Provide a QOM-based authorization API
Subsystems (2)
cover Provide a QOM-based authorization API
Many years ago I was responsible for adding the 'qemu_acl' type and associated HMP commands. Looking back at it now, it is quite a poor facility with a couple of bad limitations. First, the responsibility for creating the ACLs was left with the QEMU network service (VNC server was only thing ever doing it). This meant you could not share ACLs across multiple services. Second, there was no way to populate ACLs on the command line, you had no choice but to use the HMP commands. Third, the API was hardcoded around the idea of an in-QEMU implementation, leaving no scope for plugging in alternative implementations backed by, for exmaple, LDAP or PAM.
This series introduces a much better authorization API design to QEMU that addresses all these problems, and maintains back compatibility. It of course is based on the QOM framework, so that immediately gives us ability to create objects via the CLI, HMP or QMP. There is an abstract base clss "QAuthZ" which defines the basic API for QEMU network services to use, and a specific implementation "QAuthZ" simple which replicates the functionality of 'qemu_acl'. It is thus possible to add other impls, without changing any other part of QEMU in the future. Finally, the user is responsible for creating the ACL objects, so they can have one ACL associated with all their TLS enabled network services.
There was only one small problem with this, specifically the -object CLI arg and HMP 'object_add' command had no way to let the user specify non-scalar properties for objects. eg if an object had a property which is a list of structs, you are out of luck if you want to create it without using QMP.
Thus the first three patches do some work around QAPI / QOM to make it possible to specify non-scalar properties with the -object CLI arg and HMP 'object_add' command. See the respective patches for illustration of the syntax used.
The patches 4 and 5 introduce the new base class and specific implementation.
Patch 6 kills the old qemu_acl code, updating any existing callers of it to use the QAuthZSimple QOM class instead.
Patches 7-10 add support for associating ACLs with the network services supporting TLS encryption (NBD, chardev and VNC).
Aside from the outstanding migration TLS patches, this series wraps up the feature based work I have for TLS in this release cycle.
Daniel P. Berrange (10):
qdict: implement a qdict_crumple method for un-flattening a dict
qapi: allow QmpInputVisitor to auto-cast types
qom: support arbitrary non-scalar properties with -object
util: add QAuthZ object as an authorization base class
util: add QAuthZSimple object type for a simple access control list
acl: delete existing ACL implementation
qemu-nbd: add support for ACLs for TLS clients
nbd: allow an ACL to be set with nbd-server-start QMP command
chardev: add support for ACLs for TLS clients
vnc: allow specifying a custom ACL object name
MAINTAINERS | 7 + Makefile | 9 +- Makefile.objs | 2 + Makefile.target | 2 + blockdev-nbd.c | 10 +- crypto/tlssession.c | 28 +++- hmp.c | 20 +-- include/qapi/qmp-input-visitor.h | 3 + include/qapi/qmp/qdict.h | 1 + include/qemu/acl.h | 74 --------- include/qemu/authz-simple.h | 107 +++++++++++++ include/qemu/authz.h | 81 ++++++++++ monitor.c | 161 ++++++++++++-------- qapi-schema.json | 8 +- qapi/block.json | 4 +- qapi/qmp-input-visitor.c | 96 ++++++++++-- qapi/util.json | 31 ++++ qemu-char.c | 16 +- qemu-nbd.c | 13 +- qemu-nbd.texi | 4 + qmp-commands.hx | 2 +- qobject/qdict.c | 180 ++++++++++++++++++++++ qom/object_interfaces.c | 20 ++- tests/.gitignore | 1 + tests/Makefile | 5 +- tests/check-qdict.c | 39 +++++ tests/check-qom-proplist.c | 316 ++++++++++++++++++++++++++++++++++++++- tests/test-authz-simple.c | 156 +++++++++++++++++++ tests/test-crypto-tlssession.c | 13 +- tests/test-io-channel-tls.c | 14 +- tests/test-qmp-input-visitor.c | 115 +++++++++++++- ui/vnc-auth-sasl.c | 2 +- ui/vnc-auth-sasl.h | 4 +- ui/vnc.c | 76 ++++++++-- util/Makefile.objs | 4 +- util/acl.c | 188 ----------------------- util/authz-simple.c | 255 +++++++++++++++++++++++++++++++ util/authz.c | 45 ++++++ 38 files changed, 1709 insertions(+), 403 deletions(-) delete mode 100644 include/qemu/acl.h create mode 100644 include/qemu/authz-simple.h create mode 100644 include/qemu/authz.h create mode 100644 qapi/util.json create mode 100644 tests/test-authz-simple.c delete mode 100644 util/acl.c create mode 100644 util/authz-simple.c create mode 100644 util/authz.c
Signature · 1 line
2.5.0
PATCH 1/10 qdict: implement a qdict_crumple method for un-flattening a dict
The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.
The qdict_crumple() method aims todo the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.
If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{
'foo.0.bar': 'one',
'foo.0.wizz': '1',
'foo.1.bar': 'two',
'foo.1.wizz': '2'
}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{
'foo' => [
{ 'bar': 'one', 'wizz': '1' }
{ 'bar': 'two', 'wizz': '2' } ],
}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{
'foo..bar': 'wizz',
'bar.foo..bar': 'eek',
'bar.hello': 'world'
}Will end up as
{
'foo.bar': 'wizz',
'bar': {
'foo.bar': 'eek',
'hello': 'world'
}
}The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nested used when the same object is defined over QMP.
--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 180 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 39 ++++++++++ 3 files changed, 220 insertions(+)
include/qapi/qmp/qdict.h +1 -0
@@ -75,6 +75,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (QDict *src,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);
qobject/qdict.c +180 -0
192 lines not on this page · raw message
tests/check-qdict.c +39 -0
@@ -596,6 +596,43 @@ static void qdict_join_test(void)QDECREF (dict2); } + +static void qdict_crumple_test (void ) +{ +QDict *src, *dst, *rule, *eek; +QList *rules; + + src =qdict_new (); +qdict_put (src,"rule.0.match" ,qstring_from_str ("fred" )); +qdict_put (src,"rule.0.policy" ,qstring_from_str ("allow" )); +qdict_put (src,"rule.1.match" ,qstring_from_str ("bob" )); +qdict_put (src,"rule.1.policy" ,qstring_from_str ("deny" )); +qdict_put (src,"foo..bar" ,qstring_from_str ("wibble" )); +qdict_put (src,"eek.foo..bar" ,qstring_from_str ("wizz" )); + + dst = (QDict *)qdict_crumple (src, &error_abort); + + g_assert_cmpint(qdict_size(dst), ==, 3); + + rules =qdict_get_qlist (dst,"rule" ); + + g_assert_cmpint(qlist_size(rules), ==, 2); + + rule =qobject_to_qdict (qlist_pop (rules)); + g_assert_cmpstr("fred", ==, qdict_get_str(rule, "match")); + g_assert_cmpstr("allow", ==, qdict_get_str(rule, "policy")); + + rule =qobject_to_qdict (qlist_pop (rules)); + g_assert_cmpstr("bob", ==, qdict_get_str(rule, "match")); + g_assert_cmpstr("deny", ==, qdict_get_str(rule, "policy")); + + g_assert_cmpstr("wibble", ==, qdict_get_str(dst, "foo.bar")); + + eek =qdict_get_qdict (dst,"eek" ); + g_assert_cmpstr("wizz", ==, qdict_get_str(eek, "foo.bar")); +} + +/* * Errors test-cases */
@@ -743,6 +780,8 @@ int main(int argc, char **argv)g_test_add_func ("/errors/put_exists" , qdict_put_exists_test);g_test_add_func ("/errors/get_not_exists" , qdict_get_not_exists_test); +g_test_add_func ("/public/crumple" , qdict_crumple_test); +/* The Big one */ if (g_test_slow ()) {g_test_add_func ("/stress/test" , qdict_stress_test);
Signature · 1 line
2.5.0
Re: [Qemu-devel] [PATCH v1 01/10] qdict: implement a qdict_crumple method for un-flattening a dict
Quoted 5 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims todo the reverse, taking a flat
s/todo/to do/
Quoted 6 lines
qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.
Interesting idea. I haven't closely reviewed the patch yet, but do have a comment on the commit message:
Quoted 3 lines
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
Hmm. We forbid the use of '.' inside QAPI key names in most places, but have an exception that downstream extensions use '.'. So, I guess the only place you'd really use this is trying to target a downstream-extension key name:
Quoted 4 lines
{
'foo..bar': 'wizz',
'bar.foo..bar': 'eek',as in '__com..example_bar': 'eek' to set the downstream key-name of '__com.example_bar'. I'm not sure the special casing of '.' is worth it, particularly since qdict_flatten() doesn't have it in the reverse direction.
Quoted 6 lines
The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nested used when the same object is defined over QMP.
Signed-off-by Daniel P. Berrange <[email protected]>---
I hope to offer a more detailed review next week (I'm trying to get some of my own patches polished today).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v1 01/10] qdict: implement a qdict_crumple method for un-flattening a dict
Quoted 20 lines · Eric Blake
Quoted 5 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims todo the reverse, taking a flats/todo/to do/
Quoted 6 lines
qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.
Interesting idea. I haven't closely reviewed the patch yet, but do have a comment on the commit message:
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
Hmm. We forbid the use of '.' inside QAPI key names in most places, but have an exception that downstream extensions use '.'. So, I guess the only place you'd really use this is trying to target a downstream-extension key name:
I originally put code into object_property_add() to forbid use of a '.' in a property name, but quickly found we have alot of such usage already :-( On a x86_64 default machine I get:
Property sse4.1 on class qemu64-x86_64-cpu
Property sse4.2 on class qemu64-x86_64-cpu
Property pc.ram[*] on class container
Property pc.ram[0] on class container
Property pc.bios[*] on class container
Property pc.bios[0] on class container
Property pc.rom[*] on class container
Property pc.rom[0] on class container
Property fwcfg.dma[*] on class fw_cfg_io
Property fwcfg.dma[0] on class fw_cfg_io
Property pci.0 on class i440FX-pcihost
Property isa.0 on class PIIX3
Property vga.vram[*] on class VGA
Property vga.vram[0] on class VGA
Property vga.mmio[*] on class container
Property vga.mmio[0] on class container
Property vga.rom[*] on class VGA
Property vga.rom[0] on class VGA
Property e1000.rom[*] on class e1000
Property e1000.rom[0] on class e1000
Property ide.0 on class piix3-ide
Property ide.1 on class piix3-ide
Now none of those are implementing the UserCreatable interface right now, so wouldn't be a problem for -object usage, but I felt if we allow '.' in property names for devices, we ought to make sure the code deals with it everything.
So it seems forbiding '.' in property names won't fly :-(
Regards,
Daniel
Signature · 4 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH v1 01/10] qdict: implement a qdict_crumple method for un-flattening a dict
In reply to Daniel P. BerrangeQuoted 82 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims todo the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{ 'foo.0.bar': 'one', 'foo.0.wizz': '1', 'foo.1.bar': 'two', 'foo.1.wizz': '2'}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{ 'foo' => [ { 'bar': 'one', 'wizz': '1' } { 'bar': 'two', 'wizz': '2' } ],}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{ 'foo..bar': 'wizz', 'bar.foo..bar': 'eek', 'bar.hello': 'world' }Will end up as
{ 'foo.bar': 'wizz', 'bar': { 'foo.bar': 'eek', 'hello': 'world' } }The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nested used when the same object is defined over QMP.
Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 180 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 39 ++++++++++ 3 files changed, 220 insertions(+)include/qapi/qmp/qdict.h +1
-0@@ -75,6 +75,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (QDict *src,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);qobject/qdict.c +0
-0
qobject/qdict.c void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start) · line 614
Quoted 7 lines
qobject/qdict.c +1
-0@@ -608,6 +608,7 @@ void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start) } } +static int qdict_count_prefixed_entries (const QDict *src,const char *start) {const QDictEntry *entry;
This hunk seems unintended.
Quoted 25 lines
qobject/qdict.c +21
-0@@ -682,6 +683,185 @@ void qdict_array_split(QDict *src, QList **dst) } } + +/** +* qdict_crumple: +* +* Reverses the flattening done by qdict_flatten by +* crumpling the dicts into a nested structure. Similar +* qdict_array_split, but copes with arbitrary nesting +* of dicts & arrays, not meely one level of arrays +* +* { 'foo.0.bar': 'one', 'foo.0.wizz': '1', +* 'foo.1.bar': 'two', 'foo.1.wizz': '2' } +* +* => +* +* { +* 'foo' => [ +* { 'bar': 'one', 'wizz': '1' } +* { 'bar': 'two', 'wizz': '2' } +* ], +* } +*
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 716
Quoted 10 lines
qobject/qdict.c +10
-0+*/ +QObject *qdict_crumple (QDict *src,Error **errp) +{ +const QDictEntry *entry, *next; +const char *p =NULL ; +QDict *tmp1 =NULL , *tmp2 =NULL ; +QObject *dst =NULL , *child; +bool isList = false; +ssize_t listMax =-1 ; +size_t listLen =0 ;
As far as I'm aware we don't have any strict policy on names, but in general structs use UpperCamelCase and variables and functions use c_style_naming.
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 721
Quoted 5 lines
qobject/qdict.c +5
-0+size_t i, j; +int64_t val; +char *key; + + tmp1 =qdict_new ();
I guess I'd like clearer variable names.
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 731
Quoted 10 lines
qobject/qdict.c +10
-0+ entry =qdict_first (src); + +/* Step 1: extract everything as nested dicts */ +while (entry !=NULL ) { + next =qdict_next (src, entry); +qobject_incref (entry->value); + +/* Find first '.' separator, but treat '..' as +* an escape sequence */ + p =NULL ;
How about at least "dot" or "separator"?
Quoted 14 lines
qobject/qdict.c +14
-0+do { +if (p) { + p +=2 ; + }else { + p = entry->key; + } + p =strchr (p,'.' ); + }while (p && *(p +1 ) =='.' ); + +if (p) { + key =g_strndup (entry->key, + p - entry->key); + }else { + key =g_strdup (entry->key);
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 755
Quoted 10 lines
qobject/qdict.c +10
-0+ } + +for (i =0 , j =0 ; key[i] !='\0' ; i++, j++) { +if (key[i] =='.' && + key[i +1 ] =='.' ) { + i++; + } + key[j] = key[i]; + } + key[j] ='\0' ;
I general I think it would be nice to put this escaping code into an own static function which returns a strdup'd key for the QDict and this entry's key in that QDict.
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 762
Quoted 7 lines
qobject/qdict.c +7
-0+ +if (p) { + tmp2 =qdict_get_qdict (tmp1, key); + p++; +if (!tmp2) { + tmp2 =qdict_new (); +qdict_put (tmp1, key, tmp2);
This...
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 766
Quoted 4 lines
qobject/qdict.c +4
-0+ } +qdict_put_obj (tmp2, p, entry->value); + }else { +qdict_put_obj (tmp1, key, entry->value);
...and this may silently overwrite entries, e.g. when crumpling
{ "x": 42, "x.y": 23 }
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 767
Quoted 1 line
qobject/qdict.c +1
-0+ }
key is leaked.
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 773
Quoted 6 lines
qobject/qdict.c +6
-0+ + entry = next; + } + +/* Step 2: crumple the new dicts we just created */ + tmp2 =qdict_new ();
Please use more variables with clearer names.
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 779
Quoted 6 lines
qobject/qdict.c +6
-0+ entry =qdict_first (tmp1); +while (entry !=NULL ) { + next =qdict_next (tmp1, entry); + +if (qobject_type (entry->value) ==QTYPE_QDICT ) { + child =qdict_crumple ((QDict *)entry->value, errp);
The cast works, but I'd prefer qobject_to_qdict() nonetheless.
Quoted 5 lines
qobject/qdict.c +5
-0+if (!child) { + goto error; + } + +qdict_put_obj (tmp2, entry->key, child);
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 794
Quoted 10 lines
qobject/qdict.c +10
-0+ }else { +qobject_incref (entry->value); +qdict_put_obj (tmp2, entry->key, entry->value); + } + + entry = next; + } +QDECREF (tmp1); + +/* Step 3: detect if we need to turn our dict into list */
You could use qdict_array_entries(tmp2, "") > 0 for this.
If you do want to make { "0": 42, "2": 23 } and { "0": 42, "x": 23 } errors (my qdict_unflatten() just kept those QDicts), then you'd have to check on qdict_array_entries() error whether the QDict contained an integer key, but that would still be simpler than the following loop and the check in step 4.
Quoted 26 lines
qobject/qdict.c +26
-0+ entry =qdict_first (tmp2); +while (entry !=NULL ) { + next =qdict_next (tmp2, entry); + + errno =0 ; +if (qemu_strtoll (entry->key,NULL ,10 , &val) ==0 ) { +if (!dst) { + dst = (QObject *)qlist_new (); + isList = true; + }else if (!isList) { +error_setg (errp, +"Key '%s' is for a list, but previous key is " +"for a dict" , entry->key); + goto error; + } + listLen++; +if (val > listMax) { + listMax = val; + } + }else { +if (!dst) { + dst = (QObject *)tmp2; +qobject_incref (dst); + isList = false; + }else if (isList) { + error_setg(errp,
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 830
Quoted 10 lines
qobject/qdict.c +10
-0+"Key '%s' is for a dict, but previous key is " + "for a list", entry->key); + goto error; + } + } + + entry = next; + } + +/* Step 4: Turn the dict into a list */
Why not just use qdict_array_split(tmp2, &dst);?
Max
Quoted 95 lines
qobject/qdict.c +34
-0+if (isList) { +if (listLen != (listMax +1 )) { +error_setg (errp,"List indexes are not continuous, " +"saw %zu elements but %zu largest index" , + listLen, listMax); + goto error; + } + +for (i =0 ; i < listLen; i++) { +char *key =g_strdup_printf ("%zu" , i); + + child =qdict_get (tmp2, key); +g_free (key); +if (!child) { +error_setg (errp,"Unexpected missing list entry %zu" , i); + goto error; + } + +qobject_incref (child); +qlist_append_obj ((QList *)dst, child); + } + } +QDECREF (tmp2); + +return dst; + + error: +QDECREF (tmp2); +QDECREF (tmp1); +qobject_decref (dst); +return NULL ; +} + +/** * qdict_array_entries(): Returns the number of direct array entries if the * sub-QDict of src specified by the prefix in subqdict (or src itself for tests/check-qdict.c +39
-0@@ -596,6 +596,43 @@ static void qdict_join_test(void)QDECREF (dict2); } + +static void qdict_crumple_test (void ) +{ +QDict *src, *dst, *rule, *eek; +QList *rules; + + src =qdict_new (); +qdict_put (src,"rule.0.match" ,qstring_from_str ("fred" )); +qdict_put (src,"rule.0.policy" ,qstring_from_str ("allow" )); +qdict_put (src,"rule.1.match" ,qstring_from_str ("bob" )); +qdict_put (src,"rule.1.policy" ,qstring_from_str ("deny" )); +qdict_put (src,"foo..bar" ,qstring_from_str ("wibble" )); +qdict_put (src,"eek.foo..bar" ,qstring_from_str ("wizz" )); + + dst = (QDict *)qdict_crumple (src, &error_abort); + + g_assert_cmpint(qdict_size(dst), ==, 3); + + rules =qdict_get_qlist (dst,"rule" ); + + g_assert_cmpint(qlist_size(rules), ==, 2); + + rule =qobject_to_qdict (qlist_pop (rules)); + g_assert_cmpstr("fred", ==, qdict_get_str(rule, "match")); + g_assert_cmpstr("allow", ==, qdict_get_str(rule, "policy")); + + rule =qobject_to_qdict (qlist_pop (rules)); + g_assert_cmpstr("bob", ==, qdict_get_str(rule, "match")); + g_assert_cmpstr("deny", ==, qdict_get_str(rule, "policy")); + + g_assert_cmpstr("wibble", ==, qdict_get_str(dst, "foo.bar")); + + eek =qdict_get_qdict (dst,"eek" ); + g_assert_cmpstr("wizz", ==, qdict_get_str(eek, "foo.bar")); +} + +/* * Errors test-cases */ @@ -743,6 +780,8 @@ int main(int argc, char **argv)g_test_add_func ("/errors/put_exists" , qdict_put_exists_test);g_test_add_func ("/errors/get_not_exists" , qdict_get_not_exists_test); +g_test_add_func ("/public/crumple" , qdict_crumple_test); +/* The Big one */ if (g_test_slow ()) {g_test_add_func ("/stress/test" , qdict_stress_test);
Re: [Qemu-devel] [PATCH v1 01/10] qdict: implement a qdict_crumple method for un-flattening a dict
Quoted 67 lines · Max Reitz
Quoted 65 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims todo the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{ 'foo.0.bar': 'one', 'foo.0.wizz': '1', 'foo.1.bar': 'two', 'foo.1.wizz': '2'}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{ 'foo' => [ { 'bar': 'one', 'wizz': '1' } { 'bar': 'two', 'wizz': '2' } ],}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{ 'foo..bar': 'wizz', 'bar.foo..bar': 'eek', 'bar.hello': 'world' }Will end up as
{ 'foo.bar': 'wizz', 'bar': { 'foo.bar': 'eek', 'hello': 'world' } }The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nested used when the same object is defined over QMP.
Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 180 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 39 ++++++++++ 3 files changed, 220 insertions(+)
Quoted 13 lines
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 716
Quoted 9 lines
qobject/qdict.c +9
-0+QObject *qdict_crumple (QDict *src,Error **errp) +{ +const QDictEntry *entry, *next; +const char *p =NULL ; +QDict *tmp1 =NULL , *tmp2 =NULL ; +QObject *dst =NULL , *child; +bool isList = false; +ssize_t listMax =-1 ; +size_t listLen =0 ;As far as I'm aware we don't have any strict policy on names, but in general structs use UpperCamelCase and variables and functions use
c_style_naming.
Yep, ok
Quoted 7 lines
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 721
Quoted 5 lines
qobject/qdict.c +5
-0+size_t i, j; +int64_t val; +char *key; + + tmp1 =qdict_new ();I guess I'd like clearer variable names.
Sure
Quoted 13 lines
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 731
Quoted 11 lines
qobject/qdict.c +10
-0+ entry =qdict_first (src); + +/* Step 1: extract everything as nested dicts */ +while (entry !=NULL ) { + next =qdict_next (src, entry); +qobject_incref (entry->value); + +/* Find first '.' separator, but treat '..' as +* an escape sequence */ + p =NULL ;How about at least "dot" or "separator"?
Sure.
Quoted 20 lines
Quoted 20 lines
qobject/qdict.c +20
-0+do { +if (p) { + p +=2 ; + }else { + p = entry->key; + } + p =strchr (p,'.' ); + }while (p && *(p +1 ) =='.' ); + +if (p) { + key =g_strndup (entry->key, + p - entry->key); + }else { + key =g_strdup (entry->key); + } + +for (i =0 , j =0 ; key[i] !='\0' ; i++, j++) { +if (key[i] =='.' && + key[i +1 ] =='.' ) { + i++;
Quoted 8 lines
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 755
Quoted 4 lines
qobject/qdict.c +4
-0+ } + key[j] = key[i]; + } + key[j] ='\0' ;I general I think it would be nice to put this escaping code into an own static function which returns a strdup'd key for the QDict and this entry's key in that QDict.
Yep, good idea.
Quoted 16 lines
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 762
Quoted 8 lines
qobject/qdict.c +7
-0+ +if (p) { + tmp2 =qdict_get_qdict (tmp1, key); + p++; +if (!tmp2) { + tmp2 =qdict_new (); +qdict_put (tmp1, key, tmp2);This...
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 766
Quoted 4 lines
qobject/qdict.c +4
-0+ } +qdict_put_obj (tmp2, p, entry->value); + }else { +qdict_put_obj (tmp1, key, entry->value);
Quoted 7 lines
...and this may silently overwrite entries, e.g. when crumpling
{ "x": 42, "x.y": 23 }
Quoted 1 line
+ }
key is leaked.
Oh that should result in an error being raised as it is a contradictory input.
Quoted 9 lines
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 773
Quoted 6 lines
qobject/qdict.c +6
-0+ + entry = next; + } + +/* Step 2: crumple the new dicts we just created */ + tmp2 =qdict_new ();Please use more variables with clearer names.
Quoted 8 lines
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 779
Quoted 6 lines
qobject/qdict.c +6
-0+ entry =qdict_first (tmp1); +while (entry !=NULL ) { + next =qdict_next (tmp1, entry); + +if (qobject_type (entry->value) ==QTYPE_QDICT ) { + child =qdict_crumple ((QDict *)entry->value, errp);The cast works, but I'd prefer
qobject_to_qdict()nonetheless.
Ok
Quoted 19 lines
Quoted 6 lines
qobject/qdict.c +5
-0+if (!child) { + goto error; + } + +qdict_put_obj (tmp2, entry->key, child);qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 794
Quoted 10 lines
qobject/qdict.c +10
-0+ }else { +qobject_incref (entry->value); +qdict_put_obj (tmp2, entry->key, entry->value); + } + + entry = next; + } +QDECREF (tmp1); + +/* Step 3: detect if we need to turn our dict into list */ You could use
qdict_array_entries(tmp2, "")> 0 for this.
Quoted 5 lines
If you do want to make { "0": 42, "2": 23 } and { "0": 42, "x": 23 } errors (my
qdict_unflatten()just kept those QDicts), then you'd have to check onqdict_array_entries()error whether the QDict contained an integer key, but that would still be simpler than the following loop and the check in step 4.
If I use qdict_array_entries() then it does 2 O(N) iterations of the input qdict, and to check the errors, I'll have todo yet another iteration. My inlined code here does everything in a single O(N) iteration instead of 3. I think that's compelling enough to reason to not use that function.
Quoted 30 lines
Quoted 30 lines
qobject/qdict.c +30
-0+ entry =qdict_first (tmp2); +while (entry !=NULL ) { + next =qdict_next (tmp2, entry); + + errno =0 ; +if (qemu_strtoll (entry->key,NULL ,10 , &val) ==0 ) { +if (!dst) { + dst = (QObject *)qlist_new (); + isList = true; + }else if (!isList) { +error_setg (errp, +"Key '%s' is for a list, but previous key is " +"for a dict" , entry->key); + goto error; + } + listLen++; +if (val > listMax) { + listMax = val; + } + }else { +if (!dst) { + dst = (QObject *)tmp2; +qobject_incref (dst); + isList = false; + }else if (isList) { +error_setg (errp, +"Key '%s' is for a dict, but previous key is " +"for a list" , entry->key); + goto error; + }
Quoted 8 lines
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 830
Quoted 6 lines
qobject/qdict.c +6
-0+ } + + entry = next; + } + +/* Step 4: Turn the dict into a list */ Why not just use
qdict_array_split(tmp2, &dst);?
Again qdict_array_split() is a pretty inefficiently written function. It does multiple nested iterations over its input giving it O(n^2) time, compared to O(n) for my code here.
The only user of qdict_array_entries() and qdict_array_split() is the block quorum driver. From a brief look at that code I think it could probably be changed to just call this new qdict_crumple() method, and those 2 inefficient functions could be deleted, so we don't have duplication of code.
Quoted 26 lines
Quoted 26 lines
qobject/qdict.c +26
-0+if (isList) { +if (listLen != (listMax +1 )) { +error_setg (errp,"List indexes are not continuous, " +"saw %zu elements but %zu largest index" , + listLen, listMax); + goto error; + } + +for (i =0 ; i < listLen; i++) { +char *key =g_strdup_printf ("%zu" , i); + + child =qdict_get (tmp2, key); +g_free (key); +if (!child) { +error_setg (errp,"Unexpected missing list entry %zu" , i); + goto error; + } + +qobject_incref (child); +qlist_append_obj ((QList *)dst, child); + } + } +QDECREF (tmp2); + +return dst; +
Quoted 8 lines
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 864
Quoted 8 lines
+ error:
qobject/qdict.c +7
-0+QDECREF (tmp2); +QDECREF (tmp1); +qobject_decref (dst); +return NULL ; +} + +
Regards,
Daniel
Signature · 4 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH v1 01/10] qdict: implement a qdict_crumple method for un-flattening a dict
Quoted 16 lines · Daniel P. Berrange
Quoted 15 lines · Max Reitz
Quoted 14 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims todo the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
[...]
Quoted 24 lines
Quoted 18 lines
Quoted 5 lines
qobject/qdict.c +5
-0+if (!child) { + goto error; + } + +qdict_put_obj (tmp2, entry->key, child);qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 794
Quoted 10 lines
qobject/qdict.c +10
-0+ }else { +qobject_incref (entry->value); +qdict_put_obj (tmp2, entry->key, entry->value); + } + + entry = next; + } +QDECREF (tmp1); + +/* Step 3: detect if we need to turn our dict into list */ You could use
qdict_array_entries(tmp2, "")> 0 for this.Quoted 5 lines
If you do want to make { "0": 42, "2": 23 } and { "0": 42, "x": 23 } errors (my
qdict_unflatten()just kept those QDicts), then you'd have to check onqdict_array_entries()error whether the QDict contained an integer key, but that would still be simpler than the following loop and the check in step 4.
Quoted 5 lines
If I use
qdict_array_entries()then it does 2 O(N) iterations of the input qdict, and to check the errors, I'll have todo yet another iteration. My inlined code here does everything in a single O(N) iteration instead of 3. I think that's compelling enough to reason to not use that function.
O(3N) = O(N) :-)
Making qdict_array_entries() O(N) (pretending that O(N) and O(2N) were different things) for this case is trivial: Just omit the second loop if "" has been passed as the subqdict.
So then you'd end up with "O(2N)", if you do the error checking.
So you are right, there is a reason not to use qdict_array_entries(), but I'd personally still use it. I only have very limited knowledge of the whole qemu code base, but it doesn't appear to me as if QDict functions are ever used in a hot path or as if QDicts ever contain a large number of elements (with large being more than, say, 1000), especially not the kind of QDicts you'd want to crumple.
Because of that, I chose to use these existing functions, even while knowing that they are probably not optimal for this case.
You did propose removing qdict_array_entries() and qdict_array_split() in favor of just using this function in their stead (see my reply below), so if we do so, reusing them would obviously not be ideal any more. In that case, I'd still put this into its own static function if possible.
tl;dr: I don't think runtime complexity is an issue here, but if we are going to drop qdict_array_entries() and qdict_array_split() anyway, then using them is a bad idea, obviously. It would then still be nice to put this into an own function.
Quoted 39 lines
Quoted 30 lines
Quoted 30 lines
qobject/qdict.c +30
-0+ entry =qdict_first (tmp2); +while (entry !=NULL ) { + next =qdict_next (tmp2, entry); + + errno =0 ; +if (qemu_strtoll (entry->key,NULL ,10 , &val) ==0 ) { +if (!dst) { + dst = (QObject *)qlist_new (); + isList = true; + }else if (!isList) { +error_setg (errp, +"Key '%s' is for a list, but previous key is " +"for a dict" , entry->key); + goto error; + } + listLen++; +if (val > listMax) { + listMax = val; + } + }else { +if (!dst) { + dst = (QObject *)tmp2; +qobject_incref (dst); + isList = false; + }else if (isList) { +error_setg (errp, +"Key '%s' is for a dict, but previous key is " +"for a list" , entry->key); + goto error; + }Quoted 8 lines
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 830
Quoted 6 lines
qobject/qdict.c +6
-0+ } + + entry = next; + } + +/* Step 4: Turn the dict into a list */ Why not just use
qdict_array_split(tmp2, &dst);?
Quoted 3 lines
Again
qdict_array_split()is a pretty inefficiently written function. It does multiple nested iterations over its input giving it O(n^2) time, compared to O(n) for my code here.
Strictly speaking, it's not O(n^2) but O(nm), where n is qdict_size(src) and m is the size of the resulting QList.
(Side note: This function only is O(n) if qdict_get() is O(1), which it is in best case. In worst case, it's O(n), and then this code becomes O(n^2).)
Again, I don't think that O(nm) is much worse than O(n) for the use case at hand, but if you are going to drop qdict_array_split() anyway, then, again, it doesn't make sense to call it at all.
You could again put the code inside of the "if (isList) {}" block into an own function, but it's not that long so I'd be fine with it staying here (although it is certainly self-contained enough to look good in an own function :-)).
Quoted 5 lines
The only user of
qdict_array_entries()andqdict_array_split()is the block quorum driver. From a brief look at that code I think it could probably be changed to just call this newqdict_crumple()method, and those 2 inefficient functions could be deleted, so we don't have duplication of code.
I'm not sure. First, you do not want to crumple recursively there, so you'd have to re-flatten all the QList elements after you have crumpled them. Could be solved by adding a "bool recurse" parameter to this function.
Second, yes, qdict_array_entries() is used by quorum. However, it does (no longer) use qdict_array_split(); the users of that are util/qemu-config.c and blockdev.c. Replacing those with a non-recursing call to qdict_crumple() should be trivial, but quorum is going to be a bit more difficult. You could make it just call qdict_crumple(), get the size of the resulting list and then discard it, but that seems a bit over the top.
All in all, I think you're right in that this function can replace the (potentially) worse qdict_array_split() function (if the caller can stop it from recursing), but I don't know whether we can get rid of qdict_array_entries() so easily.
Max
Quoted 38 lines
Quoted 26 lines
Quoted 26 lines
qobject/qdict.c +26
-0+if (isList) { +if (listLen != (listMax +1 )) { +error_setg (errp,"List indexes are not continuous, " +"saw %zu elements but %zu largest index" , + listLen, listMax); + goto error; + } + +for (i =0 ; i < listLen; i++) { +char *key =g_strdup_printf ("%zu" , i); + + child =qdict_get (tmp2, key); +g_free (key); +if (!child) { +error_setg (errp,"Unexpected missing list entry %zu" , i); + goto error; + } + +qobject_incref (child); +qlist_append_obj ((QList *)dst, child); + } + } +QDECREF (tmp2); + +return dst; +Quoted 8 lines
qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 864
Quoted 8 lines
+ error:
qobject/qdict.c +7
-0+QDECREF (tmp2); +QDECREF (tmp1); +qobject_decref (dst); +return NULL ; +} + +Regards,
Daniel
Re: [Qemu-devel] [PATCH v1 01/10] qdict: implement a qdict_crumple method for un-flattening a dict
Quoted 74 lines · Max Reitz
Quoted 16 lines · Daniel P. Berrange
Quoted 15 lines · Max Reitz
Quoted 14 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims todo the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
[...]
Quoted 24 lines
Quoted 18 lines
Quoted 5 lines
qobject/qdict.c +5
-0+if (!child) { + goto error; + } + +qdict_put_obj (tmp2, entry->key, child);qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 794
Quoted 10 lines
qobject/qdict.c +10
-0+ }else { +qobject_incref (entry->value); +qdict_put_obj (tmp2, entry->key, entry->value); + } + + entry = next; + } +QDECREF (tmp1); + +/* Step 3: detect if we need to turn our dict into list */ You could use
qdict_array_entries(tmp2, "")> 0 for this.Quoted 5 lines
If you do want to make { "0": 42, "2": 23 } and { "0": 42, "x": 23 } errors (my
qdict_unflatten()just kept those QDicts), then you'd have to check onqdict_array_entries()error whether the QDict contained an integer key, but that would still be simpler than the following loop and the check in step 4.Quoted 5 lines
If I use
qdict_array_entries()then it does 2 O(N) iterations of the input qdict, and to check the errors, I'll have todo yet another iteration. My inlined code here does everything in a single O(N) iteration instead of 3. I think that's compelling enough to reason to not use that function.O(3N) = O(N) :-)
Making
qdict_array_entries()O(N) (pretending that O(N) and O(2N) were different things) for this case is trivial: Just omit the second loop if "" has been passed as the subqdict.So then you'd end up with "O(2N)", if you do the error checking.
So you are right, there is a reason not to use
qdict_array_entries(), but I'd personally still use it. I only have very limited knowledge of the whole qemu code base, but it doesn't appear to me as if QDict functions are ever used in a hot path or as if QDicts ever contain a large number of elements (with large being more than, say, 1000), especially not the kind of QDicts you'd want to crumple.Because of that, I chose to use these existing functions, even while knowing that they are probably not optimal for this case.
You did propose removing
qdict_array_entries()andqdict_array_split()in favor of just using this function in their stead (see my reply below), so if we do so, reusing them would obviously not be ideal any more. In that case, I'd still put this into its own static function if possible.
Quoted 4 lines
tl;dr: I don't think runtime complexity is an issue here, but if we are going to drop
qdict_array_entries()andqdict_array_split()anyway, then using them is a bad idea, obviously. It would then still be nice to put this into an own function.
So looking at the current usage of qdict_flatten, qdict_extract_subqdict, qdict_array_split and qdict_array_entries, it is basically only the block layer.
The root cause of this usage all stems from that fact that historically the block layer was driven off QemuOpts CLI args which are a flat data structure. Over time we've tried to bolt on recursive data structure support, but the main APIs are still accepting QDicts whose contents are flat. Increasingly I see the internal code is based on QAPI which is an inherantly nested data structure, as a result the block layer is getting more & more places where it has to process the flat data structure to extract nested bits (qdict_extract_subqdict, qdict_array_entries and qdict_array_split). Conversely when fed data coming from QMP it has to flatten the data structure with qdict_flatten.
With this new qdict_crumple() method (or equivalently you qdict_unflatten) it strikes me we can significantly simplify the block layer to always use a nested QDict internally. All that would be required is to call the qdict_crumple() method in the places which have the flat QemuOpts derived QDict. At that point we would potentially be able to delete all of qdict_flatten, qdict_extract_subqdict, qdict_array_split and qdict_array_entries
Of course I'm not suggesting we do that for 2.6, but it actually doesn't look like it would be that hard todo this conversion.
Quoted 18 lines
Quoted 5 lines
The only user of
qdict_array_entries()andqdict_array_split()is the block quorum driver. From a brief look at that code I think it could probably be changed to just call this newqdict_crumple()method, and those 2 inefficient functions could be deleted, so we don't have duplication of code.I'm not sure. First, you do not want to crumple recursively there, so you'd have to re-flatten all the QList elements after you have crumpled them. Could be solved by adding a "bool recurse" parameter to this function.
Second, yes,
qdict_array_entries()is used by quorum. However, it does (no longer) useqdict_array_split(); the users of that areutil/qemu-config.candblockdev.c. Replacing those with a non-recursing call toqdict_crumple()should be trivial, but quorum is going to be a bit more difficult. You could make it just callqdict_crumple(), get the size of the resulting list and then discard it, but that seems a bit over the top.
Quoted 4 lines
All in all, I think you're right in that this function can replace the (potentially) worse
qdict_array_split()function (if the caller can stop it from recursing), but I don't know whether we can get rid ofqdict_array_entries()so easily.
Yeah, that would require the big block layer conversion to always use a nested QDict and never use a flat QDict.
Regards,
Daniel
Signature · 4 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH v1 01/10] qdict: implement a qdict_crumple method for un-flattening a dict
Quoted 23 lines · Daniel P. Berrange
So looking at the current usage of
qdict_flatten,qdict_extract_subqdict,qdict_array_splitandqdict_array_entries, it is basically only the block layer.The root cause of this usage all stems from that fact that historically the block layer was driven off QemuOpts CLI args which are a flat data structure. Over time we've tried to bolt on recursive data structure support, but the main APIs are still accepting QDicts whose contents are flat. Increasingly I see the internal code is based on QAPI which is an inherantly nested data structure, as a result the block layer is getting more & more places where it has to process the flat data structure to extract nested bits (
qdict_extract_subqdict,qdict_array_entriesandqdict_array_split). Conversely when fed data coming from QMP it has to flatten the data structure withqdict_flatten.With this new
qdict_crumple()method (or equivalently youqdict_unflatten) it strikes me we can significantly simplify the block layer to always use a nested QDict internally. All that would be required is to call theqdict_crumple()method in the places which have the flat QemuOpts derived QDict. At that point we would potentially be able to delete all ofqdict_flatten,qdict_extract_subqdict,qdict_array_splitandqdict_array_entries
Quoted 2 lines
Of course I'm not suggesting we do that for 2.6, but it actually doesn't look like it would be that hard todo this conversion.
Agreed that it's too late for 2.6, but would make a good project for 2.7. For that matter, rather than passing a QDict around, I'd like to see if we could just directly use the QAPI types instead (the way you just recently converted from QDict to SocketAddress).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 2/10 qapi: allow QmpInputVisitor to auto-cast types
In reply to Daniel P. BerrangeCurrently the QmpInputVisitor assumes that all scalar values are directly represented as their final types. ie it assumes an 'int' is using QInt, and a 'bool' is using QBool.
This extends it so that QString is optionally permitted for any of the non-string scalar types. This behaviour is turned on by requesting the 'autocast' flag in the constructor.
This makes it possible to use QmpInputVisitor with a QDict produced from QemuOpts, where everything is in string format.
--- include/qapi/qmp-input-visitor.h | 3 + qapi/qmp-input-visitor.c | 96 +++++++++++++++++++++++++++----- tests/test-qmp-input-visitor.c | 115 ++++++++++++++++++++++++++++++++++++++- 3 files changed, 196 insertions(+), 18 deletions(-)
include/qapi/qmp-input-visitor.h +3 -0
@@ -21,6 +21,9 @@ typedef struct QmpInputVisitor QmpInputVisitor;QmpInputVisitor *qmp_input_visitor_new (QObject *obj);QmpInputVisitor *qmp_input_visitor_new_strict (QObject *obj); +QmpInputVisitor *qmp_input_visitor_new_full (QObject *obj, +bool strict, +bool autocast);void qmp_input_visitor_cleanup (QmpInputVisitor *v);
qapi/qmp-input-visitor.c +81 -15
147 lines not on this page · raw message
tests/test-qmp-input-visitor.c +112 -3
173 lines not on this page · raw message
Signature · 1 line
2.5.0
PATCH 3/10 qom: support arbitrary non-scalar properties with -object
In reply to Daniel P. BerrangeThe current -object command line syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.
This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.
It is not practical to extend the OptsVisitor to support more complex data structures while also maintaining the existing list handling behaviour that is relied upon by other areas of QEMU.
Fortunately there is no existing object that implements the UserCreatable interface that relies on the list handling behaviour, so it is possible to swap out the OptsVisitor for a different visitor implementation, so -object supports non-scalar properties, thus leaving other users of OptsVisitor unaffected.
The previously added qdict_crumple() method is able to take a qdict containing a flat set of properties and turn that into a arbitrarily nested set of dicts and lists. By combining qemu_opts_to_qdict and qdict_crumple() together, we can turn the opt string into a data structure that is practically identical to that passed over QMP when defining an object. The only difference is that all the scalar values are represented as strings, rather than strings, ints and bools. This is sufficient to let us replace the OptsVisitor with the QMPInputVisitor for use with -object.
Thus -object can now support non-scalar properties, for example the QMP object
{
"execute": "object-add",
"arguments": {
"qom-type": "demo",
"id": "demo0",
"parameters": {
"foo": [
{ "bar": "one", "wizz": "1" },
{ "bar": "two", "wizz": "2" }
]
}
}}
Would be creatable via the CLI now using
$QEMU \
-object demo,id=demo0,\
foo.0.bar=one,foo.0.wizz=1,\
foo.1.bar=two,foo.1.wizz=2This is also wired up to work for the 'object_add' command in the HMP monitor with the same syntax.
(hmp) object_add demo,id=demo0,\
foo.0.bar=one,foo.0.wizz=1,\
foo.1.bar=two,foo.1.wizz=2--- hmp.c | 18 +-- qom/object_interfaces.c | 20 ++- tests/check-qom-proplist.c | 316 ++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 334 insertions(+), 20 deletions(-)
hmp.c +5 -13
@@ -25,7 +25,7 @@#include "qemu/sockets.h" #include "monitor/monitor.h" #include "monitor/qdev.h" -#include "qapi/opts-visitor.h" +#include "qapi/qmp-input-visitor.h" #include "qapi/qmp/qerror.h" #include "qapi/string-output-visitor.h" #include "qapi/util.h"
@@ -1656,20 +1656,12 @@ void hmp_netdev_del(Monitor *mon, const QDict *qdict)void hmp_object_add (Monitor *mon,const QDict *qdict) {Error *err =NULL ; -QemuOpts *opts; -OptsVisitor *ov; +QmpInputVisitor *qiv;Object *obj =NULL ; - opts =qemu_opts_from_qdict (qemu_find_opts ("object" ), qdict, &err); -if (err) { -hmp_handle_error (mon, &err); -return ; - } - - ov =opts_visitor_new (opts); - obj =user_creatable_add (qdict,opts_get_visitor (ov), &err); -opts_visitor_cleanup (ov); -qemu_opts_del (opts); + qiv =qmp_input_visitor_new_full ((QObject *)qdict, true, true); + obj =user_creatable_add (qdict,qmp_input_get_visitor (qiv), &err); +qmp_input_visitor_cleanup (qiv);if (err) {hmp_handle_error (mon, &err);
qom/object_interfaces.c +14 -6
@@ -1,9 +1,9 @@#include "qemu/osdep.h" #include "qom/object_interfaces.h" #include "qemu/module.h" +#include "qemu/option.h" #include "qapi-visit.h" -#include "qapi/qmp-output-visitor.h" -#include "qapi/opts-visitor.h" +#include "qapi/qmp-input-visitor.h" void user_creatable_complete (Object *obj,Error **errp) {
@@ -120,6 +120,7 @@ Object *user_creatable_add_type(const char *type, const char *id, obj =object_new (type);if (qdict) {for (e =qdict_first (qdict); e; e =qdict_next (qdict, e)) { +object_property_set (obj, v, e->key, &local_err);if (local_err) { goto out;
@@ -151,15 +152,22 @@ out:Object *user_creatable_add_opts (QemuOpts *opts,Error **errp) { -OptsVisitor *ov; +QmpInputVisitor *qiv;QDict *pdict; +QObject *pobj;Object *obj =NULL ; - ov =opts_visitor_new (opts); pdict =qemu_opts_to_qdict (opts,NULL ); + pobj =qdict_crumple (pdict, errp); +if (!pobj) { + goto cleanup; + } + qiv =qmp_input_visitor_new_full (pobj, true, true); - obj =user_creatable_add (pdict,opts_get_visitor (ov), errp); -opts_visitor_cleanup (ov); + obj =user_creatable_add ((QDict *)pobj,qmp_input_get_visitor (qiv), errp); +qmp_input_visitor_cleanup (qiv); +qobject_decref (pobj); + cleanup:QDECREF (pdict);return obj; }
tests/check-qom-proplist.c +315 -1
402 lines not on this page · raw message
Signature · 1 line
2.5.0
PATCH 4/10 util: add QAuthZ object as an authorization base class
In reply to Daniel P. BerrangeThe current qemu_acl module provides a simple access control list facility inside QEMU, which is used via a set of monitor commands acl_show, acl_policy, acl_add, acl_remove & acl_reset.
Note there is no ability to create ACLs - the network services (eg VNC server) were expected to create ACLs that they want to check.
There is also no way to define ACLs on the command line, nor potentially integrate with external authorization systems like polkit, pam, ldap lookup, etc.
The QAuthZ object defines a minimal abstract QOM class that can be subclassed for creating different authorization providers.
--- MAINTAINERS | 7 +++++ Makefile | 1 + Makefile.objs | 2 ++ Makefile.target | 2 ++ include/qemu/authz.h | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++ util/Makefile.objs | 2 ++ util/authz.c | 45 +++++++++++++++++++++++++++++ 7 files changed, 140 insertions(+) create mode 100644 include/qemu/authz.h create mode 100644 util/authz.c
MAINTAINERS +7 -0
@@ -1276,6 +1276,13 @@ S: Maintained F: include/qemu/sockets.h F: util/qemu-sockets.c +Authorization +M: Daniel P. Berrange <[email protected]> +S: Maintained +F: util/authz* +F: include/qemu/authz* +F: tests/test-authz-* + Usermode Emulation ------------------ Overall
Makefile +1 -0
@@ -150,6 +150,7 @@ endif dummy := $(call unnest-vars,, \ stub-obj-y \ util-obj-y \ + util-qom-obj-y \ qga-obj-y \ ivshmem-client-obj-y \ ivshmem-server-obj-y \
Makefile.objs +2 -0
@@ -4,6 +4,8 @@ stub-obj-y = stubs/ util-obj-y = util/ qobject/ qapi/ util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o +util-qom-obj-y += util/ + ####################################################################### # block-obj-y is code used by both qemu system emulation and qemu-img
Makefile.target +2 -0
@@ -171,6 +171,7 @@ include $(SRC_PATH)/Makefile.objs dummy := $(call unnest-vars,,target-obj-y) target-obj-y-save := $(target-obj-y) dummy := $(call unnest-vars,.., \ + util-qom-obj-y \ block-obj-y \ block-obj-m \ crypto-obj-y \
@@ -183,6 +184,7 @@ target-obj-y := $(target-obj-y-save) all-obj-y += $(common-obj-y) all-obj-y += $(target-obj-y) all-obj-y += $(qom-obj-y) +all-obj-y += $(util-qom-obj-y) all-obj-$(CONFIG_SOFTMMU) += $(block-obj-y) all-obj-$(CONFIG_USER_ONLY) += $(crypto-aes-obj-y) all-obj-$(CONFIG_SOFTMMU) += $(crypto-obj-y)
include/qemu/authz.h new file +81 -0
81 lines not on this page · raw message
util/Makefile.objs +2 -0
@@ -32,3 +32,5 @@ util-obj-y += buffer.o util-obj-y += timed-average.o util-obj-y += base64.o util-obj-y += log.o + +util-qom-obj-y += authz.o
util/authz.c new file +45 -0
45 lines not on this page · raw message
Signature · 1 line
2.5.0
PATCH 5/10 util: add QAuthZSimple object type for a simple access control list
In reply to Daniel P. BerrangeAdd a QAuthZSimple object type that implements the QAuthZ interface. This simple built-in implementation maintains a trivial access control list with a sequence of match rules and a final default policy. This replicates the functionality currently provided by the qemu_acl module.
To create an instance of this object via the QMP monitor, the syntax used would be
{
"execute": "object-add",
"arguments": {
"qom-type": "authz-simple",
"id": "auth0",
"parameters": {
"rules": [
{ "match": "fred", "policy": "allow" },
{ "match": "bob", "policy": "allow" },
{ "match": "danb", "policy": "deny" },
{ "match": "dan*", "policy": "allow" }
],
"policy": "deny"
}
}
}Or via the -object command line
$QEMU \
-object authz-simple,id=acl0,policy=deny,\
match.0.name=fred,match.0.policy=allow, \
match.1.name=bob,match.1.policy=allow, \
match.2.name=danb,match.2.policy=deny, \
match.3.name=dan*,match.3.policy=allowThis sets up an authorization rule that allows 'fred', 'bob' and anyone whose name starts with 'dan', except for 'danb'. Everyone unmatched is denied.
--- Makefile | 2 +- include/qemu/authz-simple.h | 107 +++++++++++++++++++ qapi-schema.json | 6 +- qapi/util.json | 31 ++++++ tests/.gitignore | 1 + tests/Makefile | 3 + tests/test-authz-simple.c | 156 +++++++++++++++++++++++++++ util/Makefile.objs | 1 + util/authz-simple.c | 255 ++++++++++++++++++++++++++++++++++++++++++++ 9 files changed, 560 insertions(+), 2 deletions(-) create mode 100644 include/qemu/authz-simple.h create mode 100644 qapi/util.json create mode 100644 tests/test-authz-simple.c create mode 100644 util/authz-simple.c
Makefile +1 -1
@@ -274,7 +274,7 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \ $(SRC_PATH)/qapi/block.json $(SRC_PATH)/qapi/block-core.json \ $(SRC_PATH)/qapi/event.json $(SRC_PATH)/qapi/introspect.json \ $(SRC_PATH)/qapi/crypto.json $(SRC_PATH)/qapi/rocker.json \ - $(SRC_PATH)/qapi/trace.json + $(SRC_PATH)/qapi/trace.json $(SRC_PATH)/qapi/util.json qapi-types.c qapi-types.h :\ $(qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
include/qemu/authz-simple.h new file +107 -0
107 lines not on this page · raw message
qapi-schema.json +5 -1
@@ -5,6 +5,9 @@ # QAPI common definitions { 'include': 'qapi/common.json' } +# QAPI util definitions +{ 'include': 'qapi/util.json' } + # QAPI crypto definitions { 'include': 'qapi/crypto.json' }
@@ -3603,7 +3606,8 @@ # Since 2.5 ## { 'struct': 'DummyForceArrays', - 'data': { 'unused': ['X86CPUFeatureWordInfo'] } } + 'data': { 'unused': ['X86CPUFeatureWordInfo'], + 'iamalsounused': ['QAuthZSimpleRule'] } } ##
qapi/util.json new file +31 -0
@@ -0,0 +1,31 @@ +# -*- Mode: Python -*- +# +# QAPI util definitions + +## +# QAuthZSimplePolicy: +# +# The authorization policy result +# +# @deny: deny access +# @allow: allow access +# +# Since: 2.6 +## +{ 'enum': 'QAuthZSimplePolicy', + 'prefix': 'QAUTHZ_SIMPLE_POLICY', + 'data': ['deny', 'allow']} + +## +# QAuthZSimpleRule: +# +# A single authorization rule. +# +# @match: a glob to match against a user identity +# @policy: the result to return if @match evaluates to true +# +# Since: 2.6 +## +{ 'struct': 'QAuthZSimpleRule', + 'data': {'match': 'str', + 'policy': 'QAuthZSimplePolicy'}}
tests/.gitignore +1 -0
@@ -8,6 +8,7 @@ check-qom-interface check-qom-proplist rcutorture test-aio +test-authz-simple test-base64 test-bitops test-blockjob-txn
tests/Makefile +3 -0
@@ -92,6 +92,7 @@ check-unit-$(CONFIG_GNUTLS) += tests/test-io-channel-tls$(EXESUF) check-unit-y += tests/test-io-channel-command$(EXESUF) check-unit-y += tests/test-io-channel-buffer$(EXESUF) check-unit-y += tests/test-base64$(EXESUF) +check-unit-y += tests/test-authz-simple$(EXESUF) check-block-$(CONFIG_POSIX) += tests/qemu-iotests-quick.sh
@@ -431,6 +432,8 @@ tests/test-timed-average$(EXESUF): tests/test-timed-average.o qemu-timer.o \ $(test-util-obj-y) tests/test-base64$(EXESUF): tests/test-base64.o \ libqemuutil.a libqemustub.a +tests/test-authz-simple$(EXESUF): tests/test-authz-simple.o \ + libqemuutil.a libqemustub.a $(util-qom-obj-y) $(qom-obj-y) tests/test-qapi-types.c tests/test-qapi-types.h :\ $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
tests/test-authz-simple.c new file +156 -0
156 lines not on this page · raw message
util/Makefile.objs +1 -0
@@ -34,3 +34,4 @@ util-obj-y += base64.o util-obj-y += log.o util-qom-obj-y += authz.o +util-qom-obj-y += authz-simple.o
util/authz-simple.c new file +255 -0
255 lines not on this page · raw message
Signature · 1 line
2.5.0
PATCH 6/10 acl: delete existing ACL implementation
In reply to Daniel P. BerrangeThe 'qemu_acl' type was a previous non-QOM based attempt to provide an authorization facility in QEMU. Because it is non-QOM based it cannot be created via the command line and requires special monitor commands to manipulate it.
The new QAuthZ and QAuthZSimple QOM classes provide a superset of the functionality in qemu_acl, so the latter can now be deleted. The HMP 'acl_*' monitor commands are converted to use the new QAuthZSimple data type instead in order to provide backwards compatibility, but their use is discouraged.
--- Makefile | 6 +- crypto/tlssession.c | 28 ++++-- include/qemu/acl.h | 74 ---------------- monitor.c | 161 ++++++++++++++++++++++------------- tests/Makefile | 2 +- tests/test-crypto-tlssession.c | 13 +-- tests/test-io-channel-tls.c | 14 +-- ui/vnc-auth-sasl.c | 2 +- ui/vnc-auth-sasl.h | 4 +- ui/vnc.c | 11 ++- util/Makefile.objs | 1 - util/acl.c | 188 ----------------------------------------- 12 files changed, 156 insertions(+), 348 deletions(-) delete mode 100644 include/qemu/acl.h delete mode 100644 util/acl.c
Makefile +3 -3
@@ -235,9 +235,9 @@ util/module.o-cflags = -D'CONFIG_BLOCK_MODULES=$(block-modules)' qemu-img.o: qemu-img-cmds.h -qemu-img$(EXESUF): qemu-img.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) libqemuutil.a libqemustub.a -qemu-nbd$(EXESUF): qemu-nbd.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) libqemuutil.a libqemustub.a -qemu-io$(EXESUF): qemu-io.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) libqemuutil.a libqemustub.a +qemu-img$(EXESUF): qemu-img.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(util-qom-obj-y) libqemuutil.a libqemustub.a +qemu-nbd$(EXESUF): qemu-nbd.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(util-qom-obj-y) libqemuutil.a libqemustub.a +qemu-io$(EXESUF): qemu-io.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(util-qom-obj-y) libqemuutil.a libqemustub.a qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o
crypto/tlssession.c +23 -5
@@ -22,7 +22,7 @@#include "crypto/tlssession.h" #include "crypto/tlscredsanon.h" #include "crypto/tlscredsx509.h" -#include "qemu/acl.h" +#include "qemu/authz.h" #include "trace.h" #ifdef CONFIG_GNUTLS
@@ -207,6 +207,7 @@ qcrypto_tls_session_check_certificate(QCryptoTLSSession *session,unsigned int nCerts, i;time_t now;gnutls_x509_crt_t cert =NULL ; +Error *err =NULL ; now =time (NULL );if (now == ((time_t)-1 )) {
@@ -295,16 +296,33 @@ qcrypto_tls_session_check_certificate(QCryptoTLSSession *session, goto error; }if (session->aclname) { -qemu_acl *acl =qemu_acl_find (session->aclname); -int allow; -if (!acl) { +QAuthZ *acl; +Object *obj; +Object *container; +bool allow; + + container =object_get_objects_root (); + obj =object_resolve_path_component (container, + session->aclname); +if (!obj) {error_setg (errp,"Cannot find ACL %s" , session->aclname); goto error; } - allow =qemu_acl_party_is_allowed (acl, session->peername); +if (!object_dynamic_cast (obj,TYPE_QAUTHZ )) { +error_setg (errp,"Object '%s' is not a QAuthZ subclass" , + session->aclname); + goto error; + } + acl =QAUTHZ (obj); + + allow =qauthz_is_allowed (acl, session->peername, &err); +if (err) { +error_propagate (errp, err); + goto error; + }if (!allow) {error_setg (errp,"TLS x509 ACL check for %s is denied" , session->peername);
include/qemu/acl.h deleted +0 -74
74 lines not on this page · raw message
monitor.c +101 -60
221 lines not on this page · raw message
tests/Makefile +1 -1
@@ -392,7 +392,7 @@ test-qom-obj-y = $(qom-obj-y) $(test-util-obj-y) test-qapi-obj-y = tests/test-qapi-visit.o tests/test-qapi-types.o \ tests/test-qapi-event.o tests/test-qmp-introspect.o \ $(test-qom-obj-y) -test-crypto-obj-y = $(crypto-obj-y) $(test-qom-obj-y) +test-crypto-obj-y = $(crypto-obj-y) $(util-qom-obj-y) $(test-qom-obj-y) test-io-obj-y = $(io-obj-y) $(test-crypto-obj-y) test-block-obj-y = $(block-obj-y) $(test-io-obj-y)
tests/test-crypto-tlssession.c +8 -5
@@ -25,7 +25,7 @@#include "crypto/tlssession.h" #include "qom/object_interfaces.h" #include "qemu/sockets.h" -#include "qemu/acl.h" +#include "qemu/authz-simple.h" #ifdef QCRYPTO_HAVE_TLS_TEST_SUPPORT
@@ -110,7 +110,7 @@ static void test_crypto_tls_session(const void *opaque)QCryptoTLSCreds *serverCreds;QCryptoTLSSession *clientSess =NULL ;QCryptoTLSSession *serverSess =NULL ; -qemu_acl *acl; +QAuthZSimple *auth;const char *const *wildcards;int channel[2 ];bool clientShake = false;
@@ -169,11 +169,13 @@ static void test_crypto_tls_session(const void *opaque) &err);g_assert (serverCreds !=NULL ); - acl =qemu_acl_init ("tlssessionacl" ); -qemu_acl_reset (acl); + auth =qauthz_simple_new ("tlssessionacl" , +QAUTHZ_SIMPLE_POLICY_DENY , + &error_abort); wildcards = data->wildcards;while (wildcards && *wildcards) { -qemu_acl_append (acl,0 , *wildcards); +qauthz_simple_append_rule (auth, *wildcards, +QAUTHZ_SIMPLE_POLICY_ALLOW ); wildcards++; }
@@ -263,6 +265,7 @@ static void test_crypto_tls_session(const void *opaque)object_unparent (OBJECT (serverCreds));object_unparent (OBJECT (clientCreds)); +object_unparent (OBJECT (auth));qcrypto_tls_session_free (serverSess);qcrypto_tls_session_free (clientSess);
tests/test-io-channel-tls.c +9 -5
@@ -28,7 +28,7 @@#include "io/channel-socket.h" #include "io-channel-helpers.h" #include "crypto/tlscredsx509.h" -#include "qemu/acl.h" +#include "qemu/authz-simple.h" #include "qom/object_interfaces.h" #ifdef QCRYPTO_HAVE_TLS_TEST_SUPPORT
@@ -115,7 +115,7 @@ static void test_io_channel_tls(const void *opaque)QIOChannelTLS *serverChanTLS;QIOChannelSocket *clientChanSock;QIOChannelSocket *serverChanSock; -qemu_acl *acl; +QAuthZSimple *auth;const char *const *wildcards;int channel[2 ];struct QIOChannelTLSHandshakeData clientHandshake = { false, false };
@@ -166,11 +166,13 @@ static void test_io_channel_tls(const void *opaque) &err);g_assert (serverCreds !=NULL ); - acl =qemu_acl_init ("channeltlsacl" ); -qemu_acl_reset (acl); + auth =qauthz_simple_new ("channeltlsacl" , +QAUTHZ_SIMPLE_POLICY_DENY , + &error_abort); wildcards = data->wildcards;while (wildcards && *wildcards) { -qemu_acl_append (acl,0 , *wildcards); +qauthz_simple_append_rule (auth, *wildcards, +QAUTHZ_SIMPLE_POLICY_ALLOW ); wildcards++; }
@@ -256,6 +258,8 @@ static void test_io_channel_tls(const void *opaque)object_unref (OBJECT (serverChanSock));object_unref (OBJECT (clientChanSock)); +object_unparent (OBJECT (auth)); +close (channel[0 ]);close (channel[1 ]); }
ui/vnc-auth-sasl.c +1 -1
@@ -149,7 +149,7 @@ static int vnc_auth_sasl_check_access(VncState *vs)return 0 ; } - allow =qemu_acl_party_is_allowed (vs->vd->sasl.acl, vs->sasl.username); + allow =qauthz_is_allowed (vs->vd->sasl.acl, vs->sasl.username,NULL );VNC_DEBUG ("SASL client %s %s by ACL\n" , vs->sasl.username, allow ?"allowed" :"denied" );
ui/vnc-auth-sasl.h +2 -2
@@ -32,7 +32,7 @@typedef struct VncStateSASL VncStateSASL ;typedef struct VncDisplaySASL VncDisplaySASL ; -#include "qemu/acl.h" +#include "qemu/authz.h" #include "qemu/main-loop.h" struct VncStateSASL {
@@ -61,7 +61,7 @@ struct VncStateSASL { };struct VncDisplaySASL { -qemu_acl *acl; +QAuthZ *acl; };void vnc_sasl_client_cleanup (VncState *vs);
ui/vnc.c +8 -3
@@ -33,7 +33,7 @@#include "qemu/error-report.h" #include "qemu/sockets.h" #include "qemu/timer.h" -#include "qemu/acl.h" +#include "qemu/authz-simple.h" #include "qemu/config-file.h" #include "qapi/qmp/qerror.h" #include "qapi/qmp/types.h"
@@ -3701,7 +3701,9 @@ void vnc_display_open(const char *id, Error **errp) }else { vs->tlsaclname =g_strdup_printf ("vnc.%s.x509dname" , vs->id); } -qemu_acl_init (vs->tlsaclname); +qauthz_simple_new (vs->tlsaclname, +QAUTHZ_SIMPLE_POLICY_DENY , + &error_abort); }#ifdef CONFIG_VNC_SASL if (acl && sasl) {
@@ -3712,7 +3714,10 @@ void vnc_display_open(const char *id, Error **errp) } else { aclname =g_strdup_printf ("vnc.%s.username" , vs->id); } - vs->sasl.acl =qemu_acl_init (aclname); + vs->sasl.acl = +QAUTHZ (qauthz_simple_new (aclname, +QAUTHZ_SIMPLE_POLICY_DENY , + &error_abort));g_free (aclname); }#endif
util/Makefile.objs +0 -1
@@ -13,7 +13,6 @@ util-obj-y += envlist.o path.o module.o util-obj-$(call lnot,$(CONFIG_INT128)) += host-utils.o util-obj-y += bitmap.o bitops.o hbitmap.o util-obj-y += fifo8.o -util-obj-y += acl.o util-obj-y += error.o qemu-error.o util-obj-y += id.o util-obj-y += iov.o qemu-config.o qemu-sockets.o uri.o notify.o
util/acl.c deleted +0 -188
188 lines not on this page · raw message
Signature · 1 line
2.5.0
PATCH 7/10 qemu-nbd: add support for ACLs for TLS clients
In reply to Daniel P. BerrangeCurrently any client which can complete the TLS handshake is able to use the NBD server. The server admin can turn on the 'verify-peer' option for the x509 creds to require the client to provide a x509 certificate. This means the client will have to acquire a certificate from the CA before they are permitted to use the NBD server. This is still a fairly weak bar.
This adds a '--tls-acl ACL-ID' option to the qemu-nbd command which takes the ID of a previously added 'QAuthZ' object instance. This ACL will be used to validate the client's x509 distinguished name. Clients failing the ACL will not be permitted to use the NBD server.
For example to setup an ACL that only allows connection from a client whose x509 certificate distinguished name contains 'CN=fred', you would use:
qemu-nbd -object tls-creds-x509,id=tls0,dir=/home/berrange/qemutls,\
endpoint=server,verify-peer=yes \
-object authz-simple,id=acl0,policy=deny,\
rules.0.match=*CN=fred,rules.0.policy=allow \
-tls-creds tls0 \
-tls-acl acl0
....other qemu-nbd args...--- qemu-nbd.c | 13 ++++++++++++- qemu-nbd.texi | 4 ++++ 2 files changed, 16 insertions(+), 1 deletion(-)
qemu-nbd.c +12 -1
@@ -43,6 +43,7 @@#define QEMU_NBD_OPT_DETECT_ZEROES 4#define QEMU_NBD_OPT_OBJECT 5#define QEMU_NBD_OPT_TLSCREDS 6 +#define QEMU_NBD_OPT_TLSACL 7static NBDExport *exp;static bool newproto;
@@ -56,6 +57,7 @@ static int nb_fds;static QIOChannelSocket *server_ioc;static int server_watch =-1 ;static QCryptoTLSCreds *tlscreds; +static const char *tlsacl;static void usage (const char *name) {
@@ -344,7 +346,7 @@ static gboolean nbd_accept(QIOChannel *ioc, GIOCondition cond, gpointer opaque) nb_fds++;nbd_update_server_watch ();nbd_client_new (newproto ?NULL : exp, cioc, - tlscreds,NULL , nbd_client_closed); + tlscreds, tlsacl, nbd_client_closed);object_unref (OBJECT (cioc));return TRUE;
@@ -475,6 +477,7 @@ int main(int argc, char **argv) { "object", 1, NULL, QEMU_NBD_OPT_OBJECT }, { "export-name", 1, NULL, 'x' }, { "tls-creds", 1, NULL, QEMU_NBD_OPT_TLSCREDS }, + { "tls-acl", 1, NULL, QEMU_NBD_OPT_TLSACL }, { NULL, 0, NULL, 0 } };int ch;
@@ -672,6 +675,9 @@ int main(int argc, char **argv)case QEMU_NBD_OPT_TLSCREDS : tlscredsid = optarg;break ; +case QEMU_NBD_OPT_TLSACL : + tlsacl = optarg; +break ; } }
@@ -708,6 +714,11 @@ int main(int argc, char **argv) error_get_pretty(local_err));exit (EXIT_FAILURE ); } + }else { +if (tlsacl) { +error_report ("--tls-acl is not permitted without --tls-creds" ); +exit (EXIT_FAILURE ); + } }if (disconnect) {
qemu-nbd.texi +4 -0
@@ -81,6 +81,10 @@ the new style NBD protocol negotiation Enable mandatory TLS encryption for the server by setting the ID of the TLS credentials object previously created with the --object option. +@item --tls-acl=ID +Specify the ID of a qauthz object previously created with the +--object option. This will be used to authorize users who +connect against their x509 distinguish name. @item -v, --verbose Display extra debugging information @item -h, --help
Signature · 1 line
2.5.0
PATCH 8/10 nbd: allow an ACL to be set with nbd-server-start QMP command
In reply to Daniel P. BerrangeAs with the previous patch to qemu-nbd, the nbd-server-start QMP command also needs to be able to specify an ACL when enabling TLS encryption.
First the client must create a QAuthZ object instance using the 'object-add' command:
{
'execute': 'object-add',
'arguments': {
'qom-type': 'authz-simple',
'id': 'tls0',
'parameters': {
'policy': 'deny',
'rules': [
{
'match': '*CN=fred',
'policy': 'allow'
}
]
}
}
}They can then reference this in the new 'tls-acl' parameter when executing the 'nbd-server-start' command.
{
'execute': 'nbd-server-start',
'arguments': {
'addr': {
'type': 'inet',
'host': '127.0.0.1',
'port': '9000'
},
'tls-creds': 'tls0',
'tls-acl': 'tlsacl0'
}
}--- blockdev-nbd.c | 10 +++++++++- hmp.c | 2 +- qapi/block.json | 4 +++- qmp-commands.hx | 2 +- 4 files changed, 14 insertions(+), 4 deletions(-)
blockdev-nbd.c +9 -1
@@ -24,6 +24,7 @@ typedef struct NBDServerData {QIOChannelSocket *listen_ioc;int watch;QCryptoTLSCreds *tlscreds; +char *tlsacl; } NBDServerData;static NBDServerData *nbd_server;
@@ -45,7 +46,8 @@ static gboolean nbd_accept(QIOChannel *ioc, GIOCondition condition, }nbd_client_new (NULL , cioc, - nbd_server->tlscreds,NULL , + nbd_server->tlscreds, + nbd_server->tlsacl, nbd_client_put);object_unref (OBJECT (cioc));return TRUE;
@@ -65,6 +67,7 @@ static void nbd_server_free(NBDServerData *server)if (server->tlscreds) {object_unref (OBJECT (server->tlscreds)); } +g_free (server->tlsacl);g_free (server); }
@@ -101,6 +104,7 @@ static QCryptoTLSCreds *nbd_get_tls_creds(const char *id, Error **errp)void qmp_nbd_server_start (SocketAddress *addr,bool has_tls_creds,const char *tls_creds, +bool has_tls_acl,const char *tls_acl,Error **errp) {if (nbd_server) {
@@ -128,6 +132,10 @@ void qmp_nbd_server_start(SocketAddress *addr, } } +if (has_tls_acl) { + nbd_server->tlsacl =g_strdup (tls_acl); + } + nbd_server->watch = qio_channel_add_watch( QIO_CHANNEL(nbd_server->listen_ioc), G_IO_IN,
hmp.c +1 -1
@@ -1785,7 +1785,7 @@ void hmp_nbd_server_start(Monitor *mon, const QDict *qdict) goto exit; } -qmp_nbd_server_start (addr, false,NULL , &local_err); +qmp_nbd_server_start (addr, false,NULL , false,NULL , &local_err);qapi_free_SocketAddress (addr);if (local_err !=NULL ) { goto exit;
qapi/block.json +3 -1
@@ -147,6 +147,7 @@ # # @addr: Address on which to listen. # @tls-creds: (optional) ID of the TLS credentials object. Since 2.6 +# @tls-acl: (optional) ID of the QAuthZ authorization object. Since 2.6 # # Returns: error if the server is already running. #
@@ -154,7 +155,8 @@ ## { 'command': 'nbd-server-start', 'data': { 'addr': 'SocketAddress', - '*tls-creds': 'str'} } + '*tls-creds': 'str', + '*tls-acl': 'str'} } ## # @nbd-server-add:
qmp-commands.hx +1 -1
@@ -3825,7 +3825,7 @@ EQMP { .name = "nbd-server-start", - .args_type = "addr:q,tls-creds:s?", + .args_type = "addr:q,tls-creds:s?,tls-acl:s?", .mhandler.cmd_new = qmp_marshal_nbd_server_start, }, {
Signature · 1 line
2.5.0
PATCH 9/10 chardev: add support for ACLs for TLS clients
In reply to Daniel P. BerrangeCurrently any client which can complete the TLS handshake is able to use a chardev server. The server admin can turn on the 'verify-peer' option for the x509 creds to require the client to provide a x509 certificate. This means the client will have to acquire a certificate from the CA before they are permitted to use the chardev server. This is still a fairly weak bar.
This adds a 'tls-acl=ACL-ID' option to the socket chardev backend which takes the ID of a previously added 'QAuthZ' object instance. This ACL will be used to validate the client's x509 distinguished name. Clients failing the ACL will not be permitted to use the chardev server.
For example to setup an ACL that only allows connection from a client whose x509 certificate distinguished name contains 'CN=fred', you would use:
$QEMU -object tls-creds-x509,id=tls0,dir=/home/berrange/qemutls,\
endpoint=server,verify-peer=yes \
-object authz-simple,id=acl0,policy=deny,\
rules.0.match=*CN=fred,rules.0.policy=allow \
-chardev socket,host=127.0.0.1,port=9000,server,\
tls-creds=tls0,tls-acl=acl0 \
...other qemud args...--- qapi-schema.json | 2 ++ qemu-char.c | 16 +++++++++++++++- 2 files changed, 17 insertions(+), 1 deletion(-)
qapi-schema.json +2 -0
@@ -3150,6 +3150,7 @@ # @addr: socket address to listen on (server=true) # or connect to (server=false) # @tls-creds: #optional the ID of the TLS credentials object (since 2.6) +# @tls-acl: #optional the ID of the QAuthZ authorization object (since 2.6) # @server: #optional create server socket (default: true) # @wait: #optional wait for incoming connection on server # sockets (default: false).
@@ -3165,6 +3166,7 @@ ## { 'struct': 'ChardevSocket', 'data': { 'addr' : 'SocketAddress', '*tls-creds' : 'str', + '*tls-acl' : 'str', '*server' : 'bool', '*wait' : 'bool', '*nodelay' : 'bool',
qemu-char.c +15 -1
@@ -2533,6 +2533,7 @@ typedef struct {QIOChannelSocket *listen_ioc;guint listen_tag;QCryptoTLSCreds *tls_creds; +char *tls_acl;int connected;int max_size;int do_telnetopt;
@@ -2963,7 +2964,7 @@ static void tcp_chr_tls_init(CharDriverState *chr)if (s->is_listen) { tioc =qio_channel_tls_new_server ( s->ioc, s->tls_creds, -NULL ,/* XXX Use an ACL */ + s->tls_acl, &err); }else { tioc = qio_channel_tls_new_client(
@@ -3084,6 +3085,7 @@ static void tcp_chr_close(CharDriverState *chr)if (s->tls_creds) {object_unref (OBJECT (s->tls_creds)); } +g_free (s->tls_acl);if (s->write_msgfds_num) {g_free (s->write_msgfds); }
@@ -3615,6 +3617,7 @@ static void qemu_chr_parse_socket(QemuOpts *opts, ChardevBackend *backend,const char *host =qemu_opt_get (opts,"host" );const char *port =qemu_opt_get (opts,"port" );const char *tls_creds =qemu_opt_get (opts,"tls-creds" ); +const char *tls_acl =qemu_opt_get (opts,"tls-acl" );SocketAddress *addr;if (!path) {
@@ -3633,6 +3636,11 @@ static void qemu_chr_parse_socket(QemuOpts *opts, ChardevBackend *backend, } } +if (tls_acl && !tls_creds) { +error_setg (errp,"ACL should not be provided when TLS is disabled" ); +return ; + } + backend->u.socket =g_new0 (ChardevSocket,1 );qemu_chr_parse_common (opts,qapi_ChardevSocket_base (backend->u.socket));
@@ -3647,6 +3655,7 @@ static void qemu_chr_parse_socket(QemuOpts *opts, ChardevBackend *backend, backend->u.socket->has_reconnect = true; backend->u.socket->reconnect = reconnect; backend->u.socket->tls_creds =g_strdup (tls_creds); + backend->u.socket->tls_acl =g_strdup (tls_acl); addr =g_new0 (SocketAddress,1 );if (path) {
@@ -4077,6 +4086,9 @@ QemuOptsList qemu_chardev_opts = { .name = "tls-creds", .type = QEMU_OPT_STRING, },{ + .name = "tls-acl", + .type = QEMU_OPT_STRING, + },{ .name = "width", .type = QEMU_OPT_NUMBER, },{
@@ -4324,6 +4336,7 @@ static CharDriverState *qmp_chardev_open_socket(const char *id, } } } + s->tls_acl =g_strdup (sock->tls_acl);qapi_copy_SocketAddress (&s->addr, sock->addr);
@@ -4369,6 +4382,7 @@ static CharDriverState *qmp_chardev_open_socket(const char *id,if (s->tls_creds) {object_unref (OBJECT (s->tls_creds)); } +g_free (s->tls_acl);g_free (s);qemu_chr_free_common (chr);return NULL ;
Signature · 1 line
2.5.0
PATCH 10/10 vnc: allow specifying a custom ACL object name
In reply to Daniel P. BerrangeThe VNC server has historically had support for ACLs to check both the SASL username and the TLS x509 distinguished name. The VNC server was responsible for creating the initial ACL, and the client app was then responsible for populating it with rules using the HMP 'acl_add' command.
This is not satisfactory for a variety of reasons. There is no way to populate the ACLs from the command line, users are forced to use the HMP. With multiple network services all supporting TLS and ACLs now, it is desirable to be able to define a single ACL that is referenced by all services.
To address these limitations, two new options are added to the VNC server CLI. The 'tls-acl' option takes the ID of a QAuthZ object to use for checking TLS x509 distinguished names, and the 'sasl-acl' option takes the ID of another object to use for checking SASL usernames.
In this example, we setup two ACLs. The first allows any client with a certificate issued by the 'RedHat' organization in the 'London' locality. The second ACL allows clients with either the '[email protected]' or '[email protected]' kerberos usernames. Both ACLs must pass for the user to be allowed.
$QEMU -object tls-creds-x509,id=tls0,dir=/home/berrange/qemutls,\
endpoint=server,verify-peer=yes \
-object authz-simple,id=acl0,policy=deny,\
rules.0.match=O=RedHat,,L=London,rules.0.policy=allow \
-object authz-simple,id=acl0,policy=deny,\
[email protected],rules.0.policy=allow \
[email protected],rules.0.policy=allow \
-vnc 0.0.0.0:1,tls-creds=tls0,tls-acl=tlsacl0,
sasl,sasl-acl=saslacl0 \
...other QEMU args...--- ui/vnc.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 60 insertions(+), 13 deletions(-)
ui/vnc.c +60 -13
@@ -3258,6 +3258,12 @@ static QemuOptsList qemu_vnc_opts = { .name = "acl", .type = QEMU_OPT_BOOL, },{ + .name = "tls-acl", + .type = QEMU_OPT_STRING, + },{ + .name = "sasl-acl", + .type = QEMU_OPT_STRING, + },{ .name = "lossy", .type = QEMU_OPT_BOOL, },{
@@ -3480,6 +3486,10 @@ void vnc_display_open(const char *id, Error **errp)int saslErr;#endif int acl =0 ; +const char *tlsacl; +#ifdef CONFIG_VNC_SASL +const char *saslacl; +#endif int lock_key_sync =1 ;if (!vs) {
@@ -3666,6 +3676,21 @@ void vnc_display_open(const char *id, Error **errp) } } acl =qemu_opt_get_bool (opts,"acl" , false); + tlsacl =qemu_opt_get (opts,"tls-acl" ); +if (acl && tlsacl) { +error_setg (errp,"'acl' option is mutually exclusive with the " +"'tls-acl' options" ); + goto fail; + } + +#ifdef CONFIG_VNC_SASL + saslacl =qemu_opt_get (opts,"sasl-acl" ); +if (acl && saslacl) { +error_setg (errp,"'acl' option is mutually exclusive with the " +"'sasl-acl' options" ); + goto fail; + } +#endif share =qemu_opt_get (opts,"share" );if (share) {
@@ -3695,7 +3720,9 @@ void vnc_display_open(const char *id, Error **errp) vs->non_adaptive = true; } -if (acl) { +if (tlsacl) { + vs->tlsaclname =g_strdup (tlsacl); + }else if (acl) {if (strcmp (vs->id,"default" ) ==0 ) { vs->tlsaclname =g_strdup ("vnc.x509dname" ); }else {
@@ -3706,19 +3733,39 @@ void vnc_display_open(const char *id, Error **errp) &error_abort); }#ifdef CONFIG_VNC_SASL -if (acl && sasl) { -char *aclname; +if (sasl) { +if (saslacl) { +Object *container, *acl; + container =object_get_objects_root (); + acl =object_resolve_path_component (container, saslacl); +if (!acl) { +error_setg (errp,"Cannot find ACL %s" , saslacl); + goto fail; + } -if (strcmp (vs->id,"default" ) ==0 ) { - aclname =g_strdup ("vnc.username" ); - }else { - aclname =g_strdup_printf ("vnc.%s.username" , vs->id); - } - vs->sasl.acl = -QAUTHZ (qauthz_simple_new (aclname, -QAUTHZ_SIMPLE_POLICY_DENY , - &error_abort)); -g_free (aclname); +if (!object_dynamic_cast (acl,TYPE_QAUTHZ )) { +error_setg (errp,"Object '%s' is not a QAuthZ subclass" , + saslacl); + goto fail; + } + vs->sasl.acl =QAUTHZ (acl); + }else if (acl) { +char *aclname; + +if (strcmp (vs->id,"default" ) ==0 ) { + aclname =g_strdup ("vnc.username" ); + }else { + aclname =g_strdup_printf ("vnc.%s.username" , vs->id); + } + vs->sasl.acl = +QAUTHZ (qauthz_simple_new (aclname, +QAUTHZ_SIMPLE_POLICY_DENY , + &error_abort)); +g_free (aclname); + } + }else if (saslacl) { +error_setg (errp,"SASL ACL provided when SASL is disabled" ); + goto fail; }#endif
Signature · 1 line
2.5.0