Skip to contentSkip to conversation

Provide a QOM-based authorization API

PATCH Superseded Daniel P. Berrange 3 participants 25 messages Jun 2, 2016 – Jun 16, 2016

Subsystems (2)

v5 of 14 Superseded Latest is v14 →

lore
Share & tools
lore
#0 lore raw

v5 cover Provide a QOM-based authorization API

This is a followup of previously posted work in 2.6 cycle:

v1: https://lists.gnu.org/archive/html/qemu-devel/2016-02/msg04618.html
v2: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg01454.html
v3: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg02498.html
v4: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01661.html

Many years ago I was responsible for adding the 'qemu_acl' type and associated HMP commands. Looking back at it now, it is quite a poor facility with a couple of bad limitations. First, the responsibility for creating the ACLs was left with the QEMU network service (VNC server was only thing ever doing it). This meant you could not share ACLs across multiple services. Second, there was no way to populate ACLs on the command line, you had no choice but to use the HMP commands. Third, the API was hardcoded around the idea of an in-QEMU implementation, leaving no scope for plugging in alternative implementations backed by, for example, LDAP or PAM.

This series introduces a much better authorization API design to QEMU that addresses all these problems, and maintains back compatibility. It of course is based on the QOM framework, so that immediately gives us ability to create objects via the CLI, HMP or QMP. There is an abstract base clss "QAuthZ" which defines the basic API for QEMU network services to use, and a specific implementation "QAuthZ" simple which replicates the functionality of 'qemu_acl'. It is thus possible to add other impls, without changing any other part of QEMU in the future. Finally, the user is responsible for creating the ACL objects, so they can have one ACL associated with all their TLS enabled network services.

There was only one small problem with this, specifically the -object CLI arg and HMP 'object_add' command had no way to let the user specify non-scalar properties for objects. eg if an object had a property which is a list of structs, you are out of luck if you want to create it without using QMP.

Thus the first three patches do some work around QAPI / QOM to make it possible to specify non-scalar properties with the -object CLI arg and HMP 'object_add' command. See the respective patches for illustration of the syntax used. Some of Max's recent block patches also depend on the qdict_crumple method in patch 1.

The patches 4 and 5 introduce the new base class and specific implementation.

Patch 6 kills the old qemu_acl code, updating any existing callers of it to use the QAuthZSimple QOM class instead.

Patches 7-11 add support for associating ACLs with the network services supporting TLS encryption (NBD, migration, chardev and VNC).

Changed in v5:

- Resolved conflicts with Eric's visitor refactoring which made it stricter about struct begin/end calls
- Added support for ACLs to migration code now its TLS support is merged.
- Fixed typos in example in commit message

Changed in v4:

- Ensure examples use shell escaping for '*' (Eric)
- Add more tests for crumple impl (Eric)
- Raise error if sasl-acl/tls-acl are requested but sasl/tls auth are not enabled (Eric)
- Document return codes for auth check more clearly (Eric)
- Don't silently turn a glob match into a strcmp if fnmatch is not present (Eric)
- Other misc small typos/fixes (Eric)

Changed in v3:

- Created separate qdict_list_size method (Max)
- Added unit tests for case of empty dict (Max)
- Fix variable names to use underscore separator (Max)
- Fix potential free of uninitialized variables (Max)
- Use QObject APIs for casts, instead of C type casts (Max)

Changed in v2:

- Adapt to changes in qapi visitor APIs
- Add a 'bool recursive' flag to qdict_crumple (Max)
- Fix memory leaks in qdict_crumple (Max)
- Split out key splitting code from qdict_crumple (Max)
- Use saner variable names in qdict_crumple (Max)
- Added some tests for bad inputs to qdict_crumple

Daniel P. Berrange (11):
qdict: implement a qdict_crumple method for un-flattening a dict
qapi: allow QmpInputVisitor to auto-cast types
qom: support arbitrary non-scalar properties with -object
util: add QAuthZ object as an authorization base class
util: add QAuthZSimple object type for a simple access control list
acl: delete existing ACL implementation
qemu-nbd: add support for ACLs for TLS clients
nbd: allow an ACL to be set with nbd-server-start QMP command
migration: add support for a "tls-acl" migration parameter
chardev: add support for ACLs for TLS clients
vnc: allow specifying a custom ACL object name

 MAINTAINERS                        |   7 +
 Makefile                           |   9 +-
 Makefile.objs                      |   2 +
 Makefile.target                    |   2 +
 blockdev-nbd.c                     |  10 +-
 crypto/tlssession.c                |  28 +++-
 docs/qapi-code-gen.txt             |   2 +-
 hmp.c                              |  28 ++--
 include/qapi/qmp-input-visitor.h   |   6 +-
 include/qapi/qmp/qdict.h           |   1 +
 include/qemu/acl.h                 |  74 ---------
 include/qemu/authz-simple.h        | 115 ++++++++++++++
 include/qemu/authz.h               |  89 +++++++++++
 include/qom/object_interfaces.h    |  10 +-
 migration/migration.c              |   7 +
 migration/tls.c                    |   2 +-
 monitor.c                          | 181 ++++++++++++++-------
 qapi-schema.json                   |  28 +++-
 qapi/block.json                    |   4 +-
 qapi/opts-visitor.c                |   1 +
 qapi/qmp-input-visitor.c           |  89 +++++++++--
 qapi/util.json                     |  47 ++++++
 qemu-char.c                        |  11 +-
 qemu-nbd.c                         |  13 +-
 qemu-nbd.texi                      |   4 +
 qmp-commands.hx                    |   2 +-
 qmp.c                              |   4 +-
 qobject/qdict.c                    | 282 +++++++++++++++++++++++++++++++++
 qom/object_interfaces.c            |  49 ++++--
 qom/qom-qobject.c                  |   2 +-
 replay/replay-input.c              |   2 +-
 scripts/qapi-commands.py           |   2 +-
 tests/.gitignore                   |   1 +
 tests/Makefile                     |   5 +-
 tests/check-qdict.c                | 229 +++++++++++++++++++++++++++
 tests/check-qnull.c                |   2 +-
 tests/check-qom-proplist.c         | 317 ++++++++++++++++++++++++++++++++++++-
 tests/test-authz-simple.c          | 183 +++++++++++++++++++++
 tests/test-crypto-tlssession.c     |  13 +-
 tests/test-io-channel-tls.c        |  14 +-
 tests/test-qmp-commands.c          |   2 +-
 tests/test-qmp-input-strict.c      |   2 +-
 tests/test-qmp-input-visitor.c     | 115 +++++++++++++-
 tests/test-visitor-serialization.c |   2 +-
 ui/vnc-auth-sasl.c                 |   2 +-
 ui/vnc-auth-sasl.h                 |   4 +-
 ui/vnc.c                           |  80 ++++++++--
 util/Makefile.objs                 |   4 +-
 util/acl.c                         | 188 ----------------------
 util/authz-simple.c                | 314 ++++++++++++++++++++++++++++++++++++
 util/authz.c                       |  46 ++++++
 util/qemu-sockets.c                |   2 +-
 52 files changed, 2206 insertions(+), 432 deletions(-)
 delete mode 100644 include/qemu/acl.h
 create mode 100644 include/qemu/authz-simple.h
 create mode 100644 include/qemu/authz.h
 create mode 100644 qapi/util.json
 create mode 100644 tests/test-authz-simple.c
 delete mode 100644 util/acl.c
 create mode 100644 util/authz-simple.c
 create mode 100644 util/authz.c
Signature · 1 line
2.5.5
#1 lore raw

PATCH v5 1/11 qdict: implement a qdict_crumple method for un-flattening a dict

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

{
'foo.0.bar': 'one',
'foo.0.wizz': '1',
'foo.1.bar': 'two',
'foo.1.wizz': '2'
}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

{
'foo': [

     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }

],
}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 282 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 229 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 512 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..8a3ac13 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +282 -0

294 lines not on this page · raw message

tests/check-qdict.c +229 -0

247 lines not on this page · raw message

Signature · 1 line
2.5.5
#15 lore raw

PATCH v5 4/11 util: add QAuthZ object as an authorization base class

In reply to Daniel P. Berrange

The current qemu_acl module provides a simple access control list facility inside QEMU, which is used via a set of monitor commands acl_show, acl_policy, acl_add, acl_remove & acl_reset.

Note there is no ability to create ACLs - the network services (eg VNC server) were expected to create ACLs that they want to check.

There is also no way to define ACLs on the command line, nor potentially integrate with external authorization systems like polkit, pam, ldap lookup, etc.

The QAuthZ object defines a minimal abstract QOM class that can be subclassed for creating different authorization providers.

---
 MAINTAINERS          |  7 +++++
 Makefile             |  1 +
 Makefile.objs        |  2 ++
 Makefile.target      |  2 ++
 include/qemu/authz.h | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 util/Makefile.objs   |  2 ++
 util/authz.c         | 46 +++++++++++++++++++++++++++
 7 files changed, 149 insertions(+)
 create mode 100644 include/qemu/authz.h
 create mode 100644 util/authz.c
MAINTAINERS +7 -0
index 3c949d5..6ae3676 100644
@@ -1306,6 +1306,13 @@ F: include/qemu/throttle.h
 F: util/throttle.c
 L: [email protected]
 
+Authorization
+M: Daniel P. Berrange <[email protected]>
+S: Maintained
+F: util/authz*
+F: include/qemu/authz*
+F: tests/test-authz-*
+
 Usermode Emulation
 ------------------
 Overall
Makefile +1 -0
index 3a3c5dc..de83cfe 100644
@@ -147,6 +147,7 @@ endif
 dummy := $(call unnest-vars,, \
                 stub-obj-y \
                 util-obj-y \
+                util-qom-obj-y \
                 qga-obj-y \
                 ivshmem-client-obj-y \
                 ivshmem-server-obj-y \
Makefile.objs +2 -0
index da49b71..24db7f7 100644
@@ -4,6 +4,8 @@ stub-obj-y = stubs/ crypto/
 util-obj-y = util/ qobject/ qapi/
 util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o
 
+util-qom-obj-y += util/
+
 #######################################################################
 # block-obj-y is code used by both qemu system emulation and qemu-img
 
Makefile.target +2 -0
index 5b80dd7..329796a 100644
@@ -176,6 +176,7 @@ include $(SRC_PATH)/Makefile.objs
 dummy := $(call unnest-vars,,target-obj-y)
 target-obj-y-save := $(target-obj-y)
 dummy := $(call unnest-vars,.., \
+               util-qom-obj-y \
                block-obj-y \
                block-obj-m \
                crypto-obj-y \
@@ -188,6 +189,7 @@ target-obj-y := $(target-obj-y-save)
 all-obj-y += $(common-obj-y)
 all-obj-y += $(target-obj-y)
 all-obj-y += $(qom-obj-y)
+all-obj-y += $(util-qom-obj-y)
 all-obj-$(CONFIG_SOFTMMU) += $(block-obj-y)
 all-obj-$(CONFIG_USER_ONLY) += $(crypto-aes-obj-y)
 all-obj-$(CONFIG_SOFTMMU) += $(crypto-obj-y)
include/qemu/authz.h new file +89 -0

89 lines not on this page · raw message

util/Makefile.objs +2 -0
index a8a777e..b29fb45 100644
@@ -32,3 +32,5 @@ util-obj-y += buffer.o
 util-obj-y += timed-average.o
 util-obj-y += base64.o
 util-obj-y += log.o
+
+util-qom-obj-y += authz.o
util/authz.c new file +46 -0

46 lines not on this page · raw message

Signature · 1 line
2.5.5
#12 lore raw

PATCH v5 3/11 qom: support arbitrary non-scalar properties with -object

In reply to Daniel P. Berrange

The current -object command line syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.

This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.

It is not practical to extend the OptsVisitor to support more complex data structures while also maintaining the existing list handling behaviour that is relied upon by other areas of QEMU.

Fortunately there is no existing object that implements the UserCreatable interface that relies on the list handling behaviour, so it is possible to swap out the OptsVisitor for a different visitor implementation, so -object supports non-scalar properties, thus leaving other users of OptsVisitor unaffected.

The previously added qdict_crumple() method is able to take a qdict containing a flat set of properties and turn that into a arbitrarily nested set of dicts and lists. By combining qemu_opts_to_qdict and qdict_crumple() together, we can turn the opt string into a data structure that is practically identical to that passed over QMP when defining an object. The only difference is that all the scalar values are represented as strings, rather than strings, ints and bools. This is sufficient to let us replace the OptsVisitor with the QMPInputVisitor for use with -object.

Thus -object can now support non-scalar properties, for example the QMP object

  {
    "execute": "object-add",
    "arguments": {
      "qom-type": "demo",
      "id": "demo0",
      "parameters": {
        "foo": [
          { "bar": "one", "wizz": "1" },
          { "bar": "two", "wizz": "2" }
        ]
      }
    }
  }

Would be creatable via the CLI now using

    $QEMU \
      -object demo,id=demo0,\
              foo.0.bar=one,foo.0.wizz=1,\
              foo.1.bar=two,foo.1.wizz=2

Notice that this syntax is intentionally compatible with that currently used by block drivers.

This is also wired up to work for the 'object_add' command in the HMP monitor with the same syntax.

  (hmp) object_add demo,id=demo0,\
                   foo.0.bar=one,foo.0.wizz=1,\
                   foo.1.bar=two,foo.1.wizz=2

NB indentation should not be used with HMP commands, this is just for convenient formatting in this commit message.

---
 hmp.c                           |  18 +--
 include/qom/object_interfaces.h |  10 +-
 qmp.c                           |   2 +-
 qom/object_interfaces.c         |  49 +++++--
 tests/check-qom-proplist.c      | 317 +++++++++++++++++++++++++++++++++++++++-
 5 files changed, 363 insertions(+), 33 deletions(-)
hmp.c +5 -13
index a4b1d3d..1972bef 100644
@@ -25,7 +25,7 @@
 #include "qemu/sockets.h"
 #include "monitor/monitor.h"
 #include "monitor/qdev.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qmp-input-visitor.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/string-output-visitor.h"
 #include "qapi/util.h"
@@ -1717,20 +1717,12 @@ void hmp_netdev_del(Monitor *mon, const QDict *qdict)
 void hmp_object_add(Monitor *mon, const QDict *qdict)
 {
     Error *err = NULL;
-    QemuOpts *opts;
-    OptsVisitor *ov;
+    QmpInputVisitor *qiv;
     Object *obj = NULL;
 
-    opts = qemu_opts_from_qdict(qemu_find_opts("object"), qdict, &err);
-    if (err) {
-        hmp_handle_error(mon, &err);
-        return;
-    }
-
-    ov = opts_visitor_new(opts);
-    obj = user_creatable_add(qdict, opts_get_visitor(ov), &err);
-    opts_visitor_cleanup(ov);
-    qemu_opts_del(opts);
+    qiv = qmp_input_visitor_new((QObject *)qdict, true, true);
+    obj = user_creatable_add(qdict, qmp_input_get_visitor(qiv), &err);
+    qmp_input_visitor_cleanup(qiv);
 
     if (err) {
         hmp_handle_error(mon, &err);
include/qom/object_interfaces.h +8 -2
index 8b17f4d..997563a 100644
@@ -96,18 +96,24 @@ Object *user_creatable_add(const QDict *qdict,
  * user_creatable_add_type:
  * @type: the object type name
  * @id: the unique ID for the object
+ * @nested: whether to recurse into the visitor for properties
  * @qdict: the object properties
  * @v: the visitor
  * @errp: if an error occurs, a pointer to an area to store the error
  *
  * Create an instance of the user creatable object @type, placing
  * it in the object composition tree with name @id, initializing
- * it with properties from @qdict
+ * it with properties from @qdict.
+ *
+ * If the visitor is already positioned to read the properties
+ * in @qdict, @nested should be false. Conversely, if it is
+ * neccessary to open/close a struct to read the properties in
+ * @qdict, @nested should be true.
  *
  * Returns: the newly created object or NULL on error
  */
 Object *user_creatable_add_type(const char *type, const char *id,
-                                const QDict *qdict,
+                                bool nested, const QDict *qdict,
                                 Visitor *v, Error **errp);
 
 /**
qmp.c +1 -1
index dcd0953..dc5fff0 100644
@@ -666,7 +666,7 @@ void qmp_object_add(const char *type, const char *id,
     }
 
     qiv = qmp_input_visitor_new(props, true, false);
-    obj = user_creatable_add_type(type, id, pdict,
+    obj = user_creatable_add_type(type, id, true, pdict,
                                   qmp_input_get_visitor(qiv), errp);
     qmp_input_visitor_cleanup(qiv);
     if (obj) {
qom/object_interfaces.c +33 -16

94 lines not on this page · raw message

tests/check-qom-proplist.c +316 -1

409 lines not on this page · raw message

Signature · 1 line
2.5.5
#16 lore raw

PATCH v5 5/11 util: add QAuthZSimple object type for a simple access control list

In reply to Daniel P. Berrange

Add a QAuthZSimple object type that implements the QAuthZ interface. This simple built-in implementation maintains a trivial access control list with a sequence of match rules and a final default policy. This replicates the functionality currently provided by the qemu_acl module.

To create an instance of this object via the QMP monitor, the syntax used would be

  {
    "execute": "object-add",
    "arguments": {
      "qom-type": "authz-simple",
      "id": "auth0",
      "parameters": {
        "rules": [
           { "match": "fred", "policy": "allow", "format": "exact" },
           { "match": "bob", "policy": "allow", "format": "exact" },
           { "match": "danb", "policy": "deny", "format": "glob" },
           { "match": "dan*", "policy": "allow", "format": "exact" },
        ],
        "policy": "deny"
      }
    }
  }

Or via the -object command line

  $QEMU \
     -object authz-simple,id=acl0,policy=deny,\
             rules.0.match=fred,rules.0.policy=allow,rules.0.format=exact,\
             rules.1.match=bob,rules.1.policy=allow,rules.1.format=exact,\
             rules.2.match=danb,rules.2.policy=deny,rules.2.format=glob,\
             rules.3.match=dan\*,rules.3.policy=allow,rules.3.format=exact

This sets up an authorization rule that allows 'fred', 'bob' and anyone whose name starts with 'dan', except for 'danb'. Everyone unmatched is denied.

---
 Makefile                    |   2 +-
 include/qemu/authz-simple.h | 115 ++++++++++++++++
 qapi-schema.json            |   6 +-
 qapi/util.json              |  47 +++++++
 tests/.gitignore            |   1 +
 tests/Makefile              |   3 +
 tests/test-authz-simple.c   | 183 ++++++++++++++++++++++++++
 util/Makefile.objs          |   1 +
 util/authz-simple.c         | 314 ++++++++++++++++++++++++++++++++++++++++++++
 9 files changed, 670 insertions(+), 2 deletions(-)
 create mode 100644 include/qemu/authz-simple.h
 create mode 100644 qapi/util.json
 create mode 100644 tests/test-authz-simple.c
 create mode 100644 util/authz-simple.c
Makefile +1 -1
index de83cfe..ae3ffac 100644
@@ -271,7 +271,7 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/block.json $(SRC_PATH)/qapi/block-core.json \
                $(SRC_PATH)/qapi/event.json $(SRC_PATH)/qapi/introspect.json \
                $(SRC_PATH)/qapi/crypto.json $(SRC_PATH)/qapi/rocker.json \
-               $(SRC_PATH)/qapi/trace.json
+               $(SRC_PATH)/qapi/trace.json $(SRC_PATH)/qapi/util.json
 
 qapi-types.c qapi-types.h :\
 $(qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
include/qemu/authz-simple.h new file +115 -0

115 lines not on this page · raw message

qapi-schema.json +5 -1
index 8483bdf..337a6ce 100644
@@ -5,6 +5,9 @@
 # QAPI common definitions
 { 'include': 'qapi/common.json' }
 
+# QAPI util definitions
+{ 'include': 'qapi/util.json' }
+
 # QAPI crypto definitions
 { 'include': 'qapi/crypto.json' }
 
@@ -3729,7 +3732,8 @@
 # Since 2.5
 ##
 { 'struct': 'DummyForceArrays',
-  'data': { 'unused': ['X86CPUFeatureWordInfo'] } }
+  'data': { 'unused1': ['X86CPUFeatureWordInfo'],
+            'unused2': ['QAuthZSimpleRule'] } }
 
 
 ##
qapi/util.json new file +47 -0

47 lines not on this page · raw message

tests/.gitignore +1 -0
index a06a8ba..2e204a0 100644
@@ -9,6 +9,7 @@ check-qom-interface
 check-qom-proplist
 rcutorture
 test-aio
+test-authz-simple
 test-base64
 test-bitops
 test-blockjob-txn
tests/Makefile +3 -0
index 4bc041c..c4c5fc5 100644
@@ -103,6 +103,7 @@ check-unit-y += tests/test-crypto-xts$(EXESUF)
 check-unit-y += tests/test-crypto-block$(EXESUF)
 gcov-files-test-logging-y = tests/test-logging.c
 check-unit-y += tests/test-logging$(EXESUF)
+check-unit-y += tests/test-authz-simple$(EXESUF)
 
 check-block-$(CONFIG_POSIX) += tests/qemu-iotests-quick.sh
 
@@ -446,6 +447,8 @@ tests/test-timed-average$(EXESUF): tests/test-timed-average.o qemu-timer.o \
 	$(test-util-obj-y)
 tests/test-base64$(EXESUF): tests/test-base64.o \
 	libqemuutil.a libqemustub.a
+tests/test-authz-simple$(EXESUF): tests/test-authz-simple.o \
+	libqemuutil.a libqemustub.a $(util-qom-obj-y) $(qom-obj-y)
 
 tests/test-logging$(EXESUF): tests/test-logging.o $(test-util-obj-y)
 
tests/test-authz-simple.c new file +183 -0

183 lines not on this page · raw message

util/Makefile.objs +1 -0
index b29fb45..4870905 100644
@@ -34,3 +34,4 @@ util-obj-y += base64.o
 util-obj-y += log.o
 
 util-qom-obj-y += authz.o
+util-qom-obj-y += authz-simple.o
util/authz-simple.c new file +314 -0

314 lines not on this page · raw message

Signature · 1 line
2.5.5
#6 lore raw

PATCH v5 2/11 qapi: allow QmpInputVisitor to auto-cast types

In reply to Daniel P. Berrange

Currently the QmpInputVisitor assumes that all scalar values are directly represented as their final types. ie it assumes an 'int' is using QInt, and a 'bool' is using QBool.

This extends it so that QString is optionally permitted for any of the non-string scalar types. This behaviour is turned on by requesting the 'autocast' flag in the constructor.

This makes it possible to use QmpInputVisitor with a QDict produced from QemuOpts, where everything is in string format.

---
 docs/qapi-code-gen.txt             |   2 +-
 include/qapi/qmp-input-visitor.h   |   6 +-
 qapi/opts-visitor.c                |   1 +
 qapi/qmp-input-visitor.c           |  89 ++++++++++++++++++++++------
 qmp.c                              |   2 +-
 qom/qom-qobject.c                  |   2 +-
 replay/replay-input.c              |   2 +-
 scripts/qapi-commands.py           |   2 +-
 tests/check-qnull.c                |   2 +-
 tests/test-qmp-commands.c          |   2 +-
 tests/test-qmp-input-strict.c      |   2 +-
 tests/test-qmp-input-visitor.c     | 115 ++++++++++++++++++++++++++++++++++++-
 tests/test-visitor-serialization.c |   2 +-
 util/qemu-sockets.c                |   2 +-
 14 files changed, 201 insertions(+), 30 deletions(-)
docs/qapi-code-gen.txt +1 -1
index d7d6987..e21773e 100644
@@ -1008,7 +1008,7 @@ Example:
     {
         Error *err = NULL;
         UserDefOne *retval;
-        QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true);
+        QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true, false);
         QapiDeallocVisitor *qdv;
         Visitor *v;
         UserDefOneList *arg1 = NULL;
include/qapi/qmp-input-visitor.h +5 -1
index b0624d8..d35a053 100644
@@ -24,8 +24,12 @@ typedef struct QmpInputVisitor QmpInputVisitor;
  *
  * Set @strict to reject a parse that doesn't consume all keys of a
  * dictionary; otherwise excess input is ignored.
+ * Set @autocast to automatically convert string values into more
+ * specific types (numbers, bools, etc)
  */
-QmpInputVisitor *qmp_input_visitor_new(QObject *obj, bool strict);
+QmpInputVisitor *qmp_input_visitor_new(QObject *obj,
+                                       bool strict,
+                                       bool autocast);
 
 void qmp_input_visitor_cleanup(QmpInputVisitor *v);
 
qapi/opts-visitor.c +1 -0
index 4cf1cf8..00e4b1a 100644
@@ -347,6 +347,7 @@ opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp)
     }
 
     if (opt->str) {
+        /* Keep these values in sync with same code in qmp-input-visitor.c */
         if (strcmp(opt->str, "on") == 0 ||
             strcmp(opt->str, "yes") == 0 ||
             strcmp(opt->str, "y") == 0) {
qapi/qmp-input-visitor.c +73 -16

155 lines not on this page · raw message

qmp.c +1 -1
index 3165f87..dcd0953 100644
@@ -665,7 +665,7 @@ void qmp_object_add(const char *type, const char *id,
         }
     }
 
-    qiv = qmp_input_visitor_new(props, true);
+    qiv = qmp_input_visitor_new(props, true, false);
     obj = user_creatable_add_type(type, id, pdict,
                                   qmp_input_get_visitor(qiv), errp);
     qmp_input_visitor_cleanup(qiv);
qom/qom-qobject.c +1 -1
index b66088d..99666ce 100644
@@ -23,7 +23,7 @@ void object_property_set_qobject(Object *obj, QObject *value,
 {
     QmpInputVisitor *qiv;
     /* TODO: Should we reject, rather than ignore, excess input? */
-    qiv = qmp_input_visitor_new(value, false);
+    qiv = qmp_input_visitor_new(value, false, false);
     object_property_set(obj, qmp_input_get_visitor(qiv), name, errp);
 
     qmp_input_visitor_cleanup(qiv);
replay/replay-input.c +1 -1
index 03e99d5..de82a59 100644
@@ -37,7 +37,7 @@ static InputEvent *qapi_clone_InputEvent(InputEvent *src)
         return NULL;
     }
 
-    qiv = qmp_input_visitor_new(obj, true);
+    qiv = qmp_input_visitor_new(obj, true, false);
     iv = qmp_input_get_visitor(qiv);
     visit_type_InputEvent(iv, NULL, &dst, &error_abort);
     qmp_input_visitor_cleanup(qiv);
scripts/qapi-commands.py +1 -1
index 8c6acb3..e48995d 100644
@@ -115,7 +115,7 @@ def gen_marshal(name, arg_type, ret_type):
 
     if arg_type and arg_type.members:
         ret += mcgen('''
-    QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true);
+    QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true, false);
     QapiDeallocVisitor *qdv;
     Visitor *v;
     %(c_name)s arg = {0};
tests/check-qnull.c +1 -1
index fd9c68f..4c11755 100644
@@ -49,7 +49,7 @@ static void qnull_visit_test(void)
 
     g_assert(qnull_.refcnt == 1);
     obj = qnull();
-    qiv = qmp_input_visitor_new(obj, true);
+    qiv = qmp_input_visitor_new(obj, true, false);
     qobject_decref(obj);
     visit_type_null(qmp_input_get_visitor(qiv), NULL, &error_abort);
     qmp_input_visitor_cleanup(qiv);
tests/test-qmp-commands.c +1 -1
index 5c3edd7..c86b282 100644
@@ -222,7 +222,7 @@ static void test_dealloc_partial(void)
         ud2_dict = qdict_new();
         qdict_put_obj(ud2_dict, "string0", QOBJECT(qstring_from_str(text)));
 
-        qiv = qmp_input_visitor_new(QOBJECT(ud2_dict), true);
+        qiv = qmp_input_visitor_new(QOBJECT(ud2_dict), true, false);
         visit_type_UserDefTwo(qmp_input_get_visitor(qiv), NULL, &ud2, &err);
         qmp_input_visitor_cleanup(qiv);
         QDECREF(ud2_dict);
tests/test-qmp-input-strict.c +1 -1
index 4602529..f7f1f00 100644
@@ -55,7 +55,7 @@ static Visitor *validate_test_init_internal(TestInputVisitorData *data,
     data->obj = qobject_from_jsonv(json_string, ap);
     g_assert(data->obj);
 
-    data->qiv = qmp_input_visitor_new(data->obj, true);
+    data->qiv = qmp_input_visitor_new(data->obj, true, false);
     g_assert(data->qiv);
 
     v = qmp_input_get_visitor(data->qiv);
tests/test-qmp-input-visitor.c +112 -3

173 lines not on this page · raw message

tests/test-visitor-serialization.c +1 -1
index 7b14b5a..db618d8 100644
@@ -1038,7 +1038,7 @@ static void qmp_deserialize(void **native_out, void *datap,
     obj = qobject_from_json(qstring_get_str(output_json));
 
     QDECREF(output_json);
-    d->qiv = qmp_input_visitor_new(obj, true);
+    d->qiv = qmp_input_visitor_new(obj, true, false);
     qobject_decref(obj_orig);
     qobject_decref(obj);
     visit(qmp_input_get_visitor(d->qiv), native_out, errp);
util/qemu-sockets.c +1 -1
index 0d6cd1f..51c6a8e 100644
@@ -1145,7 +1145,7 @@ void qapi_copy_SocketAddress(SocketAddress **p_dest,
         return;
     }
 
-    qiv = qmp_input_visitor_new(obj, true);
+    qiv = qmp_input_visitor_new(obj, true, false);
     iv = qmp_input_get_visitor(qiv);
     visit_type_SocketAddress(iv, NULL, p_dest, &error_abort);
     qmp_input_visitor_cleanup(qiv);
Signature · 1 line
2.5.5
#21 lore raw

PATCH v5 10/11 chardev: add support for ACLs for TLS clients

In reply to Daniel P. Berrange

Currently any client which can complete the TLS handshake is able to use a chardev server. The server admin can turn on the 'verify-peer' option for the x509 creds to require the client to provide a x509 certificate. This means the client will have to acquire a certificate from the CA before they are permitted to use the chardev server. This is still a fairly weak bar.

This adds a 'tls-acl=ACL-ID' option to the socket chardev backend which takes the ID of a previously added 'QAuthZ' object instance. This ACL will be used to validate the client's x509 distinguished name. Clients failing the ACL will not be permitted to use the chardev server.

For example to setup an ACL that only allows connection from a client whose x509 certificate distinguished name contains 'CN=fred', you would use:

  $QEMU -object tls-creds-x509,id=tls0,dir=/home/berrange/qemutls,\
                endpoint=server,verify-peer=yes \
        -object authz-simple,id=acl0,policy=deny,\
                rules.0.match=\*CN=fred,rules.0.policy=allow \
        -chardev socket,host=127.0.0.1,port=9000,server,\
	         tls-creds=tls0,tls-acl=acl0 \
        ...other qemud args...
---
 qapi-schema.json |  2 ++
 qemu-char.c      | 11 ++++++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)
qapi-schema.json +2 -0
index e7ec2a1..b231e2f 100644
@@ -3293,6 +3293,7 @@
 # @addr: socket address to listen on (server=true)
 #        or connect to (server=false)
 # @tls-creds: #optional the ID of the TLS credentials object (since 2.6)
+# @tls-acl: #optional the ID of the QAuthZ authorization object (since 2.6)
 # @server: #optional create server socket (default: true)
 # @wait: #optional wait for incoming connection on server
 #        sockets (default: false).
@@ -3308,6 +3309,7 @@
 ##
 { 'struct': 'ChardevSocket', 'data': { 'addr'       : 'SocketAddress',
                                      '*tls-creds'  : 'str',
+                                     '*tls-acl'    : 'str',
                                      '*server'    : 'bool',
                                      '*wait'      : 'bool',
                                      '*nodelay'   : 'bool',
qemu-char.c +10 -1
index b597ee1..e3ced21 100644
@@ -2604,6 +2604,7 @@ typedef struct {
     QIOChannelSocket *listen_ioc;
     guint listen_tag;
     QCryptoTLSCreds *tls_creds;
+    char *tls_acl;
     int connected;
     int max_size;
     int do_telnetopt;
@@ -3047,7 +3048,7 @@ static void tcp_chr_tls_init(CharDriverState *chr)
     if (s->is_listen) {
         tioc = qio_channel_tls_new_server(
             s->ioc, s->tls_creds,
-            NULL, /* XXX Use an ACL */
+            s->tls_acl,
             &err);
     } else {
         tioc = qio_channel_tls_new_client(
@@ -3169,6 +3170,7 @@ static void tcp_chr_close(CharDriverState *chr)
     if (s->tls_creds) {
         object_unref(OBJECT(s->tls_creds));
     }
+    g_free(s->tls_acl);
     if (s->write_msgfds_num) {
         g_free(s->write_msgfds);
     }
@@ -3668,6 +3670,7 @@ static void qemu_chr_parse_socket(QemuOpts *opts, ChardevBackend *backend,
     const char *host = qemu_opt_get(opts, "host");
     const char *port = qemu_opt_get(opts, "port");
     const char *tls_creds = qemu_opt_get(opts, "tls-creds");
+    const char *tls_acl = qemu_opt_get(opts, "tls-acl");
     SocketAddress *addr;
     ChardevSocket *sock;
 
@@ -3701,6 +3704,7 @@ static void qemu_chr_parse_socket(QemuOpts *opts, ChardevBackend *backend,
     sock->has_reconnect = true;
     sock->reconnect = reconnect;
     sock->tls_creds = g_strdup(tls_creds);
+    sock->tls_acl = g_strdup(tls_acl);
 
     addr = g_new0(SocketAddress, 1);
     if (path) {
@@ -4155,6 +4159,9 @@ QemuOptsList qemu_chardev_opts = {
             .name = "tls-creds",
             .type = QEMU_OPT_STRING,
         },{
+            .name = "tls-acl",
+            .type = QEMU_OPT_STRING,
+        },{
             .name = "width",
             .type = QEMU_OPT_NUMBER,
         },{
@@ -4398,6 +4405,7 @@ static CharDriverState *qmp_chardev_open_socket(const char *id,
             }
         }
     }
+    s->tls_acl = g_strdup(sock->tls_acl);
 
     qapi_copy_SocketAddress(&s->addr, sock->addr);
 
@@ -4461,6 +4469,7 @@ static CharDriverState *qmp_chardev_open_socket(const char *id,
     if (s->tls_creds) {
         object_unref(OBJECT(s->tls_creds));
     }
+    g_free(s->tls_acl);
     g_free(s);
     qemu_chr_free_common(chr);
     return NULL;
Signature · 1 line
2.5.5
#17 lore raw

PATCH v5 6/11 acl: delete existing ACL implementation

In reply to Daniel P. Berrange

The 'qemu_acl' type was a previous non-QOM based attempt to provide an authorization facility in QEMU. Because it is non-QOM based it cannot be created via the command line and requires special monitor commands to manipulate it.

The new QAuthZ and QAuthZSimple QOM classes provide a superset of the functionality in qemu_acl, so the latter can now be deleted. The HMP 'acl_*' monitor commands are converted to use the new QAuthZSimple data type instead in order to provide backwards compatibility, but their use is discouraged.

---
 Makefile                       |   6 +-
 crypto/tlssession.c            |  28 ++++--
 include/qemu/acl.h             |  74 ----------------
 monitor.c                      | 181 ++++++++++++++++++++++++++-------------
 tests/Makefile                 |   2 +-
 tests/test-crypto-tlssession.c |  13 +--
 tests/test-io-channel-tls.c    |  14 +--
 ui/vnc-auth-sasl.c             |   2 +-
 ui/vnc-auth-sasl.h             |   4 +-
 ui/vnc.c                       |  11 ++-
 util/Makefile.objs             |   1 -
 util/acl.c                     | 188 -----------------------------------------
 12 files changed, 176 insertions(+), 348 deletions(-)
 delete mode 100644 include/qemu/acl.h
 delete mode 100644 util/acl.c
Makefile +3 -3
index ae3ffac..97e13da 100644
@@ -232,9 +232,9 @@ util/module.o-cflags = -D'CONFIG_BLOCK_MODULES=$(block-modules)'
 
 qemu-img.o: qemu-img-cmds.h
 
-qemu-img$(EXESUF): qemu-img.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) libqemuutil.a libqemustub.a
-qemu-nbd$(EXESUF): qemu-nbd.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) libqemuutil.a libqemustub.a
-qemu-io$(EXESUF): qemu-io.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) libqemuutil.a libqemustub.a
+qemu-img$(EXESUF): qemu-img.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(util-qom-obj-y) libqemuutil.a libqemustub.a
+qemu-nbd$(EXESUF): qemu-nbd.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(util-qom-obj-y) libqemuutil.a libqemustub.a
+qemu-io$(EXESUF): qemu-io.o $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(util-qom-obj-y) libqemuutil.a libqemustub.a
 
 qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o libqemuutil.a libqemustub.a
 
crypto/tlssession.c +23 -5
index a543e5a..78bb0f5 100644
@@ -23,7 +23,7 @@
 #include "crypto/tlscredsanon.h"
 #include "crypto/tlscredsx509.h"
 #include "qapi/error.h"
-#include "qemu/acl.h"
+#include "qemu/authz.h"
 #include "trace.h"
 
 #ifdef CONFIG_GNUTLS
@@ -208,6 +208,7 @@ qcrypto_tls_session_check_certificate(QCryptoTLSSession *session,
     unsigned int nCerts, i;
     time_t now;
     gnutls_x509_crt_t cert = NULL;
+    Error *err = NULL;
 
     now = time(NULL);
     if (now == ((time_t)-1)) {
@@ -296,16 +297,33 @@ qcrypto_tls_session_check_certificate(QCryptoTLSSession *session,
                 goto error;
             }
             if (session->aclname) {
-                qemu_acl *acl = qemu_acl_find(session->aclname);
-                int allow;
-                if (!acl) {
+                QAuthZ *acl;
+                Object *obj;
+                Object *container;
+                bool allow;
+
+                container = object_get_objects_root();
+                obj = object_resolve_path_component(container,
+                                                    session->aclname);
+                if (!obj) {
                     error_setg(errp, "Cannot find ACL %s",
                                session->aclname);
                     goto error;
                 }
 
-                allow = qemu_acl_party_is_allowed(acl, session->peername);
+                if (!object_dynamic_cast(obj, TYPE_QAUTHZ)) {
+                    error_setg(errp, "Object '%s' is not a QAuthZ subclass",
+                               session->aclname);
+                    goto error;
+                }
 
+                acl = QAUTHZ(obj);
+
+                allow = qauthz_is_allowed(acl, session->peername, &err);
+                if (err) {
+                    error_propagate(errp, err);
+                    goto error;
+                }
                 if (!allow) {
                     error_setg(errp, "TLS x509 ACL check for %s is denied",
                                session->peername);
include/qemu/acl.h deleted +0 -74

74 lines not on this page · raw message

monitor.c +121 -60

240 lines not on this page · raw message

tests/Makefile +1 -1
index c4c5fc5..3d62b3f 100644
@@ -405,7 +405,7 @@ test-qom-obj-y = $(qom-obj-y) $(test-util-obj-y)
 test-qapi-obj-y = tests/test-qapi-visit.o tests/test-qapi-types.o \
 	tests/test-qapi-event.o tests/test-qmp-introspect.o \
 	$(test-qom-obj-y)
-test-crypto-obj-y = $(crypto-obj-y) $(test-qom-obj-y)
+test-crypto-obj-y = $(crypto-obj-y) $(util-qom-obj-y) $(test-qom-obj-y)
 test-io-obj-y = $(io-obj-y) $(test-crypto-obj-y)
 test-block-obj-y = $(block-obj-y) $(test-io-obj-y)
 
tests/test-crypto-tlssession.c +8 -5
index 1a4a066..d3a30e9 100644
@@ -26,7 +26,7 @@
 #include "qom/object_interfaces.h"
 #include "qapi/error.h"
 #include "qemu/sockets.h"
-#include "qemu/acl.h"
+#include "qemu/authz-simple.h"
 
 #ifdef QCRYPTO_HAVE_TLS_TEST_SUPPORT
 
@@ -111,7 +111,7 @@ static void test_crypto_tls_session(const void *opaque)
     QCryptoTLSCreds *serverCreds;
     QCryptoTLSSession *clientSess = NULL;
     QCryptoTLSSession *serverSess = NULL;
-    qemu_acl *acl;
+    QAuthZSimple *auth;
     const char * const *wildcards;
     int channel[2];
     bool clientShake = false;
@@ -170,11 +170,13 @@ static void test_crypto_tls_session(const void *opaque)
         &err);
     g_assert(serverCreds != NULL);
 
-    acl = qemu_acl_init("tlssessionacl");
-    qemu_acl_reset(acl);
+    auth = qauthz_simple_new("tlssessionacl",
+                             QAUTHZ_SIMPLE_POLICY_DENY,
+                             &error_abort);
     wildcards = data->wildcards;
     while (wildcards && *wildcards) {
-        qemu_acl_append(acl, 0, *wildcards);
+        qauthz_simple_append_rule(auth, *wildcards,
+                                  QAUTHZ_SIMPLE_POLICY_ALLOW);
         wildcards++;
     }
 
@@ -264,6 +266,7 @@ static void test_crypto_tls_session(const void *opaque)
 
     object_unparent(OBJECT(serverCreds));
     object_unparent(OBJECT(clientCreds));
+    object_unparent(OBJECT(auth));
 
     qcrypto_tls_session_free(serverSess);
     qcrypto_tls_session_free(clientSess);
tests/test-io-channel-tls.c +9 -5
index 3c361a7..390d3e9 100644
@@ -28,7 +28,7 @@
 #include "io/channel-socket.h"
 #include "io-channel-helpers.h"
 #include "crypto/tlscredsx509.h"
-#include "qemu/acl.h"
+#include "qemu/authz-simple.h"
 #include "qom/object_interfaces.h"
 
 #ifdef QCRYPTO_HAVE_TLS_TEST_SUPPORT
@@ -115,7 +115,7 @@ static void test_io_channel_tls(const void *opaque)
     QIOChannelTLS *serverChanTLS;
     QIOChannelSocket *clientChanSock;
     QIOChannelSocket *serverChanSock;
-    qemu_acl *acl;
+    QAuthZSimple *auth;
     const char * const *wildcards;
     int channel[2];
     struct QIOChannelTLSHandshakeData clientHandshake = { false, false };
@@ -166,11 +166,13 @@ static void test_io_channel_tls(const void *opaque)
         &err);
     g_assert(serverCreds != NULL);
 
-    acl = qemu_acl_init("channeltlsacl");
-    qemu_acl_reset(acl);
+    auth = qauthz_simple_new("channeltlsacl",
+                             QAUTHZ_SIMPLE_POLICY_DENY,
+                             &error_abort);
     wildcards = data->wildcards;
     while (wildcards && *wildcards) {
-        qemu_acl_append(acl, 0, *wildcards);
+        qauthz_simple_append_rule(auth, *wildcards,
+                                  QAUTHZ_SIMPLE_POLICY_ALLOW);
         wildcards++;
     }
 
@@ -256,6 +258,8 @@ static void test_io_channel_tls(const void *opaque)
     object_unref(OBJECT(serverChanSock));
     object_unref(OBJECT(clientChanSock));
 
+    object_unparent(OBJECT(auth));
+
     close(channel[0]);
     close(channel[1]);
 }
ui/vnc-auth-sasl.c +1 -1
index 5ae29c1..b1064a7 100644
@@ -150,7 +150,7 @@ static int vnc_auth_sasl_check_access(VncState *vs)
         return 0;
     }
 
-    allow = qemu_acl_party_is_allowed(vs->vd->sasl.acl, vs->sasl.username);
+    allow = qauthz_is_allowed(vs->vd->sasl.acl, vs->sasl.username, NULL);
 
     VNC_DEBUG("SASL client %s %s by ACL\n", vs->sasl.username,
               allow ? "allowed" : "denied");
ui/vnc-auth-sasl.h +2 -2
index 3f59da6..a6ffb7e 100644
@@ -32,7 +32,7 @@
 typedef struct VncStateSASL VncStateSASL;
 typedef struct VncDisplaySASL VncDisplaySASL;
 
-#include "qemu/acl.h"
+#include "qemu/authz.h"
 #include "qemu/main-loop.h"
 
 struct VncStateSASL {
@@ -61,7 +61,7 @@ struct VncStateSASL {
 };
 
 struct VncDisplaySASL {
-    qemu_acl *acl;
+    QAuthZ *acl;
 };
 
 void vnc_sasl_client_cleanup(VncState *vs);
ui/vnc.c +8 -3
index d2ebf1f..ba07715 100644
@@ -33,7 +33,7 @@
 #include "qemu/error-report.h"
 #include "qemu/sockets.h"
 #include "qemu/timer.h"
-#include "qemu/acl.h"
+#include "qemu/authz-simple.h"
 #include "qemu/config-file.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/types.h"
@@ -3707,7 +3707,9 @@ void vnc_display_open(const char *id, Error **errp)
         } else {
             vs->tlsaclname = g_strdup_printf("vnc.%s.x509dname", vs->id);
         }
-        qemu_acl_init(vs->tlsaclname);
+        qauthz_simple_new(vs->tlsaclname,
+                          QAUTHZ_SIMPLE_POLICY_DENY,
+                          &error_abort);
     }
 #ifdef CONFIG_VNC_SASL
     if (acl && sasl) {
@@ -3718,7 +3720,10 @@ void vnc_display_open(const char *id, Error **errp)
         } else {
             aclname = g_strdup_printf("vnc.%s.username", vs->id);
         }
-        vs->sasl.acl = qemu_acl_init(aclname);
+        vs->sasl.acl =
+            QAUTHZ(qauthz_simple_new(aclname,
+                                     QAUTHZ_SIMPLE_POLICY_DENY,
+                                     &error_abort));
         g_free(aclname);
     }
 #endif
util/Makefile.objs +0 -1
index 4870905..44078c1 100644
@@ -13,7 +13,6 @@ util-obj-y += envlist.o path.o module.o
 util-obj-$(call lnot,$(CONFIG_INT128)) += host-utils.o
 util-obj-y += bitmap.o bitops.o hbitmap.o
 util-obj-y += fifo8.o
-util-obj-y += acl.o
 util-obj-y += error.o qemu-error.o
 util-obj-y += id.o
 util-obj-y += iov.o qemu-config.o qemu-sockets.o uri.o notify.o
util/acl.c deleted +0 -188

188 lines not on this page · raw message

Signature · 1 line
2.5.5
#18 lore raw

PATCH v5 7/11 qemu-nbd: add support for ACLs for TLS clients

In reply to Daniel P. Berrange

Currently any client which can complete the TLS handshake is able to use the NBD server. The server admin can turn on the 'verify-peer' option for the x509 creds to require the client to provide a x509 certificate. This means the client will have to acquire a certificate from the CA before they are permitted to use the NBD server. This is still a fairly weak bar.

This adds a '--tls-acl ACL-ID' option to the qemu-nbd command which takes the ID of a previously added 'QAuthZ' object instance. This ACL will be used to validate the client's x509 distinguished name. Clients failing the ACL will not be permitted to use the NBD server.

For example to setup an ACL that only allows connection from a client whose x509 certificate distinguished name contains 'CN=fred', you would use:

  qemu-nbd -object tls-creds-x509,id=tls0,dir=/home/berrange/qemutls,\
                   endpoint=server,verify-peer=yes \
           -object authz-simple,id=acl0,policy=deny,\
	           rules.0.match=*CN=fred,rules.0.policy=allow \
           -tls-creds tls0 \
           -tls-acl acl0
	   ....other qemu-nbd args...
---
 qemu-nbd.c    | 13 ++++++++++++-
 qemu-nbd.texi |  4 ++++
 2 files changed, 16 insertions(+), 1 deletion(-)
qemu-nbd.c +12 -1
index 6554f0a..2450214 100644
@@ -46,6 +46,7 @@
 #define QEMU_NBD_OPT_OBJECT        260
 #define QEMU_NBD_OPT_TLSCREDS      261
 #define QEMU_NBD_OPT_IMAGE_OPTS    262
+#define QEMU_NBD_OPT_TLSACL        263
 
 #define MBR_SIZE 512
 
@@ -61,6 +62,7 @@ static int nb_fds;
 static QIOChannelSocket *server_ioc;
 static int server_watch = -1;
 static QCryptoTLSCreds *tlscreds;
+static const char *tlsacl;
 
 static void usage(const char *name)
 {
@@ -354,7 +356,7 @@ static gboolean nbd_accept(QIOChannel *ioc, GIOCondition cond, gpointer opaque)
     nb_fds++;
     nbd_update_server_watch();
     nbd_client_new(newproto ? NULL : exp, cioc,
-                   tlscreds, NULL, nbd_client_closed);
+                   tlscreds, tlsacl, nbd_client_closed);
     object_unref(OBJECT(cioc));
 
     return TRUE;
@@ -498,6 +500,7 @@ int main(int argc, char **argv)
         { "export-name", required_argument, NULL, 'x' },
         { "tls-creds", required_argument, NULL, QEMU_NBD_OPT_TLSCREDS },
         { "image-opts", no_argument, NULL, QEMU_NBD_OPT_IMAGE_OPTS },
+        { "tls-acl", no_argument, NULL, QEMU_NBD_OPT_TLSACL },
         { NULL, 0, NULL, 0 }
     };
     int ch;
@@ -703,6 +706,9 @@ int main(int argc, char **argv)
         case QEMU_NBD_OPT_IMAGE_OPTS:
             imageOpts = true;
             break;
+        case QEMU_NBD_OPT_TLSACL:
+            tlsacl = optarg;
+            break;
         }
     }
 
@@ -738,6 +744,11 @@ int main(int argc, char **argv)
                          error_get_pretty(local_err));
             exit(EXIT_FAILURE);
         }
+    } else {
+        if (tlsacl) {
+            error_report("--tls-acl is not permitted without --tls-creds");
+            exit(EXIT_FAILURE);
+        }
     }
 
     if (disconnect) {
qemu-nbd.texi +4 -0
index 9f23343..69f32cb 100644
@@ -86,6 +86,10 @@ the new style NBD protocol negotiation
 Enable mandatory TLS encryption for the server by setting the ID
 of the TLS credentials object previously created with the --object
 option.
+@item --tls-acl=ID
+Specify the ID of a qauthz object previously created with the
+--object option. This will be used to authorize users who
+connect against their x509 distinguish name.
 @item -v, --verbose
 Display extra debugging information
 @item -h, --help
Signature · 1 line
2.5.5
#20 lore raw

PATCH v5 9/11 migration: add support for a "tls-acl" migration parameter

In reply to Daniel P. Berrange

The QEMU instance that runs as the server for the migration data transport (ie the target QEMU) needs to be able to configure access control so it can prevent unauthorized clients initiating an incoming migration. This adds a new 'tls-acl' migration parameter that is used to provide the QOM ID of a QAuthZ subclass instance that provides the access control check. This ACL is checked against the x509 certificate obtained during the TLS handshake.

---
 hmp.c                 |  8 ++++++++
 migration/migration.c |  7 +++++++
 migration/tls.c       |  2 +-
 qapi-schema.json      | 20 +++++++++++++++++---
 4 files changed, 33 insertions(+), 4 deletions(-)
hmp.c +8 -0
index e0d0d8c..4cbf37c 100644
@@ -300,6 +300,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, " %s: '%s'",
             MigrationParameter_lookup[MIGRATION_PARAMETER_TLS_HOSTNAME],
             params->tls_hostname ? : "");
+        monitor_printf(mon, " %s: '%s'",
+            MigrationParameter_lookup[MIGRATION_PARAMETER_TLS_ACL],
+            params->tls_acl ? : "");
         monitor_printf(mon, "\n");
     }
 
@@ -1259,6 +1262,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
     bool has_cpu_throttle_increment = false;
     bool has_tls_creds = false;
     bool has_tls_hostname = false;
+    bool has_tls_acl = false;
     bool use_int_value = false;
     int i;
 
@@ -1290,6 +1294,9 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
             case MIGRATION_PARAMETER_TLS_HOSTNAME:
                 has_tls_hostname = true;
                 break;
+            case MIGRATION_PARAMETER_TLS_ACL:
+                has_tls_acl = true;
+                break;
             }
 
             if (use_int_value) {
@@ -1307,6 +1314,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
                                        has_cpu_throttle_increment, valueint,
                                        has_tls_creds, valuestr,
                                        has_tls_hostname, valuestr,
+                                       has_tls_acl, valuestr,
                                        &err);
             break;
         }
migration/migration.c +7 -0
index 7ecbade..b5e8e2f 100644
@@ -566,6 +566,7 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
     params->cpu_throttle_increment = s->parameters.cpu_throttle_increment;
     params->tls_creds = g_strdup(s->parameters.tls_creds);
     params->tls_hostname = g_strdup(s->parameters.tls_hostname);
+    params->tls_acl = g_strdup(s->parameters.tls_acl);
 
     return params;
 }
@@ -771,6 +772,8 @@ void qmp_migrate_set_parameters(bool has_compress_level,
                                 const char *tls_creds,
                                 bool has_tls_hostname,
                                 const char *tls_hostname,
+                                bool has_tls_acl,
+                                const char *tls_acl,
                                 Error **errp)
 {
     MigrationState *s = migrate_get_current();
@@ -830,6 +833,10 @@ void qmp_migrate_set_parameters(bool has_compress_level,
         g_free(s->parameters.tls_hostname);
         s->parameters.tls_hostname = g_strdup(tls_hostname);
     }
+    if (has_tls_acl) {
+        g_free(s->parameters.tls_acl);
+        s->parameters.tls_acl = g_strdup(tls_acl);
+    }
 }
 
 
migration/tls.c +1 -1
index 75f959f..968fe16 100644
@@ -92,7 +92,7 @@ void migration_tls_set_incoming_channel(MigrationState *s,
 
     tioc = qio_channel_tls_new_server(
         ioc, creds,
-        NULL, /* XXX pass ACL name */
+        s->parameters.tls_acl,
         errp);
     if (!tioc) {
         return;
qapi-schema.json +17 -3
index 337a6ce..e7ec2a1 100644
@@ -636,12 +636,16 @@
 #                hostname must be provided so that the server's x509
 #                certificate identity canbe validated. (Since 2.7)
 #
+# @tls-acl: ID of the 'authz' object subclass that provides access control
+#           checking of the TLS x509 certificate distinguished name. (Since
+#           2.7)
+#
 # Since: 2.4
 ##
 { 'enum': 'MigrationParameter',
   'data': ['compress-level', 'compress-threads', 'decompress-threads',
            'cpu-throttle-initial', 'cpu-throttle-increment',
-           'tls-creds', 'tls-hostname'] }
+           'tls-creds', 'tls-hostname', 'tls-acl'] }
 
 #
 # @migrate-set-parameters
@@ -677,6 +681,10 @@
 #                hostname must be provided so that the server's x509
 #                certificate identity canbe validated. (Since 2.7)
 #
+# @tls-acl: ID of the 'authz' object subclass that provides access control
+#           checking of the TLS x509 certificate distinguished name. (Since
+#           2.7)
+#
 # Since: 2.4
 ##
 { 'command': 'migrate-set-parameters',
@@ -686,7 +694,8 @@
             '*cpu-throttle-initial': 'int',
             '*cpu-throttle-increment': 'int',
             '*tls-creds': 'str',
-            '*tls-hostname': 'str'} }
+            '*tls-hostname': 'str',
+            '*tls-acl': 'str'} }
 
 #
 # @MigrationParameters
@@ -720,6 +729,10 @@
 #                hostname must be provided so that the server's x509
 #                certificate identity canbe validated. (Since 2.6)
 #
+# @tls-acl: ID of the 'authz' object subclass that provides access control
+#           checking of the TLS x509 certificate distinguished name. (Since
+#           2.7)
+#
 # Since: 2.4
 ##
 { 'struct': 'MigrationParameters',
@@ -729,7 +742,8 @@
             'cpu-throttle-initial': 'int',
             'cpu-throttle-increment': 'int',
             'tls-creds': 'str',
-            'tls-hostname': 'str'} }
+            'tls-hostname': 'str',
+            'tls-acl': 'str'} }
 ##
 # @query-migrate-parameters
 #
Signature · 1 line
2.5.5
#19 lore raw

PATCH v5 8/11 nbd: allow an ACL to be set with nbd-server-start QMP command

In reply to Daniel P. Berrange

As with the previous patch to qemu-nbd, the nbd-server-start QMP command also needs to be able to specify an ACL when enabling TLS encryption.

First the client must create a QAuthZ object instance using the 'object-add' command:

   {
     'execute': 'object-add',
     'arguments': {
       'qom-type': 'authz-simple',
       'id': 'tls0',
       'parameters': {
         'policy': 'deny',
         'rules': [
           {
             'match': '*CN=fred',
             'policy': 'allow'
           }
         ]
       }
     }
   }

They can then reference this in the new 'tls-acl' parameter when executing the 'nbd-server-start' command.

   {
     'execute': 'nbd-server-start',
     'arguments': {
       'addr': {
           'type': 'inet',
           'host': '127.0.0.1',
           'port': '9000'
       },
       'tls-creds': 'tls0',
       'tls-acl': 'tlsacl0'
     }
   }
---
 blockdev-nbd.c  | 10 +++++++++-
 hmp.c           |  2 +-
 qapi/block.json |  4 +++-
 qmp-commands.hx |  2 +-
 4 files changed, 14 insertions(+), 4 deletions(-)
blockdev-nbd.c +9 -1
index 12cae0e..ae5335e 100644
@@ -24,6 +24,7 @@ typedef struct NBDServerData {
     QIOChannelSocket *listen_ioc;
     int watch;
     QCryptoTLSCreds *tlscreds;
+    char *tlsacl;
 } NBDServerData;
 
 static NBDServerData *nbd_server;
@@ -45,7 +46,8 @@ static gboolean nbd_accept(QIOChannel *ioc, GIOCondition condition,
     }
 
     nbd_client_new(NULL, cioc,
-                   nbd_server->tlscreds, NULL,
+                   nbd_server->tlscreds,
+                   nbd_server->tlsacl,
                    nbd_client_put);
     object_unref(OBJECT(cioc));
     return TRUE;
@@ -65,6 +67,7 @@ static void nbd_server_free(NBDServerData *server)
     if (server->tlscreds) {
         object_unref(OBJECT(server->tlscreds));
     }
+    g_free(server->tlsacl);
 
     g_free(server);
 }
@@ -101,6 +104,7 @@ static QCryptoTLSCreds *nbd_get_tls_creds(const char *id, Error **errp)
 
 void qmp_nbd_server_start(SocketAddress *addr,
                           bool has_tls_creds, const char *tls_creds,
+                          bool has_tls_acl, const char *tls_acl,
                           Error **errp)
 {
     if (nbd_server) {
@@ -128,6 +132,10 @@ void qmp_nbd_server_start(SocketAddress *addr,
         }
     }
 
+    if (has_tls_acl) {
+        nbd_server->tlsacl = g_strdup(tls_acl);
+    }
+
     nbd_server->watch = qio_channel_add_watch(
         QIO_CHANNEL(nbd_server->listen_ioc),
         G_IO_IN,
hmp.c +1 -1
index 1972bef..e0d0d8c 100644
@@ -1846,7 +1846,7 @@ void hmp_nbd_server_start(Monitor *mon, const QDict *qdict)
         goto exit;
     }
 
-    qmp_nbd_server_start(addr, false, NULL, &local_err);
+    qmp_nbd_server_start(addr, false, NULL, false, NULL, &local_err);
     qapi_free_SocketAddress(addr);
     if (local_err != NULL) {
         goto exit;
qapi/block.json +3 -1
index 937337d..a674865 100644
@@ -147,6 +147,7 @@
 #
 # @addr: Address on which to listen.
 # @tls-creds: (optional) ID of the TLS credentials object. Since 2.6
+# @tls-acl: (optional) ID of the QAuthZ authorization object. Since 2.6
 #
 # Returns: error if the server is already running.
 #
@@ -154,7 +155,8 @@
 ##
 { 'command': 'nbd-server-start',
   'data': { 'addr': 'SocketAddress',
-            '*tls-creds': 'str'} }
+            '*tls-creds': 'str',
+            '*tls-acl': 'str'} }
 
 ##
 # @nbd-server-add:
qmp-commands.hx +1 -1
index 28801a2..ad10b1e 100644
@@ -3859,7 +3859,7 @@ EQMP
 
     {
         .name       = "nbd-server-start",
-        .args_type  = "addr:q,tls-creds:s?",
+        .args_type  = "addr:q,tls-creds:s?,tls-acl:s?",
         .mhandler.cmd_new = qmp_marshal_nbd_server_start,
     },
     {
Signature · 1 line
2.5.5
#22 lore raw

PATCH v5 11/11 vnc: allow specifying a custom ACL object name

In reply to Daniel P. Berrange

The VNC server has historically had support for ACLs to check both the SASL username and the TLS x509 distinguished name. The VNC server was responsible for creating the initial ACL, and the client app was then responsible for populating it with rules using the HMP 'acl_add' command.

This is not satisfactory for a variety of reasons. There is no way to populate the ACLs from the command line, users are forced to use the HMP. With multiple network services all supporting TLS and ACLs now, it is desirable to be able to define a single ACL that is referenced by all services.

To address these limitations, two new options are added to the VNC server CLI. The 'tls-acl' option takes the ID of a QAuthZ object to use for checking TLS x509 distinguished names, and the 'sasl-acl' option takes the ID of another object to use for checking SASL usernames.

In this example, we setup two ACLs. The first allows any client with a certificate issued by the 'RedHat' organization in the 'London' locality. The second ACL allows clients with either the '[email protected]' or '[email protected]' kerberos usernames. Both ACLs must pass for the user to be allowed.

    $QEMU -object tls-creds-x509,id=tls0,dir=/home/berrange/qemutls,\
                  endpoint=server,verify-peer=yes \
          -object authz-simple,id=tlsacl0,policy=deny,\
                  rules.0.match=O=RedHat,,L=London,rules.0.policy=allow \
          -object authz-simple,id=saslacl0,policy=deny,\
                  [email protected],rules.0.policy=allow \
                  [email protected],rules.0.policy=allow \
          -vnc 0.0.0.0:1,tls-creds=tls0,tls-acl=tlsacl0,
	       sasl,sasl-acl=saslacl0 \
          ...other QEMU args...
---
 ui/vnc.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 64 insertions(+), 13 deletions(-)
ui/vnc.c +64 -13
index ba07715..939c3d3 100644
@@ -3263,6 +3263,12 @@ static QemuOptsList qemu_vnc_opts = {
             .name = "acl",
             .type = QEMU_OPT_BOOL,
         },{
+            .name = "tls-acl",
+            .type = QEMU_OPT_STRING,
+        },{
+            .name = "sasl-acl",
+            .type = QEMU_OPT_STRING,
+        },{
             .name = "lossy",
             .type = QEMU_OPT_BOOL,
         },{
@@ -3485,6 +3491,8 @@ void vnc_display_open(const char *id, Error **errp)
     int saslErr;
 #endif
     int acl = 0;
+    const char *tlsacl;
+    const char *saslacl;
     int lock_key_sync = 1;
 
     if (!vs) {
@@ -3672,6 +3680,27 @@ void vnc_display_open(const char *id, Error **errp)
         }
     }
     acl = qemu_opt_get_bool(opts, "acl", false);
+    tlsacl = qemu_opt_get(opts, "tls-acl");
+    if (acl && tlsacl) {
+        error_setg(errp, "'acl' option is mutually exclusive with the "
+                   "'tls-acl' options");
+        goto fail;
+    }
+    if (tlsacl && !vs->tlscreds) {
+        error_setg(errp, "'tls-acl' provided but TLS is not enabled");
+        goto fail;
+    }
+
+    saslacl = qemu_opt_get(opts, "sasl-acl");
+    if (acl && saslacl) {
+        error_setg(errp, "'acl' option is mutually exclusive with the "
+                   "'sasl-acl' options");
+        goto fail;
+    }
+    if (saslacl && !sasl) {
+        error_setg(errp, "'sasl-acl' provided but SASL auth is not enabled");
+        goto fail;
+    }
 
     share = qemu_opt_get(opts, "share");
     if (share) {
@@ -3701,7 +3730,9 @@ void vnc_display_open(const char *id, Error **errp)
         vs->non_adaptive = true;
     }
 
-    if (acl) {
+    if (tlsacl) {
+        vs->tlsaclname = g_strdup(tlsacl);
+    } else if (acl) {
         if (strcmp(vs->id, "default") == 0) {
             vs->tlsaclname = g_strdup("vnc.x509dname");
         } else {
@@ -3712,19 +3743,39 @@ void vnc_display_open(const char *id, Error **errp)
                           &error_abort);
     }
 #ifdef CONFIG_VNC_SASL
-    if (acl && sasl) {
-        char *aclname;
+    if (sasl) {
+        if (saslacl) {
+            Object *container, *acl;
+            container = object_get_objects_root();
+            acl = object_resolve_path_component(container, saslacl);
+            if (!acl) {
+                error_setg(errp, "Cannot find ACL %s", saslacl);
+                goto fail;
+            }
 
-        if (strcmp(vs->id, "default") == 0) {
-            aclname = g_strdup("vnc.username");
-        } else {
-            aclname = g_strdup_printf("vnc.%s.username", vs->id);
-        }
-        vs->sasl.acl =
-            QAUTHZ(qauthz_simple_new(aclname,
-                                     QAUTHZ_SIMPLE_POLICY_DENY,
-                                     &error_abort));
-        g_free(aclname);
+            if (!object_dynamic_cast(acl, TYPE_QAUTHZ)) {
+                error_setg(errp, "Object '%s' is not a QAuthZ subclass",
+                           saslacl);
+                goto fail;
+            }
+            vs->sasl.acl = QAUTHZ(acl);
+        } else if (acl) {
+            char *aclname;
+
+            if (strcmp(vs->id, "default") == 0) {
+                aclname = g_strdup("vnc.username");
+            } else {
+                aclname = g_strdup_printf("vnc.%s.username", vs->id);
+            }
+            vs->sasl.acl =
+                QAUTHZ(qauthz_simple_new(aclname,
+                                         QAUTHZ_SIMPLE_POLICY_DENY,
+                                         &error_abort));
+            g_free(aclname);
+        }
+    } else if (saslacl) {
+        error_setg(errp, "SASL ACL provided when SASL is disabled");
+        goto fail;
     }
 #endif
 
Signature · 1 line
2.5.5
#23 lore raw
In reply to Daniel P. Berrange
Quoted 37 lines · Daniel P. Berrange

This is a followup of previously posted work in 2.6 cycle:

v1: https://lists.gnu.org/archive/html/qemu-devel/2016-02/msg04618.html
v2: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg01454.html
v3: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg02498.html
v4: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01661.html

Many years ago I was responsible for adding the 'qemu_acl' type and associated HMP commands. Looking back at it now, it is quite a poor facility with a couple of bad limitations. First, the responsibility for creating the ACLs was left with the QEMU network service (VNC server was only thing ever doing it). This meant you could not share ACLs across multiple services. Second, there was no way to populate ACLs on the command line, you had no choice but to use the HMP commands. Third, the API was hardcoded around the idea of an in-QEMU implementation, leaving no scope for plugging in alternative implementations backed by, for example, LDAP or PAM.

This series introduces a much better authorization API design to QEMU that addresses all these problems, and maintains back compatibility. It of course is based on the QOM framework, so that immediately gives us ability to create objects via the CLI, HMP or QMP. There is an abstract base clss "QAuthZ" which defines the basic API for QEMU network services to use, and a specific implementation "QAuthZ" simple which replicates the functionality of 'qemu_acl'. It is thus possible to add other impls, without changing any other part of QEMU in the future. Finally, the user is responsible for creating the ACL objects, so they can have one ACL associated with all their TLS enabled network services.

There was only one small problem with this, specifically the -object CLI arg and HMP 'object_add' command had no way to let the user specify non-scalar properties for objects. eg if an object had a property which is a list of structs, you are out of luck if you want to create it without using QMP.

Quoted 6 lines

Thus the first three patches do some work around QAPI / QOM to make it possible to specify non-scalar properties with the -object CLI arg and HMP 'object_add' command. See the respective patches for illustration of the syntax used. Some of Max's recent block patches also depend on the qdict_crumple method in patch 1.

Ping, does anyone have any feedback on these first 3 patches in particular. If we get agreement on those and get them merged, then we can feed the rest of this series to the individual subsystem maintainers that are affected, and also unblock Max's patch series.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#7 lore raw

Re: [Qemu-devel] [PATCH v5 02/11] qapi: allow QmpInputVisitor to auto-cast types

In reply to Daniel P. Berrange
Quoted 5 lines · Daniel P. Berrange

Currently the QmpInputVisitor assumes that all scalar values are directly represented as their final types. ie it assumes an 'int' is using QInt, and a 'bool' is using QBool.

Quoted 8 lines

This extends it so that QString is optionally permitted for any of the non-string scalar types. This behaviour is turned on by requesting the 'autocast' flag in the constructor.

This makes it possible to use QmpInputVisitor with a QDict produced from QemuOpts, where everything is in string format.

Perhaps this should instead be a separate QmpStringInputVisitor visitor that _only_ accepts strings? You can reuse most of the QmpInputVisitor by putting it in the same file, because the struct and list visitors are compatible.

I wonder if OptsVisitor could also be replaced by qemu_opts_to_qdict, an optional crumple step and the QmpStringInputVisitor (self-answer:
probably not because of the magic integer range parsing).

Paolo

Quoted 532 lines
---
 docs/qapi-code-gen.txt             |   2 +-
 include/qapi/qmp-input-visitor.h   |   6 +-
 qapi/opts-visitor.c                |   1 +
 qapi/qmp-input-visitor.c           |  89 ++++++++++++++++++++++------
 qmp.c                              |   2 +-
 qom/qom-qobject.c                  |   2 +-
 replay/replay-input.c              |   2 +-
 scripts/qapi-commands.py           |   2 +-
 tests/check-qnull.c                |   2 +-
 tests/test-qmp-commands.c          |   2 +-
 tests/test-qmp-input-strict.c      |   2 +-
 tests/test-qmp-input-visitor.c     | 115 ++++++++++++++++++++++++++++++++++++-
 tests/test-visitor-serialization.c |   2 +-
 util/qemu-sockets.c                |   2 +-
 14 files changed, 201 insertions(+), 30 deletions(-)
docs/qapi-code-gen.txt +1 -1
index d7d6987..e21773e 100644
@@ -1008,7 +1008,7 @@ Example:
     {
         Error *err = NULL;
         UserDefOne *retval;
-        QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true);
+        QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true, false);
         QapiDeallocVisitor *qdv;
         Visitor *v;
         UserDefOneList *arg1 = NULL;
include/qapi/qmp-input-visitor.h +5 -1
index b0624d8..d35a053 100644
@@ -24,8 +24,12 @@ typedef struct QmpInputVisitor QmpInputVisitor;
  *
  * Set @strict to reject a parse that doesn't consume all keys of a
  * dictionary; otherwise excess input is ignored.
+ * Set @autocast to automatically convert string values into more
+ * specific types (numbers, bools, etc)
  */
-QmpInputVisitor *qmp_input_visitor_new(QObject *obj, bool strict);
+QmpInputVisitor *qmp_input_visitor_new(QObject *obj,
+                                       bool strict,
+                                       bool autocast);
 
 void qmp_input_visitor_cleanup(QmpInputVisitor *v);
 
qapi/opts-visitor.c +1 -0
index 4cf1cf8..00e4b1a 100644
@@ -347,6 +347,7 @@ opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp)
     }
 
     if (opt->str) {
+        /* Keep these values in sync with same code in qmp-input-visitor.c */
         if (strcmp(opt->str, "on") == 0 ||
             strcmp(opt->str, "yes") == 0 ||
             strcmp(opt->str, "y") == 0) {
qapi/qmp-input-visitor.c +73 -16
index aea90a1..92023b1 100644
@@ -20,6 +20,7 @@
 #include "qemu-common.h"
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/qerror.h"
+#include "qemu/cutils.h"
 
 #define QIV_STACK_SIZE 1024
 
@@ -45,6 +46,7 @@ struct QmpInputVisitor
 
     /* True to reject parse in visit_end_struct() if unvisited keys remain. */
     bool strict;
+    bool autocast;
 };
 
 static QmpInputVisitor *to_qiv(Visitor *v)
@@ -254,15 +256,25 @@ static void qmp_input_type_int64(Visitor *v, const char *name, int64_t *obj,
                                  Error **errp)
 {
     QmpInputVisitor *qiv = to_qiv(v);
-    QInt *qint = qobject_to_qint(qmp_input_get_object(qiv, name, true));
+    QObject *qobj = qmp_input_get_object(qiv, name, true);
+    QInt *qint;
+    QString *qstr;
 
-    if (!qint) {
-        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
-                   "integer");
+    qint = qobject_to_qint(qobj);
+    if (qint) {
+        *obj = qint_get_int(qint);
         return;
     }
 
-    *obj = qint_get_int(qint);
+    qstr = qobject_to_qstring(qobj);
+    if (qstr && qstr->string && qiv->autocast) {
+        if (qemu_strtoll(qstr->string, NULL, 10, obj) == 0) {
+            return;
+        }
+    }
+
+    error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+               "integer");
 }
 
 static void qmp_input_type_uint64(Visitor *v, const char *name, uint64_t *obj,
@@ -270,30 +282,61 @@ static void qmp_input_type_uint64(Visitor *v, const char *name, uint64_t *obj,
 {
     /* FIXME: qobject_to_qint mishandles values over INT64_MAX */
     QmpInputVisitor *qiv = to_qiv(v);
-    QInt *qint = qobject_to_qint(qmp_input_get_object(qiv, name, true));
+    QObject *qobj = qmp_input_get_object(qiv, name, true);
+    QInt *qint;
+    QString *qstr;
 
-    if (!qint) {
-        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
-                   "integer");
+    qint = qobject_to_qint(qobj);
+    if (qint) {
+        *obj = qint_get_int(qint);
         return;
     }
 
-    *obj = qint_get_int(qint);
+    qstr = qobject_to_qstring(qobj);
+    if (qstr && qstr->string && qiv->autocast) {
+        if (qemu_strtoull(qstr->string, NULL, 10, obj) == 0) {
+            return;
+        }
+    }
+
+    error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+               "integer");
 }
 
 static void qmp_input_type_bool(Visitor *v, const char *name, bool *obj,
                                 Error **errp)
 {
     QmpInputVisitor *qiv = to_qiv(v);
-    QBool *qbool = qobject_to_qbool(qmp_input_get_object(qiv, name, true));
+    QObject *qobj = qmp_input_get_object(qiv, name, true);
+    QBool *qbool;
+    QString *qstr;
 
-    if (!qbool) {
-        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
-                   "boolean");
+    qbool = qobject_to_qbool(qobj);
+    if (qbool) {
+        *obj = qbool_get_bool(qbool);
         return;
     }
 
-    *obj = qbool_get_bool(qbool);
+
+    qstr = qobject_to_qstring(qobj);
+    if (qstr && qstr->string && qiv->autocast) {
+        /* Keep these values in sync with same code in opts-visitor.c */
+        if (!strcasecmp(qstr->string, "on") ||
+            !strcasecmp(qstr->string, "yes") ||
+            !strcasecmp(qstr->string, "true")) {
+            *obj = true;
+            return;
+        }
+        if (!strcasecmp(qstr->string, "off") ||
+            !strcasecmp(qstr->string, "no") ||
+            !strcasecmp(qstr->string, "false")) {
+            *obj = false;
+            return;
+        }
+    }
+
+    error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+               "boolean");
 }
 
 static void qmp_input_type_str(Visitor *v, const char *name, char **obj,
@@ -319,6 +362,8 @@ static void qmp_input_type_number(Visitor *v, const char *name, double *obj,
     QObject *qobj = qmp_input_get_object(qiv, name, true);
     QInt *qint;
     QFloat *qfloat;
+    QString *qstr;
+    char *endp;
 
     qint = qobject_to_qint(qobj);
     if (qint) {
@@ -332,6 +377,15 @@ static void qmp_input_type_number(Visitor *v, const char *name, double *obj,
         return;
     }
 
+    qstr = qobject_to_qstring(qobj);
+    if (qstr && qstr->string && qiv->autocast) {
+        errno = 0;
+        *obj = strtod(qstr->string, &endp);
+        if (errno == 0 && endp != qstr->string && *endp == '\0') {
+            return;
+        }
+    }
+
     error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
                "number");
 }
@@ -381,7 +435,9 @@ void qmp_input_visitor_cleanup(QmpInputVisitor *v)
     g_free(v);
 }
 
-QmpInputVisitor *qmp_input_visitor_new(QObject *obj, bool strict)
+QmpInputVisitor *qmp_input_visitor_new(QObject *obj,
+                                       bool strict,
+                                       bool autocast)
 {
     QmpInputVisitor *v;
 
@@ -404,6 +460,7 @@ QmpInputVisitor *qmp_input_visitor_new(QObject *obj, bool strict)
     v->visitor.type_null = qmp_input_type_null;
     v->visitor.optional = qmp_input_optional;
     v->strict = strict;
+    v->autocast = autocast;
 
     v->root = obj;
     qobject_incref(obj);
qmp.c +1 -1
index 3165f87..dcd0953 100644
@@ -665,7 +665,7 @@ void qmp_object_add(const char *type, const char *id,
         }
     }
 
-    qiv = qmp_input_visitor_new(props, true);
+    qiv = qmp_input_visitor_new(props, true, false);
     obj = user_creatable_add_type(type, id, pdict,
                                   qmp_input_get_visitor(qiv), errp);
     qmp_input_visitor_cleanup(qiv);
qom/qom-qobject.c +1 -1
index b66088d..99666ce 100644
@@ -23,7 +23,7 @@ void object_property_set_qobject(Object *obj, QObject *value,
 {
     QmpInputVisitor *qiv;
     /* TODO: Should we reject, rather than ignore, excess input? */
-    qiv = qmp_input_visitor_new(value, false);
+    qiv = qmp_input_visitor_new(value, false, false);
     object_property_set(obj, qmp_input_get_visitor(qiv), name, errp);
 
     qmp_input_visitor_cleanup(qiv);
replay/replay-input.c +1 -1
index 03e99d5..de82a59 100644
@@ -37,7 +37,7 @@ static InputEvent *qapi_clone_InputEvent(InputEvent *src)
         return NULL;
     }
 
-    qiv = qmp_input_visitor_new(obj, true);
+    qiv = qmp_input_visitor_new(obj, true, false);
     iv = qmp_input_get_visitor(qiv);
     visit_type_InputEvent(iv, NULL, &dst, &error_abort);
     qmp_input_visitor_cleanup(qiv);
scripts/qapi-commands.py +1 -1
index 8c6acb3..e48995d 100644
@@ -115,7 +115,7 @@ def gen_marshal(name, arg_type, ret_type):
 
     if arg_type and arg_type.members:
         ret += mcgen('''
-    QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true);
+    QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true, false);
     QapiDeallocVisitor *qdv;
     Visitor *v;
     %(c_name)s arg = {0};
tests/check-qnull.c +1 -1
index fd9c68f..4c11755 100644
@@ -49,7 +49,7 @@ static void qnull_visit_test(void)
 
     g_assert(qnull_.refcnt == 1);
     obj = qnull();
-    qiv = qmp_input_visitor_new(obj, true);
+    qiv = qmp_input_visitor_new(obj, true, false);
     qobject_decref(obj);
     visit_type_null(qmp_input_get_visitor(qiv), NULL, &error_abort);
     qmp_input_visitor_cleanup(qiv);
tests/test-qmp-commands.c +1 -1
index 5c3edd7..c86b282 100644
@@ -222,7 +222,7 @@ static void test_dealloc_partial(void)
         ud2_dict = qdict_new();
         qdict_put_obj(ud2_dict, "string0", QOBJECT(qstring_from_str(text)));
 
-        qiv = qmp_input_visitor_new(QOBJECT(ud2_dict), true);
+        qiv = qmp_input_visitor_new(QOBJECT(ud2_dict), true, false);
         visit_type_UserDefTwo(qmp_input_get_visitor(qiv), NULL, &ud2, &err);
         qmp_input_visitor_cleanup(qiv);
         QDECREF(ud2_dict);
tests/test-qmp-input-strict.c +1 -1
index 4602529..f7f1f00 100644
@@ -55,7 +55,7 @@ static Visitor *validate_test_init_internal(TestInputVisitorData *data,
     data->obj = qobject_from_jsonv(json_string, ap);
     g_assert(data->obj);
 
-    data->qiv = qmp_input_visitor_new(data->obj, true);
+    data->qiv = qmp_input_visitor_new(data->obj, true, false);
     g_assert(data->qiv);
 
     v = qmp_input_get_visitor(data->qiv);
tests/test-qmp-input-visitor.c +112 -3
index cee07ce..5691dc3 100644
@@ -41,6 +41,7 @@ static void visitor_input_teardown(TestInputVisitorData *data,
    function so that the JSON string used by the tests are kept in the test
    functions (and not in main()). */
 static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data,
+                                                 bool strict, bool autocast,
                                                  const char *json_string,
                                                  va_list *ap)
 {
@@ -51,7 +52,7 @@ static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data,
     data->obj = qobject_from_jsonv(json_string, ap);
     g_assert(data->obj);
 
-    data->qiv = qmp_input_visitor_new(data->obj, false);
+    data->qiv = qmp_input_visitor_new(data->obj, strict, autocast);
     g_assert(data->qiv);
 
     v = qmp_input_get_visitor(data->qiv);
@@ -60,6 +61,21 @@ static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data,
     return v;
 }
 
+static GCC_FMT_ATTR(4, 5)
+Visitor *visitor_input_test_init_full(TestInputVisitorData *data,
+                                      bool strict, bool autocast,
+                                      const char *json_string, ...)
+{
+    Visitor *v;
+    va_list ap;
+
+    va_start(ap, json_string);
+    v = visitor_input_test_init_internal(data, strict, autocast,
+                                         json_string, &ap);
+    va_end(ap);
+    return v;
+}
+
 static GCC_FMT_ATTR(2, 3)
 Visitor *visitor_input_test_init(TestInputVisitorData *data,
                                  const char *json_string, ...)
@@ -68,7 +84,8 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,
     va_list ap;
 
     va_start(ap, json_string);
-    v = visitor_input_test_init_internal(data, json_string, &ap);
+    v = visitor_input_test_init_internal(data, false, false,
+                                         json_string, &ap);
     va_end(ap);
     return v;
 }
@@ -83,7 +100,8 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,
 static Visitor *visitor_input_test_init_raw(TestInputVisitorData *data,
                                             const char *json_string)
 {
-    return visitor_input_test_init_internal(data, json_string, NULL);
+    return visitor_input_test_init_internal(data, false, false,
+                                            json_string, NULL);
 }
 
 static void test_visitor_in_int(TestInputVisitorData *data,
@@ -115,6 +133,33 @@ static void test_visitor_in_int_overflow(TestInputVisitorData *data,
     error_free_or_abort(&err);
 }
 
+static void test_visitor_in_int_autocast(TestInputVisitorData *data,
+                                         const void *unused)
+{
+    int64_t res = 0, value = -42;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, false, true,
+                                     "\"-42\"");
+
+    visit_type_int(v, NULL, &res, &error_abort);
+    g_assert_cmpint(res, ==, value);
+}
+
+static void test_visitor_in_int_noautocast(TestInputVisitorData *data,
+                                           const void *unused)
+{
+    int64_t res = 0;
+    Visitor *v;
+    Error *err = NULL;
+
+    v = visitor_input_test_init(data, "\"-42\"");
+
+    visit_type_int(v, NULL, &res, &err);
+    g_assert(err != NULL);
+    error_free(err);
+}
+
 static void test_visitor_in_bool(TestInputVisitorData *data,
                                  const void *unused)
 {
@@ -127,6 +172,32 @@ static void test_visitor_in_bool(TestInputVisitorData *data,
     g_assert_cmpint(res, ==, true);
 }
 
+static void test_visitor_in_bool_autocast(TestInputVisitorData *data,
+                                          const void *unused)
+{
+    bool res = false;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, false, true, "\"true\"");
+
+    visit_type_bool(v, NULL, &res, &error_abort);
+    g_assert_cmpint(res, ==, true);
+}
+
+static void test_visitor_in_bool_noautocast(TestInputVisitorData *data,
+                                          const void *unused)
+{
+    bool res = false;
+    Visitor *v;
+    Error *err = NULL;
+
+    v = visitor_input_test_init(data, "\"true\"");
+
+    visit_type_bool(v, NULL, &res, &err);
+    g_assert(err != NULL);
+    error_free(err);
+}
+
 static void test_visitor_in_number(TestInputVisitorData *data,
                                    const void *unused)
 {
@@ -139,6 +210,32 @@ static void test_visitor_in_number(TestInputVisitorData *data,
     g_assert_cmpfloat(res, ==, value);
 }
 
+static void test_visitor_in_number_autocast(TestInputVisitorData *data,
+                                            const void *unused)
+{
+    double res = 0, value = 3.14;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, false, true, "\"3.14\"");
+
+    visit_type_number(v, NULL, &res, &error_abort);
+    g_assert_cmpfloat(res, ==, value);
+}
+
+static void test_visitor_in_number_noautocast(TestInputVisitorData *data,
+                                              const void *unused)
+{
+    double res = 0;
+    Visitor *v;
+    Error *err = NULL;
+
+    v = visitor_input_test_init(data, "\"3.14\"");
+
+    visit_type_number(v, NULL, &res, &err);
+    g_assert(err != NULL);
+    error_free(err);
+}
+
 static void test_visitor_in_string(TestInputVisitorData *data,
                                    const void *unused)
 {
@@ -835,10 +932,22 @@ int main(int argc, char **argv)
                            &in_visitor_data, test_visitor_in_int);
     input_visitor_test_add("/visitor/input/int_overflow",
                            &in_visitor_data, test_visitor_in_int_overflow);
+    input_visitor_test_add("/visitor/input/int_autocast",
+                           &in_visitor_data, test_visitor_in_int_autocast);
+    input_visitor_test_add("/visitor/input/int_noautocast",
+                           &in_visitor_data, test_visitor_in_int_noautocast);
     input_visitor_test_add("/visitor/input/bool",
                            &in_visitor_data, test_visitor_in_bool);
+    input_visitor_test_add("/visitor/input/bool_autocast",
+                           &in_visitor_data, test_visitor_in_bool_autocast);
+    input_visitor_test_add("/visitor/input/bool_noautocast",
+                           &in_visitor_data, test_visitor_in_bool_noautocast);
     input_visitor_test_add("/visitor/input/number",
                            &in_visitor_data, test_visitor_in_number);
+    input_visitor_test_add("/visitor/input/number_autocast",
+                           &in_visitor_data, test_visitor_in_number_autocast);
+    input_visitor_test_add("/visitor/input/number_noautocast",
+                           &in_visitor_data, test_visitor_in_number_noautocast);
     input_visitor_test_add("/visitor/input/string",
                            &in_visitor_data, test_visitor_in_string);
     input_visitor_test_add("/visitor/input/enum",
tests/test-visitor-serialization.c +1 -1
index 7b14b5a..db618d8 100644
@@ -1038,7 +1038,7 @@ static void qmp_deserialize(void **native_out, void *datap,
     obj = qobject_from_json(qstring_get_str(output_json));
 
     QDECREF(output_json);
-    d->qiv = qmp_input_visitor_new(obj, true);
+    d->qiv = qmp_input_visitor_new(obj, true, false);
     qobject_decref(obj_orig);
     qobject_decref(obj);
     visit(qmp_input_get_visitor(d->qiv), native_out, errp);
util/qemu-sockets.c +1 -1
index 0d6cd1f..51c6a8e 100644
@@ -1145,7 +1145,7 @@ void qapi_copy_SocketAddress(SocketAddress **p_dest,
         return;
     }
 
-    qiv = qmp_input_visitor_new(obj, true);
+    qiv = qmp_input_visitor_new(obj, true, false);
     iv = qmp_input_get_visitor(qiv);
     visit_type_SocketAddress(iv, NULL, p_dest, &error_abort);
     qmp_input_visitor_cleanup(qiv);
#24 lore raw
In reply to Daniel P. Berrange
Quoted 5 lines · Daniel P. Berrange

Ping, does anyone have any feedback on these first 3 patches in particular. If we get agreement on those and get them merged, then we can feed the rest of this series to the individual subsystem maintainers that are affected, and also unblock Max's patch series.

I'll try to get to them this week.

#2 lore raw

Re: [Qemu-devel] [PATCH v5 01/11] qdict: implement a qdict_crumple method for un-flattening a dict

In reply to Daniel P. Berrange

I apologize for the lateness of this review.

Quoted 54 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }
Quoted 5 lines

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

Not an objection to your patch, just an observation: we're digging ourselves ever deeper into the QemuOpts / QDict hole. We've pushed QemuOpts far beyond its original purpose "parse key=value,... option arguments". In my opinion, we'd better parse straight to QAPI-generated structs.

Quoted 43 lines
---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 282 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 229 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 512 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..8a3ac13 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +10 -0
index 60f158c..ad6a563 100644
@@ -17,6 +17,7 @@
 #include "qapi/qmp/qbool.h"
 #include "qapi/qmp/qstring.h"
 #include "qapi/qmp/qobject.h"
+#include "qapi/error.h"
 #include "qemu/queue.h"
 #include "qemu-common.h"
 #include "qemu/cutils.h"
@@ -683,6 +684,287 @@ void qdict_array_split(QDict *src, QList **dst)
     }
 }
 
+
+/**
+ * qdict_split_flat_key:
+ * @key: the key string to split
+ * @prefix: non-NULL pointer to hold extracted prefix
+ * @suffix: non-NULL pointer to hold extracted suffix
+ *
+ * Given a flattened key such as 'foo.0.bar', split it
+ * into two parts at the first '.' separator. Allows

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 705

Quoted 10 lines
qobject/qdict.c +10 -0
+ * double dot ('..') to escape the normal separator.
+ *
+ * eg
+ *    'foo.0.bar' -> prefix='foo' and suffix='0.bar'
+ *    'foo..0.bar' -> prefix='foo.0' and suffix='bar'
+ *
+ * The '..' sequence will be unescaped in the returned
+ * 'prefix' string. The 'suffix' string will be left
+ * in escaped format, so it can be fed back into the
+ * qdict_split_flat_key() key as the input later.

Why is the suffix strdup'ed then?

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 708

Quoted 3 lines
qobject/qdict.c +3 -0
+ *
+ * The caller is responsible for freeing the strings
+ * returned in @prefix and @suffix using g_free().

Wrap comment lines around column 70, please.

Quoted 26 lines
qobject/qdict.c +26 -0
+ */
+static void qdict_split_flat_key(const char *key, char **prefix, char **suffix)
+{
+    const char *separator;
+    size_t i, j;
+
+    /* Find first '.' separator, but if there is a pair '..'
+     * that acts as an escape, so skip over '..' */
+    separator = NULL;
+    do {
+        if (separator) {
+            separator += 2;
+        } else {
+            separator = key;
+        }
+        separator = strchr(separator, '.');
+    } while (separator && separator[1] == '.');
+
+    if (separator) {
+        *prefix = g_strndup(key,
+                            separator - key);
+        *suffix = g_strdup(separator + 1);
+    } else {
+        *prefix = g_strdup(key);
+        *suffix = NULL;
+    }

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 744

Quoted 10 lines
qobject/qdict.c +10 -0
+
+    /* Unescape the '..' sequence into '.' */
+    for (i = 0, j = 0; (*prefix)[i] != '\0'; i++, j++) {
+        if ((*prefix)[i] == '.') {
+            assert((*prefix)[i + 1] == '.');
+            i++;
+        }
+        (*prefix)[j] = (*prefix)[i];
+    }
+    (*prefix)[j] = '\0';

Runs through the key twice, once in g_strndup(), and once here. Would be easy enough to avoid, but since this function isn't performance critical, it's okay as is.

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 750

Quoted 6 lines
qobject/qdict.c +6 -0
+}
+
+
+/**
+ * qdict_list_size:
+ * @maybe_list: dict that may be only list elements

Huh? How can a dictionary "be only list elements"?

Do you mean "the dictionary to test?"

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 758

Quoted 8 lines
qobject/qdict.c +8 -0
+ *
+ * Determine whether all keys in @maybe_list are
+ * valid list elements. If they are all valid,
+ * then this returns the number of elements. If
+ * they all look like non-numeric keys, then returns
+ * zero. If there is a mix of numeric and non-numeric
+ * keys, then an error is set as it is both a list
+ * and a dict at once.

This is well-defined only if empty @maybe_list is considered to have dict nature, not list nature. Else, return value zero could be the length of the empty list or the special "has dict nature" value.

Please spell out behavior for empty @maybe_list.

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 760

Quoted 2 lines
qobject/qdict.c +2 -0
+ *
+ * Returns: number of list elements, 0 if a dict, -1 on error

Awkward function name. qdict_list_size_if_list() would be clear.

But I'd simply turn this into a predicate qdict_is_list(), and have the caller use qdict_size() to get the number of elements.

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 772

Quoted 12 lines
qobject/qdict.c +12 -0
+ */
+static ssize_t qdict_list_size(QDict *maybe_list, Error **errp)
+{
+    const QDictEntry *entry, *next;
+    ssize_t len = 0;
+    ssize_t max = -1;
+    int is_list = -1;
+    int64_t val;
+
+    entry = qdict_first(maybe_list);
+    while (entry != NULL) {
+        next = qdict_next(maybe_list, entry);

Please keep the loop control in one place:

       for (entry = qdict_first(maybe_list); entry; entry = qdict_next(entry)) {

I'd rename some variables for less verbiage:

       for (ent = qdict_first(dict); ent; ent = qdict_next(ent)) {

Your loop control also works when the loop body deletes @entry from @maybe_list. Seeing such loop control in a function that isn't supposed to change the its argument makes the reviewer go "WTF?!?" :)

Quoted 20 lines
qobject/qdict.c +20 -0
+
+        if (qemu_strtoll(entry->key, NULL, 10, &val) == 0) {
+            if (is_list == -1) {
+                is_list = 1;
+            } else if (!is_list) {
+                error_setg(errp,
+                           "Cannot crumple a dictionary with a mix of list "
+                           "and non-list keys");
+                return -1;
+            }
+            len++;
+            if (val > max) {
+                max = val;
+            }
+        } else {
+            if (is_list == -1) {
+                is_list = 0;
+            } else if (is_list) {
+                error_setg(errp,
+                           "Cannot crumple a dictionary with a mix of list "

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 802

Quoted 10 lines
qobject/qdict.c +10 -0
+                           "and non-list keys");
+                return -1;
+            }
+        }
+
+        entry = next;
+    }
+
+    if (is_list == -1) {
+        is_list = 0;

This can happen only when @maybe_list is empty. Okay, but perhaps you'd like to assert(!qdict_size(maybe_list)).

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 809

Quoted 7 lines
qobject/qdict.c +7 -0
+    }
+
+    if (len != (max + 1)) {
+        error_setg(errp, "List indexes are not continuous, "
+                   "saw %zd elements but %zd largest index",
+                   len, max);
+        return -1;

contiguous?

What if we saw indexes 0, 2, 2?

Hmm, see [*] below.

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 817

Quoted 8 lines
qobject/qdict.c +8 -0
+    }
+
+    return is_list ? len : 0;
+}
+
+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary to crumple

"Flat" means all values are scalar. Should we spell that out?

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 821

Quoted 4 lines
qobject/qdict.c +4 -0
+ * @recursive: true to recursively crumple nested dictionaries
+ *
+ * Takes a flat dictionary whose keys use '.' separator to
+ * indicate nesting, and values are scalars, crumplings it

s/, crumplings/, and crumples/

Quoted 14 lines
qobject/qdict.c +14 -0
+ * into a nested structure. If the @recursive parameter is
+ * false, then only the first level of structure implied
+ * by the keys will be crumpled. If @recursive is true,
+ * then the input will be recursively crumpled  to expand
+ * all levels of structure in the keys.
+ *
+ * To include a literal '.' in a key name, it must be escaped
+ * as '..'
+ *
+ * For example, an input of:
+ *
+ * { 'foo.0.bar': 'one', 'foo.0.wizz': '1',
+ *   'foo.1.bar': 'two', 'foo.1.wizz': '2' }
+ *

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 845

Quoted 10 lines
qobject/qdict.c +10 -0
+ * will result in any output of:
+ *
+ * {
+ *   'foo': [
+ *      { 'bar': 'one', 'wizz': '1' },
+ *      { 'bar': 'two', 'wizz': '2' }
+ *   ],
+ * }
+ *
+ * Returns: either a QDict or QList for the nested data structure

I think you should discuss how this can fail.

Quoted 6 lines
qobject/qdict.c +6 -0
+ */
+QObject *qdict_crumple(QDict *src, bool recursive, Error **errp)
+{
+    const QDictEntry *entry, *next;
+    QDict *two_level, *multi_level = NULL;
+    QObject *dst = NULL, *child;

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 861

Quoted 10 lines
qobject/qdict.c +10 -0
+    ssize_t list_len;
+    size_t i;
+    char *prefix = NULL, *suffix = NULL;
+
+    two_level = qdict_new();
+    entry = qdict_first(src);
+
+    /* Step 1: split our totally flat dict into a two level dict */
+    while (entry != NULL) {
+        next = qdict_next(src, entry);

Again, keep the loop control in one place.

Quoted 23 lines
qobject/qdict.c +23 -0
+
+        if (qobject_type(entry->value) == QTYPE_QDICT ||
+            qobject_type(entry->value) == QTYPE_QLIST) {
+            error_setg(errp, "Value %s is not a scalar",
+                       entry->key);
+            goto error;
+        }
+
+        qdict_split_flat_key(entry->key, &prefix, &suffix);
+
+        child = qdict_get(two_level, prefix);
+        if (suffix) {
+            if (child) {
+                if (qobject_type(child) != QTYPE_QDICT) {
+                    error_setg(errp, "Key %s prefix is already set as a scalar",
+                               prefix);
+                    goto error;
+                }
+            } else {
+                child = QOBJECT(qdict_new());
+                qdict_put_obj(two_level, prefix, child);
+            }
+            qobject_incref(entry->value);

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 894

Quoted 10 lines
qobject/qdict.c +10 -0
+            qdict_put_obj(qobject_to_qdict(child), suffix, entry->value);
+        } else {
+            if (child) {
+                error_setg(errp, "Key %s prefix is already set as a dict",
+                           prefix);
+                goto error;
+            }
+            qobject_incref(entry->value);
+            qdict_put_obj(two_level, prefix, entry->value);
+        }

Works, because we put only QDicts we've created ourselves (first qdict_put_obj() above) and values we got from @src (second qdict_put_obj()), and we fail when such a value isn't scalar.

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 896

Quoted 2 lines
qobject/qdict.c +2 -0
+
+        g_free(suffix);

As I suspected, qdict_split_flat_key() strdup'ing the suffix is useless.

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 898

Quoted 2 lines
qobject/qdict.c +2 -0
+        g_free(prefix);
+        suffix = prefix = NULL;

Dead stores.

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 909

Quoted 11 lines
qobject/qdict.c +11 -0
+
+        entry = next;
+    }
+
+    /* Step 2: optionally process the two level dict recursively
+     * into a multi-level dict */
+    if (recursive) {
+        multi_level = qdict_new();
+        entry = qdict_first(two_level);
+        while (entry != NULL) {
+            next = qdict_next(two_level, entry);

Again, keep the loop control in one place.

Quoted 27 lines
qobject/qdict.c +27 -0
+
+            if (qobject_type(entry->value) == QTYPE_QDICT) {
+                child = qdict_crumple(qobject_to_qdict(entry->value),
+                                      recursive, errp);
+                if (!child) {
+                    goto error;
+                }
+
+                qdict_put_obj(multi_level, entry->key, child);
+            } else {
+                qobject_incref(entry->value);
+                qdict_put_obj(multi_level, entry->key, entry->value);
+            }
+
+            entry = next;
+        }
+        QDECREF(two_level);
+    } else {
+        multi_level = two_level;
+    }
+    two_level = NULL;
+
+    /* Step 3: detect if we need to turn our dict into list */
+    list_len = qdict_list_size(multi_level, errp);
+    if (list_len < 0) {
+        goto error;
+    }

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 946

Quoted 10 lines
qobject/qdict.c +10 -0
+
+    if (list_len) {
+        dst = QOBJECT(qlist_new());
+
+        for (i = 0; i < list_len; i++) {
+            char *key = g_strdup_printf("%zu", i);
+
+            child = qdict_get(multi_level, key);
+            g_free(key);
+            assert(child);

qdict_list_size() accepts as list index any (string) key qemu_strtoll() accepts. If %zu formats it back into the same string, we'll find it here. Else we die. Please spell this out in the function contract.

[*] I'm afraid we also die if qdict_list_size()'s "List indexes are not continuous" check gets fooled. Suggest to drop that check, and replace this assertion by error_setg(errp, "Malformed list indexes"). Admittedly not the nicest error message; perhaps you can come up with a better one.

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 951

Quoted 5 lines
qobject/qdict.c +5 -0
+
+            qobject_incref(child);
+            qlist_append_obj(qobject_to_qlist(dst), child);
+        }
+        QDECREF(multi_level);

Do we need

           multi_level = NULL;

here?

Quoted 240 lines
qobject/qdict.c +16 -0
+    } else {
+        dst = QOBJECT(multi_level);
+    }
+
+    return dst;
+
+ error:
+    g_free(suffix);
+    g_free(prefix);
+    QDECREF(multi_level);
+    QDECREF(two_level);
+    qobject_decref(dst);
+    return NULL;
+}
+
+
 /**
  * qdict_array_entries(): Returns the number of direct array entries if the
  * sub-QDict of src specified by the prefix in subqdict (or src itself for
tests/check-qdict.c +206 -0
index a43056c..0d12f40 100644
@@ -15,6 +15,7 @@
 #include "qapi/qmp/qint.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
+#include "qapi/error.h"
 #include "qemu-common.h"
 
 /*
@@ -596,6 +597,223 @@ static void qdict_join_test(void)
     QDECREF(dict2);
 }
 
+
+static void qdict_crumple_test_nonrecursive(void)
+{
+    QDict *src, *dst, *rules, *vnc;
+    QObject *child, *res;
+
+    src = qdict_new();
+    qdict_put(src, "vnc.listen.addr", qstring_from_str("127.0.0.1"));
+    qdict_put(src, "vnc.listen.port", qstring_from_str("5901"));
+    qdict_put(src, "rule.0.match", qstring_from_str("fred"));
+    qdict_put(src, "rule.0.policy", qstring_from_str("allow"));
+    qdict_put(src, "rule.1.match", qstring_from_str("bob"));
+    qdict_put(src, "rule.1.policy", qstring_from_str("deny"));
+
+    res = qdict_crumple(src, false, &error_abort);
+
+    g_assert_cmpint(qobject_type(res), ==, QTYPE_QDICT);
+
+    dst = qobject_to_qdict(res);
+
+    g_assert_cmpint(qdict_size(dst), ==, 2);
+
+    child = qdict_get(dst, "vnc");
+    g_assert_cmpint(qobject_type(child), ==, QTYPE_QDICT);
+    vnc = qdict_get_qdict(dst, "vnc");
+
+    g_assert_cmpint(qdict_size(vnc), ==, 2);
+
+    g_assert_cmpstr("127.0.0.1", ==, qdict_get_str(vnc, "listen.addr"));
+    g_assert_cmpstr("5901", ==, qdict_get_str(vnc, "listen.port"));
+
+    child = qdict_get(dst, "rule");
+    g_assert_cmpint(qobject_type(child), ==, QTYPE_QDICT);
+    rules = qdict_get_qdict(dst, "rule");
+
+    g_assert_cmpint(qdict_size(rules), ==, 4);
+
+    g_assert_cmpstr("fred", ==, qdict_get_str(rules, "0.match"));
+    g_assert_cmpstr("allow", ==, qdict_get_str(rules, "0.policy"));
+    g_assert_cmpstr("bob", ==, qdict_get_str(rules, "1.match"));
+    g_assert_cmpstr("deny", ==, qdict_get_str(rules, "1.policy"));
+
+    QDECREF(src);
+    QDECREF(dst);
+}
+
+
+static void qdict_crumple_test_listtop(void)
+{
+    QDict *src, *rule;
+    QList *rules;
+    QObject *res;
+
+    src = qdict_new();
+    qdict_put(src, "0.match.name", qstring_from_str("Fred"));
+    qdict_put(src, "0.match.email", qstring_from_str("[email protected]"));
+    qdict_put(src, "0.policy", qstring_from_str("allow"));
+    qdict_put(src, "1.match.name", qstring_from_str("Bob"));
+    qdict_put(src, "1.match.email", qstring_from_str("[email protected]"));
+    qdict_put(src, "1.policy", qstring_from_str("deny"));
+
+    res = qdict_crumple(src, false, &error_abort);
+
+    g_assert_cmpint(qobject_type(res), ==, QTYPE_QLIST);
+
+    rules = qobject_to_qlist(res);
+
+    g_assert_cmpint(qlist_size(rules), ==, 2);
+
+    g_assert_cmpint(qobject_type(qlist_peek(rules)), ==, QTYPE_QDICT);
+    rule = qobject_to_qdict(qlist_pop(rules));
+    g_assert_cmpint(qdict_size(rule), ==, 3);
+
+    g_assert_cmpstr("Fred", ==, qdict_get_str(rule, "match.name"));
+    g_assert_cmpstr("[email protected]", ==, qdict_get_str(rule, "match.email"));
+    g_assert_cmpstr("allow", ==, qdict_get_str(rule, "policy"));
+    QDECREF(rule);
+
+    g_assert_cmpint(qobject_type(qlist_peek(rules)), ==, QTYPE_QDICT);
+    rule = qobject_to_qdict(qlist_pop(rules));
+    g_assert_cmpint(qdict_size(rule), ==, 3);
+
+    g_assert_cmpstr("Bob", ==, qdict_get_str(rule, "match.name"));
+    g_assert_cmpstr("[email protected]", ==, qdict_get_str(rule, "match.email"));
+    g_assert_cmpstr("deny", ==, qdict_get_str(rule, "policy"));
+    QDECREF(rule);
+
+    QDECREF(src);
+    qobject_decref(res);
+}
+
+
+static void qdict_crumple_test_recursive(void)
+{
+    QDict *src, *dst, *rule, *vnc, *acl, *listen;
+    QObject *child, *res;
+    QList *rules;
+
+    src = qdict_new();
+    qdict_put(src, "vnc.listen.addr", qstring_from_str("127.0.0.1"));
+    qdict_put(src, "vnc.listen.port", qstring_from_str("5901"));
+    qdict_put(src, "vnc.acl.rules.0.match", qstring_from_str("fred"));
+    qdict_put(src, "vnc.acl.rules.0.policy", qstring_from_str("allow"));
+    qdict_put(src, "vnc.acl.rules.1.match", qstring_from_str("bob"));
+    qdict_put(src, "vnc.acl.rules.1.policy", qstring_from_str("deny"));
+    qdict_put(src, "vnc.acl.default", qstring_from_str("deny"));
+
+    res = qdict_crumple(src, true, &error_abort);
+
+    g_assert_cmpint(qobject_type(res), ==, QTYPE_QDICT);
+
+    dst = qobject_to_qdict(res);
+
+    g_assert_cmpint(qdict_size(dst), ==, 1);
+
+    child = qdict_get(dst, "vnc");
+    g_assert_cmpint(qobject_type(child), ==, QTYPE_QDICT);
+    vnc = qobject_to_qdict(child);
+
+    child = qdict_get(vnc, "listen");
+    g_assert_cmpint(qobject_type(child), ==, QTYPE_QDICT);
+    listen = qobject_to_qdict(child);
+    g_assert_cmpstr("127.0.0.1", ==, qdict_get_str(listen, "addr"));
+    g_assert_cmpstr("5901", ==, qdict_get_str(listen, "port"));
+
+    child = qdict_get(vnc, "acl");
+    g_assert_cmpint(qobject_type(child), ==, QTYPE_QDICT);
+    acl = qobject_to_qdict(child);
+
+    child = qdict_get(acl, "rules");
+    g_assert_cmpint(qobject_type(child), ==, QTYPE_QLIST);
+    rules = qobject_to_qlist(child);
+    g_assert_cmpint(qlist_size(rules), ==, 2);
+
+    rule = qobject_to_qdict(qlist_pop(rules));
+    g_assert_cmpint(qdict_size(rule), ==, 2);
+    g_assert_cmpstr("fred", ==, qdict_get_str(rule, "match"));
+    g_assert_cmpstr("allow", ==, qdict_get_str(rule, "policy"));
+    QDECREF(rule);
+
+    rule = qobject_to_qdict(qlist_pop(rules));
+    g_assert_cmpint(qdict_size(rule), ==, 2);
+    g_assert_cmpstr("bob", ==, qdict_get_str(rule, "match"));
+    g_assert_cmpstr("deny", ==, qdict_get_str(rule, "policy"));
+    QDECREF(rule);
+
+    QDECREF(src);
+    QDECREF(dst);
+}
+
+
+static void qdict_crumple_test_empty(void)
+{
+    QDict *src, *dst;
+
+    src = qdict_new();
+
+    dst = (QDict *)qdict_crumple(src, true, &error_abort);
+
+    g_assert_cmpint(qdict_size(dst), ==, 0);
+
+    QDECREF(src);
+    QDECREF(dst);
+}
+
+
+static void qdict_crumple_test_bad_inputs(void)
+{
+    QDict *src;
+    Error *error = NULL;
+
+    src = qdict_new();
+    /* rule.0 can't be both a string and a dict */
+    qdict_put(src, "rule.0", qstring_from_str("fred"));
+    qdict_put(src, "rule.0.policy", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);
+
+    src = qdict_new();
+    /* rule can't be both a list and a dict */
+    qdict_put(src, "rule.0", qstring_from_str("fred"));
+    qdict_put(src, "rule.a", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);
+
+    src = qdict_new();
+    /* The input should be flat, ie no dicts or lists */
+    qdict_put(src, "rule.a", qdict_new());
+    qdict_put(src, "rule.b", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);
+
+

tests/check-qdict.c static void qdict_join_test(void) · line 814

Quoted 10 lines
tests/check-qdict.c +10 -0
+    src = qdict_new();
+    /* List indexes must not have gaps */
+    qdict_put(src, "rule.0", qdict_new());
+    qdict_put(src, "rule.3", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);

Suggest to add test case

       /* List indexes must not have gaps (more creative version) */
       qdict_put(src, "rule.0", ...);
       qdict_put(src, "rule.2", ...);
       qdict_put(src, "rule.2", ...);

and

       /* List indexes must be in %zu format */
       qdict_put(src, "rule.+0", ...);
Quoted 23 lines
tests/check-qdict.c +13 -0
+}
+
 /*
  * Errors test-cases
  */
@@ -743,6 +961,17 @@ int main(int argc, char **argv)
     g_test_add_func("/errors/put_exists", qdict_put_exists_test);
     g_test_add_func("/errors/get_not_exists", qdict_get_not_exists_test);
 
+    g_test_add_func("/public/crumple/nonrecursive",
+                    qdict_crumple_test_nonrecursive);
+    g_test_add_func("/public/crumple/recursive",
+                    qdict_crumple_test_recursive);
+    g_test_add_func("/public/crumple/listtop",
+                    qdict_crumple_test_listtop);
+    g_test_add_func("/public/crumple/empty",
+                    qdict_crumple_test_empty);
+    g_test_add_func("/public/crumple/bad_inputs",
+                    qdict_crumple_test_bad_inputs);
+
     /* The Big one */
     if (g_test_slow()) {
         g_test_add_func("/stress/test", qdict_stress_test);
#3 lore raw

Re: [Qemu-devel] [PATCH v5 01/11] qdict: implement a qdict_crumple method for un-flattening a dict

Quoted 62 lines · Markus Armbruster

I apologize for the lateness of this review.

Quoted 58 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

Quoted 7 lines
Quoted 1 line

Not an objection to your patch, just an observation: we're digging ourselves ever deeper into the QemuOpts / QDict hole. We've pushed QemuOpts far beyond its original purpose "parse key=value,... option arguments". In my opinion, we'd better parse straight to QAPI-generated structs.

NB we're not actually digging the hole any deeper here. The BlockDev code already fully dug the hole. This code is just generalizing the code for dealing with the hole (intentionally designed to 100% mirror the syntax that BlockDev already defined), so that we get consistent handling across the codebase, rather than having many separate subtley different holes :-) I have patches that I've not posted yet, which start to convert some of the blockdev code over to use this method.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#9 lore raw

Re: [Qemu-devel] [PATCH v5 02/11] qapi: allow QmpInputVisitor to auto-cast types

In reply to Daniel P. Berrange
Quoted 5 lines · Daniel P. Berrange

Currently the QmpInputVisitor assumes that all scalar values are directly represented as their final types. ie it assumes an 'int' is using QInt, and a 'bool' is using QBool.

Quoted 8 lines

This extends it so that QString is optionally permitted for any of the non-string scalar types. This behaviour is turned on by requesting the 'autocast' flag in the constructor.

This makes it possible to use QmpInputVisitor with a QDict produced from QemuOpts, where everything is in string format.

We're still digging.

Quoted 54 lines
---
 docs/qapi-code-gen.txt             |   2 +-
 include/qapi/qmp-input-visitor.h   |   6 +-
 qapi/opts-visitor.c                |   1 +
 qapi/qmp-input-visitor.c           |  89 ++++++++++++++++++++++------
 qmp.c                              |   2 +-
 qom/qom-qobject.c                  |   2 +-
 replay/replay-input.c              |   2 +-
 scripts/qapi-commands.py           |   2 +-
 tests/check-qnull.c                |   2 +-
 tests/test-qmp-commands.c          |   2 +-
 tests/test-qmp-input-strict.c      |   2 +-
 tests/test-qmp-input-visitor.c     | 115 ++++++++++++++++++++++++++++++++++++-
 tests/test-visitor-serialization.c |   2 +-
 util/qemu-sockets.c                |   2 +-
 14 files changed, 201 insertions(+), 30 deletions(-)
docs/qapi-code-gen.txt +1 -1
index d7d6987..e21773e 100644
@@ -1008,7 +1008,7 @@ Example:
     {
         Error *err = NULL;
         UserDefOne *retval;
-        QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true);
+        QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true, false);
         QapiDeallocVisitor *qdv;
         Visitor *v;
         UserDefOneList *arg1 = NULL;
include/qapi/qmp-input-visitor.h +5 -1
index b0624d8..d35a053 100644
@@ -24,8 +24,12 @@ typedef struct QmpInputVisitor QmpInputVisitor;
  *
  * Set @strict to reject a parse that doesn't consume all keys of a
  * dictionary; otherwise excess input is ignored.
+ * Set @autocast to automatically convert string values into more
+ * specific types (numbers, bools, etc)
  */
-QmpInputVisitor *qmp_input_visitor_new(QObject *obj, bool strict);
+QmpInputVisitor *qmp_input_visitor_new(QObject *obj,
+                                       bool strict,
+                                       bool autocast);
 
 void qmp_input_visitor_cleanup(QmpInputVisitor *v);
 
qapi/opts-visitor.c +0 -0
index 4cf1cf8..00e4b1a 100644

qapi/opts-visitor.c opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp) · line 350

Quoted 4 lines
qapi/opts-visitor.c +1 -0
@@ -347,6 +347,7 @@ opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp)
     }
 
     if (opt->str) {
+        /* Keep these values in sync with same code in qmp-input-visitor.c */

Also with parse_option_bool(). That's the canonical place, in fact.

Quoted 35 lines
qapi/opts-visitor.c +0 -0
         if (strcmp(opt->str, "on") == 0 ||
             strcmp(opt->str, "yes") == 0 ||
             strcmp(opt->str, "y") == 0) {
qapi/qmp-input-visitor.c +5 -4
index aea90a1..92023b1 100644
@@ -20,6 +20,7 @@
 #include "qemu-common.h"
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/qerror.h"
+#include "qemu/cutils.h"
 
 #define QIV_STACK_SIZE 1024
 
@@ -45,6 +46,7 @@ struct QmpInputVisitor
 
     /* True to reject parse in visit_end_struct() if unvisited keys remain. */
     bool strict;
+    bool autocast;
 };
 
 static QmpInputVisitor *to_qiv(Visitor *v)
@@ -254,15 +256,25 @@ static void qmp_input_type_int64(Visitor *v, const char *name, int64_t *obj,
                                  Error **errp)
 {
     QmpInputVisitor *qiv = to_qiv(v);
-    QInt *qint = qobject_to_qint(qmp_input_get_object(qiv, name, true));
+    QObject *qobj = qmp_input_get_object(qiv, name, true);
+    QInt *qint;
+    QString *qstr;
 
-    if (!qint) {
-        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
-                   "integer");

qapi/qmp-input-visitor.c static void qmp_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 271

Quoted 10 lines
qapi/qmp-input-visitor.c +6 -1
+    qint = qobject_to_qint(qobj);
+    if (qint) {
+        *obj = qint_get_int(qint);
         return;
     }
 
-    *obj = qint_get_int(qint);
+    qstr = qobject_to_qstring(qobj);
+    if (qstr && qstr->string && qiv->autocast) {
+        if (qemu_strtoll(qstr->string, NULL, 10, obj) == 0) {

In the commit message, you mentioned intended use: "with a QDict produced from QemuOpts, where everything is in string format." I figure you mean opts with empty opts->list->desc[]. For those, we accept any key and parse all values as strings.

The idea with such QemuOpts is to *delay* parsing until we know how to parse. The delay should be transparent to the user. In particular, values should be parsed the same whether the parsing is delayed or not.

The canonical QemuOpts value parser is qemu_opt_parse(). It parses integers with parse_option_number(). That function parses like stroull(qstr->string, ..., 0). Two differences:

* stroll() vs. strtoull(): they differ in ERANGE behavior. This might be tolerable, but I haven't thought it through.

* Base 0 vs 10: different syntax and semantics. Oops.

Quoted 21 lines
qapi/qmp-input-visitor.c +9 -4
+            return;
+        }
+    }
+
+    error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+               "integer");
 }
 
 static void qmp_input_type_uint64(Visitor *v, const char *name, uint64_t *obj,
@@ -270,30 +282,61 @@ static void qmp_input_type_uint64(Visitor *v, const char *name, uint64_t *obj,
 {
     /* FIXME: qobject_to_qint mishandles values over INT64_MAX */
     QmpInputVisitor *qiv = to_qiv(v);
-    QInt *qint = qobject_to_qint(qmp_input_get_object(qiv, name, true));
+    QObject *qobj = qmp_input_get_object(qiv, name, true);
+    QInt *qint;
+    QString *qstr;
 
-    if (!qint) {
-        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
-                   "integer");

qapi/qmp-input-visitor.c static void qmp_input_type_uint64(Visitor *v, const char *name, uint64_t *obj, · line 297

Quoted 10 lines
qapi/qmp-input-visitor.c +6 -1
+    qint = qobject_to_qint(qobj);
+    if (qint) {
+        *obj = qint_get_int(qint);
         return;
     }
 
-    *obj = qint_get_int(qint);
+    qstr = qobject_to_qstring(qobj);
+    if (qstr && qstr->string && qiv->autocast) {
+        if (qemu_strtoull(qstr->string, NULL, 10, obj) == 0) {

Same issue with base 0 vs. 10.

Quoted 21 lines
qapi/qmp-input-visitor.c +10 -4
+            return;
+        }
+    }
+
+    error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+               "integer");
 }
 
 static void qmp_input_type_bool(Visitor *v, const char *name, bool *obj,
                                 Error **errp)
 {
     QmpInputVisitor *qiv = to_qiv(v);
-    QBool *qbool = qobject_to_qbool(qmp_input_get_object(qiv, name, true));
+    QObject *qobj = qmp_input_get_object(qiv, name, true);
+    QBool *qbool;
+    QString *qstr;
 
-    if (!qbool) {
-        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
-                   "boolean");
+    qbool = qobject_to_qbool(qobj);

qapi/qmp-input-visitor.c static void qmp_input_type_uint64(Visitor *v, const char *name, uint64_t *obj, · line 323

Quoted 10 lines
qapi/qmp-input-visitor.c +6 -1
+    if (qbool) {
+        *obj = qbool_get_bool(qbool);
         return;
     }
 
-    *obj = qbool_get_bool(qbool);
+
+    qstr = qobject_to_qstring(qobj);
+    if (qstr && qstr->string && qiv->autocast) {
+        /* Keep these values in sync with same code in opts-visitor.c */

Also with parse_option_bool().

qapi/qmp-input-visitor.c static void qmp_input_type_uint64(Visitor *v, const char *name, uint64_t *obj, · line 335

Quoted 12 lines
qapi/qmp-input-visitor.c +12 -0
+        if (!strcasecmp(qstr->string, "on") ||
+            !strcasecmp(qstr->string, "yes") ||
+            !strcasecmp(qstr->string, "true")) {
+            *obj = true;
+            return;
+        }
+        if (!strcasecmp(qstr->string, "off") ||
+            !strcasecmp(qstr->string, "no") ||
+            !strcasecmp(qstr->string, "false")) {
+            *obj = false;
+            return;
+        }

You follow opts-visitor.c. It accepts more than parse_option_bool(). Glorious.

I guess the only way out is to add another onion dome to qemu-option.c and accept yes, no, true, false there as well.

Quoted 22 lines
qapi/qmp-input-visitor.c +8 -0
+    }
+
+    error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+               "boolean");
 }
 
 static void qmp_input_type_str(Visitor *v, const char *name, char **obj,
@@ -319,6 +362,8 @@ static void qmp_input_type_number(Visitor *v, const char *name, double *obj,
     QObject *qobj = qmp_input_get_object(qiv, name, true);
     QInt *qint;
     QFloat *qfloat;
+    QString *qstr;
+    char *endp;
 
     qint = qobject_to_qint(qobj);
     if (qint) {
@@ -332,6 +377,15 @@ static void qmp_input_type_number(Visitor *v, const char *name, double *obj,
         return;
     }
 
+    qstr = qobject_to_qstring(qobj);
+    if (qstr && qstr->string && qiv->autocast) {

qapi/qmp-input-visitor.c static void qmp_input_type_number(Visitor *v, const char *name, double *obj, · line 391

Quoted 10 lines
qapi/qmp-input-visitor.c +7 -0
+        errno = 0;
+        *obj = strtod(qstr->string, &endp);
+        if (errno == 0 && endp != qstr->string && *endp == '\0') {
+            return;
+        }
+    }
+
     error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
                "number");
 }

QemuOpts do not support floating-point numbers. Doesn't make accepting them here wrong.

Quoted 322 lines
qapi/qmp-input-visitor.c +4 -1
@@ -381,7 +435,9 @@ void qmp_input_visitor_cleanup(QmpInputVisitor *v)
     g_free(v);
 }
 
-QmpInputVisitor *qmp_input_visitor_new(QObject *obj, bool strict)
+QmpInputVisitor *qmp_input_visitor_new(QObject *obj,
+                                       bool strict,
+                                       bool autocast)
 {
     QmpInputVisitor *v;
 
@@ -404,6 +460,7 @@ QmpInputVisitor *qmp_input_visitor_new(QObject *obj, bool strict)
     v->visitor.type_null = qmp_input_type_null;
     v->visitor.optional = qmp_input_optional;
     v->strict = strict;
+    v->autocast = autocast;
 
     v->root = obj;
     qobject_incref(obj);
qmp.c +1 -1
index 3165f87..dcd0953 100644
@@ -665,7 +665,7 @@ void qmp_object_add(const char *type, const char *id,
         }
     }
 
-    qiv = qmp_input_visitor_new(props, true);
+    qiv = qmp_input_visitor_new(props, true, false);
     obj = user_creatable_add_type(type, id, pdict,
                                   qmp_input_get_visitor(qiv), errp);
     qmp_input_visitor_cleanup(qiv);
qom/qom-qobject.c +1 -1
index b66088d..99666ce 100644
@@ -23,7 +23,7 @@ void object_property_set_qobject(Object *obj, QObject *value,
 {
     QmpInputVisitor *qiv;
     /* TODO: Should we reject, rather than ignore, excess input? */
-    qiv = qmp_input_visitor_new(value, false);
+    qiv = qmp_input_visitor_new(value, false, false);
     object_property_set(obj, qmp_input_get_visitor(qiv), name, errp);
 
     qmp_input_visitor_cleanup(qiv);
replay/replay-input.c +1 -1
index 03e99d5..de82a59 100644
@@ -37,7 +37,7 @@ static InputEvent *qapi_clone_InputEvent(InputEvent *src)
         return NULL;
     }
 
-    qiv = qmp_input_visitor_new(obj, true);
+    qiv = qmp_input_visitor_new(obj, true, false);
     iv = qmp_input_get_visitor(qiv);
     visit_type_InputEvent(iv, NULL, &dst, &error_abort);
     qmp_input_visitor_cleanup(qiv);
scripts/qapi-commands.py +1 -1
index 8c6acb3..e48995d 100644
@@ -115,7 +115,7 @@ def gen_marshal(name, arg_type, ret_type):
 
     if arg_type and arg_type.members:
         ret += mcgen('''
-    QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true);
+    QmpInputVisitor *qiv = qmp_input_visitor_new(QOBJECT(args), true, false);
     QapiDeallocVisitor *qdv;
     Visitor *v;
     %(c_name)s arg = {0};
tests/check-qnull.c +1 -1
index fd9c68f..4c11755 100644
@@ -49,7 +49,7 @@ static void qnull_visit_test(void)
 
     g_assert(qnull_.refcnt == 1);
     obj = qnull();
-    qiv = qmp_input_visitor_new(obj, true);
+    qiv = qmp_input_visitor_new(obj, true, false);
     qobject_decref(obj);
     visit_type_null(qmp_input_get_visitor(qiv), NULL, &error_abort);
     qmp_input_visitor_cleanup(qiv);
tests/test-qmp-commands.c +1 -1
index 5c3edd7..c86b282 100644
@@ -222,7 +222,7 @@ static void test_dealloc_partial(void)
         ud2_dict = qdict_new();
         qdict_put_obj(ud2_dict, "string0", QOBJECT(qstring_from_str(text)));
 
-        qiv = qmp_input_visitor_new(QOBJECT(ud2_dict), true);
+        qiv = qmp_input_visitor_new(QOBJECT(ud2_dict), true, false);
         visit_type_UserDefTwo(qmp_input_get_visitor(qiv), NULL, &ud2, &err);
         qmp_input_visitor_cleanup(qiv);
         QDECREF(ud2_dict);
tests/test-qmp-input-strict.c +1 -1
index 4602529..f7f1f00 100644
@@ -55,7 +55,7 @@ static Visitor *validate_test_init_internal(TestInputVisitorData *data,
     data->obj = qobject_from_jsonv(json_string, ap);
     g_assert(data->obj);
 
-    data->qiv = qmp_input_visitor_new(data->obj, true);
+    data->qiv = qmp_input_visitor_new(data->obj, true, false);
     g_assert(data->qiv);
 
     v = qmp_input_get_visitor(data->qiv);
tests/test-qmp-input-visitor.c +112 -3
index cee07ce..5691dc3 100644
@@ -41,6 +41,7 @@ static void visitor_input_teardown(TestInputVisitorData *data,
    function so that the JSON string used by the tests are kept in the test
    functions (and not in main()). */
 static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data,
+                                                 bool strict, bool autocast,
                                                  const char *json_string,
                                                  va_list *ap)
 {
@@ -51,7 +52,7 @@ static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data,
     data->obj = qobject_from_jsonv(json_string, ap);
     g_assert(data->obj);
 
-    data->qiv = qmp_input_visitor_new(data->obj, false);
+    data->qiv = qmp_input_visitor_new(data->obj, strict, autocast);
     g_assert(data->qiv);
 
     v = qmp_input_get_visitor(data->qiv);
@@ -60,6 +61,21 @@ static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data,
     return v;
 }
 
+static GCC_FMT_ATTR(4, 5)
+Visitor *visitor_input_test_init_full(TestInputVisitorData *data,
+                                      bool strict, bool autocast,
+                                      const char *json_string, ...)
+{
+    Visitor *v;
+    va_list ap;
+
+    va_start(ap, json_string);
+    v = visitor_input_test_init_internal(data, strict, autocast,
+                                         json_string, &ap);
+    va_end(ap);
+    return v;
+}
+
 static GCC_FMT_ATTR(2, 3)
 Visitor *visitor_input_test_init(TestInputVisitorData *data,
                                  const char *json_string, ...)
@@ -68,7 +84,8 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,
     va_list ap;
 
     va_start(ap, json_string);
-    v = visitor_input_test_init_internal(data, json_string, &ap);
+    v = visitor_input_test_init_internal(data, false, false,
+                                         json_string, &ap);
     va_end(ap);
     return v;
 }
@@ -83,7 +100,8 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,
 static Visitor *visitor_input_test_init_raw(TestInputVisitorData *data,
                                             const char *json_string)
 {
-    return visitor_input_test_init_internal(data, json_string, NULL);
+    return visitor_input_test_init_internal(data, false, false,
+                                            json_string, NULL);
 }
 
 static void test_visitor_in_int(TestInputVisitorData *data,
@@ -115,6 +133,33 @@ static void test_visitor_in_int_overflow(TestInputVisitorData *data,
     error_free_or_abort(&err);
 }
 
+static void test_visitor_in_int_autocast(TestInputVisitorData *data,
+                                         const void *unused)
+{
+    int64_t res = 0, value = -42;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, false, true,
+                                     "\"-42\"");
+
+    visit_type_int(v, NULL, &res, &error_abort);
+    g_assert_cmpint(res, ==, value);
+}
+
+static void test_visitor_in_int_noautocast(TestInputVisitorData *data,
+                                           const void *unused)
+{
+    int64_t res = 0;
+    Visitor *v;
+    Error *err = NULL;
+
+    v = visitor_input_test_init(data, "\"-42\"");
+
+    visit_type_int(v, NULL, &res, &err);
+    g_assert(err != NULL);
+    error_free(err);
+}
+
 static void test_visitor_in_bool(TestInputVisitorData *data,
                                  const void *unused)
 {
@@ -127,6 +172,32 @@ static void test_visitor_in_bool(TestInputVisitorData *data,
     g_assert_cmpint(res, ==, true);
 }
 
+static void test_visitor_in_bool_autocast(TestInputVisitorData *data,
+                                          const void *unused)
+{
+    bool res = false;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, false, true, "\"true\"");
+
+    visit_type_bool(v, NULL, &res, &error_abort);
+    g_assert_cmpint(res, ==, true);
+}
+
+static void test_visitor_in_bool_noautocast(TestInputVisitorData *data,
+                                          const void *unused)
+{
+    bool res = false;
+    Visitor *v;
+    Error *err = NULL;
+
+    v = visitor_input_test_init(data, "\"true\"");
+
+    visit_type_bool(v, NULL, &res, &err);
+    g_assert(err != NULL);
+    error_free(err);
+}
+
 static void test_visitor_in_number(TestInputVisitorData *data,
                                    const void *unused)
 {
@@ -139,6 +210,32 @@ static void test_visitor_in_number(TestInputVisitorData *data,
     g_assert_cmpfloat(res, ==, value);
 }
 
+static void test_visitor_in_number_autocast(TestInputVisitorData *data,
+                                            const void *unused)
+{
+    double res = 0, value = 3.14;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, false, true, "\"3.14\"");
+
+    visit_type_number(v, NULL, &res, &error_abort);
+    g_assert_cmpfloat(res, ==, value);
+}
+
+static void test_visitor_in_number_noautocast(TestInputVisitorData *data,
+                                              const void *unused)
+{
+    double res = 0;
+    Visitor *v;
+    Error *err = NULL;
+
+    v = visitor_input_test_init(data, "\"3.14\"");
+
+    visit_type_number(v, NULL, &res, &err);
+    g_assert(err != NULL);
+    error_free(err);
+}
+
 static void test_visitor_in_string(TestInputVisitorData *data,
                                    const void *unused)
 {
@@ -835,10 +932,22 @@ int main(int argc, char **argv)
                            &in_visitor_data, test_visitor_in_int);
     input_visitor_test_add("/visitor/input/int_overflow",
                            &in_visitor_data, test_visitor_in_int_overflow);
+    input_visitor_test_add("/visitor/input/int_autocast",
+                           &in_visitor_data, test_visitor_in_int_autocast);
+    input_visitor_test_add("/visitor/input/int_noautocast",
+                           &in_visitor_data, test_visitor_in_int_noautocast);
     input_visitor_test_add("/visitor/input/bool",
                            &in_visitor_data, test_visitor_in_bool);
+    input_visitor_test_add("/visitor/input/bool_autocast",
+                           &in_visitor_data, test_visitor_in_bool_autocast);
+    input_visitor_test_add("/visitor/input/bool_noautocast",
+                           &in_visitor_data, test_visitor_in_bool_noautocast);
     input_visitor_test_add("/visitor/input/number",
                            &in_visitor_data, test_visitor_in_number);
+    input_visitor_test_add("/visitor/input/number_autocast",
+                           &in_visitor_data, test_visitor_in_number_autocast);
+    input_visitor_test_add("/visitor/input/number_noautocast",
+                           &in_visitor_data, test_visitor_in_number_noautocast);
     input_visitor_test_add("/visitor/input/string",
                            &in_visitor_data, test_visitor_in_string);
     input_visitor_test_add("/visitor/input/enum",
tests/test-visitor-serialization.c +1 -1
index 7b14b5a..db618d8 100644
@@ -1038,7 +1038,7 @@ static void qmp_deserialize(void **native_out, void *datap,
     obj = qobject_from_json(qstring_get_str(output_json));
 
     QDECREF(output_json);
-    d->qiv = qmp_input_visitor_new(obj, true);
+    d->qiv = qmp_input_visitor_new(obj, true, false);
     qobject_decref(obj_orig);
     qobject_decref(obj);
     visit(qmp_input_get_visitor(d->qiv), native_out, errp);
util/qemu-sockets.c +1 -1
index 0d6cd1f..51c6a8e 100644
@@ -1145,7 +1145,7 @@ void qapi_copy_SocketAddress(SocketAddress **p_dest,
         return;
     }
 
-    qiv = qmp_input_visitor_new(obj, true);
+    qiv = qmp_input_visitor_new(obj, true, false);
     iv = qmp_input_get_visitor(qiv);
     visit_type_SocketAddress(iv, NULL, p_dest, &error_abort);
     qmp_input_visitor_cleanup(qiv);
#13 lore raw

Re: [Qemu-devel] [PATCH v5 03/11] qom: support arbitrary non-scalar properties with -object

In reply to Daniel P. Berrange
Quoted 11 lines · Daniel P. Berrange

The current -object command line syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.

This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.

Quoted 4 lines

It is not practical to extend the OptsVisitor to support more complex data structures while also maintaining the existing list handling behaviour that is relied upon by other areas of QEMU.

It should be practical to create a new option input visitor that parses command line syntax straight into a QAPI visit, without the list magic, and with fewer or no restrictions. Then we could start defining complex command line arguments as QAPI types, parse them straight into generated QAPI types, and dispense with the QDict wrangling.

But I can't blame you for solving a simpler problem instead. In particular since there's ample precedence for QDict wrangling.

Quoted 49 lines

Fortunately there is no existing object that implements the UserCreatable interface that relies on the list handling behaviour, so it is possible to swap out the OptsVisitor for a different visitor implementation, so -object supports non-scalar properties, thus leaving other users of OptsVisitor unaffected.

The previously added qdict_crumple() method is able to take a qdict containing a flat set of properties and turn that into a arbitrarily nested set of dicts and lists. By combining qemu_opts_to_qdict and qdict_crumple() together, we can turn the opt string into a data structure that is practically identical to that passed over QMP when defining an object. The only difference is that all the scalar values are represented as strings, rather than strings, ints and bools. This is sufficient to let us replace the OptsVisitor with the QMPInputVisitor for use with -object.

Thus -object can now support non-scalar properties, for example the QMP object

  {
    "execute": "object-add",
    "arguments": {
      "qom-type": "demo",
      "id": "demo0",
      "parameters": {
        "foo": [
          { "bar": "one", "wizz": "1" },
          { "bar": "two", "wizz": "2" }
        ]
      }
    }
  }

Would be creatable via the CLI now using

    $QEMU \
      -object demo,id=demo0,\
              foo.0.bar=one,foo.0.wizz=1,\
              foo.1.bar=two,foo.1.wizz=2

Notice that this syntax is intentionally compatible with that currently used by block drivers.

This is also wired up to work for the 'object_add' command in the HMP monitor with the same syntax.

Quoted 6 lines
  (hmp) object_add demo,id=demo0,\
                   foo.0.bar=one,foo.0.wizz=1,\
                   foo.1.bar=two,foo.1.wizz=2

NB indentation should not be used with HMP commands, this is just for convenient formatting in this commit message.

Cannot be used, actually (and neither can line continuation), but I get what you mean.

Quoted 34 lines
---
 hmp.c                           |  18 +--
 include/qom/object_interfaces.h |  10 +-
 qmp.c                           |   2 +-
 qom/object_interfaces.c         |  49 +++++--
 tests/check-qom-proplist.c      | 317 +++++++++++++++++++++++++++++++++++++++-
 5 files changed, 363 insertions(+), 33 deletions(-)
hmp.c +2 -6
index a4b1d3d..1972bef 100644
@@ -25,7 +25,7 @@
 #include "qemu/sockets.h"
 #include "monitor/monitor.h"
 #include "monitor/qdev.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qmp-input-visitor.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/string-output-visitor.h"
 #include "qapi/util.h"
@@ -1717,20 +1717,12 @@ void hmp_netdev_del(Monitor *mon, const QDict *qdict)
 void hmp_object_add(Monitor *mon, const QDict *qdict)
 {
     Error *err = NULL;
-    QemuOpts *opts;
-    OptsVisitor *ov;
+    QmpInputVisitor *qiv;
     Object *obj = NULL;
 
-    opts = qemu_opts_from_qdict(qemu_find_opts("object"), qdict, &err);
-    if (err) {
-        hmp_handle_error(mon, &err);

hmp.c void hmp_netdev_del(Monitor *mon, const QDict *qdict) · line 1725

Quoted 10 lines
hmp.c +3 -7
-        return;
-    }
-
-    ov = opts_visitor_new(opts);
-    obj = user_creatable_add(qdict, opts_get_visitor(ov), &err);
-    opts_visitor_cleanup(ov);
-    qemu_opts_del(opts);
+    qiv = qmp_input_visitor_new((QObject *)qdict, true, true);
+    obj = user_creatable_add(qdict, qmp_input_get_visitor(qiv), &err);
+    qmp_input_visitor_cleanup(qiv);

Any patch that gets rid of a qemu_opts_from_qdict() deserves support.

Quoted 14 lines
hmp.c +0 -0
     if (err) {
         hmp_handle_error(mon, &err);
include/qom/object_interfaces.h +1 -0
index 8b17f4d..997563a 100644
@@ -96,18 +96,24 @@ Object *user_creatable_add(const QDict *qdict,
  * user_creatable_add_type:
  * @type: the object type name
  * @id: the unique ID for the object
+ * @nested: whether to recurse into the visitor for properties
  * @qdict: the object properties
  * @v: the visitor

include/qom/object_interfaces.h Object *user_creatable_add(const QDict *qdict, · line 110

Quoted 10 lines
include/qom/object_interfaces.h +5 -1
  * @errp: if an error occurs, a pointer to an area to store the error
  *
  * Create an instance of the user creatable object @type, placing
  * it in the object composition tree with name @id, initializing
- * it with properties from @qdict
+ * it with properties from @qdict.
+ *
+ * If the visitor is already positioned to read the properties
+ * in @qdict, @nested should be false. Conversely, if it is
+ * neccessary to open/close a struct to read the properties in

necessary

include/qom/object_interfaces.h Object *user_creatable_add(const QDict *qdict, · line 111

Quoted 1 line
include/qom/object_interfaces.h +1 -0
+ * @qdict, @nested should be true.

I don't like this interface much, but I'm not the maintainer here. Have you considered creating a function that does nested=false and can be used by the nested=true function?

Quoted 116 lines
include/qom/object_interfaces.h +1 -1
  *
  * Returns: the newly created object or NULL on error
  */
 Object *user_creatable_add_type(const char *type, const char *id,
-                                const QDict *qdict,
+                                bool nested, const QDict *qdict,
                                 Visitor *v, Error **errp);
 
 /**
qmp.c +1 -1
index dcd0953..dc5fff0 100644
@@ -666,7 +666,7 @@ void qmp_object_add(const char *type, const char *id,
     }
 
     qiv = qmp_input_visitor_new(props, true, false);
-    obj = user_creatable_add_type(type, id, pdict,
+    obj = user_creatable_add_type(type, id, true, pdict,
                                   qmp_input_get_visitor(qiv), errp);
     qmp_input_visitor_cleanup(qiv);
     if (obj) {
qom/object_interfaces.c +26 -16
index 51e62e2..0883c91 100644
@@ -3,6 +3,7 @@
 #include "qom/object_interfaces.h"
 #include "qemu/module.h"
 #include "qapi-visit.h"
+#include "qapi/qmp-input-visitor.h"
 #include "qapi/qmp-output-visitor.h"
 #include "qapi/opts-visitor.h"
 
@@ -63,12 +64,16 @@ Object *user_creatable_add(const QDict *qdict,
     if (local_err) {
         goto out_visit;
     }
-    visit_check_struct(v, &local_err);
+
+    obj = user_creatable_add_type(type, id, false, pdict, v, &local_err);
     if (local_err) {
         goto out_visit;
     }
 
-    obj = user_creatable_add_type(type, id, pdict, v, &local_err);
+    visit_check_struct(v, &local_err);
+    if (local_err) {
+        goto out_visit;
+    }
 
 out_visit:
     visit_end_struct(v);
@@ -87,7 +92,7 @@ out:
 
 
 Object *user_creatable_add_type(const char *type, const char *id,
-                                const QDict *qdict,
+                                bool nested, const QDict *qdict,
                                 Visitor *v, Error **errp)
 {
     Object *obj;
@@ -114,9 +119,11 @@ Object *user_creatable_add_type(const char *type, const char *id,
 
     assert(qdict);
     obj = object_new(type);
-    visit_start_struct(v, NULL, NULL, 0, &local_err);
-    if (local_err) {
-        goto out;
+    if (nested) {
+        visit_start_struct(v, NULL, NULL, 0, &local_err);
+        if (local_err) {
+            goto out;
+        }
     }
     for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) {
         object_property_set(obj, v, e->key, &local_err);
@@ -124,12 +131,14 @@ Object *user_creatable_add_type(const char *type, const char *id,
             break;
         }
     }
-    if (!local_err) {
-        visit_check_struct(v, &local_err);
-    }
-    visit_end_struct(v);
-    if (local_err) {
-        goto out;
+    if (nested) {
+        if (!local_err) {
+            visit_check_struct(v, &local_err);
+        }
+        visit_end_struct(v);
+        if (local_err) {
+            goto out;
+        }
     }
 
     object_property_add_child(object_get_objects_root(),
@@ -156,15 +165,23 @@ out:
 
 Object *user_creatable_add_opts(QemuOpts *opts, Error **errp)
 {
-    OptsVisitor *ov;
+    QmpInputVisitor *qiv;
     QDict *pdict;
+    QObject *pobj;
     Object *obj = NULL;
 
-    ov = opts_visitor_new(opts);
     pdict = qemu_opts_to_qdict(opts, NULL);
 
-    obj = user_creatable_add(pdict, opts_get_visitor(ov), errp);
-    opts_visitor_cleanup(ov);
+    pobj = qdict_crumple(pdict, true, errp);
+    if (!pobj) {
+        goto cleanup;

qom/object_interfaces.c out: · line 187

Quoted 10 lines
qom/object_interfaces.c +7 -0
+    }
+    qiv = qmp_input_visitor_new(pobj, true, true);
+
+    obj = user_creatable_add((QDict *)pobj, qmp_input_get_visitor(qiv), errp);
+    qmp_input_visitor_cleanup(qiv);
+    qobject_decref(pobj);
+ cleanup:
     QDECREF(pdict);
     return obj;
 }

[...]

I'm not familiar enough with this code to provide a real review with reasonable effort. It looks sane to ignorant me.

#4 lore raw

Re: [Qemu-devel] [PATCH v5 01/11] qdict: implement a qdict_crumple method for un-flattening a dict

In reply to Markus Armbruster
Quoted 84 lines · Markus Armbruster

I apologize for the lateness of this review.

Quoted 71 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

+/**
+ * qdict_split_flat_key:
+ * @key: the key string to split
+ * @prefix: non-NULL pointer to hold extracted prefix
+ * @suffix: non-NULL pointer to hold extracted suffix
+ *
+ * Given a flattened key such as 'foo.0.bar', split it
+ * into two parts at the first '.' separator. Allows
+ * double dot ('..') to escape the normal separator.
+ *

Quoted 8 lines

+ * eg
+ * 'foo.0.bar' -> prefix='foo' and suffix='0.bar'
+ * 'foo..0.bar' -> prefix='foo.0' and suffix='bar'
+ *
+ * The '..' sequence will be unescaped in the returned
+ * 'prefix' string. The 'suffix' string will be left
+ * in escaped format, so it can be fed back into the
+ * qdict_split_flat_key() key as the input later.

Quoted 1 line

Why is the suffix strdup'ed then?

It doesn't need to be - i'll make it const.

Quoted 10 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 750

Quoted 6 lines
qobject/qdict.c +6 -0
+}
+
+
+/**
+ * qdict_list_size:
+ * @maybe_list: dict that may be only list elements

Huh? How can a dictionary "be only list elements"?

Do you mean "the dictionary to test?"

I'll say "dict to search for keys representing list elements."

Quoted 9 lines
Quoted 8 lines

+ *
+ * Determine whether all keys in @maybe_list are
+ * valid list elements. If they are all valid,
+ * then this returns the number of elements. If
+ * they all look like non-numeric keys, then returns
+ * zero. If there is a mix of numeric and non-numeric
+ * keys, then an error is set as it is both a list
+ * and a dict at once.

Quoted 5 lines

This is well-defined only if empty @maybe_list is considered to have dict nature, not list nature. Else, return value zero could be the length of the empty list or the special "has dict nature" value.

Please spell out behavior for empty @maybe_list.

Yep, empty list will imply qdict nature and so return zero.

Quoted 7 lines
Quoted 2 lines

+ *
+ * Returns: number of list elements, 0 if a dict, -1 on error

Awkward function name. qdict_list_size_if_list() would be clear.

But I'd simply turn this into a predicate qdict_is_list(), and have the caller use qdict_size() to get the number of elements.

I thought that qdict_size() would cause another iteration, but I see now it just returns a cached size. So I'll switch to qidct_is_list().

Quoted 16 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 772

Quoted 11 lines
qobject/qdict.c +11 -0
+static ssize_t qdict_list_size(QDict *maybe_list, Error **errp)
+{
+    const QDictEntry *entry, *next;
+    ssize_t len = 0;
+    ssize_t max = -1;
+    int is_list = -1;
+    int64_t val;
+
+    entry = qdict_first(maybe_list);
+    while (entry != NULL) {
+        next = qdict_next(maybe_list, entry);

Please keep the loop control in one place:

       for (entry = qdict_first(maybe_list); entry; entry = qdict_next(entry)) {
Quoted 7 lines

I'd rename some variables for less verbiage:

       for (ent = qdict_first(dict); ent; ent = qdict_next(ent)) {

Your loop control also works when the loop body deletes @entry from @maybe_list. Seeing such loop control in a function that isn't supposed to change the its argument makes the reviewer go "WTF?!?" :)

This pattern was left from an earlier version where all the code was in one method. I'll change it to a for() loop now.

Quoted 26 lines
Quoted 26 lines
qobject/qdict.c +25 -0
+
+        if (qemu_strtoll(entry->key, NULL, 10, &val) == 0) {
+            if (is_list == -1) {
+                is_list = 1;
+            } else if (!is_list) {
+                error_setg(errp,
+                           "Cannot crumple a dictionary with a mix of list "
+                           "and non-list keys");
+                return -1;
+            }
+            len++;
+            if (val > max) {
+                max = val;
+            }
+        } else {
+            if (is_list == -1) {
+                is_list = 0;
+            } else if (is_list) {
+                error_setg(errp,
+                           "Cannot crumple a dictionary with a mix of list "
+                           "and non-list keys");
+                return -1;
+            }
+        }
+
Quoted 8 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 802

Quoted 5 lines
qobject/qdict.c +5 -0
+        entry = next;
+    }
+
+    if (is_list == -1) {
+        is_list = 0;

This can happen only when @maybe_list is empty. Okay, but perhaps you'd like to assert(!qdict_size(maybe_list)).

Ok

Quoted 9 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 809

Quoted 8 lines
qobject/qdict.c +7 -0
+    }
+
+    if (len != (max + 1)) {
+        error_setg(errp, "List indexes are not continuous, "
+                   "saw %zd elements but %zd largest index",
+                   len, max);
+        return -1;
Quoted 3 lines

contiguous?

What if we saw indexes 0, 2, 2?

That would imply that the dict had two entries with the same key, which by definition is impossible with a dict data structure.

Quoted 10 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 817

Quoted 8 lines
qobject/qdict.c +8 -0
+    }
+
+    return is_list ? len : 0;
+}
+
+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary to crumple

"Flat" means all values are scalar. Should we spell that out?

Yep, ok.

Quoted 32 lines
Quoted 4 lines

+ * @recursive: true to recursively crumple nested dictionaries
+ *
+ * Takes a flat dictionary whose keys use '.' separator to
+ * indicate nesting, and values are scalars, crumplings it

s/, crumplings/, and crumples/

Quoted 16 lines

+ * into a nested structure. If the @recursive parameter is
+ * false, then only the first level of structure implied
+ * by the keys will be crumpled. If @recursive is true,
+ * then the input will be recursively crumpled to expand
+ * all levels of structure in the keys.
+ *
+ * To include a literal '.' in a key name, it must be escaped
+ * as '..'
+ *
+ * For example, an input of:
+ *
+ * { 'foo.0.bar': 'one', 'foo.0.wizz': '1',
+ * 'foo.1.bar': 'two', 'foo.1.wizz': '2' }
+ *
+ * will result in any output of:
+ *

Quoted 8 lines

+ * {
+ * 'foo': [
+ * { 'bar': 'one', 'wizz': '1' },
+ * { 'bar': 'two', 'wizz': '2' }
+ * ],
+ * }
+ *
+ * Returns: either a QDict or QList for the nested data structure

Quoted 1 line

I think you should discuss how this can fail.

Will do.

Quoted 59 lines
Quoted 5 lines
qobject/qdict.c +5 -0
+QObject *qdict_crumple(QDict *src, bool recursive, Error **errp)
+{
+    const QDictEntry *entry, *next;
+    QDict *two_level, *multi_level = NULL;
+    QObject *dst = NULL, *child;

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 861

Quoted 10 lines
qobject/qdict.c +10 -0
+    ssize_t list_len;
+    size_t i;
+    char *prefix = NULL, *suffix = NULL;
+
+    two_level = qdict_new();
+    entry = qdict_first(src);
+
+    /* Step 1: split our totally flat dict into a two level dict */
+    while (entry != NULL) {
+        next = qdict_next(src, entry);

Again, keep the loop control in one place.

Quoted 23 lines
qobject/qdict.c +23 -0
+
+        if (qobject_type(entry->value) == QTYPE_QDICT ||
+            qobject_type(entry->value) == QTYPE_QLIST) {
+            error_setg(errp, "Value %s is not a scalar",
+                       entry->key);
+            goto error;
+        }
+
+        qdict_split_flat_key(entry->key, &prefix, &suffix);
+
+        child = qdict_get(two_level, prefix);
+        if (suffix) {
+            if (child) {
+                if (qobject_type(child) != QTYPE_QDICT) {
+                    error_setg(errp, "Key %s prefix is already set as a scalar",
+                               prefix);
+                    goto error;
+                }
+            } else {
+                child = QOBJECT(qdict_new());
+                qdict_put_obj(two_level, prefix, child);
+            }
+            qobject_incref(entry->value);

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 894

Quoted 10 lines
qobject/qdict.c +10 -0
+            qdict_put_obj(qobject_to_qdict(child), suffix, entry->value);
+        } else {
+            if (child) {
+                error_setg(errp, "Key %s prefix is already set as a dict",
+                           prefix);
+                goto error;
+            }
+            qobject_incref(entry->value);
+            qdict_put_obj(two_level, prefix, entry->value);
+        }

Works, because we put only QDicts we've created ourselves (first qdict_put_obj() above) and values we got from @src (second qdict_put_obj()), and we fail when such a value isn't scalar.

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+        g_free(suffix);
Quoted 6 lines

As I suspected, qdict_split_flat_key() strdup'ing the suffix is useless.

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 898

Quoted 2 lines
qobject/qdict.c +2 -0
+        g_free(prefix);
+        suffix = prefix = NULL;

Dead stores.

No, they're not dead. The end of this method has a 'g_free(prefix)' so we must be sure to clear this out to prevent a double-free if a later codebase jumps to the error label.

Quoted 11 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 909

Quoted 10 lines
qobject/qdict.c +10 -0
+        entry = next;
+    }
+
+    /* Step 2: optionally process the two level dict recursively
+     * into a multi-level dict */
+    if (recursive) {
+        multi_level = qdict_new();
+        entry = qdict_first(two_level);
+        while (entry != NULL) {
+            next = qdict_next(two_level, entry);
Quoted 1 line

Again, keep the loop control in one place.

OK

Quoted 34 lines
Quoted 34 lines
qobject/qdict.c +33 -0
+
+            if (qobject_type(entry->value) == QTYPE_QDICT) {
+                child = qdict_crumple(qobject_to_qdict(entry->value),
+                                      recursive, errp);
+                if (!child) {
+                    goto error;
+                }
+
+                qdict_put_obj(multi_level, entry->key, child);
+            } else {
+                qobject_incref(entry->value);
+                qdict_put_obj(multi_level, entry->key, entry->value);
+            }
+
+            entry = next;
+        }
+        QDECREF(two_level);
+    } else {
+        multi_level = two_level;
+    }
+    two_level = NULL;
+
+    /* Step 3: detect if we need to turn our dict into list */
+    list_len = qdict_list_size(multi_level, errp);
+    if (list_len < 0) {
+        goto error;
+    }
+
+    if (list_len) {
+        dst = QOBJECT(qlist_new());
+
+        for (i = 0; i < list_len; i++) {
+            char *key = g_strdup_printf("%zu", i);
Quoted 8 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 946

Quoted 4 lines
qobject/qdict.c +4 -0
+
+            child = qdict_get(multi_level, key);
+            g_free(key);
+            assert(child);

qdict_list_size() accepts as list index any (string) key qemu_strtoll() accepts. If %zu formats it back into the same string, we'll find it here. Else we die. Please spell this out in the function contract.

OK.

Quoted 5 lines

[*] I'm afraid we also die if qdict_list_size()'s "List indexes are not continuous" check gets fooled. Suggest to drop that check, and replace this assertion by error_setg(errp, "Malformed list indexes"). Admittedly not the nicest error message; perhaps you can come up with a better one.

We can't be fooled per my note earlier

Quoted 7 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 951

Quoted 6 lines
qobject/qdict.c +5 -0
+
+            qobject_incref(child);
+            qlist_append_obj(qobject_to_qlist(dst), child);
+        }
+        QDECREF(multi_level);
Quoted 5 lines

Do we need

           multi_level = NULL;

here?

It isn't needed right now, since we're at the end of the method now and nothing will touch this var again. Setting it to NULL is a worthwhile safety net for future refactoring though.

Quoted 83 lines
Quoted 70 lines
qobject/qdict.c +16 -0
+    } else {
+        dst = QOBJECT(multi_level);
+    }
+
+    return dst;
+
+ error:
+    g_free(suffix);
+    g_free(prefix);
+    QDECREF(multi_level);
+    QDECREF(two_level);
+    qobject_decref(dst);
+    return NULL;
+}
+
+
 /**
  * qdict_array_entries(): Returns the number of direct array entries if the
  * sub-QDict of src specified by the prefix in subqdict (or src itself for
tests/check-qdict.c +40 -0
index a43056c..0d12f40 100644
@@ -15,6 +15,7 @@
 #include "qapi/qmp/qint.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
+#include "qapi/error.h"
 #include "qemu-common.h"
 
+static void qdict_crumple_test_bad_inputs(void)
+{
+    QDict *src;
+    Error *error = NULL;
+
+    src = qdict_new();
+    /* rule.0 can't be both a string and a dict */
+    qdict_put(src, "rule.0", qstring_from_str("fred"));
+    qdict_put(src, "rule.0.policy", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);
+
+    src = qdict_new();
+    /* rule can't be both a list and a dict */
+    qdict_put(src, "rule.0", qstring_from_str("fred"));
+    qdict_put(src, "rule.a", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);
+
+    src = qdict_new();
+    /* The input should be flat, ie no dicts or lists */
+    qdict_put(src, "rule.a", qdict_new());
+    qdict_put(src, "rule.b", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);
+
+

tests/check-qdict.c static void qdict_join_test(void) · line 814

Quoted 10 lines
tests/check-qdict.c +10 -0
+    src = qdict_new();
+    /* List indexes must not have gaps */
+    qdict_put(src, "rule.0", qdict_new());
+    qdict_put(src, "rule.3", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);

Suggest to add test case

Quoted 4 lines
       /* List indexes must not have gaps (more creative version) */
       qdict_put(src, "rule.0", ...);
       qdict_put(src, "rule.2", ...);
       qdict_put(src, "rule.2", ...);

That's surely impossible, as dict keys have to be unique.

Quoted 4 lines

and

       /* List indexes must be in %zu format */
       qdict_put(src, "rule.+0", ...);

Ok.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#10 lore raw

Re: [Qemu-devel] [PATCH v5 02/11] qapi: allow QmpInputVisitor to auto-cast types

In reply to Markus Armbruster
Quoted 44 lines · Markus Armbruster
Quoted 5 lines · Daniel P. Berrange

Currently the QmpInputVisitor assumes that all scalar values are directly represented as their final types. ie it assumes an 'int' is using QInt, and a 'bool' is using QBool.

Quoted 8 lines

This extends it so that QString is optionally permitted for any of the non-string scalar types. This behaviour is turned on by requesting the 'autocast' flag in the constructor.

This makes it possible to use QmpInputVisitor with a QDict produced from QemuOpts, where everything is in string format.

We're still digging.

Quoted 26 lines
---
 docs/qapi-code-gen.txt             |   2 +-
 include/qapi/qmp-input-visitor.h   |   6 +-
 qapi/opts-visitor.c                |   1 +
 qapi/qmp-input-visitor.c           |  89 ++++++++++++++++++++++------
 qmp.c                              |   2 +-
 qom/qom-qobject.c                  |   2 +-
 replay/replay-input.c              |   2 +-
 scripts/qapi-commands.py           |   2 +-
 tests/check-qnull.c                |   2 +-
 tests/test-qmp-commands.c          |   2 +-
 tests/test-qmp-input-strict.c      |   2 +-
 tests/test-qmp-input-visitor.c     | 115 ++++++++++++++++++++++++++++++++++++-
 tests/test-visitor-serialization.c |   2 +-
 util/qemu-sockets.c                |   2 +-
 14 files changed, 201 insertions(+), 30 deletions(-)
qapi/opts-visitor.c +0 -0
index 4cf1cf8..00e4b1a 100644
@@ -347,6 +347,7 @@ opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp)
     }
 
Quoted 4 lines

qapi/opts-visitor.c opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp) · line 350

Quoted 2 lines
qapi/opts-visitor.c +1 -0
     if (opt->str) {
+        /* Keep these values in sync with same code in qmp-input-visitor.c */

Also with parse_option_bool(). That's the canonical place, in fact.

....except parse_option_bool() doesn't allow "yes", "no", "y", "n" as valid values - it only permits "on" and "off" :-( I guess I could make the parse_option_bool() method non-static, make it accept these missing values, and then call it from all places so we have guaranteed consistency.

Quoted 56 lines
Quoted 36 lines
qapi/opts-visitor.c +0 -0
         if (strcmp(opt->str, "on") == 0 ||
             strcmp(opt->str, "yes") == 0 ||
             strcmp(opt->str, "y") == 0) {
qapi/qmp-input-visitor.c +5 -4
index aea90a1..92023b1 100644
@@ -20,6 +20,7 @@
 #include "qemu-common.h"
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/qerror.h"
+#include "qemu/cutils.h"
 
 #define QIV_STACK_SIZE 1024
 
@@ -45,6 +46,7 @@ struct QmpInputVisitor
 
     /* True to reject parse in visit_end_struct() if unvisited keys remain. */
     bool strict;
+    bool autocast;
 };
 
 static QmpInputVisitor *to_qiv(Visitor *v)
@@ -254,15 +256,25 @@ static void qmp_input_type_int64(Visitor *v, const char *name, int64_t *obj,
                                  Error **errp)
 {
     QmpInputVisitor *qiv = to_qiv(v);
-    QInt *qint = qobject_to_qint(qmp_input_get_object(qiv, name, true));
+    QObject *qobj = qmp_input_get_object(qiv, name, true);
+    QInt *qint;
+    QString *qstr;
 
-    if (!qint) {
-        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
-                   "integer");

qapi/qmp-input-visitor.c static void qmp_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 271

Quoted 10 lines
qapi/qmp-input-visitor.c +6 -1
+    qint = qobject_to_qint(qobj);
+    if (qint) {
+        *obj = qint_get_int(qint);
         return;
     }
 
-    *obj = qint_get_int(qint);
+    qstr = qobject_to_qstring(qobj);
+    if (qstr && qstr->string && qiv->autocast) {
+        if (qemu_strtoll(qstr->string, NULL, 10, obj) == 0) {

In the commit message, you mentioned intended use: "with a QDict produced from QemuOpts, where everything is in string format." I figure you mean opts with empty opts->list->desc[]. For those, we accept any key and parse all values as strings.

The idea with such QemuOpts is to *delay* parsing until we know how to parse. The delay should be transparent to the user. In particular, values should be parsed the same whether the parsing is delayed or not.

Quoted 8 lines

The canonical QemuOpts value parser is qemu_opt_parse(). It parses integers with parse_option_number(). That function parses like stroull(qstr->string, ..., 0). Two differences:

* stroll() vs. strtoull(): they differ in ERANGE behavior. This might be tolerable, but I haven't thought it through.

* Base 0 vs 10: different syntax and semantics. Oops.

Sigh, I originally had my code using strtoull() directly as the parse_option_number() method does, but had to change it to make checkpatch.pl stfu thus creating incosistency. This is a great example of why I hate the fact that we only validate our style rules against new patches and not our existing codebase :-(

Anyway, it seems to be something where we should refactor code to use the same parsing method from both places. eg by making the parse_option_number method non-static and just calling it from here.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#8 lore raw

Re: [Qemu-devel] [PATCH v5 02/11] qapi: allow QmpInputVisitor to auto-cast types

In reply to Paolo Bonzini
Quoted 13 lines · Paolo Bonzini
Quoted 10 lines · Daniel P. Berrange

Currently the QmpInputVisitor assumes that all scalar values are directly represented as their final types. ie it assumes an 'int' is using QInt, and a 'bool' is using QBool.

This extends it so that QString is optionally permitted for any of the non-string scalar types. This behaviour is turned on by requesting the 'autocast' flag in the constructor.

Quoted 8 lines
Quoted 3 lines

This makes it possible to use QmpInputVisitor with a QDict produced from QemuOpts, where everything is in string format.

Perhaps this should instead be a separate QmpStringInputVisitor visitor that _only_ accepts strings? You can reuse most of the QmpInputVisitor by putting it in the same file, because the struct and list visitors are compatible.

Yes, that actually works out quite nicely indeed, and in fact showed up a bug in my unit tests too :-)

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#14 lore raw

Re: [Qemu-devel] [PATCH v5 03/11] qom: support arbitrary non-scalar properties with -object

In reply to Markus Armbruster
Quoted 18 lines · Markus Armbruster
Quoted 11 lines · Daniel P. Berrange

The current -object command line syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.

This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.

Quoted 4 lines

It is not practical to extend the OptsVisitor to support more complex data structures while also maintaining the existing list handling behaviour that is relied upon by other areas of QEMU.

Quoted 5 lines

It should be practical to create a new option input visitor that parses command line syntax straight into a QAPI visit, without the list magic, and with fewer or no restrictions. Then we could start defining complex command line arguments as QAPI types, parse them straight into generated QAPI types, and dispense with the QDict wrangling.

Funnily enough I did actually start out trying to implement pretty much exactly that. I found that in my new opts visitor, I was essentially parsing the QemuOpts into QDict/QLists to handle the recursion from the visitor. At this point I was starting to duplicate much code from QDict and QmpInputVisitor, so it abandoned that approach and went for the more general QemuOpts -> QDict crumping instead, since it the goal to be achieved with simpler code overall.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#5 lore raw

Re: [Qemu-devel] [PATCH v5 01/11] qdict: implement a qdict_crumple method for un-flattening a dict

In reply to Daniel P. Berrange
Quoted 100 lines · Daniel P. Berrange
Quoted 84 lines · Markus Armbruster

I apologize for the lateness of this review.

Quoted 71 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

+/**
+ * qdict_split_flat_key:
+ * @key: the key string to split
+ * @prefix: non-NULL pointer to hold extracted prefix
+ * @suffix: non-NULL pointer to hold extracted suffix
+ *
+ * Given a flattened key such as 'foo.0.bar', split it
+ * into two parts at the first '.' separator. Allows
+ * double dot ('..') to escape the normal separator.
+ *

Quoted 8 lines

+ * eg
+ * 'foo.0.bar' -> prefix='foo' and suffix='0.bar'
+ * 'foo..0.bar' -> prefix='foo.0' and suffix='bar'
+ *
+ * The '..' sequence will be unescaped in the returned
+ * 'prefix' string. The 'suffix' string will be left
+ * in escaped format, so it can be fed back into the
+ * qdict_split_flat_key() key as the input later.

Quoted 1 line

Why is the suffix strdup'ed then?

It doesn't need to be - i'll make it const.

Quoted 9 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 750

Quoted 6 lines
qobject/qdict.c +6 -0
+}
+
+
+/**
+ * qdict_list_size:
+ * @maybe_list: dict that may be only list elements

Huh? How can a dictionary "be only list elements"?

Quoted 3 lines
Quoted 1 line

Do you mean "the dictionary to test?"

I'll say "dict to search for keys representing list elements."

Yes, that's better.

Quoted 102 lines
Quoted 9 lines
Quoted 8 lines

+ *
+ * Determine whether all keys in @maybe_list are
+ * valid list elements. If they are all valid,
+ * then this returns the number of elements. If
+ * they all look like non-numeric keys, then returns
+ * zero. If there is a mix of numeric and non-numeric
+ * keys, then an error is set as it is both a list
+ * and a dict at once.

Quoted 5 lines

This is well-defined only if empty @maybe_list is considered to have dict nature, not list nature. Else, return value zero could be the length of the empty list or the special "has dict nature" value.

Please spell out behavior for empty @maybe_list.

Yep, empty list will imply qdict nature and so return zero.

Quoted 7 lines
Quoted 2 lines

+ *
+ * Returns: number of list elements, 0 if a dict, -1 on error

Awkward function name. qdict_list_size_if_list() would be clear.

But I'd simply turn this into a predicate qdict_is_list(), and have the caller use qdict_size() to get the number of elements.

I thought that qdict_size() would cause another iteration, but I see now it just returns a cached size. So I'll switch to qidct_is_list().

Quoted 16 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 772

Quoted 11 lines
qobject/qdict.c +11 -0
+static ssize_t qdict_list_size(QDict *maybe_list, Error **errp)
+{
+    const QDictEntry *entry, *next;
+    ssize_t len = 0;
+    ssize_t max = -1;
+    int is_list = -1;
+    int64_t val;
+
+    entry = qdict_first(maybe_list);
+    while (entry != NULL) {
+        next = qdict_next(maybe_list, entry);

Please keep the loop control in one place:

       for (entry = qdict_first(maybe_list); entry; entry = qdict_next(entry)) {
Quoted 7 lines

I'd rename some variables for less verbiage:

       for (ent = qdict_first(dict); ent; ent = qdict_next(ent)) {

Your loop control also works when the loop body deletes @entry from @maybe_list. Seeing such loop control in a function that isn't supposed to change the its argument makes the reviewer go "WTF?!?" :)

This pattern was left from an earlier version where all the code was in one method. I'll change it to a for() loop now.

Quoted 26 lines
Quoted 26 lines
qobject/qdict.c +25 -0
+
+        if (qemu_strtoll(entry->key, NULL, 10, &val) == 0) {
+            if (is_list == -1) {
+                is_list = 1;
+            } else if (!is_list) {
+                error_setg(errp,
+                           "Cannot crumple a dictionary with a mix of list "
+                           "and non-list keys");
+                return -1;
+            }
+            len++;
+            if (val > max) {
+                max = val;
+            }
+        } else {
+            if (is_list == -1) {
+                is_list = 0;
+            } else if (is_list) {
+                error_setg(errp,
+                           "Cannot crumple a dictionary with a mix of list "
+                           "and non-list keys");
+                return -1;
+            }
+        }
+
Quoted 8 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 802

Quoted 5 lines
qobject/qdict.c +5 -0
+        entry = next;
+    }
+
+    if (is_list == -1) {
+        is_list = 0;

This can happen only when @maybe_list is empty. Okay, but perhaps you'd like to assert(!qdict_size(maybe_list)).

Ok

Quoted 9 lines
Quoted 8 lines
qobject/qdict.c +7 -0
+    }
+
+    if (len != (max + 1)) {
+        error_setg(errp, "List indexes are not continuous, "
+                   "saw %zd elements but %zd largest index",
+                   len, max);
+        return -1;
Quoted 7 lines
Quoted 3 lines

contiguous?

What if we saw indexes 0, 2, 2?

That would imply that the dict had two entries with the same key, which by definition is impossible with a dict data structure.

I got confused :)

Quoted 196 lines
Quoted 10 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 817

Quoted 8 lines
qobject/qdict.c +8 -0
+    }
+
+    return is_list ? len : 0;
+}
+
+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary to crumple

"Flat" means all values are scalar. Should we spell that out?

Yep, ok.

Quoted 32 lines
Quoted 4 lines

+ * @recursive: true to recursively crumple nested dictionaries
+ *
+ * Takes a flat dictionary whose keys use '.' separator to
+ * indicate nesting, and values are scalars, crumplings it

s/, crumplings/, and crumples/

Quoted 16 lines

+ * into a nested structure. If the @recursive parameter is
+ * false, then only the first level of structure implied
+ * by the keys will be crumpled. If @recursive is true,
+ * then the input will be recursively crumpled to expand
+ * all levels of structure in the keys.
+ *
+ * To include a literal '.' in a key name, it must be escaped
+ * as '..'
+ *
+ * For example, an input of:
+ *
+ * { 'foo.0.bar': 'one', 'foo.0.wizz': '1',
+ * 'foo.1.bar': 'two', 'foo.1.wizz': '2' }
+ *
+ * will result in any output of:
+ *

Quoted 8 lines

+ * {
+ * 'foo': [
+ * { 'bar': 'one', 'wizz': '1' },
+ * { 'bar': 'two', 'wizz': '2' }
+ * ],
+ * }
+ *
+ * Returns: either a QDict or QList for the nested data structure

Quoted 1 line

I think you should discuss how this can fail.

Will do.

Quoted 59 lines
Quoted 5 lines
qobject/qdict.c +5 -0
+QObject *qdict_crumple(QDict *src, bool recursive, Error **errp)
+{
+    const QDictEntry *entry, *next;
+    QDict *two_level, *multi_level = NULL;
+    QObject *dst = NULL, *child;

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 861

Quoted 10 lines
qobject/qdict.c +10 -0
+    ssize_t list_len;
+    size_t i;
+    char *prefix = NULL, *suffix = NULL;
+
+    two_level = qdict_new();
+    entry = qdict_first(src);
+
+    /* Step 1: split our totally flat dict into a two level dict */
+    while (entry != NULL) {
+        next = qdict_next(src, entry);

Again, keep the loop control in one place.

Quoted 23 lines
qobject/qdict.c +23 -0
+
+        if (qobject_type(entry->value) == QTYPE_QDICT ||
+            qobject_type(entry->value) == QTYPE_QLIST) {
+            error_setg(errp, "Value %s is not a scalar",
+                       entry->key);
+            goto error;
+        }
+
+        qdict_split_flat_key(entry->key, &prefix, &suffix);
+
+        child = qdict_get(two_level, prefix);
+        if (suffix) {
+            if (child) {
+                if (qobject_type(child) != QTYPE_QDICT) {
+                    error_setg(errp, "Key %s prefix is already set as a scalar",
+                               prefix);
+                    goto error;
+                }
+            } else {
+                child = QOBJECT(qdict_new());
+                qdict_put_obj(two_level, prefix, child);
+            }
+            qobject_incref(entry->value);

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 894

Quoted 10 lines
qobject/qdict.c +10 -0
+            qdict_put_obj(qobject_to_qdict(child), suffix, entry->value);
+        } else {
+            if (child) {
+                error_setg(errp, "Key %s prefix is already set as a dict",
+                           prefix);
+                goto error;
+            }
+            qobject_incref(entry->value);
+            qdict_put_obj(two_level, prefix, entry->value);
+        }

Works, because we put only QDicts we've created ourselves (first qdict_put_obj() above) and values we got from @src (second qdict_put_obj()), and we fail when such a value isn't scalar.

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+        g_free(suffix);
Quoted 6 lines

As I suspected, qdict_split_flat_key() strdup'ing the suffix is useless.

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 898

Quoted 2 lines
qobject/qdict.c +2 -0
+        g_free(prefix);
+        suffix = prefix = NULL;

Dead stores.

No, they're not dead. The end of this method has a 'g_free(prefix)' so we must be sure to clear this out to prevent a double-free if a later codebase jumps to the error label.

Quoted 11 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 909

Quoted 10 lines
qobject/qdict.c +10 -0
+        entry = next;
+    }
+
+    /* Step 2: optionally process the two level dict recursively
+     * into a multi-level dict */
+    if (recursive) {
+        multi_level = qdict_new();
+        entry = qdict_first(two_level);
+        while (entry != NULL) {
+            next = qdict_next(two_level, entry);
Quoted 1 line

Again, keep the loop control in one place.

OK

Quoted 34 lines
Quoted 34 lines
qobject/qdict.c +33 -0
+
+            if (qobject_type(entry->value) == QTYPE_QDICT) {
+                child = qdict_crumple(qobject_to_qdict(entry->value),
+                                      recursive, errp);
+                if (!child) {
+                    goto error;
+                }
+
+                qdict_put_obj(multi_level, entry->key, child);
+            } else {
+                qobject_incref(entry->value);
+                qdict_put_obj(multi_level, entry->key, entry->value);
+            }
+
+            entry = next;
+        }
+        QDECREF(two_level);
+    } else {
+        multi_level = two_level;
+    }
+    two_level = NULL;
+
+    /* Step 3: detect if we need to turn our dict into list */
+    list_len = qdict_list_size(multi_level, errp);
+    if (list_len < 0) {
+        goto error;
+    }
+
+    if (list_len) {
+        dst = QOBJECT(qlist_new());
+
+        for (i = 0; i < list_len; i++) {
+            char *key = g_strdup_printf("%zu", i);
Quoted 8 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 946

Quoted 4 lines
qobject/qdict.c +4 -0
+
+            child = qdict_get(multi_level, key);
+            g_free(key);
+            assert(child);

qdict_list_size() accepts as list index any (string) key qemu_strtoll() accepts. If %zu formats it back into the same string, we'll find it here. Else we die. Please spell this out in the function contract.

OK.

Quoted 5 lines

[*] I'm afraid we also die if qdict_list_size()'s "List indexes are not continuous" check gets fooled. Suggest to drop that check, and replace this assertion by error_setg(errp, "Malformed list indexes"). Admittedly not the nicest error message; perhaps you can come up with a better one.

We can't be fooled per my note earlier

Quoted 9 lines

qobject/qdict.c void qdict_array_split(QDict *src, QList **dst) · line 951

Quoted 6 lines
qobject/qdict.c +5 -0
+
+            qobject_incref(child);
+            qlist_append_obj(qobject_to_qlist(dst), child);
+        }
+        QDECREF(multi_level);

Do we need

Quoted 7 lines
Quoted 3 lines
           multi_level = NULL;

here?

It isn't needed right now, since we're at the end of the method now and nothing will touch this var again. Setting it to NULL is a worthwhile safety net for future refactoring though.

I generally don't bother to zap pointers "just in case". I saw the QDECREF() after the error label, and missed the fact that we can't reach it from here. Your patch is fine as is.

Quoted 82 lines
Quoted 82 lines
Quoted 69 lines
qobject/qdict.c +16 -0
+    } else {
+        dst = QOBJECT(multi_level);
+    }
+
+    return dst;
+
+ error:
+    g_free(suffix);
+    g_free(prefix);
+    QDECREF(multi_level);
+    QDECREF(two_level);
+    qobject_decref(dst);
+    return NULL;
+}
+
+
 /**
  * qdict_array_entries(): Returns the number of direct array entries if the
  * sub-QDict of src specified by the prefix in subqdict (or src itself for
tests/check-qdict.c +40 -0
index a43056c..0d12f40 100644
@@ -15,6 +15,7 @@
 #include "qapi/qmp/qint.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
+#include "qapi/error.h"
 #include "qemu-common.h"
 
+static void qdict_crumple_test_bad_inputs(void)
+{
+    QDict *src;
+    Error *error = NULL;
+
+    src = qdict_new();
+    /* rule.0 can't be both a string and a dict */
+    qdict_put(src, "rule.0", qstring_from_str("fred"));
+    qdict_put(src, "rule.0.policy", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);
+
+    src = qdict_new();
+    /* rule can't be both a list and a dict */
+    qdict_put(src, "rule.0", qstring_from_str("fred"));
+    qdict_put(src, "rule.a", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);
+
+    src = qdict_new();
+    /* The input should be flat, ie no dicts or lists */
+    qdict_put(src, "rule.a", qdict_new());
+    qdict_put(src, "rule.b", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);
+
+

tests/check-qdict.c static void qdict_join_test(void) · line 814

Quoted 10 lines
tests/check-qdict.c +10 -0
+    src = qdict_new();
+    /* List indexes must not have gaps */
+    qdict_put(src, "rule.0", qdict_new());
+    qdict_put(src, "rule.3", qstring_from_str("allow"));
+
+    g_assert(qdict_crumple(src, true, &error) == NULL);
+    g_assert(error != NULL);
+    error_free(error);
+    error = NULL;
+    QDECREF(src);

Suggest to add test case

Quoted 6 lines
Quoted 4 lines
       /* List indexes must not have gaps (more creative version) */
       qdict_put(src, "rule.0", ...);
       qdict_put(src, "rule.2", ...);
       qdict_put(src, "rule.2", ...);

That's surely impossible, as dict keys have to be unique.

It's surely possible that patch review melts my brain :)

Quoted 4 lines
Quoted 4 lines

and

       /* List indexes must be in %zu format */
       qdict_put(src, "rule.+0", ...);
#11 lore raw

Re: [Qemu-devel] [PATCH v5 02/11] qapi: allow QmpInputVisitor to auto-cast types

In reply to Daniel P. Berrange
Quoted 50 lines · Daniel P. Berrange
Quoted 44 lines · Markus Armbruster
Quoted 5 lines · Daniel P. Berrange

Currently the QmpInputVisitor assumes that all scalar values are directly represented as their final types. ie it assumes an 'int' is using QInt, and a 'bool' is using QBool.

Quoted 8 lines

This extends it so that QString is optionally permitted for any of the non-string scalar types. This behaviour is turned on by requesting the 'autocast' flag in the constructor.

This makes it possible to use QmpInputVisitor with a QDict produced from QemuOpts, where everything is in string format.

We're still digging.

Quoted 26 lines
---
 docs/qapi-code-gen.txt             |   2 +-
 include/qapi/qmp-input-visitor.h   |   6 +-
 qapi/opts-visitor.c                |   1 +
 qapi/qmp-input-visitor.c           |  89 ++++++++++++++++++++++------
 qmp.c                              |   2 +-
 qom/qom-qobject.c                  |   2 +-
 replay/replay-input.c              |   2 +-
 scripts/qapi-commands.py           |   2 +-
 tests/check-qnull.c                |   2 +-
 tests/test-qmp-commands.c          |   2 +-
 tests/test-qmp-input-strict.c      |   2 +-
 tests/test-qmp-input-visitor.c     | 115 ++++++++++++++++++++++++++++++++++++-
 tests/test-visitor-serialization.c |   2 +-
 util/qemu-sockets.c                |   2 +-
 14 files changed, 201 insertions(+), 30 deletions(-)
qapi/opts-visitor.c +0 -0
index 4cf1cf8..00e4b1a 100644
@@ -347,6 +347,7 @@ opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp)
     }
 
Quoted 4 lines

qapi/opts-visitor.c opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp) · line 350

Quoted 2 lines
qapi/opts-visitor.c +1 -0
     if (opt->str) {
+        /* Keep these values in sync with same code in qmp-input-visitor.c */

Also with parse_option_bool(). That's the canonical place, in fact.

Quoted 4 lines

....except parse_option_bool() doesn't allow "yes", "no", "y", "n" as valid values - it only permits "on" and "off" :-( I guess I could make the parse_option_bool() method non-static, make it accept these missing values, and then call it from all places so we have guaranteed consistency.

Or factor out its parsing guts and put them into cutils.c. Similar to how qemu_strtol() & friends are (or should be) the common number parsers.

But that's optional. All I'm asking for this patch is an updated comment.

Quoted 71 lines
Quoted 56 lines
Quoted 36 lines
qapi/opts-visitor.c +0 -0
         if (strcmp(opt->str, "on") == 0 ||
             strcmp(opt->str, "yes") == 0 ||
             strcmp(opt->str, "y") == 0) {
qapi/qmp-input-visitor.c +5 -4
index aea90a1..92023b1 100644
@@ -20,6 +20,7 @@
 #include "qemu-common.h"
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/qerror.h"
+#include "qemu/cutils.h"
 
 #define QIV_STACK_SIZE 1024
 
@@ -45,6 +46,7 @@ struct QmpInputVisitor
 
     /* True to reject parse in visit_end_struct() if unvisited keys remain. */
     bool strict;
+    bool autocast;
 };
 
 static QmpInputVisitor *to_qiv(Visitor *v)
@@ -254,15 +256,25 @@ static void qmp_input_type_int64(Visitor *v, const char *name, int64_t *obj,
                                  Error **errp)
 {
     QmpInputVisitor *qiv = to_qiv(v);
-    QInt *qint = qobject_to_qint(qmp_input_get_object(qiv, name, true));
+    QObject *qobj = qmp_input_get_object(qiv, name, true);
+    QInt *qint;
+    QString *qstr;
 
-    if (!qint) {
-        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
-                   "integer");

qapi/qmp-input-visitor.c static void qmp_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 271

Quoted 10 lines
qapi/qmp-input-visitor.c +6 -1
+    qint = qobject_to_qint(qobj);
+    if (qint) {
+        *obj = qint_get_int(qint);
         return;
     }
 
-    *obj = qint_get_int(qint);
+    qstr = qobject_to_qstring(qobj);
+    if (qstr && qstr->string && qiv->autocast) {
+        if (qemu_strtoll(qstr->string, NULL, 10, obj) == 0) {

In the commit message, you mentioned intended use: "with a QDict produced from QemuOpts, where everything is in string format." I figure you mean opts with empty opts->list->desc[]. For those, we accept any key and parse all values as strings.

The idea with such QemuOpts is to *delay* parsing until we know how to parse. The delay should be transparent to the user. In particular, values should be parsed the same whether the parsing is delayed or not.

Quoted 8 lines

The canonical QemuOpts value parser is qemu_opt_parse(). It parses integers with parse_option_number(). That function parses like stroull(qstr->string, ..., 0). Two differences:

* stroll() vs. strtoull(): they differ in ERANGE behavior. This might be tolerable, but I haven't thought it through.

* Base 0 vs 10: different syntax and semantics. Oops.

Sigh, I originally had my code using strtoull() directly as the parse_option_number() method does, but had to change it to make checkpatch.pl stfu thus creating incosistency. This is a great example of why I hate the fact that we only validate our style rules against new patches and not our existing codebase :-(

Quoted 4 lines

Anyway, it seems to be something where we should refactor code to use the same parsing method from both places. eg by making the parse_option_number method non-static and just calling it from here.

To get this patch accepted, you need to fix the inconsistency. Refactoring to improve our chances the inconsistency stays fixed is welcome, but optional.