Skip to contentSkip to conversation

QAPI/QOM work for non-scalar object properties

PATCH Applied Daniel P. Berrange 8 participants 109 messages Sep 30, 2016 – Jul 12, 2017 Page 1 of 2

Subsystems (4)
lore
Share & tools
lore
#1 lore raw

PATCH v14 1/21 option: make parse_option_bool/number non-static

In reply to Daniel P. Berrange

The opts-visitor.c opts_type_bool() method has code for parsing a string to set a bool value, as does the qemu-option.c parse_option_bool() method, except it handles fewer cases.

To enable consistency across the codebase, extend parse_option_bool() to handle "yes", "no", "y" and "n", and make it non-static. Convert the opts visitor to call this method directly.

Also make parse_option_number() non-static to allow for similar reuse later.

---
 include/qemu/option.h         |  4 ++++
 qapi/opts-visitor.c           | 19 +------------------
 tests/qemu-iotests/051.out    |  6 +++---
 tests/qemu-iotests/051.pc.out |  6 +++---
 tests/qemu-iotests/137.out    |  4 ++--
 util/qemu-option.c            | 27 ++++++++++++++++-----------
 6 files changed, 29 insertions(+), 37 deletions(-)
include/qemu/option.h +4 -0
index 1f9e3f9..2a5266f 100644
@@ -37,6 +37,10 @@ int get_param_value(char *buf, int buf_size,
                     const char *tag, const char *str);
 
 
+void parse_option_bool(const char *name, const char *value, bool *ret,
+                       Error **errp);
+void parse_option_number(const char *name, const char *value,
+                         uint64_t *ret, Error **errp);
 void parse_option_size(const char *name, const char *value,
                        uint64_t *ret, Error **errp);
 bool has_help_option(const char *param);
qapi/opts-visitor.c +1 -18
index 1048bbc..084f7cc 100644
@@ -334,7 +334,6 @@ opts_type_str(Visitor *v, const char *name, char **obj, Error **errp)
 }
 
 
-/* mimics qemu-option.c::parse_option_bool() */
 static void
 opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp)
 {
@@ -346,23 +345,7 @@ opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp)
         return;
     }
 
-    if (opt->str) {
-        if (strcmp(opt->str, "on") == 0 ||
-            strcmp(opt->str, "yes") == 0 ||
-            strcmp(opt->str, "y") == 0) {
-            *obj = true;
-        } else if (strcmp(opt->str, "off") == 0 ||
-            strcmp(opt->str, "no") == 0 ||
-            strcmp(opt->str, "n") == 0) {
-            *obj = false;
-        } else {
-            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, opt->name,
-                       "on|yes|y|off|no|n");
-            return;
-        }
-    } else {
-        *obj = true;
-    }
+    parse_option_bool(opt->name, opt->str, obj, errp);
 
     processed(ov, name);
 }
tests/qemu-iotests/051.out +3 -3
index 408d613..ffdc0b9 100644
@@ -86,13 +86,13 @@ QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) qququiquit
 
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=
-QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: Parameter 'lazy-refcounts' expects 'on' or 'off'
+QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42
-QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42: Parameter 'lazy-refcounts' expects 'on' or 'off'
+QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo
-QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo: Parameter 'lazy-refcounts' expects 'on' or 'off'
+QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 
 
 === With version 2 images enabling lazy refcounts must fail ===
tests/qemu-iotests/051.pc.out +3 -3
index ec6d222..6abf1ab 100644
@@ -86,13 +86,13 @@ QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) qququiquit
 
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=
-QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: Parameter 'lazy-refcounts' expects 'on' or 'off'
+QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42
-QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42: Parameter 'lazy-refcounts' expects 'on' or 'off'
+QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo
-QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo: Parameter 'lazy-refcounts' expects 'on' or 'off'
+QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 
 
 === With version 2 images enabling lazy refcounts must fail ===
tests/qemu-iotests/137.out +2 -2
index c0e7534..61c1f90 100644
@@ -15,7 +15,7 @@ read 33554432/33554432 bytes at offset 0
 
 === Try setting some invalid values ===
 
-Parameter 'lazy-refcounts' expects 'on' or 'off'
+Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 cache-size, l2-cache-size and refcount-cache-size may not be set the same time
 l2-cache-size may not exceed cache-size
 refcount-cache-size may not exceed cache-size
@@ -40,7 +40,7 @@ incompatible_features     0x0
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
 wrote 65536/65536 bytes at offset 0
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-Parameter 'lazy-refcounts' expects 'on' or 'off'
+Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 qcow2: Marking image as corrupt: Preventing invalid write on metadata (overlaps with qcow2_header); further corruption events will be suppressed
 write failed: Input/output error
 *** done
util/qemu-option.c +16 -11
index 3467dc2..41b356c 100644
@@ -125,25 +125,30 @@ int get_param_value(char *buf, int buf_size,
     return get_next_param_value(buf, buf_size, tag, &str);
 }
 
-static void parse_option_bool(const char *name, const char *value, bool *ret,
-                              Error **errp)
+void parse_option_bool(const char *name, const char *value, bool *ret,
+                       Error **errp)
 {
     if (value != NULL) {
-        if (!strcmp(value, "on")) {
-            *ret = 1;
-        } else if (!strcmp(value, "off")) {
-            *ret = 0;
+        if (strcmp(value, "on") == 0 ||
+            strcmp(value, "yes") == 0 ||
+            strcmp(value, "y") == 0) {
+            *ret = true;
+        } else if (strcmp(value, "off") == 0 ||
+            strcmp(value, "no") == 0 ||
+            strcmp(value, "n") == 0) {
+            *ret = false;
         } else {
-            error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
-                       name, "'on' or 'off'");
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name,
+                       "'on', 'yes', 'y', 'off', 'no' or 'n'");
+            return;
         }
     } else {
-        *ret = 1;
+        *ret = true;
     }
 }
 
-static void parse_option_number(const char *name, const char *value,
-                                uint64_t *ret, Error **errp)
+void parse_option_number(const char *name, const char *value,
+                         uint64_t *ret, Error **errp)
 {
     char *postfix;
     uint64_t number;
Signature · 1 line
2.7.4
#0 lore raw

v14 cover QAPI/QOM work for non-scalar object properties

An update of a series previously posted

v1: https://lists.gnu.org/archive/html/qemu-devel/2016-02/msg04618.html
v2: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg01454.html
v3: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg02498.html
v4: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01661.html
v5: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg00485.html
v6: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg03876.html
v7: https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg00919.html
v8: https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg03115.html
v9: https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg02653.html
v10: https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg02694.html
v11: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg00652.html
v12: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg03559.html
v13: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg04212.html
v14: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg07037.html

This series provides the infrastructure to allow use of non-scalar properties with the -object CLI arg, and object_add monitor commands. eg a property which is a list of structs. The syntax used for this is intentionally compatible with the syntax used by the block layer. This will allow the qdict_crumple method to be used by the block layer to convert from QemuOpts into structured QAPI block layer structs at a future date. It is already used by one of Max's patch series, by Kevin's -blockdev series, and by my own ACL series.

This patch series has grown a fair bit larger since previous postings (19^H^H21 patches instead of 6). This was to avoid trying to squish too much into one patch, to make it easier to review.

It does not neccessarily need to be merged in one go. There are three logical blocks of patches here.

Patches 1-8 are the core support for visiting QemuOpts using the QObjectInputVisitor class. Merging this would unblock at least Kevin's blockdev series, and probably Max's series too.

The patches 9-14 add enhancements to the code to support various special cases / edge cases needed to provide fully semantic replacement of the OptsVisitor

Patches 15-21 deal with converting code over to use the new visitor logic.

Once this is merged, there is also scope to further enhance the QObjectInputVisitor to deal with visiting single-properties, which would let us delete StringInputVisitor too. This would mean we have a single input visitor class, instead of the current three.

Changed in v15:

- Re-arranged patch series to allow easier incremental merge of core functionality
- Extra error checking of some scenarios to be avoided (Eric)
- Misc typos (Eric)
- Rename QEMU_OPTS_REPEAT_POLICY_LIST to QEMU_OPTS_REPEAT_POLICY_ALL, and add QEMU_OPTS_REPEAT_POLICY_ERROR (Eric)

Changed in v14:

- Extend qemu_opts_to_qdict to optionally handle case of repeated keys without dropping them.
- Support parsing of integer ranges for "list of int" visits for back compat with OptsVisitor
- Support visiting nested structs which are represented as a flat keyspace for back compat with OptsVisitor
- Add trace events for visitor functions to facilitate debugging
- Add a QObjectInputVisitor constructor which takes a QemuOpts object, to avoid repeated code patterns in callers
- Greater test coverage of -object to validate correct handling of "list of ints" legacy OptsVisitor syntax
- Convert all other use of OptsVisitor to QObjectInputVisitor
- Delete OptsVisitor

Changed in v13:

- Fix typos (Kevin)
- Remove unneeded line breaks (Kevin)

Changed in v12:

- More user friendly error message for mixing dict/list keys (Kevin)
- Report error instead of assert for non-contiguous list keys (Kevin)
- Fix tests for non-contiguous list keys (Kevin)
- Add tests for escaping of '.' when crumpling (Kevin)
- Fix remaining references to Qmp(In|Out)putVisitor (Markus)
- Misc typos / whitespace fixes (Eric, Kevin)
- Avoid touching 'ret' when parsing int64 fails (Eric)
- Testing of more edge cases in QObjectInputVisitor (Eric)
- Simplify API doc format (Markus)
- Use parse_option_size instead of qemu_strtosz_suffix for consistency (Kevin)
- Use safer qobject_to_qdict casts (Eric)

Changed in v11:

- Split QAPI/QOM patches off from the access control patches

Changed in v10:

- Fixed stupid build mistake

Changed in v9:

- Rename QmpInputVisitor -> QObjectInputVisitor (Markus/Eric)
- Rename QmpOutputVisitor -> QObjectOutputVisitor (Markus/Eric)
- Drop "strict" param from qobject_string_visitor_new() (Marus)
- Misc docs typos
- Add a visitor able to use strict or string types (for Eric's netdev series)
- Add a authorization API implementation that uses PAM

Changed in v8:

- Rebase due to merge of Visitor API changes (Eric)

Changed in v7:

- Misc typos in API docs (Marc-André)
- Fix parsing of properties using type_size visitor (Marc-André)
- Mark based auth class as abstract (Marc-André)
- Fix QAPI version annotations to say 2.7 (Marc-André)

Changed in v6:

- Switch from while() to for() loop for iterating over dicts (Markus)
- Avoid redundant strdup (Markus)
- Rewrap comments at 70 chars (Markus)
- Change qdict_list_size() to qdict_is_list() (Markus)
- Misc docs changes (Markus)
- Change QmpInputVisitor so the code for handling the string types is separate from code using native scalar types (Paolo)
- Centralize code parsing bool strings (Markus)
- Centralize code parsing int strings (Markus)

Changed in v5:

- Resolved conflicts with Eric's visitor refactoring which made it stricter about struct begin/end calls
- Added support for ACLs to migration code now its TLS support is merged.
- Fixed typos in example in commit message

Changed in v4:
- Ensure examples use shell escaping for '*' (Eric)
- Add more tests for crumple impl (Eric)
- Raise error if sasl-acl/tls-acl are requested but sasl/tls auth are not enabled (Eric)
- Document return codes for auth check more clearly (Eric)
- Don't silently turn a glob match into a strcmp
- Other misc small typos/fixes (Eric)

Changed in v3:

- Created separate qdict_list_size method (Max)
- Added unit tests for case of empty dict (Max)
- Fix variable names to use underscore separator (Max)
- Fix potential free of uninitialized variables (Max)
- Use QObject APIs for casts, instead of C type casts (Max)

Changed in v2:

- Adapt to changes in qapi visitor APIs
- Add a 'bool recursive' flag to qdict_crumple (Max)
- Fix memory leaks in qdict_crumple (Max)
- Split out key splitting code from qdict_crumple (Max)
- Use saner variable names in qdict_crumple (Max)
- Added some tests for bad inputs to qdict_crumple

Daniel P. Berrange (21):
option: make parse_option_bool/number non-static
qdict: implement a qdict_crumple method for un-flattening a dict
qapi: add trace events for visitor
qapi: rename QmpInputVisitor to QObjectInputVisitor
qapi: rename QmpOutputVisitor to QObjectOutputVisitor
qapi: don't pass two copies of TestInputVisitorData to tests
qapi: permit scalar type conversions in QObjectInputVisitor
qapi: allow QObjectInputVisitor to be created with QemuOpts
qapi: permit auto-creating single element lists
qapi: permit auto-creating nested structs
qapi: add integer range support for QObjectInputVisitor
option: allow qemu_opts_to_qdict to merge repeated options
qdict: allow qdict_crumple to accept compound types as values
qapi: allow repeated opts with qobject_input_visitor_new_opts
qom: support non-scalar properties with -object
hmp: support non-scalar properties with object_add
numa: convert to use QObjectInputVisitor for -numa
block: convert crypto driver to use QObjectInputVisitor
acpi: convert to QObjectInputVisitor for -acpi parsing
net: convert to QObjectInputVisitor for -net/-netdev parsing
qapi: delete unused OptsVisitor code

 Makefile.objs                                      |   1 +
 block/crypto.c                                     |  14 +-
 block/qapi.c                                       |   4 +-
 blockdev.c                                         |  11 +-
 docs/qapi-code-gen.txt                             |   4 +-
 hmp.c                                              |  25 +-
 hw/acpi/core.c                                     |  14 +-
 include/qapi/opts-visitor.h                        |  40 --
 include/qapi/qmp-input-visitor.h                   |  30 -
 include/qapi/qmp/qdict.h                           |   1 +
 include/qapi/qobject-input-visitor.h               | 130 ++++
 ...p-output-visitor.h => qobject-output-visitor.h} |  10 +-
 include/qapi/visitor.h                             |   6 +-
 include/qemu/option.h                              |  17 +-
 include/sysemu/numa_int.h                          |  11 +
 monitor.c                                          |   5 +-
 net/net.c                                          |  18 +-
 numa.c                                             |  36 +-
 qapi/Makefile.objs                                 |   6 +-
 qapi/opts-visitor.c                                | 561 ---------------
 qapi/qapi-clone-visitor.c                          |   2 +-
 qapi/qapi-visit-core.c                             |  27 +
 qapi/qmp-input-visitor.c                           | 412 -----------
 qapi/qmp-output-visitor.c                          | 256 -------
 qapi/qobject-input-visitor.c                       | 785 +++++++++++++++++++++
 qapi/qobject-output-visitor.c                      | 254 +++++++
 qapi/trace-events                                  |  33 +
 qemu-img.c                                         |  12 +-
 qemu-io-cmds.c                                     |   3 +-
 qemu-io.c                                          |   6 +-
 qemu-nbd.c                                         |   3 +-
 qmp.c                                              |   4 +-
 qobject/qdict.c                                    | 302 ++++++++
 qom/object_interfaces.c                            |  42 +-
 qom/qom-qobject.c                                  |   8 +-
 scripts/qapi-commands.py                           |   8 +-
 scripts/qapi-event.py                              |   4 +-
 stubs/Makefile.objs                                |   5 +
 stubs/exec.c                                       |   6 +
 stubs/hostmem.c                                    |  14 +
 stubs/memory.c                                     |  41 ++
 stubs/qdev.c                                       |   8 +
 stubs/vl.c                                         |   8 +
 stubs/vmstate.c                                    |   4 +
 target-s390x/cpu_models.c                          |   4 +-
 tests/.gitignore                                   |   6 +-
 tests/Makefile.include                             |  27 +-
 tests/check-qdict.c                                | 436 ++++++++++++
 tests/check-qnull.c                                |   8 +-
 tests/check-qom-proplist.c                         | 367 +++++++++-
 tests/qemu-iotests/051.out                         |   6 +-
 tests/qemu-iotests/051.pc.out                      |   6 +-
 tests/qemu-iotests/137.out                         |   4 +-
 tests/test-numa.c                                  | 116 +++
 tests/test-opts-visitor.c                          | 268 -------
 tests/test-qemu-opts.c                             | 132 ++++
 tests/test-qmp-commands.c                          |   4 +-
 ...-input-strict.c => test-qobject-input-strict.c} |   6 +-
 ...nput-visitor.c => test-qobject-input-visitor.c} | 637 +++++++++++++++--
 ...put-visitor.c => test-qobject-output-visitor.c} |   6 +-
 tests/test-replication.c                           |   9 +-
 tests/test-string-input-visitor.c                  |   2 +-
 tests/test-string-output-visitor.c                 |   2 +-
 tests/test-visitor-serialization.c                 |   8 +-
 util/qemu-option.c                                 |  91 ++-
 util/qemu-sockets.c                                |   4 +-
 vl.c                                               |   1 -
 67 files changed, 3566 insertions(+), 1775 deletions(-)
 delete mode 100644 include/qapi/opts-visitor.h
 delete mode 100644 include/qapi/qmp-input-visitor.h
 create mode 100644 include/qapi/qobject-input-visitor.h
 rename include/qapi/{qmp-output-visitor.h => qobject-output-visitor.h} (66%)
 create mode 100644 include/sysemu/numa_int.h
 delete mode 100644 qapi/opts-visitor.c
 delete mode 100644 qapi/qmp-input-visitor.c
 delete mode 100644 qapi/qmp-output-visitor.c
 create mode 100644 qapi/qobject-input-visitor.c
 create mode 100644 qapi/qobject-output-visitor.c
 create mode 100644 qapi/trace-events
 create mode 100644 stubs/exec.c
 create mode 100644 stubs/hostmem.c
 create mode 100644 stubs/memory.c
 create mode 100644 stubs/qdev.c
 create mode 100644 stubs/vl.c
 create mode 100644 tests/test-numa.c
 delete mode 100644 tests/test-opts-visitor.c
 rename tests/{test-qmp-input-strict.c => test-qobject-input-strict.c} (98%)
 rename tests/{test-qmp-input-visitor.c => test-qobject-input-visitor.c} (57%)
 rename tests/{test-qmp-output-visitor.c => test-qobject-output-visitor.c} (99%)
Signature · 1 line
2.7.4
#5 lore raw

PATCH v14 2/21 qdict: implement a qdict_crumple method for un-flattening a dict

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

{
'foo.0.bar': 'one',
'foo.0.wizz': '1',
'foo.1.bar': 'two',
'foo.1.wizz': '2'
}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

{
'foo': [

     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }

],
}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 289 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 261 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 551 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..e0d24e1 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +289 -0

301 lines not on this page · raw message

tests/check-qdict.c +261 -0

279 lines not on this page · raw message

Signature · 1 line
2.7.4
#16 lore raw

PATCH v14 3/21 qapi: add trace events for visitor

In reply to Daniel P. Berrange

Allow tracing of the operation of visitors

---
 Makefile.objs          |  1 +
 qapi/qapi-visit-core.c | 27 +++++++++++++++++++++++++++
 qapi/trace-events      | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+)
 create mode 100644 qapi/trace-events
Makefile.objs +1 -0
index a8e0224..b3e8aef 100644
@@ -160,3 +160,4 @@ trace-events-y += target-s390x/trace-events
 trace-events-y += target-ppc/trace-events
 trace-events-y += qom/trace-events
 trace-events-y += linux-user/trace-events
+trace-events-y += qapi/trace-events
qapi/qapi-visit-core.c +27 -0
index 55f5876..bfcb276 100644
@@ -19,10 +19,12 @@
 #include "qapi/qmp/qerror.h"
 #include "qapi/visitor.h"
 #include "qapi/visitor-impl.h"
+#include "trace.h"
 
 void visit_complete(Visitor *v, void *opaque)
 {
     assert(v->type != VISITOR_OUTPUT || v->complete);
+    trace_visit_complete(v, opaque);
     if (v->complete) {
         v->complete(v, opaque);
     }
@@ -30,6 +32,7 @@ void visit_complete(Visitor *v, void *opaque)
 
 void visit_free(Visitor *v)
 {
+    trace_visit_free(v);
     if (v) {
         v->free(v);
     }
@@ -40,6 +43,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,
 {
     Error *err = NULL;
 
+    trace_visit_start_struct(v, name, obj, size);
     if (obj) {
         assert(size);
         assert(!(v->type & VISITOR_OUTPUT) || *obj);
@@ -53,6 +57,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,
 
 void visit_check_struct(Visitor *v, Error **errp)
 {
+    trace_visit_check_struct(v);
     if (v->check_struct) {
         v->check_struct(v, errp);
     }
@@ -60,6 +65,7 @@ void visit_check_struct(Visitor *v, Error **errp)
 
 void visit_end_struct(Visitor *v, void **obj)
 {
+    trace_visit_end_struct(v, obj);
     v->end_struct(v, obj);
 }
 
@@ -69,6 +75,7 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,
     Error *err = NULL;
 
     assert(!list || size >= sizeof(GenericList));
+    trace_visit_start_list(v, name, list, size);
     v->start_list(v, name, list, size, &err);
     if (list && (v->type & VISITOR_INPUT)) {
         assert(!(err && *list));
@@ -79,11 +86,13 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,
 GenericList *visit_next_list(Visitor *v, GenericList *tail, size_t size)
 {
     assert(tail && size >= sizeof(GenericList));
+    trace_visit_next_list(v, tail, size);
     return v->next_list(v, tail, size);
 }
 
 void visit_end_list(Visitor *v, void **obj)
 {
+    trace_visit_end_list(v, obj);
     v->end_list(v, obj);
 }
 
@@ -95,6 +104,7 @@ void visit_start_alternate(Visitor *v, const char *name,
 
     assert(obj && size >= sizeof(GenericAlternate));
     assert(!(v->type & VISITOR_OUTPUT) || *obj);
+    trace_visit_start_alternate(v, name, obj, size, promote_int);
     if (v->start_alternate) {
         v->start_alternate(v, name, obj, size, promote_int, &err);
     }
@@ -106,6 +116,7 @@ void visit_start_alternate(Visitor *v, const char *name,
 
 void visit_end_alternate(Visitor *v, void **obj)
 {
+    trace_visit_end_alternate(v, obj);
     if (v->end_alternate) {
         v->end_alternate(v, obj);
     }
@@ -113,6 +124,7 @@ void visit_end_alternate(Visitor *v, void **obj)
 
 bool visit_optional(Visitor *v, const char *name, bool *present)
 {
+    trace_visit_optional(v, name, present);
     if (v->optional) {
         v->optional(v, name, present);
     }
@@ -127,6 +139,7 @@ bool visit_is_input(Visitor *v)
 void visit_type_int(Visitor *v, const char *name, int64_t *obj, Error **errp)
 {
     assert(obj);
+    trace_visit_type_int(v, name, obj);
     v->type_int64(v, name, obj, errp);
 }
 
@@ -151,6 +164,7 @@ void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj,
                       Error **errp)
 {
     uint64_t value = *obj;
+    trace_visit_type_uint8(v, name, obj);
     visit_type_uintN(v, &value, name, UINT8_MAX, "uint8_t", errp);
     *obj = value;
 }
@@ -159,6 +173,7 @@ void visit_type_uint16(Visitor *v, const char *name, uint16_t *obj,
                        Error **errp)
 {
     uint64_t value = *obj;
+    trace_visit_type_uint16(v, name, obj);
     visit_type_uintN(v, &value, name, UINT16_MAX, "uint16_t", errp);
     *obj = value;
 }
@@ -167,6 +182,7 @@ void visit_type_uint32(Visitor *v, const char *name, uint32_t *obj,
                        Error **errp)
 {
     uint64_t value = *obj;
+    trace_visit_type_uint32(v, name, obj);
     visit_type_uintN(v, &value, name, UINT32_MAX, "uint32_t", errp);
     *obj = value;
 }
@@ -175,6 +191,7 @@ void visit_type_uint64(Visitor *v, const char *name, uint64_t *obj,
                        Error **errp)
 {
     assert(obj);
+    trace_visit_type_uint64(v, name, obj);
     v->type_uint64(v, name, obj, errp);
 }
 
@@ -199,6 +216,7 @@ static void visit_type_intN(Visitor *v, int64_t *obj, const char *name,
 void visit_type_int8(Visitor *v, const char *name, int8_t *obj, Error **errp)
 {
     int64_t value = *obj;
+    trace_visit_type_int8(v, name, obj);
     visit_type_intN(v, &value, name, INT8_MIN, INT8_MAX, "int8_t", errp);
     *obj = value;
 }
@@ -207,6 +225,7 @@ void visit_type_int16(Visitor *v, const char *name, int16_t *obj,
                       Error **errp)
 {
     int64_t value = *obj;
+    trace_visit_type_int16(v, name, obj);
     visit_type_intN(v, &value, name, INT16_MIN, INT16_MAX, "int16_t", errp);
     *obj = value;
 }
@@ -215,6 +234,7 @@ void visit_type_int32(Visitor *v, const char *name, int32_t *obj,
                       Error **errp)
 {
     int64_t value = *obj;
+    trace_visit_type_int32(v, name, obj);
     visit_type_intN(v, &value, name, INT32_MIN, INT32_MAX, "int32_t", errp);
     *obj = value;
 }
@@ -223,6 +243,7 @@ void visit_type_int64(Visitor *v, const char *name, int64_t *obj,
                       Error **errp)
 {
     assert(obj);
+    trace_visit_type_int64(v, name, obj);
     v->type_int64(v, name, obj, errp);
 }
 
@@ -230,6 +251,7 @@ void visit_type_size(Visitor *v, const char *name, uint64_t *obj,
                      Error **errp)
 {
     assert(obj);
+    trace_visit_type_size(v, name, obj);
     if (v->type_size) {
         v->type_size(v, name, obj, errp);
     } else {
@@ -240,6 +262,7 @@ void visit_type_size(Visitor *v, const char *name, uint64_t *obj,
 void visit_type_bool(Visitor *v, const char *name, bool *obj, Error **errp)
 {
     assert(obj);
+    trace_visit_type_bool(v, name, obj);
     v->type_bool(v, name, obj, errp);
 }
 
@@ -252,6 +275,7 @@ void visit_type_str(Visitor *v, const char *name, char **obj, Error **errp)
      * can enable:
     assert(!(v->type & VISITOR_OUTPUT) || *obj);
      */
+    trace_visit_type_str(v, name, obj);
     v->type_str(v, name, obj, &err);
     if (v->type & VISITOR_INPUT) {
         assert(!err != !*obj);
@@ -263,6 +287,7 @@ void visit_type_number(Visitor *v, const char *name, double *obj,
                        Error **errp)
 {
     assert(obj);
+    trace_visit_type_number(v, name, obj);
     v->type_number(v, name, obj, errp);
 }
 
@@ -272,6 +297,7 @@ void visit_type_any(Visitor *v, const char *name, QObject **obj, Error **errp)
 
     assert(obj);
     assert(v->type != VISITOR_OUTPUT || *obj);
+    trace_visit_type_any(v, name, obj);
     v->type_any(v, name, obj, &err);
     if (v->type == VISITOR_INPUT) {
         assert(!err != !*obj);
@@ -281,6 +307,7 @@ void visit_type_any(Visitor *v, const char *name, QObject **obj, Error **errp)
 
 void visit_type_null(Visitor *v, const char *name, Error **errp)
 {
+    trace_visit_type_null(v, name);
     v->type_null(v, name, errp);
 }
 
qapi/trace-events new file +33 -0
new file mode 100644
index 0000000..2c5d3bc
@@ -0,0 +1,33 @@
+# qapi-visit-core.c
+visit_free(void *v) "v=%p"
+visit_complete(void *v, void *opaque) "v=%p opaque=%p"
+
+visit_start_struct(void *v, const char *name, void *obj, size_t size) "v=%p name=%s obj=%p size=%zu"
+visit_check_struct(void *v) "v=%p"
+visit_end_struct(void *v, void *obj) "v=%p obj=%p"
+
+visit_start_list(void *v, const char *name, void *obj, size_t size) "v=%p name=%s obj=%p size=%zu"
+visit_next_list(void *v, void *tail, size_t size) "v=%p tail=%p size=%zu"
+visit_end_list(void *v, void *obj) "v=%p obj=%p"
+
+visit_start_alternate(void *v, const char *name, void *obj, size_t size, bool promote_int) "v=%p name=%s obj=%p size=%zu promote_int=%d"
+visit_end_alternate(void *v, void *obj) "v=%p obj=%p"
+
+visit_optional(void *v, const char *name, bool *present) "v=%p name=%s present=%p"
+
+visit_type_enum(void *v, const char *name, int *obj) "v=%p name=%s obj=%p"
+visit_type_int(void *v, const char *name, int64_t *obj) "v=%p name=%s obj=%p"
+visit_type_uint8(void *v, const char *name, uint8_t *obj) "v=%p name=%s obj=%p"
+visit_type_uint16(void *v, const char *name, uint16_t *obj) "v=%p name=%s obj=%p"
+visit_type_uint32(void *v, const char *name, uint32_t *obj) "v=%p name=%s obj=%p"
+visit_type_uint64(void *v, const char *name, uint64_t *obj) "v=%p name=%s obj=%p"
+visit_type_int8(void *v, const char *name, int8_t *obj) "v=%p name=%s obj=%p"
+visit_type_int16(void *v, const char *name, int16_t *obj) "v=%p name=%s obj=%p"
+visit_type_int32(void *v, const char *name, int32_t *obj) "v=%p name=%s obj=%p"
+visit_type_int64(void *v, const char *name, int64_t *obj) "v=%p name=%s obj=%p"
+visit_type_size(void *v, const char *name, uint64_t *obj) "v=%p name=%s obj=%p"
+visit_type_bool(void *v, const char *name, bool *obj) "v=%p name=%s obj=%p"
+visit_type_str(void *v, const char *name, char **obj) "v=%p name=%s obj=%p"
+visit_type_number(void *v, const char *name, double *obj) "v=%p name=%s obj=%p"
+visit_type_any(void *v, const char *name, void *obj) "v=%p name=%s obj=%p"
+visit_type_null(void *v, const char *name) "v=%p name=%s"
Signature · 1 line
2.7.4
#26 lore raw

PATCH v14 6/21 qapi: don't pass two copies of TestInputVisitorData to tests

In reply to Daniel P. Berrange

The input_visitor_test_add() method was accepting an instance of 'TestInputVisitorData' and passing it as the 'user_data' parameter to test functions. The main 'TestInputVisitorData' instance that was actually used, was meanwhile being allocated automatically by the test framework fixture setup.

The 'user_data' parameter is going to be needed for tests added in later patches, so getting rid of the current mistaken usage now allows this.

---
 tests/test-qobject-input-visitor.c | 76 ++++++++++++++++----------------------
 1 file changed, 32 insertions(+), 44 deletions(-)
tests/test-qobject-input-visitor.c +32 -44
index 0e65e63..26c5012 100644
@@ -747,10 +747,11 @@ static void test_visitor_in_native_list_number(TestInputVisitorData *data,
 }
 
 static void input_visitor_test_add(const char *testpath,
-                                   TestInputVisitorData *data,
-                                   void (*test_func)(TestInputVisitorData *data, const void *user_data))
+                                   const void *user_data,
+                                   void (*test_func)(TestInputVisitorData *data,
+                                                     const void *user_data))
 {
-    g_test_add(testpath, TestInputVisitorData, data, NULL, test_func,
+    g_test_add(testpath, TestInputVisitorData, user_data, NULL, test_func,
                visitor_input_teardown);
 }
 
@@ -833,77 +834,64 @@ static void test_visitor_in_wrong_type(TestInputVisitorData *data,
 
 int main(int argc, char **argv)
 {
-    TestInputVisitorData in_visitor_data;
-
     g_test_init(&argc, &argv, NULL);
 
     input_visitor_test_add("/visitor/input/int",
-                           &in_visitor_data, test_visitor_in_int);
+                           NULL, test_visitor_in_int);
     input_visitor_test_add("/visitor/input/int_overflow",
-                           &in_visitor_data, test_visitor_in_int_overflow);
+                           NULL, test_visitor_in_int_overflow);
     input_visitor_test_add("/visitor/input/bool",
-                           &in_visitor_data, test_visitor_in_bool);
+                           NULL, test_visitor_in_bool);
     input_visitor_test_add("/visitor/input/number",
-                           &in_visitor_data, test_visitor_in_number);
+                           NULL, test_visitor_in_number);
     input_visitor_test_add("/visitor/input/string",
-                           &in_visitor_data, test_visitor_in_string);
+                           NULL, test_visitor_in_string);
     input_visitor_test_add("/visitor/input/enum",
-                           &in_visitor_data, test_visitor_in_enum);
+                           NULL, test_visitor_in_enum);
     input_visitor_test_add("/visitor/input/struct",
-                           &in_visitor_data, test_visitor_in_struct);
+                           NULL, test_visitor_in_struct);
     input_visitor_test_add("/visitor/input/struct-nested",
-                           &in_visitor_data, test_visitor_in_struct_nested);
+                           NULL, test_visitor_in_struct_nested);
     input_visitor_test_add("/visitor/input/list",
-                           &in_visitor_data, test_visitor_in_list);
+                           NULL, test_visitor_in_list);
     input_visitor_test_add("/visitor/input/any",
-                           &in_visitor_data, test_visitor_in_any);
+                           NULL, test_visitor_in_any);
     input_visitor_test_add("/visitor/input/null",
-                           &in_visitor_data, test_visitor_in_null);
+                           NULL, test_visitor_in_null);
     input_visitor_test_add("/visitor/input/union-flat",
-                           &in_visitor_data, test_visitor_in_union_flat);
+                           NULL, test_visitor_in_union_flat);
     input_visitor_test_add("/visitor/input/alternate",
-                           &in_visitor_data, test_visitor_in_alternate);
+                           NULL, test_visitor_in_alternate);
     input_visitor_test_add("/visitor/input/errors",
-                           &in_visitor_data, test_visitor_in_errors);
+                           NULL, test_visitor_in_errors);
     input_visitor_test_add("/visitor/input/wrong-type",
-                           &in_visitor_data, test_visitor_in_wrong_type);
+                           NULL, test_visitor_in_wrong_type);
     input_visitor_test_add("/visitor/input/alternate-number",
-                           &in_visitor_data, test_visitor_in_alternate_number);
+                           NULL, test_visitor_in_alternate_number);
     input_visitor_test_add("/visitor/input/native_list/int",
-                           &in_visitor_data,
-                           test_visitor_in_native_list_int);
+                           NULL, test_visitor_in_native_list_int);
     input_visitor_test_add("/visitor/input/native_list/int8",
-                           &in_visitor_data,
-                           test_visitor_in_native_list_int8);
+                           NULL, test_visitor_in_native_list_int8);
     input_visitor_test_add("/visitor/input/native_list/int16",
-                           &in_visitor_data,
-                           test_visitor_in_native_list_int16);
+                           NULL, test_visitor_in_native_list_int16);
     input_visitor_test_add("/visitor/input/native_list/int32",
-                           &in_visitor_data,
-                           test_visitor_in_native_list_int32);
+                           NULL, test_visitor_in_native_list_int32);
     input_visitor_test_add("/visitor/input/native_list/int64",
-                           &in_visitor_data,
-                           test_visitor_in_native_list_int64);
+                           NULL, test_visitor_in_native_list_int64);
     input_visitor_test_add("/visitor/input/native_list/uint8",
-                           &in_visitor_data,
-                           test_visitor_in_native_list_uint8);
+                           NULL, test_visitor_in_native_list_uint8);
     input_visitor_test_add("/visitor/input/native_list/uint16",
-                           &in_visitor_data,
-                           test_visitor_in_native_list_uint16);
+                           NULL, test_visitor_in_native_list_uint16);
     input_visitor_test_add("/visitor/input/native_list/uint32",
-                           &in_visitor_data,
-                           test_visitor_in_native_list_uint32);
+                           NULL, test_visitor_in_native_list_uint32);
     input_visitor_test_add("/visitor/input/native_list/uint64",
-                           &in_visitor_data,
-                           test_visitor_in_native_list_uint64);
+                           NULL, test_visitor_in_native_list_uint64);
     input_visitor_test_add("/visitor/input/native_list/bool",
-                           &in_visitor_data, test_visitor_in_native_list_bool);
+                           NULL, test_visitor_in_native_list_bool);
     input_visitor_test_add("/visitor/input/native_list/str",
-                           &in_visitor_data,
-                           test_visitor_in_native_list_string);
+                           NULL, test_visitor_in_native_list_string);
     input_visitor_test_add("/visitor/input/native_list/number",
-                           &in_visitor_data,
-                           test_visitor_in_native_list_number);
+                           NULL, test_visitor_in_native_list_number);
 
     g_test_run();
 
Signature · 1 line
2.7.4
#22 lore raw

PATCH v14 4/21 qapi: rename QmpInputVisitor to QObjectInputVisitor

In reply to Daniel P. Berrange

The QmpInputVisitor has no direct dependency on QMP. It is valid to use it anywhere that one has a QObject. Rename it to better reflect its functionality as a generic QObject to QAPI converter.

---
 docs/qapi-code-gen.txt                             |   2 +-
 ...qmp-input-visitor.h => qobject-input-visitor.h} |  10 +-
 include/qapi/visitor.h                             |   6 +-
 monitor.c                                          |   2 +-
 qapi/Makefile.objs                                 |   2 +-
 ...qmp-input-visitor.c => qobject-input-visitor.c} | 171 +++++++++++----------
 qmp.c                                              |   4 +-
 qom/qom-qobject.c                                  |   4 +-
 scripts/qapi-commands.py                           |   4 +-
 target-s390x/cpu_models.c                          |   4 +-
 tests/.gitignore                                   |   4 +-
 tests/Makefile.include                             |  12 +-
 tests/check-qnull.c                                |   4 +-
 tests/test-qmp-commands.c                          |   4 +-
 ...-input-strict.c => test-qobject-input-strict.c} |   6 +-
 ...nput-visitor.c => test-qobject-input-visitor.c} |   6 +-
 tests/test-string-input-visitor.c                  |   2 +-
 tests/test-visitor-serialization.c                 |   4 +-
 util/qemu-sockets.c                                |   2 +-
 19 files changed, 128 insertions(+), 125 deletions(-)
 rename include/qapi/{qmp-input-visitor.h => qobject-input-visitor.h} (63%)
 rename qapi/{qmp-input-visitor.c => qobject-input-visitor.c} (56%)
 rename tests/{test-qmp-input-strict.c => test-qobject-input-strict.c} (98%)
 rename tests/{test-qmp-input-visitor.c => test-qobject-input-visitor.c} (99%)
docs/qapi-code-gen.txt +1 -1
index 5d4c2cd..d2604b6 100644
@@ -1024,7 +1024,7 @@ Example:
         Visitor *v;
         UserDefOneList *arg1 = NULL;
 
-        v = qmp_input_visitor_new(QOBJECT(args), true);
+        v = qobject_input_visitor_new(QOBJECT(args), true);
         visit_start_struct(v, NULL, NULL, 0, &err);
         if (err) {
             goto out;
include/qapi/qobject-input-visitor.h renamed from include/qapi/qmp-input-visitor.h +5 -5
similarity index 63%
rename from include/qapi/qmp-input-visitor.h
rename to include/qapi/qobject-input-visitor.h
index f3ff5f3..cde328d 100644
@@ -11,20 +11,20 @@
  *
  */
 
-#ifndef QMP_INPUT_VISITOR_H
-#define QMP_INPUT_VISITOR_H
+#ifndef QOBJECT_INPUT_VISITOR_H
+#define QOBJECT_INPUT_VISITOR_H
 
 #include "qapi/visitor.h"
 #include "qapi/qmp/qobject.h"
 
-typedef struct QmpInputVisitor QmpInputVisitor;
+typedef struct QObjectInputVisitor QObjectInputVisitor;
 
 /*
- * Return a new input visitor that converts QMP to QAPI.
+ * Return a new input visitor that converts a QObject to a QAPI object.
  *
  * Set @strict to reject a parse that doesn't consume all keys of a
  * dictionary; otherwise excess input is ignored.
  */
-Visitor *qmp_input_visitor_new(QObject *obj, bool strict);
+Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
 
 #endif
include/qapi/visitor.h +3 -3
index 6c77a91..9bb6cba 100644
@@ -25,14 +25,14 @@
  * for doing work at each node of a QAPI graph; it can also be used
  * for a virtual walk, where there is no actual QAPI C struct.
  *
- * There are four kinds of visitor classes: input visitors (QMP,
+ * There are four kinds of visitor classes: input visitors (QObject,
  * string, and QemuOpts) parse an external representation and build
- * the corresponding QAPI graph, output visitors (QMP and string) take
+ * the corresponding QAPI graph, output visitors (QObject and string) take
  * a completed QAPI graph and generate an external representation, the
  * dealloc visitor can take a QAPI graph (possibly partially
  * constructed) and recursively free its resources, and the clone
  * visitor performs a deep clone of one QAPI object to another.  While
- * the dealloc and QMP input/output visitors are general, the string,
+ * the dealloc and QObject input/output visitors are general, the string,
  * QemuOpts, and clone visitors have some implementation limitations;
  * see the documentation for each visitor for more details on what it
  * supports.  Also, see visitor-impl.h for the callback contracts
monitor.c +1 -1
index 83c4edf..14089cc 100644
@@ -957,7 +957,7 @@ EventInfoList *qmp_query_events(Error **errp)
  * directly into QObject instead of first parsing it with
  * visit_type_SchemaInfoList() into a SchemaInfoList, then marshal it
  * to QObject with generated output marshallers, every time.  Instead,
- * we do it in test-qmp-input-visitor.c, just to make sure
+ * we do it in test-qobject-input-visitor.c, just to make sure
  * qapi-introspect.py's output actually conforms to the schema.
  */
 static void qmp_query_qmp_schema(QDict *qdict, QObject **ret_data,
qapi/Makefile.objs +1 -1
index 7ea4aeb..6ec7bdc 100644
@@ -1,4 +1,4 @@
-util-obj-y = qapi-visit-core.o qapi-dealloc-visitor.o qmp-input-visitor.o
+util-obj-y = qapi-visit-core.o qapi-dealloc-visitor.o qobject-input-visitor.o
 util-obj-y += qmp-output-visitor.o qmp-registry.o qmp-dispatch.o
 util-obj-y += string-input-visitor.o string-output-visitor.o
 util-obj-y += opts-visitor.o qapi-clone-visitor.o
qapi/qobject-input-visitor.c renamed from qapi/qmp-input-visitor.c +87 -84

336 lines not on this page · raw message

qmp.c +2 -2
index 621f6ae..642bf41 100644
@@ -31,7 +31,7 @@
 #include "qom/qom-qobject.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qobject.h"
-#include "qapi/qmp-input-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "hw/boards.h"
 #include "qom/object_interfaces.h"
 #include "hw/mem/pc-dimm.h"
@@ -672,7 +672,7 @@ void qmp_object_add(const char *type, const char *id,
         }
     }
 
-    v = qmp_input_visitor_new(props, true);
+    v = qobject_input_visitor_new(props, true);
     obj = user_creatable_add_type(type, id, pdict, v, errp);
     visit_free(v);
     if (obj) {
qom/qom-qobject.c +2 -2
index c225abc..81959e0 100644
@@ -15,7 +15,7 @@
 #include "qom/object.h"
 #include "qom/qom-qobject.h"
 #include "qapi/visitor.h"
-#include "qapi/qmp-input-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/qmp-output-visitor.h"
 
 void object_property_set_qobject(Object *obj, QObject *value,
@@ -23,7 +23,7 @@ void object_property_set_qobject(Object *obj, QObject *value,
 {
     Visitor *v;
     /* TODO: Should we reject, rather than ignore, excess input? */
-    v = qmp_input_visitor_new(value, false);
+    v = qobject_input_visitor_new(value, false);
     object_property_set(obj, v, name, errp);
     visit_free(v);
 }
scripts/qapi-commands.py +2 -2
index 2f603b0..04158a3 100644
@@ -130,7 +130,7 @@ def gen_marshal(name, arg_type, boxed, ret_type):
         push_indent()
 
     ret += mcgen('''
-    v = qmp_input_visitor_new(QOBJECT(args), true);
+    v = qobject_input_visitor_new(QOBJECT(args), true);
     visit_start_struct(v, NULL, NULL, 0, &err);
     if (err) {
         goto out;
@@ -294,7 +294,7 @@ fdef.write(mcgen('''
 #include "qapi/qmp/dispatch.h"
 #include "qapi/visitor.h"
 #include "qapi/qmp-output-visitor.h"
-#include "qapi/qmp-input-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/dealloc-visitor.h"
 #include "%(prefix)sqapi-types.h"
 #include "%(prefix)sqapi-visit.h"
target-s390x/cpu_models.c +2 -2
index 3ff6a70..c1e729d 100644
@@ -17,7 +17,7 @@
 #include "qapi/visitor.h"
 #include "qemu/error-report.h"
 #include "qapi/qmp/qerror.h"
-#include "qapi/qmp-input-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/qmp/qbool.h"
 #ifndef CONFIG_USER_ONLY
 #include "sysemu/arch_init.h"
@@ -345,7 +345,7 @@ static void cpu_model_from_info(S390CPUModel *model, const CpuModelInfo *info,
     }
 
     if (qdict) {
-        visitor = qmp_input_visitor_new(info->props, true);
+        visitor = qobject_input_visitor_new(info->props, true);
         visit_start_struct(visitor, NULL, NULL, 0, errp);
         if (*errp) {
             object_unref(obj);
tests/.gitignore +2 -2
index 0f0c79b..bc076eb 100644
@@ -57,8 +57,8 @@ test-qht-par
 test-qmp-commands
 test-qmp-commands.h
 test-qmp-event
-test-qmp-input-strict
-test-qmp-input-visitor
+test-qobject-input-strict
+test-qobject-input-visitor
 test-qmp-introspect.[ch]
 test-qmp-marshal.c
 test-qmp-output-visitor
tests/Makefile.include +6 -6
index 8162f6f..eb2c8b0 100644
@@ -24,9 +24,9 @@ check-unit-y += tests/test-qmp-output-visitor$(EXESUF)
 gcov-files-test-qmp-output-visitor-y = qapi/qmp-output-visitor.c
 check-unit-y += tests/test-clone-visitor$(EXESUF)
 gcov-files-test-clone-visitor-y = qapi/qapi-clone-visitor.c
-check-unit-y += tests/test-qmp-input-visitor$(EXESUF)
-gcov-files-test-qmp-input-visitor-y = qapi/qmp-input-visitor.c
-check-unit-y += tests/test-qmp-input-strict$(EXESUF)
+check-unit-y += tests/test-qobject-input-visitor$(EXESUF)
+gcov-files-test-qobject-input-visitor-y = qapi/qobject-input-visitor.c
+check-unit-y += tests/test-qobject-input-strict$(EXESUF)
 check-unit-y += tests/test-qmp-commands$(EXESUF)
 gcov-files-test-qmp-commands-y = qapi/qmp-dispatch.c
 check-unit-y += tests/test-string-input-visitor$(EXESUF)
@@ -441,7 +441,7 @@ test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \
 	tests/test-coroutine.o tests/test-string-output-visitor.o \
 	tests/test-string-input-visitor.o tests/test-qmp-output-visitor.o \
 	tests/test-clone-visitor.o \
-	tests/test-qmp-input-visitor.o tests/test-qmp-input-strict.o \
+	tests/test-qobject-input-visitor.o tests/test-qobject-input-strict.o \
 	tests/test-qmp-commands.o tests/test-visitor-serialization.o \
 	tests/test-x86-cpuid.o tests/test-mul64.o tests/test-int128.o \
 	tests/test-opts-visitor.o tests/test-qmp-event.o \
@@ -544,8 +544,8 @@ tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(te
 tests/test-qmp-event$(EXESUF): tests/test-qmp-event.o $(test-qapi-obj-y)
 tests/test-qmp-output-visitor$(EXESUF): tests/test-qmp-output-visitor.o $(test-qapi-obj-y)
 tests/test-clone-visitor$(EXESUF): tests/test-clone-visitor.o $(test-qapi-obj-y)
-tests/test-qmp-input-visitor$(EXESUF): tests/test-qmp-input-visitor.o $(test-qapi-obj-y)
-tests/test-qmp-input-strict$(EXESUF): tests/test-qmp-input-strict.o $(test-qapi-obj-y)
+tests/test-qobject-input-visitor$(EXESUF): tests/test-qobject-input-visitor.o $(test-qapi-obj-y)
+tests/test-qobject-input-strict$(EXESUF): tests/test-qobject-input-strict.o $(test-qapi-obj-y)
 tests/test-qmp-commands$(EXESUF): tests/test-qmp-commands.o tests/test-qmp-marshal.o $(test-qapi-obj-y)
 tests/test-visitor-serialization$(EXESUF): tests/test-visitor-serialization.o $(test-qapi-obj-y)
 tests/test-opts-visitor$(EXESUF): tests/test-opts-visitor.o $(test-qapi-obj-y)
tests/check-qnull.c +2 -2
index dc906b1..eeb803a 100644
@@ -10,7 +10,7 @@
 
 #include "qapi/qmp/qobject.h"
 #include "qemu-common.h"
-#include "qapi/qmp-input-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/qmp-output-visitor.h"
 #include "qapi/error.h"
 
@@ -47,7 +47,7 @@ static void qnull_visit_test(void)
 
     g_assert(qnull_.refcnt == 1);
     obj = qnull();
-    v = qmp_input_visitor_new(obj, true);
+    v = qobject_input_visitor_new(obj, true);
     qobject_decref(obj);
     visit_type_null(v, NULL, &error_abort);
     visit_free(v);
tests/test-qmp-commands.c +2 -2
index 81cbe54..ff94481 100644
@@ -4,7 +4,7 @@
 #include "test-qmp-commands.h"
 #include "qapi/qmp/dispatch.h"
 #include "qemu/module.h"
-#include "qapi/qmp-input-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "tests/test-qapi-types.h"
 #include "tests/test-qapi-visit.h"
 
@@ -244,7 +244,7 @@ static void test_dealloc_partial(void)
         ud2_dict = qdict_new();
         qdict_put_obj(ud2_dict, "string0", QOBJECT(qstring_from_str(text)));
 
-        v = qmp_input_visitor_new(QOBJECT(ud2_dict), true);
+        v = qobject_input_visitor_new(QOBJECT(ud2_dict), true);
         visit_type_UserDefTwo(v, NULL, &ud2, &err);
         visit_free(v);
         QDECREF(ud2_dict);
tests/test-qobject-input-strict.c renamed from tests/test-qmp-input-strict.c +3 -3
similarity index 98%
rename from tests/test-qmp-input-strict.c
rename to tests/test-qobject-input-strict.c
index 814550a..8250365 100644
@@ -1,5 +1,5 @@
 /*
- * QMP Input Visitor unit-tests (strict mode).
+ * QObject Input Visitor unit-tests (strict mode).
  *
  * Copyright (C) 2011-2012, 2015 Red Hat Inc.
  *
@@ -15,7 +15,7 @@
 
 #include "qemu-common.h"
 #include "qapi/error.h"
-#include "qapi/qmp-input-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "test-qapi-types.h"
 #include "test-qapi-visit.h"
 #include "qapi/qmp/types.h"
@@ -53,7 +53,7 @@ static Visitor *validate_test_init_internal(TestInputVisitorData *data,
     data->obj = qobject_from_jsonv(json_string, ap);
     g_assert(data->obj);
 
-    data->qiv = qmp_input_visitor_new(data->obj, true);
+    data->qiv = qobject_input_visitor_new(data->obj, true);
     g_assert(data->qiv);
     return data->qiv;
 }
tests/test-qobject-input-visitor.c renamed from tests/test-qmp-input-visitor.c +3 -3
similarity index 99%
rename from tests/test-qmp-input-visitor.c
rename to tests/test-qobject-input-visitor.c
index f583dce..0e65e63 100644
@@ -1,5 +1,5 @@
 /*
- * QMP Input Visitor unit-tests.
+ * QObject Input Visitor unit-tests.
  *
  * Copyright (C) 2011-2016 Red Hat Inc.
  *
@@ -14,7 +14,7 @@
 
 #include "qemu-common.h"
 #include "qapi/error.h"
-#include "qapi/qmp-input-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "test-qapi-types.h"
 #include "test-qapi-visit.h"
 #include "qapi/qmp/types.h"
@@ -49,7 +49,7 @@ static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data,
     data->obj = qobject_from_jsonv(json_string, ap);
     g_assert(data->obj);
 
-    data->qiv = qmp_input_visitor_new(data->obj, false);
+    data->qiv = qobject_input_visitor_new(data->obj, false);
     g_assert(data->qiv);
     return data->qiv;
 }
tests/test-string-input-visitor.c +1 -1
index a679fbc..7f10e25 100644
@@ -4,7 +4,7 @@
  * Copyright (C) 2012 Red Hat Inc.
  *
  * Authors:
- *  Paolo Bonzini <[email protected]> (based on test-qmp-input-visitor)
+ *  Paolo Bonzini <[email protected]> (based on test-qobject-input-visitor)
  *
  * This work is licensed under the terms of the GNU GPL, version 2 or later.
  * See the COPYING file in the top-level directory.
tests/test-visitor-serialization.c +2 -2
index dba4670..51df428 100644
@@ -20,7 +20,7 @@
 #include "qapi/error.h"
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/qjson.h"
-#include "qapi/qmp-input-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/qmp-output-visitor.h"
 #include "qapi/string-input-visitor.h"
 #include "qapi/string-output-visitor.h"
@@ -1040,7 +1040,7 @@ static void qmp_deserialize(void **native_out, void *datap,
     obj = qobject_from_json(qstring_get_str(output_json));
 
     QDECREF(output_json);
-    d->qiv = qmp_input_visitor_new(obj, true);
+    d->qiv = qobject_input_visitor_new(obj, true);
     qobject_decref(obj_orig);
     qobject_decref(obj);
     visit(d->qiv, native_out, errp);
util/qemu-sockets.c +1 -1
index 6db48b3..9e30a21 100644
@@ -21,7 +21,7 @@
 #include "qapi/error.h"
 #include "qemu/sockets.h"
 #include "qemu/main-loop.h"
-#include "qapi/qmp-input-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/qmp-output-visitor.h"
 #include "qapi-visit.h"
 #include "qemu/cutils.h"
Signature · 1 line
2.7.4
#35 lore raw

PATCH v14 8/21 qapi: allow QObjectInputVisitor to be created with QemuOpts

In reply to Daniel P. Berrange

Instead of requiring all callers to go through the mutli-step process of turning QemuOpts into a suitable QObject for visiting, add a new constructor that encapsulates this logic. This will allow QObjectInputVisitor to be a drop-in replacement for the existing OptsVisitor with minimal code changes for callers.

NB, at this point it is only supporting opts syntax which explicitly matches the QAPI schema structure, so is not yet a true drop-in replacement for OptsVisitor. The patches that follow will add the special cases requird for full backwards compatibility with OptsVisitor.

---
 include/qapi/qobject-input-visitor.h | 15 +++++++++++++++
 include/qemu/option.h                |  2 +-
 qapi/qobject-input-visitor.c         | 25 +++++++++++++++++++++++++
 util/qemu-option.c                   |  2 +-
 4 files changed, 42 insertions(+), 2 deletions(-)
include/qapi/qobject-input-visitor.h +15 -0
index 5022297..f134d90 100644
@@ -51,4 +51,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  */
 Visitor *qobject_input_visitor_new_autocast(QObject *obj);
 
+
+/**
+ * Create a new input visitor that converts @opts to a QAPI object.
+ *
+ * The QemuOpts will be converted into a QObject using the
+ * qdict_crumple() method to automatically create structs
+ * and lists. The resulting QDict will then be passed to the
+ * qobject_input_visitor_new_autocast() method. See the docs
+ * of that method for further details on processing behaviour.
+ *
+ * The returned input visitor should be released by calling
+ * visit_free() when no longer required.
+ */
+Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, Error **errp);
+
 #endif
include/qemu/option.h +1 -1
index 2a5266f..29f3f18 100644
@@ -125,7 +125,7 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char *params,
                             int permit_abbrev);
 QemuOpts *qemu_opts_from_qdict(QemuOptsList *list, const QDict *qdict,
                                Error **errp);
-QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict);
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict);
 void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp);
 
 typedef int (*qemu_opts_loopfunc)(void *opaque, QemuOpts *opts, Error **errp);
qapi/qobject-input-visitor.c +25 -0
index cf41df6..d9269c9 100644
@@ -545,3 +545,28 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
 
     return &v->visitor;
 }
+
+
+Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
+                                        Error **errp)
+{
+    QDict *pdict;
+    QObject *pobj = NULL;
+    Visitor *v = NULL;
+
+    pdict = qemu_opts_to_qdict(opts, NULL);
+    if (!pdict) {
+        goto cleanup;
+    }
+
+    pobj = qdict_crumple(pdict, true, errp);
+    if (!pobj) {
+        goto cleanup;
+    }
+
+    v = qobject_input_visitor_new_autocast(pobj);
+ cleanup:
+    qobject_decref(pobj);
+    QDECREF(pdict);
+    return v;
+}
util/qemu-option.c +1 -1
index 41b356c..418cbb6 100644
@@ -1058,7 +1058,7 @@ void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp)
  * TODO We'll want to use types appropriate for opt->desc->type, but
  * this is enough for now.
  */
-QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict)
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict)
 {
     QemuOpt *opt;
     QObject *val;
Signature · 1 line
2.7.4
#24 lore raw

PATCH v14 5/21 qapi: rename QmpOutputVisitor to QObjectOutputVisitor

In reply to Daniel P. Berrange

The QmpOutputVisitor has no direct dependency on QMP. It is valid to use it anywhere that one wants a QObject. Rename it to better reflect its functionality as a generic QAPI to QObject converter.

---
 block/qapi.c                                       |   4 +-
 blockdev.c                                         |   4 +-
 docs/qapi-code-gen.txt                             |   2 +-
 ...p-output-visitor.h => qobject-output-visitor.h} |  10 +-
 qapi/Makefile.objs                                 |   2 +-
 qapi/qapi-clone-visitor.c                          |   2 +-
 qapi/qmp-output-visitor.c                          | 256 ---------------------
 qapi/qobject-output-visitor.c                      | 254 ++++++++++++++++++++
 qemu-img.c                                         |   8 +-
 qom/object_interfaces.c                            |   2 +-
 qom/qom-qobject.c                                  |   4 +-
 scripts/qapi-commands.py                           |   4 +-
 scripts/qapi-event.py                              |   4 +-
 tests/.gitignore                                   |   2 +-
 tests/Makefile.include                             |   8 +-
 tests/check-qnull.c                                |   4 +-
 ...put-visitor.c => test-qobject-output-visitor.c} |   6 +-
 tests/test-string-output-visitor.c                 |   2 +-
 tests/test-visitor-serialization.c                 |   4 +-
 util/qemu-sockets.c                                |   2 +-
 20 files changed, 291 insertions(+), 293 deletions(-)
 rename include/qapi/{qmp-output-visitor.h => qobject-output-visitor.h} (66%)
 delete mode 100644 qapi/qmp-output-visitor.c
 create mode 100644 qapi/qobject-output-visitor.c
 rename tests/{test-qmp-output-visitor.c => test-qobject-output-visitor.c} (99%)
block/qapi.c +2 -2
index 6f947e3..f35c89f 100644
@@ -29,7 +29,7 @@
 #include "block/write-threshold.h"
 #include "qmp-commands.h"
 #include "qapi-visit.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "qapi/qmp/types.h"
 #include "sysemu/block-backend.h"
 #include "qemu/cutils.h"
@@ -691,7 +691,7 @@ void bdrv_image_info_specific_dump(fprintf_function func_fprintf, void *f,
                                    ImageInfoSpecific *info_spec)
 {
     QObject *obj, *data;
-    Visitor *v = qmp_output_visitor_new(&obj);
+    Visitor *v = qobject_output_visitor_new(&obj);
 
     visit_type_ImageInfoSpecific(v, NULL, &info_spec, &error_abort);
     visit_complete(v, &obj);
blockdev.c +2 -2
index 29c6561..814d49f 100644
@@ -43,7 +43,7 @@
 #include "qapi/qmp/types.h"
 #include "qapi-visit.h"
 #include "qapi/qmp/qerror.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "qapi/util.h"
 #include "sysemu/sysemu.h"
 #include "block/block_int.h"
@@ -3828,7 +3828,7 @@ void qmp_blockdev_add(BlockdevOptions *options, Error **errp)
 {
     BlockDriverState *bs;
     QObject *obj;
-    Visitor *v = qmp_output_visitor_new(&obj);
+    Visitor *v = qobject_output_visitor_new(&obj);
     QDict *qdict;
     Error *local_err = NULL;
 
docs/qapi-code-gen.txt +1 -1
index d2604b6..2841c51 100644
@@ -1005,7 +1005,7 @@ Example:
         Error *err = NULL;
         Visitor *v;
 
-        v = qmp_output_visitor_new(ret_out);
+        v = qobject_output_visitor_new(ret_out);
         visit_type_UserDefOne(v, "unused", &ret_in, &err);
         if (!err) {
             visit_complete(v, ret_out);
include/qapi/qobject-output-visitor.h renamed from include/qapi/qmp-output-visitor.h +5 -5
similarity index 66%
rename from include/qapi/qmp-output-visitor.h
rename to include/qapi/qobject-output-visitor.h
index 040fdda..358c959 100644
@@ -11,20 +11,20 @@
  *
  */
 
-#ifndef QMP_OUTPUT_VISITOR_H
-#define QMP_OUTPUT_VISITOR_H
+#ifndef QOBJECT_OUTPUT_VISITOR_H
+#define QOBJECT_OUTPUT_VISITOR_H
 
 #include "qapi/visitor.h"
 #include "qapi/qmp/qobject.h"
 
-typedef struct QmpOutputVisitor QmpOutputVisitor;
+typedef struct QObjectOutputVisitor QObjectOutputVisitor;
 
 /*
- * Create a new QMP output visitor.
+ * Create a new QOBJECT output visitor.
  *
  * If everything else succeeds, pass @result to visit_complete() to
  * collect the result of the visit.
  */
-Visitor *qmp_output_visitor_new(QObject **result);
+Visitor *qobject_output_visitor_new(QObject **result);
 
 #endif
qapi/Makefile.objs +1 -1
index 6ec7bdc..33906ff 100644
@@ -1,5 +1,5 @@
 util-obj-y = qapi-visit-core.o qapi-dealloc-visitor.o qobject-input-visitor.o
-util-obj-y += qmp-output-visitor.o qmp-registry.o qmp-dispatch.o
+util-obj-y += qobject-output-visitor.o qmp-registry.o qmp-dispatch.o
 util-obj-y += string-input-visitor.o string-output-visitor.o
 util-obj-y += opts-visitor.o qapi-clone-visitor.o
 util-obj-y += qmp-event.o
qapi/qapi-clone-visitor.c +1 -1
index 0bb8216..34086cb 100644
@@ -110,7 +110,7 @@ static void qapi_clone_type_str(Visitor *v, const char *name, char **obj,
     assert(qcv->depth);
     /*
      * Pointer was already cloned by g_memdup; create fresh copy.
-     * Note that as long as qmp-output-visitor accepts NULL instead of
+     * Note that as long as qobject-output-visitor accepts NULL instead of
      * "", then we must do likewise. However, we want to obey the
      * input visitor semantics of never producing NULL when the empty
      * string is intended.
qapi/qmp-output-visitor.c deleted +0 -256

256 lines not on this page · raw message

qapi/qobject-output-visitor.c new file +254 -0

254 lines not on this page · raw message

qemu-img.c +4 -4
index ceffefe..851422a 100644
@@ -25,7 +25,7 @@
 #include "qemu-version.h"
 #include "qapi/error.h"
 #include "qapi-visit.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qjson.h"
 #include "qemu/cutils.h"
@@ -500,7 +500,7 @@ static void dump_json_image_check(ImageCheck *check, bool quiet)
 {
     QString *str;
     QObject *obj;
-    Visitor *v = qmp_output_visitor_new(&obj);
+    Visitor *v = qobject_output_visitor_new(&obj);
 
     visit_type_ImageCheck(v, NULL, &check, &error_abort);
     visit_complete(v, &obj);
@@ -2193,7 +2193,7 @@ static void dump_json_image_info_list(ImageInfoList *list)
 {
     QString *str;
     QObject *obj;
-    Visitor *v = qmp_output_visitor_new(&obj);
+    Visitor *v = qobject_output_visitor_new(&obj);
 
     visit_type_ImageInfoList(v, NULL, &list, &error_abort);
     visit_complete(v, &obj);
@@ -2209,7 +2209,7 @@ static void dump_json_image_info(ImageInfo *info)
 {
     QString *str;
     QObject *obj;
-    Visitor *v = qmp_output_visitor_new(&obj);
+    Visitor *v = qobject_output_visitor_new(&obj);
 
     visit_type_ImageInfo(v, NULL, &info, &error_abort);
     visit_complete(v, &obj);
qom/object_interfaces.c +1 -1
index bf59846..ded4d84 100644
@@ -3,7 +3,7 @@
 #include "qom/object_interfaces.h"
 #include "qemu/module.h"
 #include "qapi-visit.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "qapi/opts-visitor.h"
 
 void user_creatable_complete(Object *obj, Error **errp)
qom/qom-qobject.c +2 -2
index 81959e0..447e4a0 100644
@@ -16,7 +16,7 @@
 #include "qom/qom-qobject.h"
 #include "qapi/visitor.h"
 #include "qapi/qobject-input-visitor.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 
 void object_property_set_qobject(Object *obj, QObject *value,
                                  const char *name, Error **errp)
@@ -35,7 +35,7 @@ QObject *object_property_get_qobject(Object *obj, const char *name,
     Error *local_err = NULL;
     Visitor *v;
 
-    v = qmp_output_visitor_new(&ret);
+    v = qobject_output_visitor_new(&ret);
     object_property_get(obj, v, name, &local_err);
     if (!local_err) {
         visit_complete(v, &ret);
scripts/qapi-commands.py +2 -2
index 04158a3..09e8467 100644
@@ -68,7 +68,7 @@ static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out,
     Error *err = NULL;
     Visitor *v;
 
-    v = qmp_output_visitor_new(ret_out);
+    v = qobject_output_visitor_new(ret_out);
     visit_type_%(c_name)s(v, "unused", &ret_in, &err);
     if (!err) {
         visit_complete(v, ret_out);
@@ -293,7 +293,7 @@ fdef.write(mcgen('''
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/dispatch.h"
 #include "qapi/visitor.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "qapi/qobject-input-visitor.h"
 #include "qapi/dealloc-visitor.h"
 #include "%(prefix)sqapi-types.h"
scripts/qapi-event.py +2 -2
index 38d8211..f4eb7f8 100644
@@ -98,7 +98,7 @@ def gen_event_send(name, arg_type, boxed):
 
     if arg_type and not arg_type.is_empty():
         ret += mcgen('''
-    v = qmp_output_visitor_new(&obj);
+    v = qobject_output_visitor_new(&obj);
 ''')
         if not arg_type.is_implicit():
             ret += mcgen('''
@@ -209,7 +209,7 @@ fdef.write(mcgen('''
 #include "qemu-common.h"
 #include "%(prefix)sqapi-event.h"
 #include "%(prefix)sqapi-visit.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "qapi/qmp-event.h"
 
 ''',
tests/.gitignore +1 -1
index bc076eb..8ee11c8 100644
@@ -61,7 +61,7 @@ test-qobject-input-strict
 test-qobject-input-visitor
 test-qmp-introspect.[ch]
 test-qmp-marshal.c
-test-qmp-output-visitor
+test-qobject-output-visitor
 test-rcu-list
 test-replication
 test-rfifolock
tests/Makefile.include +4 -4
index eb2c8b0..605f276 100644
@@ -20,8 +20,8 @@ check-unit-y += tests/check-qnull$(EXESUF)
 gcov-files-check-qnull-y = qobject/qnull.c
 check-unit-y += tests/check-qjson$(EXESUF)
 gcov-files-check-qjson-y = qobject/qjson.c
-check-unit-y += tests/test-qmp-output-visitor$(EXESUF)
-gcov-files-test-qmp-output-visitor-y = qapi/qmp-output-visitor.c
+check-unit-y += tests/test-qobject-output-visitor$(EXESUF)
+gcov-files-test-qobject-output-visitor-y = qapi/qobject-output-visitor.c
 check-unit-y += tests/test-clone-visitor$(EXESUF)
 gcov-files-test-clone-visitor-y = qapi/qapi-clone-visitor.c
 check-unit-y += tests/test-qobject-input-visitor$(EXESUF)
@@ -439,7 +439,7 @@ test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \
 	tests/check-qlist.o tests/check-qfloat.o tests/check-qnull.o \
 	tests/check-qjson.o \
 	tests/test-coroutine.o tests/test-string-output-visitor.o \
-	tests/test-string-input-visitor.o tests/test-qmp-output-visitor.o \
+	tests/test-string-input-visitor.o tests/test-qobject-output-visitor.o \
 	tests/test-clone-visitor.o \
 	tests/test-qobject-input-visitor.o tests/test-qobject-input-strict.o \
 	tests/test-qmp-commands.o tests/test-visitor-serialization.o \
@@ -542,7 +542,7 @@ $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-int
 tests/test-string-output-visitor$(EXESUF): tests/test-string-output-visitor.o $(test-qapi-obj-y)
 tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(test-qapi-obj-y)
 tests/test-qmp-event$(EXESUF): tests/test-qmp-event.o $(test-qapi-obj-y)
-tests/test-qmp-output-visitor$(EXESUF): tests/test-qmp-output-visitor.o $(test-qapi-obj-y)
+tests/test-qobject-output-visitor$(EXESUF): tests/test-qobject-output-visitor.o $(test-qapi-obj-y)
 tests/test-clone-visitor$(EXESUF): tests/test-clone-visitor.o $(test-qapi-obj-y)
 tests/test-qobject-input-visitor$(EXESUF): tests/test-qobject-input-visitor.o $(test-qapi-obj-y)
 tests/test-qobject-input-strict$(EXESUF): tests/test-qobject-input-strict.o $(test-qapi-obj-y)
tests/check-qnull.c +2 -2
index eeb803a..b50bb8a 100644
@@ -11,7 +11,7 @@
 #include "qapi/qmp/qobject.h"
 #include "qemu-common.h"
 #include "qapi/qobject-input-visitor.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "qapi/error.h"
 
 /*
@@ -52,7 +52,7 @@ static void qnull_visit_test(void)
     visit_type_null(v, NULL, &error_abort);
     visit_free(v);
 
-    v = qmp_output_visitor_new(&obj);
+    v = qobject_output_visitor_new(&obj);
     visit_type_null(v, NULL, &error_abort);
     visit_complete(v, &obj);
     g_assert(obj == &qnull_);
tests/test-qobject-output-visitor.c renamed from tests/test-qmp-output-visitor.c +3 -3
similarity index 99%
rename from tests/test-qmp-output-visitor.c
rename to tests/test-qobject-output-visitor.c
index 513d71f..4e2d79c 100644
@@ -1,5 +1,5 @@
 /*
- * QMP Output Visitor unit-tests.
+ * QObject Output Visitor unit-tests.
  *
  * Copyright (C) 2011-2016 Red Hat Inc.
  *
@@ -14,7 +14,7 @@
 
 #include "qemu-common.h"
 #include "qapi/error.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "test-qapi-types.h"
 #include "test-qapi-visit.h"
 #include "qapi/qmp/types.h"
@@ -28,7 +28,7 @@ typedef struct TestOutputVisitorData {
 static void visitor_output_setup(TestOutputVisitorData *data,
                                  const void *unused)
 {
-    data->ov = qmp_output_visitor_new(&data->obj);
+    data->ov = qobject_output_visitor_new(&data->obj);
     g_assert(data->ov);
 }
 
tests/test-string-output-visitor.c +1 -1
index 444844a..e736db3 100644
@@ -4,7 +4,7 @@
  * Copyright (C) 2012 Red Hat Inc.
  *
  * Authors:
- *  Paolo Bonzini <[email protected]> (based on test-qmp-output-visitor)
+ *  Paolo Bonzini <[email protected]> (based on test-qobject-output-visitor)
  *
  * This work is licensed under the terms of the GNU GPL, version 2 or later.
  * See the COPYING file in the top-level directory.
tests/test-visitor-serialization.c +2 -2
index 51df428..66b2b1c 100644
@@ -21,7 +21,7 @@
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/qjson.h"
 #include "qapi/qobject-input-visitor.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "qapi/string-input-visitor.h"
 #include "qapi/string-output-visitor.h"
 #include "qapi-types.h"
@@ -1022,7 +1022,7 @@ static void qmp_serialize(void *native_in, void **datap,
 {
     QmpSerializeData *d = g_malloc0(sizeof(*d));
 
-    d->qov = qmp_output_visitor_new(&d->obj);
+    d->qov = qobject_output_visitor_new(&d->obj);
     visit(d->qov, &native_in, errp);
     *datap = d;
 }
util/qemu-sockets.c +1 -1
index 9e30a21..4cef549 100644
@@ -22,7 +22,7 @@
 #include "qemu/sockets.h"
 #include "qemu/main-loop.h"
 #include "qapi/qobject-input-visitor.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "qapi-visit.h"
 #include "qemu/cutils.h"
 
Signature · 1 line
2.7.4
#29 lore raw

PATCH v14 7/21 qapi: permit scalar type conversions in QObjectInputVisitor

In reply to Daniel P. Berrange

Currently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.

To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.

---
 include/qapi/qobject-input-visitor.h |  32 +++++-
 qapi/qobject-input-visitor.c         | 132 ++++++++++++++++++++++++
 tests/test-qobject-input-visitor.c   | 194 ++++++++++++++++++++++++++++++++++-
 3 files changed, 350 insertions(+), 8 deletions(-)
include/qapi/qobject-input-visitor.h +28 -4
index cde328d..5022297 100644
@@ -19,12 +19,36 @@
 
 typedef struct QObjectInputVisitor QObjectInputVisitor;
 
-/*
- * Return a new input visitor that converts a QObject to a QAPI object.
+/**
+ * Create a new input visitor that converts @obj to a QAPI object.
+ *
+ * Any scalar values in the @obj input data structure should be in the
+ * required type already. i.e. if visiting a bool, the value should
+ * already be a QBool instance.
  *
- * Set @strict to reject a parse that doesn't consume all keys of a
- * dictionary; otherwise excess input is ignored.
+ * If @strict is set to true, then an error will be reported if any
+ * dict keys are not consumed during visitation. If @strict is false
+ * then extra dict keys are silently ignored.
+ *
+ * The returned input visitor should be released by calling
+ * visit_free() when no longer required.
  */
 Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
 
+/**
+ * Create a new input visitor that converts @obj to a QAPI object.
+ *
+ * Any scalar values in the @obj input data structure should always be
+ * represented as strings. i.e. if visiting a boolean, the value should
+ * be a QString whose contents represent a valid boolean.
+ *
+ * The visitor always operates in strict mode, requiring all dict keys
+ * to be consumed during visitation. An error will be reported if this
+ * does not happen.
+ *
+ * The returned input visitor should be released by calling
+ * visit_free() when no longer required.
+ */
+Visitor *qobject_input_visitor_new_autocast(QObject *obj);
+
 #endif
qapi/qobject-input-visitor.c +132 -0

171 lines not on this page · raw message

tests/test-qobject-input-visitor.c +190 -4

256 lines not on this page · raw message

Signature · 1 line
2.7.4
#40 lore raw

PATCH v14 9/21 qapi: permit auto-creating single element lists

In reply to Daniel P. Berrange

When converting QemuOpts to a QObject, there is no information about compound types available, so when visiting a list, the corresponding QObject is not guaranteed to be a QList. We therefore need to be able to auto-create a single element QList from whatever type we find.

This mode should only be enabled if you have compatibility requirements for

-arg foo=hello,foo=world

to be treated as equivalent to the preferred syntax:

-arg foo.0=hello,foo.1=world

---
 include/qapi/qobject-input-visitor.h | 20 +++++++-
 qapi/qobject-input-visitor.c         | 27 +++++++++--
 tests/test-qobject-input-visitor.c   | 88 +++++++++++++++++++++++++++++++-----
 3 files changed, 117 insertions(+), 18 deletions(-)
include/qapi/qobject-input-visitor.h +18 -2
index f134d90..1809f48 100644
@@ -42,6 +42,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * represented as strings. i.e. if visiting a boolean, the value should
  * be a QString whose contents represent a valid boolean.
  *
+ * If @autocreate_list is true, then as an alternative to a normal QList,
+ * list values can be stored as a QString or QDict instead, which will
+ * be interpreted as representing single element lists. This should only
+ * by used if compatibility is required with the OptsVisitor which allowed
+ * repeated keys, without list indexes, to represent lists. e.g. set this
+ * to true if you have compatibility requirements for
+ *
+ *   -arg foo=hello,foo=world
+ *
+ * to be treated as equivalent to the preferred syntax:
+ *
+ *   -arg foo.0=hello,foo.1=world
+ *
  * The visitor always operates in strict mode, requiring all dict keys
  * to be consumed during visitation. An error will be reported if this
  * does not happen.
@@ -49,7 +62,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * The returned input visitor should be released by calling
  * visit_free() when no longer required.
  */
-Visitor *qobject_input_visitor_new_autocast(QObject *obj);
+Visitor *qobject_input_visitor_new_autocast(QObject *obj,
+                                            bool autocreate_list);
 
 
 /**
@@ -64,6 +78,8 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj);
  * The returned input visitor should be released by calling
  * visit_free() when no longer required.
  */
-Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, Error **errp);
+Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
+                                        bool autocreate_list,
+                                        Error **errp);
 
 #endif
qapi/qobject-input-visitor.c +23 -4
index d9269c9..d88e9f9 100644
@@ -48,6 +48,10 @@ struct QObjectInputVisitor
 
     /* True to reject parse in visit_end_struct() if unvisited keys remain. */
     bool strict;
+
+    /* Whether we can auto-create single element lists when
+     * encountering a non-QList type */
+    bool autocreate_list;
 };
 
 static QObjectInputVisitor *to_qiv(Visitor *v)
@@ -108,6 +112,7 @@ static const QListEntry *qobject_input_push(QObjectInputVisitor *qiv,
     assert(obj);
     tos->obj = obj;
     tos->qapi = qapi;
+    qobject_incref(obj);
 
     if (qiv->strict && qobject_type(obj) == QTYPE_QDICT) {
         h = g_hash_table_new(g_str_hash, g_str_equal);
@@ -147,6 +152,7 @@ static void qobject_input_stack_object_free(StackObject *tos)
     if (tos->h) {
         g_hash_table_unref(tos->h);
     }
+    qobject_decref(tos->obj);
 
     g_free(tos);
 }
@@ -197,7 +203,7 @@ static void qobject_input_start_list(Visitor *v, const char *name,
     QObject *qobj = qobject_input_get_object(qiv, name, true);
     const QListEntry *entry;
 
-    if (!qobj || qobject_type(qobj) != QTYPE_QLIST) {
+    if (!qobj || (!qiv->autocreate_list && qobject_type(qobj) != QTYPE_QLIST)) {
         if (list) {
             *list = NULL;
         }
@@ -206,7 +212,16 @@ static void qobject_input_start_list(Visitor *v, const char *name,
         return;
     }
 
-    entry = qobject_input_push(qiv, qobj, list, errp);
+    if (qobject_type(qobj) != QTYPE_QLIST) {
+        QList *tmplist = qlist_new();
+        qlist_append_obj(tmplist, qobj);
+        qobject_incref(qobj);
+        entry = qobject_input_push(qiv, QOBJECT(tmplist), list, errp);
+        QDECREF(tmplist);
+    } else {
+        entry = qobject_input_push(qiv, qobj, list, errp);
+    }
+
     if (list) {
         if (entry) {
             *list = g_malloc0(size);
@@ -514,7 +529,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)
     return &v->visitor;
 }
 
-Visitor *qobject_input_visitor_new_autocast(QObject *obj)
+Visitor *qobject_input_visitor_new_autocast(QObject *obj,
+                                            bool autocreate_list)
 {
     QObjectInputVisitor *v;
 
@@ -539,6 +555,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
     v->visitor.optional = qobject_input_optional;
     v->visitor.free = qobject_input_free;
     v->strict = true;
+    v->autocreate_list = autocreate_list;
 
     v->root = obj;
     qobject_incref(obj);
@@ -548,6 +565,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
 
 
 Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
+                                        bool autocreate_list,
                                         Error **errp)
 {
     QDict *pdict;
@@ -564,7 +582,8 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
         goto cleanup;
     }
 
-    v = qobject_input_visitor_new_autocast(pobj);
+    v = qobject_input_visitor_new_autocast(pobj,
+                                           autocreate_list);
  cleanup:
     qobject_decref(pobj);
     QDECREF(pdict);
tests/test-qobject-input-visitor.c +76 -12

187 lines not on this page · raw message

Signature · 1 line
2.7.4
#45 lore raw

PATCH v14 10/21 qapi: permit auto-creating nested structs

In reply to Daniel P. Berrange

Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the qdict_crumple() call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.

Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.

The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach

  -net vlan=1,id=foo,name=bar,opts.type=tap,\
       opts.data.fd=3,opts.data.script=ifup

The legacy syntax will just be presenting

-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup

So we need to auto-create 3 levels of struct when visiting.

The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.

---
 include/qapi/qobject-input-visitor.h |  22 +++++-
 qapi/qobject-input-visitor.c         |  59 ++++++++++++++--
 tests/test-qobject-input-visitor.c   | 130 ++++++++++++++++++++++++++++++++---
 3 files changed, 194 insertions(+), 17 deletions(-)
include/qapi/qobject-input-visitor.h +20 -2
index 1809f48..94051f3 100644
@@ -45,7 +45,7 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * If @autocreate_list is true, then as an alternative to a normal QList,
  * list values can be stored as a QString or QDict instead, which will
  * be interpreted as representing single element lists. This should only
- * by used if compatibility is required with the OptsVisitor which allowed
+ * be used if compatibility is required with the OptsVisitor which allowed
  * repeated keys, without list indexes, to represent lists. e.g. set this
  * to true if you have compatibility requirements for
  *
@@ -55,6 +55,22 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  *
  *   -arg foo.0=hello,foo.1=world
  *
+ * If @autocreate_struct_levels is non-zero, then when visiting structs,
+ * if the corresponding QDict is not found, it will automatically create
+ * a QDict containing all remaining unvisited options. This should only
+ * be used if compatibility is required with the OptsVisitor which flatten
+ * structs so that all keys were at the same level. e.g. set this to a
+ * non-zero number if you compatibility requirements for
+ *
+ *   -arg type=person,surname=blogs,forename=fred
+ *
+ * to be treated as equivalent to the perferred syntax
+ *
+ *   -arg type=person,data.surname=blogs,data.forename=fred
+ *
+ * The value given determines how many levels of structs are allowed to
+ * be flattened in this way.
+ *
  * The visitor always operates in strict mode, requiring all dict keys
  * to be consumed during visitation. An error will be reported if this
  * does not happen.
@@ -63,7 +79,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * visit_free() when no longer required.
  */
 Visitor *qobject_input_visitor_new_autocast(QObject *obj,
-                                            bool autocreate_list);
+                                            bool autocreate_list,
+                                            size_t autocreate_struct_levels);
 
 
 /**
@@ -80,6 +97,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,
  */
 Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
                                         bool autocreate_list,
+                                        size_t autocreate_struct_levels,
                                         Error **errp);
 
 #endif
qapi/qobject-input-visitor.c +54 -5

130 lines not on this page · raw message

tests/test-qobject-input-visitor.c +120 -10

174 lines not on this page · raw message

Signature · 1 line
2.7.4
#56 lore raw

PATCH v14 11/21 qapi: add integer range support for QObjectInputVisitor

In reply to Daniel P. Berrange

The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:

-arg foo=5,foo=6,foo=7
-arg foo=5-7

This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.

This means that

-arg foo=5-7

is treated as equivalent to

-arg foo.0=5,foo.1=6,foo.2=7

Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.

---
 include/qapi/qobject-input-visitor.h |  23 ++++-
 qapi/qobject-input-visitor.c         | 158 ++++++++++++++++++++++++++--
 tests/test-qobject-input-visitor.c   | 195 +++++++++++++++++++++++++++++++++--
 3 files changed, 360 insertions(+), 16 deletions(-)
include/qapi/qobject-input-visitor.h +22 -1
index 94051f3..242b767 100644
@@ -19,6 +19,12 @@
 
 typedef struct QObjectInputVisitor QObjectInputVisitor;
 
+/* Inclusive upper bound on the size of any flattened range. This is a safety
+ * (= anti-annoyance) measure; wrong ranges should not cause long startup
+ * delays nor exhaust virtual memory.
+ */
+#define QIV_RANGE_MAX 65536
+
 /**
  * Create a new input visitor that converts @obj to a QAPI object.
  *
@@ -71,6 +77,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * The value given determines how many levels of structs are allowed to
  * be flattened in this way.
  *
+ * If @permit_int_ranges is true, then when visiting a list of integers,
+ * the integer value strings may encode ranges eg a single element
+ * containing "5-7" is treated as if there were three elements "5", "6",
+ * "7". This should only be used if compatibility is required with the
+ * OptsVisitor which would allow integer ranges. e.g. set this to true
+ * if you have compatibility requirements for
+ *
+ *   -arg val=5-8
+ *
+ * to be treated as equivalent to the preferred syntax:
+ *
+ *   -arg val.0=5,val.1=6,val.2=7,val.3=8
+ *
  * The visitor always operates in strict mode, requiring all dict keys
  * to be consumed during visitation. An error will be reported if this
  * does not happen.
@@ -80,7 +99,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  */
 Visitor *qobject_input_visitor_new_autocast(QObject *obj,
                                             bool autocreate_list,
-                                            size_t autocreate_struct_levels);
+                                            size_t autocreate_struct_levels,
+                                            bool permit_int_ranges);
 
 
 /**
@@ -98,6 +118,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,
 Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
                                         bool autocreate_list,
                                         size_t autocreate_struct_levels,
+                                        bool permit_int_ranges,
                                         Error **errp);
 
 #endif
qapi/qobject-input-visitor.c +149 -9

236 lines not on this page · raw message

tests/test-qobject-input-visitor.c +189 -6

252 lines not on this page · raw message

Signature · 1 line
2.7.4
#73 lore raw

PATCH v14 13/21 qdict: allow qdict_crumple to accept compound types as values

In reply to Daniel P. Berrange

Currently qdict_crumple requires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.

In order to have backwards compatibility with the OptsVisitor, qemu_opt_to_qdict() has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given to qdict_crumple().

To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed

foo=hello,foo=world,foo=wibble
foo.0=hello,foo.1=world,foo.2=wibble

but this is forbidden

foo=hello,foo=world,foo.2=wibble

---
 qobject/qdict.c     |  45 +++++++++-----
 tests/check-qdict.c | 175 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 204 insertions(+), 16 deletions(-)
qobject/qdict.c +29 -16
index c38e90e..83384b2 100644
@@ -814,15 +814,16 @@ static int qdict_is_list(QDict *maybe_list, Error **errp)
 
 /**
  * qdict_crumple:
- * @src: the original flat dictionary (only scalar values) to crumple
+ * @src: the original dictionary to crumple
  * @recursive: true to recursively crumple nested dictionaries
  *
- * Takes a flat dictionary whose keys use '.' separator to indicate
- * nesting, and values are scalars, and crumples it into a nested
- * structure. If the @recursive parameter is false, then only the
- * first level of structure implied by the keys will be crumpled. If
- * @recursive is true, then the input will be recursively crumpled to
- * expand all levels of structure in the keys.
+ * Takes a dictionary whose keys use '.' separator to indicate
+ * nesting, crumples it into a nested structure. The values in
+ * @src are permitted to be scalars or compound typee. If the
+ * @recursive parameter is false, then only the first level of
+ * structure implied by the keys will be crumpled. If @recursive
+ * is true, then the input will be recursively crumpled to expand
+ * all levels of structure in the keys.
  *
  * To include a literal '.' in a key name, it must be escaped as '..'
  *
@@ -843,7 +844,10 @@ static int qdict_is_list(QDict *maybe_list, Error **errp)
  * The following scenarios in the input dict will result in an
  * error being returned:
  *
- *  - Any values in @src are non-scalar types
+ *  - If a key in @src has a value that is non-scalar, and
+ *    a later key implies creation of a compound type. e.g.
+ *    if 'foo' point to a 'QList' or 'QDict', then it is
+ *    not permitted to also have 'foo.NNN' keys later.
  *  - If keys in @src imply that a particular level is both a
  *    list and a dict. e.g., "foo.0.bar" and "foo.eek.bar".
  *  - If keys in @src imply that a particular level is a list,
@@ -864,21 +868,26 @@ QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp)
     char *prefix = NULL;
     const char *suffix = NULL;
     int is_list;
+    QDict *compound;
 
+    compound = qdict_new();
     two_level = qdict_new();
 
     /* Step 1: split our totally flat dict into a two level dict */
     for (ent = qdict_first(src); ent != NULL; ent = qdict_next(src, ent)) {
-        if (qobject_type(ent->value) == QTYPE_QDICT ||
-            qobject_type(ent->value) == QTYPE_QLIST) {
-            error_setg(errp, "Value %s is not a scalar",
-                       ent->key);
-            goto error;
-        }
-
         qdict_split_flat_key(ent->key, &prefix, &suffix);
 
         child = qdict_get(two_level, prefix);
+
+        if (qdict_haskey(compound, prefix)) {
+            error_setg(errp, "Key %s is already set as a list/dict", prefix);
+            goto error;
+        }
+        if (qobject_type(ent->value) == QTYPE_QLIST ||
+            qobject_type(ent->value) == QTYPE_QDICT) {
+            qobject_incref(ent->value);
+            qdict_put_obj(compound, prefix, ent->value);
+        }
         if (suffix) {
             if (child) {
                 if (qobject_type(child) != QTYPE_QDICT) {
@@ -913,7 +922,8 @@ QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp)
         for (ent = qdict_first(two_level); ent != NULL;
              ent = qdict_next(two_level, ent)) {
 
-            if (qobject_type(ent->value) == QTYPE_QDICT) {
+            if (qobject_type(ent->value) == QTYPE_QDICT &&
+                !qdict_haskey(compound, ent->key)) {
                 child = qdict_crumple(qobject_to_qdict(ent->value),
                                       recursive, errp);
                 if (!child) {
@@ -961,10 +971,13 @@ QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp)
         dst = QOBJECT(multi_level);
     }
 
+    QDECREF(compound);
+
     return dst;
 
  error:
     g_free(prefix);
+    QDECREF(compound);
     QDECREF(multi_level);
     QDECREF(two_level);
     qobject_decref(dst);
tests/check-qdict.c +175 -0

187 lines not on this page · raw message

Signature · 1 line
2.7.4
#87 lore raw

PATCH v14 14/21 qapi: allow repeated opts with qobject_input_visitor_new_opts

In reply to Daniel P. Berrange

The qobject_input_visitor_new_opts() method gains a new parameter to control whether it allows repeated option keys in the input QemuOpts or not.

---
 include/qapi/qobject-input-visitor.h | 6 ++++++
 qapi/qobject-input-visitor.c         | 5 ++++-
 2 files changed, 10 insertions(+), 1 deletion(-)
include/qapi/qobject-input-visitor.h +6 -0
index 242b767..bc5062a 100644
@@ -112,6 +112,11 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,
  * qobject_input_visitor_new_autocast() method. See the docs
  * of that method for further details on processing behaviour.
  *
+ * If the @permit_repeated_opts parameter is true, then the input
+ * @opts is allowed to contain repeated keys and they will be
+ * turned into a QList. If it is false, then repeated keys will
+ * result in an error being reported.
+ *
  * The returned input visitor should be released by calling
  * visit_free() when no longer required.
  */
@@ -119,6 +124,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
                                         bool autocreate_list,
                                         size_t autocreate_struct_levels,
                                         bool permit_int_ranges,
+                                        bool permit_repeated_opts,
                                         Error **errp);
 
 #endif
qapi/qobject-input-visitor.c +4 -1
index 2287d11..5a3872c 100644
@@ -753,6 +753,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
                                         bool autocreate_list,
                                         size_t autocreate_struct_levels,
                                         bool permit_int_ranges,
+                                        bool permit_repeated_opts,
                                         Error **errp)
 {
     QDict *pdict;
@@ -760,7 +761,9 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
     Visitor *v = NULL;
 
     pdict = qemu_opts_to_qdict(opts, NULL,
-                               QEMU_OPTS_REPEAT_POLICY_LAST,
+                               permit_repeated_opts ?
+                               QEMU_OPTS_REPEAT_POLICY_ALL :
+                               QEMU_OPTS_REPEAT_POLICY_ERROR,
                                errp);
     if (!pdict) {
         goto cleanup;
Signature · 1 line
2.7.4
#63 lore raw

PATCH v14 12/21 option: allow qemu_opts_to_qdict to merge repeated options

In reply to Daniel P. Berrange

If given an option string such as

size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind

the qemu_opts_to_qdict() method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:

    size=QString("1024")
    nodes=QString("1-2")
    policy=QString("bind")

With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain

    size=QString("1024")
    nodes=QList([QString("10"),
                 QString("4-5"),
                 QString("1-2")])
    policy=QString("bind")

Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.

In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.

All existing callers are all converted to explicitly request the historical behaviour of only reporting the last key. Later patches will make use of the new modes.

---
 blockdev.c                   |   7 ++-
 include/qemu/option.h        |  13 ++++-
 monitor.c                    |   3 +-
 qapi/qobject-input-visitor.c |   4 +-
 qemu-img.c                   |   4 +-
 qemu-io-cmds.c               |   3 +-
 qemu-io.c                    |   6 +-
 qemu-nbd.c                   |   3 +-
 qom/object_interfaces.c      |   3 +-
 tests/test-qemu-opts.c       | 132 +++++++++++++++++++++++++++++++++++++++++++
 tests/test-replication.c     |   9 ++-
 util/qemu-option.c           |  64 ++++++++++++++++++---
 12 files changed, 229 insertions(+), 22 deletions(-)
blockdev.c +4 -3
index 814d49f..a999d46 100644
@@ -914,7 +914,8 @@ DriveInfo *drive_new(QemuOpts *all_opts, BlockInterfaceType block_default_type)
 
     /* Get a QDict for processing the options */
     bs_opts = qdict_new();
-    qemu_opts_to_qdict(all_opts, bs_opts);
+    qemu_opts_to_qdict(all_opts, bs_opts, QEMU_OPTS_REPEAT_POLICY_LAST,
+                       &error_abort);
 
     legacy_opts = qemu_opts_create(&qemu_legacy_drive_opts, NULL, 0,
                                    &error_abort);
@@ -3804,8 +3805,8 @@ void hmp_drive_add_node(Monitor *mon, const char *optstr)
         return;
     }
 
-    qdict = qemu_opts_to_qdict(opts, NULL);
-
+    qdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                               &error_abort);
     if (!qdict_get_try_str(qdict, "node-name")) {
         QDECREF(qdict);
         error_report("'node-name' needs to be specified");
include/qemu/option.h +12 -1
index 29f3f18..ffd841d 100644
@@ -125,7 +125,18 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char *params,
                             int permit_abbrev);
 QemuOpts *qemu_opts_from_qdict(QemuOptsList *list, const QDict *qdict,
                                Error **errp);
-QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict);
+typedef enum {
+    /* Last occurrence of a duplicate option silently replaces earlier */
+    QEMU_OPTS_REPEAT_POLICY_LAST,
+    /* Each occurrence of a duplicate option converts value to a QList */
+    QEMU_OPTS_REPEAT_POLICY_ALL,
+    /* First occurrence of a duplicate option causes an error */
+    QEMU_OPTS_REPEAT_POLICY_ERROR,
+} QemuOptsRepeatPolicy;
+
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict,
+                          QemuOptsRepeatPolicy repeatPolicy,
+                          Error **errp);
 void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp);
 
 typedef int (*qemu_opts_loopfunc)(void *opaque, QemuOpts *opts, Error **errp);
monitor.c +2 -1
index 14089cc..84e79d4 100644
@@ -2642,7 +2642,8 @@ static QDict *monitor_parse_arguments(Monitor *mon,
                 if (!opts) {
                     goto fail;
                 }
-                qemu_opts_to_qdict(opts, qdict);
+                qemu_opts_to_qdict(opts, qdict, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                   &error_abort);
                 qemu_opts_del(opts);
             }
             break;
qapi/qobject-input-visitor.c +3 -1
index 6b3d0f2..2287d11 100644
@@ -759,7 +759,9 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
     QObject *pobj = NULL;
     Visitor *v = NULL;
 
-    pdict = qemu_opts_to_qdict(opts, NULL);
+    pdict = qemu_opts_to_qdict(opts, NULL,
+                               QEMU_OPTS_REPEAT_POLICY_LAST,
+                               errp);
     if (!pdict) {
         goto cleanup;
     }
qemu-img.c +3 -1
index 851422a..f47ea75 100644
@@ -273,7 +273,9 @@ static BlockBackend *img_open_opts(const char *optstr,
     QDict *options;
     Error *local_err = NULL;
     BlockBackend *blk;
-    options = qemu_opts_to_qdict(opts, NULL);
+    options = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                 &error_abort);
+
     blk = blk_new_open(NULL, NULL, options, flags, &local_err);
     if (!blk) {
         error_reportf_err(local_err, "Could not open '%s': ", optstr);
qemu-io-cmds.c +2 -1
index 3a3838a..ce654f4 100644
@@ -1952,7 +1952,8 @@ static int reopen_f(BlockBackend *blk, int argc, char **argv)
     }
 
     qopts = qemu_opts_find(&reopen_opts, NULL);
-    opts = qopts ? qemu_opts_to_qdict(qopts, NULL) : NULL;
+    opts = qopts ? qemu_opts_to_qdict(qopts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                      &error_abort) : NULL;
     qemu_opts_reset(&reopen_opts);
 
     brq = bdrv_reopen_queue(NULL, bs, opts, flags);
qemu-io.c +4 -2
index db129ea..bb374a6 100644
@@ -207,7 +207,8 @@ static int open_f(BlockBackend *blk, int argc, char **argv)
     }
 
     qopts = qemu_opts_find(&empty_opts, NULL);
-    opts = qopts ? qemu_opts_to_qdict(qopts, NULL) : NULL;
+    opts = qopts ? qemu_opts_to_qdict(qopts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                      &error_abort) : NULL;
     qemu_opts_reset(&empty_opts);
 
     if (optind == argc - 1) {
@@ -593,7 +594,8 @@ int main(int argc, char **argv)
             if (!qopts) {
                 exit(1);
             }
-            opts = qemu_opts_to_qdict(qopts, NULL);
+            opts = qemu_opts_to_qdict(qopts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                      &error_abort);
             openfile(NULL, flags, writethrough, opts);
         } else {
             if (format) {
qemu-nbd.c +2 -1
index 99297a5..73b40b1 100644
@@ -859,7 +859,8 @@ int main(int argc, char **argv)
             qemu_opts_reset(&file_opts);
             exit(EXIT_FAILURE);
         }
-        options = qemu_opts_to_qdict(opts, NULL);
+        options = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                     &error_abort);
         qemu_opts_reset(&file_opts);
         blk = blk_new_open(NULL, NULL, options, flags, &local_err);
     } else {
qom/object_interfaces.c +2 -1
index ded4d84..fdc406b 100644
@@ -161,7 +161,8 @@ Object *user_creatable_add_opts(QemuOpts *opts, Error **errp)
     Object *obj = NULL;
 
     v = opts_visitor_new(opts);
-    pdict = qemu_opts_to_qdict(opts, NULL);
+    pdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                               &error_abort);
 
     obj = user_creatable_add(pdict, v, errp);
     visit_free(v);
tests/test-qemu-opts.c +132 -0

144 lines not on this page · raw message

tests/test-replication.c +6 -3
index 0997bd8..e267f9a 100644
@@ -181,7 +181,8 @@ static BlockBackend *start_primary(void)
     opts = qemu_opts_parse_noisily(&qemu_drive_opts, cmdline, false);
     g_free(cmdline);
 
-    qdict = qemu_opts_to_qdict(opts, NULL);
+    qdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                               &error_abort);
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_DIRECT, "off");
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_NO_FLUSH, "off");
 
@@ -311,7 +312,8 @@ static BlockBackend *start_secondary(void)
     opts = qemu_opts_parse_noisily(&qemu_drive_opts, cmdline, false);
     g_free(cmdline);
 
-    qdict = qemu_opts_to_qdict(opts, NULL);
+    qdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                               &error_abort);
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_DIRECT, "off");
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_NO_FLUSH, "off");
 
@@ -336,7 +338,8 @@ static BlockBackend *start_secondary(void)
     opts = qemu_opts_parse_noisily(&qemu_drive_opts, cmdline, false);
     g_free(cmdline);
 
-    qdict = qemu_opts_to_qdict(opts, NULL);
+    qdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                               &error_abort);
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_DIRECT, "off");
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_NO_FLUSH, "off");
 
util/qemu-option.c +57 -7

80 lines not on this page · raw message

Signature · 1 line
2.7.4
#89 lore raw

PATCH v14 15/21 qom: support non-scalar properties with -object

In reply to Daniel P. Berrange

The current -object command line syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.

This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.

The QObjectInputVisitor now has functionality that allows it to replace OptsVisitor, maintaining full backwards compatibility for previous CLI syntax, while also allowing use of new syntax for structs.

Thus -object can now support non-scalar properties, for example the QMP object:

  {
    "execute": "object-add",
    "arguments": {
      "qom-type": "demo",
      "id": "demo0",
      "parameters": {
        "foo": [
          { "bar": "one", "wizz": "1" },
          { "bar": "two", "wizz": "2" }
        ]
      }
    }
  }

Would be creatable via the CLI now using

    $QEMU \
      -object demo,id=demo0,\
              foo.0.bar=one,foo.0.wizz=1,\
              foo.1.bar=two,foo.1.wizz=2

Notice that this syntax is intentionally compatible with that currently used by block drivers. NB the indentation seen here after line continuations should not be used in reality, it is just for clarity in this commit message.

---
 qapi/qobject-input-visitor.c |   2 +-
 qom/object_interfaces.c      |  37 ++++-
 tests/check-qom-proplist.c   | 367 ++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 397 insertions(+), 9 deletions(-)
qapi/qobject-input-visitor.c +1 -1
index 5a3872c..f1030d5 100644
@@ -204,7 +204,7 @@ static void qobject_input_start_struct(Visitor *v, const char *name, void **obj,
         *obj = NULL;
     }
 
-    if (!qobj && (qiv->struct_level < qiv->autocreate_struct_levels)) {
+    if (!qobj && (qiv->struct_level <= qiv->autocreate_struct_levels)) {
         /* Create a new dict that contains all the currently
          * unvisited items */
         if (tos) {
qom/object_interfaces.c +30 -7
index fdc406b..88a1e88 100644
@@ -4,7 +4,8 @@
 #include "qemu/module.h"
 #include "qapi-visit.h"
 #include "qapi/qobject-output-visitor.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
+#include "qemu/option.h"
 
 void user_creatable_complete(Object *obj, Error **errp)
 {
@@ -63,12 +64,16 @@ Object *user_creatable_add(const QDict *qdict,
     if (local_err) {
         goto out_visit;
     }
-    visit_check_struct(v, &local_err);
+
+    obj = user_creatable_add_type(type, id, pdict, v, &local_err);
     if (local_err) {
         goto out_visit;
     }
 
-    obj = user_creatable_add_type(type, id, pdict, v, &local_err);
+    visit_check_struct(v, &local_err);
+    if (local_err) {
+        goto out_visit;
+    }
 
 out_visit:
     visit_end_struct(v, NULL);
@@ -114,7 +119,7 @@ Object *user_creatable_add_type(const char *type, const char *id,
 
     assert(qdict);
     obj = object_new(type);
-    visit_start_struct(v, NULL, NULL, 0, &local_err);
+    visit_start_struct(v, "props", NULL, 0, &local_err);
     if (local_err) {
         goto out;
     }
@@ -158,14 +163,32 @@ Object *user_creatable_add_opts(QemuOpts *opts, Error **errp)
 {
     Visitor *v;
     QDict *pdict;
+    QObject *pobj;
     Object *obj = NULL;
 
-    v = opts_visitor_new(opts);
-    pdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+    pdict = qemu_opts_to_qdict(opts, NULL,
+                               QEMU_OPTS_REPEAT_POLICY_ALL,
                                &error_abort);
 
-    obj = user_creatable_add(pdict, v, errp);
+    pobj = qdict_crumple(pdict, true, errp);
+    if (!pobj) {
+        goto cleanup;
+    }
+    /*
+     * We need autocreate_list=true + permit_int_ranges=true
+     * in order to maintain compat with OptsVisitor creation
+     * of the 'numa' object which had an int16List property.
+     *
+     * We need autocreate_structs=1, because at the CLI level
+     * we have the object type + properties in the same flat
+     * struct, even though at the QMP level they are nested.
+     */
+    v = qobject_input_visitor_new_autocast(pobj, true, 1, true);
+
+    obj = user_creatable_add(qobject_to_qdict(pobj), v, errp);
     visit_free(v);
+    qobject_decref(pobj);
+ cleanup:
     QDECREF(pdict);
     return obj;
 }
tests/check-qom-proplist.c +366 -1

466 lines not on this page · raw message

Signature · 1 line
2.7.4
#98 lore raw

PATCH v14 17/21 numa: convert to use QObjectInputVisitor for -numa

In reply to Daniel P. Berrange

Switch away from using OptsVisitor to parse the -numa argument processing. This enables use of the modern list syntax for specifying CPUs. e.g. the old syntax

-numa node,nodeid=0,cpus=0-3,cpus=8-11,mem=107

is equivalent to

  -numa node,nodeid=0,cpus.0=0,cpus.1=1,cpus.2=2,cpus.3=3,\
        cpus.4=8,cpus.5=9,cpus.6=10,cpus.7=11,mem=107

Furthermore, the cli arg can now follow the QAPI schema nesting, so the above is equivalent to the canonical syntax:

  -numa type=node,data.nodeid=0,data.cpus.0=0,data.cpus.1=1,\
        data.cpus.2=2,data.cpus.3=3,data.cpus.4=8,data.cpus.5=9,\
	data.cpus.6=10,data.cpus.7=11,data.mem=107

A test case is added to cover argument parsing to validate that both the old and new syntax is correctly handled.

---
 include/sysemu/numa_int.h |  11 +++++
 numa.c                    |  36 +++++++++-----
 stubs/Makefile.objs       |   5 ++
 stubs/exec.c              |   6 +++
 stubs/hostmem.c           |  14 ++++++
 stubs/memory.c            |  41 ++++++++++++++++
 stubs/qdev.c              |   8 ++++
 stubs/vl.c                |   8 ++++
 stubs/vmstate.c           |   4 ++
 tests/Makefile.include    |   2 +
 tests/test-numa.c         | 116 ++++++++++++++++++++++++++++++++++++++++++++++
 11 files changed, 240 insertions(+), 11 deletions(-)
 create mode 100644 include/sysemu/numa_int.h
 create mode 100644 stubs/exec.c
 create mode 100644 stubs/hostmem.c
 create mode 100644 stubs/memory.c
 create mode 100644 stubs/qdev.c
 create mode 100644 stubs/vl.c
 create mode 100644 tests/test-numa.c
include/sysemu/numa_int.h new file +11 -0
new file mode 100644
index 0000000..93160da
@@ -0,0 +1,11 @@
+#ifndef SYSEMU_NUMA_INT_H
+#define SYSEMU_NUMA_INT_H
+
+#include "sysemu/numa.h"
+
+extern int have_memdevs;
+extern int max_numa_nodeid;
+
+int parse_numa(void *opaque, QemuOpts *opts, Error **errp);
+
+#endif
numa.c +25 -11
index 6289f46..653ebf1 100644
@@ -23,14 +23,14 @@
  */
 
 #include "qemu/osdep.h"
-#include "sysemu/numa.h"
+#include "sysemu/numa_int.h"
 #include "exec/cpu-common.h"
 #include "qemu/bitmap.h"
 #include "qom/cpu.h"
 #include "qemu/error-report.h"
 #include "include/exec/cpu-common.h" /* for RAM_ADDR_FMT */
 #include "qapi-visit.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "hw/boards.h"
 #include "sysemu/hostmem.h"
 #include "qmp-commands.h"
@@ -45,10 +45,10 @@ QemuOptsList qemu_numa_opts = {
     .desc = { { 0 } } /* validated with OptsVisitor */
 };
 
-static int have_memdevs = -1;
-static int max_numa_nodeid; /* Highest specified NUMA node ID, plus one.
-                             * For all nodes, nodeid < max_numa_nodeid
-                             */
+int have_memdevs = -1;
+int max_numa_nodeid; /* Highest specified NUMA node ID, plus one.
+                      * For all nodes, nodeid < max_numa_nodeid
+                      */
 int nb_numa_nodes;
 NodeInfo numa_info[MAX_NODES];
 
@@ -189,6 +189,9 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
     if (node->has_mem) {
         uint64_t mem_size = node->mem;
         const char *mem_str = qemu_opt_get(opts, "mem");
+        if (!mem_str) {
+            mem_str = qemu_opt_get(opts, "data.mem");
+        }
         /* Fix up legacy suffix-less format */
         if (g_ascii_isdigit(mem_str[strlen(mem_str) - 1])) {
             mem_size <<= 20;
@@ -211,16 +214,27 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
     max_numa_nodeid = MAX(max_numa_nodeid, nodenr + 1);
 }
 
-static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
+int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
 {
     NumaOptions *object = NULL;
     Error *err = NULL;
+    Visitor *v;
 
-    {
-        Visitor *v = opts_visitor_new(opts);
-        visit_type_NumaOptions(v, NULL, &object, &err);
-        visit_free(v);
+    /*
+     * Needs autocreate_list=true, permit_int_ranges=true and
+     * permit_repeated_opts=true in order to support existing
+     * syntax for 'cpus' parameter which is an int list.
+     *
+     * Needs autocreate_struct_levels=1, because existing syntax
+     * used a nested struct in the QMP schema with a flat namespace
+     * in the CLI args.
+     */
+    v = qobject_input_visitor_new_opts(opts, true, 1, true, true, &err);
+    if (err) {
+        goto end;
     }
+    visit_type_NumaOptions(v, NULL, &object, &err);
+    visit_free(v);
 
     if (err) {
         goto end;
stubs/Makefile.objs +5 -0
index c5850e8..661b48a 100644
@@ -48,3 +48,8 @@ stub-obj-y += iohandler.o
 stub-obj-y += smbios_type_38.o
 stub-obj-y += ipmi.o
 stub-obj-y += pc_madt_cpu_entry.o
+stub-obj-y += vl.o
+stub-obj-y += exec.o
+stub-obj-y += memory.o
+stub-obj-y += hostmem.o
+stub-obj-y += qdev.o
stubs/exec.c new file +6 -0
new file mode 100644
index 0000000..e37f002
@@ -0,0 +1,6 @@
+
+#include "qemu/osdep.h"
+#include "exec/cpu-common.h"
+#include "qom/cpu.h"
+
+struct CPUTailQ cpus = QTAILQ_HEAD_INITIALIZER(cpus);
stubs/hostmem.c new file +14 -0
new file mode 100644
index 0000000..301d853
@@ -0,0 +1,14 @@
+
+#include "qemu/osdep.h"
+#include "sysemu/hostmem.h"
+
+void host_memory_backend_set_mapped(HostMemoryBackend *backend, bool mapped)
+{
+}
+
+
+MemoryRegion *host_memory_backend_get_memory(HostMemoryBackend *backend,
+                                             Error **errp)
+{
+    return NULL;
+}
stubs/memory.c new file +41 -0

41 lines not on this page · raw message

stubs/qdev.c new file +8 -0
new file mode 100644
index 0000000..e28fdf2
@@ -0,0 +1,8 @@
+
+#include "qemu/osdep.h"
+#include "hw/qdev-core.h"
+
+Object *qdev_get_machine(void)
+{
+    return NULL;
+}
stubs/vl.c new file +8 -0
new file mode 100644
index 0000000..698fa80
@@ -0,0 +1,8 @@
+
+#include "qemu/osdep.h"
+#include "exec/cpu-common.h"
+
+int max_cpus = 1;
+ram_addr_t ram_size;
+const char *mem_path;
+int mem_prealloc;
stubs/vmstate.c +4 -0
index 94b831e..f2708ee 100644
@@ -23,3 +23,7 @@ void vmstate_unregister(DeviceState *dev,
                         void *opaque)
 {
 }
+
+void vmstate_register_ram_global(struct MemoryRegion *memory)
+{
+}
tests/Makefile.include +2 -0
index 605f276..72f71ea 100644
@@ -116,6 +116,7 @@ check-unit-$(CONFIG_REPLICATION) += tests/test-replication$(EXESUF)
 check-unit-y += tests/test-bufferiszero$(EXESUF)
 gcov-files-check-bufferiszero-y = util/bufferiszero.c
 check-unit-y += tests/test-uuid$(EXESUF)
+check-unit-y += tests/test-numa$(EXESUF)
 
 check-block-$(CONFIG_POSIX) += tests/qemu-iotests-quick.sh
 
@@ -584,6 +585,7 @@ tests/test-crypto-pbkdf$(EXESUF): tests/test-crypto-pbkdf.o $(test-crypto-obj-y)
 tests/test-crypto-ivgen$(EXESUF): tests/test-crypto-ivgen.o $(test-crypto-obj-y)
 tests/test-crypto-afsplit$(EXESUF): tests/test-crypto-afsplit.o $(test-crypto-obj-y)
 tests/test-crypto-block$(EXESUF): tests/test-crypto-block.o $(test-crypto-obj-y)
+tests/test-numa$(EXESUF): tests/test-numa.o  numa.o $(test-qom-obj-y)
 
 libqos-obj-y = tests/libqos/pci.o tests/libqos/fw_cfg.o tests/libqos/malloc.o
 libqos-obj-y += tests/libqos/i2c.o tests/libqos/libqos.o
tests/test-numa.c new file +116 -0

116 lines not on this page · raw message

Signature · 1 line
2.7.4
#100 lore raw

PATCH v14 18/21 block: convert crypto driver to use QObjectInputVisitor

In reply to Daniel P. Berrange

The crypto block driver currently uses OptsVisitor to convert from the block driver open/create options into QCryptoBlockOpenOptions/QCryptoBlockCreateOptions. This is easily replaced by use of QObjectInputVisitor with no need to enable any compatibility options, since the structs dealt with contain only scalars.

---
 block/crypto.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
block/crypto.c +11 -3
index 7aa7eb5..12b0b9f 100644
@@ -23,7 +23,7 @@
 #include "block/block_int.h"
 #include "sysemu/block-backend.h"
 #include "crypto/block.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi-visit.h"
 #include "qapi/error.h"
 
@@ -206,7 +206,11 @@ block_crypto_open_opts_init(QCryptoBlockFormat format,
     ret = g_new0(QCryptoBlockOpenOptions, 1);
     ret->format = format;
 
-    v = opts_visitor_new(opts);
+    v = qobject_input_visitor_new_opts(opts, false, 0, false, false,
+                                       &local_err);
+    if (local_err) {
+        goto out;
+    }
 
     visit_start_struct(v, NULL, NULL, 0, &local_err);
     if (local_err) {
@@ -252,7 +256,11 @@ block_crypto_create_opts_init(QCryptoBlockFormat format,
     ret = g_new0(QCryptoBlockCreateOptions, 1);
     ret->format = format;
 
-    v = opts_visitor_new(opts);
+    v = qobject_input_visitor_new_opts(opts, false, 0, false, false,
+                                       &local_err);
+    if (local_err) {
+        goto out;
+    }
 
     visit_start_struct(v, NULL, NULL, 0, &local_err);
     if (local_err) {
Signature · 1 line
2.7.4
#96 lore raw

PATCH v14 16/21 hmp: support non-scalar properties with object_add

In reply to Daniel P. Berrange

The current object_add HMP syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.

This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.

The QObjectInputVisitor now has functionality that allows it to replace OptsVisitor, maintaining full backwards compatibility for previous CLI syntax, while also allowing use of new syntax for structs.

Thus -object can now support non-scalar properties, for example the QMP object:

  {
    "execute": "object-add",
    "arguments": {
      "qom-type": "demo",
      "id": "demo0",
      "parameters": {
        "foo": [
          { "bar": "one", "wizz": "1" },
          { "bar": "two", "wizz": "2" }
        ]
      }
    }
  }

Would be creatable via the HMP now using

   object_add demo,id=demo0,\
              foo.0.bar=one,foo.0.wizz=1,\
              foo.1.bar=two,foo.1.wizz=2

Notice that this syntax is intentionally compatible with that currently used by block drivers. NB the indentation seen here after line continuations should not be used in reality, it is just for clarity in this commit message.

---
 hmp.c | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)
hmp.c +17 -8
index 336e7bf..b32d8c8 100644
@@ -25,7 +25,7 @@
 #include "qemu/sockets.h"
 #include "monitor/monitor.h"
 #include "monitor/qdev.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/string-output-visitor.h"
 #include "qapi/util.h"
@@ -1696,21 +1696,30 @@ void hmp_netdev_del(Monitor *mon, const QDict *qdict)
 void hmp_object_add(Monitor *mon, const QDict *qdict)
 {
     Error *err = NULL;
-    QemuOpts *opts;
     Visitor *v;
     Object *obj = NULL;
+    QObject *pobj;
 
-    opts = qemu_opts_from_qdict(qemu_find_opts("object"), qdict, &err);
+    pobj = qdict_crumple(qdict, true, &err);
     if (err) {
-        hmp_handle_error(mon, &err);
-        return;
+        goto cleanup;
     }
 
-    v = opts_visitor_new(opts);
-    obj = user_creatable_add(qdict, v, &err);
+    /*
+     * We need autocreate_list=true + permit_int_ranges=true
+     * in order to maintain compat with OptsVisitor creation
+     * of the 'numa' object which had an int16List property.
+     *
+     * We need autocreate_structs=1, because at the CLI level
+     * we have the object type + properties in the same flat
+     * struct, even though at the QMP level they are nested.
+     */
+    v = qobject_input_visitor_new_autocast(pobj, true, 1, true);
+    obj = user_creatable_add(qobject_to_qdict(pobj), v, &err);
     visit_free(v);
-    qemu_opts_del(opts);
 
+ cleanup:
+    qobject_decref(pobj);
     if (err) {
         hmp_handle_error(mon, &err);
     }
Signature · 1 line
2.7.4
#101 lore raw

PATCH v14 19/21 acpi: convert to QObjectInputVisitor for -acpi parsing

In reply to Daniel P. Berrange

The -acpi command line option parsing uses the OptsVisitor currently. This is easily replaced by the QObjectInputVisitor instead. There is no need to enable any of the compatibility options, since the AcpiTableOptions QAPI struct only contains scalar properties.

---
 hw/acpi/core.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
hw/acpi/core.c +7 -7
index e890a5d..67c0abe 100644
@@ -25,7 +25,7 @@
 #include "hw/acpi/acpi.h"
 #include "hw/nvram/fw_cfg.h"
 #include "qemu/config-file.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi-visit.h"
 #include "qapi-event.h"
 
@@ -237,14 +237,14 @@ void acpi_table_add(const QemuOpts *opts, Error **errp)
     char **cur;
     size_t bloblen = 0;
     char unsigned *blob = NULL;
+    Visitor *v;
 
-    {
-        Visitor *v;
-
-        v = opts_visitor_new(opts);
-        visit_type_AcpiTableOptions(v, NULL, &hdrs, &err);
-        visit_free(v);
+    v = qobject_input_visitor_new_opts(opts, false, 0, false, false, &err);
+    if (err) {
+        goto out;
     }
+    visit_type_AcpiTableOptions(v, NULL, &hdrs, &err);
+    visit_free(v);
 
     if (err) {
         goto out;
Signature · 1 line
2.7.4
#102 lore raw

PATCH v14 20/21 net: convert to QObjectInputVisitor for -net/-netdev parsing

In reply to Daniel P. Berrange

The -net/-netdev command line parsing code uses OptsVisitor for parsing options to populate NetLegacy or NetDev struct respectively. Although those structs have nesting, the OptsVisitor flattens them, so we must enable compatibility options to auto-create structs. This allows the legacy syntax

-net tap,id=net0,vlan=3,fd=3,script=/bin/ifup-qemu

to be treated as equivalent to the modern QAPI based syntax

-net id=net0,vlan=3,opts.type=tap,opts.data.fd=3,opts.data.script=/bin/ifup-qemu

---
 net/net.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
net/net.c +16 -2
index ec984bf..de6bf8e 100644
@@ -43,7 +43,7 @@
 #include "qemu/iov.h"
 #include "qemu/main-loop.h"
 #include "qapi-visit.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "sysemu/sysemu.h"
 #include "net/filter.h"
 #include "qapi/string-output-visitor.h"
@@ -1069,7 +1069,21 @@ int net_client_init(QemuOpts *opts, bool is_netdev, Error **errp)
     void *object = NULL;
     Error *err = NULL;
     int ret = -1;
-    Visitor *v = opts_visitor_new(opts);
+    /*
+     * Needs autocreate_lists=true in order support existing
+     * syntax for list options where the bare key is repeated
+     *
+     * Needs autocreate_struct_levels=3 in order to deal with
+     * 3 level nesting in NetLegacy option args, which was
+     * exposed as a flat namespace with OptVisitor
+     */
+    Visitor *v = qobject_input_visitor_new_opts(opts, true, 3, false, true,
+                                                &err);
+
+    if (err) {
+        error_propagate(errp, err);
+        return -1;
+    }
 
     {
         /* Parse convenience option format ip6-net=fec0::0[/64] */
Signature · 1 line
2.7.4
#105 lore raw

PATCH v14 21/21 qapi: delete unused OptsVisitor code

In reply to Daniel P. Berrange

Now that all code has been converted to QObjectInputVisitor's QemuOpts compatibility mode, there is no longer any reason to keep OptsVisitor alive.

---
 include/qapi/opts-visitor.h |  40 ----
 qapi/Makefile.objs          |   2 +-
 qapi/opts-visitor.c         | 544 --------------------------------------------
 tests/Makefile.include      |   5 +-
 tests/test-opts-visitor.c   | 268 ----------------------
 vl.c                        |   1 -
 6 files changed, 2 insertions(+), 858 deletions(-)
 delete mode 100644 include/qapi/opts-visitor.h
 delete mode 100644 qapi/opts-visitor.c
 delete mode 100644 tests/test-opts-visitor.c
include/qapi/opts-visitor.h deleted +0 -40
deleted file mode 100644
index 6462c96..0000000
@@ -1,40 +0,0 @@
-/*
- * Options Visitor
- *
- * Copyright Red Hat, Inc. 2012
- *
- * Author: Laszlo Ersek <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef OPTS_VISITOR_H
-#define OPTS_VISITOR_H
-
-#include "qapi/visitor.h"
-#include "qemu/option.h"
-
-/* Inclusive upper bound on the size of any flattened range. This is a safety
- * (= anti-annoyance) measure; wrong ranges should not cause long startup
- * delays nor exhaust virtual memory.
- */
-#define OPTS_VISITOR_RANGE_MAX 65536
-
-typedef struct OptsVisitor OptsVisitor;
-
-/* Contrarily to qemu-option.c::parse_option_number(), OptsVisitor's "int"
- * parser relies on strtoll() instead of strtoull(). Consequences:
- * - string representations of negative numbers yield negative values,
- * - values below INT64_MIN or LLONG_MIN are rejected,
- * - values above INT64_MAX or LLONG_MAX are rejected.
- *
- * The Opts input visitor does not implement support for visiting QAPI
- * alternates, numbers (other than integers), null, or arbitrary
- * QTypes.  It also requires a non-null list argument to
- * visit_start_list().
- */
-Visitor *opts_visitor_new(const QemuOpts *opts);
-
-#endif
qapi/Makefile.objs +1 -1
index 33906ff..4b820a7 100644
@@ -1,6 +1,6 @@
 util-obj-y = qapi-visit-core.o qapi-dealloc-visitor.o qobject-input-visitor.o
 util-obj-y += qobject-output-visitor.o qmp-registry.o qmp-dispatch.o
 util-obj-y += string-input-visitor.o string-output-visitor.o
-util-obj-y += opts-visitor.o qapi-clone-visitor.o
+util-obj-y += qapi-clone-visitor.o
 util-obj-y += qmp-event.o
 util-obj-y += qapi-util.o
qapi/opts-visitor.c deleted +0 -544

544 lines not on this page · raw message

tests/Makefile.include +1 -4
index 72f71ea..df39976 100644
@@ -35,8 +35,6 @@ check-unit-y += tests/test-string-output-visitor$(EXESUF)
 gcov-files-test-string-output-visitor-y = qapi/string-output-visitor.c
 check-unit-y += tests/test-qmp-event$(EXESUF)
 gcov-files-test-qmp-event-y += qapi/qmp-event.c
-check-unit-y += tests/test-opts-visitor$(EXESUF)
-gcov-files-test-opts-visitor-y = qapi/opts-visitor.c
 check-unit-y += tests/test-coroutine$(EXESUF)
 gcov-files-test-coroutine-y = coroutine-$(CONFIG_COROUTINE_BACKEND).c
 check-unit-y += tests/test-visitor-serialization$(EXESUF)
@@ -445,7 +443,7 @@ test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \
 	tests/test-qobject-input-visitor.o tests/test-qobject-input-strict.o \
 	tests/test-qmp-commands.o tests/test-visitor-serialization.o \
 	tests/test-x86-cpuid.o tests/test-mul64.o tests/test-int128.o \
-	tests/test-opts-visitor.o tests/test-qmp-event.o \
+	tests/test-qmp-event.o \
 	tests/rcutorture.o tests/test-rcu-list.o \
 	tests/test-qdist.o \
 	tests/test-qht.o tests/qht-bench.o tests/test-qht-par.o
@@ -549,7 +547,6 @@ tests/test-qobject-input-visitor$(EXESUF): tests/test-qobject-input-visitor.o $(
 tests/test-qobject-input-strict$(EXESUF): tests/test-qobject-input-strict.o $(test-qapi-obj-y)
 tests/test-qmp-commands$(EXESUF): tests/test-qmp-commands.o tests/test-qmp-marshal.o $(test-qapi-obj-y)
 tests/test-visitor-serialization$(EXESUF): tests/test-visitor-serialization.o $(test-qapi-obj-y)
-tests/test-opts-visitor$(EXESUF): tests/test-opts-visitor.o $(test-qapi-obj-y)
 
 tests/test-mul64$(EXESUF): tests/test-mul64.o $(test-util-obj-y)
 tests/test-bitops$(EXESUF): tests/test-bitops.o $(test-util-obj-y)
tests/test-opts-visitor.c deleted +0 -268

268 lines not on this page · raw message

vl.c +0 -1
index ab0349b..3a214c1 100644
@@ -115,7 +115,6 @@ int main(int argc, char **argv)
 
 #include "ui/qemu-spice.h"
 #include "qapi/string-input-visitor.h"
-#include "qapi/opts-visitor.h"
 #include "qom/object_interfaces.h"
 #include "qapi-event.h"
 #include "exec/semihost.h"
Signature · 1 line
2.7.4
#106 lore raw
In reply to Daniel P. Berrange
Your series failed automatic build test. Please find the testing commands and their output below. If you have docker installed, you can probably reproduce it locally.

Hi,

Your series failed automatic build test. Please find the testing commands and their output below. If you have docker installed, you can probably reproduce it locally.

Type: series
Message-id: [email protected]
Subject: [Qemu-devel] [PATCH v14 00/21] QAPI/QOM work for non-scalar object properties

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
make J=8 docker-test-quick@centos6
make J=8 docker-test-mingw@fedora
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
- [tag update] patchew/[email protected] -> patchew/[email protected]
Switched to a new branch 'test'
9b89b2d qapi: delete unused OptsVisitor code
7d7d027 net: convert to QObjectInputVisitor for -net/-netdev parsing f7fd6c1 acpi: convert to QObjectInputVisitor for -acpi parsing fecaabf block: convert crypto driver to use QObjectInputVisitor 7941b3f numa: convert to use QObjectInputVisitor for -numa 3c6762d hmp: support non-scalar properties with object_add efb01de qom: support non-scalar properties with -object 3c90b78 qapi: allow repeated opts with qobject_input_visitor_new_opts 81f728f qdict: allow qdict_crumple to accept compound types as values 8353c22 option: allow qemu_opts_to_qdict to merge repeated options 0948fcd qapi: add integer range support for QObjectInputVisitor 6da7117 qapi: permit auto-creating nested structs
1170f4b qapi: permit auto-creating single element lists 2ec3eac qapi: allow QObjectInputVisitor to be created with QemuOpts b56363f qapi: permit scalar type conversions in QObjectInputVisitor 6fa7850 qapi: don't pass two copies of TestInputVisitorData to tests d1b24d3 qapi: rename QmpOutputVisitor to QObjectOutputVisitor 794fec6 qapi: rename QmpInputVisitor to QObjectInputVisitor 43b5ede qapi: add trace events for visitor
587d218 qdict: implement a qdict_crumple method for un-flattening a dict 67cbb95 option: make parse_option_bool/number non-static

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc' Cloning into 'dtc'...
Submodule path 'dtc': checked out '65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf'
BUILD centos6
ARCHIVE qemu.tgz
ARCHIVE dtc.tgz
COPY RUNNER
RUN test-quick in centos6
Packages installed:
SDL-devel-1.2.14-7.el6_7.1.x86_64
ccache-3.1.6-2.el6.x86_64
epel-release-6-8.noarch
gcc-4.4.7-17.el6.x86_64
git-1.7.1-4.el6_7.1.x86_64
glib2-devel-2.28.8-5.el6.x86_64
libfdt-devel-1.4.0-1.el6.x86_64
make-3.81-23.el6.x86_64
package g++ is not installed
pixman-devel-0.32.8-1.el6.x86_64
tar-1.23-15.el6_8.x86_64
zlib-devel-1.2.3-29.el6.x86_64

Environment variables:
PACKAGES=libfdt-devel ccache tar git make gcc g++ zlib-devel glib2-devel SDL-devel pixman-devel epel-release
HOSTNAME=9320e4514cd6
TERM=xterm
MAKEFLAGS= -j8
HISTSIZE=1000
J=8
USER=root
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
MAIL=/var/spool/mail/root PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PWD=/
LANG=en_US.UTF-8
TARGET_LIST=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
FEATURES= dtc
DEBUG=
G_BROKEN_FILENAMES=1
CCACHE_HASHDIR=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/src/tests/docker/install No C++ compiler available; disabling C++ specific optional code

Install prefix    /tmp/qemu-test/src/tests/docker/install
BIOS directory    /tmp/qemu-test/src/tests/docker/install/share/qemu
binary directory  /tmp/qemu-test/src/tests/docker/install/bin

library directory /tmp/qemu-test/src/tests/docker/install/lib module directory /tmp/qemu-test/src/tests/docker/install/lib/qemu libexec directory /tmp/qemu-test/src/tests/docker/install/libexec include directory /tmp/qemu-test/src/tests/docker/install/include config directory /tmp/qemu-test/src/tests/docker/install/etc
local state directory /tmp/qemu-test/src/tests/docker/install/var
Manual directory /tmp/qemu-test/src/tests/docker/install/share/man ELF interp prefix /usr/gnemul/qemu-%M

Source path       /tmp/qemu-test/src
C compiler        cc
Host C compiler   cc
C++ compiler      
Objective-C compiler cc
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/include/pixman-1    -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all
LDFLAGS           -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       x86_64-softmmu aarch64-softmmu
tcg debug enabled no
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
pixman            system
SDL support       yes (1.2.14)
GTK support       no 
GTK GL support    no
VTE support       no 
TLS priority      NORMAL
GNUTLS support    no
GNUTLS rnd        no
libgcrypt         no
libgcrypt kdf     no
nettle            no 
nettle kdf        no
libtasn1          no
curses support    no
virgl support     no
curl support      no
mingw32 support   no
Audio drivers     oss
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    no
VNC support       yes
VNC SASL support  no
VNC JPEG support  no
VNC PNG support   no
xen support       no
brlapi support    no
bluez  support    no
Documentation     no
PIE               yes
vde support       no
netmap support    no
Linux AIO support no
ATTR/XATTR support yes
Install blobs     yes
KVM support       yes
RDMA support      no
TCG interpreter   no
fdt support       yes
preadv support    yes
fdatasync         yes
madvise           yes
posix_madvise     yes

libcap-ng support no
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes

Trace backends    log
spice support     no 
rbd support       no
xfsctl support    no
smartcard support no
libusb            no
usb net redir     no
OpenGL support    no
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no

build guest agent yes
QGA VSS support no
QGA w32 disk info no
QGA MSI support no
seccomp support no
coroutine backend ucontext
coroutine pool yes
GlusterFS support no
Archipelago support no

gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   no
TPM passthrough   yes
QOM debugging     yes
lzo support       no
snappy support    no
bzip2 support     no

NUMA host support no
tcmalloc support no
jemalloc support no
avx2 optimization no
replication support yes

  GEN   x86_64-softmmu/config-devices.mak.tmp
  GEN   aarch64-softmmu/config-devices.mak.tmp
  GEN   config-host.h
  GEN   qemu-options.def
  GEN   qmp-commands.h
  GEN   qapi-types.h
  GEN   qapi-visit.h
  GEN   qapi-event.h
  GEN   x86_64-softmmu/config-devices.mak
  GEN   aarch64-softmmu/config-devices.mak
  GEN   qmp-introspect.h
  GEN   module_block.h
  GEN   tests/test-qapi-types.h
  GEN   tests/test-qapi-visit.h
  GEN   tests/test-qmp-commands.h
  GEN   tests/test-qapi-event.h
  GEN   tests/test-qmp-introspect.h
  GEN   config-all-devices.mak
  GEN   trace/generated-events.h
  GEN   trace/generated-tracers.h
  GEN   trace/generated-tcg-tracers.h
  GEN   trace/generated-helpers-wrappers.h
  GEN   trace/generated-helpers.h
  CC    tests/qemu-iotests/socket_scm_helper.o
  GEN   qga/qapi-generated/qga-qapi-types.h
  GEN   qga/qapi-generated/qga-qapi-visit.h
  GEN   qga/qapi-generated/qga-qmp-commands.h
  GEN   qga/qapi-generated/qga-qapi-types.c
  GEN   qga/qapi-generated/qga-qapi-visit.c
  GEN   qga/qapi-generated/qga-qmp-marshal.c
  GEN   qmp-introspect.c
  GEN   qapi-types.c
  GEN   qapi-visit.c
  GEN   qapi-event.c
  CC    qapi/qapi-visit-core.o
  CC    qapi/qapi-dealloc-visitor.o
  CC    qapi/qobject-input-visitor.o
  CC    qapi/qobject-output-visitor.o
  CC    qapi/qmp-registry.o
  CC    qapi/qmp-dispatch.o
  CC    qapi/string-input-visitor.o
  CC    qapi/string-output-visitor.o
/tmp/qemu-test/src/qapi/qobject-input-visitor.c: In function ‘qobject_input_get_object’:
/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: warning: implicit declaration of function ‘g_hash_table_contains’
/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: warning: nested extern declaration of ‘g_hash_table_contains’
  CC    qapi/qapi-clone-visitor.o
  CC    qapi/qmp-event.o
  CC    qapi/qapi-util.o
  CC    qobject/qnull.o
  CC    qobject/qint.o
  CC    qobject/qstring.o
  CC    qobject/qdict.o
  CC    qobject/qlist.o
  CC    qobject/qfloat.o
  CC    qobject/qbool.o
  CC    qobject/qjson.o
  CC    qobject/qobject.o
  CC    qobject/json-lexer.o
  CC    qobject/json-streamer.o
  CC    qobject/json-parser.o
  GEN   trace/generated-events.c
  CC    trace/control.o
  CC    trace/qmp.o
  CC    util/osdep.o
  CC    util/cutils.o
  CC    util/unicode.o
  CC    util/qemu-timer-common.o
  CC    util/bufferiszero.o
  CC    util/compatfd.o
  CC    util/event_notifier-posix.o
  CC    util/mmap-alloc.o
  CC    util/oslib-posix.o
  CC    util/qemu-openpty.o
  CC    util/qemu-thread-posix.o
  CC    util/memfd.o
  CC    util/path.o
  CC    util/envlist.o
  CC    util/module.o
  CC    util/bitmap.o
  CC    util/bitops.o
  CC    util/hbitmap.o
  CC    util/fifo8.o
  CC    util/acl.o
  CC    util/error.o
  CC    util/qemu-error.o
  CC    util/id.o
  CC    util/iov.o
  CC    util/qemu-config.o
  CC    util/qemu-sockets.o
  CC    util/uri.o
  CC    util/notify.o
  CC    util/qemu-option.o
  CC    util/qemu-progress.o
  CC    util/hexdump.o
  CC    util/crc32c.o
  CC    util/uuid.o
  CC    util/throttle.o
  CC    util/getauxval.o
  CC    util/readline.o
  CC    util/rfifolock.o
  CC    util/rcu.o
  CC    util/qemu-coroutine.o
  CC    util/qemu-coroutine-lock.o
  CC    util/qemu-coroutine-io.o
  CC    util/qemu-coroutine-sleep.o
  CC    util/coroutine-ucontext.o
  CC    util/buffer.o
  CC    util/timed-average.o
  CC    util/base64.o
  CC    util/log.o
  CC    util/qdist.o
  CC    util/qht.o
  CC    util/range.o
  CC    crypto/pbkdf-stub.o
  CC    stubs/arch-query-cpu-def.o
  CC    stubs/arch-query-cpu-model-expansion.o
  CC    stubs/arch-query-cpu-model-comparison.o
  CC    stubs/arch-query-cpu-model-baseline.o
  CC    stubs/bdrv-next-monitor-owned.o
  CC    stubs/blk-commit-all.o
  CC    stubs/blockdev-close-all-bdrv-states.o
  CC    stubs/clock-warp.o
/tmp/qemu-test/src/util/qht.c: In function ‘qht_reset_size’:
/tmp/qemu-test/src/util/qht.c:413: warning: ‘new’ may be used uninitialized in this function
  CC    stubs/cpu-get-clock.o
  CC    stubs/cpu-get-icount.o
  CC    stubs/dump.o
  CC    stubs/fdset-add-fd.o
  CC    stubs/fdset-find-fd.o
  CC    stubs/fdset-get-fd.o
  CC    stubs/fdset-remove-fd.o
  CC    stubs/gdbstub.o
  CC    stubs/get-fd.o
  CC    stubs/get-next-serial.o
  CC    stubs/get-vm-name.o
  CC    stubs/iothread-lock.o
  CC    stubs/is-daemonized.o
  CC    stubs/machine-init-done.o
  CC    stubs/migr-blocker.o
  CC    stubs/mon-is-qmp.o
  CC    stubs/mon-printf.o
  CC    stubs/monitor-init.o
  CC    stubs/notify-event.o
  CC    stubs/qtest.o
  CC    stubs/replay.o
  CC    stubs/replay-user.o
  CC    stubs/reset.o
  CC    stubs/runstate-check.o
  CC    stubs/set-fd-handler.o
  CC    stubs/slirp.o
  CC    stubs/sysbus.o
  CC    stubs/trace-control.o
  CC    stubs/uuid.o
  CC    stubs/vm-stop.o
  CC    stubs/vmstate.o
  CC    stubs/cpus.o
  CC    stubs/kvm.o
  CC    stubs/qmp_pc_dimm_device_list.o
  CC    stubs/target-monitor-defs.o
  CC    stubs/target-get-monitor-def.o
  CC    stubs/vhost.o
  CC    stubs/iohandler.o
  CC    stubs/smbios_type_38.o
  CC    stubs/ipmi.o
  CC    stubs/pc_madt_cpu_entry.o
  CC    stubs/vl.o
  CC    stubs/exec.o
  CC    stubs/memory.o
  CC    stubs/hostmem.o
  CC    stubs/qdev.o
  CC    contrib/ivshmem-client/ivshmem-client.o
  CC    contrib/ivshmem-client/main.o
  CC    contrib/ivshmem-server/ivshmem-server.o
  CC    contrib/ivshmem-server/main.o
  CC    qemu-nbd.o
  CC    async.o
  CC    thread-pool.o
  CC    block.o
  CC    blockjob.o
  CC    main-loop.o
  CC    iohandler.o
  CC    qemu-timer.o
  CC    aio-posix.o
  CC    qemu-io-cmds.o
  CC    block/raw_bsd.o
  CC    replication.o
  CC    block/qcow.o
  CC    block/vdi.o
  CC    block/vmdk.o
  CC    block/cloop.o
  CC    block/bochs.o
  CC    block/vpc.o
  CC    block/vvfat.o
  CC    block/dmg.o
  CC    block/qcow2.o
  CC    block/qcow2-refcount.o
  CC    block/qcow2-cluster.o
  CC    block/qcow2-snapshot.o
  CC    block/qcow2-cache.o
  CC    block/qed.o
  CC    block/qed-gencb.o
  CC    block/qed-l2-cache.o
  CC    block/qed-table.o
  CC    block/qed-cluster.o
  CC    block/qed-check.o
  CC    block/vhdx.o
  CC    block/vhdx-endian.o
  CC    block/vhdx-log.o
  CC    block/quorum.o
  CC    block/parallels.o
  CC    block/blkdebug.o
  CC    block/blkverify.o
  CC    block/blkreplay.o
  CC    block/block-backend.o
  CC    block/snapshot.o
  CC    block/qapi.o
  CC    block/raw-posix.o
  CC    block/null.o
  CC    block/mirror.o
  CC    block/commit.o
  CC    block/io.o
  CC    block/throttle-groups.o
  CC    block/nbd.o
  CC    block/nbd-client.o
  CC    block/sheepdog.o
  CC    block/accounting.o
  CC    block/dirty-bitmap.o
  CC    block/write-threshold.o
  CC    block/backup.o
  CC    block/replication.o
  CC    block/crypto.o
  CC    nbd/server.o
  CC    nbd/client.o
  CC    nbd/common.o
  CC    crypto/init.o
  CC    crypto/hash.o
  CC    crypto/hash-glib.o
  CC    crypto/aes.o
  CC    crypto/desrfb.o
  CC    crypto/cipher.o
  CC    crypto/tlscreds.o
  CC    crypto/tlscredsanon.o
  CC    crypto/tlscredsx509.o
  CC    crypto/tlssession.o
  CC    crypto/secret.o
  CC    crypto/random-platform.o
  CC    crypto/pbkdf.o
  CC    crypto/ivgen.o
  CC    crypto/ivgen-essiv.o
  CC    crypto/ivgen-plain.o
  CC    crypto/ivgen-plain64.o
  CC    crypto/afsplit.o
  CC    crypto/xts.o
  CC    crypto/block.o
  CC    crypto/block-qcow.o
  CC    crypto/block-luks.o
  CC    io/channel.o
  CC    io/channel-buffer.o
  CC    io/channel-command.o
  CC    io/channel-file.o
  CC    io/channel-socket.o
  CC    io/channel-tls.o
  CC    io/channel-watch.o
  CC    io/channel-websock.o
  CC    io/channel-util.o
  CC    io/task.o
  CC    qom/object.o
  CC    qom/container.o
  CC    qom/qom-qobject.o
  CC    qom/object_interfaces.o
  GEN   qemu-img-cmds.h
  CC    qemu-io.o
  CC    qemu-bridge-helper.o
  CC    blockdev.o
  CC    blockdev-nbd.o
  CC    iothread.o
  CC    qdev-monitor.o
  CC    device-hotplug.o
  CC    os-posix.o
  CC    qemu-char.o
  CC    page_cache.o
  CC    accel.o
  CC    bt-host.o
  CC    bt-vhci.o
  CC    dma-helpers.o
  CC    vl.o
  CC    tpm.o
  CC    device_tree.o
  GEN   qmp-marshal.c
  CC    qmp.o
  CC    hmp.o
  CC    tcg-runtime.o
  CC    cpus-common.o
  CC    audio/audio.o
  CC    audio/noaudio.o
  CC    audio/wavaudio.o
  CC    audio/mixeng.o
  CC    audio/sdlaudio.o
  CC    audio/ossaudio.o
  CC    audio/wavcapture.o
  CC    backends/rng.o
  CC    backends/rng-egd.o
  CC    backends/rng-random.o
  CC    backends/msmouse.o
  CC    backends/testdev.o
  CC    backends/tpm.o
  CC    backends/hostmem.o
  CC    backends/hostmem-ram.o
  CC    backends/hostmem-file.o
  CC    block/stream.o
  CC    disas/arm.o
  CC    disas/i386.o
  CC    fsdev/qemu-fsdev-dummy.o
  CC    fsdev/qemu-fsdev-opts.o
  CC    hw/acpi/core.o
  CC    hw/acpi/piix4.o
  CC    hw/acpi/pcihp.o
  CC    hw/acpi/ich9.o
  CC    hw/acpi/tco.o
  CC    hw/acpi/cpu_hotplug.o
  CC    hw/acpi/memory_hotplug.o
  CC    hw/acpi/memory_hotplug_acpi_table.o
  CC    hw/acpi/cpu.o
  CC    hw/acpi/acpi_interface.o
  CC    hw/acpi/bios-linker-loader.o
  CC    hw/acpi/aml-build.o
  CC    hw/acpi/ipmi.o
  CC    hw/audio/sb16.o
  CC    hw/audio/es1370.o
  CC    hw/audio/ac97.o
  CC    hw/audio/fmopl.o
  CC    hw/audio/adlib.o
  CC    hw/audio/gus.o
  CC    hw/audio/gusemu_hal.o
  CC    hw/audio/gusemu_mixer.o
  CC    hw/audio/cs4231a.o
  CC    hw/audio/intel-hda.o
  CC    hw/audio/hda-codec.o
  CC    hw/audio/pcspk.o
  CC    hw/audio/wm8750.o
  CC    hw/audio/pl041.o
  CC    hw/audio/lm4549.o
  CC    hw/audio/marvell_88w8618.o
  CC    hw/block/block.o
  CC    hw/block/cdrom.o
  CC    hw/block/hd-geometry.o
  CC    hw/block/fdc.o
  CC    hw/block/m25p80.o
  CC    hw/block/nand.o
  CC    hw/block/pflash_cfi01.o
  CC    hw/block/pflash_cfi02.o
  CC    hw/block/ecc.o
  CC    hw/block/onenand.o
  CC    hw/block/nvme.o
  CC    hw/bt/core.o
  CC    hw/bt/l2cap.o
  CC    hw/bt/sdp.o
  CC    hw/bt/hci.o
  CC    hw/bt/hid.o
  CC    hw/bt/hci-csr.o
  CC    hw/char/ipoctal232.o
  CC    hw/char/parallel.o
  CC    hw/char/pl011.o
  CC    hw/char/serial.o
  CC    hw/char/serial-isa.o
  CC    hw/char/serial-pci.o
  CC    hw/char/virtio-console.o
  CC    hw/char/cadence_uart.o
  CC    hw/char/debugcon.o
  CC    hw/char/imx_serial.o
  CC    hw/core/qdev.o
  CC    hw/core/qdev-properties.o
  CC    hw/core/bus.o
  CC    hw/core/fw-path-provider.o
  CC    hw/core/irq.o
  CC    hw/core/hotplug.o
  CC    hw/core/ptimer.o
  CC    hw/core/sysbus.o
  CC    hw/core/machine.o
  CC    hw/core/null-machine.o
  CC    hw/core/loader.o
  CC    hw/core/qdev-properties-system.o
  CC    hw/core/register.o
  CC    hw/core/platform-bus.o
  CC    hw/display/ads7846.o
  CC    hw/display/cirrus_vga.o
  CC    hw/display/pl110.o
  CC    hw/display/ssd0303.o
  CC    hw/display/ssd0323.o
  CC    hw/display/vga-pci.o
  CC    hw/display/vga-isa.o
  CC    hw/display/vmware_vga.o
  CC    hw/display/blizzard.o
  CC    hw/display/exynos4210_fimd.o
  CC    hw/display/framebuffer.o
  CC    hw/display/tc6393xb.o
  CC    hw/dma/pl080.o
  CC    hw/dma/pl330.o
  CC    hw/dma/i8257.o
  CC    hw/dma/xlnx-zynq-devcfg.o
  CC    hw/gpio/max7310.o
  CC    hw/gpio/pl061.o
  CC    hw/gpio/zaurus.o
  CC    hw/gpio/gpio_key.o
  CC    hw/i2c/core.o
  CC    hw/i2c/smbus.o
  CC    hw/i2c/smbus_eeprom.o
  CC    hw/i2c/i2c-ddc.o
  CC    hw/i2c/versatile_i2c.o
  CC    hw/i2c/smbus_ich9.o
  CC    hw/i2c/pm_smbus.o
  CC    hw/i2c/bitbang_i2c.o
  CC    hw/i2c/exynos4210_i2c.o
  CC    hw/i2c/imx_i2c.o
  CC    hw/i2c/aspeed_i2c.o
  CC    hw/ide/core.o
  CC    hw/ide/atapi.o
  CC    hw/ide/qdev.o
  CC    hw/ide/pci.o
  CC    hw/ide/isa.o
  CC    hw/ide/piix.o
  CC    hw/ide/microdrive.o
  CC    hw/ide/ahci.o
  CC    hw/ide/ich.o
  CC    hw/input/hid.o
  CC    hw/input/lm832x.o
  CC    hw/input/pckbd.o
  CC    hw/input/pl050.o
  CC    hw/input/ps2.o
  CC    hw/input/stellaris_input.o
  CC    hw/input/tsc2005.o
  CC    hw/input/vmmouse.o
  CC    hw/input/virtio-input.o
  CC    hw/input/virtio-input-hid.o
  CC    hw/input/virtio-input-host.o
  CC    hw/intc/i8259_common.o
  CC    hw/intc/i8259.o
  CC    hw/intc/pl190.o
  CC    hw/intc/imx_avic.o
  CC    hw/intc/realview_gic.o
  CC    hw/intc/ioapic_common.o
  CC    hw/intc/arm_gic_common.o
  CC    hw/intc/arm_gic.o
  CC    hw/intc/arm_gicv2m.o
  CC    hw/intc/arm_gicv3_common.o
  CC    hw/intc/arm_gicv3.o
  CC    hw/intc/arm_gicv3_dist.o
  CC    hw/intc/arm_gicv3_redist.o
  CC    hw/ipack/ipack.o
  CC    hw/ipack/tpci200.o
  CC    hw/ipmi/ipmi.o
  CC    hw/ipmi/ipmi_bmc_sim.o
  CC    hw/ipmi/ipmi_bmc_extern.o
  CC    hw/ipmi/isa_ipmi_kcs.o
  CC    hw/ipmi/isa_ipmi_bt.o
  CC    hw/isa/isa-bus.o
  CC    hw/isa/apm.o
  CC    hw/mem/pc-dimm.o
  CC    hw/mem/nvdimm.o
  CC    hw/misc/applesmc.o
  CC    hw/misc/max111x.o
  CC    hw/misc/tmp105.o
  CC    hw/misc/debugexit.o
  CC    hw/misc/sga.o
  CC    hw/misc/pc-testdev.o
  CC    hw/misc/pci-testdev.o
  CC    hw/misc/arm_l2x0.o
  CC    hw/misc/arm_integrator_debug.o
  CC    hw/misc/a9scu.o
  CC    hw/misc/arm11scu.o
  CC    hw/net/ne2000.o
  CC    hw/net/eepro100.o
  CC    hw/net/pcnet-pci.o
  CC    hw/net/pcnet.o
  CC    hw/net/e1000.o
  CC    hw/net/e1000x_common.o
  CC    hw/net/net_tx_pkt.o
  CC    hw/net/net_rx_pkt.o
  CC    hw/net/e1000e.o
  CC    hw/net/e1000e_core.o
  CC    hw/net/rtl8139.o
  CC    hw/net/vmxnet3.o
  CC    hw/net/smc91c111.o
  CC    hw/net/lan9118.o
  CC    hw/net/ne2000-isa.o
  CC    hw/net/xgmac.o
  CC    hw/net/allwinner_emac.o
  CC    hw/net/imx_fec.o
  CC    hw/net/cadence_gem.o
  CC    hw/net/stellaris_enet.o
  CC    hw/net/rocker/rocker.o
  CC    hw/net/rocker/rocker_fp.o
  CC    hw/net/rocker/rocker_desc.o
  CC    hw/net/rocker/rocker_world.o
  CC    hw/net/rocker/rocker_of_dpa.o
  CC    hw/nvram/eeprom93xx.o
  CC    hw/nvram/fw_cfg.o
  CC    hw/pci-bridge/pci_bridge_dev.o
  CC    hw/pci-bridge/pci_expander_bridge.o
  CC    hw/pci-bridge/xio3130_upstream.o
  CC    hw/pci-bridge/xio3130_downstream.o
  CC    hw/pci-bridge/ioh3420.o
  CC    hw/pci-bridge/i82801b11.o
  CC    hw/pci-host/pam.o
  CC    hw/pci-host/versatile.o
  CC    hw/pci-host/piix.o
  CC    hw/pci-host/q35.o
  CC    hw/pci-host/gpex.o
  CC    hw/pci/pci.o
  CC    hw/pci/pci_bridge.o
  CC    hw/pci/msix.o
  CC    hw/pci/msi.o
/tmp/qemu-test/src/hw/nvram/fw_cfg.c: In function ‘fw_cfg_dma_transfer’:
/tmp/qemu-test/src/hw/nvram/fw_cfg.c:330: warning: ‘read’ may be used uninitialized in this function
  CC    hw/pci/shpc.o
  CC    hw/pci/slotid_cap.o
  CC    hw/pci/pci_host.o
  CC    hw/pci/pcie_host.o
  CC    hw/pci/pcie.o
  CC    hw/pci/pcie_aer.o
  CC    hw/pci/pcie_port.o
  CC    hw/pci/pci-stub.o
  CC    hw/pcmcia/pcmcia.o
  CC    hw/scsi/scsi-disk.o
  CC    hw/scsi/scsi-generic.o
  CC    hw/scsi/scsi-bus.o
  CC    hw/scsi/lsi53c895a.o
  CC    hw/scsi/mptsas.o
  CC    hw/scsi/mptconfig.o
  CC    hw/scsi/mptendian.o
  CC    hw/scsi/megasas.o
  CC    hw/scsi/vmw_pvscsi.o
  CC    hw/scsi/esp.o
  CC    hw/scsi/esp-pci.o
  CC    hw/sd/pl181.o
  CC    hw/sd/ssi-sd.o
  CC    hw/sd/sd.o
  CC    hw/sd/core.o
  CC    hw/sd/sdhci.o
  CC    hw/smbios/smbios.o
  CC    hw/smbios/smbios_type_38.o
  CC    hw/ssi/pl022.o
  CC    hw/ssi/ssi.o
  CC    hw/ssi/xilinx_spips.o
  CC    hw/ssi/aspeed_smc.o
  CC    hw/timer/arm_timer.o
  CC    hw/timer/arm_mptimer.o
  CC    hw/timer/a9gtimer.o
  CC    hw/timer/cadence_ttc.o
  CC    hw/timer/ds1338.o
  CC    hw/timer/hpet.o
  CC    hw/timer/i8254_common.o
  CC    hw/timer/i8254.o
  CC    hw/timer/pl031.o
  CC    hw/timer/twl92230.o
  CC    hw/timer/imx_epit.o
  CC    hw/timer/imx_gpt.o
  CC    hw/timer/stm32f2xx_timer.o
  CC    hw/timer/aspeed_timer.o
  CC    hw/tpm/tpm_tis.o
  CC    hw/tpm/tpm_passthrough.o
  CC    hw/tpm/tpm_util.o
  CC    hw/usb/core.o
  CC    hw/usb/combined-packet.o
  CC    hw/usb/bus.o
  CC    hw/usb/libhw.o
  CC    hw/usb/desc.o
  CC    hw/usb/desc-msos.o
  CC    hw/usb/hcd-uhci.o
  CC    hw/usb/hcd-ohci.o
  CC    hw/usb/hcd-ehci.o
  CC    hw/usb/hcd-ehci-pci.o
  CC    hw/usb/hcd-xhci.o
  CC    hw/usb/hcd-ehci-sysbus.o
  CC    hw/usb/hcd-musb.o
  CC    hw/usb/dev-hub.o
  CC    hw/usb/dev-hid.o
  CC    hw/usb/dev-wacom.o
  CC    hw/usb/dev-storage.o
  CC    hw/usb/dev-uas.o
  CC    hw/usb/dev-audio.o
  CC    hw/usb/dev-serial.o
  CC    hw/usb/dev-network.o
  CC    hw/usb/dev-bluetooth.o
  CC    hw/usb/dev-smartcard-reader.o
  CC    hw/usb/dev-mtp.o
  CC    hw/usb/host-stub.o
  CC    hw/virtio/virtio-rng.o
  CC    hw/virtio/virtio-pci.o
  CC    hw/virtio/virtio-bus.o
  CC    hw/virtio/virtio-mmio.o
  CC    hw/watchdog/watchdog.o
  CC    hw/watchdog/wdt_i6300esb.o
  CC    hw/watchdog/wdt_ib700.o
  CC    migration/migration.o
  CC    migration/socket.o
  CC    migration/fd.o
  CC    migration/exec.o
  CC    migration/tls.o
  CC    migration/vmstate.o
  CC    migration/qemu-file.o
  CC    migration/qemu-file-channel.o
  CC    migration/xbzrle.o
  CC    migration/postcopy-ram.o
  CC    migration/qjson.o
  CC    migration/block.o
  CC    net/net.o
  CC    net/queue.o
  CC    net/checksum.o
  CC    net/util.o
  CC    net/hub.o
  CC    net/socket.o
  CC    net/dump.o
  CC    net/eth.o
  CC    net/l2tpv3.o
  CC    net/tap.o
  CC    net/vhost-user.o
  CC    net/tap-linux.o
  CC    net/slirp.o
  CC    net/filter.o
  CC    net/filter-buffer.o
  CC    net/filter-mirror.o
  CC    net/colo-compare.o
  CC    net/colo.o
  CC    net/filter-rewriter.o
  CC    qom/cpu.o
  CC    replay/replay.o
  CC    replay/replay-internal.o
  CC    replay/replay-events.o
  CC    replay/replay-time.o
  CC    replay/replay-input.o
  CC    replay/replay-char.o
  CC    replay/replay-snapshot.o
/tmp/qemu-test/src/replay/replay-internal.c: In function ‘replay_put_array’:
/tmp/qemu-test/src/replay/replay-internal.c:65: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result
  CC    slirp/cksum.o
  CC    slirp/if.o
  CC    slirp/ip_icmp.o
  CC    slirp/ip6_icmp.o
  CC    slirp/ip6_input.o
  CC    slirp/ip6_output.o
  CC    slirp/ip_input.o
  CC    slirp/ip_output.o
  CC    slirp/dnssearch.o
  CC    slirp/dhcpv6.o
  CC    slirp/slirp.o
  CC    slirp/mbuf.o
  CC    slirp/misc.o
  CC    slirp/sbuf.o
  CC    slirp/socket.o
  CC    slirp/tcp_input.o
  CC    slirp/tcp_output.o
  CC    slirp/tcp_subr.o
  CC    slirp/tcp_timer.o
  CC    slirp/udp.o
  CC    slirp/udp6.o
  CC    slirp/bootp.o
  CC    slirp/tftp.o
  CC    slirp/arp_table.o

/tmp/qemu-test/src/slirp/tcp_input.c: In function ‘tcp_input’:
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_p’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_len’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_tos’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_id’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_off’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_ttl’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_sum’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_src.s_addr’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_dst.s_addr’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:220: warning: ‘save_ip6.ip_nh’ may be used uninitialized in this function

  CC    slirp/ndp_table.o
  CC    ui/keymaps.o
  CC    ui/console.o
  CC    ui/cursor.o
  CC    ui/qemu-pixman.o
  CC    ui/input.o
  CC    ui/input-keymap.o
  CC    ui/input-legacy.o
  CC    ui/input-linux.o
  CC    ui/sdl.o
  CC    ui/sdl_zoom.o
  CC    ui/x_keymap.o
  CC    ui/vnc.o
  CC    ui/vnc-enc-zlib.o
  CC    ui/vnc-enc-hextile.o
  CC    ui/vnc-enc-tight.o
  CC    ui/vnc-palette.o
  CC    ui/vnc-enc-zrle.o
  CC    ui/vnc-auth-vencrypt.o
  CC    ui/vnc-ws.o
  CC    ui/vnc-jobs.o
  AS    optionrom/multiboot.o
  AS    optionrom/linuxboot.o
  CC    optionrom/linuxboot_dma.o

cc: unrecognized option '-no-integrated-as'
cc: unrecognized option '-no-integrated-as'
AS optionrom/kvmvapic.o
Building optionrom/multiboot.img
Building optionrom/linuxboot.img
Building optionrom/linuxboot_dma.img
Building optionrom/kvmvapic.img
Building optionrom/multiboot.raw
Building optionrom/linuxboot.raw
LINK tests/qemu-iotests/socket_scm_helper
Building optionrom/linuxboot_dma.raw
Building optionrom/kvmvapic.raw
Signing optionrom/multiboot.bin
Signing optionrom/linuxboot.bin
Signing optionrom/linuxboot_dma.bin
CC qga/commands.o
Signing optionrom/kvmvapic.bin

  CC    qga/guest-agent-command-state.o
  CC    qga/main.o
  CC    qga/commands-posix.o
  CC    qga/channel-posix.o
  CC    qga/qapi-generated/qga-qapi-types.o
  CC    qga/qapi-generated/qga-qapi-visit.o
  CC    qga/qapi-generated/qga-qmp-marshal.o
  CC    qmp-introspect.o
  CC    qapi-types.o
  CC    qapi-visit.o
  CC    qapi-event.o
  AR    libqemustub.a
  CC    qemu-img.o
  CC    qmp-marshal.o
  CC    trace/generated-events.o
  AR    libqemuutil.a
  LINK  qemu-ga
  LINK  ivshmem-client
  LINK  ivshmem-server
  LINK  qemu-nbd
  LINK  qemu-img
  LINK  qemu-io
  LINK  qemu-bridge-helper

libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':
/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make: *** [qemu-ga] Error 1
make: *** Waiting for unfinished jobs....
libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':
/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make: *** [qemu-nbd] Error 1
libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':
/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make: *** [qemu-img] Error 1
libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':
/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make: *** [qemu-io] Error 1

  GEN   x86_64-softmmu/hmp-commands.h
  GEN   x86_64-softmmu/hmp-commands-info.h
  GEN   x86_64-softmmu/config-target.h
  GEN   aarch64-softmmu/hmp-commands.h
  GEN   aarch64-softmmu/hmp-commands-info.h
  GEN   aarch64-softmmu/config-target.h
  CC    x86_64-softmmu/exec.o
  CC    x86_64-softmmu/translate-all.o
  CC    x86_64-softmmu/cpu-exec.o
  CC    x86_64-softmmu/translate-common.o
  CC    x86_64-softmmu/cpu-exec-common.o
  CC    x86_64-softmmu/tcg/tcg.o
  CC    x86_64-softmmu/tcg/tcg-op.o
  CC    aarch64-softmmu/exec.o
  CC    x86_64-softmmu/tcg/optimize.o
  CC    x86_64-softmmu/tcg/tcg-common.o
  CC    x86_64-softmmu/fpu/softfloat.o
  CC    x86_64-softmmu/disas.o
  CC    x86_64-softmmu/arch_init.o
  CC    x86_64-softmmu/cpus.o
  CC    x86_64-softmmu/monitor.o
  CC    aarch64-softmmu/translate-all.o
  CC    aarch64-softmmu/cpu-exec.o
  CC    x86_64-softmmu/gdbstub.o
  CC    aarch64-softmmu/translate-common.o
  CC    x86_64-softmmu/balloon.o
  CC    x86_64-softmmu/ioport.o
  CC    x86_64-softmmu/numa.o
  CC    aarch64-softmmu/cpu-exec-common.o
  CC    aarch64-softmmu/tcg/tcg.o
  CC    aarch64-softmmu/tcg/tcg-op.o
  CC    aarch64-softmmu/tcg/optimize.o
  CC    x86_64-softmmu/qtest.o
  CC    aarch64-softmmu/tcg/tcg-common.o
  CC    x86_64-softmmu/bootdevice.o
  CC    aarch64-softmmu/fpu/softfloat.o
  CC    x86_64-softmmu/kvm-all.o
  CC    aarch64-softmmu/disas.o
  CC    x86_64-softmmu/memory.o
  CC    x86_64-softmmu/cputlb.o
  CC    x86_64-softmmu/memory_mapping.o
  GEN   aarch64-softmmu/gdbstub-xml.c
  CC    aarch64-softmmu/kvm-stub.o
  CC    x86_64-softmmu/dump.o
  CC    x86_64-softmmu/migration/ram.o
  CC    x86_64-softmmu/migration/savevm.o
  CC    aarch64-softmmu/arch_init.o
  CC    aarch64-softmmu/cpus.o
  CC    aarch64-softmmu/monitor.o
  CC    aarch64-softmmu/gdbstub.o
  CC    x86_64-softmmu/xen-common-stub.o
  CC    aarch64-softmmu/balloon.o
  CC    aarch64-softmmu/ioport.o
  CC    x86_64-softmmu/xen-hvm-stub.o
  CC    aarch64-softmmu/numa.o
  CC    x86_64-softmmu/hw/acpi/nvdimm.o
  CC    x86_64-softmmu/hw/block/virtio-blk.o
  CC    aarch64-softmmu/qtest.o
  CC    aarch64-softmmu/bootdevice.o
  CC    x86_64-softmmu/hw/block/dataplane/virtio-blk.o
  CC    x86_64-softmmu/hw/char/virtio-serial-bus.o
  CC    aarch64-softmmu/memory.o
  CC    aarch64-softmmu/cputlb.o
  CC    aarch64-softmmu/memory_mapping.o
  CC    aarch64-softmmu/dump.o
  CC    aarch64-softmmu/migration/ram.o
  CC    aarch64-softmmu/migration/savevm.o
  CC    x86_64-softmmu/hw/core/nmi.o
  CC    aarch64-softmmu/xen-common-stub.o
  CC    x86_64-softmmu/hw/cpu/core.o
  CC    x86_64-softmmu/hw/display/vga.o
  CC    aarch64-softmmu/xen-hvm-stub.o
  CC    aarch64-softmmu/hw/block/virtio-blk.o
  CC    x86_64-softmmu/hw/display/virtio-gpu.o
  CC    x86_64-softmmu/hw/display/virtio-gpu-3d.o
  CC    aarch64-softmmu/hw/block/dataplane/virtio-blk.o
  CC    aarch64-softmmu/hw/char/exynos4210_uart.o
  CC    aarch64-softmmu/hw/char/omap_uart.o
  CC    x86_64-softmmu/hw/display/virtio-gpu-pci.o
  CC    aarch64-softmmu/hw/char/digic-uart.o
  CC    x86_64-softmmu/hw/display/virtio-vga.o
  CC    aarch64-softmmu/hw/char/stm32f2xx_usart.o
  CC    aarch64-softmmu/hw/char/bcm2835_aux.o
  CC    aarch64-softmmu/hw/char/virtio-serial-bus.o
  CC    aarch64-softmmu/hw/core/nmi.o
  CC    x86_64-softmmu/hw/intc/apic.o
  CC    x86_64-softmmu/hw/intc/apic_common.o
  CC    x86_64-softmmu/hw/intc/ioapic.o
  CC    x86_64-softmmu/hw/isa/lpc_ich9.o
  CC    x86_64-softmmu/hw/misc/vmport.o
  CC    aarch64-softmmu/hw/cpu/arm11mpcore.o
  CC    x86_64-softmmu/hw/misc/ivshmem.o
  CC    x86_64-softmmu/hw/misc/pvpanic.o
  CC    aarch64-softmmu/hw/cpu/realview_mpcore.o
  CC    aarch64-softmmu/hw/cpu/a9mpcore.o
  CC    aarch64-softmmu/hw/cpu/a15mpcore.o
  CC    aarch64-softmmu/hw/cpu/core.o
  CC    x86_64-softmmu/hw/misc/edu.o
  CC    aarch64-softmmu/hw/display/omap_dss.o
  CC    x86_64-softmmu/hw/misc/hyperv_testdev.o
  CC    x86_64-softmmu/hw/net/virtio-net.o
  CC    aarch64-softmmu/hw/display/omap_lcdc.o
  CC    x86_64-softmmu/hw/net/vhost_net.o
  CC    aarch64-softmmu/hw/display/pxa2xx_lcd.o
  CC    aarch64-softmmu/hw/display/bcm2835_fb.o
  CC    x86_64-softmmu/hw/scsi/virtio-scsi.o
  CC    x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC    aarch64-softmmu/hw/display/vga.o
  CC    x86_64-softmmu/hw/scsi/vhost-scsi.o
  CC    aarch64-softmmu/hw/display/virtio-gpu.o
  CC    aarch64-softmmu/hw/display/virtio-gpu-3d.o
  CC    aarch64-softmmu/hw/display/virtio-gpu-pci.o
  CC    x86_64-softmmu/hw/timer/mc146818rtc.o
  CC    aarch64-softmmu/hw/display/dpcd.o
  CC    x86_64-softmmu/hw/vfio/common.o
  CC    x86_64-softmmu/hw/vfio/pci.o
  CC    aarch64-softmmu/hw/display/xlnx_dp.o
  CC    aarch64-softmmu/hw/dma/xlnx_dpdma.o
  CC    aarch64-softmmu/hw/dma/omap_dma.o
  CC    aarch64-softmmu/hw/dma/soc_dma.o
  CC    aarch64-softmmu/hw/dma/pxa2xx_dma.o
  CC    aarch64-softmmu/hw/dma/bcm2835_dma.o
  CC    aarch64-softmmu/hw/gpio/omap_gpio.o
  CC    x86_64-softmmu/hw/vfio/pci-quirks.o
  CC    x86_64-softmmu/hw/vfio/platform.o
  CC    aarch64-softmmu/hw/gpio/imx_gpio.o
  CC    x86_64-softmmu/hw/vfio/calxeda-xgmac.o
  CC    aarch64-softmmu/hw/i2c/omap_i2c.o
  CC    x86_64-softmmu/hw/vfio/amd-xgbe.o
  CC    aarch64-softmmu/hw/input/pxa2xx_keypad.o
  CC    aarch64-softmmu/hw/input/tsc210x.o
  CC    x86_64-softmmu/hw/vfio/spapr.o
  CC    x86_64-softmmu/hw/virtio/virtio.o
  CC    aarch64-softmmu/hw/intc/armv7m_nvic.o
  CC    aarch64-softmmu/hw/intc/exynos4210_gic.o
  CC    x86_64-softmmu/hw/virtio/virtio-balloon.o
  CC    x86_64-softmmu/hw/virtio/vhost.o
  CC    x86_64-softmmu/hw/virtio/vhost-backend.o
  CC    x86_64-softmmu/hw/virtio/vhost-user.o
  CC    x86_64-softmmu/hw/virtio/vhost-vsock.o
  CC    aarch64-softmmu/hw/intc/exynos4210_combiner.o
  CC    aarch64-softmmu/hw/intc/omap_intc.o
  CC    x86_64-softmmu/hw/i386/multiboot.o
  CC    x86_64-softmmu/hw/i386/pc.o
  CC    x86_64-softmmu/hw/i386/pc_piix.o
  CC    x86_64-softmmu/hw/i386/pc_q35.o
  CC    x86_64-softmmu/hw/i386/pc_sysfw.o
  CC    x86_64-softmmu/hw/i386/x86-iommu.o
  CC    aarch64-softmmu/hw/intc/bcm2835_ic.o
  CC    aarch64-softmmu/hw/intc/bcm2836_control.o
  CC    aarch64-softmmu/hw/intc/allwinner-a10-pic.o
  CC    aarch64-softmmu/hw/intc/aspeed_vic.o
  CC    x86_64-softmmu/hw/i386/intel_iommu.o
  CC    x86_64-softmmu/hw/i386/amd_iommu.o
  CC    aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC    x86_64-softmmu/hw/i386/kvmvapic.o
  CC    x86_64-softmmu/hw/i386/acpi-build.o
/tmp/qemu-test/src/hw/i386/pc_piix.c: In function ‘igd_passthrough_isa_bridge_create’:
/tmp/qemu-test/src/hw/i386/pc_piix.c:1046: warning: ‘pch_rev_id’ may be used uninitialized in this function
  CC    x86_64-softmmu/hw/i386/pci-assign-load-rom.o
  CC    x86_64-softmmu/hw/i386/kvm/clock.o
  CC    aarch64-softmmu/hw/misc/ivshmem.o
  CC    aarch64-softmmu/hw/misc/arm_sysctl.o
  CC    x86_64-softmmu/hw/i386/kvm/apic.o
  CC    x86_64-softmmu/hw/i386/kvm/i8259.o
  CC    x86_64-softmmu/hw/i386/kvm/ioapic.o
  CC    aarch64-softmmu/hw/misc/cbus.o
  CC    aarch64-softmmu/hw/misc/exynos4210_pmu.o
  CC    aarch64-softmmu/hw/misc/imx_ccm.o
/tmp/qemu-test/src/hw/i386/acpi-build.c: In function ‘build_append_pci_bus_devices’:
/tmp/qemu-test/src/hw/i386/acpi-build.c:472: warning: ‘notify_method’ may be used uninitialized in this function
  CC    aarch64-softmmu/hw/misc/imx31_ccm.o
  CC    aarch64-softmmu/hw/misc/imx25_ccm.o
  CC    aarch64-softmmu/hw/misc/imx6_ccm.o
  CC    x86_64-softmmu/hw/i386/kvm/i8254.o
  CC    x86_64-softmmu/hw/i386/kvm/pci-assign.o
  CC    aarch64-softmmu/hw/misc/imx6_src.o
  CC    x86_64-softmmu/target-i386/translate.o
  CC    aarch64-softmmu/hw/misc/mst_fpga.o
  CC    aarch64-softmmu/hw/misc/omap_clk.o
  CC    aarch64-softmmu/hw/misc/omap_gpmc.o
  CC    aarch64-softmmu/hw/misc/omap_l4.o
  CC    x86_64-softmmu/target-i386/helper.o
  CC    aarch64-softmmu/hw/misc/omap_sdrc.o
  CC    aarch64-softmmu/hw/misc/omap_tap.o
  CC    aarch64-softmmu/hw/misc/bcm2835_mbox.o
  CC    aarch64-softmmu/hw/misc/bcm2835_property.o
  CC    x86_64-softmmu/target-i386/cpu.o
  CC    x86_64-softmmu/target-i386/bpt_helper.o
  CC    x86_64-softmmu/target-i386/excp_helper.o
  CC    aarch64-softmmu/hw/misc/zynq_slcr.o
  CC    aarch64-softmmu/hw/misc/zynq-xadc.o
  CC    aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC    aarch64-softmmu/hw/misc/edu.o
  CC    x86_64-softmmu/target-i386/fpu_helper.o
  CC    x86_64-softmmu/target-i386/cc_helper.o
  CC    aarch64-softmmu/hw/misc/auxbus.o
  CC    aarch64-softmmu/hw/misc/aspeed_scu.o
  CC    x86_64-softmmu/target-i386/int_helper.o
  CC    x86_64-softmmu/target-i386/svm_helper.o
  CC    aarch64-softmmu/hw/misc/aspeed_sdmc.o
  CC    aarch64-softmmu/hw/net/virtio-net.o
  CC    aarch64-softmmu/hw/net/vhost_net.o
  CC    x86_64-softmmu/target-i386/smm_helper.o
  CC    aarch64-softmmu/hw/pcmcia/pxa2xx.o
  CC    x86_64-softmmu/target-i386/misc_helper.o
  CC    aarch64-softmmu/hw/scsi/virtio-scsi.o
  CC    aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC    x86_64-softmmu/target-i386/mem_helper.o
  CC    x86_64-softmmu/target-i386/seg_helper.o
  CC    aarch64-softmmu/hw/scsi/vhost-scsi.o
  CC    aarch64-softmmu/hw/sd/omap_mmc.o
  CC    x86_64-softmmu/target-i386/mpx_helper.o
  CC    x86_64-softmmu/target-i386/gdbstub.o
  CC    aarch64-softmmu/hw/sd/pxa2xx_mmci.o
  CC    aarch64-softmmu/hw/ssi/omap_spi.o
  CC    aarch64-softmmu/hw/ssi/imx_spi.o
  CC    x86_64-softmmu/target-i386/machine.o
  CC    aarch64-softmmu/hw/timer/exynos4210_mct.o
  CC    x86_64-softmmu/target-i386/arch_memory_mapping.o
  CC    x86_64-softmmu/target-i386/arch_dump.o
  CC    x86_64-softmmu/target-i386/monitor.o
  CC    x86_64-softmmu/target-i386/kvm.o
  CC    x86_64-softmmu/target-i386/hyperv.o
  CC    aarch64-softmmu/hw/timer/exynos4210_pwm.o
  GEN   trace/generated-helpers.c
  CC    x86_64-softmmu/trace/control-target.o
  CC    aarch64-softmmu/hw/timer/exynos4210_rtc.o
  CC    aarch64-softmmu/hw/timer/omap_gptimer.o
  CC    x86_64-softmmu/trace/generated-helpers.o
  CC    aarch64-softmmu/hw/timer/omap_synctimer.o
  CC    aarch64-softmmu/hw/timer/pxa2xx_timer.o
  CC    aarch64-softmmu/hw/timer/digic-timer.o
  CC    aarch64-softmmu/hw/timer/allwinner-a10-pit.o
  CC    aarch64-softmmu/hw/usb/tusb6010.o
  CC    aarch64-softmmu/hw/vfio/common.o
  CC    aarch64-softmmu/hw/vfio/pci.o
  CC    aarch64-softmmu/hw/vfio/pci-quirks.o
  CC    aarch64-softmmu/hw/vfio/platform.o
  CC    aarch64-softmmu/hw/vfio/calxeda-xgmac.o
  CC    aarch64-softmmu/hw/vfio/amd-xgbe.o
  CC    aarch64-softmmu/hw/vfio/spapr.o
  LINK  x86_64-softmmu/qemu-system-x86_64
  CC    aarch64-softmmu/hw/virtio/virtio.o
  CC    aarch64-softmmu/hw/virtio/virtio-balloon.o
  CC    aarch64-softmmu/hw/virtio/vhost.o
  CC    aarch64-softmmu/hw/virtio/vhost-backend.o
  CC    aarch64-softmmu/hw/virtio/vhost-user.o
  CC    aarch64-softmmu/hw/virtio/vhost-vsock.o
  CC    aarch64-softmmu/hw/arm/boot.o
  CC    aarch64-softmmu/hw/arm/collie.o
  CC    aarch64-softmmu/hw/arm/exynos4_boards.o
  CC    aarch64-softmmu/hw/arm/gumstix.o

../libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':
/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make[1]: *** [qemu-system-x86_64] Error 1
make: *** [subdir-x86_64-softmmu] Error 2

  CC    aarch64-softmmu/hw/arm/highbank.o
  CC    aarch64-softmmu/hw/arm/digic_boards.o
  CC    aarch64-softmmu/hw/arm/integratorcp.o
  CC    aarch64-softmmu/hw/arm/mainstone.o
  CC    aarch64-softmmu/hw/arm/musicpal.o
  CC    aarch64-softmmu/hw/arm/nseries.o
  CC    aarch64-softmmu/hw/arm/omap_sx1.o
  CC    aarch64-softmmu/hw/arm/palm.o
  CC    aarch64-softmmu/hw/arm/realview.o
  CC    aarch64-softmmu/hw/arm/spitz.o
  CC    aarch64-softmmu/hw/arm/stellaris.o
  CC    aarch64-softmmu/hw/arm/tosa.o
  CC    aarch64-softmmu/hw/arm/versatilepb.o
  CC    aarch64-softmmu/hw/arm/vexpress.o
  CC    aarch64-softmmu/hw/arm/virt.o
  CC    aarch64-softmmu/hw/arm/xilinx_zynq.o
  CC    aarch64-softmmu/hw/arm/z2.o
  CC    aarch64-softmmu/hw/arm/virt-acpi-build.o
  CC    aarch64-softmmu/hw/arm/netduino2.o
  CC    aarch64-softmmu/hw/arm/sysbus-fdt.o
  CC    aarch64-softmmu/hw/arm/armv7m.o
  CC    aarch64-softmmu/hw/arm/exynos4210.o
  CC    aarch64-softmmu/hw/arm/pxa2xx.o
  CC    aarch64-softmmu/hw/arm/pxa2xx_gpio.o
  CC    aarch64-softmmu/hw/arm/pxa2xx_pic.o
  CC    aarch64-softmmu/hw/arm/digic.o
  CC    aarch64-softmmu/hw/arm/omap1.o
  CC    aarch64-softmmu/hw/arm/omap2.o
  CC    aarch64-softmmu/hw/arm/strongarm.o
  CC    aarch64-softmmu/hw/arm/allwinner-a10.o
  CC    aarch64-softmmu/hw/arm/cubieboard.o
  CC    aarch64-softmmu/hw/arm/bcm2835_peripherals.o
  CC    aarch64-softmmu/hw/arm/bcm2836.o
  CC    aarch64-softmmu/hw/arm/raspi.o
  CC    aarch64-softmmu/hw/arm/stm32f205_soc.o
  CC    aarch64-softmmu/hw/arm/xlnx-zynqmp.o
  CC    aarch64-softmmu/hw/arm/xlnx-ep108.o
  CC    aarch64-softmmu/hw/arm/fsl-imx25.o
  CC    aarch64-softmmu/hw/arm/imx25_pdk.o
  CC    aarch64-softmmu/hw/arm/fsl-imx31.o
  CC    aarch64-softmmu/hw/arm/kzm.o
  CC    aarch64-softmmu/hw/arm/fsl-imx6.o
  CC    aarch64-softmmu/hw/arm/sabrelite.o
  CC    aarch64-softmmu/hw/arm/aspeed_soc.o
  CC    aarch64-softmmu/hw/arm/aspeed.o
  CC    aarch64-softmmu/target-arm/arm-semi.o
  CC    aarch64-softmmu/target-arm/machine.o
  CC    aarch64-softmmu/target-arm/psci.o
  CC    aarch64-softmmu/target-arm/arch_dump.o
  CC    aarch64-softmmu/target-arm/monitor.o
  CC    aarch64-softmmu/target-arm/kvm-stub.o
  CC    aarch64-softmmu/target-arm/translate.o
  CC    aarch64-softmmu/target-arm/op_helper.o
  CC    aarch64-softmmu/target-arm/helper.o
  CC    aarch64-softmmu/target-arm/cpu.o
  CC    aarch64-softmmu/target-arm/neon_helper.o
  CC    aarch64-softmmu/target-arm/iwmmxt_helper.o
  CC    aarch64-softmmu/target-arm/gdbstub.o
  CC    aarch64-softmmu/target-arm/cpu64.o
  CC    aarch64-softmmu/target-arm/translate-a64.o
  CC    aarch64-softmmu/target-arm/helper-a64.o
  CC    aarch64-softmmu/target-arm/gdbstub64.o
  CC    aarch64-softmmu/target-arm/crypto_helper.o
  CC    aarch64-softmmu/target-arm/arm-powerctl.o
  GEN   trace/generated-helpers.c
  CC    aarch64-softmmu/trace/control-target.o
  CC    aarch64-softmmu/gdbstub-xml.o
  CC    aarch64-softmmu/trace/generated-helpers.o

/tmp/qemu-test/src/target-arm/translate-a64.c: In function ‘handle_shri_with_rndacc’:
/tmp/qemu-test/src/target-arm/translate-a64.c:6333: warning: ‘tcg_src_hi’ may be used uninitialized in this function /tmp/qemu-test/src/target-arm/translate-a64.c: In function ‘disas_simd_scalar_two_reg_misc’:
/tmp/qemu-test/src/target-arm/translate-a64.c:8060: warning: ‘rmode’ may be used uninitialized in this function
LINK aarch64-softmmu/qemu-system-aarch64
../libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':
/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make[1]: *** [qemu-system-aarch64] Error 1
make: *** [subdir-aarch64-softmmu] Error 2
tests/docker/Makefile.include:107: recipe for target 'docker-run-test-quick@centos6' failed
make: *** [docker-run-test-quick@centos6] Error 2
=== OUTPUT END ===

Test command exited with code: 2

---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to [email protected]

Reviewed-by on v14 3/21
#17 lore raw

Re: [Qemu-devel] [PATCH v14 03/21] qapi: add trace events for visitor

In reply to Daniel P. Berrange
Quoted 9 lines · Daniel P. Berrange

Allow tracing of the operation of visitors

---
 Makefile.objs          |  1 +
 qapi/qapi-visit-core.c | 27 +++++++++++++++++++++++++++
 qapi/trace-events      | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+)
 create mode 100644 qapi/trace-events

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+visit_start_alternate(void *v, const char *name, void *obj, size_t size, bool promote_int) "v=%p name=%s obj=%p size=%zu promote_int=%d"

Unrelated to your patch, but should the trace engines be taught how to honor line wrapping in the trace files?

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
Reviewed-by on v14 6/21
#27 lore raw

Re: [Qemu-devel] [PATCH v14 06/21] qapi: don't pass two copies of TestInputVisitorData to tests

In reply to Daniel P. Berrange
Quoted 6 lines · Daniel P. Berrange

The input_visitor_test_add() method was accepting an instance of 'TestInputVisitorData' and passing it as the 'user_data' parameter to test functions. The main 'TestInputVisitorData' instance that was actually used, was meanwhile being allocated automatically by the test framework fixture setup.

Quoted 8 lines

The 'user_data' parameter is going to be needed for tests added in later patches, so getting rid of the current mistaken usage now allows this.

---
 tests/test-qobject-input-visitor.c | 76 ++++++++++++++++----------------------
 1 file changed, 32 insertions(+), 44 deletions(-)

The improved commit message makes the difference :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
Reviewed-by on v14 7/21
#30 lore raw

Re: [Qemu-devel] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor

In reply to Daniel P. Berrange
Quoted 14 lines · Daniel P. Berrange

Currently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.

To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.

Quoted 9 lines
---
 include/qapi/qobject-input-visitor.h |  32 +++++-
 qapi/qobject-input-visitor.c         | 132 ++++++++++++++++++++++++
 tests/test-qobject-input-visitor.c   | 194 ++++++++++++++++++++++++++++++++++-
 3 files changed, 350 insertions(+), 8 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
Reviewed-by on v14 8/21
#36 lore raw

Re: [Qemu-devel] [PATCH v14 08/21] qapi: allow QObjectInputVisitor to be created with QemuOpts

In reply to Daniel P. Berrange
Quoted 10 lines · Daniel P. Berrange

Instead of requiring all callers to go through the mutli-step process of turning QemuOpts into a suitable QObject for visiting, add a new constructor that encapsulates this logic. This will allow QObjectInputVisitor to be a drop-in replacement for the existing OptsVisitor with minimal code changes for callers.

NB, at this point it is only supporting opts syntax which explicitly matches the QAPI schema structure, so is not yet a true drop-in replacement for OptsVisitor. The patches that follow will add the special cases requird for full backwards

s/requird/required/

include/qemu/option.h void qemu_opts_set_defaults(QemuOptsList *list, const char *params, · line 129

Quoted 13 lines

compatibility with OptsVisitor.

---

+++ b/include/qemu/option.h

include/qemu/option.h +1 -1
@@ -125,7 +125,7 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char *params,
                             int permit_abbrev);
 QemuOpts *qemu_opts_from_qdict(QemuOptsList *list, const QDict *qdict,
                                Error **errp);
-QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict);
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict);
 void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp);

Should the const-correctness of this parameter be hoisted to any earlier patch? But here is fine, as the first place where the compiler requires it.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
Reviewed-by on v14 9/21
#41 lore raw

Re: [Qemu-devel] [PATCH v14 09/21] qapi: permit auto-creating single element lists

In reply to Daniel P. Berrange
Quoted 13 lines · Daniel P. Berrange

When converting QemuOpts to a QObject, there is no information about compound types available, so when visiting a list, the corresponding QObject is not guaranteed to be a QList. We therefore need to be able to auto-create a single element QList from whatever type we find.

This mode should only be enabled if you have compatibility requirements for

-arg foo=hello,foo=world

to be treated as equivalent to the preferred syntax:

Quoted 9 lines
   -arg foo.0=hello,foo.1=world
---
 include/qapi/qobject-input-visitor.h | 20 +++++++-
 qapi/qobject-input-visitor.c         | 27 +++++++++--
 tests/test-qobject-input-visitor.c   | 88 +++++++++++++++++++++++++++++++-----
 3 files changed, 117 insertions(+), 18 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#46 lore raw

Re: [Qemu-devel] [PATCH v14 10/21] qapi: permit auto-creating nested structs

In reply to Daniel P. Berrange
Quoted 3 lines · Daniel P. Berrange

Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting

singular/plural mismatch; either s/a // or s/structs/struct/

Quoted 17 lines

the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the qdict_crumple() call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.

Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.

The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach

Quoted 6 lines
  -net vlan=1,id=foo,name=bar,opts.type=tap,\
       opts.data.fd=3,opts.data.script=ifup

The legacy syntax will just be presenting

-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup

So basically, any key=values that were not consumed as known keys in the parent struct are reparsed as the (presumably lone) remaining QDict child, recursing as needed to follow the QAPI nesting.

Does this still work if the command line is presented in a different order:

-net script=ifup,vlan=1,type=tap,fd=3,id=foo,name=bar

My guess is yes, but if the testsuite doesn't cover it, you may want to add a test.

Quoted 19 lines

So we need to auto-create 3 levels of struct when visiting.

The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.

---
 include/qapi/qobject-input-visitor.h |  22 +++++-
 qapi/qobject-input-visitor.c         |  59 ++++++++++++++--
 tests/test-qobject-input-visitor.c   | 130 ++++++++++++++++++++++++++++++++---
 3 files changed, 194 insertions(+), 17 deletions(-)
include/qapi/qobject-input-visitor.h +0 -0
index 1809f48..94051f3 100644

include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict); · line 48

Quoted 5 lines
include/qapi/qobject-input-visitor.h +1 -1
@@ -45,7 +45,7 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * If @autocreate_list is true, then as an alternative to a normal QList,
  * list values can be stored as a QString or QDict instead, which will
  * be interpreted as representing single element lists. This should only
- * by used if compatibility is required with the OptsVisitor which allowed
+ * be used if compatibility is required with the OptsVisitor which allowed

This typo fix should be hoisted earlier in the series when it was introduced.

Quoted 7 lines
include/qapi/qobject-input-visitor.h +0 -0
  * repeated keys, without list indexes, to represent lists. e.g. set this
  * to true if you have compatibility requirements for
  *
@@ -55,6 +55,22 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  *
  *   -arg foo.0=hello,foo.1=world
  *

include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict); · line 67

Quoted 10 lines
include/qapi/qobject-input-visitor.h +10 -0
+ * If @autocreate_struct_levels is non-zero, then when visiting structs,
+ * if the corresponding QDict is not found, it will automatically create
+ * a QDict containing all remaining unvisited options. This should only
+ * be used if compatibility is required with the OptsVisitor which flatten
+ * structs so that all keys were at the same level. e.g. set this to a
+ * non-zero number if you compatibility requirements for
+ *
+ *   -arg type=person,surname=blogs,forename=fred
+ *
+ * to be treated as equivalent to the perferred syntax

s/perferred/preferred/

Quoted 5 lines
include/qapi/qobject-input-visitor.h +5 -0
+static void test_visitor_in_struct_autocreate(TestInputVisitorData *data,
+                                              const void *unused)
+{
+    Visitor *v;
+    int64_t vlan;

include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict);

Quoted 10 lines
include/qapi/qobject-input-visitor.h +10 -0
+    char *id = NULL;
+    char *type;
+    int64_t fd;
+    char *script = NULL;
+
+    v = visitor_input_test_init_internal(
+        data, true, true, false, 3,
+        "{ 'vlan': '1', 'id': 'foo', 'type': 'tap', 'fd': '3', "
+        "'script': 'ifup' }", NULL);
+

This is where intentionally mixing up the parameter orders might be worthwhile.

Overall, the idea looks reasonable.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#107 lore raw
In reply to [email protected]
Quoted 7 lines · On 09/30/2016 10:45 AM, [email protected] wrote:

Hi,

Your series failed automatic build test. Please find the testing commands and their output below. If you have docker installed, you can probably reproduce it locally.

Quoted 8 lines

CC qapi/string-output-visitor.o
/tmp/qemu-test/src/qapi/qobject-input-visitor.c: In function ‘qobject_input_get_object’:
/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: warning: implicit declaration of function ‘g_hash_table_contains’ /tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: warning: nested extern declaration of ‘g_hash_table_contains’

tests/docker/Makefile.include:107: recipe for target 'docker-run-test-quick@centos6' failed
make: *** [docker-run-test-quick@centos6] Error 2

Cool - the autobuilder is now detecting use of too-new glib functions :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#57 lore raw

Re: [Qemu-devel] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor

In reply to Daniel P. Berrange
Quoted 25 lines · Daniel P. Berrange

The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:

-arg foo=5,foo=6,foo=7
-arg foo=5-7

This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.

This means that

-arg foo=5-7

is treated as equivalent to

-arg foo.0=5,foo.1=6,foo.2=7

Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.

---

include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict); · line 81

Quoted 5 lines
include/qapi/qobject-input-visitor.h +2 -0
@@ -71,6 +77,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * The value given determines how many levels of structs are allowed to
  * be flattened in this way.
  *
+ * If @permit_int_ranges is true, then when visiting a list of integers,
+ * the integer value strings may encode ranges eg a single element

My usual complaint that 'e.g.' is usually spelled with dots :)

include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict); · line 84

Quoted 3 lines
include/qapi/qobject-input-visitor.h +3 -0
+ * containing "5-7" is treated as if there were three elements "5", "6",
+ * "7". This should only be used if compatibility is required with the
+ * OptsVisitor which would allow integer ranges. e.g. set this to true

Ah. Here you DID use dots, but used the wrong abbreviation. Substituting "For example" in this location does not make as much sense as "i.e." would. Or go for simplicity; it reads just fine as:

"...would allow integer ranges; so only set this to true if..."

Quoted 25 lines
(quoted diff) +24 -0
+ * if you have compatibility requirements for
+ *
+ *   -arg val=5-8
+ *
+ * to be treated as equivalent to the preferred syntax:
+ *
+ *   -arg val.0=5,val.1=6,val.2=7,val.3=8
+ *
 
+
+    /*
+     * If we have string that represents an integer range (5-24),
+     * parse the end of the range and set things up so we'll process
+     * the rest of the range before consuming another element
+     * from the QList.
+     */
+    if (end && *end) {
+        if (!qiv->permit_int_ranges) {
+            error_setg(errp,
+                       "Integer ranges are not permitted here");
+            return;
+        }
+        if (!inlist) {
+            error_setg(errp,
+                       "Integer ranges are only permitted when "

Review Visitor *qobject_input_visitor_new(QObject *obj, bool strict);

Quoted 10 lines
(quoted diff) +10 -0
+                       "visiting list parameters");
+            return;
+        }
+        if (*end != '-') {
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name,
+                       "a number range");
+            return;
+        }
+        end++;
+        if (qemu_strtoll(end, NULL, 0, &ret) < 0) {

Signed parse...

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 399

Quoted 4 lines
qapi/qobject-input-visitor.c +4 -0
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
+            return;
+        }
+        if (*obj > ret) {

...and signed comparison...

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 402

Quoted 3 lines
qapi/qobject-input-visitor.c +3 -0
+            error_setg(errp,
+                       "Parameter '%s' range start %" PRIu64
+                       " must be less than (or equal to) %" PRIu64,

...but unsigned numbers in the error message. Looks a bit odd (but not necessarily worse than what our OptsVisitor was already doing).

Quoted 7 lines
qapi/qobject-input-visitor.c +7 -0
+                       name, *obj, ret);
+            return;
+        }
+
+        if ((*obj <= (INT64_MAX - QIV_RANGE_MAX)) &&
+            (ret >= (*obj + QIV_RANGE_MAX))) {
+            error_setg(errp,

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 419

Quoted 10 lines
qapi/qobject-input-visitor.c +9 -0
+                       "Parameter '%s' range must be less than %d",
+                       name, QIV_RANGE_MAX);
+            return;
+        }
+        if (*obj != ret) {
+            tos->range_val = *obj;
+            tos->range_limit = ret;
+        }
+    }
 }

Overall, I found this a bit easier to read than the OptsVisitor state machine. Which makes me worried that we may have some subtle translation bugs; but I hope it's the same end result.

My consolation is that if there are any differences, they would be in the corner cases, where a user is probably going to get surprising behavior from the command line anyway, and shouldn't have been requesting ranges in that corner case.

qapi/qobject-input-visitor.c static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, · line 440

Quoted 6 lines

static void qobject_input_type_uint64(Visitor *v, const char *name,

qapi/qobject-input-visitor.c +0 -0
@@ -366,21 +438,85 @@ static void qobject_input_type_uint64_autocast(Visitor *v, const char *name,
                                                uint64_t *obj, Error **errp)
 {
     QObjectInputVisitor *qiv = to_qiv(v);

Feels like a lot of code duplication; but I don't know if there's any sane way to share more of the code while handling both signed and unsigned types.

The testsuite additions are reassuring; I confirmed that they match test-opts-visitor.c in coverage.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
Reviewed-by on v14 12/21
#64 lore raw

Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options

In reply to Daniel P. Berrange
Quoted 17 lines · Daniel P. Berrange

If given an option string such as

size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind

the qemu_opts_to_qdict() method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:

    size=QString("1024")
    nodes=QString("1-2")
    policy=QString("bind")

With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain

Quoted 5 lines
    size=QString("1024")
    nodes=QList([QString("10"),
                 QString("4-5"),
                 QString("1-2")])
    policy=QString("bind")

I think the last time around you were converting keys (turning "nodes" into "nodes.0"; I think your idea here of keeping a single key tied to QList makes more sense.

Quoted 20 lines

Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.

In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.

All existing callers are all converted to explicitly request the historical behaviour of only reporting the last key. Later patches will make use of the new modes.

---

include/qemu/option.h void qemu_opts_set_defaults(QemuOptsList *list, const char *params, · line 136

Quoted 10 lines
include/qemu/option.h +9 -1
-QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict);
+typedef enum {
+    /* Last occurrence of a duplicate option silently replaces earlier */
+    QEMU_OPTS_REPEAT_POLICY_LAST,
+    /* Each occurrence of a duplicate option converts value to a QList */
+    QEMU_OPTS_REPEAT_POLICY_ALL,
+    /* First occurrence of a duplicate option causes an error */
+    QEMU_OPTS_REPEAT_POLICY_ERROR,
+} QemuOptsRepeatPolicy;
+

Nicer. Thanks!

Quoted 36 lines

+++ b/tests/test-qemu-opts.c

tests/test-qemu-opts.c +30 -0
@@ -421,6 +421,130 @@ static void test_qemu_opts_set(void)
     g_assert(opts == NULL);
 }
 
 
+static void test_qemu_opts_to_qdict_repeat_all(void)
+{
+    QemuOpts *opts;
+    QDict *dict;
+    QList *list;
+    const QListEntry *ent;
+    QString *str;
+
+    /* dynamically initialized (parsed) opts */
+    opts = qemu_opts_parse(&opts_list_03,
+                           "size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind",
+                           false, NULL);
+    g_assert(opts);
+
+    dict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_ALL,
+                              &error_abort);
+    g_assert(dict);
+
+    g_assert_cmpstr(qdict_get_str(dict, "size"), ==, "1024");
+    g_assert(qdict_haskey(dict, "nodes"));
+    list = qobject_to_qlist(qdict_get(dict, "nodes"));
+    g_assert(list);
+
+    ent = qlist_first(list);
+    g_assert(ent);
+    str = qobject_to_qstring(ent->value);
+    g_assert(str);
+    g_assert_cmpstr(qstring_get_str(str), ==, "10");
+
+    ent = qlist_next(ent);

tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 464

Quoted 10 lines
tests/test-qemu-opts.c +10 -0
+    g_assert(ent);
+    str = qobject_to_qstring(ent->value);
+    g_assert(str);
+    g_assert_cmpstr(qstring_get_str(str), ==, "4-5");
+
+    ent = qlist_next(ent);
+    g_assert(ent);
+    str = qobject_to_qstring(ent->value);
+    g_assert(str);
+    g_assert_cmpstr(qstring_get_str(str), ==, "1-2");

Yes, this is different from v13, but does look nicer with the value turning into QList instead of the key being rewritten into dotted form.

Quoted 10 lines
util/qemu-option.c +6 -0
+++ b/util/qemu-option.c
@@ -1057,23 +1057,73 @@ void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp)
  * The QDict values are of type QString.
  * TODO We'll want to use types appropriate for opt->desc->type, but
  * this is enough for now.
+ *
+ * If the @opts contains multiple occurrences of the same key,
+ * then the @repeatPolicy parameter determines how they are to
+ * be handled. Traditional behaviour was to only store the
+ * last occurrence, but if @repeatPolicy is set to

util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1073

Quoted 10 lines
util/qemu-option.c +8 -1
+ * QEMU_OPTS_REPEAT_POLICY_ALL, then a QList will be returned
+ * containing all values, for any key with multiple occurrences.
+ * The QEMU_OPTS_REPEAT_POLICY_ERROR value can be used to fail
+ * conversion with an error if multiple occurrences of a key
+ * are seen.
  */
-QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict)
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict,
+                          QemuOptsRepeatPolicy repeatPolicy,
+                          Error **errp)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
Reviewed-by on v14 6/21
#28 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 06/21] qapi: don't pass two copies of TestInputVisitorData to tests

In reply to Daniel P. Berrange

Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:

Quoted 11 lines

The input_visitor_test_add() method was accepting an instance of 'TestInputVisitorData' and passing it as the 'user_data' parameter to test functions. The main 'TestInputVisitorData' instance that was actually used, was meanwhile being allocated automatically by the test framework fixture setup.

The 'user_data' parameter is going to be needed for tests added in later patches, so getting rid of the current mistaken usage now allows this.

Reviewed-by on v14 8/21
#37 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 08/21] qapi: allow QObjectInputVisitor to be created with QemuOpts

In reply to Daniel P. Berrange

Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:

Quoted 6 lines

Instead of requiring all callers to go through the mutli-step process of turning QemuOpts into a suitable QObject for visiting, add a new constructor that encapsulates this logic. This will allow QObjectInputVisitor to be a drop-in replacement for the existing OptsVisitor with minimal code changes for callers.

Quoted 7 lines

NB, at this point it is only supporting opts syntax which explicitly matches the QAPI schema structure, so is not yet a true drop-in replacement for OptsVisitor. The patches that follow will add the special cases requird for full backwards compatibility with OptsVisitor.

#47 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs

In reply to Daniel P. Berrange

Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:

Quoted 25 lines

Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the qdict_crumple() call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.

Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.

The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach

  -net vlan=1,id=foo,name=bar,opts.type=tap,\
       opts.data.fd=3,opts.data.script=ifup

The legacy syntax will just be presenting

Quoted 7 lines

-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup

So we need to auto-create 3 levels of struct when visiting.

The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.

So you're actually introducing the modern syntax only now?

I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.

So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.

Kevin

#48 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs

Quoted 28 lines · Kevin Wolf

Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:

Quoted 27 lines

Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the qdict_crumple() call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.

Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.

The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach

  -net vlan=1,id=foo,name=bar,opts.type=tap,\
       opts.data.fd=3,opts.data.script=ifup

The legacy syntax will just be presenting

-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup

Quoted 7 lines
Quoted 5 lines

So we need to auto-create 3 levels of struct when visiting.

The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.

So you're actually introducing the modern syntax only now?

No, the modern syntax is fully implemented by patch 8.

This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.

Quoted 7 lines

I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.

So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.

Changing that will impact on the QMP data structure, so I don't think we can do that.

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#49 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs

Am 06.10.2016 um 17:18 hat Daniel P. Berrange geschrieben:

Quoted 31 lines · Kevin Wolf
Quoted 30 lines

Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:

Quoted 29 lines

Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the qdict_crumple() call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.

Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.

The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach

  -net vlan=1,id=foo,name=bar,opts.type=tap,\
       opts.data.fd=3,opts.data.script=ifup

The legacy syntax will just be presenting

-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup

So we need to auto-create 3 levels of struct when visiting.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.

So you're actually introducing the modern syntax only now?

No, the modern syntax is fully implemented by patch 8.

"now" in the sense of "in this series" (because this means that there is no external API to preserve yet).

Quoted 10 lines

This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.

Quoted 3 lines

I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.

Quoted 7 lines
Quoted 4 lines

So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.

Changing that will impact on the QMP data structure, so I don't think we can do that.

I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.

Kevin

#50 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs

Quoted 36 lines · Kevin Wolf

Am 06.10.2016 um 17:18 hat Daniel P. Berrange geschrieben:

Quoted 35 lines · Kevin Wolf
Quoted 34 lines

Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:

Quoted 32 lines

Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the qdict_crumple() call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.

Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.

The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach

  -net vlan=1,id=foo,name=bar,opts.type=tap,\
       opts.data.fd=3,opts.data.script=ifup

The legacy syntax will just be presenting

-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup

So we need to auto-create 3 levels of struct when visiting.

The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

So you're actually introducing the modern syntax only now?

No, the modern syntax is fully implemented by patch 8.

"now" in the sense of "in this series" (because this means that there is no external API to preserve yet).

Well the syntax implemented in patch 8 is designed to 100% mirror the QAPI schema structure nesting. I don't think we want to change that behaviour.

If there are certain QAPI schemas structs can still have freedom to change though, its fine to reconsider them

Quoted 16 lines
Quoted 16 lines

This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.

Quoted 7 lines

I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.

So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.

Quoted 6 lines
Quoted 2 lines

Changing that will impact on the QMP data structure, so I don't think we can do that.

I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.

Ok, yes, it seems "NetLegacy" is only used in CLI arg parsing, so we do have some freedom there.

This patch was also needed for -numa handling too - again we might have some flexibility to flatten that.

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#51 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs

Am 06.10.2016 um 17:39 hat Daniel P. Berrange geschrieben:

Quoted 23 lines · Kevin Wolf
Quoted 22 lines

Am 06.10.2016 um 17:18 hat Daniel P. Berrange geschrieben:

Quoted 21 lines · Kevin Wolf
Quoted 19 lines

Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:

Quoted 9 lines

The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach

  -net vlan=1,id=foo,name=bar,opts.type=tap,\
       opts.data.fd=3,opts.data.script=ifup

The legacy syntax will just be presenting

Quoted 7 lines

-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup

So we need to auto-create 3 levels of struct when visiting.

The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.

So you're actually introducing the modern syntax only now?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

No, the modern syntax is fully implemented by patch 8.

"now" in the sense of "in this series" (because this means that there is no external API to preserve yet).

Well the syntax implemented in patch 8 is designed to 100% mirror the QAPI schema structure nesting. I don't think we want to change that behaviour.

I think patch 8 is fine as it is.

Quoted 27 lines

If there are certain QAPI schemas structs can still have freedom to change though, its fine to reconsider them

Quoted 16 lines
Quoted 16 lines

This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.

Quoted 7 lines

I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.

So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.

Quoted 6 lines
Quoted 2 lines

Changing that will impact on the QMP data structure, so I don't think we can do that.

I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.

Quoted 5 lines

Ok, yes, it seems "NetLegacy" is only used in CLI arg parsing, so we do have some freedom there.

This patch was also needed for -numa handling too - again we might have some flexibility to flatten that.

NumaOptions is also unused in QMP, so it's the same thing.

If these two are the only options that need the behaviour, I would prefer if we changed the QAPI schema to flatten them, and then we could save ourselves some complexity by dropping this patch.

Kevin

#52 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs

Quoted 65 lines · Kevin Wolf

Am 06.10.2016 um 17:39 hat Daniel P. Berrange geschrieben:

Quoted 23 lines · Kevin Wolf
Quoted 22 lines

Am 06.10.2016 um 17:18 hat Daniel P. Berrange geschrieben:

Quoted 21 lines · Kevin Wolf
Quoted 19 lines

Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:

Quoted 9 lines

The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach

  -net vlan=1,id=foo,name=bar,opts.type=tap,\
       opts.data.fd=3,opts.data.script=ifup

The legacy syntax will just be presenting

Quoted 7 lines

-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup

So we need to auto-create 3 levels of struct when visiting.

The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.

So you're actually introducing the modern syntax only now?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

No, the modern syntax is fully implemented by patch 8.

"now" in the sense of "in this series" (because this means that there is no external API to preserve yet).

Well the syntax implemented in patch 8 is designed to 100% mirror the QAPI schema structure nesting. I don't think we want to change that behaviour.

I think patch 8 is fine as it is.

Quoted 30 lines

If there are certain QAPI schemas structs can still have freedom to change though, its fine to reconsider them

Quoted 16 lines
Quoted 16 lines

This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.

Quoted 7 lines

I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.

So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.

Quoted 6 lines
Quoted 2 lines

Changing that will impact on the QMP data structure, so I don't think we can do that.

I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.

Ok, yes, it seems "NetLegacy" is only used in CLI arg parsing, so we do have some freedom there.

Quoted 8 lines
Quoted 2 lines

This patch was also needed for -numa handling too - again we might have some flexibility to flatten that.

NumaOptions is also unused in QMP, so it's the same thing.

If these two are the only options that need the behaviour, I would prefer if we changed the QAPI schema to flatten them, and then we could save ourselves some complexity by dropping this patch.

Agreed, the fewer hacks like this that we need the better.

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#54 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs

In reply to Kevin Wolf
Quoted 11 lines · Kevin Wolf
Quoted 7 lines
Quoted 4 lines

So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.

Changing that will impact on the QMP data structure, so I don't think we can do that.

I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.

In fact, in all my work to move netdev_add towards QAPI, I intentionally special-cased NetLegacy to be unchanged, because it was not being used by QMP at the time, and I didn't want any QMP changes to netdev to break command line usage of NetLegacy.

We still have the annoying problem that my last patch for converting netdev_add to QAPI didn't make 2.7, because we hadn't sorted out whether we wanted to be able to handle back-compat of a user that requested "1" vs. 1 (the QemuOpts code accepted either spelling, by virtue of the fact that QDict to opts conversion rewrote the parsed QMP object into all strings); and maybe this series solves that issue. But the issue for netdev_add (which IS visible to QMP) is slightly different than the issue for NetLegacy (which does not have QMP ties other than using QAPI to define a struct and glue code in net.c to map it back to normal netdev code).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#19 lore raw

Re: [Qemu-devel] [PATCH v14 03/21] qapi: add trace events for visitor

In reply to Daniel P. Berrange
Quoted 24 lines · Daniel P. Berrange

Allow tracing of the operation of visitors

---
 Makefile.objs          |  1 +
 qapi/qapi-visit-core.c | 27 +++++++++++++++++++++++++++
 qapi/trace-events      | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+)
 create mode 100644 qapi/trace-events
Makefile.objs +1 -0
index a8e0224..b3e8aef 100644
@@ -160,3 +160,4 @@ trace-events-y += target-s390x/trace-events
 trace-events-y += target-ppc/trace-events
 trace-events-y += qom/trace-events
 trace-events-y += linux-user/trace-events
+trace-events-y += qapi/trace-events
qapi/qapi-visit-core.c +0 -0
index 55f5876..bfcb276 100644

qapi/qapi-visit-core.c line 27

Quoted 9 lines
qapi/qapi-visit-core.c +2 -0
@@ -19,10 +19,12 @@
 #include "qapi/qmp/qerror.h"
 #include "qapi/visitor.h"
 #include "qapi/visitor-impl.h"
+#include "trace.h"
 
 void visit_complete(Visitor *v, void *opaque)
 {
     assert(v->type != VISITOR_OUTPUT || v->complete);
+    trace_visit_complete(v, opaque);

Trace after or before checking the precondition? Preferences, anyone?

Quoted 90 lines
qapi/qapi-visit-core.c +11 -0
     if (v->complete) {
         v->complete(v, opaque);
     }
@@ -30,6 +32,7 @@ void visit_complete(Visitor *v, void *opaque)
 
 void visit_free(Visitor *v)
 {
+    trace_visit_free(v);
     if (v) {
         v->free(v);
     }
@@ -40,6 +43,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,
 {
     Error *err = NULL;
 
+    trace_visit_start_struct(v, name, obj, size);
     if (obj) {
         assert(size);
         assert(!(v->type & VISITOR_OUTPUT) || *obj);
@@ -53,6 +57,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,
 
 void visit_check_struct(Visitor *v, Error **errp)
 {
+    trace_visit_check_struct(v);
     if (v->check_struct) {
         v->check_struct(v, errp);
     }
@@ -60,6 +65,7 @@ void visit_check_struct(Visitor *v, Error **errp)
 
 void visit_end_struct(Visitor *v, void **obj)
 {
+    trace_visit_end_struct(v, obj);
     v->end_struct(v, obj);
 }
 
@@ -69,6 +75,7 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,
     Error *err = NULL;
 
     assert(!list || size >= sizeof(GenericList));
+    trace_visit_start_list(v, name, list, size);
     v->start_list(v, name, list, size, &err);
     if (list && (v->type & VISITOR_INPUT)) {
         assert(!(err && *list));
@@ -79,11 +86,13 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,
 GenericList *visit_next_list(Visitor *v, GenericList *tail, size_t size)
 {
     assert(tail && size >= sizeof(GenericList));
+    trace_visit_next_list(v, tail, size);
     return v->next_list(v, tail, size);
 }
 
 void visit_end_list(Visitor *v, void **obj)
 {
+    trace_visit_end_list(v, obj);
     v->end_list(v, obj);
 }
 
@@ -95,6 +104,7 @@ void visit_start_alternate(Visitor *v, const char *name,
 
     assert(obj && size >= sizeof(GenericAlternate));
     assert(!(v->type & VISITOR_OUTPUT) || *obj);
+    trace_visit_start_alternate(v, name, obj, size, promote_int);
     if (v->start_alternate) {
         v->start_alternate(v, name, obj, size, promote_int, &err);
     }
@@ -106,6 +116,7 @@ void visit_start_alternate(Visitor *v, const char *name,
 
 void visit_end_alternate(Visitor *v, void **obj)
 {
+    trace_visit_end_alternate(v, obj);
     if (v->end_alternate) {
         v->end_alternate(v, obj);
     }
@@ -113,6 +124,7 @@ void visit_end_alternate(Visitor *v, void **obj)
 
 bool visit_optional(Visitor *v, const char *name, bool *present)
 {
+    trace_visit_optional(v, name, present);
     if (v->optional) {
         v->optional(v, name, present);
     }
@@ -127,6 +139,7 @@ bool visit_is_input(Visitor *v)
 void visit_type_int(Visitor *v, const char *name, int64_t *obj, Error **errp)
 {
     assert(obj);
+    trace_visit_type_int(v, name, obj);
     v->type_int64(v, name, obj, errp);
 }
 

qapi/qapi-visit-core.c void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj, · line 170

Quoted 7 lines
qapi/qapi-visit-core.c +1 -0
@@ -151,6 +164,7 @@ void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj,
                       Error **errp)
 {
     uint64_t value = *obj;
+    trace_visit_type_uint8(v, name, obj);
     visit_type_uintN(v, &value, name, UINT8_MAX, "uint8_t", errp);
     *obj = value;
 }

Putting the trace in the middle of the "value = *obj; visit_type_uintN(v, &value, ...); *obj = value" pattern makes it less visible.

Preserving the pattern requires replacing the initializer by an assignment:

       uint64_t value;

       trace_visit_type_uint8(v, name, obj);
       value = *obj;
       visit_type_uintN(v, &value, name, UINT8_MAX, "uint8_t", errp);
       *obj = value;

Looks slightly more legible to me. What do you think?

[...]

#20 lore raw

Re: [Qemu-devel] [PATCH v14 03/21] qapi: add trace events for visitor

Quoted 31 lines · Markus Armbruster
Quoted 29 lines · Daniel P. Berrange

Allow tracing of the operation of visitors

---
 Makefile.objs          |  1 +
 qapi/qapi-visit-core.c | 27 +++++++++++++++++++++++++++
 qapi/trace-events      | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+)
 create mode 100644 qapi/trace-events
Makefile.objs +1 -0
index a8e0224..b3e8aef 100644
@@ -160,3 +160,4 @@ trace-events-y += target-s390x/trace-events
 trace-events-y += target-ppc/trace-events
 trace-events-y += qom/trace-events
 trace-events-y += linux-user/trace-events
+trace-events-y += qapi/trace-events
qapi/qapi-visit-core.c +1 -0
index 55f5876..bfcb276 100644
@@ -19,10 +19,12 @@
 #include "qapi/qmp/qerror.h"
 #include "qapi/visitor.h"
 #include "qapi/visitor-impl.h"
+#include "trace.h"
 
Quoted 6 lines

qapi/qapi-visit-core.c line 27

Quoted 4 lines
qapi/qapi-visit-core.c +1 -0
 void visit_complete(Visitor *v, void *opaque)
 {
     assert(v->type != VISITOR_OUTPUT || v->complete);
+    trace_visit_complete(v, opaque);

Trace after or before checking the precondition? Preferences, anyone?

I'm ambivalent, as the assert will crash you either way so whether you get a trace event just before the crash seems mostly irrelevant to me - the abort stack trace is what you'll use to diagnose the crash.

Quoted 106 lines
Quoted 91 lines
qapi/qapi-visit-core.c +11 -0
     if (v->complete) {
         v->complete(v, opaque);
     }
@@ -30,6 +32,7 @@ void visit_complete(Visitor *v, void *opaque)
 
 void visit_free(Visitor *v)
 {
+    trace_visit_free(v);
     if (v) {
         v->free(v);
     }
@@ -40,6 +43,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,
 {
     Error *err = NULL;
 
+    trace_visit_start_struct(v, name, obj, size);
     if (obj) {
         assert(size);
         assert(!(v->type & VISITOR_OUTPUT) || *obj);
@@ -53,6 +57,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,
 
 void visit_check_struct(Visitor *v, Error **errp)
 {
+    trace_visit_check_struct(v);
     if (v->check_struct) {
         v->check_struct(v, errp);
     }
@@ -60,6 +65,7 @@ void visit_check_struct(Visitor *v, Error **errp)
 
 void visit_end_struct(Visitor *v, void **obj)
 {
+    trace_visit_end_struct(v, obj);
     v->end_struct(v, obj);
 }
 
@@ -69,6 +75,7 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,
     Error *err = NULL;
 
     assert(!list || size >= sizeof(GenericList));
+    trace_visit_start_list(v, name, list, size);
     v->start_list(v, name, list, size, &err);
     if (list && (v->type & VISITOR_INPUT)) {
         assert(!(err && *list));
@@ -79,11 +86,13 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,
 GenericList *visit_next_list(Visitor *v, GenericList *tail, size_t size)
 {
     assert(tail && size >= sizeof(GenericList));
+    trace_visit_next_list(v, tail, size);
     return v->next_list(v, tail, size);
 }
 
 void visit_end_list(Visitor *v, void **obj)
 {
+    trace_visit_end_list(v, obj);
     v->end_list(v, obj);
 }
 
@@ -95,6 +104,7 @@ void visit_start_alternate(Visitor *v, const char *name,
 
     assert(obj && size >= sizeof(GenericAlternate));
     assert(!(v->type & VISITOR_OUTPUT) || *obj);
+    trace_visit_start_alternate(v, name, obj, size, promote_int);
     if (v->start_alternate) {
         v->start_alternate(v, name, obj, size, promote_int, &err);
     }
@@ -106,6 +116,7 @@ void visit_start_alternate(Visitor *v, const char *name,
 
 void visit_end_alternate(Visitor *v, void **obj)
 {
+    trace_visit_end_alternate(v, obj);
     if (v->end_alternate) {
         v->end_alternate(v, obj);
     }
@@ -113,6 +124,7 @@ void visit_end_alternate(Visitor *v, void **obj)
 
 bool visit_optional(Visitor *v, const char *name, bool *present)
 {
+    trace_visit_optional(v, name, present);
     if (v->optional) {
         v->optional(v, name, present);
     }
@@ -127,6 +139,7 @@ bool visit_is_input(Visitor *v)
 void visit_type_int(Visitor *v, const char *name, int64_t *obj, Error **errp)
 {
     assert(obj);
+    trace_visit_type_int(v, name, obj);
     v->type_int64(v, name, obj, errp);
 }
 

qapi/qapi-visit-core.c void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj, · line 170

Quoted 7 lines
qapi/qapi-visit-core.c +1 -0
@@ -151,6 +164,7 @@ void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj,
                       Error **errp)
 {
     uint64_t value = *obj;
+    trace_visit_type_uint8(v, name, obj);
     visit_type_uintN(v, &value, name, UINT8_MAX, "uint8_t", errp);
     *obj = value;
 }

Putting the trace in the middle of the "value = *obj; visit_type_uintN(v, &value, ...); *obj = value" pattern makes it less visible.

Preserving the pattern requires replacing the initializer by an assignment:

Quoted 8 lines
       uint64_t value;

       trace_visit_type_uint8(v, name, obj);
       value = *obj;
       visit_type_uintN(v, &value, name, UINT8_MAX, "uint8_t", errp);
       *obj = value;

Looks slightly more legible to me. What do you think?

I'm fine with whatever you prefer.

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#31 lore raw

Re: [Qemu-devel] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor

In reply to Daniel P. Berrange
Quoted 3 lines · Daniel P. Berrange

Currently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using

i.e.

Quoted 11 lines

QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.

To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.

Quoted 8 lines
---
 include/qapi/qobject-input-visitor.h |  32 +++++-
 qapi/qobject-input-visitor.c         | 132 ++++++++++++++++++++++++
 tests/test-qobject-input-visitor.c   | 194 ++++++++++++++++++++++++++++++++++-
 3 files changed, 350 insertions(+), 8 deletions(-)

Quite some code. Here's how I'm persuading myself to like it.

We need to parse two and a half different languages into QAPI objects:
JSON, command line key=value,... (QemuOpts), and just value (same syntax as in QemuOpts).

Two language differences stand out:

1. JSON is recursive, key=value,... is flat (but see below), and

2. JSON values have syntactically obvious JSON types, key=value values are just strings.

To parse JSON into a QAPI object, we first parse it into a QObject, then use the QObject input visitor to convert the QObject into a QAPI object.

To parse key=value, we first parse it into a QemuOpts, then use either the options visitor to convert the QemuOpts into a QAPI object. We can also use the string input visitor to convert just an individual key's value. Or any string for that matter.

Note that we always use the QemuOpts *string* value, never the integer or boolean value QemuOpts additionally provides when the key has been specified by a QemuOptDesc with a non-string type. Such QemuOptDesc are best avoided when we parse the string again with a visitor. For when it isn't avoided, the visitors' parsers need to match the QemuOpts parser. Yes, this is a mess.

The flatness of key=value,... has become overly limiting. We've grown workarounds for lists in both the options and the string input visitor. Independent ones *sigh*. More recently, the block layer's dotted key convention has provided a way to overcome flatness completely.

We could implement this convention in the options visitor. Instead, you're picking a different route: you reuse existing qemu_opts_to_qdict() to go from QemuOpts to flat qdict, create qdict_crumple() to unflatten using dotted key convention, then reuse the existing QObject input visitor to go to QAPI object. This decomposes the task into smaller ones, some of them solved already, at a tolerable cost in efficiency.

Except the QObject input visitor won't do as is, because of language difference 2. This patch makes it cope.

The need to do this negates much of the code reuse advantage. Doesn't look so hot by itself. However, you then do more work and replace the options visitor, with an obvious path to replacing the string input visitor as well.

Quoted 12 lines
include/qapi/qobject-input-visitor.h +2 -2
index cde328d..5022297 100644
@@ -19,12 +19,36 @@
 
 typedef struct QObjectInputVisitor QObjectInputVisitor;
 
-/*
- * Return a new input visitor that converts a QObject to a QAPI object.
+/**
+ * Create a new input visitor that converts @obj to a QAPI object.

include/qapi/qobject-input-visitor.h line 31

Quoted 10 lines
include/qapi/qobject-input-visitor.h +7 -2
+ *
+ * Any scalar values in the @obj input data structure should be in the
+ * required type already. i.e. if visiting a bool, the value should
+ * already be a QBool instance.
  *
- * Set @strict to reject a parse that doesn't consume all keys of a
- * dictionary; otherwise excess input is ignored.
+ * If @strict is set to true, then an error will be reported if any
+ * dict keys are not consumed during visitation. If @strict is false
+ * then extra dict keys are silently ignored.

Aside: I want to get rid of non-strict, badly. Quoting commit 240f64b "qapi: Use strict QMP input visitor in more places":

The only remaining uses of non-strict input visits are:

- QMP 'qom-set' (which eventually executes object_property_set_qobject()) - mark it as something to revisit in the future (I didn't want to spend any more time on this patch auditing if we have any QOM dictionary properties that might be impacted, and couldn't easily prove whether this code path is shared with anything else).

- test-qmp-input-visitor: explicit tests of non-strict mode. If we later get rid of users that don't need strictness, then this test should be merged with test-qmp-input-strict

Quoted 11 lines
include/qapi/qobject-input-visitor.h +8 -0
+ *
+ * The returned input visitor should be released by calling
+ * visit_free() when no longer required.
  */
 Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
 
+/**
+ * Create a new input visitor that converts @obj to a QAPI object.
+ *
+ * Any scalar values in the @obj input data structure should always be
+ * represented as strings. i.e. if visiting a boolean, the value should

include/qapi/qobject-input-visitor.h line 52

Quoted 10 lines
include/qapi/qobject-input-visitor.h +10 -0
+ * be a QString whose contents represent a valid boolean.
+ *
+ * The visitor always operates in strict mode, requiring all dict keys
+ * to be consumed during visitation. An error will be reported if this
+ * does not happen.
+ *
+ * The returned input visitor should be released by calling
+ * visit_free() when no longer required.
+ */
+Visitor *qobject_input_visitor_new_autocast(QObject *obj);

Not thrilled about _autocast. It kind of suggests that scalar values are converted as necessary. They are, but the full truth is that all scalar values must be strings. _string_scalars is even longer, though.

Quoted 15 lines
include/qapi/qobject-input-visitor.h +1 -0
+
 #endif
qapi/qobject-input-visitor.c +1 -0
index 5ff3db3..cf41df6 100644
@@ -20,6 +20,7 @@
 #include "qemu-common.h"
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/qerror.h"
+#include "qemu/cutils.h"
 
 #define QIV_STACK_SIZE 1024
 

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 273

Quoted 10 lines
qapi/qobject-input-visitor.c +7 -0
@@ -263,6 +264,28 @@ static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj,
     *obj = qint_get_int(qint);
 }
 
+
+static void qobject_input_type_int64_autocast(Visitor *v, const char *name,
+                                              int64_t *obj, Error **errp)
+{
+    QObjectInputVisitor *qiv = to_qiv(v);
+    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                                true));

Needs a rebase for commit 1382d4a. Same elsewhere.

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 276

Quoted 3 lines
qapi/qobject-input-visitor.c +3 -0
+    int64_t ret;
+
+    if (!qstr || !qstr->string) {

I don't think !qstr->string can happen. Same elsewhere.

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 280

Quoted 4 lines
qapi/qobject-input-visitor.c +4 -0
+        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "string");
+        return;
+    }

So far, this is basically qobject_input_type_str() less the g_strdup(). Worth factoring out?

Now we're entering out parsing swamp:

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 285

Quoted 5 lines
qapi/qobject-input-visitor.c +5 -0
+
+    if (qemu_strtoll(qstr->string, NULL, 0, &ret) < 0) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
+        return;
+    }

To serve as replacement for the options visitor, this needs to parse exactly like opts_type_int64().

It should also match the JSON parser as far as possible, to minimize difference between the two QObject input visitor variants, and the QemuOpts parser, for command line consistency.

opts_type_int64() uses strtoll() directly. It carefully checks for no conversion (both ways, EINVAL and endptr == str), range, and string not fully consumed.

Your code uses qemu_strtoll(). Bug#1: qemu_strtoll() assumes long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values. Bug#2: your code fails to check the string is fully consumed.

The JSON parser also uses strtoll() directly, in parse_literal(). When we get there, we know that the string consists only of decimal digits, possibly prefixed by a minus sign. Therefore, strtoll() can only fail with ERANGE, and parse_literal() handles that correctly.

QemuOpts doesn't do signed integers.

Quoted 19 lines
qapi/qobject-input-visitor.c +12 -0
+    *obj = ret;
+}
+
 static void qobject_input_type_uint64(Visitor *v, const char *name,
                                       uint64_t *obj, Error **errp)
 {
@@ -279,6 +302,27 @@ static void qobject_input_type_uint64(Visitor *v, const char *name,
     *obj = qint_get_int(qint);
 }
 
+static void qobject_input_type_uint64_autocast(Visitor *v, const char *name,
+                                               uint64_t *obj, Error **errp)
+{
+    QObjectInputVisitor *qiv = to_qiv(v);
+    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                                true));
+    unsigned long long ret;
+
+    if (!qstr || !qstr->string) {

qapi/qobject-input-visitor.c static void qobject_input_type_uint64(Visitor *v, const char *name, · line 323

Quoted 10 lines
qapi/qobject-input-visitor.c +10 -0
+        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "string");
+        return;
+    }
+
+    if (parse_uint_full(qstr->string, &ret, 0) < 0) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
+        return;
+    }
+    *obj = ret;

Differently wrong :)

To serve as replacement for the options visitor, this needs to parse exactly like opts_type_uint64().

Again, this should also match the JSON parser and the QemuOpts parser as far as possible.

opts_type_uint64() uses parse_uint(). It carefully checks for no conversion (EINVAL; parse_uint() normalizes), range, and string not fully consumed.

You use parse_uint_full(). You therefore don't have bug#2 here. You do have bug#1: you assume unsigned long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values.

There's a discrepancy with the JSON parser, but it's not your fault: the JSON parser represents JSON numbers that fit into int64_t to QInt, and any others as QFloat. In particular, the integers between INT64_MAX+1 and UINT64_MAX are represented as QFloat. qmp_input_type_uint64() accepts only QInt. You can declare things as uint64 in the schema, but you're still limited to 0..UINT64_MAX in QMP.

QemuOpts uses strtoull() directly, in parse_option_number(). Bug (not yours): it happily ignores errors other than "string not fully consumed".

Quoted 13 lines
qapi/qobject-input-visitor.c +6 -0
+}
+
 static void qobject_input_type_bool(Visitor *v, const char *name, bool *obj,
                                     Error **errp)
 {
@@ -294,6 +338,22 @@ static void qobject_input_type_bool(Visitor *v, const char *name, bool *obj,
     *obj = qbool_get_bool(qbool);
 }
 
+static void qobject_input_type_bool_autocast(Visitor *v, const char *name,
+                                             bool *obj, Error **errp)
+{
+    QObjectInputVisitor *qiv = to_qiv(v);

qapi/qobject-input-visitor.c static void qobject_input_type_bool(Visitor *v, const char *name, bool *obj, · line 354

Quoted 10 lines
qapi/qobject-input-visitor.c +10 -0
+    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                                true));
+
+    if (!qstr || !qstr->string) {
+        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "string");
+        return;
+    }
+
+    parse_option_bool(name, qstr->string, obj, errp);

To serve as replacement for the options visitor, this needs to parse exactly like opts_type_bool(). Which parses with parse_option_bool(), too. Good.

QemuOpts parses the same way.

JSON syntax is different.

Quoted 23 lines
qapi/qobject-input-visitor.c +16 -0
+}
+
 static void qobject_input_type_str(Visitor *v, const char *name, char **obj,
                                    Error **errp)
 {
@@ -335,6 +395,30 @@ static void qobject_input_type_number(Visitor *v, const char *name, double *obj,
                "number");
 }
 
+static void qobject_input_type_number_autocast(Visitor *v, const char *name,
+                                               double *obj, Error **errp)
+{
+    QObjectInputVisitor *qiv = to_qiv(v);
+    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                                true));
+    char *endp;
+
+    if (!qstr || !qstr->string) {
+        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "string");
+        return;
+    }
+

qapi/qobject-input-visitor.c static void qobject_input_type_number(Visitor *v, const char *name, double *obj, · line 421

Quoted 10 lines
qapi/qobject-input-visitor.c +10 -0
+    errno = 0;
+    *obj = strtod(qstr->string, &endp);
+    if (errno == 0 && endp != qstr->string && *endp == '\0') {
+        return;
+    }
+
+    error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+               "number");
+}
+

The options visitor does not implement this method.

Your error checking looks good.

The JSON parser also uses strtod(), in parse_literal(). When we get there, we know that the string is a valid JSON number. Therefore, strtod() can only fail with ERANGE. Bug (not yours): we ignore that.

QemuOpts doesn't do floating-point.

Quoted 11 lines
qapi/qobject-input-visitor.c +4 -0
 static void qobject_input_type_any(Visitor *v, const char *name, QObject **obj,
                                    Error **errp)
 {
@@ -356,6 +440,22 @@ static void qobject_input_type_null(Visitor *v, const char *name, Error **errp)
     }
 }
 
+static void qobject_input_type_size_autocast(Visitor *v, const char *name,
+                                             uint64_t *obj, Error **errp)
+{
+    QObjectInputVisitor *qiv = to_qiv(v);

qapi/qobject-input-visitor.c static void qobject_input_type_null(Visitor *v, const char *name, Error **errp) · line 456

Quoted 10 lines
qapi/qobject-input-visitor.c +10 -0
+    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                                true));
+
+    if (!qstr || !qstr->string) {
+        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "string");
+        return;
+    }
+
+    parse_option_size(name, qstr->string, obj, errp);

The options visitor uses qemu_strtosz_suffix(). Are you sure parse_option_size() is a compatible replacement? (I'm getting tired...)

Quoted 300 lines
qapi/qobject-input-visitor.c +34 -0
+}
+
 static void qobject_input_optional(Visitor *v, const char *name, bool *present)
 {
     QObjectInputVisitor *qiv = to_qiv(v);
@@ -413,3 +513,35 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)
 
     return &v->visitor;
 }
+
+Visitor *qobject_input_visitor_new_autocast(QObject *obj)
+{
+    QObjectInputVisitor *v;
+
+    v = g_malloc0(sizeof(*v));
+
+    v->visitor.type = VISITOR_INPUT;
+    v->visitor.start_struct = qobject_input_start_struct;
+    v->visitor.check_struct = qobject_input_check_struct;
+    v->visitor.end_struct = qobject_input_pop;
+    v->visitor.start_list = qobject_input_start_list;
+    v->visitor.next_list = qobject_input_next_list;
+    v->visitor.end_list = qobject_input_pop;
+    v->visitor.start_alternate = qobject_input_start_alternate;
+    v->visitor.type_int64 = qobject_input_type_int64_autocast;
+    v->visitor.type_uint64 = qobject_input_type_uint64_autocast;
+    v->visitor.type_bool = qobject_input_type_bool_autocast;
+    v->visitor.type_str = qobject_input_type_str;
+    v->visitor.type_number = qobject_input_type_number_autocast;
+    v->visitor.type_any = qobject_input_type_any;
+    v->visitor.type_null = qobject_input_type_null;
+    v->visitor.type_size = qobject_input_type_size_autocast;
+    v->visitor.optional = qobject_input_optional;
+    v->visitor.free = qobject_input_free;
+    v->strict = true;
+
+    v->root = obj;
+    qobject_incref(obj);
+
+    return &v->visitor;
+}
tests/test-qobject-input-visitor.c +183 -4
index 26c5012..d5b8044 100644
@@ -41,6 +41,7 @@ static void visitor_input_teardown(TestInputVisitorData *data,
    function so that the JSON string used by the tests are kept in the test
    functions (and not in main()). */
 static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data,
+                                                 bool strict, bool autocast,
                                                  const char *json_string,
                                                  va_list *ap)
 {
@@ -49,11 +50,31 @@ static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data,
     data->obj = qobject_from_jsonv(json_string, ap);
     g_assert(data->obj);
 
-    data->qiv = qobject_input_visitor_new(data->obj, false);
+    if (autocast) {
+        assert(strict);
+        data->qiv = qobject_input_visitor_new_autocast(data->obj);
+    } else {
+        data->qiv = qobject_input_visitor_new(data->obj, strict);
+    }
     g_assert(data->qiv);
     return data->qiv;
 }
 
+static GCC_FMT_ATTR(4, 5)
+Visitor *visitor_input_test_init_full(TestInputVisitorData *data,
+                                      bool strict, bool autocast,
+                                      const char *json_string, ...)
+{
+    Visitor *v;
+    va_list ap;
+
+    va_start(ap, json_string);
+    v = visitor_input_test_init_internal(data, strict, autocast,
+                                         json_string, &ap);
+    va_end(ap);
+    return v;
+}
+
 static GCC_FMT_ATTR(2, 3)
 Visitor *visitor_input_test_init(TestInputVisitorData *data,
                                  const char *json_string, ...)
@@ -62,7 +83,8 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,
     va_list ap;
 
     va_start(ap, json_string);
-    v = visitor_input_test_init_internal(data, json_string, &ap);
+    v = visitor_input_test_init_internal(data, true, false,
+                                         json_string, &ap);
     va_end(ap);
     return v;
 }
@@ -77,7 +99,8 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,
 static Visitor *visitor_input_test_init_raw(TestInputVisitorData *data,
                                             const char *json_string)
 {
-    return visitor_input_test_init_internal(data, json_string, NULL);
+    return visitor_input_test_init_internal(data, true, false,
+                                            json_string, NULL);
 }
 
 static void test_visitor_in_int(TestInputVisitorData *data,
@@ -109,6 +132,45 @@ static void test_visitor_in_int_overflow(TestInputVisitorData *data,
     error_free_or_abort(&err);
 }
 
+static void test_visitor_in_int_autocast(TestInputVisitorData *data,
+                                         const void *unused)
+{
+    int64_t res = 0, value = -42;
+    Error *err = NULL;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, true, true,
+                                     "%" PRId64, value);
+    visit_type_int(v, NULL, &res, &err);
+    error_free_or_abort(&err);
+}
+
+static void test_visitor_in_int_str_autocast(TestInputVisitorData *data,
+                                             const void *unused)
+{
+    int64_t res = 0, value = -42;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, true, true,
+                                     "\"-42\"");
+
+    visit_type_int(v, NULL, &res, &error_abort);
+    g_assert_cmpint(res, ==, value);
+}
+
+static void test_visitor_in_int_str_noautocast(TestInputVisitorData *data,
+                                               const void *unused)
+{
+    int64_t res = 0;
+    Visitor *v;
+    Error *err = NULL;
+
+    v = visitor_input_test_init(data, "\"-42\"");
+
+    visit_type_int(v, NULL, &res, &err);
+    error_free_or_abort(&err);
+}
+
 static void test_visitor_in_bool(TestInputVisitorData *data,
                                  const void *unused)
 {
@@ -121,6 +183,44 @@ static void test_visitor_in_bool(TestInputVisitorData *data,
     g_assert_cmpint(res, ==, true);
 }
 
+static void test_visitor_in_bool_autocast(TestInputVisitorData *data,
+                                          const void *unused)
+{
+    bool res = false;
+    Error *err = NULL;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, true, true, "true");
+
+    visit_type_bool(v, NULL, &res, &err);
+    error_free_or_abort(&err);
+}
+
+static void test_visitor_in_bool_str_autocast(TestInputVisitorData *data,
+                                              const void *unused)
+{
+    bool res = false;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, true, true, "\"yes\"");
+
+    visit_type_bool(v, NULL, &res, &error_abort);
+    g_assert_cmpint(res, ==, true);
+}
+
+static void test_visitor_in_bool_str_noautocast(TestInputVisitorData *data,
+                                                const void *unused)
+{
+    bool res = false;
+    Visitor *v;
+    Error *err = NULL;
+
+    v = visitor_input_test_init(data, "\"true\"");
+
+    visit_type_bool(v, NULL, &res, &err);
+    error_free_or_abort(&err);
+}
+
 static void test_visitor_in_number(TestInputVisitorData *data,
                                    const void *unused)
 {
@@ -133,6 +233,69 @@ static void test_visitor_in_number(TestInputVisitorData *data,
     g_assert_cmpfloat(res, ==, value);
 }
 
+static void test_visitor_in_number_autocast(TestInputVisitorData *data,
+                                            const void *unused)
+{
+    double res = 0, value = 3.14;
+    Error *err = NULL;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, true, true, "%f", value);
+
+    visit_type_number(v, NULL, &res, &err);
+    error_free_or_abort(&err);
+}
+
+static void test_visitor_in_number_str_autocast(TestInputVisitorData *data,
+                                                const void *unused)
+{
+    double res = 0, value = 3.14;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, true, true, "\"3.14\"");
+
+    visit_type_number(v, NULL, &res, &error_abort);
+    g_assert_cmpfloat(res, ==, value);
+}
+
+static void test_visitor_in_number_str_noautocast(TestInputVisitorData *data,
+                                                  const void *unused)
+{
+    double res = 0;
+    Visitor *v;
+    Error *err = NULL;
+
+    v = visitor_input_test_init(data, "\"3.14\"");
+
+    visit_type_number(v, NULL, &res, &err);
+    error_free_or_abort(&err);
+}
+
+static void test_visitor_in_size_str_autocast(TestInputVisitorData *data,
+                                              const void *unused)
+{
+    uint64_t res, value = 500 * 1024 * 1024;
+    Visitor *v;
+
+    v = visitor_input_test_init_full(data, true, true, "\"500M\"");
+
+    visit_type_size(v, NULL, &res, &error_abort);
+    g_assert_cmpfloat(res, ==, value);
+}
+
+static void test_visitor_in_size_str_noautocast(TestInputVisitorData *data,
+                                                const void *unused)
+{
+    uint64_t res = 0;
+    Visitor *v;
+    Error *err = NULL;
+
+    v = visitor_input_test_init(data, "\"500M\"");
+
+    visit_type_size(v, NULL, &res, &err);
+    error_free_or_abort(&err);
+}
+
 static void test_visitor_in_string(TestInputVisitorData *data,
                                    const void *unused)
 {
@@ -289,7 +452,8 @@ static void test_visitor_in_null(TestInputVisitorData *data,
      * when input is not null.
      */
 
-    v = visitor_input_test_init(data, "{ 'a': null, 'b': '' }");
+    v = visitor_input_test_init_full(data, false, false,
+                                     "{ 'a': null, 'b': '' }");
     visit_start_struct(v, NULL, NULL, 0, &error_abort);
     visit_type_null(v, "a", &error_abort);
     visit_type_str(v, "a", &tmp, &err);
@@ -840,10 +1004,32 @@ int main(int argc, char **argv)
                            NULL, test_visitor_in_int);
     input_visitor_test_add("/visitor/input/int_overflow",
                            NULL, test_visitor_in_int_overflow);
+    input_visitor_test_add("/visitor/input/int_autocast",
+                           NULL, test_visitor_in_int_autocast);
+    input_visitor_test_add("/visitor/input/int_str_autocast",
+                           NULL, test_visitor_in_int_str_autocast);
+    input_visitor_test_add("/visitor/input/int_str_noautocast",
+                           NULL, test_visitor_in_int_str_noautocast);
     input_visitor_test_add("/visitor/input/bool",
                            NULL, test_visitor_in_bool);
+    input_visitor_test_add("/visitor/input/bool_autocast",
+                           NULL, test_visitor_in_bool_autocast);
+    input_visitor_test_add("/visitor/input/bool_str_autocast",
+                           NULL, test_visitor_in_bool_str_autocast);
+    input_visitor_test_add("/visitor/input/bool_str_noautocast",
+                           NULL, test_visitor_in_bool_str_noautocast);
     input_visitor_test_add("/visitor/input/number",
                            NULL, test_visitor_in_number);
+    input_visitor_test_add("/visitor/input/number_autocast",
+                           NULL, test_visitor_in_number_autocast);
+    input_visitor_test_add("/visitor/input/number_str_autocast",

tests/test-qobject-input-visitor.c int main(int argc, char **argv) · line 1035

Quoted 10 lines
tests/test-qobject-input-visitor.c +7 -0
+                           NULL, test_visitor_in_number_str_autocast);
+    input_visitor_test_add("/visitor/input/number_str_noautocast",
+                           NULL, test_visitor_in_number_str_noautocast);
+    input_visitor_test_add("/visitor/input/size_str_autocast",
+                           NULL, test_visitor_in_size_str_autocast);
+    input_visitor_test_add("/visitor/input/size_str_noautocast",
+                           NULL, test_visitor_in_size_str_noautocast);
     input_visitor_test_add("/visitor/input/string",
                            NULL, test_visitor_in_string);
     input_visitor_test_add("/visitor/input/enum",

I don't like the test code duplication. The alternative would be a function that rewrites all the scalars in a QObject to strings. Probably not worthwhile. We'll live with the duplication then.

#38 lore raw

Re: [Qemu-devel] [PATCH v14 08/21] qapi: allow QObjectInputVisitor to be created with QemuOpts

In reply to Daniel P. Berrange
Quoted 1 line · Daniel P. Berrange

Instead of requiring all callers to go through the mutli-step

multi-step

Quoted 61 lines

process of turning QemuOpts into a suitable QObject for visiting, add a new constructor that encapsulates this logic. This will allow QObjectInputVisitor to be a drop-in replacement for the existing OptsVisitor with minimal code changes for callers.

NB, at this point it is only supporting opts syntax which explicitly matches the QAPI schema structure, so is not yet a true drop-in replacement for OptsVisitor. The patches that follow will add the special cases requird for full backwards compatibility with OptsVisitor.

---
 include/qapi/qobject-input-visitor.h | 15 +++++++++++++++
 include/qemu/option.h                |  2 +-
 qapi/qobject-input-visitor.c         | 25 +++++++++++++++++++++++++
 util/qemu-option.c                   |  2 +-
 4 files changed, 42 insertions(+), 2 deletions(-)
include/qapi/qobject-input-visitor.h +15 -0
index 5022297..f134d90 100644
@@ -51,4 +51,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  */
 Visitor *qobject_input_visitor_new_autocast(QObject *obj);
 
+
+/**
+ * Create a new input visitor that converts @opts to a QAPI object.
+ *
+ * The QemuOpts will be converted into a QObject using the
+ * qdict_crumple() method to automatically create structs
+ * and lists. The resulting QDict will then be passed to the
+ * qobject_input_visitor_new_autocast() method. See the docs
+ * of that method for further details on processing behaviour.
+ *
+ * The returned input visitor should be released by calling
+ * visit_free() when no longer required.
+ */
+Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, Error **errp);
+
 #endif
include/qemu/option.h +1 -1
index 2a5266f..29f3f18 100644
@@ -125,7 +125,7 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char *params,
                             int permit_abbrev);
 QemuOpts *qemu_opts_from_qdict(QemuOptsList *list, const QDict *qdict,
                                Error **errp);
-QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict);
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict);
 void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp);
 
 typedef int (*qemu_opts_loopfunc)(void *opaque, QemuOpts *opts, Error **errp);
qapi/qobject-input-visitor.c +0 -0
index cf41df6..d9269c9 100644

qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_autocast(QObject *obj) · line 554

Quoted 10 lines
qapi/qobject-input-visitor.c +7 -0
@@ -545,3 +545,28 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
 
     return &v->visitor;
 }
+
+
+Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
+                                        Error **errp)
+{
+    QDict *pdict;
+    QObject *pobj = NULL;

@pdict and @pobj are unusual names. Let's stick to the more common @dict and @obj.

Quoted 23 lines
qapi/qobject-input-visitor.c +18 -0
+    Visitor *v = NULL;
+
+    pdict = qemu_opts_to_qdict(opts, NULL);
+    if (!pdict) {
+        goto cleanup;
+    }
+
+    pobj = qdict_crumple(pdict, true, errp);
+    if (!pobj) {
+        goto cleanup;
+    }
+
+    v = qobject_input_visitor_new_autocast(pobj);
+ cleanup:
+    qobject_decref(pobj);
+    QDECREF(pdict);
+    return v;
+}
util/qemu-option.c +0 -0
index 41b356c..418cbb6 100644

util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1059

Quoted 2 lines
util/qemu-option.c +0 -0
@@ -1058,7 +1058,7 @@ void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp)
  * TODO We'll want to use types appropriate for opt->desc->type, but
  * this is enough for now.

Implementing this TODO could break users that pass it to qobject_input_visitor_new_autocast(), because despite its name, the _autocast visitor expects *only* string scalars.

I guess (but have not checked) that these users have null opt->desc since their opts->list->desc[] is empty.

We should either drop the TODO (needs review of the other users), or update it to reflect the new situation. Perhaps users of qobject_input_visitor_new_autocast() should additionally assert opts->list->desc[] is empty.

Quoted 6 lines
util/qemu-option.c +1 -1
  */
-QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict)
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict)
 {
     QemuOpt *opt;
     QObject *val;
#42 lore raw

Re: [Qemu-devel] [PATCH v14 09/21] qapi: permit auto-creating single element lists

In reply to Daniel P. Berrange
Quoted 2 lines · Daniel P. Berrange

When converting QemuOpts to a QObject, there is no information about compound types available,

Yes, that's a drawback of splitting the conversion into a QemuOpts -> QObject part that is oblivious of types, and a QObject -> QAPI object part that knows the types.

Quoted 5 lines

so when visiting a list, the
corresponding QObject is not guaranteed to be a QList. We therefore need to be able to auto-create a single element QList from whatever type we find.

Quoted 8 lines

This mode should only be enabled if you have compatibility requirements for

-arg foo=hello,foo=world

to be treated as equivalent to the preferred syntax:

   -arg foo.0=hello,foo.1=world

Not sure this is "preferred". "More powerfully warty" is probably closer to the truth ;)

How is "-arg foo=hello,foo=world" treated if this mode isn't enabled?

What would be the drawbacks of doing this always instead of only when we "have compatibility requirements"?

Quoted 75 lines
---
 include/qapi/qobject-input-visitor.h | 20 +++++++-
 qapi/qobject-input-visitor.c         | 27 +++++++++--
 tests/test-qobject-input-visitor.c   | 88 +++++++++++++++++++++++++++++++-----
 3 files changed, 117 insertions(+), 18 deletions(-)
include/qapi/qobject-input-visitor.h +18 -2
index f134d90..1809f48 100644
@@ -42,6 +42,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * represented as strings. i.e. if visiting a boolean, the value should
  * be a QString whose contents represent a valid boolean.
  *
+ * If @autocreate_list is true, then as an alternative to a normal QList,
+ * list values can be stored as a QString or QDict instead, which will
+ * be interpreted as representing single element lists. This should only
+ * by used if compatibility is required with the OptsVisitor which allowed
+ * repeated keys, without list indexes, to represent lists. e.g. set this
+ * to true if you have compatibility requirements for
+ *
+ *   -arg foo=hello,foo=world
+ *
+ * to be treated as equivalent to the preferred syntax:
+ *
+ *   -arg foo.0=hello,foo.1=world
+ *
  * The visitor always operates in strict mode, requiring all dict keys
  * to be consumed during visitation. An error will be reported if this
  * does not happen.
@@ -49,7 +62,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * The returned input visitor should be released by calling
  * visit_free() when no longer required.
  */
-Visitor *qobject_input_visitor_new_autocast(QObject *obj);
+Visitor *qobject_input_visitor_new_autocast(QObject *obj,
+                                            bool autocreate_list);
 
 
 /**
@@ -64,6 +78,8 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj);
  * The returned input visitor should be released by calling
  * visit_free() when no longer required.
  */
-Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, Error **errp);
+Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
+                                        bool autocreate_list,
+                                        Error **errp);
 
 #endif
qapi/qobject-input-visitor.c +5 -0
index d9269c9..d88e9f9 100644
@@ -48,6 +48,10 @@ struct QObjectInputVisitor
 
     /* True to reject parse in visit_end_struct() if unvisited keys remain. */
     bool strict;
+
+    /* Whether we can auto-create single element lists when
+     * encountering a non-QList type */
+    bool autocreate_list;
 };
 
 static QObjectInputVisitor *to_qiv(Visitor *v)
@@ -108,6 +112,7 @@ static const QListEntry *qobject_input_push(QObjectInputVisitor *qiv,
     assert(obj);
     tos->obj = obj;
     tos->qapi = qapi;
+    qobject_incref(obj);
 
     if (qiv->strict && qobject_type(obj) == QTYPE_QDICT) {
         h = g_hash_table_new(g_str_hash, g_str_equal);

qapi/qobject-input-visitor.c static void qobject_input_stack_object_free(StackObject *tos) · line 158

Quoted 7 lines
qapi/qobject-input-visitor.c +1 -0
@@ -147,6 +152,7 @@ static void qobject_input_stack_object_free(StackObject *tos)
     if (tos->h) {
         g_hash_table_unref(tos->h);
     }
+    qobject_decref(tos->obj);
 
     g_free(tos);
 }

Can you explain the reference counting change?

qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 206

Quoted 6 lines
qapi/qobject-input-visitor.c +1 -1
@@ -197,7 +203,7 @@ static void qobject_input_start_list(Visitor *v, const char *name,
     QObject *qobj = qobject_input_get_object(qiv, name, true);
     const QListEntry *entry;
 
-    if (!qobj || qobject_type(qobj) != QTYPE_QLIST) {
+    if (!qobj || (!qiv->autocreate_list && qobject_type(qobj) != QTYPE_QLIST)) {

Long line, but I believe it'll go away when you rebase for commit 1382d4a.

Quoted 11 lines
qapi/qobject-input-visitor.c +3 -1
         if (list) {
             *list = NULL;
         }
@@ -206,7 +212,16 @@ static void qobject_input_start_list(Visitor *v, const char *name,
         return;
     }
 
-    entry = qobject_input_push(qiv, qobj, list, errp);
+    if (qobject_type(qobj) != QTYPE_QLIST) {
+        QList *tmplist = qlist_new();
+        qlist_append_obj(tmplist, qobj);

qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 227

Quoted 10 lines
qapi/qobject-input-visitor.c +7 -0
+        qobject_incref(qobj);
+        entry = qobject_input_push(qiv, QOBJECT(tmplist), list, errp);
+        QDECREF(tmplist);
+    } else {
+        entry = qobject_input_push(qiv, qobj, list, errp);
+    }
+
     if (list) {
         if (entry) {
             *list = g_malloc0(size);

Buries autolist behavior in the middle of things. What about doing it first, so it's more separate?

       QObjectInputVisitor *qiv = to_qiv(v);
       QObject *qobj = qobject_input_get_object_(qiv, name, true, errp);
       const QListEntry *entry;

       if (!qobj) {
           return;
       }

+ if (qiv->autocreate_list && qobject_type(qobj) != QTYPE_QLIST) {

  +        QList *auto_list = qlist_new();
  +        qlist_append_obj(auto_list, qobj);
  +        qobj = auto_list;
  +    }
  +
       if (qobject_type(qobj) != QTYPE_QLIST) {

I ignored reference counting here, because I don't yet understand how and why you're changing it.

Quoted 27 lines
qapi/qobject-input-visitor.c +4 -1
@@ -514,7 +529,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)
     return &v->visitor;
 }
 
-Visitor *qobject_input_visitor_new_autocast(QObject *obj)
+Visitor *qobject_input_visitor_new_autocast(QObject *obj,
+                                            bool autocreate_list)
 {
     QObjectInputVisitor *v;
 
@@ -539,6 +555,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
     v->visitor.optional = qobject_input_optional;
     v->visitor.free = qobject_input_free;
     v->strict = true;
+    v->autocreate_list = autocreate_list;
 
     v->root = obj;
     qobject_incref(obj);
@@ -548,6 +565,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
 
 
 Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
+                                        bool autocreate_list,
                                         Error **errp)
 {
     QDict *pdict;

qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 589

Quoted 9 lines
qapi/qobject-input-visitor.c +2 -1
@@ -564,7 +582,8 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
         goto cleanup;
     }
 
-    v = qobject_input_visitor_new_autocast(pobj);
+    v = qobject_input_visitor_new_autocast(pobj,
+                                           autocreate_list);
  cleanup:
     qobject_decref(pobj);
     QDECREF(pdict);

[Skipping test updates for now...]

#53 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs

In reply to Daniel P. Berrange
Quoted 63 lines · Daniel P. Berrange
Quoted 62 lines · Kevin Wolf

Am 06.10.2016 um 17:39 hat Daniel P. Berrange geschrieben:

Quoted 23 lines · Kevin Wolf
Quoted 22 lines

Am 06.10.2016 um 17:18 hat Daniel P. Berrange geschrieben:

Quoted 21 lines · Kevin Wolf
Quoted 19 lines

Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:

Quoted 9 lines

The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach

  -net vlan=1,id=foo,name=bar,opts.type=tap,\
       opts.data.fd=3,opts.data.script=ifup

The legacy syntax will just be presenting

Quoted 7 lines

-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup

So we need to auto-create 3 levels of struct when visiting.

The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.

So you're actually introducing the modern syntax only now?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

No, the modern syntax is fully implemented by patch 8.

"now" in the sense of "in this series" (because this means that there is no external API to preserve yet).

Well the syntax implemented in patch 8 is designed to 100% mirror the QAPI schema structure nesting. I don't think we want to change that behaviour.

I think patch 8 is fine as it is.

Quoted 27 lines

If there are certain QAPI schemas structs can still have freedom to change though, its fine to reconsider them

Quoted 22 lines
Quoted 11 lines

This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.

Quoted 3 lines

I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.

Quoted 7 lines
Quoted 4 lines

So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.

Changing that will impact on the QMP data structure, so I don't think we can do that.

I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Ok, yes, it seems "NetLegacy" is only used in CLI arg parsing, so we do have some freedom there.

This patch was also needed for -numa handling too - again we might have some flexibility to flatten that.

NumaOptions is also unused in QMP, so it's the same thing.

For better or worse, the QAPI schema doesn't separate externally visible stuff from purely internal stuff. Useful trick to see what's external:

    $ python -B scripts/qapi-introspect.py -cu -p scratch- qapi-schema.json

This generates scratch-qmp-introspect.c describing the external QMP interface with unmasked type names (-u). Anything not mentioned there can be changed freely.

I'd like to see patches flattening simple unions that aren't externally visible. Flat carries a bit of notational overhead in the schema, but I hope to address that once the QAPI queue is under control.

Quoted 5 lines
Quoted 3 lines

If these two are the only options that need the behaviour, I would prefer if we changed the QAPI schema to flatten them, and then we could save ourselves some complexity by dropping this patch.

Agreed, the fewer hacks like this that we need the better.

Indeed. Please find out which hacks we actually need.

I believe the best way to keep the warts in check is getting rid of the options visitor (done in this series) and the string visitor (hopefully enabled by this series).

#55 lore raw

Re: [Qemu-devel] [PATCH v14 10/21] qapi: permit auto-creating nested structs

In reply to Daniel P. Berrange
Quoted 45 lines · Daniel P. Berrange

Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the qdict_crumple() call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.

Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.

The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach

  -net vlan=1,id=foo,name=bar,opts.type=tap,\
       opts.data.fd=3,opts.data.script=ifup

The legacy syntax will just be presenting

-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup

So we need to auto-create 3 levels of struct when visiting.

The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.

---
 include/qapi/qobject-input-visitor.h |  22 +++++-
 qapi/qobject-input-visitor.c         |  59 ++++++++++++++--
 tests/test-qobject-input-visitor.c   | 130 ++++++++++++++++++++++++++++++++---
 3 files changed, 194 insertions(+), 17 deletions(-)
include/qapi/qobject-input-visitor.h +0 -0
index 1809f48..94051f3 100644

include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict); · line 51

Quoted 8 lines
include/qapi/qobject-input-visitor.h +1 -1
@@ -45,7 +45,7 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * If @autocreate_list is true, then as an alternative to a normal QList,
  * list values can be stored as a QString or QDict instead, which will
  * be interpreted as representing single element lists. This should only
- * by used if compatibility is required with the OptsVisitor which allowed
+ * be used if compatibility is required with the OptsVisitor which allowed
  * repeated keys, without list indexes, to represent lists. e.g. set this
  * to true if you have compatibility requirements for
  *

Typo introduced in the previous patch, please fix it there.

Skipping the rest of this patch until it's clear we need it.

[...]

#32 lore raw

Re: [Qemu-devel] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor

In reply to Markus Armbruster
Quoted 22 lines · Markus Armbruster
Quoted 3 lines · Daniel P. Berrange

Currently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using

i.e.

Quoted 14 lines

QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.

To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.

[...]

Quoted 37 lines
Quoted 13 lines
qapi/qobject-input-visitor.c +1 -0
index 5ff3db3..cf41df6 100644
@@ -20,6 +20,7 @@
 #include "qemu-common.h"
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/qerror.h"
+#include "qemu/cutils.h"
 
 #define QIV_STACK_SIZE 1024
 

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 273

Quoted 10 lines
qapi/qobject-input-visitor.c +7 -0
@@ -263,6 +264,28 @@ static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj,
     *obj = qint_get_int(qint);
 }
 
+
+static void qobject_input_type_int64_autocast(Visitor *v, const char *name,
+                                              int64_t *obj, Error **errp)
+{
+    QObjectInputVisitor *qiv = to_qiv(v);
+    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                                true));

Needs a rebase for commit 1382d4a. Same elsewhere.

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 276

Quoted 3 lines
qapi/qobject-input-visitor.c +3 -0
+    int64_t ret;
+
+    if (!qstr || !qstr->string) {

I don't think !qstr->string can happen. Same elsewhere.

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 280

Quoted 4 lines
qapi/qobject-input-visitor.c +4 -0
+        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "string");
+        return;
+    }
Quoted 8 lines

So far, this is basically qobject_input_type_str() less the g_strdup(). Worth factoring out?

Now we're entering out parsing swamp:

Quoted 3 lines
qapi/qobject-input-visitor.c +3 -0
+
+    if (qemu_strtoll(qstr->string, NULL, 0, &ret) < 0) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");

"integer", actually.

Quoted 36 lines

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 285

Quoted 2 lines
qapi/qobject-input-visitor.c +2 -0
+        return;
+    }

To serve as replacement for the options visitor, this needs to parse exactly like opts_type_int64().

It should also match the JSON parser as far as possible, to minimize difference between the two QObject input visitor variants, and the QemuOpts parser, for command line consistency.

opts_type_int64() uses strtoll() directly. It carefully checks for no conversion (both ways, EINVAL and endptr == str), range, and string not fully consumed.

Your code uses qemu_strtoll(). Bug#1: qemu_strtoll() assumes long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values. Bug#2: your code fails to check the string is fully consumed.

The JSON parser also uses strtoll() directly, in parse_literal(). When we get there, we know that the string consists only of decimal digits, possibly prefixed by a minus sign. Therefore, strtoll() can only fail with ERANGE, and parse_literal() handles that correctly.

QemuOpts doesn't do signed integers.

Quoted 10 lines
qapi/qobject-input-visitor.c +3 -0
+    *obj = ret;
+}
+
 static void qobject_input_type_uint64(Visitor *v, const char *name,
                                       uint64_t *obj, Error **errp)
 {
@@ -279,6 +302,27 @@ static void qobject_input_type_uint64(Visitor *v, const char *name,
     *obj = qint_get_int(qint);
 }
 
Quoted 16 lines
Quoted 6 lines
qapi/qobject-input-visitor.c +6 -0
+static void qobject_input_type_uint64_autocast(Visitor *v, const char *name,
+                                               uint64_t *obj, Error **errp)
+{
+    QObjectInputVisitor *qiv = to_qiv(v);
+    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                                true));

qapi/qobject-input-visitor.c static void qobject_input_type_uint64(Visitor *v, const char *name, · line 320

Quoted 10 lines
qapi/qobject-input-visitor.c +10 -0
+    unsigned long long ret;
+
+    if (!qstr || !qstr->string) {
+        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "string");
+        return;
+    }
+
+    if (parse_uint_full(qstr->string, &ret, 0) < 0) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");

"integer".

Quoted 37 lines

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 286

Quoted 3 lines
qapi/qobject-input-visitor.c +3 -0
+        return;
+    }
+    *obj = ret;

Differently wrong :)

To serve as replacement for the options visitor, this needs to parse exactly like opts_type_uint64().

Again, this should also match the JSON parser and the QemuOpts parser as far as possible.

opts_type_uint64() uses parse_uint(). It carefully checks for no conversion (EINVAL; parse_uint() normalizes), range, and string not fully consumed.

You use parse_uint_full(). You therefore don't have bug#2 here. You do have bug#1: you assume unsigned long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values.

There's a discrepancy with the JSON parser, but it's not your fault: the JSON parser represents JSON numbers that fit into int64_t to QInt, and any others as QFloat. In particular, the integers between INT64_MAX+1 and UINT64_MAX are represented as QFloat. qmp_input_type_uint64() accepts only QInt. You can declare things as uint64 in the schema, but you're still limited to 0..UINT64_MAX in QMP.

QemuOpts uses strtoull() directly, in parse_option_number(). Bug (not yours): it happily ignores errors other than "string not fully consumed".

Quoted 5 lines
qapi/qobject-input-visitor.c +2 -0
+}
+
 static void qobject_input_type_bool(Visitor *v, const char *name, bool *obj,
                                     Error **errp)
 {

[...]

#33 lore raw

Re: [Qemu-devel] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor

Quoted 65 lines · Markus Armbruster
Quoted 22 lines · Markus Armbruster
Quoted 3 lines · Daniel P. Berrange

Currently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using

i.e.

Quoted 14 lines

QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.

To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.

[...]

Quoted 40 lines
Quoted 13 lines
qapi/qobject-input-visitor.c +1 -0
index 5ff3db3..cf41df6 100644
@@ -20,6 +20,7 @@
 #include "qemu-common.h"
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/qerror.h"
+#include "qemu/cutils.h"
 
 #define QIV_STACK_SIZE 1024
 

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 273

Quoted 10 lines
qapi/qobject-input-visitor.c +7 -0
@@ -263,6 +264,28 @@ static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj,
     *obj = qint_get_int(qint);
 }
 
+
+static void qobject_input_type_int64_autocast(Visitor *v, const char *name,
+                                              int64_t *obj, Error **errp)
+{
+    QObjectInputVisitor *qiv = to_qiv(v);
+    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                                true));

Needs a rebase for commit 1382d4a. Same elsewhere.

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 276

Quoted 3 lines
qapi/qobject-input-visitor.c +3 -0
+    int64_t ret;
+
+    if (!qstr || !qstr->string) {

I don't think !qstr->string can happen. Same elsewhere.

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 280

Quoted 4 lines
qapi/qobject-input-visitor.c +4 -0
+        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "string");
+        return;
+    }

So far, this is basically qobject_input_type_str() less the g_strdup(). Worth factoring out?

Quoted 7 lines
Quoted 5 lines

Now we're entering out parsing swamp:

Quoted 3 lines
qapi/qobject-input-visitor.c +3 -0
+
+    if (qemu_strtoll(qstr->string, NULL, 0, &ret) < 0) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");

"integer", actually.

Wait! You're using QERR_INVALID_PARAMETER_VALUE here, so it's "an integer".

To be pedantically correct, we'd have to complain about the type when qemu_strtoll() fails to parse, and about the value when it parses okay, but the value is out of range.

Quoted 46 lines
Quoted 46 lines

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 285

Quoted 2 lines
qapi/qobject-input-visitor.c +2 -0
+        return;
+    }

To serve as replacement for the options visitor, this needs to parse exactly like opts_type_int64().

It should also match the JSON parser as far as possible, to minimize difference between the two QObject input visitor variants, and the QemuOpts parser, for command line consistency.

opts_type_int64() uses strtoll() directly. It carefully checks for no conversion (both ways, EINVAL and endptr == str), range, and string not fully consumed.

Your code uses qemu_strtoll(). Bug#1: qemu_strtoll() assumes long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values. Bug#2: your code fails to check the string is fully consumed.

The JSON parser also uses strtoll() directly, in parse_literal(). When we get there, we know that the string consists only of decimal digits, possibly prefixed by a minus sign. Therefore, strtoll() can only fail with ERANGE, and parse_literal() handles that correctly.

QemuOpts doesn't do signed integers.

Quoted 20 lines
qapi/qobject-input-visitor.c +13 -0
+    *obj = ret;
+}
+
 static void qobject_input_type_uint64(Visitor *v, const char *name,
                                       uint64_t *obj, Error **errp)
 {
@@ -279,6 +302,27 @@ static void qobject_input_type_uint64(Visitor *v, const char *name,
     *obj = qint_get_int(qint);
 }
 
+static void qobject_input_type_uint64_autocast(Visitor *v, const char *name,
+                                               uint64_t *obj, Error **errp)
+{
+    QObjectInputVisitor *qiv = to_qiv(v);
+    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                                true));
+    unsigned long long ret;
+
+    if (!qstr || !qstr->string) {
+        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
Quoted 8 lines
Quoted 6 lines

qapi/qobject-input-visitor.c static void qobject_input_type_uint64(Visitor *v, const char *name, · line 320

Quoted 6 lines
qapi/qobject-input-visitor.c +6 -0
+                   "string");
+        return;
+    }
+
+    if (parse_uint_full(qstr->string, &ret, 0) < 0) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");

"integer".

Likewise.

Quoted 38 lines
Quoted 37 lines

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 286

Quoted 3 lines
qapi/qobject-input-visitor.c +3 -0
+        return;
+    }
+    *obj = ret;

Differently wrong :)

To serve as replacement for the options visitor, this needs to parse exactly like opts_type_uint64().

Again, this should also match the JSON parser and the QemuOpts parser as far as possible.

opts_type_uint64() uses parse_uint(). It carefully checks for no conversion (EINVAL; parse_uint() normalizes), range, and string not fully consumed.

You use parse_uint_full(). You therefore don't have bug#2 here. You do have bug#1: you assume unsigned long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values.

There's a discrepancy with the JSON parser, but it's not your fault: the JSON parser represents JSON numbers that fit into int64_t to QInt, and any others as QFloat. In particular, the integers between INT64_MAX+1 and UINT64_MAX are represented as QFloat. qmp_input_type_uint64() accepts only QInt. You can declare things as uint64 in the schema, but you're still limited to 0..UINT64_MAX in QMP.

QemuOpts uses strtoull() directly, in parse_option_number(). Bug (not yours): it happily ignores errors other than "string not fully consumed".

Quoted 5 lines
qapi/qobject-input-visitor.c +2 -0
+}
+
 static void qobject_input_type_bool(Visitor *v, const char *name, bool *obj,
                                     Error **errp)
 {

[...]

#34 lore raw

Re: [Qemu-devel] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor

Quoted 73 lines · Markus Armbruster
Quoted 71 lines · Markus Armbruster
Quoted 22 lines · Markus Armbruster
Quoted 3 lines · Daniel P. Berrange

Currently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using

i.e.

Quoted 14 lines

QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.

To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.

[...]

Quoted 45 lines
Quoted 13 lines
qapi/qobject-input-visitor.c +1 -0
index 5ff3db3..cf41df6 100644
@@ -20,6 +20,7 @@
 #include "qemu-common.h"
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/qerror.h"
+#include "qemu/cutils.h"
 
 #define QIV_STACK_SIZE 1024
 

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 273

Quoted 10 lines
qapi/qobject-input-visitor.c +7 -0
@@ -263,6 +264,28 @@ static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj,
     *obj = qint_get_int(qint);
 }
 
+
+static void qobject_input_type_int64_autocast(Visitor *v, const char *name,
+                                              int64_t *obj, Error **errp)
+{
+    QObjectInputVisitor *qiv = to_qiv(v);
+    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                                true));

Needs a rebase for commit 1382d4a. Same elsewhere.

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 276

Quoted 3 lines
qapi/qobject-input-visitor.c +3 -0
+    int64_t ret;
+
+    if (!qstr || !qstr->string) {

I don't think !qstr->string can happen. Same elsewhere.

qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 280

Quoted 4 lines
qapi/qobject-input-visitor.c +4 -0
+        error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "string");
+        return;
+    }

So far, this is basically qobject_input_type_str() less the g_strdup(). Worth factoring out?

Now we're entering out parsing swamp:

Quoted 3 lines
qapi/qobject-input-visitor.c +3 -0
+
+    if (qemu_strtoll(qstr->string, NULL, 0, &ret) < 0) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
Quoted 8 lines
Quoted 1 line

"integer", actually.

Wait! You're using QERR_INVALID_PARAMETER_VALUE here, so it's "an integer".

To be pedantically correct, we'd have to complain about the type when qemu_strtoll() fails to parse, and about the value when it parses okay, but the value is out of range.

Nevermind, I got confused. The type is actually always string here, so your use of QERR_INVALID_PARAMETER_VALUE is appropriate.

[...]

#58 lore raw

Re: [Qemu-devel] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor

In reply to Daniel P. Berrange
Quoted 88 lines · Daniel P. Berrange

The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:

-arg foo=5,foo=6,foo=7
-arg foo=5-7

This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.

This means that

-arg foo=5-7

is treated as equivalent to

-arg foo.0=5,foo.1=6,foo.2=7

Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.

---
 include/qapi/qobject-input-visitor.h |  23 ++++-
 qapi/qobject-input-visitor.c         | 158 ++++++++++++++++++++++++++--
 tests/test-qobject-input-visitor.c   | 195 +++++++++++++++++++++++++++++++++--
 3 files changed, 360 insertions(+), 16 deletions(-)
include/qapi/qobject-input-visitor.h +22 -1
index 94051f3..242b767 100644
@@ -19,6 +19,12 @@
 
 typedef struct QObjectInputVisitor QObjectInputVisitor;
 
+/* Inclusive upper bound on the size of any flattened range. This is a safety
+ * (= anti-annoyance) measure; wrong ranges should not cause long startup
+ * delays nor exhaust virtual memory.
+ */
+#define QIV_RANGE_MAX 65536
+
 /**
  * Create a new input visitor that converts @obj to a QAPI object.
  *
@@ -71,6 +77,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * The value given determines how many levels of structs are allowed to
  * be flattened in this way.
  *
+ * If @permit_int_ranges is true, then when visiting a list of integers,
+ * the integer value strings may encode ranges eg a single element
+ * containing "5-7" is treated as if there were three elements "5", "6",
+ * "7". This should only be used if compatibility is required with the
+ * OptsVisitor which would allow integer ranges. e.g. set this to true
+ * if you have compatibility requirements for
+ *
+ *   -arg val=5-8
+ *
+ * to be treated as equivalent to the preferred syntax:
+ *
+ *   -arg val.0=5,val.1=6,val.2=7,val.3=8
+ *
  * The visitor always operates in strict mode, requiring all dict keys
  * to be consumed during visitation. An error will be reported if this
  * does not happen.
@@ -80,7 +99,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  */
 Visitor *qobject_input_visitor_new_autocast(QObject *obj,
                                             bool autocreate_list,
-                                            size_t autocreate_struct_levels);
+                                            size_t autocreate_struct_levels,
+                                            bool permit_int_ranges);
 
 
 /**
@@ -98,6 +118,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,
 Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
                                         bool autocreate_list,
                                         size_t autocreate_struct_levels,
+                                        bool permit_int_ranges,
                                         Error **errp);
 
 #endif
qapi/qobject-input-visitor.c +0 -0
index 1be4865..6b3d0f2 100644

qapi/qobject-input-visitor.c typedef struct StackObject · line 35

Quoted 5 lines
qapi/qobject-input-visitor.c +2 -0
@@ -31,6 +31,8 @@ typedef struct StackObject
 
     GHashTable *h;           /* If obj is dict: unvisited keys */
     const QListEntry *entry; /* If obj is list: unvisited tail */
+    uint64_t range_val;
+    uint64_t range_limit;

It's either ugly unions or ugly type casts. The options visitor picked ugly unions, you're picking ugly type casts. Matter of taste, as long as the type casts are all kosher.

Quoted 32 lines
qapi/qobject-input-visitor.c +7 -3
     QSLIST_ENTRY(StackObject) node;
 } StackObject;
@@ -60,6 +62,10 @@ struct QObjectInputVisitor
      * consider auto-creating a struct containing
      * remaining unvisited items */
     size_t autocreate_struct_levels;
+
+    /* Whether int lists can have single values representing
+     * ranges of values */
+    bool permit_int_ranges;
 };
 
 static QObjectInputVisitor *to_qiv(Visitor *v)
@@ -282,7 +288,7 @@ static GenericList *qobject_input_next_list(Visitor *v, GenericList *tail,
     QObjectInputVisitor *qiv = to_qiv(v);
     StackObject *so = QSLIST_FIRST(&qiv->stack);
 
-    if (!so->entry) {
+    if ((so->range_val == so->range_limit) && !so->entry) {
         return NULL;
     }
     tail->next = g_malloc0(size);
@@ -329,21 +335,87 @@ static void qobject_input_type_int64_autocast(Visitor *v, const char *name,
                                               int64_t *obj, Error **errp)
 {
     QObjectInputVisitor *qiv = to_qiv(v);
-    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
-                                                                true));
+    QString *qstr;
     int64_t ret;
+    const char *end = NULL;

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 350

Quoted 10 lines
qapi/qobject-input-visitor.c +10 -0
+    StackObject *tos;
+    bool inlist = false;
+
+    /* Preferentially generate values from a range, before
+     * trying to consume another QList element */
+    tos = QSLIST_FIRST(&qiv->stack);
+    if (tos) {
+        if ((int64_t)tos->range_val < (int64_t)tos->range_limit) {
+            *obj = tos->range_val + 1;
+            tos->range_val++;

Roundabout way to write

               *obj = tos->range_val++;

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 355

Quoted 5 lines
qapi/qobject-input-visitor.c +5 -0
+            return;
+        } else {
+            inlist = tos->entry != NULL;
+        }
+    }

I'd prefer

       tos = QSLIST_FIRST(&qiv->stack);
       inlist = tos && tos->entry;
       if (inlist) {
           if ((int64_t)tos->range_val < (int64_t)tos->range_limit) {
               *obj = tos->range_val++;
               return;
           }
       }

Note for later:

    !tos || (int64_t)tos->range_val >= (int64_t)tos->range_limit

holds here.

Quoted 24 lines
qapi/qobject-input-visitor.c +12 -1
+    qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                       true));
     if (!qstr || !qstr->string) {
         error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
                    "string");
         return;
     }
 
-    if (qemu_strtoll(qstr->string, NULL, 0, &ret) < 0) {
+    if (qemu_strtoll(qstr->string, &end, 0, &ret) < 0) {
         error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
         return;
     }
     *obj = ret;
+
+    /*
+     * If we have string that represents an integer range (5-24),
+     * parse the end of the range and set things up so we'll process
+     * the rest of the range before consuming another element
+     * from the QList.
+     */
+    if (end && *end) {
+        if (!qiv->permit_int_ranges) {

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 388

Quoted 10 lines
qapi/qobject-input-visitor.c +10 -0
+            error_setg(errp,
+                       "Integer ranges are not permitted here");
+            return;
+        }
+        if (!inlist) {
+            error_setg(errp,
+                       "Integer ranges are only permitted when "
+                       "visiting list parameters");
+            return;
+        }

Note for later: since inlist is tos && tos->entry, we know tos is non-null here. Recommend assert(tos) to make it more obvious.

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 391

Quoted 3 lines
qapi/qobject-input-visitor.c +3 -0
+        if (*end != '-') {
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name,
+                       "a number range");

"an integer or integer range", actually.

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 396

Quoted 5 lines
qapi/qobject-input-visitor.c +5 -0
+            return;
+        }
+        end++;
+        if (qemu_strtoll(end, NULL, 0, &ret) < 0) {
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");

"an integer".

Quoted 7 lines
qapi/qobject-input-visitor.c +7 -0
+            return;
+        }
+        if (*obj > ret) {
+            error_setg(errp,
+                       "Parameter '%s' range start %" PRIu64
+                       " must be less than (or equal to) %" PRIu64,
+                       name, *obj, ret);

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 413

Quoted 10 lines
qapi/qobject-input-visitor.c +10 -0
+            return;
+        }
+
+        if ((*obj <= (INT64_MAX - QIV_RANGE_MAX)) &&
+            (ret >= (*obj + QIV_RANGE_MAX))) {
+            error_setg(errp,
+                       "Parameter '%s' range must be less than %d",
+                       name, QIV_RANGE_MAX);
+            return;
+        }

I see you improved the error messages over the option visitor's. Good.

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 417

Quoted 4 lines
qapi/qobject-input-visitor.c +4 -0
+        if (*obj != ret) {
+            tos->range_val = *obj;
+            tos->range_limit = ret;
+        }

Does this need to be conditional?

Dereferencing tos is safe (see note above).

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 417

Quoted 1 line
qapi/qobject-input-visitor.c +1 -0
+    }

No else, thus tos->range_val and tos->range_limit remain unchanged. Works because we know that !tos || tos->range_val >= tos->range_limit (see note above). If it wasn't, the next visit wouldn't consume another object as it should. Considering the distance to the place that ensures this, I'd prefer to spell this out here with an assertion, or perhaps by assigning to tos->range_val and tos->range_limit.

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 418

Quoted 1 line
qapi/qobject-input-visitor.c +0 -0
 }

Overall, less hairy than the options visitor.

Quoted 79 lines
qapi/qobject-input-visitor.c +58 -3
 static void qobject_input_type_uint64(Visitor *v, const char *name,
@@ -366,21 +438,85 @@ static void qobject_input_type_uint64_autocast(Visitor *v, const char *name,
                                                uint64_t *obj, Error **errp)
 {
     QObjectInputVisitor *qiv = to_qiv(v);
-    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
-                                                                true));
+    QString *qstr;
     unsigned long long ret;
+    char *end = NULL;
+    StackObject *tos;
+    bool inlist = false;
+
+    /* Preferentially generate values from a range, before
+     * trying to consume another QList element */
+    tos = QSLIST_FIRST(&qiv->stack);
+    if (tos) {
+        if (tos->range_val < tos->range_limit) {
+            *obj = tos->range_val + 1;
+            tos->range_val++;
+            return;
+        } else {
+            inlist = tos->entry != NULL;
+        }
+    }
 
+    qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                       true));
     if (!qstr || !qstr->string) {
         error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
                    "string");
         return;
     }
 
-    if (parse_uint_full(qstr->string, &ret, 0) < 0) {
+    if (parse_uint(qstr->string, &ret, &end, 0) < 0) {
         error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
         return;
     }
     *obj = ret;
+
+    /*
+     * If we have string that represents an integer range (5-24),
+     * parse the end of the range and set things up so we'll process
+     * the rest of the range before consuming another element
+     * from the QList.
+     */
+    if (end && *end) {
+        if (!qiv->permit_int_ranges) {
+            error_setg(errp,
+                       "Integer ranges are not permitted here");
+            return;
+        }
+        if (!inlist) {
+            error_setg(errp,
+                       "Integer ranges are only permitted when "
+                       "visiting list parameters");
+            return;
+        }
+        if (*end != '-') {
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name,
+                       "a number range");
+            return;
+        }
+        end++;
+        if (parse_uint_full(end, &ret, 0) < 0) {
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
+            return;
+        }
+        if (*obj > ret) {
+            error_setg(errp,
+                       "Parameter '%s' range start %" PRIu64
+                       " must be less than (or equal to) %llu",
+                       name, *obj, ret);
+            return;
+        }
+        if ((ret - *obj) > (QIV_RANGE_MAX - 1)) {
+            error_setg(errp,

qapi/qobject-input-visitor.c static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, · line 520

Quoted 10 lines
qapi/qobject-input-visitor.c +9 -0
+                       "Parameter '%s' range must be less than %d",
+                       name, QIV_RANGE_MAX);
+            return;
+        }
+        if (*obj != ret) {
+            tos->range_val = *obj;
+            tos->range_limit = ret;
+        }
+    }
 }

Duplicates the signed code, which is sad, but I don't have better ideas.

Except this one: are we actually using both the signed and the unsigned case now? If not, can we get rid of the one we don't use?

Quoted 29 lines
qapi/qobject-input-visitor.c +4 -1
 static void qobject_input_type_bool(Visitor *v, const char *name, bool *obj,
@@ -576,7 +712,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)
 
 Visitor *qobject_input_visitor_new_autocast(QObject *obj,
                                             bool autocreate_list,
-                                            size_t autocreate_struct_levels)
+                                            size_t autocreate_struct_levels,
+                                            bool permit_int_ranges)
 {
     QObjectInputVisitor *v;
 
@@ -603,6 +740,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,
     v->strict = true;
     v->autocreate_list = autocreate_list;
     v->autocreate_struct_levels = autocreate_struct_levels;
+    v->permit_int_ranges = permit_int_ranges;
 
     v->root = obj;
     qobject_incref(obj);
@@ -614,6 +752,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,
 Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
                                         bool autocreate_list,
                                         size_t autocreate_struct_levels,
+                                        bool permit_int_ranges,
                                         Error **errp)
 {
     QDict *pdict;

qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 778

Quoted 9 lines
qapi/qobject-input-visitor.c +2 -1
@@ -632,7 +771,8 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
 
     v = qobject_input_visitor_new_autocast(pobj,
                                            autocreate_list,
-                                           autocreate_struct_levels);
+                                           autocreate_struct_levels,
+                                           permit_int_ranges);
  cleanup:
     qobject_decref(pobj);
     QDECREF(pdict);

[Skipping test updates for now...]

#59 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor

Am 12.10.2016 um 17:50 hat Markus Armbruster geschrieben:

Quoted 47 lines
Quoted 32 lines · Daniel P. Berrange

The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:

-arg foo=5,foo=6,foo=7
-arg foo=5-7

This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.

This means that

-arg foo=5-7

is treated as equivalent to

-arg foo.0=5,foo.1=6,foo.2=7

Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.

qapi/qobject-input-visitor.c +2 -2
@@ -329,21 +335,87 @@ static void qobject_input_type_int64_autocast(Visitor *v, const char *name,
                                               int64_t *obj, Error **errp)
 {
     QObjectInputVisitor *qiv = to_qiv(v);
-    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
-                                                                true));
+    QString *qstr;
     int64_t ret;
+    const char *end = NULL;

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 350

Quoted 10 lines
qapi/qobject-input-visitor.c +10 -0
+    StackObject *tos;
+    bool inlist = false;
+
+    /* Preferentially generate values from a range, before
+     * trying to consume another QList element */
+    tos = QSLIST_FIRST(&qiv->stack);
+    if (tos) {
+        if ((int64_t)tos->range_val < (int64_t)tos->range_limit) {
+            *obj = tos->range_val + 1;
+            tos->range_val++;

Roundabout way to write

Quoted 1 line
               *obj = tos->range_val++;

*obj = ++tos->range_val, actually.

Kevin

#65 lore raw

Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options

In reply to Daniel P. Berrange
Quoted 31 lines · Daniel P. Berrange

If given an option string such as

size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind

the qemu_opts_to_qdict() method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:

    size=QString("1024")
    nodes=QString("1-2")
    policy=QString("bind")

With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain

    size=QString("1024")
    nodes=QList([QString("10"),
                 QString("4-5"),
                 QString("1-2")])
    policy=QString("bind")

Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.

Quoted 3 lines

In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.

To serve as a replacement for the options visitor, this needs to be able to behave exactly the same together with a suitably hacked up QObject input visitor. Before I dive into the actual patch, let me summarize QemuOpts and options visitor behavior.

Warning, this is going to get ugly.

QemuOpts faithfully represents a key=value,... string as a list of QemuOpt. Each QemuOpt represents one key=value. They are in the same order. If key occurs multiple times in the string, it occurs just the same in the list.

*Except* key "id" is special: it's stored outside the list, and all but the first one are silently ignored.

Most users only ever get the last value of a key. Any non-last key=value are silently ignored.

We actually exploit this behavior to do defaults, by *prepending* them to the list. See the use of qemu_opts_set_defaults() in main().

A few users get all values of keys (other than key "id"):

* -device, in qdev_device_add() with callback set_property().

We first get "driver" and "bus" normally (silently ignoring non-last values, as usual). All other keys are device properties. To set them, we get all (key, value), ignore keys "driver" and "bus", and set the rest. If a key occurs multiple times, it gets set multiple times. This effectively ignores all but the last one, silently.

* -semihosting-config, in main() with callback add_semihosting_arg().

We first get a bunch of keys normally. Key "arg" is special: it may be repeated to build a list. To implement that, we get all (key, value), ignore keys other than "arg", and accumulate the values.

* -machine & friends, in main() with callback machine_set_property()

Similar to -device, only for machines, with "type" instead of "driver" and "bus".

* -spice, in qemu_spice_init() with callback add_channel()

Keys "tls-channel" and "plaintext-channel" may be used repeated to specify multiple channels. To implement that, we get all (key, value), ignore keys other than "tls-channel" and "plaintext-channel", and set up a channel for each of the others.

* -writeconfig, in config_write_opts() with callback config_write_opt()

We write out all keys in order.

* The options visitor, in opts_start_struct()

We convert the list of (key, value) to a hash table of (key, list of values). Most of the time, the list of values has exactly one element.

When the visitor's user asks for a scalar, we return the last element of the list of values, in lookup_scalar().

When the user asks for list elements, we return the elements of the list of values in order, in opts_next_list(), or if there are none, the empty list in opts_start_list().

Unlike the options visitor, this patch (judging from your description) makes a list only when keys are repeated. The QObject visitor will have to cope with finding both scalars and lists. When it finds a scalar, but needs a list, it'll have to wrap it in a list (PATCH 09, I think). When it finds a list, but needs a scalar, it'll have to fish it out of the list (where is that?).

Quoted 5 lines

All existing callers are all converted to explicitly request the historical behaviour of only reporting the last key. Later patches will make use of the new modes.

Out of steam for today.

#60 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor

In reply to Kevin Wolf
Quoted 48 lines · Kevin Wolf

Am 12.10.2016 um 17:50 hat Markus Armbruster geschrieben:

Quoted 47 lines
Quoted 32 lines · Daniel P. Berrange

The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:

-arg foo=5,foo=6,foo=7
-arg foo=5-7

This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.

This means that

-arg foo=5-7

is treated as equivalent to

-arg foo.0=5,foo.1=6,foo.2=7

Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.

qapi/qobject-input-visitor.c +2 -2
@@ -329,21 +335,87 @@ static void qobject_input_type_int64_autocast(Visitor *v, const char *name,
                                               int64_t *obj, Error **errp)
 {
     QObjectInputVisitor *qiv = to_qiv(v);
-    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
-                                                                true));
+    QString *qstr;
     int64_t ret;
+    const char *end = NULL;

qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 350

Quoted 10 lines
qapi/qobject-input-visitor.c +10 -0
+    StackObject *tos;
+    bool inlist = false;
+
+    /* Preferentially generate values from a range, before
+     * trying to consume another QList element */
+    tos = QSLIST_FIRST(&qiv->stack);
+    if (tos) {
+        if ((int64_t)tos->range_val < (int64_t)tos->range_limit) {
+            *obj = tos->range_val + 1;
+            tos->range_val++;

Roundabout way to write

Quoted 3 lines
Quoted 1 line
               *obj = tos->range_val++;

*obj = ++tos->range_val, actually.

Of course, thanks.

#39 lore raw

Re: [Qemu-devel] [PATCH v14 08/21] qapi: allow QObjectInputVisitor to be created with QemuOpts

In reply to Markus Armbruster
Quoted 18 lines · Markus Armbruster
Quoted 1 line · Daniel P. Berrange

Instead of requiring all callers to go through the mutli-step

multi-step

Quoted 12 lines

process of turning QemuOpts into a suitable QObject for visiting, add a new constructor that encapsulates this logic. This will allow QObjectInputVisitor to be a drop-in replacement for the existing OptsVisitor with minimal code changes for callers.

NB, at this point it is only supporting opts syntax which explicitly matches the QAPI schema structure, so is not yet a true drop-in replacement for OptsVisitor. The patches that follow will add the special cases requird for full backwards compatibility with OptsVisitor.

[...]

Quoted 6 lines
Quoted 6 lines
qapi/qobject-input-visitor.c +0 -0
index cf41df6..d9269c9 100644
@@ -545,3 +545,28 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
 
Quoted 18 lines

qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_autocast(QObject *obj) · line 554

Quoted 9 lines
qapi/qobject-input-visitor.c +7 -0
     return &v->visitor;
 }
+
+
+Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
+                                        Error **errp)
+{
+    QDict *pdict;
+    QObject *pobj = NULL;

@pdict and @pobj are unusual names. Let's stick to the more common @dict and @obj.

Quoted 5 lines
qapi/qobject-input-visitor.c +5 -0
+    Visitor *v = NULL;
+
+    pdict = qemu_opts_to_qdict(opts, NULL);
+    if (!pdict) {
+        goto cleanup;

Returns null without setting an error, which is wrong. Fortunately, qemu_opts_to_qdict() cannot fail. Please drop the broken error handling.

Quoted 13 lines
Quoted 13 lines
qapi/qobject-input-visitor.c +13 -0
+    }
+
+    pobj = qdict_crumple(pdict, true, errp);
+    if (!pobj) {
+        goto cleanup;
+    }
+
+    v = qobject_input_visitor_new_autocast(pobj);
+ cleanup:
+    qobject_decref(pobj);
+    QDECREF(pdict);
+    return v;
+}

[...]

#70 lore raw

Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options

In reply to Daniel P. Berrange
Quoted 23 lines · Daniel P. Berrange

If given an option string such as

size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind

the qemu_opts_to_qdict() method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:

    size=QString("1024")
    nodes=QString("1-2")
    policy=QString("bind")

With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain

    size=QString("1024")
    nodes=QList([QString("10"),
                 QString("4-5"),
                 QString("1-2")])
    policy=QString("bind")
Quoted 7 lines

Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.

Actually three cases, not two:

0. qdict does not contain the key means empty list.

1. qdict contains the key with a QString value means list of one element.

2. qdict contains the key with a QList value means list of more than one element.

I consider this weird. However, it's usefully weird with at least your QObject input visitor.

Quoted 3 lines

In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.

Got users for this policy in the pipeline?

Quoted 50 lines

All existing callers are all converted to explicitly request the historical behaviour of only reporting the last key. Later patches will make use of the new modes.

---
 blockdev.c                   |   7 ++-
 include/qemu/option.h        |  13 ++++-
 monitor.c                    |   3 +-
 qapi/qobject-input-visitor.c |   4 +-
 qemu-img.c                   |   4 +-
 qemu-io-cmds.c               |   3 +-
 qemu-io.c                    |   6 +-
 qemu-nbd.c                   |   3 +-
 qom/object_interfaces.c      |   3 +-
 tests/test-qemu-opts.c       | 132 +++++++++++++++++++++++++++++++++++++++++++
 tests/test-replication.c     |   9 ++-
 util/qemu-option.c           |  64 ++++++++++++++++++---
 12 files changed, 229 insertions(+), 22 deletions(-)
blockdev.c +4 -3
index 814d49f..a999d46 100644
@@ -914,7 +914,8 @@ DriveInfo *drive_new(QemuOpts *all_opts, BlockInterfaceType block_default_type)
 
     /* Get a QDict for processing the options */
     bs_opts = qdict_new();
-    qemu_opts_to_qdict(all_opts, bs_opts);
+    qemu_opts_to_qdict(all_opts, bs_opts, QEMU_OPTS_REPEAT_POLICY_LAST,
+                       &error_abort);
 
     legacy_opts = qemu_opts_create(&qemu_legacy_drive_opts, NULL, 0,
                                    &error_abort);
@@ -3804,8 +3805,8 @@ void hmp_drive_add_node(Monitor *mon, const char *optstr)
         return;
     }
 
-    qdict = qemu_opts_to_qdict(opts, NULL);
-
+    qdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                               &error_abort);
     if (!qdict_get_try_str(qdict, "node-name")) {
         QDECREF(qdict);
         error_report("'node-name' needs to be specified");
include/qemu/option.h +0 -0
index 29f3f18..ffd841d 100644

include/qemu/option.h void qemu_opts_set_defaults(QemuOptsList *list, const char *params, · line 128

Quoted 4 lines
include/qemu/option.h +0 -1
@@ -125,7 +125,18 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char *params,
                             int permit_abbrev);
 QemuOpts *qemu_opts_from_qdict(QemuOptsList *list, const QDict *qdict,
                                Error **errp);
-QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict);

Blank line here, please.

Quoted 34 lines
include/qemu/option.h +12 -0
+typedef enum {
+    /* Last occurrence of a duplicate option silently replaces earlier */
+    QEMU_OPTS_REPEAT_POLICY_LAST,
+    /* Each occurrence of a duplicate option converts value to a QList */
+    QEMU_OPTS_REPEAT_POLICY_ALL,
+    /* First occurrence of a duplicate option causes an error */
+    QEMU_OPTS_REPEAT_POLICY_ERROR,
+} QemuOptsRepeatPolicy;
+
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict,
+                          QemuOptsRepeatPolicy repeatPolicy,
+                          Error **errp);
 void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp);
 
 typedef int (*qemu_opts_loopfunc)(void *opaque, QemuOpts *opts, Error **errp);
monitor.c +2 -1
index 14089cc..84e79d4 100644
@@ -2642,7 +2642,8 @@ static QDict *monitor_parse_arguments(Monitor *mon,
                 if (!opts) {
                     goto fail;
                 }
-                qemu_opts_to_qdict(opts, qdict);
+                qemu_opts_to_qdict(opts, qdict, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                   &error_abort);
                 qemu_opts_del(opts);
             }
             break;
qapi/qobject-input-visitor.c +0 -0
index 6b3d0f2..2287d11 100644

qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 763

Quoted 6 lines
qapi/qobject-input-visitor.c +2 -1
@@ -759,7 +759,9 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
     QObject *pobj = NULL;
     Visitor *v = NULL;
 
-    pdict = qemu_opts_to_qdict(opts, NULL);
+    pdict = qemu_opts_to_qdict(opts, NULL,
+                               QEMU_OPTS_REPEAT_POLICY_LAST,

Join the previous two lines, please.

qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 764

Quoted 1 line
qapi/qobject-input-visitor.c +1 -0
+                               errp);

Unlike the other callers, this one doesn't pass &error_abort. The error handling is actually dead code. Let's pass &error_abort and be done with it.

Quoted 23 lines
qapi/qobject-input-visitor.c +0 -0
     if (!pdict) {
         goto cleanup;
     }
qemu-img.c +3 -1
index 851422a..f47ea75 100644
@@ -273,7 +273,9 @@ static BlockBackend *img_open_opts(const char *optstr,
     QDict *options;
     Error *local_err = NULL;
     BlockBackend *blk;
-    options = qemu_opts_to_qdict(opts, NULL);
+    options = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                 &error_abort);
+
     blk = blk_new_open(NULL, NULL, options, flags, &local_err);
     if (!blk) {
         error_reportf_err(local_err, "Could not open '%s': ", optstr);
qemu-io-cmds.c +0 -0
index 3a3838a..ce654f4 100644

qemu-io-cmds.c static int reopen_f(BlockBackend *blk, int argc, char **argv) · line 1956

Quoted 6 lines
qemu-io-cmds.c +2 -1
@@ -1952,7 +1952,8 @@ static int reopen_f(BlockBackend *blk, int argc, char **argv)
     }
 
     qopts = qemu_opts_find(&reopen_opts, NULL);
-    opts = qopts ? qemu_opts_to_qdict(qopts, NULL) : NULL;
+    opts = qopts ? qemu_opts_to_qdict(qopts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                      &error_abort) : NULL;

Short ternaries are more legible than if statements, but this one is pushing the limit now.

Quoted 8 lines
qemu-io-cmds.c +0 -0
     qemu_opts_reset(&reopen_opts);
 
     brq = bdrv_reopen_queue(NULL, bs, opts, flags);
qemu-io.c +0 -0
index db129ea..bb374a6 100644

qemu-io.c static int open_f(BlockBackend *blk, int argc, char **argv) · line 211

Quoted 6 lines
qemu-io.c +2 -1
@@ -207,7 +207,8 @@ static int open_f(BlockBackend *blk, int argc, char **argv)
     }
 
     qopts = qemu_opts_find(&empty_opts, NULL);
-    opts = qopts ? qemu_opts_to_qdict(qopts, NULL) : NULL;
+    opts = qopts ? qemu_opts_to_qdict(qopts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                      &error_abort) : NULL;

Likewise.

qemu-io.c int main(int argc, char **argv) · line 598

Quoted 10 lines
qemu-io.c +2 -1
     qemu_opts_reset(&empty_opts);
 
     if (optind == argc - 1) {
@@ -593,7 +594,8 @@ int main(int argc, char **argv)
             if (!qopts) {
                 exit(1);
             }
-            opts = qemu_opts_to_qdict(qopts, NULL);
+            opts = qemu_opts_to_qdict(qopts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                      &error_abort);

Even this line's getting ugly. Can we find a prefix that's less loquacious than QEMU_OPTS_REPEAT_POLICY_, but still self-explaining?

Quoted 36 lines
qemu-io.c +0 -0
             openfile(NULL, flags, writethrough, opts);
         } else {
             if (format) {
qemu-nbd.c +2 -1
index 99297a5..73b40b1 100644
@@ -859,7 +859,8 @@ int main(int argc, char **argv)
             qemu_opts_reset(&file_opts);
             exit(EXIT_FAILURE);
         }
-        options = qemu_opts_to_qdict(opts, NULL);
+        options = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                                     &error_abort);
         qemu_opts_reset(&file_opts);
         blk = blk_new_open(NULL, NULL, options, flags, &local_err);
     } else {
qom/object_interfaces.c +2 -1
index ded4d84..fdc406b 100644
@@ -161,7 +161,8 @@ Object *user_creatable_add_opts(QemuOpts *opts, Error **errp)
     Object *obj = NULL;
 
     v = opts_visitor_new(opts);
-    pdict = qemu_opts_to_qdict(opts, NULL);
+    pdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                               &error_abort);
 
     obj = user_creatable_add(pdict, v, errp);
     visit_free(v);
tests/test-qemu-opts.c +0 -0
index a505a3e..3b59978 100644

tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 430

Quoted 10 lines
tests/test-qemu-opts.c +7 -0
@@ -421,6 +421,130 @@ static void test_qemu_opts_set(void)
     g_assert(opts == NULL);
 }
 
+
+static void test_qemu_opts_to_qdict_repeat_last(void)
+{
+    QemuOpts *opts;
+    QDict *dict;
+
+    /* dynamically initialized (parsed) opts */

What are you trying to say with this comment? Hmm, you merely copied it from the existing tests. I'd drop the existing instances of this comment instead.

tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 434

Quoted 4 lines
tests/test-qemu-opts.c +4 -0
+    opts = qemu_opts_parse(&opts_list_03,
+                           "size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind",
+                           false, NULL);
+    g_assert(opts);

You could pass &error_abort.

tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 440

Quoted 6 lines
tests/test-qemu-opts.c +6 -0
+
+    dict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                              &error_abort);
+    g_assert(dict);
+
+

One blank line, not two, please.

tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 446

Quoted 6 lines
tests/test-qemu-opts.c +6 -0
+    g_assert_cmpstr(qdict_get_str(dict, "size"), ==, "1024");
+    g_assert_cmpstr(qdict_get_str(dict, "nodes"), ==, "1-2");
+    g_assert(!qdict_haskey(dict, "nodes.0"));
+    g_assert(!qdict_haskey(dict, "nodes.1"));
+    g_assert(!qdict_haskey(dict, "nodes.2"));
+    g_assert_cmpstr(qdict_get_str(dict, "policy"), ==, "bind");

Checking for absence of "nodes.0"... feels odd. A better idea is to check we got exactly the keys we expect, and no more. Here's a simple way to do that:

       g_assert(qdict_size(dict) == 3);
Quoted 19 lines
tests/test-qemu-opts.c +19 -0
+    QDECREF(dict);
+
+    qemu_opts_del(opts);
+    qemu_opts_reset(&opts_list_03);
+}
+
+
+static void test_qemu_opts_to_qdict_repeat_all(void)
+{
+    QemuOpts *opts;
+    QDict *dict;
+    QList *list;
+    const QListEntry *ent;
+    QString *str;
+
+    /* dynamically initialized (parsed) opts */
+    opts = qemu_opts_parse(&opts_list_03,
+                           "size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind",
+                           false, NULL);

tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 475

Quoted 10 lines
tests/test-qemu-opts.c +10 -0
+    g_assert(opts);
+
+    dict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_ALL,
+                              &error_abort);
+    g_assert(dict);
+
+    g_assert_cmpstr(qdict_get_str(dict, "size"), ==, "1024");
+    g_assert(qdict_haskey(dict, "nodes"));
+    list = qobject_to_qlist(qdict_get(dict, "nodes"));
+    g_assert(list);

Works, but I'd do

       obj = qdict_get(dict, "nodes");
       g_assert(obj);
       list = qobject_to_qlist(obj);
       g_assert(list);

tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 481

Quoted 6 lines
tests/test-qemu-opts.c +6 -0
+
+    ent = qlist_first(list);
+    g_assert(ent);
+    str = qobject_to_qstring(ent->value);
+    g_assert(str);
+    g_assert_cmpstr(qstring_get_str(str), ==, "10");

Since g_assert_cmpstr() does the right thing when @str is null, g_assert(str) is redundant. More of the same below.

Quoted 7 lines
tests/test-qemu-opts.c +7 -0
+
+    ent = qlist_next(ent);
+    g_assert(ent);
+    str = qobject_to_qstring(ent->value);
+    g_assert(str);
+    g_assert_cmpstr(qstring_get_str(str), ==, "4-5");
+

tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 498

Quoted 10 lines
tests/test-qemu-opts.c +10 -0
+    ent = qlist_next(ent);
+    g_assert(ent);
+    str = qobject_to_qstring(ent->value);
+    g_assert(str);
+    g_assert_cmpstr(qstring_get_str(str), ==, "1-2");
+
+    ent = qlist_next(ent);
+    g_assert(!ent);
+
+    g_assert_cmpstr(qdict_get_str(dict, "policy"), ==, "bind");
       g_assert(qdict_size(dict) == 3);
Quoted 33 lines
tests/test-qemu-opts.c +33 -0
+    QDECREF(dict);
+
+    qemu_opts_del(opts);
+    qemu_opts_reset(&opts_list_03);
+}
+
+static void test_qemu_opts_to_qdict_repeat_err_fail(void)
+{
+    QemuOpts *opts;
+    QDict *dict;
+    Error *err = NULL;
+
+    /* dynamically initialized (parsed) opts */
+    opts = qemu_opts_parse(&opts_list_03,
+                           "size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind",
+                           false, NULL);
+    g_assert(opts);
+
+    dict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_ERROR,
+                              &err);
+    error_free_or_abort(&err);
+    g_assert(!dict);
+
+    qemu_opts_del(opts);
+    qemu_opts_reset(&opts_list_03);
+}
+
+static void test_qemu_opts_to_qdict_repeat_err_ok(void)
+{
+    QemuOpts *opts;
+    QDict *dict;
+
+    /* dynamically initialized (parsed) opts */

tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 541

Quoted 10 lines
tests/test-qemu-opts.c +10 -0
+    opts = qemu_opts_parse(&opts_list_03,
+                           "size=1024,nodes=10,policy=bind",
+                           false, NULL);
+    g_assert(opts);
+
+    dict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_ERROR,
+                              &error_abort);
+    g_assert_cmpstr(qdict_get_str(dict, "size"), ==, "1024");
+    g_assert_cmpstr(qdict_get_str(dict, "nodes"), ==, "10");
+    g_assert_cmpstr(qdict_get_str(dict, "policy"), ==, "bind");
       g_assert(qdict_size(dict) == 3);
Quoted 63 lines
tests/test-qemu-opts.c +14 -0
+
+    QDECREF(dict);
+    qemu_opts_del(opts);
+    qemu_opts_reset(&opts_list_03);
+}
+
 int main(int argc, char *argv[])
 {
     register_opts();
@@ -435,6 +559,14 @@ int main(int argc, char *argv[])
     g_test_add_func("/qemu-opts/opt_unset", test_qemu_opt_unset);
     g_test_add_func("/qemu-opts/opts_reset", test_qemu_opts_reset);
     g_test_add_func("/qemu-opts/opts_set", test_qemu_opts_set);
+    g_test_add_func("/qemu-opts/to_qdict/repeat-last",
+                    test_qemu_opts_to_qdict_repeat_last);
+    g_test_add_func("/qemu-opts/to_qdict/repeat-all",
+                    test_qemu_opts_to_qdict_repeat_all);
+    g_test_add_func("/qemu-opts/to_qdict/repeat-err-fail",
+                    test_qemu_opts_to_qdict_repeat_err_fail);
+    g_test_add_func("/qemu-opts/to_qdict/repeat-err-ok",
+                    test_qemu_opts_to_qdict_repeat_err_ok);
     g_test_run();
     return 0;
 }
tests/test-replication.c +6 -3
index 0997bd8..e267f9a 100644
@@ -181,7 +181,8 @@ static BlockBackend *start_primary(void)
     opts = qemu_opts_parse_noisily(&qemu_drive_opts, cmdline, false);
     g_free(cmdline);
 
-    qdict = qemu_opts_to_qdict(opts, NULL);
+    qdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                               &error_abort);
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_DIRECT, "off");
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_NO_FLUSH, "off");
 
@@ -311,7 +312,8 @@ static BlockBackend *start_secondary(void)
     opts = qemu_opts_parse_noisily(&qemu_drive_opts, cmdline, false);
     g_free(cmdline);
 
-    qdict = qemu_opts_to_qdict(opts, NULL);
+    qdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                               &error_abort);
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_DIRECT, "off");
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_NO_FLUSH, "off");
 
@@ -336,7 +338,8 @@ static BlockBackend *start_secondary(void)
     opts = qemu_opts_parse_noisily(&qemu_drive_opts, cmdline, false);
     g_free(cmdline);
 
-    qdict = qemu_opts_to_qdict(opts, NULL);
+    qdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+                               &error_abort);
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_DIRECT, "off");
     qdict_set_default_str(qdict, BDRV_OPT_CACHE_NO_FLUSH, "off");
 
util/qemu-option.c +0 -0
index 418cbb6..0129ede 100644

util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1059

Quoted 3 lines
util/qemu-option.c +0 -0
@@ -1057,23 +1057,73 @@ void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp)
  * The QDict values are of type QString.
  * TODO We'll want to use types appropriate for opt->desc->type, but
  * this is enough for now.

While there, we could document @obj and @dict properly. But see below on @dict.

util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1069

Quoted 10 lines
util/qemu-option.c +10 -0
+ *
+ * If the @opts contains multiple occurrences of the same key,
+ * then the @repeatPolicy parameter determines how they are to
+ * be handled. Traditional behaviour was to only store the
+ * last occurrence, but if @repeatPolicy is set to
+ * QEMU_OPTS_REPEAT_POLICY_ALL, then a QList will be returned
+ * containing all values, for any key with multiple occurrences.
+ * The QEMU_OPTS_REPEAT_POLICY_ERROR value can be used to fail
+ * conversion with an error if multiple occurrences of a key
+ * are seen.

What the traditional behavior was is quite interesting in a commit message. Less so in a comment. Suggest:

* be handled. With @repeatPolicy QEMU_OPTS_REPEAT_POLICY_LAST,
* all values but the last are silently ignored. With
* QEMU_OPTS_REPEAT_POLICY_ALL, a QList will be returned
* containing all values, for any key with multiple occurrences.
* With QEMU_OPTS_REPEAT_POLICY_ERROR, the conversion fails.

Let's spell out that the function can only fail with QEMU_OPTS_REPEAT_POLICY_ERROR:

* This is the only way this function can fail.

util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1076

Quoted 9 lines
util/qemu-option.c +4 -2
  */
-QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict)
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict,
+                          QemuOptsRepeatPolicy repeatPolicy,
+                          Error **errp)
 {
     QemuOpt *opt;
-    QObject *val;
+    QObject *val, *prevval;

I'd prefer prev_val.

util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1084

Quoted 10 lines
util/qemu-option.c +6 -2
+    QList *list;
+    QDict *ret;
 
-    if (!qdict) {
-        qdict = qdict_new();
+    if (qdict) {
+        ret = qdict;
+    } else {
+        ret = qdict_new();
     }

You need this change to get the reference counting on failure right. This makes me question the qdict parameter. I can see two non-null arguments for it outside tests. The one in drive_new() is basically stupid:

    bs_opts = qdict_new();
    qemu_opts_to_qdict(all_opts, bs_opts, QEMU_OPTS_REPEAT_POLICY_LAST,
                       &error_abort);

Easily replaced by

bs_opts = qemu_opts_to_qdict(all_opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
&error_abort);

The one in monitor_parse_arguments()

qemu_opts_to_qdict(opts, qdict, QEMU_OPTS_REPEAT_POLICY_LAST,
&error_abort);

could be replaced by

                QDECREF(qdict);
                qdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
                                   &error_abort);

at a negligible loss of efficiency. We could then drop the parameter, simplifying the function's contract.

Let's insert a blank line here...

util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1087

Quoted 4 lines
util/qemu-option.c +1 -1
     if (opts->id) {
-        qdict_put(qdict, "id", qstring_from_str(opts->id));
+        qdict_put(ret, "id", qstring_from_str(opts->id));
     }

... and here.

Quoted 14 lines
util/qemu-option.c +11 -1
     QTAILQ_FOREACH(opt, &opts->head, next) {
         val = QOBJECT(qstring_from_str(opt->str));
-        qdict_put_obj(qdict, opt->name, val);
+
+        if (qdict_haskey(ret, opt->name)) {
+            switch (repeatPolicy) {
+            case QEMU_OPTS_REPEAT_POLICY_ERROR:
+                error_setg(errp, "Option '%s' appears more than once",
+                           opt->name);
+                qobject_decref(val);
+                if (!qdict) {
+                    QDECREF(ret);
+                }
+                return NULL;

util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1110

Quoted 10 lines
util/qemu-option.c +10 -0
+
+            case QEMU_OPTS_REPEAT_POLICY_ALL:
+                prevval = qdict_get(ret, opt->name);
+                if (qobject_type(prevval) == QTYPE_QLIST) {
+                    /* Already identified this key as a list */
+                    list = qobject_to_qlist(prevval);
+                } else {
+                    /* Replace current scalar with a list */
+                    list = qlist_new();
+                    qobject_incref(prevval);

Where is this reference decremented?

Quoted 4 lines
util/qemu-option.c +4 -0
+                    qlist_append_obj(list, prevval);
+                    qdict_put_obj(ret, opt->name, QOBJECT(list));
+                }
+                qlist_append_obj(list, val);

util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1124

Quoted 10 lines
util/qemu-option.c +10 -0
+                break;
+
+            case QEMU_OPTS_REPEAT_POLICY_LAST:
+                /* Just discard previously set value */
+                qdict_put_obj(ret, opt->name, val);
+                break;
+            }
+        } else {
+            qdict_put_obj(ret, opt->name, val);
+        }

A possible alternative:

           val = QOBJECT(qstring_from_str(opt->str));

           if (qdict_haskey(ret, opt->name)) {
               switch (repeatPolicy) {
               case QEMU_OPTS_REPEAT_POLICY_ERROR:
                   error_setg(errp, "Option '%s' appears more than once",
                              opt->name);
                   qobject_decref(val);
                   if (!qdict) {
                       QDECREF(ret);
                   }
                   return NULL;

               case QEMU_OPTS_REPEAT_POLICY_ALL:
                   prevval = qdict_get(ret, opt->name);
                   if (qobject_type(prevval) == QTYPE_QLIST) {
                       /* Already identified this key as a list */
                       list = qobject_to_qlist(prevval);
                   } else {
                       /* Replace current scalar with a list */
                       list = qlist_new();
                       qobject_incref(prevval);
                       qlist_append_obj(list, prevval);
                   }
                   qlist_append_obj(list, val);
                   val = QOBJECT(list);
                   break;

               case QEMU_OPTS_REPEAT_POLICY_LAST:
                   break;
               }
           }
           qdict_put_obj(ret, opt->name, val);

This shows the common part of the behavior more clearly. Matter of taste, you get to use your artistic license.

Quoted 6 lines
util/qemu-option.c +1 -1
     }
-    return qdict;
+    return ret;
 }
 
 /* Validate parsed opts against descriptions where no
#66 lore raw

Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options

In reply to Markus Armbruster
Quoted 49 lines · Markus Armbruster
Quoted 31 lines · Daniel P. Berrange

If given an option string such as

size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind

the qemu_opts_to_qdict() method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:

    size=QString("1024")
    nodes=QString("1-2")
    policy=QString("bind")

With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain

    size=QString("1024")
    nodes=QList([QString("10"),
                 QString("4-5"),
                 QString("1-2")])
    policy=QString("bind")

Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.

Quoted 3 lines

In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.

To serve as a replacement for the options visitor, this needs to be able to behave exactly the same together with a suitably hacked up QObject input visitor. Before I dive into the actual patch, let me summarize QemuOpts and options visitor behavior.

Warning, this is going to get ugly.

QemuOpts faithfully represents a key=value,... string as a list of QemuOpt. Each QemuOpt represents one key=value. They are in the same order. If key occurs multiple times in the string, it occurs just the same in the list.

Quoted 8 lines

*Except* key "id" is special: it's stored outside the list, and all but the first one are silently ignored.

Most users only ever get the last value of a key. Any non-last key=value are silently ignored.

We actually exploit this behavior to do defaults, by *prepending* them to the list. See the use of qemu_opts_set_defaults() in main().

This prepending of defaults assumes all users ignore values other than the last. It breaks if any user gets non-last values.

Quoted 38 lines

A few users get all values of keys (other than key "id"):

* -device, in qdev_device_add() with callback set_property().

We first get "driver" and "bus" normally (silently ignoring non-last values, as usual). All other keys are device properties. To set them, we get all (key, value), ignore keys "driver" and "bus", and set the rest. If a key occurs multiple times, it gets set multiple times. This effectively ignores all but the last one, silently.

* -semihosting-config, in main() with callback add_semihosting_arg().

We first get a bunch of keys normally. Key "arg" is special: it may be repeated to build a list. To implement that, we get all (key, value), ignore keys other than "arg", and accumulate the values.

* -machine & friends, in main() with callback machine_set_property()

Similar to -device, only for machines, with "type" instead of "driver" and "bus".

* -spice, in qemu_spice_init() with callback add_channel()

Keys "tls-channel" and "plaintext-channel" may be used repeated to specify multiple channels. To implement that, we get all (key, value), ignore keys other than "tls-channel" and "plaintext-channel", and set up a channel for each of the others.

* -writeconfig, in config_write_opts() with callback config_write_opt()

We write out all keys in order.

* The options visitor, in opts_start_struct()

We convert the list of (key, value) to a hash table of (key, list of values). Most of the time, the list of values has exactly one element.

Quoted 6 lines

When the visitor's user asks for a scalar, we return the last element of the list of values, in lookup_scalar().

When the user asks for list elements, we return the elements of the list of values in order, in opts_next_list(), or if there are none, the empty list in opts_start_list().

Note that the only way to get non-last values is to iterate over all (key, value). The combination of "getting a specific key's value gets the last one" and "iterating over all keys gets all values" is poor interface design. The latter feature got pressed into service to do list-valued keys. When qemu_opts_set_defaults() got added (commit 4f6dd9a), the bad interaction with the list-valued keys hack wasn't considered, probably because the whole thing had become too byzantine to fully understand.

Quoted 14 lines

Unlike the options visitor, this patch (judging from your description) makes a list only when keys are repeated. The QObject visitor will have to cope with finding both scalars and lists. When it finds a scalar, but needs a list, it'll have to wrap it in a list (PATCH 09, I think). When it finds a list, but needs a scalar, it'll have to fish it out of the list (where is that?).

Quoted 5 lines

All existing callers are all converted to explicitly request the historical behaviour of only reporting the last key. Later patches will make use of the new modes.

Out of steam for today.

#74 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

In reply to Daniel P. Berrange
Cc Kevin for discussion of QemuOpts dotted key convention
Quoted 12 lines · Daniel P. Berrange

Currently qdict_crumple requires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.

In order to have backwards compatibility with the OptsVisitor, qemu_opt_to_qdict() has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given to qdict_crumple().

To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed

Quoted 6 lines
   foo=hello,foo=world,foo=wibble
   foo.0=hello,foo.1=world,foo.2=wibble

but this is forbidden

foo=hello,foo=world,foo.2=wibble

I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.

The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?

Note that this second way creates entirely new failure modes and restrictions. Let me show using an example derived from one in qdict_crumple()'s contract:

foo.0.bar=bla,foo.eek.bar=blubb

Without the dotted key convention, this is perfectly fine: key "foo.0.bar" has the single value "bla", and key "foo.eek.bar" has the single value "blubb". Equivalent JSON would be

{ "foo.0.bar": "bla", "foo.eek.bar": "blubb" }

With just the struct convention, it's still fine: it obviously means the same as JSON

{ "foo": { "0": { "bar": "bla" }, "eek": { "bar": "blubb" } } }

Adding the list convention makes it invalid. It also outlaws a bunch of keys that would be just fine in JSON, namely any that get recognized as list index. Raise your hand if you're willing to bet real money on your predictions of what will be recognized as list index, without looking at the code. I'm not.

I'm afraid I have growing doubts regarding the QemuOpts dotted key convention in general.

The convention makes '.' a special character in keys, but only sometimes. If the key gets consumed by something that uses dotted key convention, '.' is special, and to get a non-special '.', you need to escape it by doubling. Else, it's not.

Since the same key can be used differently by different code, the same '.' could in theory be both special and non-special. In practice, this would be madness.

Adopting the dotted key convention for an existing QemuOpts option, say -object [PATCH 15], *breaks* existing command line usage of keys containing '.', because you now have to escape the '.'. Dan, I'm afraid you need to show that no such keys exist, or if they exist they don't matter.

I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.

It's probably too late to back out the dotted key convention completely. Kevin?

Can we still back out the list part of the convention, and use repeated keys instead?

If we're stuck with some form of the dotted key convention, can we at least make it a more integral part of QemuOpts rather than something bolted on as an afterthought? Here's my thinking on how that might be done:

* Have a QemuOptsList flag @flat.

* If @flat, QemuOpts behaves as it always has: the special characters are ',' and '=', and parsing a key=value,... string produces a list where each element represents one key=value from the string, in the same order.

* If not @flat, '.' becomes an additional special character, and parsing a key=value,... string produces a dictionary, similar to the one you get now by converting with qemu_opts_to_qdict() and filtering through qdict_crumple().

The difference to now is that you either always crumple, or not at all, and the meaning of '.' is unambiguous.

I wish we had refrained from saddling QemuOpts with even more magic. Compared to this swamp, use of JSON on the command line looks rather appealing to me.

#75 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

Am 13.10.2016 um 14:35 hat Markus Armbruster geschrieben:

Quoted 21 lines
Cc Kevin for discussion of QemuOpts dotted key convention
Quoted 17 lines · Daniel P. Berrange

Currently qdict_crumple requires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.

In order to have backwards compatibility with the OptsVisitor, qemu_opt_to_qdict() has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given to qdict_crumple().

To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed

   foo=hello,foo=world,foo=wibble
   foo.0=hello,foo.1=world,foo.2=wibble

but this is forbidden

Quoted 8 lines
Quoted 1 line

foo=hello,foo=world,foo.2=wibble

I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.

The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?

Probably primarily because someone didn't realise this when introducing the dotted syntax. Also because flat QDicts can't represent this.

But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.

Compare this:

    driver=quorum,
    child.0.driver=file,child.0.filename=disk1.img,
    child.1.driver=host_device,child.1.filename=/dev/sdb,
    child.2.driver=nbd,child.2.host=localhost

And this:

driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost

For starters, can we really trust the order in QemuOpts so that the right driver and filename are associated with each other?

We would also have code to associate the third child.driver with the first child.host (because file and host_device don't have a host property). And this isn't even touching optional arguments yet. Would you really want to implement or review this?

Quoted 15 lines

Note that this second way creates entirely new failure modes and restrictions. Let me show using an example derived from one in qdict_crumple()'s contract:

foo.0.bar=bla,foo.eek.bar=blubb

Without the dotted key convention, this is perfectly fine: key "foo.0.bar" has the single value "bla", and key "foo.eek.bar" has the single value "blubb". Equivalent JSON would be

{ "foo.0.bar": "bla", "foo.eek.bar": "blubb" }

With just the struct convention, it's still fine: it obviously means the same as JSON

Quoted 7 lines

{ "foo": { "0": { "bar": "bla" }, "eek": { "bar": "blubb" } } }

Adding the list convention makes it invalid. It also outlaws a bunch of keys that would be just fine in JSON, namely any that get recognized as list index. Raise your hand if you're willing to bet real money on your predictions of what will be recognized as list index, without looking at the code. I'm not.

If you're adding dict keys to the schema that could be reasonably parsed as a list index, I think you're doing something wrong. I would agree that this is a problem if we were talking about user-supplied values, but it's just keys that are defined by qemu.

Quoted 7 lines

I'm afraid I have growing doubts regarding the QemuOpts dotted key convention in general.

The convention makes '.' a special character in keys, but only sometimes. If the key gets consumed by something that uses dotted key convention, '.' is special, and to get a non-special '.', you need to escape it by doubling. Else, it's not.

Do we really implement escaping by doubling dots? This would be news to me.

Do we even have a reason to escape dots, i.e. are there any options in the schema whose keys contain a dot? I think we took care to avoid such things.

Quoted 10 lines

Since the same key can be used differently by different code, the same '.' could in theory be both special and non-special. In practice, this would be madness.

Adopting the dotted key convention for an existing QemuOpts option, say -object [PATCH 15], *breaks* existing command line usage of keys containing '.', because you now have to escape the '.'. Dan, I'm afraid you need to show that no such keys exist, or if they exist they don't matter.

Quoted 3 lines

I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.

I wanted to have a look at this example, but I can only find the string "ide.0" used as a bus name in the sources, that is, a value rather than a key.

Do you have a pointer to the property definition that you mean?

Quoted 2 lines

It's probably too late to back out the dotted key convention completely. Kevin?

Yes, far too late.

Also, I haven't yet seen even just an idea for an alternative solution.

Quoted 2 lines

Can we still back out the list part of the convention, and use repeated keys instead?

See above, repeated keys are messy and can't provide the same thing.

Also it's been in use since quorum was introduced in commit c88a1de51, February 2014. Pretty sure that this means no anyway.

Quoted 12 lines

If we're stuck with some form of the dotted key convention, can we at least make it a more integral part of QemuOpts rather than something bolted on as an afterthought? Here's my thinking on how that might be done:

* Have a QemuOptsList flag @flat.

* If @flat, QemuOpts behaves as it always has: the special characters are ',' and '=', and parsing a key=value,... string produces a list where each element represents one key=value from the string, in the same order.

Quoted 7 lines

* If not @flat, '.' becomes an additional special character, and parsing a key=value,... string produces a dictionary, similar to the one you get now by converting with qemu_opts_to_qdict() and filtering through qdict_crumple().

The difference to now is that you either always crumple, or not at all, and the meaning of '.' is unambiguous.

Integrating qdict_crumple() functionality in QemuOpts sounds like a reasonable next step after Dan's series.

Quoted 3 lines

I wish we had refrained from saddling QemuOpts with even more magic. Compared to this swamp, use of JSON on the command line looks rather appealing to me.

I'd bet most users would disagree with you because they use only simple cases.

In more complicated setups, dotted syntax still works (unlike repeated options), but becomes a bit verbose because of the repetition of long prefixes. If we wanted to attack this, we would have to introduce two more special characters (brackets) for a more concise nesting syntax.

Kevin

#76 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

Quoted 27 lines · Kevin Wolf

Am 13.10.2016 um 14:35 hat Markus Armbruster geschrieben:

Quoted 26 lines
Cc Kevin for discussion of QemuOpts dotted key convention
Quoted 12 lines · Daniel P. Berrange

Currently qdict_crumple requires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.

In order to have backwards compatibility with the OptsVisitor, qemu_opt_to_qdict() has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given to qdict_crumple().

To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed

Quoted 6 lines
   foo=hello,foo=world,foo=wibble
   foo.0=hello,foo.1=world,foo.2=wibble

but this is forbidden

foo=hello,foo=world,foo.2=wibble

I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.

Quoted 6 lines
Quoted 3 lines

The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?

Probably primarily because someone didn't realise this when introducing the dotted syntax.

Can't even blame "someone" for that; it's an obscure, underdocumented feature of an interface that's collapsing under its load of obscure, underdocumented features.

On the other hand, that's not exactly a state that allows for *more* obscure features.

Quoted 1 line

Also because flat QDicts can't represent this.

Explain?

Quoted 11 lines

But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.

Compare this:

    driver=quorum,
    child.0.driver=file,child.0.filename=disk1.img,
    child.1.driver=host_device,child.1.filename=/dev/sdb,
    child.2.driver=nbd,child.2.host=localhost
Quoted 6 lines

And this:

driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost

Aside: both are about equally illegible, to be honest.

Quoted 2 lines

For starters, can we really trust the order in QemuOpts so that the right driver and filename are associated with each other?

The order is trustworthy, but...

Quoted 4 lines

We would also have code to associate the third child.driver with the first child.host (because file and host_device don't have a host property). And this isn't even touching optional arguments yet. Would you really want to implement or review this?

... you're right, doing lists by repeating keys breaks down when combined with the dotted key convention's use of repetition to do structured values.

Permit me to digress.

QemuOpts wasn't designed for list-values keys. Doing lists by repetition was clever.

QemuOpts wasn't designed for structured values. Doing structured values by a dotted key convention plus repetition was clever.

And there's the problem: too much cleverness, not enough "this is being pushed way beyond its design limits, time to replace it".

For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:

    { "driver": "quorum",
      "child": [ { "driver": "file", "filename": "disk1.img" },
                 { "driver": "host_device", "filename=/dev/sdb" },
                 { "driver": "nbd", "host": "localhost" } ] }

Note how the structure is obvious. It isn't with dotted keys, not even if you order them sensibly (which users inevitably won't).

Also not that the value needs to be parsed by QemuOpts! You can't leave it to the consumer of the QemuOpts, because if you did, you'd have to escape the commas.

If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try

driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]

I'd go with some existing syntax, though. The one we already use is JSON.

End of digression.

Quoted 23 lines
Quoted 15 lines

Note that this second way creates entirely new failure modes and restrictions. Let me show using an example derived from one in qdict_crumple()'s contract:

foo.0.bar=bla,foo.eek.bar=blubb

Without the dotted key convention, this is perfectly fine: key "foo.0.bar" has the single value "bla", and key "foo.eek.bar" has the single value "blubb". Equivalent JSON would be

{ "foo.0.bar": "bla", "foo.eek.bar": "blubb" }

With just the struct convention, it's still fine: it obviously means the same as JSON

Quoted 7 lines

{ "foo": { "0": { "bar": "bla" }, "eek": { "bar": "blubb" } } }

Adding the list convention makes it invalid. It also outlaws a bunch of keys that would be just fine in JSON, namely any that get recognized as list index. Raise your hand if you're willing to bet real money on your predictions of what will be recognized as list index, without looking at the code. I'm not.

Quoted 4 lines

If you're adding dict keys to the schema that could be reasonably parsed as a list index, I think you're doing something wrong. I would agree that this is a problem if we were talking about user-supplied values, but it's just keys that are defined by qemu.

Your argument would be stronger if we didn't have such a disappointing track record on sensible naming. We've failed at picking a consistent rules for names, let alone enforcing them. Instead, we have local rules, enforced to varying degrees, and of course inconsistent with each other. We even have code to mangle names, so that they conform to some other set of rules.

We should not extend an interface in a way that makes it rely on certain rules for names without writing down and enforcing these rules.

You did it *locally* for block stuff, and got away with it, because you control the keys (hopefully), and nobody (including probably you and very much me) wants to think about this particular mess unless it's absolutely, obviously, embarrassingly necessary.

Which is what he get to do now, because now we're discussing to adopt your local thing for general use, with keys no single person really knows, let alone controls.

Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.

We can avoid rule 2 by requiring '.' to be escaped. Dan's qdict_crumple() currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.

The names in question are the names declared in QemuOptDesc (easy enough to find) plus any names that the code may use with empty QemuOptDesc (harder). Empty QemuOptDesc are used with the following option groups:

* "drive", qemu_drive_opts in blockdev.c
Also empty_opts in qemu-io.c and qemu_drive_opts in test-replication.c

* "numa", qemu_numa_opts in numa.c

* "device", qemu_device_opts in qdev-monitor.c

This one pulls in qdev properties. Properties violating rule 2 exist.

* "object", qemu_object_opts in vl.c
Also qemu_object_opts in qemu-img.c, qemu-io.c and qemu-nbd.c (WTF?)

This one pulls in properties of user-creatable objects.

* "source", qemu_soource_opts (sic!) in qemu-img.c

* "reopen", reopen_opts in qemu-io-cmds.c

* "file", file_opts in qemu-io.c and qemu-nbd.c

* "machine", qemu_machine_opts in vl.c

This one pulls in machine properties.

* "tpmdev", qemu_tpmdev_opts in vl.c

* "netdev", qemu_netdev_opts in net.c

* "net", qemu_net_opts in net.c

* "userdef", userdef_opts in tests-opts-visitor.c

* "opts_list_03", opts_list_03 in test-qemu-opts.c

* "acpi", qemu_acpi_opts in hw/acpi/core.c

* "smbios", qemu_smbios_opts in smbios.c

Empty QemuOptDesc is used this widely because QemuOpts is too primitive to support the needs of its users. Unlike a QAPI schema. Just sayin'.

Finding the names and assessing how they're impacted by the dotted key convention is going to be a substantial chunk of work.

Quoted 10 lines
Quoted 7 lines

I'm afraid I have growing doubts regarding the QemuOpts dotted key convention in general.

The convention makes '.' a special character in keys, but only sometimes. If the key gets consumed by something that uses dotted key convention, '.' is special, and to get a non-special '.', you need to escape it by doubling. Else, it's not.

Do we really implement escaping by doubling dots? This would be news to me.

Dan's PATCH 02/21 does.

Quoted 3 lines

Do we even have a reason to escape dots, i.e. are there any options in the schema whose keys contain a dot? I think we took care to avoid such things.

I'm afraid we did not.

Quoted 14 lines
Quoted 10 lines

Since the same key can be used differently by different code, the same '.' could in theory be both special and non-special. In practice, this would be madness.

Adopting the dotted key convention for an existing QemuOpts option, say -object [PATCH 15], *breaks* existing command line usage of keys containing '.', because you now have to escape the '.'. Dan, I'm afraid you need to show that no such keys exist, or if they exist they don't matter.

Quoted 3 lines

I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.

Quoted 5 lines

I wanted to have a look at this example, but I can only find the string "ide.0" used as a bus name in the sources, that is, a value rather than a key.

Do you have a pointer to the property definition that you mean?

We've gotten better at hiding property definitions...

"qemu-system-ppc -device macio-ide,help" shows the property:

    macio-ide.ide.0=child<IDE>

The device is not marked no-user, but the obvious attempt to use it fails:

    $ qemu-system-ppc -device macio-ide
    qemu-system-ppc: Option '-device macio-ide' cannot be handled by this machine

This is the Alex's dynamic system bus thingy, which put most of the system bus devices in help even when they cannot be used.

Perhaps you can make a case why this property cannot be used via QemuOpts. If so, great! There are about three dozen more for you to check, followed by fourteen more option groups with empty QemuOptDesc, and a bunch of non-empty QemuOptDesc ;-P

Quoted 4 lines
Quoted 2 lines

It's probably too late to back out the dotted key convention completely. Kevin?

Yes, far too late.

If we're stuck with the existing use of dotted key convention, we're stuck with it. But that's no excuse for adding more uses.

If we decide we want to use dotted key convention more widely, we must document and enforce the naming conventions it requires at least in all the places where we use it. As discussed above, this might involve compatibility breaks, hopefully minor.

Quoted 1 line

Also, I haven't yet seen even just an idea for an alternative solution.

See digression above.

Quoted 31 lines
Quoted 2 lines

Can we still back out the list part of the convention, and use repeated keys instead?

See above, repeated keys are messy and can't provide the same thing.

Also it's been in use since quorum was introduced in commit c88a1de51, February 2014. Pretty sure that this means no anyway.

Quoted 12 lines

If we're stuck with some form of the dotted key convention, can we at least make it a more integral part of QemuOpts rather than something bolted on as an afterthought? Here's my thinking on how that might be done:

* Have a QemuOptsList flag @flat.

* If @flat, QemuOpts behaves as it always has: the special characters are ',' and '=', and parsing a key=value,... string produces a list where each element represents one key=value from the string, in the same order.

Quoted 7 lines

* If not @flat, '.' becomes an additional special character, and parsing a key=value,... string produces a dictionary, similar to the one you get now by converting with qemu_opts_to_qdict() and filtering through qdict_crumple().

The difference to now is that you either always crumple, or not at all, and the meaning of '.' is unambiguous.

Integrating qdict_crumple() functionality in QemuOpts sounds like a reasonable next step after Dan's series.

Quoted 6 lines
Quoted 3 lines

I wish we had refrained from saddling QemuOpts with even more magic. Compared to this swamp, use of JSON on the command line looks rather appealing to me.

I'd bet most users would disagree with you because they use only simple cases.

Use of JSON on the command line when and where you need nesting, not everywhere. The simple cases can stay exactly as they are.

Quoted 6 lines

In more complicated setups, dotted syntax still works (unlike repeated options), but becomes a bit verbose because of the repetition of long prefixes. If we wanted to attack this, we would have to introduce two more special characters (brackets) for a more concise nesting syntax.

Kevin

#77 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

Quoted 18 lines

For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:

    { "driver": "quorum",
      "child": [ { "driver": "file", "filename": "disk1.img" },
                 { "driver": "host_device", "filename=/dev/sdb" },
                 { "driver": "nbd", "host": "localhost" } ] }

Note how the structure is obvious. It isn't with dotted keys, not even if you order them sensibly (which users inevitably won't).

Also not that the value needs to be parsed by QemuOpts! You can't leave it to the consumer of the QemuOpts, because if you did, you'd have to escape the commas.

If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try

Quoted 7 lines

driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]

I'd go with some existing syntax, though. The one we already use is JSON.

In fact there is already "filename=json:{...}" support in the block layer. By the way, abuse of QemuOpts dates back to http://wiki.qemu.org/Features/QCFG.

Quoted 9 lines

Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.

We can avoid rule 2 by requiring '.' to be escaped. Dan's qdict_crumple() currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.

Quoted 3 lines

* "device", qemu_device_opts in qdev-monitor.c

This one pulls in qdev properties. Properties violating rule 2 exist.

Which are they? Only bus names?

Quoted 16 lines

* "object", qemu_object_opts in vl.c

This one pulls in properties of user-creatable objects.

* "machine", qemu_machine_opts in vl.c

This one pulls in machine properties.

Quoted 8 lines
Quoted 3 lines

I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.

I wanted to have a look at this example, but I can only find the string "ide.0" used as a bus name in the sources, that is, a value rather than a key.

Quoted 7 lines
Quoted 1 line

Do you have a pointer to the property definition that you mean?

We've gotten better at hiding property definitions...

"qemu-system-ppc -device macio-ide,help" shows the property:

    macio-ide.ide.0=child<IDE>

It is a bug that this property is shown in the help, because it's not assignable (same for all other child<> properties). I'd rather declare other occurrences of "." in user-accessible property names to be bugs, and break the ABI if there are any.

Paolo

#79 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

In reply to Markus Armbruster
Quoted 31 lines · Markus Armbruster
Quoted 11 lines

But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.

Compare this:

    driver=quorum,
    child.0.driver=file,child.0.filename=disk1.img,
    child.1.driver=host_device,child.1.filename=/dev/sdb,
    child.2.driver=nbd,child.2.host=localhost
Quoted 6 lines

And this:

driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost

Aside: both are about equally illegible, to be honest.

Permit me to digress.

QemuOpts wasn't designed for list-values keys. Doing lists by repetition was clever.

QemuOpts wasn't designed for structured values. Doing structured values by a dotted key convention plus repetition was clever.

And there's the problem: too much cleverness, not enough "this is being pushed way beyond its design limits, time to replace it".

Quoted 7 lines

For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:

    { "driver": "quorum",
      "child": [ { "driver": "file", "filename": "disk1.img" },
                 { "driver": "host_device", "filename=/dev/sdb" },
                 { "driver": "nbd", "host": "localhost" } ] }

Possible hack solution:

QemuOpts already special-cases id=. What if we ALSO make it special-case a leading json=? Shown here with shell quoting, the above example of creating a Quorum -drive argument could then be:

-drive json='

    { "driver": "quorum",
      "child": [ { "driver": "file", "filename": "disk1.img" },
                 { "driver": "host_device", "filename=/dev/sdb" },
                 { "driver": "nbd", "host": "localhost" } ] }

'

As far as I know, we don't have 'json' as any existing QemuOpts key (do we? A full audit may be better than my quick git grep '"json"'). Thus, if QemuOpts sees a leading json=, it hands off the rest of the string to the same parser as we use for QMP, where we no longer have to escape commas (even nicer than the drive hack where we support filename=json:{...} but have to double up all commas to make it through the QemuOpts layer). Encountering json= as anything other than the first option would be an error, and you would be unable to combine a json= option with any other old-style option. In other words, the use of leading json= would be the switch for whether to do old-style parsing or to use a saner syntax for everything else that needs structure, on a per-argument basis.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#78 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

In reply to Paolo Bonzini
Quoted 23 lines · Paolo Bonzini
Quoted 23 lines

For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:

    { "driver": "quorum",
      "child": [ { "driver": "file", "filename": "disk1.img" },
                 { "driver": "host_device", "filename=/dev/sdb" },
                 { "driver": "nbd", "host": "localhost" } ] }

Note how the structure is obvious. It isn't with dotted keys, not even if you order them sensibly (which users inevitably won't).

Also not that the value needs to be parsed by QemuOpts! You can't leave it to the consumer of the QemuOpts, because if you did, you'd have to escape the commas.

If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try

driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]

Quoted 4 lines
Quoted 2 lines

I'd go with some existing syntax, though. The one we already use is JSON.

In fact there is already "filename=json:{...}" support in the block layer.

Yes.

Quoted 1 line

By the way, abuse of QemuOpts dates back to http://wiki.qemu.org/Features/QCFG.

That page is from 2011, when QAPI was being implemented. The idea to bring the power of the QAPI schema to bear on the command line is sound, but QCFG never made it past the ideas stage, I'm afraid. It uses either dotted keys or braces for nested structs. It doesn't cover lists.

Quoted 9 lines
Quoted 8 lines

Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.

We can avoid rule 2 by requiring '.' to be escaped. Dan's qdict_crumple() currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.

Quoted 5 lines
Quoted 3 lines

* "device", qemu_device_opts in qdev-monitor.c

This one pulls in qdev properties. Properties violating rule 2 exist.

Which are they? Only bus names?

Finding properties is difficult, because we (foolishly!) define them in code rather than data. My testing with "info qdm" for all targets finds:

    = xlnx.axi-dma =
    xlnx.axi-dma[0]
    = xlnx.xps-ethernetlite =
    xlnx.xps-ethernetlite[0]
    = xlnx.xps-intc =
    xlnx.xps-intc[0]
    = xlnx.xps-uartlite =
    xlnx.xps-uartlite[0]
    = xlnx.axi-dma =
    xlnx.axi-dma[0]
    = xlnx.xps-ethernetlite =
    xlnx.xps-ethernetlite[0]
    = xlnx.xps-intc =
    xlnx.xps-intc[0]
    = xlnx.xps-uartlite =
    xlnx.xps-uartlite[0]
    = cuda =
    adb.0
    = raven-pcihost =
    pci.0
    = macio-ide =
    ide.0
    = mpc8544-guts =
    mpc8544.guts[0]
    = xlnx.xps-ethernetlite =
    xlnx.xps-ethernetlite[0]
    = xlnx.xps-intc =
    xlnx.xps-intc[0]
    = xlnx.xps-uartlite =
    xlnx.xps-uartlite[0]
    = cuda =
    adb.0
    = raven-pcihost =
    pci.0
    = macio-ide =
    ide.0
    = mpc8544-guts =
    mpc8544.guts[0]
    = xlnx.xps-ethernetlite =
    xlnx.xps-ethernetlite[0]
    = xlnx.xps-intc =
    xlnx.xps-intc[0]
    = xlnx.xps-uartlite =
    xlnx.xps-uartlite[0]
    = s390-sclp-event-facility =
    s390-sclp-events-bus.0
    = cgthree =
    cg3.reg[0]
    cg3.prom[0]
    = SUNW,tcx =
    tcx.thc[0]
    tcx.rblit[0]
    tcx.dac[0]
    tcx.alt[0]
    tcx.tec[0]
    tcx.rstip[0]
    tcx.blit[0]
    tcx.dhc[0]
    tcx.prom[0]
    tcx.stip[0]
Quoted 18 lines
Quoted 18 lines

* "object", qemu_object_opts in vl.c

This one pulls in properties of user-creatable objects.

* "machine", qemu_machine_opts in vl.c

This one pulls in machine properties.

Quoted 9 lines
Quoted 3 lines

I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.

I wanted to have a look at this example, but I can only find the string "ide.0" used as a bus name in the sources, that is, a value rather than a key.

Do you have a pointer to the property definition that you mean?

Quoted 8 lines
Quoted 5 lines

We've gotten better at hiding property definitions...

"qemu-system-ppc -device macio-ide,help" shows the property:

    macio-ide.ide.0=child<IDE>

It is a bug that this property is shown in the help, because it's not assignable (same for all other child<> properties).

Let's fix it then.

Quoted 3 lines

I'd rather declare
other occurrences of "." in user-accessible property names to be bugs, and break the ABI if there are any.

I propose to adopt QAPI's rules on permissible names: "All names must begin with a letter, and contain only ASCII letters, digits, dash, and underscore" (docs/qapi-code-gen.txt). QAPI's naming rules get adopted anyway on QAPIfication, so why wait? :)

Note that this may affect names generated by automatic arrayification (commit 3396590), because those contain [ and ].

#81 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

In reply to Markus Armbruster

Am 17.10.2016 um 16:50 hat Markus Armbruster geschrieben:

Quoted 36 lines
Quoted 27 lines · Kevin Wolf

Am 13.10.2016 um 14:35 hat Markus Armbruster geschrieben:

Quoted 26 lines
Cc Kevin for discussion of QemuOpts dotted key convention
Quoted 12 lines · Daniel P. Berrange

Currently qdict_crumple requires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.

In order to have backwards compatibility with the OptsVisitor, qemu_opt_to_qdict() has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given to qdict_crumple().

To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed

Quoted 6 lines
   foo=hello,foo=world,foo=wibble
   foo.0=hello,foo.1=world,foo.2=wibble

but this is forbidden

foo=hello,foo=world,foo.2=wibble

I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.

Quoted 6 lines
Quoted 3 lines

The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?

Probably primarily because someone didn't realise this when introducing the dotted syntax.

Quoted 6 lines

Can't even blame "someone" for that; it's an obscure, underdocumented feature of an interface that's collapsing under its load of obscure, underdocumented features.

On the other hand, that's not exactly a state that allows for *more* obscure features.

I don't really think we're introducing more obscure features here, but rather trying to implement a single, and rather straightforward, way as the standard.

Dotted syntax for hierarchy has actually plenty of precedence in qemu if you look a bit closer (the block layer, -global, -device foo,help, even the bus names you mentioned below are really just flattened lists), so we're only making things more consistent.

Quoted 3 lines
Quoted 1 line

Also because flat QDicts can't represent this.

Explain?

Repeated options are parsed into QLists. If you keep it at that without flattening you have at least a QDict that contains a QList that contains simple types. This is not flat any more.

Of course, you could argue that flat QDicts are the wrong data structure in the first place and instead of flatting everything we should have done the equivalent of qdict_crumple from the beginning, but they were the natural extension of what was already there and happened to work good enough, so this is what we're currently using.

Quoted 11 lines
Quoted 11 lines

But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.

Compare this:

    driver=quorum,
    child.0.driver=file,child.0.filename=disk1.img,
    child.1.driver=host_device,child.1.filename=/dev/sdb,
    child.2.driver=nbd,child.2.host=localhost
Quoted 8 lines
Quoted 6 lines

And this:

driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost

Aside: both are about equally illegible, to be honest.

Not sure about equally, but let's agree on "both are illegible".

Quoted 33 lines
Quoted 2 lines

For starters, can we really trust the order in QemuOpts so that the right driver and filename are associated with each other?

The order is trustworthy, but...

Quoted 4 lines

We would also have code to associate the third child.driver with the first child.host (because file and host_device don't have a host property). And this isn't even touching optional arguments yet. Would you really want to implement or review this?

... you're right, doing lists by repeating keys breaks down when combined with the dotted key convention's use of repetition to do structured values.

Permit me to digress.

QemuOpts wasn't designed for list-values keys. Doing lists by repetition was clever.

QemuOpts wasn't designed for structured values. Doing structured values by a dotted key convention plus repetition was clever.

And there's the problem: too much cleverness, not enough "this is being pushed way beyond its design limits, time to replace it".

For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:

    { "driver": "quorum",
      "child": [ { "driver": "file", "filename": "disk1.img" },
                 { "driver": "host_device", "filename=/dev/sdb" },
                 { "driver": "nbd", "host": "localhost" } ] }
Quoted 6 lines

Note how the structure is obvious. It isn't with dotted keys, not even if you order them sensibly (which users inevitably won't).

Also not that the value needs to be parsed by QemuOpts! You can't leave it to the consumer of the QemuOpts, because if you did, you'd have to escape the commas.

Okay, so I like JSON. It's a great format for our monitor protocol. We even have pretty printers so that it's more or less readable as output for human users. However, there is one thing for which it is horrible:
Getting user input.

Yes, getting rid of the comma escaping is a first step, but typing JSON on the command line remains a PITA. Mostly because of the quotes (which you probably have to escape), but also things like the useless outer brackets.

Quoted 7 lines

If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try

driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]

This looks a lot more agreeable as something that I have to type on the command line.

What's more, this is a direct extension of the existing syntax. You complained about how the step from simple configurations with -hda to more complex ones with -drive completely changes the syntax (and rightly so). Going from simple QemuOpts syntax to doing JSON as soon as you get structured values and lists would be another similar step. In contrast, the new syntax you're suggesting above is a natural extension of what's there.

Quoted 2 lines

I'd go with some existing syntax, though. The one we already use is JSON.

The one we already use in this context is comma separated key/value pairs as parsed by QemuOpts.

Quoted 8 lines

Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.

We can avoid rule 2 by requiring '.' to be escaped. Dan's qdict_crumple() currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.

I don't think we should support any escaping and rather forbid '.' completely in names.

Quoted 4 lines

The names in question are the names declared in QemuOptDesc (easy enough to find) plus any names that the code may use with empty QemuOptDesc (harder). Empty QemuOptDesc are used with the following option groups:

Quoted 8 lines

* "drive", qemu_drive_opts in blockdev.c
Also empty_opts in qemu-io.c and qemu_drive_opts in test-replication.c

* "numa", qemu_numa_opts in numa.c

* "device", qemu_device_opts in qdev-monitor.c

This one pulls in qdev properties. Properties violating rule 2 exist.

You posted a list of them in another email. Apart from not being user-configurable, I think it actually makes sense to interpret these as being structured.

A lot of them are really array/list properties with a single entry. It seems we have some inconsistency there as to whether it the syntax is 'list.index' or 'list[index]'. The former matches the dotted syntax rule and is used in external interfaces (e.g. in bus=...), and I seem to remember the latter is a newer QOM thing.

Would it be too late to change the latter into the former? In other words, is the difference externally visible outside of qom-*, which we declared a non-ABI, as far as I know?

Quoted 23 lines

* "object", qemu_object_opts in vl.c
Also qemu_object_opts in qemu-img.c, qemu-io.c and qemu-nbd.c (WTF?)

This one pulls in properties of user-creatable objects.

* "source", qemu_soource_opts (sic!) in qemu-img.c

* "reopen", reopen_opts in qemu-io-cmds.c

* "file", file_opts in qemu-io.c and qemu-nbd.c

* "machine", qemu_machine_opts in vl.c

This one pulls in machine properties.

* "tpmdev", qemu_tpmdev_opts in vl.c

* "netdev", qemu_netdev_opts in net.c

* "net", qemu_net_opts in net.c

* "userdef", userdef_opts in tests-opts-visitor.c

Quoted 8 lines

* "opts_list_03", opts_list_03 in test-qemu-opts.c

* "acpi", qemu_acpi_opts in hw/acpi/core.c

* "smbios", qemu_smbios_opts in smbios.c

Empty QemuOptDesc is used this widely because QemuOpts is too primitive to support the needs of its users. Unlike a QAPI schema. Just sayin'.

I don't think you have to convince anyone that QemuOpts is too limited. But today it's still a piece that is needed to get from the command line to something superior like QAPI. This is the whole reason why we're doing all of this.

Completely QAPIfication of everything and going directly from the command line to QAPI objects would be nice, but somehow I feel it's not a realistic hope to get this as a short term solution.

Quoted 2 lines

Finding the names and assessing how they're impacted by the dotted key convention is going to be a substantial chunk of work.

Possibly.

The other option would be to take it step by step and let every user of QemuOpts specify on creation whether dots are used for structure or should be taken literally. Then you can convert user by user and assess them one at a time (which is still a substantial chunk for -object and -device, but at least you can concentrate on only these for now).

If you're concerned about compatibility issues if we find a dot in a name only in one of the later users: We can handle it. If you then stumble across an obscure "weird.name" property where the dot is really part of the name and not already representing structure, you can still artificially reinterpret it as a nested structure with a single field even if logically that's not what it is... Might be a bit ugly, but keeps it working.

Kevin

#80 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

In reply to Eric Blake
Quoted 53 lines · Eric Blake
Quoted 31 lines · Markus Armbruster
Quoted 11 lines

But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.

Compare this:

    driver=quorum,
    child.0.driver=file,child.0.filename=disk1.img,
    child.1.driver=host_device,child.1.filename=/dev/sdb,
    child.2.driver=nbd,child.2.host=localhost
Quoted 6 lines

And this:

driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost

Aside: both are about equally illegible, to be honest.

Permit me to digress.

QemuOpts wasn't designed for list-values keys. Doing lists by repetition was clever.

QemuOpts wasn't designed for structured values. Doing structured values by a dotted key convention plus repetition was clever.

And there's the problem: too much cleverness, not enough "this is being pushed way beyond its design limits, time to replace it".

Quoted 7 lines

For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:

    { "driver": "quorum",
      "child": [ { "driver": "file", "filename": "disk1.img" },
                 { "driver": "host_device", "filename=/dev/sdb" },
                 { "driver": "nbd", "host": "localhost" } ] }

Possible hack solution:

QemuOpts already special-cases id=. What if we ALSO make it special-case a leading json=? Shown here with shell quoting, the above example of creating a Quorum -drive argument could then be:

-drive json='

    { "driver": "quorum",
      "child": [ { "driver": "file", "filename": "disk1.img" },
                 { "driver": "host_device", "filename=/dev/sdb" },
                 { "driver": "nbd", "host": "localhost" } ] }

'

Quoted 12 lines

As far as I know, we don't have 'json' as any existing QemuOpts key (do we? A full audit may be better than my quick git grep '"json"'). Thus, if QemuOpts sees a leading json=, it hands off the rest of the string to the same parser as we use for QMP, where we no longer have to escape commas (even nicer than the drive hack where we support filename=json:{...} but have to double up all commas to make it through the QemuOpts layer). Encountering json= as anything other than the first option would be an error, and you would be unable to combine a json= option with any other old-style option. In other words, the use of leading json= would be the switch for whether to do old-style parsing or to use a saner syntax for everything else that needs structure, on a per-argument basis.

Slight variation: omit the 'json=' and recognize the '{':

    -drive '{ "driver": "quorum",
              "child": [ { "driver": "file", "filename": "disk1.img" },
                         { "driver": "host_device", "filename=/dev/sdb" },
                         { "driver": "nbd", "host": "localhost" } ] }'

As always when extending haphazardly defined syntax, the question to ask is whether this makes the syntax (more) ambiguous.

So what is the option argument syntax now? The abstract syntax is simple enough: "list of (key, value) pairs". For the concrete syntax, we need to study opts_do_parse().

Each iteration of its loop accepts an abstract (key, value). It first looks for the leftmost '=' and ','. Cases:

1. There is no '=', or it is to the right of the leftmost ','. In other words, this (key, value) can only be key with an implied value or a value with an implied key.

1a. If this is the first iteration, and we accept an implied key, this is a value for the implied key. We consume everything up to the first non-escaped ','. This may be more than the leftmost ',' we found above. The consumed string with escapes processed is the value.

1b. Else, this is a key with an implied value. We consume everything up to the leftmost ',' (no escaping here).

1b1. If the consumed string starts with "no", the key is everything after the "no" and the value is "off".

1b2. Else the key is the string and the value is "on".

2. This is a key and a value. We first consume everything up to the leftmost '=' (no escaping here). The consumed string is the key. We then consume '='. Finally, we consume everything up to the first non-escaped ','The consumed string with escapes processed is the value.

Thus, the option argument starts either with a key (case 1b1, 2), "no" (case 1b2) or a value (case 1a).

Adding JSON object syntax (which always starts with '{') is ambiguous when a key can start with '{' (case 1b1, 2) or when a value can (case 1a).

Keys starting with '{' are basically foolish. Let's outlaw them by adopting QAPI's name rules.

Values starting with '{' are possible. The implied keys I can remember don't have such values, though. If we interpret '{' as JSON, then users can't omit the key when the value starts with '{'. Not pretty, but I'd say it's tolerable.

#6 lore raw

Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict

In reply to Daniel P. Berrange
Quoted 84 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 289 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 261 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 551 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..e0d24e1 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +0 -0
index 60f158c..c38e90e 100644

[...]

Quoted 4 lines

+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionaries

Is recursive=false used outside tests in this series?

Quoted 40 lines

+ *
+ * Takes a flat dictionary whose keys use '.' separator to indicate
+ * nesting, and values are scalars, and crumples it into a nested
+ * structure. If the @recursive parameter is false, then only the
+ * first level of structure implied by the keys will be crumpled. If
+ * @recursive is true, then the input will be recursively crumpled to
+ * expand all levels of structure in the keys.
+ *
+ * To include a literal '.' in a key name, it must be escaped as '..'
+ *
+ * For example, an input of:
+ *
+ * { 'foo.0.bar': 'one', 'foo.0.wizz': '1',
+ * 'foo.1.bar': 'two', 'foo.1.wizz': '2' }
+ *
+ * will result in an output of:
+ *
+ * {
+ * 'foo': [
+ * { 'bar': 'one', 'wizz': '1' },
+ * { 'bar': 'two', 'wizz': '2' }
+ * ],
+ * }
+ *
+ * The following scenarios in the input dict will result in an
+ * error being returned:
+ *
+ * - Any values in @src are non-scalar types
+ * - If keys in @src imply that a particular level is both a
+ * list and a dict. e.g., "foo.0.bar" and "foo.eek.bar".
+ * - If keys in @src imply that a particular level is a list,
+ * but the indices are non-contiguous. e.g. "foo.0.bar" and
+ * "foo.2.bar" without any "foo.1.bar" present.
+ * - If keys in @src represent list indexes, but are not in
+ * the "%zu" format. e.g. "foo.+0.bar"
+ *
+ * Returns: either a QDict or QList for the nested data structure, or NULL
+ * on error
+ */
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp)

[...]

#82 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

In reply to Kevin Wolf
Quoted 53 lines · Kevin Wolf

Am 17.10.2016 um 16:50 hat Markus Armbruster geschrieben:

Quoted 36 lines
Quoted 27 lines · Kevin Wolf

Am 13.10.2016 um 14:35 hat Markus Armbruster geschrieben:

Quoted 26 lines
Cc Kevin for discussion of QemuOpts dotted key convention
Quoted 12 lines · Daniel P. Berrange

Currently qdict_crumple requires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.

In order to have backwards compatibility with the OptsVisitor, qemu_opt_to_qdict() has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given to qdict_crumple().

To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed

Quoted 6 lines
   foo=hello,foo=world,foo=wibble
   foo.0=hello,foo.1=world,foo.2=wibble

but this is forbidden

foo=hello,foo=world,foo.2=wibble

I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.

Quoted 6 lines
Quoted 3 lines

The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?

Probably primarily because someone didn't realise this when introducing the dotted syntax.

Quoted 6 lines

Can't even blame "someone" for that; it's an obscure, underdocumented feature of an interface that's collapsing under its load of obscure, underdocumented features.

On the other hand, that's not exactly a state that allows for *more* obscure features.

I don't really think we're introducing more obscure features here, but rather trying to implement a single, and rather straightforward, way as the standard.

Dotted syntax for hierarchy has actually plenty of precedence in qemu if you look a bit closer (the block layer, -global, -device foo,help, even the bus names you mentioned below are really just flattened lists), so we're only making things more consistent.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Also because flat QDicts can't represent this.

Explain?

Repeated options are parsed into QLists. If you keep it at that without flattening you have at least a QDict that contains a QList that contains simple types. This is not flat any more.

*All* options are parsed into a (non-QList) list. That's what is flat.

Only when you start crumpling things go beyond flat, and QDict/QList come into play.

Quoted 5 lines

Of course, you could argue that flat QDicts are the wrong data structure in the first place and instead of flatting everything we should have done the equivalent of qdict_crumple from the beginning, but they were the natural extension of what was already there and happened to work good enough, so this is what we're currently using.

We didn't "flatten everything", because QemuOpts is and has always been flat to begin with. Rather we've started to crumple a few things, and in a separate layer.

I now think this is the wrong approach. We have clearly outgrown flat options. We should redo QemuOpts to support what we need instead of bolting on an optional crumpling layer.

I guess I reached the place you described, just on a different path :)

Quoted 67 lines
Quoted 11 lines
Quoted 11 lines

But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.

Compare this:

    driver=quorum,
    child.0.driver=file,child.0.filename=disk1.img,
    child.1.driver=host_device,child.1.filename=/dev/sdb,
    child.2.driver=nbd,child.2.host=localhost
Quoted 8 lines
Quoted 6 lines

And this:

driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost

Aside: both are about equally illegible, to be honest.

Not sure about equally, but let's agree on "both are illegible".

Quoted 33 lines
Quoted 2 lines

For starters, can we really trust the order in QemuOpts so that the right driver and filename are associated with each other?

The order is trustworthy, but...

Quoted 4 lines

We would also have code to associate the third child.driver with the first child.host (because file and host_device don't have a host property). And this isn't even touching optional arguments yet. Would you really want to implement or review this?

... you're right, doing lists by repeating keys breaks down when combined with the dotted key convention's use of repetition to do structured values.

Permit me to digress.

QemuOpts wasn't designed for list-values keys. Doing lists by repetition was clever.

QemuOpts wasn't designed for structured values. Doing structured values by a dotted key convention plus repetition was clever.

And there's the problem: too much cleverness, not enough "this is being pushed way beyond its design limits, time to replace it".

For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:

    { "driver": "quorum",
      "child": [ { "driver": "file", "filename": "disk1.img" },
                 { "driver": "host_device", "filename=/dev/sdb" },
                 { "driver": "nbd", "host": "localhost" } ] }
Quoted 6 lines

Note how the structure is obvious. It isn't with dotted keys, not even if you order them sensibly (which users inevitably won't).

Also not that the value needs to be parsed by QemuOpts! You can't leave it to the consumer of the QemuOpts, because if you did, you'd have to escape the commas.

Okay, so I like JSON. It's a great format for our monitor protocol. We even have pretty printers so that it's more or less readable as output for human users. However, there is one thing for which it is horrible:
Getting user input.

Quoted 4 lines

Yes, getting rid of the comma escaping is a first step, but typing JSON on the command line remains a PITA. Mostly because of the quotes (which you probably have to escape), but also things like the useless outer brackets.

As long as you don't need "'" in your JSON, you can simply enclose in "'" and be done. Since "'" can only occur in JSON strings, and the same strings would be present in any other syntax, any other syntax would be pretty much the same PITA.

Quoted 11 lines
Quoted 7 lines

If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try

driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]

This looks a lot more agreeable as something that I have to type on the command line.

Quoted 7 lines

What's more, this is a direct extension of the existing syntax. You complained about how the step from simple configurations with -hda to more complex ones with -drive completely changes the syntax (and rightly so). Going from simple QemuOpts syntax to doing JSON as soon as you get structured values and lists would be another similar step. In contrast, the new syntax you're suggesting above is a natural extension of what's there.

Point taken. I just don't like inventing syntax, because as a rule, way too much syntax gets invented, and almost always badly.

Quoted 5 lines
Quoted 2 lines

I'd go with some existing syntax, though. The one we already use is JSON.

The one we already use in this context is comma separated key/value pairs as parsed by QemuOpts.

Which is flat, and flat doesn't cut it anymore.

If you make it non-flat with dotted key convention, the dotted key convention becomes part of the syntax. Counts as inventing syntax in my book.

Quoted 11 lines
Quoted 8 lines

Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.

We can avoid rule 2 by requiring '.' to be escaped. Dan's qdict_crumple() currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.

I don't think we should support any escaping and rather forbid '.' completely in names.

I think we should adopt QAPI's naming rules.

Quoted 13 lines
Quoted 4 lines

The names in question are the names declared in QemuOptDesc (easy enough to find) plus any names that the code may use with empty QemuOptDesc (harder). Empty QemuOptDesc are used with the following option groups:

Quoted 8 lines

* "drive", qemu_drive_opts in blockdev.c
Also empty_opts in qemu-io.c and qemu_drive_opts in test-replication.c

* "numa", qemu_numa_opts in numa.c

* "device", qemu_device_opts in qdev-monitor.c

This one pulls in qdev properties. Properties violating rule 2 exist.

Quoted 7 lines

You posted a list of them in another email. Apart from not being user-configurable, I think it actually makes sense to interpret these as being structured.

A lot of them are really array/list properties with a single entry. It seems we have some inconsistency there as to whether it the syntax is 'list.index' or 'list[index]'.

Yes.

Quoted 3 lines

The former matches the dotted syntax
rule and is used in external interfaces (e.g. in bus=...), and I seem to remember the latter is a newer QOM thing.

Commit 3396590 "qom: Add automatic arrayification to object_property_add()". I tried to strangle it in the crib, unsuccessfully:
Message-Id: <[email protected]>
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg01905.html

Quoted 3 lines

Would it be too late to change the latter into the former? In other words, is the difference externally visible outside of qom-*, which we declared a non-ABI, as far as I know?

Paolo?

Quoted 32 lines
Quoted 23 lines

* "object", qemu_object_opts in vl.c
Also qemu_object_opts in qemu-img.c, qemu-io.c and qemu-nbd.c (WTF?)

This one pulls in properties of user-creatable objects.

* "source", qemu_soource_opts (sic!) in qemu-img.c

* "reopen", reopen_opts in qemu-io-cmds.c

* "file", file_opts in qemu-io.c and qemu-nbd.c

* "machine", qemu_machine_opts in vl.c

This one pulls in machine properties.

* "tpmdev", qemu_tpmdev_opts in vl.c

* "netdev", qemu_netdev_opts in net.c

* "net", qemu_net_opts in net.c

* "userdef", userdef_opts in tests-opts-visitor.c

Quoted 8 lines

* "opts_list_03", opts_list_03 in test-qemu-opts.c

* "acpi", qemu_acpi_opts in hw/acpi/core.c

* "smbios", qemu_smbios_opts in smbios.c

Empty QemuOptDesc is used this widely because QemuOpts is too primitive to support the needs of its users. Unlike a QAPI schema. Just sayin'.

Quoted 8 lines

I don't think you have to convince anyone that QemuOpts is too limited. But today it's still a piece that is needed to get from the command line to something superior like QAPI. This is the whole reason why we're doing all of this.

Completely QAPIfication of everything and going directly from the command line to QAPI objects would be nice, but somehow I feel it's not a realistic hope to get this as a short term solution.

No argument. However, we should try hard to avoid short term hacks that make a QAPI-based solution harder. I believe the hard part will be retaining all the special cases that got in for convenience, or by accident, then were repurposed for whatever. And that's why I'm resisting additional syntactic/semantic cleverness *especially* when it's used only sometimes. Stuff that's used only sometimes can easily conflict with stuff used at other times.

I don't think we understand the existing cleverness sufficiently to predict interactions with new cleverness with confidence.

Quoted 10 lines
Quoted 2 lines

Finding the names and assessing how they're impacted by the dotted key convention is going to be a substantial chunk of work.

Possibly.

The other option would be to take it step by step and let every user of QemuOpts specify on creation whether dots are used for structure or should be taken literally. Then you can convert user by user and assess them one at a time (which is still a substantial chunk for -object and -device, but at least you can concentrate on only these for now).

I'm wary of protracted iterative conversion. We're better at starting these than finishing them. The started-but-not-finished state is exactly the state that scares me here: still more special cases basically nobody fully understands.

Quoted 7 lines

If you're concerned about compatibility issues if we find a dot in a name only in one of the later users: We can handle it. If you then stumble across an obscure "weird.name" property where the dot is really part of the name and not already representing structure, you can still artificially reinterpret it as a nested structure with a single field even if logically that's not what it is... Might be a bit ugly, but keeps it working.

Yes, that's how we'd do backward compatibility, if needed.

So, what can we do? Perhaps something like this:

* Pick a concrete syntax that supports nested stuff. Proposals on the table so far are

1. Dotted key convention as used in the block layer

2. Array syntax as used in QOM's automatic arrayification (not a complete solution, mentioned because it conflicts with the previous)

3. JSON

4. JSON with ':' replaced by '=', double-quotes and the outermost pair of braces dropped.

* Replace the QemuOpts internal representation: QDict instead of list.

* Replace the QemuOpts parser.

* Add suitable struct- and list-valued accessors.

qdict_crumple(qemu_opt_to_qdict(...), true, &errp) gets replaced by a trivial "get the whole dictionary" operation.

Other uses of qemu_opt_to_qdict() need to be rewritten.

* "Repetition builds a list" backward compatibility. It's used in only a few places.

* Possibly "keys with funny characters" backward compatibility.

Feels doable to me. 2.8 would be ambitious, though: soft freeze in in less than two weeks. However, getting this series into 2.8 has started to feel ambitious to me, too.

#88 lore raw

Re: [Qemu-devel] [PATCH v14 14/21] qapi: allow repeated opts with qobject_input_visitor_new_opts

In reply to Daniel P. Berrange
Quoted 47 lines · Daniel P. Berrange

The qobject_input_visitor_new_opts() method gains a new parameter to control whether it allows repeated option keys in the input QemuOpts or not.

---
 include/qapi/qobject-input-visitor.h | 6 ++++++
 qapi/qobject-input-visitor.c         | 5 ++++-
 2 files changed, 10 insertions(+), 1 deletion(-)
include/qapi/qobject-input-visitor.h +6 -0
index 242b767..bc5062a 100644
@@ -112,6 +112,11 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,
  * qobject_input_visitor_new_autocast() method. See the docs
  * of that method for further details on processing behaviour.
  *
+ * If the @permit_repeated_opts parameter is true, then the input
+ * @opts is allowed to contain repeated keys and they will be
+ * turned into a QList. If it is false, then repeated keys will
+ * result in an error being reported.
+ *
  * The returned input visitor should be released by calling
  * visit_free() when no longer required.
  */
@@ -119,6 +124,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
                                         bool autocreate_list,
                                         size_t autocreate_struct_levels,
                                         bool permit_int_ranges,
+                                        bool permit_repeated_opts,
                                         Error **errp);
 
 #endif
qapi/qobject-input-visitor.c +1 -0
index 2287d11..5a3872c 100644
@@ -753,6 +753,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
                                         bool autocreate_list,
                                         size_t autocreate_struct_levels,
                                         bool permit_int_ranges,
+                                        bool permit_repeated_opts,
                                         Error **errp)
 {
     QDict *pdict;

qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 769

Quoted 10 lines
qapi/qobject-input-visitor.c +3 -1
@@ -760,7 +761,9 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
     Visitor *v = NULL;
 
     pdict = qemu_opts_to_qdict(opts, NULL,
-                               QEMU_OPTS_REPEAT_POLICY_LAST,
+                               permit_repeated_opts ?
+                               QEMU_OPTS_REPEAT_POLICY_ALL :
+                               QEMU_OPTS_REPEAT_POLICY_ERROR,
                                errp);
     if (!pdict) {
         goto cleanup;

@permit_repeated_opts applies to the whole @opts. However, the "repeat to build a list" feature is actually used more selectively. For instance, -spice wants it for "tls-channel" and "plaintext-channel", but not for its other keys. There, the code uses qemu_opt_get() or similar, which gets the last value.

The options visitor works the same. Internally, it stores only lists. When the visit is for a scalar, it gets the last value from the list. When the visit is for a list, it sets up things so that the list element visits get the values in order.

As long as no key is list-valued, you can keep @permit_repeated_opts false, and have duplicate keys rejected, for whatever that's worth.

As soon as you have a list-valued key, you need to set @permit_repeated_opts to true. But then *no* duplicate keys are rejected. If consistency with @permit_repeated_opts = false is needed, then something else needs to detect unwanted lists and reject them.

What use cases for @permit_repeated_opts = false do you envisage?

To serve as replacement for the options visitor, this needs to treat duplicates the same. As we saw in PATCH 12, QEMU_OPTS_REPEAT_POLICY_ALL makes a list when the key is repeated, while the options visitor makes a list always. I guess some other patch hides this difference.

#83 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

In reply to Markus Armbruster

Am 18.10.2016 um 17:35 hat Markus Armbruster geschrieben:

Quoted 10 lines
Quoted 5 lines · Kevin Wolf

Of course, you could argue that flat QDicts are the wrong data structure in the first place and instead of flatting everything we should have done the equivalent of qdict_crumple from the beginning, but they were the natural extension of what was already there and happened to work good enough, so this is what we're currently using.

We didn't "flatten everything", because QemuOpts is and has always been flat to begin with. Rather we've started to crumple a few things, and in a separate layer.

That's the QemuOpts point of view.

I was talking from a block layer view. There we get data in two different formats, QMP and the command line. The first is structured, the second is flat. We need to make both uniform before we can pass them to the actual block layer functions.

The current code chose to flatten what we get from QMP blockdev-add and use that throughout the block layer. We could instead have decided that we leave the blockdev-add input as it is, crumple what we get from QemuOpts and use nested QObjects throughout the block layer.

Quoted 5 lines

I now think this is the wrong approach. We have clearly outgrown flat options. We should redo QemuOpts to support what we need instead of bolting on an optional crumpling layer.

I guess I reached the place you described, just on a different path :)

Yes, I think the conclusion is easy to agree on.

Quoted 10 lines
Quoted 9 lines

Okay, so I like JSON. It's a great format for our monitor protocol. We even have pretty printers so that it's more or less readable as output for human users. However, there is one thing for which it is horrible:
Getting user input.

Yes, getting rid of the comma escaping is a first step, but typing JSON on the command line remains a PITA. Mostly because of the quotes (which you probably have to escape), but also things like the useless outer brackets.

Quoted 4 lines

As long as you don't need "'" in your JSON, you can simply enclose in "'" and be done. Since "'" can only occur in JSON strings, and the same strings would be present in any other syntax, any other syntax would be pretty much the same PITA.

I've written enough scripts (qemu-iotests cases) that produce JSON with shell variables in it, so if anything you can use "" quoting for the shell and make use of qemu's extension that '' is accepted in JSON, too.

Anyway, the quotes aren't only nasty because of the escaping, but also just because I have to type them.

Quoted 25 lines
Quoted 11 lines
Quoted 7 lines

If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try

driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]

This looks a lot more agreeable as something that I have to type on the command line.

Quoted 7 lines

What's more, this is a direct extension of the existing syntax. You complained about how the step from simple configurations with -hda to more complex ones with -drive completely changes the syntax (and rightly so). Going from simple QemuOpts syntax to doing JSON as soon as you get structured values and lists would be another similar step. In contrast, the new syntax you're suggesting above is a natural extension of what's there.

Point taken. I just don't like inventing syntax, because as a rule, way too much syntax gets invented, and almost always badly.

Quoted 3 lines
Quoted 2 lines

I'd go with some existing syntax, though. The one we already use is JSON.

Quoted 8 lines
Quoted 2 lines

The one we already use in this context is comma separated key/value pairs as parsed by QemuOpts.

Which is flat, and flat doesn't cut it anymore.

If you make it non-flat with dotted key convention, the dotted key convention becomes part of the syntax. Counts as inventing syntax in my book.

Yes, it is. Though in the context of command line options, dotted syntax is an invention already made, whereas JSON would be a new invention. (Well, not completely, because for block devices we already have json: - but that works a little different again.)

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.

We can avoid rule 2 by requiring '.' to be escaped. Dan's qdict_crumple() currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.

Quoted 4 lines
Quoted 2 lines

I don't think we should support any escaping and rather forbid '.' completely in names.

I think we should adopt QAPI's naming rules.

Which includes what I said, so fine with me.

Quoted 17 lines
Quoted 7 lines

If you're concerned about compatibility issues if we find a dot in a name only in one of the later users: We can handle it. If you then stumble across an obscure "weird.name" property where the dot is really part of the name and not already representing structure, you can still artificially reinterpret it as a nested structure with a single field even if logically that's not what it is... Might be a bit ugly, but keeps it working.

Yes, that's how we'd do backward compatibility, if needed.

So, what can we do? Perhaps something like this:

* Pick a concrete syntax that supports nested stuff. Proposals on the table so far are

1. Dotted key convention as used in the block layer

Quoted 8 lines

2. Array syntax as used in QOM's automatic arrayification (not a complete solution, mentioned because it conflicts with the previous)

3. JSON

4. JSON with ':' replaced by '=', double-quotes and the outermost pair of braces dropped.

Biased block layer view on what we ideally want to have in the end:
Anthony's QCFG proposal plus a [] syntax for lists.

This is more or less what you describe in option 4, except that the existing dotted syntax is still supported. It is part of the ABI by now, so we can't get rid of it anyway, and supporting it everywhere is not only more consistent but probably also easier than making it a special case.

The tricky part here is the {} and [] syntax for dicts and lists:
Parsing this requires a schema if you don't want to restrict string values to not contain any of these characters. Such a restriction would be incompatible, so we probably can't make it.

Essentially this means that we're back to "QAPIfy everything" before we can make this work. Leaves us with implementing only the dotted syntax subset for now, which is equally powerful, even if a lot more verbose.

Quoted 11 lines

* Replace the QemuOpts internal representation: QDict instead of list.

* Replace the QemuOpts parser.

* Add suitable struct- and list-valued accessors.

qdict_crumple(qemu_opt_to_qdict(...), true, &errp) gets replaced by a trivial "get the whole dictionary" operation.

Other uses of qemu_opt_to_qdict() need to be rewritten.

Quoted 8 lines

* "Repetition builds a list" backward compatibility. It's used in only a few places.

* Possibly "keys with funny characters" backward compatibility.

Feels doable to me. 2.8 would be ambitious, though: soft freeze in in less than two weeks. However, getting this series into 2.8 has started to feel ambitious to me, too.

Let's merge qdict_crumple() at least, this would allow us to add the SSH and NBD blockdev-add support for 2.8, possibly NFS to. No command line syntax involved at all there. I can take the patch through my tree if nobody objects.

With -blockdev I'll have to wait for the final decision, but if we decide to use a syntax that includes dotted syntax (possibly as a subset), I think this series would provide the right external interface even if there is still some infrastructure cleanup work to do internally.

Kevin

#84 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

Quoted 19 lines · Kevin Wolf

Am 18.10.2016 um 17:35 hat Markus Armbruster geschrieben:

Quoted 10 lines
Quoted 5 lines · Kevin Wolf

Of course, you could argue that flat QDicts are the wrong data structure in the first place and instead of flatting everything we should have done the equivalent of qdict_crumple from the beginning, but they were the natural extension of what was already there and happened to work good enough, so this is what we're currently using.

We didn't "flatten everything", because QemuOpts is and has always been flat to begin with. Rather we've started to crumple a few things, and in a separate layer.

That's the QemuOpts point of view.

I was talking from a block layer view. There we get data in two different formats, QMP and the command line. The first is structured, the second is flat. We need to make both uniform before we can pass them to the actual block layer functions.

Quoted 4 lines

The current code chose to flatten what we get from QMP blockdev-add and use that throughout the block layer. We could instead have decided that we leave the blockdev-add input as it is, crumple what we get from QemuOpts and use nested QObjects throughout the block layer.

I would very much like to see BlockdevOptions structs passed around the block drivers, instead of QemuOpts - I think it'd lead to much clearer code than the QemuOpts stuff we have today IMHO

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#85 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

In reply to Markus Armbruster
Quoted 11 lines · Markus Armbruster
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.

Compare this:

    driver=quorum,
    child.0.driver=file,child.0.filename=disk1.img,
    child.1.driver=host_device,child.1.filename=/dev/sdb,
    child.2.driver=nbd,child.2.host=localhost
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

And this:

driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost

Aside: both are about equally illegible, to be honest.

Furthermore, both of these currently require really long command lines. Overnight, I was thinking about whether it would be worth improving QemuOpts to ignore whitespace after (unquoted) commas, so you can do:

-drive 'driver=quorum,
child.0.driver=file, child.0.filename=disk1.img,
child.1.driver=host_device, child.1.filename=/dev/sdb,
child.2.driver=nbd, child.2.host=localhost
'

I think this one should be just fine - we don't have ANY keys that start with leading whitespace, so ignoring space will still let us parse out key names, but allow for much more legibility in the command lines. And this is true whether we use dotted notation...

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try

driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]

This looks a lot more agreeable as something that I have to type on the command line.

...or new syntax (where of course the new syntax has already demonstrated that we want to support strategic ignoring of whitespace).

So I'll probably propose a patch along those lines soon, as it seems like an orthogonal improvement.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#90 lore raw

Re: [Qemu-devel] [PATCH v14 15/21] qom: support non-scalar properties with -object

In reply to Daniel P. Berrange
Quoted 58 lines · Daniel P. Berrange

The current -object command line syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.

This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.

The QObjectInputVisitor now has functionality that allows it to replace OptsVisitor, maintaining full backwards compatibility for previous CLI syntax, while also allowing use of new syntax for structs.

Thus -object can now support non-scalar properties, for example the QMP object:

  {
    "execute": "object-add",
    "arguments": {
      "qom-type": "demo",
      "id": "demo0",
      "parameters": {
        "foo": [
          { "bar": "one", "wizz": "1" },
          { "bar": "two", "wizz": "2" }
        ]
      }
    }
  }

Would be creatable via the CLI now using

    $QEMU \
      -object demo,id=demo0,\
              foo.0.bar=one,foo.0.wizz=1,\
              foo.1.bar=two,foo.1.wizz=2

Notice that this syntax is intentionally compatible with that currently used by block drivers. NB the indentation seen here after line continuations should not be used in reality, it is just for clarity in this commit message.

---
 qapi/qobject-input-visitor.c |   2 +-
 qom/object_interfaces.c      |  37 ++++-
 tests/check-qom-proplist.c   | 367 ++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 397 insertions(+), 9 deletions(-)
qapi/qobject-input-visitor.c +0 -0
index 5a3872c..f1030d5 100644

qapi/qobject-input-visitor.c static void qobject_input_start_struct(Visitor *v, const char *name, void **obj, · line 210

Quoted 8 lines
qapi/qobject-input-visitor.c +1 -1
@@ -204,7 +204,7 @@ static void qobject_input_start_struct(Visitor *v, const char *name, void **obj,
         *obj = NULL;
     }
 
-    if (!qobj && (qiv->struct_level < qiv->autocreate_struct_levels)) {
+    if (!qobj && (qiv->struct_level <= qiv->autocreate_struct_levels)) {
         /* Create a new dict that contains all the currently
          * unvisited items */
         if (tos) {

Uh, can you explain this hunk? The < comes from PATCH 10.

Quoted 23 lines
qom/object_interfaces.c +4 -2
index fdc406b..88a1e88 100644
@@ -4,7 +4,8 @@
 #include "qemu/module.h"
 #include "qapi-visit.h"
 #include "qapi/qobject-output-visitor.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
+#include "qemu/option.h"
 
 void user_creatable_complete(Object *obj, Error **errp)
 {
@@ -63,12 +64,16 @@ Object *user_creatable_add(const QDict *qdict,
     if (local_err) {
         goto out_visit;
     }
-    visit_check_struct(v, &local_err);
+
+    obj = user_creatable_add_type(type, id, pdict, v, &local_err);
     if (local_err) {
         goto out_visit;

qom/object_interfaces.c Object *user_creatable_add(const QDict *qdict, · line 79

Quoted 10 lines
qom/object_interfaces.c +4 -1
     }
 
-    obj = user_creatable_add_type(type, id, pdict, v, &local_err);
+    visit_check_struct(v, &local_err);
+    if (local_err) {
+        goto out_visit;
+    }
 
 out_visit:
     visit_end_struct(v, NULL);

Can you explain why you swap the order of visit_check_struct() and user_creatable_add_type()? It smells like a bug fix to me...

Odd: opts_check_struct() does nothing unless depth == 0. But depth is at least 1 between opts_start_struct() and opts_end_struct(). Bug?

qom/object_interfaces.c Object *user_creatable_add_type(const char *type, const char *id, · line 125

Quoted 9 lines
qom/object_interfaces.c +1 -1
@@ -114,7 +119,7 @@ Object *user_creatable_add_type(const char *type, const char *id,
 
     assert(qdict);
     obj = object_new(type);
-    visit_start_struct(v, NULL, NULL, 0, &local_err);
+    visit_start_struct(v, "props", NULL, 0, &local_err);
     if (local_err) {
         goto out;
     }

Why this hunk?

Quoted 12 lines
qom/object_interfaces.c +3 -2
@@ -158,14 +163,32 @@ Object *user_creatable_add_opts(QemuOpts *opts, Error **errp)
 {
     Visitor *v;
     QDict *pdict;
+    QObject *pobj;
     Object *obj = NULL;
 
-    v = opts_visitor_new(opts);
-    pdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+    pdict = qemu_opts_to_qdict(opts, NULL,
+                               QEMU_OPTS_REPEAT_POLICY_ALL,
                                &error_abort);

qom/object_interfaces.c Object *user_creatable_add_opts(QemuOpts *opts, Error **errp) · line 180

Quoted 10 lines
qom/object_interfaces.c +8 -1
 
-    obj = user_creatable_add(pdict, v, errp);
+    pobj = qdict_crumple(pdict, true, errp);
+    if (!pobj) {
+        goto cleanup;
+    }
+    /*
+     * We need autocreate_list=true + permit_int_ranges=true
+     * in order to maintain compat with OptsVisitor creation
+     * of the 'numa' object which had an int16List property.

I can't find a "numa" object in the object-add sense, only a NumaOptions QAPI object created by -numa. Is that what you mean?

qom/object_interfaces.c Object *user_creatable_add_opts(QemuOpts *opts, Error **errp) · line 184

Quoted 4 lines
qom/object_interfaces.c +4 -0
+     *
+     * We need autocreate_structs=1, because at the CLI level
+     * we have the object type + properties in the same flat
+     * struct, even though at the QMP level they are nested.

We screwed up QMP object-add. device_add, netdev_add and object_add all treat additional arguments as properties *except* for QMP object-add, which has them wrapped in a JSON object instead.

Aside: if someone foolishly adds a property named "qom-type" or "id", it'll work in QMP but not in HMP or -object.

The inconsistency complicates the code even before this patch:

* Core: user_creatable_add_type() takes properties in *two* forms: as a visitor and as a qdict. It visits a struct with the members given by the qdict's keys. That's its only use of the qdict.

* QMP: qmp_object_add() gets properties as a separate QDict. It creates a QObject visitor for it, and passes it to user_creatable_add_type() along with the QDict.

* -object: user_creatable_add_opts() gets the option argument as a QemuOpts. It creates an options visitor for it *and* converts it to a QDict, then passes both to user_creatable_add().

* user_creatable_add() visits a struct. It first visits the non-property arguments "qom-type" and "id". It passes the visitor and a shallow copy of the QDict with the non-property entries deleted to user_creatable_add_type() to visit the remaining struct members.

* HMP: hmp_object_add() gets its arguments as a QDict. It converts it to QemuOpts and creates an options visitor for it *boggle*. It passes the visitor along with original QDict to user_creatable_add().

I still can't quite see what's requiring autocreate_structs. But I hope we can get rid of it by cleaning up this mess a bit. Here's how I'd try:

* Move visit_start_struct() and visit_end_struct() from user_creatable_add_type() to its two callers qmp_object_add() and user_creatable_add().

* Make user_creatable_add_type() ignore QDict keys "qom-type" and "id".

* QMP: any attempt to specify property "qom-type" or "id" now fails, because visit_check_struct() flags them as unexpected.

* user_creatable_add(): ditch the silly QDict copying.

* hmp_object_add(): ditch the silly conversion to QemuOpts, and create a QObject visitor instead.

qom/object_interfaces.c Object *user_creatable_add_opts(QemuOpts *opts, Error **errp) · line 194

Quoted 10 lines
qom/object_interfaces.c +6 -0
+     */
+    v = qobject_input_visitor_new_autocast(pobj, true, 1, true);
+
+    obj = user_creatable_add(qobject_to_qdict(pobj), v, errp);
     visit_free(v);
+    qobject_decref(pobj);
+ cleanup:
     QDECREF(pdict);
     return obj;
 }

[Skipping test updates for now...]

#97 lore raw

Re: [Qemu-devel] [PATCH v14 16/21] hmp: support non-scalar properties with object_add

In reply to Daniel P. Berrange
Quoted 78 lines · Daniel P. Berrange

The current object_add HMP syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.

This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.

The QObjectInputVisitor now has functionality that allows it to replace OptsVisitor, maintaining full backwards compatibility for previous CLI syntax, while also allowing use of new syntax for structs.

Thus -object can now support non-scalar properties, for example the QMP object:

  {
    "execute": "object-add",
    "arguments": {
      "qom-type": "demo",
      "id": "demo0",
      "parameters": {
        "foo": [
          { "bar": "one", "wizz": "1" },
          { "bar": "two", "wizz": "2" }
        ]
      }
    }
  }

Would be creatable via the HMP now using

   object_add demo,id=demo0,\
              foo.0.bar=one,foo.0.wizz=1,\
              foo.1.bar=two,foo.1.wizz=2

Notice that this syntax is intentionally compatible with that currently used by block drivers. NB the indentation seen here after line continuations should not be used in reality, it is just for clarity in this commit message.

---
 hmp.c | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)
hmp.c +4 -5
index 336e7bf..b32d8c8 100644
@@ -25,7 +25,7 @@
 #include "qemu/sockets.h"
 #include "monitor/monitor.h"
 #include "monitor/qdev.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/string-output-visitor.h"
 #include "qapi/util.h"
@@ -1696,21 +1696,30 @@ void hmp_netdev_del(Monitor *mon, const QDict *qdict)
 void hmp_object_add(Monitor *mon, const QDict *qdict)
 {
     Error *err = NULL;
-    QemuOpts *opts;
     Visitor *v;
     Object *obj = NULL;
+    QObject *pobj;
 
-    opts = qemu_opts_from_qdict(qemu_find_opts("object"), qdict, &err);
+    pobj = qdict_crumple(qdict, true, &err);
     if (err) {
-        hmp_handle_error(mon, &err);
-        return;
+        goto cleanup;

hmp.c void hmp_netdev_del(Monitor *mon, const QDict *qdict) · line 1713

Quoted 10 lines
hmp.c +6 -2
     }
 
-    v = opts_visitor_new(opts);
-    obj = user_creatable_add(qdict, v, &err);
+    /*
+     * We need autocreate_list=true + permit_int_ranges=true
+     * in order to maintain compat with OptsVisitor creation
+     * of the 'numa' object which had an int16List property.
+     *
+     * We need autocreate_structs=1, because at the CLI level

autocreate_struct_levels=1

Same in previous patch.

Quoted 13 lines
hmp.c +7 -1
+     * we have the object type + properties in the same flat
+     * struct, even though at the QMP level they are nested.
+     */
+    v = qobject_input_visitor_new_autocast(pobj, true, 1, true);
+    obj = user_creatable_add(qobject_to_qdict(pobj), v, &err);
     visit_free(v);
-    qemu_opts_del(opts);
 
+ cleanup:
+    qobject_decref(pobj);
     if (err) {
         hmp_handle_error(mon, &err);
     }
#99 lore raw

Re: [Qemu-devel] [PATCH v14 17/21] numa: convert to use QObjectInputVisitor for -numa

In reply to Daniel P. Berrange
Quoted 120 lines · Daniel P. Berrange

Switch away from using OptsVisitor to parse the -numa argument processing. This enables use of the modern list syntax for specifying CPUs. e.g. the old syntax

-numa node,nodeid=0,cpus=0-3,cpus=8-11,mem=107

is equivalent to

  -numa node,nodeid=0,cpus.0=0,cpus.1=1,cpus.2=2,cpus.3=3,\
        cpus.4=8,cpus.5=9,cpus.6=10,cpus.7=11,mem=107

Furthermore, the cli arg can now follow the QAPI schema nesting, so the above is equivalent to the canonical syntax:

  -numa type=node,data.nodeid=0,data.cpus.0=0,data.cpus.1=1,\
        data.cpus.2=2,data.cpus.3=3,data.cpus.4=8,data.cpus.5=9,\
	data.cpus.6=10,data.cpus.7=11,data.mem=107

A test case is added to cover argument parsing to validate that both the old and new syntax is correctly handled.

---
 include/sysemu/numa_int.h |  11 +++++
 numa.c                    |  36 +++++++++-----
 stubs/Makefile.objs       |   5 ++
 stubs/exec.c              |   6 +++
 stubs/hostmem.c           |  14 ++++++
 stubs/memory.c            |  41 ++++++++++++++++
 stubs/qdev.c              |   8 ++++
 stubs/vl.c                |   8 ++++
 stubs/vmstate.c           |   4 ++
 tests/Makefile.include    |   2 +
 tests/test-numa.c         | 116 ++++++++++++++++++++++++++++++++++++++++++++++
 11 files changed, 240 insertions(+), 11 deletions(-)
 create mode 100644 include/sysemu/numa_int.h
 create mode 100644 stubs/exec.c
 create mode 100644 stubs/hostmem.c
 create mode 100644 stubs/memory.c
 create mode 100644 stubs/qdev.c
 create mode 100644 stubs/vl.c
 create mode 100644 tests/test-numa.c
include/sysemu/numa_int.h new file +11 -0
new file mode 100644
index 0000000..93160da
@@ -0,0 +1,11 @@
+#ifndef SYSEMU_NUMA_INT_H
+#define SYSEMU_NUMA_INT_H
+
+#include "sysemu/numa.h"
+
+extern int have_memdevs;
+extern int max_numa_nodeid;
+
+int parse_numa(void *opaque, QemuOpts *opts, Error **errp);
+
+#endif
numa.c +11 -9
index 6289f46..653ebf1 100644
@@ -23,14 +23,14 @@
  */
 
 #include "qemu/osdep.h"
-#include "sysemu/numa.h"
+#include "sysemu/numa_int.h"
 #include "exec/cpu-common.h"
 #include "qemu/bitmap.h"
 #include "qom/cpu.h"
 #include "qemu/error-report.h"
 #include "include/exec/cpu-common.h" /* for RAM_ADDR_FMT */
 #include "qapi-visit.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "hw/boards.h"
 #include "sysemu/hostmem.h"
 #include "qmp-commands.h"
@@ -45,10 +45,10 @@ QemuOptsList qemu_numa_opts = {
     .desc = { { 0 } } /* validated with OptsVisitor */
 };
 
-static int have_memdevs = -1;
-static int max_numa_nodeid; /* Highest specified NUMA node ID, plus one.
-                             * For all nodes, nodeid < max_numa_nodeid
-                             */
+int have_memdevs = -1;
+int max_numa_nodeid; /* Highest specified NUMA node ID, plus one.
+                      * For all nodes, nodeid < max_numa_nodeid
+                      */
 int nb_numa_nodes;
 NodeInfo numa_info[MAX_NODES];
 
@@ -189,6 +189,9 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
     if (node->has_mem) {
         uint64_t mem_size = node->mem;
         const char *mem_str = qemu_opt_get(opts, "mem");
+        if (!mem_str) {
+            mem_str = qemu_opt_get(opts, "data.mem");
+        }
         /* Fix up legacy suffix-less format */
         if (g_ascii_isdigit(mem_str[strlen(mem_str) - 1])) {
             mem_size <<= 20;
@@ -211,16 +214,27 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)
     max_numa_nodeid = MAX(max_numa_nodeid, nodenr + 1);
 }
 
-static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
+int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
 {
     NumaOptions *object = NULL;
     Error *err = NULL;
+    Visitor *v;
 
-    {
-        Visitor *v = opts_visitor_new(opts);

numa.c static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp) · line 230

Quoted 10 lines
numa.c +8 -2
-        visit_type_NumaOptions(v, NULL, &object, &err);
-        visit_free(v);
+    /*
+     * Needs autocreate_list=true, permit_int_ranges=true and
+     * permit_repeated_opts=true in order to support existing
+     * syntax for 'cpus' parameter which is an int list.
+     *
+     * Needs autocreate_struct_levels=1, because existing syntax
+     * used a nested struct in the QMP schema with a flat namespace
+     * in the CLI args.

Which QAPI definition(s) do you allude to here?

If it's just union NumaOptions, that's not ABI. I'd simply flatten it. Sketch appended.

numa.c static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp) · line 240

Quoted 10 lines
numa.c +6 -0
+     */
+    v = qobject_input_visitor_new_opts(opts, true, 1, true, true, &err);
+    if (err) {
+        goto end;
     }
+    visit_type_NumaOptions(v, NULL, &object, &err);
+    visit_free(v);
 
     if (err) {
         goto end;

[Skipping test updates for now...]

numa.c +2 -2
index 9c09e45..792fa2a 100644
@@ -227,8 +227,8 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
     }
 
     switch (object->type) {
-    case NUMA_OPTIONS_KIND_NODE:
-        numa_node_parse(object->u.node.data, opts, &err);
+    case NUMA_OPTIONS_TYPE_NODE:
+        numa_node_parse(&object->u.node, opts, &err);
         if (err) {
             goto end;
         }
qapi-schema.json +3 -0
index ded1179..4838258 100644
@@ -4255,7 +4255,10 @@
 #
 # Since 2.1
 ##
+{ 'enum': 'NumaOptionsType', 'data': ['node'] }
 { 'union': 'NumaOptions',
+  'base': { 'type': 'NumaOptionsType' },
+  'discriminator': 'type',
   'data': {
     'node': 'NumaNodeOptions' }}
 
#103 lore raw

Re: [Qemu-devel] [PATCH v14 20/21] net: convert to QObjectInputVisitor for -net/-netdev parsing

In reply to Daniel P. Berrange
Quoted 45 lines · Daniel P. Berrange

The -net/-netdev command line parsing code uses OptsVisitor for parsing options to populate NetLegacy or NetDev struct respectively. Although those structs have nesting, the OptsVisitor flattens them, so we must enable compatibility options to auto-create structs. This allows the legacy syntax

-net tap,id=net0,vlan=3,fd=3,script=/bin/ifup-qemu

to be treated as equivalent to the modern QAPI based syntax

-net id=net0,vlan=3,opts.type=tap,opts.data.fd=3,opts.data.script=/bin/ifup-qemu

---
 net/net.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
net/net.c +9 -2
index ec984bf..de6bf8e 100644
@@ -43,7 +43,7 @@
 #include "qemu/iov.h"
 #include "qemu/main-loop.h"
 #include "qapi-visit.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "sysemu/sysemu.h"
 #include "net/filter.h"
 #include "qapi/string-output-visitor.h"
@@ -1069,7 +1069,21 @@ int net_client_init(QemuOpts *opts, bool is_netdev, Error **errp)
     void *object = NULL;
     Error *err = NULL;
     int ret = -1;
-    Visitor *v = opts_visitor_new(opts);
+    /*
+     * Needs autocreate_lists=true in order support existing
+     * syntax for list options where the bare key is repeated
+     *
+     * Needs autocreate_struct_levels=3 in order to deal with
+     * 3 level nesting in NetLegacy option args, which was
+     * exposed as a flat namespace with OptVisitor
+     */

net/net.c int net_client_init(QemuOpts *opts, bool is_netdev, Error **errp) · line 1089

Quoted 10 lines
net/net.c +7 -0
+    Visitor *v = qobject_input_visitor_new_opts(opts, true, 3, false, true,
+                                                &err);
+
+    if (err) {
+        error_propagate(errp, err);
+        return -1;
+    }
 
     {
         /* Parse convenience option format ip6-net=fec0::0[/64] */

Neither NetLegacy nor Netdev are ABI, so if I understand the problem, perhaps I can find a way around it. Let's figure out what exactly requires levels=3.

Your commit message shows opts.* and opts.data.* in -net. Anything else?

I can see opts.* in the QAPI schema: it's NetLegacy member @opts, of union type NetLegacyOptions. It's boxed on the wire, as any member of complex type is.

I can also see opts.data.*: NetLegacyOptions is a simple union, which wraps 'data': { ... } around the variant on the wire.

The latter wrapping is easy to avoid: make NetLegacyOptions a flat union.

The former wrapping isn't so easy. QAPI can't unbox complex members. The only way to include a type unboxed is making it the base type. But base types must be struct, and @opts is a union.

However, if you take a step back you can see that the (flat) argument of -net / -netdev is fundamentally just a (flat) union! The nesting is an artifact of the clumsy way it's defined in the QAPI schema. Sketch doing it less clumsily appended; look ma, no nesting! Compile-tested only.

net/net.c +23 -24
index ec984bf..f2cbf28 100644
@@ -984,55 +984,54 @@ static int net_client_init1(const void *object, bool is_netdev, Error **errp)
         }
     } else {
         const NetLegacy *net = object;
-        const NetLegacyOptions *opts = net->opts;
         legacy.id = net->id;
         netdev = &legacy;
         /* missing optional values have been initialized to "all bits zero" */
         name = net->has_id ? net->id : net->name;
 
         /* Map the old options to the new flat type */
-        switch (opts->type) {
-        case NET_LEGACY_OPTIONS_KIND_NONE:
+        switch (net->type) {
+        case NET_LEGACY_TYPE_NONE:
             return 0; /* nothing to do */
-        case NET_LEGACY_OPTIONS_KIND_NIC:
+        case NET_LEGACY_TYPE_NIC:
             legacy.type = NET_CLIENT_DRIVER_NIC;
-            legacy.u.nic = *opts->u.nic.data;
+            legacy.u.nic = net->u.nic;
             break;
-        case NET_LEGACY_OPTIONS_KIND_USER:
+        case NET_LEGACY_TYPE_USER:
             legacy.type = NET_CLIENT_DRIVER_USER;
-            legacy.u.user = *opts->u.user.data;
+            legacy.u.user = net->u.user;
             break;
-        case NET_LEGACY_OPTIONS_KIND_TAP:
+        case NET_LEGACY_TYPE_TAP:
             legacy.type = NET_CLIENT_DRIVER_TAP;
-            legacy.u.tap = *opts->u.tap.data;
+            legacy.u.tap = net->u.tap;
             break;
-        case NET_LEGACY_OPTIONS_KIND_L2TPV3:
+        case NET_LEGACY_TYPE_L2TPV3:
             legacy.type = NET_CLIENT_DRIVER_L2TPV3;
-            legacy.u.l2tpv3 = *opts->u.l2tpv3.data;
+            legacy.u.l2tpv3 = net->u.l2tpv3;
             break;
-        case NET_LEGACY_OPTIONS_KIND_SOCKET:
+        case NET_LEGACY_TYPE_SOCKET:
             legacy.type = NET_CLIENT_DRIVER_SOCKET;
-            legacy.u.socket = *opts->u.socket.data;
+            legacy.u.socket = net->u.socket;
             break;
-        case NET_LEGACY_OPTIONS_KIND_VDE:
+        case NET_LEGACY_TYPE_VDE:
             legacy.type = NET_CLIENT_DRIVER_VDE;
-            legacy.u.vde = *opts->u.vde.data;
+            legacy.u.vde = net->u.vde;
             break;
-        case NET_LEGACY_OPTIONS_KIND_DUMP:
+        case NET_LEGACY_TYPE_DUMP:
             legacy.type = NET_CLIENT_DRIVER_DUMP;
-            legacy.u.dump = *opts->u.dump.data;
+            legacy.u.dump = net->u.dump;
             break;
-        case NET_LEGACY_OPTIONS_KIND_BRIDGE:
+        case NET_LEGACY_TYPE_BRIDGE:
             legacy.type = NET_CLIENT_DRIVER_BRIDGE;
-            legacy.u.bridge = *opts->u.bridge.data;
+            legacy.u.bridge = net->u.bridge;
             break;
-        case NET_LEGACY_OPTIONS_KIND_NETMAP:
+        case NET_LEGACY_TYPE_NETMAP:
             legacy.type = NET_CLIENT_DRIVER_NETMAP;
-            legacy.u.netmap = *opts->u.netmap.data;
+            legacy.u.netmap = net->u.netmap;
             break;
-        case NET_LEGACY_OPTIONS_KIND_VHOST_USER:
+        case NET_LEGACY_TYPE_VHOST_USER:
             legacy.type = NET_CLIENT_DRIVER_VHOST_USER;
-            legacy.u.vhost_user = *opts->u.vhost_user.data;
+            legacy.u.vhost_user = net->u.vhost_user;
             break;
         default:
             abort();
@@ -1047,7 +1046,7 @@ static int net_client_init1(const void *object, bool is_netdev, Error **errp)
 
         /* Do not add to a vlan if it's a nic with a netdev= parameter. */
         if (netdev->type != NET_CLIENT_DRIVER_NIC ||
-            !opts->u.nic.data->has_netdev) {
+            !net->u.nic.has_netdev) {
             peer = net_hub_add_port(net->has_vlan ? net->vlan : 0, NULL);
         }
     }
qapi-schema.json +8 -13
index 4838258..d863418 100644
@@ -2915,21 +2915,16 @@
 #
 # Since 1.2
 ##
-{ 'struct': 'NetLegacy',
-  'data': {
+{ 'enum': 'NetLegacyType',
+  'data': [ 'none', 'nic', 'user', 'tap', 'l2tpv3', 'socket', 'vde',
+            'dump', 'bridge', 'netmap', 'vhost-user' ] }
+{ 'union': 'NetLegacy',
+  'base': {
+    'type': 'NetLegacyType',
     '*vlan': 'int32',
     '*id':   'str',
-    '*name': 'str',
-    'opts':  'NetLegacyOptions' } }
-
-##
-# @NetLegacyOptions
-#
-# Like Netdev, but for use only by the legacy command line options
-#
-# Since 1.2
-##
-{ 'union': 'NetLegacyOptions',
+    '*name': 'str' },
+  'discriminator': 'type',
   'data': {
     'none':     'NetdevNoneOptions',
     'nic':      'NetLegacyNicOptions',
#104 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH v14 20/21] net: convert to QObjectInputVisitor for -net/-netdev parsing

Quoted 21 lines · Markus Armbruster
Quoted 21 lines
(quoted diff) +15 -1
@@ -1069,7 +1069,21 @@ int net_client_init(QemuOpts *opts, bool is_netdev, Error **errp)
     void *object = NULL;
     Error *err = NULL;
     int ret = -1;
-    Visitor *v = opts_visitor_new(opts);
+    /*
+     * Needs autocreate_lists=true in order support existing
+     * syntax for list options where the bare key is repeated
+     *
+     * Needs autocreate_struct_levels=3 in order to deal with
+     * 3 level nesting in NetLegacy option args, which was
+     * exposed as a flat namespace with OptVisitor
+     */
+    Visitor *v = qobject_input_visitor_new_opts(opts, true, 3, false, true,
+                                                &err);
+
+    if (err) {
+        error_propagate(errp, err);
+        return -1;
+    }
 
Quoted 6 lines

Review int net_client_init(QemuOpts *opts, bool is_netdev, Error **errp) · line 1089

Quoted 2 lines
(quoted diff) +0 -0
     {
         /* Parse convenience option format ip6-net=fec0::0[/64] */

Neither NetLegacy nor Netdev are ABI, so if I understand the problem, perhaps I can find a way around it. Let's figure out what exactly requires levels=3.

Netdev is not ABI only because we decided to NOT apply the last patch of QAPI-fying it in 2.7 while deciding to handle the back-compat (non-?)issues that existing netdev_add QMP command accepts both 1 and "1", but using the Netdev type would accept only 1.

While working towards making netdev_add use QAPI, I intentionally left NetLegacy unchanged; but since NetLegacy is solely used by the command line, feel free to rearrange that type as long as CLI back-compat is kept.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#7 lore raw

Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict

In reply to Markus Armbruster
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 289 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 261 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 551 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..e0d24e1 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +0 -0
index 60f158c..c38e90e 100644

[...]

Quoted 6 lines
Quoted 4 lines

+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionaries

Is recursive=false used outside tests in this series?

No, its not used.

It was suggested in a way earlier version by Max, but not sure if his code uses it or not.

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#43 lore raw

Re: [Qemu-devel] [PATCH v14 09/21] qapi: permit auto-creating single element lists

In reply to Markus Armbruster
Quoted 17 lines · Markus Armbruster
Quoted 2 lines · Daniel P. Berrange

When converting QemuOpts to a QObject, there is no information about compound types available,

Yes, that's a drawback of splitting the conversion into a QemuOpts -> QObject part that is oblivious of types, and a QObject -> QAPI object part that knows the types.

Quoted 8 lines

so when visiting a list, the
corresponding QObject is not guaranteed to be a QList. We therefore need to be able to auto-create a single element QList from whatever type we find.

This mode should only be enabled if you have compatibility requirements for

Quoted 8 lines
Quoted 5 lines

-arg foo=hello,foo=world

to be treated as equivalent to the preferred syntax:

   -arg foo.0=hello,foo.1=world

Not sure this is "preferred". "More powerfully warty" is probably closer to the truth ;)

Well, I call it "preferred" in the sense that that option syntax directly maps to the QAPI syntax in an unambigous manner. ie given the arg value alone "foo.0=hello,foo.1=world" you can clearly determine that "foo" is a list. With the compat syntax you cannot distinguish list from scalar, without knowing the QAPI schema.

Quoted 1 line

How is "-arg foo=hello,foo=world" treated if this mode isn't enabled?

The default behaviour would be that only the last key is present in the dict, eg foo=world, and then if you tried to visit a list, the visitor would complain that its got a QString instead of QList for the key 'foo'.

This is related to patch 14

Quoted 2 lines

What would be the drawbacks of doing this always instead of only when we "have compatibility requirements"?

Essentially we'd be permanently allowing 2 distinct syntaxes for dealing with lists, for all options. I felt it desirable that we have only a single syntax and only allow this alt syntax in the backcompat cases.

Quoted 29 lines
Quoted 29 lines
qapi/qobject-input-visitor.c +6 -0
index d9269c9..d88e9f9 100644
@@ -48,6 +48,10 @@ struct QObjectInputVisitor
 
     /* True to reject parse in visit_end_struct() if unvisited keys remain. */
     bool strict;
+
+    /* Whether we can auto-create single element lists when
+     * encountering a non-QList type */
+    bool autocreate_list;
 };
 
 static QObjectInputVisitor *to_qiv(Visitor *v)
@@ -108,6 +112,7 @@ static const QListEntry *qobject_input_push(QObjectInputVisitor *qiv,
     assert(obj);
     tos->obj = obj;
     tos->qapi = qapi;
+    qobject_incref(obj);
 
     if (qiv->strict && qobject_type(obj) == QTYPE_QDICT) {
         h = g_hash_table_new(g_str_hash, g_str_equal);
@@ -147,6 +152,7 @@ static void qobject_input_stack_object_free(StackObject *tos)
     if (tos->h) {
         g_hash_table_unref(tos->h);
     }
+    qobject_decref(tos->obj);
 
Quoted 4 lines

qapi/qobject-input-visitor.c static void qobject_input_stack_object_free(StackObject *tos) · line 158

Quoted 2 lines
qapi/qobject-input-visitor.c +0 -0
     g_free(tos);
 }

Can you explain the reference counting change?

Previously the stack stored a borrowed reference, since it didn't ever need responsibility for free'ing the object when popping the stack. This is no longer the case if you look a few lines later....

Quoted 18 lines

qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 206

Quoted 7 lines
qapi/qobject-input-visitor.c +1 -1
@@ -197,7 +203,7 @@ static void qobject_input_start_list(Visitor *v, const char *name,
     QObject *qobj = qobject_input_get_object(qiv, name, true);
     const QListEntry *entry;
 
-    if (!qobj || qobject_type(qobj) != QTYPE_QLIST) {
+    if (!qobj || (!qiv->autocreate_list && qobject_type(qobj) != QTYPE_QLIST)) {

Long line, but I believe it'll go away when you rebase for commit 1382d4a.

Quoted 7 lines
qapi/qobject-input-visitor.c +0 -0
         if (list) {
             *list = NULL;
         }
@@ -206,7 +212,16 @@ static void qobject_input_start_list(Visitor *v, const char *name,
         return;
     }
 
Quoted 7 lines

qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 220

Quoted 7 lines
qapi/qobject-input-visitor.c +6 -1
-    entry = qobject_input_push(qiv, qobj, list, errp);
+    if (qobject_type(qobj) != QTYPE_QLIST) {
+        QList *tmplist = qlist_new();
+        qlist_append_obj(tmplist, qobj);
+        qobject_incref(qobj);
+        entry = qobject_input_push(qiv, QOBJECT(tmplist), list, errp);
+        QDECREF(tmplist);

... here we are storing the 'qmplist' in the stack, and so when popping the stack, we must free that object. We thus need the stack to always hold its own reference, so when popping it can decref and (potentially) release the last reference.

Quoted 10 lines

qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 227

Quoted 7 lines
qapi/qobject-input-visitor.c +4 -0
+    } else {
+        entry = qobject_input_push(qiv, qobj, list, errp);
+    }
+
     if (list) {
         if (entry) {
             *list = g_malloc0(size);

Buries autolist behavior in the middle of things. What about doing it first, so it's more separate?

I'm not sure I understand what you mean here ?

Quoted 50 lines
       QObjectInputVisitor *qiv = to_qiv(v);
       QObject *qobj = qobject_input_get_object_(qiv, name, true, errp);
       const QListEntry *entry;

       if (!qobj) {
           return;
       }

+ if (qiv->autocreate_list && qobject_type(qobj) != QTYPE_QLIST) {

  +        QList *auto_list = qlist_new();
  +        qlist_append_obj(auto_list, qobj);
  +        qobj = auto_list;
  +    }
  +
       if (qobject_type(qobj) != QTYPE_QLIST) {

I ignored reference counting here, because I don't yet understand how and why you're changing it.

Quoted 30 lines
qapi/qobject-input-visitor.c +4 -1
@@ -514,7 +529,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)
     return &v->visitor;
 }
 
-Visitor *qobject_input_visitor_new_autocast(QObject *obj)
+Visitor *qobject_input_visitor_new_autocast(QObject *obj,
+                                            bool autocreate_list)
 {
     QObjectInputVisitor *v;
 
@@ -539,6 +555,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
     v->visitor.optional = qobject_input_optional;
     v->visitor.free = qobject_input_free;
     v->strict = true;
+    v->autocreate_list = autocreate_list;
 
     v->root = obj;
     qobject_incref(obj);
@@ -548,6 +565,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
 
 
 Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
+                                        bool autocreate_list,
                                         Error **errp)
 {
     QDict *pdict;
@@ -564,7 +582,8 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
         goto cleanup;
     }
 
Quoted 7 lines

qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 589

Quoted 6 lines
qapi/qobject-input-visitor.c +2 -1
-    v = qobject_input_visitor_new_autocast(pobj);
+    v = qobject_input_visitor_new_autocast(pobj,
+                                           autocreate_list);
  cleanup:
     qobject_decref(pobj);
     QDECREF(pdict);

[Skipping test updates for now...]

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#61 lore raw

Re: [Qemu-devel] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor

In reply to Markus Armbruster
Quoted 122 lines · Markus Armbruster
Quoted 107 lines · Daniel P. Berrange

The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:

-arg foo=5,foo=6,foo=7
-arg foo=5-7

This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.

This means that

-arg foo=5-7

is treated as equivalent to

-arg foo.0=5,foo.1=6,foo.2=7

Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.

---
 include/qapi/qobject-input-visitor.h |  23 ++++-
 qapi/qobject-input-visitor.c         | 158 ++++++++++++++++++++++++++--
 tests/test-qobject-input-visitor.c   | 195 +++++++++++++++++++++++++++++++++--
 3 files changed, 360 insertions(+), 16 deletions(-)

static void qobject_input_type_uint64(Visitor *v, const char *name,

qapi/qobject-input-visitor.c +58 -3
@@ -366,21 +438,85 @@ static void qobject_input_type_uint64_autocast(Visitor *v, const char *name,
                                                uint64_t *obj, Error **errp)
 {
     QObjectInputVisitor *qiv = to_qiv(v);
-    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
-                                                                true));
+    QString *qstr;
     unsigned long long ret;
+    char *end = NULL;
+    StackObject *tos;
+    bool inlist = false;
+
+    /* Preferentially generate values from a range, before
+     * trying to consume another QList element */
+    tos = QSLIST_FIRST(&qiv->stack);
+    if (tos) {
+        if (tos->range_val < tos->range_limit) {
+            *obj = tos->range_val + 1;
+            tos->range_val++;
+            return;
+        } else {
+            inlist = tos->entry != NULL;
+        }
+    }
 
+    qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                       true));
     if (!qstr || !qstr->string) {
         error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
                    "string");
         return;
     }
 
-    if (parse_uint_full(qstr->string, &ret, 0) < 0) {
+    if (parse_uint(qstr->string, &ret, &end, 0) < 0) {
         error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
         return;
     }
     *obj = ret;
+
+    /*
+     * If we have string that represents an integer range (5-24),
+     * parse the end of the range and set things up so we'll process
+     * the rest of the range before consuming another element
+     * from the QList.
+     */
+    if (end && *end) {
+        if (!qiv->permit_int_ranges) {
+            error_setg(errp,
+                       "Integer ranges are not permitted here");
+            return;
+        }
+        if (!inlist) {
+            error_setg(errp,
+                       "Integer ranges are only permitted when "
+                       "visiting list parameters");
+            return;
+        }
+        if (*end != '-') {
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name,
+                       "a number range");
+            return;
+        }
+        end++;
+        if (parse_uint_full(end, &ret, 0) < 0) {
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
+            return;
+        }
+        if (*obj > ret) {
+            error_setg(errp,
+                       "Parameter '%s' range start %" PRIu64
+                       " must be less than (or equal to) %llu",
+                       name, *obj, ret);
+            return;
+        }
+        if ((ret - *obj) > (QIV_RANGE_MAX - 1)) {
+            error_setg(errp,

qapi/qobject-input-visitor.c static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, · line 520

Quoted 10 lines
qapi/qobject-input-visitor.c +9 -0
+                       "Parameter '%s' range must be less than %d",
+                       name, QIV_RANGE_MAX);
+            return;
+        }
+        if (*obj != ret) {
+            tos->range_val = *obj;
+            tos->range_limit = ret;
+        }
+    }
 }

Duplicates the signed code, which is sad, but I don't have better ideas.

Quoted 2 lines

Except this one: are we actually using both the signed and the unsigned case now? If not, can we get rid of the one we don't use?

Out of the args that I converted in this series, I only see uint16List used. eg for -numa and -object hostmem

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#67 lore raw

Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options

In reply to Markus Armbruster
Quoted 103 lines · Markus Armbruster
Quoted 31 lines · Daniel P. Berrange

If given an option string such as

size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind

the qemu_opts_to_qdict() method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:

    size=QString("1024")
    nodes=QString("1-2")
    policy=QString("bind")

With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain

    size=QString("1024")
    nodes=QList([QString("10"),
                 QString("4-5"),
                 QString("1-2")])
    policy=QString("bind")

Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.

Quoted 3 lines

In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.

To serve as a replacement for the options visitor, this needs to be able to behave exactly the same together with a suitably hacked up QObject input visitor. Before I dive into the actual patch, let me summarize QemuOpts and options visitor behavior.

Warning, this is going to get ugly.

QemuOpts faithfully represents a key=value,... string as a list of QemuOpt. Each QemuOpt represents one key=value. They are in the same order. If key occurs multiple times in the string, it occurs just the same in the list.

*Except* key "id" is special: it's stored outside the list, and all but the first one are silently ignored.

Most users only ever get the last value of a key. Any non-last key=value are silently ignored.

We actually exploit this behavior to do defaults, by *prepending* them to the list. See the use of qemu_opts_set_defaults() in main().

A few users get all values of keys (other than key "id"):

* -device, in qdev_device_add() with callback set_property().

We first get "driver" and "bus" normally (silently ignoring non-last values, as usual). All other keys are device properties. To set them, we get all (key, value), ignore keys "driver" and "bus", and set the rest. If a key occurs multiple times, it gets set multiple times. This effectively ignores all but the last one, silently.

* -semihosting-config, in main() with callback add_semihosting_arg().

We first get a bunch of keys normally. Key "arg" is special: it may be repeated to build a list. To implement that, we get all (key, value), ignore keys other than "arg", and accumulate the values.

* -machine & friends, in main() with callback machine_set_property()

Similar to -device, only for machines, with "type" instead of "driver" and "bus".

* -spice, in qemu_spice_init() with callback add_channel()

Keys "tls-channel" and "plaintext-channel" may be used repeated to specify multiple channels. To implement that, we get all (key, value), ignore keys other than "tls-channel" and "plaintext-channel", and set up a channel for each of the others.

* -writeconfig, in config_write_opts() with callback config_write_opt()

We write out all keys in order.

* The options visitor, in opts_start_struct()

We convert the list of (key, value) to a hash table of (key, list of values). Most of the time, the list of values has exactly one element.

When the visitor's user asks for a scalar, we return the last element of the list of values, in lookup_scalar().

When the user asks for list elements, we return the elements of the list of values in order, in opts_next_list(), or if there are none, the empty list in opts_start_list().

Quoted 6 lines

Unlike the options visitor, this patch (judging from your description) makes a list only when keys are repeated. The QObject visitor will have to cope with finding both scalars and lists. When it finds a scalar, but needs a list, it'll have to wrap it in a list (PATCH 09, I think). When it finds a list, but needs a scalar, it'll have to fish it out of the list (where is that?).

If my code finds a list but wants a scalar, it is reporting an error.

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#71 lore raw

Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options

In reply to Markus Armbruster
Quoted 43 lines · Markus Armbruster
Quoted 23 lines · Daniel P. Berrange

If given an option string such as

size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind

the qemu_opts_to_qdict() method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:

    size=QString("1024")
    nodes=QString("1-2")
    policy=QString("bind")

With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain

    size=QString("1024")
    nodes=QList([QString("10"),
                 QString("4-5"),
                 QString("1-2")])
    policy=QString("bind")
Quoted 7 lines

Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.

Actually three cases, not two:

0. qdict does not contain the key means empty list.

1. qdict contains the key with a QString value means list of one element.

2. qdict contains the key with a QList value means list of more than one element.

Quoted 8 lines

I consider this weird. However, it's usefully weird with at least your QObject input visitor.

Quoted 3 lines

In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.

Got users for this policy in the pipeline?

I in fact used it in the QObjectInputVisitor, when the autocreate_list is not set.

I guess strictly speaking this is not back-compatible if someone is passing repeated keys, but I judged that rather than silently ignoring this incorrect usage, it was acceptable to report an error.

Quoted 18 lines
Quoted 18 lines
     QTAILQ_FOREACH(opt, &opts->head, next) {
         val = QOBJECT(qstring_from_str(opt->str));
util/qemu-option.c +15 -1
-        qdict_put_obj(qdict, opt->name, val);
+
+        if (qdict_haskey(ret, opt->name)) {
+            switch (repeatPolicy) {
+            case QEMU_OPTS_REPEAT_POLICY_ERROR:
+                error_setg(errp, "Option '%s' appears more than once",
+                           opt->name);
+                qobject_decref(val);
+                if (!qdict) {
+                    QDECREF(ret);
+                }
+                return NULL;
+
+            case QEMU_OPTS_REPEAT_POLICY_ALL:
+                prevval = qdict_get(ret, opt->name);
+                if (qobject_type(prevval) == QTYPE_QLIST) {
Quoted 8 lines

util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1110

Quoted 6 lines
util/qemu-option.c +6 -0
+                    /* Already identified this key as a list */
+                    list = qobject_to_qlist(prevval);
+                } else {
+                    /* Replace current scalar with a list */
+                    list = qlist_new();
+                    qobject_incref(prevval);

Where is this reference decremented?

'prevval' is a borrowed reference from 'ret', against the key opt->name.

qdict_put_obj decrements the reference we borrowed from ret against the key opt->name.

qlist_append_obj() takes ownership of the reference it is passed, so we must qobject_incref() to avoid qdict_put_obj free'ing prevval.

When we call qdict_put_obj() we're replacing the value currently associted

Quoted 52 lines
Quoted 5 lines
util/qemu-option.c +4 -0
+                    qlist_append_obj(list, prevval);
+                    qdict_put_obj(ret, opt->name, QOBJECT(list));
+                }
+                qlist_append_obj(list, val);

util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1124

Quoted 10 lines
util/qemu-option.c +10 -0
+                break;
+
+            case QEMU_OPTS_REPEAT_POLICY_LAST:
+                /* Just discard previously set value */
+                qdict_put_obj(ret, opt->name, val);
+                break;
+            }
+        } else {
+            qdict_put_obj(ret, opt->name, val);
+        }

A possible alternative:

           val = QOBJECT(qstring_from_str(opt->str));

           if (qdict_haskey(ret, opt->name)) {
               switch (repeatPolicy) {
               case QEMU_OPTS_REPEAT_POLICY_ERROR:
                   error_setg(errp, "Option '%s' appears more than once",
                              opt->name);
                   qobject_decref(val);
                   if (!qdict) {
                       QDECREF(ret);
                   }
                   return NULL;

               case QEMU_OPTS_REPEAT_POLICY_ALL:
                   prevval = qdict_get(ret, opt->name);
                   if (qobject_type(prevval) == QTYPE_QLIST) {
                       /* Already identified this key as a list */
                       list = qobject_to_qlist(prevval);
                   } else {
                       /* Replace current scalar with a list */
                       list = qlist_new();
                       qobject_incref(prevval);
                       qlist_append_obj(list, prevval);
                   }
                   qlist_append_obj(list, val);
                   val = QOBJECT(list);
                   break;

               case QEMU_OPTS_REPEAT_POLICY_LAST:
                   break;
               }
           }
           qdict_put_obj(ret, opt->name, val);
Quoted 2 lines

This shows the common part of the behavior more clearly. Matter of taste, you get to use your artistic license.

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#86 lore raw

Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

In reply to Markus Armbruster
Quoted 21 lines · Markus Armbruster
Cc Kevin for discussion of QemuOpts dotted key convention
Quoted 17 lines · Daniel P. Berrange

Currently qdict_crumple requires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.

In order to have backwards compatibility with the OptsVisitor, qemu_opt_to_qdict() has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given to qdict_crumple().

To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed

   foo=hello,foo=world,foo=wibble
   foo.0=hello,foo.1=world,foo.2=wibble

but this is forbidden

Quoted 8 lines
Quoted 1 line

foo=hello,foo=world,foo.2=wibble

I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.

The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?

Two reasons I did this. First blockdev already uses this foo.0=val syntax, and I wanted to be compatible with blockdev, so it could be converted to use this new code.

Second, using foo.0 syntax means that you can unambigously determine whether a key is going to be a scalar or a list. This lets the qdict_crumple() method convert the QemuOpts to a QDict without needing to know anything about the QAPI schema.

Of course I later had to add hacks to the visitor to cope with the bare repeated key syntax, so I lost some of that benefit.

Personally I still prefer the unambiguous syntax as it lets us give clear error messages when users do unexpected things, instead of say, silently ignoring all but the last key.

Quoted 35 lines

Note that this second way creates entirely new failure modes and restrictions. Let me show using an example derived from one in qdict_crumple()'s contract:

foo.0.bar=bla,foo.eek.bar=blubb

Without the dotted key convention, this is perfectly fine: key "foo.0.bar" has the single value "bla", and key "foo.eek.bar" has the single value "blubb". Equivalent JSON would be

{ "foo.0.bar": "bla", "foo.eek.bar": "blubb" }

With just the struct convention, it's still fine: it obviously means the same as JSON

{ "foo": { "0": { "bar": "bla" }, "eek": { "bar": "blubb" } } }

Adding the list convention makes it invalid. It also outlaws a bunch of keys that would be just fine in JSON, namely any that get recognized as list index. Raise your hand if you're willing to bet real money on your predictions of what will be recognized as list index, without looking at the code. I'm not.

I'm afraid I have growing doubts regarding the QemuOpts dotted key convention in general.

The convention makes '.' a special character in keys, but only sometimes. If the key gets consumed by something that uses dotted key convention, '.' is special, and to get a non-special '.', you need to escape it by doubling. Else, it's not.

Since the same key can be used differently by different code, the same '.' could in theory be both special and non-special. In practice, this would be madness.

Quoted 5 lines

Adopting the dotted key convention for an existing QemuOpts option, say -object [PATCH 15], *breaks* existing command line usage of keys containing '.', because you now have to escape the '.'. Dan, I'm afraid you need to show that no such keys exist, or if they exist they don't matter.

I checked the things that I converted (eg -net, -object, -numa, etc), but I didn't check -device since that's processed using different code.

Quoted 8 lines

I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.

It's probably too late to back out the dotted key convention completely. Kevin?

Quoted 7 lines

Can we still back out the list part of the convention, and use repeated keys instead?

If we're stuck with some form of the dotted key convention, can we at least make it a more integral part of QemuOpts rather than something bolted on as an afterthought? Here's my thinking on how that might be done:

The only issue with dropping the dotted list convention is compat with the block layer code - we couldn't easily use this new visitor logic to turn -drive into a QAPI BlockOptions object. Kevin's new -blockdev arg would potentially be ok with it since its a new arg, but IIUC, we would have to do some cleanup inside various block driver impls, since block layer doesn't use the QAPI objects internally - they all get converted back into QemuOpts :-(

Quoted 13 lines

* Have a QemuOptsList flag @flat.

* If @flat, QemuOpts behaves as it always has: the special characters are ',' and '=', and parsing a key=value,... string produces a list where each element represents one key=value from the string, in the same order.

* If not @flat, '.' becomes an additional special character, and parsing a key=value,... string produces a dictionary, similar to the one you get now by converting with qemu_opts_to_qdict() and filtering through qdict_crumple().

Quoted 6 lines

The difference to now is that you either always crumple, or not at all, and the meaning of '.' is unambiguous.

I wish we had refrained from saddling QemuOpts with even more magic. Compared to this swamp, use of JSON on the command line looks rather appealing to me.

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#8 lore raw

Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict

In reply to Daniel P. Berrange
Quoted 93 lines · Daniel P. Berrange
Quoted 92 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 289 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 261 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 551 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..e0d24e1 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +0 -0
index 60f158c..c38e90e 100644

[...]

Quoted 4 lines

+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionaries

Quoted 6 lines
Quoted 1 line

Is recursive=false used outside tests in this series?

No, its not used.

It was suggested in a way earlier version by Max, but not sure if his code uses it or not.

In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.

Max, do you actually need this one? If yes, please explain your use case.

#21 lore raw

Re: [Qemu-devel] [PATCH v14 03/21] qapi: add trace events for visitor

In reply to Daniel P. Berrange
Quoted 39 lines · Daniel P. Berrange
Quoted 31 lines · Markus Armbruster
Quoted 29 lines · Daniel P. Berrange

Allow tracing of the operation of visitors

---
 Makefile.objs          |  1 +
 qapi/qapi-visit-core.c | 27 +++++++++++++++++++++++++++
 qapi/trace-events      | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+)
 create mode 100644 qapi/trace-events
Makefile.objs +1 -0
index a8e0224..b3e8aef 100644
@@ -160,3 +160,4 @@ trace-events-y += target-s390x/trace-events
 trace-events-y += target-ppc/trace-events
 trace-events-y += qom/trace-events
 trace-events-y += linux-user/trace-events
+trace-events-y += qapi/trace-events
qapi/qapi-visit-core.c +1 -0
index 55f5876..bfcb276 100644
@@ -19,10 +19,12 @@
 #include "qapi/qmp/qerror.h"
 #include "qapi/visitor.h"
 #include "qapi/visitor-impl.h"
+#include "trace.h"
 
Quoted 6 lines

qapi/qapi-visit-core.c line 27

Quoted 4 lines
qapi/qapi-visit-core.c +1 -0
 void visit_complete(Visitor *v, void *opaque)
 {
     assert(v->type != VISITOR_OUTPUT || v->complete);
+    trace_visit_complete(v, opaque);

Trace after or before checking the precondition? Preferences, anyone?

Quoted 4 lines

I'm ambivalent, as the assert will crash you either way so whether you get a trace event just before the crash seems mostly irrelevant to me - the abort stack trace is what you'll use to diagnose the crash.

I have a slight preference for trace first, because trace is one call, while precondition checking can be several. However, grep shows neither order is prevalent. Not worth a respin.

Quoted 107 lines
Quoted 107 lines
Quoted 90 lines
qapi/qapi-visit-core.c +11 -0
     if (v->complete) {
         v->complete(v, opaque);
     }
@@ -30,6 +32,7 @@ void visit_complete(Visitor *v, void *opaque)
 
 void visit_free(Visitor *v)
 {
+    trace_visit_free(v);
     if (v) {
         v->free(v);
     }
@@ -40,6 +43,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,
 {
     Error *err = NULL;
 
+    trace_visit_start_struct(v, name, obj, size);
     if (obj) {
         assert(size);
         assert(!(v->type & VISITOR_OUTPUT) || *obj);
@@ -53,6 +57,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,
 
 void visit_check_struct(Visitor *v, Error **errp)
 {
+    trace_visit_check_struct(v);
     if (v->check_struct) {
         v->check_struct(v, errp);
     }
@@ -60,6 +65,7 @@ void visit_check_struct(Visitor *v, Error **errp)
 
 void visit_end_struct(Visitor *v, void **obj)
 {
+    trace_visit_end_struct(v, obj);
     v->end_struct(v, obj);
 }
 
@@ -69,6 +75,7 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,
     Error *err = NULL;
 
     assert(!list || size >= sizeof(GenericList));
+    trace_visit_start_list(v, name, list, size);
     v->start_list(v, name, list, size, &err);
     if (list && (v->type & VISITOR_INPUT)) {
         assert(!(err && *list));
@@ -79,11 +86,13 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,
 GenericList *visit_next_list(Visitor *v, GenericList *tail, size_t size)
 {
     assert(tail && size >= sizeof(GenericList));
+    trace_visit_next_list(v, tail, size);
     return v->next_list(v, tail, size);
 }
 
 void visit_end_list(Visitor *v, void **obj)
 {
+    trace_visit_end_list(v, obj);
     v->end_list(v, obj);
 }
 
@@ -95,6 +104,7 @@ void visit_start_alternate(Visitor *v, const char *name,
 
     assert(obj && size >= sizeof(GenericAlternate));
     assert(!(v->type & VISITOR_OUTPUT) || *obj);
+    trace_visit_start_alternate(v, name, obj, size, promote_int);
     if (v->start_alternate) {
         v->start_alternate(v, name, obj, size, promote_int, &err);
     }
@@ -106,6 +116,7 @@ void visit_start_alternate(Visitor *v, const char *name,
 
 void visit_end_alternate(Visitor *v, void **obj)
 {
+    trace_visit_end_alternate(v, obj);
     if (v->end_alternate) {
         v->end_alternate(v, obj);
     }
@@ -113,6 +124,7 @@ void visit_end_alternate(Visitor *v, void **obj)
 
 bool visit_optional(Visitor *v, const char *name, bool *present)
 {
+    trace_visit_optional(v, name, present);
     if (v->optional) {
         v->optional(v, name, present);
     }
@@ -127,6 +139,7 @@ bool visit_is_input(Visitor *v)
 void visit_type_int(Visitor *v, const char *name, int64_t *obj, Error **errp)
 {
     assert(obj);
+    trace_visit_type_int(v, name, obj);
     v->type_int64(v, name, obj, errp);
 }
 

qapi/qapi-visit-core.c void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj, · line 170

Quoted 7 lines
qapi/qapi-visit-core.c +1 -0
@@ -151,6 +164,7 @@ void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj,
                       Error **errp)
 {
     uint64_t value = *obj;
+    trace_visit_type_uint8(v, name, obj);
     visit_type_uintN(v, &value, name, UINT8_MAX, "uint8_t", errp);
     *obj = value;
 }

Putting the trace in the middle of the "value = *obj; visit_type_uintN(v, &value, ...); *obj = value" pattern makes it less visible.

Preserving the pattern requires replacing the initializer by an assignment:

       uint64_t value;
Quoted 8 lines
Quoted 6 lines
       trace_visit_type_uint8(v, name, obj);
       value = *obj;
       visit_type_uintN(v, &value, name, UINT8_MAX, "uint8_t", errp);
       *obj = value;

Looks slightly more legible to me. What do you think?

I'm fine with whatever you prefer.

I can touch up this one on commit.

#62 lore raw

Re: [Qemu-devel] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor

In reply to Daniel P. Berrange
Quoted 123 lines · Daniel P. Berrange
Quoted 122 lines · Markus Armbruster
Quoted 107 lines · Daniel P. Berrange

The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:

-arg foo=5,foo=6,foo=7
-arg foo=5-7

This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.

This means that

-arg foo=5-7

is treated as equivalent to

-arg foo.0=5,foo.1=6,foo.2=7

Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.

---
 include/qapi/qobject-input-visitor.h |  23 ++++-
 qapi/qobject-input-visitor.c         | 158 ++++++++++++++++++++++++++--
 tests/test-qobject-input-visitor.c   | 195 +++++++++++++++++++++++++++++++++--
 3 files changed, 360 insertions(+), 16 deletions(-)

static void qobject_input_type_uint64(Visitor *v, const char *name,

qapi/qobject-input-visitor.c +58 -3
@@ -366,21 +438,85 @@ static void qobject_input_type_uint64_autocast(Visitor *v, const char *name,
                                                uint64_t *obj, Error **errp)
 {
     QObjectInputVisitor *qiv = to_qiv(v);
-    QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
-                                                                true));
+    QString *qstr;
     unsigned long long ret;
+    char *end = NULL;
+    StackObject *tos;
+    bool inlist = false;
+
+    /* Preferentially generate values from a range, before
+     * trying to consume another QList element */
+    tos = QSLIST_FIRST(&qiv->stack);
+    if (tos) {
+        if (tos->range_val < tos->range_limit) {
+            *obj = tos->range_val + 1;
+            tos->range_val++;
+            return;
+        } else {
+            inlist = tos->entry != NULL;
+        }
+    }
 
+    qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+                                                       true));
     if (!qstr || !qstr->string) {
         error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
                    "string");
         return;
     }
 
-    if (parse_uint_full(qstr->string, &ret, 0) < 0) {
+    if (parse_uint(qstr->string, &ret, &end, 0) < 0) {
         error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
         return;
     }
     *obj = ret;
+
+    /*
+     * If we have string that represents an integer range (5-24),
+     * parse the end of the range and set things up so we'll process
+     * the rest of the range before consuming another element
+     * from the QList.
+     */
+    if (end && *end) {
+        if (!qiv->permit_int_ranges) {
+            error_setg(errp,
+                       "Integer ranges are not permitted here");
+            return;
+        }
+        if (!inlist) {
+            error_setg(errp,
+                       "Integer ranges are only permitted when "
+                       "visiting list parameters");
+            return;
+        }
+        if (*end != '-') {
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name,
+                       "a number range");
+            return;
+        }
+        end++;
+        if (parse_uint_full(end, &ret, 0) < 0) {
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
+            return;
+        }
+        if (*obj > ret) {
+            error_setg(errp,
+                       "Parameter '%s' range start %" PRIu64
+                       " must be less than (or equal to) %llu",
+                       name, *obj, ret);
+            return;
+        }
+        if ((ret - *obj) > (QIV_RANGE_MAX - 1)) {
+            error_setg(errp,

qapi/qobject-input-visitor.c static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, · line 520

Quoted 10 lines
qapi/qobject-input-visitor.c +9 -0
+                       "Parameter '%s' range must be less than %d",
+                       name, QIV_RANGE_MAX);
+            return;
+        }
+        if (*obj != ret) {
+            tos->range_val = *obj;
+            tos->range_limit = ret;
+        }
+    }
 }

Duplicates the signed code, which is sad, but I don't have better ideas.

Quoted 5 lines
Quoted 2 lines

Except this one: are we actually using both the signed and the unsigned case now? If not, can we get rid of the one we don't use?

Out of the args that I converted in this series, I only see uint16List used. eg for -numa and -object hostmem

Let's drop the signed copy then.

#68 lore raw

Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options

In reply to Daniel P. Berrange
Quoted 42 lines · Daniel P. Berrange
Quoted 41 lines · Markus Armbruster
Quoted 31 lines · Daniel P. Berrange

If given an option string such as

size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind

the qemu_opts_to_qdict() method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:

    size=QString("1024")
    nodes=QString("1-2")
    policy=QString("bind")

With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain

    size=QString("1024")
    nodes=QList([QString("10"),
                 QString("4-5"),
                 QString("1-2")])
    policy=QString("bind")

Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.

Quoted 3 lines

In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.

To serve as a replacement for the options visitor, this needs to be able to behave exactly the same together with a suitably hacked up QObject input visitor. Before I dive into the actual patch, let me summarize QemuOpts and options visitor behavior.

[...]

Quoted 9 lines
Quoted 6 lines

Unlike the options visitor, this patch (judging from your description) makes a list only when keys are repeated. The QObject visitor will have to cope with finding both scalars and lists. When it finds a scalar, but needs a list, it'll have to wrap it in a list (PATCH 09, I think). When it finds a list, but needs a scalar, it'll have to fish it out of the list (where is that?).

If my code finds a list but wants a scalar, it is reporting an error.

I'm afraid that breaks things like

    --numa node,cpus=0,cpus=2,mem=512,nodeid=0,nodeid=0

The options visitor inteprets cpus=0,cpus=2 as a list [0,2] (because NumaNodeOptions member @cpus has a list type) and nodeid=0,nodid=0 as integer 0 (because member @nodeid has a scalar type).

#69 lore raw

Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options

Quoted 55 lines · Markus Armbruster
Quoted 42 lines · Daniel P. Berrange
Quoted 41 lines · Markus Armbruster
Quoted 31 lines · Daniel P. Berrange

If given an option string such as

size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind

the qemu_opts_to_qdict() method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:

    size=QString("1024")
    nodes=QString("1-2")
    policy=QString("bind")

With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain

    size=QString("1024")
    nodes=QList([QString("10"),
                 QString("4-5"),
                 QString("1-2")])
    policy=QString("bind")

Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.

Quoted 3 lines

In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.

To serve as a replacement for the options visitor, this needs to be able to behave exactly the same together with a suitably hacked up QObject input visitor. Before I dive into the actual patch, let me summarize QemuOpts and options visitor behavior.

[...]

Quoted 9 lines
Quoted 6 lines

Unlike the options visitor, this patch (judging from your description) makes a list only when keys are repeated. The QObject visitor will have to cope with finding both scalars and lists. When it finds a scalar, but needs a list, it'll have to wrap it in a list (PATCH 09, I think). When it finds a list, but needs a scalar, it'll have to fish it out of the list (where is that?).

If my code finds a list but wants a scalar, it is reporting an error.

Quoted 7 lines

I'm afraid that breaks things like

    --numa node,cpus=0,cpus=2,mem=512,nodeid=0,nodeid=0

The options visitor inteprets cpus=0,cpus=2 as a list [0,2] (because NumaNodeOptions member @cpus has a list type) and nodeid=0,nodid=0 as integer 0 (because member @nodeid has a scalar type).

Right, but that command line syntax is arguably broken already - the app/user invoking QEMU is buggy, so I felt it was justifiable to change from silently ignoring the app error to telling them of their mistake.

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#72 lore raw

Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options

In reply to Daniel P. Berrange
Quoted 53 lines · Daniel P. Berrange
Quoted 43 lines · Markus Armbruster
Quoted 23 lines · Daniel P. Berrange

If given an option string such as

size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind

the qemu_opts_to_qdict() method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:

    size=QString("1024")
    nodes=QString("1-2")
    policy=QString("bind")

With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain

    size=QString("1024")
    nodes=QList([QString("10"),
                 QString("4-5"),
                 QString("1-2")])
    policy=QString("bind")
Quoted 7 lines

Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.

Actually three cases, not two:

0. qdict does not contain the key means empty list.

1. qdict contains the key with a QString value means list of one element.

2. qdict contains the key with a QList value means list of more than one element.

Quoted 8 lines

I consider this weird. However, it's usefully weird with at least your QObject input visitor.

Quoted 3 lines

In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.

Got users for this policy in the pipeline?

Quoted 7 lines

I in fact used it in the QObjectInputVisitor, when the autocreate_list is not set.

I guess strictly speaking this is not back-compatible if someone is passing repeated keys, but I judged that rather than silently ignoring this incorrect usage, it was acceptable to report an error.

While usage like

-machine usb=on,usb=off

could perhaps be declared erroneous post hoc, the "last one wins" feature also has genuinely useful applications. For instance, tack a correction to a long command line:

-machine usb=on [... lots of other stuff ...] -machine usb=off

Or modify canned configuration:

-readconfig vm1.config -machine usb=off
# vm1.config sets usb=on, -machine overrides

-readconfig vm1.config -set drive.disk0.file=tmp.qcow2 # vm1.config defines drive "disk0", -set overrides its image file

I don't think we can break them all.

[...]

#44 lore raw

Re: [Qemu-devel] [PATCH v14 09/21] qapi: permit auto-creating single element lists

In reply to Daniel P. Berrange
Quoted 42 lines · Daniel P. Berrange
Quoted 17 lines · Markus Armbruster
Quoted 2 lines · Daniel P. Berrange

When converting QemuOpts to a QObject, there is no information about compound types available,

Yes, that's a drawback of splitting the conversion into a QemuOpts -> QObject part that is oblivious of types, and a QObject -> QAPI object part that knows the types.

Quoted 8 lines

so when visiting a list, the
corresponding QObject is not guaranteed to be a QList. We therefore need to be able to auto-create a single element QList from whatever type we find.

This mode should only be enabled if you have compatibility requirements for

Quoted 8 lines
Quoted 5 lines

-arg foo=hello,foo=world

to be treated as equivalent to the preferred syntax:

   -arg foo.0=hello,foo.1=world

Not sure this is "preferred". "More powerfully warty" is probably closer to the truth ;)

Well, I call it "preferred" in the sense that that option syntax directly maps to the QAPI syntax in an unambigous manner. ie given the arg value alone "foo.0=hello,foo.1=world" you can clearly determine that "foo" is a list. With the compat syntax you cannot distinguish list from scalar, without knowing the QAPI schema.

Quoted 1 line

How is "-arg foo=hello,foo=world" treated if this mode isn't enabled?

The default behaviour would be that only the last key is present in the dict, eg foo=world, and then if you tried to visit a list, the visitor would complain that its got a QString instead of QList for the key 'foo'.

This is related to patch 14

Quoted 7 lines
Quoted 2 lines

What would be the drawbacks of doing this always instead of only when we "have compatibility requirements"?

Essentially we'd be permanently allowing 2 distinct syntaxes for dealing with lists, for all options. I felt it desirable that we have only a single syntax and only allow this alt syntax in the backcompat cases.

Fair point.

The bolted-on extensions (options visitor's integer list syntax, block layer's dotted key convention) have only ever worked in the places that choose to use them. Even the integrated support for repeated keys is only used for lists in the places that choose to use it that way.

Quoted 57 lines
Quoted 29 lines
Quoted 29 lines
qapi/qobject-input-visitor.c +6 -0
index d9269c9..d88e9f9 100644
@@ -48,6 +48,10 @@ struct QObjectInputVisitor
 
     /* True to reject parse in visit_end_struct() if unvisited keys remain. */
     bool strict;
+
+    /* Whether we can auto-create single element lists when
+     * encountering a non-QList type */
+    bool autocreate_list;
 };
 
 static QObjectInputVisitor *to_qiv(Visitor *v)
@@ -108,6 +112,7 @@ static const QListEntry *qobject_input_push(QObjectInputVisitor *qiv,
     assert(obj);
     tos->obj = obj;
     tos->qapi = qapi;
+    qobject_incref(obj);
 
     if (qiv->strict && qobject_type(obj) == QTYPE_QDICT) {
         h = g_hash_table_new(g_str_hash, g_str_equal);
@@ -147,6 +152,7 @@ static void qobject_input_stack_object_free(StackObject *tos)
     if (tos->h) {
         g_hash_table_unref(tos->h);
     }
+    qobject_decref(tos->obj);
 
Quoted 4 lines

qapi/qobject-input-visitor.c static void qobject_input_stack_object_free(StackObject *tos) · line 158

Quoted 2 lines
qapi/qobject-input-visitor.c +0 -0
     g_free(tos);
 }

Can you explain the reference counting change?

Previously the stack stored a borrowed reference, since it didn't ever need responsibility for free'ing the object when popping the stack. This is no longer the case if you look a few lines later....

Quoted 18 lines

qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 206

Quoted 7 lines
qapi/qobject-input-visitor.c +1 -1
@@ -197,7 +203,7 @@ static void qobject_input_start_list(Visitor *v, const char *name,
     QObject *qobj = qobject_input_get_object(qiv, name, true);
     const QListEntry *entry;
 
-    if (!qobj || qobject_type(qobj) != QTYPE_QLIST) {
+    if (!qobj || (!qiv->autocreate_list && qobject_type(qobj) != QTYPE_QLIST)) {

Long line, but I believe it'll go away when you rebase for commit 1382d4a.

Quoted 7 lines
qapi/qobject-input-visitor.c +0 -0
         if (list) {
             *list = NULL;
         }
@@ -206,7 +212,16 @@ static void qobject_input_start_list(Visitor *v, const char *name,
         return;
     }
 
Quoted 12 lines
Quoted 7 lines

qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 220

Quoted 7 lines
qapi/qobject-input-visitor.c +6 -1
-    entry = qobject_input_push(qiv, qobj, list, errp);
+    if (qobject_type(qobj) != QTYPE_QLIST) {
+        QList *tmplist = qlist_new();
+        qlist_append_obj(tmplist, qobj);
+        qobject_incref(qobj);
+        entry = qobject_input_push(qiv, QOBJECT(tmplist), list, errp);
+        QDECREF(tmplist);

... here we are storing the 'qmplist' in the stack, and so when popping the stack, we must free that object. We thus need the stack to always hold its own reference, so when popping it can decref and (potentially) release the last reference.

Aha.

Manual reference counting is a PITA.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
qapi/qobject-input-visitor.c +4 -0
+    } else {
+        entry = qobject_input_push(qiv, qobj, list, errp);
+    }
+
     if (list) {
         if (entry) {
             *list = g_malloc0(size);
Quoted 4 lines
Quoted 2 lines

Buries autolist behavior in the middle of things. What about doing it first, so it's more separate?

I'm not sure I understand what you mean here ?

I'm suggesting a structure like this:

1. Map special case to normal case
2. Deal with normal case

Keeping the normal case and the special case separate like that can make it easier to understand either.

Here's my attempt to do it:

Quoted 8 lines
Quoted 8 lines
       QObjectInputVisitor *qiv = to_qiv(v);
       QObject *qobj = qobject_input_get_object_(qiv, name, true, errp);
       const QListEntry *entry;

       if (!qobj) {
           return;
       }
Quoted 6 lines
Quoted 6 lines

+ if (qiv->autocreate_list && qobject_type(qobj) != QTYPE_QLIST) {

  +        QList *auto_list = qlist_new();
  +        qlist_append_obj(auto_list, qobj);
  +        qobj = auto_list;

+ }
+

The new code is the mapping. The normal case code below remains unchanged. It's just a sketch; I didn't spend a single brainwave on the reference counting.

Quoted 45 lines
Quoted 35 lines
       if (qobject_type(qobj) != QTYPE_QLIST) {

I ignored reference counting here, because I don't yet understand how and why you're changing it.

Quoted 30 lines
qapi/qobject-input-visitor.c +4 -1
@@ -514,7 +529,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)
     return &v->visitor;
 }
 
-Visitor *qobject_input_visitor_new_autocast(QObject *obj)
+Visitor *qobject_input_visitor_new_autocast(QObject *obj,
+                                            bool autocreate_list)
 {
     QObjectInputVisitor *v;
 
@@ -539,6 +555,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
     v->visitor.optional = qobject_input_optional;
     v->visitor.free = qobject_input_free;
     v->strict = true;
+    v->autocreate_list = autocreate_list;
 
     v->root = obj;
     qobject_incref(obj);
@@ -548,6 +565,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
 
 
 Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
+                                        bool autocreate_list,
                                         Error **errp)
 {
     QDict *pdict;
@@ -564,7 +582,8 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
         goto cleanup;
     }
 
Quoted 7 lines

qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 589

Quoted 6 lines
qapi/qobject-input-visitor.c +2 -1
-    v = qobject_input_visitor_new_autocast(pobj);
+    v = qobject_input_visitor_new_autocast(pobj,
+                                           autocreate_list);
  cleanup:
     qobject_decref(pobj);
     QDECREF(pdict);

[Skipping test updates for now...]

Regards,
Daniel

#2 lore raw

Re: [Qemu-devel] [PATCH v14 01/21] option: make parse_option_bool/number non-static

In reply to Daniel P. Berrange
Quoted 10 lines · Daniel P. Berrange

The opts-visitor.c opts_type_bool() method has code for parsing a string to set a bool value, as does the qemu-option.c parse_option_bool() method, except it handles fewer cases.

To enable consistency across the codebase, extend parse_option_bool() to handle "yes", "no", "y" and "n", and make it non-static. Convert the opts visitor to call this method directly.

Quoted 8 lines

Also make parse_option_number() non-static to allow for similar reuse later.

On first glance, this patch makes the boolean values recognized on the command line consistent regardless of which part of the code is used to recognize them, by extending the QemuOpts parser to accept the option visitor's additional convenience values.

Once we've done that in a release, we can't go back. No problem if it actually achieved consistency. But it doesn't: there are other parsers that recognize only "on" and "off". A quick grep finds select_display(), bdrv_open_inherit(), netfilter_set_status().

bdrv_open_inherit() is particularly instructive: -drive gets parsed by QemuOpts (evidence: your patch makes read-only=y work), but by the time the options arrive here, they're a QDict. Curiously, bdrv_open_inherit() expects the value of key "read-only" to be a *string*, and it checks for "on". Fortunately, something on the way to bdrv_open_inherit() replaces the "y" by "on", so this works. My point
is: this patch is trickier than it looks on first glance.

There's also HMP, which continues to accept only "on" and "off".

We might want to treat the option visitor's additional boolean values like its other syntax extensions: keep for compatibility, but confine to existing uses.

I think we should decide that when we merge the rest of your option visitor replacement work, hopefully in 2.9.

#3 lore raw

Re: [Qemu-devel] [PATCH v14 01/21] option: make parse_option_bool/number non-static

Quoted 10 lines · Markus Armbruster

On first glance, this patch makes the boolean values recognized on the command line consistent regardless of which part of the code is used to recognize them, by extending the QemuOpts parser to accept the option visitor's additional convenience values.

Once we've done that in a release, we can't go back. No problem if it actually achieved consistency. But it doesn't: there are other parsers that recognize only "on" and "off". A quick grep finds select_display(), bdrv_open_inherit(), netfilter_set_status().

Would it be wrong to convert these to use the same function, to get us to the point where they are consistent instead of self-limited? Yes, it means that newer qemu will accept spellings that older doesn't, and that we can't later prune the set of spellings back down, but trying to remain backwards-compatible to every different set of inputs on a command-by-command basis is harder than just making all commands accept all spellings.

Quoted 10 lines

bdrv_open_inherit() is particularly instructive: -drive gets parsed by QemuOpts (evidence: your patch makes read-only=y work), but by the time the options arrive here, they're a QDict. Curiously, bdrv_open_inherit() expects the value of key "read-only" to be a *string*, and it checks for "on". Fortunately, something on the way to bdrv_open_inherit() replaces the "y" by "on", so this works. My point
is: this patch is trickier than it looks on first glance.

There's also HMP, which continues to accept only "on" and "off".

Particularly for HMP, where we DON'T have backwards-compatibility constraints, I'd argue that ease-of-use is the driving factor and that HMP should allow ALL spellings of boolean arguments.

Quoted 4 lines

We might want to treat the option visitor's additional boolean values like its other syntax extensions: keep for compatibility, but confine to existing uses.

As in - new interfaces ONLY get to pass "true" or "false", and only existing interfaces get to also pass "y" or "on"?

Quoted 5 lines

I think we should decide that when we merge the rest of your option visitor replacement work, hopefully in 2.9.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#4 lore raw

Re: [Qemu-devel] [PATCH v14 01/21] option: make parse_option_bool/number non-static

Quoted 13 lines · Eric Blake
Quoted 10 lines · Markus Armbruster

On first glance, this patch makes the boolean values recognized on the command line consistent regardless of which part of the code is used to recognize them, by extending the QemuOpts parser to accept the option visitor's additional convenience values.

Once we've done that in a release, we can't go back. No problem if it actually achieved consistency. But it doesn't: there are other parsers that recognize only "on" and "off". A quick grep finds select_display(), bdrv_open_inherit(), netfilter_set_status().

Quoted 7 lines

Would it be wrong to convert these to use the same function, to get us to the point where they are consistent instead of self-limited? Yes, it means that newer qemu will accept spellings that older doesn't, and that we can't later prune the set of spellings back down, but trying to remain backwards-compatible to every different set of inputs on a command-by-command basis is harder than just making all commands accept all spellings.

I have two concerns:

1. Accepting alternate spellings in just a few places is slightly bad, accepting them is most, but not all places is worse. Converting all the parsers as you suggest would take care of this one.

Quoted 7 lines
Quoted 7 lines

bdrv_open_inherit() is particularly instructive: -drive gets parsed by QemuOpts (evidence: your patch makes read-only=y work), but by the time the options arrive here, they're a QDict. Curiously, bdrv_open_inherit() expects the value of key "read-only" to be a *string*, and it checks for "on". Fortunately, something on the way to bdrv_open_inherit() replaces the "y" by "on", so this works. My point
is: this patch is trickier than it looks on first glance.

2. I'm scared some user of QemuOpts declares QEMU_OPT_BOOL, but then checks the *string*. Yes, QemuOpts lets you do that. Making the parser accept more strings would break such code. Need a careful code review to put this concern to rest.

Quoted 5 lines
Quoted 1 line

There's also HMP, which continues to accept only "on" and "off".

Particularly for HMP, where we DON'T have backwards-compatibility constraints, I'd argue that ease-of-use is the driving factor and that HMP should allow ALL spellings of boolean arguments.

Personally, I find alternate spellings more confusing than helpful, but I agree that once we adopt them for the command line (consistently, not just in a few places), we should also adopt them for HMP.

Quoted 6 lines
Quoted 3 lines

We might want to treat the option visitor's additional boolean values like its other syntax extensions: keep for compatibility, but confine to existing uses.

As in - new interfaces ONLY get to pass "true" or "false", and only existing interfaces get to also pass "y" or "on"?

Very few places accept anything but "on" and "off", namely the ones that use the options or the string visitor. Dan's work kills of the former, and hopefully permits us killing off the latter. The alternate spellings will then be confined in one place, guarded by a compatibility flag you're not supposed to set in new code. Looks workable to me.

Quoted 2 lines
Quoted 2 lines

I think we should decide that when we merge the rest of your option visitor replacement work, hopefully in 2.9.

I'm not rejecting this patch, I'm merely punting the ABI-affecting parts of this series to 2.9.

#108 lore raw
In reply to Daniel P. Berrange
Quoted 41 lines · Daniel P. Berrange

An update of a series previously posted

v1: https://lists.gnu.org/archive/html/qemu-devel/2016-02/msg04618.html
v2: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg01454.html
v3: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg02498.html
v4: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01661.html
v5: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg00485.html
v6: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg03876.html
v7: https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg00919.html
v8: https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg03115.html
v9: https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg02653.html
v10: https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg02694.html
v11: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg00652.html
v12: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg03559.html
v13: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg04212.html
v14: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg07037.html

This series provides the infrastructure to allow use of non-scalar properties with the -object CLI arg, and object_add monitor commands. eg a property which is a list of structs. The syntax used for this is intentionally compatible with the syntax used by the block layer. This will allow the qdict_crumple method to be used by the block layer to convert from QemuOpts into structured QAPI block layer structs at a future date. It is already used by one of Max's patch series, by Kevin's -blockdev series, and by my own ACL series.

This patch series has grown a fair bit larger since previous postings (19^H^H21 patches instead of 6). This was to avoid trying to squish too much into one patch, to make it easier to review.

It does not neccessarily need to be merged in one go. There are three logical blocks of patches here.

Patches 1-8 are the core support for visiting QemuOpts using the QObjectInputVisitor class. Merging this would unblock at least Kevin's blockdev series, and probably Max's series too.

The patches 9-14 add enhancements to the code to support various special cases / edge cases needed to provide fully semantic replacement of the OptsVisitor

Quoted 8 lines

Patches 15-21 deal with converting code over to use the new visitor logic.

Once this is merged, there is also scope to further enhance the QObjectInputVisitor to deal with visiting single-properties, which would let us delete StringInputVisitor too. This would mean we have a single input visitor class, instead of the current three.

PATCH 03-06 applied to qapi-next. For PATCH 02, the question whether we need @recursive is still open. I hope Max can help me figure it out, so I can apply it next week. I think we'll rebase the rest on QemuOpts work I want for 2.9. Thanks!

#9 lore raw

Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict

In reply to Markus Armbruster
Quoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 289 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 261 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 551 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..e0d24e1 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +0 -0
index 60f158c..c38e90e 100644

[...]

Quoted 6 lines
Quoted 4 lines

+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionaries

Is recursive=false used outside tests in this series?

No, its not used.

Quoted 8 lines
Quoted 2 lines

It was suggested in a way earlier version by Max, but not sure if his code uses it or not.

In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.

Max, do you actually need this one? If yes, please explain your use case.

As far as I can tell from a quick glance, I made the point for v1 that qdict_crumple() could be simplified by using qdict_array_split() and qdict_array_entries().

Dan then (correctly) said that using these functions would worsen runtime performance of qdict_crumple() and that instead we can replace qdict_array_split() by qdict_crumple(). However, for that to work, we need to be able to make qdict_crumple() non-recursive (because qdict_array_split() is non-recursive).

Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.

However, clearly there are no patches out yet that replace qdict_array_split() by qdict_crumple(recursive=false), and I certainly won't write any until qdict_crumple() is merged.

Max

#10 lore raw

Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict

Quoted 119 lines · Max Reitz
Quoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 289 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 261 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 551 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..e0d24e1 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +0 -0
index 60f158c..c38e90e 100644

[...]

Quoted 6 lines
Quoted 4 lines

+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionaries

Is recursive=false used outside tests in this series?

No, its not used.

Quoted 8 lines
Quoted 2 lines

It was suggested in a way earlier version by Max, but not sure if his code uses it or not.

In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.

Max, do you actually need this one? If yes, please explain your use case.

As far as I can tell from a quick glance, I made the point for v1 that qdict_crumple() could be simplified by using qdict_array_split() and qdict_array_entries().

Dan then (correctly) said that using these functions would worsen runtime performance of qdict_crumple() and that instead we can replace qdict_array_split() by qdict_crumple(). However, for that to work, we need to be able to make qdict_crumple() non-recursive (because qdict_array_split() is non-recursive).

Quoted 6 lines

Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.

In the long run, the block layer should use proper C types instead of mucking around with QDict. That's what QAPI is for.

Quoted 3 lines

However, clearly there are no patches out yet that replace qdict_array_split() by qdict_crumple(recursive=false), and I certainly won't write any until qdict_crumple() is merged.

All right, I'll go hunting for prior versions of qdict_crumple() to see whether I find a suitable one without @recursive.

#11 lore raw

Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict

Quoted 131 lines · Markus Armbruster
Quoted 119 lines · Max Reitz
Quoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 289 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 261 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 551 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..e0d24e1 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +0 -0
index 60f158c..c38e90e 100644

[...]

Quoted 6 lines
Quoted 4 lines

+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionaries

Is recursive=false used outside tests in this series?

No, its not used.

Quoted 8 lines
Quoted 2 lines

It was suggested in a way earlier version by Max, but not sure if his code uses it or not.

In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.

Max, do you actually need this one? If yes, please explain your use case.

As far as I can tell from a quick glance, I made the point for v1 that qdict_crumple() could be simplified by using qdict_array_split() and qdict_array_entries().

Dan then (correctly) said that using these functions would worsen runtime performance of qdict_crumple() and that instead we can replace qdict_array_split() by qdict_crumple(). However, for that to work, we need to be able to make qdict_crumple() non-recursive (because qdict_array_split() is non-recursive).

Quoted 6 lines

Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.

In the long run, the block layer should use proper C types instead of mucking around with QDict. That's what QAPI is for.

Quoted 6 lines
Quoted 3 lines

However, clearly there are no patches out yet that replace qdict_array_split() by qdict_crumple(recursive=false), and I certainly won't write any until qdict_crumple() is merged.

All right, I'll go hunting for prior versions of qdict_crumple() to see whether I find a suitable one without @recursive.

Please don't do that - there have been a tonne of other fixes and improvements since the version that introduced @recursive. We need to just chop out that parameter from the v15 version instead.

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|
#12 lore raw

Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict

Quoted 136 lines · Daniel P. Berrange
Quoted 135 lines · Markus Armbruster
Quoted 119 lines · Max Reitz
Quoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 289 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 261 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 551 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..e0d24e1 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +0 -0
index 60f158c..c38e90e 100644

[...]

Quoted 6 lines
Quoted 4 lines

+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionaries

Is recursive=false used outside tests in this series?

No, its not used.

Quoted 8 lines
Quoted 2 lines

It was suggested in a way earlier version by Max, but not sure if his code uses it or not.

In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.

Max, do you actually need this one? If yes, please explain your use case.

As far as I can tell from a quick glance, I made the point for v1 that qdict_crumple() could be simplified by using qdict_array_split() and qdict_array_entries().

Dan then (correctly) said that using these functions would worsen runtime performance of qdict_crumple() and that instead we can replace qdict_array_split() by qdict_crumple(). However, for that to work, we need to be able to make qdict_crumple() non-recursive (because qdict_array_split() is non-recursive).

Quoted 6 lines

Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.

In the long run, the block layer should use proper C types instead of mucking around with QDict. That's what QAPI is for.

Quoted 3 lines

However, clearly there are no patches out yet that replace qdict_array_split() by qdict_crumple(recursive=false), and I certainly won't write any until qdict_crumple() is merged.

Quoted 6 lines
Quoted 2 lines

All right, I'll go hunting for prior versions of qdict_crumple() to see whether I find a suitable one without @recursive.

Please don't do that - there have been a tonne of other fixes and improvements since the version that introduced @recursive. We need to just chop out that parameter from the v15 version instead.

I realized that once I looked at the interdiffs.

Chopping off @recursive is easy enough. So is dropping all tests that pass false, but I'm not sure what that does to test coverage. Any advice?

I'm sorry I didn't catch this in my prior reviews (v5+v8).

hmp.c +1 -1
index b32d8c8..3845d1b 100644
@@ -1700,7 +1700,7 @@ void hmp_object_add(Monitor *mon, const QDict *qdict)
     Object *obj = NULL;
     QObject *pobj;
 
-    pobj = qdict_crumple(qdict, true, &err);
+    pobj = qdict_crumple(qdict, &err);
     if (err) {
         goto cleanup;
     }
include/qapi/qmp/qdict.h +1 -1
index e0d24e1..fe9a4c5 100644
@@ -73,7 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
-QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
+QObject *qdict_crumple(const QDict *src, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qapi/qobject-input-visitor.c +1 -1
index f1030d5..c743a5b 100644
@@ -769,7 +769,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
         goto cleanup;
     }
 
-    pobj = qdict_crumple(pdict, true, errp);
+    pobj = qdict_crumple(pdict, errp);
     if (!pobj) {
         goto cleanup;
     }
qobject/qdict.c +17 -26

68 lines not on this page · raw message

qom/object_interfaces.c +1 -1
index 88a1e88..0ea22fd 100644
@@ -170,7 +170,7 @@ Object *user_creatable_add_opts(QemuOpts *opts, Error **errp)
                                QEMU_OPTS_REPEAT_POLICY_ALL,
                                &error_abort);
 
-    pobj = qdict_crumple(pdict, true, errp);
+    pobj = qdict_crumple(pdict, errp);
     if (!pobj) {
         goto cleanup;
     }
tests/check-qdict.c +10 -148

268 lines not on this page · raw message

#13 lore raw

Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict

In reply to Max Reitz
Quoted 126 lines · Max Reitz
Quoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 289 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 261 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 551 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..e0d24e1 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +0 -0
index 60f158c..c38e90e 100644

[...]

Quoted 6 lines
Quoted 4 lines

+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionaries

Is recursive=false used outside tests in this series?

No, its not used.

Quoted 8 lines
Quoted 2 lines

It was suggested in a way earlier version by Max, but not sure if his code uses it or not.

In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.

Max, do you actually need this one? If yes, please explain your use case.

As far as I can tell from a quick glance, I made the point for v1 that qdict_crumple() could be simplified by using qdict_array_split() and qdict_array_entries().

Dan then (correctly) said that using these functions would worsen runtime performance of qdict_crumple() and that instead we can replace qdict_array_split() by qdict_crumple(). However, for that to work, we need to be able to make qdict_crumple() non-recursive (because qdict_array_split() is non-recursive).

Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.

Quoted 3 lines

However, clearly there are no patches out yet that replace qdict_array_split() by qdict_crumple(recursive=false), and I certainly won't write any until qdict_crumple() is merged.

I prefer not to maintain code that isn't actually used. Since Dan is enjoying a few days off, and the soft freeze is closing in, I'm proposing to squash in the following:

* Drop @recursive, in the most straightforward way possible.

* Drop all tests that pass false to @recursive. This might reduce coverage, but we can fix that on top.

* While there, collapse some vertical whitespace, for consistency with spacing elsewhere in the same files.

Resulting fixup patch appended. I'd appreciate a quick look-over before I do a pull request. Current state at
http://repo.or.cz/w/qemu/armbru.git branch qapi-not-next.

Quoted 1 line

From df87de10a12bd65d2ddc47514e951712de7c06f0 Mon Sep 17 00:00:00 2001

From: Markus Armbruster <[email protected]>
Date: Tue, 25 Oct 2016 10:09:46 +0200
Subject: [PATCH] fixup! qdict: implement a qdict_crumple method for
un-flattening a dict

---
 include/qapi/qmp/qdict.h |   2 +-
 qobject/qdict.c          |  42 ++++++---------
 tests/check-qdict.c      | 133 +++--------------------------------------------
 3 files changed, 23 insertions(+), 154 deletions(-)
include/qapi/qmp/qdict.h +1 -1
index e0d24e1..fe9a4c5 100644
@@ -73,7 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
-QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
+QObject *qdict_crumple(const QDict *src, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +15 -27

85 lines not on this page · raw message

tests/check-qdict.c +7 -126

261 lines not on this page · raw message

Signature · 1 line
2.5.5
#14 lore raw

Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict

Quoted 141 lines · Markus Armbruster
Quoted 126 lines · Max Reitz
Quoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange

The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.

The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.

If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'

}

will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.

 {
   'foo': [
     { 'bar': 'one', 'wizz': '1' },
     { 'bar': 'two', 'wizz': '2' }
   ],

}

If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict

  {
     'foo..bar': 'wizz',
     'bar.foo..bar': 'eek',
     'bar.hello': 'world'
  }

Will end up as

  {
     'foo.bar': 'wizz',
     'bar': {
        'foo.bar': 'eek',
        'hello': 'world'
     }
  }

The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.

---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c          | 289 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c      | 261 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 551 insertions(+)
include/qapi/qmp/qdict.h +1 -0
index 71b8eb0..e0d24e1 100644
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
qobject/qdict.c +0 -0
index 60f158c..c38e90e 100644

[...]

Quoted 6 lines
Quoted 4 lines

+/**
+ * qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionaries

Is recursive=false used outside tests in this series?

No, its not used.

Quoted 8 lines
Quoted 2 lines

It was suggested in a way earlier version by Max, but not sure if his code uses it or not.

In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.

Max, do you actually need this one? If yes, please explain your use case.

As far as I can tell from a quick glance, I made the point for v1 that qdict_crumple() could be simplified by using qdict_array_split() and qdict_array_entries().

Dan then (correctly) said that using these functions would worsen runtime performance of qdict_crumple() and that instead we can replace qdict_array_split() by qdict_crumple(). However, for that to work, we need to be able to make qdict_crumple() non-recursive (because qdict_array_split() is non-recursive).

Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.

Quoted 3 lines

However, clearly there are no patches out yet that replace qdict_array_split() by qdict_crumple(recursive=false), and I certainly won't write any until qdict_crumple() is merged.

I prefer not to maintain code that isn't actually used. Since Dan is enjoying a few days off, and the soft freeze is closing in, I'm proposing to squash in the following:

* Drop @recursive, in the most straightforward way possible.

* Drop all tests that pass false to @recursive. This might reduce coverage, but we can fix that on top.

Quoted 6 lines

* While there, collapse some vertical whitespace, for consistency with spacing elsewhere in the same files.

Resulting fixup patch appended. I'd appreciate a quick look-over before I do a pull request. Current state at
http://repo.or.cz/w/qemu/armbru.git branch qapi-not-next.

Had a quick look, and it seems fine to me.

Regards,
Daniel

Signature · 3 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|