QAPI/QOM work for non-scalar object properties
Subsystems (4)
PATCH v14 1/21 option: make parse_option_bool/number non-static
In reply to Daniel P. BerrangeThe opts-visitor.c opts_type_bool() method has code for parsing a string to set a bool value, as does the qemu-option.c parse_option_bool() method, except it handles fewer cases.
To enable consistency across the codebase, extend parse_option_bool() to handle "yes", "no", "y" and "n", and make it non-static. Convert the opts visitor to call this method directly.
Also make parse_option_number() non-static to allow for similar reuse later.
--- include/qemu/option.h | 4 ++++ qapi/opts-visitor.c | 19 +------------------ tests/qemu-iotests/051.out | 6 +++--- tests/qemu-iotests/051.pc.out | 6 +++--- tests/qemu-iotests/137.out | 4 ++-- util/qemu-option.c | 27 ++++++++++++++++----------- 6 files changed, 29 insertions(+), 37 deletions(-)
include/qemu/option.h +4 -0
@@ -37,6 +37,10 @@ int get_param_value(char *buf, int buf_size, const char *tag, const char *str); +void parse_option_bool (const char *name,const char *value,bool *ret, +Error **errp); +void parse_option_number (const char *name,const char *value, +uint64_t *ret,Error **errp);void parse_option_size (const char *name,const char *value,uint64_t *ret,Error **errp);bool has_help_option (const char *param);
qapi/opts-visitor.c +1 -18
@@ -334,7 +334,6 @@ opts_type_str(Visitor *v, const char *name, char **obj, Error **errp) } -/* mimics qemu-option.c::parse_option_bool() */ static void opts_type_bool (Visitor *v,const char *name,bool *obj,Error **errp) {
@@ -346,23 +345,7 @@ opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp)return ; } -if (opt->str) { -if (strcmp (opt->str,"on" ) ==0 || -strcmp (opt->str,"yes" ) ==0 || -strcmp (opt->str,"y" ) ==0 ) { - *obj = true; - }else if (strcmp (opt->str,"off" ) ==0 || -strcmp (opt->str,"no" ) ==0 || -strcmp (opt->str,"n" ) ==0 ) { - *obj = false; - }else { -error_setg (errp,QERR_INVALID_PARAMETER_VALUE , opt->name, -"on|yes|y|off|no|n" ); -return ; - } - }else { - *obj = true; - } +parse_option_bool (opt->name, opt->str, obj, errp);processed (ov, name); }
tests/qemu-iotests/051.out +3 -3
@@ -86,13 +86,13 @@ QEMU X.Y.Z monitor - type 'help' for more information (qemu) q[K[Dqu[K[D[Dqui[K[D[D[Dquit[K Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts= -QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: Parameter 'lazy-refcounts' expects 'on' or 'off' +QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n' Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42 -QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42: Parameter 'lazy-refcounts' expects 'on' or 'off' +QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n' Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo -QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo: Parameter 'lazy-refcounts' expects 'on' or 'off' +QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n' === With version 2 images enabling lazy refcounts must fail ===
tests/qemu-iotests/051.pc.out +3 -3
@@ -86,13 +86,13 @@ QEMU X.Y.Z monitor - type 'help' for more information (qemu) q[K[Dqu[K[D[Dqui[K[D[D[Dquit[K Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts= -QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: Parameter 'lazy-refcounts' expects 'on' or 'off' +QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n' Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42 -QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42: Parameter 'lazy-refcounts' expects 'on' or 'off' +QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n' Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo -QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo: Parameter 'lazy-refcounts' expects 'on' or 'off' +QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo: Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n' === With version 2 images enabling lazy refcounts must fail ===
tests/qemu-iotests/137.out +2 -2
@@ -15,7 +15,7 @@ read 33554432/33554432 bytes at offset 0 === Try setting some invalid values === -Parameter 'lazy-refcounts' expects 'on' or 'off' +Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n' cache-size, l2-cache-size and refcount-cache-size may not be set the same time l2-cache-size may not exceed cache-size refcount-cache-size may not exceed cache-size
@@ -40,7 +40,7 @@ incompatible_features 0x0 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 wrote 65536/65536 bytes at offset 0 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -Parameter 'lazy-refcounts' expects 'on' or 'off' +Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n' qcow2: Marking image as corrupt: Preventing invalid write on metadata (overlaps with qcow2_header); further corruption events will be suppressed write failed: Input/output error *** done
util/qemu-option.c +16 -11
@@ -125,25 +125,30 @@ int get_param_value(char *buf, int buf_size,return get_next_param_value (buf, buf_size, tag, &str); } -static void parse_option_bool (const char *name,const char *value,bool *ret, -Error **errp) +void parse_option_bool (const char *name,const char *value,bool *ret, +Error **errp) {if (value !=NULL ) { -if (!strcmp (value,"on" )) { - *ret =1 ; - }else if (!strcmp (value,"off" )) { - *ret =0 ; +if (strcmp (value,"on" ) ==0 || +strcmp (value,"yes" ) ==0 || +strcmp (value,"y" ) ==0 ) { + *ret = true; + }else if (strcmp (value,"off" ) ==0 || +strcmp (value,"no" ) ==0 || +strcmp (value,"n" ) ==0 ) { + *ret = false; }else { -error_setg (errp,QERR_INVALID_PARAMETER_VALUE , - name,"'on' or 'off'" ); +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name, +"'on', 'yes', 'y', 'off', 'no' or 'n'" ); +return ; } }else { - *ret =1 ; + *ret = true; } } -static void parse_option_number (const char *name,const char *value, -uint64_t *ret,Error **errp) +void parse_option_number (const char *name,const char *value, +uint64_t *ret,Error **errp) {char *postfix;uint64_t number;
Signature · 1 line
2.7.4
v14 cover QAPI/QOM work for non-scalar object properties
An update of a series previously posted
v1: https://lists.gnu.org/archive/html/qemu-devel/2016-02/msg04618.html
v2: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg01454.html
v3: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg02498.html
v4: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01661.html
v5: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg00485.html
v6: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg03876.html
v7: https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg00919.html
v8: https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg03115.html
v9: https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg02653.html
v10: https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg02694.html
v11: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg00652.html
v12: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg03559.html
v13: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg04212.html
v14: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg07037.html
This series provides the infrastructure to allow use of non-scalar properties with the -object CLI arg, and object_add monitor commands. eg a property which is a list of structs. The syntax used for this is intentionally compatible with the syntax used by the block layer. This will allow the qdict_crumple method to be used by the block layer to convert from QemuOpts into structured QAPI block layer structs at a future date. It is already used by one of Max's patch series, by Kevin's -blockdev series, and by my own ACL series.
This patch series has grown a fair bit larger since previous postings (19^H^H21 patches instead of 6). This was to avoid trying to squish too much into one patch, to make it easier to review.
It does not neccessarily need to be merged in one go. There are three logical blocks of patches here.
Patches 1-8 are the core support for visiting QemuOpts using the QObjectInputVisitor class. Merging this would unblock at least Kevin's blockdev series, and probably Max's series too.
The patches 9-14 add enhancements to the code to support various special cases / edge cases needed to provide fully semantic replacement of the OptsVisitor
Patches 15-21 deal with converting code over to use the new visitor logic.
Once this is merged, there is also scope to further enhance the QObjectInputVisitor to deal with visiting single-properties, which would let us delete StringInputVisitor too. This would mean we have a single input visitor class, instead of the current three.
Changed in v15:
- Re-arranged patch series to allow easier incremental merge of core functionality
- Extra error checking of some scenarios to be avoided (Eric)
- Misc typos (Eric)
- Rename QEMU_OPTS_REPEAT_POLICY_LIST to QEMU_OPTS_REPEAT_POLICY_ALL, and add QEMU_OPTS_REPEAT_POLICY_ERROR (Eric)
Changed in v14:
- Extend qemu_opts_to_qdict to optionally handle case of repeated keys without dropping them.
- Support parsing of integer ranges for "list of int" visits for back compat with OptsVisitor
- Support visiting nested structs which are represented as a flat keyspace for back compat with OptsVisitor
- Add trace events for visitor functions to facilitate debugging
- Add a QObjectInputVisitor constructor which takes a QemuOpts object, to avoid repeated code patterns in callers
- Greater test coverage of -object to validate correct handling of "list of ints" legacy OptsVisitor syntax
- Convert all other use of OptsVisitor to QObjectInputVisitor
- Delete OptsVisitor
Changed in v13:
- Fix typos (Kevin)
- Remove unneeded line breaks (Kevin)
Changed in v12:
- More user friendly error message for mixing dict/list keys (Kevin)
- Report error instead of assert for non-contiguous list keys (Kevin)
- Fix tests for non-contiguous list keys (Kevin)
- Add tests for escaping of '.' when crumpling (Kevin)
- Fix remaining references to Qmp(In|Out)putVisitor (Markus)
- Misc typos / whitespace fixes (Eric, Kevin)
- Avoid touching 'ret' when parsing int64 fails (Eric)
- Testing of more edge cases in QObjectInputVisitor (Eric)
- Simplify API doc format (Markus)
- Use parse_option_size instead of qemu_strtosz_suffix for consistency (Kevin)
- Use safer qobject_to_qdict casts (Eric)
Changed in v11:
- Split QAPI/QOM patches off from the access control patches
Changed in v10:
- Fixed stupid build mistake
Changed in v9:
- Rename QmpInputVisitor -> QObjectInputVisitor (Markus/Eric)
- Rename QmpOutputVisitor -> QObjectOutputVisitor (Markus/Eric)
- Drop "strict" param from qobject_string_visitor_new() (Marus)
- Misc docs typos
- Add a visitor able to use strict or string types (for Eric's netdev series)
- Add a authorization API implementation that uses PAM
Changed in v8:
- Rebase due to merge of Visitor API changes (Eric)
Changed in v7:
- Misc typos in API docs (Marc-André)
- Fix parsing of properties using type_size visitor (Marc-André)
- Mark based auth class as abstract (Marc-André)
- Fix QAPI version annotations to say 2.7 (Marc-André)
Changed in v6:
- Switch from while() to for() loop for iterating over dicts (Markus)
- Avoid redundant strdup (Markus)
- Rewrap comments at 70 chars (Markus)
- Change qdict_list_size() to qdict_is_list() (Markus)
- Misc docs changes (Markus)
- Change QmpInputVisitor so the code for handling the string types is separate from code using native scalar types (Paolo)
- Centralize code parsing bool strings (Markus)
- Centralize code parsing int strings (Markus)
Changed in v5:
- Resolved conflicts with Eric's visitor refactoring which made it stricter about struct begin/end calls
- Added support for ACLs to migration code now its TLS support is merged.
- Fixed typos in example in commit message
Changed in v4:
- Ensure examples use shell escaping for '*' (Eric)
- Add more tests for crumple impl (Eric)
- Raise error if sasl-acl/tls-acl are requested but sasl/tls auth are not enabled (Eric)
- Document return codes for auth check more clearly (Eric)
- Don't silently turn a glob match into a strcmp
- Other misc small typos/fixes (Eric)
Changed in v3:
- Created separate qdict_list_size method (Max)
- Added unit tests for case of empty dict (Max)
- Fix variable names to use underscore separator (Max)
- Fix potential free of uninitialized variables (Max)
- Use QObject APIs for casts, instead of C type casts (Max)
Changed in v2:
- Adapt to changes in qapi visitor APIs
- Add a 'bool recursive' flag to qdict_crumple (Max)
- Fix memory leaks in qdict_crumple (Max)
- Split out key splitting code from qdict_crumple (Max)
- Use saner variable names in qdict_crumple (Max)
- Added some tests for bad inputs to qdict_crumple
Daniel P. Berrange (21):
option: make parse_option_bool/number non-static
qdict: implement a qdict_crumple method for un-flattening a dict
qapi: add trace events for visitor
qapi: rename QmpInputVisitor to QObjectInputVisitor
qapi: rename QmpOutputVisitor to QObjectOutputVisitor
qapi: don't pass two copies of TestInputVisitorData to tests
qapi: permit scalar type conversions in QObjectInputVisitor
qapi: allow QObjectInputVisitor to be created with QemuOpts
qapi: permit auto-creating single element lists
qapi: permit auto-creating nested structs
qapi: add integer range support for QObjectInputVisitor
option: allow qemu_opts_to_qdict to merge repeated options
qdict: allow qdict_crumple to accept compound types as values
qapi: allow repeated opts with qobject_input_visitor_new_opts
qom: support non-scalar properties with -object
hmp: support non-scalar properties with object_add
numa: convert to use QObjectInputVisitor for -numa
block: convert crypto driver to use QObjectInputVisitor
acpi: convert to QObjectInputVisitor for -acpi parsing
net: convert to QObjectInputVisitor for -net/-netdev parsing
qapi: delete unused OptsVisitor code
Makefile.objs | 1 +
block/crypto.c | 14 +-
block/qapi.c | 4 +-
blockdev.c | 11 +-
docs/qapi-code-gen.txt | 4 +-
hmp.c | 25 +-
hw/acpi/core.c | 14 +-
include/qapi/opts-visitor.h | 40 --
include/qapi/qmp-input-visitor.h | 30 -
include/qapi/qmp/qdict.h | 1 +
include/qapi/qobject-input-visitor.h | 130 ++++
...p-output-visitor.h => qobject-output-visitor.h} | 10 +-
include/qapi/visitor.h | 6 +-
include/qemu/option.h | 17 +-
include/sysemu/numa_int.h | 11 +
monitor.c | 5 +-
net/net.c | 18 +-
numa.c | 36 +-
qapi/Makefile.objs | 6 +-
qapi/opts-visitor.c | 561 ---------------
qapi/qapi-clone-visitor.c | 2 +-
qapi/qapi-visit-core.c | 27 +
qapi/qmp-input-visitor.c | 412 -----------
qapi/qmp-output-visitor.c | 256 -------
qapi/qobject-input-visitor.c | 785 +++++++++++++++++++++
qapi/qobject-output-visitor.c | 254 +++++++
qapi/trace-events | 33 +
qemu-img.c | 12 +-
qemu-io-cmds.c | 3 +-
qemu-io.c | 6 +-
qemu-nbd.c | 3 +-
qmp.c | 4 +-
qobject/qdict.c | 302 ++++++++
qom/object_interfaces.c | 42 +-
qom/qom-qobject.c | 8 +-
scripts/qapi-commands.py | 8 +-
scripts/qapi-event.py | 4 +-
stubs/Makefile.objs | 5 +
stubs/exec.c | 6 +
stubs/hostmem.c | 14 +
stubs/memory.c | 41 ++
stubs/qdev.c | 8 +
stubs/vl.c | 8 +
stubs/vmstate.c | 4 +
target-s390x/cpu_models.c | 4 +-
tests/.gitignore | 6 +-
tests/Makefile.include | 27 +-
tests/check-qdict.c | 436 ++++++++++++
tests/check-qnull.c | 8 +-
tests/check-qom-proplist.c | 367 +++++++++-
tests/qemu-iotests/051.out | 6 +-
tests/qemu-iotests/051.pc.out | 6 +-
tests/qemu-iotests/137.out | 4 +-
tests/test-numa.c | 116 +++
tests/test-opts-visitor.c | 268 -------
tests/test-qemu-opts.c | 132 ++++
tests/test-qmp-commands.c | 4 +-
...-input-strict.c => test-qobject-input-strict.c} | 6 +-
...nput-visitor.c => test-qobject-input-visitor.c} | 637 +++++++++++++++--
...put-visitor.c => test-qobject-output-visitor.c} | 6 +-
tests/test-replication.c | 9 +-
tests/test-string-input-visitor.c | 2 +-
tests/test-string-output-visitor.c | 2 +-
tests/test-visitor-serialization.c | 8 +-
util/qemu-option.c | 91 ++-
util/qemu-sockets.c | 4 +-
vl.c | 1 -
67 files changed, 3566 insertions(+), 1775 deletions(-)
delete mode 100644 include/qapi/opts-visitor.h
delete mode 100644 include/qapi/qmp-input-visitor.h
create mode 100644 include/qapi/qobject-input-visitor.h
rename include/qapi/{qmp-output-visitor.h => qobject-output-visitor.h} (66%)
create mode 100644 include/sysemu/numa_int.h
delete mode 100644 qapi/opts-visitor.c
delete mode 100644 qapi/qmp-input-visitor.c
delete mode 100644 qapi/qmp-output-visitor.c
create mode 100644 qapi/qobject-input-visitor.c
create mode 100644 qapi/qobject-output-visitor.c
create mode 100644 qapi/trace-events
create mode 100644 stubs/exec.c
create mode 100644 stubs/hostmem.c
create mode 100644 stubs/memory.c
create mode 100644 stubs/qdev.c
create mode 100644 stubs/vl.c
create mode 100644 tests/test-numa.c
delete mode 100644 tests/test-opts-visitor.c
rename tests/{test-qmp-input-strict.c => test-qobject-input-strict.c} (98%)
rename tests/{test-qmp-input-visitor.c => test-qobject-input-visitor.c} (57%)
rename tests/{test-qmp-output-visitor.c => test-qobject-output-visitor.c} (99%)Signature · 1 line
2.7.4
PATCH v14 2/21 qdict: implement a qdict_crumple method for un-flattening a dict
The qdict_flatten() method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.
The qdict_crumple() method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.
If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{
'foo.0.bar': 'one',
'foo.0.wizz': '1',
'foo.1.bar': 'two',
'foo.1.wizz': '2'
}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{
'foo': [
{ 'bar': 'one', 'wizz': '1' },
{ 'bar': 'two', 'wizz': '2' } ],
}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{
'foo..bar': 'wizz',
'bar.foo..bar': 'eek',
'bar.hello': 'world'
}Will end up as
{
'foo.bar': 'wizz',
'bar': {
'foo.bar': 'eek',
'hello': 'world'
}
}The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.
--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 261 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 551 insertions(+)
include/qapi/qmp/qdict.h +1 -0
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);
qobject/qdict.c +289 -0
301 lines not on this page · raw message
tests/check-qdict.c +261 -0
279 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v14 3/21 qapi: add trace events for visitor
In reply to Daniel P. BerrangeAllow tracing of the operation of visitors
--- Makefile.objs | 1 + qapi/qapi-visit-core.c | 27 +++++++++++++++++++++++++++ qapi/trace-events | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 61 insertions(+) create mode 100644 qapi/trace-events
Makefile.objs +1 -0
@@ -160,3 +160,4 @@ trace-events-y += target-s390x/trace-events trace-events-y += target-ppc/trace-events trace-events-y += qom/trace-events trace-events-y += linux-user/trace-events +trace-events-y += qapi/trace-events
qapi/qapi-visit-core.c +27 -0
@@ -19,10 +19,12 @@#include "qapi/qmp/qerror.h" #include "qapi/visitor.h" #include "qapi/visitor-impl.h" +#include "trace.h" void visit_complete (Visitor *v,void *opaque) {assert (v->type !=VISITOR_OUTPUT || v->complete); +trace_visit_complete (v, opaque);if (v->complete) { v->complete (v, opaque); }
@@ -30,6 +32,7 @@ void visit_complete(Visitor *v, void *opaque)void visit_free (Visitor *v) { +trace_visit_free (v);if (v) { v->free (v); }
@@ -40,6 +43,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj, {Error *err =NULL ; +trace_visit_start_struct (v, name, obj, size);if (obj) {assert (size);assert (!(v->type &VISITOR_OUTPUT ) || *obj);
@@ -53,6 +57,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,void visit_check_struct (Visitor *v,Error **errp) { +trace_visit_check_struct (v);if (v->check_struct) { v->check_struct (v, errp); }
@@ -60,6 +65,7 @@ void visit_check_struct(Visitor *v, Error **errp)void visit_end_struct (Visitor *v,void **obj) { +trace_visit_end_struct (v, obj); v->end_struct (v, obj); }
@@ -69,6 +75,7 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,Error *err =NULL ;assert (!list || size >=sizeof (GenericList)); +trace_visit_start_list (v, name, list, size); v->start_list (v, name, list, size, &err);if (list && (v->type &VISITOR_INPUT )) {assert (!(err && *list));
@@ -79,11 +86,13 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,GenericList *visit_next_list (Visitor *v,GenericList *tail,size_t size) {assert (tail && size >=sizeof (GenericList)); +trace_visit_next_list (v, tail, size);return v->next_list (v, tail, size); }void visit_end_list (Visitor *v,void **obj) { +trace_visit_end_list (v, obj); v->end_list (v, obj); }
@@ -95,6 +104,7 @@ void visit_start_alternate(Visitor *v, const char *name,assert (obj && size >=sizeof (GenericAlternate));assert (!(v->type &VISITOR_OUTPUT ) || *obj); +trace_visit_start_alternate (v, name, obj, size, promote_int);if (v->start_alternate) { v->start_alternate (v, name, obj, size, promote_int, &err); }
@@ -106,6 +116,7 @@ void visit_start_alternate(Visitor *v, const char *name,void visit_end_alternate (Visitor *v,void **obj) { +trace_visit_end_alternate (v, obj);if (v->end_alternate) { v->end_alternate (v, obj); }
@@ -113,6 +124,7 @@ void visit_end_alternate(Visitor *v, void **obj)bool visit_optional (Visitor *v,const char *name,bool *present) { +trace_visit_optional (v, name, present);if (v->optional) { v->optional (v, name, present); }
@@ -127,6 +139,7 @@ bool visit_is_input(Visitor *v)void visit_type_int (Visitor *v,const char *name,int64_t *obj,Error **errp) {assert (obj); +trace_visit_type_int (v, name, obj); v->type_int64 (v, name, obj, errp); }
@@ -151,6 +164,7 @@ void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj, Error **errp) {uint64_t value = *obj; +trace_visit_type_uint8 (v, name, obj);visit_type_uintN (v, &value, name,UINT8_MAX ,"uint8_t" , errp); *obj = value; }
@@ -159,6 +173,7 @@ void visit_type_uint16(Visitor *v, const char *name, uint16_t *obj, Error **errp) {uint64_t value = *obj; +trace_visit_type_uint16 (v, name, obj);visit_type_uintN (v, &value, name,UINT16_MAX ,"uint16_t" , errp); *obj = value; }
@@ -167,6 +182,7 @@ void visit_type_uint32(Visitor *v, const char *name, uint32_t *obj, Error **errp) {uint64_t value = *obj; +trace_visit_type_uint32 (v, name, obj);visit_type_uintN (v, &value, name,UINT32_MAX ,"uint32_t" , errp); *obj = value; }
@@ -175,6 +191,7 @@ void visit_type_uint64(Visitor *v, const char *name, uint64_t *obj, Error **errp) {assert (obj); +trace_visit_type_uint64 (v, name, obj); v->type_uint64 (v, name, obj, errp); }
@@ -199,6 +216,7 @@ static void visit_type_intN(Visitor *v, int64_t *obj, const char *name,void visit_type_int8 (Visitor *v,const char *name,int8_t *obj,Error **errp) {int64_t value = *obj; +trace_visit_type_int8 (v, name, obj);visit_type_intN (v, &value, name,INT8_MIN ,INT8_MAX ,"int8_t" , errp); *obj = value; }
@@ -207,6 +225,7 @@ void visit_type_int16(Visitor *v, const char *name, int16_t *obj, Error **errp) {int64_t value = *obj; +trace_visit_type_int16 (v, name, obj);visit_type_intN (v, &value, name,INT16_MIN ,INT16_MAX ,"int16_t" , errp); *obj = value; }
@@ -215,6 +234,7 @@ void visit_type_int32(Visitor *v, const char *name, int32_t *obj, Error **errp) {int64_t value = *obj; +trace_visit_type_int32 (v, name, obj);visit_type_intN (v, &value, name,INT32_MIN ,INT32_MAX ,"int32_t" , errp); *obj = value; }
@@ -223,6 +243,7 @@ void visit_type_int64(Visitor *v, const char *name, int64_t *obj, Error **errp) {assert (obj); +trace_visit_type_int64 (v, name, obj); v->type_int64 (v, name, obj, errp); }
@@ -230,6 +251,7 @@ void visit_type_size(Visitor *v, const char *name, uint64_t *obj, Error **errp) {assert (obj); +trace_visit_type_size (v, name, obj);if (v->type_size) { v->type_size (v, name, obj, errp); }else {
@@ -240,6 +262,7 @@ void visit_type_size(Visitor *v, const char *name, uint64_t *obj,void visit_type_bool (Visitor *v,const char *name,bool *obj,Error **errp) {assert (obj); +trace_visit_type_bool (v, name, obj); v->type_bool (v, name, obj, errp); }
@@ -252,6 +275,7 @@ void visit_type_str(Visitor *v, const char *name, char **obj, Error **errp)* can enable: assert(!(v->type & VISITOR_OUTPUT) || *obj); */ +trace_visit_type_str (v, name, obj); v->type_str (v, name, obj, &err);if (v->type &VISITOR_INPUT ) {assert (!err != !*obj);
@@ -263,6 +287,7 @@ void visit_type_number(Visitor *v, const char *name, double *obj, Error **errp) {assert (obj); +trace_visit_type_number (v, name, obj); v->type_number (v, name, obj, errp); }
@@ -272,6 +297,7 @@ void visit_type_any(Visitor *v, const char *name, QObject **obj, Error **errp)assert (obj);assert (v->type !=VISITOR_OUTPUT || *obj); +trace_visit_type_any (v, name, obj); v->type_any (v, name, obj, &err);if (v->type ==VISITOR_INPUT ) {assert (!err != !*obj);
@@ -281,6 +307,7 @@ void visit_type_any(Visitor *v, const char *name, QObject **obj, Error **errp)void visit_type_null (Visitor *v,const char *name,Error **errp) { +trace_visit_type_null (v, name); v->type_null (v, name, errp); }
qapi/trace-events new file +33 -0
@@ -0,0 +1,33 @@ +# qapi-visit-core.c +visit_free(void *v) "v=%p" +visit_complete(void *v, void *opaque) "v=%p opaque=%p" + +visit_start_struct(void *v, const char *name, void *obj, size_t size) "v=%p name=%s obj=%p size=%zu" +visit_check_struct(void *v) "v=%p" +visit_end_struct(void *v, void *obj) "v=%p obj=%p" + +visit_start_list(void *v, const char *name, void *obj, size_t size) "v=%p name=%s obj=%p size=%zu" +visit_next_list(void *v, void *tail, size_t size) "v=%p tail=%p size=%zu" +visit_end_list(void *v, void *obj) "v=%p obj=%p" + +visit_start_alternate(void *v, const char *name, void *obj, size_t size, bool promote_int) "v=%p name=%s obj=%p size=%zu promote_int=%d" +visit_end_alternate(void *v, void *obj) "v=%p obj=%p" + +visit_optional(void *v, const char *name, bool *present) "v=%p name=%s present=%p" + +visit_type_enum(void *v, const char *name, int *obj) "v=%p name=%s obj=%p" +visit_type_int(void *v, const char *name, int64_t *obj) "v=%p name=%s obj=%p" +visit_type_uint8(void *v, const char *name, uint8_t *obj) "v=%p name=%s obj=%p" +visit_type_uint16(void *v, const char *name, uint16_t *obj) "v=%p name=%s obj=%p" +visit_type_uint32(void *v, const char *name, uint32_t *obj) "v=%p name=%s obj=%p" +visit_type_uint64(void *v, const char *name, uint64_t *obj) "v=%p name=%s obj=%p" +visit_type_int8(void *v, const char *name, int8_t *obj) "v=%p name=%s obj=%p" +visit_type_int16(void *v, const char *name, int16_t *obj) "v=%p name=%s obj=%p" +visit_type_int32(void *v, const char *name, int32_t *obj) "v=%p name=%s obj=%p" +visit_type_int64(void *v, const char *name, int64_t *obj) "v=%p name=%s obj=%p" +visit_type_size(void *v, const char *name, uint64_t *obj) "v=%p name=%s obj=%p" +visit_type_bool(void *v, const char *name, bool *obj) "v=%p name=%s obj=%p" +visit_type_str(void *v, const char *name, char **obj) "v=%p name=%s obj=%p" +visit_type_number(void *v, const char *name, double *obj) "v=%p name=%s obj=%p" +visit_type_any(void *v, const char *name, void *obj) "v=%p name=%s obj=%p" +visit_type_null(void *v, const char *name) "v=%p name=%s"
Signature · 1 line
2.7.4
PATCH v14 6/21 qapi: don't pass two copies of TestInputVisitorData to tests
In reply to Daniel P. BerrangeThe input_visitor_test_add() method was accepting an instance of 'TestInputVisitorData' and passing it as the 'user_data' parameter to test functions. The main 'TestInputVisitorData' instance that was actually used, was meanwhile being allocated automatically by the test framework fixture setup.
The 'user_data' parameter is going to be needed for tests added in later patches, so getting rid of the current mistaken usage now allows this.
--- tests/test-qobject-input-visitor.c | 76 ++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 44 deletions(-)
tests/test-qobject-input-visitor.c +32 -44
@@ -747,10 +747,11 @@ static void test_visitor_in_native_list_number(TestInputVisitorData *data, }static void input_visitor_test_add (const char *testpath, -TestInputVisitorData *data, -void (*test_func)(TestInputVisitorData *data,const void *user_data)) +const void *user_data, +void (*test_func)(TestInputVisitorData *data, +const void *user_data)) { -g_test_add (testpath, TestInputVisitorData, data,NULL , test_func, +g_test_add (testpath, TestInputVisitorData, user_data,NULL , test_func, visitor_input_teardown); }
@@ -833,77 +834,64 @@ static void test_visitor_in_wrong_type(TestInputVisitorData *data,int main (int argc,char **argv) { -TestInputVisitorData in_visitor_data; -g_test_init (&argc, &argv,NULL );input_visitor_test_add ("/visitor/input/int" , - &in_visitor_data, test_visitor_in_int); +NULL , test_visitor_in_int);input_visitor_test_add ("/visitor/input/int_overflow" , - &in_visitor_data, test_visitor_in_int_overflow); +NULL , test_visitor_in_int_overflow);input_visitor_test_add ("/visitor/input/bool" , - &in_visitor_data, test_visitor_in_bool); +NULL , test_visitor_in_bool);input_visitor_test_add ("/visitor/input/number" , - &in_visitor_data, test_visitor_in_number); +NULL , test_visitor_in_number);input_visitor_test_add ("/visitor/input/string" , - &in_visitor_data, test_visitor_in_string); +NULL , test_visitor_in_string);input_visitor_test_add ("/visitor/input/enum" , - &in_visitor_data, test_visitor_in_enum); +NULL , test_visitor_in_enum);input_visitor_test_add ("/visitor/input/struct" , - &in_visitor_data, test_visitor_in_struct); +NULL , test_visitor_in_struct);input_visitor_test_add ("/visitor/input/struct-nested" , - &in_visitor_data, test_visitor_in_struct_nested); +NULL , test_visitor_in_struct_nested);input_visitor_test_add ("/visitor/input/list" , - &in_visitor_data, test_visitor_in_list); +NULL , test_visitor_in_list);input_visitor_test_add ("/visitor/input/any" , - &in_visitor_data, test_visitor_in_any); +NULL , test_visitor_in_any);input_visitor_test_add ("/visitor/input/null" , - &in_visitor_data, test_visitor_in_null); +NULL , test_visitor_in_null);input_visitor_test_add ("/visitor/input/union-flat" , - &in_visitor_data, test_visitor_in_union_flat); +NULL , test_visitor_in_union_flat);input_visitor_test_add ("/visitor/input/alternate" , - &in_visitor_data, test_visitor_in_alternate); +NULL , test_visitor_in_alternate);input_visitor_test_add ("/visitor/input/errors" , - &in_visitor_data, test_visitor_in_errors); +NULL , test_visitor_in_errors);input_visitor_test_add ("/visitor/input/wrong-type" , - &in_visitor_data, test_visitor_in_wrong_type); +NULL , test_visitor_in_wrong_type);input_visitor_test_add ("/visitor/input/alternate-number" , - &in_visitor_data, test_visitor_in_alternate_number); +NULL , test_visitor_in_alternate_number);input_visitor_test_add ("/visitor/input/native_list/int" , - &in_visitor_data, - test_visitor_in_native_list_int); +NULL , test_visitor_in_native_list_int);input_visitor_test_add ("/visitor/input/native_list/int8" , - &in_visitor_data, - test_visitor_in_native_list_int8); +NULL , test_visitor_in_native_list_int8);input_visitor_test_add ("/visitor/input/native_list/int16" , - &in_visitor_data, - test_visitor_in_native_list_int16); +NULL , test_visitor_in_native_list_int16);input_visitor_test_add ("/visitor/input/native_list/int32" , - &in_visitor_data, - test_visitor_in_native_list_int32); +NULL , test_visitor_in_native_list_int32);input_visitor_test_add ("/visitor/input/native_list/int64" , - &in_visitor_data, - test_visitor_in_native_list_int64); +NULL , test_visitor_in_native_list_int64);input_visitor_test_add ("/visitor/input/native_list/uint8" , - &in_visitor_data, - test_visitor_in_native_list_uint8); +NULL , test_visitor_in_native_list_uint8);input_visitor_test_add ("/visitor/input/native_list/uint16" , - &in_visitor_data, - test_visitor_in_native_list_uint16); +NULL , test_visitor_in_native_list_uint16);input_visitor_test_add ("/visitor/input/native_list/uint32" , - &in_visitor_data, - test_visitor_in_native_list_uint32); +NULL , test_visitor_in_native_list_uint32);input_visitor_test_add ("/visitor/input/native_list/uint64" , - &in_visitor_data, - test_visitor_in_native_list_uint64); +NULL , test_visitor_in_native_list_uint64);input_visitor_test_add ("/visitor/input/native_list/bool" , - &in_visitor_data, test_visitor_in_native_list_bool); +NULL , test_visitor_in_native_list_bool);input_visitor_test_add ("/visitor/input/native_list/str" , - &in_visitor_data, - test_visitor_in_native_list_string); +NULL , test_visitor_in_native_list_string);input_visitor_test_add ("/visitor/input/native_list/number" , - &in_visitor_data, - test_visitor_in_native_list_number); +NULL , test_visitor_in_native_list_number);g_test_run ();
Signature · 1 line
2.7.4
PATCH v14 4/21 qapi: rename QmpInputVisitor to QObjectInputVisitor
In reply to Daniel P. BerrangeThe QmpInputVisitor has no direct dependency on QMP. It is valid to use it anywhere that one has a QObject. Rename it to better reflect its functionality as a generic QObject to QAPI converter.
---
docs/qapi-code-gen.txt | 2 +-
...qmp-input-visitor.h => qobject-input-visitor.h} | 10 +-
include/qapi/visitor.h | 6 +-
monitor.c | 2 +-
qapi/Makefile.objs | 2 +-
...qmp-input-visitor.c => qobject-input-visitor.c} | 171 +++++++++++----------
qmp.c | 4 +-
qom/qom-qobject.c | 4 +-
scripts/qapi-commands.py | 4 +-
target-s390x/cpu_models.c | 4 +-
tests/.gitignore | 4 +-
tests/Makefile.include | 12 +-
tests/check-qnull.c | 4 +-
tests/test-qmp-commands.c | 4 +-
...-input-strict.c => test-qobject-input-strict.c} | 6 +-
...nput-visitor.c => test-qobject-input-visitor.c} | 6 +-
tests/test-string-input-visitor.c | 2 +-
tests/test-visitor-serialization.c | 4 +-
util/qemu-sockets.c | 2 +-
19 files changed, 128 insertions(+), 125 deletions(-)
rename include/qapi/{qmp-input-visitor.h => qobject-input-visitor.h} (63%)
rename qapi/{qmp-input-visitor.c => qobject-input-visitor.c} (56%)
rename tests/{test-qmp-input-strict.c => test-qobject-input-strict.c} (98%)
rename tests/{test-qmp-input-visitor.c => test-qobject-input-visitor.c} (99%)docs/qapi-code-gen.txt +1 -1
@@ -1024,7 +1024,7 @@ Example: Visitor *v; UserDefOneList *arg1 = NULL; - v = qmp_input_visitor_new(QOBJECT(args), true); + v = qobject_input_visitor_new(QOBJECT(args), true); visit_start_struct(v, NULL, NULL, 0, &err); if (err) { goto out;
include/qapi/qobject-input-visitor.h renamed from include/qapi/qmp-input-visitor.h +5 -5
@@ -11,20 +11,20 @@* */ -#ifndef QMP_INPUT_VISITOR_H -#define QMP_INPUT_VISITOR_H +#ifndef QOBJECT_INPUT_VISITOR_H +#define QOBJECT_INPUT_VISITOR_H #include "qapi/visitor.h" #include "qapi/qmp/qobject.h" -typedef struct QmpInputVisitor QmpInputVisitor ; +typedef struct QObjectInputVisitor QObjectInputVisitor ;/* -* Return a new input visitor that converts QMP to QAPI. +* Return a new input visitor that converts a QObject to a QAPI object. * * Set @strict to reject a parse that doesn't consume all keys of a * dictionary; otherwise excess input is ignored. */ -Visitor *qmp_input_visitor_new (QObject *obj,bool strict); +Visitor *qobject_input_visitor_new (QObject *obj,bool strict);#endif
include/qapi/visitor.h +3 -3
@@ -25,14 +25,14 @@* for doing work at each node of a QAPI graph; it can also be used * for a virtual walk, where there is no actual QAPI C struct. * -* There are four kinds of visitor classes: input visitors (QMP, +* There are four kinds of visitor classes: input visitors (QObject, * string, and QemuOpts) parse an external representation and build -* the corresponding QAPI graph, output visitors (QMP and string) take +* the corresponding QAPI graph, output visitors (QObject and string) take * a completed QAPI graph and generate an external representation, the * dealloc visitor can take a QAPI graph (possibly partially * constructed) and recursively free its resources, and the clone * visitor performs a deep clone of one QAPI object to another. While -* the dealloc and QMP input/output visitors are general, the string, +* the dealloc and QObject input/output visitors are general, the string, * QemuOpts, and clone visitors have some implementation limitations; * see the documentation for each visitor for more details on what it * supports. Also, see visitor-impl.h for the callback contracts
monitor.c +1 -1
@@ -957,7 +957,7 @@ EventInfoList *qmp_query_events(Error **errp)* directly into QObject instead of first parsing it with * visit_type_SchemaInfoList() into a SchemaInfoList, then marshal it * to QObject with generated output marshallers, every time. Instead, -* we do it in test-qmp-input-visitor.c, just to make sure +* we do it in test-qobject-input-visitor.c, just to make sure * qapi-introspect.py's output actually conforms to the schema. */ static void qmp_query_qmp_schema(QDict *qdict, QObject **ret_data,
qapi/Makefile.objs +1 -1
@@ -1,4 +1,4 @@ -util-obj-y = qapi-visit-core.o qapi-dealloc-visitor.o qmp-input-visitor.o +util-obj-y = qapi-visit-core.o qapi-dealloc-visitor.o qobject-input-visitor.o util-obj-y += qmp-output-visitor.o qmp-registry.o qmp-dispatch.o util-obj-y += string-input-visitor.o string-output-visitor.o util-obj-y += opts-visitor.o qapi-clone-visitor.o
qapi/qobject-input-visitor.c renamed from qapi/qmp-input-visitor.c +87 -84
336 lines not on this page · raw message
qmp.c +2 -2
@@ -31,7 +31,7 @@#include "qom/qom-qobject.h" #include "qapi/qmp/qerror.h" #include "qapi/qmp/qobject.h" -#include "qapi/qmp-input-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "hw/boards.h" #include "qom/object_interfaces.h" #include "hw/mem/pc-dimm.h"
@@ -672,7 +672,7 @@ void qmp_object_add(const char *type, const char *id, } } - v =qmp_input_visitor_new (props, true); + v =qobject_input_visitor_new (props, true); obj =user_creatable_add_type (type, id, pdict, v, errp);visit_free (v);if (obj) {
qom/qom-qobject.c +2 -2
@@ -15,7 +15,7 @@#include "qom/object.h" #include "qom/qom-qobject.h" #include "qapi/visitor.h" -#include "qapi/qmp-input-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "qapi/qmp-output-visitor.h" void object_property_set_qobject(Object *obj, QObject *value,
@@ -23,7 +23,7 @@ void object_property_set_qobject(Object *obj, QObject *value, {Visitor *v;/* TODO: Should we reject, rather than ignore, excess input? */ - v =qmp_input_visitor_new (value, false); + v =qobject_input_visitor_new (value, false);object_property_set (obj, v, name, errp);visit_free (v); }
scripts/qapi-commands.py +2 -2
@@ -130,7 +130,7 @@ def gen_marshal(name, arg_type, boxed, ret_type): push_indent() ret += mcgen(''' - v = qmp_input_visitor_new(QOBJECT(args), true); + v = qobject_input_visitor_new(QOBJECT(args), true); visit_start_struct(v, NULL, NULL, 0, &err); if (err) { goto out;
@@ -294,7 +294,7 @@ fdef.write(mcgen(''' #include "qapi/qmp/dispatch.h" #include "qapi/visitor.h" #include "qapi/qmp-output-visitor.h" -#include "qapi/qmp-input-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "qapi/dealloc-visitor.h" #include "%(prefix)sqapi-types.h" #include "%(prefix)sqapi-visit.h"
target-s390x/cpu_models.c +2 -2
@@ -17,7 +17,7 @@#include "qapi/visitor.h" #include "qemu/error-report.h" #include "qapi/qmp/qerror.h" -#include "qapi/qmp-input-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "qapi/qmp/qbool.h" #ifndef CONFIG_USER_ONLY #include "sysemu/arch_init.h"
@@ -345,7 +345,7 @@ static void cpu_model_from_info(S390CPUModel *model, const CpuModelInfo *info, }if (qdict) { - visitor =qmp_input_visitor_new (info->props, true); + visitor =qobject_input_visitor_new (info->props, true);visit_start_struct (visitor,NULL ,NULL ,0 , errp);if (*errp) {object_unref (obj);
tests/.gitignore +2 -2
@@ -57,8 +57,8 @@ test-qht-par test-qmp-commands test-qmp-commands.h test-qmp-event -test-qmp-input-strict -test-qmp-input-visitor +test-qobject-input-strict +test-qobject-input-visitor test-qmp-introspect.[ch] test-qmp-marshal.c test-qmp-output-visitor
tests/Makefile.include +6 -6
@@ -24,9 +24,9 @@ check-unit-y += tests/test-qmp-output-visitor$(EXESUF) gcov-files-test-qmp-output-visitor-y = qapi/qmp-output-visitor.c check-unit-y += tests/test-clone-visitor$(EXESUF) gcov-files-test-clone-visitor-y = qapi/qapi-clone-visitor.c -check-unit-y += tests/test-qmp-input-visitor$(EXESUF) -gcov-files-test-qmp-input-visitor-y = qapi/qmp-input-visitor.c -check-unit-y += tests/test-qmp-input-strict$(EXESUF) +check-unit-y += tests/test-qobject-input-visitor$(EXESUF) +gcov-files-test-qobject-input-visitor-y = qapi/qobject-input-visitor.c +check-unit-y += tests/test-qobject-input-strict$(EXESUF) check-unit-y += tests/test-qmp-commands$(EXESUF) gcov-files-test-qmp-commands-y = qapi/qmp-dispatch.c check-unit-y += tests/test-string-input-visitor$(EXESUF)
@@ -441,7 +441,7 @@ test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \ tests/test-coroutine.o tests/test-string-output-visitor.o \ tests/test-string-input-visitor.o tests/test-qmp-output-visitor.o \ tests/test-clone-visitor.o \ - tests/test-qmp-input-visitor.o tests/test-qmp-input-strict.o \ + tests/test-qobject-input-visitor.o tests/test-qobject-input-strict.o \ tests/test-qmp-commands.o tests/test-visitor-serialization.o \ tests/test-x86-cpuid.o tests/test-mul64.o tests/test-int128.o \ tests/test-opts-visitor.o tests/test-qmp-event.o \
@@ -544,8 +544,8 @@ tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(te tests/test-qmp-event$(EXESUF): tests/test-qmp-event.o $(test-qapi-obj-y) tests/test-qmp-output-visitor$(EXESUF): tests/test-qmp-output-visitor.o $(test-qapi-obj-y) tests/test-clone-visitor$(EXESUF): tests/test-clone-visitor.o $(test-qapi-obj-y) -tests/test-qmp-input-visitor$(EXESUF): tests/test-qmp-input-visitor.o $(test-qapi-obj-y) -tests/test-qmp-input-strict$(EXESUF): tests/test-qmp-input-strict.o $(test-qapi-obj-y) +tests/test-qobject-input-visitor$(EXESUF): tests/test-qobject-input-visitor.o $(test-qapi-obj-y) +tests/test-qobject-input-strict$(EXESUF): tests/test-qobject-input-strict.o $(test-qapi-obj-y) tests/test-qmp-commands$(EXESUF): tests/test-qmp-commands.o tests/test-qmp-marshal.o $(test-qapi-obj-y) tests/test-visitor-serialization$(EXESUF): tests/test-visitor-serialization.o $(test-qapi-obj-y) tests/test-opts-visitor$(EXESUF): tests/test-opts-visitor.o $(test-qapi-obj-y)
tests/check-qnull.c +2 -2
@@ -10,7 +10,7 @@#include "qapi/qmp/qobject.h" #include "qemu-common.h" -#include "qapi/qmp-input-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "qapi/qmp-output-visitor.h" #include "qapi/error.h"
@@ -47,7 +47,7 @@ static void qnull_visit_test(void)g_assert (qnull_.refcnt ==1 ); obj =qnull (); - v =qmp_input_visitor_new (obj, true); + v =qobject_input_visitor_new (obj, true);qobject_decref (obj);visit_type_null (v,NULL , &error_abort);visit_free (v);
tests/test-qmp-commands.c +2 -2
@@ -4,7 +4,7 @@#include "test-qmp-commands.h" #include "qapi/qmp/dispatch.h" #include "qemu/module.h" -#include "qapi/qmp-input-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "tests/test-qapi-types.h" #include "tests/test-qapi-visit.h"
@@ -244,7 +244,7 @@ static void test_dealloc_partial(void) ud2_dict =qdict_new ();qdict_put_obj (ud2_dict,"string0" ,QOBJECT (qstring_from_str (text))); - v =qmp_input_visitor_new (QOBJECT (ud2_dict), true); + v =qobject_input_visitor_new (QOBJECT (ud2_dict), true);visit_type_UserDefTwo (v,NULL , &ud2, &err);visit_free (v);QDECREF (ud2_dict);
tests/test-qobject-input-strict.c renamed from tests/test-qmp-input-strict.c +3 -3
@@ -1,5 +1,5 @@/* -* QMP Input Visitor unit-tests (strict mode). +* QObject Input Visitor unit-tests (strict mode). * * Copyright (C) 2011-2012, 2015 Red Hat Inc. *
@@ -15,7 +15,7 @@#include "qemu-common.h" #include "qapi/error.h" -#include "qapi/qmp-input-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "test-qapi-types.h" #include "test-qapi-visit.h" #include "qapi/qmp/types.h"
@@ -53,7 +53,7 @@ static Visitor *validate_test_init_internal(TestInputVisitorData *data, data->obj =qobject_from_jsonv (json_string, ap);g_assert (data->obj); - data->qiv =qmp_input_visitor_new (data->obj, true); + data->qiv =qobject_input_visitor_new (data->obj, true);g_assert (data->qiv);return data->qiv; }
tests/test-qobject-input-visitor.c renamed from tests/test-qmp-input-visitor.c +3 -3
@@ -1,5 +1,5 @@/* -* QMP Input Visitor unit-tests. +* QObject Input Visitor unit-tests. * * Copyright (C) 2011-2016 Red Hat Inc. *
@@ -14,7 +14,7 @@#include "qemu-common.h" #include "qapi/error.h" -#include "qapi/qmp-input-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "test-qapi-types.h" #include "test-qapi-visit.h" #include "qapi/qmp/types.h"
@@ -49,7 +49,7 @@ static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data, data->obj =qobject_from_jsonv (json_string, ap);g_assert (data->obj); - data->qiv =qmp_input_visitor_new (data->obj, false); + data->qiv =qobject_input_visitor_new (data->obj, false);g_assert (data->qiv);return data->qiv; }
tests/test-string-input-visitor.c +1 -1
@@ -4,7 +4,7 @@* Copyright (C) 2012 Red Hat Inc. * * Authors: -* Paolo Bonzini <[email protected]> (based on test-qmp-input-visitor) +* Paolo Bonzini <[email protected]> (based on test-qobject-input-visitor) * * This work is licensed under the terms of the GNU GPL, version 2 or later. * See the COPYING file in the top-level directory.
tests/test-visitor-serialization.c +2 -2
@@ -20,7 +20,7 @@#include "qapi/error.h" #include "qapi/qmp/types.h" #include "qapi/qmp/qjson.h" -#include "qapi/qmp-input-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "qapi/qmp-output-visitor.h" #include "qapi/string-input-visitor.h" #include "qapi/string-output-visitor.h"
@@ -1040,7 +1040,7 @@ static void qmp_deserialize(void **native_out, void *datap, obj =qobject_from_json (qstring_get_str (output_json));QDECREF (output_json); - d->qiv =qmp_input_visitor_new (obj, true); + d->qiv =qobject_input_visitor_new (obj, true);qobject_decref (obj_orig);qobject_decref (obj);visit (d->qiv, native_out, errp);
util/qemu-sockets.c +1 -1
@@ -21,7 +21,7 @@#include "qapi/error.h" #include "qemu/sockets.h" #include "qemu/main-loop.h" -#include "qapi/qmp-input-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "qapi/qmp-output-visitor.h" #include "qapi-visit.h" #include "qemu/cutils.h"
Signature · 1 line
2.7.4
PATCH v14 8/21 qapi: allow QObjectInputVisitor to be created with QemuOpts
In reply to Daniel P. BerrangeInstead of requiring all callers to go through the mutli-step process of turning QemuOpts into a suitable QObject for visiting, add a new constructor that encapsulates this logic. This will allow QObjectInputVisitor to be a drop-in replacement for the existing OptsVisitor with minimal code changes for callers.
NB, at this point it is only supporting opts syntax which explicitly matches the QAPI schema structure, so is not yet a true drop-in replacement for OptsVisitor. The patches that follow will add the special cases requird for full backwards compatibility with OptsVisitor.
--- include/qapi/qobject-input-visitor.h | 15 +++++++++++++++ include/qemu/option.h | 2 +- qapi/qobject-input-visitor.c | 25 +++++++++++++++++++++++++ util/qemu-option.c | 2 +- 4 files changed, 42 insertions(+), 2 deletions(-)
include/qapi/qobject-input-visitor.h +15 -0
@@ -51,4 +51,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);*/ Visitor *qobject_input_visitor_new_autocast (QObject *obj); + +/** +* Create a new input visitor that converts @opts to a QAPI object. +* +* The QemuOpts will be converted into a QObject using the +* qdict_crumple() method to automatically create structs +* and lists. The resulting QDict will then be passed to the +* qobject_input_visitor_new_autocast() method. See the docs +* of that method for further details on processing behaviour. +* +* The returned input visitor should be released by calling +* visit_free() when no longer required. +*/ +Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts,Error **errp); +#endif
include/qemu/option.h +1 -1
@@ -125,7 +125,7 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char *params, int permit_abbrev);QemuOpts *qemu_opts_from_qdict (QemuOptsList *list,const QDict *qdict,Error **errp); -QDict *qemu_opts_to_qdict (QemuOpts *opts,QDict *qdict); +QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict);void qemu_opts_absorb_qdict (QemuOpts *opts,QDict *qdict,Error **errp);typedef int (*qemu_opts_loopfunc )(void *opaque,QemuOpts *opts,Error **errp);
qapi/qobject-input-visitor.c +25 -0
@@ -545,3 +545,28 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)return &v->visitor; } + + +Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts, +Error **errp) +{ +QDict *pdict; +QObject *pobj =NULL ; +Visitor *v =NULL ; + + pdict =qemu_opts_to_qdict (opts,NULL ); +if (!pdict) { + goto cleanup; + } + + pobj =qdict_crumple (pdict, true, errp); +if (!pobj) { + goto cleanup; + } + + v =qobject_input_visitor_new_autocast (pobj); + cleanup: +qobject_decref (pobj); +QDECREF (pdict); +return v; +}
util/qemu-option.c +1 -1
@@ -1058,7 +1058,7 @@ void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp)* TODO We'll want to use types appropriate for opt->desc->type, but * this is enough for now. */ -QDict *qemu_opts_to_qdict (QemuOpts *opts,QDict *qdict) +QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict) {QemuOpt *opt;QObject *val;
Signature · 1 line
2.7.4
PATCH v14 5/21 qapi: rename QmpOutputVisitor to QObjectOutputVisitor
In reply to Daniel P. BerrangeThe QmpOutputVisitor has no direct dependency on QMP. It is valid to use it anywhere that one wants a QObject. Rename it to better reflect its functionality as a generic QAPI to QObject converter.
---
block/qapi.c | 4 +-
blockdev.c | 4 +-
docs/qapi-code-gen.txt | 2 +-
...p-output-visitor.h => qobject-output-visitor.h} | 10 +-
qapi/Makefile.objs | 2 +-
qapi/qapi-clone-visitor.c | 2 +-
qapi/qmp-output-visitor.c | 256 ---------------------
qapi/qobject-output-visitor.c | 254 ++++++++++++++++++++
qemu-img.c | 8 +-
qom/object_interfaces.c | 2 +-
qom/qom-qobject.c | 4 +-
scripts/qapi-commands.py | 4 +-
scripts/qapi-event.py | 4 +-
tests/.gitignore | 2 +-
tests/Makefile.include | 8 +-
tests/check-qnull.c | 4 +-
...put-visitor.c => test-qobject-output-visitor.c} | 6 +-
tests/test-string-output-visitor.c | 2 +-
tests/test-visitor-serialization.c | 4 +-
util/qemu-sockets.c | 2 +-
20 files changed, 291 insertions(+), 293 deletions(-)
rename include/qapi/{qmp-output-visitor.h => qobject-output-visitor.h} (66%)
delete mode 100644 qapi/qmp-output-visitor.c
create mode 100644 qapi/qobject-output-visitor.c
rename tests/{test-qmp-output-visitor.c => test-qobject-output-visitor.c} (99%)block/qapi.c +2 -2
@@ -29,7 +29,7 @@#include "block/write-threshold.h" #include "qmp-commands.h" #include "qapi-visit.h" -#include "qapi/qmp-output-visitor.h" +#include "qapi/qobject-output-visitor.h" #include "qapi/qmp/types.h" #include "sysemu/block-backend.h" #include "qemu/cutils.h"
@@ -691,7 +691,7 @@ void bdrv_image_info_specific_dump(fprintf_function func_fprintf, void *f, ImageInfoSpecific *info_spec) {QObject *obj, *data; -Visitor *v =qmp_output_visitor_new (&obj); +Visitor *v =qobject_output_visitor_new (&obj);visit_type_ImageInfoSpecific (v,NULL , &info_spec, &error_abort);visit_complete (v, &obj);
blockdev.c +2 -2
@@ -43,7 +43,7 @@#include "qapi/qmp/types.h" #include "qapi-visit.h" #include "qapi/qmp/qerror.h" -#include "qapi/qmp-output-visitor.h" +#include "qapi/qobject-output-visitor.h" #include "qapi/util.h" #include "sysemu/sysemu.h" #include "block/block_int.h"
@@ -3828,7 +3828,7 @@ void qmp_blockdev_add(BlockdevOptions *options, Error **errp) {BlockDriverState *bs;QObject *obj; -Visitor *v =qmp_output_visitor_new (&obj); +Visitor *v =qobject_output_visitor_new (&obj);QDict *qdict;Error *local_err =NULL ;
docs/qapi-code-gen.txt +1 -1
@@ -1005,7 +1005,7 @@ Example: Error *err = NULL; Visitor *v; - v = qmp_output_visitor_new(ret_out); + v = qobject_output_visitor_new(ret_out); visit_type_UserDefOne(v, "unused", &ret_in, &err); if (!err) { visit_complete(v, ret_out);
include/qapi/qobject-output-visitor.h renamed from include/qapi/qmp-output-visitor.h +5 -5
@@ -11,20 +11,20 @@* */ -#ifndef QMP_OUTPUT_VISITOR_H -#define QMP_OUTPUT_VISITOR_H +#ifndef QOBJECT_OUTPUT_VISITOR_H +#define QOBJECT_OUTPUT_VISITOR_H #include "qapi/visitor.h" #include "qapi/qmp/qobject.h" -typedef struct QmpOutputVisitor QmpOutputVisitor ; +typedef struct QObjectOutputVisitor QObjectOutputVisitor ;/* -* Create a new QMP output visitor. +* Create a new QOBJECT output visitor. * * If everything else succeeds, pass @result to visit_complete() to * collect the result of the visit. */ -Visitor *qmp_output_visitor_new (QObject **result); +Visitor *qobject_output_visitor_new (QObject **result);#endif
qapi/Makefile.objs +1 -1
@@ -1,5 +1,5 @@ util-obj-y = qapi-visit-core.o qapi-dealloc-visitor.o qobject-input-visitor.o -util-obj-y += qmp-output-visitor.o qmp-registry.o qmp-dispatch.o +util-obj-y += qobject-output-visitor.o qmp-registry.o qmp-dispatch.o util-obj-y += string-input-visitor.o string-output-visitor.o util-obj-y += opts-visitor.o qapi-clone-visitor.o util-obj-y += qmp-event.o
qapi/qapi-clone-visitor.c +1 -1
@@ -110,7 +110,7 @@ static void qapi_clone_type_str(Visitor *v, const char *name, char **obj,assert (qcv->depth);/* * Pointer was already cloned by g_memdup; create fresh copy. -* Note that as long as qmp-output-visitor accepts NULL instead of +* Note that as long as qobject-output-visitor accepts NULL instead of * "", then we must do likewise. However, we want to obey the * input visitor semantics of never producing NULL when the empty * string is intended.
qapi/qmp-output-visitor.c deleted +0 -256
256 lines not on this page · raw message
qapi/qobject-output-visitor.c new file +254 -0
254 lines not on this page · raw message
qemu-img.c +4 -4
@@ -25,7 +25,7 @@#include "qemu-version.h" #include "qapi/error.h" #include "qapi-visit.h" -#include "qapi/qmp-output-visitor.h" +#include "qapi/qobject-output-visitor.h" #include "qapi/qmp/qerror.h" #include "qapi/qmp/qjson.h" #include "qemu/cutils.h"
@@ -500,7 +500,7 @@ static void dump_json_image_check(ImageCheck *check, bool quiet) {QString *str;QObject *obj; -Visitor *v =qmp_output_visitor_new (&obj); +Visitor *v =qobject_output_visitor_new (&obj);visit_type_ImageCheck (v,NULL , &check, &error_abort);visit_complete (v, &obj);
@@ -2193,7 +2193,7 @@ static void dump_json_image_info_list(ImageInfoList *list) {QString *str;QObject *obj; -Visitor *v =qmp_output_visitor_new (&obj); +Visitor *v =qobject_output_visitor_new (&obj);visit_type_ImageInfoList (v,NULL , &list, &error_abort);visit_complete (v, &obj);
@@ -2209,7 +2209,7 @@ static void dump_json_image_info(ImageInfo *info) {QString *str;QObject *obj; -Visitor *v =qmp_output_visitor_new (&obj); +Visitor *v =qobject_output_visitor_new (&obj);visit_type_ImageInfo (v,NULL , &info, &error_abort);visit_complete (v, &obj);
qom/object_interfaces.c +1 -1
@@ -3,7 +3,7 @@#include "qom/object_interfaces.h" #include "qemu/module.h" #include "qapi-visit.h" -#include "qapi/qmp-output-visitor.h" +#include "qapi/qobject-output-visitor.h" #include "qapi/opts-visitor.h" void user_creatable_complete (Object *obj,Error **errp)
qom/qom-qobject.c +2 -2
@@ -16,7 +16,7 @@#include "qom/qom-qobject.h" #include "qapi/visitor.h" #include "qapi/qobject-input-visitor.h" -#include "qapi/qmp-output-visitor.h" +#include "qapi/qobject-output-visitor.h" void object_property_set_qobject (Object *obj,QObject *value,const char *name,Error **errp)
@@ -35,7 +35,7 @@ QObject *object_property_get_qobject(Object *obj, const char *name,Error *local_err =NULL ;Visitor *v; - v =qmp_output_visitor_new (&ret); + v =qobject_output_visitor_new (&ret);object_property_get (obj, v, name, &local_err);if (!local_err) {visit_complete (v, &ret);
scripts/qapi-commands.py +2 -2
@@ -68,7 +68,7 @@ static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error *err = NULL; Visitor *v; - v = qmp_output_visitor_new(ret_out); + v = qobject_output_visitor_new(ret_out); visit_type_%(c_name)s(v, "unused", &ret_in, &err); if (!err) { visit_complete(v, ret_out);
@@ -293,7 +293,7 @@ fdef.write(mcgen(''' #include "qapi/qmp/types.h" #include "qapi/qmp/dispatch.h" #include "qapi/visitor.h" -#include "qapi/qmp-output-visitor.h" +#include "qapi/qobject-output-visitor.h" #include "qapi/qobject-input-visitor.h" #include "qapi/dealloc-visitor.h" #include "%(prefix)sqapi-types.h"
scripts/qapi-event.py +2 -2
@@ -98,7 +98,7 @@ def gen_event_send(name, arg_type, boxed): if arg_type and not arg_type.is_empty(): ret += mcgen(''' - v = qmp_output_visitor_new(&obj); + v = qobject_output_visitor_new(&obj); ''') if not arg_type.is_implicit(): ret += mcgen('''
@@ -209,7 +209,7 @@ fdef.write(mcgen(''' #include "qemu-common.h" #include "%(prefix)sqapi-event.h" #include "%(prefix)sqapi-visit.h" -#include "qapi/qmp-output-visitor.h" +#include "qapi/qobject-output-visitor.h" #include "qapi/qmp-event.h" ''',
tests/.gitignore +1 -1
@@ -61,7 +61,7 @@ test-qobject-input-strict test-qobject-input-visitor test-qmp-introspect.[ch] test-qmp-marshal.c -test-qmp-output-visitor +test-qobject-output-visitor test-rcu-list test-replication test-rfifolock
tests/Makefile.include +4 -4
@@ -20,8 +20,8 @@ check-unit-y += tests/check-qnull$(EXESUF) gcov-files-check-qnull-y = qobject/qnull.c check-unit-y += tests/check-qjson$(EXESUF) gcov-files-check-qjson-y = qobject/qjson.c -check-unit-y += tests/test-qmp-output-visitor$(EXESUF) -gcov-files-test-qmp-output-visitor-y = qapi/qmp-output-visitor.c +check-unit-y += tests/test-qobject-output-visitor$(EXESUF) +gcov-files-test-qobject-output-visitor-y = qapi/qobject-output-visitor.c check-unit-y += tests/test-clone-visitor$(EXESUF) gcov-files-test-clone-visitor-y = qapi/qapi-clone-visitor.c check-unit-y += tests/test-qobject-input-visitor$(EXESUF)
@@ -439,7 +439,7 @@ test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \ tests/check-qlist.o tests/check-qfloat.o tests/check-qnull.o \ tests/check-qjson.o \ tests/test-coroutine.o tests/test-string-output-visitor.o \ - tests/test-string-input-visitor.o tests/test-qmp-output-visitor.o \ + tests/test-string-input-visitor.o tests/test-qobject-output-visitor.o \ tests/test-clone-visitor.o \ tests/test-qobject-input-visitor.o tests/test-qobject-input-strict.o \ tests/test-qmp-commands.o tests/test-visitor-serialization.o \
@@ -542,7 +542,7 @@ $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-int tests/test-string-output-visitor$(EXESUF): tests/test-string-output-visitor.o $(test-qapi-obj-y) tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(test-qapi-obj-y) tests/test-qmp-event$(EXESUF): tests/test-qmp-event.o $(test-qapi-obj-y) -tests/test-qmp-output-visitor$(EXESUF): tests/test-qmp-output-visitor.o $(test-qapi-obj-y) +tests/test-qobject-output-visitor$(EXESUF): tests/test-qobject-output-visitor.o $(test-qapi-obj-y) tests/test-clone-visitor$(EXESUF): tests/test-clone-visitor.o $(test-qapi-obj-y) tests/test-qobject-input-visitor$(EXESUF): tests/test-qobject-input-visitor.o $(test-qapi-obj-y) tests/test-qobject-input-strict$(EXESUF): tests/test-qobject-input-strict.o $(test-qapi-obj-y)
tests/check-qnull.c +2 -2
@@ -11,7 +11,7 @@#include "qapi/qmp/qobject.h" #include "qemu-common.h" #include "qapi/qobject-input-visitor.h" -#include "qapi/qmp-output-visitor.h" +#include "qapi/qobject-output-visitor.h" #include "qapi/error.h" /*
@@ -52,7 +52,7 @@ static void qnull_visit_test(void)visit_type_null (v,NULL , &error_abort);visit_free (v); - v =qmp_output_visitor_new (&obj); + v =qobject_output_visitor_new (&obj);visit_type_null (v,NULL , &error_abort);visit_complete (v, &obj);g_assert (obj == &qnull_);
tests/test-qobject-output-visitor.c renamed from tests/test-qmp-output-visitor.c +3 -3
@@ -1,5 +1,5 @@/* -* QMP Output Visitor unit-tests. +* QObject Output Visitor unit-tests. * * Copyright (C) 2011-2016 Red Hat Inc. *
@@ -14,7 +14,7 @@#include "qemu-common.h" #include "qapi/error.h" -#include "qapi/qmp-output-visitor.h" +#include "qapi/qobject-output-visitor.h" #include "test-qapi-types.h" #include "test-qapi-visit.h" #include "qapi/qmp/types.h"
@@ -28,7 +28,7 @@ typedef struct TestOutputVisitorData {static void visitor_output_setup (TestOutputVisitorData *data,const void *unused) { - data->ov =qmp_output_visitor_new (&data->obj); + data->ov =qobject_output_visitor_new (&data->obj);g_assert (data->ov); }
tests/test-string-output-visitor.c +1 -1
@@ -4,7 +4,7 @@* Copyright (C) 2012 Red Hat Inc. * * Authors: -* Paolo Bonzini <[email protected]> (based on test-qmp-output-visitor) +* Paolo Bonzini <[email protected]> (based on test-qobject-output-visitor) * * This work is licensed under the terms of the GNU GPL, version 2 or later. * See the COPYING file in the top-level directory.
tests/test-visitor-serialization.c +2 -2
@@ -21,7 +21,7 @@#include "qapi/qmp/types.h" #include "qapi/qmp/qjson.h" #include "qapi/qobject-input-visitor.h" -#include "qapi/qmp-output-visitor.h" +#include "qapi/qobject-output-visitor.h" #include "qapi/string-input-visitor.h" #include "qapi/string-output-visitor.h" #include "qapi-types.h"
@@ -1022,7 +1022,7 @@ static void qmp_serialize(void *native_in, void **datap, {QmpSerializeData *d =g_malloc0 (sizeof (*d)); - d->qov =qmp_output_visitor_new (&d->obj); + d->qov =qobject_output_visitor_new (&d->obj);visit (d->qov, &native_in, errp); *datap = d; }
util/qemu-sockets.c +1 -1
@@ -22,7 +22,7 @@#include "qemu/sockets.h" #include "qemu/main-loop.h" #include "qapi/qobject-input-visitor.h" -#include "qapi/qmp-output-visitor.h" +#include "qapi/qobject-output-visitor.h" #include "qapi-visit.h" #include "qemu/cutils.h"
Signature · 1 line
2.7.4
PATCH v14 7/21 qapi: permit scalar type conversions in QObjectInputVisitor
In reply to Daniel P. BerrangeCurrently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.
To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.
--- include/qapi/qobject-input-visitor.h | 32 +++++- qapi/qobject-input-visitor.c | 132 ++++++++++++++++++++++++ tests/test-qobject-input-visitor.c | 194 ++++++++++++++++++++++++++++++++++- 3 files changed, 350 insertions(+), 8 deletions(-)
include/qapi/qobject-input-visitor.h +28 -4
@@ -19,12 +19,36 @@typedef struct QObjectInputVisitor QObjectInputVisitor ; -/* -* Return a new input visitor that converts a QObject to a QAPI object. +/** +* Create a new input visitor that converts @obj to a QAPI object. +* +* Any scalar values in the @obj input data structure should be in the +* required type already. i.e. if visiting a bool, the value should +* already be a QBool instance. * -* Set @strict to reject a parse that doesn't consume all keys of a -* dictionary; otherwise excess input is ignored. +* If @strict is set to true, then an error will be reported if any +* dict keys are not consumed during visitation. If @strict is false +* then extra dict keys are silently ignored. +* +* The returned input visitor should be released by calling +* visit_free() when no longer required. */ Visitor *qobject_input_visitor_new (QObject *obj,bool strict); +/** +* Create a new input visitor that converts @obj to a QAPI object. +* +* Any scalar values in the @obj input data structure should always be +* represented as strings. i.e. if visiting a boolean, the value should +* be a QString whose contents represent a valid boolean. +* +* The visitor always operates in strict mode, requiring all dict keys +* to be consumed during visitation. An error will be reported if this +* does not happen. +* +* The returned input visitor should be released by calling +* visit_free() when no longer required. +*/ +Visitor *qobject_input_visitor_new_autocast (QObject *obj); +#endif
qapi/qobject-input-visitor.c +132 -0
171 lines not on this page · raw message
tests/test-qobject-input-visitor.c +190 -4
256 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v14 9/21 qapi: permit auto-creating single element lists
In reply to Daniel P. BerrangeWhen converting QemuOpts to a QObject, there is no information about compound types available, so when visiting a list, the corresponding QObject is not guaranteed to be a QList. We therefore need to be able to auto-create a single element QList from whatever type we find.
This mode should only be enabled if you have compatibility requirements for
-arg foo=hello,foo=world
to be treated as equivalent to the preferred syntax:
-arg foo.0=hello,foo.1=world
--- include/qapi/qobject-input-visitor.h | 20 +++++++- qapi/qobject-input-visitor.c | 27 +++++++++-- tests/test-qobject-input-visitor.c | 88 +++++++++++++++++++++++++++++++----- 3 files changed, 117 insertions(+), 18 deletions(-)
include/qapi/qobject-input-visitor.h +18 -2
@@ -42,6 +42,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* represented as strings. i.e. if visiting a boolean, the value should * be a QString whose contents represent a valid boolean. * +* If @autocreate_list is true, then as an alternative to a normal QList, +* list values can be stored as a QString or QDict instead, which will +* be interpreted as representing single element lists. This should only +* by used if compatibility is required with the OptsVisitor which allowed +* repeated keys, without list indexes, to represent lists. e.g. set this +* to true if you have compatibility requirements for +* +* -arg foo=hello,foo=world +* +* to be treated as equivalent to the preferred syntax: +* +* -arg foo.0=hello,foo.1=world +* * The visitor always operates in strict mode, requiring all dict keys * to be consumed during visitation. An error will be reported if this * does not happen.
@@ -49,7 +62,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* The returned input visitor should be released by calling * visit_free() when no longer required. */ -Visitor *qobject_input_visitor_new_autocast (QObject *obj); +Visitor *qobject_input_visitor_new_autocast (QObject *obj, +bool autocreate_list);/**
@@ -64,6 +78,8 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj);* The returned input visitor should be released by calling * visit_free() when no longer required. */ -Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts,Error **errp); +Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts, +bool autocreate_list, +Error **errp);#endif
qapi/qobject-input-visitor.c +23 -4
@@ -48,6 +48,10 @@ struct QObjectInputVisitor/* True to reject parse in visit_end_struct() if unvisited keys remain. */ bool strict; + +/* Whether we can auto-create single element lists when +* encountering a non-QList type */ +bool autocreate_list; };static QObjectInputVisitor *to_qiv (Visitor *v)
@@ -108,6 +112,7 @@ static const QListEntry *qobject_input_push(QObjectInputVisitor *qiv,assert (obj); tos->obj = obj; tos->qapi = qapi; +qobject_incref (obj);if (qiv->strict &&qobject_type (obj) ==QTYPE_QDICT ) { h =g_hash_table_new (g_str_hash, g_str_equal);
@@ -147,6 +152,7 @@ static void qobject_input_stack_object_free(StackObject *tos)if (tos->h) {g_hash_table_unref (tos->h); } +qobject_decref (tos->obj);g_free (tos); }
@@ -197,7 +203,7 @@ static void qobject_input_start_list(Visitor *v, const char *name,QObject *qobj =qobject_input_get_object (qiv, name, true);const QListEntry *entry; -if (!qobj ||qobject_type (qobj) !=QTYPE_QLIST ) { +if (!qobj || (!qiv->autocreate_list &&qobject_type (qobj) !=QTYPE_QLIST )) {if (list) { *list =NULL ; }
@@ -206,7 +212,16 @@ static void qobject_input_start_list(Visitor *v, const char *name,return ; } - entry =qobject_input_push (qiv, qobj, list, errp); +if (qobject_type (qobj) !=QTYPE_QLIST ) { +QList *tmplist =qlist_new (); +qlist_append_obj (tmplist, qobj); +qobject_incref (qobj); + entry =qobject_input_push (qiv,QOBJECT (tmplist), list, errp); +QDECREF (tmplist); + }else { + entry =qobject_input_push (qiv, qobj, list, errp); + } +if (list) {if (entry) { *list =g_malloc0 (size);
@@ -514,7 +529,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)return &v->visitor; } -Visitor *qobject_input_visitor_new_autocast (QObject *obj) +Visitor *qobject_input_visitor_new_autocast (QObject *obj, +bool autocreate_list) {QObjectInputVisitor *v;
@@ -539,6 +555,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj) v->visitor.optional = qobject_input_optional; v->visitor.free = qobject_input_free; v->strict = true; + v->autocreate_list = autocreate_list; v->root = obj;qobject_incref (obj);
@@ -548,6 +565,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts, +bool autocreate_list,Error **errp) {QDict *pdict;
@@ -564,7 +582,8 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, goto cleanup; } - v =qobject_input_visitor_new_autocast (pobj); + v =qobject_input_visitor_new_autocast (pobj, + autocreate_list); cleanup:qobject_decref (pobj);QDECREF (pdict);
tests/test-qobject-input-visitor.c +76 -12
187 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v14 10/21 qapi: permit auto-creating nested structs
In reply to Daniel P. BerrangeSome of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the qdict_crumple() call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.
Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.
The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach
-net vlan=1,id=foo,name=bar,opts.type=tap,\
opts.data.fd=3,opts.data.script=ifupThe legacy syntax will just be presenting
-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup
So we need to auto-create 3 levels of struct when visiting.
The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.
--- include/qapi/qobject-input-visitor.h | 22 +++++- qapi/qobject-input-visitor.c | 59 ++++++++++++++-- tests/test-qobject-input-visitor.c | 130 ++++++++++++++++++++++++++++++++--- 3 files changed, 194 insertions(+), 17 deletions(-)
include/qapi/qobject-input-visitor.h +20 -2
@@ -45,7 +45,7 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* If @autocreate_list is true, then as an alternative to a normal QList, * list values can be stored as a QString or QDict instead, which will * be interpreted as representing single element lists. This should only -* by used if compatibility is required with the OptsVisitor which allowed +* be used if compatibility is required with the OptsVisitor which allowed * repeated keys, without list indexes, to represent lists. e.g. set this * to true if you have compatibility requirements for *
@@ -55,6 +55,22 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* * -arg foo.0=hello,foo.1=world * +* If @autocreate_struct_levels is non-zero, then when visiting structs, +* if the corresponding QDict is not found, it will automatically create +* a QDict containing all remaining unvisited options. This should only +* be used if compatibility is required with the OptsVisitor which flatten +* structs so that all keys were at the same level. e.g. set this to a +* non-zero number if you compatibility requirements for +* +* -arg type=person,surname=blogs,forename=fred +* +* to be treated as equivalent to the perferred syntax +* +* -arg type=person,data.surname=blogs,data.forename=fred +* +* The value given determines how many levels of structs are allowed to +* be flattened in this way. +* * The visitor always operates in strict mode, requiring all dict keys * to be consumed during visitation. An error will be reported if this * does not happen.
@@ -63,7 +79,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* visit_free() when no longer required. */ Visitor *qobject_input_visitor_new_autocast (QObject *obj, -bool autocreate_list); +bool autocreate_list, +size_t autocreate_struct_levels);/**
@@ -80,6 +97,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,*/ Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts,bool autocreate_list, +size_t autocreate_struct_levels,Error **errp);#endif
qapi/qobject-input-visitor.c +54 -5
130 lines not on this page · raw message
tests/test-qobject-input-visitor.c +120 -10
174 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v14 11/21 qapi: add integer range support for QObjectInputVisitor
In reply to Daniel P. BerrangeThe traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:
-arg foo=5,foo=6,foo=7
-arg foo=5-7
This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.
This means that
-arg foo=5-7
is treated as equivalent to
-arg foo.0=5,foo.1=6,foo.2=7
Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.
--- include/qapi/qobject-input-visitor.h | 23 ++++- qapi/qobject-input-visitor.c | 158 ++++++++++++++++++++++++++-- tests/test-qobject-input-visitor.c | 195 +++++++++++++++++++++++++++++++++-- 3 files changed, 360 insertions(+), 16 deletions(-)
include/qapi/qobject-input-visitor.h +22 -1
@@ -19,6 +19,12 @@typedef struct QObjectInputVisitor QObjectInputVisitor ; +/* Inclusive upper bound on the size of any flattened range. This is a safety +* (= anti-annoyance) measure; wrong ranges should not cause long startup +* delays nor exhaust virtual memory. +*/ +#define QIV_RANGE_MAX 65536 +/** * Create a new input visitor that converts @obj to a QAPI object. *
@@ -71,6 +77,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* The value given determines how many levels of structs are allowed to * be flattened in this way. * +* If @permit_int_ranges is true, then when visiting a list of integers, +* the integer value strings may encode ranges eg a single element +* containing "5-7" is treated as if there were three elements "5", "6", +* "7". This should only be used if compatibility is required with the +* OptsVisitor which would allow integer ranges. e.g. set this to true +* if you have compatibility requirements for +* +* -arg val=5-8 +* +* to be treated as equivalent to the preferred syntax: +* +* -arg val.0=5,val.1=6,val.2=7,val.3=8 +* * The visitor always operates in strict mode, requiring all dict keys * to be consumed during visitation. An error will be reported if this * does not happen.
@@ -80,7 +99,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);*/ Visitor *qobject_input_visitor_new_autocast (QObject *obj,bool autocreate_list, -size_t autocreate_struct_levels); +size_t autocreate_struct_levels, +bool permit_int_ranges);/**
@@ -98,6 +118,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts,bool autocreate_list,size_t autocreate_struct_levels, +bool permit_int_ranges,Error **errp);#endif
qapi/qobject-input-visitor.c +149 -9
236 lines not on this page · raw message
tests/test-qobject-input-visitor.c +189 -6
252 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v14 13/21 qdict: allow qdict_crumple to accept compound types as values
In reply to Daniel P. BerrangeCurrently qdict_crumple requires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.
In order to have backwards compatibility with the OptsVisitor, qemu_opt_to_qdict() has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given to qdict_crumple().
To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed
foo=hello,foo=world,foo=wibble
foo.0=hello,foo.1=world,foo.2=wibble
but this is forbidden
foo=hello,foo=world,foo.2=wibble
--- qobject/qdict.c | 45 +++++++++----- tests/check-qdict.c | 175 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 204 insertions(+), 16 deletions(-)
qobject/qdict.c +29 -16
@@ -814,15 +814,16 @@ static int qdict_is_list(QDict *maybe_list, Error **errp)/** * qdict_crumple: -* @src: the original flat dictionary (only scalar values) to crumple +* @src: the original dictionary to crumple * @recursive: true to recursively crumple nested dictionaries * -* Takes a flat dictionary whose keys use '.' separator to indicate -* nesting, and values are scalars, and crumples it into a nested -* structure. If the @recursive parameter is false, then only the -* first level of structure implied by the keys will be crumpled. If -* @recursive is true, then the input will be recursively crumpled to -* expand all levels of structure in the keys. +* Takes a dictionary whose keys use '.' separator to indicate +* nesting, crumples it into a nested structure. The values in +* @src are permitted to be scalars or compound typee. If the +* @recursive parameter is false, then only the first level of +* structure implied by the keys will be crumpled. If @recursive +* is true, then the input will be recursively crumpled to expand +* all levels of structure in the keys. * * To include a literal '.' in a key name, it must be escaped as '..' *
@@ -843,7 +844,10 @@ static int qdict_is_list(QDict *maybe_list, Error **errp)* The following scenarios in the input dict will result in an * error being returned: * -* - Any values in @src are non-scalar types +* - If a key in @src has a value that is non-scalar, and +* a later key implies creation of a compound type. e.g. +* if 'foo' point to a 'QList' or 'QDict', then it is +* not permitted to also have 'foo.NNN' keys later. * - If keys in @src imply that a particular level is both a * list and a dict. e.g., "foo.0.bar" and "foo.eek.bar". * - If keys in @src imply that a particular level is a list,
@@ -864,21 +868,26 @@ QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp)char *prefix =NULL ;const char *suffix =NULL ;int is_list; +QDict *compound; + compound =qdict_new (); two_level =qdict_new ();/* Step 1: split our totally flat dict into a two level dict */ for (ent =qdict_first (src); ent !=NULL ; ent =qdict_next (src, ent)) { -if (qobject_type (ent->value) ==QTYPE_QDICT || -qobject_type (ent->value) ==QTYPE_QLIST ) { -error_setg (errp,"Value %s is not a scalar" , - ent->key); - goto error; - } -qdict_split_flat_key (ent->key, &prefix, &suffix); child =qdict_get (two_level, prefix); + +if (qdict_haskey (compound, prefix)) { +error_setg (errp,"Key %s is already set as a list/dict" , prefix); + goto error; + } +if (qobject_type (ent->value) ==QTYPE_QLIST || +qobject_type (ent->value) ==QTYPE_QDICT ) { +qobject_incref (ent->value); +qdict_put_obj (compound, prefix, ent->value); + }if (suffix) {if (child) {if (qobject_type (child) !=QTYPE_QDICT ) {
@@ -913,7 +922,8 @@ QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp)for (ent =qdict_first (two_level); ent !=NULL ; ent =qdict_next (two_level, ent)) { -if (qobject_type (ent->value) ==QTYPE_QDICT ) { +if (qobject_type (ent->value) ==QTYPE_QDICT && + !qdict_haskey (compound, ent->key)) { child =qdict_crumple (qobject_to_qdict (ent->value), recursive, errp);if (!child) {
@@ -961,10 +971,13 @@ QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp) dst =QOBJECT (multi_level); } +QDECREF (compound); +return dst; error:g_free (prefix); +QDECREF (compound);QDECREF (multi_level);QDECREF (two_level);qobject_decref (dst);
tests/check-qdict.c +175 -0
187 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v14 14/21 qapi: allow repeated opts with qobject_input_visitor_new_opts
In reply to Daniel P. BerrangeThe qobject_input_visitor_new_opts() method gains a new parameter to control whether it allows repeated option keys in the input QemuOpts or not.
--- include/qapi/qobject-input-visitor.h | 6 ++++++ qapi/qobject-input-visitor.c | 5 ++++- 2 files changed, 10 insertions(+), 1 deletion(-)
include/qapi/qobject-input-visitor.h +6 -0
@@ -112,6 +112,11 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,* qobject_input_visitor_new_autocast() method. See the docs * of that method for further details on processing behaviour. * +* If the @permit_repeated_opts parameter is true, then the input +* @opts is allowed to contain repeated keys and they will be +* turned into a QList. If it is false, then repeated keys will +* result in an error being reported. +* * The returned input visitor should be released by calling * visit_free() when no longer required. */
@@ -119,6 +124,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,bool autocreate_list, size_t autocreate_struct_levels, bool permit_int_ranges, + bool permit_repeated_opts, Error **errp);#endif
qapi/qobject-input-visitor.c +4 -1
@@ -753,6 +753,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,bool autocreate_list, size_t autocreate_struct_levels, bool permit_int_ranges, + bool permit_repeated_opts, Error **errp) { QDict *pdict;
@@ -760,7 +761,9 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,Visitor *v =NULL ; pdict =qemu_opts_to_qdict (opts,NULL , -QEMU_OPTS_REPEAT_POLICY_LAST , + permit_repeated_opts ? +QEMU_OPTS_REPEAT_POLICY_ALL : +QEMU_OPTS_REPEAT_POLICY_ERROR , errp);if (!pdict) { goto cleanup;
Signature · 1 line
2.7.4
PATCH v14 12/21 option: allow qemu_opts_to_qdict to merge repeated options
In reply to Daniel P. BerrangeIf given an option string such as
size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind
the qemu_opts_to_qdict() method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:
size=QString("1024")
nodes=QString("1-2")
policy=QString("bind")With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain
size=QString("1024")
nodes=QList([QString("10"),
QString("4-5"),
QString("1-2")])
policy=QString("bind")Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.
In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.
All existing callers are all converted to explicitly request the historical behaviour of only reporting the last key. Later patches will make use of the new modes.
--- blockdev.c | 7 ++- include/qemu/option.h | 13 ++++- monitor.c | 3 +- qapi/qobject-input-visitor.c | 4 +- qemu-img.c | 4 +- qemu-io-cmds.c | 3 +- qemu-io.c | 6 +- qemu-nbd.c | 3 +- qom/object_interfaces.c | 3 +- tests/test-qemu-opts.c | 132 +++++++++++++++++++++++++++++++++++++++++++ tests/test-replication.c | 9 ++- util/qemu-option.c | 64 ++++++++++++++++++--- 12 files changed, 229 insertions(+), 22 deletions(-)
blockdev.c +4 -3
@@ -914,7 +914,8 @@ DriveInfo *drive_new(QemuOpts *all_opts, BlockInterfaceType block_default_type)/* Get a QDict for processing the options */ bs_opts =qdict_new (); -qemu_opts_to_qdict (all_opts, bs_opts); +qemu_opts_to_qdict (all_opts, bs_opts,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort); legacy_opts =qemu_opts_create (&qemu_legacy_drive_opts,NULL ,0 , &error_abort);
@@ -3804,8 +3805,8 @@ void hmp_drive_add_node(Monitor *mon, const char *optstr)return ; } - qdict =qemu_opts_to_qdict (opts,NULL ); - + qdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);if (!qdict_get_try_str (qdict,"node-name" )) {QDECREF (qdict);error_report ("'node-name' needs to be specified" );
include/qemu/option.h +12 -1
@@ -125,7 +125,18 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char *params, int permit_abbrev);QemuOpts *qemu_opts_from_qdict (QemuOptsList *list,const QDict *qdict,Error **errp); -QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict); +typedef enum { +/* Last occurrence of a duplicate option silently replaces earlier */ +QEMU_OPTS_REPEAT_POLICY_LAST , +/* Each occurrence of a duplicate option converts value to a QList */ +QEMU_OPTS_REPEAT_POLICY_ALL , +/* First occurrence of a duplicate option causes an error */ +QEMU_OPTS_REPEAT_POLICY_ERROR , +}QemuOptsRepeatPolicy ; + +QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict, +QemuOptsRepeatPolicy repeatPolicy, +Error **errp);void qemu_opts_absorb_qdict (QemuOpts *opts,QDict *qdict,Error **errp);typedef int (*qemu_opts_loopfunc )(void *opaque,QemuOpts *opts,Error **errp);
monitor.c +2 -1
@@ -2642,7 +2642,8 @@ static QDict *monitor_parse_arguments(Monitor *mon,if (!opts) { goto fail; } -qemu_opts_to_qdict (opts, qdict); +qemu_opts_to_qdict (opts, qdict,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);qemu_opts_del (opts); }break ;
qapi/qobject-input-visitor.c +3 -1
@@ -759,7 +759,9 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,QObject *pobj =NULL ;Visitor *v =NULL ; - pdict =qemu_opts_to_qdict (opts,NULL ); + pdict =qemu_opts_to_qdict (opts,NULL , +QEMU_OPTS_REPEAT_POLICY_LAST , + errp);if (!pdict) { goto cleanup; }
qemu-img.c +3 -1
@@ -273,7 +273,9 @@ static BlockBackend *img_open_opts(const char *optstr,QDict *options;Error *local_err =NULL ;BlockBackend *blk; - options =qemu_opts_to_qdict (opts,NULL ); + options =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort); + blk =blk_new_open (NULL ,NULL , options, flags, &local_err);if (!blk) {error_reportf_err (local_err,"Could not open '%s': " , optstr);
qemu-io-cmds.c +2 -1
@@ -1952,7 +1952,8 @@ static int reopen_f(BlockBackend *blk, int argc, char **argv) } qopts =qemu_opts_find (&reopen_opts,NULL ); - opts = qopts ?qemu_opts_to_qdict (qopts,NULL ) :NULL ; + opts = qopts ?qemu_opts_to_qdict (qopts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort) :NULL ;qemu_opts_reset (&reopen_opts); brq =bdrv_reopen_queue (NULL , bs, opts, flags);
qemu-io.c +4 -2
@@ -207,7 +207,8 @@ static int open_f(BlockBackend *blk, int argc, char **argv) } qopts =qemu_opts_find (&empty_opts,NULL ); - opts = qopts ?qemu_opts_to_qdict (qopts,NULL ) :NULL ; + opts = qopts ?qemu_opts_to_qdict (qopts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort) :NULL ;qemu_opts_reset (&empty_opts);if (optind == argc -1 ) {
@@ -593,7 +594,8 @@ int main(int argc, char **argv)if (!qopts) {exit (1 ); } - opts =qemu_opts_to_qdict (qopts,NULL ); + opts =qemu_opts_to_qdict (qopts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);openfile (NULL , flags, writethrough, opts); }else {if (format) {
qemu-nbd.c +2 -1
@@ -859,7 +859,8 @@ int main(int argc, char **argv)qemu_opts_reset (&file_opts);exit (EXIT_FAILURE ); } - options =qemu_opts_to_qdict (opts,NULL ); + options =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);qemu_opts_reset (&file_opts); blk =blk_new_open (NULL ,NULL , options, flags, &local_err); }else {
qom/object_interfaces.c +2 -1
@@ -161,7 +161,8 @@ Object *user_creatable_add_opts(QemuOpts *opts, Error **errp)Object *obj =NULL ; v =opts_visitor_new (opts); - pdict =qemu_opts_to_qdict (opts,NULL ); + pdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort); obj =user_creatable_add (pdict, v, errp);visit_free (v);
tests/test-qemu-opts.c +132 -0
144 lines not on this page · raw message
tests/test-replication.c +6 -3
@@ -181,7 +181,8 @@ static BlockBackend *start_primary(void) opts =qemu_opts_parse_noisily (&qemu_drive_opts, cmdline, false);g_free (cmdline); - qdict =qemu_opts_to_qdict (opts,NULL ); + qdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);qdict_set_default_str (qdict,BDRV_OPT_CACHE_DIRECT ,"off" );qdict_set_default_str (qdict,BDRV_OPT_CACHE_NO_FLUSH ,"off" );
@@ -311,7 +312,8 @@ static BlockBackend *start_secondary(void) opts =qemu_opts_parse_noisily (&qemu_drive_opts, cmdline, false);g_free (cmdline); - qdict =qemu_opts_to_qdict (opts,NULL ); + qdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);qdict_set_default_str (qdict,BDRV_OPT_CACHE_DIRECT ,"off" );qdict_set_default_str (qdict,BDRV_OPT_CACHE_NO_FLUSH ,"off" );
@@ -336,7 +338,8 @@ static BlockBackend *start_secondary(void) opts =qemu_opts_parse_noisily (&qemu_drive_opts, cmdline, false);g_free (cmdline); - qdict =qemu_opts_to_qdict (opts,NULL ); + qdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);qdict_set_default_str (qdict,BDRV_OPT_CACHE_DIRECT ,"off" );qdict_set_default_str (qdict,BDRV_OPT_CACHE_NO_FLUSH ,"off" );
util/qemu-option.c +57 -7
80 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v14 15/21 qom: support non-scalar properties with -object
In reply to Daniel P. BerrangeThe current -object command line syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.
This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.
The QObjectInputVisitor now has functionality that allows it to replace OptsVisitor, maintaining full backwards compatibility for previous CLI syntax, while also allowing use of new syntax for structs.
Thus -object can now support non-scalar properties, for example the QMP object:
{
"execute": "object-add",
"arguments": {
"qom-type": "demo",
"id": "demo0",
"parameters": {
"foo": [
{ "bar": "one", "wizz": "1" },
{ "bar": "two", "wizz": "2" }
]
}
}
}Would be creatable via the CLI now using
$QEMU \
-object demo,id=demo0,\
foo.0.bar=one,foo.0.wizz=1,\
foo.1.bar=two,foo.1.wizz=2Notice that this syntax is intentionally compatible with that currently used by block drivers. NB the indentation seen here after line continuations should not be used in reality, it is just for clarity in this commit message.
--- qapi/qobject-input-visitor.c | 2 +- qom/object_interfaces.c | 37 ++++- tests/check-qom-proplist.c | 367 ++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 397 insertions(+), 9 deletions(-)
qapi/qobject-input-visitor.c +1 -1
@@ -204,7 +204,7 @@ static void qobject_input_start_struct(Visitor *v, const char *name, void **obj, *obj =NULL ; } -if (!qobj && (qiv->struct_level < qiv->autocreate_struct_levels)) { +if (!qobj && (qiv->struct_level <= qiv->autocreate_struct_levels)) {/* Create a new dict that contains all the currently * unvisited items */ if (tos) {
qom/object_interfaces.c +30 -7
@@ -4,7 +4,8 @@#include "qemu/module.h" #include "qapi-visit.h" #include "qapi/qobject-output-visitor.h" -#include "qapi/opts-visitor.h" +#include "qapi/qobject-input-visitor.h" +#include "qemu/option.h" void user_creatable_complete (Object *obj,Error **errp) {
@@ -63,12 +64,16 @@ Object *user_creatable_add(const QDict *qdict,if (local_err) { goto out_visit; } -visit_check_struct (v, &local_err); + + obj =user_creatable_add_type (type, id, pdict, v, &local_err);if (local_err) { goto out_visit; } - obj =user_creatable_add_type (type, id, pdict, v, &local_err); +visit_check_struct (v, &local_err); +if (local_err) { + goto out_visit; + } out_visit:visit_end_struct (v,NULL );
@@ -114,7 +119,7 @@ Object *user_creatable_add_type(const char *type, const char *id,assert (qdict); obj =object_new (type); -visit_start_struct (v,NULL ,NULL ,0 , &local_err); +visit_start_struct (v,"props" ,NULL ,0 , &local_err);if (local_err) { goto out; }
@@ -158,14 +163,32 @@ Object *user_creatable_add_opts(QemuOpts *opts, Error **errp) {Visitor *v;QDict *pdict; +QObject *pobj;Object *obj =NULL ; - v =opts_visitor_new (opts); - pdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + pdict =qemu_opts_to_qdict (opts,NULL , +QEMU_OPTS_REPEAT_POLICY_ALL , &error_abort); - obj =user_creatable_add (pdict, v, errp); + pobj =qdict_crumple (pdict, true, errp); +if (!pobj) { + goto cleanup; + } +/* +* We need autocreate_list=true + permit_int_ranges=true +* in order to maintain compat with OptsVisitor creation +* of the 'numa' object which had an int16List property. +* +* We need autocreate_structs=1, because at the CLI level +* we have the object type + properties in the same flat +* struct, even though at the QMP level they are nested. +*/ + v =qobject_input_visitor_new_autocast (pobj, true,1 , true); + + obj =user_creatable_add (qobject_to_qdict (pobj), v, errp);visit_free (v); +qobject_decref (pobj); + cleanup:QDECREF (pdict);return obj; }
tests/check-qom-proplist.c +366 -1
466 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v14 17/21 numa: convert to use QObjectInputVisitor for -numa
In reply to Daniel P. BerrangeSwitch away from using OptsVisitor to parse the -numa argument processing. This enables use of the modern list syntax for specifying CPUs. e.g. the old syntax
-numa node,nodeid=0,cpus=0-3,cpus=8-11,mem=107
is equivalent to
-numa node,nodeid=0,cpus.0=0,cpus.1=1,cpus.2=2,cpus.3=3,\
cpus.4=8,cpus.5=9,cpus.6=10,cpus.7=11,mem=107Furthermore, the cli arg can now follow the QAPI schema nesting, so the above is equivalent to the canonical syntax:
-numa type=node,data.nodeid=0,data.cpus.0=0,data.cpus.1=1,\
data.cpus.2=2,data.cpus.3=3,data.cpus.4=8,data.cpus.5=9,\
data.cpus.6=10,data.cpus.7=11,data.mem=107A test case is added to cover argument parsing to validate that both the old and new syntax is correctly handled.
--- include/sysemu/numa_int.h | 11 +++++ numa.c | 36 +++++++++----- stubs/Makefile.objs | 5 ++ stubs/exec.c | 6 +++ stubs/hostmem.c | 14 ++++++ stubs/memory.c | 41 ++++++++++++++++ stubs/qdev.c | 8 ++++ stubs/vl.c | 8 ++++ stubs/vmstate.c | 4 ++ tests/Makefile.include | 2 + tests/test-numa.c | 116 ++++++++++++++++++++++++++++++++++++++++++++++ 11 files changed, 240 insertions(+), 11 deletions(-) create mode 100644 include/sysemu/numa_int.h create mode 100644 stubs/exec.c create mode 100644 stubs/hostmem.c create mode 100644 stubs/memory.c create mode 100644 stubs/qdev.c create mode 100644 stubs/vl.c create mode 100644 tests/test-numa.c
include/sysemu/numa_int.h new file +11 -0
@@ -0,0 +1,11 @@ +#ifndef SYSEMU_NUMA_INT_H +#define SYSEMU_NUMA_INT_H + +#include "sysemu/numa.h" + +extern int have_memdevs; +extern int max_numa_nodeid; + +int parse_numa (void *opaque,QemuOpts *opts,Error **errp); + +#endif
numa.c +25 -11
@@ -23,14 +23,14 @@*/ #include "qemu/osdep.h" -#include "sysemu/numa.h" +#include "sysemu/numa_int.h" #include "exec/cpu-common.h" #include "qemu/bitmap.h" #include "qom/cpu.h" #include "qemu/error-report.h" #include "include/exec/cpu-common.h" /* for RAM_ADDR_FMT */ #include "qapi-visit.h" -#include "qapi/opts-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "hw/boards.h" #include "sysemu/hostmem.h" #include "qmp-commands.h"
@@ -45,10 +45,10 @@ QemuOptsList qemu_numa_opts = { .desc = { { 0 } } /* validated with OptsVisitor */ }; -static int have_memdevs =-1 ; -static int max_numa_nodeid;/* Highest specified NUMA node ID, plus one. -* For all nodes, nodeid < max_numa_nodeid -*/ +int have_memdevs =-1 ; +int max_numa_nodeid;/* Highest specified NUMA node ID, plus one. +* For all nodes, nodeid < max_numa_nodeid +*/ int nb_numa_nodes;NodeInfo numa_info[MAX_NODES ];
@@ -189,6 +189,9 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)if (node->has_mem) {uint64_t mem_size = node->mem;const char *mem_str =qemu_opt_get (opts,"mem" ); +if (!mem_str) { + mem_str =qemu_opt_get (opts,"data.mem" ); + }/* Fix up legacy suffix-less format */ if (g_ascii_isdigit (mem_str[strlen (mem_str) -1 ])) { mem_size <<=20 ;
@@ -211,16 +214,27 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp) max_numa_nodeid =MAX (max_numa_nodeid, nodenr +1 ); } -static int parse_numa (void *opaque,QemuOpts *opts,Error **errp) +int parse_numa (void *opaque,QemuOpts *opts,Error **errp) {NumaOptions *object =NULL ;Error *err =NULL ; +Visitor *v; - { -Visitor *v =opts_visitor_new (opts); -visit_type_NumaOptions (v,NULL , &object, &err); -visit_free (v); +/* +* Needs autocreate_list=true, permit_int_ranges=true and +* permit_repeated_opts=true in order to support existing +* syntax for 'cpus' parameter which is an int list. +* +* Needs autocreate_struct_levels=1, because existing syntax +* used a nested struct in the QMP schema with a flat namespace +* in the CLI args. +*/ + v =qobject_input_visitor_new_opts (opts, true,1 , true, true, &err); +if (err) { + goto end; } +visit_type_NumaOptions (v,NULL , &object, &err); +visit_free (v);if (err) { goto end;
stubs/Makefile.objs +5 -0
@@ -48,3 +48,8 @@ stub-obj-y += iohandler.o stub-obj-y += smbios_type_38.o stub-obj-y += ipmi.o stub-obj-y += pc_madt_cpu_entry.o +stub-obj-y += vl.o +stub-obj-y += exec.o +stub-obj-y += memory.o +stub-obj-y += hostmem.o +stub-obj-y += qdev.o
stubs/exec.c new file +6 -0
@@ -0,0 +1,6 @@ + +#include "qemu/osdep.h" +#include "exec/cpu-common.h" +#include "qom/cpu.h" + +struct CPUTailQ cpus =QTAILQ_HEAD_INITIALIZER (cpus);
stubs/hostmem.c new file +14 -0
@@ -0,0 +1,14 @@ + +#include "qemu/osdep.h" +#include "sysemu/hostmem.h" + +void host_memory_backend_set_mapped (HostMemoryBackend *backend,bool mapped) +{ +} + + +MemoryRegion *host_memory_backend_get_memory (HostMemoryBackend *backend, +Error **errp) +{ +return NULL ; +}
stubs/memory.c new file +41 -0
41 lines not on this page · raw message
stubs/qdev.c new file +8 -0
@@ -0,0 +1,8 @@ + +#include "qemu/osdep.h" +#include "hw/qdev-core.h" + +Object *qdev_get_machine (void ) +{ +return NULL ; +}
stubs/vl.c new file +8 -0
@@ -0,0 +1,8 @@ + +#include "qemu/osdep.h" +#include "exec/cpu-common.h" + +int max_cpus =1 ; +ram_addr_t ram_size; +const char *mem_path; +int mem_prealloc;
stubs/vmstate.c +4 -0
@@ -23,3 +23,7 @@ void vmstate_unregister(DeviceState *dev, void *opaque) { } + +void vmstate_register_ram_global (struct MemoryRegion *memory) +{ +}
tests/Makefile.include +2 -0
@@ -116,6 +116,7 @@ check-unit-$(CONFIG_REPLICATION) += tests/test-replication$(EXESUF) check-unit-y += tests/test-bufferiszero$(EXESUF) gcov-files-check-bufferiszero-y = util/bufferiszero.c check-unit-y += tests/test-uuid$(EXESUF) +check-unit-y += tests/test-numa$(EXESUF) check-block-$(CONFIG_POSIX) += tests/qemu-iotests-quick.sh
@@ -584,6 +585,7 @@ tests/test-crypto-pbkdf$(EXESUF): tests/test-crypto-pbkdf.o $(test-crypto-obj-y) tests/test-crypto-ivgen$(EXESUF): tests/test-crypto-ivgen.o $(test-crypto-obj-y) tests/test-crypto-afsplit$(EXESUF): tests/test-crypto-afsplit.o $(test-crypto-obj-y) tests/test-crypto-block$(EXESUF): tests/test-crypto-block.o $(test-crypto-obj-y) +tests/test-numa$(EXESUF): tests/test-numa.o numa.o $(test-qom-obj-y) libqos-obj-y = tests/libqos/pci.o tests/libqos/fw_cfg.o tests/libqos/malloc.o libqos-obj-y += tests/libqos/i2c.o tests/libqos/libqos.o
tests/test-numa.c new file +116 -0
116 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v14 18/21 block: convert crypto driver to use QObjectInputVisitor
In reply to Daniel P. BerrangeThe crypto block driver currently uses OptsVisitor to convert from the block driver open/create options into QCryptoBlockOpenOptions/QCryptoBlockCreateOptions. This is easily replaced by use of QObjectInputVisitor with no need to enable any compatibility options, since the structs dealt with contain only scalars.
--- block/crypto.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
block/crypto.c +11 -3
@@ -23,7 +23,7 @@#include "block/block_int.h" #include "sysemu/block-backend.h" #include "crypto/block.h" -#include "qapi/opts-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "qapi-visit.h" #include "qapi/error.h"
@@ -206,7 +206,11 @@ block_crypto_open_opts_init(QCryptoBlockFormat format, ret =g_new0 (QCryptoBlockOpenOptions,1 ); ret->format = format; - v =opts_visitor_new (opts); + v =qobject_input_visitor_new_opts (opts, false,0 , false, false, + &local_err); +if (local_err) { + goto out; + }visit_start_struct (v,NULL ,NULL ,0 , &local_err);if (local_err) {
@@ -252,7 +256,11 @@ block_crypto_create_opts_init(QCryptoBlockFormat format, ret =g_new0 (QCryptoBlockCreateOptions,1 ); ret->format = format; - v =opts_visitor_new (opts); + v =qobject_input_visitor_new_opts (opts, false,0 , false, false, + &local_err); +if (local_err) { + goto out; + }visit_start_struct (v,NULL ,NULL ,0 , &local_err);if (local_err) {
Signature · 1 line
2.7.4
PATCH v14 16/21 hmp: support non-scalar properties with object_add
In reply to Daniel P. BerrangeThe current object_add HMP syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.
This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.
The QObjectInputVisitor now has functionality that allows it to replace OptsVisitor, maintaining full backwards compatibility for previous CLI syntax, while also allowing use of new syntax for structs.
Thus -object can now support non-scalar properties, for example the QMP object:
{
"execute": "object-add",
"arguments": {
"qom-type": "demo",
"id": "demo0",
"parameters": {
"foo": [
{ "bar": "one", "wizz": "1" },
{ "bar": "two", "wizz": "2" }
]
}
}
}Would be creatable via the HMP now using
object_add demo,id=demo0,\
foo.0.bar=one,foo.0.wizz=1,\
foo.1.bar=two,foo.1.wizz=2Notice that this syntax is intentionally compatible with that currently used by block drivers. NB the indentation seen here after line continuations should not be used in reality, it is just for clarity in this commit message.
--- hmp.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-)
hmp.c +17 -8
@@ -25,7 +25,7 @@#include "qemu/sockets.h" #include "monitor/monitor.h" #include "monitor/qdev.h" -#include "qapi/opts-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "qapi/qmp/qerror.h" #include "qapi/string-output-visitor.h" #include "qapi/util.h"
@@ -1696,21 +1696,30 @@ void hmp_netdev_del(Monitor *mon, const QDict *qdict)void hmp_object_add (Monitor *mon,const QDict *qdict) {Error *err =NULL ; -QemuOpts *opts;Visitor *v;Object *obj =NULL ; +QObject *pobj; - opts =qemu_opts_from_qdict (qemu_find_opts ("object" ), qdict, &err); + pobj =qdict_crumple (qdict, true, &err);if (err) { -hmp_handle_error (mon, &err); -return ; + goto cleanup; } - v =opts_visitor_new (opts); - obj =user_creatable_add (qdict, v, &err); +/* +* We need autocreate_list=true + permit_int_ranges=true +* in order to maintain compat with OptsVisitor creation +* of the 'numa' object which had an int16List property. +* +* We need autocreate_structs=1, because at the CLI level +* we have the object type + properties in the same flat +* struct, even though at the QMP level they are nested. +*/ + v =qobject_input_visitor_new_autocast (pobj, true,1 , true); + obj =user_creatable_add (qobject_to_qdict (pobj), v, &err);visit_free (v); -qemu_opts_del (opts); + cleanup: +qobject_decref (pobj);if (err) {hmp_handle_error (mon, &err); }
Signature · 1 line
2.7.4
PATCH v14 19/21 acpi: convert to QObjectInputVisitor for -acpi parsing
In reply to Daniel P. BerrangeThe -acpi command line option parsing uses the OptsVisitor currently. This is easily replaced by the QObjectInputVisitor instead. There is no need to enable any of the compatibility options, since the AcpiTableOptions QAPI struct only contains scalar properties.
--- hw/acpi/core.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
hw/acpi/core.c +7 -7
@@ -25,7 +25,7 @@#include "hw/acpi/acpi.h" #include "hw/nvram/fw_cfg.h" #include "qemu/config-file.h" -#include "qapi/opts-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "qapi-visit.h" #include "qapi-event.h"
@@ -237,14 +237,14 @@ void acpi_table_add(const QemuOpts *opts, Error **errp)char **cur;size_t bloblen =0 ;char unsigned *blob =NULL ; +Visitor *v; - { -Visitor *v; - - v =opts_visitor_new (opts); -visit_type_AcpiTableOptions (v,NULL , &hdrs, &err); -visit_free (v); + v =qobject_input_visitor_new_opts (opts, false,0 , false, false, &err); +if (err) { + goto out; } +visit_type_AcpiTableOptions (v,NULL , &hdrs, &err); +visit_free (v);if (err) { goto out;
Signature · 1 line
2.7.4
PATCH v14 20/21 net: convert to QObjectInputVisitor for -net/-netdev parsing
In reply to Daniel P. BerrangeThe -net/-netdev command line parsing code uses OptsVisitor for parsing options to populate NetLegacy or NetDev struct respectively. Although those structs have nesting, the OptsVisitor flattens them, so we must enable compatibility options to auto-create structs. This allows the legacy syntax
-net tap,id=net0,vlan=3,fd=3,script=/bin/ifup-qemu
to be treated as equivalent to the modern QAPI based syntax
-net id=net0,vlan=3,opts.type=tap,opts.data.fd=3,opts.data.script=/bin/ifup-qemu
--- net/net.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)
net/net.c +16 -2
@@ -43,7 +43,7 @@#include "qemu/iov.h" #include "qemu/main-loop.h" #include "qapi-visit.h" -#include "qapi/opts-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "sysemu/sysemu.h" #include "net/filter.h" #include "qapi/string-output-visitor.h"
@@ -1069,7 +1069,21 @@ int net_client_init(QemuOpts *opts, bool is_netdev, Error **errp)void *object =NULL ;Error *err =NULL ;int ret =-1 ; -Visitor *v =opts_visitor_new (opts); +/* +* Needs autocreate_lists=true in order support existing +* syntax for list options where the bare key is repeated +* +* Needs autocreate_struct_levels=3 in order to deal with +* 3 level nesting in NetLegacy option args, which was +* exposed as a flat namespace with OptVisitor +*/ +Visitor *v =qobject_input_visitor_new_opts (opts, true,3 , false, true, + &err); + +if (err) { +error_propagate (errp, err); +return -1 ; + } {/* Parse convenience option format ip6-net=fec0::0[/64] */
Signature · 1 line
2.7.4
PATCH v14 21/21 qapi: delete unused OptsVisitor code
In reply to Daniel P. BerrangeNow that all code has been converted to QObjectInputVisitor's QemuOpts compatibility mode, there is no longer any reason to keep OptsVisitor alive.
--- include/qapi/opts-visitor.h | 40 ---- qapi/Makefile.objs | 2 +- qapi/opts-visitor.c | 544 -------------------------------------------- tests/Makefile.include | 5 +- tests/test-opts-visitor.c | 268 ---------------------- vl.c | 1 - 6 files changed, 2 insertions(+), 858 deletions(-) delete mode 100644 include/qapi/opts-visitor.h delete mode 100644 qapi/opts-visitor.c delete mode 100644 tests/test-opts-visitor.c
include/qapi/opts-visitor.h deleted +0 -40
@@ -1,40 +0,0 @@ -/* -* Options Visitor -* -* Copyright Red Hat, Inc. 2012 -* -* Author: Laszlo Ersek <[email protected]> -* -* This work is licensed under the terms of the GNU LGPL, version 2.1 or later. -* See the COPYING.LIB file in the top-level directory. -* -*/ - -#ifndef OPTS_VISITOR_H -#define OPTS_VISITOR_H - -#include "qapi/visitor.h" -#include "qemu/option.h" - -/* Inclusive upper bound on the size of any flattened range. This is a safety -* (= anti-annoyance) measure; wrong ranges should not cause long startup -* delays nor exhaust virtual memory. -*/ -#define OPTS_VISITOR_RANGE_MAX 65536 - -typedef struct OptsVisitor OptsVisitor ; - -/* Contrarily to qemu-option.c::parse_option_number(), OptsVisitor's "int" -* parser relies on strtoll() instead of strtoull(). Consequences: -* - string representations of negative numbers yield negative values, -* - values below INT64_MIN or LLONG_MIN are rejected, -* - values above INT64_MAX or LLONG_MAX are rejected. -* -* The Opts input visitor does not implement support for visiting QAPI -* alternates, numbers (other than integers), null, or arbitrary -* QTypes. It also requires a non-null list argument to -* visit_start_list(). -*/ -Visitor *opts_visitor_new (const QemuOpts *opts); - -#endif
qapi/Makefile.objs +1 -1
@@ -1,6 +1,6 @@ util-obj-y = qapi-visit-core.o qapi-dealloc-visitor.o qobject-input-visitor.o util-obj-y += qobject-output-visitor.o qmp-registry.o qmp-dispatch.o util-obj-y += string-input-visitor.o string-output-visitor.o -util-obj-y += opts-visitor.o qapi-clone-visitor.o +util-obj-y += qapi-clone-visitor.o util-obj-y += qmp-event.o util-obj-y += qapi-util.o
qapi/opts-visitor.c deleted +0 -544
544 lines not on this page · raw message
tests/Makefile.include +1 -4
@@ -35,8 +35,6 @@ check-unit-y += tests/test-string-output-visitor$(EXESUF) gcov-files-test-string-output-visitor-y = qapi/string-output-visitor.c check-unit-y += tests/test-qmp-event$(EXESUF) gcov-files-test-qmp-event-y += qapi/qmp-event.c -check-unit-y += tests/test-opts-visitor$(EXESUF) -gcov-files-test-opts-visitor-y = qapi/opts-visitor.c check-unit-y += tests/test-coroutine$(EXESUF) gcov-files-test-coroutine-y = coroutine-$(CONFIG_COROUTINE_BACKEND).c check-unit-y += tests/test-visitor-serialization$(EXESUF)
@@ -445,7 +443,7 @@ test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \ tests/test-qobject-input-visitor.o tests/test-qobject-input-strict.o \ tests/test-qmp-commands.o tests/test-visitor-serialization.o \ tests/test-x86-cpuid.o tests/test-mul64.o tests/test-int128.o \ - tests/test-opts-visitor.o tests/test-qmp-event.o \ + tests/test-qmp-event.o \ tests/rcutorture.o tests/test-rcu-list.o \ tests/test-qdist.o \ tests/test-qht.o tests/qht-bench.o tests/test-qht-par.o
@@ -549,7 +547,6 @@ tests/test-qobject-input-visitor$(EXESUF): tests/test-qobject-input-visitor.o $( tests/test-qobject-input-strict$(EXESUF): tests/test-qobject-input-strict.o $(test-qapi-obj-y) tests/test-qmp-commands$(EXESUF): tests/test-qmp-commands.o tests/test-qmp-marshal.o $(test-qapi-obj-y) tests/test-visitor-serialization$(EXESUF): tests/test-visitor-serialization.o $(test-qapi-obj-y) -tests/test-opts-visitor$(EXESUF): tests/test-opts-visitor.o $(test-qapi-obj-y) tests/test-mul64$(EXESUF): tests/test-mul64.o $(test-util-obj-y) tests/test-bitops$(EXESUF): tests/test-bitops.o $(test-util-obj-y)
tests/test-opts-visitor.c deleted +0 -268
268 lines not on this page · raw message
vl.c +0 -1
@@ -115,7 +115,6 @@ int main(int argc, char **argv)#include "ui/qemu-spice.h" #include "qapi/string-input-visitor.h" -#include "qapi/opts-visitor.h" #include "qom/object_interfaces.h" #include "qapi-event.h" #include "exec/semihost.h"
Signature · 1 line
2.7.4
Your series failed automatic build test. Please find the testing commands and their output below. If you have docker installed, you can probably reproduce it locally.
Hi,
Your series failed automatic build test. Please find the testing commands and their output below. If you have docker installed, you can probably reproduce it locally.
Type: series
Message-id: [email protected]
Subject: [Qemu-devel] [PATCH v14 00/21] QAPI/QOM work for non-scalar object properties
=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
make J=8 docker-test-quick@centos6
make J=8 docker-test-mingw@fedora
=== TEST SCRIPT END ===
Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
- [tag update] patchew/[email protected] -> patchew/[email protected]
Switched to a new branch 'test'
9b89b2d qapi: delete unused OptsVisitor code
7d7d027 net: convert to QObjectInputVisitor for -net/-netdev parsing f7fd6c1 acpi: convert to QObjectInputVisitor for -acpi parsing fecaabf block: convert crypto driver to use QObjectInputVisitor 7941b3f numa: convert to use QObjectInputVisitor for -numa 3c6762d hmp: support non-scalar properties with object_add efb01de qom: support non-scalar properties with -object 3c90b78 qapi: allow repeated opts with qobject_input_visitor_new_opts 81f728f qdict: allow qdict_crumple to accept compound types as values 8353c22 option: allow qemu_opts_to_qdict to merge repeated options 0948fcd qapi: add integer range support for QObjectInputVisitor 6da7117 qapi: permit auto-creating nested structs
1170f4b qapi: permit auto-creating single element lists 2ec3eac qapi: allow QObjectInputVisitor to be created with QemuOpts b56363f qapi: permit scalar type conversions in QObjectInputVisitor 6fa7850 qapi: don't pass two copies of TestInputVisitorData to tests d1b24d3 qapi: rename QmpOutputVisitor to QObjectOutputVisitor 794fec6 qapi: rename QmpInputVisitor to QObjectInputVisitor 43b5ede qapi: add trace events for visitor
587d218 qdict: implement a qdict_crumple method for un-flattening a dict 67cbb95 option: make parse_option_bool/number non-static
=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc' Cloning into 'dtc'...
Submodule path 'dtc': checked out '65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf'
BUILD centos6
ARCHIVE qemu.tgz
ARCHIVE dtc.tgz
COPY RUNNER
RUN test-quick in centos6
Packages installed:
SDL-devel-1.2.14-7.el6_7.1.x86_64
ccache-3.1.6-2.el6.x86_64
epel-release-6-8.noarch
gcc-4.4.7-17.el6.x86_64
git-1.7.1-4.el6_7.1.x86_64
glib2-devel-2.28.8-5.el6.x86_64
libfdt-devel-1.4.0-1.el6.x86_64
make-3.81-23.el6.x86_64
package g++ is not installed
pixman-devel-0.32.8-1.el6.x86_64
tar-1.23-15.el6_8.x86_64
zlib-devel-1.2.3-29.el6.x86_64
Environment variables:
PACKAGES=libfdt-devel ccache tar git make gcc g++ zlib-devel glib2-devel SDL-devel pixman-devel epel-release
HOSTNAME=9320e4514cd6
TERM=xterm
MAKEFLAGS= -j8
HISTSIZE=1000
J=8
USER=root
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
MAIL=/var/spool/mail/root PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PWD=/
LANG=en_US.UTF-8
TARGET_LIST=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
FEATURES= dtc
DEBUG=
G_BROKEN_FILENAMES=1
CCACHE_HASHDIR=
_=/usr/bin/env
Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/src/tests/docker/install No C++ compiler available; disabling C++ specific optional code
Install prefix /tmp/qemu-test/src/tests/docker/install BIOS directory /tmp/qemu-test/src/tests/docker/install/share/qemu binary directory /tmp/qemu-test/src/tests/docker/install/bin
library directory /tmp/qemu-test/src/tests/docker/install/lib module directory /tmp/qemu-test/src/tests/docker/install/lib/qemu libexec directory /tmp/qemu-test/src/tests/docker/install/libexec include directory /tmp/qemu-test/src/tests/docker/install/include config directory /tmp/qemu-test/src/tests/docker/install/etc
local state directory /tmp/qemu-test/src/tests/docker/install/var
Manual directory /tmp/qemu-test/src/tests/docker/install/share/man ELF interp prefix /usr/gnemul/qemu-%M
Source path /tmp/qemu-test/src C compiler cc Host C compiler cc C++ compiler Objective-C compiler cc ARFLAGS rv CFLAGS -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g QEMU_CFLAGS -I/usr/include/pixman-1 -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all LDFLAGS -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g make make install install python python -B smbd /usr/sbin/smbd module support no host CPU x86_64 host big endian no target list x86_64-softmmu aarch64-softmmu tcg debug enabled no gprof enabled no sparse enabled no strip binaries yes profiler no static build no pixman system SDL support yes (1.2.14) GTK support no GTK GL support no VTE support no TLS priority NORMAL GNUTLS support no GNUTLS rnd no libgcrypt no libgcrypt kdf no nettle no nettle kdf no libtasn1 no curses support no virgl support no curl support no mingw32 support no Audio drivers oss Block whitelist (rw) Block whitelist (ro) VirtFS support no VNC support yes VNC SASL support no VNC JPEG support no VNC PNG support no xen support no brlapi support no bluez support no Documentation no PIE yes vde support no netmap support no Linux AIO support no ATTR/XATTR support yes Install blobs yes KVM support yes RDMA support no TCG interpreter no fdt support yes preadv support yes fdatasync yes madvise yes posix_madvise yes
libcap-ng support no
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes
Trace backends log spice support no rbd support no xfsctl support no smartcard support no libusb no usb net redir no OpenGL support no OpenGL dmabufs no libiscsi support no libnfs support no
build guest agent yes
QGA VSS support no
QGA w32 disk info no
QGA MSI support no
seccomp support no
coroutine backend ucontext
coroutine pool yes
GlusterFS support no
Archipelago support no
gcov gcov gcov enabled no TPM support yes libssh2 support no TPM passthrough yes QOM debugging yes lzo support no snappy support no bzip2 support no
NUMA host support no
tcmalloc support no
jemalloc support no
avx2 optimization no
replication support yes
GEN x86_64-softmmu/config-devices.mak.tmp GEN aarch64-softmmu/config-devices.mak.tmp GEN config-host.h GEN qemu-options.def GEN qmp-commands.h GEN qapi-types.h GEN qapi-visit.h GEN qapi-event.h GEN x86_64-softmmu/config-devices.mak GEN aarch64-softmmu/config-devices.mak GEN qmp-introspect.h GEN module_block.h GEN tests/test-qapi-types.h GEN tests/test-qapi-visit.h GEN tests/test-qmp-commands.h GEN tests/test-qapi-event.h GEN tests/test-qmp-introspect.h GEN config-all-devices.mak GEN trace/generated-events.h GEN trace/generated-tracers.h GEN trace/generated-tcg-tracers.h GEN trace/generated-helpers-wrappers.h GEN trace/generated-helpers.h CC tests/qemu-iotests/socket_scm_helper.o GEN qga/qapi-generated/qga-qapi-types.h GEN qga/qapi-generated/qga-qapi-visit.h GEN qga/qapi-generated/qga-qmp-commands.h GEN qga/qapi-generated/qga-qapi-types.c GEN qga/qapi-generated/qga-qapi-visit.c GEN qga/qapi-generated/qga-qmp-marshal.c GEN qmp-introspect.c GEN qapi-types.c GEN qapi-visit.c GEN qapi-event.c CC qapi/qapi-visit-core.o CC qapi/qapi-dealloc-visitor.o CC qapi/qobject-input-visitor.o CC qapi/qobject-output-visitor.o CC qapi/qmp-registry.o CC qapi/qmp-dispatch.o CC qapi/string-input-visitor.o CC qapi/string-output-visitor.o /tmp/qemu-test/src/qapi/qobject-input-visitor.c: In function ‘qobject_input_get_object’: /tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: warning: implicit declaration of function ‘g_hash_table_contains’ /tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: warning: nested extern declaration of ‘g_hash_table_contains’ CC qapi/qapi-clone-visitor.o CC qapi/qmp-event.o CC qapi/qapi-util.o CC qobject/qnull.o CC qobject/qint.o CC qobject/qstring.o CC qobject/qdict.o CC qobject/qlist.o CC qobject/qfloat.o CC qobject/qbool.o CC qobject/qjson.o CC qobject/qobject.o CC qobject/json-lexer.o CC qobject/json-streamer.o CC qobject/json-parser.o GEN trace/generated-events.c CC trace/control.o CC trace/qmp.o CC util/osdep.o CC util/cutils.o CC util/unicode.o CC util/qemu-timer-common.o CC util/bufferiszero.o CC util/compatfd.o CC util/event_notifier-posix.o CC util/mmap-alloc.o CC util/oslib-posix.o CC util/qemu-openpty.o CC util/qemu-thread-posix.o CC util/memfd.o CC util/path.o CC util/envlist.o CC util/module.o CC util/bitmap.o CC util/bitops.o CC util/hbitmap.o CC util/fifo8.o CC util/acl.o CC util/error.o CC util/qemu-error.o CC util/id.o CC util/iov.o CC util/qemu-config.o CC util/qemu-sockets.o CC util/uri.o CC util/notify.o CC util/qemu-option.o CC util/qemu-progress.o CC util/hexdump.o CC util/crc32c.o CC util/uuid.o CC util/throttle.o CC util/getauxval.o CC util/readline.o CC util/rfifolock.o CC util/rcu.o CC util/qemu-coroutine.o CC util/qemu-coroutine-lock.o CC util/qemu-coroutine-io.o CC util/qemu-coroutine-sleep.o CC util/coroutine-ucontext.o CC util/buffer.o CC util/timed-average.o CC util/base64.o CC util/log.o CC util/qdist.o CC util/qht.o CC util/range.o CC crypto/pbkdf-stub.o CC stubs/arch-query-cpu-def.o CC stubs/arch-query-cpu-model-expansion.o CC stubs/arch-query-cpu-model-comparison.o CC stubs/arch-query-cpu-model-baseline.o CC stubs/bdrv-next-monitor-owned.o CC stubs/blk-commit-all.o CC stubs/blockdev-close-all-bdrv-states.o CC stubs/clock-warp.o /tmp/qemu-test/src/util/qht.c: In function ‘qht_reset_size’: /tmp/qemu-test/src/util/qht.c:413: warning: ‘new’ may be used uninitialized in this function CC stubs/cpu-get-clock.o CC stubs/cpu-get-icount.o CC stubs/dump.o CC stubs/fdset-add-fd.o CC stubs/fdset-find-fd.o CC stubs/fdset-get-fd.o CC stubs/fdset-remove-fd.o CC stubs/gdbstub.o CC stubs/get-fd.o CC stubs/get-next-serial.o CC stubs/get-vm-name.o CC stubs/iothread-lock.o CC stubs/is-daemonized.o CC stubs/machine-init-done.o CC stubs/migr-blocker.o CC stubs/mon-is-qmp.o CC stubs/mon-printf.o CC stubs/monitor-init.o CC stubs/notify-event.o CC stubs/qtest.o CC stubs/replay.o CC stubs/replay-user.o CC stubs/reset.o CC stubs/runstate-check.o CC stubs/set-fd-handler.o CC stubs/slirp.o CC stubs/sysbus.o CC stubs/trace-control.o CC stubs/uuid.o CC stubs/vm-stop.o CC stubs/vmstate.o CC stubs/cpus.o CC stubs/kvm.o CC stubs/qmp_pc_dimm_device_list.o CC stubs/target-monitor-defs.o CC stubs/target-get-monitor-def.o CC stubs/vhost.o CC stubs/iohandler.o CC stubs/smbios_type_38.o CC stubs/ipmi.o CC stubs/pc_madt_cpu_entry.o CC stubs/vl.o CC stubs/exec.o CC stubs/memory.o CC stubs/hostmem.o CC stubs/qdev.o CC contrib/ivshmem-client/ivshmem-client.o CC contrib/ivshmem-client/main.o CC contrib/ivshmem-server/ivshmem-server.o CC contrib/ivshmem-server/main.o CC qemu-nbd.o CC async.o CC thread-pool.o CC block.o CC blockjob.o CC main-loop.o CC iohandler.o CC qemu-timer.o CC aio-posix.o CC qemu-io-cmds.o CC block/raw_bsd.o CC replication.o CC block/qcow.o CC block/vdi.o CC block/vmdk.o CC block/cloop.o CC block/bochs.o CC block/vpc.o CC block/vvfat.o CC block/dmg.o CC block/qcow2.o CC block/qcow2-refcount.o CC block/qcow2-cluster.o CC block/qcow2-snapshot.o CC block/qcow2-cache.o CC block/qed.o CC block/qed-gencb.o CC block/qed-l2-cache.o CC block/qed-table.o CC block/qed-cluster.o CC block/qed-check.o CC block/vhdx.o CC block/vhdx-endian.o CC block/vhdx-log.o CC block/quorum.o CC block/parallels.o CC block/blkdebug.o CC block/blkverify.o CC block/blkreplay.o CC block/block-backend.o CC block/snapshot.o CC block/qapi.o CC block/raw-posix.o CC block/null.o CC block/mirror.o CC block/commit.o CC block/io.o CC block/throttle-groups.o CC block/nbd.o CC block/nbd-client.o CC block/sheepdog.o CC block/accounting.o CC block/dirty-bitmap.o CC block/write-threshold.o CC block/backup.o CC block/replication.o CC block/crypto.o CC nbd/server.o CC nbd/client.o CC nbd/common.o CC crypto/init.o CC crypto/hash.o CC crypto/hash-glib.o CC crypto/aes.o CC crypto/desrfb.o CC crypto/cipher.o CC crypto/tlscreds.o CC crypto/tlscredsanon.o CC crypto/tlscredsx509.o CC crypto/tlssession.o CC crypto/secret.o CC crypto/random-platform.o CC crypto/pbkdf.o CC crypto/ivgen.o CC crypto/ivgen-essiv.o CC crypto/ivgen-plain.o CC crypto/ivgen-plain64.o CC crypto/afsplit.o CC crypto/xts.o CC crypto/block.o CC crypto/block-qcow.o CC crypto/block-luks.o CC io/channel.o CC io/channel-buffer.o CC io/channel-command.o CC io/channel-file.o CC io/channel-socket.o CC io/channel-tls.o CC io/channel-watch.o CC io/channel-websock.o CC io/channel-util.o CC io/task.o CC qom/object.o CC qom/container.o CC qom/qom-qobject.o CC qom/object_interfaces.o GEN qemu-img-cmds.h CC qemu-io.o CC qemu-bridge-helper.o CC blockdev.o CC blockdev-nbd.o CC iothread.o CC qdev-monitor.o CC device-hotplug.o CC os-posix.o CC qemu-char.o CC page_cache.o CC accel.o CC bt-host.o CC bt-vhci.o CC dma-helpers.o CC vl.o CC tpm.o CC device_tree.o GEN qmp-marshal.c CC qmp.o CC hmp.o CC tcg-runtime.o CC cpus-common.o CC audio/audio.o CC audio/noaudio.o CC audio/wavaudio.o CC audio/mixeng.o CC audio/sdlaudio.o CC audio/ossaudio.o CC audio/wavcapture.o CC backends/rng.o CC backends/rng-egd.o CC backends/rng-random.o CC backends/msmouse.o CC backends/testdev.o CC backends/tpm.o CC backends/hostmem.o CC backends/hostmem-ram.o CC backends/hostmem-file.o CC block/stream.o CC disas/arm.o CC disas/i386.o CC fsdev/qemu-fsdev-dummy.o CC fsdev/qemu-fsdev-opts.o CC hw/acpi/core.o CC hw/acpi/piix4.o CC hw/acpi/pcihp.o CC hw/acpi/ich9.o CC hw/acpi/tco.o CC hw/acpi/cpu_hotplug.o CC hw/acpi/memory_hotplug.o CC hw/acpi/memory_hotplug_acpi_table.o CC hw/acpi/cpu.o CC hw/acpi/acpi_interface.o CC hw/acpi/bios-linker-loader.o CC hw/acpi/aml-build.o CC hw/acpi/ipmi.o CC hw/audio/sb16.o CC hw/audio/es1370.o CC hw/audio/ac97.o CC hw/audio/fmopl.o CC hw/audio/adlib.o CC hw/audio/gus.o CC hw/audio/gusemu_hal.o CC hw/audio/gusemu_mixer.o CC hw/audio/cs4231a.o CC hw/audio/intel-hda.o CC hw/audio/hda-codec.o CC hw/audio/pcspk.o CC hw/audio/wm8750.o CC hw/audio/pl041.o CC hw/audio/lm4549.o CC hw/audio/marvell_88w8618.o CC hw/block/block.o CC hw/block/cdrom.o CC hw/block/hd-geometry.o CC hw/block/fdc.o CC hw/block/m25p80.o CC hw/block/nand.o CC hw/block/pflash_cfi01.o CC hw/block/pflash_cfi02.o CC hw/block/ecc.o CC hw/block/onenand.o CC hw/block/nvme.o CC hw/bt/core.o CC hw/bt/l2cap.o CC hw/bt/sdp.o CC hw/bt/hci.o CC hw/bt/hid.o CC hw/bt/hci-csr.o CC hw/char/ipoctal232.o CC hw/char/parallel.o CC hw/char/pl011.o CC hw/char/serial.o CC hw/char/serial-isa.o CC hw/char/serial-pci.o CC hw/char/virtio-console.o CC hw/char/cadence_uart.o CC hw/char/debugcon.o CC hw/char/imx_serial.o CC hw/core/qdev.o CC hw/core/qdev-properties.o CC hw/core/bus.o CC hw/core/fw-path-provider.o CC hw/core/irq.o CC hw/core/hotplug.o CC hw/core/ptimer.o CC hw/core/sysbus.o CC hw/core/machine.o CC hw/core/null-machine.o CC hw/core/loader.o CC hw/core/qdev-properties-system.o CC hw/core/register.o CC hw/core/platform-bus.o CC hw/display/ads7846.o CC hw/display/cirrus_vga.o CC hw/display/pl110.o CC hw/display/ssd0303.o CC hw/display/ssd0323.o CC hw/display/vga-pci.o CC hw/display/vga-isa.o CC hw/display/vmware_vga.o CC hw/display/blizzard.o CC hw/display/exynos4210_fimd.o CC hw/display/framebuffer.o CC hw/display/tc6393xb.o CC hw/dma/pl080.o CC hw/dma/pl330.o CC hw/dma/i8257.o CC hw/dma/xlnx-zynq-devcfg.o CC hw/gpio/max7310.o CC hw/gpio/pl061.o CC hw/gpio/zaurus.o CC hw/gpio/gpio_key.o CC hw/i2c/core.o CC hw/i2c/smbus.o CC hw/i2c/smbus_eeprom.o CC hw/i2c/i2c-ddc.o CC hw/i2c/versatile_i2c.o CC hw/i2c/smbus_ich9.o CC hw/i2c/pm_smbus.o CC hw/i2c/bitbang_i2c.o CC hw/i2c/exynos4210_i2c.o CC hw/i2c/imx_i2c.o CC hw/i2c/aspeed_i2c.o CC hw/ide/core.o CC hw/ide/atapi.o CC hw/ide/qdev.o CC hw/ide/pci.o CC hw/ide/isa.o CC hw/ide/piix.o CC hw/ide/microdrive.o CC hw/ide/ahci.o CC hw/ide/ich.o CC hw/input/hid.o CC hw/input/lm832x.o CC hw/input/pckbd.o CC hw/input/pl050.o CC hw/input/ps2.o CC hw/input/stellaris_input.o CC hw/input/tsc2005.o CC hw/input/vmmouse.o CC hw/input/virtio-input.o CC hw/input/virtio-input-hid.o CC hw/input/virtio-input-host.o CC hw/intc/i8259_common.o CC hw/intc/i8259.o CC hw/intc/pl190.o CC hw/intc/imx_avic.o CC hw/intc/realview_gic.o CC hw/intc/ioapic_common.o CC hw/intc/arm_gic_common.o CC hw/intc/arm_gic.o CC hw/intc/arm_gicv2m.o CC hw/intc/arm_gicv3_common.o CC hw/intc/arm_gicv3.o CC hw/intc/arm_gicv3_dist.o CC hw/intc/arm_gicv3_redist.o CC hw/ipack/ipack.o CC hw/ipack/tpci200.o CC hw/ipmi/ipmi.o CC hw/ipmi/ipmi_bmc_sim.o CC hw/ipmi/ipmi_bmc_extern.o CC hw/ipmi/isa_ipmi_kcs.o CC hw/ipmi/isa_ipmi_bt.o CC hw/isa/isa-bus.o CC hw/isa/apm.o CC hw/mem/pc-dimm.o CC hw/mem/nvdimm.o CC hw/misc/applesmc.o CC hw/misc/max111x.o CC hw/misc/tmp105.o CC hw/misc/debugexit.o CC hw/misc/sga.o CC hw/misc/pc-testdev.o CC hw/misc/pci-testdev.o CC hw/misc/arm_l2x0.o CC hw/misc/arm_integrator_debug.o CC hw/misc/a9scu.o CC hw/misc/arm11scu.o CC hw/net/ne2000.o CC hw/net/eepro100.o CC hw/net/pcnet-pci.o CC hw/net/pcnet.o CC hw/net/e1000.o CC hw/net/e1000x_common.o CC hw/net/net_tx_pkt.o CC hw/net/net_rx_pkt.o CC hw/net/e1000e.o CC hw/net/e1000e_core.o CC hw/net/rtl8139.o CC hw/net/vmxnet3.o CC hw/net/smc91c111.o CC hw/net/lan9118.o CC hw/net/ne2000-isa.o CC hw/net/xgmac.o CC hw/net/allwinner_emac.o CC hw/net/imx_fec.o CC hw/net/cadence_gem.o CC hw/net/stellaris_enet.o CC hw/net/rocker/rocker.o CC hw/net/rocker/rocker_fp.o CC hw/net/rocker/rocker_desc.o CC hw/net/rocker/rocker_world.o CC hw/net/rocker/rocker_of_dpa.o CC hw/nvram/eeprom93xx.o CC hw/nvram/fw_cfg.o CC hw/pci-bridge/pci_bridge_dev.o CC hw/pci-bridge/pci_expander_bridge.o CC hw/pci-bridge/xio3130_upstream.o CC hw/pci-bridge/xio3130_downstream.o CC hw/pci-bridge/ioh3420.o CC hw/pci-bridge/i82801b11.o CC hw/pci-host/pam.o CC hw/pci-host/versatile.o CC hw/pci-host/piix.o CC hw/pci-host/q35.o CC hw/pci-host/gpex.o CC hw/pci/pci.o CC hw/pci/pci_bridge.o CC hw/pci/msix.o CC hw/pci/msi.o /tmp/qemu-test/src/hw/nvram/fw_cfg.c: In function ‘fw_cfg_dma_transfer’: /tmp/qemu-test/src/hw/nvram/fw_cfg.c:330: warning: ‘read’ may be used uninitialized in this function CC hw/pci/shpc.o CC hw/pci/slotid_cap.o CC hw/pci/pci_host.o CC hw/pci/pcie_host.o CC hw/pci/pcie.o CC hw/pci/pcie_aer.o CC hw/pci/pcie_port.o CC hw/pci/pci-stub.o CC hw/pcmcia/pcmcia.o CC hw/scsi/scsi-disk.o CC hw/scsi/scsi-generic.o CC hw/scsi/scsi-bus.o CC hw/scsi/lsi53c895a.o CC hw/scsi/mptsas.o CC hw/scsi/mptconfig.o CC hw/scsi/mptendian.o CC hw/scsi/megasas.o CC hw/scsi/vmw_pvscsi.o CC hw/scsi/esp.o CC hw/scsi/esp-pci.o CC hw/sd/pl181.o CC hw/sd/ssi-sd.o CC hw/sd/sd.o CC hw/sd/core.o CC hw/sd/sdhci.o CC hw/smbios/smbios.o CC hw/smbios/smbios_type_38.o CC hw/ssi/pl022.o CC hw/ssi/ssi.o CC hw/ssi/xilinx_spips.o CC hw/ssi/aspeed_smc.o CC hw/timer/arm_timer.o CC hw/timer/arm_mptimer.o CC hw/timer/a9gtimer.o CC hw/timer/cadence_ttc.o CC hw/timer/ds1338.o CC hw/timer/hpet.o CC hw/timer/i8254_common.o CC hw/timer/i8254.o CC hw/timer/pl031.o CC hw/timer/twl92230.o CC hw/timer/imx_epit.o CC hw/timer/imx_gpt.o CC hw/timer/stm32f2xx_timer.o CC hw/timer/aspeed_timer.o CC hw/tpm/tpm_tis.o CC hw/tpm/tpm_passthrough.o CC hw/tpm/tpm_util.o CC hw/usb/core.o CC hw/usb/combined-packet.o CC hw/usb/bus.o CC hw/usb/libhw.o CC hw/usb/desc.o CC hw/usb/desc-msos.o CC hw/usb/hcd-uhci.o CC hw/usb/hcd-ohci.o CC hw/usb/hcd-ehci.o CC hw/usb/hcd-ehci-pci.o CC hw/usb/hcd-xhci.o CC hw/usb/hcd-ehci-sysbus.o CC hw/usb/hcd-musb.o CC hw/usb/dev-hub.o CC hw/usb/dev-hid.o CC hw/usb/dev-wacom.o CC hw/usb/dev-storage.o CC hw/usb/dev-uas.o CC hw/usb/dev-audio.o CC hw/usb/dev-serial.o CC hw/usb/dev-network.o CC hw/usb/dev-bluetooth.o CC hw/usb/dev-smartcard-reader.o CC hw/usb/dev-mtp.o CC hw/usb/host-stub.o CC hw/virtio/virtio-rng.o CC hw/virtio/virtio-pci.o CC hw/virtio/virtio-bus.o CC hw/virtio/virtio-mmio.o CC hw/watchdog/watchdog.o CC hw/watchdog/wdt_i6300esb.o CC hw/watchdog/wdt_ib700.o CC migration/migration.o CC migration/socket.o CC migration/fd.o CC migration/exec.o CC migration/tls.o CC migration/vmstate.o CC migration/qemu-file.o CC migration/qemu-file-channel.o CC migration/xbzrle.o CC migration/postcopy-ram.o CC migration/qjson.o CC migration/block.o CC net/net.o CC net/queue.o CC net/checksum.o CC net/util.o CC net/hub.o CC net/socket.o CC net/dump.o CC net/eth.o CC net/l2tpv3.o CC net/tap.o CC net/vhost-user.o CC net/tap-linux.o CC net/slirp.o CC net/filter.o CC net/filter-buffer.o CC net/filter-mirror.o CC net/colo-compare.o CC net/colo.o CC net/filter-rewriter.o CC qom/cpu.o CC replay/replay.o CC replay/replay-internal.o CC replay/replay-events.o CC replay/replay-time.o CC replay/replay-input.o CC replay/replay-char.o CC replay/replay-snapshot.o /tmp/qemu-test/src/replay/replay-internal.c: In function ‘replay_put_array’: /tmp/qemu-test/src/replay/replay-internal.c:65: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result CC slirp/cksum.o CC slirp/if.o CC slirp/ip_icmp.o CC slirp/ip6_icmp.o CC slirp/ip6_input.o CC slirp/ip6_output.o CC slirp/ip_input.o CC slirp/ip_output.o CC slirp/dnssearch.o CC slirp/dhcpv6.o CC slirp/slirp.o CC slirp/mbuf.o CC slirp/misc.o CC slirp/sbuf.o CC slirp/socket.o CC slirp/tcp_input.o CC slirp/tcp_output.o CC slirp/tcp_subr.o CC slirp/tcp_timer.o CC slirp/udp.o CC slirp/udp6.o CC slirp/bootp.o CC slirp/tftp.o CC slirp/arp_table.o
/tmp/qemu-test/src/slirp/tcp_input.c: In function ‘tcp_input’:/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_p’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_len’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_tos’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_id’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_off’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_ttl’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_sum’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_src.s_addr’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_dst.s_addr’ may be used uninitialized in this function /tmp/qemu-test/src/slirp/tcp_input.c:220: warning: ‘save_ip6.ip_nh’ may be used uninitialized in this function
CC slirp/ndp_table.o CC ui/keymaps.o CC ui/console.o CC ui/cursor.o CC ui/qemu-pixman.o CC ui/input.o CC ui/input-keymap.o CC ui/input-legacy.o CC ui/input-linux.o CC ui/sdl.o CC ui/sdl_zoom.o CC ui/x_keymap.o CC ui/vnc.o CC ui/vnc-enc-zlib.o CC ui/vnc-enc-hextile.o CC ui/vnc-enc-tight.o CC ui/vnc-palette.o CC ui/vnc-enc-zrle.o CC ui/vnc-auth-vencrypt.o CC ui/vnc-ws.o CC ui/vnc-jobs.o AS optionrom/multiboot.o AS optionrom/linuxboot.o CC optionrom/linuxboot_dma.o
cc: unrecognized option '-no-integrated-as'
cc: unrecognized option '-no-integrated-as'
AS optionrom/kvmvapic.o
Building optionrom/multiboot.img
Building optionrom/linuxboot.img
Building optionrom/linuxboot_dma.img
Building optionrom/kvmvapic.img
Building optionrom/multiboot.raw
Building optionrom/linuxboot.raw
LINK tests/qemu-iotests/socket_scm_helper
Building optionrom/linuxboot_dma.raw
Building optionrom/kvmvapic.raw
Signing optionrom/multiboot.bin
Signing optionrom/linuxboot.bin
Signing optionrom/linuxboot_dma.bin
CC qga/commands.o
Signing optionrom/kvmvapic.bin
CC qga/guest-agent-command-state.o CC qga/main.o CC qga/commands-posix.o CC qga/channel-posix.o CC qga/qapi-generated/qga-qapi-types.o CC qga/qapi-generated/qga-qapi-visit.o CC qga/qapi-generated/qga-qmp-marshal.o CC qmp-introspect.o CC qapi-types.o CC qapi-visit.o CC qapi-event.o AR libqemustub.a CC qemu-img.o CC qmp-marshal.o CC trace/generated-events.o AR libqemuutil.a LINK qemu-ga LINK ivshmem-client LINK ivshmem-server LINK qemu-nbd LINK qemu-img LINK qemu-io LINK qemu-bridge-helper
libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make: *** [qemu-ga] Error 1
make: *** Waiting for unfinished jobs....libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make: *** [qemu-nbd] Error 1libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make: *** [qemu-img] Error 1libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make: *** [qemu-io] Error 1
GEN x86_64-softmmu/hmp-commands.h GEN x86_64-softmmu/hmp-commands-info.h GEN x86_64-softmmu/config-target.h GEN aarch64-softmmu/hmp-commands.h GEN aarch64-softmmu/hmp-commands-info.h GEN aarch64-softmmu/config-target.h CC x86_64-softmmu/exec.o CC x86_64-softmmu/translate-all.o CC x86_64-softmmu/cpu-exec.o CC x86_64-softmmu/translate-common.o CC x86_64-softmmu/cpu-exec-common.o CC x86_64-softmmu/tcg/tcg.o CC x86_64-softmmu/tcg/tcg-op.o CC aarch64-softmmu/exec.o CC x86_64-softmmu/tcg/optimize.o CC x86_64-softmmu/tcg/tcg-common.o CC x86_64-softmmu/fpu/softfloat.o CC x86_64-softmmu/disas.o CC x86_64-softmmu/arch_init.o CC x86_64-softmmu/cpus.o CC x86_64-softmmu/monitor.o CC aarch64-softmmu/translate-all.o CC aarch64-softmmu/cpu-exec.o CC x86_64-softmmu/gdbstub.o CC aarch64-softmmu/translate-common.o CC x86_64-softmmu/balloon.o CC x86_64-softmmu/ioport.o CC x86_64-softmmu/numa.o CC aarch64-softmmu/cpu-exec-common.o CC aarch64-softmmu/tcg/tcg.o CC aarch64-softmmu/tcg/tcg-op.o CC aarch64-softmmu/tcg/optimize.o CC x86_64-softmmu/qtest.o CC aarch64-softmmu/tcg/tcg-common.o CC x86_64-softmmu/bootdevice.o CC aarch64-softmmu/fpu/softfloat.o CC x86_64-softmmu/kvm-all.o CC aarch64-softmmu/disas.o CC x86_64-softmmu/memory.o CC x86_64-softmmu/cputlb.o CC x86_64-softmmu/memory_mapping.o GEN aarch64-softmmu/gdbstub-xml.c CC aarch64-softmmu/kvm-stub.o CC x86_64-softmmu/dump.o CC x86_64-softmmu/migration/ram.o CC x86_64-softmmu/migration/savevm.o CC aarch64-softmmu/arch_init.o CC aarch64-softmmu/cpus.o CC aarch64-softmmu/monitor.o CC aarch64-softmmu/gdbstub.o CC x86_64-softmmu/xen-common-stub.o CC aarch64-softmmu/balloon.o CC aarch64-softmmu/ioport.o CC x86_64-softmmu/xen-hvm-stub.o CC aarch64-softmmu/numa.o CC x86_64-softmmu/hw/acpi/nvdimm.o CC x86_64-softmmu/hw/block/virtio-blk.o CC aarch64-softmmu/qtest.o CC aarch64-softmmu/bootdevice.o CC x86_64-softmmu/hw/block/dataplane/virtio-blk.o CC x86_64-softmmu/hw/char/virtio-serial-bus.o CC aarch64-softmmu/memory.o CC aarch64-softmmu/cputlb.o CC aarch64-softmmu/memory_mapping.o CC aarch64-softmmu/dump.o CC aarch64-softmmu/migration/ram.o CC aarch64-softmmu/migration/savevm.o CC x86_64-softmmu/hw/core/nmi.o CC aarch64-softmmu/xen-common-stub.o CC x86_64-softmmu/hw/cpu/core.o CC x86_64-softmmu/hw/display/vga.o CC aarch64-softmmu/xen-hvm-stub.o CC aarch64-softmmu/hw/block/virtio-blk.o CC x86_64-softmmu/hw/display/virtio-gpu.o CC x86_64-softmmu/hw/display/virtio-gpu-3d.o CC aarch64-softmmu/hw/block/dataplane/virtio-blk.o CC aarch64-softmmu/hw/char/exynos4210_uart.o CC aarch64-softmmu/hw/char/omap_uart.o CC x86_64-softmmu/hw/display/virtio-gpu-pci.o CC aarch64-softmmu/hw/char/digic-uart.o CC x86_64-softmmu/hw/display/virtio-vga.o CC aarch64-softmmu/hw/char/stm32f2xx_usart.o CC aarch64-softmmu/hw/char/bcm2835_aux.o CC aarch64-softmmu/hw/char/virtio-serial-bus.o CC aarch64-softmmu/hw/core/nmi.o CC x86_64-softmmu/hw/intc/apic.o CC x86_64-softmmu/hw/intc/apic_common.o CC x86_64-softmmu/hw/intc/ioapic.o CC x86_64-softmmu/hw/isa/lpc_ich9.o CC x86_64-softmmu/hw/misc/vmport.o CC aarch64-softmmu/hw/cpu/arm11mpcore.o CC x86_64-softmmu/hw/misc/ivshmem.o CC x86_64-softmmu/hw/misc/pvpanic.o CC aarch64-softmmu/hw/cpu/realview_mpcore.o CC aarch64-softmmu/hw/cpu/a9mpcore.o CC aarch64-softmmu/hw/cpu/a15mpcore.o CC aarch64-softmmu/hw/cpu/core.o CC x86_64-softmmu/hw/misc/edu.o CC aarch64-softmmu/hw/display/omap_dss.o CC x86_64-softmmu/hw/misc/hyperv_testdev.o CC x86_64-softmmu/hw/net/virtio-net.o CC aarch64-softmmu/hw/display/omap_lcdc.o CC x86_64-softmmu/hw/net/vhost_net.o CC aarch64-softmmu/hw/display/pxa2xx_lcd.o CC aarch64-softmmu/hw/display/bcm2835_fb.o CC x86_64-softmmu/hw/scsi/virtio-scsi.o CC x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o CC aarch64-softmmu/hw/display/vga.o CC x86_64-softmmu/hw/scsi/vhost-scsi.o CC aarch64-softmmu/hw/display/virtio-gpu.o CC aarch64-softmmu/hw/display/virtio-gpu-3d.o CC aarch64-softmmu/hw/display/virtio-gpu-pci.o CC x86_64-softmmu/hw/timer/mc146818rtc.o CC aarch64-softmmu/hw/display/dpcd.o CC x86_64-softmmu/hw/vfio/common.o CC x86_64-softmmu/hw/vfio/pci.o CC aarch64-softmmu/hw/display/xlnx_dp.o CC aarch64-softmmu/hw/dma/xlnx_dpdma.o CC aarch64-softmmu/hw/dma/omap_dma.o CC aarch64-softmmu/hw/dma/soc_dma.o CC aarch64-softmmu/hw/dma/pxa2xx_dma.o CC aarch64-softmmu/hw/dma/bcm2835_dma.o CC aarch64-softmmu/hw/gpio/omap_gpio.o CC x86_64-softmmu/hw/vfio/pci-quirks.o CC x86_64-softmmu/hw/vfio/platform.o CC aarch64-softmmu/hw/gpio/imx_gpio.o CC x86_64-softmmu/hw/vfio/calxeda-xgmac.o CC aarch64-softmmu/hw/i2c/omap_i2c.o CC x86_64-softmmu/hw/vfio/amd-xgbe.o CC aarch64-softmmu/hw/input/pxa2xx_keypad.o CC aarch64-softmmu/hw/input/tsc210x.o CC x86_64-softmmu/hw/vfio/spapr.o CC x86_64-softmmu/hw/virtio/virtio.o CC aarch64-softmmu/hw/intc/armv7m_nvic.o CC aarch64-softmmu/hw/intc/exynos4210_gic.o CC x86_64-softmmu/hw/virtio/virtio-balloon.o CC x86_64-softmmu/hw/virtio/vhost.o CC x86_64-softmmu/hw/virtio/vhost-backend.o CC x86_64-softmmu/hw/virtio/vhost-user.o CC x86_64-softmmu/hw/virtio/vhost-vsock.o CC aarch64-softmmu/hw/intc/exynos4210_combiner.o CC aarch64-softmmu/hw/intc/omap_intc.o CC x86_64-softmmu/hw/i386/multiboot.o CC x86_64-softmmu/hw/i386/pc.o CC x86_64-softmmu/hw/i386/pc_piix.o CC x86_64-softmmu/hw/i386/pc_q35.o CC x86_64-softmmu/hw/i386/pc_sysfw.o CC x86_64-softmmu/hw/i386/x86-iommu.o CC aarch64-softmmu/hw/intc/bcm2835_ic.o CC aarch64-softmmu/hw/intc/bcm2836_control.o CC aarch64-softmmu/hw/intc/allwinner-a10-pic.o CC aarch64-softmmu/hw/intc/aspeed_vic.o CC x86_64-softmmu/hw/i386/intel_iommu.o CC x86_64-softmmu/hw/i386/amd_iommu.o CC aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o CC x86_64-softmmu/hw/i386/kvmvapic.o CC x86_64-softmmu/hw/i386/acpi-build.o /tmp/qemu-test/src/hw/i386/pc_piix.c: In function ‘igd_passthrough_isa_bridge_create’: /tmp/qemu-test/src/hw/i386/pc_piix.c:1046: warning: ‘pch_rev_id’ may be used uninitialized in this function CC x86_64-softmmu/hw/i386/pci-assign-load-rom.o CC x86_64-softmmu/hw/i386/kvm/clock.o CC aarch64-softmmu/hw/misc/ivshmem.o CC aarch64-softmmu/hw/misc/arm_sysctl.o CC x86_64-softmmu/hw/i386/kvm/apic.o CC x86_64-softmmu/hw/i386/kvm/i8259.o CC x86_64-softmmu/hw/i386/kvm/ioapic.o CC aarch64-softmmu/hw/misc/cbus.o CC aarch64-softmmu/hw/misc/exynos4210_pmu.o CC aarch64-softmmu/hw/misc/imx_ccm.o /tmp/qemu-test/src/hw/i386/acpi-build.c: In function ‘build_append_pci_bus_devices’: /tmp/qemu-test/src/hw/i386/acpi-build.c:472: warning: ‘notify_method’ may be used uninitialized in this function CC aarch64-softmmu/hw/misc/imx31_ccm.o CC aarch64-softmmu/hw/misc/imx25_ccm.o CC aarch64-softmmu/hw/misc/imx6_ccm.o CC x86_64-softmmu/hw/i386/kvm/i8254.o CC x86_64-softmmu/hw/i386/kvm/pci-assign.o CC aarch64-softmmu/hw/misc/imx6_src.o CC x86_64-softmmu/target-i386/translate.o CC aarch64-softmmu/hw/misc/mst_fpga.o CC aarch64-softmmu/hw/misc/omap_clk.o CC aarch64-softmmu/hw/misc/omap_gpmc.o CC aarch64-softmmu/hw/misc/omap_l4.o CC x86_64-softmmu/target-i386/helper.o CC aarch64-softmmu/hw/misc/omap_sdrc.o CC aarch64-softmmu/hw/misc/omap_tap.o CC aarch64-softmmu/hw/misc/bcm2835_mbox.o CC aarch64-softmmu/hw/misc/bcm2835_property.o CC x86_64-softmmu/target-i386/cpu.o CC x86_64-softmmu/target-i386/bpt_helper.o CC x86_64-softmmu/target-i386/excp_helper.o CC aarch64-softmmu/hw/misc/zynq_slcr.o CC aarch64-softmmu/hw/misc/zynq-xadc.o CC aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o CC aarch64-softmmu/hw/misc/edu.o CC x86_64-softmmu/target-i386/fpu_helper.o CC x86_64-softmmu/target-i386/cc_helper.o CC aarch64-softmmu/hw/misc/auxbus.o CC aarch64-softmmu/hw/misc/aspeed_scu.o CC x86_64-softmmu/target-i386/int_helper.o CC x86_64-softmmu/target-i386/svm_helper.o CC aarch64-softmmu/hw/misc/aspeed_sdmc.o CC aarch64-softmmu/hw/net/virtio-net.o CC aarch64-softmmu/hw/net/vhost_net.o CC x86_64-softmmu/target-i386/smm_helper.o CC aarch64-softmmu/hw/pcmcia/pxa2xx.o CC x86_64-softmmu/target-i386/misc_helper.o CC aarch64-softmmu/hw/scsi/virtio-scsi.o CC aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o CC x86_64-softmmu/target-i386/mem_helper.o CC x86_64-softmmu/target-i386/seg_helper.o CC aarch64-softmmu/hw/scsi/vhost-scsi.o CC aarch64-softmmu/hw/sd/omap_mmc.o CC x86_64-softmmu/target-i386/mpx_helper.o CC x86_64-softmmu/target-i386/gdbstub.o CC aarch64-softmmu/hw/sd/pxa2xx_mmci.o CC aarch64-softmmu/hw/ssi/omap_spi.o CC aarch64-softmmu/hw/ssi/imx_spi.o CC x86_64-softmmu/target-i386/machine.o CC aarch64-softmmu/hw/timer/exynos4210_mct.o CC x86_64-softmmu/target-i386/arch_memory_mapping.o CC x86_64-softmmu/target-i386/arch_dump.o CC x86_64-softmmu/target-i386/monitor.o CC x86_64-softmmu/target-i386/kvm.o CC x86_64-softmmu/target-i386/hyperv.o CC aarch64-softmmu/hw/timer/exynos4210_pwm.o GEN trace/generated-helpers.c CC x86_64-softmmu/trace/control-target.o CC aarch64-softmmu/hw/timer/exynos4210_rtc.o CC aarch64-softmmu/hw/timer/omap_gptimer.o CC x86_64-softmmu/trace/generated-helpers.o CC aarch64-softmmu/hw/timer/omap_synctimer.o CC aarch64-softmmu/hw/timer/pxa2xx_timer.o CC aarch64-softmmu/hw/timer/digic-timer.o CC aarch64-softmmu/hw/timer/allwinner-a10-pit.o CC aarch64-softmmu/hw/usb/tusb6010.o CC aarch64-softmmu/hw/vfio/common.o CC aarch64-softmmu/hw/vfio/pci.o CC aarch64-softmmu/hw/vfio/pci-quirks.o CC aarch64-softmmu/hw/vfio/platform.o CC aarch64-softmmu/hw/vfio/calxeda-xgmac.o CC aarch64-softmmu/hw/vfio/amd-xgbe.o CC aarch64-softmmu/hw/vfio/spapr.o LINK x86_64-softmmu/qemu-system-x86_64 CC aarch64-softmmu/hw/virtio/virtio.o CC aarch64-softmmu/hw/virtio/virtio-balloon.o CC aarch64-softmmu/hw/virtio/vhost.o CC aarch64-softmmu/hw/virtio/vhost-backend.o CC aarch64-softmmu/hw/virtio/vhost-user.o CC aarch64-softmmu/hw/virtio/vhost-vsock.o CC aarch64-softmmu/hw/arm/boot.o CC aarch64-softmmu/hw/arm/collie.o CC aarch64-softmmu/hw/arm/exynos4_boards.o CC aarch64-softmmu/hw/arm/gumstix.o
../libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make[1]: *** [qemu-system-x86_64] Error 1
make: *** [subdir-x86_64-softmmu] Error 2
CC aarch64-softmmu/hw/arm/highbank.o CC aarch64-softmmu/hw/arm/digic_boards.o CC aarch64-softmmu/hw/arm/integratorcp.o CC aarch64-softmmu/hw/arm/mainstone.o CC aarch64-softmmu/hw/arm/musicpal.o CC aarch64-softmmu/hw/arm/nseries.o CC aarch64-softmmu/hw/arm/omap_sx1.o CC aarch64-softmmu/hw/arm/palm.o CC aarch64-softmmu/hw/arm/realview.o CC aarch64-softmmu/hw/arm/spitz.o CC aarch64-softmmu/hw/arm/stellaris.o CC aarch64-softmmu/hw/arm/tosa.o CC aarch64-softmmu/hw/arm/versatilepb.o CC aarch64-softmmu/hw/arm/vexpress.o CC aarch64-softmmu/hw/arm/virt.o CC aarch64-softmmu/hw/arm/xilinx_zynq.o CC aarch64-softmmu/hw/arm/z2.o CC aarch64-softmmu/hw/arm/virt-acpi-build.o CC aarch64-softmmu/hw/arm/netduino2.o CC aarch64-softmmu/hw/arm/sysbus-fdt.o CC aarch64-softmmu/hw/arm/armv7m.o CC aarch64-softmmu/hw/arm/exynos4210.o CC aarch64-softmmu/hw/arm/pxa2xx.o CC aarch64-softmmu/hw/arm/pxa2xx_gpio.o CC aarch64-softmmu/hw/arm/pxa2xx_pic.o CC aarch64-softmmu/hw/arm/digic.o CC aarch64-softmmu/hw/arm/omap1.o CC aarch64-softmmu/hw/arm/omap2.o CC aarch64-softmmu/hw/arm/strongarm.o CC aarch64-softmmu/hw/arm/allwinner-a10.o CC aarch64-softmmu/hw/arm/cubieboard.o CC aarch64-softmmu/hw/arm/bcm2835_peripherals.o CC aarch64-softmmu/hw/arm/bcm2836.o CC aarch64-softmmu/hw/arm/raspi.o CC aarch64-softmmu/hw/arm/stm32f205_soc.o CC aarch64-softmmu/hw/arm/xlnx-zynqmp.o CC aarch64-softmmu/hw/arm/xlnx-ep108.o CC aarch64-softmmu/hw/arm/fsl-imx25.o CC aarch64-softmmu/hw/arm/imx25_pdk.o CC aarch64-softmmu/hw/arm/fsl-imx31.o CC aarch64-softmmu/hw/arm/kzm.o CC aarch64-softmmu/hw/arm/fsl-imx6.o CC aarch64-softmmu/hw/arm/sabrelite.o CC aarch64-softmmu/hw/arm/aspeed_soc.o CC aarch64-softmmu/hw/arm/aspeed.o CC aarch64-softmmu/target-arm/arm-semi.o CC aarch64-softmmu/target-arm/machine.o CC aarch64-softmmu/target-arm/psci.o CC aarch64-softmmu/target-arm/arch_dump.o CC aarch64-softmmu/target-arm/monitor.o CC aarch64-softmmu/target-arm/kvm-stub.o CC aarch64-softmmu/target-arm/translate.o CC aarch64-softmmu/target-arm/op_helper.o CC aarch64-softmmu/target-arm/helper.o CC aarch64-softmmu/target-arm/cpu.o CC aarch64-softmmu/target-arm/neon_helper.o CC aarch64-softmmu/target-arm/iwmmxt_helper.o CC aarch64-softmmu/target-arm/gdbstub.o CC aarch64-softmmu/target-arm/cpu64.o CC aarch64-softmmu/target-arm/translate-a64.o CC aarch64-softmmu/target-arm/helper-a64.o CC aarch64-softmmu/target-arm/gdbstub64.o CC aarch64-softmmu/target-arm/crypto_helper.o CC aarch64-softmmu/target-arm/arm-powerctl.o GEN trace/generated-helpers.c CC aarch64-softmmu/trace/control-target.o CC aarch64-softmmu/gdbstub-xml.o CC aarch64-softmmu/trace/generated-helpers.o
/tmp/qemu-test/src/target-arm/translate-a64.c: In function ‘handle_shri_with_rndacc’:/tmp/qemu-test/src/target-arm/translate-a64.c:6333: warning: ‘tcg_src_hi’ may be used uninitialized in this function /tmp/qemu-test/src/target-arm/translate-a64.c: In function ‘disas_simd_scalar_two_reg_misc’:/tmp/qemu-test/src/target-arm/translate-a64.c:8060: warning: ‘rmode’ may be used uninitialized in this function
LINK aarch64-softmmu/qemu-system-aarch64
../libqemuutil.a(qobject-input-visitor.o): In function `qobject_input_get_object':/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: undefined reference to `g_hash_table_contains'
collect2: ld returned 1 exit status
make[1]: *** [qemu-system-aarch64] Error 1
make: *** [subdir-aarch64-softmmu] Error 2
tests/docker/Makefile.include:107: recipe for target 'docker-run-test-quick@centos6' failed
make: *** [docker-run-test-quick@centos6] Error 2
=== OUTPUT END ===
Test command exited with code: 2
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to [email protected]
Re: [Qemu-devel] [PATCH v14 03/21] qapi: add trace events for visitor
In reply to Daniel P. BerrangeQuoted 9 lines · Daniel P. Berrange
Allow tracing of the operation of visitors
Signed-off-by Daniel P. Berrange <[email protected]>--- Makefile.objs | 1 + qapi/qapi-visit-core.c | 27 +++++++++++++++++++++++++++ qapi/trace-events | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 61 insertions(+) create mode 100644 qapi/trace-events
Review
Quoted 2 lines
(quoted diff) +2
-0+ +visit_start_alternate(void *v, const char *name, void *obj, size_t size, bool promote_int) "v=%p name=%s obj=%p size=%zu promote_int=%d"
Unrelated to your patch, but should the trace engines be taught how to honor line wrapping in the trace files?
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 06/21] qapi: don't pass two copies of TestInputVisitorData to tests
In reply to Daniel P. BerrangeQuoted 6 lines · Daniel P. Berrange
The
input_visitor_test_add()method was accepting an instance of 'TestInputVisitorData' and passing it as the 'user_data' parameter to test functions. The main 'TestInputVisitorData' instance that was actually used, was meanwhile being allocated automatically by the test framework fixture setup.
Quoted 8 lines
The '
user_data' parameter is going to be needed for tests added in later patches, so getting rid of the current mistaken usage now allows this.Signed-off-by Daniel P. Berrange <[email protected]>--- tests/test-qobject-input-visitor.c | 76 ++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 44 deletions(-)
The improved commit message makes the difference :)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor
In reply to Daniel P. BerrangeQuoted 14 lines · Daniel P. Berrange
Currently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.
To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.
Quoted 9 lines
Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qobject-input-visitor.h | 32 +++++- qapi/qobject-input-visitor.c | 132 ++++++++++++++++++++++++ tests/test-qobject-input-visitor.c | 194 ++++++++++++++++++++++++++++++++++- 3 files changed, 350 insertions(+), 8 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 08/21] qapi: allow QObjectInputVisitor to be created with QemuOpts
In reply to Daniel P. BerrangeQuoted 10 lines · Daniel P. Berrange
Instead of requiring all callers to go through the mutli-step process of turning QemuOpts into a suitable QObject for visiting, add a new constructor that encapsulates this logic. This will allow QObjectInputVisitor to be a drop-in replacement for the existing OptsVisitor with minimal code changes for callers.
NB, at this point it is only supporting opts syntax which explicitly matches the QAPI schema structure, so is not yet a true drop-in replacement for OptsVisitor. The patches that follow will add the special cases requird for full backwards
s/requird/required/
include/qemu/option.h void qemu_opts_set_defaults(QemuOptsList *list, const char *params, · line 129
Quoted 13 lines
compatibility with OptsVisitor.
Signed-off-by Daniel P. Berrange <[email protected]>---
+++
b/include/qemu/option.hinclude/qemu/option.h +1
-1@@ -125,7 +125,7 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char *params, int permit_abbrev);QemuOpts *qemu_opts_from_qdict (QemuOptsList *list,const QDict *qdict,Error **errp); -QDict *qemu_opts_to_qdict (QemuOpts *opts,QDict *qdict); +QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict);void qemu_opts_absorb_qdict (QemuOpts *opts,QDict *qdict,Error **errp);
Should the const-correctness of this parameter be hoisted to any earlier patch? But here is fine, as the first place where the compiler requires it.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 09/21] qapi: permit auto-creating single element lists
In reply to Daniel P. BerrangeQuoted 13 lines · Daniel P. Berrange
When converting QemuOpts to a QObject, there is no information about compound types available, so when visiting a list, the corresponding QObject is not guaranteed to be a QList. We therefore need to be able to auto-create a single element QList from whatever type we find.
This mode should only be enabled if you have compatibility requirements for
-arg foo=hello,foo=world
to be treated as equivalent to the preferred syntax:
Quoted 9 lines
-arg foo.0=hello,foo.1=worldSigned-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qobject-input-visitor.h | 20 +++++++- qapi/qobject-input-visitor.c | 27 +++++++++-- tests/test-qobject-input-visitor.c | 88 +++++++++++++++++++++++++++++++----- 3 files changed, 117 insertions(+), 18 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 10/21] qapi: permit auto-creating nested structs
In reply to Daniel P. BerrangeQuoted 3 lines · Daniel P. Berrange
Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting
singular/plural mismatch; either s/a // or s/structs/struct/
Quoted 17 lines
the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the
qdict_crumple()call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.
The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach
Quoted 6 lines
-net vlan=1,id=foo,name=bar,opts.type=tap,\ opts.data.fd=3,opts.data.script=ifupThe legacy syntax will just be presenting
-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup
So basically, any key=values that were not consumed as known keys in the parent struct are reparsed as the (presumably lone) remaining QDict child, recursing as needed to follow the QAPI nesting.
Does this still work if the command line is presented in a different order:
-net script=ifup,vlan=1,type=tap,fd=3,id=foo,name=bar
My guess is yes, but if the testsuite doesn't cover it, you may want to add a test.
Quoted 19 lines
So we need to auto-create 3 levels of struct when visiting.
The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.
Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qobject-input-visitor.h | 22 +++++- qapi/qobject-input-visitor.c | 59 ++++++++++++++-- tests/test-qobject-input-visitor.c | 130 ++++++++++++++++++++++++++++++++--- 3 files changed, 194 insertions(+), 17 deletions(-)include/qapi/qobject-input-visitor.h +0
-0
include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict); · line 48
Quoted 5 lines
include/qapi/qobject-input-visitor.h +1
-1@@ -45,7 +45,7 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* If @autocreate_list is true, then as an alternative to a normal QList, * list values can be stored as a QString or QDict instead, which will * be interpreted as representing single element lists. This should only -* by used if compatibility is required with the OptsVisitor which allowed +* be used if compatibility is required with the OptsVisitor which allowed
This typo fix should be hoisted earlier in the series when it was introduced.
Quoted 7 lines
include/qapi/qobject-input-visitor.h +0
-0* repeated keys, without list indexes, to represent lists. e.g. set this * to true if you have compatibility requirements for * @@ -55,6 +55,22 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* * -arg foo.0=hello,foo.1=world *
include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict); · line 67
Quoted 10 lines
include/qapi/qobject-input-visitor.h +10
-0+* If @autocreate_struct_levels is non-zero, then when visiting structs, +* if the corresponding QDict is not found, it will automatically create +* a QDict containing all remaining unvisited options. This should only +* be used if compatibility is required with the OptsVisitor which flatten +* structs so that all keys were at the same level. e.g. set this to a +* non-zero number if you compatibility requirements for +* +* -arg type=person,surname=blogs,forename=fred +* +* to be treated as equivalent to the perferred syntax
s/perferred/preferred/
Quoted 5 lines
include/qapi/qobject-input-visitor.h +5
-0+static void test_visitor_in_struct_autocreate (TestInputVisitorData *data, +const void *unused) +{ +Visitor *v; +int64_t vlan;
include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
Quoted 10 lines
include/qapi/qobject-input-visitor.h +10
-0+char *id =NULL ; +char *type; +int64_t fd; +char *script =NULL ; + + v =visitor_input_test_init_internal ( + data, true, true, false,3 , +"{ 'vlan': '1', 'id': 'foo', 'type': 'tap', 'fd': '3', " +"'script': 'ifup' }" ,NULL ); +
This is where intentionally mixing up the parameter orders might be worthwhile.
Overall, the idea looks reasonable.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Quoted 7 lines · On 09/30/2016 10:45 AM, [email protected] wrote:
Hi,
Your series failed automatic build test. Please find the testing commands and their output below. If you have docker installed, you can probably reproduce it locally.
Quoted 8 lines
CC
qapi/string-output-visitor.o/tmp/qemu-test/src/qapi/qobject-input-visitor.c: In function ‘qobject_input_get_object’:/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: warning: implicit declaration of function ‘g_hash_table_contains’/tmp/qemu-test/src/qapi/qobject-input-visitor.c:97: warning: nested extern declaration of ‘g_hash_table_contains’tests/docker/Makefile.include:107: recipe for target 'docker-run-test-quick@centos6' failed
make: *** [docker-run-test-quick@centos6] Error 2
Cool - the autobuilder is now detecting use of too-new glib functions :)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor
In reply to Daniel P. BerrangeQuoted 25 lines · Daniel P. Berrange
The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:
-arg foo=5,foo=6,foo=7
-arg foo=5-7This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.
This means that
-arg foo=5-7
is treated as equivalent to
-arg foo.0=5,foo.1=6,foo.2=7
Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.
Signed-off-by Daniel P. Berrange <[email protected]>---
include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict); · line 81
Quoted 5 lines
include/qapi/qobject-input-visitor.h +2
-0@@ -71,6 +77,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* The value given determines how many levels of structs are allowed to * be flattened in this way. * +* If @permit_int_ranges is true, then when visiting a list of integers, +* the integer value strings may encode ranges eg a single element
My usual complaint that 'e.g.' is usually spelled with dots :)
include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict); · line 84
Quoted 3 lines
include/qapi/qobject-input-visitor.h +3
-0+* containing "5-7" is treated as if there were three elements "5", "6", +* "7". This should only be used if compatibility is required with the +* OptsVisitor which would allow integer ranges. e.g. set this to true
Ah. Here you DID use dots, but used the wrong abbreviation. Substituting "For example" in this location does not make as much sense as "i.e." would. Or go for simplicity; it reads just fine as:
"...would allow integer ranges; so only set this to true if..."
Quoted 25 lines
(quoted diff) +24
-0+ * if you have compatibility requirements for + * + * -arg val=5-8 + * + * to be treated as equivalent to the preferred syntax: + * + * -arg val.0=5,val.1=6,val.2=7,val.3=8 + * + + /* + * If we have string that represents an integer range (5-24), + * parse the end of the range and set things up so we'll process + * the rest of the range before consuming another element + * from the QList. + */ + if (end && *end) { + if (!qiv->permit_int_ranges) { + error_setg(errp, + "Integer ranges are not permitted here"); + return; + } + if (!inlist) { + error_setg(errp, + "Integer ranges are only permitted when "
Review Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
Quoted 10 lines
(quoted diff) +10
-0+ "visiting list parameters"); + return; + } + if (*end != '-') { + error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, + "a number range"); + return; + } + end++; + if (qemu_strtoll(end, NULL, 0, &ret) < 0) {
Signed parse...
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 399
Quoted 4 lines
qapi/qobject-input-visitor.c +4
-0+error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" ); +return ; + } +if (*obj > ret) {
...and signed comparison...
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 402
Quoted 3 lines
qapi/qobject-input-visitor.c +3
-0+ error_setg(errp, + "Parameter '%s' range start %" PRIu64 + " must be less than (or equal to) %" PRIu64,
...but unsigned numbers in the error message. Looks a bit odd (but not necessarily worse than what our OptsVisitor was already doing).
Quoted 7 lines
qapi/qobject-input-visitor.c +7
-0+ name, *obj, ret); +return ; + } + +if ((*obj <= (INT64_MAX -QIV_RANGE_MAX )) && + (ret >= (*obj +QIV_RANGE_MAX ))) { + error_setg(errp,
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 419
Quoted 10 lines
qapi/qobject-input-visitor.c +9
-0+"Parameter '%s' range must be less than %d" , + name, QIV_RANGE_MAX); +return ; + } +if (*obj != ret) { + tos->range_val = *obj; + tos->range_limit = ret; + } + } }
Overall, I found this a bit easier to read than the OptsVisitor state machine. Which makes me worried that we may have some subtle translation bugs; but I hope it's the same end result.
My consolation is that if there are any differences, they would be in the corner cases, where a user is probably going to get surprising behavior from the command line anyway, and shouldn't have been requesting ranges in that corner case.
qapi/qobject-input-visitor.c static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, · line 440
Quoted 6 lines
static void qobject_input_type_uint64(Visitor *v, const char *name,
qapi/qobject-input-visitor.c +0
-0@@ -366,21 +438,85 @@ static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, uint64_t *obj, Error **errp) { QObjectInputVisitor *qiv = to_qiv(v);
Feels like a lot of code duplication; but I don't know if there's any sane way to share more of the code while handling both signed and unsigned types.
The testsuite additions are reassuring; I confirmed that they match test-opts-visitor.c in coverage.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options
In reply to Daniel P. BerrangeQuoted 17 lines · Daniel P. Berrange
If given an option string such as
size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind
the
qemu_opts_to_qdict()method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:size=QString("1024") nodes=QString("1-2") policy=QString("bind")With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain
Quoted 5 lines
size=QString("1024")
nodes=QList([QString("10"),
QString("4-5"),
QString("1-2")])
policy=QString("bind")I think the last time around you were converting keys (turning "nodes" into "nodes.0"; I think your idea here of keeping a single key tied to QList makes more sense.
Quoted 20 lines
Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.
In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.
All existing callers are all converted to explicitly request the historical behaviour of only reporting the last key. Later patches will make use of the new modes.
Signed-off-by Daniel P. Berrange <[email protected]>---
include/qemu/option.h void qemu_opts_set_defaults(QemuOptsList *list, const char *params, · line 136
Quoted 10 lines
include/qemu/option.h +9
-1-QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict); +typedef enum { +/* Last occurrence of a duplicate option silently replaces earlier */ +QEMU_OPTS_REPEAT_POLICY_LAST , +/* Each occurrence of a duplicate option converts value to a QList */ +QEMU_OPTS_REPEAT_POLICY_ALL , +/* First occurrence of a duplicate option causes an error */ +QEMU_OPTS_REPEAT_POLICY_ERROR , +}QemuOptsRepeatPolicy ; +
Nicer. Thanks!
Quoted 36 lines
+++
b/tests/test-qemu-opts.ctests/test-qemu-opts.c +30
-0@@ -421,6 +421,130 @@ static void test_qemu_opts_set(void)g_assert (opts ==NULL ); } +static void test_qemu_opts_to_qdict_repeat_all (void ) +{ +QemuOpts *opts; +QDict *dict; +QList *list; +const QListEntry *ent; +QString *str; + +/* dynamically initialized (parsed) opts */ + opts =qemu_opts_parse (&opts_list_03, +"size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind" , + false,NULL ); +g_assert (opts); + + dict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_ALL , + &error_abort); +g_assert (dict); + + g_assert_cmpstr(qdict_get_str(dict, "size"), ==, "1024"); +g_assert (qdict_haskey (dict,"nodes" )); + list =qobject_to_qlist (qdict_get (dict,"nodes" )); +g_assert (list); + + ent =qlist_first (list); +g_assert (ent); + str =qobject_to_qstring (ent->value); +g_assert (str); + g_assert_cmpstr(qstring_get_str(str), ==, "10"); + + ent =qlist_next (ent);
tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 464
Quoted 10 lines
tests/test-qemu-opts.c +10
-0+g_assert (ent); + str =qobject_to_qstring (ent->value); +g_assert (str); + g_assert_cmpstr(qstring_get_str(str), ==, "4-5"); + + ent =qlist_next (ent); +g_assert (ent); + str =qobject_to_qstring (ent->value); +g_assert (str); + g_assert_cmpstr(qstring_get_str(str), ==, "1-2");
Yes, this is different from v13, but does look nicer with the value turning into QList instead of the key being rewritten into dotted form.
Quoted 10 lines
util/qemu-option.c +6
-0+++ b/util/qemu-option.c@@ -1057,23 +1057,73 @@ void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp)* The QDict values are of type QString. * TODO We'll want to use types appropriate for opt->desc->type, but * this is enough for now. +* +* If the @opts contains multiple occurrences of the same key, +* then the @repeatPolicy parameter determines how they are to +* be handled. Traditional behaviour was to only store the +* last occurrence, but if @repeatPolicy is set to
util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1073
Quoted 10 lines
util/qemu-option.c +8
-1+* QEMU_OPTS_REPEAT_POLICY_ALL, then a QList will be returned +* containing all values, for any key with multiple occurrences. +* The QEMU_OPTS_REPEAT_POLICY_ERROR value can be used to fail +* conversion with an error if multiple occurrences of a key +* are seen. */ -QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict) +QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict, +QemuOptsRepeatPolicy repeatPolicy, +Error **errp)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [Qemu-block] [PATCH v14 03/21] qapi: add trace events for visitor
In reply to Daniel P. BerrangeAm 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:
Quoted 3 lines
Allow tracing of the operation of visitors
Signed-off-by Daniel P. Berrange <[email protected]>
Re: [Qemu-devel] [Qemu-block] [PATCH v14 06/21] qapi: don't pass two copies of TestInputVisitorData to tests
In reply to Daniel P. BerrangeAm 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:
Quoted 11 lines
The
input_visitor_test_add()method was accepting an instance of 'TestInputVisitorData' and passing it as the 'user_data' parameter to test functions. The main 'TestInputVisitorData' instance that was actually used, was meanwhile being allocated automatically by the test framework fixture setup.The '
user_data' parameter is going to be needed for tests added in later patches, so getting rid of the current mistaken usage now allows this.Signed-off-by Daniel P. Berrange <[email protected]>
Re: [Qemu-devel] [Qemu-block] [PATCH v14 08/21] qapi: allow QObjectInputVisitor to be created with QemuOpts
In reply to Daniel P. BerrangeAm 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:
Quoted 6 lines
Instead of requiring all callers to go through the mutli-step process of turning QemuOpts into a suitable QObject for visiting, add a new constructor that encapsulates this logic. This will allow QObjectInputVisitor to be a drop-in replacement for the existing OptsVisitor with minimal code changes for callers.
Quoted 7 lines
NB, at this point it is only supporting opts syntax which explicitly matches the QAPI schema structure, so is not yet a true drop-in replacement for OptsVisitor. The patches that follow will add the special cases requird for full backwards compatibility with OptsVisitor.
Signed-off-by Daniel P. Berrange <[email protected]>
Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs
In reply to Daniel P. BerrangeAm 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:
Quoted 25 lines
Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the
qdict_crumple()call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.
The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach
-net vlan=1,id=foo,name=bar,opts.type=tap,\ opts.data.fd=3,opts.data.script=ifupThe legacy syntax will just be presenting
Quoted 7 lines
-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup
So we need to auto-create 3 levels of struct when visiting.
The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.
So you're actually introducing the modern syntax only now?
I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.
So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.
Kevin
Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs
Quoted 28 lines · Kevin Wolf
Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:
Quoted 27 lines
Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the
qdict_crumple()call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.
The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach
-net vlan=1,id=foo,name=bar,opts.type=tap,\ opts.data.fd=3,opts.data.script=ifupThe legacy syntax will just be presenting
-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup
Quoted 7 lines
Quoted 5 lines
So we need to auto-create 3 levels of struct when visiting.
The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.
So you're actually introducing the modern syntax only now?
No, the modern syntax is fully implemented by patch 8.
This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.
Quoted 7 lines
I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.
So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.
Changing that will impact on the QMP data structure, so I don't think we can do that.
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs
Am 06.10.2016 um 17:18 hat Daniel P. Berrange geschrieben:
Quoted 31 lines · Kevin Wolf
Quoted 30 lines
Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:
Quoted 29 lines
Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the
qdict_crumple()call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.
The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach
-net vlan=1,id=foo,name=bar,opts.type=tap,\ opts.data.fd=3,opts.data.script=ifupThe legacy syntax will just be presenting
-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup
So we need to auto-create 3 levels of struct when visiting.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.
So you're actually introducing the modern syntax only now?
No, the modern syntax is fully implemented by patch 8.
"now" in the sense of "in this series" (because this means that there is no external API to preserve yet).
Quoted 10 lines
This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.
Quoted 3 lines
I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.
Quoted 7 lines
Quoted 4 lines
So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.
Changing that will impact on the QMP data structure, so I don't think we can do that.
I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.
Kevin
Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs
Quoted 36 lines · Kevin Wolf
Am 06.10.2016 um 17:18 hat Daniel P. Berrange geschrieben:
Quoted 35 lines · Kevin Wolf
Quoted 34 lines
Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:
Quoted 32 lines
Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the
qdict_crumple()call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.
The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach
-net vlan=1,id=foo,name=bar,opts.type=tap,\ opts.data.fd=3,opts.data.script=ifupThe legacy syntax will just be presenting
-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup
So we need to auto-create 3 levels of struct when visiting.
The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
So you're actually introducing the modern syntax only now?
No, the modern syntax is fully implemented by patch 8.
"now" in the sense of "in this series" (because this means that there is no external API to preserve yet).
Well the syntax implemented in patch 8 is designed to 100% mirror the QAPI schema structure nesting. I don't think we want to change that behaviour.
If there are certain QAPI schemas structs can still have freedom to change though, its fine to reconsider them
Quoted 16 lines
Quoted 16 lines
This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.
Quoted 7 lines
I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.
So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.
Quoted 6 lines
Quoted 2 lines
Changing that will impact on the QMP data structure, so I don't think we can do that.
I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.
Ok, yes, it seems "NetLegacy" is only used in CLI arg parsing, so we do have some freedom there.
This patch was also needed for -numa handling too - again we might have some flexibility to flatten that.
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs
Am 06.10.2016 um 17:39 hat Daniel P. Berrange geschrieben:
Quoted 23 lines · Kevin Wolf
Quoted 22 lines
Am 06.10.2016 um 17:18 hat Daniel P. Berrange geschrieben:
Quoted 21 lines · Kevin Wolf
Quoted 19 lines
Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:
Quoted 9 lines
The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach
-net vlan=1,id=foo,name=bar,opts.type=tap,\ opts.data.fd=3,opts.data.script=ifupThe legacy syntax will just be presenting
Quoted 7 lines
-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup
So we need to auto-create 3 levels of struct when visiting.
The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.
So you're actually introducing the modern syntax only now?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
No, the modern syntax is fully implemented by patch 8.
"now" in the sense of "in this series" (because this means that there is no external API to preserve yet).
Well the syntax implemented in patch 8 is designed to 100% mirror the QAPI schema structure nesting. I don't think we want to change that behaviour.
I think patch 8 is fine as it is.
Quoted 27 lines
If there are certain QAPI schemas structs can still have freedom to change though, its fine to reconsider them
Quoted 16 lines
Quoted 16 lines
This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.
Quoted 7 lines
I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.
So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.
Quoted 6 lines
Quoted 2 lines
Changing that will impact on the QMP data structure, so I don't think we can do that.
I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.
Quoted 5 lines
Ok, yes, it seems "NetLegacy" is only used in CLI arg parsing, so we do have some freedom there.
This patch was also needed for -numa handling too - again we might have some flexibility to flatten that.
NumaOptions is also unused in QMP, so it's the same thing.
If these two are the only options that need the behaviour, I would prefer if we changed the QAPI schema to flatten them, and then we could save ourselves some complexity by dropping this patch.
Kevin
Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs
Quoted 65 lines · Kevin Wolf
Am 06.10.2016 um 17:39 hat Daniel P. Berrange geschrieben:
Quoted 23 lines · Kevin Wolf
Quoted 22 lines
Am 06.10.2016 um 17:18 hat Daniel P. Berrange geschrieben:
Quoted 21 lines · Kevin Wolf
Quoted 19 lines
Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:
Quoted 9 lines
The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach
-net vlan=1,id=foo,name=bar,opts.type=tap,\ opts.data.fd=3,opts.data.script=ifupThe legacy syntax will just be presenting
Quoted 7 lines
-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup
So we need to auto-create 3 levels of struct when visiting.
The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.
So you're actually introducing the modern syntax only now?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
No, the modern syntax is fully implemented by patch 8.
"now" in the sense of "in this series" (because this means that there is no external API to preserve yet).
Well the syntax implemented in patch 8 is designed to 100% mirror the QAPI schema structure nesting. I don't think we want to change that behaviour.
I think patch 8 is fine as it is.
Quoted 30 lines
If there are certain QAPI schemas structs can still have freedom to change though, its fine to reconsider them
Quoted 16 lines
Quoted 16 lines
This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.
Quoted 7 lines
I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.
So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.
Quoted 6 lines
Quoted 2 lines
Changing that will impact on the QMP data structure, so I don't think we can do that.
I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.
Ok, yes, it seems "NetLegacy" is only used in CLI arg parsing, so we do have some freedom there.
Quoted 8 lines
Quoted 2 lines
This patch was also needed for -numa handling too - again we might have some flexibility to flatten that.
NumaOptions is also unused in QMP, so it's the same thing.
If these two are the only options that need the behaviour, I would prefer if we changed the QAPI schema to flatten them, and then we could save ourselves some complexity by dropping this patch.
Agreed, the fewer hacks like this that we need the better.
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs
In reply to Kevin WolfQuoted 11 lines · Kevin Wolf
Quoted 7 lines
Quoted 4 lines
So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.
Changing that will impact on the QMP data structure, so I don't think we can do that.
I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.
In fact, in all my work to move netdev_add towards QAPI, I intentionally special-cased NetLegacy to be unchanged, because it was not being used by QMP at the time, and I didn't want any QMP changes to netdev to break command line usage of NetLegacy.
We still have the annoying problem that my last patch for converting netdev_add to QAPI didn't make 2.7, because we hadn't sorted out whether we wanted to be able to handle back-compat of a user that requested "1" vs. 1 (the QemuOpts code accepted either spelling, by virtue of the fact that QDict to opts conversion rewrote the parsed QMP object into all strings); and maybe this series solves that issue. But the issue for netdev_add (which IS visible to QMP) is slightly different than the issue for NetLegacy (which does not have QMP ties other than using QAPI to define a struct and glue code in net.c to map it back to normal netdev code).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 03/21] qapi: add trace events for visitor
In reply to Daniel P. BerrangeQuoted 24 lines · Daniel P. Berrange
Allow tracing of the operation of visitors
Signed-off-by Daniel P. Berrange <[email protected]>--- Makefile.objs | 1 + qapi/qapi-visit-core.c | 27 +++++++++++++++++++++++++++ qapi/trace-events | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 61 insertions(+) create mode 100644 qapi/trace-eventsMakefile.objs +1
-0@@ -160,3 +160,4 @@ trace-events-y += target-s390x/trace-events trace-events-y += target-ppc/trace-events trace-events-y += qom/trace-events trace-events-y += linux-user/trace-events +trace-events-y += qapi/trace-eventsqapi/qapi-visit-core.c +0
-0
qapi/qapi-visit-core.c line 27
Quoted 9 lines
qapi/qapi-visit-core.c +2
-0@@ -19,10 +19,12 @@#include "qapi/qmp/qerror.h" #include "qapi/visitor.h" #include "qapi/visitor-impl.h" +#include "trace.h" void visit_complete (Visitor *v,void *opaque) {assert (v->type !=VISITOR_OUTPUT || v->complete); +trace_visit_complete (v, opaque);
Trace after or before checking the precondition? Preferences, anyone?
Quoted 90 lines
qapi/qapi-visit-core.c +11
-0if (v->complete) { v->complete (v, opaque); }@@ -30,6 +32,7 @@ void visit_complete(Visitor *v, void *opaque)void visit_free (Visitor *v) { +trace_visit_free (v);if (v) { v->free (v); }@@ -40,6 +43,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj, {Error *err =NULL ; +trace_visit_start_struct (v, name, obj, size);if (obj) {assert (size);assert (!(v->type &VISITOR_OUTPUT ) || *obj);@@ -53,6 +57,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,void visit_check_struct (Visitor *v,Error **errp) { +trace_visit_check_struct (v);if (v->check_struct) { v->check_struct (v, errp); }@@ -60,6 +65,7 @@ void visit_check_struct(Visitor *v, Error **errp)void visit_end_struct (Visitor *v,void **obj) { +trace_visit_end_struct (v, obj); v->end_struct (v, obj); }@@ -69,6 +75,7 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,Error *err =NULL ;assert (!list || size >=sizeof (GenericList)); +trace_visit_start_list (v, name, list, size); v->start_list (v, name, list, size, &err);if (list && (v->type &VISITOR_INPUT )) {assert (!(err && *list));@@ -79,11 +86,13 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,GenericList *visit_next_list (Visitor *v,GenericList *tail,size_t size) {assert (tail && size >=sizeof (GenericList)); +trace_visit_next_list (v, tail, size);return v->next_list (v, tail, size); }void visit_end_list (Visitor *v,void **obj) { +trace_visit_end_list (v, obj); v->end_list (v, obj); }@@ -95,6 +104,7 @@ void visit_start_alternate(Visitor *v, const char *name,assert (obj && size >=sizeof (GenericAlternate));assert (!(v->type &VISITOR_OUTPUT ) || *obj); +trace_visit_start_alternate (v, name, obj, size, promote_int);if (v->start_alternate) { v->start_alternate (v, name, obj, size, promote_int, &err); }@@ -106,6 +116,7 @@ void visit_start_alternate(Visitor *v, const char *name,void visit_end_alternate (Visitor *v,void **obj) { +trace_visit_end_alternate (v, obj);if (v->end_alternate) { v->end_alternate (v, obj); }@@ -113,6 +124,7 @@ void visit_end_alternate(Visitor *v, void **obj)bool visit_optional (Visitor *v,const char *name,bool *present) { +trace_visit_optional (v, name, present);if (v->optional) { v->optional (v, name, present); }@@ -127,6 +139,7 @@ bool visit_is_input(Visitor *v)void visit_type_int (Visitor *v,const char *name,int64_t *obj,Error **errp) {assert (obj); +trace_visit_type_int (v, name, obj); v->type_int64 (v, name, obj, errp); }
qapi/qapi-visit-core.c void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj, · line 170
Quoted 7 lines
qapi/qapi-visit-core.c +1
-0@@ -151,6 +164,7 @@ void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj, Error **errp) {uint64_t value = *obj; +trace_visit_type_uint8 (v, name, obj);visit_type_uintN (v, &value, name,UINT8_MAX ,"uint8_t" , errp); *obj = value; }
Putting the trace in the middle of the "value = *obj; visit_type_uintN(v, &value, ...); *obj = value" pattern makes it less visible.
Preserving the pattern requires replacing the initializer by an assignment:
uint64_t value;
trace_visit_type_uint8(v, name, obj);
value = *obj;
visit_type_uintN(v, &value, name, UINT8_MAX, "uint8_t", errp);
*obj = value;Looks slightly more legible to me. What do you think?
[...]
Re: [Qemu-devel] [PATCH v14 03/21] qapi: add trace events for visitor
Quoted 31 lines · Markus Armbruster
Quoted 29 lines · Daniel P. Berrange
Allow tracing of the operation of visitors
Signed-off-by Daniel P. Berrange <[email protected]>--- Makefile.objs | 1 + qapi/qapi-visit-core.c | 27 +++++++++++++++++++++++++++ qapi/trace-events | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 61 insertions(+) create mode 100644 qapi/trace-eventsMakefile.objs +1
-0@@ -160,3 +160,4 @@ trace-events-y += target-s390x/trace-events trace-events-y += target-ppc/trace-events trace-events-y += qom/trace-events trace-events-y += linux-user/trace-events +trace-events-y += qapi/trace-eventsqapi/qapi-visit-core.c +1
-0@@ -19,10 +19,12 @@#include "qapi/qmp/qerror.h" #include "qapi/visitor.h" #include "qapi/visitor-impl.h" +#include "trace.h"
Quoted 6 lines
qapi/qapi-visit-core.c line 27
Quoted 4 lines
qapi/qapi-visit-core.c +1
-0void visit_complete (Visitor *v,void *opaque) {assert (v->type !=VISITOR_OUTPUT || v->complete); +trace_visit_complete (v, opaque);Trace after or before checking the precondition? Preferences, anyone?
I'm ambivalent, as the assert will crash you either way so whether you get a trace event just before the crash seems mostly irrelevant to me - the abort stack trace is what you'll use to diagnose the crash.
Quoted 106 lines
Quoted 91 lines
qapi/qapi-visit-core.c +11
-0if (v->complete) { v->complete (v, opaque); }@@ -30,6 +32,7 @@ void visit_complete(Visitor *v, void *opaque)void visit_free (Visitor *v) { +trace_visit_free (v);if (v) { v->free (v); }@@ -40,6 +43,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj, {Error *err =NULL ; +trace_visit_start_struct (v, name, obj, size);if (obj) {assert (size);assert (!(v->type &VISITOR_OUTPUT ) || *obj);@@ -53,6 +57,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,void visit_check_struct (Visitor *v,Error **errp) { +trace_visit_check_struct (v);if (v->check_struct) { v->check_struct (v, errp); }@@ -60,6 +65,7 @@ void visit_check_struct(Visitor *v, Error **errp)void visit_end_struct (Visitor *v,void **obj) { +trace_visit_end_struct (v, obj); v->end_struct (v, obj); }@@ -69,6 +75,7 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,Error *err =NULL ;assert (!list || size >=sizeof (GenericList)); +trace_visit_start_list (v, name, list, size); v->start_list (v, name, list, size, &err);if (list && (v->type &VISITOR_INPUT )) {assert (!(err && *list));@@ -79,11 +86,13 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,GenericList *visit_next_list (Visitor *v,GenericList *tail,size_t size) {assert (tail && size >=sizeof (GenericList)); +trace_visit_next_list (v, tail, size);return v->next_list (v, tail, size); }void visit_end_list (Visitor *v,void **obj) { +trace_visit_end_list (v, obj); v->end_list (v, obj); }@@ -95,6 +104,7 @@ void visit_start_alternate(Visitor *v, const char *name,assert (obj && size >=sizeof (GenericAlternate));assert (!(v->type &VISITOR_OUTPUT ) || *obj); +trace_visit_start_alternate (v, name, obj, size, promote_int);if (v->start_alternate) { v->start_alternate (v, name, obj, size, promote_int, &err); }@@ -106,6 +116,7 @@ void visit_start_alternate(Visitor *v, const char *name,void visit_end_alternate (Visitor *v,void **obj) { +trace_visit_end_alternate (v, obj);if (v->end_alternate) { v->end_alternate (v, obj); }@@ -113,6 +124,7 @@ void visit_end_alternate(Visitor *v, void **obj)bool visit_optional (Visitor *v,const char *name,bool *present) { +trace_visit_optional (v, name, present);if (v->optional) { v->optional (v, name, present); }@@ -127,6 +139,7 @@ bool visit_is_input(Visitor *v)void visit_type_int (Visitor *v,const char *name,int64_t *obj,Error **errp) {assert (obj); +trace_visit_type_int (v, name, obj); v->type_int64 (v, name, obj, errp); }qapi/qapi-visit-core.c void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj, · line 170
Quoted 7 lines
qapi/qapi-visit-core.c +1
-0@@ -151,6 +164,7 @@ void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj, Error **errp) {uint64_t value = *obj; +trace_visit_type_uint8 (v, name, obj);visit_type_uintN (v, &value, name,UINT8_MAX ,"uint8_t" , errp); *obj = value; }Putting the trace in the middle of the "value = *obj;
visit_type_uintN(v, &value, ...); *obj = value" pattern makes it less visible.Preserving the pattern requires replacing the initializer by an assignment:
Quoted 8 lines
uint64_t value; trace_visit_type_uint8(v, name, obj); value = *obj; visit_type_uintN(v, &value, name, UINT8_MAX, "uint8_t", errp); *obj = value;Looks slightly more legible to me. What do you think?
I'm fine with whatever you prefer.
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor
In reply to Daniel P. BerrangeQuoted 3 lines · Daniel P. Berrange
Currently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using
i.e.
Quoted 11 lines
QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.
To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.
Quoted 8 lines
Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qobject-input-visitor.h | 32 +++++- qapi/qobject-input-visitor.c | 132 ++++++++++++++++++++++++ tests/test-qobject-input-visitor.c | 194 ++++++++++++++++++++++++++++++++++- 3 files changed, 350 insertions(+), 8 deletions(-)
Quite some code. Here's how I'm persuading myself to like it.
We need to parse two and a half different languages into QAPI objects:
JSON, command line key=value,... (QemuOpts), and just value (same syntax as in QemuOpts).
Two language differences stand out:
1. JSON is recursive, key=value,... is flat (but see below), and
2. JSON values have syntactically obvious JSON types, key=value values are just strings.
To parse JSON into a QAPI object, we first parse it into a QObject, then use the QObject input visitor to convert the QObject into a QAPI object.
To parse key=value, we first parse it into a QemuOpts, then use either the options visitor to convert the QemuOpts into a QAPI object. We can also use the string input visitor to convert just an individual key's value. Or any string for that matter.
Note that we always use the QemuOpts *string* value, never the integer or boolean value QemuOpts additionally provides when the key has been specified by a QemuOptDesc with a non-string type. Such QemuOptDesc are best avoided when we parse the string again with a visitor. For when it isn't avoided, the visitors' parsers need to match the QemuOpts parser. Yes, this is a mess.
The flatness of key=value,... has become overly limiting. We've grown workarounds for lists in both the options and the string input visitor. Independent ones *sigh*. More recently, the block layer's dotted key convention has provided a way to overcome flatness completely.
We could implement this convention in the options visitor. Instead, you're picking a different route: you reuse existing qemu_opts_to_qdict() to go from QemuOpts to flat qdict, create qdict_crumple() to unflatten using dotted key convention, then reuse the existing QObject input visitor to go to QAPI object. This decomposes the task into smaller ones, some of them solved already, at a tolerable cost in efficiency.
Except the QObject input visitor won't do as is, because of language difference 2. This patch makes it cope.
The need to do this negates much of the code reuse advantage. Doesn't look so hot by itself. However, you then do more work and replace the options visitor, with an obvious path to replacing the string input visitor as well.
Quoted 12 lines
include/qapi/qobject-input-visitor.h +2
-2@@ -19,12 +19,36 @@typedef struct QObjectInputVisitor QObjectInputVisitor ; -/* -* Return a new input visitor that converts a QObject to a QAPI object. +/** +* Create a new input visitor that converts @obj to a QAPI object.
include/qapi/qobject-input-visitor.h line 31
Quoted 10 lines
include/qapi/qobject-input-visitor.h +7
-2+* +* Any scalar values in the @obj input data structure should be in the +* required type already. i.e. if visiting a bool, the value should +* already be a QBool instance. * -* Set @strict to reject a parse that doesn't consume all keys of a -* dictionary; otherwise excess input is ignored. +* If @strict is set to true, then an error will be reported if any +* dict keys are not consumed during visitation. If @strict is false +* then extra dict keys are silently ignored.
Aside: I want to get rid of non-strict, badly. Quoting commit 240f64b "qapi: Use strict QMP input visitor in more places":
The only remaining uses of non-strict input visits are:
- QMP 'qom-set' (which eventually executes object_property_set_qobject()) - mark it as something to revisit in the future (I didn't want to spend any more time on this patch auditing if we have any QOM dictionary properties that might be impacted, and couldn't easily prove whether this code path is shared with anything else).
- test-qmp-input-visitor: explicit tests of non-strict mode. If we later get rid of users that don't need strictness, then this test should be merged with test-qmp-input-strict
Quoted 11 lines
include/qapi/qobject-input-visitor.h +8
-0+* +* The returned input visitor should be released by calling +* visit_free() when no longer required. */ Visitor *qobject_input_visitor_new (QObject *obj,bool strict); +/** +* Create a new input visitor that converts @obj to a QAPI object. +* +* Any scalar values in the @obj input data structure should always be +* represented as strings. i.e. if visiting a boolean, the value should
include/qapi/qobject-input-visitor.h line 52
Quoted 10 lines
include/qapi/qobject-input-visitor.h +10
-0+* be a QString whose contents represent a valid boolean. +* +* The visitor always operates in strict mode, requiring all dict keys +* to be consumed during visitation. An error will be reported if this +* does not happen. +* +* The returned input visitor should be released by calling +* visit_free() when no longer required. +*/ +Visitor *qobject_input_visitor_new_autocast (QObject *obj);
Not thrilled about _autocast. It kind of suggests that scalar values are converted as necessary. They are, but the full truth is that all scalar values must be strings. _string_scalars is even longer, though.
Quoted 15 lines
include/qapi/qobject-input-visitor.h +1
-0+#endif qapi/qobject-input-visitor.c +1
-0@@ -20,6 +20,7 @@#include "qemu-common.h" #include "qapi/qmp/types.h" #include "qapi/qmp/qerror.h" +#include "qemu/cutils.h" #define QIV_STACK_SIZE 1024
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 273
Quoted 10 lines
qapi/qobject-input-visitor.c +7
-0@@ -263,6 +264,28 @@ static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, *obj =qint_get_int (qint); } + +static void qobject_input_type_int64_autocast (Visitor *v,const char *name, +int64_t *obj,Error **errp) +{ +QObjectInputVisitor *qiv =to_qiv (v); +QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true));
Needs a rebase for commit 1382d4a. Same elsewhere.
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 276
Quoted 3 lines
qapi/qobject-input-visitor.c +3
-0+int64_t ret; + +if (!qstr || !qstr->string) {
I don't think !qstr->string can happen. Same elsewhere.
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 280
Quoted 4 lines
qapi/qobject-input-visitor.c +4
-0+error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" , +"string" ); +return ; + }
So far, this is basically qobject_input_type_str() less the g_strdup(). Worth factoring out?
Now we're entering out parsing swamp:
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 285
Quoted 5 lines
qapi/qobject-input-visitor.c +5
-0+ +if (qemu_strtoll (qstr->string,NULL ,0 , &ret) <0 ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" ); +return ; + }
To serve as replacement for the options visitor, this needs to parse exactly like opts_type_int64().
It should also match the JSON parser as far as possible, to minimize difference between the two QObject input visitor variants, and the QemuOpts parser, for command line consistency.
opts_type_int64() uses strtoll() directly. It carefully checks for no conversion (both ways, EINVAL and endptr == str), range, and string not fully consumed.
Your code uses qemu_strtoll(). Bug#1: qemu_strtoll() assumes long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values. Bug#2: your code fails to check the string is fully consumed.
The JSON parser also uses strtoll() directly, in parse_literal(). When we get there, we know that the string consists only of decimal digits, possibly prefixed by a minus sign. Therefore, strtoll() can only fail with ERANGE, and parse_literal() handles that correctly.
QemuOpts doesn't do signed integers.
Quoted 19 lines
qapi/qobject-input-visitor.c +12
-0+ *obj = ret; +} +static void qobject_input_type_uint64 (Visitor *v,const char *name,uint64_t *obj,Error **errp) {@@ -279,6 +302,27 @@ static void qobject_input_type_uint64(Visitor *v, const char *name, *obj =qint_get_int (qint); } +static void qobject_input_type_uint64_autocast (Visitor *v,const char *name, +uint64_t *obj,Error **errp) +{ +QObjectInputVisitor *qiv =to_qiv (v); +QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true)); +unsigned long long ret; + +if (!qstr || !qstr->string) {
qapi/qobject-input-visitor.c static void qobject_input_type_uint64(Visitor *v, const char *name, · line 323
Quoted 10 lines
qapi/qobject-input-visitor.c +10
-0+error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" , +"string" ); +return ; + } + +if (parse_uint_full (qstr->string, &ret,0 ) <0 ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" ); +return ; + } + *obj = ret;
Differently wrong :)
To serve as replacement for the options visitor, this needs to parse exactly like opts_type_uint64().
Again, this should also match the JSON parser and the QemuOpts parser as far as possible.
opts_type_uint64() uses parse_uint(). It carefully checks for no conversion (EINVAL; parse_uint() normalizes), range, and string not fully consumed.
You use parse_uint_full(). You therefore don't have bug#2 here. You do have bug#1: you assume unsigned long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values.
There's a discrepancy with the JSON parser, but it's not your fault: the JSON parser represents JSON numbers that fit into int64_t to QInt, and any others as QFloat. In particular, the integers between INT64_MAX+1 and UINT64_MAX are represented as QFloat. qmp_input_type_uint64() accepts only QInt. You can declare things as uint64 in the schema, but you're still limited to 0..UINT64_MAX in QMP.
QemuOpts uses strtoull() directly, in parse_option_number(). Bug (not yours): it happily ignores errors other than "string not fully consumed".
Quoted 13 lines
qapi/qobject-input-visitor.c +6
-0+} +static void qobject_input_type_bool (Visitor *v,const char *name,bool *obj,Error **errp) {@@ -294,6 +338,22 @@ static void qobject_input_type_bool(Visitor *v, const char *name, bool *obj, *obj =qbool_get_bool (qbool); } +static void qobject_input_type_bool_autocast (Visitor *v,const char *name, +bool *obj,Error **errp) +{ +QObjectInputVisitor *qiv =to_qiv (v);
qapi/qobject-input-visitor.c static void qobject_input_type_bool(Visitor *v, const char *name, bool *obj, · line 354
Quoted 10 lines
qapi/qobject-input-visitor.c +10
-0+QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true)); + +if (!qstr || !qstr->string) { +error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" , +"string" ); +return ; + } + +parse_option_bool (name, qstr->string, obj, errp);
To serve as replacement for the options visitor, this needs to parse exactly like opts_type_bool(). Which parses with parse_option_bool(), too. Good.
QemuOpts parses the same way.
JSON syntax is different.
Quoted 23 lines
qapi/qobject-input-visitor.c +16
-0+} +static void qobject_input_type_str (Visitor *v,const char *name,char **obj,Error **errp) {@@ -335,6 +395,30 @@ static void qobject_input_type_number(Visitor *v, const char *name, double *obj, "number"); } +static void qobject_input_type_number_autocast (Visitor *v,const char *name, +double *obj,Error **errp) +{ +QObjectInputVisitor *qiv =to_qiv (v); +QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true)); +char *endp; + +if (!qstr || !qstr->string) { +error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" , +"string" ); +return ; + } +
qapi/qobject-input-visitor.c static void qobject_input_type_number(Visitor *v, const char *name, double *obj, · line 421
Quoted 10 lines
qapi/qobject-input-visitor.c +10
-0+ errno =0 ; + *obj =strtod (qstr->string, &endp); +if (errno ==0 && endp != qstr->string && *endp =='\0' ) { +return ; + } + +error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" , +"number" ); +} +
The options visitor does not implement this method.
Your error checking looks good.
The JSON parser also uses strtod(), in parse_literal(). When we get there, we know that the string is a valid JSON number. Therefore, strtod() can only fail with ERANGE. Bug (not yours): we ignore that.
QemuOpts doesn't do floating-point.
Quoted 11 lines
qapi/qobject-input-visitor.c +4
-0static void qobject_input_type_any (Visitor *v,const char *name,QObject **obj,Error **errp) {@@ -356,6 +440,22 @@ static void qobject_input_type_null(Visitor *v, const char *name, Error **errp) } } +static void qobject_input_type_size_autocast (Visitor *v,const char *name, +uint64_t *obj,Error **errp) +{ +QObjectInputVisitor *qiv =to_qiv (v);
qapi/qobject-input-visitor.c static void qobject_input_type_null(Visitor *v, const char *name, Error **errp) · line 456
Quoted 10 lines
qapi/qobject-input-visitor.c +10
-0+QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true)); + +if (!qstr || !qstr->string) { +error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" , +"string" ); +return ; + } + +parse_option_size (name, qstr->string, obj, errp);
The options visitor uses qemu_strtosz_suffix(). Are you sure parse_option_size() is a compatible replacement? (I'm getting tired...)
Quoted 300 lines
qapi/qobject-input-visitor.c +34
-0+} +static void qobject_input_optional (Visitor *v,const char *name,bool *present) {QObjectInputVisitor *qiv =to_qiv (v);@@ -413,3 +513,35 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)return &v->visitor; } + +Visitor *qobject_input_visitor_new_autocast (QObject *obj) +{ +QObjectInputVisitor *v; + + v =g_malloc0 (sizeof (*v)); + + v->visitor.type =VISITOR_INPUT ; + v->visitor.start_struct = qobject_input_start_struct; + v->visitor.check_struct = qobject_input_check_struct; + v->visitor.end_struct = qobject_input_pop; + v->visitor.start_list = qobject_input_start_list; + v->visitor.next_list = qobject_input_next_list; + v->visitor.end_list = qobject_input_pop; + v->visitor.start_alternate = qobject_input_start_alternate; + v->visitor.type_int64 = qobject_input_type_int64_autocast; + v->visitor.type_uint64 = qobject_input_type_uint64_autocast; + v->visitor.type_bool = qobject_input_type_bool_autocast; + v->visitor.type_str = qobject_input_type_str; + v->visitor.type_number = qobject_input_type_number_autocast; + v->visitor.type_any = qobject_input_type_any; + v->visitor.type_null = qobject_input_type_null; + v->visitor.type_size = qobject_input_type_size_autocast; + v->visitor.optional = qobject_input_optional; + v->visitor.free = qobject_input_free; + v->strict = true; + + v->root = obj; +qobject_incref (obj); + +return &v->visitor; +}tests/test-qobject-input-visitor.c +183
-4@@ -41,6 +41,7 @@ static void visitor_input_teardown(TestInputVisitorData *data,function so that the JSON string used by the tests are kept in the test functions (and not in main()). */ static Visitor *visitor_input_test_init_internal (TestInputVisitorData *data, +bool strict,bool autocast,const char *json_string,va_list *ap) {@@ -49,11 +50,31 @@ static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data, data->obj =qobject_from_jsonv (json_string, ap);g_assert (data->obj); - data->qiv =qobject_input_visitor_new (data->obj, false); +if (autocast) { +assert (strict); + data->qiv =qobject_input_visitor_new_autocast (data->obj); + }else { + data->qiv =qobject_input_visitor_new (data->obj, strict); + }g_assert (data->qiv);return data->qiv; } +static GCC_FMT_ATTR(4, 5) +Visitor *visitor_input_test_init_full (TestInputVisitorData *data, +bool strict,bool autocast, +const char *json_string, ...) +{ +Visitor *v; +va_list ap; + +va_start (ap, json_string); + v =visitor_input_test_init_internal (data, strict, autocast, + json_string, &ap); +va_end (ap); +return v; +} + static GCC_FMT_ATTR(2, 3) Visitor *visitor_input_test_init(TestInputVisitorData *data, const char *json_string, ...)@@ -62,7 +83,8 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,va_list ap;va_start (ap, json_string); - v =visitor_input_test_init_internal (data, json_string, &ap); + v =visitor_input_test_init_internal (data, true, false, + json_string, &ap);va_end (ap);return v; }@@ -77,7 +99,8 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,static Visitor *visitor_input_test_init_raw (TestInputVisitorData *data,const char *json_string) { -return visitor_input_test_init_internal (data, json_string,NULL ); +return visitor_input_test_init_internal (data, true, false, + json_string,NULL ); } static void test_visitor_in_int(TestInputVisitorData *data,@@ -109,6 +132,45 @@ static void test_visitor_in_int_overflow(TestInputVisitorData *data,error_free_or_abort (&err); } +static void test_visitor_in_int_autocast (TestInputVisitorData *data, +const void *unused) +{ +int64_t res =0 , value =-42 ; +Error *err =NULL ; +Visitor *v; + + v =visitor_input_test_init_full (data, true, true, +"%" PRId64, value); +visit_type_int (v,NULL , &res, &err); +error_free_or_abort (&err); +} + +static void test_visitor_in_int_str_autocast (TestInputVisitorData *data, +const void *unused) +{ +int64_t res =0 , value =-42 ; +Visitor *v; + + v =visitor_input_test_init_full (data, true, true, +"\"-42\"" ); + +visit_type_int (v,NULL , &res, &error_abort); + g_assert_cmpint(res, ==, value); +} + +static void test_visitor_in_int_str_noautocast (TestInputVisitorData *data, +const void *unused) +{ +int64_t res =0 ; +Visitor *v; +Error *err =NULL ; + + v =visitor_input_test_init (data,"\"-42\"" ); + +visit_type_int (v,NULL , &res, &err); +error_free_or_abort (&err); +} +static void test_visitor_in_bool (TestInputVisitorData *data,const void *unused) {@@ -121,6 +183,44 @@ static void test_visitor_in_bool(TestInputVisitorData *data, g_assert_cmpint(res, ==, true); } +static void test_visitor_in_bool_autocast (TestInputVisitorData *data, +const void *unused) +{ +bool res = false; +Error *err =NULL ; +Visitor *v; + + v =visitor_input_test_init_full (data, true, true,"true" ); + +visit_type_bool (v,NULL , &res, &err); +error_free_or_abort (&err); +} + +static void test_visitor_in_bool_str_autocast (TestInputVisitorData *data, +const void *unused) +{ +bool res = false; +Visitor *v; + + v =visitor_input_test_init_full (data, true, true,"\"yes\"" ); + +visit_type_bool (v,NULL , &res, &error_abort); + g_assert_cmpint(res, ==, true); +} + +static void test_visitor_in_bool_str_noautocast (TestInputVisitorData *data, +const void *unused) +{ +bool res = false; +Visitor *v; +Error *err =NULL ; + + v =visitor_input_test_init (data,"\"true\"" ); + +visit_type_bool (v,NULL , &res, &err); +error_free_or_abort (&err); +} +static void test_visitor_in_number (TestInputVisitorData *data,const void *unused) {@@ -133,6 +233,69 @@ static void test_visitor_in_number(TestInputVisitorData *data, g_assert_cmpfloat(res, ==, value); } +static void test_visitor_in_number_autocast (TestInputVisitorData *data, +const void *unused) +{ +double res =0 , value =3.14 ; +Error *err =NULL ; +Visitor *v; + + v =visitor_input_test_init_full (data, true, true,"%f" , value); + +visit_type_number (v,NULL , &res, &err); +error_free_or_abort (&err); +} + +static void test_visitor_in_number_str_autocast (TestInputVisitorData *data, +const void *unused) +{ +double res =0 , value =3.14 ; +Visitor *v; + + v =visitor_input_test_init_full (data, true, true,"\"3.14\"" ); + +visit_type_number (v,NULL , &res, &error_abort); + g_assert_cmpfloat(res, ==, value); +} + +static void test_visitor_in_number_str_noautocast (TestInputVisitorData *data, +const void *unused) +{ +double res =0 ; +Visitor *v; +Error *err =NULL ; + + v =visitor_input_test_init (data,"\"3.14\"" ); + +visit_type_number (v,NULL , &res, &err); +error_free_or_abort (&err); +} + +static void test_visitor_in_size_str_autocast (TestInputVisitorData *data, +const void *unused) +{ +uint64_t res, value =500 *1024 *1024 ; +Visitor *v; + + v =visitor_input_test_init_full (data, true, true,"\"500M\"" ); + +visit_type_size (v,NULL , &res, &error_abort); + g_assert_cmpfloat(res, ==, value); +} + +static void test_visitor_in_size_str_noautocast (TestInputVisitorData *data, +const void *unused) +{ +uint64_t res =0 ; +Visitor *v; +Error *err =NULL ; + + v =visitor_input_test_init (data,"\"500M\"" ); + +visit_type_size (v,NULL , &res, &err); +error_free_or_abort (&err); +} +static void test_visitor_in_string (TestInputVisitorData *data,const void *unused) {@@ -289,7 +452,8 @@ static void test_visitor_in_null(TestInputVisitorData *data,* when input is not null. */ - v =visitor_input_test_init (data,"{ 'a': null, 'b': '' }" ); + v =visitor_input_test_init_full (data, false, false, +"{ 'a': null, 'b': '' }" );visit_start_struct (v,NULL ,NULL ,0 , &error_abort);visit_type_null (v,"a" , &error_abort);visit_type_str (v,"a" , &tmp, &err);@@ -840,10 +1004,32 @@ int main(int argc, char **argv) NULL, test_visitor_in_int);input_visitor_test_add ("/visitor/input/int_overflow" ,NULL , test_visitor_in_int_overflow); +input_visitor_test_add ("/visitor/input/int_autocast" , +NULL , test_visitor_in_int_autocast); +input_visitor_test_add ("/visitor/input/int_str_autocast" , +NULL , test_visitor_in_int_str_autocast); +input_visitor_test_add ("/visitor/input/int_str_noautocast" , +NULL , test_visitor_in_int_str_noautocast);input_visitor_test_add ("/visitor/input/bool" ,NULL , test_visitor_in_bool); +input_visitor_test_add ("/visitor/input/bool_autocast" , +NULL , test_visitor_in_bool_autocast); +input_visitor_test_add ("/visitor/input/bool_str_autocast" , +NULL , test_visitor_in_bool_str_autocast); +input_visitor_test_add ("/visitor/input/bool_str_noautocast" , +NULL , test_visitor_in_bool_str_noautocast);input_visitor_test_add ("/visitor/input/number" ,NULL , test_visitor_in_number); +input_visitor_test_add ("/visitor/input/number_autocast" , +NULL , test_visitor_in_number_autocast); + input_visitor_test_add("/visitor/input/number_str_autocast",
tests/test-qobject-input-visitor.c int main(int argc, char **argv) · line 1035
Quoted 10 lines
tests/test-qobject-input-visitor.c +7
-0+ NULL, test_visitor_in_number_str_autocast); +input_visitor_test_add ("/visitor/input/number_str_noautocast" , +NULL , test_visitor_in_number_str_noautocast); +input_visitor_test_add ("/visitor/input/size_str_autocast" , +NULL , test_visitor_in_size_str_autocast); +input_visitor_test_add ("/visitor/input/size_str_noautocast" , +NULL , test_visitor_in_size_str_noautocast);input_visitor_test_add ("/visitor/input/string" ,NULL , test_visitor_in_string); input_visitor_test_add("/visitor/input/enum",
I don't like the test code duplication. The alternative would be a function that rewrites all the scalars in a QObject to strings. Probably not worthwhile. We'll live with the duplication then.
Re: [Qemu-devel] [PATCH v14 08/21] qapi: allow QObjectInputVisitor to be created with QemuOpts
In reply to Daniel P. BerrangeQuoted 1 line · Daniel P. Berrange
Instead of requiring all callers to go through the mutli-step
multi-step
Quoted 61 lines
process of turning QemuOpts into a suitable QObject for visiting, add a new constructor that encapsulates this logic. This will allow QObjectInputVisitor to be a drop-in replacement for the existing OptsVisitor with minimal code changes for callers.
NB, at this point it is only supporting opts syntax which explicitly matches the QAPI schema structure, so is not yet a true drop-in replacement for OptsVisitor. The patches that follow will add the special cases requird for full backwards compatibility with OptsVisitor.
Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qobject-input-visitor.h | 15 +++++++++++++++ include/qemu/option.h | 2 +- qapi/qobject-input-visitor.c | 25 +++++++++++++++++++++++++ util/qemu-option.c | 2 +- 4 files changed, 42 insertions(+), 2 deletions(-)include/qapi/qobject-input-visitor.h +15
-0@@ -51,4 +51,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);*/ Visitor *qobject_input_visitor_new_autocast (QObject *obj); + +/** +* Create a new input visitor that converts @opts to a QAPI object. +* +* The QemuOpts will be converted into a QObject using the +* qdict_crumple() method to automatically create structs +* and lists. The resulting QDict will then be passed to the +* qobject_input_visitor_new_autocast() method. See the docs +* of that method for further details on processing behaviour. +* +* The returned input visitor should be released by calling +* visit_free() when no longer required. +*/ +Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts,Error **errp); +#endif include/qemu/option.h +1
-1@@ -125,7 +125,7 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char *params, int permit_abbrev);QemuOpts *qemu_opts_from_qdict (QemuOptsList *list,const QDict *qdict,Error **errp); -QDict *qemu_opts_to_qdict (QemuOpts *opts,QDict *qdict); +QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict);void qemu_opts_absorb_qdict (QemuOpts *opts,QDict *qdict,Error **errp);typedef int (*qemu_opts_loopfunc )(void *opaque,QemuOpts *opts,Error **errp);qapi/qobject-input-visitor.c +0
-0
qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_autocast(QObject *obj) · line 554
Quoted 10 lines
qapi/qobject-input-visitor.c +7
-0@@ -545,3 +545,28 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)return &v->visitor; } + + +Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts, +Error **errp) +{ +QDict *pdict; +QObject *pobj =NULL ;
@pdict and @pobj are unusual names. Let's stick to the more common @dict and @obj.
Quoted 23 lines
qapi/qobject-input-visitor.c +18
-0+Visitor *v =NULL ; + + pdict =qemu_opts_to_qdict (opts,NULL ); +if (!pdict) { + goto cleanup; + } + + pobj =qdict_crumple (pdict, true, errp); +if (!pobj) { + goto cleanup; + } + + v =qobject_input_visitor_new_autocast (pobj); + cleanup: +qobject_decref (pobj); +QDECREF (pdict); +return v; +}util/qemu-option.c +0
-0
util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1059
Quoted 2 lines
util/qemu-option.c +0
-0@@ -1058,7 +1058,7 @@ void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp)* TODO We'll want to use types appropriate for opt->desc->type, but * this is enough for now.
Implementing this TODO could break users that pass it to qobject_input_visitor_new_autocast(), because despite its name, the _autocast visitor expects *only* string scalars.
I guess (but have not checked) that these users have null opt->desc since their opts->list->desc[] is empty.
We should either drop the TODO (needs review of the other users), or update it to reflect the new situation. Perhaps users of qobject_input_visitor_new_autocast() should additionally assert opts->list->desc[] is empty.
Quoted 6 lines
util/qemu-option.c +1
-1*/ -QDict *qemu_opts_to_qdict (QemuOpts *opts,QDict *qdict) +QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict) {QemuOpt *opt;QObject *val;
Re: [Qemu-devel] [PATCH v14 09/21] qapi: permit auto-creating single element lists
In reply to Daniel P. BerrangeQuoted 2 lines · Daniel P. Berrange
When converting QemuOpts to a QObject, there is no information about compound types available,
Yes, that's a drawback of splitting the conversion into a QemuOpts -> QObject part that is oblivious of types, and a QObject -> QAPI object part that knows the types.
Quoted 5 lines
so when visiting a list, the
corresponding QObject is not guaranteed to be a QList. We therefore need to be able to auto-create a single element QList from whatever type we find.
Quoted 8 lines
This mode should only be enabled if you have compatibility requirements for
-arg foo=hello,foo=world
to be treated as equivalent to the preferred syntax:
-arg foo.0=hello,foo.1=world
Not sure this is "preferred". "More powerfully warty" is probably closer to the truth ;)
How is "-arg foo=hello,foo=world" treated if this mode isn't enabled?
What would be the drawbacks of doing this always instead of only when we "have compatibility requirements"?
Quoted 75 lines
Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qobject-input-visitor.h | 20 +++++++- qapi/qobject-input-visitor.c | 27 +++++++++-- tests/test-qobject-input-visitor.c | 88 +++++++++++++++++++++++++++++++----- 3 files changed, 117 insertions(+), 18 deletions(-)include/qapi/qobject-input-visitor.h +18
-2@@ -42,6 +42,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* represented as strings. i.e. if visiting a boolean, the value should * be a QString whose contents represent a valid boolean. * +* If @autocreate_list is true, then as an alternative to a normal QList, +* list values can be stored as a QString or QDict instead, which will +* be interpreted as representing single element lists. This should only +* by used if compatibility is required with the OptsVisitor which allowed +* repeated keys, without list indexes, to represent lists. e.g. set this +* to true if you have compatibility requirements for +* +* -arg foo=hello,foo=world +* +* to be treated as equivalent to the preferred syntax: +* +* -arg foo.0=hello,foo.1=world +* * The visitor always operates in strict mode, requiring all dict keys * to be consumed during visitation. An error will be reported if this * does not happen. @@ -49,7 +62,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* The returned input visitor should be released by calling * visit_free() when no longer required. */ -Visitor *qobject_input_visitor_new_autocast (QObject *obj); +Visitor *qobject_input_visitor_new_autocast (QObject *obj, +bool autocreate_list);/** @@ -64,6 +78,8 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj);* The returned input visitor should be released by calling * visit_free() when no longer required. */ -Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts,Error **errp); +Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts, +bool autocreate_list, +Error **errp);#endif qapi/qobject-input-visitor.c +5
-0@@ -48,6 +48,10 @@ struct QObjectInputVisitor/* True to reject parse in visit_end_struct() if unvisited keys remain. */ bool strict; + +/* Whether we can auto-create single element lists when +* encountering a non-QList type */ +bool autocreate_list; };static QObjectInputVisitor *to_qiv (Visitor *v)@@ -108,6 +112,7 @@ static const QListEntry *qobject_input_push(QObjectInputVisitor *qiv,assert (obj); tos->obj = obj; tos->qapi = qapi; +qobject_incref (obj);if (qiv->strict &&qobject_type (obj) ==QTYPE_QDICT ) { h =g_hash_table_new (g_str_hash, g_str_equal);
qapi/qobject-input-visitor.c static void qobject_input_stack_object_free(StackObject *tos) · line 158
Quoted 7 lines
qapi/qobject-input-visitor.c +1
-0@@ -147,6 +152,7 @@ static void qobject_input_stack_object_free(StackObject *tos)if (tos->h) {g_hash_table_unref (tos->h); } +qobject_decref (tos->obj);g_free (tos); }
Can you explain the reference counting change?
qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 206
Quoted 6 lines
qapi/qobject-input-visitor.c +1
-1@@ -197,7 +203,7 @@ static void qobject_input_start_list(Visitor *v, const char *name,QObject *qobj =qobject_input_get_object (qiv, name, true);const QListEntry *entry; -if (!qobj ||qobject_type (qobj) !=QTYPE_QLIST ) { +if (!qobj || (!qiv->autocreate_list &&qobject_type (qobj) !=QTYPE_QLIST )) {
Long line, but I believe it'll go away when you rebase for commit 1382d4a.
Quoted 11 lines
qapi/qobject-input-visitor.c +3
-1if (list) { *list =NULL ; }@@ -206,7 +212,16 @@ static void qobject_input_start_list(Visitor *v, const char *name,return ; } - entry =qobject_input_push (qiv, qobj, list, errp); +if (qobject_type (qobj) !=QTYPE_QLIST ) { +QList *tmplist =qlist_new (); +qlist_append_obj (tmplist, qobj);
qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 227
Quoted 10 lines
qapi/qobject-input-visitor.c +7
-0+qobject_incref (qobj); + entry =qobject_input_push (qiv,QOBJECT (tmplist), list, errp); +QDECREF (tmplist); + }else { + entry =qobject_input_push (qiv, qobj, list, errp); + } +if (list) {if (entry) { *list =g_malloc0 (size);
Buries autolist behavior in the middle of things. What about doing it first, so it's more separate?
QObjectInputVisitor *qiv = to_qiv(v);
QObject *qobj = qobject_input_get_object_(qiv, name, true, errp);
const QListEntry *entry;
if (!qobj) {
return;
} + if (qiv->autocreate_list && qobject_type(qobj) != QTYPE_QLIST) {
+ QList *auto_list = qlist_new();
+ qlist_append_obj(auto_list, qobj);
+ qobj = auto_list;
+ }
+
if (qobject_type(qobj) != QTYPE_QLIST) {I ignored reference counting here, because I don't yet understand how and why you're changing it.
Quoted 27 lines
qapi/qobject-input-visitor.c +4
-1@@ -514,7 +529,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)return &v->visitor; } -Visitor *qobject_input_visitor_new_autocast (QObject *obj) +Visitor *qobject_input_visitor_new_autocast (QObject *obj, +bool autocreate_list) {QObjectInputVisitor *v;@@ -539,6 +555,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj) v->visitor.optional = qobject_input_optional; v->visitor.free = qobject_input_free; v->strict = true; + v->autocreate_list = autocreate_list; v->root = obj;qobject_incref (obj);@@ -548,6 +565,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts, +bool autocreate_list,Error **errp) {QDict *pdict;
qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 589
Quoted 9 lines
qapi/qobject-input-visitor.c +2
-1@@ -564,7 +582,8 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, goto cleanup; } - v =qobject_input_visitor_new_autocast (pobj); + v =qobject_input_visitor_new_autocast (pobj, + autocreate_list); cleanup:qobject_decref (pobj);QDECREF (pdict);
[Skipping test updates for now...]
Re: [Qemu-devel] [Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs
In reply to Daniel P. BerrangeQuoted 63 lines · Daniel P. Berrange
Quoted 62 lines · Kevin Wolf
Am 06.10.2016 um 17:39 hat Daniel P. Berrange geschrieben:
Quoted 23 lines · Kevin Wolf
Quoted 22 lines
Am 06.10.2016 um 17:18 hat Daniel P. Berrange geschrieben:
Quoted 21 lines · Kevin Wolf
Quoted 19 lines
Am 30.09.2016 um 16:45 hat Daniel P. Berrange geschrieben:
Quoted 9 lines
The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach
-net vlan=1,id=foo,name=bar,opts.type=tap,\ opts.data.fd=3,opts.data.script=ifupThe legacy syntax will just be presenting
Quoted 7 lines
-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup
So we need to auto-create 3 levels of struct when visiting.
The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.
So you're actually introducing the modern syntax only now?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
No, the modern syntax is fully implemented by patch 8.
"now" in the sense of "in this series" (because this means that there is no external API to preserve yet).
Well the syntax implemented in patch 8 is designed to 100% mirror the QAPI schema structure nesting. I don't think we want to change that behaviour.
I think patch 8 is fine as it is.
Quoted 27 lines
If there are certain QAPI schemas structs can still have freedom to change though, its fine to reconsider them
Quoted 22 lines
Quoted 11 lines
This patch is about adding hacks for the legacy syntax used by the OptsVisitor. The OptsVisitor didn't interpret struct nesting at all, so everything looked flat - this only works as long as you don't have the same key used in multiple structs at different levels, so is not useful as a general approach - it only works by luck really.
Quoted 3 lines
I don't think an "opts.data." prefix makes a lot of sense. I suspect the schema looks this way only because we didn't have flat unions in 1.2.
Quoted 7 lines
Quoted 4 lines
So, considering that it is a purely internally used type not visible in QMP, would it make sense to change NetLegacy to be a flat union instead, with NetLegacyOptions as the common base? Then you get the same flat namespace that we always had and that makes much more sense as an API.
Changing that will impact on the QMP data structure, so I don't think we can do that.
I don't see this type used in QMP at all. It's only used for command line parsing, and only with the OptsVisitor, so I think we're fine if we flatten it now.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Ok, yes, it seems "NetLegacy" is only used in CLI arg parsing, so we do have some freedom there.
This patch was also needed for -numa handling too - again we might have some flexibility to flatten that.
NumaOptions is also unused in QMP, so it's the same thing.
For better or worse, the QAPI schema doesn't separate externally visible stuff from purely internal stuff. Useful trick to see what's external:
$ python -B scripts/qapi-introspect.py -cu -p scratch- qapi-schema.json
This generates scratch-qmp-introspect.c describing the external QMP interface with unmasked type names (-u). Anything not mentioned there can be changed freely.
I'd like to see patches flattening simple unions that aren't externally visible. Flat carries a bit of notational overhead in the schema, but I hope to address that once the QAPI queue is under control.
Quoted 5 lines
Quoted 3 lines
If these two are the only options that need the behaviour, I would prefer if we changed the QAPI schema to flatten them, and then we could save ourselves some complexity by dropping this patch.
Agreed, the fewer hacks like this that we need the better.
Indeed. Please find out which hacks we actually need.
I believe the best way to keep the warts in check is getting rid of the options visitor (done in this series) and the string visitor (hopefully enabled by this series).
Re: [Qemu-devel] [PATCH v14 10/21] qapi: permit auto-creating nested structs
In reply to Daniel P. BerrangeQuoted 45 lines · Daniel P. Berrange
Some of the historical command line opts that had their keys in in a completely flat namespace are now represented by QAPI schemas that use a nested structs. When converting the QemuOpts to QObject, there is no information about compound types available, so the QObject will be completely flat, even after the
qdict_crumple()call. So when starting a struct, we may not have a QDict available in the input data, so we auto-create a QDict containing all the currently unvisited input data keys. Not all historical command line opts require this, so the behaviour is opt-in, by specifying how many levels of structs are permitted to be auto-created.Note that this only works if the child struct is the last field to the visited in the parent struct. This is always the case for currently existing legacy command line options.
The example is the NetLegacy type which has 3 levels of structs. The modern way to represent this in QemuOpts would be the dot-separated component approach
-net vlan=1,id=foo,name=bar,opts.type=tap,\ opts.data.fd=3,opts.data.script=ifupThe legacy syntax will just be presenting
-net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup
So we need to auto-create 3 levels of struct when visiting.
The implementation here will enable visiting in both the modern and legacy syntax, compared to OptsVisitor which only allows the legacy syntax.
Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qobject-input-visitor.h | 22 +++++- qapi/qobject-input-visitor.c | 59 ++++++++++++++-- tests/test-qobject-input-visitor.c | 130 ++++++++++++++++++++++++++++++++--- 3 files changed, 194 insertions(+), 17 deletions(-)include/qapi/qobject-input-visitor.h +0
-0
include/qapi/qobject-input-visitor.h Visitor *qobject_input_visitor_new(QObject *obj, bool strict); · line 51
Quoted 8 lines
include/qapi/qobject-input-visitor.h +1
-1@@ -45,7 +45,7 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* If @autocreate_list is true, then as an alternative to a normal QList, * list values can be stored as a QString or QDict instead, which will * be interpreted as representing single element lists. This should only -* by used if compatibility is required with the OptsVisitor which allowed +* be used if compatibility is required with the OptsVisitor which allowed * repeated keys, without list indexes, to represent lists. e.g. set this * to true if you have compatibility requirements for *
Typo introduced in the previous patch, please fix it there.
Skipping the rest of this patch until it's clear we need it.
[...]
Re: [Qemu-devel] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor
In reply to Markus ArmbrusterQuoted 22 lines · Markus Armbruster
Quoted 3 lines · Daniel P. Berrange
Currently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using
i.e.
Quoted 14 lines
QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.
To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.
Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>
[...]
Quoted 37 lines
Quoted 13 lines
qapi/qobject-input-visitor.c +1
-0@@ -20,6 +20,7 @@#include "qemu-common.h" #include "qapi/qmp/types.h" #include "qapi/qmp/qerror.h" +#include "qemu/cutils.h" #define QIV_STACK_SIZE 1024qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 273
Quoted 10 lines
qapi/qobject-input-visitor.c +7
-0@@ -263,6 +264,28 @@ static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, *obj =qint_get_int (qint); } + +static void qobject_input_type_int64_autocast (Visitor *v,const char *name, +int64_t *obj,Error **errp) +{ +QObjectInputVisitor *qiv =to_qiv (v); +QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true));Needs a rebase for commit 1382d4a. Same elsewhere.
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 276
Quoted 3 lines
qapi/qobject-input-visitor.c +3
-0+int64_t ret; + +if (!qstr || !qstr->string) {I don't think !qstr->string can happen. Same elsewhere.
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 280
Quoted 4 lines
qapi/qobject-input-visitor.c +4
-0+error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" , +"string" ); +return ; + }
Quoted 8 lines
So far, this is basically
qobject_input_type_str()less theg_strdup(). Worth factoring out?Now we're entering out parsing swamp:
Quoted 3 lines
qapi/qobject-input-visitor.c +3
-0+ +if (qemu_strtoll (qstr->string,NULL ,0 , &ret) <0 ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" );
"integer", actually.
Quoted 36 lines
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 285
Quoted 2 lines
qapi/qobject-input-visitor.c +2
-0+return ; + }To serve as replacement for the options visitor, this needs to parse exactly like
opts_type_int64().It should also match the JSON parser as far as possible, to minimize difference between the two QObject input visitor variants, and the QemuOpts parser, for command line consistency.
opts_type_int64()usesstrtoll()directly. It carefully checks for no conversion (both ways, EINVAL and endptr == str), range, and string not fully consumed.Your code uses
qemu_strtoll(). Bug#1:qemu_strtoll()assumes long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values. Bug#2: your code fails to check the string is fully consumed.The JSON parser also uses
strtoll()directly, inparse_literal(). When we get there, we know that the string consists only of decimal digits, possibly prefixed by a minus sign. Therefore,strtoll()can only fail with ERANGE, andparse_literal()handles that correctly.QemuOpts doesn't do signed integers.
Quoted 10 lines
qapi/qobject-input-visitor.c +3
-0+ *obj = ret; +} +static void qobject_input_type_uint64 (Visitor *v,const char *name,uint64_t *obj,Error **errp) {@@ -279,6 +302,27 @@ static void qobject_input_type_uint64(Visitor *v, const char *name, *obj =qint_get_int (qint); }
Quoted 16 lines
Quoted 6 lines
qapi/qobject-input-visitor.c +6
-0+static void qobject_input_type_uint64_autocast (Visitor *v,const char *name, +uint64_t *obj,Error **errp) +{ +QObjectInputVisitor *qiv =to_qiv (v); +QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true));qapi/qobject-input-visitor.c static void qobject_input_type_uint64(Visitor *v, const char *name, · line 320
Quoted 10 lines
qapi/qobject-input-visitor.c +10
-0+unsigned long long ret; + +if (!qstr || !qstr->string) { +error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" , +"string" ); +return ; + } + +if (parse_uint_full (qstr->string, &ret,0 ) <0 ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" );
"integer".
Quoted 37 lines
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 286
Quoted 3 lines
qapi/qobject-input-visitor.c +3
-0+return ; + } + *obj = ret;Differently wrong :)
To serve as replacement for the options visitor, this needs to parse exactly like
opts_type_uint64().Again, this should also match the JSON parser and the QemuOpts parser as far as possible.
opts_type_uint64()usesparse_uint(). It carefully checks for no conversion (EINVAL;parse_uint()normalizes), range, and string not fully consumed.You use
parse_uint_full(). You therefore don't have bug#2 here. You do have bug#1: you assume unsigned long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values.There's a discrepancy with the JSON parser, but it's not your fault: the JSON parser represents JSON numbers that fit into
int64_tto QInt, and any others as QFloat. In particular, the integers between INT64_MAX+1 andUINT64_MAXare represented as QFloat.qmp_input_type_uint64()accepts only QInt. You can declare things as uint64 in the schema, but you're still limited to 0..UINT64_MAX in QMP.QemuOpts uses
strtoull()directly, inparse_option_number(). Bug (not yours): it happily ignores errors other than "string not fully consumed".Quoted 5 lines
qapi/qobject-input-visitor.c +2
-0+} +static void qobject_input_type_bool (Visitor *v,const char *name,bool *obj,Error **errp) {
[...]
Re: [Qemu-devel] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor
Quoted 65 lines · Markus Armbruster
Quoted 22 lines · Markus Armbruster
Quoted 3 lines · Daniel P. Berrange
Currently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using
i.e.
Quoted 14 lines
QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.
To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.
Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>[...]
Quoted 40 lines
Quoted 13 lines
qapi/qobject-input-visitor.c +1
-0@@ -20,6 +20,7 @@#include "qemu-common.h" #include "qapi/qmp/types.h" #include "qapi/qmp/qerror.h" +#include "qemu/cutils.h" #define QIV_STACK_SIZE 1024qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 273
Quoted 10 lines
qapi/qobject-input-visitor.c +7
-0@@ -263,6 +264,28 @@ static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, *obj =qint_get_int (qint); } + +static void qobject_input_type_int64_autocast (Visitor *v,const char *name, +int64_t *obj,Error **errp) +{ +QObjectInputVisitor *qiv =to_qiv (v); +QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true));Needs a rebase for commit 1382d4a. Same elsewhere.
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 276
Quoted 3 lines
qapi/qobject-input-visitor.c +3
-0+int64_t ret; + +if (!qstr || !qstr->string) {I don't think !qstr->string can happen. Same elsewhere.
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 280
Quoted 4 lines
qapi/qobject-input-visitor.c +4
-0+error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" , +"string" ); +return ; + }So far, this is basically
qobject_input_type_str()less theg_strdup(). Worth factoring out?
Quoted 7 lines
Quoted 5 lines
Now we're entering out parsing swamp:
Quoted 3 lines
qapi/qobject-input-visitor.c +3
-0+ +if (qemu_strtoll (qstr->string,NULL ,0 , &ret) <0 ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" );"integer", actually.
Wait! You're using QERR_INVALID_PARAMETER_VALUE here, so it's "an integer".
To be pedantically correct, we'd have to complain about the type when qemu_strtoll() fails to parse, and about the value when it parses okay, but the value is out of range.
Quoted 46 lines
Quoted 46 lines
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 285
Quoted 2 lines
qapi/qobject-input-visitor.c +2
-0+return ; + }To serve as replacement for the options visitor, this needs to parse exactly like
opts_type_int64().It should also match the JSON parser as far as possible, to minimize difference between the two QObject input visitor variants, and the QemuOpts parser, for command line consistency.
opts_type_int64()usesstrtoll()directly. It carefully checks for no conversion (both ways, EINVAL and endptr == str), range, and string not fully consumed.Your code uses
qemu_strtoll(). Bug#1:qemu_strtoll()assumes long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values. Bug#2: your code fails to check the string is fully consumed.The JSON parser also uses
strtoll()directly, inparse_literal(). When we get there, we know that the string consists only of decimal digits, possibly prefixed by a minus sign. Therefore,strtoll()can only fail with ERANGE, andparse_literal()handles that correctly.QemuOpts doesn't do signed integers.
Quoted 20 lines
qapi/qobject-input-visitor.c +13
-0+ *obj = ret; +} +static void qobject_input_type_uint64 (Visitor *v,const char *name,uint64_t *obj,Error **errp) {@@ -279,6 +302,27 @@ static void qobject_input_type_uint64(Visitor *v, const char *name, *obj =qint_get_int (qint); } +static void qobject_input_type_uint64_autocast (Visitor *v,const char *name, +uint64_t *obj,Error **errp) +{ +QObjectInputVisitor *qiv =to_qiv (v); +QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true)); +unsigned long long ret; + +if (!qstr || !qstr->string) { + error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
Quoted 8 lines
Quoted 6 lines
qapi/qobject-input-visitor.c static void qobject_input_type_uint64(Visitor *v, const char *name, · line 320
Quoted 6 lines
qapi/qobject-input-visitor.c +6
-0+ "string"); +return ; + } + +if (parse_uint_full (qstr->string, &ret,0 ) <0 ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" );"integer".
Likewise.
Quoted 38 lines
Quoted 37 lines
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 286
Quoted 3 lines
qapi/qobject-input-visitor.c +3
-0+return ; + } + *obj = ret;Differently wrong :)
To serve as replacement for the options visitor, this needs to parse exactly like
opts_type_uint64().Again, this should also match the JSON parser and the QemuOpts parser as far as possible.
opts_type_uint64()usesparse_uint(). It carefully checks for no conversion (EINVAL;parse_uint()normalizes), range, and string not fully consumed.You use
parse_uint_full(). You therefore don't have bug#2 here. You do have bug#1: you assume unsigned long long is exactly 64 bits. If it's wider, we fail to diagnose overflow. If it's narrower, we can't parse all values.There's a discrepancy with the JSON parser, but it's not your fault: the JSON parser represents JSON numbers that fit into
int64_tto QInt, and any others as QFloat. In particular, the integers between INT64_MAX+1 andUINT64_MAXare represented as QFloat.qmp_input_type_uint64()accepts only QInt. You can declare things as uint64 in the schema, but you're still limited to 0..UINT64_MAX in QMP.QemuOpts uses
strtoull()directly, inparse_option_number(). Bug (not yours): it happily ignores errors other than "string not fully consumed".Quoted 5 lines
qapi/qobject-input-visitor.c +2
-0+} +static void qobject_input_type_bool (Visitor *v,const char *name,bool *obj,Error **errp) {[...]
Re: [Qemu-devel] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor
Quoted 73 lines · Markus Armbruster
Quoted 71 lines · Markus Armbruster
Quoted 22 lines · Markus Armbruster
Quoted 3 lines · Daniel P. Berrange
Currently the QObjectInputVisitor assumes that all scalar values are directly represented as the final types declared by the thing being visited. ie it assumes an 'int' is using
i.e.
Quoted 14 lines
QInt, and a 'bool' is using QBool, etc. This is good when QObjectInputVisitor is fed a QObject that came from a JSON document on the QMP monitor, as it will strictly validate correctness.
To allow QObjectInputVisitor to be reused for visiting a QObject originating from QemuOpts, an alternative mode is needed where all the scalars types are represented as QString and converted on the fly to the final desired type.
Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>[...]
Quoted 45 lines
Quoted 13 lines
qapi/qobject-input-visitor.c +1
-0@@ -20,6 +20,7 @@#include "qemu-common.h" #include "qapi/qmp/types.h" #include "qapi/qmp/qerror.h" +#include "qemu/cutils.h" #define QIV_STACK_SIZE 1024qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 273
Quoted 10 lines
qapi/qobject-input-visitor.c +7
-0@@ -263,6 +264,28 @@ static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, *obj =qint_get_int (qint); } + +static void qobject_input_type_int64_autocast (Visitor *v,const char *name, +int64_t *obj,Error **errp) +{ +QObjectInputVisitor *qiv =to_qiv (v); +QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true));Needs a rebase for commit 1382d4a. Same elsewhere.
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 276
Quoted 3 lines
qapi/qobject-input-visitor.c +3
-0+int64_t ret; + +if (!qstr || !qstr->string) {I don't think !qstr->string can happen. Same elsewhere.
qapi/qobject-input-visitor.c static void qobject_input_type_int64(Visitor *v, const char *name, int64_t *obj, · line 280
Quoted 4 lines
qapi/qobject-input-visitor.c +4
-0+error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" , +"string" ); +return ; + }So far, this is basically
qobject_input_type_str()less theg_strdup(). Worth factoring out?Now we're entering out parsing swamp:
Quoted 3 lines
qapi/qobject-input-visitor.c +3
-0+ +if (qemu_strtoll (qstr->string,NULL ,0 , &ret) <0 ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" );
Quoted 8 lines
Quoted 1 line
"integer", actually.
Wait! You're using
QERR_INVALID_PARAMETER_VALUEhere, so it's "an integer".To be pedantically correct, we'd have to complain about the type when
qemu_strtoll()fails to parse, and about the value when it parses okay, but the value is out of range.
Nevermind, I got confused. The type is actually always string here, so your use of QERR_INVALID_PARAMETER_VALUE is appropriate.
[...]
Re: [Qemu-devel] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor
In reply to Daniel P. BerrangeQuoted 88 lines · Daniel P. Berrange
The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:
-arg foo=5,foo=6,foo=7
-arg foo=5-7This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.
This means that
-arg foo=5-7
is treated as equivalent to
-arg foo.0=5,foo.1=6,foo.2=7
Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.
Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qobject-input-visitor.h | 23 ++++- qapi/qobject-input-visitor.c | 158 ++++++++++++++++++++++++++-- tests/test-qobject-input-visitor.c | 195 +++++++++++++++++++++++++++++++++-- 3 files changed, 360 insertions(+), 16 deletions(-)include/qapi/qobject-input-visitor.h +22
-1@@ -19,6 +19,12 @@typedef struct QObjectInputVisitor QObjectInputVisitor ; +/* Inclusive upper bound on the size of any flattened range. This is a safety +* (= anti-annoyance) measure; wrong ranges should not cause long startup +* delays nor exhaust virtual memory. +*/ +#define QIV_RANGE_MAX 65536 +/** * Create a new input visitor that converts @obj to a QAPI object. * @@ -71,6 +77,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);* The value given determines how many levels of structs are allowed to * be flattened in this way. * +* If @permit_int_ranges is true, then when visiting a list of integers, +* the integer value strings may encode ranges eg a single element +* containing "5-7" is treated as if there were three elements "5", "6", +* "7". This should only be used if compatibility is required with the +* OptsVisitor which would allow integer ranges. e.g. set this to true +* if you have compatibility requirements for +* +* -arg val=5-8 +* +* to be treated as equivalent to the preferred syntax: +* +* -arg val.0=5,val.1=6,val.2=7,val.3=8 +* * The visitor always operates in strict mode, requiring all dict keys * to be consumed during visitation. An error will be reported if this * does not happen. @@ -80,7 +99,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);*/ Visitor *qobject_input_visitor_new_autocast (QObject *obj,bool autocreate_list, -size_t autocreate_struct_levels); +size_t autocreate_struct_levels, +bool permit_int_ranges);/** @@ -98,6 +118,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts,bool autocreate_list,size_t autocreate_struct_levels, +bool permit_int_ranges,Error **errp);#endif qapi/qobject-input-visitor.c +0
-0
qapi/qobject-input-visitor.c typedef struct StackObject · line 35
Quoted 5 lines
qapi/qobject-input-visitor.c +2
-0@@ -31,6 +31,8 @@ typedef struct StackObjectGHashTable *h;/* If obj is dict: unvisited keys */ const QListEntry *entry;/* If obj is list: unvisited tail */ +uint64_t range_val; +uint64_t range_limit;
It's either ugly unions or ugly type casts. The options visitor picked ugly unions, you're picking ugly type casts. Matter of taste, as long as the type casts are all kosher.
Quoted 32 lines
qapi/qobject-input-visitor.c +7
-3QSLIST_ENTRY (StackObject) node; } StackObject;@@ -60,6 +62,10 @@ struct QObjectInputVisitor* consider auto-creating a struct containing * remaining unvisited items */ size_t autocreate_struct_levels; + +/* Whether int lists can have single values representing +* ranges of values */ +bool permit_int_ranges; };static QObjectInputVisitor *to_qiv (Visitor *v)@@ -282,7 +288,7 @@ static GenericList *qobject_input_next_list(Visitor *v, GenericList *tail,QObjectInputVisitor *qiv =to_qiv (v);StackObject *so =QSLIST_FIRST (&qiv->stack); -if (!so->entry) { +if ((so->range_val == so->range_limit) && !so->entry) {return NULL ; } tail->next =g_malloc0 (size);@@ -329,21 +335,87 @@ static void qobject_input_type_int64_autocast(Visitor *v, const char *name, int64_t *obj, Error **errp) { QObjectInputVisitor *qiv = to_qiv(v); -QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, - true)); +QString *qstr;int64_t ret; +const char *end =NULL ;
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 350
Quoted 10 lines
qapi/qobject-input-visitor.c +10
-0+StackObject *tos; +bool inlist = false; + +/* Preferentially generate values from a range, before +* trying to consume another QList element */ + tos =QSLIST_FIRST (&qiv->stack); +if (tos) { +if ((int64_t )tos->range_val < (int64_t )tos->range_limit) { + *obj = tos->range_val +1 ; + tos->range_val++;
Roundabout way to write
*obj = tos->range_val++;
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 355
Quoted 5 lines
qapi/qobject-input-visitor.c +5
-0+return ; + }else { + inlist = tos->entry !=NULL ; + } + }
I'd prefer
tos = QSLIST_FIRST(&qiv->stack);
inlist = tos && tos->entry;
if (inlist) {
if ((int64_t)tos->range_val < (int64_t)tos->range_limit) {
*obj = tos->range_val++;
return;
}
}Note for later:
!tos || (int64_t)tos->range_val >= (int64_t)tos->range_limit
holds here.
Quoted 24 lines
qapi/qobject-input-visitor.c +12
-1+ qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true));if (!qstr || !qstr->string) {error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" ,"string" );return ; } -if (qemu_strtoll (qstr->string,NULL ,0 , &ret) <0 ) { +if (qemu_strtoll (qstr->string, &end,0 , &ret) <0 ) {error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" );return ; } *obj = ret; + +/* +* If we have string that represents an integer range (5-24), +* parse the end of the range and set things up so we'll process +* the rest of the range before consuming another element +* from the QList. +*/ +if (end && *end) { +if (!qiv->permit_int_ranges) {
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 388
Quoted 10 lines
qapi/qobject-input-visitor.c +10
-0+error_setg (errp, +"Integer ranges are not permitted here" ); +return ; + } +if (!inlist) { +error_setg (errp, +"Integer ranges are only permitted when " +"visiting list parameters" ); +return ; + }
Note for later: since inlist is tos && tos->entry, we know tos is non-null here. Recommend assert(tos) to make it more obvious.
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 391
Quoted 3 lines
qapi/qobject-input-visitor.c +3
-0+if (*end !='-' ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name, +"a number range" );
"an integer or integer range", actually.
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 396
Quoted 5 lines
qapi/qobject-input-visitor.c +5
-0+return ; + } + end++; +if (qemu_strtoll (end,NULL ,0 , &ret) <0 ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" );
"an integer".
Quoted 7 lines
qapi/qobject-input-visitor.c +7
-0+return ; + } +if (*obj > ret) { +error_setg (errp, +"Parameter '%s' range start %" PRIu64 +" must be less than (or equal to) %" PRIu64, + name, *obj, ret);
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 413
Quoted 10 lines
qapi/qobject-input-visitor.c +10
-0+return ; + } + +if ((*obj <= (INT64_MAX -QIV_RANGE_MAX )) && + (ret >= (*obj +QIV_RANGE_MAX ))) { +error_setg (errp, +"Parameter '%s' range must be less than %d" , + name,QIV_RANGE_MAX ); +return ; + }
I see you improved the error messages over the option visitor's. Good.
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 417
Quoted 4 lines
qapi/qobject-input-visitor.c +4
-0+if (*obj != ret) { + tos->range_val = *obj; + tos->range_limit = ret; + }
Does this need to be conditional?
Dereferencing tos is safe (see note above).
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 417
Quoted 1 line
qapi/qobject-input-visitor.c +1
-0+ }
No else, thus tos->range_val and tos->range_limit remain unchanged. Works because we know that !tos || tos->range_val >= tos->range_limit (see note above). If it wasn't, the next visit wouldn't consume another object as it should. Considering the distance to the place that ensures this, I'd prefer to spell this out here with an assertion, or perhaps by assigning to tos->range_val and tos->range_limit.
qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 418
Quoted 1 line
qapi/qobject-input-visitor.c +0
-0}
Overall, less hairy than the options visitor.
Quoted 79 lines
qapi/qobject-input-visitor.c +58
-3static void qobject_input_type_uint64(Visitor *v, const char *name,@@ -366,21 +438,85 @@ static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, uint64_t *obj, Error **errp) { QObjectInputVisitor *qiv = to_qiv(v); -QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, - true)); +QString *qstr;unsigned long long ret; +char *end =NULL ; +StackObject *tos; +bool inlist = false; + +/* Preferentially generate values from a range, before +* trying to consume another QList element */ + tos =QSLIST_FIRST (&qiv->stack); +if (tos) { +if (tos->range_val < tos->range_limit) { + *obj = tos->range_val +1 ; + tos->range_val++; +return ; + }else { + inlist = tos->entry !=NULL ; + } + } + qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true));if (!qstr || !qstr->string) {error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" ,"string" );return ; } -if (parse_uint_full (qstr->string, &ret,0 ) <0 ) { +if (parse_uint (qstr->string, &ret, &end,0 ) <0 ) {error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" );return ; } *obj = ret; + +/* +* If we have string that represents an integer range (5-24), +* parse the end of the range and set things up so we'll process +* the rest of the range before consuming another element +* from the QList. +*/ +if (end && *end) { +if (!qiv->permit_int_ranges) { +error_setg (errp, +"Integer ranges are not permitted here" ); +return ; + } +if (!inlist) { +error_setg (errp, +"Integer ranges are only permitted when " +"visiting list parameters" ); +return ; + } +if (*end !='-' ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name, +"a number range" ); +return ; + } + end++; +if (parse_uint_full (end, &ret,0 ) <0 ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" ); +return ; + } +if (*obj > ret) { +error_setg (errp, +"Parameter '%s' range start %" PRIu64 +" must be less than (or equal to) %llu" , + name, *obj, ret); +return ; + } +if ((ret - *obj) > (QIV_RANGE_MAX -1 )) { + error_setg(errp,
qapi/qobject-input-visitor.c static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, · line 520
Quoted 10 lines
qapi/qobject-input-visitor.c +9
-0+"Parameter '%s' range must be less than %d" , + name, QIV_RANGE_MAX); +return ; + } +if (*obj != ret) { + tos->range_val = *obj; + tos->range_limit = ret; + } + } }
Duplicates the signed code, which is sad, but I don't have better ideas.
Except this one: are we actually using both the signed and the unsigned case now? If not, can we get rid of the one we don't use?
Quoted 29 lines
qapi/qobject-input-visitor.c +4
-1static void qobject_input_type_bool(Visitor *v, const char *name, bool *obj,@@ -576,7 +712,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)Visitor *qobject_input_visitor_new_autocast (QObject *obj,bool autocreate_list, -size_t autocreate_struct_levels) +size_t autocreate_struct_levels, +bool permit_int_ranges) {QObjectInputVisitor *v;@@ -603,6 +740,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj, v->strict = true; v->autocreate_list = autocreate_list; v->autocreate_struct_levels = autocreate_struct_levels; + v->permit_int_ranges = permit_int_ranges; v->root = obj;qobject_incref (obj);@@ -614,6 +752,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts,bool autocreate_list,size_t autocreate_struct_levels, +bool permit_int_ranges,Error **errp) {QDict *pdict;
qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 778
Quoted 9 lines
qapi/qobject-input-visitor.c +2
-1@@ -632,7 +771,8 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, v =qobject_input_visitor_new_autocast (pobj, autocreate_list, - autocreate_struct_levels); + autocreate_struct_levels, + permit_int_ranges); cleanup:qobject_decref (pobj);QDECREF (pdict);
[Skipping test updates for now...]
Re: [Qemu-devel] [Qemu-block] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor
Am 12.10.2016 um 17:50 hat Markus Armbruster geschrieben:
Quoted 47 lines
Quoted 32 lines · Daniel P. Berrange
The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:
-arg foo=5,foo=6,foo=7
-arg foo=5-7This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.
This means that
-arg foo=5-7
is treated as equivalent to
-arg foo.0=5,foo.1=6,foo.2=7
Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.
Signed-off-by Daniel P. Berrange <[email protected]>qapi/qobject-input-visitor.c +2
-2@@ -329,21 +335,87 @@ static void qobject_input_type_int64_autocast(Visitor *v, const char *name, int64_t *obj, Error **errp) { QObjectInputVisitor *qiv = to_qiv(v); -QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, - true)); +QString *qstr;int64_t ret; +const char *end =NULL ;qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 350
Quoted 10 lines
qapi/qobject-input-visitor.c +10
-0+StackObject *tos; +bool inlist = false; + +/* Preferentially generate values from a range, before +* trying to consume another QList element */ + tos =QSLIST_FIRST (&qiv->stack); +if (tos) { +if ((int64_t )tos->range_val < (int64_t )tos->range_limit) { + *obj = tos->range_val +1 ; + tos->range_val++;Roundabout way to write
Quoted 1 line
*obj = tos->range_val++;
*obj = ++tos->range_val, actually.
Kevin
Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options
In reply to Daniel P. BerrangeQuoted 31 lines · Daniel P. Berrange
If given an option string such as
size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind
the
qemu_opts_to_qdict()method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:size=QString("1024") nodes=QString("1-2") policy=QString("bind")With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain
size=QString("1024") nodes=QList([QString("10"), QString("4-5"), QString("1-2")]) policy=QString("bind")Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.
Quoted 3 lines
In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.
To serve as a replacement for the options visitor, this needs to be able to behave exactly the same together with a suitably hacked up QObject input visitor. Before I dive into the actual patch, let me summarize QemuOpts and options visitor behavior.
Warning, this is going to get ugly.
QemuOpts faithfully represents a key=value,... string as a list of QemuOpt. Each QemuOpt represents one key=value. They are in the same order. If key occurs multiple times in the string, it occurs just the same in the list.
*Except* key "id" is special: it's stored outside the list, and all but the first one are silently ignored.
Most users only ever get the last value of a key. Any non-last key=value are silently ignored.
We actually exploit this behavior to do defaults, by *prepending* them to the list. See the use of qemu_opts_set_defaults() in main().
A few users get all values of keys (other than key "id"):
* -device, in qdev_device_add() with callback set_property().
We first get "driver" and "bus" normally (silently ignoring non-last values, as usual). All other keys are device properties. To set them, we get all (key, value), ignore keys "driver" and "bus", and set the rest. If a key occurs multiple times, it gets set multiple times. This effectively ignores all but the last one, silently.
* -semihosting-config, in main() with callback add_semihosting_arg().
We first get a bunch of keys normally. Key "arg" is special: it may be repeated to build a list. To implement that, we get all (key, value), ignore keys other than "arg", and accumulate the values.
* -machine & friends, in main() with callback machine_set_property()
Similar to -device, only for machines, with "type" instead of "driver" and "bus".
* -spice, in qemu_spice_init() with callback add_channel()
Keys "tls-channel" and "plaintext-channel" may be used repeated to specify multiple channels. To implement that, we get all (key, value), ignore keys other than "tls-channel" and "plaintext-channel", and set up a channel for each of the others.
* -writeconfig, in config_write_opts() with callback config_write_opt()
We write out all keys in order.
* The options visitor, in opts_start_struct()
We convert the list of (key, value) to a hash table of (key, list of values). Most of the time, the list of values has exactly one element.
When the visitor's user asks for a scalar, we return the last element of the list of values, in lookup_scalar().
When the user asks for list elements, we return the elements of the list of values in order, in opts_next_list(), or if there are none, the empty list in opts_start_list().
Unlike the options visitor, this patch (judging from your description) makes a list only when keys are repeated. The QObject visitor will have to cope with finding both scalars and lists. When it finds a scalar, but needs a list, it'll have to wrap it in a list (PATCH 09, I think). When it finds a list, but needs a scalar, it'll have to fish it out of the list (where is that?).
Quoted 5 lines
All existing callers are all converted to explicitly request the historical behaviour of only reporting the last key. Later patches will make use of the new modes.
Signed-off-by Daniel P. Berrange <[email protected]>
Out of steam for today.
Re: [Qemu-devel] [Qemu-block] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor
In reply to Kevin WolfQuoted 48 lines · Kevin Wolf
Am 12.10.2016 um 17:50 hat Markus Armbruster geschrieben:
Quoted 47 lines
Quoted 32 lines · Daniel P. Berrange
The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:
-arg foo=5,foo=6,foo=7
-arg foo=5-7This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.
This means that
-arg foo=5-7
is treated as equivalent to
-arg foo.0=5,foo.1=6,foo.2=7
Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.
Signed-off-by Daniel P. Berrange <[email protected]>qapi/qobject-input-visitor.c +2
-2@@ -329,21 +335,87 @@ static void qobject_input_type_int64_autocast(Visitor *v, const char *name, int64_t *obj, Error **errp) { QObjectInputVisitor *qiv = to_qiv(v); -QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, - true)); +QString *qstr;int64_t ret; +const char *end =NULL ;qapi/qobject-input-visitor.c static void qobject_input_type_int64_autocast(Visitor *v, const char *name, · line 350
Quoted 10 lines
qapi/qobject-input-visitor.c +10
-0+StackObject *tos; +bool inlist = false; + +/* Preferentially generate values from a range, before +* trying to consume another QList element */ + tos =QSLIST_FIRST (&qiv->stack); +if (tos) { +if ((int64_t )tos->range_val < (int64_t )tos->range_limit) { + *obj = tos->range_val +1 ; + tos->range_val++;Roundabout way to write
Quoted 3 lines
Quoted 1 line
*obj = tos->range_val++;*obj = ++tos->range_val, actually.
Of course, thanks.
Re: [Qemu-devel] [PATCH v14 08/21] qapi: allow QObjectInputVisitor to be created with QemuOpts
In reply to Markus ArmbrusterQuoted 18 lines · Markus Armbruster
Quoted 1 line · Daniel P. Berrange
Instead of requiring all callers to go through the mutli-step
multi-step
Quoted 12 lines
process of turning QemuOpts into a suitable QObject for visiting, add a new constructor that encapsulates this logic. This will allow QObjectInputVisitor to be a drop-in replacement for the existing OptsVisitor with minimal code changes for callers.
NB, at this point it is only supporting opts syntax which explicitly matches the QAPI schema structure, so is not yet a true drop-in replacement for OptsVisitor. The patches that follow will add the special cases requird for full backwards compatibility with OptsVisitor.
Signed-off-by Daniel P. Berrange <[email protected]>
[...]
Quoted 6 lines
Quoted 6 lines
qapi/qobject-input-visitor.c +0
-0@@ -545,3 +545,28 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
Quoted 18 lines
qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_autocast(QObject *obj) · line 554
Quoted 9 lines
qapi/qobject-input-visitor.c +7
-0return &v->visitor; } + + +Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts, +Error **errp) +{ +QDict *pdict; +QObject *pobj =NULL ;@pdict and @pobj are unusual names. Let's stick to the more common @dict and @obj.
Quoted 5 lines
qapi/qobject-input-visitor.c +5
-0+Visitor *v =NULL ; + + pdict =qemu_opts_to_qdict (opts,NULL ); +if (!pdict) { + goto cleanup;
Returns null without setting an error, which is wrong. Fortunately, qemu_opts_to_qdict() cannot fail. Please drop the broken error handling.
Quoted 13 lines
Quoted 13 lines
qapi/qobject-input-visitor.c +13
-0+ } + + pobj =qdict_crumple (pdict, true, errp); +if (!pobj) { + goto cleanup; + } + + v =qobject_input_visitor_new_autocast (pobj); + cleanup: +qobject_decref (pobj); +QDECREF (pdict); +return v; +}
[...]
Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options
In reply to Daniel P. BerrangeQuoted 23 lines · Daniel P. Berrange
If given an option string such as
size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind
the
qemu_opts_to_qdict()method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:size=QString("1024") nodes=QString("1-2") policy=QString("bind")With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain
size=QString("1024") nodes=QList([QString("10"), QString("4-5"), QString("1-2")]) policy=QString("bind")
Quoted 7 lines
Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.
Actually three cases, not two:
0. qdict does not contain the key means empty list.
1. qdict contains the key with a QString value means list of one element.
2. qdict contains the key with a QList value means list of more than one element.
I consider this weird. However, it's usefully weird with at least your QObject input visitor.
Quoted 3 lines
In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.
Got users for this policy in the pipeline?
Quoted 50 lines
All existing callers are all converted to explicitly request the historical behaviour of only reporting the last key. Later patches will make use of the new modes.
Signed-off-by Daniel P. Berrange <[email protected]>--- blockdev.c | 7 ++- include/qemu/option.h | 13 ++++- monitor.c | 3 +- qapi/qobject-input-visitor.c | 4 +- qemu-img.c | 4 +- qemu-io-cmds.c | 3 +- qemu-io.c | 6 +- qemu-nbd.c | 3 +- qom/object_interfaces.c | 3 +- tests/test-qemu-opts.c | 132 +++++++++++++++++++++++++++++++++++++++++++ tests/test-replication.c | 9 ++- util/qemu-option.c | 64 ++++++++++++++++++--- 12 files changed, 229 insertions(+), 22 deletions(-)blockdev.c +4
-3@@ -914,7 +914,8 @@ DriveInfo *drive_new(QemuOpts *all_opts, BlockInterfaceType block_default_type)/* Get a QDict for processing the options */ bs_opts =qdict_new (); -qemu_opts_to_qdict (all_opts, bs_opts); +qemu_opts_to_qdict (all_opts, bs_opts,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort); legacy_opts =qemu_opts_create (&qemu_legacy_drive_opts,NULL ,0 , &error_abort);@@ -3804,8 +3805,8 @@ void hmp_drive_add_node(Monitor *mon, const char *optstr)return ; } - qdict =qemu_opts_to_qdict (opts,NULL ); - + qdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);if (!qdict_get_try_str (qdict,"node-name" )) {QDECREF (qdict);error_report ("'node-name' needs to be specified" );include/qemu/option.h +0
-0
include/qemu/option.h void qemu_opts_set_defaults(QemuOptsList *list, const char *params, · line 128
Quoted 4 lines
include/qemu/option.h +0
-1@@ -125,7 +125,18 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char *params, int permit_abbrev);QemuOpts *qemu_opts_from_qdict (QemuOptsList *list,const QDict *qdict,Error **errp); -QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict);
Blank line here, please.
Quoted 34 lines
include/qemu/option.h +12
-0+typedef enum { +/* Last occurrence of a duplicate option silently replaces earlier */ +QEMU_OPTS_REPEAT_POLICY_LAST , +/* Each occurrence of a duplicate option converts value to a QList */ +QEMU_OPTS_REPEAT_POLICY_ALL , +/* First occurrence of a duplicate option causes an error */ +QEMU_OPTS_REPEAT_POLICY_ERROR , +}QemuOptsRepeatPolicy ; + +QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict, +QemuOptsRepeatPolicy repeatPolicy, +Error **errp);void qemu_opts_absorb_qdict (QemuOpts *opts,QDict *qdict,Error **errp);typedef int (*qemu_opts_loopfunc )(void *opaque,QemuOpts *opts,Error **errp);monitor.c +2
-1@@ -2642,7 +2642,8 @@ static QDict *monitor_parse_arguments(Monitor *mon,if (!opts) { goto fail; } -qemu_opts_to_qdict (opts, qdict); +qemu_opts_to_qdict (opts, qdict,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);qemu_opts_del (opts); }break ;qapi/qobject-input-visitor.c +0
-0
qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 763
Quoted 6 lines
qapi/qobject-input-visitor.c +2
-1@@ -759,7 +759,9 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,QObject *pobj =NULL ;Visitor *v =NULL ; - pdict =qemu_opts_to_qdict (opts,NULL ); + pdict = qemu_opts_to_qdict(opts, NULL, + QEMU_OPTS_REPEAT_POLICY_LAST,
Join the previous two lines, please.
qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 764
Quoted 1 line
qapi/qobject-input-visitor.c +1
-0+ errp);
Unlike the other callers, this one doesn't pass &error_abort. The error handling is actually dead code. Let's pass &error_abort and be done with it.
Quoted 23 lines
qapi/qobject-input-visitor.c +0
-0if (!pdict) { goto cleanup; }qemu-img.c +3
-1@@ -273,7 +273,9 @@ static BlockBackend *img_open_opts(const char *optstr,QDict *options;Error *local_err =NULL ;BlockBackend *blk; - options =qemu_opts_to_qdict (opts,NULL ); + options =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort); + blk =blk_new_open (NULL ,NULL , options, flags, &local_err);if (!blk) {error_reportf_err (local_err,"Could not open '%s': " , optstr);qemu-io-cmds.c +0
-0
qemu-io-cmds.c static int reopen_f(BlockBackend *blk, int argc, char **argv) · line 1956
Quoted 6 lines
qemu-io-cmds.c +2
-1@@ -1952,7 +1952,8 @@ static int reopen_f(BlockBackend *blk, int argc, char **argv) } qopts =qemu_opts_find (&reopen_opts,NULL ); - opts = qopts ?qemu_opts_to_qdict (qopts,NULL ) :NULL ; + opts = qopts ?qemu_opts_to_qdict (qopts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort) :NULL ;
Short ternaries are more legible than if statements, but this one is pushing the limit now.
Quoted 8 lines
qemu-io-cmds.c +0
-0qemu_opts_reset (&reopen_opts); brq =bdrv_reopen_queue (NULL , bs, opts, flags);qemu-io.c +0
-0
qemu-io.c static int open_f(BlockBackend *blk, int argc, char **argv) · line 211
Quoted 6 lines
qemu-io.c +2
-1@@ -207,7 +207,8 @@ static int open_f(BlockBackend *blk, int argc, char **argv) } qopts =qemu_opts_find (&empty_opts,NULL ); - opts = qopts ?qemu_opts_to_qdict (qopts,NULL ) :NULL ; + opts = qopts ?qemu_opts_to_qdict (qopts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort) :NULL ;
Likewise.
qemu-io.c int main(int argc, char **argv) · line 598
Quoted 10 lines
qemu-io.c +2
-1qemu_opts_reset (&empty_opts);if (optind == argc -1 ) {@@ -593,7 +594,8 @@ int main(int argc, char **argv)if (!qopts) {exit (1 ); } - opts =qemu_opts_to_qdict (qopts,NULL ); + opts =qemu_opts_to_qdict (qopts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);
Even this line's getting ugly. Can we find a prefix that's less loquacious than QEMU_OPTS_REPEAT_POLICY_, but still self-explaining?
Quoted 36 lines
qemu-io.c +0
-0openfile (NULL , flags, writethrough, opts); }else {if (format) {qemu-nbd.c +2
-1@@ -859,7 +859,8 @@ int main(int argc, char **argv)qemu_opts_reset (&file_opts);exit (EXIT_FAILURE ); } - options =qemu_opts_to_qdict (opts,NULL ); + options =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);qemu_opts_reset (&file_opts); blk =blk_new_open (NULL ,NULL , options, flags, &local_err); }else {qom/object_interfaces.c +2
-1@@ -161,7 +161,8 @@ Object *user_creatable_add_opts(QemuOpts *opts, Error **errp)Object *obj =NULL ; v =opts_visitor_new (opts); - pdict =qemu_opts_to_qdict (opts,NULL ); + pdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort); obj =user_creatable_add (pdict, v, errp);visit_free (v);tests/test-qemu-opts.c +0
-0
tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 430
Quoted 10 lines
tests/test-qemu-opts.c +7
-0@@ -421,6 +421,130 @@ static void test_qemu_opts_set(void)g_assert (opts ==NULL ); } + +static void test_qemu_opts_to_qdict_repeat_last (void ) +{ +QemuOpts *opts; +QDict *dict; + +/* dynamically initialized (parsed) opts */
What are you trying to say with this comment? Hmm, you merely copied it from the existing tests. I'd drop the existing instances of this comment instead.
tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 434
Quoted 4 lines
tests/test-qemu-opts.c +4
-0+ opts =qemu_opts_parse (&opts_list_03, +"size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind" , + false,NULL ); +g_assert (opts);
You could pass &error_abort.
tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 440
Quoted 6 lines
tests/test-qemu-opts.c +6
-0+ + dict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort); +g_assert (dict); + +
One blank line, not two, please.
tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 446
Quoted 6 lines
tests/test-qemu-opts.c +6
-0+ g_assert_cmpstr(qdict_get_str(dict, "size"), ==, "1024"); + g_assert_cmpstr(qdict_get_str(dict, "nodes"), ==, "1-2"); +g_assert (!qdict_haskey (dict,"nodes.0" )); +g_assert (!qdict_haskey (dict,"nodes.1" )); +g_assert (!qdict_haskey (dict,"nodes.2" )); + g_assert_cmpstr(qdict_get_str(dict, "policy"), ==, "bind");
Checking for absence of "nodes.0"... feels odd. A better idea is to check we got exactly the keys we expect, and no more. Here's a simple way to do that:
g_assert(qdict_size(dict) == 3);
Quoted 19 lines
tests/test-qemu-opts.c +19
-0+QDECREF (dict); + +qemu_opts_del (opts); +qemu_opts_reset (&opts_list_03); +} + + +static void test_qemu_opts_to_qdict_repeat_all (void ) +{ +QemuOpts *opts; +QDict *dict; +QList *list; +const QListEntry *ent; +QString *str; + +/* dynamically initialized (parsed) opts */ + opts =qemu_opts_parse (&opts_list_03, +"size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind" , + false,NULL );
tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 475
Quoted 10 lines
tests/test-qemu-opts.c +10
-0+g_assert (opts); + + dict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_ALL , + &error_abort); +g_assert (dict); + + g_assert_cmpstr(qdict_get_str(dict, "size"), ==, "1024"); +g_assert (qdict_haskey (dict,"nodes" )); + list =qobject_to_qlist (qdict_get (dict,"nodes" )); +g_assert (list);
Works, but I'd do
obj = qdict_get(dict, "nodes");
g_assert(obj);
list = qobject_to_qlist(obj);
g_assert(list);tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 481
Quoted 6 lines
tests/test-qemu-opts.c +6
-0+ + ent =qlist_first (list); +g_assert (ent); + str =qobject_to_qstring (ent->value); +g_assert (str); + g_assert_cmpstr(qstring_get_str(str), ==, "10");
Since g_assert_cmpstr() does the right thing when @str is null, g_assert(str) is redundant. More of the same below.
Quoted 7 lines
tests/test-qemu-opts.c +7
-0+ + ent =qlist_next (ent); +g_assert (ent); + str =qobject_to_qstring (ent->value); +g_assert (str); + g_assert_cmpstr(qstring_get_str(str), ==, "4-5"); +
tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 498
Quoted 10 lines
tests/test-qemu-opts.c +10
-0+ ent =qlist_next (ent); +g_assert (ent); + str =qobject_to_qstring (ent->value); +g_assert (str); + g_assert_cmpstr(qstring_get_str(str), ==, "1-2"); + + ent =qlist_next (ent); +g_assert (!ent); + + g_assert_cmpstr(qdict_get_str(dict, "policy"), ==, "bind");
g_assert(qdict_size(dict) == 3);
Quoted 33 lines
tests/test-qemu-opts.c +33
-0+QDECREF (dict); + +qemu_opts_del (opts); +qemu_opts_reset (&opts_list_03); +} + +static void test_qemu_opts_to_qdict_repeat_err_fail (void ) +{ +QemuOpts *opts; +QDict *dict; +Error *err =NULL ; + +/* dynamically initialized (parsed) opts */ + opts =qemu_opts_parse (&opts_list_03, +"size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind" , + false,NULL ); +g_assert (opts); + + dict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_ERROR , + &err); +error_free_or_abort (&err); +g_assert (!dict); + +qemu_opts_del (opts); +qemu_opts_reset (&opts_list_03); +} + +static void test_qemu_opts_to_qdict_repeat_err_ok (void ) +{ +QemuOpts *opts; +QDict *dict; + +/* dynamically initialized (parsed) opts */
tests/test-qemu-opts.c static void test_qemu_opts_set(void) · line 541
Quoted 10 lines
tests/test-qemu-opts.c +10
-0+ opts =qemu_opts_parse (&opts_list_03, +"size=1024,nodes=10,policy=bind" , + false,NULL ); +g_assert (opts); + + dict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_ERROR , + &error_abort); + g_assert_cmpstr(qdict_get_str(dict, "size"), ==, "1024"); + g_assert_cmpstr(qdict_get_str(dict, "nodes"), ==, "10"); + g_assert_cmpstr(qdict_get_str(dict, "policy"), ==, "bind");
g_assert(qdict_size(dict) == 3);
Quoted 63 lines
tests/test-qemu-opts.c +14
-0+ +QDECREF (dict); +qemu_opts_del (opts); +qemu_opts_reset (&opts_list_03); +} +int main (int argc,char *argv[]) {register_opts ();@@ -435,6 +559,14 @@ int main(int argc, char *argv[])g_test_add_func ("/qemu-opts/opt_unset" , test_qemu_opt_unset);g_test_add_func ("/qemu-opts/opts_reset" , test_qemu_opts_reset);g_test_add_func ("/qemu-opts/opts_set" , test_qemu_opts_set); +g_test_add_func ("/qemu-opts/to_qdict/repeat-last" , + test_qemu_opts_to_qdict_repeat_last); +g_test_add_func ("/qemu-opts/to_qdict/repeat-all" , + test_qemu_opts_to_qdict_repeat_all); +g_test_add_func ("/qemu-opts/to_qdict/repeat-err-fail" , + test_qemu_opts_to_qdict_repeat_err_fail); +g_test_add_func ("/qemu-opts/to_qdict/repeat-err-ok" , + test_qemu_opts_to_qdict_repeat_err_ok);g_test_run ();return 0 ; }tests/test-replication.c +6
-3@@ -181,7 +181,8 @@ static BlockBackend *start_primary(void) opts =qemu_opts_parse_noisily (&qemu_drive_opts, cmdline, false);g_free (cmdline); - qdict =qemu_opts_to_qdict (opts,NULL ); + qdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);qdict_set_default_str (qdict,BDRV_OPT_CACHE_DIRECT ,"off" );qdict_set_default_str (qdict,BDRV_OPT_CACHE_NO_FLUSH ,"off" );@@ -311,7 +312,8 @@ static BlockBackend *start_secondary(void) opts =qemu_opts_parse_noisily (&qemu_drive_opts, cmdline, false);g_free (cmdline); - qdict =qemu_opts_to_qdict (opts,NULL ); + qdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);qdict_set_default_str (qdict,BDRV_OPT_CACHE_DIRECT ,"off" );qdict_set_default_str (qdict,BDRV_OPT_CACHE_NO_FLUSH ,"off" );@@ -336,7 +338,8 @@ static BlockBackend *start_secondary(void) opts =qemu_opts_parse_noisily (&qemu_drive_opts, cmdline, false);g_free (cmdline); - qdict =qemu_opts_to_qdict (opts,NULL ); + qdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + &error_abort);qdict_set_default_str (qdict,BDRV_OPT_CACHE_DIRECT ,"off" );qdict_set_default_str (qdict,BDRV_OPT_CACHE_NO_FLUSH ,"off" );util/qemu-option.c +0
-0
util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1059
Quoted 3 lines
util/qemu-option.c +0
-0@@ -1057,23 +1057,73 @@ void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp)* The QDict values are of type QString. * TODO We'll want to use types appropriate for opt->desc->type, but * this is enough for now.
While there, we could document @obj and @dict properly. But see below on @dict.
util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1069
Quoted 10 lines
util/qemu-option.c +10
-0+* +* If the @opts contains multiple occurrences of the same key, +* then the @repeatPolicy parameter determines how they are to +* be handled. Traditional behaviour was to only store the +* last occurrence, but if @repeatPolicy is set to +* QEMU_OPTS_REPEAT_POLICY_ALL, then a QList will be returned +* containing all values, for any key with multiple occurrences. +* The QEMU_OPTS_REPEAT_POLICY_ERROR value can be used to fail +* conversion with an error if multiple occurrences of a key +* are seen.
What the traditional behavior was is quite interesting in a commit message. Less so in a comment. Suggest:
* be handled. With @repeatPolicy QEMU_OPTS_REPEAT_POLICY_LAST,
* all values but the last are silently ignored. With
* QEMU_OPTS_REPEAT_POLICY_ALL, a QList will be returned
* containing all values, for any key with multiple occurrences.
* With QEMU_OPTS_REPEAT_POLICY_ERROR, the conversion fails.
Let's spell out that the function can only fail with QEMU_OPTS_REPEAT_POLICY_ERROR:
* This is the only way this function can fail.
util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1076
Quoted 9 lines
util/qemu-option.c +4
-2*/ -QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict) +QDict *qemu_opts_to_qdict (const QemuOpts *opts,QDict *qdict, +QemuOptsRepeatPolicy repeatPolicy, +Error **errp) {QemuOpt *opt; -QObject *val; +QObject *val, *prevval;
I'd prefer prev_val.
util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1084
Quoted 10 lines
util/qemu-option.c +6
-2+QList *list; +QDict *ret; -if (!qdict) { - qdict =qdict_new (); +if (qdict) { + ret = qdict; + }else { + ret =qdict_new (); }
You need this change to get the reference counting on failure right. This makes me question the qdict parameter. I can see two non-null arguments for it outside tests. The one in drive_new() is basically stupid:
bs_opts = qdict_new();
qemu_opts_to_qdict(all_opts, bs_opts, QEMU_OPTS_REPEAT_POLICY_LAST,
&error_abort);Easily replaced by
bs_opts = qemu_opts_to_qdict(all_opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
&error_abort);
The one in monitor_parse_arguments()
qemu_opts_to_qdict(opts, qdict, QEMU_OPTS_REPEAT_POLICY_LAST,
&error_abort);
could be replaced by
QDECREF(qdict);
qdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
&error_abort);at a negligible loss of efficiency. We could then drop the parameter, simplifying the function's contract.
Let's insert a blank line here...
util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1087
Quoted 4 lines
util/qemu-option.c +1
-1if (opts->id) { -qdict_put (qdict,"id" ,qstring_from_str (opts->id)); +qdict_put (ret,"id" ,qstring_from_str (opts->id)); }
... and here.
Quoted 14 lines
util/qemu-option.c +11
-1QTAILQ_FOREACH(opt, &opts->head, next) { val =QOBJECT (qstring_from_str (opt->str)); -qdict_put_obj (qdict, opt->name, val); + +if (qdict_haskey (ret, opt->name)) { +switch (repeatPolicy) { +case QEMU_OPTS_REPEAT_POLICY_ERROR : +error_setg (errp,"Option '%s' appears more than once" , + opt->name); +qobject_decref (val); +if (!qdict) { +QDECREF (ret); + } +return NULL ;
util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1110
Quoted 10 lines
util/qemu-option.c +10
-0+ +case QEMU_OPTS_REPEAT_POLICY_ALL : + prevval =qdict_get (ret, opt->name); +if (qobject_type (prevval) ==QTYPE_QLIST ) { +/* Already identified this key as a list */ + list =qobject_to_qlist (prevval); + }else { +/* Replace current scalar with a list */ + list =qlist_new (); +qobject_incref (prevval);
Where is this reference decremented?
Quoted 4 lines
util/qemu-option.c +4
-0+qlist_append_obj (list, prevval); +qdict_put_obj (ret, opt->name,QOBJECT (list)); + } +qlist_append_obj (list, val);
util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1124
Quoted 10 lines
util/qemu-option.c +10
-0+break ; + +case QEMU_OPTS_REPEAT_POLICY_LAST : +/* Just discard previously set value */ +qdict_put_obj (ret, opt->name, val); +break ; + } + }else { +qdict_put_obj (ret, opt->name, val); + }
A possible alternative:
val = QOBJECT(qstring_from_str(opt->str));
if (qdict_haskey(ret, opt->name)) {
switch (repeatPolicy) {
case QEMU_OPTS_REPEAT_POLICY_ERROR:
error_setg(errp, "Option '%s' appears more than once",
opt->name);
qobject_decref(val);
if (!qdict) {
QDECREF(ret);
}
return NULL;
case QEMU_OPTS_REPEAT_POLICY_ALL:
prevval = qdict_get(ret, opt->name);
if (qobject_type(prevval) == QTYPE_QLIST) {
/* Already identified this key as a list */
list = qobject_to_qlist(prevval);
} else {
/* Replace current scalar with a list */
list = qlist_new();
qobject_incref(prevval);
qlist_append_obj(list, prevval);
}
qlist_append_obj(list, val);
val = QOBJECT(list);
break;
case QEMU_OPTS_REPEAT_POLICY_LAST:
break;
}
}
qdict_put_obj(ret, opt->name, val);This shows the common part of the behavior more clearly. Matter of taste, you get to use your artistic license.
Quoted 6 lines
util/qemu-option.c +1
-1} -return qdict; +return ret; }/* Validate parsed opts against descriptions where no
Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options
In reply to Markus ArmbrusterQuoted 49 lines · Markus Armbruster
Quoted 31 lines · Daniel P. Berrange
If given an option string such as
size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind
the
qemu_opts_to_qdict()method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:size=QString("1024") nodes=QString("1-2") policy=QString("bind")With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain
size=QString("1024") nodes=QList([QString("10"), QString("4-5"), QString("1-2")]) policy=QString("bind")Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.
Quoted 3 lines
In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.
To serve as a replacement for the options visitor, this needs to be able to behave exactly the same together with a suitably hacked up QObject input visitor. Before I dive into the actual patch, let me summarize QemuOpts and options visitor behavior.
Warning, this is going to get ugly.
QemuOpts faithfully represents a key=value,... string as a list of QemuOpt. Each QemuOpt represents one key=value. They are in the same order. If key occurs multiple times in the string, it occurs just the same in the list.
Quoted 8 lines
*Except* key "id" is special: it's stored outside the list, and all but the first one are silently ignored.
Most users only ever get the last value of a key. Any non-last key=value are silently ignored.
We actually exploit this behavior to do defaults, by *prepending* them to the list. See the use of
qemu_opts_set_defaults()inmain().
This prepending of defaults assumes all users ignore values other than the last. It breaks if any user gets non-last values.
Quoted 38 lines
A few users get all values of keys (other than key "id"):
* -device, in
qdev_device_add()with callbackset_property().We first get "driver" and "bus" normally (silently ignoring non-last values, as usual). All other keys are device properties. To set them, we get all (key, value), ignore keys "driver" and "bus", and set the rest. If a key occurs multiple times, it gets set multiple times. This effectively ignores all but the last one, silently.
* -semihosting-config, in
main()with callbackadd_semihosting_arg().We first get a bunch of keys normally. Key "arg" is special: it may be repeated to build a list. To implement that, we get all (key, value), ignore keys other than "arg", and accumulate the values.
* -machine & friends, in
main()with callbackmachine_set_property()Similar to -device, only for machines, with "type" instead of "driver" and "bus".
* -spice, in
qemu_spice_init()with callbackadd_channel()Keys "tls-channel" and "plaintext-channel" may be used repeated to specify multiple channels. To implement that, we get all (key, value), ignore keys other than "tls-channel" and "plaintext-channel", and set up a channel for each of the others.
* -writeconfig, in
config_write_opts()with callbackconfig_write_opt()We write out all keys in order.
* The options visitor, in
opts_start_struct()We convert the list of (key, value) to a hash table of (key, list of values). Most of the time, the list of values has exactly one element.
Quoted 6 lines
When the visitor's user asks for a scalar, we return the last element of the list of values, in
lookup_scalar().When the user asks for list elements, we return the elements of the list of values in order, in
opts_next_list(), or if there are none, the empty list inopts_start_list().
Note that the only way to get non-last values is to iterate over all (key, value). The combination of "getting a specific key's value gets the last one" and "iterating over all keys gets all values" is poor interface design. The latter feature got pressed into service to do list-valued keys. When qemu_opts_set_defaults() got added (commit 4f6dd9a), the bad interaction with the list-valued keys hack wasn't considered, probably because the whole thing had become too byzantine to fully understand.
Quoted 14 lines
Unlike the options visitor, this patch (judging from your description) makes a list only when keys are repeated. The QObject visitor will have to cope with finding both scalars and lists. When it finds a scalar, but needs a list, it'll have to wrap it in a list (PATCH 09, I think). When it finds a list, but needs a scalar, it'll have to fish it out of the list (where is that?).
Quoted 5 lines
All existing callers are all converted to explicitly request the historical behaviour of only reporting the last key. Later patches will make use of the new modes.
Signed-off-by Daniel P. Berrange <[email protected]>Out of steam for today.
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
In reply to Daniel P. BerrangeQuoted 12 lines · Daniel P. Berrange
Currently
qdict_crumplerequires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.In order to have backwards compatibility with the OptsVisitor,
qemu_opt_to_qdict()has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given toqdict_crumple().To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed
Quoted 6 lines
foo=hello,foo=world,foo=wibble foo.0=hello,foo.1=world,foo.2=wibblebut this is forbidden
foo=hello,foo=world,foo.2=wibble
I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.
The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?
Note that this second way creates entirely new failure modes and restrictions. Let me show using an example derived from one in qdict_crumple()'s contract:
foo.0.bar=bla,foo.eek.bar=blubb
Without the dotted key convention, this is perfectly fine: key "foo.0.bar" has the single value "bla", and key "foo.eek.bar" has the single value "blubb". Equivalent JSON would be
{ "foo.0.bar": "bla", "foo.eek.bar": "blubb" }
With just the struct convention, it's still fine: it obviously means the same as JSON
{ "foo": { "0": { "bar": "bla" }, "eek": { "bar": "blubb" } } }
Adding the list convention makes it invalid. It also outlaws a bunch of keys that would be just fine in JSON, namely any that get recognized as list index. Raise your hand if you're willing to bet real money on your predictions of what will be recognized as list index, without looking at the code. I'm not.
I'm afraid I have growing doubts regarding the QemuOpts dotted key convention in general.
The convention makes '.' a special character in keys, but only sometimes. If the key gets consumed by something that uses dotted key convention, '.' is special, and to get a non-special '.', you need to escape it by doubling. Else, it's not.
Since the same key can be used differently by different code, the same '.' could in theory be both special and non-special. In practice, this would be madness.
Adopting the dotted key convention for an existing QemuOpts option, say -object [PATCH 15], *breaks* existing command line usage of keys containing '.', because you now have to escape the '.'. Dan, I'm afraid you need to show that no such keys exist, or if they exist they don't matter.
I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.
It's probably too late to back out the dotted key convention completely. Kevin?
Can we still back out the list part of the convention, and use repeated keys instead?
If we're stuck with some form of the dotted key convention, can we at least make it a more integral part of QemuOpts rather than something bolted on as an afterthought? Here's my thinking on how that might be done:
* Have a QemuOptsList flag @flat.
* If @flat, QemuOpts behaves as it always has: the special characters are ',' and '=', and parsing a key=value,... string produces a list where each element represents one key=value from the string, in the same order.
* If not @flat, '.' becomes an additional special character, and parsing a key=value,... string produces a dictionary, similar to the one you get now by converting with qemu_opts_to_qdict() and filtering through qdict_crumple().
The difference to now is that you either always crumple, or not at all, and the meaning of '.' is unambiguous.
I wish we had refrained from saddling QemuOpts with even more magic. Compared to this swamp, use of JSON on the command line looks rather appealing to me.
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
Am 13.10.2016 um 14:35 hat Markus Armbruster geschrieben:
Quoted 21 lines
Cc Kevin for discussion of QemuOpts dotted key conventionQuoted 17 lines · Daniel P. Berrange
Currently
qdict_crumplerequires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.In order to have backwards compatibility with the OptsVisitor,
qemu_opt_to_qdict()has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given toqdict_crumple().To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed
foo=hello,foo=world,foo=wibble foo.0=hello,foo.1=world,foo.2=wibblebut this is forbidden
Quoted 8 lines
Quoted 1 line
foo=hello,foo=world,foo.2=wibble
I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.
The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?
Probably primarily because someone didn't realise this when introducing the dotted syntax. Also because flat QDicts can't represent this.
But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.
Compare this:
driver=quorum,
child.0.driver=file,child.0.filename=disk1.img,
child.1.driver=host_device,child.1.filename=/dev/sdb,
child.2.driver=nbd,child.2.host=localhostAnd this:
driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost
For starters, can we really trust the order in QemuOpts so that the right driver and filename are associated with each other?
We would also have code to associate the third child.driver with the first child.host (because file and host_device don't have a host property). And this isn't even touching optional arguments yet. Would you really want to implement or review this?
Quoted 15 lines
Note that this second way creates entirely new failure modes and restrictions. Let me show using an example derived from one in qdict_crumple()'s contract:
foo.0.bar=bla,foo.eek.bar=blubb
Without the dotted key convention, this is perfectly fine: key "foo.0.bar" has the single value "bla", and key "foo.eek.bar" has the single value "blubb". Equivalent JSON would be
{ "foo.0.bar": "bla", "foo.eek.bar": "blubb" }
With just the struct convention, it's still fine: it obviously means the same as JSON
Quoted 7 lines
{ "foo": { "0": { "bar": "bla" }, "eek": { "bar": "blubb" } } }
Adding the list convention makes it invalid. It also outlaws a bunch of keys that would be just fine in JSON, namely any that get recognized as list index. Raise your hand if you're willing to bet real money on your predictions of what will be recognized as list index, without looking at the code. I'm not.
If you're adding dict keys to the schema that could be reasonably parsed as a list index, I think you're doing something wrong. I would agree that this is a problem if we were talking about user-supplied values, but it's just keys that are defined by qemu.
Quoted 7 lines
I'm afraid I have growing doubts regarding the QemuOpts dotted key convention in general.
The convention makes '.' a special character in keys, but only sometimes. If the key gets consumed by something that uses dotted key convention, '.' is special, and to get a non-special '.', you need to escape it by doubling. Else, it's not.
Do we really implement escaping by doubling dots? This would be news to me.
Do we even have a reason to escape dots, i.e. are there any options in the schema whose keys contain a dot? I think we took care to avoid such things.
Quoted 10 lines
Since the same key can be used differently by different code, the same '.' could in theory be both special and non-special. In practice, this would be madness.
Adopting the dotted key convention for an existing QemuOpts option, say -object [PATCH 15], *breaks* existing command line usage of keys containing '.', because you now have to escape the '.'. Dan, I'm afraid you need to show that no such keys exist, or if they exist they don't matter.
Quoted 3 lines
I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.
I wanted to have a look at this example, but I can only find the string "ide.0" used as a bus name in the sources, that is, a value rather than a key.
Do you have a pointer to the property definition that you mean?
Quoted 2 lines
It's probably too late to back out the dotted key convention completely. Kevin?
Yes, far too late.
Also, I haven't yet seen even just an idea for an alternative solution.
Quoted 2 lines
Can we still back out the list part of the convention, and use repeated keys instead?
See above, repeated keys are messy and can't provide the same thing.
Also it's been in use since quorum was introduced in commit c88a1de51, February 2014. Pretty sure that this means no anyway.
Quoted 12 lines
If we're stuck with some form of the dotted key convention, can we at least make it a more integral part of QemuOpts rather than something bolted on as an afterthought? Here's my thinking on how that might be done:
* Have a QemuOptsList flag @flat.
* If @flat, QemuOpts behaves as it always has: the special characters are ',' and '=', and parsing a key=value,... string produces a list where each element represents one key=value from the string, in the same order.
Quoted 7 lines
* If not @flat, '.' becomes an additional special character, and parsing a key=value,... string produces a dictionary, similar to the one you get now by converting with
qemu_opts_to_qdict()and filtering throughqdict_crumple().The difference to now is that you either always crumple, or not at all, and the meaning of '.' is unambiguous.
Integrating qdict_crumple() functionality in QemuOpts sounds like a reasonable next step after Dan's series.
Quoted 3 lines
I wish we had refrained from saddling QemuOpts with even more magic. Compared to this swamp, use of JSON on the command line looks rather appealing to me.
I'd bet most users would disagree with you because they use only simple cases.
In more complicated setups, dotted syntax still works (unlike repeated options), but becomes a bit verbose because of the repetition of long prefixes. If we wanted to attack this, we would have to introduce two more special characters (brackets) for a more concise nesting syntax.
Kevin
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
Quoted 27 lines · Kevin Wolf
Am 13.10.2016 um 14:35 hat Markus Armbruster geschrieben:
Quoted 26 lines
Cc Kevin for discussion of QemuOpts dotted key conventionQuoted 12 lines · Daniel P. Berrange
Currently
qdict_crumplerequires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.In order to have backwards compatibility with the OptsVisitor,
qemu_opt_to_qdict()has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given toqdict_crumple().To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed
Quoted 6 lines
foo=hello,foo=world,foo=wibble foo.0=hello,foo.1=world,foo.2=wibblebut this is forbidden
foo=hello,foo=world,foo.2=wibble
I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.
Quoted 6 lines
Quoted 3 lines
The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?
Probably primarily because someone didn't realise this when introducing the dotted syntax.
Can't even blame "someone" for that; it's an obscure, underdocumented feature of an interface that's collapsing under its load of obscure, underdocumented features.
On the other hand, that's not exactly a state that allows for *more* obscure features.
Quoted 1 line
Also because flat QDicts can't represent this.
Explain?
Quoted 11 lines
But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.
Compare this:
driver=quorum, child.0.driver=file,child.0.filename=disk1.img, child.1.driver=host_device,child.1.filename=/dev/sdb, child.2.driver=nbd,child.2.host=localhost
Quoted 6 lines
And this:
driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost
Aside: both are about equally illegible, to be honest.
Quoted 2 lines
For starters, can we really trust the order in QemuOpts so that the right driver and filename are associated with each other?
The order is trustworthy, but...
Quoted 4 lines
We would also have code to associate the third child.driver with the first child.host (because file and
host_devicedon't have a host property). And this isn't even touching optional arguments yet. Would you really want to implement or review this?
... you're right, doing lists by repeating keys breaks down when combined with the dotted key convention's use of repetition to do structured values.
Permit me to digress.
QemuOpts wasn't designed for list-values keys. Doing lists by repetition was clever.
QemuOpts wasn't designed for structured values. Doing structured values by a dotted key convention plus repetition was clever.
And there's the problem: too much cleverness, not enough "this is being pushed way beyond its design limits, time to replace it".
For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:
{ "driver": "quorum",
"child": [ { "driver": "file", "filename": "disk1.img" },
{ "driver": "host_device", "filename=/dev/sdb" },
{ "driver": "nbd", "host": "localhost" } ] }Note how the structure is obvious. It isn't with dotted keys, not even if you order them sensibly (which users inevitably won't).
Also not that the value needs to be parsed by QemuOpts! You can't leave it to the consumer of the QemuOpts, because if you did, you'd have to escape the commas.
If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try
driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]
I'd go with some existing syntax, though. The one we already use is JSON.
End of digression.
Quoted 23 lines
Quoted 15 lines
Note that this second way creates entirely new failure modes and restrictions. Let me show using an example derived from one in qdict_crumple()'s contract:
foo.0.bar=bla,foo.eek.bar=blubb
Without the dotted key convention, this is perfectly fine: key "foo.0.bar" has the single value "bla", and key "foo.eek.bar" has the single value "blubb". Equivalent JSON would be
{ "foo.0.bar": "bla", "foo.eek.bar": "blubb" }
With just the struct convention, it's still fine: it obviously means the same as JSON
Quoted 7 lines
{ "foo": { "0": { "bar": "bla" }, "eek": { "bar": "blubb" } } }
Adding the list convention makes it invalid. It also outlaws a bunch of keys that would be just fine in JSON, namely any that get recognized as list index. Raise your hand if you're willing to bet real money on your predictions of what will be recognized as list index, without looking at the code. I'm not.
Quoted 4 lines
If you're adding dict keys to the schema that could be reasonably parsed as a list index, I think you're doing something wrong. I would agree that this is a problem if we were talking about user-supplied values, but it's just keys that are defined by qemu.
Your argument would be stronger if we didn't have such a disappointing track record on sensible naming. We've failed at picking a consistent rules for names, let alone enforcing them. Instead, we have local rules, enforced to varying degrees, and of course inconsistent with each other. We even have code to mangle names, so that they conform to some other set of rules.
We should not extend an interface in a way that makes it rely on certain rules for names without writing down and enforcing these rules.
You did it *locally* for block stuff, and got away with it, because you control the keys (hopefully), and nobody (including probably you and very much me) wants to think about this particular mess unless it's absolutely, obviously, embarrassingly necessary.
Which is what he get to do now, because now we're discussing to adopt your local thing for general use, with keys no single person really knows, let alone controls.
Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.
We can avoid rule 2 by requiring '.' to be escaped. Dan's qdict_crumple() currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.
The names in question are the names declared in QemuOptDesc (easy enough to find) plus any names that the code may use with empty QemuOptDesc (harder). Empty QemuOptDesc are used with the following option groups:
* "drive", qemu_drive_opts in blockdev.c
Also empty_opts in qemu-io.c and qemu_drive_opts in test-replication.c
* "numa", qemu_numa_opts in numa.c
* "device", qemu_device_opts in qdev-monitor.c
This one pulls in qdev properties. Properties violating rule 2 exist.
* "object", qemu_object_opts in vl.c
Also qemu_object_opts in qemu-img.c, qemu-io.c and qemu-nbd.c (WTF?)
This one pulls in properties of user-creatable objects.
* "source", qemu_soource_opts (sic!) in qemu-img.c
* "reopen", reopen_opts in qemu-io-cmds.c
* "file", file_opts in qemu-io.c and qemu-nbd.c
* "machine", qemu_machine_opts in vl.c
This one pulls in machine properties.
* "tpmdev", qemu_tpmdev_opts in vl.c
* "netdev", qemu_netdev_opts in net.c
* "net", qemu_net_opts in net.c
* "userdef", userdef_opts in tests-opts-visitor.c
* "opts_list_03", opts_list_03 in test-qemu-opts.c
* "acpi", qemu_acpi_opts in hw/acpi/core.c
* "smbios", qemu_smbios_opts in smbios.c
Empty QemuOptDesc is used this widely because QemuOpts is too primitive to support the needs of its users. Unlike a QAPI schema. Just sayin'.
Finding the names and assessing how they're impacted by the dotted key convention is going to be a substantial chunk of work.
Quoted 10 lines
Quoted 7 lines
I'm afraid I have growing doubts regarding the QemuOpts dotted key convention in general.
The convention makes '.' a special character in keys, but only sometimes. If the key gets consumed by something that uses dotted key convention, '.' is special, and to get a non-special '.', you need to escape it by doubling. Else, it's not.
Do we really implement escaping by doubling dots? This would be news to me.
Dan's PATCH 02/21 does.
Quoted 3 lines
Do we even have a reason to escape dots, i.e. are there any options in the schema whose keys contain a dot? I think we took care to avoid such things.
I'm afraid we did not.
Quoted 14 lines
Quoted 10 lines
Since the same key can be used differently by different code, the same '.' could in theory be both special and non-special. In practice, this would be madness.
Adopting the dotted key convention for an existing QemuOpts option, say -object [PATCH 15], *breaks* existing command line usage of keys containing '.', because you now have to escape the '.'. Dan, I'm afraid you need to show that no such keys exist, or if they exist they don't matter.
Quoted 3 lines
I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.
Quoted 5 lines
I wanted to have a look at this example, but I can only find the string "ide.0" used as a bus name in the sources, that is, a value rather than a key.
Do you have a pointer to the property definition that you mean?
We've gotten better at hiding property definitions...
"qemu-system-ppc -device macio-ide,help" shows the property:
macio-ide.ide.0=child<IDE>
The device is not marked no-user, but the obvious attempt to use it fails:
$ qemu-system-ppc -device macio-ide
qemu-system-ppc: Option '-device macio-ide' cannot be handled by this machineThis is the Alex's dynamic system bus thingy, which put most of the system bus devices in help even when they cannot be used.
Perhaps you can make a case why this property cannot be used via QemuOpts. If so, great! There are about three dozen more for you to check, followed by fourteen more option groups with empty QemuOptDesc, and a bunch of non-empty QemuOptDesc ;-P
Quoted 4 lines
Quoted 2 lines
It's probably too late to back out the dotted key convention completely. Kevin?
Yes, far too late.
If we're stuck with the existing use of dotted key convention, we're stuck with it. But that's no excuse for adding more uses.
If we decide we want to use dotted key convention more widely, we must document and enforce the naming conventions it requires at least in all the places where we use it. As discussed above, this might involve compatibility breaks, hopefully minor.
Quoted 1 line
Also, I haven't yet seen even just an idea for an alternative solution.
See digression above.
Quoted 31 lines
Quoted 2 lines
Can we still back out the list part of the convention, and use repeated keys instead?
See above, repeated keys are messy and can't provide the same thing.
Also it's been in use since quorum was introduced in commit
c88a1de51, February 2014. Pretty sure that this means no anyway.Quoted 12 lines
If we're stuck with some form of the dotted key convention, can we at least make it a more integral part of QemuOpts rather than something bolted on as an afterthought? Here's my thinking on how that might be done:
* Have a QemuOptsList flag @flat.
* If @flat, QemuOpts behaves as it always has: the special characters are ',' and '=', and parsing a key=value,... string produces a list where each element represents one key=value from the string, in the same order.
Quoted 7 lines
* If not @flat, '.' becomes an additional special character, and parsing a key=value,... string produces a dictionary, similar to the one you get now by converting with
qemu_opts_to_qdict()and filtering throughqdict_crumple().The difference to now is that you either always crumple, or not at all, and the meaning of '.' is unambiguous.
Integrating
qdict_crumple()functionality in QemuOpts sounds like a reasonable next step after Dan's series.
Quoted 6 lines
Quoted 3 lines
I wish we had refrained from saddling QemuOpts with even more magic. Compared to this swamp, use of JSON on the command line looks rather appealing to me.
I'd bet most users would disagree with you because they use only simple cases.
Use of JSON on the command line when and where you need nesting, not everywhere. The simple cases can stay exactly as they are.
Quoted 6 lines
In more complicated setups, dotted syntax still works (unlike repeated options), but becomes a bit verbose because of the repetition of long prefixes. If we wanted to attack this, we would have to introduce two more special characters (brackets) for a more concise nesting syntax.
Kevin
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
Quoted 18 lines
For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:
{ "driver": "quorum", "child": [ { "driver": "file", "filename": "disk1.img" }, { "driver": "host_device", "filename=/dev/sdb" }, { "driver": "nbd", "host": "localhost" } ] }Note how the structure is obvious. It isn't with dotted keys, not even if you order them sensibly (which users inevitably won't).
Also not that the value needs to be parsed by QemuOpts! You can't leave it to the consumer of the QemuOpts, because if you did, you'd have to escape the commas.
If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try
Quoted 7 lines
driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]I'd go with some existing syntax, though. The one we already use is JSON.
In fact there is already "filename=json:{...}" support in the block layer. By the way, abuse of QemuOpts dates back to http://wiki.qemu.org/Features/QCFG.
Quoted 9 lines
Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.
We can avoid rule 2 by requiring '.' to be escaped. Dan's
qdict_crumple()currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.
Quoted 3 lines
* "device",
qemu_device_optsinqdev-monitor.cThis one pulls in qdev properties. Properties violating rule 2 exist.
Which are they? Only bus names?
Quoted 16 lines
* "object",
qemu_object_optsinvl.cThis one pulls in properties of user-creatable objects.
* "machine",
qemu_machine_optsinvl.cThis one pulls in machine properties.
Quoted 8 lines
Quoted 3 lines
I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.
I wanted to have a look at this example, but I can only find the string "ide.0" used as a bus name in the sources, that is, a value rather than a key.
Quoted 7 lines
Quoted 1 line
Do you have a pointer to the property definition that you mean?
We've gotten better at hiding property definitions...
"qemu-system-ppc -device macio-ide,help" shows the property:
macio-ide.ide.0=child<IDE>
It is a bug that this property is shown in the help, because it's not assignable (same for all other child<> properties). I'd rather declare other occurrences of "." in user-accessible property names to be bugs, and break the ABI if there are any.
Paolo
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
In reply to Markus ArmbrusterQuoted 31 lines · Markus Armbruster
Quoted 11 lines
But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.
Compare this:
driver=quorum, child.0.driver=file,child.0.filename=disk1.img, child.1.driver=host_device,child.1.filename=/dev/sdb, child.2.driver=nbd,child.2.host=localhostQuoted 6 lines
And this:
driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost
Aside: both are about equally illegible, to be honest.
Permit me to digress.
QemuOpts wasn't designed for list-values keys. Doing lists by repetition was clever.
QemuOpts wasn't designed for structured values. Doing structured values by a dotted key convention plus repetition was clever.
And there's the problem: too much cleverness, not enough "this is being pushed way beyond its design limits, time to replace it".
Quoted 7 lines
For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:
{ "driver": "quorum", "child": [ { "driver": "file", "filename": "disk1.img" }, { "driver": "host_device", "filename=/dev/sdb" }, { "driver": "nbd", "host": "localhost" } ] }
Possible hack solution:
QemuOpts already special-cases id=. What if we ALSO make it special-case a leading json=? Shown here with shell quoting, the above example of creating a Quorum -drive argument could then be:
-drive json='
{ "driver": "quorum",
"child": [ { "driver": "file", "filename": "disk1.img" },
{ "driver": "host_device", "filename=/dev/sdb" },
{ "driver": "nbd", "host": "localhost" } ] }'
As far as I know, we don't have 'json' as any existing QemuOpts key (do we? A full audit may be better than my quick git grep '"json"'). Thus, if QemuOpts sees a leading json=, it hands off the rest of the string to the same parser as we use for QMP, where we no longer have to escape commas (even nicer than the drive hack where we support filename=json:{...} but have to double up all commas to make it through the QemuOpts layer). Encountering json= as anything other than the first option would be an error, and you would be unable to combine a json= option with any other old-style option. In other words, the use of leading json= would be the switch for whether to do old-style parsing or to use a saner syntax for everything else that needs structure, on a per-argument basis.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
In reply to Paolo BonziniQuoted 23 lines · Paolo Bonzini
Quoted 23 lines
For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:
{ "driver": "quorum", "child": [ { "driver": "file", "filename": "disk1.img" }, { "driver": "host_device", "filename=/dev/sdb" }, { "driver": "nbd", "host": "localhost" } ] }Note how the structure is obvious. It isn't with dotted keys, not even if you order them sensibly (which users inevitably won't).
Also not that the value needs to be parsed by QemuOpts! You can't leave it to the consumer of the QemuOpts, because if you did, you'd have to escape the commas.
If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try
driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]
Quoted 4 lines
Quoted 2 lines
I'd go with some existing syntax, though. The one we already use is JSON.
In fact there is already "filename=json:{...}" support in the block layer.
Yes.
Quoted 1 line
By the way, abuse of QemuOpts dates back to http://wiki.qemu.org/Features/QCFG.
That page is from 2011, when QAPI was being implemented. The idea to bring the power of the QAPI schema to bear on the command line is sound, but QCFG never made it past the ideas stage, I'm afraid. It uses either dotted keys or braces for nested structs. It doesn't cover lists.
Quoted 9 lines
Quoted 8 lines
Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.
We can avoid rule 2 by requiring '.' to be escaped. Dan's
qdict_crumple()currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.
Quoted 5 lines
Quoted 3 lines
* "device",
qemu_device_optsinqdev-monitor.cThis one pulls in qdev properties. Properties violating rule 2 exist.
Which are they? Only bus names?
Finding properties is difficult, because we (foolishly!) define them in code rather than data. My testing with "info qdm" for all targets finds:
= xlnx.axi-dma =
xlnx.axi-dma[0]
= xlnx.xps-ethernetlite =
xlnx.xps-ethernetlite[0]
= xlnx.xps-intc =
xlnx.xps-intc[0]
= xlnx.xps-uartlite =
xlnx.xps-uartlite[0]
= xlnx.axi-dma =
xlnx.axi-dma[0]
= xlnx.xps-ethernetlite =
xlnx.xps-ethernetlite[0]
= xlnx.xps-intc =
xlnx.xps-intc[0]
= xlnx.xps-uartlite =
xlnx.xps-uartlite[0]
= cuda =
adb.0
= raven-pcihost =
pci.0
= macio-ide =
ide.0
= mpc8544-guts =
mpc8544.guts[0]
= xlnx.xps-ethernetlite =
xlnx.xps-ethernetlite[0]
= xlnx.xps-intc =
xlnx.xps-intc[0]
= xlnx.xps-uartlite =
xlnx.xps-uartlite[0]
= cuda =
adb.0
= raven-pcihost =
pci.0
= macio-ide =
ide.0
= mpc8544-guts =
mpc8544.guts[0]
= xlnx.xps-ethernetlite =
xlnx.xps-ethernetlite[0]
= xlnx.xps-intc =
xlnx.xps-intc[0]
= xlnx.xps-uartlite =
xlnx.xps-uartlite[0]
= s390-sclp-event-facility =
s390-sclp-events-bus.0
= cgthree =
cg3.reg[0]
cg3.prom[0]
= SUNW,tcx =
tcx.thc[0]
tcx.rblit[0]
tcx.dac[0]
tcx.alt[0]
tcx.tec[0]
tcx.rstip[0]
tcx.blit[0]
tcx.dhc[0]
tcx.prom[0]
tcx.stip[0]Quoted 18 lines
Quoted 18 lines
* "object",
qemu_object_optsinvl.cThis one pulls in properties of user-creatable objects.
* "machine",
qemu_machine_optsinvl.cThis one pulls in machine properties.
Quoted 9 lines
Quoted 3 lines
I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.
I wanted to have a look at this example, but I can only find the string "ide.0" used as a bus name in the sources, that is, a value rather than a key.
Do you have a pointer to the property definition that you mean?
Quoted 8 lines
Quoted 5 lines
We've gotten better at hiding property definitions...
"qemu-system-ppc -device macio-ide,help" shows the property:
macio-ide.ide.0=child<IDE>It is a bug that this property is shown in the help, because it's not assignable (same for all other child<> properties).
Let's fix it then.
Quoted 3 lines
I'd rather declare
other occurrences of "." in user-accessible property names to be bugs, and break the ABI if there are any.
I propose to adopt QAPI's rules on permissible names: "All names must begin with a letter, and contain only ASCII letters, digits, dash, and underscore" (docs/qapi-code-gen.txt). QAPI's naming rules get adopted anyway on QAPIfication, so why wait? :)
Note that this may affect names generated by automatic arrayification (commit 3396590), because those contain [ and ].
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
In reply to Markus ArmbrusterAm 17.10.2016 um 16:50 hat Markus Armbruster geschrieben:
Quoted 36 lines
Quoted 27 lines · Kevin Wolf
Am 13.10.2016 um 14:35 hat Markus Armbruster geschrieben:
Quoted 26 lines
Cc Kevin for discussion of QemuOpts dotted key conventionQuoted 12 lines · Daniel P. Berrange
Currently
qdict_crumplerequires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.In order to have backwards compatibility with the OptsVisitor,
qemu_opt_to_qdict()has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given toqdict_crumple().To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed
Quoted 6 lines
foo=hello,foo=world,foo=wibble foo.0=hello,foo.1=world,foo.2=wibblebut this is forbidden
foo=hello,foo=world,foo.2=wibble
I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.
Quoted 6 lines
Quoted 3 lines
The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?
Probably primarily because someone didn't realise this when introducing the dotted syntax.
Quoted 6 lines
Can't even blame "someone" for that; it's an obscure, underdocumented feature of an interface that's collapsing under its load of obscure, underdocumented features.
On the other hand, that's not exactly a state that allows for *more* obscure features.
I don't really think we're introducing more obscure features here, but rather trying to implement a single, and rather straightforward, way as the standard.
Dotted syntax for hierarchy has actually plenty of precedence in qemu if you look a bit closer (the block layer, -global, -device foo,help, even the bus names you mentioned below are really just flattened lists), so we're only making things more consistent.
Quoted 3 lines
Quoted 1 line
Also because flat QDicts can't represent this.
Explain?
Repeated options are parsed into QLists. If you keep it at that without flattening you have at least a QDict that contains a QList that contains simple types. This is not flat any more.
Of course, you could argue that flat QDicts are the wrong data structure in the first place and instead of flatting everything we should have done the equivalent of qdict_crumple from the beginning, but they were the natural extension of what was already there and happened to work good enough, so this is what we're currently using.
Quoted 11 lines
Quoted 11 lines
But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.
Compare this:
driver=quorum, child.0.driver=file,child.0.filename=disk1.img, child.1.driver=host_device,child.1.filename=/dev/sdb, child.2.driver=nbd,child.2.host=localhost
Quoted 8 lines
Quoted 6 lines
And this:
driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost
Aside: both are about equally illegible, to be honest.
Not sure about equally, but let's agree on "both are illegible".
Quoted 33 lines
Quoted 2 lines
For starters, can we really trust the order in QemuOpts so that the right driver and filename are associated with each other?
The order is trustworthy, but...
Quoted 4 lines
We would also have code to associate the third child.driver with the first child.host (because file and
host_devicedon't have a host property). And this isn't even touching optional arguments yet. Would you really want to implement or review this?... you're right, doing lists by repeating keys breaks down when combined with the dotted key convention's use of repetition to do structured values.
Permit me to digress.
QemuOpts wasn't designed for list-values keys. Doing lists by repetition was clever.
QemuOpts wasn't designed for structured values. Doing structured values by a dotted key convention plus repetition was clever.
And there's the problem: too much cleverness, not enough "this is being pushed way beyond its design limits, time to replace it".
For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:
{ "driver": "quorum", "child": [ { "driver": "file", "filename": "disk1.img" }, { "driver": "host_device", "filename=/dev/sdb" }, { "driver": "nbd", "host": "localhost" } ] }
Quoted 6 lines
Note how the structure is obvious. It isn't with dotted keys, not even if you order them sensibly (which users inevitably won't).
Also not that the value needs to be parsed by QemuOpts! You can't leave it to the consumer of the QemuOpts, because if you did, you'd have to escape the commas.
Okay, so I like JSON. It's a great format for our monitor protocol. We even have pretty printers so that it's more or less readable as output for human users. However, there is one thing for which it is horrible:
Getting user input.
Yes, getting rid of the comma escaping is a first step, but typing JSON on the command line remains a PITA. Mostly because of the quotes (which you probably have to escape), but also things like the useless outer brackets.
Quoted 7 lines
If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try
driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]
This looks a lot more agreeable as something that I have to type on the command line.
What's more, this is a direct extension of the existing syntax. You complained about how the step from simple configurations with -hda to more complex ones with -drive completely changes the syntax (and rightly so). Going from simple QemuOpts syntax to doing JSON as soon as you get structured values and lists would be another similar step. In contrast, the new syntax you're suggesting above is a natural extension of what's there.
Quoted 2 lines
I'd go with some existing syntax, though. The one we already use is JSON.
The one we already use in this context is comma separated key/value pairs as parsed by QemuOpts.
Quoted 8 lines
Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.
We can avoid rule 2 by requiring '.' to be escaped. Dan's
qdict_crumple()currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.
I don't think we should support any escaping and rather forbid '.' completely in names.
Quoted 4 lines
The names in question are the names declared in QemuOptDesc (easy enough to find) plus any names that the code may use with empty QemuOptDesc (harder). Empty QemuOptDesc are used with the following option groups:
Quoted 8 lines
* "drive",
qemu_drive_optsinblockdev.c
Alsoempty_optsinqemu-io.candqemu_drive_optsintest-replication.c* "numa",
qemu_numa_optsinnuma.c* "device",
qemu_device_optsinqdev-monitor.cThis one pulls in qdev properties. Properties violating rule 2 exist.
You posted a list of them in another email. Apart from not being user-configurable, I think it actually makes sense to interpret these as being structured.
A lot of them are really array/list properties with a single entry. It seems we have some inconsistency there as to whether it the syntax is 'list.index' or 'list[index]'. The former matches the dotted syntax rule and is used in external interfaces (e.g. in bus=...), and I seem to remember the latter is a newer QOM thing.
Would it be too late to change the latter into the former? In other words, is the difference externally visible outside of qom-*, which we declared a non-ABI, as far as I know?
Quoted 23 lines
* "object",
qemu_object_optsinvl.c
Alsoqemu_object_optsinqemu-img.c,qemu-io.candqemu-nbd.c(WTF?)This one pulls in properties of user-creatable objects.
* "source",
qemu_soource_opts(sic!) inqemu-img.c* "reopen",
reopen_optsinqemu-io-cmds.c* "file",
file_optsinqemu-io.candqemu-nbd.c* "machine",
qemu_machine_optsinvl.cThis one pulls in machine properties.
* "tpmdev",
qemu_tpmdev_optsinvl.c* "netdev",
qemu_netdev_optsinnet.c* "net",
qemu_net_optsinnet.c* "userdef",
userdef_optsintests-opts-visitor.c
Quoted 8 lines
* "
opts_list_03",opts_list_03intest-qemu-opts.c* "acpi",
qemu_acpi_optsinhw/acpi/core.c* "smbios",
qemu_smbios_optsinsmbios.cEmpty QemuOptDesc is used this widely because QemuOpts is too primitive to support the needs of its users. Unlike a QAPI schema. Just sayin'.
I don't think you have to convince anyone that QemuOpts is too limited. But today it's still a piece that is needed to get from the command line to something superior like QAPI. This is the whole reason why we're doing all of this.
Completely QAPIfication of everything and going directly from the command line to QAPI objects would be nice, but somehow I feel it's not a realistic hope to get this as a short term solution.
Quoted 2 lines
Finding the names and assessing how they're impacted by the dotted key convention is going to be a substantial chunk of work.
Possibly.
The other option would be to take it step by step and let every user of QemuOpts specify on creation whether dots are used for structure or should be taken literally. Then you can convert user by user and assess them one at a time (which is still a substantial chunk for -object and -device, but at least you can concentrate on only these for now).
If you're concerned about compatibility issues if we find a dot in a name only in one of the later users: We can handle it. If you then stumble across an obscure "weird.name" property where the dot is really part of the name and not already representing structure, you can still artificially reinterpret it as a nested structure with a single field even if logically that's not what it is... Might be a bit ugly, but keeps it working.
Kevin
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
In reply to Eric BlakeQuoted 53 lines · Eric Blake
Quoted 31 lines · Markus Armbruster
Quoted 11 lines
But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.
Compare this:
driver=quorum, child.0.driver=file,child.0.filename=disk1.img, child.1.driver=host_device,child.1.filename=/dev/sdb, child.2.driver=nbd,child.2.host=localhostQuoted 6 lines
And this:
driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost
Aside: both are about equally illegible, to be honest.
Permit me to digress.
QemuOpts wasn't designed for list-values keys. Doing lists by repetition was clever.
QemuOpts wasn't designed for structured values. Doing structured values by a dotted key convention plus repetition was clever.
And there's the problem: too much cleverness, not enough "this is being pushed way beyond its design limits, time to replace it".
Quoted 7 lines
For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:
{ "driver": "quorum", "child": [ { "driver": "file", "filename": "disk1.img" }, { "driver": "host_device", "filename=/dev/sdb" }, { "driver": "nbd", "host": "localhost" } ] }Possible hack solution:
QemuOpts already special-cases id=. What if we ALSO make it special-case a leading json=? Shown here with shell quoting, the above example of creating a Quorum -drive argument could then be:
-drive json='
{ "driver": "quorum", "child": [ { "driver": "file", "filename": "disk1.img" }, { "driver": "host_device", "filename=/dev/sdb" }, { "driver": "nbd", "host": "localhost" } ] }'
Quoted 12 lines
As far as I know, we don't have 'json' as any existing QemuOpts key (do we? A full audit may be better than my quick git grep '"json"'). Thus, if QemuOpts sees a leading json=, it hands off the rest of the string to the same parser as we use for QMP, where we no longer have to escape commas (even nicer than the drive hack where we support filename=json:{...} but have to double up all commas to make it through the QemuOpts layer). Encountering json= as anything other than the first option would be an error, and you would be unable to combine a json= option with any other old-style option. In other words, the use of leading json= would be the switch for whether to do old-style parsing or to use a saner syntax for everything else that needs structure, on a per-argument basis.
Slight variation: omit the 'json=' and recognize the '{':
-drive '{ "driver": "quorum",
"child": [ { "driver": "file", "filename": "disk1.img" },
{ "driver": "host_device", "filename=/dev/sdb" },
{ "driver": "nbd", "host": "localhost" } ] }'As always when extending haphazardly defined syntax, the question to ask is whether this makes the syntax (more) ambiguous.
So what is the option argument syntax now? The abstract syntax is simple enough: "list of (key, value) pairs". For the concrete syntax, we need to study opts_do_parse().
Each iteration of its loop accepts an abstract (key, value). It first looks for the leftmost '=' and ','. Cases:
1. There is no '=', or it is to the right of the leftmost ','. In other words, this (key, value) can only be key with an implied value or a value with an implied key.
1a. If this is the first iteration, and we accept an implied key, this is a value for the implied key. We consume everything up to the first non-escaped ','. This may be more than the leftmost ',' we found above. The consumed string with escapes processed is the value.
1b. Else, this is a key with an implied value. We consume everything up to the leftmost ',' (no escaping here).
1b1. If the consumed string starts with "no", the key is everything after the "no" and the value is "off".
1b2. Else the key is the string and the value is "on".
2. This is a key and a value. We first consume everything up to the leftmost '=' (no escaping here). The consumed string is the key. We then consume '='. Finally, we consume everything up to the first non-escaped ','The consumed string with escapes processed is the value.
Thus, the option argument starts either with a key (case 1b1, 2), "no" (case 1b2) or a value (case 1a).
Adding JSON object syntax (which always starts with '{') is ambiguous when a key can start with '{' (case 1b1, 2) or when a value can (case 1a).
Keys starting with '{' are basically foolish. Let's outlaw them by adopting QAPI's name rules.
Values starting with '{' are possible. The implied keys I can remember don't have such values, though. If we interpret '{' as JSON, then users can't omit the key when the value starts with '{'. Not pretty, but I'd say it's tolerable.
Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict
In reply to Daniel P. BerrangeQuoted 84 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{ 'foo.0.bar': 'one', 'foo.0.wizz': '1', 'foo.1.bar': 'two', 'foo.1.wizz': '2'}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{ 'foo': [ { 'bar': 'one', 'wizz': '1' }, { 'bar': 'two', 'wizz': '2' } ],}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{ 'foo..bar': 'wizz', 'bar.foo..bar': 'eek', 'bar.hello': 'world' }Will end up as
{ 'foo.bar': 'wizz', 'bar': { 'foo.bar': 'eek', 'hello': 'world' } }The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.
Reviewed-by Eric Blake <[email protected]>Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 261 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 551 insertions(+)include/qapi/qmp/qdict.h +1
-0@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);qobject/qdict.c +0
-0
[...]
Quoted 4 lines
+/**
+ *qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionaries
Is recursive=false used outside tests in this series?
Quoted 40 lines
+ *
+ * Takes a flat dictionary whose keys use '.' separator to indicate
+ * nesting, and values are scalars, and crumples it into a nested
+ * structure. If the @recursive parameter is false, then only the
+ * first level of structure implied by the keys will be crumpled. If
+ * @recursive is true, then the input will be recursively crumpled to
+ * expand all levels of structure in the keys.
+ *
+ * To include a literal '.' in a key name, it must be escaped as '..'
+ *
+ * For example, an input of:
+ *
+ * { 'foo.0.bar': 'one', 'foo.0.wizz': '1',
+ * 'foo.1.bar': 'two', 'foo.1.wizz': '2' }
+ *
+ * will result in an output of:
+ *
+ * {
+ * 'foo': [
+ * { 'bar': 'one', 'wizz': '1' },
+ * { 'bar': 'two', 'wizz': '2' }
+ * ],
+ * }
+ *
+ * The following scenarios in the input dict will result in an
+ * error being returned:
+ *
+ * - Any values in @src are non-scalar types
+ * - If keys in @src imply that a particular level is both a
+ * list and a dict. e.g., "foo.0.bar" and "foo.eek.bar".
+ * - If keys in @src imply that a particular level is a list,
+ * but the indices are non-contiguous. e.g. "foo.0.bar" and
+ * "foo.2.bar" without any "foo.1.bar" present.
+ * - If keys in @src represent list indexes, but are not in
+ * the "%zu" format. e.g. "foo.+0.bar"
+ *
+ * Returns: either a QDict or QList for the nested data structure, or NULL
+ * on error
+ */
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp)
[...]
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
In reply to Kevin WolfQuoted 53 lines · Kevin Wolf
Am 17.10.2016 um 16:50 hat Markus Armbruster geschrieben:
Quoted 36 lines
Quoted 27 lines · Kevin Wolf
Am 13.10.2016 um 14:35 hat Markus Armbruster geschrieben:
Quoted 26 lines
Cc Kevin for discussion of QemuOpts dotted key conventionQuoted 12 lines · Daniel P. Berrange
Currently
qdict_crumplerequires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.In order to have backwards compatibility with the OptsVisitor,
qemu_opt_to_qdict()has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given toqdict_crumple().To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed
Quoted 6 lines
foo=hello,foo=world,foo=wibble foo.0=hello,foo.1=world,foo.2=wibblebut this is forbidden
foo=hello,foo=world,foo.2=wibble
I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.
Quoted 6 lines
Quoted 3 lines
The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?
Probably primarily because someone didn't realise this when introducing the dotted syntax.
Quoted 6 lines
Can't even blame "someone" for that; it's an obscure, underdocumented feature of an interface that's collapsing under its load of obscure, underdocumented features.
On the other hand, that's not exactly a state that allows for *more* obscure features.
I don't really think we're introducing more obscure features here, but rather trying to implement a single, and rather straightforward, way as the standard.
Dotted syntax for hierarchy has actually plenty of precedence in qemu if you look a bit closer (the block layer, -global, -device foo,help, even the bus names you mentioned below are really just flattened lists), so we're only making things more consistent.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Also because flat QDicts can't represent this.
Explain?
Repeated options are parsed into QLists. If you keep it at that without flattening you have at least a QDict that contains a QList that contains simple types. This is not flat any more.
*All* options are parsed into a (non-QList) list. That's what is flat.
Only when you start crumpling things go beyond flat, and QDict/QList come into play.
Quoted 5 lines
Of course, you could argue that flat QDicts are the wrong data structure in the first place and instead of flatting everything we should have done the equivalent of
qdict_crumplefrom the beginning, but they were the natural extension of what was already there and happened to work good enough, so this is what we're currently using.
We didn't "flatten everything", because QemuOpts is and has always been flat to begin with. Rather we've started to crumple a few things, and in a separate layer.
I now think this is the wrong approach. We have clearly outgrown flat options. We should redo QemuOpts to support what we need instead of bolting on an optional crumpling layer.
I guess I reached the place you described, just on a different path :)
Quoted 67 lines
Quoted 11 lines
Quoted 11 lines
But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.
Compare this:
driver=quorum, child.0.driver=file,child.0.filename=disk1.img, child.1.driver=host_device,child.1.filename=/dev/sdb, child.2.driver=nbd,child.2.host=localhostQuoted 8 lines
Quoted 6 lines
And this:
driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost
Aside: both are about equally illegible, to be honest.
Not sure about equally, but let's agree on "both are illegible".
Quoted 33 lines
Quoted 2 lines
For starters, can we really trust the order in QemuOpts so that the right driver and filename are associated with each other?
The order is trustworthy, but...
Quoted 4 lines
We would also have code to associate the third child.driver with the first child.host (because file and
host_devicedon't have a host property). And this isn't even touching optional arguments yet. Would you really want to implement or review this?... you're right, doing lists by repeating keys breaks down when combined with the dotted key convention's use of repetition to do structured values.
Permit me to digress.
QemuOpts wasn't designed for list-values keys. Doing lists by repetition was clever.
QemuOpts wasn't designed for structured values. Doing structured values by a dotted key convention plus repetition was clever.
And there's the problem: too much cleverness, not enough "this is being pushed way beyond its design limits, time to replace it".
For me, a replacement should do structured values by providing suitable *value* syntax instead of hacking it into the keys:
{ "driver": "quorum", "child": [ { "driver": "file", "filename": "disk1.img" }, { "driver": "host_device", "filename=/dev/sdb" }, { "driver": "nbd", "host": "localhost" } ] }Quoted 6 lines
Note how the structure is obvious. It isn't with dotted keys, not even if you order them sensibly (which users inevitably won't).
Also not that the value needs to be parsed by QemuOpts! You can't leave it to the consumer of the QemuOpts, because if you did, you'd have to escape the commas.
Okay, so I like JSON. It's a great format for our monitor protocol. We even have pretty printers so that it's more or less readable as output for human users. However, there is one thing for which it is horrible:
Getting user input.
Quoted 4 lines
Yes, getting rid of the comma escaping is a first step, but typing JSON on the command line remains a PITA. Mostly because of the quotes (which you probably have to escape), but also things like the useless outer brackets.
As long as you don't need "'" in your JSON, you can simply enclose in "'" and be done. Since "'" can only occur in JSON strings, and the same strings would be present in any other syntax, any other syntax would be pretty much the same PITA.
Quoted 11 lines
Quoted 7 lines
If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try
driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]This looks a lot more agreeable as something that I have to type on the command line.
Quoted 7 lines
What's more, this is a direct extension of the existing syntax. You complained about how the step from simple configurations with -hda to more complex ones with -drive completely changes the syntax (and rightly so). Going from simple QemuOpts syntax to doing JSON as soon as you get structured values and lists would be another similar step. In contrast, the new syntax you're suggesting above is a natural extension of what's there.
Point taken. I just don't like inventing syntax, because as a rule, way too much syntax gets invented, and almost always badly.
Quoted 5 lines
Quoted 2 lines
I'd go with some existing syntax, though. The one we already use is JSON.
The one we already use in this context is comma separated key/value pairs as parsed by QemuOpts.
Which is flat, and flat doesn't cut it anymore.
If you make it non-flat with dotted key convention, the dotted key convention becomes part of the syntax. Counts as inventing syntax in my book.
Quoted 11 lines
Quoted 8 lines
Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.
We can avoid rule 2 by requiring '.' to be escaped. Dan's
qdict_crumple()currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.I don't think we should support any escaping and rather forbid '.' completely in names.
I think we should adopt QAPI's naming rules.
Quoted 13 lines
Quoted 4 lines
The names in question are the names declared in QemuOptDesc (easy enough to find) plus any names that the code may use with empty QemuOptDesc (harder). Empty QemuOptDesc are used with the following option groups:
Quoted 8 lines
* "drive",
qemu_drive_optsinblockdev.c
Alsoempty_optsinqemu-io.candqemu_drive_optsintest-replication.c* "numa",
qemu_numa_optsinnuma.c* "device",
qemu_device_optsinqdev-monitor.cThis one pulls in qdev properties. Properties violating rule 2 exist.
Quoted 7 lines
You posted a list of them in another email. Apart from not being user-configurable, I think it actually makes sense to interpret these as being structured.
A lot of them are really array/list properties with a single entry. It seems we have some inconsistency there as to whether it the syntax is 'list.index' or 'list[index]'.
Yes.
Quoted 3 lines
The former matches the dotted syntax
rule and is used in external interfaces (e.g. in bus=...), and I seem to remember the latter is a newer QOM thing.
Commit 3396590 "qom: Add automatic arrayification to object_property_add()". I tried to strangle it in the crib, unsuccessfully:
Message-Id: <[email protected]>
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg01905.html
Quoted 3 lines
Would it be too late to change the latter into the former? In other words, is the difference externally visible outside of qom-*, which we declared a non-ABI, as far as I know?
Paolo?
Quoted 32 lines
Quoted 23 lines
* "object",
qemu_object_optsinvl.c
Alsoqemu_object_optsinqemu-img.c,qemu-io.candqemu-nbd.c(WTF?)This one pulls in properties of user-creatable objects.
* "source",
qemu_soource_opts(sic!) inqemu-img.c* "reopen",
reopen_optsinqemu-io-cmds.c* "file",
file_optsinqemu-io.candqemu-nbd.c* "machine",
qemu_machine_optsinvl.cThis one pulls in machine properties.
* "tpmdev",
qemu_tpmdev_optsinvl.c* "netdev",
qemu_netdev_optsinnet.c* "net",
qemu_net_optsinnet.c* "userdef",
userdef_optsintests-opts-visitor.cQuoted 8 lines
* "
opts_list_03",opts_list_03intest-qemu-opts.c* "acpi",
qemu_acpi_optsinhw/acpi/core.c* "smbios",
qemu_smbios_optsinsmbios.cEmpty QemuOptDesc is used this widely because QemuOpts is too primitive to support the needs of its users. Unlike a QAPI schema. Just sayin'.
Quoted 8 lines
I don't think you have to convince anyone that QemuOpts is too limited. But today it's still a piece that is needed to get from the command line to something superior like QAPI. This is the whole reason why we're doing all of this.
Completely QAPIfication of everything and going directly from the command line to QAPI objects would be nice, but somehow I feel it's not a realistic hope to get this as a short term solution.
No argument. However, we should try hard to avoid short term hacks that make a QAPI-based solution harder. I believe the hard part will be retaining all the special cases that got in for convenience, or by accident, then were repurposed for whatever. And that's why I'm resisting additional syntactic/semantic cleverness *especially* when it's used only sometimes. Stuff that's used only sometimes can easily conflict with stuff used at other times.
I don't think we understand the existing cleverness sufficiently to predict interactions with new cleverness with confidence.
Quoted 10 lines
Quoted 2 lines
Finding the names and assessing how they're impacted by the dotted key convention is going to be a substantial chunk of work.
Possibly.
The other option would be to take it step by step and let every user of QemuOpts specify on creation whether dots are used for structure or should be taken literally. Then you can convert user by user and assess them one at a time (which is still a substantial chunk for -object and -device, but at least you can concentrate on only these for now).
I'm wary of protracted iterative conversion. We're better at starting these than finishing them. The started-but-not-finished state is exactly the state that scares me here: still more special cases basically nobody fully understands.
Quoted 7 lines
If you're concerned about compatibility issues if we find a dot in a name only in one of the later users: We can handle it. If you then stumble across an obscure "weird.name" property where the dot is really part of the name and not already representing structure, you can still artificially reinterpret it as a nested structure with a single field even if logically that's not what it is... Might be a bit ugly, but keeps it working.
Yes, that's how we'd do backward compatibility, if needed.
So, what can we do? Perhaps something like this:
* Pick a concrete syntax that supports nested stuff. Proposals on the table so far are
1. Dotted key convention as used in the block layer
2. Array syntax as used in QOM's automatic arrayification (not a complete solution, mentioned because it conflicts with the previous)
3. JSON
4. JSON with ':' replaced by '=', double-quotes and the outermost pair of braces dropped.
* Replace the QemuOpts internal representation: QDict instead of list.
* Replace the QemuOpts parser.
* Add suitable struct- and list-valued accessors.
qdict_crumple(qemu_opt_to_qdict(...), true, &errp) gets replaced by a trivial "get the whole dictionary" operation.
Other uses of qemu_opt_to_qdict() need to be rewritten.
* "Repetition builds a list" backward compatibility. It's used in only a few places.
* Possibly "keys with funny characters" backward compatibility.
Feels doable to me. 2.8 would be ambitious, though: soft freeze in in less than two weeks. However, getting this series into 2.8 has started to feel ambitious to me, too.
Re: [Qemu-devel] [PATCH v14 14/21] qapi: allow repeated opts with qobject_input_visitor_new_opts
In reply to Daniel P. BerrangeQuoted 47 lines · Daniel P. Berrange
The
qobject_input_visitor_new_opts()method gains a new parameter to control whether it allows repeated option keys in the input QemuOpts or not.Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qobject-input-visitor.h | 6 ++++++ qapi/qobject-input-visitor.c | 5 ++++- 2 files changed, 10 insertions(+), 1 deletion(-)include/qapi/qobject-input-visitor.h +6
-0@@ -112,6 +112,11 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,* qobject_input_visitor_new_autocast() method. See the docs * of that method for further details on processing behaviour. * +* If the @permit_repeated_opts parameter is true, then the input +* @opts is allowed to contain repeated keys and they will be +* turned into a QList. If it is false, then repeated keys will +* result in an error being reported. +* * The returned input visitor should be released by calling * visit_free() when no longer required. */ @@ -119,6 +124,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,bool autocreate_list, size_t autocreate_struct_levels, bool permit_int_ranges, + bool permit_repeated_opts, Error **errp);#endif qapi/qobject-input-visitor.c +1
-0@@ -753,6 +753,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,bool autocreate_list, size_t autocreate_struct_levels, bool permit_int_ranges, + bool permit_repeated_opts, Error **errp) { QDict *pdict;
qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 769
Quoted 10 lines
qapi/qobject-input-visitor.c +3
-1@@ -760,7 +761,9 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,Visitor *v =NULL ; pdict =qemu_opts_to_qdict (opts,NULL , -QEMU_OPTS_REPEAT_POLICY_LAST , + permit_repeated_opts ? +QEMU_OPTS_REPEAT_POLICY_ALL : +QEMU_OPTS_REPEAT_POLICY_ERROR , errp);if (!pdict) { goto cleanup;
@permit_repeated_opts applies to the whole @opts. However, the "repeat to build a list" feature is actually used more selectively. For instance, -spice wants it for "tls-channel" and "plaintext-channel", but not for its other keys. There, the code uses qemu_opt_get() or similar, which gets the last value.
The options visitor works the same. Internally, it stores only lists. When the visit is for a scalar, it gets the last value from the list. When the visit is for a list, it sets up things so that the list element visits get the values in order.
As long as no key is list-valued, you can keep @permit_repeated_opts false, and have duplicate keys rejected, for whatever that's worth.
As soon as you have a list-valued key, you need to set @permit_repeated_opts to true. But then *no* duplicate keys are rejected. If consistency with @permit_repeated_opts = false is needed, then something else needs to detect unwanted lists and reject them.
What use cases for @permit_repeated_opts = false do you envisage?
To serve as replacement for the options visitor, this needs to treat duplicates the same. As we saw in PATCH 12, QEMU_OPTS_REPEAT_POLICY_ALL makes a list when the key is repeated, while the options visitor makes a list always. I guess some other patch hides this difference.
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
In reply to Markus ArmbrusterAm 18.10.2016 um 17:35 hat Markus Armbruster geschrieben:
Quoted 10 lines
Quoted 5 lines · Kevin Wolf
Of course, you could argue that flat QDicts are the wrong data structure in the first place and instead of flatting everything we should have done the equivalent of
qdict_crumplefrom the beginning, but they were the natural extension of what was already there and happened to work good enough, so this is what we're currently using.We didn't "flatten everything", because QemuOpts is and has always been flat to begin with. Rather we've started to crumple a few things, and in a separate layer.
That's the QemuOpts point of view.
I was talking from a block layer view. There we get data in two different formats, QMP and the command line. The first is structured, the second is flat. We need to make both uniform before we can pass them to the actual block layer functions.
The current code chose to flatten what we get from QMP blockdev-add and use that throughout the block layer. We could instead have decided that we leave the blockdev-add input as it is, crumple what we get from QemuOpts and use nested QObjects throughout the block layer.
Quoted 5 lines
I now think this is the wrong approach. We have clearly outgrown flat options. We should redo QemuOpts to support what we need instead of bolting on an optional crumpling layer.
I guess I reached the place you described, just on a different path :)
Yes, I think the conclusion is easy to agree on.
Quoted 10 lines
Quoted 9 lines
Okay, so I like JSON. It's a great format for our monitor protocol. We even have pretty printers so that it's more or less readable as output for human users. However, there is one thing for which it is horrible:
Getting user input.Yes, getting rid of the comma escaping is a first step, but typing JSON on the command line remains a PITA. Mostly because of the quotes (which you probably have to escape), but also things like the useless outer brackets.
Quoted 4 lines
As long as you don't need "'" in your JSON, you can simply enclose in "'" and be done. Since "'" can only occur in JSON strings, and the same strings would be present in any other syntax, any other syntax would be pretty much the same PITA.
I've written enough scripts (qemu-iotests cases) that produce JSON with shell variables in it, so if anything you can use "" quoting for the shell and make use of qemu's extension that '' is accepted in JSON, too.
Anyway, the quotes aren't only nasty because of the escaping, but also just because I have to type them.
Quoted 25 lines
Quoted 11 lines
Quoted 7 lines
If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try
driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]This looks a lot more agreeable as something that I have to type on the command line.
Quoted 7 lines
What's more, this is a direct extension of the existing syntax. You complained about how the step from simple configurations with -hda to more complex ones with -drive completely changes the syntax (and rightly so). Going from simple QemuOpts syntax to doing JSON as soon as you get structured values and lists would be another similar step. In contrast, the new syntax you're suggesting above is a natural extension of what's there.
Point taken. I just don't like inventing syntax, because as a rule, way too much syntax gets invented, and almost always badly.
Quoted 3 lines
Quoted 2 lines
I'd go with some existing syntax, though. The one we already use is JSON.
Quoted 8 lines
Quoted 2 lines
The one we already use in this context is comma separated key/value pairs as parsed by QemuOpts.
Which is flat, and flat doesn't cut it anymore.
If you make it non-flat with dotted key convention, the dotted key convention becomes part of the syntax. Counts as inventing syntax in my book.
Yes, it is. Though in the context of command line options, dotted syntax is an invention already made, whereas JSON would be a new invention. (Well, not completely, because for block devices we already have json: - but that works a little different again.)
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Your dotted key convention requires two rules: 1. names must not look like integers, and 2. names must not contain '.'.
We can avoid rule 2 by requiring '.' to be escaped. Dan's
qdict_crumple()currently does that, to your surprise. Adding the escaping to existing options is a compatibility break, however. So, if names with '.' already exist, we can either break compatibility by renaming them, or break it by requiring the '.' to be escaped.
Quoted 4 lines
Quoted 2 lines
I don't think we should support any escaping and rather forbid '.' completely in names.
I think we should adopt QAPI's naming rules.
Which includes what I said, so fine with me.
Quoted 17 lines
Quoted 7 lines
If you're concerned about compatibility issues if we find a dot in a name only in one of the later users: We can handle it. If you then stumble across an obscure "weird.name" property where the dot is really part of the name and not already representing structure, you can still artificially reinterpret it as a nested structure with a single field even if logically that's not what it is... Might be a bit ugly, but keeps it working.
Yes, that's how we'd do backward compatibility, if needed.
So, what can we do? Perhaps something like this:
* Pick a concrete syntax that supports nested stuff. Proposals on the table so far are
1. Dotted key convention as used in the block layer
Quoted 8 lines
2. Array syntax as used in QOM's automatic arrayification (not a complete solution, mentioned because it conflicts with the previous)
3. JSON
4. JSON with ':' replaced by '=', double-quotes and the outermost pair of braces dropped.
Biased block layer view on what we ideally want to have in the end:
Anthony's QCFG proposal plus a [] syntax for lists.
This is more or less what you describe in option 4, except that the existing dotted syntax is still supported. It is part of the ABI by now, so we can't get rid of it anyway, and supporting it everywhere is not only more consistent but probably also easier than making it a special case.
The tricky part here is the {} and [] syntax for dicts and lists:
Parsing this requires a schema if you don't want to restrict string values to not contain any of these characters. Such a restriction would be incompatible, so we probably can't make it.
Essentially this means that we're back to "QAPIfy everything" before we can make this work. Leaves us with implementing only the dotted syntax subset for now, which is equally powerful, even if a lot more verbose.
Quoted 11 lines
* Replace the QemuOpts internal representation: QDict instead of list.
* Replace the QemuOpts parser.
* Add suitable struct- and list-valued accessors.
qdict_crumple(qemu_opt_to_qdict(...), true, &errp)gets replaced by a trivial "get the whole dictionary" operation.Other uses of
qemu_opt_to_qdict()need to be rewritten.
Quoted 8 lines
* "Repetition builds a list" backward compatibility. It's used in only a few places.
* Possibly "keys with funny characters" backward compatibility.
Feels doable to me. 2.8 would be ambitious, though: soft freeze in in less than two weeks. However, getting this series into 2.8 has started to feel ambitious to me, too.
Let's merge qdict_crumple() at least, this would allow us to add the SSH and NBD blockdev-add support for 2.8, possibly NFS to. No command line syntax involved at all there. I can take the patch through my tree if nobody objects.
With -blockdev I'll have to wait for the final decision, but if we decide to use a syntax that includes dotted syntax (possibly as a subset), I think this series would provide the right external interface even if there is still some infrastructure cleanup work to do internally.
Kevin
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
Quoted 19 lines · Kevin Wolf
Am 18.10.2016 um 17:35 hat Markus Armbruster geschrieben:
Quoted 10 lines
Quoted 5 lines · Kevin Wolf
Of course, you could argue that flat QDicts are the wrong data structure in the first place and instead of flatting everything we should have done the equivalent of
qdict_crumplefrom the beginning, but they were the natural extension of what was already there and happened to work good enough, so this is what we're currently using.We didn't "flatten everything", because QemuOpts is and has always been flat to begin with. Rather we've started to crumple a few things, and in a separate layer.
That's the QemuOpts point of view.
I was talking from a block layer view. There we get data in two different formats, QMP and the command line. The first is structured, the second is flat. We need to make both uniform before we can pass them to the actual block layer functions.
Quoted 4 lines
The current code chose to flatten what we get from QMP blockdev-add and use that throughout the block layer. We could instead have decided that we leave the blockdev-add input as it is, crumple what we get from QemuOpts and use nested QObjects throughout the block layer.
I would very much like to see BlockdevOptions structs passed around the block drivers, instead of QemuOpts - I think it'd lead to much clearer code than the QemuOpts stuff we have today IMHO
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [Qemu-block] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
In reply to Markus ArmbrusterQuoted 11 lines · Markus Armbruster
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
But even if I realised that QemuOpts support this syntax, I think we would still have to use the dotted syntax because it's explicit about the index and we need that because the list can contains dicts.
Compare this:
driver=quorum, child.0.driver=file,child.0.filename=disk1.img, child.1.driver=host_device,child.1.filename=/dev/sdb, child.2.driver=nbd,child.2.host=localhost
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
And this:
driver=quorum, child.driver=file,child.filename=disk1.img, child.driver=host_device,child.filename=/dev/sdb, child.driver=nbd,child.host=localhost
Aside: both are about equally illegible, to be honest.
Furthermore, both of these currently require really long command lines. Overnight, I was thinking about whether it would be worth improving QemuOpts to ignore whitespace after (unquoted) commas, so you can do:
-drive 'driver=quorum,
child.0.driver=file, child.0.filename=disk1.img,
child.1.driver=host_device, child.1.filename=/dev/sdb,
child.2.driver=nbd, child.2.host=localhost
'
I think this one should be just fine - we don't have ANY keys that start with leading whitespace, so ignoring space will still let us parse out key names, but allow for much more legibility in the command lines. And this is true whether we use dotted notation...
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
If you'd rather invent syntax closer to QemuOpts than reuse JSON, you could try
driver=quorum,
child=[{ driver=file, filename=disk1.img },
{ driver=host_device, filename=/dev/sdb }, { driver=nbd, host=localhost } ]This looks a lot more agreeable as something that I have to type on the command line.
...or new syntax (where of course the new syntax has already demonstrated that we want to support strategic ignoring of whitespace).
So I'll probably propose a patch along those lines soon, as it seems like an orthogonal improvement.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 15/21] qom: support non-scalar properties with -object
In reply to Daniel P. BerrangeQuoted 58 lines · Daniel P. Berrange
The current -object command line syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.
This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.
The QObjectInputVisitor now has functionality that allows it to replace OptsVisitor, maintaining full backwards compatibility for previous CLI syntax, while also allowing use of new syntax for structs.
Thus -object can now support non-scalar properties, for example the QMP object:
{ "execute": "object-add", "arguments": { "qom-type": "demo", "id": "demo0", "parameters": { "foo": [ { "bar": "one", "wizz": "1" }, { "bar": "two", "wizz": "2" } ] } } }Would be creatable via the CLI now using
$QEMU \ -object demo,id=demo0,\ foo.0.bar=one,foo.0.wizz=1,\ foo.1.bar=two,foo.1.wizz=2Notice that this syntax is intentionally compatible with that currently used by block drivers. NB the indentation seen here after line continuations should not be used in reality, it is just for clarity in this commit message.
Signed-off-by Daniel P. Berrange <[email protected]>--- qapi/qobject-input-visitor.c | 2 +- qom/object_interfaces.c | 37 ++++- tests/check-qom-proplist.c | 367 ++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 397 insertions(+), 9 deletions(-)qapi/qobject-input-visitor.c +0
-0
qapi/qobject-input-visitor.c static void qobject_input_start_struct(Visitor *v, const char *name, void **obj, · line 210
Quoted 8 lines
qapi/qobject-input-visitor.c +1
-1@@ -204,7 +204,7 @@ static void qobject_input_start_struct(Visitor *v, const char *name, void **obj, *obj =NULL ; } -if (!qobj && (qiv->struct_level < qiv->autocreate_struct_levels)) { +if (!qobj && (qiv->struct_level <= qiv->autocreate_struct_levels)) {/* Create a new dict that contains all the currently * unvisited items */ if (tos) {
Uh, can you explain this hunk? The < comes from PATCH 10.
Quoted 23 lines
qom/object_interfaces.c +4
-2@@ -4,7 +4,8 @@#include "qemu/module.h" #include "qapi-visit.h" #include "qapi/qobject-output-visitor.h" -#include "qapi/opts-visitor.h" +#include "qapi/qobject-input-visitor.h" +#include "qemu/option.h" void user_creatable_complete (Object *obj,Error **errp) {@@ -63,12 +64,16 @@ Object *user_creatable_add(const QDict *qdict,if (local_err) { goto out_visit; } -visit_check_struct (v, &local_err); + + obj =user_creatable_add_type (type, id, pdict, v, &local_err);if (local_err) { goto out_visit;
qom/object_interfaces.c Object *user_creatable_add(const QDict *qdict, · line 79
Quoted 10 lines
qom/object_interfaces.c +4
-1} - obj =user_creatable_add_type (type, id, pdict, v, &local_err); +visit_check_struct (v, &local_err); +if (local_err) { + goto out_visit; + } out_visit:visit_end_struct (v,NULL );
Can you explain why you swap the order of visit_check_struct() and user_creatable_add_type()? It smells like a bug fix to me...
Odd: opts_check_struct() does nothing unless depth == 0. But depth is at least 1 between opts_start_struct() and opts_end_struct(). Bug?
qom/object_interfaces.c Object *user_creatable_add_type(const char *type, const char *id, · line 125
Quoted 9 lines
qom/object_interfaces.c +1
-1@@ -114,7 +119,7 @@ Object *user_creatable_add_type(const char *type, const char *id,assert (qdict); obj =object_new (type); -visit_start_struct (v,NULL ,NULL ,0 , &local_err); +visit_start_struct (v,"props" ,NULL ,0 , &local_err);if (local_err) { goto out; }
Why this hunk?
Quoted 12 lines
qom/object_interfaces.c +3
-2@@ -158,14 +163,32 @@ Object *user_creatable_add_opts(QemuOpts *opts, Error **errp) {Visitor *v;QDict *pdict; +QObject *pobj;Object *obj =NULL ; - v =opts_visitor_new (opts); - pdict =qemu_opts_to_qdict (opts,NULL ,QEMU_OPTS_REPEAT_POLICY_LAST , + pdict =qemu_opts_to_qdict (opts,NULL , +QEMU_OPTS_REPEAT_POLICY_ALL , &error_abort);
qom/object_interfaces.c Object *user_creatable_add_opts(QemuOpts *opts, Error **errp) · line 180
Quoted 10 lines
qom/object_interfaces.c +8
-1- obj =user_creatable_add (pdict, v, errp); + pobj =qdict_crumple (pdict, true, errp); +if (!pobj) { + goto cleanup; + } +/* +* We need autocreate_list=true + permit_int_ranges=true +* in order to maintain compat with OptsVisitor creation +* of the 'numa' object which had an int16List property.
I can't find a "numa" object in the object-add sense, only a NumaOptions QAPI object created by -numa. Is that what you mean?
qom/object_interfaces.c Object *user_creatable_add_opts(QemuOpts *opts, Error **errp) · line 184
Quoted 4 lines
qom/object_interfaces.c +4
-0+* +* We need autocreate_structs=1, because at the CLI level +* we have the object type + properties in the same flat +* struct, even though at the QMP level they are nested.
We screwed up QMP object-add. device_add, netdev_add and object_add all treat additional arguments as properties *except* for QMP object-add, which has them wrapped in a JSON object instead.
Aside: if someone foolishly adds a property named "qom-type" or "id", it'll work in QMP but not in HMP or -object.
The inconsistency complicates the code even before this patch:
* Core: user_creatable_add_type() takes properties in *two* forms: as a visitor and as a qdict. It visits a struct with the members given by the qdict's keys. That's its only use of the qdict.
* QMP: qmp_object_add() gets properties as a separate QDict. It creates a QObject visitor for it, and passes it to user_creatable_add_type() along with the QDict.
* -object: user_creatable_add_opts() gets the option argument as a QemuOpts. It creates an options visitor for it *and* converts it to a QDict, then passes both to user_creatable_add().
* user_creatable_add() visits a struct. It first visits the non-property arguments "qom-type" and "id". It passes the visitor and a shallow copy of the QDict with the non-property entries deleted to user_creatable_add_type() to visit the remaining struct members.
* HMP: hmp_object_add() gets its arguments as a QDict. It converts it to QemuOpts and creates an options visitor for it *boggle*. It passes the visitor along with original QDict to user_creatable_add().
I still can't quite see what's requiring autocreate_structs. But I hope we can get rid of it by cleaning up this mess a bit. Here's how I'd try:
* Move visit_start_struct() and visit_end_struct() from user_creatable_add_type() to its two callers qmp_object_add() and user_creatable_add().
* Make user_creatable_add_type() ignore QDict keys "qom-type" and "id".
* QMP: any attempt to specify property "qom-type" or "id" now fails, because visit_check_struct() flags them as unexpected.
* user_creatable_add(): ditch the silly QDict copying.
* hmp_object_add(): ditch the silly conversion to QemuOpts, and create a QObject visitor instead.
qom/object_interfaces.c Object *user_creatable_add_opts(QemuOpts *opts, Error **errp) · line 194
Quoted 10 lines
qom/object_interfaces.c +6
-0+*/ + v =qobject_input_visitor_new_autocast (pobj, true,1 , true); + + obj =user_creatable_add (qobject_to_qdict (pobj), v, errp);visit_free (v); +qobject_decref (pobj); + cleanup:QDECREF (pdict);return obj; }
[Skipping test updates for now...]
Re: [Qemu-devel] [PATCH v14 16/21] hmp: support non-scalar properties with object_add
In reply to Daniel P. BerrangeQuoted 78 lines · Daniel P. Berrange
The current
object_addHMP syntax only allows for creation of objects with scalar properties, or a list with a fixed scalar element type. Objects which have properties that are represented as structs in the QAPI schema cannot be created using -object.This is a design limitation of the way the OptsVisitor is written. It simply iterates over the QemuOpts values as a flat list. The support for lists is enabled by allowing the same key to be repeated in the opts string.
The QObjectInputVisitor now has functionality that allows it to replace OptsVisitor, maintaining full backwards compatibility for previous CLI syntax, while also allowing use of new syntax for structs.
Thus -object can now support non-scalar properties, for example the QMP object:
{ "execute": "object-add", "arguments": { "qom-type": "demo", "id": "demo0", "parameters": { "foo": [ { "bar": "one", "wizz": "1" }, { "bar": "two", "wizz": "2" } ] } } }Would be creatable via the HMP now using
object_add demo,id=demo0,\ foo.0.bar=one,foo.0.wizz=1,\ foo.1.bar=two,foo.1.wizz=2Notice that this syntax is intentionally compatible with that currently used by block drivers. NB the indentation seen here after line continuations should not be used in reality, it is just for clarity in this commit message.
Signed-off-by Daniel P. Berrange <[email protected]>--- hmp.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-)hmp.c +4
-5@@ -25,7 +25,7 @@#include "qemu/sockets.h" #include "monitor/monitor.h" #include "monitor/qdev.h" -#include "qapi/opts-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "qapi/qmp/qerror.h" #include "qapi/string-output-visitor.h" #include "qapi/util.h" @@ -1696,21 +1696,30 @@ void hmp_netdev_del(Monitor *mon, const QDict *qdict)void hmp_object_add (Monitor *mon,const QDict *qdict) {Error *err =NULL ; -QemuOpts *opts;Visitor *v;Object *obj =NULL ; +QObject *pobj; - opts =qemu_opts_from_qdict (qemu_find_opts ("object" ), qdict, &err); + pobj =qdict_crumple (qdict, true, &err);if (err) { -hmp_handle_error (mon, &err); -return ; + goto cleanup;
hmp.c void hmp_netdev_del(Monitor *mon, const QDict *qdict) · line 1713
Quoted 10 lines
hmp.c +6
-2} - v =opts_visitor_new (opts); - obj =user_creatable_add (qdict, v, &err); +/* +* We need autocreate_list=true + permit_int_ranges=true +* in order to maintain compat with OptsVisitor creation +* of the 'numa' object which had an int16List property. +* +* We need autocreate_structs=1, because at the CLI level
autocreate_struct_levels=1
Same in previous patch.
Quoted 13 lines
hmp.c +7
-1+* we have the object type + properties in the same flat +* struct, even though at the QMP level they are nested. +*/ + v =qobject_input_visitor_new_autocast (pobj, true,1 , true); + obj =user_creatable_add (qobject_to_qdict (pobj), v, &err);visit_free (v); -qemu_opts_del (opts); + cleanup: +qobject_decref (pobj);if (err) {hmp_handle_error (mon, &err); }
Re: [Qemu-devel] [PATCH v14 17/21] numa: convert to use QObjectInputVisitor for -numa
In reply to Daniel P. BerrangeQuoted 120 lines · Daniel P. Berrange
Switch away from using OptsVisitor to parse the -numa argument processing. This enables use of the modern list syntax for specifying CPUs. e.g. the old syntax
-numa node,nodeid=0,cpus=0-3,cpus=8-11,mem=107
is equivalent to
-numa node,nodeid=0,cpus.0=0,cpus.1=1,cpus.2=2,cpus.3=3,\ cpus.4=8,cpus.5=9,cpus.6=10,cpus.7=11,mem=107Furthermore, the cli arg can now follow the QAPI schema nesting, so the above is equivalent to the canonical syntax:
-numa type=node,data.nodeid=0,data.cpus.0=0,data.cpus.1=1,\ data.cpus.2=2,data.cpus.3=3,data.cpus.4=8,data.cpus.5=9,\ data.cpus.6=10,data.cpus.7=11,data.mem=107A test case is added to cover argument parsing to validate that both the old and new syntax is correctly handled.
Signed-off-by Daniel P. Berrange <[email protected]>--- include/sysemu/numa_int.h | 11 +++++ numa.c | 36 +++++++++----- stubs/Makefile.objs | 5 ++ stubs/exec.c | 6 +++ stubs/hostmem.c | 14 ++++++ stubs/memory.c | 41 ++++++++++++++++ stubs/qdev.c | 8 ++++ stubs/vl.c | 8 ++++ stubs/vmstate.c | 4 ++ tests/Makefile.include | 2 + tests/test-numa.c | 116 ++++++++++++++++++++++++++++++++++++++++++++++ 11 files changed, 240 insertions(+), 11 deletions(-) create mode 100644 include/sysemu/numa_int.h create mode 100644 stubs/exec.c create mode 100644 stubs/hostmem.c create mode 100644 stubs/memory.c create mode 100644 stubs/qdev.c create mode 100644 stubs/vl.c create mode 100644 tests/test-numa.cinclude/sysemu/numa_int.h new file +11
-0@@ -0,0 +1,11 @@ +#ifndef SYSEMU_NUMA_INT_H +#define SYSEMU_NUMA_INT_H + +#include "sysemu/numa.h" + +extern int have_memdevs; +extern int max_numa_nodeid; + +int parse_numa (void *opaque,QemuOpts *opts,Error **errp); + +#endif numa.c +11
-9@@ -23,14 +23,14 @@*/ #include "qemu/osdep.h" -#include "sysemu/numa.h" +#include "sysemu/numa_int.h" #include "exec/cpu-common.h" #include "qemu/bitmap.h" #include "qom/cpu.h" #include "qemu/error-report.h" #include "include/exec/cpu-common.h" /* for RAM_ADDR_FMT */ #include "qapi-visit.h" -#include "qapi/opts-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "hw/boards.h" #include "sysemu/hostmem.h" #include "qmp-commands.h" @@ -45,10 +45,10 @@ QemuOptsList qemu_numa_opts = { .desc = { { 0 } } /* validated with OptsVisitor */ }; -static int have_memdevs =-1 ; -static int max_numa_nodeid;/* Highest specified NUMA node ID, plus one. -* For all nodes, nodeid < max_numa_nodeid -*/ +int have_memdevs =-1 ; +int max_numa_nodeid;/* Highest specified NUMA node ID, plus one. +* For all nodes, nodeid < max_numa_nodeid +*/ int nb_numa_nodes;NodeInfo numa_info[MAX_NODES ];@@ -189,6 +189,9 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp)if (node->has_mem) {uint64_t mem_size = node->mem;const char *mem_str =qemu_opt_get (opts,"mem" ); +if (!mem_str) { + mem_str =qemu_opt_get (opts,"data.mem" ); + }/* Fix up legacy suffix-less format */ if (g_ascii_isdigit (mem_str[strlen (mem_str) -1 ])) { mem_size <<=20 ;@@ -211,16 +214,27 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp) max_numa_nodeid =MAX (max_numa_nodeid, nodenr +1 ); } -static int parse_numa (void *opaque,QemuOpts *opts,Error **errp) +int parse_numa (void *opaque,QemuOpts *opts,Error **errp) {NumaOptions *object =NULL ;Error *err =NULL ; +Visitor *v; - { -Visitor *v =opts_visitor_new (opts);
numa.c static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp) · line 230
Quoted 10 lines
numa.c +8
-2-visit_type_NumaOptions (v,NULL , &object, &err); -visit_free (v); +/* +* Needs autocreate_list=true, permit_int_ranges=true and +* permit_repeated_opts=true in order to support existing +* syntax for 'cpus' parameter which is an int list. +* +* Needs autocreate_struct_levels=1, because existing syntax +* used a nested struct in the QMP schema with a flat namespace +* in the CLI args.
Which QAPI definition(s) do you allude to here?
If it's just union NumaOptions, that's not ABI. I'd simply flatten it. Sketch appended.
numa.c static void numa_node_parse(NumaNodeOptions *node, QemuOpts *opts, Error **errp) · line 240
Quoted 10 lines
numa.c +6
-0+*/ + v =qobject_input_visitor_new_opts (opts, true,1 , true, true, &err); +if (err) { + goto end; } +visit_type_NumaOptions (v,NULL , &object, &err); +visit_free (v);if (err) { goto end;
[Skipping test updates for now...]
numa.c +2 -2
@@ -227,8 +227,8 @@ static int parse_numa(void *opaque, QemuOpts *opts, Error **errp) }switch (object->type) { -case NUMA_OPTIONS_KIND_NODE : -numa_node_parse (object->u.node.data, opts, &err); +case NUMA_OPTIONS_TYPE_NODE : +numa_node_parse (&object->u.node, opts, &err);if (err) { goto end; }
qapi-schema.json +3 -0
@@ -4255,7 +4255,10 @@ # # Since 2.1 ## +{ 'enum': 'NumaOptionsType', 'data': ['node'] } { 'union': 'NumaOptions', + 'base': { 'type': 'NumaOptionsType' }, + 'discriminator': 'type', 'data': { 'node': 'NumaNodeOptions' }}
Re: [Qemu-devel] [PATCH v14 20/21] net: convert to QObjectInputVisitor for -net/-netdev parsing
In reply to Daniel P. BerrangeQuoted 45 lines · Daniel P. Berrange
The -net/-netdev command line parsing code uses OptsVisitor for parsing options to populate NetLegacy or NetDev struct respectively. Although those structs have nesting, the OptsVisitor flattens them, so we must enable compatibility options to auto-create structs. This allows the legacy syntax
-net tap,id=net0,vlan=3,fd=3,script=/bin/ifup-qemu
to be treated as equivalent to the modern QAPI based syntax
-net id=net0,vlan=3,opts.type=tap,opts.data.fd=3,opts.data.script=/bin/ifup-qemu
Signed-off-by Daniel P. Berrange <[email protected]>--- net/net.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)net/net.c +9
-2@@ -43,7 +43,7 @@#include "qemu/iov.h" #include "qemu/main-loop.h" #include "qapi-visit.h" -#include "qapi/opts-visitor.h" +#include "qapi/qobject-input-visitor.h" #include "sysemu/sysemu.h" #include "net/filter.h" #include "qapi/string-output-visitor.h" @@ -1069,7 +1069,21 @@ int net_client_init(QemuOpts *opts, bool is_netdev, Error **errp)void *object =NULL ;Error *err =NULL ;int ret =-1 ; -Visitor *v =opts_visitor_new (opts); +/* +* Needs autocreate_lists=true in order support existing +* syntax for list options where the bare key is repeated +* +* Needs autocreate_struct_levels=3 in order to deal with +* 3 level nesting in NetLegacy option args, which was +* exposed as a flat namespace with OptVisitor +*/
net/net.c int net_client_init(QemuOpts *opts, bool is_netdev, Error **errp) · line 1089
Quoted 10 lines
net/net.c +7
-0+Visitor *v =qobject_input_visitor_new_opts (opts, true,3 , false, true, + &err); + +if (err) { +error_propagate (errp, err); +return -1 ; + } {/* Parse convenience option format ip6-net=fec0::0[/64] */
Neither NetLegacy nor Netdev are ABI, so if I understand the problem, perhaps I can find a way around it. Let's figure out what exactly requires levels=3.
Your commit message shows opts.* and opts.data.* in -net. Anything else?
I can see opts.* in the QAPI schema: it's NetLegacy member @opts, of union type NetLegacyOptions. It's boxed on the wire, as any member of complex type is.
I can also see opts.data.*: NetLegacyOptions is a simple union, which wraps 'data': { ... } around the variant on the wire.
The latter wrapping is easy to avoid: make NetLegacyOptions a flat union.
The former wrapping isn't so easy. QAPI can't unbox complex members. The only way to include a type unboxed is making it the base type. But base types must be struct, and @opts is a union.
However, if you take a step back you can see that the (flat) argument of -net / -netdev is fundamentally just a (flat) union! The nesting is an artifact of the clumsy way it's defined in the QAPI schema. Sketch doing it less clumsily appended; look ma, no nesting! Compile-tested only.
net/net.c +23 -24
@@ -984,55 +984,54 @@ static int net_client_init1(const void *object, bool is_netdev, Error **errp) } }else {const NetLegacy *net = object; -const NetLegacyOptions *opts = net->opts; legacy.id = net->id; netdev = &legacy;/* missing optional values have been initialized to "all bits zero" */ name = net->has_id ? net->id : net->name;/* Map the old options to the new flat type */ -switch (opts->type) { -case NET_LEGACY_OPTIONS_KIND_NONE : +switch (net->type) { +case NET_LEGACY_TYPE_NONE :return 0 ;/* nothing to do */ -case NET_LEGACY_OPTIONS_KIND_NIC : +case NET_LEGACY_TYPE_NIC : legacy.type =NET_CLIENT_DRIVER_NIC ; - legacy.u.nic = *opts->u.nic.data; + legacy.u.nic = net->u.nic;break ; -case NET_LEGACY_OPTIONS_KIND_USER : +case NET_LEGACY_TYPE_USER : legacy.type =NET_CLIENT_DRIVER_USER ; - legacy.u.user = *opts->u.user.data; + legacy.u.user = net->u.user;break ; -case NET_LEGACY_OPTIONS_KIND_TAP : +case NET_LEGACY_TYPE_TAP : legacy.type =NET_CLIENT_DRIVER_TAP ; - legacy.u.tap = *opts->u.tap.data; + legacy.u.tap = net->u.tap;break ; -case NET_LEGACY_OPTIONS_KIND_L2TPV3 : +case NET_LEGACY_TYPE_L2TPV3 : legacy.type =NET_CLIENT_DRIVER_L2TPV3 ; - legacy.u.l2tpv3 = *opts->u.l2tpv3.data; + legacy.u.l2tpv3 = net->u.l2tpv3;break ; -case NET_LEGACY_OPTIONS_KIND_SOCKET : +case NET_LEGACY_TYPE_SOCKET : legacy.type =NET_CLIENT_DRIVER_SOCKET ; - legacy.u.socket = *opts->u.socket.data; + legacy.u.socket = net->u.socket;break ; -case NET_LEGACY_OPTIONS_KIND_VDE : +case NET_LEGACY_TYPE_VDE : legacy.type =NET_CLIENT_DRIVER_VDE ; - legacy.u.vde = *opts->u.vde.data; + legacy.u.vde = net->u.vde;break ; -case NET_LEGACY_OPTIONS_KIND_DUMP : +case NET_LEGACY_TYPE_DUMP : legacy.type =NET_CLIENT_DRIVER_DUMP ; - legacy.u.dump = *opts->u.dump.data; + legacy.u.dump = net->u.dump;break ; -case NET_LEGACY_OPTIONS_KIND_BRIDGE : +case NET_LEGACY_TYPE_BRIDGE : legacy.type =NET_CLIENT_DRIVER_BRIDGE ; - legacy.u.bridge = *opts->u.bridge.data; + legacy.u.bridge = net->u.bridge;break ; -case NET_LEGACY_OPTIONS_KIND_NETMAP : +case NET_LEGACY_TYPE_NETMAP : legacy.type =NET_CLIENT_DRIVER_NETMAP ; - legacy.u.netmap = *opts->u.netmap.data; + legacy.u.netmap = net->u.netmap;break ; -case NET_LEGACY_OPTIONS_KIND_VHOST_USER : +case NET_LEGACY_TYPE_VHOST_USER : legacy.type =NET_CLIENT_DRIVER_VHOST_USER ; - legacy.u.vhost_user = *opts->u.vhost_user.data; + legacy.u.vhost_user = net->u.vhost_user;break ;default :abort ();
@@ -1047,7 +1046,7 @@ static int net_client_init1(const void *object, bool is_netdev, Error **errp)/* Do not add to a vlan if it's a nic with a netdev= parameter. */ if (netdev->type !=NET_CLIENT_DRIVER_NIC || - !opts->u.nic.data->has_netdev) { + !net->u.nic.has_netdev) { peer =net_hub_add_port (net->has_vlan ? net->vlan :0 ,NULL ); } }
qapi-schema.json +8 -13
@@ -2915,21 +2915,16 @@ # # Since 1.2 ## -{ 'struct': 'NetLegacy', - 'data': { +{ 'enum': 'NetLegacyType', + 'data': [ 'none', 'nic', 'user', 'tap', 'l2tpv3', 'socket', 'vde', + 'dump', 'bridge', 'netmap', 'vhost-user' ] } +{ 'union': 'NetLegacy', + 'base': { + 'type': 'NetLegacyType', '*vlan': 'int32', '*id': 'str', - '*name': 'str', - 'opts': 'NetLegacyOptions' } } - -## -# @NetLegacyOptions -# -# Like Netdev, but for use only by the legacy command line options -# -# Since 1.2 -## -{ 'union': 'NetLegacyOptions', + '*name': 'str' }, + 'discriminator': 'type', 'data': { 'none': 'NetdevNoneOptions', 'nic': 'NetLegacyNicOptions',
Re: [Qemu-devel] [Qemu-block] [PATCH v14 20/21] net: convert to QObjectInputVisitor for -net/-netdev parsing
Quoted 21 lines · Markus Armbruster
Quoted 21 lines
(quoted diff) +15
-1@@ -1069,7 +1069,21 @@ int net_client_init(QemuOpts *opts, bool is_netdev, Error **errp) void *object = NULL; Error *err = NULL; int ret = -1; - Visitor *v = opts_visitor_new(opts); + /* + * Needs autocreate_lists=true in order support existing + * syntax for list options where the bare key is repeated + * + * Needs autocreate_struct_levels=3 in order to deal with + * 3 level nesting in NetLegacy option args, which was + * exposed as a flat namespace with OptVisitor + */ + Visitor *v = qobject_input_visitor_new_opts(opts, true, 3, false, true, + &err); + + if (err) { + error_propagate(errp, err); + return -1; + }
Quoted 6 lines
Review int net_client_init(QemuOpts *opts, bool is_netdev, Error **errp) · line 1089
Quoted 2 lines
(quoted diff) +0
-0{ /* Parse convenience option format ip6-net=fec0::0[/64] */Neither NetLegacy nor Netdev are ABI, so if I understand the problem, perhaps I can find a way around it. Let's figure out what exactly requires levels=3.
Netdev is not ABI only because we decided to NOT apply the last patch of QAPI-fying it in 2.7 while deciding to handle the back-compat (non-?)issues that existing netdev_add QMP command accepts both 1 and "1", but using the Netdev type would accept only 1.
While working towards making netdev_add use QAPI, I intentionally left NetLegacy unchanged; but since NetLegacy is solely used by the command line, feel free to rearrange that type as long as CLI back-compat is kept.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict
In reply to Markus ArmbrusterQuoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{ 'foo.0.bar': 'one', 'foo.0.wizz': '1', 'foo.1.bar': 'two', 'foo.1.wizz': '2'}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{ 'foo': [ { 'bar': 'one', 'wizz': '1' }, { 'bar': 'two', 'wizz': '2' } ],}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{ 'foo..bar': 'wizz', 'bar.foo..bar': 'eek', 'bar.hello': 'world' }Will end up as
{ 'foo.bar': 'wizz', 'bar': { 'foo.bar': 'eek', 'hello': 'world' } }The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.
Reviewed-by Eric Blake <[email protected]>Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 261 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 551 insertions(+)include/qapi/qmp/qdict.h +1
-0@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);qobject/qdict.c +0
-0[...]
Quoted 6 lines
Quoted 4 lines
+/**
+ *qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionariesIs recursive=false used outside tests in this series?
No, its not used.
It was suggested in a way earlier version by Max, but not sure if his code uses it or not.
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [PATCH v14 09/21] qapi: permit auto-creating single element lists
In reply to Markus ArmbrusterQuoted 17 lines · Markus Armbruster
Quoted 2 lines · Daniel P. Berrange
When converting QemuOpts to a QObject, there is no information about compound types available,
Yes, that's a drawback of splitting the conversion into a QemuOpts -> QObject part that is oblivious of types, and a QObject -> QAPI object part that knows the types.
Quoted 8 lines
so when visiting a list, the
corresponding QObject is not guaranteed to be a QList. We therefore need to be able to auto-create a single element QList from whatever type we find.This mode should only be enabled if you have compatibility requirements for
Quoted 8 lines
Quoted 5 lines
-arg foo=hello,foo=world
to be treated as equivalent to the preferred syntax:
-arg foo.0=hello,foo.1=worldNot sure this is "preferred". "More powerfully warty" is probably closer to the truth ;)
Well, I call it "preferred" in the sense that that option syntax directly maps to the QAPI syntax in an unambigous manner. ie given the arg value alone "foo.0=hello,foo.1=world" you can clearly determine that "foo" is a list. With the compat syntax you cannot distinguish list from scalar, without knowing the QAPI schema.
Quoted 1 line
How is "-arg foo=hello,foo=world" treated if this mode isn't enabled?
The default behaviour would be that only the last key is present in the dict, eg foo=world, and then if you tried to visit a list, the visitor would complain that its got a QString instead of QList for the key 'foo'.
This is related to patch 14
Quoted 2 lines
What would be the drawbacks of doing this always instead of only when we "have compatibility requirements"?
Essentially we'd be permanently allowing 2 distinct syntaxes for dealing with lists, for all options. I felt it desirable that we have only a single syntax and only allow this alt syntax in the backcompat cases.
Quoted 29 lines
Quoted 29 lines
qapi/qobject-input-visitor.c +6
-0@@ -48,6 +48,10 @@ struct QObjectInputVisitor/* True to reject parse in visit_end_struct() if unvisited keys remain. */ bool strict; + +/* Whether we can auto-create single element lists when +* encountering a non-QList type */ +bool autocreate_list; };static QObjectInputVisitor *to_qiv (Visitor *v)@@ -108,6 +112,7 @@ static const QListEntry *qobject_input_push(QObjectInputVisitor *qiv,assert (obj); tos->obj = obj; tos->qapi = qapi; +qobject_incref (obj);if (qiv->strict &&qobject_type (obj) ==QTYPE_QDICT ) { h =g_hash_table_new (g_str_hash, g_str_equal);@@ -147,6 +152,7 @@ static void qobject_input_stack_object_free(StackObject *tos)if (tos->h) {g_hash_table_unref (tos->h); } +qobject_decref (tos->obj);
Quoted 4 lines
qapi/qobject-input-visitor.c static void qobject_input_stack_object_free(StackObject *tos) · line 158
Quoted 2 lines
qapi/qobject-input-visitor.c +0
-0g_free (tos); }Can you explain the reference counting change?
Previously the stack stored a borrowed reference, since it didn't ever need responsibility for free'ing the object when popping the stack. This is no longer the case if you look a few lines later....
Quoted 18 lines
qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 206
Quoted 7 lines
qapi/qobject-input-visitor.c +1
-1@@ -197,7 +203,7 @@ static void qobject_input_start_list(Visitor *v, const char *name,QObject *qobj =qobject_input_get_object (qiv, name, true);const QListEntry *entry; -if (!qobj ||qobject_type (qobj) !=QTYPE_QLIST ) { +if (!qobj || (!qiv->autocreate_list &&qobject_type (qobj) !=QTYPE_QLIST )) {Long line, but I believe it'll go away when you rebase for commit 1382d4a.
Quoted 7 lines
qapi/qobject-input-visitor.c +0
-0if (list) { *list =NULL ; }@@ -206,7 +212,16 @@ static void qobject_input_start_list(Visitor *v, const char *name,return ; }
Quoted 7 lines
qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 220
Quoted 7 lines
qapi/qobject-input-visitor.c +6
-1- entry =qobject_input_push (qiv, qobj, list, errp); +if (qobject_type (qobj) !=QTYPE_QLIST ) { +QList *tmplist =qlist_new (); +qlist_append_obj (tmplist, qobj); +qobject_incref (qobj); + entry =qobject_input_push (qiv,QOBJECT (tmplist), list, errp); +QDECREF (tmplist);
... here we are storing the 'qmplist' in the stack, and so when popping the stack, we must free that object. We thus need the stack to always hold its own reference, so when popping it can decref and (potentially) release the last reference.
Quoted 10 lines
qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 227
Quoted 7 lines
qapi/qobject-input-visitor.c +4
-0+ }else { + entry =qobject_input_push (qiv, qobj, list, errp); + } +if (list) {if (entry) { *list =g_malloc0 (size);Buries autolist behavior in the middle of things. What about doing it first, so it's more separate?
I'm not sure I understand what you mean here ?
Quoted 50 lines
QObjectInputVisitor *qiv = to_qiv(v); QObject *qobj = qobject_input_get_object_(qiv, name, true, errp); const QListEntry *entry; if (!qobj) { return; }+ if (
qiv->autocreate_list&&qobject_type(qobj)!=QTYPE_QLIST) {+ QList *auto_list = qlist_new(); + qlist_append_obj(auto_list, qobj); + qobj = auto_list; + } + if (qobject_type(qobj) != QTYPE_QLIST) {I ignored reference counting here, because I don't yet understand how and why you're changing it.
Quoted 30 lines
qapi/qobject-input-visitor.c +4
-1@@ -514,7 +529,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)return &v->visitor; } -Visitor *qobject_input_visitor_new_autocast (QObject *obj) +Visitor *qobject_input_visitor_new_autocast (QObject *obj, +bool autocreate_list) {QObjectInputVisitor *v;@@ -539,6 +555,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj) v->visitor.optional = qobject_input_optional; v->visitor.free = qobject_input_free; v->strict = true; + v->autocreate_list = autocreate_list; v->root = obj;qobject_incref (obj);@@ -548,6 +565,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts, +bool autocreate_list,Error **errp) {QDict *pdict;@@ -564,7 +582,8 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, goto cleanup; }
Quoted 7 lines
qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 589
Quoted 6 lines
qapi/qobject-input-visitor.c +2
-1- v =qobject_input_visitor_new_autocast (pobj); + v =qobject_input_visitor_new_autocast (pobj, + autocreate_list); cleanup:qobject_decref (pobj);QDECREF (pdict);[Skipping test updates for now...]
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor
In reply to Markus ArmbrusterQuoted 122 lines · Markus Armbruster
Quoted 107 lines · Daniel P. Berrange
The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:
-arg foo=5,foo=6,foo=7
-arg foo=5-7This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.
This means that
-arg foo=5-7
is treated as equivalent to
-arg foo.0=5,foo.1=6,foo.2=7
Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.
Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qobject-input-visitor.h | 23 ++++- qapi/qobject-input-visitor.c | 158 ++++++++++++++++++++++++++-- tests/test-qobject-input-visitor.c | 195 +++++++++++++++++++++++++++++++++-- 3 files changed, 360 insertions(+), 16 deletions(-)static void qobject_input_type_uint64(Visitor *v, const char *name,
qapi/qobject-input-visitor.c +58
-3@@ -366,21 +438,85 @@ static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, uint64_t *obj, Error **errp) { QObjectInputVisitor *qiv = to_qiv(v); -QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, - true)); +QString *qstr;unsigned long long ret; +char *end =NULL ; +StackObject *tos; +bool inlist = false; + +/* Preferentially generate values from a range, before +* trying to consume another QList element */ + tos =QSLIST_FIRST (&qiv->stack); +if (tos) { +if (tos->range_val < tos->range_limit) { + *obj = tos->range_val +1 ; + tos->range_val++; +return ; + }else { + inlist = tos->entry !=NULL ; + } + } + qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true));if (!qstr || !qstr->string) {error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" ,"string" );return ; } -if (parse_uint_full (qstr->string, &ret,0 ) <0 ) { +if (parse_uint (qstr->string, &ret, &end,0 ) <0 ) {error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" );return ; } *obj = ret; + +/* +* If we have string that represents an integer range (5-24), +* parse the end of the range and set things up so we'll process +* the rest of the range before consuming another element +* from the QList. +*/ +if (end && *end) { +if (!qiv->permit_int_ranges) { +error_setg (errp, +"Integer ranges are not permitted here" ); +return ; + } +if (!inlist) { +error_setg (errp, +"Integer ranges are only permitted when " +"visiting list parameters" ); +return ; + } +if (*end !='-' ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name, +"a number range" ); +return ; + } + end++; +if (parse_uint_full (end, &ret,0 ) <0 ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" ); +return ; + } +if (*obj > ret) { +error_setg (errp, +"Parameter '%s' range start %" PRIu64 +" must be less than (or equal to) %llu" , + name, *obj, ret); +return ; + } +if ((ret - *obj) > (QIV_RANGE_MAX -1 )) { + error_setg(errp,qapi/qobject-input-visitor.c static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, · line 520
Quoted 10 lines
qapi/qobject-input-visitor.c +9
-0+"Parameter '%s' range must be less than %d" , + name, QIV_RANGE_MAX); +return ; + } +if (*obj != ret) { + tos->range_val = *obj; + tos->range_limit = ret; + } + } }Duplicates the signed code, which is sad, but I don't have better ideas.
Quoted 2 lines
Except this one: are we actually using both the signed and the unsigned case now? If not, can we get rid of the one we don't use?
Out of the args that I converted in this series, I only see uint16List used. eg for -numa and -object hostmem
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options
In reply to Markus ArmbrusterQuoted 103 lines · Markus Armbruster
Quoted 31 lines · Daniel P. Berrange
If given an option string such as
size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind
the
qemu_opts_to_qdict()method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:size=QString("1024") nodes=QString("1-2") policy=QString("bind")With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain
size=QString("1024") nodes=QList([QString("10"), QString("4-5"), QString("1-2")]) policy=QString("bind")Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.
Quoted 3 lines
In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.
To serve as a replacement for the options visitor, this needs to be able to behave exactly the same together with a suitably hacked up QObject input visitor. Before I dive into the actual patch, let me summarize QemuOpts and options visitor behavior.
Warning, this is going to get ugly.
QemuOpts faithfully represents a key=value,... string as a list of QemuOpt. Each QemuOpt represents one key=value. They are in the same order. If key occurs multiple times in the string, it occurs just the same in the list.
*Except* key "id" is special: it's stored outside the list, and all but the first one are silently ignored.
Most users only ever get the last value of a key. Any non-last key=value are silently ignored.
We actually exploit this behavior to do defaults, by *prepending* them to the list. See the use of
qemu_opts_set_defaults()inmain().A few users get all values of keys (other than key "id"):
* -device, in
qdev_device_add()with callbackset_property().We first get "driver" and "bus" normally (silently ignoring non-last values, as usual). All other keys are device properties. To set them, we get all (key, value), ignore keys "driver" and "bus", and set the rest. If a key occurs multiple times, it gets set multiple times. This effectively ignores all but the last one, silently.
* -semihosting-config, in
main()with callbackadd_semihosting_arg().We first get a bunch of keys normally. Key "arg" is special: it may be repeated to build a list. To implement that, we get all (key, value), ignore keys other than "arg", and accumulate the values.
* -machine & friends, in
main()with callbackmachine_set_property()Similar to -device, only for machines, with "type" instead of "driver" and "bus".
* -spice, in
qemu_spice_init()with callbackadd_channel()Keys "tls-channel" and "plaintext-channel" may be used repeated to specify multiple channels. To implement that, we get all (key, value), ignore keys other than "tls-channel" and "plaintext-channel", and set up a channel for each of the others.
* -writeconfig, in
config_write_opts()with callbackconfig_write_opt()We write out all keys in order.
* The options visitor, in
opts_start_struct()We convert the list of (key, value) to a hash table of (key, list of values). Most of the time, the list of values has exactly one element.
When the visitor's user asks for a scalar, we return the last element of the list of values, in
lookup_scalar().When the user asks for list elements, we return the elements of the list of values in order, in
opts_next_list(), or if there are none, the empty list inopts_start_list().
Quoted 6 lines
Unlike the options visitor, this patch (judging from your description) makes a list only when keys are repeated. The QObject visitor will have to cope with finding both scalars and lists. When it finds a scalar, but needs a list, it'll have to wrap it in a list (PATCH 09, I think). When it finds a list, but needs a scalar, it'll have to fish it out of the list (where is that?).
If my code finds a list but wants a scalar, it is reporting an error.
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options
In reply to Markus ArmbrusterQuoted 43 lines · Markus Armbruster
Quoted 23 lines · Daniel P. Berrange
If given an option string such as
size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind
the
qemu_opts_to_qdict()method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:size=QString("1024") nodes=QString("1-2") policy=QString("bind")With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain
size=QString("1024") nodes=QList([QString("10"), QString("4-5"), QString("1-2")]) policy=QString("bind")Quoted 7 lines
Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.
Actually three cases, not two:
0. qdict does not contain the key means empty list.
1. qdict contains the key with a QString value means list of one element.
2. qdict contains the key with a QList value means list of more than one element.
Quoted 8 lines
I consider this weird. However, it's usefully weird with at least your QObject input visitor.
Quoted 3 lines
In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.
Got users for this policy in the pipeline?
I in fact used it in the QObjectInputVisitor, when the autocreate_list is not set.
I guess strictly speaking this is not back-compatible if someone is passing repeated keys, but I judged that rather than silently ignoring this incorrect usage, it was acceptable to report an error.
Quoted 18 lines
Quoted 18 lines
QTAILQ_FOREACH(opt, &opts->head, next) { val = QOBJECT(qstring_from_str(opt->str));util/qemu-option.c +15
-1-qdict_put_obj (qdict, opt->name, val); + +if (qdict_haskey (ret, opt->name)) { +switch (repeatPolicy) { +case QEMU_OPTS_REPEAT_POLICY_ERROR : +error_setg (errp,"Option '%s' appears more than once" , + opt->name); +qobject_decref (val); +if (!qdict) { +QDECREF (ret); + } +return NULL ; + +case QEMU_OPTS_REPEAT_POLICY_ALL : + prevval =qdict_get (ret, opt->name); +if (qobject_type (prevval) ==QTYPE_QLIST ) {
Quoted 8 lines
util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1110
Quoted 6 lines
util/qemu-option.c +6
-0+/* Already identified this key as a list */ + list =qobject_to_qlist (prevval); + }else { +/* Replace current scalar with a list */ + list =qlist_new (); +qobject_incref (prevval);Where is this reference decremented?
'prevval' is a borrowed reference from 'ret', against the key opt->name.
qdict_put_obj decrements the reference we borrowed from ret against the key opt->name.
qlist_append_obj() takes ownership of the reference it is passed, so we must qobject_incref() to avoid qdict_put_obj free'ing prevval.
When we call qdict_put_obj() we're replacing the value currently associted
Quoted 52 lines
Quoted 5 lines
util/qemu-option.c +4
-0+qlist_append_obj (list, prevval); +qdict_put_obj (ret, opt->name,QOBJECT (list)); + } +qlist_append_obj (list, val);util/qemu-option.c void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) · line 1124
Quoted 10 lines
util/qemu-option.c +10
-0+break ; + +case QEMU_OPTS_REPEAT_POLICY_LAST : +/* Just discard previously set value */ +qdict_put_obj (ret, opt->name, val); +break ; + } + }else { +qdict_put_obj (ret, opt->name, val); + }A possible alternative:
val = QOBJECT(qstring_from_str(opt->str)); if (qdict_haskey(ret, opt->name)) { switch (repeatPolicy) { case QEMU_OPTS_REPEAT_POLICY_ERROR: error_setg(errp, "Option '%s' appears more than once", opt->name); qobject_decref(val); if (!qdict) { QDECREF(ret); } return NULL; case QEMU_OPTS_REPEAT_POLICY_ALL: prevval = qdict_get(ret, opt->name); if (qobject_type(prevval) == QTYPE_QLIST) { /* Already identified this key as a list */ list = qobject_to_qlist(prevval); } else { /* Replace current scalar with a list */ list = qlist_new(); qobject_incref(prevval); qlist_append_obj(list, prevval); } qlist_append_obj(list, val); val = QOBJECT(list); break; case QEMU_OPTS_REPEAT_POLICY_LAST: break; } } qdict_put_obj(ret, opt->name, val);
Quoted 2 lines
This shows the common part of the behavior more clearly. Matter of taste, you get to use your artistic license.
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values
In reply to Markus ArmbrusterQuoted 21 lines · Markus Armbruster
Cc Kevin for discussion of QemuOpts dotted key conventionQuoted 17 lines · Daniel P. Berrange
Currently
qdict_crumplerequires a totally flat QDict as its input. i.e. all values in the QDict must be scalar types.In order to have backwards compatibility with the OptsVisitor,
qemu_opt_to_qdict()has a new mode where it may return a QList for values in the QDict, if there was a repeated key. We thus need to allow compound types to appear as values in the input dict given toqdict_crumple().To avoid confusion, we sanity check that the user has not mixed the old and new syntax at the same time. e.g. these are allowed
foo=hello,foo=world,foo=wibble foo.0=hello,foo.1=world,foo.2=wibblebut this is forbidden
Quoted 8 lines
Quoted 1 line
foo=hello,foo=world,foo.2=wibble
I understand the need for foo.bar=val. It makes it possible to specify nested dictionaries with QemuOpts.
The case for foo.0=val is less clear. QemuOpts already supports lists, by repeating keys. Why do we need a second, wordier way to specify them?
Two reasons I did this. First blockdev already uses this foo.0=val syntax, and I wanted to be compatible with blockdev, so it could be converted to use this new code.
Second, using foo.0 syntax means that you can unambigously determine whether a key is going to be a scalar or a list. This lets the qdict_crumple() method convert the QemuOpts to a QDict without needing to know anything about the QAPI schema.
Of course I later had to add hacks to the visitor to cope with the bare repeated key syntax, so I lost some of that benefit.
Personally I still prefer the unambiguous syntax as it lets us give clear error messages when users do unexpected things, instead of say, silently ignoring all but the last key.
Quoted 35 lines
Note that this second way creates entirely new failure modes and restrictions. Let me show using an example derived from one in qdict_crumple()'s contract:
foo.0.bar=bla,foo.eek.bar=blubb
Without the dotted key convention, this is perfectly fine: key "foo.0.bar" has the single value "bla", and key "foo.eek.bar" has the single value "blubb". Equivalent JSON would be
{ "foo.0.bar": "bla", "foo.eek.bar": "blubb" }
With just the struct convention, it's still fine: it obviously means the same as JSON
{ "foo": { "0": { "bar": "bla" }, "eek": { "bar": "blubb" } } }
Adding the list convention makes it invalid. It also outlaws a bunch of keys that would be just fine in JSON, namely any that get recognized as list index. Raise your hand if you're willing to bet real money on your predictions of what will be recognized as list index, without looking at the code. I'm not.
I'm afraid I have growing doubts regarding the QemuOpts dotted key convention in general.
The convention makes '.' a special character in keys, but only sometimes. If the key gets consumed by something that uses dotted key convention, '.' is special, and to get a non-special '.', you need to escape it by doubling. Else, it's not.
Since the same key can be used differently by different code, the same '.' could in theory be both special and non-special. In practice, this would be madness.
Quoted 5 lines
Adopting the dotted key convention for an existing QemuOpts option, say -object [PATCH 15], *breaks* existing command line usage of keys containing '.', because you now have to escape the '.'. Dan, I'm afraid you need to show that no such keys exist, or if they exist they don't matter.
I checked the things that I converted (eg -net, -object, -numa, etc), but I didn't check -device since that's processed using different code.
Quoted 8 lines
I know we have keys containing '.' elsewhere, e.g. device "macio-ide" property "ide.0". Our chronic inability to consistently restrict names in ABI to something sane is beyond foolish.
It's probably too late to back out the dotted key convention completely. Kevin?
Quoted 7 lines
Can we still back out the list part of the convention, and use repeated keys instead?
If we're stuck with some form of the dotted key convention, can we at least make it a more integral part of QemuOpts rather than something bolted on as an afterthought? Here's my thinking on how that might be done:
The only issue with dropping the dotted list convention is compat with the block layer code - we couldn't easily use this new visitor logic to turn -drive into a QAPI BlockOptions object. Kevin's new -blockdev arg would potentially be ok with it since its a new arg, but IIUC, we would have to do some cleanup inside various block driver impls, since block layer doesn't use the QAPI objects internally - they all get converted back into QemuOpts :-(
Quoted 13 lines
* Have a QemuOptsList flag @flat.
* If @flat, QemuOpts behaves as it always has: the special characters are ',' and '=', and parsing a key=value,... string produces a list where each element represents one key=value from the string, in the same order.
* If not @flat, '.' becomes an additional special character, and parsing a key=value,... string produces a dictionary, similar to the one you get now by converting with
qemu_opts_to_qdict()and filtering throughqdict_crumple().
Quoted 6 lines
The difference to now is that you either always crumple, or not at all, and the meaning of '.' is unambiguous.
I wish we had refrained from saddling QemuOpts with even more magic. Compared to this swamp, use of JSON on the command line looks rather appealing to me.
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict
In reply to Daniel P. BerrangeQuoted 93 lines · Daniel P. Berrange
Quoted 92 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{ 'foo.0.bar': 'one', 'foo.0.wizz': '1', 'foo.1.bar': 'two', 'foo.1.wizz': '2'}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{ 'foo': [ { 'bar': 'one', 'wizz': '1' }, { 'bar': 'two', 'wizz': '2' } ],}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{ 'foo..bar': 'wizz', 'bar.foo..bar': 'eek', 'bar.hello': 'world' }Will end up as
{ 'foo.bar': 'wizz', 'bar': { 'foo.bar': 'eek', 'hello': 'world' } }The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.
Reviewed-by Eric Blake <[email protected]>Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 261 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 551 insertions(+)include/qapi/qmp/qdict.h +1
-0@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);qobject/qdict.c +0
-0[...]
Quoted 4 lines
+/**
+ *qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionaries
Quoted 6 lines
Quoted 1 line
Is recursive=false used outside tests in this series?
No, its not used.
It was suggested in a way earlier version by Max, but not sure if his code uses it or not.
In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.
Max, do you actually need this one? If yes, please explain your use case.
Re: [Qemu-devel] [PATCH v14 03/21] qapi: add trace events for visitor
In reply to Daniel P. BerrangeQuoted 39 lines · Daniel P. Berrange
Quoted 31 lines · Markus Armbruster
Quoted 29 lines · Daniel P. Berrange
Allow tracing of the operation of visitors
Signed-off-by Daniel P. Berrange <[email protected]>--- Makefile.objs | 1 + qapi/qapi-visit-core.c | 27 +++++++++++++++++++++++++++ qapi/trace-events | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 61 insertions(+) create mode 100644 qapi/trace-eventsMakefile.objs +1
-0@@ -160,3 +160,4 @@ trace-events-y += target-s390x/trace-events trace-events-y += target-ppc/trace-events trace-events-y += qom/trace-events trace-events-y += linux-user/trace-events +trace-events-y += qapi/trace-eventsqapi/qapi-visit-core.c +1
-0@@ -19,10 +19,12 @@#include "qapi/qmp/qerror.h" #include "qapi/visitor.h" #include "qapi/visitor-impl.h" +#include "trace.h" Quoted 6 lines
qapi/qapi-visit-core.c line 27
Quoted 4 lines
qapi/qapi-visit-core.c +1
-0void visit_complete (Visitor *v,void *opaque) {assert (v->type !=VISITOR_OUTPUT || v->complete); +trace_visit_complete (v, opaque);Trace after or before checking the precondition? Preferences, anyone?
Quoted 4 lines
I'm ambivalent, as the assert will crash you either way so whether you get a trace event just before the crash seems mostly irrelevant to me - the abort stack trace is what you'll use to diagnose the crash.
I have a slight preference for trace first, because trace is one call, while precondition checking can be several. However, grep shows neither order is prevalent. Not worth a respin.
Quoted 107 lines
Quoted 107 lines
Quoted 90 lines
qapi/qapi-visit-core.c +11
-0if (v->complete) { v->complete (v, opaque); }@@ -30,6 +32,7 @@ void visit_complete(Visitor *v, void *opaque)void visit_free (Visitor *v) { +trace_visit_free (v);if (v) { v->free (v); }@@ -40,6 +43,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj, {Error *err =NULL ; +trace_visit_start_struct (v, name, obj, size);if (obj) {assert (size);assert (!(v->type &VISITOR_OUTPUT ) || *obj);@@ -53,6 +57,7 @@ void visit_start_struct(Visitor *v, const char *name, void **obj,void visit_check_struct (Visitor *v,Error **errp) { +trace_visit_check_struct (v);if (v->check_struct) { v->check_struct (v, errp); }@@ -60,6 +65,7 @@ void visit_check_struct(Visitor *v, Error **errp)void visit_end_struct (Visitor *v,void **obj) { +trace_visit_end_struct (v, obj); v->end_struct (v, obj); }@@ -69,6 +75,7 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,Error *err =NULL ;assert (!list || size >=sizeof (GenericList)); +trace_visit_start_list (v, name, list, size); v->start_list (v, name, list, size, &err);if (list && (v->type &VISITOR_INPUT )) {assert (!(err && *list));@@ -79,11 +86,13 @@ void visit_start_list(Visitor *v, const char *name, GenericList **list,GenericList *visit_next_list (Visitor *v,GenericList *tail,size_t size) {assert (tail && size >=sizeof (GenericList)); +trace_visit_next_list (v, tail, size);return v->next_list (v, tail, size); }void visit_end_list (Visitor *v,void **obj) { +trace_visit_end_list (v, obj); v->end_list (v, obj); }@@ -95,6 +104,7 @@ void visit_start_alternate(Visitor *v, const char *name,assert (obj && size >=sizeof (GenericAlternate));assert (!(v->type &VISITOR_OUTPUT ) || *obj); +trace_visit_start_alternate (v, name, obj, size, promote_int);if (v->start_alternate) { v->start_alternate (v, name, obj, size, promote_int, &err); }@@ -106,6 +116,7 @@ void visit_start_alternate(Visitor *v, const char *name,void visit_end_alternate (Visitor *v,void **obj) { +trace_visit_end_alternate (v, obj);if (v->end_alternate) { v->end_alternate (v, obj); }@@ -113,6 +124,7 @@ void visit_end_alternate(Visitor *v, void **obj)bool visit_optional (Visitor *v,const char *name,bool *present) { +trace_visit_optional (v, name, present);if (v->optional) { v->optional (v, name, present); }@@ -127,6 +139,7 @@ bool visit_is_input(Visitor *v)void visit_type_int (Visitor *v,const char *name,int64_t *obj,Error **errp) {assert (obj); +trace_visit_type_int (v, name, obj); v->type_int64 (v, name, obj, errp); }qapi/qapi-visit-core.c void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj, · line 170
Quoted 7 lines
qapi/qapi-visit-core.c +1
-0@@ -151,6 +164,7 @@ void visit_type_uint8(Visitor *v, const char *name, uint8_t *obj, Error **errp) {uint64_t value = *obj; +trace_visit_type_uint8 (v, name, obj);visit_type_uintN (v, &value, name,UINT8_MAX ,"uint8_t" , errp); *obj = value; }Putting the trace in the middle of the "value = *obj;
visit_type_uintN(v, &value, ...); *obj = value" pattern makes it less visible.Preserving the pattern requires replacing the initializer by an assignment:
uint64_t value;
Quoted 8 lines
Quoted 6 lines
trace_visit_type_uint8(v, name, obj); value = *obj; visit_type_uintN(v, &value, name, UINT8_MAX, "uint8_t", errp); *obj = value;Looks slightly more legible to me. What do you think?
I'm fine with whatever you prefer.
I can touch up this one on commit.
Re: [Qemu-devel] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor
In reply to Daniel P. BerrangeQuoted 123 lines · Daniel P. Berrange
Quoted 122 lines · Markus Armbruster
Quoted 107 lines · Daniel P. Berrange
The traditional CLI arg syntax allows two ways to specify integer lists, either one value per key, or a range of values per key. eg the following are identical:
-arg foo=5,foo=6,foo=7
-arg foo=5-7This extends the QObjectInputVisitor so that it is able to parse ranges and turn them into distinct list entries.
This means that
-arg foo=5-7
is treated as equivalent to
-arg foo.0=5,foo.1=6,foo.2=7
Edge case tests are copied from test-opts-visitor to ensure identical behaviour when parsing.
Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qobject-input-visitor.h | 23 ++++- qapi/qobject-input-visitor.c | 158 ++++++++++++++++++++++++++-- tests/test-qobject-input-visitor.c | 195 +++++++++++++++++++++++++++++++++-- 3 files changed, 360 insertions(+), 16 deletions(-)static void qobject_input_type_uint64(Visitor *v, const char *name,
qapi/qobject-input-visitor.c +58
-3@@ -366,21 +438,85 @@ static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, uint64_t *obj, Error **errp) { QObjectInputVisitor *qiv = to_qiv(v); -QString *qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, - true)); +QString *qstr;unsigned long long ret; +char *end =NULL ; +StackObject *tos; +bool inlist = false; + +/* Preferentially generate values from a range, before +* trying to consume another QList element */ + tos =QSLIST_FIRST (&qiv->stack); +if (tos) { +if (tos->range_val < tos->range_limit) { + *obj = tos->range_val +1 ; + tos->range_val++; +return ; + }else { + inlist = tos->entry !=NULL ; + } + } + qstr =qobject_to_qstring (qobject_input_get_object (qiv, name, + true));if (!qstr || !qstr->string) {error_setg (errp,QERR_INVALID_PARAMETER_TYPE , name ? name :"null" ,"string" );return ; } -if (parse_uint_full (qstr->string, &ret,0 ) <0 ) { +if (parse_uint (qstr->string, &ret, &end,0 ) <0 ) {error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" );return ; } *obj = ret; + +/* +* If we have string that represents an integer range (5-24), +* parse the end of the range and set things up so we'll process +* the rest of the range before consuming another element +* from the QList. +*/ +if (end && *end) { +if (!qiv->permit_int_ranges) { +error_setg (errp, +"Integer ranges are not permitted here" ); +return ; + } +if (!inlist) { +error_setg (errp, +"Integer ranges are only permitted when " +"visiting list parameters" ); +return ; + } +if (*end !='-' ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name, +"a number range" ); +return ; + } + end++; +if (parse_uint_full (end, &ret,0 ) <0 ) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE , name,"a number" ); +return ; + } +if (*obj > ret) { +error_setg (errp, +"Parameter '%s' range start %" PRIu64 +" must be less than (or equal to) %llu" , + name, *obj, ret); +return ; + } +if ((ret - *obj) > (QIV_RANGE_MAX -1 )) { + error_setg(errp,qapi/qobject-input-visitor.c static void qobject_input_type_uint64_autocast(Visitor *v, const char *name, · line 520
Quoted 10 lines
qapi/qobject-input-visitor.c +9
-0+"Parameter '%s' range must be less than %d" , + name, QIV_RANGE_MAX); +return ; + } +if (*obj != ret) { + tos->range_val = *obj; + tos->range_limit = ret; + } + } }Duplicates the signed code, which is sad, but I don't have better ideas.
Quoted 5 lines
Quoted 2 lines
Except this one: are we actually using both the signed and the unsigned case now? If not, can we get rid of the one we don't use?
Out of the args that I converted in this series, I only see uint16List used. eg for -numa and -object hostmem
Let's drop the signed copy then.
Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options
In reply to Daniel P. BerrangeQuoted 42 lines · Daniel P. Berrange
Quoted 41 lines · Markus Armbruster
Quoted 31 lines · Daniel P. Berrange
If given an option string such as
size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind
the
qemu_opts_to_qdict()method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:size=QString("1024") nodes=QString("1-2") policy=QString("bind")With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain
size=QString("1024") nodes=QList([QString("10"), QString("4-5"), QString("1-2")]) policy=QString("bind")Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.
Quoted 3 lines
In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.
To serve as a replacement for the options visitor, this needs to be able to behave exactly the same together with a suitably hacked up QObject input visitor. Before I dive into the actual patch, let me summarize QemuOpts and options visitor behavior.
[...]
Quoted 9 lines
Quoted 6 lines
Unlike the options visitor, this patch (judging from your description) makes a list only when keys are repeated. The QObject visitor will have to cope with finding both scalars and lists. When it finds a scalar, but needs a list, it'll have to wrap it in a list (PATCH 09, I think). When it finds a list, but needs a scalar, it'll have to fish it out of the list (where is that?).
If my code finds a list but wants a scalar, it is reporting an error.
I'm afraid that breaks things like
--numa node,cpus=0,cpus=2,mem=512,nodeid=0,nodeid=0
The options visitor inteprets cpus=0,cpus=2 as a list [0,2] (because NumaNodeOptions member @cpus has a list type) and nodeid=0,nodid=0 as integer 0 (because member @nodeid has a scalar type).
Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options
Quoted 55 lines · Markus Armbruster
Quoted 42 lines · Daniel P. Berrange
Quoted 41 lines · Markus Armbruster
Quoted 31 lines · Daniel P. Berrange
If given an option string such as
size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind
the
qemu_opts_to_qdict()method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:size=QString("1024") nodes=QString("1-2") policy=QString("bind")With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain
size=QString("1024") nodes=QList([QString("10"), QString("4-5"), QString("1-2")]) policy=QString("bind")Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.
Quoted 3 lines
In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.
To serve as a replacement for the options visitor, this needs to be able to behave exactly the same together with a suitably hacked up QObject input visitor. Before I dive into the actual patch, let me summarize QemuOpts and options visitor behavior.
[...]
Quoted 9 lines
Quoted 6 lines
Unlike the options visitor, this patch (judging from your description) makes a list only when keys are repeated. The QObject visitor will have to cope with finding both scalars and lists. When it finds a scalar, but needs a list, it'll have to wrap it in a list (PATCH 09, I think). When it finds a list, but needs a scalar, it'll have to fish it out of the list (where is that?).
If my code finds a list but wants a scalar, it is reporting an error.
Quoted 7 lines
I'm afraid that breaks things like
--numa node,cpus=0,cpus=2,mem=512,nodeid=0,nodeid=0The options visitor inteprets cpus=0,cpus=2 as a list [0,2] (because NumaNodeOptions member @cpus has a list type) and nodeid=0,nodid=0 as integer 0 (because member @nodeid has a scalar type).
Right, but that command line syntax is arguably broken already - the app/user invoking QEMU is buggy, so I felt it was justifiable to change from silently ignoring the app error to telling them of their mistake.
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options
In reply to Daniel P. BerrangeQuoted 53 lines · Daniel P. Berrange
Quoted 43 lines · Markus Armbruster
Quoted 23 lines · Daniel P. Berrange
If given an option string such as
size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind
the
qemu_opts_to_qdict()method will currently overwrite the values for repeated option keys, so only the last value is in the returned dict:size=QString("1024") nodes=QString("1-2") policy=QString("bind")With this change the caller can optionally ask for all the repeated values to be stored in a QList. In the above example that would result in 'nodes' being a QList, so the returned dict would contain
size=QString("1024") nodes=QList([QString("10"), QString("4-5"), QString("1-2")]) policy=QString("bind")Quoted 7 lines
Note that the conversion has no way of knowing whether any given key is expected to be a list upfront - it can only figure that out when seeing the first duplicated key. Thus the caller has to be prepared to deal with the fact that if a key 'foo' is a list, then the returned qdict may contain either a QString or a QList for the key 'foo'.
Actually three cases, not two:
0. qdict does not contain the key means empty list.
1. qdict contains the key with a QString value means list of one element.
2. qdict contains the key with a QList value means list of more than one element.
Quoted 8 lines
I consider this weird. However, it's usefully weird with at least your QObject input visitor.
Quoted 3 lines
In a third mode, it is possible to ask for repeated options to be reported as an error, rather than silently dropping all but the last one.
Got users for this policy in the pipeline?
Quoted 7 lines
I in fact used it in the QObjectInputVisitor, when the
autocreate_listis not set.I guess strictly speaking this is not back-compatible if someone is passing repeated keys, but I judged that rather than silently ignoring this incorrect usage, it was acceptable to report an error.
While usage like
-machine usb=on,usb=off
could perhaps be declared erroneous post hoc, the "last one wins" feature also has genuinely useful applications. For instance, tack a correction to a long command line:
-machine usb=on [... lots of other stuff ...] -machine usb=off
Or modify canned configuration:
-readconfig vm1.config -machine usb=off
# vm1.config sets usb=on, -machine overrides
-readconfig vm1.config -set drive.disk0.file=tmp.qcow2 # vm1.config defines drive "disk0", -set overrides its image file
I don't think we can break them all.
[...]
Re: [Qemu-devel] [PATCH v14 09/21] qapi: permit auto-creating single element lists
In reply to Daniel P. BerrangeQuoted 42 lines · Daniel P. Berrange
Quoted 17 lines · Markus Armbruster
Quoted 2 lines · Daniel P. Berrange
When converting QemuOpts to a QObject, there is no information about compound types available,
Yes, that's a drawback of splitting the conversion into a QemuOpts -> QObject part that is oblivious of types, and a QObject -> QAPI object part that knows the types.
Quoted 8 lines
so when visiting a list, the
corresponding QObject is not guaranteed to be a QList. We therefore need to be able to auto-create a single element QList from whatever type we find.This mode should only be enabled if you have compatibility requirements for
Quoted 8 lines
Quoted 5 lines
-arg foo=hello,foo=world
to be treated as equivalent to the preferred syntax:
-arg foo.0=hello,foo.1=worldNot sure this is "preferred". "More powerfully warty" is probably closer to the truth ;)
Well, I call it "preferred" in the sense that that option syntax directly maps to the QAPI syntax in an unambigous manner. ie given the arg value alone "foo.0=hello,foo.1=world" you can clearly determine that "foo" is a list. With the compat syntax you cannot distinguish list from scalar, without knowing the QAPI schema.
Quoted 1 line
How is "-arg foo=hello,foo=world" treated if this mode isn't enabled?
The default behaviour would be that only the last key is present in the dict, eg foo=world, and then if you tried to visit a list, the visitor would complain that its got a QString instead of QList for the key 'foo'.
This is related to patch 14
Quoted 7 lines
Quoted 2 lines
What would be the drawbacks of doing this always instead of only when we "have compatibility requirements"?
Essentially we'd be permanently allowing 2 distinct syntaxes for dealing with lists, for all options. I felt it desirable that we have only a single syntax and only allow this alt syntax in the backcompat cases.
Fair point.
The bolted-on extensions (options visitor's integer list syntax, block layer's dotted key convention) have only ever worked in the places that choose to use them. Even the integrated support for repeated keys is only used for lists in the places that choose to use it that way.
Quoted 57 lines
Quoted 29 lines
Quoted 29 lines
qapi/qobject-input-visitor.c +6
-0@@ -48,6 +48,10 @@ struct QObjectInputVisitor/* True to reject parse in visit_end_struct() if unvisited keys remain. */ bool strict; + +/* Whether we can auto-create single element lists when +* encountering a non-QList type */ +bool autocreate_list; };static QObjectInputVisitor *to_qiv (Visitor *v)@@ -108,6 +112,7 @@ static const QListEntry *qobject_input_push(QObjectInputVisitor *qiv,assert (obj); tos->obj = obj; tos->qapi = qapi; +qobject_incref (obj);if (qiv->strict &&qobject_type (obj) ==QTYPE_QDICT ) { h =g_hash_table_new (g_str_hash, g_str_equal);@@ -147,6 +152,7 @@ static void qobject_input_stack_object_free(StackObject *tos)if (tos->h) {g_hash_table_unref (tos->h); } +qobject_decref (tos->obj);Quoted 4 lines
qapi/qobject-input-visitor.c static void qobject_input_stack_object_free(StackObject *tos) · line 158
Quoted 2 lines
qapi/qobject-input-visitor.c +0
-0g_free (tos); }Can you explain the reference counting change?
Previously the stack stored a borrowed reference, since it didn't ever need responsibility for free'ing the object when popping the stack. This is no longer the case if you look a few lines later....
Quoted 18 lines
qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 206
Quoted 7 lines
qapi/qobject-input-visitor.c +1
-1@@ -197,7 +203,7 @@ static void qobject_input_start_list(Visitor *v, const char *name,QObject *qobj =qobject_input_get_object (qiv, name, true);const QListEntry *entry; -if (!qobj ||qobject_type (qobj) !=QTYPE_QLIST ) { +if (!qobj || (!qiv->autocreate_list &&qobject_type (qobj) !=QTYPE_QLIST )) {Long line, but I believe it'll go away when you rebase for commit 1382d4a.
Quoted 7 lines
qapi/qobject-input-visitor.c +0
-0if (list) { *list =NULL ; }@@ -206,7 +212,16 @@ static void qobject_input_start_list(Visitor *v, const char *name,return ; }
Quoted 12 lines
Quoted 7 lines
qapi/qobject-input-visitor.c static void qobject_input_start_list(Visitor *v, const char *name, · line 220
Quoted 7 lines
qapi/qobject-input-visitor.c +6
-1- entry =qobject_input_push (qiv, qobj, list, errp); +if (qobject_type (qobj) !=QTYPE_QLIST ) { +QList *tmplist =qlist_new (); +qlist_append_obj (tmplist, qobj); +qobject_incref (qobj); + entry =qobject_input_push (qiv,QOBJECT (tmplist), list, errp); +QDECREF (tmplist);... here we are storing the 'qmplist' in the stack, and so when popping the stack, we must free that object. We thus need the stack to always hold its own reference, so when popping it can decref and (potentially) release the last reference.
Aha.
Manual reference counting is a PITA.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
qapi/qobject-input-visitor.c +4
-0+ }else { + entry =qobject_input_push (qiv, qobj, list, errp); + } +if (list) {if (entry) { *list =g_malloc0 (size);
Quoted 4 lines
Quoted 2 lines
Buries autolist behavior in the middle of things. What about doing it first, so it's more separate?
I'm not sure I understand what you mean here ?
I'm suggesting a structure like this:
1. Map special case to normal case
2. Deal with normal case
Keeping the normal case and the special case separate like that can make it easier to understand either.
Here's my attempt to do it:
Quoted 8 lines
Quoted 8 lines
QObjectInputVisitor *qiv = to_qiv(v); QObject *qobj = qobject_input_get_object_(qiv, name, true, errp); const QListEntry *entry; if (!qobj) { return; }
Quoted 6 lines
Quoted 6 lines
+ if (
qiv->autocreate_list&&qobject_type(qobj)!=QTYPE_QLIST) {+ QList *auto_list = qlist_new(); + qlist_append_obj(auto_list, qobj); + qobj = auto_list;+ }
+
The new code is the mapping. The normal case code below remains unchanged. It's just a sketch; I didn't spend a single brainwave on the reference counting.
Quoted 45 lines
Quoted 35 lines
if (qobject_type(qobj) != QTYPE_QLIST) {I ignored reference counting here, because I don't yet understand how and why you're changing it.
Quoted 30 lines
qapi/qobject-input-visitor.c +4
-1@@ -514,7 +529,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict)return &v->visitor; } -Visitor *qobject_input_visitor_new_autocast (QObject *obj) +Visitor *qobject_input_visitor_new_autocast (QObject *obj, +bool autocreate_list) {QObjectInputVisitor *v;@@ -539,6 +555,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj) v->visitor.optional = qobject_input_optional; v->visitor.free = qobject_input_free; v->strict = true; + v->autocreate_list = autocreate_list; v->root = obj;qobject_incref (obj);@@ -548,6 +565,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)Visitor *qobject_input_visitor_new_opts (const QemuOpts *opts, +bool autocreate_list,Error **errp) {QDict *pdict;@@ -564,7 +582,8 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, goto cleanup; }Quoted 7 lines
qapi/qobject-input-visitor.c Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, · line 589
Quoted 6 lines
qapi/qobject-input-visitor.c +2
-1- v =qobject_input_visitor_new_autocast (pobj); + v =qobject_input_visitor_new_autocast (pobj, + autocreate_list); cleanup:qobject_decref (pobj);QDECREF (pdict);[Skipping test updates for now...]
Regards,
Daniel
Re: [Qemu-devel] [PATCH v14 01/21] option: make parse_option_bool/number non-static
In reply to Daniel P. BerrangeQuoted 10 lines · Daniel P. Berrange
The
opts-visitor.copts_type_bool()method has code for parsing a string to set a bool value, as does theqemu-option.cparse_option_bool()method, except it handles fewer cases.To enable consistency across the codebase, extend
parse_option_bool()to handle "yes", "no", "y" and "n", and make it non-static. Convert the opts visitor to call this method directly.
Quoted 8 lines
Also make
parse_option_number()non-static to allow for similar reuse later.Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Markus Armbruster <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>
On first glance, this patch makes the boolean values recognized on the command line consistent regardless of which part of the code is used to recognize them, by extending the QemuOpts parser to accept the option visitor's additional convenience values.
Once we've done that in a release, we can't go back. No problem if it actually achieved consistency. But it doesn't: there are other parsers that recognize only "on" and "off". A quick grep finds select_display(), bdrv_open_inherit(), netfilter_set_status().
bdrv_open_inherit() is particularly instructive: -drive gets parsed by QemuOpts (evidence: your patch makes read-only=y work), but by the time the options arrive here, they're a QDict. Curiously, bdrv_open_inherit() expects the value of key "read-only" to be a *string*, and it checks for "on". Fortunately, something on the way to bdrv_open_inherit() replaces the "y" by "on", so this works. My point
is: this patch is trickier than it looks on first glance.
There's also HMP, which continues to accept only "on" and "off".
We might want to treat the option visitor's additional boolean values like its other syntax extensions: keep for compatibility, but confine to existing uses.
I think we should decide that when we merge the rest of your option visitor replacement work, hopefully in 2.9.
Re: [Qemu-devel] [PATCH v14 01/21] option: make parse_option_bool/number non-static
Quoted 10 lines · Markus Armbruster
On first glance, this patch makes the boolean values recognized on the command line consistent regardless of which part of the code is used to recognize them, by extending the QemuOpts parser to accept the option visitor's additional convenience values.
Once we've done that in a release, we can't go back. No problem if it actually achieved consistency. But it doesn't: there are other parsers that recognize only "on" and "off". A quick grep finds
select_display(),bdrv_open_inherit(),netfilter_set_status().
Would it be wrong to convert these to use the same function, to get us to the point where they are consistent instead of self-limited? Yes, it means that newer qemu will accept spellings that older doesn't, and that we can't later prune the set of spellings back down, but trying to remain backwards-compatible to every different set of inputs on a command-by-command basis is harder than just making all commands accept all spellings.
Quoted 10 lines
bdrv_open_inherit()is particularly instructive: -drive gets parsed by QemuOpts (evidence: your patch makes read-only=y work), but by the time the options arrive here, they're a QDict. Curiously,bdrv_open_inherit()expects the value of key "read-only" to be a *string*, and it checks for "on". Fortunately, something on the way tobdrv_open_inherit()replaces the "y" by "on", so this works. My point
is: this patch is trickier than it looks on first glance.There's also HMP, which continues to accept only "on" and "off".
Particularly for HMP, where we DON'T have backwards-compatibility constraints, I'd argue that ease-of-use is the driving factor and that HMP should allow ALL spellings of boolean arguments.
Quoted 4 lines
We might want to treat the option visitor's additional boolean values like its other syntax extensions: keep for compatibility, but confine to existing uses.
As in - new interfaces ONLY get to pass "true" or "false", and only existing interfaces get to also pass "y" or "on"?
Quoted 5 lines
I think we should decide that when we merge the rest of your option visitor replacement work, hopefully in 2.9.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v14 01/21] option: make parse_option_bool/number non-static
Quoted 13 lines · Eric Blake
Quoted 10 lines · Markus Armbruster
On first glance, this patch makes the boolean values recognized on the command line consistent regardless of which part of the code is used to recognize them, by extending the QemuOpts parser to accept the option visitor's additional convenience values.
Once we've done that in a release, we can't go back. No problem if it actually achieved consistency. But it doesn't: there are other parsers that recognize only "on" and "off". A quick grep finds
select_display(),bdrv_open_inherit(),netfilter_set_status().
Quoted 7 lines
Would it be wrong to convert these to use the same function, to get us to the point where they are consistent instead of self-limited? Yes, it means that newer qemu will accept spellings that older doesn't, and that we can't later prune the set of spellings back down, but trying to remain backwards-compatible to every different set of inputs on a command-by-command basis is harder than just making all commands accept all spellings.
I have two concerns:
1. Accepting alternate spellings in just a few places is slightly bad, accepting them is most, but not all places is worse. Converting all the parsers as you suggest would take care of this one.
Quoted 7 lines
Quoted 7 lines
bdrv_open_inherit()is particularly instructive: -drive gets parsed by QemuOpts (evidence: your patch makes read-only=y work), but by the time the options arrive here, they're a QDict. Curiously,bdrv_open_inherit()expects the value of key "read-only" to be a *string*, and it checks for "on". Fortunately, something on the way tobdrv_open_inherit()replaces the "y" by "on", so this works. My point
is: this patch is trickier than it looks on first glance.
2. I'm scared some user of QemuOpts declares QEMU_OPT_BOOL, but then checks the *string*. Yes, QemuOpts lets you do that. Making the parser accept more strings would break such code. Need a careful code review to put this concern to rest.
Quoted 5 lines
Quoted 1 line
There's also HMP, which continues to accept only "on" and "off".
Particularly for HMP, where we DON'T have backwards-compatibility constraints, I'd argue that ease-of-use is the driving factor and that HMP should allow ALL spellings of boolean arguments.
Personally, I find alternate spellings more confusing than helpful, but I agree that once we adopt them for the command line (consistently, not just in a few places), we should also adopt them for HMP.
Quoted 6 lines
Quoted 3 lines
We might want to treat the option visitor's additional boolean values like its other syntax extensions: keep for compatibility, but confine to existing uses.
As in - new interfaces ONLY get to pass "true" or "false", and only existing interfaces get to also pass "y" or "on"?
Very few places accept anything but "on" and "off", namely the ones that use the options or the string visitor. Dan's work kills of the former, and hopefully permits us killing off the latter. The alternate spellings will then be confined in one place, guarded by a compatibility flag you're not supposed to set in new code. Looks workable to me.
Quoted 2 lines
Quoted 2 lines
I think we should decide that when we merge the rest of your option visitor replacement work, hopefully in 2.9.
I'm not rejecting this patch, I'm merely punting the ABI-affecting parts of this series to 2.9.
Quoted 41 lines · Daniel P. Berrange
An update of a series previously posted
v1: https://lists.gnu.org/archive/html/qemu-devel/2016-02/msg04618.html
v2: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg01454.html
v3: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg02498.html
v4: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01661.html
v5: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg00485.html
v6: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg03876.html
v7: https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg00919.html
v8: https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg03115.html
v9: https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg02653.html
v10: https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg02694.html
v11: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg00652.html
v12: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg03559.html
v13: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg04212.html
v14: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg07037.htmlThis series provides the infrastructure to allow use of non-scalar properties with the -object CLI arg, and
object_addmonitor commands. eg a property which is a list of structs. The syntax used for this is intentionally compatible with the syntax used by the block layer. This will allow theqdict_crumplemethod to be used by the block layer to convert from QemuOpts into structured QAPI block layer structs at a future date. It is already used by one of Max's patch series, by Kevin's -blockdev series, and by my own ACL series.This patch series has grown a fair bit larger since previous postings (19^H^H21 patches instead of 6). This was to avoid trying to squish too much into one patch, to make it easier to review.
It does not neccessarily need to be merged in one go. There are three logical blocks of patches here.
Patches 1-8 are the core support for visiting QemuOpts using the QObjectInputVisitor class. Merging this would unblock at least Kevin's blockdev series, and probably Max's series too.
The patches 9-14 add enhancements to the code to support various special cases / edge cases needed to provide fully semantic replacement of the OptsVisitor
Quoted 8 lines
Patches 15-21 deal with converting code over to use the new visitor logic.
Once this is merged, there is also scope to further enhance the QObjectInputVisitor to deal with visiting single-properties, which would let us delete StringInputVisitor too. This would mean we have a single input visitor class, instead of the current three.
PATCH 03-06 applied to qapi-next. For PATCH 02, the question whether we need @recursive is still open. I hope Max can help me figure it out, so I can apply it next week. I think we'll rebase the rest on QemuOpts work I want for 2.9. Thanks!
Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict
In reply to Markus ArmbrusterQuoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{ 'foo.0.bar': 'one', 'foo.0.wizz': '1', 'foo.1.bar': 'two', 'foo.1.wizz': '2'}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{ 'foo': [ { 'bar': 'one', 'wizz': '1' }, { 'bar': 'two', 'wizz': '2' } ],}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{ 'foo..bar': 'wizz', 'bar.foo..bar': 'eek', 'bar.hello': 'world' }Will end up as
{ 'foo.bar': 'wizz', 'bar': { 'foo.bar': 'eek', 'hello': 'world' } }The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.
Reviewed-by Eric Blake <[email protected]>Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 261 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 551 insertions(+)include/qapi/qmp/qdict.h +1
-0@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);qobject/qdict.c +0
-0[...]
Quoted 6 lines
Quoted 4 lines
+/**
+ *qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionariesIs recursive=false used outside tests in this series?
No, its not used.
Quoted 8 lines
Quoted 2 lines
It was suggested in a way earlier version by Max, but not sure if his code uses it or not.
In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.
Max, do you actually need this one? If yes, please explain your use case.
As far as I can tell from a quick glance, I made the point for v1 that qdict_crumple() could be simplified by using qdict_array_split() and qdict_array_entries().
Dan then (correctly) said that using these functions would worsen runtime performance of qdict_crumple() and that instead we can replace qdict_array_split() by qdict_crumple(). However, for that to work, we need to be able to make qdict_crumple() non-recursive (because qdict_array_split() is non-recursive).
Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.
However, clearly there are no patches out yet that replace qdict_array_split() by qdict_crumple(recursive=false), and I certainly won't write any until qdict_crumple() is merged.
Max
Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict
Quoted 119 lines · Max Reitz
Quoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{ 'foo.0.bar': 'one', 'foo.0.wizz': '1', 'foo.1.bar': 'two', 'foo.1.wizz': '2'}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{ 'foo': [ { 'bar': 'one', 'wizz': '1' }, { 'bar': 'two', 'wizz': '2' } ],}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{ 'foo..bar': 'wizz', 'bar.foo..bar': 'eek', 'bar.hello': 'world' }Will end up as
{ 'foo.bar': 'wizz', 'bar': { 'foo.bar': 'eek', 'hello': 'world' } }The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.
Reviewed-by Eric Blake <[email protected]>Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 261 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 551 insertions(+)include/qapi/qmp/qdict.h +1
-0@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);qobject/qdict.c +0
-0[...]
Quoted 6 lines
Quoted 4 lines
+/**
+ *qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionariesIs recursive=false used outside tests in this series?
No, its not used.
Quoted 8 lines
Quoted 2 lines
It was suggested in a way earlier version by Max, but not sure if his code uses it or not.
In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.
Max, do you actually need this one? If yes, please explain your use case.
As far as I can tell from a quick glance, I made the point for v1 that
qdict_crumple()could be simplified by usingqdict_array_split()andqdict_array_entries().Dan then (correctly) said that using these functions would worsen runtime performance of
qdict_crumple()and that instead we can replaceqdict_array_split()byqdict_crumple(). However, for that to work, we need to be able to makeqdict_crumple()non-recursive (becauseqdict_array_split()is non-recursive).
Quoted 6 lines
Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.
In the long run, the block layer should use proper C types instead of mucking around with QDict. That's what QAPI is for.
Quoted 3 lines
However, clearly there are no patches out yet that replace
qdict_array_split()byqdict_crumple(recursive=false), and I certainly won't write any untilqdict_crumple()is merged.
All right, I'll go hunting for prior versions of qdict_crumple() to see whether I find a suitable one without @recursive.
Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict
Quoted 131 lines · Markus Armbruster
Quoted 119 lines · Max Reitz
Quoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{ 'foo.0.bar': 'one', 'foo.0.wizz': '1', 'foo.1.bar': 'two', 'foo.1.wizz': '2'}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{ 'foo': [ { 'bar': 'one', 'wizz': '1' }, { 'bar': 'two', 'wizz': '2' } ],}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{ 'foo..bar': 'wizz', 'bar.foo..bar': 'eek', 'bar.hello': 'world' }Will end up as
{ 'foo.bar': 'wizz', 'bar': { 'foo.bar': 'eek', 'hello': 'world' } }The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.
Reviewed-by Eric Blake <[email protected]>Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 261 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 551 insertions(+)include/qapi/qmp/qdict.h +1
-0@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);qobject/qdict.c +0
-0[...]
Quoted 6 lines
Quoted 4 lines
+/**
+ *qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionariesIs recursive=false used outside tests in this series?
No, its not used.
Quoted 8 lines
Quoted 2 lines
It was suggested in a way earlier version by Max, but not sure if his code uses it or not.
In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.
Max, do you actually need this one? If yes, please explain your use case.
As far as I can tell from a quick glance, I made the point for v1 that
qdict_crumple()could be simplified by usingqdict_array_split()andqdict_array_entries().Dan then (correctly) said that using these functions would worsen runtime performance of
qdict_crumple()and that instead we can replaceqdict_array_split()byqdict_crumple(). However, for that to work, we need to be able to makeqdict_crumple()non-recursive (becauseqdict_array_split()is non-recursive).Quoted 6 lines
Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.
In the long run, the block layer should use proper C types instead of mucking around with QDict. That's what QAPI is for.
Quoted 6 lines
Quoted 3 lines
However, clearly there are no patches out yet that replace
qdict_array_split()byqdict_crumple(recursive=false), and I certainly won't write any untilqdict_crumple()is merged.All right, I'll go hunting for prior versions of
qdict_crumple()to see whether I find a suitable one without @recursive.
Please don't do that - there have been a tonne of other fixes and improvements since the version that introduced @recursive. We need to just chop out that parameter from the v15 version instead.
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|
Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict
Quoted 136 lines · Daniel P. Berrange
Quoted 135 lines · Markus Armbruster
Quoted 119 lines · Max Reitz
Quoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{ 'foo.0.bar': 'one', 'foo.0.wizz': '1', 'foo.1.bar': 'two', 'foo.1.wizz': '2'}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{ 'foo': [ { 'bar': 'one', 'wizz': '1' }, { 'bar': 'two', 'wizz': '2' } ],}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{ 'foo..bar': 'wizz', 'bar.foo..bar': 'eek', 'bar.hello': 'world' }Will end up as
{ 'foo.bar': 'wizz', 'bar': { 'foo.bar': 'eek', 'hello': 'world' } }The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.
Reviewed-by Eric Blake <[email protected]>Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 261 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 551 insertions(+)include/qapi/qmp/qdict.h +1
-0@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);qobject/qdict.c +0
-0[...]
Quoted 6 lines
Quoted 4 lines
+/**
+ *qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionariesIs recursive=false used outside tests in this series?
No, its not used.
Quoted 8 lines
Quoted 2 lines
It was suggested in a way earlier version by Max, but not sure if his code uses it or not.
In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.
Max, do you actually need this one? If yes, please explain your use case.
As far as I can tell from a quick glance, I made the point for v1 that
qdict_crumple()could be simplified by usingqdict_array_split()andqdict_array_entries().Dan then (correctly) said that using these functions would worsen runtime performance of
qdict_crumple()and that instead we can replaceqdict_array_split()byqdict_crumple(). However, for that to work, we need to be able to makeqdict_crumple()non-recursive (becauseqdict_array_split()is non-recursive).Quoted 6 lines
Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.
In the long run, the block layer should use proper C types instead of mucking around with QDict. That's what QAPI is for.
Quoted 3 lines
However, clearly there are no patches out yet that replace
qdict_array_split()byqdict_crumple(recursive=false), and I certainly won't write any untilqdict_crumple()is merged.
Quoted 6 lines
Quoted 2 lines
All right, I'll go hunting for prior versions of
qdict_crumple()to see whether I find a suitable one without @recursive.Please don't do that - there have been a tonne of other fixes and improvements since the version that introduced @recursive. We need to just chop out that parameter from the v15 version instead.
I realized that once I looked at the interdiffs.
Chopping off @recursive is easy enough. So is dropping all tests that pass false, but I'm not sure what that does to test coverage. Any advice?
I'm sorry I didn't catch this in my prior reviews (v5+v8).
hmp.c +1 -1
@@ -1700,7 +1700,7 @@ void hmp_object_add(Monitor *mon, const QDict *qdict)Object *obj =NULL ;QObject *pobj; - pobj =qdict_crumple (qdict, true, &err); + pobj =qdict_crumple (qdict, &err);if (err) { goto cleanup; }
include/qapi/qmp/qdict.h +1 -1
@@ -73,7 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); -QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp); +QObject *qdict_crumple (const QDict *src,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);
qapi/qobject-input-visitor.c +1 -1
@@ -769,7 +769,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, goto cleanup; } - pobj =qdict_crumple (pdict, true, errp); + pobj =qdict_crumple (pdict, errp);if (!pobj) { goto cleanup; }
qobject/qdict.c +17 -26
68 lines not on this page · raw message
qom/object_interfaces.c +1 -1
@@ -170,7 +170,7 @@ Object *user_creatable_add_opts(QemuOpts *opts, Error **errp)QEMU_OPTS_REPEAT_POLICY_ALL , &error_abort); - pobj =qdict_crumple (pdict, true, errp); + pobj =qdict_crumple (pdict, errp);if (!pobj) { goto cleanup; }
tests/check-qdict.c +10 -148
268 lines not on this page · raw message
Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict
In reply to Max ReitzQuoted 126 lines · Max Reitz
Quoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{ 'foo.0.bar': 'one', 'foo.0.wizz': '1', 'foo.1.bar': 'two', 'foo.1.wizz': '2'}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{ 'foo': [ { 'bar': 'one', 'wizz': '1' }, { 'bar': 'two', 'wizz': '2' } ],}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{ 'foo..bar': 'wizz', 'bar.foo..bar': 'eek', 'bar.hello': 'world' }Will end up as
{ 'foo.bar': 'wizz', 'bar': { 'foo.bar': 'eek', 'hello': 'world' } }The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.
Reviewed-by Eric Blake <[email protected]>Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 261 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 551 insertions(+)include/qapi/qmp/qdict.h +1
-0@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);qobject/qdict.c +0
-0[...]
Quoted 6 lines
Quoted 4 lines
+/**
+ *qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionariesIs recursive=false used outside tests in this series?
No, its not used.
Quoted 8 lines
Quoted 2 lines
It was suggested in a way earlier version by Max, but not sure if his code uses it or not.
In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.
Max, do you actually need this one? If yes, please explain your use case.
As far as I can tell from a quick glance, I made the point for v1 that
qdict_crumple()could be simplified by usingqdict_array_split()andqdict_array_entries().Dan then (correctly) said that using these functions would worsen runtime performance of
qdict_crumple()and that instead we can replaceqdict_array_split()byqdict_crumple(). However, for that to work, we need to be able to makeqdict_crumple()non-recursive (becauseqdict_array_split()is non-recursive).Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.
Quoted 3 lines
However, clearly there are no patches out yet that replace
qdict_array_split()byqdict_crumple(recursive=false), and I certainly won't write any untilqdict_crumple()is merged.
I prefer not to maintain code that isn't actually used. Since Dan is enjoying a few days off, and the soft freeze is closing in, I'm proposing to squash in the following:
* Drop @recursive, in the most straightforward way possible.
* Drop all tests that pass false to @recursive. This might reduce coverage, but we can fix that on top.
* While there, collapse some vertical whitespace, for consistency with spacing elsewhere in the same files.
Resulting fixup patch appended. I'd appreciate a quick look-over before I do a pull request. Current state at
http://repo.or.cz/w/qemu/armbru.git branch qapi-not-next.
Quoted 1 line
From
df87de10a12bd65d2ddc47514e951712de7c06f0Mon Sep 17 00:00:00 2001
From: Markus Armbruster <[email protected]>
Date: Tue, 25 Oct 2016 10:09:46 +0200
Subject: [PATCH] fixup! qdict: implement a qdict_crumple method for
un-flattening a dict
--- include/qapi/qmp/qdict.h | 2 +- qobject/qdict.c | 42 ++++++--------- tests/check-qdict.c | 133 +++-------------------------------------------- 3 files changed, 23 insertions(+), 154 deletions(-)
include/qapi/qmp/qdict.h +1 -1
@@ -73,7 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); -QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp); +QObject *qdict_crumple (const QDict *src,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);
qobject/qdict.c +15 -27
85 lines not on this page · raw message
tests/check-qdict.c +7 -126
261 lines not on this page · raw message
Signature · 1 line
2.5.5
Re: [Qemu-devel] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict
Quoted 141 lines · Markus Armbruster
Quoted 126 lines · Max Reitz
Quoted 99 lines · Markus Armbruster
Quoted 97 lines · Daniel P. Berrange
Quoted 87 lines · Markus Armbruster
Quoted 84 lines · Daniel P. Berrange
The
qdict_flatten()method will take a dict whose elements are further nested dicts/lists and flatten them by concatenating keys.The
qdict_crumple()method aims to do the reverse, taking a flat qdict, and turning it into a set of nested dicts/lists. It will apply nesting based on the key name, with a '.' indicating a new level in the hierarchy. If the keys in the nested structure are all numeric, it will create a list, otherwise it will create a dict.If the keys are a mixture of numeric and non-numeric, or the numeric keys are not in strictly ascending order, an error will be reported.
As an example, a flat dict containing
{ 'foo.0.bar': 'one', 'foo.0.wizz': '1', 'foo.1.bar': 'two', 'foo.1.wizz': '2'}
will get turned into a dict with one element 'foo' whose value is a list. The list elements will each in turn be dicts.
{ 'foo': [ { 'bar': 'one', 'wizz': '1' }, { 'bar': 'two', 'wizz': '2' } ],}
If the key is intended to contain a literal '.', then it must be escaped as '..'. ie a flat dict
{ 'foo..bar': 'wizz', 'bar.foo..bar': 'eek', 'bar.hello': 'world' }Will end up as
{ 'foo.bar': 'wizz', 'bar': { 'foo.bar': 'eek', 'hello': 'world' } }The intent of this function is that it allows a set of QemuOpts to be turned into a nested data structure that mirrors the nesting used when the same object is defined over QMP.
Reviewed-by Eric Blake <[email protected]>Reviewed-by Kevin Wolf <[email protected]>Reviewed-by Marc-André Lureau <[email protected]>Signed-off-by Daniel P. Berrange <[email protected]>--- include/qapi/qmp/qdict.h | 1 + qobject/qdict.c | 289 +++++++++++++++++++++++++++++++++++++++++++++++ tests/check-qdict.c | 261 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 551 insertions(+)include/qapi/qmp/qdict.h +1
-0@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);void qdict_extract_subqdict (QDict *src,QDict **dst,const char *start);void qdict_array_split (QDict *src,QList **dst);int qdict_array_entries (QDict *src,const char *subqdict); +QObject *qdict_crumple (const QDict *src,bool recursive,Error **errp);void qdict_join (QDict *dest,QDict *src,bool overwrite);qobject/qdict.c +0
-0[...]
Quoted 6 lines
Quoted 4 lines
+/**
+ *qdict_crumple:
+ * @src: the original flat dictionary (only scalar values) to crumple
+ * @recursive: true to recursively crumple nested dictionariesIs recursive=false used outside tests in this series?
No, its not used.
Quoted 8 lines
Quoted 2 lines
It was suggested in a way earlier version by Max, but not sure if his code uses it or not.
In general, I prefer features to be added right before they're used, and I really dislike adding features "just in case". YAGNI.
Max, do you actually need this one? If yes, please explain your use case.
As far as I can tell from a quick glance, I made the point for v1 that
qdict_crumple()could be simplified by usingqdict_array_split()andqdict_array_entries().Dan then (correctly) said that using these functions would worsen runtime performance of
qdict_crumple()and that instead we can replaceqdict_array_split()byqdict_crumple(). However, for that to work, we need to be able to makeqdict_crumple()non-recursive (becauseqdict_array_split()is non-recursive).Dan also said that in the long run we want to keep the tree structure in the block layer instead of flattening everything down which would rid us of several other QDict functions (and would make non-recursive behavior obsolete again). I believe that this is an idea for the (far?) future, as can be seen from the discussion you and others had on this very topic in this version here.
Quoted 3 lines
However, clearly there are no patches out yet that replace
qdict_array_split()byqdict_crumple(recursive=false), and I certainly won't write any untilqdict_crumple()is merged.I prefer not to maintain code that isn't actually used. Since Dan is enjoying a few days off, and the soft freeze is closing in, I'm proposing to squash in the following:
* Drop @recursive, in the most straightforward way possible.
* Drop all tests that pass false to @recursive. This might reduce coverage, but we can fix that on top.
Quoted 6 lines
* While there, collapse some vertical whitespace, for consistency with spacing elsewhere in the same files.
Resulting fixup patch appended. I'd appreciate a quick look-over before I do a pull request. Current state at
http://repo.or.cz/w/qemu/armbru.git branch qapi-not-next.
Had a quick look, and it seems fine to me.
Regards,
Daniel
Signature · 3 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o- http://search.cpan.org/~danberr/ :|