qapi: QMP introspection
v2 cover qapi: QMP introspection
Still RFC, because it has a couple of FIXMEs and TODOs.
* PATCH 01-19 are preliminary fixes and cleanups. Not really RFC.
* PATCH 20-38 basically replace the interemediate representation. The replacement isn't complete, but fully functional. See PATCH 21 for rationale and future work.
* PATCH 39-44 replace the '**' type bypass by a proper top type called 'any'.
* PATCH 45 is the introspection RFC.
* PATCH 46-47 are minor variations to hide a bit more detail from clients.
I'm afraid the size of my series will strain reviewers. I did my level best to split things up into reviewable pieces.
Series depends on my "[PATCH for-2.4 0/2] qom: Fix misuse of Error API" to pass tests.
You can also fetch it from my git://repo.or.cz/qemu/armbru.git branch qapi-introspect.
I'll be offline for two weeks starting next Monday.
Bombs away!
v2:
* Old PATCH 01-18 have been merged
* PATCH 19 exploded. Comparing against RFC v1 is almost certainly a waste of time, except perhaps for the QAPI schema introspect.json. The awkward alternate Value got replaced by the proper top type 'any'.
* For even more kaboom, I fixed and cleaned up along the way.
Markus Armbruster (47):
qapi: Clarify docs on including the same file multiple times
qapi: Clean up cgen() and mcgen()
qapi: Simplify guardname()
qapi-event: Clean up how name of enum QAPIEvent is made
qapi: Reject -p arguments that break qapi-event.py
qapi: Drop unused and useless parameters and variables
qapi: Generate a nicer struct for flat unions
qapi-visit: Fix generated code when schema has forward refs
qapi-visit: Replace list implicit_structs by set
qapi-visit: Fix two name arguments passed to visitors tests/qapi-schema: Document alternate's enum lacks visit function tests/qapi-schema: Document events with with base don't work tests/qapi-schema: Restore test case for flat union base bug
qapi-tests: New tests for union, alternate command arguments
qapi: Fix to reject union command arguments
qapi-commands: Fix gen_err_check(e) for e and e != 'local_err'
qapi-commands: Inline gen_marshal_output_call()
qapi-commands: Don't feed output of mcgen() to mcgen() again
qapi: Generated code cleanup
qapi: Rename class QAPISchema to QAPISchemaParser
qapi: New QAPISchema intermediate reperesentation
qapi: QAPISchema code generation helper methods
qapi: New QAPISchemaVisitor
tests/qapi-schema: Convert test harness to QAPISchemaVisitor
qapi: Make generators work on sorted schema expressions
qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
qapi-commands: Convert to QAPISchemaVisitor
qapi: Replace dirty is_c_ptr() by method c_null()
qapi: De-duplicate enum code generation
qapi-event: Eliminate global variable event_enum_value
qapi-event: Convert to QAPISchemaVisitor, fixing data with base
qapi: Clean up after recent conversions to QAPISchemaVisitor
qapi-visit: Rearrange code a bit
qapi-commands: Rearrange code
qapi: Rename qmp_marshal_input_FOO() to qmp_marshal_FOO()
qapi: De-duplicate parameter list generation
qapi-commands: De-duplicate output marshaling functions
qapi: Improve built-in type documentation
qapi: Introduce a first class 'any' type
qom: Don't use 'gen': false for qom-get, qom-set, object-add
qapi-schema: Fix up misleading specification of netdev_add
qmp: Improve netdev_add usage example in the manual
qapi: Pseudo-type '**' is now unused, drop it
qapi: New QMP command query-schema for QMP schema introspection
qapi-introspect: Map all integer types to 'int'
qapi-introspect: Hide type names
.gitignore | 1 +
Makefile | 9 +-
Makefile.objs | 4 +-
docs/qapi-code-gen.txt | 53 +-
docs/writing-qmp-commands.txt | 8 +-
include/monitor/monitor.h | 3 -
include/qapi/visitor-impl.h | 2 +
include/qapi/visitor.h | 1 +
monitor.c | 17 +-
qapi-schema.json | 25 +-
qapi/introspect.json | 69 +++
qapi/qapi-dealloc-visitor.c | 9 +
qapi/qapi-visit-core.c | 6 +
qapi/qmp-input-visitor.c | 11 +
qapi/qmp-output-visitor.c | 9 +
qmp-commands.hx | 266 +++++----
qmp.c | 20 +-
scripts/qapi-commands.py | 355 ++++++-----
scripts/qapi-event.py | 243 +++-----
scripts/qapi-introspect.py | 189 ++++++
scripts/qapi-types.py | 378 +++++-------
scripts/qapi-visit.py | 368 +++++-------
scripts/qapi.py | 658 +++++++++++++++++----
tests/.gitignore | 1 +
tests/Makefile | 15 +-
tests/qapi-schema/alternate-good.out | 16 +-
tests/qapi-schema/args-alternate.err | 1 +
...type-bypass-no-gen.exit => args-alternate.exit} | 0
tests/qapi-schema/args-alternate.json | 4 +
.../{type-bypass-no-gen.out => args-alternate.out} | 0
tests/qapi-schema/args-returns-any.err | 1 +
tests/qapi-schema/args-returns-any.exit | 1 +
tests/qapi-schema/args-returns-any.json | 6 +
.../{type-bypass.err => args-returns-any.out} | 0
tests/qapi-schema/args-union.err | 1 +
tests/qapi-schema/args-union.exit | 1 +
tests/qapi-schema/args-union.json | 4 +
tests/qapi-schema/args-union.out | 0
tests/qapi-schema/comments.out | 5 +-
tests/qapi-schema/data-member-array.out | 14 +-
tests/qapi-schema/empty.out | 4 +-
tests/qapi-schema/enum-empty.out | 5 +-
tests/qapi-schema/event-case.out | 5 +-
tests/qapi-schema/flat-union-base-star.err | 2 +-
tests/qapi-schema/flat-union-base-star.json | 2 +-
tests/qapi-schema/flat-union-reverse-define.out | 22 +-
tests/qapi-schema/ident-with-escape.out | 8 +-
tests/qapi-schema/include-relpath.out | 5 +-
tests/qapi-schema/include-repetition.out | 5 +-
tests/qapi-schema/include-simple.out | 5 +-
tests/qapi-schema/indented-expr.out | 8 +-
tests/qapi-schema/qapi-schema-test.json | 33 +-
tests/qapi-schema/qapi-schema-test.out | 187 ++++--
tests/qapi-schema/returns-int.out | 6 +-
tests/qapi-schema/test-qapi.py | 37 +-
tests/qapi-schema/type-bypass-no-gen.err | 1 -
tests/qapi-schema/type-bypass-no-gen.json | 2 -
tests/qapi-schema/type-bypass.exit | 1 -
tests/qapi-schema/type-bypass.json | 2 -
tests/qapi-schema/type-bypass.out | 3 -
tests/test-qmp-event.c | 2 +-
tests/test-qmp-input-strict.c | 2 +-
tests/test-qmp-input-visitor.c | 82 ++-
tests/test-qmp-output-visitor.c | 55 +-
64 files changed, 2011 insertions(+), 1247 deletions(-)
create mode 100644 qapi/introspect.json
create mode 100644 scripts/qapi-introspect.py
create mode 100644 tests/qapi-schema/args-alternate.err
rename tests/qapi-schema/{type-bypass-no-gen.exit => args-alternate.exit} (100%)
create mode 100644 tests/qapi-schema/args-alternate.json
rename tests/qapi-schema/{type-bypass-no-gen.out => args-alternate.out} (100%)
create mode 100644 tests/qapi-schema/args-returns-any.err
create mode 100644 tests/qapi-schema/args-returns-any.exit
create mode 100644 tests/qapi-schema/args-returns-any.json
rename tests/qapi-schema/{type-bypass.err => args-returns-any.out} (100%)
create mode 100644 tests/qapi-schema/args-union.err
create mode 100644 tests/qapi-schema/args-union.exit
create mode 100644 tests/qapi-schema/args-union.json
create mode 100644 tests/qapi-schema/args-union.out
delete mode 100644 tests/qapi-schema/type-bypass-no-gen.err
delete mode 100644 tests/qapi-schema/type-bypass-no-gen.json
delete mode 100644 tests/qapi-schema/type-bypass.exit
delete mode 100644 tests/qapi-schema/type-bypass.json
delete mode 100644 tests/qapi-schema/type-bypass.outSignature · 1 line
1.9.3
RFC PATCH v2 1/47 qapi: Clarify docs on including the same file multiple times
It's idempotent.
--- docs/qapi-code-gen.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
docs/qapi-code-gen.txt +1 -1
@@ -163,7 +163,7 @@ The QAPI schema definitions can be modularized using the 'include' directive: The directive is evaluated recursively, and include paths are relative to the file using the directive. Multiple includes of the same file are -safe. No other keys should appear in the expression, and the include +idempotent. No other keys should appear in the expression, and the include value should be a string. As a matter of style, it is a good idea to have all files be
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 01/47] qapi: Clarify docs on including the same file multiple times
Quoted 6 lines · Markus Armbruster
It's idempotent.
Signed-off-by Markus Armbruster <[email protected]>--- docs/qapi-code-gen.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 2/47 qapi: Clean up cgen() and mcgen()
In reply to Markus ArmbrusterCommit 05dfb26 added eatspace stripping to mcgen(). Move it to cgen(), just in case somebody gets tempted to use cgen() directly instead of via mcgen().
cgen() indents blank lines. No such lines get generated right now, but fix it anyway.
We use triple-quoted strings for program text, like this:
'''
Program text
any number of lines
'''Keeps the program text relatively readable, but puts an extra newline at either end. mcgen() "fixes" that by dropping the first and last line outright. Drop only the newlines.
This unmasks a bug in qapi-commands.py: four quotes instead of three. Fix it up.
Output doesn't change
--- scripts/qapi-commands.py | 2 +- scripts/qapi.py | 19 ++++++++++++------- 2 files changed, 13 insertions(+), 8 deletions(-)
scripts/qapi-commands.py +1 -1
@@ -56,7 +56,7 @@ def gen_sync_call(name, args, ret_type, indent=0): name=c_name(name), args=arglist, retval=retval).rstrip() if ret_type: ret += "\n" + gen_err_check('local_err') - ret += "\n" + mcgen('''' + ret += "\n" + mcgen(''' %(marshal_output_call)s ''', marshal_output_call=gen_marshal_output_call(name, ret_type)).rstrip()
scripts/qapi.py +12 -7
@@ -943,16 +943,21 @@ def pop_indent(indent_amount=4): global indent_level indent_level -= indent_amount +# Generate @code with @kwds interpolated. +# Obey indent_level, and strip eatspace. def cgen(code, **kwds): - indent = genindent(indent_level) - lines = code.split('\n') - lines = map(lambda x: indent + x, lines) - return '\n'.join(lines) % kwds + '\n' - -def mcgen(code, **kwds): - raw = cgen('\n'.join(code.split('\n')[1:-1]), **kwds) + raw = code % kwds + if indent_level: + indent = genindent(indent_level) + raw = re.subn("^.", indent + '\g<0>', raw, 0, re.MULTILINE) + raw = raw[0] return re.sub(re.escape(eatspace) + ' *', '', raw) +def mcgen(code, **kwds): + if code[0] == '\n': + code = code[1:] + return cgen(code, **kwds) + def basename(filename): return filename.split("/")[-1]
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 02/47] qapi: Clean up cgen() and mcgen()
Quoted 14 lines · Markus Armbruster
Commit 05dfb26 added eatspace stripping to
mcgen(). Move it tocgen(), just in case somebody gets tempted to usecgen()directly instead of viamcgen().
cgen()indents blank lines. No such lines get generated right now, but fix it anyway.We use triple-quoted strings for program text, like this:
''' Program text any number of lines '''
Quoted 8 lines
Keeps the program text relatively readable, but puts an extra newline at either end.
mcgen()"fixes" that by dropping the first and last line outright. Drop only the newlines.This unmasks a bug in
qapi-commands.py: four quotes instead of three. Fix it up.Output doesn't change
Confirmed.
Quoted 7 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 2 +- scripts/qapi.py | 19 ++++++++++++------- 2 files changed, 13 insertions(+), 8 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 3/47 qapi: Simplify guardname()
In reply to Markus ArmbrusterThe guards around built-in declarations lose their _H. It never made much sense anyway.
--- scripts/qapi.py | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)
scripts/qapi.py +3 -7
@@ -958,14 +958,9 @@ def mcgen(code, **kwds): code = code[1:] return cgen(code, **kwds) -def basename(filename): - return filename.split("/")[-1] def guardname(filename): - guard = basename(filename).rsplit(".", 1)[0] - for substr in [".", " ", "-"]: - guard = guard.replace(substr, "_") - return guard.upper() + '_H' + return c_name(filename, protect=False).upper() def guardstart(name): return mcgen('''
@@ -1035,6 +1030,7 @@ def parse_command_line(extra_options = "", extra_long_options = []): def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, c_comment, h_comment): + guard = guardname(prefix + h_file) c_file = output_dir + prefix + c_file h_file = output_dir + prefix + h_file
@@ -1067,7 +1063,7 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, #define %(guard)s ''', - comment = h_comment, guard = guardname(h_file))) + comment = h_comment, guard = guard)) return (fdef, fdecl)
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 03/47] qapi: Simplify guardname()
Quoted 2 lines · Markus Armbruster
The guards around built-in declarations lose their _H. It never made much sense anyway.
Correct - the _H tail on the guard for the overall file makes sense (when the overall file ends in .h), but for an unrelated use in the middle of the file, it has no bearing on any file name and does not need a _H tail.
Quoted 5 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)
For an example of what it changes, qapi-types.h sees changes like this (while the original #ifndef QAPI_TYPES_H guard at the front of the file remains unchanged, which is good):
qapi-types.h renamed from qapi-types.h.old +3 -3
@@ -20,8 +20,8 @@#include <stdint.h> -#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL_H -#define QAPI_TYPES_BUILTIN_STRUCT_DECL_H +#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL +#define QAPI_TYPES_BUILTIN_STRUCT_DECL typedef struct int32List
@@ -141,7 +141,7 @@ struct uint32List *next; } uint32List; -#endif /* QAPI_TYPES_BUILTIN_STRUCT_DECL_H */ +#endif /* QAPI_TYPES_BUILTIN_STRUCT_DECL */ extern const char *const ErrorClass_lookup[];
...
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 4/47 qapi-event: Clean up how name of enum QAPIEvent is made
In reply to Markus ArmbrusterUse c_name() instead of ad hoc code. Doesn't upcase the -p prefix, which is an improvement in my book. Unbreaks prefix containing '.', but other funny characters remain broken. To be fixed next.
--- scripts/qapi-event.py | 2 +- tests/test-qmp-event.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
scripts/qapi-event.py +1 -1
@@ -267,7 +267,7 @@ fdecl.write(mcgen(''' exprs = parse_schema(input_file) -event_enum_name = prefix.upper().replace('-', '_') + "QAPIEvent" +event_enum_name = c_name(prefix + "QAPIEvent", protect=False) event_enum_values = [] event_enum_strings = []
tests/test-qmp-event.c +1 -1
@@ -94,7 +94,7 @@ static bool qdict_cmp_simple(QDict *a, QDict *b)/* This function is hooked as final emit function, which can verify the correctness. */ -static void event_test_emit (TEST_QAPIEvent event,QDict *d,Error **errp) +static void event_test_emit (test_QAPIEvent event,QDict *d,Error **errp) {QObject *obj;QDict *t;
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 04/47] qapi-event: Clean up how name of enum QAPIEvent is made
Quoted 9 lines · Markus Armbruster
Use
c_name()instead of ad hoc code. Doesn't upcase the -p prefix, which is an improvement in my book. Unbreaks prefix containing '.', but other funny characters remain broken. To be fixed next.Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-event.py | 2 +- tests/test-qmp-event.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
No change to the generated qapi-event.[ch], so it looks like only the testsuite is affected. [In fact, as far as I can tell, only docs/qapi-code-gen.txt and tests/Makefile even take advantage of the '-p prefix' argument.] Fine by me.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 5/47 qapi: Reject -p arguments that break qapi-event.py
In reply to Markus Armbrusterqapi-event.py breaks when you ask for a funny prefix like '@'. Protect it.
--- scripts/qapi.py | 6 ++++++ 1 file changed, 6 insertions(+)
scripts/qapi.py +6 -0
@@ -1003,6 +1003,12 @@ def parse_command_line(extra_options = "", extra_long_options = []): for oa in opts: o, a = oa if o in ("-p", "--prefix"): + match = re.match('([A-Za-z_.-][A-Za-z0-9_.-]*)?', a) + if match.end() != len(a): + print >>sys.stderr, \ + "%s: 'funny character '%s' in argument of -prefix" \ + % (sys.argv[0], a[match.end()]) + sys.exit(1) prefix = a elif o in ("-o", "--output-dir"): output_dir = a + "/"
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 05/47] qapi: Reject -p arguments that break qapi-event.py
Quoted 2 lines · Markus Armbruster
qapi-event.pybreaks when you ask for a funny prefix like '@'. Protect it.
Only possible from the command line (not triggered by our makefiles); but doesn't hurt.
Quoted 13 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 6 ++++++ 1 file changed, 6 insertions(+)scripts/qapi.py +0
-0
scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1006
Quoted 4 lines
scripts/qapi.py +1
-0@@ -1003,6 +1003,12 @@ def parse_command_line(extra_options = "", extra_long_options = []): for oa in opts: o, a = oa if o in ("-p", "--prefix"): + match = re.match('([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)
I can understand allowing a leading _, but why bother allowing a leading '.' or '-'? Those will get normalized to _, but in all honesty, no one should ever be doing that.
I'd be just as happy with the shorter:
match = re.match('([A-Za-z_][A-Za-z0-9_.-]*)?', a)
scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1009
Quoted 3 lines
scripts/qapi.py +3
-0+ if match.end() != len(a): + print >>sys.stderr, \ + "%s: 'funny character '%s' in argument of -prefix" \
'qemu' is unusual for accepting -single-dash-long-opts; I don't think python getopts does the same by default. Please spell this error message --prefix.
scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1014
Quoted 6 lines
scripts/qapi.py +2
-0+ % (sys.argv[0], a[match.end()]) + sys.exit(1) prefix = a elif o in ("-o", "--output-dir"): output_dir = a + "/"
With the second spelling fix, and optionally with the shorter regex,
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 05/47] qapi: Reject -p arguments that break qapi-event.py
Quoted 2 lines · Eric Blake
'qemu' is unusual for accepting -single-dash-long-opts; I don't think python getopts does the same by default.
Or to rephrase, qemu acts as though it uses GNU getopt_long_only() (some programs like gcc do likewise), but MOST programs that take long opts behave like they use getopt_long(). It is only the getopt_long_only() variant that accepts long options with a single dash, when it is unambiguous with POSIX getopt() parsing of multiple smashed-together short options.
[well, qemu rolls its own option parser instead of using getopt_long_only(), but you get the picture]
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 05/47] qapi: Reject -p arguments that break qapi-event.py
In reply to Eric BlakeQuoted 15 lines · Eric Blake
Quoted 2 lines · Markus Armbruster
qapi-event.pybreaks when you ask for a funny prefix like '@'. Protect it.Only possible from the command line (not triggered by our makefiles); but doesn't hurt.
Quoted 8 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 6 ++++++ 1 file changed, 6 insertions(+)
Quoted 13 lines
scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1006
Quoted 9 lines
scripts/qapi.py +1
-0@@ -1003,6 +1003,12 @@ def parse_command_line(extra_options = "", extra_long_options = []): for oa in opts: o, a = oa if o in ("-p", "--prefix"): + match = re.match('([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)I can understand allowing a leading _, but why bother allowing a leading '.' or '-'? Those will get normalized to _, but in all honesty, no one should ever be doing that.
My patch rejects exactly the prefixes that won't work.
Quoted 3 lines
I'd be just as happy with the shorter:
match =
re.match('([A-Za-z_][A-Za-z0-9_.-]*)?', a)
This additionally rejects a few rather foolish ones.
I have a slight preference for the tool staying out of policing foolish prefixes.
Quoted 7 lines
scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1009
Quoted 3 lines
scripts/qapi.py +3
-0+ if match.end() != len(a): + print >>sys.stderr, \ + "%s: 'funny character '%s' in argument of -prefix" \'qemu' is unusual for accepting -single-dash-long-opts; I don't think python getopts does the same by default. Please spell this error message --prefix.
Typo, will fix.
Quoted 10 lines
scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1014
Quoted 6 lines
scripts/qapi.py +2
-0+ % (sys.argv[0], a[match.end()]) + sys.exit(1) prefix = a elif o in ("-o", "--output-dir"): output_dir = a + "/"With the second spelling fix, and optionally with the shorter regex,
Reviewed-by Eric Blake <[email protected]>
Thanks!
RFC PATCH v2 6/47 qapi: Drop unused and useless parameters and variables
In reply to Markus Armbrustergen_sync_call()'s parameter indent is useless: gen_sync_call() uses it only as optional argument for push_indent() and pop_indent(), their default is four, and gen_sync_call()'s only caller passes four.
gen_visitor_input_containers_decl()'s parameter obj is always "QOBJECT(args)".
--- scripts/qapi-commands.py | 27 +++++++++++++-------------- scripts/qapi-event.py | 1 - scripts/qapi-types.py | 1 - scripts/qapi-visit.py | 47 ++++++++++++++++++++++------------------------- scripts/qapi.py | 1 - 5 files changed, 35 insertions(+), 42 deletions(-)
scripts/qapi-commands.py +13 -14
@@ -38,7 +38,7 @@ if (local_err) { ''') return '' -def gen_sync_call(name, args, ret_type, indent=0): +def gen_sync_call(name, args, ret_type): ret = "" arglist="" retval=""
@@ -48,7 +48,7 @@ def gen_sync_call(name, args, ret_type, indent=0): if optional: arglist += "has_%s, " % c_name(argname) arglist += "%s, " % (c_name(argname)) - push_indent(indent) + push_indent() ret = mcgen(''' %(retval)sqmp_%(name)s(%(args)s&local_err);
@@ -60,7 +60,7 @@ def gen_sync_call(name, args, ret_type, indent=0): %(marshal_output_call)s ''', marshal_output_call=gen_marshal_output_call(name, ret_type)).rstrip() - pop_indent(indent) + pop_indent() return ret.rstrip()
@@ -69,17 +69,16 @@ def gen_marshal_output_call(name, ret_type): return "" return "qmp_marshal_output_%s(retval, ret, &local_err);" % c_name(name) -def gen_visitor_input_containers_decl(args, obj): +def gen_visitor_input_containers_decl(args): ret = "" push_indent() if len(args) > 0: ret += mcgen(''' -QmpInputVisitor *mi = qmp_input_visitor_new_strict(%(obj)s); +QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args)); QapiDeallocVisitor *md; Visitor *v; -''', - obj=obj) +''') pop_indent() return ret.rstrip()
@@ -161,7 +160,7 @@ qapi_dealloc_visitor_cleanup(md); pop_indent() return ret.rstrip() -def gen_marshal_output(name, args, ret_type, middle_mode): +def gen_marshal_output(name, ret_type): if not ret_type: return ""
@@ -194,14 +193,14 @@ out: return ret -def gen_marshal_input_decl(name, args, ret_type, middle_mode): +def gen_marshal_input_decl(name, middle_mode): ret = 'void qmp_marshal_input_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name) if not middle_mode: ret = "static " + ret return ret def gen_marshal_input(name, args, ret_type, middle_mode): - hdr = gen_marshal_input_decl(name, args, ret_type, middle_mode) + hdr = gen_marshal_input_decl(name, middle_mode) ret = mcgen(''' %(header)s
@@ -228,7 +227,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode): %(visitor_input_block)s ''', - visitor_input_containers_decl=gen_visitor_input_containers_decl(args, "QOBJECT(args)"), + visitor_input_containers_decl=gen_visitor_input_containers_decl(args), visitor_input_vars_decl=gen_visitor_input_vars_decl(args), visitor_input_block=gen_visitor_input_block(args)) else:
@@ -240,7 +239,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode): ret += mcgen(''' %(sync_call)s ''', - sync_call=gen_sync_call(name, args, ret_type, indent=4)) + sync_call=gen_sync_call(name, args, ret_type)) if re.search('^ *goto out\\;', ret, re.MULTILINE): ret += mcgen('''
@@ -360,11 +359,11 @@ for cmd in commands: ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n" fdecl.write(ret) if ret_type: - ret = gen_marshal_output(cmd['command'], arglist, ret_type, middle_mode) + "\n" + ret = gen_marshal_output(cmd['command'], ret_type) + "\n" fdef.write(ret) if middle_mode: - fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], arglist, ret_type, middle_mode)) + fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], middle_mode)) ret = gen_marshal_input(cmd['command'], arglist, ret_type, middle_mode) + "\n" fdef.write(ret)
scripts/qapi-event.py +0 -1
@@ -199,7 +199,6 @@ const char *%(event_enum_name)s_lookup[] = { ''', event_enum_name = event_enum_name) - i = 0 for string in event_enum_strings: ret += mcgen(''' "%(string)s",
scripts/qapi-types.py +0 -1
@@ -108,7 +108,6 @@ def generate_enum_lookup(name, values): const char * const %(name)s_lookup[] = { ''', name=c_name(name)) - i = 0 for value in values: index = c_enum_const(name, value) ret += mcgen('''
scripts/qapi-visit.py +22 -25
136 lines not on this page · raw message
scripts/qapi.py +0 -1
@@ -507,7 +507,6 @@ def check_command(expr, expr_info): def check_event(expr, expr_info): global events name = expr['event'] - params = expr.get('data') if name.upper() == 'MAX': raise QAPIExprError(expr_info, "Event name 'MAX' cannot be created")
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 06/47] qapi: Drop unused and useless parameters and variables
Quoted 6 lines · Markus Armbruster
gen_sync_call()'s parameter indent is useless:
gen_sync_call()uses it only as optional argument forpush_indent()andpop_indent(), their default is four, and gen_sync_call()'s only caller passes four.gen_visitor_input_containers_decl()'s parameter obj is always "QOBJECT(args)".
It might be nice to call out that several other functions are also stripped of unused arguments. I was assuming that only two functions (and their callers) would be modified, but the patch touched more:
Quoted 11 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 27 +++++++++++++-------------- scripts/qapi-event.py | 1 - scripts/qapi-types.py | 1 - scripts/qapi-visit.py | 47 ++++++++++++++++++++++------------------------- scripts/qapi.py | 1 - 5 files changed, 35 insertions(+), 42 deletions(-)
scripts/qapi-commands.py qapi_dealloc_visitor_cleanup(md); · line 165
Quoted 7 lines
scripts/qapi-commands.py +1
-1@@ -161,7 +160,7 @@ qapi_dealloc_visitor_cleanup(md); pop_indent() return ret.rstrip() -def gen_marshal_output(name, args, ret_type, middle_mode): +def gen_marshal_output(name, ret_type): if not ret_type: return ""
For example, gen_marshal_output() was not mentioned in the commit message.
scripts/qapi-commands.py out: · line 196
Quoted 7 lines
scripts/qapi-commands.py +1
-1@@ -194,14 +193,14 @@ out: return ret -def gen_marshal_input_decl(name, args, ret_type, middle_mode): +def gen_marshal_input_decl(name, middle_mode):
Or gen_marshal_input_decl()
scripts/qapi-event.py const char *%(event_enum_name)s_lookup[] = { · line 202
Quoted 7 lines
scripts/qapi-event.py +1
-1+++ b/scripts/qapi-event.py@@ -199,7 +199,6 @@ const char *%(event_enum_name)s_lookup[] = { ''', event_enum_name = event_enum_name) - i = 0 for string in event_enum_strings:
I guess the subject line covered deletion of unused internal variables.
Quoted 8 lines
scripts/qapi-visit.py +2
-1+++ b/scripts/qapi-visit.py@@ -441,44 +440,42 @@ fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) elif expr.has_key('union'): ret = generate_visit_union(expr) - ret += generate_visit_list(expr['union'], expr['data']) + ret += generate_visit_list(expr['union'])
scripts/qapi-visit.py fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) · line 450
Quoted 10 lines
scripts/qapi-visit.py +2
-3fdef.write(ret) enum_define = discriminator_find_enum_define(expr) ret = "" if not enum_define: - ret = generate_decl_enum('%sKind' % expr['union'], - expr['data'].keys()) - ret += generate_declaration(expr['union'], expr['data']) + ret = generate_decl_enum('%sKind' % expr['union']) + ret += generate_declaration(expr['union'])
As long as you are touching this, generate_decl_enum(expr['union'] + 'Kind') would read nicer.
scripts/qapi-visit.py fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) · line 469
Quoted 11 lines
scripts/qapi-visit.py +2
-4fdecl.write(ret) elif expr.has_key('alternate'): ret = generate_visit_alternate(expr['alternate'], expr['data']) - ret += generate_visit_list(expr['alternate'], expr['data']) + ret += generate_visit_list(expr['alternate']) fdef.write(ret) - ret = generate_decl_enum('%sKind' % expr['alternate'], - expr['data'].keys()) - ret += generate_declaration(expr['alternate'], expr['data']) + ret = generate_decl_enum('%sKind' % expr['alternate'])
Same here.
At any rate, no change to generated output. So assuming you are happy with the commit message, or take my advice to improve it, the code cleanup itself is fine.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 06/47] qapi: Drop unused and useless parameters and variables
Quoted 11 lines · Eric Blake
Quoted 6 lines · Markus Armbruster
gen_sync_call()'s parameter indent is useless:
gen_sync_call()uses it only as optional argument forpush_indent()andpop_indent(), their default is four, and gen_sync_call()'s only caller passes four.gen_visitor_input_containers_decl()'s parameter obj is always "QOBJECT(args)".
It might be nice to call out that several other functions are also stripped of unused arguments. I was assuming that only two functions (and their callers) would be modified, but the patch touched more:
I'll rephrase the commit message to make this more obvious.
Quoted 59 lines
Quoted 10 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 27 +++++++++++++-------------- scripts/qapi-event.py | 1 - scripts/qapi-types.py | 1 - scripts/qapi-visit.py | 47 ++++++++++++++++++++++------------------------- scripts/qapi.py | 1 - 5 files changed, 35 insertions(+), 42 deletions(-)scripts/qapi-commands.py qapi_dealloc_visitor_cleanup(md); · line 165
Quoted 7 lines
scripts/qapi-commands.py +1
-1@@ -161,7 +160,7 @@ qapi_dealloc_visitor_cleanup(md); pop_indent() return ret.rstrip() -def gen_marshal_output(name, args, ret_type, middle_mode): +def gen_marshal_output(name, ret_type): if not ret_type: return ""For example,
gen_marshal_output()was not mentioned in the commit message.scripts/qapi-commands.py out: · line 196
Quoted 7 lines
scripts/qapi-commands.py +1
-1@@ -194,14 +193,14 @@ out: return ret -def gen_marshal_input_decl(name, args, ret_type, middle_mode): +def gen_marshal_input_decl(name, middle_mode):Or
gen_marshal_input_decl()scripts/qapi-event.py const char *%(event_enum_name)s_lookup[] = { · line 202
Quoted 7 lines
scripts/qapi-event.py +1
-1+++ b/scripts/qapi-event.py@@ -199,7 +199,6 @@ const char *%(event_enum_name)s_lookup[] = { ''', event_enum_name = event_enum_name) - i = 0 for string in event_enum_strings:I guess the subject line covered deletion of unused internal variables.
Quoted 8 lines
scripts/qapi-visit.py +2
-1+++ b/scripts/qapi-visit.py@@ -441,44 +440,42 @@ fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) elif expr.has_key('union'): ret = generate_visit_union(expr) - ret += generate_visit_list(expr['union'], expr['data']) + ret += generate_visit_list(expr['union'])scripts/qapi-visit.py fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) · line 450
Quoted 10 lines
scripts/qapi-visit.py +2
-3fdef.write(ret) enum_define = discriminator_find_enum_define(expr) ret = "" if not enum_define: - ret = generate_decl_enum('%sKind' % expr['union'], - expr['data'].keys()) - ret += generate_declaration(expr['union'], expr['data']) + ret = generate_decl_enum('%sKind' % expr['union']) + ret += generate_declaration(expr['union'])
Quoted 2 lines
As long as you are touching this,
generate_decl_enum(expr['union'] + 'Kind')would read nicer.
It'll go away in PATCH 27. Let's keep this patch simple.
Quoted 7 lines
Quoted 7 lines
scripts/qapi-visit.py +1
-1fdecl.write(ret) elif expr.has_key('alternate'): ret = generate_visit_alternate(expr['alternate'], expr['data']) - ret += generate_visit_list(expr['alternate'], expr['data']) + ret += generate_visit_list(expr['alternate']) fdef.write(ret)
Quoted 6 lines
scripts/qapi-visit.py fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) · line 469
Quoted 4 lines
scripts/qapi-visit.py +1
-3- ret = generate_decl_enum('%sKind' % expr['alternate'], - expr['data'].keys()) - ret += generate_declaration(expr['alternate'], expr['data']) + ret = generate_decl_enum('%sKind' % expr['alternate'])Same here.
Likewise.
Quoted 5 lines
At any rate, no change to generated output. So assuming you are happy with the commit message, or take my advice to improve it, the code cleanup itself is fine.
Reviewed-by Eric Blake <[email protected]>
Thanks!
RFC PATCH v2 7/47 qapi: Generate a nicer struct for flat unions
In reply to Markus ArmbrusterThe struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.
Example: qapi-schema-test.json has
{ 'struct': 'UserDefUnionBase',
'data': { 'string': 'str', 'enum1': 'EnumOne' } }
{ 'union': 'UserDefFlatUnion',
'base': 'UserDefUnionBase',
'discriminator': 'enum1',
'data': { 'value1' : 'UserDefA',
'value2' : 'UserDefB',
'value3' : 'UserDefB' } }We generate:
struct UserDefFlatUnion
{
EnumOne kind;
union {
void *data;
UserDefA *value1;
UserDefB *value2;
UserDefB *value3;
};
char *string;
};Change to put all base members at the beginning, unadulterated. Not only is this easier to understand, it also permits casting the flat union to its base, if that should become useful.
We now generate:
struct UserDefFlatUnion
{
char *string;
EnumOne enum1;
/* union tag is EnumOne enum1 */
union {
void *data;
UserDefA *value1;
UserDefB *value2;
UserDefB *value3;
};
};--- scripts/qapi-types.py | 32 ++++++++++++++++++-------------- scripts/qapi-visit.py | 7 +++++-- tests/test-qmp-input-visitor.c | 2 +- tests/test-qmp-output-visitor.c | 2 +- 4 files changed, 25 insertions(+), 18 deletions(-)
scripts/qapi-types.py +18 -14
@@ -200,12 +200,27 @@ def generate_union(expr, meta): ret = mcgen(''' struct %(name)s { +''', + name=name) + if base: + base_fields = find_struct(base)['data'] + ret += generate_struct_fields(base_fields) + ret += mcgen(''' + /* union tag is %(discriminator_type_name)s %(c_name)s */ +''', + discriminator_type_name=c_name(discriminator_type_name), + c_name=c_name(discriminator)) + else: + assert not discriminator + ret += mcgen(''' %(discriminator_type_name)s kind; +''', + discriminator_type_name=c_name(discriminator_type_name)) + + ret += mcgen(''' union { void *data; -''', - name=name, - discriminator_type_name=c_name(discriminator_type_name)) +''') for key in typeinfo: ret += mcgen('''
@@ -216,17 +231,6 @@ struct %(name)s ret += mcgen(''' }; -''') - - if base: - assert discriminator - base_fields = find_struct(base)['data'].copy() - del base_fields[discriminator] - ret += generate_struct_fields(base_fields) - else: - assert not discriminator - - ret += mcgen(''' }; ''') if meta == 'alternate':
scripts/qapi-visit.py +5 -2
@@ -288,20 +288,23 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e name=c_name(name)) if not discriminator: + tag = 'kind' disc_key = "type" else: + tag = discriminator disc_key = discriminator ret += mcgen(''' - visit_type_%(disc_type)s(m, &(*obj)->kind, "%(disc_key)s", &err); + visit_type_%(disc_type)s(m, &(*obj)->%(c_tag)s, "%(disc_key)s", &err); if (err) { goto out_obj; } if (!visit_start_union(m, !!(*obj)->data, &err) || err) { goto out_obj; } - switch ((*obj)->kind) { + switch ((*obj)->%(c_tag)s) { ''', disc_type = disc_type, + c_tag=c_name(tag), disc_key = disc_key) for key in members:
tests/test-qmp-input-visitor.c +1 -1
@@ -313,7 +313,7 @@ static void test_visitor_in_union_flat(TestInputVisitorData *data,visit_type_UserDefFlatUnion (v, &tmp,NULL , &err);g_assert (err ==NULL ); - g_assert_cmpint(tmp->kind, ==, ENUM_ONE_VALUE1); + g_assert_cmpint(tmp->enum1, ==, ENUM_ONE_VALUE1); g_assert_cmpstr(tmp->string, ==, "str");/* TODO g_assert_cmpint(tmp->integer, ==, 41); */ g_assert_cmpint(tmp->value1->boolean, ==, true);
tests/test-qmp-output-visitor.c +1 -1
@@ -437,7 +437,7 @@ static void test_visitor_out_union_flat(TestOutputVisitorData *data,Error *err =NULL ;UserDefFlatUnion *tmp =g_malloc0 (sizeof (UserDefFlatUnion)); - tmp->kind =ENUM_ONE_VALUE1 ; + tmp->enum1 =ENUM_ONE_VALUE1 ; tmp->string =g_strdup ("str" ); tmp->value1 =g_malloc0 (sizeof (UserDefA));/* TODO when generator bug is fixed: tmp->integer = 41; */
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
Quoted 5 lines · Markus Armbruster
The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.
Quoted 8 lines
Change to put all base members at the beginning, unadulterated. Not only is this easier to understand, it also permits casting the flat union to its base, if that should become useful.
We now generate:
struct UserDefFlatUnion {
It might be worth tweaking the generator to output a C comment either here (at the start of the larger struct)...
Quoted 2 lines
char *string;
EnumOne enum1;...or here (at the end of the base struct) mentioning that UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to make it easier when reading the generated code to trace back to that "inheritance" relationship. Right now, there is nothing in the generated UserDefFlatUnion that points you back to the qapi relationship of a base class. But it's not a show-stopper if you don't like my suggestion.
Quoted 8 lines
/* union tag is EnumOne enum1 */
union {
void *data;
UserDefA *value1;
UserDefB *value2;
UserDefB *value3;
};
};Only impact in files generated for qemu was to struct BlockdevOptions, in the files qapi-types.[ch], and I agree that it is an improvement. Oddly enough, it seems none of the C code cared about the field being renamed from 'kind' to 'driver' (I guess that's because a lot of our use of the blockdev code is not directly through the generated C structs, but through QObject dictionary queries).
Quoted 11 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-types.py | 32 ++++++++++++++++++-------------- scripts/qapi-visit.py | 7 +++++-- tests/test-qmp-input-visitor.c | 2 +- tests/test-qmp-output-visitor.c | 2 +- 4 files changed, 25 insertions(+), 18 deletions(-)+''',
scripts/qapi-types.py def generate_union(expr, meta):
Quoted 10 lines
scripts/qapi-types.py +4
-5+ name=name) + if base: + base_fields = find_struct(base)['data'] + ret += generate_struct_fields(base_fields) - if base: - assert discriminator - base_fields = find_struct(base)['data'].copy() - del base_fields[discriminator] - ret += generate_struct_fields(base_fields)
I also like the fact that you no longer modify the base_fields array (because you are no longer floating a single renamed element out-of-order compared to the rest of the base), and therefore avoid the need for a .copy().
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
Quoted 21 lines · Eric Blake
Quoted 5 lines · Markus Armbruster
The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.
Quoted 8 lines
Change to put all base members at the beginning, unadulterated. Not only is this easier to understand, it also permits casting the flat union to its base, if that should become useful.
We now generate:
struct UserDefFlatUnion {It might be worth tweaking the generator to output a C comment either here (at the start of the larger struct)...
Quoted 2 lines
char *string; EnumOne enum1;
Quoted 7 lines
...or here (at the end of the base struct) mentioning that UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to make it easier when reading the generated code to trace back to that "inheritance" relationship. Right now, there is nothing in the generated UserDefFlatUnion that points you back to the qapi relationship of a base class. But it's not a show-stopper if you don't like my suggestion.
I do like it. Perhaps something like
struct UserDefFlatUnion
{
/* Members inherited from UserDefUnionBase: */
char *string;
EnumOne enum1;
/* Own members: */
...
};Quoted 9 lines
Quoted 8 lines
/* union tag is EnumOne enum1 */ union { void *data; UserDefA *value1; UserDefB *value2; UserDefB *value3; }; };
Quoted 6 lines
Only impact in files generated for qemu was to struct BlockdevOptions, in the files qapi-types.[ch], and I agree that it is an improvement. Oddly enough, it seems none of the C code cared about the field being renamed from 'kind' to 'driver' (I guess that's because a lot of our use of the blockdev code is not directly through the generated C structs, but through QObject dictionary queries).
It surprised me, too.
Quoted 21 lines
Quoted 10 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-types.py | 32 ++++++++++++++++++-------------- scripts/qapi-visit.py | 7 +++++-- tests/test-qmp-input-visitor.c | 2 +- tests/test-qmp-output-visitor.c | 2 +- 4 files changed, 25 insertions(+), 18 deletions(-)+''',
scripts/qapi-types.py def generate_union(expr, meta):
Quoted 10 lines
scripts/qapi-types.py +4
-5+ name=name) + if base: + base_fields = find_struct(base)['data'] + ret += generate_struct_fields(base_fields) - if base: - assert discriminator - base_fields = find_struct(base)['data'].copy() - del base_fields[discriminator] - ret += generate_struct_fields(base_fields)
Quoted 6 lines
I also like the fact that you no longer modify the
base_fieldsarray (because you are no longer floating a single renamed element out-of-order compared to the rest of the base), and therefore avoid the need for a .copy().Reviewed-by Eric Blake <[email protected]>
Thanks!
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
Quoted 16 lines · Markus Armbruster
Quoted 6 lines
It might be worth tweaking the generator to output a C comment either here (at the start of the larger struct)...
Quoted 2 lines
char *string; EnumOne enum1;Quoted 7 lines
...or here (at the end of the base struct) mentioning that UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to make it easier when reading the generated code to trace back to that "inheritance" relationship. Right now, there is nothing in the generated UserDefFlatUnion that points you back to the qapi relationship of a base class. But it's not a show-stopper if you don't like my suggestion.
I do like it. Perhaps something like
Quoted 8 lines
struct UserDefFlatUnion
{
/* Members inherited from UserDefUnionBase: */
char *string;
EnumOne enum1;
/* Own members: */
...
};What about nested types? Whatever we pick has to look good when indirect bases are involved. Suppose we have Base -> Mid -> Derived as a struct inheritance. If we generate a comment at both start and end of visiting a base class (and visit base classes recursively), we'd have something like:
struct Derived
{
/* Members inherited from Mid */
/* Members inherited from Base */
int one;
/* End members inherited from Base */
int two;
/* End members inherited from Mid */
/* own members */
int three;
...
};Don't know if that helps you think about it more, but now I'm just painting a bikeshed that just adds the tail comment. Here's the resulting RFC:
http://thread.gmane.org/gmane.comp.emulators.qemu/353204
and the resulting generated structs:
struct BlockdevOptionsQcow2 {
- BlockdevOptionsGenericCOWFormat *base;
+ BlockdevRef *file;
+ /* End fields inherited from BlockdevOptionsGenericFormat. */
+ bool has_backing;
+ BlockdevRef *backing;
+ /* End fields inherited from BlockdevOptionsGenericCOWFormat. */
bool has_lazy_refcounts;
bool lazy_refcounts;...
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
Quoted 34 lines · Eric Blake
Quoted 16 lines · Markus Armbruster
Quoted 6 lines
It might be worth tweaking the generator to output a C comment either here (at the start of the larger struct)...
Quoted 2 lines
char *string; EnumOne enum1;Quoted 7 lines
...or here (at the end of the base struct) mentioning that UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to make it easier when reading the generated code to trace back to that "inheritance" relationship. Right now, there is nothing in the generated UserDefFlatUnion that points you back to the qapi relationship of a base class. But it's not a show-stopper if you don't like my suggestion.
I do like it. Perhaps something like
Quoted 8 lines
struct UserDefFlatUnion { /* Members inherited from UserDefUnionBase: */ char *string; EnumOne enum1; /* Own members: */ ... };What about nested types? Whatever we pick has to look good when indirect bases are involved. Suppose we have Base -> Mid -> Derived as a struct inheritance. If we generate a comment at both start and end of visiting a base class (and visit base classes recursively), we'd have something like:
struct Derived
Quoted 11 lines
{
/* Members inherited from Mid */
/* Members inherited from Base */
int one;
/* End members inherited from Base */
int two;
/* End members inherited from Mid */
/* own members */
int three;
...
};Documenting the direct ancestor in the code is more important than the others, because you can always peruse the direct ancestor to find the next one.
Quoted 9 lines
Don't know if that helps you think about it more, but now I'm just painting a bikeshed that just adds the tail comment. Here's the resulting RFC:
http://thread.gmane.org/gmane.comp.emulators.qemu/353204
and the resulting generated structs:
struct BlockdevOptionsQcow2 {
Review
Quoted 9 lines
(quoted diff) +5
-1- BlockdevOptionsGenericCOWFormat *base; + BlockdevRef *file; + /* End fields inherited from BlockdevOptionsGenericFormat. */ + bool has_backing; + BlockdevRef *backing; + /* End fields inherited from BlockdevOptionsGenericCOWFormat. */ bool has_lazy_refcounts; bool lazy_refcounts;
This and other followup patches are encouraging. I'll gladly review them when I can spare the cycles. Right now, I need to focus them on introspection.
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
In reply to Markus ArmbrusterQuoted 3 lines · Markus Armbruster
The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.
The renaming to 'kind' was a bug waiting to happen. Consider this example, which is broken before your series:
(quoted diff) +0 -0
w/tests/qapi-schema/qapi-schema-test.json
index c7eaa86..12c09e3 100644
tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -37,7 +37,7 @@ 'data': { 'string1': 'str', 'string2': 'str' } } { 'struct': 'UserDefUnionBase', - 'data': { 'string': 'str', 'enum1': 'EnumOne' } } + 'data': { 'kind': 'str', 'enum1': 'EnumOne' } } { 'union': 'UserDefFlatUnion', 'base': 'UserDefUnionBase',
leading to this compilation error during 'make check-unit':
In file included from tests/test-qmp-output-visitor.c:17:0:tests/test-qapi-types.h:617:11: error: duplicate member ‘kind’
char *kind;
^
tests/test-qapi-types.h:631:11: error: duplicate member ‘kind’
char *kind;
^Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
Quoted 28 lines · Eric Blake
Quoted 3 lines · Markus Armbruster
The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.
The renaming to 'kind' was a bug waiting to happen. Consider this example, which is broken before your series:
(quoted diff) +0
-0
w/tests/qapi-schema/qapi-schema-test.json
index c7eaa86..12c09e3 100644tests/qapi-schema/qapi-schema-test.json +1
-1@@ -37,7 +37,7 @@ 'data': { 'string1': 'str', 'string2': 'str' } } { 'struct': 'UserDefUnionBase', - 'data': { 'string': 'str', 'enum1': 'EnumOne' } } + 'data': { 'kind': 'str', 'enum1': 'EnumOne' } } { 'union': 'UserDefFlatUnion', 'base': 'UserDefUnionBase',leading to this compilation error during 'make check-unit':
In file included from
tests/test-qmp-output-visitor.c:17:0:tests/test-qapi-types.h:617:11: error: duplicate member ‘kind’
Quoted 8 lines
char *kind; ^ tests/test-qapi-types.h:631:11: error: duplicate member ‘kind’ char *kind; ^Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)
Cool! I'll work (a variation of) this test case into my series.
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
Quoted 8 lines · Markus Armbruster
Quoted 5 lines · Eric Blake
Quoted 3 lines · Markus Armbruster
The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.
Quoted 4 lines
Quoted 2 lines
Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)
Cool! I'll work (a variation of) this test case into my series.
Another name collision bug: our code generates flat unions as:
struct BlockdevOptions {
BlockdevDriver driver;
...
/* End fields inherited from BlockdevOptionsBase. */
/* union tag is BlockdevDriver driver */
union {
void *data;
BlockdevOptionsArchipelago *archipelago;...
which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '_data'.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
Quoted 24 lines · Eric Blake
Quoted 8 lines · Markus Armbruster
Quoted 5 lines · Eric Blake
Quoted 3 lines · Markus Armbruster
The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.
Quoted 4 lines
Quoted 2 lines
Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)
Cool! I'll work (a variation of) this test case into my series.
Another name collision bug: our code generates flat unions as:
struct BlockdevOptions { BlockdevDriver driver; ... /* End fields inherited from BlockdevOptionsBase. */ /* union tag is BlockdevDriver driver */ union { void *data; BlockdevOptionsArchipelago *archipelago;
Quoted 5 lines
...
which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '
_data'.
I wonder whether member data is actually used. I'll find out.
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
Quoted 12 lines · Markus Armbruster
Quoted 12 lines
Another name collision bug: our code generates flat unions as:
struct BlockdevOptions { BlockdevDriver driver; ... /* End fields inherited from BlockdevOptionsBase. */ /* union tag is BlockdevDriver driver */ union { void *data; BlockdevOptionsArchipelago *archipelago;...
Quoted 5 lines
Quoted 3 lines
which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '
_data'.I wonder whether member data is actually used. I'll find out.
The dealloc visitor uses 'data' being non-null as a flag on whether to deallocate the union even if the tag was invalid for some reason; or more importantly, if parsing consumed the tag but then detected an error while parsing the union, leaving the union branch partially allocated. To avoid a leak, we have to deallocate the branch.
But if the tag was invalid, then why did we ever allocate the union in the first place, and how do we prove we are calling the correct free-ing function? And if the tag is valid, why can't we just guarantee that the union is 0-initialized and that deleting the branch will work through the correct branch type instead of worrying about 'data'?
We still need a dummy member if it is valid to do { 'union':'Foo', 'data':{} } since C doesn't like empty unions, but an empty union seems like something we may want to reject, at which point you are probably right that deleting the data member altogether should work and still let us recover from bad partial parses without a leak.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
Quoted 26 lines · Eric Blake
Quoted 12 lines · Markus Armbruster
Quoted 12 lines
Another name collision bug: our code generates flat unions as:
struct BlockdevOptions { BlockdevDriver driver; ... /* End fields inherited from BlockdevOptionsBase. */ /* union tag is BlockdevDriver driver */ union { void *data; BlockdevOptionsArchipelago *archipelago;...
Quoted 5 lines
Quoted 3 lines
which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '
_data'.I wonder whether member data is actually used. I'll find out.
The dealloc visitor uses 'data' being non-null as a flag on whether to deallocate the union even if the tag was invalid for some reason; or more importantly, if parsing consumed the tag but then detected an error while parsing the union, leaving the union branch partially allocated. To avoid a leak, we have to deallocate the branch.
Quoted 5 lines
But if the tag was invalid, then why did we ever allocate the union in the first place, and how do we prove we are calling the correct free-ing function? And if the tag is valid, why can't we just guarantee that the union is 0-initialized and that deleting the branch will work through the correct branch type instead of worrying about 'data'?
Good questions. Someone will have to review and fix this code. Let's add a FIXME so we don't forget. Care to propose one?
Quoted 5 lines
We still need a dummy member if it is valid to do { 'union':'Foo', 'data':{} } since C doesn't like empty unions, but an empty union seems like something we may want to reject, at which point you are probably right that deleting the data member altogether should work and still let us recover from bad partial parses without a leak.
Either we reject empty unions, or we detect them and add a dummy, similar to what we do for structs since commit 83ecb22. I figure simply rejecting them is easier.
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
Quoted 12 lines · Markus Armbruster
Quoted 12 lines
Quoted 5 lines
Quoted 3 lines
which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '
_data'.I wonder whether member data is actually used. I'll find out.
The dealloc visitor uses 'data' being non-null as a flag on whether to deallocate the union even if the tag was invalid for some reason; or more importantly, if parsing consumed the tag but then detected an error while parsing the union, leaving the union branch partially allocated. To avoid a leak, we have to deallocate the branch.
Quoted 8 lines
Quoted 5 lines
But if the tag was invalid, then why did we ever allocate the union in the first place, and how do we prove we are calling the correct free-ing function? And if the tag is valid, why can't we just guarantee that the union is 0-initialized and that deleting the branch will work through the correct branch type instead of worrying about 'data'?
Good questions. Someone will have to review and fix this code. Let's add a FIXME so we don't forget. Care to propose one?
Sure; see 12.6/47 (since that is close to several other patches adding FIXME comments).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
In reply to Eric BlakeQuoted 24 lines · Eric Blake
Quoted 8 lines · Markus Armbruster
Quoted 5 lines · Eric Blake
Quoted 3 lines · Markus Armbruster
The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.
Quoted 4 lines
Quoted 2 lines
Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)
Cool! I'll work (a variation of) this test case into my series.
Another name collision bug: our code generates flat unions as:
struct BlockdevOptions { BlockdevDriver driver; ... /* End fields inherited from BlockdevOptionsBase. */ /* union tag is BlockdevDriver driver */ union { void *data; BlockdevOptionsArchipelago *archipelago;
Quoted 5 lines
...
which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '
_data'.
Works, because schema names should not begin with '_', except for downstream extensions, which begin with '__RFQDN_'. We don't enforce that, however. I'll include the appended patch.
Subject: [PATCH] qapi: Document flaws in checking of names
We don't actually enforce our "other than downstream extensions [...], all names should begin with a letter" rule. Add a FIXME.
We should reject names that differ only in '_' vs. '.' vs. '-', because they're liable to clash in generated C. Add a FIXME.
--- scripts/qapi.py | 4 ++++ 1 file changed, 4 insertions(+)
scripts/qapi.py +4 -0
@@ -341,6 +341,8 @@ def discriminator_find_enum_define(expr): return find_enum(discriminator_type) +# FIXME should enforce "other than downstream extensions [...], all +# names should begin with a letter". valid_name = re.compile('^[a-zA-Z_][a-zA-Z0-9_.-]*$') def check_name(expr_info, source, name, allow_optional = False, enum_member = False):
@@ -367,6 +369,8 @@ def check_name(expr_info, source, name, allow_optional = False, def add_name(name, info, meta, implicit = False): global all_names check_name(info, "'%s'" % meta, name) + # FIXME should reject names that differ only in '_' vs. '.' + # vs. '-', because they're liable to clash in generated C. if name in all_names: raise QAPIExprError(info, "%s '%s' is already defined"
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions
In reply to Markus ArmbrusterQuoted 36 lines · Markus Armbruster
Quoted 34 lines · Eric Blake
Quoted 3 lines · Markus Armbruster
The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.
The renaming to 'kind' was a bug waiting to happen. Consider this example, which is broken before your series:
(quoted diff) +0
-0
w/tests/qapi-schema/qapi-schema-test.json
index c7eaa86..12c09e3 100644tests/qapi-schema/qapi-schema-test.json +1
-1@@ -37,7 +37,7 @@ 'data': { 'string1': 'str', 'string2': 'str' } } { 'struct': 'UserDefUnionBase', - 'data': { 'string': 'str', 'enum1': 'EnumOne' } } + 'data': { 'kind': 'str', 'enum1': 'EnumOne' } } { 'union': 'UserDefFlatUnion', 'base': 'UserDefUnionBase',leading to this compilation error during 'make check-unit':
In file included from
tests/test-qmp-output-visitor.c:17:0:tests/test-qapi-types.h:617:11: error: duplicate member ‘kind’char *kind; ^ tests/test-qapi-types.h:631:11: error: duplicate member ‘kind’ char *kind; ^
Quoted 4 lines
Quoted 2 lines
Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)
Cool! I'll work (a variation of) this test case into my series.
I split the bug fix off this patch. I put the test in the commit message, because I feel it has little value as regression test going forward.
RFC PATCH v2 8/47 qapi-visit: Fix generated code when schema has forward refs
In reply to Markus ArmbrusterThe visit_type_implicit_FOO() are generated on demand, right before their first use. Used by visit_type_STRUCT_fields() when STRUCT has base FOO, and by visit_type_UNION() when flat UNION has member a FOO.
If the schema defines FOO after its first use as struct base or flat union member, visit_type_implicit_FOO() calls visit_type_implicit_FOO() before its definition, which doesn't compile.
Rearrange qapi-schema-test.json to demonstrate the bug.
Fix by generating the necessary forward declaration.
--- scripts/qapi-visit.py | 15 ++++++++++++++- tests/qapi-schema/qapi-schema-test.json | 30 +++++++++++++++++------------- tests/qapi-schema/qapi-schema-test.out | 10 +++++----- 3 files changed, 36 insertions(+), 19 deletions(-)
scripts/qapi-visit.py +14 -1
@@ -17,13 +17,23 @@ from qapi import * import re implicit_structs = [] +struct_fields_seen = set() def generate_visit_implicit_struct(type): global implicit_structs if type in implicit_structs: return '' implicit_structs.append(type) - return mcgen(''' + ret = '' + if type not in struct_fields_seen: + # Need a forward declaration + ret += mcgen(''' + +static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp); +''', + c_type=type_name(type)) + + ret += mcgen(''' static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp) {
@@ -38,8 +48,11 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * } ''', c_type=type_name(type)) + return ret def generate_visit_struct_fields(name, members, base = None): + struct_fields_seen.add(name) + ret = '' if base:
tests/qapi-schema/qapi-schema-test.json +17 -13
@@ -7,13 +7,13 @@ 'data': { 'enum1': 'EnumOne', '*enum2': 'EnumOne', 'enum3': 'EnumOne', '*enum4': 'EnumOne' } } # for testing nested structs -{ 'struct': 'UserDefZero', - 'data': { 'integer': 'int' } } - { 'struct': 'UserDefOne', - 'base': 'UserDefZero', + 'base': 'UserDefZero', # intentional forward reference 'data': { 'string': 'str', '*enum1': 'EnumOne' } } +{ 'struct': 'UserDefZero', + 'data': { 'integer': 'int' } } + { 'struct': 'UserDefTwoDictDict', 'data': { 'userdef': 'UserDefOne', 'string': 'str' } }
@@ -33,29 +33,33 @@ { 'struct': 'UserDefB', 'data': { 'integer': 'int' } } -{ 'struct': 'UserDefC', - 'data': { 'string1': 'str', 'string2': 'str' } } - -{ 'struct': 'UserDefUnionBase', - 'data': { 'string': 'str', 'enum1': 'EnumOne' } } - { 'union': 'UserDefFlatUnion', - 'base': 'UserDefUnionBase', + 'base': 'UserDefUnionBase', # intentional forward reference 'discriminator': 'enum1', - 'data': { 'value1' : 'UserDefA', 'value2' : 'UserDefB', 'value3' : 'UserDefB' } } + 'data': { 'value1' : 'UserDefA', + 'value2' : 'UserDefB', + 'value3' : 'UserDefB' } } # FIXME generated struct UserDefFlatUnion has members for direct base # UserDefOne, but lacks members for indirect base UserDefZero +{ 'struct': 'UserDefUnionBase', + 'data': { 'string': 'str', 'enum1': 'EnumOne' } } + # this variant of UserDefFlatUnion defaults to a union that uses fields with # allocated types to test corner cases in the cleanup/dealloc visitor { 'union': 'UserDefFlatUnion2', 'base': 'UserDefUnionBase', 'discriminator': 'enum1', - 'data': { 'value1' : 'UserDefC', 'value2' : 'UserDefB', 'value3' : 'UserDefA' } } + 'data': { 'value1' : 'UserDefC', # intentional forward reference + 'value2' : 'UserDefB', + 'value3' : 'UserDefA' } } { 'alternate': 'UserDefAlternate', 'data': { 'uda': 'UserDefA', 's': 'str', 'i': 'int' } } +{ 'struct': 'UserDefC', + 'data': { 'string1': 'str', 'string2': 'str' } } + # for testing native lists { 'union': 'UserDefNativeListUnion', 'data': { 'integer': ['int'],
tests/qapi-schema/qapi-schema-test.out +5 -5
@@ -1,17 +1,17 @@ [OrderedDict([('enum', 'EnumOne'), ('data', ['value1', 'value2', 'value3'])]), OrderedDict([('struct', 'NestedEnumsOne'), ('data', OrderedDict([('enum1', 'EnumOne'), ('*enum2', 'EnumOne'), ('enum3', 'EnumOne'), ('*enum4', 'EnumOne')]))]), - OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]), OrderedDict([('struct', 'UserDefOne'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('*enum1', 'EnumOne')]))]), + OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]), OrderedDict([('struct', 'UserDefTwoDictDict'), ('data', OrderedDict([('userdef', 'UserDefOne'), ('string', 'str')]))]), OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]), OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]), OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]), OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]), - OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]), - OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]), OrderedDict([('union', 'UserDefFlatUnion'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefA'), ('value2', 'UserDefB'), ('value3', 'UserDefB')]))]), + OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]), OrderedDict([('union', 'UserDefFlatUnion2'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefC'), ('value2', 'UserDefB'), ('value3', 'UserDefA')]))]), OrderedDict([('alternate', 'UserDefAlternate'), ('data', OrderedDict([('uda', 'UserDefA'), ('s', 'str'), ('i', 'int')]))]), + OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]), OrderedDict([('union', 'UserDefNativeListUnion'), ('data', OrderedDict([('integer', ['int']), ('s8', ['int8']), ('s16', ['int16']), ('s32', ['int32']), ('s64', ['int64']), ('u8', ['uint8']), ('u16', ['uint16']), ('u32', ['uint32']), ('u64', ['uint64']), ('number', ['number']), ('boolean', ['bool']), ('string', ['str']), ('sizes', ['size'])]))]), OrderedDict([('command', 'user_def_cmd'), ('data', OrderedDict())]), OrderedDict([('command', 'user_def_cmd1'), ('data', OrderedDict([('ud1a', 'UserDefOne')]))]),
@@ -39,15 +39,15 @@ {'enum_name': '__org.qemu_x-Union1Kind', 'enum_values': None}, {'enum_name': '__org.qemu_x-AltKind', 'enum_values': None}] [OrderedDict([('struct', 'NestedEnumsOne'), ('data', OrderedDict([('enum1', 'EnumOne'), ('*enum2', 'EnumOne'), ('enum3', 'EnumOne'), ('*enum4', 'EnumOne')]))]), - OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]), OrderedDict([('struct', 'UserDefOne'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('*enum1', 'EnumOne')]))]), + OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]), OrderedDict([('struct', 'UserDefTwoDictDict'), ('data', OrderedDict([('userdef', 'UserDefOne'), ('string', 'str')]))]), OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]), OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]), OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]), OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]), - OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]), OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]), + OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]), OrderedDict([('struct', 'UserDefOptions'), ('data', OrderedDict([('*i64', ['int']), ('*u64', ['uint64']), ('*u16', ['uint16']), ('*i64x', 'int'), ('*u64x', 'uint64')]))]), OrderedDict([('struct', 'EventStructOne'), ('data', OrderedDict([('struct1', 'UserDefOne'), ('string', 'str'), ('*enum2', 'EnumOne')]))]), OrderedDict([('struct', '__org.qemu_x-Base'), ('data', OrderedDict([('__org.qemu_x-member1', '__org.qemu_x-Enum')]))]),
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 08/47] qapi-visit: Fix generated code when schema has forward refs
Quoted 8 lines · Markus Armbruster
The
visit_type_implicit_FOO()are generated on demand, right before their first use. Used byvisit_type_STRUCT_fields()when STRUCT has base FOO, and byvisit_type_UNION()when flat UNION has member a FOO.If the schema defines FOO after its first use as struct base or flat union member,
visit_type_implicit_FOO()callsvisit_type_implicit_FOO()before its definition, which doesn't compile.
None of our public qapi .json files currently do this, so no difference to the generated code used in qemu proper.
Quoted 2 lines
Rearrange qapi-schema-test.json to demonstrate the bug.
Indeed, without testsuite exposure, nothing was relying on this fix.
Quoted 24 lines
Fix by generating the necessary forward declaration.
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-visit.py | 15 ++++++++++++++- tests/qapi-schema/qapi-schema-test.json | 30 +++++++++++++++++------------- tests/qapi-schema/qapi-schema-test.out | 10 +++++----- 3 files changed, 36 insertions(+), 19 deletions(-)+++
b/scripts/qapi-visit.pyscripts/qapi-visit.py +1
-1@@ -17,13 +17,23 @@ from qapi import * import re implicit_structs = [] +struct_fields_seen = set() def generate_visit_implicit_struct(type): global implicit_structs if type in implicit_structs: return '' implicit_structs.append(type) - return mcgen('''
...
Quoted 1 line
+ ret += mcgen('''
Oddly enough, the ''' is at the same indentation,...
scripts/qapi-visit.py static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * · line 51
Quoted 8 lines
static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp) {
scripts/qapi-visit.py +1
-0@@ -38,8 +48,11 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * } ''', c_type=type_name(type)) + return ret
so nothing needed to be reindented here :)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 08/47] qapi-visit: Fix generated code when schema has forward refs
Quoted 5 lines · Eric Blake
Quoted 4 lines · Markus Armbruster
The
visit_type_implicit_FOO()are generated on demand, right before their first use. Used byvisit_type_STRUCT_fields()when STRUCT has base FOO, and byvisit_type_UNION()when flat UNION has member a FOO.
Quoted 7 lines
Quoted 4 lines
If the schema defines FOO after its first use as struct base or flat union member,
visit_type_implicit_FOO()callsvisit_type_implicit_FOO()before its definition, which doesn't compile.None of our public qapi .json files currently do this, so no difference to the generated code used in qemu proper.
Yes, we've intentionally avoided forward references.
Nevertheless, the generators either have to catch and reject them, or make them work. Making them work is easy, so I did that.
Quoted 44 lines
Quoted 1 line
Rearrange qapi-schema-test.json to demonstrate the bug.
Indeed, without testsuite exposure, nothing was relying on this fix.
Quoted 24 lines
Fix by generating the necessary forward declaration.
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-visit.py | 15 ++++++++++++++- tests/qapi-schema/qapi-schema-test.json | 30 +++++++++++++++++------------- tests/qapi-schema/qapi-schema-test.out | 10 +++++----- 3 files changed, 36 insertions(+), 19 deletions(-)+++
b/scripts/qapi-visit.pyscripts/qapi-visit.py +1
-1@@ -17,13 +17,23 @@ from qapi import * import re implicit_structs = [] +struct_fields_seen = set() def generate_visit_implicit_struct(type): global implicit_structs if type in implicit_structs: return '' implicit_structs.append(type) - return mcgen('''...
Quoted 1 line
+ ret += mcgen('''
Oddly enough, the ''' is at the same indentation,...
scripts/qapi-visit.py static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * · line 51
Quoted 8 lines
static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp) {
scripts/qapi-visit.py +1
-0@@ -38,8 +48,11 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * } ''', c_type=type_name(type)) + return retso nothing needed to be reindented here :)
Quoted 1 line
Reviewed-by Eric Blake <[email protected]>
Thanks!
RFC PATCH v2 9/47 qapi-visit: Replace list implicit_structs by set
In reply to Markus ArmbrusterUse set because that's what it is. While there, rename to implicit_structs_seen.
--- scripts/qapi-visit.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
scripts/qapi-visit.py +3 -4
@@ -16,14 +16,13 @@ from ordereddict import OrderedDict from qapi import * import re -implicit_structs = [] +implicit_structs_seen = set() struct_fields_seen = set() def generate_visit_implicit_struct(type): - global implicit_structs - if type in implicit_structs: + if type in implicit_structs_seen: return '' - implicit_structs.append(type) + implicit_structs_seen.add(type) ret = '' if type not in struct_fields_seen: # Need a forward declaration
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 09/47] qapi-visit: Replace list implicit_structs by set
Quoted 8 lines · Markus Armbruster
Use set because that's what it is. While there, rename to
implicit_structs_seen.Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-visit.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
(I have no idea if that makes python more efficient, but it is cleaner)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 10/47 qapi-visit: Fix two name arguments passed to visitors
In reply to Markus ArmbrusterThe generated code passes mangled schema names to visit_type_enum() and union's visit_start_struct(). Fix it to pass the names unadulterated, like we do everywhere else.
Only qapi-schema-test.json actually has names where this makes a
difference: enum __org.qemu_x-Enum, flat union __org.qemu_x-Union2, simple union __org.qemu_x-Union1 and its implicit enum __org.qemu_x-Union1Kind.
--- scripts/qapi-visit.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
scripts/qapi-visit.py +6 -6
@@ -185,12 +185,12 @@ out: def generate_visit_enum(name): return mcgen(''' -void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp) +void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error **errp) { - visit_type_enum(m, (int *)obj, %(name)s_lookup, "%(name)s", name, errp); + visit_type_enum(m, (int *)obj, %(c_name)s_lookup, "%(name)s", name, errp); } ''', - name=c_name(name)) + c_name=c_name(name), name=name) def generate_visit_alternate(name, members): ret = mcgen('''
@@ -278,17 +278,17 @@ def generate_visit_union(expr): ret += mcgen(''' -void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp) +void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp) { Error *err = NULL; - visit_start_struct(m, (void **)obj, "%(name)s", name, sizeof(%(name)s), &err); + visit_start_struct(m, (void **)obj, "%(name)s", name, sizeof(%(c_name)s), &err); if (err) { goto out; } if (*obj) { ''', - name=c_name(name)) + c_name=c_name(name), name=name) if base: ret += mcgen('''
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 10/47] qapi-visit: Fix two name arguments passed to visitors
Quoted 8 lines · Markus Armbruster
The generated code passes mangled schema names to
visit_type_enum()and union'svisit_start_struct(). Fix it to pass the names unadulterated, like we do everywhere else.Only qapi-schema-test.json actually has names where this makes a
difference: enum __org.qemu_x-Enum, flat union __org.qemu_x-Union2, simple union __org.qemu_x-Union1 and its implicit enum __org.qemu_x-Union1Kind.
I'm glad I added that test - it covers things that will affect downstream extensions, but which don't affect mainline.
Quoted 6 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-visit.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 11/47 tests/qapi-schema: Document alternate's enum lacks visit function
In reply to Markus ArmbrusterWe generate a declaration, but no definition.
The QMP schema has two: Qcow2OverlapChecks and BlockdevRef. Neither visit_type_Qcow2OverlapChecksKind() nor visit_type_BlockdevRefKind() is actually used.
--- tests/qapi-schema/qapi-schema-test.json | 1 + 1 file changed, 1 insertion(+)
tests/qapi-schema/qapi-schema-test.json +1 -0
@@ -56,6 +56,7 @@ { 'alternate': 'UserDefAlternate', 'data': { 'uda': 'UserDefA', 's': 'str', 'i': 'int' } } +# FIXME only a declaration of visit_type_UserDefAlternateKind() generated { 'struct': 'UserDefC', 'data': { 'string1': 'str', 'string2': 'str' } }
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 11/47] tests/qapi-schema: Document alternate's enum lacks visit function
Quoted 10 lines · Markus Armbruster
We generate a declaration, but no definition.
The QMP schema has two: Qcow2OverlapChecks and BlockdevRef. Neither
visit_type_Qcow2OverlapChecksKind()norvisit_type_BlockdevRefKind()is actually used.Signed-off-by Markus Armbruster <[email protected]>--- tests/qapi-schema/qapi-schema-test.json | 1 + 1 file changed, 1 insertion(+)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 12/47 tests/qapi-schema: Document events with with base don't work
In reply to Markus ArmbrusterWhen event FOO's 'data' is a struct with a base, we consider only the struct's direct members, and ignore its base. The generated qapi_event_send_foo() doesn't take arguments for base members.
No such events currently exist in the QMP schema.
--- tests/qapi-schema/qapi-schema-test.json | 3 +++ 1 file changed, 3 insertions(+)
tests/qapi-schema/qapi-schema-test.json +3 -0
@@ -128,6 +128,9 @@ { 'alternate': '__org.qemu_x-Alt', 'data': { '__org.qemu_x-branch': 'str', 'b': '__org.qemu_x-Base' } } { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' } +# FIXME generated qapi_event_send___org_qemu_x_event() has only a +# parameter for data's member __org_qemu_x_member2, none for its base +# __org.qemu_x-Base's member __org_qemu_x_member1 { 'command': '__org.qemu_x-command', 'data': { 'a': ['__org.qemu_x-Enum'], 'b': ['__org.qemu_x-Struct'], 'c': '__org.qemu_x-Union2', 'd': '__org.qemu_x-Alt' },
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 12/47] tests/qapi-schema: Document events with with base don't work
Quoted 10 lines · Markus Armbruster
When event FOO's 'data' is a struct with a base, we consider only the struct's direct members, and ignore its base. The generated
qapi_event_send_foo()doesn't take arguments for base members.No such events currently exist in the QMP schema.
Signed-off-by Markus Armbruster <[email protected]>--- tests/qapi-schema/qapi-schema-test.json | 3 +++ 1 file changed, 3 insertions(+)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH 1/1 qapi: Document that input visitor semantics are prone to leaks
In reply to Markus ArmbrusterMost functions that can return a pointer or set an Error ** value are decent enough to guarantee a NULL return when reporting an error. Not so with our generated qapi visitor functions. If the caller is not careful to clean up partially-allocated objects on error, then the caller suffers a memory leak.
Properly fixing it is probably complex enough to save for a later day, so merely document it for now.
---
As mentioned elsewhere in the thread (comments on 29/47
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg06107.html)
it would be worth documenting a FIXME for a future series. I'm submitting this as 12.5/47 due to its relation to other similar shortcoming doc patches; and assuming Markus can rebase the rest of the series on top if he wants to fold it into his v3 posting at this spot. Otherwise, I can wait for his v3 and rebase it to be part of my (growing) followup series.
scripts/qapi-visit.py | 4 ++++ tests/test-qmp-input-visitor.c | 2 ++ 2 files changed, 6 insertions(+)
scripts/qapi-visit.py +4 -0
@@ -115,6 +115,10 @@ out: def generate_visit_struct_body(name): + # FIXME: if *obj is NULL on entry, and visit_start_struct() assigns to + # *obj, but then visit_type_FOO_fields() fails, we should clean up *obj + # rather than leaving it non-NULL. As currently written, the caller must + # call qapi_free_FOO() to avoid a memory leak of the partial FOO. ret = mcgen(''' Error *err = NULL;
tests/test-qmp-input-visitor.c +2 -0
@@ -636,6 +636,8 @@ static void test_visitor_in_errors(TestInputVisitorData *data,visit_type_TestStruct (v, &p,NULL , &err);g_assert (err); +/* FIXME - a failed parse should not leave a partially-allocated p +* for us to clean up; this could cause callers to leak memory. */ g_assert (p->string ==NULL );error_free (err);
Signature · 1 line
2.4.3
Re: [Qemu-devel] [RFC PATCH 12.5/47] qapi: Document that input visitor semantics are prone to leaks
Quoted 10 lines · Eric Blake
Most functions that can return a pointer or set an Error ** value are decent enough to guarantee a NULL return when reporting an error. Not so with our generated qapi visitor functions. If the caller is not careful to clean up partially-allocated objects on error, then the caller suffers a memory leak.
Properly fixing it is probably complex enough to save for a later day, so merely document it for now.
Signed-off-by Eric Blake <[email protected]>
Picked into my series. Thanks!
RFC PATCH 1/1 qapi: Document shortcoming with union 'data' branch
In reply to Markus ArmbrusterAdd a FIXME to remind us to fully audit whether removing the 'void *data' branch of each qapi union type can be done safely.
---
Another potential doc FIXME patch, this time based on 7/47
(https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg06101.html)
scripts/qapi-types.py | 8 ++++++++ 1 file changed, 8 insertions(+)
scripts/qapi-types.py +8 -0
@@ -217,6 +217,14 @@ struct %(name)s ''', discriminator_type_name=c_name(discriminator_type_name)) + # FIXME: What purpose does data serve, besides preventing a union that + # has a branch named 'data'? We use it in qapi-visit.py to decide + # whether to bypass the switch statement if visiting the discriminator + # failed; but since we 0-initialize structs, and cannot tell what + # branch of the union is in use if the discriminator is invalid, there + # should not be any data leaks even without a data pointer. Or, if + # 'data' is merely added to guarantee we don't have an empty union, + # shouldn't we enforce that at .json parse time? ret += mcgen(''' union { void *data;
Signature · 1 line
2.4.3
Re: [Qemu-devel] [RFC PATCH 12.6/47] qapi: Document shortcoming with union 'data' branch
Quoted 4 lines · Eric Blake
Add a FIXME to remind us to fully audit whether removing the 'void *data' branch of each qapi union type can be done safely.
Signed-off-by Eric Blake <[email protected]>
Picked into my series. Thanks!
RFC PATCH v2 13/47 tests/qapi-schema: Restore test case for flat union base bug
In reply to Markus ArmbrusterTest case added in commit 2fc0043, and messed up in commit 5223070.
--- tests/qapi-schema/qapi-schema-test.json | 5 +++-- tests/qapi-schema/qapi-schema-test.out | 8 ++++---- 2 files changed, 7 insertions(+), 6 deletions(-)
tests/qapi-schema/qapi-schema-test.json +3 -2
@@ -31,7 +31,7 @@ 'data': { 'boolean': 'bool' } } { 'struct': 'UserDefB', - 'data': { 'integer': 'int' } } + 'data': { 'intb': 'int' } } { 'union': 'UserDefFlatUnion', 'base': 'UserDefUnionBase', # intentional forward reference
@@ -40,9 +40,10 @@ 'value2' : 'UserDefB', 'value3' : 'UserDefB' } } # FIXME generated struct UserDefFlatUnion has members for direct base -# UserDefOne, but lacks members for indirect base UserDefZero +# UserDefUnionBase, but lacks members for indirect base UserDefZero { 'struct': 'UserDefUnionBase', + 'base': 'UserDefZero', 'data': { 'string': 'str', 'enum1': 'EnumOne' } } # this variant of UserDefFlatUnion defaults to a union that uses fields with
tests/qapi-schema/qapi-schema-test.out +4 -4
@@ -6,9 +6,9 @@ OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]), OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]), OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]), - OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]), + OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('intb', 'int')]))]), OrderedDict([('union', 'UserDefFlatUnion'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefA'), ('value2', 'UserDefB'), ('value3', 'UserDefB')]))]), - OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]), + OrderedDict([('struct', 'UserDefUnionBase'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]), OrderedDict([('union', 'UserDefFlatUnion2'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefC'), ('value2', 'UserDefB'), ('value3', 'UserDefA')]))]), OrderedDict([('alternate', 'UserDefAlternate'), ('data', OrderedDict([('uda', 'UserDefA'), ('s', 'str'), ('i', 'int')]))]), OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
@@ -45,8 +45,8 @@ OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]), OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]), OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]), - OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]), - OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]), + OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('intb', 'int')]))]), + OrderedDict([('struct', 'UserDefUnionBase'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]), OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]), OrderedDict([('struct', 'UserDefOptions'), ('data', OrderedDict([('*i64', ['int']), ('*u64', ['uint64']), ('*u16', ['uint16']), ('*i64x', 'int'), ('*u64x', 'uint64')]))]), OrderedDict([('struct', 'EventStructOne'), ('data', OrderedDict([('struct1', 'UserDefOne'), ('string', 'str'), ('*enum2', 'EnumOne')]))]),
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 13/47] tests/qapi-schema: Restore test case for flat union base bug
Quoted 8 lines · Markus Armbruster
Test case added in commit 2fc0043, and messed up in commit 5223070.
Signed-off-by Markus Armbruster <[email protected]>--- tests/qapi-schema/qapi-schema-test.json | 5 +++-- tests/qapi-schema/qapi-schema-test.out | 8 ++++---- 2 files changed, 7 insertions(+), 6 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 14/47 qapi-tests: New tests for union, alternate command arguments
In reply to Markus ArmbrusterA command's 'data' must be a struct type, given either as a dictionary, or as struct type name.
Existing test case data-int.json covers simple type 'int'. Add test cases for type names referring to union and alternate types.
The latter is caught (good), but the former is not (bug).
--- tests/Makefile | 3 ++- tests/qapi-schema/args-alternate.err | 1 + tests/qapi-schema/args-alternate.exit | 1 + tests/qapi-schema/args-alternate.json | 4 ++++ tests/qapi-schema/args-alternate.out | 0 tests/qapi-schema/args-union.err | 0 tests/qapi-schema/args-union.exit | 1 + tests/qapi-schema/args-union.json | 4 ++++ tests/qapi-schema/args-union.out | 4 ++++ 9 files changed, 17 insertions(+), 1 deletion(-) create mode 100644 tests/qapi-schema/args-alternate.err create mode 100644 tests/qapi-schema/args-alternate.exit create mode 100644 tests/qapi-schema/args-alternate.json create mode 100644 tests/qapi-schema/args-alternate.out create mode 100644 tests/qapi-schema/args-union.err create mode 100644 tests/qapi-schema/args-union.exit create mode 100644 tests/qapi-schema/args-union.json create mode 100644 tests/qapi-schema/args-union.out
tests/Makefile +2 -1
@@ -225,7 +225,8 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \ type-bypass.json type-bypass-no-gen.json type-bypass-bad-gen.json \ data-array-empty.json data-array-unknown.json data-int.json \ data-unknown.json data-member-unknown.json data-member-array.json \ - data-member-array-bad.json returns-array-bad.json returns-int.json \ + data-member-array-bad.json args-union.json args-alternate.json \ + returns-array-bad.json returns-int.json \ returns-unknown.json returns-alternate.json returns-whitelist.json \ missing-colon.json missing-comma-list.json missing-comma-object.json \ nested-struct-data.json nested-struct-returns.json non-objects.json \
tests/qapi-schema/args-alternate.err new file +1 -0
@@ -0,0 +1 @@ +tests/qapi-schema/args-alternate.json:4: 'data' for command 'oops' cannot use alternate type 'Alt'
tests/qapi-schema/args-alternate.exit new file +1 -0
@@ -0,0 +1 @@ +1
tests/qapi-schema/args-alternate.json new file +4 -0
@@ -0,0 +1,4 @@ +# we do not allow alternate arguments +# TODO should we support this? +{ 'alternate': 'Alt', 'data': { 'case1': 'int', 'case2': 'str' } } +{ 'command': 'oops', 'data': 'Alt' }
tests/qapi-schema/args-alternate.out new file +0 -0
tests/qapi-schema/args-union.err new file +0 -0
tests/qapi-schema/args-union.exit new file +1 -0
@@ -0,0 +1 @@ +0
tests/qapi-schema/args-union.json new file +4 -0
@@ -0,0 +1,4 @@ +# FIXME we should reject union arguments +# TODO should we support this? +{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } } +{ 'command': 'oops', 'data': 'Uni' }
tests/qapi-schema/args-union.out new file +4 -0
@@ -0,0 +1,4 @@ +[OrderedDict([('union', 'Uni'), ('data', OrderedDict([('case1', 'int'), ('case2', 'str')]))]), + OrderedDict([('command', 'oops'), ('data', 'Uni')])] +[{'enum_name': 'UniKind', 'enum_values': None}] +[]
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments
Quoted 5 lines · Markus Armbruster
A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.
Existing test case data-int.json covers simple type 'int'. Add test cases for type names referring to union and alternate types.
We could probably relax things to allow a union (which is always a dictionary on the wire), but I agree that allowing an alternate type is not appropriate (the goal here is that we require a dictionary). But it's also easier to be conservative now and relax later.
Quoted 18 lines
The latter is caught (good), but the former is not (bug).
Signed-off-by Markus Armbruster <[email protected]>--- tests/Makefile | 3 ++- tests/qapi-schema/args-alternate.err | 1 + tests/qapi-schema/args-alternate.exit | 1 + tests/qapi-schema/args-alternate.json | 4 ++++ tests/qapi-schema/args-alternate.out | 0 tests/qapi-schema/args-union.err | 0 tests/qapi-schema/args-union.exit | 1 + tests/qapi-schema/args-union.json | 4 ++++ tests/qapi-schema/args-union.out | 4 ++++ 9 files changed, 17 insertions(+), 1 deletion(-)+++
b/tests/qapi-schema/args-alternate.json
tests/qapi-schema/args-alternate.json line 2
Quoted 2 lines
tests/qapi-schema/args-alternate.json new file +2
-0@@ -0,0 +1,4 @@ +# we do not allow alternate arguments +# TODO should we support this?
I see no reason to allow a non-dictionary in QMP, so this TODO could be dropped.
tests/qapi-schema/args-union.json line 4
Quoted 6 lines
tests/qapi-schema/args-union.json new file +5
-0+++ b/tests/qapi-schema/args-union.json@@ -0,0 +1,4 @@ +# FIXME we should reject union arguments +# TODO should we support this? +{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } } +{ 'command': 'oops', 'data': 'Uni' }
This, on the other hand, seems valid from the wire format (it will always be a dictionary). I guess the problem is that we generate a C function signature based by calling out each member of the dictionary - but how do you do that for a union? So I see what you are doing:
marking that this test currently passes the parser, but then causes problems for generating C code, so we should either reject it up front, or fix the generator. The FIXME documents what you will do later in the series (reject it up front) and the TODO documents what we can do down the road (fix the generator to allow it).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments
Quoted 13 lines · Eric Blake
Quoted 5 lines · Markus Armbruster
A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.
Existing test case data-int.json covers simple type 'int'. Add test cases for type names referring to union and alternate types.
We could probably relax things to allow a union (which is always a dictionary on the wire), but I agree that allowing an alternate type is not appropriate (the goal here is that we require a dictionary). But it's also easier to be conservative now and relax later.
Quoted 7 lines
tests/qapi-schema/args-alternate.json line 2
Quoted 4 lines
+++
b/tests/qapi-schema/args-alternate.jsontests/qapi-schema/args-alternate.json new file +2
-0@@ -0,0 +1,4 @@ +# we do not allow alternate arguments +# TODO should we support this?I see no reason to allow a non-dictionary in QMP, so this TODO could be dropped.
Or, to be clear, we document that arguments is always a dictionary, for:
{ "execute":"command", "arguments":{} }. Allowing an alternate would break that, so it is a different level of change to allow an alternate (change the QMP protocol) than what it would be to allow a union (the QMP protocol is unchanged). Not that we can't do it if we ever have a reason, it's just that I don't see it being worth a TODO statement.
Quoted 17 lines
tests/qapi-schema/args-union.json line 4
Quoted 7 lines
tests/qapi-schema/args-union.json new file +5
-0+++ b/tests/qapi-schema/args-union.json@@ -0,0 +1,4 @@ +# FIXME we should reject union arguments +# TODO should we support this? +{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } } +{ 'command': 'oops', 'data': 'Uni' }This, on the other hand, seems valid from the wire format (it will always be a dictionary). I guess the problem is that we generate a C function signature based by calling out each member of the dictionary - but how do you do that for a union? So I see what you are doing:
marking that this test currently passes the parser, but then causes problems for generating C code, so we should either reject it up front, or fix the generator. The FIXME documents what you will do later in the series (reject it up front) and the TODO documents what we can do down the road (fix the generator to allow it).
See also 32/47 - events have the same problem.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments
Quoted 5 lines · Eric Blake
Quoted 4 lines · Eric Blake
Quoted 3 lines · Markus Armbruster
A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Existing test case data-int.json covers simple type 'int'. Add test cases for type names referring to union and alternate types.
We could probably relax things to allow a union (which is always a dictionary on the wire), but I agree that allowing an alternate type is not appropriate (the goal here is that we require a dictionary). But it's also easier to be conservative now and relax later.
Yes.
Quoted 8 lines
Quoted 7 lines
tests/qapi-schema/args-alternate.json line 2
Quoted 4 lines
+++
b/tests/qapi-schema/args-alternate.jsontests/qapi-schema/args-alternate.json new file +2
-0@@ -0,0 +1,4 @@ +# we do not allow alternate arguments +# TODO should we support this?I see no reason to allow a non-dictionary in QMP, so this TODO could be dropped.
Quoted 6 lines
Or, to be clear, we document that arguments is always a dictionary, for:
{ "execute":"command", "arguments":{} }. Allowing an alternate would break that, so it is a different level of change to allow an alternate (change the QMP protocol) than what it would be to allow a union (the QMP protocol is unchanged). Not that we can't do it if we ever have a reason, it's just that I don't see it being worth a TODO statement.
I'll drop it.
Quoted 11 lines
Quoted 11 lines
tests/qapi-schema/args-union.json line 4
Quoted 6 lines
tests/qapi-schema/args-union.json new file +5
-0+++ b/tests/qapi-schema/args-union.json@@ -0,0 +1,4 @@ +# FIXME we should reject union arguments +# TODO should we support this? +{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } } +{ 'command': 'oops', 'data': 'Uni' }This, on the other hand, seems valid from the wire format (it will always be a dictionary). I guess the problem is that we generate a C function signature based by calling out each member of the dictionary - but how do you do that for a union?
Exactly: the problem is neither conceptual nor the wire API, it's the C API we generate.
Quoted 5 lines
Quoted 5 lines
So I see what you are doing:
marking that this test currently passes the parser, but then causes problems for generating C code, so we should either reject it up front, or fix the generator. The FIXME documents what you will do later in the series (reject it up front)
Yes, in PATCH 15.
Quoted 2 lines
Quoted 2 lines
and the TODO documents what we can do down
the road (fix the generator to allow it).
I figure we'd change the C API not to explode the data type into multiple parameters. We can consider that when we have a use for it.
Quoted 1 line
See also 32/47 - events have the same problem.
I'm afraid I don't see the connection to PATCH 32.
Quoted 1 line
Quoted 1 line
Reviewed-by Eric Blake <[email protected]>
Thanks!
Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments
Quoted 19 lines · Markus Armbruster
Quoted 4 lines
Quoted 4 lines
This, on the other hand, seems valid from the wire format (it will always be a dictionary). I guess the problem is that we generate a C function signature based by calling out each member of the dictionary - but how do you do that for a union?
Exactly: the problem is neither conceptual nor the wire API, it's the C API we generate.
Quoted 5 lines
Quoted 5 lines
So I see what you are doing:
marking that this test currently passes the parser, but then causes problems for generating C code, so we should either reject it up front, or fix the generator. The FIXME documents what you will do later in the series (reject it up front)Yes, in PATCH 15.
Quoted 2 lines
Quoted 2 lines
and the TODO documents what we can do down
the road (fix the generator to allow it).
Quoted 6 lines
I figure we'd change the C API not to explode the data type into multiple parameters. We can consider that when we have a use for it.
Quoted 1 line
See also 32/47 - events have the same problem.
I'm afraid I don't see the connection to PATCH 32.
Patch 32 was where I figured out that we have the same problem where the C code we generate for an event will break if an event tries to use a union type as its data dictionary; and therefore, this patch would be an ideal time to add a test for that, and patch 15 would be an ideal time to tweak events to not allow unions (as the simple fix, where someday down the road we may relax things to allow unions in both commands and events).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments
In reply to Eric BlakeQuoted 47 lines · Eric Blake
Quoted 13 lines · Eric Blake
Quoted 5 lines · Markus Armbruster
A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.
Existing test case data-int.json covers simple type 'int'. Add test cases for type names referring to union and alternate types.
We could probably relax things to allow a union (which is always a dictionary on the wire), but I agree that allowing an alternate type is not appropriate (the goal here is that we require a dictionary). But it's also easier to be conservative now and relax later.
Quoted 7 lines
tests/qapi-schema/args-alternate.json line 2
Quoted 4 lines
+++
b/tests/qapi-schema/args-alternate.jsontests/qapi-schema/args-alternate.json new file +2
-0@@ -0,0 +1,4 @@ +# we do not allow alternate arguments +# TODO should we support this?I see no reason to allow a non-dictionary in QMP, so this TODO could be dropped.
Or, to be clear, we document that arguments is always a dictionary, for:
{ "execute":"command", "arguments":{} }. Allowing an alternate would break that, so it is a different level of change to allow an alternate (change the QMP protocol) than what it would be to allow a union (the QMP protocol is unchanged). Not that we can't do it if we ever have a reason, it's just that I don't see it being worth a TODO statement.Quoted 17 lines
tests/qapi-schema/args-union.json line 4
Quoted 7 lines
tests/qapi-schema/args-union.json new file +5
-0+++ b/tests/qapi-schema/args-union.json@@ -0,0 +1,4 @@ +# FIXME we should reject union arguments +# TODO should we support this? +{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } } +{ 'command': 'oops', 'data': 'Uni' }This, on the other hand, seems valid from the wire format (it will always be a dictionary). I guess the problem is that we generate a C function signature based by calling out each member of the dictionary - but how do you do that for a union? So I see what you are doing:
marking that this test currently passes the parser, but then causes problems for generating C code, so we should either reject it up front, or fix the generator. The FIXME documents what you will do later in the series (reject it up front) and the TODO documents what we can do down the road (fix the generator to allow it).
Quoted 1 line
See also 32/47 - events have the same problem.
I'll amend the next patch to fix them, too.
RFC PATCH v2 15/47 qapi: Fix to reject union command arguments
In reply to Markus ArmbrusterA command's 'data' must be a struct type, given either as a dictionary, or as struct type name.
Commit dd883c6 tightened the checking there, but not enough: we still accept 'union'. Fix to reject it.
We may want to support union types there, but we'll have to extend qapi-commands.py for it.
--- scripts/qapi.py | 2 +- tests/qapi-schema/args-union.err | 1 + tests/qapi-schema/args-union.exit | 2 +- tests/qapi-schema/args-union.json | 2 +- tests/qapi-schema/args-union.out | 4 ---- 5 files changed, 4 insertions(+), 7 deletions(-)
scripts/qapi.py +1 -1
@@ -495,7 +495,7 @@ def check_command(expr, expr_info): check_type(expr_info, "'data' for command '%s'" % name, expr.get('data'), allow_dict=True, allow_optional=True, - allow_metas=['union', 'struct'], allow_star=allow_star) + allow_metas=['struct'], allow_star=allow_star) returns_meta = ['union', 'struct'] if name in returns_whitelist: returns_meta += ['built-in', 'alternate', 'enum']
tests/qapi-schema/args-union.err +1 -0
@@ -0,0 +1 @@ +tests/qapi-schema/args-union.json:4: 'data' for command 'oops' cannot use union type 'Uni'
tests/qapi-schema/args-union.exit +1 -1
@@ -1 +1 @@ -0 +1
tests/qapi-schema/args-union.json +1 -1
@@ -1,4 +1,4 @@ -# FIXME we should reject union arguments +# we do not allow union arguments # TODO should we support this? { 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } } { 'command': 'oops', 'data': 'Uni' }
tests/qapi-schema/args-union.out +0 -4
@@ -1,4 +0,0 @@ -[OrderedDict([('union', 'Uni'), ('data', OrderedDict([('case1', 'int'), ('case2', 'str')]))]), - OrderedDict([('command', 'oops'), ('data', 'Uni')])] -[{'enum_name': 'UniKind', 'enum_values': None}] -[]
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 15/47] qapi: Fix to reject union command arguments
Quoted 5 lines · Markus Armbruster
A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.
Commit dd883c6 tightened the checking there, but not enough: we still accept 'union'. Fix to reject it.
I was thinking that 'all dictionaries are okay' when I wrote that commit, and obviously missed thinking about 'how would we translate unions into a C prototype'.
Quoted 4 lines
We may want to support union types there, but we'll have to extend
qapi-commands.pyfor it.
Quoted 8 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 2 +- tests/qapi-schema/args-union.err | 1 + tests/qapi-schema/args-union.exit | 2 +- tests/qapi-schema/args-union.json | 2 +- tests/qapi-schema/args-union.out | 4 ---- 5 files changed, 4 insertions(+), 7 deletions(-)
scripts/qapi.py def check_command(expr, expr_info): · line 498
Quoted 11 lines
scripts/qapi.py +1
-1@@ -495,7 +495,7 @@ def check_command(expr, expr_info): check_type(expr_info, "'data' for command '%s'" % name, expr.get('data'), allow_dict=True, allow_optional=True, - allow_metas=['union', 'struct'], allow_star=allow_star) + allow_metas=['struct'], allow_star=allow_star)
I'm still quite pleased at how my earlier work made this a one-liner fix
- the check_type() interface turned out to be rather pleasant.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 16/47 qapi-commands: Fix gen_err_check(e) for e and e != 'local_err'
In reply to Markus Armbrustergen_err_check() hard-codes 'local_err' instead of substituting the argument. Currently harmless, since all callers pass either None or 'local_err'.
--- scripts/qapi-commands.py | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
scripts/qapi-commands.py +7 -6
@@ -29,14 +29,15 @@ def generate_command_decl(name, args, ret_type): ret_type=c_type(ret_type), name=c_name(name), args=arglist).strip() -def gen_err_check(errvar): - if errvar: - return mcgen(''' -if (local_err) { +def gen_err_check(err): + if not err: + return '' + return mcgen(''' +if (%(err)s) { goto out; } -''') - return '' +''', + err=err) def gen_sync_call(name, args, ret_type): ret = ""
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 16/47] qapi-commands: Fix gen_err_check(e) for e and e != 'local_err'
Quoted 9 lines · Markus Armbruster
gen_err_check()hard-codes 'local_err' instead of substituting the argument. Currently harmless, since all callers pass either None or 'local_err'.Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 17/47 qapi-commands: Inline gen_marshal_output_call()
In reply to Markus Armbruster--- scripts/qapi-commands.py | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
scripts/qapi-commands.py +4 -8
@@ -57,19 +57,15 @@ def gen_sync_call(name, args, ret_type): name=c_name(name), args=arglist, retval=retval).rstrip() if ret_type: ret += "\n" + gen_err_check('local_err') - ret += "\n" + mcgen(''' -%(marshal_output_call)s + ret += mcgen(''' + +qmp_marshal_output_%(c_name)s(retval, ret, &local_err); ''', - marshal_output_call=gen_marshal_output_call(name, ret_type)).rstrip() + c_name=c_name(name)) pop_indent() return ret.rstrip() -def gen_marshal_output_call(name, ret_type): - if not ret_type: - return "" - return "qmp_marshal_output_%s(retval, ret, &local_err);" % c_name(name) - def gen_visitor_input_containers_decl(args): ret = ""
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 17/47] qapi-commands: Inline gen_marshal_output_call()
Quoted 5 lines · Markus Armbruster
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 18/47 qapi-commands: Don't feed output of mcgen() to mcgen() again
In reply to Markus ArmbrusterMultiple passes through mcgen() is prone to produce unwanted blank lines, which we then combat by sprinkling .rstrip() on top. Just don't do it.
--- scripts/qapi-commands.py | 52 +++++++++++++++++++----------------------------- 1 file changed, 21 insertions(+), 31 deletions(-)
scripts/qapi-commands.py +21 -31
@@ -27,7 +27,7 @@ def generate_command_decl(name, args, ret_type): %(ret_type)s qmp_%(name)s(%(args)sError **errp); ''', ret_type=c_type(ret_type), name=c_name(name), - args=arglist).strip() + args=arglist) def gen_err_check(err): if not err:
@@ -52,19 +52,17 @@ def gen_sync_call(name, args, ret_type): push_indent() ret = mcgen(''' %(retval)sqmp_%(name)s(%(args)s&local_err); - ''', - name=c_name(name), args=arglist, retval=retval).rstrip() + name=c_name(name), args=arglist, retval=retval) if ret_type: - ret += "\n" + gen_err_check('local_err') + ret += gen_err_check('local_err') ret += mcgen(''' qmp_marshal_output_%(c_name)s(retval, ret, &local_err); ''', c_name=c_name(name)) pop_indent() - return ret.rstrip() - + return ret def gen_visitor_input_containers_decl(args): ret = ""
@@ -78,7 +76,7 @@ Visitor *v; ''') pop_indent() - return ret.rstrip() + return ret def gen_visitor_input_vars_decl(args): ret = ""
@@ -101,7 +99,7 @@ bool has_%(argname)s = false; argname=c_name(argname), argtype=c_type(argtype)) pop_indent() - return ret.rstrip() + return ret def gen_visitor_input_block(args, dealloc=False): ret = ""
@@ -155,7 +153,7 @@ visit_type_%(visitor)s(v, &%(c_name)s, "%(name)s", %(errp)s); qapi_dealloc_visitor_cleanup(md); ''') pop_indent() - return ret.rstrip() + return ret def gen_marshal_output(name, ret_type): if not ret_type:
@@ -217,26 +215,17 @@ def gen_marshal_input(name, args, ret_type, middle_mode): retval=retval) if len(args) > 0: - ret += mcgen(''' -%(visitor_input_containers_decl)s -%(visitor_input_vars_decl)s - -%(visitor_input_block)s - -''', - visitor_input_containers_decl=gen_visitor_input_containers_decl(args), - visitor_input_vars_decl=gen_visitor_input_vars_decl(args), - visitor_input_block=gen_visitor_input_block(args)) + ret += gen_visitor_input_containers_decl(args) + ret += gen_visitor_input_vars_decl(args) + '\n' + ret += gen_visitor_input_block(args) + '\n' else: ret += mcgen(''' (void)args; ''') - ret += mcgen(''' -%(sync_call)s -''', - sync_call=gen_sync_call(name, args, ret_type)) + ret += gen_sync_call(name, args, ret_type) + if re.search('^ *goto out\\;', ret, re.MULTILINE): ret += mcgen('''
@@ -244,11 +233,11 @@ out: ''') ret += mcgen(''' error_propagate(errp, local_err); -%(visitor_input_block_cleanup)s +''') + ret += gen_visitor_input_block(args, dealloc=True) + ret += mcgen(''' } -''', - visitor_input_block_cleanup=gen_visitor_input_block(args, - dealloc=True)) +''') return ret def gen_registry(commands):
@@ -268,12 +257,13 @@ qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s); ret = mcgen(''' static void qmp_init_marshal(void) { -%(registry)s +''') + ret += registry + ret += mcgen(''' } qapi_init(qmp_init_marshal); -''', - registry=registry.rstrip()) +''') return ret middle_mode = False
@@ -353,7 +343,7 @@ for cmd in commands: arglist = cmd['data'] if cmd.has_key('returns'): ret_type = cmd['returns'] - ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n" + ret = generate_command_decl(cmd['command'], arglist, ret_type) fdecl.write(ret) if ret_type: ret = gen_marshal_output(cmd['command'], ret_type) + "\n"
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 18/47] qapi-commands: Don't feed output of mcgen() to mcgen() again
Quoted 8 lines · Markus Armbruster
Multiple passes through
mcgen()is prone to produce unwanted blank lines, which we then combat by sprinkling .rstrip() on top. Just don't do it.Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 52 +++++++++++++++++++----------------------------- 1 file changed, 21 insertions(+), 31 deletions(-)
Causes the generated files to be slightly shorter (good), with diffs like the following:
--- qga/qapi-generated/qga-qmp-marshal.c.old 2015-07-20
17:13:06.056297347 -0600
+++ qga/qapi-generated/qga-qmp-marshal.c 2015-07-21
11:16:09.887228043 -0600
@@ -140,7 +140,6 @@
(void)args;
qmp_guest_ping(&local_err);
error_propagate(errp, local_err);-
}
static void qmp_marshal_output_guest_get_time(int64_t ret_in, QObject
**ret_out, Error **errp)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 19/47 qapi: Generated code cleanup
In reply to Markus ArmbrusterClean up white-space, brace placement, and superfluous
--- scripts/qapi-commands.py | 1 + scripts/qapi-event.py | 3 +-- scripts/qapi-types.py | 66 +++++++++++++++++++++++------------------------- scripts/qapi-visit.py | 1 + 4 files changed, 34 insertions(+), 37 deletions(-)
scripts/qapi-commands.py +1 -0
@@ -222,6 +222,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode): ret += mcgen(''' (void)args; + ''') ret += gen_sync_call(name, args, ret_type)
scripts/qapi-event.py +1 -2
@@ -167,8 +167,7 @@ extern const char *%(event_enum_name)s_lookup[]; event_enum_name = event_enum_name) enum_decl = mcgen(''' -typedef enum %(event_enum_name)s -{ +typedef enum %(event_enum_name)s { ''', event_enum_name = event_enum_name)
scripts/qapi-types.py +31 -35
181 lines not on this page · raw message
scripts/qapi-visit.py +1 -0
@@ -62,6 +62,7 @@ def generate_visit_struct_fields(name, members, base = None): static void visit_type_%(name)s_fields(Visitor *m, %(name)s **obj, Error **errp) { Error *err = NULL; + ''', name=c_name(name)) push_indent()
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup
Quoted 1 line · Markus Armbruster
Clean up white-space, brace placement, and superfluous
Incomplete sentence. I bet it's because your editor line-wrapped, and the rest of your sentence was something like '#ifndef in a .c file.' (see [1] below), then git ate the line thinking it was a comment. I think I'm okay with cramming all of these cleanups into one patch rather than trying to do one style of cleanup per patch (blank lines, { placement, and guard cleanup), but the commit message could do a better job of explaining things.
Quoted 8 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 1 + scripts/qapi-event.py | 3 +-- scripts/qapi-types.py | 66 +++++++++++++++++++++++------------------------- scripts/qapi-visit.py | 1 + 4 files changed, 34 insertions(+), 37 deletions(-)
Missing changes to docs/qapi-code-gen.txt to reflect the improvements. Since having docs that are stale compared to implementation can be misleading, I'd like to wait for v2 before giving my R-b; but see also
[2] below.
Overall, I'm a fan of the cleanups.
I'm going to intersperse diffs of the generated files that were caused by some of these changes:
scripts/qapi-commands.py def gen_marshal_input(name, args, ret_type, middle_mode): · line 226
Quoted 11 lines
scripts/qapi-commands.py +1
-0@@ -222,6 +222,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode): ret += mcgen(''' (void)args; + ''')
This and similar hunts avoids extra blank lines. Not worth showing a diff to the generated files.
scripts/qapi-event.py extern const char *%(event_enum_name)s_lookup[]; · line 170
Quoted 13 lines
scripts/qapi-commands.py +0
-0ret += gen_sync_call(name, args, ret_type)scripts/qapi-event.py +1
-2@@ -167,8 +167,7 @@ extern const char *%(event_enum_name)s_lookup[]; event_enum_name = event_enum_name) enum_decl = mcgen(''' -typedef enum %(event_enum_name)s -{ +typedef enum %(event_enum_name)s {
Several hunks like this; gets us generally closer to qemu style; with generated diffs like:
qapi-types.h:
-typedef struct int32List
-{
+typedef struct int32List {
union {
int32_t value;
uint64_t padding;scripts/qapi-event.py struct %(name)s · line 107
Quoted 7 lines
scripts/qapi-event.py +2
-1@@ -105,7 +103,8 @@ struct %(name)s def generate_enum_lookup(name, values): ret = mcgen(''' -const char * const %(name)s_lookup[] = { + +const char *const %(name)s_lookup[] = {
[2] generated diffs like this:
qapi-types.c:
-const char * const OnOffAuto_lookup[] = {
+const char *const OnOffAuto_lookup[] = {
Hmm - we already failed to update docs/qapi-code-gen.txt in the past; we added a const in commit 2e4450ff that is missing from the documentation.
scripts/qapi-event.py for typename in builtin_types.keys(): · line 342
Quoted 13 lines
scripts/qapi-event.py +2
-2@@ -335,22 +333,22 @@ for typename in builtin_types.keys(): fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL")) for expr in exprs: - ret = "\n" + ret = "" if expr.has_key('struct'): ret += generate_fwd_struct(expr['struct']) elif expr.has_key('enum'): - ret += generate_enum(expr['enum'], expr['data']) + "\n" + ret += generate_enum(expr['enum'], expr['data']) ret += generate_fwd_enum_struct(expr['enum']) fdef.write(generate_enum_lookup(expr['enum'], expr['data']))
More work at avoiding extra blank lines.
scripts/qapi-event.py fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) · line 376
Quoted 8 lines
scripts/qapi-event.py +0
-2@@ -370,34 +368,32 @@ fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) # have the functions defined, so we use -b option to provide control # over these cases if do_builtins: - fdef.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DEF")) for typename in builtin_types.keys(): fdef.write(generate_type_cleanup(typename + "List")) - fdef.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))
[1] this is the hunk whose description got corrupted in your commit message. It gives a diff like this:
qapi-types.c:
-
-#ifndef QAPI_TYPES_BUILTIN_CLEANUP_DEF
-#define QAPI_TYPES_BUILTIN_CLEANUP_DEF
-
-
void qapi_free_int32List(int32List *obj)
We conditionally declare the functions in the .h, but the .c is not compiled multiple times, so there is no need for a header-style guard.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup
Quoted 10 lines · Eric Blake
Quoted 1 line · Markus Armbruster
Clean up white-space, brace placement, and superfluous
Incomplete sentence. I bet it's because your editor line-wrapped, and the rest of your sentence was something like '#ifndef in a .c file.' (see [1] below), then git ate the line thinking it was a comment. I think I'm okay with cramming all of these cleanups into one patch rather than trying to do one style of cleanup per patch (blank lines, { placement, and guard cleanup), but the commit message could do a better job of explaining things.
Not throwing away half of it by accident would be a start :)
Quoted 9 lines
Quoted 8 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 1 + scripts/qapi-event.py | 3 +-- scripts/qapi-types.py | 66 +++++++++++++++++++++++------------------------- scripts/qapi-visit.py | 1 + 4 files changed, 34 insertions(+), 37 deletions(-)
Quoted 4 lines
Missing changes to
docs/qapi-code-gen.txtto reflect the improvements. Since having docs that are stale compared to implementation can be misleading, I'd like to wait for v2 before giving my R-b; but see also
[2] below.
I agree we should keep docs/qapi-code-gen.txt up-to-date.
Since updating it is manual, updating it just once for a complete series can make sense.
I'll double check it's accurate at the end of the series, then decide how to do necessary updates, if any.
Quoted 59 lines
Overall, I'm a fan of the cleanups.
I'm going to intersperse diffs of the generated files that were caused by some of these changes:
scripts/qapi-commands.py def gen_marshal_input(name, args, ret_type, middle_mode): · line 226
Quoted 11 lines
scripts/qapi-commands.py +1
-0@@ -222,6 +222,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode): ret += mcgen(''' (void)args; + ''')This and similar hunts avoids extra blank lines. Not worth showing a diff to the generated files.
scripts/qapi-event.py extern const char *%(event_enum_name)s_lookup[]; · line 170
Quoted 13 lines
scripts/qapi-commands.py +0
-0ret += gen_sync_call(name, args, ret_type)scripts/qapi-event.py +1
-2@@ -167,8 +167,7 @@ extern const char *%(event_enum_name)s_lookup[]; event_enum_name = event_enum_name) enum_decl = mcgen(''' -typedef enum %(event_enum_name)s -{ +typedef enum %(event_enum_name)s {Several hunks like this; gets us generally closer to qemu style; with generated diffs like:
qapi-types.h:scripts/qapi-types.py +1
-2-typedef struct int32List -{ +typedef struct int32List { union { int32_t value; uint64_t padding;scripts/qapi-event.py struct %(name)s · line 107
Quoted 8 lines
scripts/qapi-event.py +2
-1@@ -105,7 +103,8 @@ struct %(name)s def generate_enum_lookup(name, values): ret = mcgen(''' -const char * const %(name)s_lookup[] = { + +const char *const %(name)s_lookup[] = {[2] generated diffs like this:
qapi-types.c:(quoted diff) +1
-1-const char * const OnOffAuto_lookup[] = { +const char *const OnOffAuto_lookup[] = {
Quoted 2 lines
Hmm - we already failed to update
docs/qapi-code-gen.txtin the past; we added a const in commit2e4450ffthat is missing from the documentation.
Minor review fail. Not the first time.
Quoted 37 lines
scripts/qapi-event.py for typename in builtin_types.keys(): · line 342
Quoted 13 lines
scripts/qapi-event.py +2
-2@@ -335,22 +333,22 @@ for typename in builtin_types.keys(): fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL")) for expr in exprs: - ret = "\n" + ret = "" if expr.has_key('struct'): ret += generate_fwd_struct(expr['struct']) elif expr.has_key('enum'): - ret += generate_enum(expr['enum'], expr['data']) + "\n" + ret += generate_enum(expr['enum'], expr['data']) ret += generate_fwd_enum_struct(expr['enum']) fdef.write(generate_enum_lookup(expr['enum'], expr['data']))More work at avoiding extra blank lines.
scripts/qapi-event.py fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) · line 376
Quoted 8 lines
scripts/qapi-event.py +0
-2@@ -370,34 +368,32 @@ fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) # have the functions defined, so we use -b option to provide control # over these cases if do_builtins: - fdef.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DEF")) for typename in builtin_types.keys(): fdef.write(generate_type_cleanup(typename + "List")) - fdef.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))[1] this is the hunk whose description got corrupted in your commit message. It gives a diff like this:
qapi-types.c:(quoted diff) +0
-5- -#ifndef QAPI_TYPES_BUILTIN_CLEANUP_DEF -#define QAPI_TYPES_BUILTIN_CLEANUP_DEF - - void qapi_free_int32List(int32List *obj)
Quoted 2 lines
We conditionally declare the functions in the .h, but the .c is not compiled multiple times, so there is no need for a header-style guard.
This is going to help me reconstruct my crippled commit message, thanks!
Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup
Quoted 3 lines · Markus Armbruster
Quoted 1 line · Eric Blake
On 07/01/2015 02:22 PM, Markus Armbruster wrote:
[...]
Quoted 10 lines
Quoted 10 lines
scripts/qapi-types.py struct %(name)s · line 107
Quoted 7 lines
scripts/qapi-types.py +2
-1@@ -105,7 +103,8 @@ struct %(name)s def generate_enum_lookup(name, values): ret = mcgen(''' -const char * const %(name)s_lookup[] = { + +const char *const %(name)s_lookup[] = {[2] generated diffs like this:
Quoted 8 lines
Quoted 6 lines
qapi-types.c:(quoted diff) +1
-1-const char * const OnOffAuto_lookup[] = { +const char *const OnOffAuto_lookup[] = {Hmm - we already failed to update
docs/qapi-code-gen.txtin the past; we added a const in commit2e4450ffthat is missing from the documentation.Minor review fail. Not the first time.
I take that back, it's actually not visible in qapi-code-gen.txt.
[...]
Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup
Quoted 20 lines · Markus Armbruster
Quoted 3 lines · Markus Armbruster
Quoted 1 line · Eric Blake
On 07/01/2015 02:22 PM, Markus Armbruster wrote:
[...]
Quoted 14 lines
Quoted 14 lines
scripts/qapi-types.py struct %(name)s · line 107
Quoted 7 lines
scripts/qapi-types.py +2
-1@@ -105,7 +103,8 @@ struct %(name)s def generate_enum_lookup(name, values): ret = mcgen(''' -const char * const %(name)s_lookup[] = { + +const char *const %(name)s_lookup[] = {[2] generated diffs like this:
qapi-types.c:(quoted diff) +1
-1-const char * const OnOffAuto_lookup[] = { +const char *const OnOffAuto_lookup[] = {
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Hmm - we already failed to update
docs/qapi-code-gen.txtin the past; we added a const in commit2e4450ffthat is missing from the documentation.Minor review fail. Not the first time.
I take that back, it's actually not visible in qapi-code-gen.txt.
Oh, you're right - the only mention of *_lookup[] in the docs file is the array generated for all events, and that one was output with correct spacing (until the rest of your series fixes it to share code rather than duplicate things with slight differences).
Maybe we SHOULD be showing what the generators do for enums, unions, and alternates (by expanding the example-schema.json that is then fed to all the example script usage). But that's fine as a project for another day.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup
Quoted 28 lines · Eric Blake
Quoted 20 lines · Markus Armbruster
Quoted 3 lines · Markus Armbruster
Quoted 1 line · Eric Blake
On 07/01/2015 02:22 PM, Markus Armbruster wrote:
[...]
Quoted 14 lines
Quoted 14 lines
scripts/qapi-types.py struct %(name)s · line 107
Quoted 7 lines
scripts/qapi-types.py +2
-1@@ -105,7 +103,8 @@ struct %(name)s def generate_enum_lookup(name, values): ret = mcgen(''' -const char * const %(name)s_lookup[] = { + +const char *const %(name)s_lookup[] = {[2] generated diffs like this:
qapi-types.c:(quoted diff) +1
-1-const char * const OnOffAuto_lookup[] = { +const char *const OnOffAuto_lookup[] = {Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Hmm - we already failed to update
docs/qapi-code-gen.txtin the past; we added a const in commit2e4450ffthat is missing from the documentation.Minor review fail. Not the first time.
I take that back, it's actually not visible in qapi-code-gen.txt.
Quoted 8 lines
Oh, you're right - the only mention of *_lookup[] in the docs file is the array generated for all events, and that one was output with correct spacing (until the rest of your series fixes it to share code rather than duplicate things with slight differences).
Maybe we SHOULD be showing what the generators do for enums, unions, and alternates (by expanding the example-schema.json that is then fed to all the example script usage). But that's fine as a project for another day.
The place where we try to exercise all the schema features is qapi-schema-test.json.
RFC PATCH v2 20/47 qapi: Rename class QAPISchema to QAPISchemaParser
In reply to Markus ArmbrusterI want to name a new class QAPISchema.
While there, make it a new-style class.
--- scripts/qapi.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi.py +4 -4
@@ -103,7 +103,7 @@ class QAPIExprError(Exception): return error_path(self.info['parent']) + \ "%s:%d: %s" % (self.info['file'], self.info['line'], self.msg) -class QAPISchema: +class QAPISchemaParser(object): def __init__(self, fp, previously_included = [], incl_info = None): abs_fname = os.path.abspath(fp.name)
@@ -149,8 +149,8 @@ class QAPISchema: except IOError, e: raise QAPIExprError(expr_info, '%s: %s' % (e.strerror, include)) - exprs_include = QAPISchema(fobj, previously_included, - expr_info) + exprs_include = QAPISchemaParser(fobj, previously_included, + expr_info) self.exprs.extend(exprs_include.exprs) else: expr_elem = {'expr': expr,
@@ -755,7 +755,7 @@ def check_exprs(exprs): def parse_schema(fname): try: - schema = QAPISchema(open(fname, "r")) + schema = QAPISchemaParser(open(fname, "r")) return check_exprs(schema.exprs) except (QAPISchemaError, QAPIExprError), e: print >>sys.stderr, e
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 20/47] qapi: Rename class QAPISchema to QAPISchemaParser
Quoted 8 lines · Markus Armbruster
I want to name a new class QAPISchema.
While there, make it a new-style class.
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 21/47 qapi: New QAPISchema intermediate reperesentation
In reply to Markus ArmbrusterThe QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaties) on the side.
They have clearly outgrown these simple data structures. There's lots of rummaging around in dictionaries, and information is recomputed on the fly. For the work I'm going to do, I want more clearly defined and more convenient interfaces.
Going forward, I also want less coupling between the back-ends and the syntax tree, to make messing with the syntax easier.
Create a bunch of classes to represent QAPI schemata.
Have the QAPISchema initializer call the parser, then walk the syntax tree to create the new internal representation, and finally perform semantic analysis.
Shortcut: the semantic analysis still relies on existing check_exprs() to do the actual semantic checking. All this code needs to move into the classes. Mark as TODO.
We generate array types eagerly, even though most of them aren't used. Mark as TODO.
--- scripts/qapi-commands.py | 2 +- scripts/qapi-event.py | 2 +- scripts/qapi-types.py | 2 +- scripts/qapi-visit.py | 2 +- scripts/qapi.py | 351 +++++++++++++++++++++++++++++++++++++++-- tests/qapi-schema/test-qapi.py | 2 +- 6 files changed, 347 insertions(+), 14 deletions(-)
scripts/qapi-commands.py +1 -1
@@ -276,7 +276,7 @@ for o, a in opts: if o in ("-m", "--middle"): middle_mode = True -exprs = parse_schema(input_file) +exprs = QAPISchema(input_file).get_exprs() commands = filter(lambda expr: expr.has_key('command'), exprs) commands = filter(lambda expr: not expr.has_key('gen'), commands)
scripts/qapi-event.py +1 -1
@@ -263,7 +263,7 @@ fdecl.write(mcgen(''' ''', prefix=prefix)) -exprs = parse_schema(input_file) +exprs = QAPISchema(input_file).get_exprs() event_enum_name = c_name(prefix + "QAPIEvent", protect=False) event_enum_values = []
scripts/qapi-types.py +1 -1
@@ -325,7 +325,7 @@ fdecl.write(mcgen(''' #include <stdint.h> ''')) -exprs = parse_schema(input_file) +exprs = QAPISchema(input_file).get_exprs() fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL")) for typename in builtin_types.keys():
scripts/qapi-visit.py +1 -1
@@ -442,7 +442,7 @@ fdecl.write(mcgen(''' ''', prefix=prefix)) -exprs = parse_schema(input_file) +exprs = QAPISchema(input_file).get_exprs() # to avoid header dependency hell, we always generate declarations # for built-in types in our header files and simply guard them
scripts/qapi.py +342 -9
363 lines not on this page · raw message
tests/qapi-schema/test-qapi.py +1 -1
@@ -16,7 +16,7 @@ import os import sys try: - exprs = parse_schema(sys.argv[1]) + exprs = QAPISchema(sys.argv[1]).get_exprs() except SystemExit: raise
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 21/47] qapi: New QAPISchema intermediate reperesentation
Quoted 2 lines · Markus Armbruster
The QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaties) on the side.
s/dictionaties/dictionaries/
Quoted 8 lines
They have clearly outgrown these simple data structures. There's lots of rummaging around in dictionaries, and information is recomputed on the fly. For the work I'm going to do, I want more clearly defined and more convenient interfaces.
Going forward, I also want less coupling between the back-ends and the syntax tree, to make messing with the syntax easier.
Indeed - should be a lot easier to add new qapi .json syntax sugar on the front-end, or to tweak generated C layout on the backend, without close coupling between the two. Particularly when adding front-end sugar extensions that still normalize into the same backend structs.
Quoted 7 lines
Create a bunch of classes to represent QAPI schemata.
Have the QAPISchema initializer call the parser, then walk the syntax tree to create the new internal representation, and finally perform semantic analysis.
Quoted 7 lines
Shortcut: the semantic analysis still relies on existing
check_exprs()to do the actual semantic checking. All this code needs to move into the classes. Mark as TODO.We generate array types eagerly, even though most of them aren't used. Mark as TODO.
No change to generated files at this stage in the game (this is mostly additive, then later patches shed their old ways of doing things by using this). Good division of labor between patches.
Quoted 29 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 2 +- scripts/qapi-event.py | 2 +- scripts/qapi-types.py | 2 +- scripts/qapi-visit.py | 2 +- scripts/qapi.py | 351 +++++++++++++++++++++++++++++++++++++++-- tests/qapi-schema/test-qapi.py | 2 +- 6 files changed, 347 insertions(+), 14 deletions(-)+++
b/scripts/qapi.pyscripts/qapi.py +16
-0+ +# +# Schema compiler frontend +# + +class QAPISchemaEntity(object): + def __init__(self, name, info): + assert isinstance(name, str) + self.name = name + self.info = info + def check(self, schema): + pass + +class QAPISchemaType(QAPISchemaEntity): + pass +
scripts/qapi.py def check_exprs(exprs): · line 781
Quoted 10 lines
scripts/qapi.py +10
-0+class QAPISchemaBuiltinType(QAPISchemaType): + def __init__(self, name): + QAPISchemaType.__init__(self, name, None) + +class QAPISchemaEnumType(QAPISchemaType): + def __init__(self, name, info, values): + QAPISchemaType.__init__(self, name, info) + for v in values: + assert isinstance(v, str) + self.values = values
worth a check() method to ensure values don't collide? Especially since you already do that in QAPISchemaObjectTypeMember.check()
Quoted 19 lines
scripts/qapi.py +19
-0+ +class QAPISchemaArrayType(QAPISchemaType): + def __init__(self, name, info, element_type): + QAPISchemaType.__init__(self, name, info) + assert isinstance(element_type, str) + self.element_type_name = element_type + self.element_type = None + def check(self, schema): + self.element_type = schema.lookup_type(self.element_type_name) + assert self.element_type + +class QAPISchemaObjectType(QAPISchemaType): + def __init__(self, name, info, base, local_members, variants): + QAPISchemaType.__init__(self, name, info) + assert base == None or isinstance(base, str) + for m in local_members: + assert isinstance(m, QAPISchemaObjectTypeMember) + if variants != None: + assert isinstance(variants, QAPISchemaObjectTypeVariants)
scripts/qapi.py def check_exprs(exprs): · line 810
Quoted 10 lines
scripts/qapi.py +10
-0+ self.base_name = base + self.base = None + self.local_members = local_members + self.variants = variants + self.members = None + def check(self, schema): + if self.members: + return # already checked + assert self.members == None # not running in cycles + self.members = False # mark as being checked
Interesting, but makes sense (since we allow self-referential nesting, populating our own members may require revisiting the same type). Cute that python allows both None and False as distinct non-true settings, so that you end up using the variable as a tri-state.
scripts/qapi.py def check_exprs(exprs): · line 813
Quoted 3 lines
scripts/qapi.py +3
-0+ if self.base_name: + self.base = schema.lookup_type(self.base_name) + self.base.check(schema)
Do you need 'assert self.base' here, similar to how you did it in QAPISchemaArrayType.check()? I guess you'll still get a python exception that None.check() doesn't exist if the lookup_type failed, so it just depends on what error message you want.
Quoted 1 line
+ members = list(self.base.members)
For that matter, do you want to assert isinstance(self.base, QAPISchemaObjectType), since lookup_type can return other types, but other child classes of QAPISchemaType do not have a .members?
scripts/qapi.py def check_exprs(exprs): · line 826
Quoted 12 lines
scripts/qapi.py +12
-0+ else: + members = [] + seen = {} + for m in members: + seen[m.name] = m + for m in self.local_members: + m.check(schema, members, seen) + if self.variants: + self.variants.check(schema, members, seen) + self.members = members + +class QAPISchemaObjectTypeMember(object):
Is 'object' the right base class, or should it be QAPISchemaEntity?
scripts/qapi.py def check_exprs(exprs): · line 829
Quoted 3 lines
scripts/qapi.py +3
-0+ def __init__(self, name, typ, optional): + assert isinstance(name, str) + assert isinstance(typ, str)
I guess that means all callers flatten array types into an unambiguous string first.
scripts/qapi.py def check_exprs(exprs): · line 842
Quoted 13 lines
scripts/qapi.py +13
-0+ assert isinstance(optional, bool) + self.name = name + self.type_name = typ + self.type = None + self.optional = optional + def check(self, schema, all_members, seen): + assert self.name not in seen + self.type = schema.lookup_type(self.type_name) + assert self.type + all_members.append(self) + seen[self.name] = self + +class QAPISchemaObjectTypeVariants(object):
Is 'object' the right base class, or should it be QAPISchemaEntity?
scripts/qapi.py def check_exprs(exprs): · line 854
Quoted 12 lines
scripts/qapi.py +12
-0+ def __init__(self, tag_name, tag_enum, variants): + assert tag_name == None or isinstance(tag_name, str) + assert tag_enum == None or isinstance(tag_enum, str) + for v in variants: + assert isinstance(v, QAPISchemaObjectTypeVariant) + self.tag_name = tag_name + if tag_name: + assert not tag_enum + self.tag_member = None + else: + self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum, + False)
Nice - the tag is either part of the base struct (flat union) or implicitly created (plain union). By the time we are done with check(), we then have both its type (possibly generated implicitly) and its name.
scripts/qapi.py def check_exprs(exprs): · line 860
Quoted 6 lines
scripts/qapi.py +6
-0+ self.variants = variants + def check(self, schema, members, seen): + if self.tag_name: + self.tag_member = seen[self.tag_name] + else: + self.tag_member.check(schema, members, seen)
A little bit tricky here in that flat unions (tag_name was set) must find the tag from the members already loaded from the base class, while plain unions (tag_name is None) add their own implicit member into the union. But it works.
Quoted 6 lines
scripts/qapi.py +6
-0+ assert isinstance(self.tag_member.type, QAPISchemaEnumType) + for v in self.variants: + vseen = dict(seen) + v.check(schema, self.tag_member.type, vseen) + +class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
scripts/qapi.py def check_exprs(exprs): · line 876
Quoted 10 lines
scripts/qapi.py +10
-0+ def __init__(self, name, typ, flat): + QAPISchemaObjectTypeMember.__init__(self, name, typ, False) + assert isinstance(flat, bool) + self.flat = flat + def check(self, schema, tag_type, seen): + QAPISchemaObjectTypeMember.check(self, schema, [], seen) + assert self.name in tag_type.values + if self.flat: + self.type.check(schema) + assert isinstance(self.type, QAPISchemaObjectType)
Do we really want to be tracking self.flat for each variant? After all, docs/qapi-code-gen.txt already describes the mapping from simple union into flat union (as if the flat union had a base class with single member 'kind' of the right type, then each branch of the union composed of an implicit object with a lone member 'data' of the correct type). In other words, is it any better to just normalize into that form now, such that each QAPISchemaObjectTypeVariant is merely a (often one-element) list of name:type members being added to the overall QAPISchemaObject? But I guess it remains to be seen how you use self.flat before knowing if it is worth normalizing away from it.
scripts/qapi.py def check_exprs(exprs): · line 884
Quoted 8 lines
scripts/qapi.py +8
-0+ +class QAPISchemaAlternateType(QAPISchemaType): + def __init__(self, name, info, variants): + QAPISchemaType.__init__(self, name, info) + assert isinstance(variants, QAPISchemaObjectTypeVariants) + assert not variants.tag_name + for v in variants.variants: + assert not v.flat
Hmm. You are using .flat after all.
scripts/qapi.py def check_exprs(exprs): · line 893
Quoted 9 lines
scripts/qapi.py +9
-0+ self.variants = variants + def check(self, schema): + self.variants.check(schema, [], {}) + +class QAPISchemaCommand(QAPISchemaEntity): + def __init__(self, name, info, args, rets, gen, success_response): + QAPISchemaEntity.__init__(self, name, info) + assert not args or isinstance(args, str) + assert not rets or isinstance(rets, str)
Again, this means that the caller has already had to convert qapi dictionaries into an implicit type and pass in that type name here. Works for me.
Quoted 6 lines
scripts/qapi.py +6
-0+ self.args_name = args + self.args = None + self.rets_name = rets + self.rets = None + self.gen = gen + self.success_response = success_response
scripts/qapi.py def check_exprs(exprs): · line 909
Quoted 10 lines
scripts/qapi.py +10
-0+ def check(self, schema): + if self.args_name: + self.args = schema.lookup_type(self.args_name) + assert isinstance(self.args, QAPISchemaObjectType) + assert not self.args.variants # not implemented + self.args.check(schema) + if self.rets_name: + self.rets = schema.lookup_type(self.rets_name) + assert isinstance(self.rets, QAPISchemaType) + self.rets.check(schema)
Matches our existing difference in requiring a dictionary for arguments, but allowing the return of any type (although we now have a whitelist to prevent too many additions of non-dictionaries).
Quoted 27 lines
scripts/qapi.py +27
-0+ +class QAPISchemaEvent(QAPISchemaEntity): + def __init__(self, name, info, data): + QAPISchemaEntity.__init__(self, name, info) + assert not data or isinstance(data, str) + self.data_name = data + self.data = None + def check(self, schema): + if self.data_name: + self.data = schema.lookup_type(self.data_name) + assert isinstance(self.data, QAPISchemaObjectType) + self.data.check(schema) + +class QAPISchema(object): + def __init__(self, fname): + try: + self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs) + except (QAPISchemaError, QAPIExprError), err: + print >>sys.stderr, err + exit(1) + self.entity_dict = {} + self._def_predefineds() + self._def_exprs() + self.check() + + def get_exprs(self): + return [expr_elem['expr'] for expr_elem in self.exprs]
scripts/qapi.py def check_exprs(exprs): · line 946
Quoted 10 lines
scripts/qapi.py +10
-0+ + def _def_entity(self, ent): + assert ent.name not in self.entity_dict + self.entity_dict[ent.name] = ent + + def lookup_entity(self, name, typ=None): + ent = self.entity_dict.get(name) + if typ and not isinstance(ent, typ): + return None + return ent
Ah, so you enforce a shared namespace between types, commands, and events (all three are in self.entity_dict), but can use the typ parameter to allow limiting a lookup to just types.
Quoted 19 lines
scripts/qapi.py +19
-0+ + def lookup_type(self, name): + return self.lookup_entity(name, QAPISchemaType) + + def _def_builtin_type(self, name): + self._def_entity(QAPISchemaBuiltinType(name)) + if name != '**': + self._make_array_type(name) # TODO really needed? + + def _def_predefineds(self): + for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64', + 'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']: + self._def_builtin_type(t) + + def _make_implicit_enum_type(self, name, values): + name = name + 'Kind' + self._def_entity(QAPISchemaEnumType(name, None, values)) + return name +
scripts/qapi.py def check_exprs(exprs): · line 975
Quoted 10 lines
scripts/qapi.py +10
-0+ def _make_array_type(self, element_type): + name = element_type + 'List' + if not self.lookup_type(name): + self._def_entity(QAPISchemaArrayType(name, None, element_type)) + return name + + def _make_implicit_object_type(self, name, role, members): + name = ':obj-%s-%s' % (name, role) + self._def_entity(QAPISchemaObjectType(name, None, None, members, None)) + return name
We may extend the qapi .json syntax to allow anonymous types in more places, but so far, this matches that all existing use of anonymous types are for structs, not unions.
Quoted 39 lines
scripts/qapi.py +39
-0+ + def _def_enum_type(self, expr, info): + name = expr['enum'] + data = expr['data'] + self._def_entity(QAPISchemaEnumType(name, info, data)) + self._make_array_type(name) # TODO really needed? + + def _make_member(self, name, typ): + optional = False + if name.startswith('*'): + name = name[1:] + optional = True + if isinstance(typ, list): + assert len(typ) == 1 + typ = self._make_array_type(typ[0]) + return QAPISchemaObjectTypeMember(name, typ, optional) + + def _make_members(self, data): + return [self._make_member(key, data[key]) for key in data.keys()] + + def _def_struct_type(self, expr, info): + name = expr['struct'] + base = expr.get('base') + data = expr['data'] + self._def_entity(QAPISchemaObjectType(name, info, base, + self._make_members(data), + None)) + self._make_array_type(name) # TODO really needed? + + def _make_flat_variant(self, case, typ): + return QAPISchemaObjectTypeVariant(case, typ, True) + + def _make_flat_variants(self, tag_name, data): + variants = [self._make_flat_variant(key, data[key]) + for key in data.keys()] + return QAPISchemaObjectTypeVariants(tag_name, None, variants) + + def _make_simple_variant(self, case, typ): + if isinstance(typ, list):
scripts/qapi.py def check_exprs(exprs): · line 1024
Quoted 10 lines
scripts/qapi.py +10
-0+ assert len(typ) == 1 + typ = self._make_array_type(typ[0]) + return QAPISchemaObjectTypeVariant(case, typ, False) + + def _make_simple_variants(self, type_name, data): + variants = [self._make_simple_variant(key, data[key]) + for key in data.keys()] + enum = self._make_implicit_enum_type(type_name, + [v.name for v in variants]) + return QAPISchemaObjectTypeVariants(None, enum, variants)
Again, I wonder if normalizing simple unions into flat unions (to get rid of the need for tracking .flat) would make later use any easier, or if it would just make introspection QMP output more verbose.
Quoted 22 lines
scripts/qapi.py +22
-0+ + def _def_union_type(self, expr, info): + name = expr['union'] + data = expr['data'] + tag_name = expr.get('discriminator') + if tag_name: + base = expr['base'] + variants = self._make_flat_variants(tag_name, data) + else: + base = None + variants = self._make_simple_variants(name, data) + self._def_entity(QAPISchemaObjectType(name, info, base, + self._make_members(OrderedDict()), + variants)) + self._make_array_type(name) # TODO really needed? + + def _def_alternate_type(self, expr, info): + name = expr['alternate'] + data = expr['data'] + self._def_entity(QAPISchemaAlternateType(name, info, + self._make_simple_variants(name, data))) + self._make_array_type(name) # TODO really needed?
scripts/qapi.py def check_exprs(exprs): · line 1056
Quoted 10 lines
scripts/qapi.py +10
-0+ + def _def_command(self, expr, info): + name = expr['command'] + args = expr.get('data') + rets = expr.get('returns') + gen = expr.get('gen', True) + success_response = expr.get('success-response', True) + if args and not isinstance(args, str): + args = self._make_implicit_object_type(name, 'args', + self._make_members(args))
If I write { 'command':'Foo', 'data':{} }, does that try and create an implicit type, with no members? I'm wondering if it is any simpler to write args = expr.get('data', {}), and have _make_implicit_object_type() do the right thing when presented with an empty list of members.
Quoted 29 lines
scripts/qapi.py +29
-0+ if rets and isinstance(rets, list): + assert len(rets) == 1 + rets = self._make_array_type(rets[0]) + elif rets and not isinstance(rets, str): + rets = self._make_implicit_object_type(name, 'rets', + self._make_members(rets)) + self._def_entity(QAPISchemaCommand(name, info, args, rets, gen, + success_response)) + + def _def_event(self, expr, info): + name = expr['event'] + data = expr.get('data') + if data and not isinstance(data, str): + data = self._make_implicit_object_type(name, 'data', + self._make_members(data)) + self._def_entity(QAPISchemaEvent(name, info, data)) + + def _def_exprs(self): + for expr_elem in self.exprs: + expr = expr_elem['expr'] + info = expr_elem['info'] + if 'enum' in expr: + self._def_enum_type(expr, info) + elif 'struct' in expr: + self._def_struct_type(expr, info) + elif 'union' in expr: + self._def_union_type(expr, info) + elif 'alternate' in expr: + self._def_alternate_type(expr, info)
scripts/qapi.py def check_exprs(exprs): · line 1095
Quoted 10 lines
scripts/qapi.py +10
-0+ elif 'command' in expr: + self._def_command(expr, info) + elif 'event' in expr: + self._def_event(expr, info) + else: + assert False + + def check(self): + for ent in self.entity_dict.values(): + ent.check(self)
Overall seems like a very nice layout.
I'll reserve R-b until I see how the rest of the series uses this object hierarchy, but it looks like you won't have to change very much of this patch when you do your next spin of the series.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 21/47] qapi: New QAPISchema intermediate reperesentation
Quoted 5 lines · Eric Blake
Quoted 2 lines · Markus Armbruster
The QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaties) on the side.
s/dictionaties/dictionaries/
Will fix.
Quoted 24 lines
Quoted 7 lines
They have clearly outgrown these simple data structures. There's lots of rummaging around in dictionaries, and information is recomputed on the fly. For the work I'm going to do, I want more clearly defined and more convenient interfaces.
Going forward, I also want less coupling between the back-ends and the syntax tree, to make messing with the syntax easier.
Indeed - should be a lot easier to add new qapi .json syntax sugar on the front-end, or to tweak generated C layout on the backend, without close coupling between the two. Particularly when adding front-end sugar extensions that still normalize into the same backend structs.
Quoted 11 lines
Create a bunch of classes to represent QAPI schemata.
Have the QAPISchema initializer call the parser, then walk the syntax tree to create the new internal representation, and finally perform semantic analysis.
Shortcut: the semantic analysis still relies on existing
check_exprs()to do the actual semantic checking. All this code needs to move into the classes. Mark as TODO.
Quoted 7 lines
Quoted 3 lines
We generate array types eagerly, even though most of them aren't used. Mark as TODO.
No change to generated files at this stage in the game (this is mostly additive, then later patches shed their old ways of doing things by using this). Good division of labor between patches.
I'll steal from this paragraph for my commit message.
Quoted 34 lines
Quoted 34 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 2 +- scripts/qapi-event.py | 2 +- scripts/qapi-types.py | 2 +- scripts/qapi-visit.py | 2 +- scripts/qapi.py | 351 +++++++++++++++++++++++++++++++++++++++-- tests/qapi-schema/test-qapi.py | 2 +- 6 files changed, 347 insertions(+), 14 deletions(-)+++
b/scripts/qapi.pyscripts/qapi.py +21
-0+ +# +# Schema compiler frontend +# + +class QAPISchemaEntity(object): + def __init__(self, name, info): + assert isinstance(name, str) + self.name = name + self.info = info + def check(self, schema): + pass + +class QAPISchemaType(QAPISchemaEntity): + pass + +class QAPISchemaBuiltinType(QAPISchemaType): + def __init__(self, name): + QAPISchemaType.__init__(self, name, None) + +class QAPISchemaEnumType(QAPISchemaType):
Quoted 8 lines
scripts/qapi.py def check_exprs(exprs): · line 781
Quoted 5 lines
scripts/qapi.py +5
-0+ def __init__(self, name, info, values): + QAPISchemaType.__init__(self, name, info) + for v in values: + assert isinstance(v, str) + self.values = valuesworth a
check()method to ensure values don't collide? Especially since you already do that inQAPISchemaObjectTypeMember.check()
Can do.
Aside: everything the check() methods assert must be established by semantic analysis. Moving semantic analysis into the classes as outlined in the commit message will hopefully make that obvious enough to permit dropping the assertions.
Quoted 24 lines
Quoted 24 lines
scripts/qapi.py +24
-0+ +class QAPISchemaArrayType(QAPISchemaType): + def __init__(self, name, info, element_type): + QAPISchemaType.__init__(self, name, info) + assert isinstance(element_type, str) + self.element_type_name = element_type + self.element_type = None + def check(self, schema): + self.element_type = schema.lookup_type(self.element_type_name) + assert self.element_type + +class QAPISchemaObjectType(QAPISchemaType): + def __init__(self, name, info, base, local_members, variants): + QAPISchemaType.__init__(self, name, info) + assert base == None or isinstance(base, str) + for m in local_members: + assert isinstance(m, QAPISchemaObjectTypeMember) + if variants != None: + assert isinstance(variants, QAPISchemaObjectTypeVariants) + self.base_name = base + self.base = None + self.local_members = local_members + self.variants = variants + self.members = None
Quoted 8 lines
scripts/qapi.py def check_exprs(exprs): · line 810
Quoted 5 lines
scripts/qapi.py +5
-0+ def check(self, schema): + if self.members: + return # already checked + assert self.members == None # not running in cycles + self.members = False # mark as being checkedInteresting, but makes sense (since we allow self-referential nesting, populating our own members may require revisiting the same type).
Yes. Another way to put it: for a recursive type like this, we better check the recursion terminates.
Quoted 3 lines
Cute
that python allows both None and False as distinct non-true settings, so that you end up using the variable as a tri-state.
Yup :)
Quoted 8 lines
scripts/qapi.py def check_exprs(exprs): · line 813
Quoted 3 lines
scripts/qapi.py +3
-0+ if self.base_name: + self.base = schema.lookup_type(self.base_name) + self.base.check(schema)Do you need 'assert self.base' here, similar to how you did it in
QAPISchemaArrayType.check()? I guess you'll still get a python exception thatNone.check()doesn't exist if thelookup_typefailed, so it just depends on what error message you want.
I don't see a hard need, but consistency is nice. Since I generally assert elsewhere, I'll add an assertion here.
Quoted 5 lines
Quoted 1 line
+ members = list(self.base.members)
For that matter, do you want to assert
isinstance(self.base, QAPISchemaObjectType), sincelookup_typecan return other types, but other child classes of QAPISchemaType do not have a .members?
Yes, that's the appropriate assertion.
Quoted 6 lines
Quoted 6 lines
scripts/qapi.py +6
-0+ else: + members = [] + seen = {} + for m in members: + seen[m.name] = m + for m in self.local_members:
Quoted 8 lines
scripts/qapi.py def check_exprs(exprs): · line 826
Quoted 6 lines
scripts/qapi.py +6
-0+ m.check(schema, members, seen) + if self.variants: + self.variants.check(schema, members, seen) + self.members = members + +class QAPISchemaObjectTypeMember(object):Is 'object' the right base class, or should it be QAPISchemaEntity?
No, it's really object. A schema's entities can be object types, other types, commands and events, but not members. A schema doesn't have "members", only an object type has.
Quoted 6 lines
scripts/qapi.py def check_exprs(exprs): · line 829
Quoted 3 lines
scripts/qapi.py +3
-0+ def __init__(self, name, typ, optional): + assert isinstance(name, str) + assert isinstance(typ, str)I guess that means all callers flatten array types into an unambiguous string first.
Yes.
Conscious design decision: every type in the internal representation has a unique name. If the concrete syntax lets you define an anonymous type, we make up a name in the internal representation. One less special case to worry about.
Quoted 7 lines
Quoted 7 lines
scripts/qapi.py +7
-0+ assert isinstance(optional, bool) + self.name = name + self.type_name = typ + self.type = None + self.optional = optional + def check(self, schema, all_members, seen): + assert self.name not in seen
Quoted 8 lines
scripts/qapi.py def check_exprs(exprs): · line 842
Quoted 6 lines
scripts/qapi.py +6
-0+ self.type = schema.lookup_type(self.type_name) + assert self.type + all_members.append(self) + seen[self.name] = self + +class QAPISchemaObjectTypeVariants(object):Is 'object' the right base class, or should it be QAPISchemaEntity?
As for QAPISchemaObjectTypeMember above, object is appropriate.
Quoted 13 lines
scripts/qapi.py def check_exprs(exprs): · line 854
Quoted 12 lines
scripts/qapi.py +12
-0+ def __init__(self, tag_name, tag_enum, variants): + assert tag_name == None or isinstance(tag_name, str) + assert tag_enum == None or isinstance(tag_enum, str) + for v in variants: + assert isinstance(v, QAPISchemaObjectTypeVariant) + self.tag_name = tag_name + if tag_name: + assert not tag_enum + self.tag_member = None + else: + self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum, + False)
Quoted 3 lines
Nice - the tag is either part of the base struct (flat union) or implicitly created (plain union). By the time we are done with
check(), we then have both its type (possibly generated implicitly) and its name.
... and we don't have to know where they came from anymore.
Quoted 30 lines
scripts/qapi.py def check_exprs(exprs): · line 860
Quoted 6 lines
scripts/qapi.py +6
-0+ self.variants = variants + def check(self, schema, members, seen): + if self.tag_name: + self.tag_member = seen[self.tag_name] + else: + self.tag_member.check(schema, members, seen)A little bit tricky here in that flat unions (
tag_namewas set) must find the tag from the members already loaded from the base class, while plain unions (tag_nameis None) add their own implicit member into the union. But it works.Quoted 6 lines
scripts/qapi.py +6
-0+ assert isinstance(self.tag_member.type, QAPISchemaEnumType) + for v in self.variants: + vseen = dict(seen) + v.check(schema, self.tag_member.type, vseen) + +class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):scripts/qapi.py def check_exprs(exprs): · line 876
Quoted 10 lines
scripts/qapi.py +10
-0+ def __init__(self, name, typ, flat): + QAPISchemaObjectTypeMember.__init__(self, name, typ, False) + assert isinstance(flat, bool) + self.flat = flat + def check(self, schema, tag_type, seen): + QAPISchemaObjectTypeMember.check(self, schema, [], seen) + assert self.name in tag_type.values + if self.flat: + self.type.check(schema) + assert isinstance(self.type, QAPISchemaObjectType)Do we really want to be tracking self.flat for each variant? After all,
Quoted 8 lines
docs/qapi-code-gen.txtalready describes the mapping from simple union into flat union (as if the flat union had a base class with single member 'kind' of the right type, then each branch of the union composed of an implicit object with a lone member 'data' of the correct type). In other words, is it any better to just normalize into that form now, such that each QAPISchemaObjectTypeVariant is merely a (often one-element) list of name:type members being added to the overall QAPISchemaObject?
I tried to do exactly that, but got bogged down in special cases and copped out. Then I went on vacation, and now I don't remember the exact problems anymore %-}
I guess / hope it's just relatively pointless differences in the generated C code I didn't want to get rid of at this time. The series is long and hairy enough as it is...
Quoted 2 lines
But I guess it remains to be seen how you use
self.flat before knowing if it is worth normalizing away from it.
At least introspect.json is oblivious of it.
Quoted 42 lines
scripts/qapi.py def check_exprs(exprs): · line 884
Quoted 8 lines
scripts/qapi.py +8
-0+ +class QAPISchemaAlternateType(QAPISchemaType): + def __init__(self, name, info, variants): + QAPISchemaType.__init__(self, name, info) + assert isinstance(variants, QAPISchemaObjectTypeVariants) + assert not variants.tag_name + for v in variants.variants: + assert not v.flatHmm. You are using .flat after all.
scripts/qapi.py def check_exprs(exprs): · line 893
Quoted 9 lines
scripts/qapi.py +9
-0+ self.variants = variants + def check(self, schema): + self.variants.check(schema, [], {}) + +class QAPISchemaCommand(QAPISchemaEntity): + def __init__(self, name, info, args, rets, gen, success_response): + QAPISchemaEntity.__init__(self, name, info) + assert not args or isinstance(args, str) + assert not rets or isinstance(rets, str)Again, this means that the caller has already had to convert qapi dictionaries into an implicit type and pass in that type name here. Works for me.
Quoted 6 lines
scripts/qapi.py +6
-0+ self.args_name = args + self.args = None + self.rets_name = rets + self.rets = None + self.gen = gen + self.success_response = success_responsescripts/qapi.py def check_exprs(exprs): · line 909
Quoted 10 lines
scripts/qapi.py +10
-0+ def check(self, schema): + if self.args_name: + self.args = schema.lookup_type(self.args_name) + assert isinstance(self.args, QAPISchemaObjectType) + assert not self.args.variants # not implemented + self.args.check(schema) + if self.rets_name: + self.rets = schema.lookup_type(self.rets_name) + assert isinstance(self.rets, QAPISchemaType) + self.rets.check(schema)
Quoted 3 lines
Matches our existing difference in requiring a dictionary for arguments, but allowing the return of any type (although we now have a whitelist to prevent too many additions of non-dictionaries).
Yes. In my opinion, the difference between arguments and returns is a design blemish (a function maps from domain to codomain, and both are just sets, so any differences are likely pointless baggage), but we need to work with what we've got.
Quoted 33 lines
Quoted 33 lines
scripts/qapi.py +33
-0+ +class QAPISchemaEvent(QAPISchemaEntity): + def __init__(self, name, info, data): + QAPISchemaEntity.__init__(self, name, info) + assert not data or isinstance(data, str) + self.data_name = data + self.data = None + def check(self, schema): + if self.data_name: + self.data = schema.lookup_type(self.data_name) + assert isinstance(self.data, QAPISchemaObjectType) + self.data.check(schema) + +class QAPISchema(object): + def __init__(self, fname): + try: + self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs) + except (QAPISchemaError, QAPIExprError), err: + print >>sys.stderr, err + exit(1) + self.entity_dict = {} + self._def_predefineds() + self._def_exprs() + self.check() + + def get_exprs(self): + return [expr_elem['expr'] for expr_elem in self.exprs] + + def _def_entity(self, ent): + assert ent.name not in self.entity_dict + self.entity_dict[ent.name] = ent + + def lookup_entity(self, name, typ=None):
Quoted 8 lines
scripts/qapi.py def check_exprs(exprs): · line 946
Quoted 4 lines
scripts/qapi.py +4
-0+ ent = self.entity_dict.get(name) + if typ and not isinstance(ent, typ): + return None + return entAh, so you enforce a shared namespace between types, commands, and events (all three are in self.entity_dict), but can use the typ parameter to allow limiting a lookup to just types.
Yes. It's a convenience feature.
Quoted 25 lines
Quoted 25 lines
scripts/qapi.py +25
-0+ + def lookup_type(self, name): + return self.lookup_entity(name, QAPISchemaType) + + def _def_builtin_type(self, name): + self._def_entity(QAPISchemaBuiltinType(name)) + if name != '**': + self._make_array_type(name) # TODO really needed? + + def _def_predefineds(self): + for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64', + 'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']: + self._def_builtin_type(t) + + def _make_implicit_enum_type(self, name, values): + name = name + 'Kind' + self._def_entity(QAPISchemaEnumType(name, None, values)) + return name + + def _make_array_type(self, element_type): + name = element_type + 'List' + if not self.lookup_type(name): + self._def_entity(QAPISchemaArrayType(name, None, element_type)) + return name +
Quoted 8 lines
scripts/qapi.py def check_exprs(exprs): · line 975
Quoted 4 lines
scripts/qapi.py +4
-0+ def _make_implicit_object_type(self, name, role, members): + name = ':obj-%s-%s' % (name, role) + self._def_entity(QAPISchemaObjectType(name, None, None, members, None)) + return nameWe may extend the qapi .json syntax to allow anonymous types in more places, but so far, this matches that all existing use of anonymous types are for structs, not unions.
Yes.
If we unify struct and union in the concrete syntax like we do in this internal representation, then anonymous unions (really: anonymous objects with variants) may well fall out naturally.
Quoted 45 lines
Quoted 45 lines
scripts/qapi.py +45
-0+ + def _def_enum_type(self, expr, info): + name = expr['enum'] + data = expr['data'] + self._def_entity(QAPISchemaEnumType(name, info, data)) + self._make_array_type(name) # TODO really needed? + + def _make_member(self, name, typ): + optional = False + if name.startswith('*'): + name = name[1:] + optional = True + if isinstance(typ, list): + assert len(typ) == 1 + typ = self._make_array_type(typ[0]) + return QAPISchemaObjectTypeMember(name, typ, optional) + + def _make_members(self, data): + return [self._make_member(key, data[key]) for key in data.keys()] + + def _def_struct_type(self, expr, info): + name = expr['struct'] + base = expr.get('base') + data = expr['data'] + self._def_entity(QAPISchemaObjectType(name, info, base, + self._make_members(data), + None)) + self._make_array_type(name) # TODO really needed? + + def _make_flat_variant(self, case, typ): + return QAPISchemaObjectTypeVariant(case, typ, True) + + def _make_flat_variants(self, tag_name, data): + variants = [self._make_flat_variant(key, data[key]) + for key in data.keys()] + return QAPISchemaObjectTypeVariants(tag_name, None, variants) + + def _make_simple_variant(self, case, typ): + if isinstance(typ, list): + assert len(typ) == 1 + typ = self._make_array_type(typ[0]) + return QAPISchemaObjectTypeVariant(case, typ, False) + + def _make_simple_variants(self, type_name, data): + variants = [self._make_simple_variant(key, data[key])
Quoted 8 lines
scripts/qapi.py def check_exprs(exprs): · line 1024
Quoted 4 lines
scripts/qapi.py +4
-0+ for key in data.keys()] + enum = self._make_implicit_enum_type(type_name, + [v.name for v in variants]) + return QAPISchemaObjectTypeVariants(None, enum, variants)Again, I wonder if normalizing simple unions into flat unions (to get rid of the need for tracking .flat) would make later use any easier, or if it would just make introspection QMP output more verbose.
They *are* normalized in introspection output!
Quoted 27 lines
Quoted 27 lines
scripts/qapi.py +27
-0+ + def _def_union_type(self, expr, info): + name = expr['union'] + data = expr['data'] + tag_name = expr.get('discriminator') + if tag_name: + base = expr['base'] + variants = self._make_flat_variants(tag_name, data) + else: + base = None + variants = self._make_simple_variants(name, data) + self._def_entity(QAPISchemaObjectType(name, info, base, + self._make_members(OrderedDict()), + variants)) + self._make_array_type(name) # TODO really needed? + + def _def_alternate_type(self, expr, info): + name = expr['alternate'] + data = expr['data'] + self._def_entity(QAPISchemaAlternateType(name, info, + self._make_simple_variants(name, data))) + self._make_array_type(name) # TODO really needed? + + def _def_command(self, expr, info): + name = expr['command'] + args = expr.get('data') + rets = expr.get('returns')
Quoted 8 lines
scripts/qapi.py def check_exprs(exprs): · line 1056
Quoted 5 lines
scripts/qapi.py +5
-0+ gen = expr.get('gen', True) + success_response = expr.get('success-response', True) + if args and not isinstance(args, str): + args = self._make_implicit_object_type(name, 'args', + self._make_members(args))If I write { 'command':'Foo', 'data':{} }, does that try and create an implicit type, with no members?
tests/qapi-schema/qapi-schema-test.json tests this:
{ 'command': 'user_def_cmd', 'data': {} }args = expr.get('data') gets an empty OrderedDict().
Since an empty dictionary counts as false, we skip calling _make_implicit_object_type(), and pass the empty dictionary to QAPISchemaCommand(). Works, because the class consistently treats any false args the same as None. tests/qapi-schema/qapi-schema-test.out:
command user_def_cmd None -> None
gen=True success_response=TrueI'll consider cleaning this up a bit.
Quoted 46 lines
I'm wondering if it is any simpler to
write args =expr.get('data', {}), and have_make_implicit_object_type()do the right thing when presented with an empty list of members.Quoted 29 lines
scripts/qapi.py +29
-0+ if rets and isinstance(rets, list): + assert len(rets) == 1 + rets = self._make_array_type(rets[0]) + elif rets and not isinstance(rets, str): + rets = self._make_implicit_object_type(name, 'rets', + self._make_members(rets)) + self._def_entity(QAPISchemaCommand(name, info, args, rets, gen, + success_response)) + + def _def_event(self, expr, info): + name = expr['event'] + data = expr.get('data') + if data and not isinstance(data, str): + data = self._make_implicit_object_type(name, 'data', + self._make_members(data)) + self._def_entity(QAPISchemaEvent(name, info, data)) + + def _def_exprs(self): + for expr_elem in self.exprs: + expr = expr_elem['expr'] + info = expr_elem['info'] + if 'enum' in expr: + self._def_enum_type(expr, info) + elif 'struct' in expr: + self._def_struct_type(expr, info) + elif 'union' in expr: + self._def_union_type(expr, info) + elif 'alternate' in expr: + self._def_alternate_type(expr, info)scripts/qapi.py def check_exprs(exprs): · line 1095
Quoted 10 lines
scripts/qapi.py +10
-0+ elif 'command' in expr: + self._def_command(expr, info) + elif 'event' in expr: + self._def_event(expr, info) + else: + assert False + + def check(self): + for ent in self.entity_dict.values(): + ent.check(self)Overall seems like a very nice layout.
Quoted 3 lines
I'll reserve R-b until I see how the rest of the series uses this object hierarchy, but it looks like you won't have to change very much of this patch when you do your next spin of the series.
Thanks!
Re: [Qemu-devel] [PATCH RFC v2 21/47] qapi: New QAPISchema intermediate reperesentation
Quoted 34 lines · Markus Armbruster
Quoted 20 lines
Quoted 9 lines
Create a bunch of classes to represent QAPI schemata.
Have the QAPISchema initializer call the parser, then walk the syntax tree to create the new internal representation, and finally perform semantic analysis.
scripts/qapi.py +1
-0+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):scripts/qapi.py def check_exprs(exprs): · line 876
Quoted 10 lines
scripts/qapi.py +10
-0+ def __init__(self, name, typ, flat): + QAPISchemaObjectTypeMember.__init__(self, name, typ, False) + assert isinstance(flat, bool) + self.flat = flat + def check(self, schema, tag_type, seen): + QAPISchemaObjectTypeMember.check(self, schema, [], seen) + assert self.name in tag_type.values + if self.flat: + self.type.check(schema) + assert isinstance(self.type, QAPISchemaObjectType)Quoted 9 lines
Do we really want to be tracking self.flat for each variant? After all,
docs/qapi-code-gen.txtalready describes the mapping from simple union into flat union (as if the flat union had a base class with single member 'kind' of the right type, then each branch of the union composed of an implicit object with a lone member 'data' of the correct type). In other words, is it any better to just normalize into that form now, such that each QAPISchemaObjectTypeVariant is merely a (often one-element) list of name:type members being added to the overall QAPISchemaObject?I tried to do exactly that, but got bogged down in special cases and copped out. Then I went on vacation, and now I don't remember the exact problems anymore %-}
Quoted 8 lines
I guess / hope it's just relatively pointless differences in the generated C code I didn't want to get rid of at this time. The series is long and hairy enough as it is...
Quoted 2 lines
But I guess it remains to be seen how you use
self.flat before knowing if it is worth normalizing away from it.At least introspect.json is oblivious of it.
Yeah, that was my conclusion by the end of the series - we still had enough special cases where we generate different code for simple unions than for flat unions. It would be possible to merge the generator and simplify things further, but at this point, it is best done as a follow-up series because it will touch lots of C code (anything that uses a simple union would have to convert to the common representation).
And you actually did reference .flat in the patch that first added qapi-introspect.py (good that it did not leak to the introspection output, but it did have to be considered when figuring out what to output); again, something you may want to rework before polishing this into v3, or something you may end up just documenting as a possible cleanup for a future series. But after having reviewed the whole series now, I'm able to live with your use of .flat, if only because it makes the initial conversion faster.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
scripts/qapi.py +6
-0+ + def lookup_entity(self, name, typ=None): + ent = self.entity_dict.get(name) + if typ and not isinstance(ent, typ): + return None + return ent
Quoted 5 lines
Quoted 3 lines
Ah, so you enforce a shared namespace between types, commands, and events (all three are in self.entity_dict), but can use the typ parameter to allow limiting a lookup to just types.
Yes. It's a convenience feature.
Documentation comments never hurt :) (You did mention in the cover letter that part of the reason this was still RFC was that it was lacking documentation)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 22/47 qapi: QAPISchema code generation helper methods
In reply to Markus ArmbrusterNew methods c_name(), c_type(), c_null(), json_type(), alternate_qtype().
--- scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 65 insertions(+), 7 deletions(-)
scripts/qapi.py +65 -7
@@ -763,15 +763,47 @@ class QAPISchemaEntity(object): assert isinstance(name, str) self.name = name self.info = info + def c_name(self): + return c_name(self.name) def check(self, schema): pass class QAPISchemaType(QAPISchemaEntity): - pass + def c_type(self, is_param=False): + return c_name(self.name) + pointer_suffix + def c_null(self): + return 'NULL' + def json_type(self): + return None + def alternate_qtype(self): + json2qtype = { + 'string': 'QTYPE_QSTRING', + 'number': 'QTYPE_QFLOAT', + 'int': 'QTYPE_QINT', + 'boolean': 'QTYPE_QBOOL', + 'object': 'QTYPE_QDICT' + } + return json2qtype.get(self.json_type()) class QAPISchemaBuiltinType(QAPISchemaType): - def __init__(self, name): + def __init__(self, name, json_type, c_type, c_null): QAPISchemaType.__init__(self, name, None) + assert not c_type or isinstance(c_type, str) + assert json_type in ('string', 'number', 'int', 'boolean', 'null', + 'value') + self.json_type_name = json_type + self.c_type_name = c_type + self.c_null_val = c_null + def c_name(self): + return self.name + def c_type(self, is_param=False): + if is_param and self.name == 'str': + return 'const ' + self.c_type_name + return self.c_type_name + def c_null(self): + return self.c_null_val + def json_type(self): + return self.json_type_name class QAPISchemaEnumType(QAPISchemaType): def __init__(self, name, info, values):
@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType): for v in values: assert isinstance(v, str) self.values = values + def c_type(self, is_param=False): + return c_name(self.name) + def c_null(self): + return c_enum_const(self.name, self.values[0]) + def json_type(self): + return 'string' class QAPISchemaArrayType(QAPISchemaType): def __init__(self, name, info, element_type):
@@ -822,6 +860,14 @@ class QAPISchemaObjectType(QAPISchemaType): if self.variants: self.variants.check(schema, members, seen) self.members = members + def c_name(self): + assert self.info + return QAPISchemaType.c_name(self) + def c_type(self, is_param=False): + assert self.info + return QAPISchemaType.c_type(self) + def json_type(self): + return 'object' class QAPISchemaObjectTypeMember(object): def __init__(self, name, typ, optional):
@@ -948,15 +994,27 @@ class QAPISchema(object): def lookup_type(self, name): return self.lookup_entity(name, QAPISchemaType) - def _def_builtin_type(self, name): - self._def_entity(QAPISchemaBuiltinType(name)) + def _def_builtin_type(self, name, json_type, c_type, c_null): + self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null)) if name != '**': self._make_array_type(name) # TODO really needed? def _def_predefineds(self): - for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64', - 'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']: - self._def_builtin_type(t) + for t in [('str', 'string', 'char' + pointer_suffix, 'NULL'), + ('number', 'number', 'double', '0'), + ('int', 'int', 'int64_t', '0'), + ('int8', 'int', 'int8_t', '0'), + ('int16', 'int', 'int16_t', '0'), + ('int32', 'int', 'int32_t', '0'), + ('int64', 'int', 'int64_t', '0'), + ('uint8', 'int', 'uint8_t', '0'), + ('uint16', 'int', 'uint16_t', '0'), + ('uint32', 'int', 'uint32_t', '0'), + ('uint64', 'int', 'uint64_t', '0'), + ('size', 'int', 'uint64_t', '0'), + ('bool', 'boolean', 'bool', 'false'), + ('**', 'value', None, None)]: + self._def_builtin_type(*t) def _make_implicit_enum_type(self, name, values): name = name + 'Kind'
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods
Quoted 7 lines · Markus Armbruster
New methods
c_name(),c_type(),c_null(),json_type(),alternate_qtype().Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 65 insertions(+), 7 deletions(-)
Still unused in the rest of the code, so no change to generated code yet; and thanks for separating this apart from the creation of the new object hierarchy.
scripts/qapi.py class QAPISchemaEntity(object): · line 773
Quoted 4 lines
class QAPISchemaType(QAPISchemaEntity):
scripts/qapi.py +2
-1- pass + def c_type(self, is_param=False): + return c_name(self.name) + pointer_suffix
At first, I thought is_param was unused; but reviewing the rest of the patch shows that subclasses are overriding it, so declaring it here is necessary for polymorphic calls to supply the argument and have it properly acted on.
scripts/qapi.py class QAPISchemaEntity(object): · line 785
Quoted 12 lines
scripts/qapi.py +12
-0+ def c_null(self): + return 'NULL' + def json_type(self): + return None + def alternate_qtype(self): + json2qtype = { + 'string': 'QTYPE_QSTRING', + 'number': 'QTYPE_QFLOAT', + 'int': 'QTYPE_QINT', + 'boolean': 'QTYPE_QBOOL', + 'object': 'QTYPE_QDICT' + }
Are there any style rules on whether to include or omit a trailing comma?
Quoted 4 lines
scripts/qapi.py +1
-1+ return json2qtype.get(self.json_type()) class QAPISchemaBuiltinType(QAPISchemaType): - def __init__(self, name):
scripts/qapi.py class QAPISchemaEntity(object): · line 798
Quoted 10 lines
scripts/qapi.py +9
-0+ def __init__(self, name, json_type, c_type, c_null): QAPISchemaType.__init__(self, name, None) + assert not c_type or isinstance(c_type, str) + assert json_type in ('string', 'number', 'int', 'boolean', 'null', + 'value') + self.json_type_name = json_type + self.c_type_name = c_type + self.c_null_val = c_null + def c_name(self): + return self.name
At first, I thought this was redundant with QAPISchemaEntity.c_name(), until I realized that you are intentionally overriding things here to NOT call global c_name() (so that 'int' does not get munged the 'q_int' in the generated C code). Nice, once I figured it out.
Quoted 12 lines
scripts/qapi.py +8
-0+ def c_type(self, is_param=False): + if is_param and self.name == 'str': + return 'const ' + self.c_type_name + return self.c_type_name + def c_null(self): + return self.c_null_val + def json_type(self): + return self.json_type_name class QAPISchemaEnumType(QAPISchemaType): def __init__(self, name, info, values):
scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 815
Quoted 5 lines
scripts/qapi.py +2
-0@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType): for v in values: assert isinstance(v, str) self.values = values + def c_type(self, is_param=False): + return c_name(self.name)
Probably showing my unfamiliarity with python polymorphism/overriding rules - is it necessary to declare an otherwise-unused optional parameter here if we are overriding a base method where the parameter has a sane default for our needs? That is, would c_type(self) here properly override c_type(self, is_param=False) in the parent class?
Quoted 8 lines
scripts/qapi.py +4
-0+ def c_null(self): + return c_enum_const(self.name, self.values[0]) + def json_type(self): + return 'string' class QAPISchemaArrayType(QAPISchemaType): def __init__(self, name, info, element_type):
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 868
Quoted 9 lines
scripts/qapi.py +6
-0@@ -822,6 +860,14 @@ class QAPISchemaObjectType(QAPISchemaType): if self.variants: self.variants.check(schema, members, seen) self.members = members + def c_name(self): + assert self.info + return QAPISchemaType.c_name(self) + def c_type(self, is_param=False): + assert self.info + return QAPISchemaType.c_type(self)
So, to make sure I understand correctly, implicit types (those generated from anonymous dictionaries in a command or event) have info=None, and the assertions here are making sure we never try to learn the c_name or c_type of those implicit types in generated code, but rather use them solely in generated code that just enumerates the members (C struct declarations, parameter list to functions that implement commands).
Quoted 12 lines
scripts/qapi.py +3
-2+ def json_type(self): + return 'object' class QAPISchemaObjectTypeMember(object): def __init__(self, name, typ, optional):@@ -948,15 +994,27 @@ class QAPISchema(object): def lookup_type(self, name): return self.lookup_entity(name, QAPISchemaType) - def _def_builtin_type(self, name): - self._def_entity(QAPISchemaBuiltinType(name)) + def _def_builtin_type(self, name, json_type, c_type, c_null):
scripts/qapi.py class QAPISchema(object): · line 1004
Quoted 10 lines
scripts/qapi.py +3
-3+ self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null)) if name != '**': self._make_array_type(name) # TODO really needed? def _def_predefineds(self): - for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64', - 'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']: - self._def_builtin_type(t) + for t in [('str', 'string', 'char' + pointer_suffix, 'NULL'), + ('number', 'number', 'double', '0'),
Do you want '0.0' as the default here (yes, promoting an int 0 to double does the right thing, but I like making it obvious that I know I'm using floating point).
Quoted 7 lines
scripts/qapi.py +6
-0+ ('int', 'int', 'int64_t', '0'), + ('int8', 'int', 'int8_t', '0'), + ('int16', 'int', 'int16_t', '0'), + ('int32', 'int', 'int32_t', '0'), + ('int64', 'int', 'int64_t', '0'), + ('uint8', 'int', 'uint8_t', '0'),
scripts/qapi.py class QAPISchema(object): · line 1020
Quoted 10 lines
scripts/qapi.py +7
-0+ ('uint16', 'int', 'uint16_t', '0'), + ('uint32', 'int', 'uint32_t', '0'), + ('uint64', 'int', 'uint64_t', '0'), + ('size', 'int', 'uint64_t', '0'), + ('bool', 'boolean', 'bool', 'false'), + ('**', 'value', None, None)]: + self._def_builtin_type(*t) def _make_implicit_enum_type(self, name, values): name = name + 'Kind'
Tweaks you make (if any) based on the above comments should be minor, so I'm fine with adding:
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods
Quoted 19 lines · Eric Blake
Quoted 8 lines · Markus Armbruster
New methods
c_name(),c_type(),c_null(),json_type(),alternate_qtype().Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 65 insertions(+), 7 deletions(-)Still unused in the rest of the code, so no change to generated code yet; and thanks for separating this apart from the creation of the new object hierarchy.
scripts/qapi.py class QAPISchemaEntity(object): · line 773
Quoted 4 lines
class QAPISchemaType(QAPISchemaEntity):
scripts/qapi.py +2
-1- pass + def c_type(self, is_param=False): + return c_name(self.name) + pointer_suffix
Quoted 4 lines
At first, I thought
is_paramwas unused; but reviewing the rest of the patch shows that subclasses are overriding it, so declaring it here is necessary for polymorphic calls to supply the argument and have it properly acted on.
Yup.
Quoted 13 lines
scripts/qapi.py class QAPISchemaEntity(object): · line 785
Quoted 12 lines
scripts/qapi.py +12
-0+ def c_null(self): + return 'NULL' + def json_type(self): + return None + def alternate_qtype(self): + json2qtype = { + 'string': 'QTYPE_QSTRING', + 'number': 'QTYPE_QFLOAT', + 'int': 'QTYPE_QINT', + 'boolean': 'QTYPE_QBOOL', + 'object': 'QTYPE_QDICT' + }
Quoted 1 line
Are there any style rules on whether to include or omit a trailing comma?
PEP8 appears to be silent on it.
Quoted 15 lines
Quoted 4 lines
scripts/qapi.py +1
-1+ return json2qtype.get(self.json_type()) class QAPISchemaBuiltinType(QAPISchemaType): - def __init__(self, name):scripts/qapi.py class QAPISchemaEntity(object): · line 798
Quoted 10 lines
scripts/qapi.py +9
-0+ def __init__(self, name, json_type, c_type, c_null): QAPISchemaType.__init__(self, name, None) + assert not c_type or isinstance(c_type, str) + assert json_type in ('string', 'number', 'int', 'boolean', 'null', + 'value') + self.json_type_name = json_type + self.c_type_name = c_type + self.c_null_val = c_null + def c_name(self): + return self.name
Quoted 4 lines
At first, I thought this was redundant with
QAPISchemaEntity.c_name(), until I realized that you are intentionally overriding things here to NOT call globalc_name()(so that 'int' does not get munged the 'q_int' in the generated C code). Nice, once I figured it out.
I'll consider adding comments to help future readers figure it out.
Quoted 9 lines
Quoted 9 lines
scripts/qapi.py +8
-0+ def c_type(self, is_param=False): + if is_param and self.name == 'str': + return 'const ' + self.c_type_name + return self.c_type_name + def c_null(self): + return self.c_null_val + def json_type(self): + return self.json_type_name
Quoted 14 lines
scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 815
Quoted 8 lines
class QAPISchemaEnumType(QAPISchemaType):
def __init__(self, name, info, values):scripts/qapi.py +2
-0@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType): for v in values: assert isinstance(v, str) self.values = values + def c_type(self, is_param=False): + return c_name(self.name)Probably showing my unfamiliarity with python polymorphism/overriding rules - is it necessary to declare an otherwise-unused optional parameter here if we are overriding a base method where the parameter has a sane default for our needs? That is, would
c_type(self)here properly overridec_type(self, is_param=False)in the parent class?
I don't know. But the way I wrote it, we don't need to know :)
Quoted 5 lines
Quoted 5 lines
scripts/qapi.py +4
-0+ def c_null(self): + return c_enum_const(self.name, self.values[0]) + def json_type(self): + return 'string'
Quoted 15 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 868
Quoted 12 lines
scripts/qapi.py +6
-0class QAPISchemaArrayType(QAPISchemaType): def __init__(self, name, info, element_type):@@ -822,6 +860,14 @@ class QAPISchemaObjectType(QAPISchemaType): if self.variants: self.variants.check(schema, members, seen) self.members = members + def c_name(self): + assert self.info + return QAPISchemaType.c_name(self) + def c_type(self, is_param=False): + assert self.info + return QAPISchemaType.c_type(self)So, to make sure I understand correctly, implicit types (those generated from anonymous dictionaries in a command or event) have info=None,
Correct.
Quoted 5 lines
and
the assertions here are making sure we never try to learn thec_nameorc_typeof those implicit types in generated code, but rather use them solely in generated code that just enumerates the members (C struct declarations, parameter list to functions that implement commands).
Correct again.
Quoted 16 lines
Quoted 16 lines
scripts/qapi.py +4
-2+ def json_type(self): + return 'object' class QAPISchemaObjectTypeMember(object): def __init__(self, name, typ, optional):@@ -948,15 +994,27 @@ class QAPISchema(object): def lookup_type(self, name): return self.lookup_entity(name, QAPISchemaType) - def _def_builtin_type(self, name): - self._def_entity(QAPISchemaBuiltinType(name)) + def _def_builtin_type(self, name, json_type, c_type, c_null): + self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null)) if name != '**': self._make_array_type(name) # TODO really needed?
Quoted 10 lines
scripts/qapi.py class QAPISchema(object): · line 1004
Quoted 6 lines
scripts/qapi.py +2
-3def _def_predefineds(self): - for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64', - 'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']: - self._def_builtin_type(t) + for t in [('str', 'string', 'char' + pointer_suffix, 'NULL'), + ('number', 'number', 'double', '0'),Do you want '0.0' as the default here (yes, promoting an int 0 to double does the right thing, but I like making it obvious that I know I'm using floating point).
The '0' here is used as initializer, i.e. we generate things like
double d = 0;
Changing it to '0.0' will instead generate
double d = 0.0;
No strong preference on my part.
Quoted 14 lines
Quoted 14 lines
scripts/qapi.py +13
-0+ ('int', 'int', 'int64_t', '0'), + ('int8', 'int', 'int8_t', '0'), + ('int16', 'int', 'int16_t', '0'), + ('int32', 'int', 'int32_t', '0'), + ('int64', 'int', 'int64_t', '0'), + ('uint8', 'int', 'uint8_t', '0'), + ('uint16', 'int', 'uint16_t', '0'), + ('uint32', 'int', 'uint32_t', '0'), + ('uint64', 'int', 'uint64_t', '0'), + ('size', 'int', 'uint64_t', '0'), + ('bool', 'boolean', 'bool', 'false'), + ('**', 'value', None, None)]: + self._def_builtin_type(*t)
Quoted 8 lines
scripts/qapi.py class QAPISchema(object): · line 1020
Quoted 3 lines
scripts/qapi.py +0
-0def _make_implicit_enum_type(self, name, values): name = name + 'Kind'Tweaks you make (if any) based on the above comments should be minor, so I'm fine with adding:
Reviewed-by Eric Blake <[email protected]>
Thanks!
Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods
In reply to Markus ArmbrusterQuoted 8 lines · Markus Armbruster
New methods
c_name(),c_type(),c_null(),json_type(),alternate_qtype().Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 65 insertions(+), 7 deletions(-)
I just noticed:
scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 817
Quoted 8 lines
scripts/qapi.py +4
-0@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType): for v in values: assert isinstance(v, str) self.values = values + def c_type(self, is_param=False): + return c_name(self.name) + def c_null(self): + return c_enum_const(self.name, self.values[0])
What does this return for an empty enum, as in { 'enum':'Empty', 'data':[] }? Our testsuite proves we can do that, even if our normal .json code doesn't use it.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods
Quoted 13 lines · Eric Blake
Quoted 9 lines · Markus Armbruster
New methods
c_name(),c_type(),c_null(),json_type(),alternate_qtype().Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 65 insertions(+), 7 deletions(-)I just noticed:
Quoted 11 lines
scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 817
Quoted 8 lines
scripts/qapi.py +4
-0@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType): for v in values: assert isinstance(v, str) self.values = values + def c_type(self, is_param=False): + return c_name(self.name) + def c_null(self): + return c_enum_const(self.name, self.values[0])What does this return for an empty enum, as in { 'enum':'Empty', 'data':[] }?
I suspect self.values will be [] then, and self.values[0] will bomb.
Possible fixes:
* Outlaw empty enums
* Add the implicit MAX member to self.values[] (other code may have to skip it)
* Catch the special case here, and return the implicit MAX member.
Quoted 2 lines
Our testsuite proves we can do that, even if our normal
.json code doesn't use it.
tests/qapi-schema/enum-empty.json:{ 'enum': 'MyEnum', 'data': [ ] }
Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods
Quoted 29 lines · Markus Armbruster
Quoted 13 lines · Eric Blake
Quoted 9 lines · Markus Armbruster
New methods
c_name(),c_type(),c_null(),json_type(),alternate_qtype().Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 65 insertions(+), 7 deletions(-)I just noticed:
Quoted 11 lines
scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 817
Quoted 8 lines
scripts/qapi.py +4
-0@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType): for v in values: assert isinstance(v, str) self.values = values + def c_type(self, is_param=False): + return c_name(self.name) + def c_null(self): + return c_enum_const(self.name, self.values[0])What does this return for an empty enum, as in { 'enum':'Empty', 'data':[] }?
I suspect self.values will be [] then, and self.values[0] will bomb.
Quoted 8 lines
Possible fixes:
* Outlaw empty enums
* Add the implicit MAX member to self.values[] (other code may have to skip it)
* Catch the special case here, and return the implicit MAX member.
I'm leaning toward the third option here.
Quoted 5 lines
Quoted 3 lines
Our testsuite proves we can do that, even if our normal
.json code doesn't use it.tests/qapi-schema/enum-empty.json:{ 'enum': 'MyEnum', 'data': [ ] }
As I've mentioned elsewhere, most of our tests/qapi-schema/*.json merely cover whether the parser is okay with the input, while tests/qapi-schema/qapi-schema-test.json is the one that also tests that the generated C code works; so sounds like that test should be enhanced to cover some of these corner cases we have been considering in this series (empty enum, command with no arguments and no returns, and so forth).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods
Quoted 32 lines · Eric Blake
Quoted 31 lines · Markus Armbruster
Quoted 13 lines · Eric Blake
Quoted 9 lines · Markus Armbruster
New methods
c_name(),c_type(),c_null(),json_type(),alternate_qtype().Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 65 insertions(+), 7 deletions(-)I just noticed:
Quoted 11 lines
scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 817
Quoted 8 lines
scripts/qapi.py +4
-0@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType): for v in values: assert isinstance(v, str) self.values = values + def c_type(self, is_param=False): + return c_name(self.name) + def c_null(self): + return c_enum_const(self.name, self.values[0])What does this return for an empty enum, as in { 'enum':'Empty', 'data':[] }?
I suspect self.values will be [] then, and self.values[0] will bomb.
Possible fixes:
Quoted 8 lines
Quoted 6 lines
* Outlaw empty enums
* Add the implicit MAX member to self.values[] (other code may have to skip it)
* Catch the special case here, and return the implicit MAX member.
I'm leaning toward the third option here.
Me too now, because it's easy:
return c_enum_const(self.name, (self.values + [ 'MAX' ])[0])
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines
Our testsuite proves we can do that, even if our normal
.json code doesn't use it.tests/qapi-schema/enum-empty.json:{ 'enum': 'MyEnum', 'data': [ ] }
As I've mentioned elsewhere, most of our tests/qapi-schema/*.json merely cover whether the parser is okay with the input, while
tests/qapi-schema/qapi-schema-test.jsonis the one that also tests that the generated C code works; so sounds like that test should be enhanced to cover some of these corner cases we have been considering in this series (empty enum, command with no arguments and no returns, and so forth).
Yes, that would be nice, but I feel I have to leave it for another day.
RFC PATCH v2 23/47 qapi: New QAPISchemaVisitor
In reply to Markus ArmbrusterThe visitor will help keeping the code generation code simple and reasonably separated from QAPISchema details.
--- scripts/qapi.py | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+)
scripts/qapi.py +44 -0
@@ -767,6 +767,28 @@ class QAPISchemaEntity(object): return c_name(self.name) def check(self, schema): pass + def visit(self, visitor): + pass + +class QAPISchemaVisitor(object): + def visit_begin(self): + pass + def visit_end(self): + pass + def visit_builtin_type(self, name, info, json_type): + pass + def visit_enum_type(self, name, info, values): + pass + def visit_array_type(self, name, info, element_type): + pass + def visit_object_type(self, name, info, base, members, variants): + pass + def visit_alternate_type(self, name, info, variants): + pass + def visit_command(self, name, info, args, rets, gen, success_response): + pass + def visit_event(self, name, info, data): + pass class QAPISchemaType(QAPISchemaEntity): def c_type(self, is_param=False):
@@ -804,6 +826,8 @@ class QAPISchemaBuiltinType(QAPISchemaType): return self.c_null_val def json_type(self): return self.json_type_name + def visit(self, visitor): + visitor.visit_builtin_type(self.name, self.info, self.json_type()) class QAPISchemaEnumType(QAPISchemaType): def __init__(self, name, info, values):
@@ -817,6 +841,8 @@ class QAPISchemaEnumType(QAPISchemaType): return c_enum_const(self.name, self.values[0]) def json_type(self): return 'string' + def visit(self, visitor): + visitor.visit_enum_type(self.name, self.info, self.values) class QAPISchemaArrayType(QAPISchemaType): def __init__(self, name, info, element_type):
@@ -827,6 +853,8 @@ class QAPISchemaArrayType(QAPISchemaType): def check(self, schema): self.element_type = schema.lookup_type(self.element_type_name) assert self.element_type + def visit(self, visitor): + visitor.visit_array_type(self.name, self.info, self.element_type) class QAPISchemaObjectType(QAPISchemaType): def __init__(self, name, info, base, local_members, variants):
@@ -868,6 +896,9 @@ class QAPISchemaObjectType(QAPISchemaType): return QAPISchemaType.c_type(self) def json_type(self): return 'object' + def visit(self, visitor): + visitor.visit_object_type(self.name, self.info, + self.base, self.local_members, self.variants) class QAPISchemaObjectTypeMember(object): def __init__(self, name, typ, optional):
@@ -931,6 +962,8 @@ class QAPISchemaAlternateType(QAPISchemaType): self.variants = variants def check(self, schema): self.variants.check(schema, [], {}) + def visit(self, visitor): + visitor.visit_alternate_type(self.name, self.info, self.variants) class QAPISchemaCommand(QAPISchemaEntity): def __init__(self, name, info, args, rets, gen, success_response):
@@ -953,6 +986,9 @@ class QAPISchemaCommand(QAPISchemaEntity): self.rets = schema.lookup_type(self.rets_name) assert isinstance(self.rets, QAPISchemaType) self.rets.check(schema) + def visit(self, visitor): + visitor.visit_command(self.name, self.info, self.args, self.rets, + self.gen, self.success_response) class QAPISchemaEvent(QAPISchemaEntity): def __init__(self, name, info, data):
@@ -965,6 +1001,8 @@ class QAPISchemaEvent(QAPISchemaEntity): self.data = schema.lookup_type(self.data_name) assert isinstance(self.data, QAPISchemaObjectType) self.data.check(schema) + def visit(self, visitor): + visitor.visit_event(self.name, self.info, self.data) class QAPISchema(object): def __init__(self, fname):
@@ -1152,6 +1190,12 @@ class QAPISchema(object): for ent in self.entity_dict.values(): ent.check(self) + def visit(self, visitor): + visitor.visit_begin() + for name in sorted(self.entity_dict.keys()): + self.entity_dict[name].visit(visitor) + visitor.visit_end() + # # Code generation helpers #
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 23/47] qapi: New QAPISchemaVisitor
Quoted 2 lines · Markus Armbruster
The visitor will help keeping the code generation code simple and reasonably separated from QAPISchema details.
Still no change to generated code until later patches start using this.
Quoted 6 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 24/47 tests/qapi-schema: Convert test harness to QAPISchemaVisitor
In reply to Markus ArmbrusterThe old code prints the result of parsing (list of expression dictionaries), and partial results of semantic analysis (list of enum dictionaries, list of struct dictionaries).
The new code prints a trace of a schema visit, i.e. what the back-ends are going to use. Built-in and array types are omitted, because they're boring.
--- tests/qapi-schema/alternate-good.out | 15 +- tests/qapi-schema/comments.out | 4 +- tests/qapi-schema/data-member-array.out | 13 +- tests/qapi-schema/empty.out | 3 - tests/qapi-schema/enum-empty.out | 4 +- tests/qapi-schema/event-case.out | 4 +- tests/qapi-schema/flat-union-reverse-define.out | 21 +-- tests/qapi-schema/ident-with-escape.out | 7 +- tests/qapi-schema/include-relpath.out | 4 +- tests/qapi-schema/include-repetition.out | 4 +- tests/qapi-schema/include-simple.out | 4 +- tests/qapi-schema/indented-expr.out | 7 +- tests/qapi-schema/qapi-schema-test.out | 186 +++++++++++++++++------- tests/qapi-schema/returns-int.out | 5 +- tests/qapi-schema/test-qapi.py | 37 ++++- tests/qapi-schema/type-bypass.out | 7 +- 16 files changed, 210 insertions(+), 115 deletions(-)
tests/qapi-schema/alternate-good.out +9 -6
@@ -1,6 +1,9 @@ -[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))]), - OrderedDict([('enum', 'Enum'), ('data', ['hello', 'world'])]), - OrderedDict([('alternate', 'Alt'), ('data', OrderedDict([('value', 'int'), ('string', 'Enum'), ('struct', 'Data')]))])] -[{'enum_name': 'Enum', 'enum_values': ['hello', 'world']}, - {'enum_name': 'AltKind', 'enum_values': None}] -[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))])] +alternate Alt + case value: int flat=False + case string: Enum flat=False + case struct: Data flat=False +enum AltKind ['value', 'string', 'struct'] +object Data + member number: int optional=True + member name: str optional=True +enum Enum ['hello', 'world']
tests/qapi-schema/comments.out +1 -3
@@ -1,3 +1 @@ -[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])] -[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}] -[] +enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/data-member-array.out +8 -5
@@ -1,5 +1,8 @@ -[OrderedDict([('enum', 'abc'), ('data', ['a', 'b', 'c'])]), - OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))]), - OrderedDict([('command', 'okay'), ('data', OrderedDict([('member1', ['int']), ('member2', ['def'])]))])] -[{'enum_name': 'abc', 'enum_values': ['a', 'b', 'c']}] -[OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))])] +object :obj-okay-args + member member1: intList optional=False + member member2: defList optional=False +enum abc ['a', 'b', 'c'] +object def + member array: abcList optional=False +command okay :obj-okay-args -> None + gen=True success_response=True
tests/qapi-schema/empty.out +0 -3
@@ -1,3 +0,0 @@ -[] -[] -[]
tests/qapi-schema/enum-empty.out +1 -3
@@ -1,3 +1 @@ -[OrderedDict([('enum', 'MyEnum'), ('data', [])])] -[{'enum_name': 'MyEnum', 'enum_values': []}] -[] +enum MyEnum []
tests/qapi-schema/event-case.out +1 -3
@@ -1,3 +1 @@ -[OrderedDict([('event', 'oops')])] -[] -[] +event oops None
tests/qapi-schema/flat-union-reverse-define.out +12 -9
@@ -1,9 +1,12 @@ -[OrderedDict([('union', 'TestUnion'), ('base', 'TestBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'TestTypeA'), ('value2', 'TestTypeB')]))]), - OrderedDict([('struct', 'TestBase'), ('data', OrderedDict([('enum1', 'TestEnum')]))]), - OrderedDict([('enum', 'TestEnum'), ('data', ['value1', 'value2'])]), - OrderedDict([('struct', 'TestTypeA'), ('data', OrderedDict([('string', 'str')]))]), - OrderedDict([('struct', 'TestTypeB'), ('data', OrderedDict([('integer', 'int')]))])] -[{'enum_name': 'TestEnum', 'enum_values': ['value1', 'value2']}] -[OrderedDict([('struct', 'TestBase'), ('data', OrderedDict([('enum1', 'TestEnum')]))]), - OrderedDict([('struct', 'TestTypeA'), ('data', OrderedDict([('string', 'str')]))]), - OrderedDict([('struct', 'TestTypeB'), ('data', OrderedDict([('integer', 'int')]))])] +object TestBase + member enum1: TestEnum optional=False +enum TestEnum ['value1', 'value2'] +object TestTypeA + member string: str optional=False +object TestTypeB + member integer: int optional=False +object TestUnion + base TestBase + tag enum1 + case value1: TestTypeA flat=True + case value2: TestTypeB flat=True
tests/qapi-schema/ident-with-escape.out +4 -3
@@ -1,3 +1,4 @@ -[OrderedDict([('command', 'fooA'), ('data', OrderedDict([('bar1', 'str')]))])] -[] -[] +object :obj-fooA-args + member bar1: str optional=False +command fooA :obj-fooA-args -> None + gen=True success_response=True
tests/qapi-schema/include-relpath.out +1 -3
@@ -1,3 +1 @@ -[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])] -[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}] -[] +enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-repetition.out +1 -3
@@ -1,3 +1 @@ -[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])] -[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}] -[] +enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-simple.out +1 -3
@@ -1,3 +1 @@ -[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])] -[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}] -[] +enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/indented-expr.out +4 -3
@@ -1,3 +1,4 @@ -[OrderedDict([('command', 'eins')]), OrderedDict([('command', 'zwei')])] -[] -[] +command eins None -> None + gen=True success_response=True +command zwei None -> None + gen=True success_response=True
tests/qapi-schema/qapi-schema-test.out +131 -55
186 lines not on this page · raw message
tests/qapi-schema/returns-int.out +2 -3
@@ -1,3 +1,2 @@ -[OrderedDict([('command', 'guest-get-time'), ('returns', 'int')])] -[] -[] +command guest-get-time None -> int + gen=True success_response=True
tests/qapi-schema/test-qapi.py +30 -7
@@ -15,11 +15,34 @@ from pprint import pprint import os import sys -try: - exprs = QAPISchema(sys.argv[1]).get_exprs() -except SystemExit: - raise +class QAPISchemaTestVisitor(QAPISchemaVisitor): + def visit_enum_type(self, name, info, values): + print 'enum %s %s' % (name, values) + def visit_object_type(self, name, info, base, members, variants): + print 'object %s' % name + if base: + print ' base %s' % base.name + for m in members: + print ' member %s: %s optional=%s' % (m.name, m.type.name, + m.optional) + self._print_variants(variants) + def visit_alternate_type(self, name, info, variants): + print 'alternate %s' % name + self._print_variants(variants) + def visit_command(self, name, info, args, rets, gen, success_response): + print 'command %s %s -> %s' % (name, (args and args.name), + (rets and rets.name)) + print ' gen=%s success_response=%s' % (gen, success_response) + def visit_event(self, name, info, data): + print 'event %s %s' % (name, data and data.name) -pprint(exprs) -pprint(enum_types) -pprint(struct_types) + @staticmethod + def _print_variants(variants): + if variants: + if variants.tag_name: + print ' tag %s' % variants.tag_name + for v in variants.variants: + print ' case %s: %s flat=%s' % (v.name, v.type.name, v.flat) + +schema = QAPISchema(sys.argv[1]) +schema.visit(QAPISchemaTestVisitor())
tests/qapi-schema/type-bypass.out +4 -3
@@ -1,3 +1,4 @@ -[OrderedDict([('command', 'unsafe'), ('data', OrderedDict([('arg', '**')])), ('returns', '**'), ('gen', False)])] -[] -[] +object :obj-unsafe-args + member arg: ** optional=False +command unsafe :obj-unsafe-args -> ** + gen=False success_response=True
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 24/47] tests/qapi-schema: Convert test harness to QAPISchemaVisitor
Quoted 19 lines · Markus Armbruster
The old code prints the result of parsing (list of expression dictionaries), and partial results of semantic analysis (list of enum dictionaries, list of struct dictionaries).
The new code prints a trace of a schema visit, i.e. what the back-ends are going to use. Built-in and array types are omitted, because they're boring.
Signed-off-by Markus Armbruster <[email protected]>--- tests/qapi-schema/alternate-good.out | 15 +- tests/qapi-schema/comments.out | 4 +- tests/qapi-schema/data-member-array.out | 13 +- tests/qapi-schema/empty.out | 3 - tests/qapi-schema/enum-empty.out | 4 +- tests/qapi-schema/event-case.out | 4 +- tests/qapi-schema/flat-union-reverse-define.out | 21 +-- tests/qapi-schema/ident-with-escape.out | 7 +- tests/qapi-schema/include-relpath.out | 4 +-
Quoted 8 lines
tests/qapi-schema/include-repetition.out | 4 +- tests/qapi-schema/include-simple.out | 4 +- tests/qapi-schema/indented-expr.out | 7 +- tests/qapi-schema/qapi-schema-test.out | 186 +++++++++++++++++------- tests/qapi-schema/returns-int.out | 5 +- tests/qapi-schema/test-qapi.py | 37 ++++- tests/qapi-schema/type-bypass.out | 7 +- 16 files changed, 210 insertions(+), 115 deletions(-)
We have a lot more negative than positive tests of the parser (good thing, because that meant fewer .out files to update to the new format).
No change to actual qemu code, and proves that the previous three patches have set up enough of a framework to accurately cover our testsuite.
Quoted 6 lines
tests/qapi-schema/alternate-good.out +0
-0
tests/qapi-schema/alternate-good.out line 4
Quoted 10 lines
tests/qapi-schema/alternate-good.out +4
-6@@ -1,6 +1,9 @@ -[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))]), - OrderedDict([('enum', 'Enum'), ('data', ['hello', 'world'])]), - OrderedDict([('alternate', 'Alt'), ('data', OrderedDict([('value', 'int'), ('string', 'Enum'), ('struct', 'Data')]))])] -[{'enum_name': 'Enum', 'enum_values': ['hello', 'world']}, - {'enum_name': 'AltKind', 'enum_values': None}] -[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))])] +alternate Alt + case value: int flat=False + case string: Enum flat=False + case struct: Data flat=False
I'm still not convinced whether we need .flat exposed through this much detail, or if we should just normalize plain unions into flat unions with implicit structs for each branch. Changing your design will have obvious ripple effects here.
tests/qapi-schema/alternate-good.out line 2
Quoted 9 lines
tests/qapi-schema/alternate-good.out +3
-5+++ b/tests/qapi-schema/data-member-array.out@@ -1,5 +1,8 @@ -[OrderedDict([('enum', 'abc'), ('data', ['a', 'b', 'c'])]), - OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))]), - OrderedDict([('command', 'okay'), ('data', OrderedDict([('member1', ['int']), ('member2', ['def'])]))])] -[{'enum_name': 'abc', 'enum_values': ['a', 'b', 'c']}] -[OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))])] +object :obj-okay-args + member member1: intList optional=False
Took me a moment to realize the object is an implicit one (named ':obj-okay-args') and not a typo for 'object: obj-okay-args' consistent with members being listed 'name: type'. But not worth changing things, as it is sufficiently unambiguous to serve as a valid test.
Quoted 17 lines
tests/qapi-schema/alternate-good.out +17
-0+object UserDefFlatUnion + base UserDefUnionBase + tag enum1 + case value1: UserDefA flat=True + case value2: UserDefB flat=True + case value3: UserDefB flat=True +object UserDefFlatUnion2 + base UserDefUnionBase + tag enum1 + case value1: UserDefC flat=True + case value2: UserDefB flat=True + case value3: UserDefA flat=True +object UserDefNativeListUnion + case integer: intList flat=False + case s8: int8List flat=False + case s16: int16List flat=False + case s32: int32List flat=False
tests/qapi-schema/alternate-good.out
Quoted 10 lines
tests/qapi-schema/alternate-good.out +10
-0+ case s64: int64List flat=False + case u8: uint8List flat=False + case u16: uint16List flat=False + case u32: uint32List flat=False + case u64: uint64List flat=False + case number: numberList flat=False + case boolean: boolList flat=False + case string: strList flat=False + case sizes: sizeList flat=False +enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32', 's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes']
Hmm. You are dumping the tag name and type of flat unions, but not of simple unions. I would have expected:
object UserDefNativeListUnion
member kind: UserDefNativeListUnionKind
tag kind
case integer: intList flat=False
...
The above was fallout, while below is the meat of the new visitor.
Quoted 10 lines
+++
b/tests/qapi-schema/test-qapi.pytests/qapi-schema/alternate-good.out +1
-4@@ -15,11 +15,34 @@ from pprint import pprint import os import sys -try: - exprs = QAPISchema(sys.argv[1]).get_exprs() -except SystemExit: - raise +class QAPISchemaTestVisitor(QAPISchemaVisitor):
tests/qapi-schema/alternate-good.out from pprint import pprint · line 28
Quoted 10 lines
tests/qapi-schema/alternate-good.out +10
-0+ def visit_enum_type(self, name, info, values): + print 'enum %s %s' % (name, values) + def visit_object_type(self, name, info, base, members, variants): + print 'object %s' % name + if base: + print ' base %s' % base.name + for m in members: + print ' member %s: %s optional=%s' % (m.name, m.type.name, + m.optional) + self._print_variants(variants)
So here is where information was missing when visiting a simple union. Why didn't 'kind' get injected into members? And...
Quoted 8 lines
tests/qapi-schema/alternate-good.out +8
-0+ def visit_alternate_type(self, name, info, variants): + print 'alternate %s' % name + self._print_variants(variants) + def visit_command(self, name, info, args, rets, gen, success_response): + print 'command %s %s -> %s' % (name, (args and args.name), + (rets and rets.name)) + print ' gen=%s success_response=%s' % (gen, success_response) + def visit_event(self, name, info, data):
tests/qapi-schema/alternate-good.out from pprint import pprint · line 43
Quoted 10 lines
tests/qapi-schema/alternate-good.out +6
-3+ print 'event %s %s' % (name, data and data.name) -pprint(exprs) -pprint(enum_types) -pprint(struct_types) + @staticmethod + def _print_variants(variants): + if variants: + if variants.tag_name: + print ' tag %s' % variants.tag_name
...shouldn't a simple union report 'kind' as its tag_name?
tests/qapi-schema/type-bypass.out
Quoted 6 lines
tests/qapi-schema/alternate-good.out +5
-0+ for v in variants.variants: + print ' case %s: %s flat=%s' % (v.name, v.type.name, v.flat) + +schema = QAPISchema(sys.argv[1]) +schema.visit(QAPISchemaTestVisitor())tests/qapi-schema/type-bypass.out +0
-0
Overall, fairly slick, but I have enough questions about the representation of a simple union that I'll wait for the non-RFC respin to see if you change any design.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 24/47] tests/qapi-schema: Convert test harness to QAPISchemaVisitor
Quoted 53 lines · Eric Blake
Quoted 20 lines · Markus Armbruster
The old code prints the result of parsing (list of expression dictionaries), and partial results of semantic analysis (list of enum dictionaries, list of struct dictionaries).
The new code prints a trace of a schema visit, i.e. what the back-ends are going to use. Built-in and array types are omitted, because they're boring.
Signed-off-by Markus Armbruster <[email protected]>--- tests/qapi-schema/alternate-good.out | 15 +- tests/qapi-schema/comments.out | 4 +- tests/qapi-schema/data-member-array.out | 13 +- tests/qapi-schema/empty.out | 3 - tests/qapi-schema/enum-empty.out | 4 +- tests/qapi-schema/event-case.out | 4 +- tests/qapi-schema/flat-union-reverse-define.out | 21 +-- tests/qapi-schema/ident-with-escape.out | 7 +- tests/qapi-schema/include-relpath.out | 4 +- tests/qapi-schema/include-repetition.out | 4 +-Quoted 8 lines
tests/qapi-schema/include-simple.out | 4 +- tests/qapi-schema/indented-expr.out | 7 +- tests/qapi-schema/qapi-schema-test.out | 186 +++++++++++++++++------- tests/qapi-schema/returns-int.out | 5 +- tests/qapi-schema/test-qapi.py | 37 ++++- tests/qapi-schema/type-bypass.out | 7 +- 16 files changed, 210 insertions(+), 115 deletions(-)We have a lot more negative than positive tests of the parser (good thing, because that meant fewer .out files to update to the new format).
No change to actual qemu code, and proves that the previous three patches have set up enough of a framework to accurately cover our testsuite.
Quoted 17 lines
tests/qapi-schema/alternate-good.out +0
-3@@ -1,6 +1,9 @@ -[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))]), - OrderedDict([('enum', 'Enum'), ('data', ['hello', 'world'])]), - OrderedDict([('alternate', 'Alt'), ('data', OrderedDict([('value','int'), ('string', 'Enum'), ('struct', 'Data')]))])] -[{'enum_name': 'Enum', '
enum_values': ['hello', 'world']},
- {'enum_name': 'AltKind', 'enum_values': None}] -[OrderedDict([('struct', 'Data'), ('data',OrderedDict([('*number', 'int'), ('*name', 'str')]))])]
+alternate Alt
Quoted 8 lines
tests/qapi-schema/alternate-good.out line 4
Quoted 3 lines
tests/qapi-schema/alternate-good.out +3
-0+ case value: int flat=False + case string: Enum flat=False + case struct: Data flat=FalseI'm still not convinced whether we need .flat exposed through this much detail, or if we should just normalize plain unions into flat unions with implicit structs for each branch. Changing your design will have obvious ripple effects here.
Yes. I think it's okay as long as we keep it out of external interfaces.
Quoted 9 lines
Quoted 9 lines
+++ b/tests/qapi-schema/data-member-array.out
tests/qapi-schema/alternate-good.out +0
-3@@ -1,5 +1,8 @@ -[OrderedDict([('enum', 'abc'), ('data', ['a', 'b', 'c'])]), - OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))]), - OrderedDict([('command', 'okay'), ('data',
OrderedDict([('member1', ['int']), ('member2', ['def'])]))])] -[{'enum_name': 'abc', 'enum_values': ['a', 'b', 'c']}] -[OrderedDict([('struct', 'def'), ('data', OrderedDict([('array',
Quoted 8 lines
tests/qapi-schema/alternate-good.out
Quoted 3 lines
['abc'])]))])]
tests/qapi-schema/alternate-good.out +2
-0+object :obj-okay-args + member member1: intList optional=FalseTook me a moment to realize the object is an implicit one (named ':obj-okay-args') and not a typo for 'object: obj-okay-args' consistent with members being listed 'name: type'. But not worth changing things, as it is sufficiently unambiguous to serve as a valid test.
Perhaps omitting the ':' after member names would be less confusing.
Quoted 33 lines
Quoted 27 lines
tests/qapi-schema/alternate-good.out +27
-0+object UserDefFlatUnion + base UserDefUnionBase + tag enum1 + case value1: UserDefA flat=True + case value2: UserDefB flat=True + case value3: UserDefB flat=True +object UserDefFlatUnion2 + base UserDefUnionBase + tag enum1 + case value1: UserDefC flat=True + case value2: UserDefB flat=True + case value3: UserDefA flat=True +object UserDefNativeListUnion + case integer: intList flat=False + case s8: int8List flat=False + case s16: int16List flat=False + case s32: int32List flat=False + case s64: int64List flat=False + case u8: uint8List flat=False + case u16: uint16List flat=False + case u32: uint32List flat=False + case u64: uint64List flat=False + case number: numberList flat=False + case boolean: boolList flat=False + case string: strList flat=False + case sizes: sizeList flat=False +enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32',Quoted 2 lines
's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes']
Hmm. You are dumping the tag name and type of flat unions, but not of simple unions. I would have expected:
Quoted 5 lines
object UserDefNativeListUnion
member kind: UserDefNativeListUnionKind
tag kind
case integer: intList flat=False
...
See below.
Quoted 17 lines
The above was fallout, while below is the meat of the new visitor.
Quoted 15 lines
+++
b/tests/qapi-schema/test-qapi.pytests/qapi-schema/alternate-good.out +6
-4@@ -15,11 +15,34 @@ from pprint import pprint import os import sys -try: - exprs = QAPISchema(sys.argv[1]).get_exprs() -except SystemExit: - raise +class QAPISchemaTestVisitor(QAPISchemaVisitor): + def visit_enum_type(self, name, info, values): + print 'enum %s %s' % (name, values) + def visit_object_type(self, name, info, base, members, variants): + print 'object %s' % name + if base:
Quoted 8 lines
tests/qapi-schema/alternate-good.out from pprint import pprint · line 28
Quoted 5 lines
tests/qapi-schema/alternate-good.out +5
-0+ print ' base %s' % base.name + for m in members: + print ' member %s: %s optional=%s' % (m.name, m.type.name, + m.optional) + self._print_variants(variants)So here is where information was missing when visiting a simple union. Why didn't 'kind' get injected into members? And...
Because members are the explicit members. A simple union's implicit tag member is in QAPISchemaObjectTypeVariants. See further below.
Quoted 10 lines
Quoted 10 lines
tests/qapi-schema/alternate-good.out +9
-0+ def visit_alternate_type(self, name, info, variants): + print 'alternate %s' % name + self._print_variants(variants) + def visit_command(self, name, info, args, rets, gen, success_response): + print 'command %s %s -> %s' % (name, (args and args.name), + (rets and rets.name)) + print ' gen=%s success_response=%s' % (gen, success_response) + def visit_event(self, name, info, data): + print 'event %s %s' % (name, data and data.name)
Quoted 10 lines
tests/qapi-schema/alternate-good.out from pprint import pprint · line 43
Quoted 8 lines
tests/qapi-schema/alternate-good.out +5
-3-pprint(exprs) -pprint(enum_types) -pprint(struct_types) + @staticmethod + def _print_variants(variants): + if variants: + if variants.tag_name: + print ' tag %s' % variants.tag_name...shouldn't a simple union report 'kind' as its
tag_name?
It could. Here's why it currently isn't.
QAPISchemaObjectType has members .base and .local_members, which come straight from the type definition.
These get flattened into .members.
We do create a simple union's implicit member 'kind', and store it in .variants.tag_member. But we don't add it to .local_members, only to .members, similar to how .base's members are only in .members.
We don't print .members here. Instead, we print .base, .local_members and .variants. The part printing .variants doesn't print .variants.tag_member.
We need to draw the line on what to print *somewhere*. Where exactly is of course debatable. My working idea on what to print here is "print everything that isn't derived from other stuff".
Quoted 11 lines
tests/qapi-schema/type-bypass.out
Quoted 7 lines
tests/qapi-schema/alternate-good.out +5
-0+ for v in variants.variants: + print ' case %s: %s flat=%s' % (v.name, v.type.name, v.flat) + +schema = QAPISchema(sys.argv[1]) +schema.visit(QAPISchemaTestVisitor())tests/qapi-schema/type-bypass.out +0
-0Overall, fairly slick, but I have enough questions about the representation of a simple union that I'll wait for the non-RFC respin to see if you change any design.
Thanks!
RFC PATCH v2 25/47 qapi: Make generators work on sorted schema expressions
In reply to Markus ArmbrusterOrder of expressions doesn't matter. QAPISchema().get_exprs() returns expressions in the order they're parsed.
I'm going to refactor this code. To check refactorings, I want to diff the generated code, and for that I need to preserve its order.
Since I don't feel like preserving parse order, I'm changing get_expr() to return the expressions sorted by name. This order will be trivial to preserve. It also makes the generated code slightly easier to navigate.
--- scripts/qapi.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
scripts/qapi.py +8 -1
@@ -1017,7 +1017,14 @@ class QAPISchema(object): self.check() def get_exprs(self): - return [expr_elem['expr'] for expr_elem in self.exprs] + def expr_name(expr): + name = expr.get('enum') or expr.get('union') \ + or expr.get('alternate') or expr.get('struct') \ + or expr.get('command') or expr.get('event') + assert name + return name + return sorted([expr_elem['expr'] for expr_elem in self.exprs], + key=expr_name) def _def_entity(self, ent): assert ent.name not in self.entity_dict
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 25/47] qapi: Make generators work on sorted schema expressions
Quoted 10 lines · Markus Armbruster
Order of expressions doesn't matter. QAPISchema().get_exprs() returns expressions in the order they're parsed.
I'm going to refactor this code. To check refactorings, I want to diff the generated code, and for that I need to preserve its order.
Since I don't feel like preserving parse order, I'm changing
get_expr()to return the expressions sorted by name. This order will be trivial to preserve. It also makes the generated code slightly easier to navigate.
Huge change to the generated files, but that's to be expected. Diffstat shows it is not a straight 1:1 reshuffle:
qapi-event.c | 890 +-- qapi-event.h | 190 qapi-types.c | 1996 +++---- qapi-types.h | 5420 ++++++++++---------- qapi-visit.c | 9088 +++++++++++++++++----------------- qapi-visit.h | 746 +- qga/qapi-generated/qga-qapi-types.c | 148 qga/qapi-generated/qga-qapi-types.h | 340 - qga/qapi-generated/qga-qapi-visit.c | 446 - qga/qapi-generated/qga-qapi-visit.h | 54 qga/qapi-generated/qga-qmp-commands.h | 38 qga/qapi-generated/qga-qmp-marshal.c | 864 +-- qmp-commands.h | 376 - 13 files changed, 10308 insertions(+), 10288 deletions(-)
but that's probably because you now emit forward declarations for things that occur alphabetically after their first use (since 8/47 in this series touched forward declarations), whereas before they happened to occurr in topological order from the parse.
Quoted 11 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)scripts/qapi.py +0
-0
scripts/qapi.py class QAPISchema(object): · line 1025
Quoted 10 lines
scripts/qapi.py +6
-1@@ -1017,7 +1017,14 @@ class QAPISchema(object): self.check() def get_exprs(self): - return [expr_elem['expr'] for expr_elem in self.exprs] + def expr_name(expr): + name = expr.get('enum') or expr.get('union') \ + or expr.get('alternate') or expr.get('struct') \ + or expr.get('command') or expr.get('event') + assert name + return name
When I was working on this file earlier, I half toyed with the idea of adding expr_elem['name'] holding the entity name, and expr_elem['meta'] holding what meta-type the entity represents; it might have made some of our later 6-way switches simpler. But with your hierarchy of actual objects, I'm not sure my idea helps any more.
scripts/qapi.py class QAPISchema(object): · line 1027
Quoted 2 lines
scripts/qapi.py +2
-0+ return sorted([expr_elem['expr'] for expr_elem in self.exprs], + key=expr_name)
Python has some nice compact syntactical gems - I'd hate the amount of boilerplate required to write this same filter using straight C code :)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 25/47] qapi: Make generators work on sorted schema expressions
Quoted 30 lines · Eric Blake
Quoted 10 lines · Markus Armbruster
Order of expressions doesn't matter. QAPISchema().get_exprs() returns expressions in the order they're parsed.
I'm going to refactor this code. To check refactorings, I want to diff the generated code, and for that I need to preserve its order.
Since I don't feel like preserving parse order, I'm changing
get_expr()to return the expressions sorted by name. This order will be trivial to preserve. It also makes the generated code slightly easier to navigate.Huge change to the generated files, but that's to be expected. Diffstat shows it is not a straight 1:1 reshuffle:
qapi-event.c | 890 +-- qapi-event.h | 190 qapi-types.c | 1996 +++---- qapi-types.h | 5420 ++++++++++---------- qapi-visit.c | 9088 +++++++++++++++++----------------- qapi-visit.h | 746 +- qga/qapi-generated/qga-qapi-types.c | 148 qga/qapi-generated/qga-qapi-types.h | 340 - qga/qapi-generated/qga-qapi-visit.c | 446 - qga/qapi-generated/qga-qapi-visit.h | 54 qga/qapi-generated/qga-qmp-commands.h | 38 qga/qapi-generated/qga-qmp-marshal.c | 864 +-- qmp-commands.h | 376 - 13 files changed, 10308 insertions(+), 10288 deletions(-)
Quoted 4 lines
but that's probably because you now emit forward declarations for things that occur alphabetically after their first use (since 8/47 in this series touched forward declarations), whereas before they happened to occurr in topological order from the parse.
Here's my cheap trick for sanity checking this commit: compare before and after *sorted*. Results:
* qapi-event.h
- Members of enum QAPIEvent reordered, values changed accordingly (but they shouldn't matter)
* qapi-visit.c
- A bunch of new forward declarations
* test-qapi-visit.c
- One forward declaration gone.
Quoted 12 lines
Quoted 12 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)scripts/qapi.py +0
-0@@ -1017,7 +1017,14 @@ class QAPISchema(object): self.check()
Quoted 14 lines
scripts/qapi.py class QAPISchema(object): · line 1025
Quoted 8 lines
scripts/qapi.py +6
-1def get_exprs(self): - return [expr_elem['expr'] for expr_elem in self.exprs] + def expr_name(expr): + name = expr.get('enum') or expr.get('union') \ + or expr.get('alternate') or expr.get('struct') \ + or expr.get('command') or expr.get('event') + assert name + return nameWhen I was working on this file earlier, I half toyed with the idea of adding expr_elem['name'] holding the entity name, and expr_elem['meta'] holding what meta-type the entity represents; it might have made some of our later 6-way switches simpler. But with your hierarchy of actual objects, I'm not sure my idea helps any more.
The more work we do with the new internal representation instead of the syntax tree, the less it'll help.
Quoted 7 lines
scripts/qapi.py class QAPISchema(object): · line 1027
Quoted 2 lines
scripts/qapi.py +2
-0+ return sorted([expr_elem['expr'] for expr_elem in self.exprs], + key=expr_name)Python has some nice compact syntactical gems - I'd hate the amount of boilerplate required to write this same filter using straight C code :)
Reviewed-by Eric Blake <[email protected]>
Thanks!
RFC PATCH v2 26/47 qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
In reply to Markus ArmbrusterFixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:
{ 'union': 'UserDefFlatUnion',
'base': 'UserDefUnionBase',
'discriminator': 'enum1',
'data': { 'value1' : 'UserDefA',
'value2' : 'UserDefB',
'value3' : 'UserDefB' } }
{ 'struct': 'UserDefUnionBase',
'base': 'UserDefZero',
'data': { 'string': 'str', 'enum1': 'EnumOne' } }
{ 'struct': 'UserDefZero',
'data': { 'integer': 'int' } }Patch's effect on UserDefFlatUnion:
struct UserDefFlatUnion {
+ int64_t integer;
char *string;
EnumOne enum1;
/* union tag is EnumOne enum1 */
union {
void *data;
UserDefA *value1;
UserDefB *value2;
UserDefB *value3;
};
};Flat union visitors remain broken. They'll be fixed next.
--- scripts/qapi-types.py | 268 +++++++++++++------------------- tests/qapi-schema/qapi-schema-test.json | 4 +- 2 files changed, 114 insertions(+), 158 deletions(-)
scripts/qapi-types.py +112 -156
@@ -2,88 +2,67 @@ # QAPI types generator # # Copyright IBM, Corp. 2011 +# Copyright (c) 2013-2015 Red Hat Inc. # # Authors: # Anthony Liguori <[email protected]> +# Markus Armbruster <[email protected]> # # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -from ordereddict import OrderedDict from qapi import * -def generate_fwd_builtin(name): - return mcgen(''' - -typedef struct %(name)sList { - union { - %(type)s value; - uint64_t padding; - }; - struct %(name)sList *next; -} %(name)sList; -''', - type=c_type(name), - name=name) - -def generate_fwd_struct(name): +def gen_fwd_object_or_array(name): return mcgen(''' typedef struct %(name)s %(name)s; - -typedef struct %(name)sList { - union { - %(name)s *value; - uint64_t padding; - }; - struct %(name)sList *next; -} %(name)sList; ''', name=c_name(name)) -def generate_fwd_enum_struct(name): +def gen_array(name, element_type): return mcgen(''' -typedef struct %(name)sList { +struct %(name)s { union { - %(name)s value; + %(c_type)s value; uint64_t padding; }; - struct %(name)sList *next; -} %(name)sList; + struct %(name)s *next; +}; ''', - name=c_name(name)) + name=c_name(name), c_type=element_type.c_type()) -def generate_struct_fields(members): +def gen_struct_field(name, typ, optional): ret = '' - for argname, argentry, optional in parse_args(members): - if optional: - ret += mcgen(''' + if optional: + ret += mcgen(''' bool has_%(c_name)s; ''', - c_name=c_name(argname)) - ret += mcgen(''' + c_name=c_name(name)) + ret += mcgen(''' %(c_type)s %(c_name)s; ''', - c_type=c_type(argentry), c_name=c_name(argname)) - + c_type=typ.c_type(), c_name=c_name(name)) return ret -def generate_struct(expr): +def generate_struct_fields(members): + ret = '' - structname = expr.get('struct', "") - members = expr['data'] - base = expr.get('base') + for memb in members: + ret += gen_struct_field(memb.name, memb.type, memb.optional) + return ret +def gen_struct(name, base, members): ret = mcgen(''' struct %(name)s { ''', - name=c_name(structname)) + name=c_name(name)) if base: - ret += generate_struct_fields({'base': base}) + ret += gen_struct_field('base', base, False) ret += generate_struct_fields(members)
@@ -156,46 +135,38 @@ typedef enum %(name)s { return enum_decl + lookup_decl -def generate_alternate_qtypes(expr): +def gen_alternate_qtypes_decl(name): + return mcgen(''' - name = expr['alternate'] - members = expr['data'] +extern const int %(c_name)s_qtypes[]; +''', + c_name=c_name(name)) +def gen_alternate_qtypes(name, variants): ret = mcgen(''' const int %(name)s_qtypes[QTYPE_MAX] = { ''', name=c_name(name)) - for key in members: - qtype = find_alternate_member_qtype(members[key]) - assert qtype, "Invalid alternate member" + for var in variants.variants: + qtype = var.type.alternate_qtype() + assert qtype ret += mcgen(''' [%(qtype)s] = %(enum_const)s, ''', qtype = qtype, - enum_const = c_enum_const(name + 'Kind', key)) + enum_const=c_enum_const(variants.tag_member.type.name, + var.name)) ret += mcgen(''' }; ''') return ret - -def generate_union(expr, meta): - - name = c_name(expr[meta]) - typeinfo = expr['data'] - - base = expr.get('base') - discriminator = expr.get('discriminator') - - enum_define = discriminator_find_enum_define(expr) - if enum_define: - discriminator_type_name = enum_define['enum_name'] - else: - discriminator_type_name = '%sKind' % (name) +def gen_union(name, base, variants): + name = c_name(name) ret = mcgen('''
@@ -203,56 +174,49 @@ struct %(name)s { ''', name=name) if base: - base_fields = find_struct(base)['data'] - ret += generate_struct_fields(base_fields) + ret += generate_struct_fields(base.members) ret += mcgen(''' /* union tag is %(discriminator_type_name)s %(c_name)s */ ''', - discriminator_type_name=c_name(discriminator_type_name), - c_name=c_name(discriminator)) + discriminator_type_name=c_name(variants.tag_member.type.name), + c_name=c_name(variants.tag_member.name)) else: - assert not discriminator ret += mcgen(''' %(discriminator_type_name)s kind; ''', - discriminator_type_name=c_name(discriminator_type_name)) + discriminator_type_name=c_name(variants.tag_member.type.name)) ret += mcgen(''' union { void *data; ''') - for key in typeinfo: + for var in variants.variants: ret += mcgen(''' %(c_type)s %(c_name)s; ''', - c_type=c_type(typeinfo[key]), - c_name=c_name(key)) + c_type=var.type.c_type(), + c_name=c_name(var.name)) ret += mcgen(''' }; }; ''') - if meta == 'alternate': - ret += mcgen(''' -extern const int %(name)s_qtypes[]; -''', - name=name) - return ret def generate_type_cleanup_decl(name): ret = mcgen(''' -void qapi_free_%(name)s(%(c_type)s obj); + +void qapi_free_%(name)s(%(name)s *obj); ''', - c_type=c_type(name), name=c_name(name)) + name=c_name(name)) return ret def generate_type_cleanup(name): ret = mcgen(''' -void qapi_free_%(name)s(%(c_type)s obj) +void qapi_free_%(name)s(%(name)s *obj) { QapiDeallocVisitor *md; Visitor *v;
@@ -267,9 +231,68 @@ void qapi_free_%(name)s(%(c_type)s obj) qapi_dealloc_visitor_cleanup(md); } ''', - c_type=c_type(name), name=c_name(name)) + name=c_name(name)) return ret +class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): + def __init__(self): + self.decl = None + self.defn = None + self.fwdecl = None + self.fwdefn = None + self.btin = None + def visit_begin(self): + self.decl = '' + self.defn = '' + self.fwdecl = '' + self.fwdefn = '' + self.btin = guardstart('QAPI_TYPES_BUILTIN') + def visit_end(self): + self.decl = self.fwdecl + self.decl + self.fwdecl = None + self.defn = self.fwdefn + self.defn + self.fwdefn = None + # To avoid header dependency hell, we always generate + # declarations for built-in types in our header files and + # simply guard them. + self.btin += guardend('QAPI_TYPES_BUILTIN') + self.decl = self.btin + self.decl + self.btin = None + # Doesn't work for cases where we link in multiple objects + # that have the functions defined, so generate them only with + # option -b (do_builtins). + def _gen_type_cleanup(self, name): + self.decl += generate_type_cleanup_decl(name) + self.defn += generate_type_cleanup(name) + def visit_enum_type(self, name, info, values): + self.fwdecl += generate_enum(name, values) + self.fwdefn += generate_enum_lookup(name, values) + def visit_array_type(self, name, info, element_type): + if isinstance(element_type, QAPISchemaBuiltinType): + self.btin += gen_fwd_object_or_array(name) + self.btin += gen_array(name, element_type) + self.btin += generate_type_cleanup_decl(name) + if do_builtins: + self.defn += generate_type_cleanup(name) + else: + self.fwdecl += gen_fwd_object_or_array(name) + self.decl += gen_array(name, element_type) + self._gen_type_cleanup(name) + def visit_object_type(self, name, info, base, members, variants): + if info: + self.fwdecl += gen_fwd_object_or_array(name) + if variants: + self.decl += gen_union(name, base, variants) + else: + self.decl += gen_struct(name, base, members) + self._gen_type_cleanup(name) + def visit_alternate_type(self, name, info, variants): + self.fwdecl += gen_fwd_object_or_array(name) + self.fwdefn += gen_alternate_qtypes(name, variants) + self.decl += gen_union(name, None, variants) + self.decl += gen_alternate_qtypes_decl(name) + self._gen_type_cleanup(name) + do_builtins = False (input_file, output_dir, do_c, do_h, prefix, opts) = \
@@ -325,77 +348,10 @@ fdecl.write(mcgen(''' #include <stdint.h> ''')) -exprs = QAPISchema(input_file).get_exprs() - -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_fwd_builtin(typename)) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL")) - -for expr in exprs: - ret = "" - if expr.has_key('struct'): - ret += generate_fwd_struct(expr['struct']) - elif expr.has_key('enum'): - ret += generate_enum(expr['enum'], expr['data']) - ret += generate_fwd_enum_struct(expr['enum']) - fdef.write(generate_enum_lookup(expr['enum'], expr['data'])) - elif expr.has_key('union'): - ret += generate_fwd_struct(expr['union']) - enum_define = discriminator_find_enum_define(expr) - if not enum_define: - ret += generate_enum('%sKind' % expr['union'], expr['data'].keys()) - fdef.write(generate_enum_lookup('%sKind' % expr['union'], - expr['data'].keys())) - elif expr.has_key('alternate'): - ret += generate_fwd_struct(expr['alternate']) - ret += generate_enum('%sKind' % expr['alternate'], expr['data'].keys()) - fdef.write(generate_enum_lookup('%sKind' % expr['alternate'], - expr['data'].keys())) - fdef.write(generate_alternate_qtypes(expr)) - else: - continue - fdecl.write(ret) - -# to avoid header dependency hell, we always generate declarations -# for built-in types in our header files and simply guard them -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_type_cleanup_decl(typename + "List")) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) - -# ...this doesn't work for cases where we link in multiple objects that -# have the functions defined, so we use -b option to provide control -# over these cases -if do_builtins: - for typename in builtin_types.keys(): - fdef.write(generate_type_cleanup(typename + "List")) - -for expr in exprs: - ret = "" - if expr.has_key('struct'): - ret += generate_struct(expr) + "\n" - ret += generate_type_cleanup_decl(expr['struct'] + "List") - fdef.write(generate_type_cleanup(expr['struct'] + "List")) - ret += generate_type_cleanup_decl(expr['struct']) - fdef.write(generate_type_cleanup(expr['struct'])) - elif expr.has_key('union'): - ret += generate_union(expr, 'union') + "\n" - ret += generate_type_cleanup_decl(expr['union'] + "List") - fdef.write(generate_type_cleanup(expr['union'] + "List")) - ret += generate_type_cleanup_decl(expr['union']) - fdef.write(generate_type_cleanup(expr['union'])) - elif expr.has_key('alternate'): - ret += generate_union(expr, 'alternate') + "\n" - ret += generate_type_cleanup_decl(expr['alternate'] + "List") - fdef.write(generate_type_cleanup(expr['alternate'] + "List")) - ret += generate_type_cleanup_decl(expr['alternate']) - fdef.write(generate_type_cleanup(expr['alternate'])) - elif expr.has_key('enum'): - ret += "\n" + generate_type_cleanup_decl(expr['enum'] + "List") - fdef.write(generate_type_cleanup(expr['enum'] + "List")) - else: - continue - fdecl.write(ret) +schema = QAPISchema(input_file) +gen = QAPISchemaGenTypeVisitor() +schema.visit(gen) +fdef.write(gen.defn) +fdecl.write(gen.decl) close_output(fdef, fdecl)
tests/qapi-schema/qapi-schema-test.json +2 -2
@@ -39,8 +39,8 @@ 'data': { 'value1' : 'UserDefA', 'value2' : 'UserDefB', 'value3' : 'UserDefB' } } -# FIXME generated struct UserDefFlatUnion has members for direct base -# UserDefUnionBase, but lacks members for indirect base UserDefZero +# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit +# members of indirect base UserDefZero { 'struct': 'UserDefUnionBase', 'base': 'UserDefZero',
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Quoted 6 lines · Markus Armbruster
Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:
Flat union visitors remain broken. They'll be fixed next.
Sadly, the generated files had a huge diffstat, making it very hard to determine if this change is sane:
qapi-types.c | 1412 ++++++------- qapi-types.h | 3705 ++++++++++++++++++++---------------- qga/qapi-generated/qga-qapi-types.c | 110 - qga/qapi-generated/qga-qapi-types.h | 542 +++-- 4 files changed, 3208 insertions(+), 2561 deletions(-)
At first, it looks easy, as in:
qapi-types.c:
+const int BlockdevRef_qtypes[QTYPE_MAX] = {
...
const char *const BlockdevRefKind_lookup[] = {
...
-const int BlockdevRef_qtypes[QTYPE_MAX] = {
...
(that is, the new visitor outputs the two arrays in a different order - I can live with that).
Then it gets a bit crazy when using normal diff algorithms:
-void qapi_free_int32List(int32List *obj)
+void qapi_free_ACPIOSTInfo(ACPIOSTInfo *obj)
where declarations are rearranged (so the previous commit, which attempted to sort declarations to make this one easier, did not quite succeed).
Even in qapi-types.h things were rearranged:
-#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL
-#define QAPI_TYPES_BUILTIN_STRUCT_DECL
+#ifndef QAPI_TYPES_BUILTIN
+#define QAPI_TYPES_BUILTIN
-typedef struct int32List {
+typedef struct boolList boolList;
+
+struct boolList {
I can understand the minor change in #ifdef name, and even the separation between typedef and struct declaration (even though the old approach of doing it all at once works). But the overall diff in the generated files would be easier to review if done in stages, and either make 25/47 sort closer to what this patch does, or add yet more prerequisite patches that further tweak sorting. Ideally, this patch will be a lot easier to review if the generated code is much closer to the pre-patch version (that is, separating sorting changes from other changes). On the other hand, yes, it's kind of a pain to patch old code to do things differently just before throwing it away with the new code in its place, so it becomes a judgment call of how confident we want to be that the new implementation isn't breaking things.
I was able to make a bit more sense of things by using git's 'diff patience' algorithm, which showed things more as code motion rather than one-or-two-line changes to lots of common boilerplate, but even that diffstat is still big:
qapi-types1.c | 2030 +++++++++++++++---------------- qapi-types1.h | 3707 +++++++++++++++++++++++++++++++++------------------------ 2 files changed, 3148 insertions(+), 2589 deletions(-)
That said, I'll still at least review this code by inspection, and things still compile fine, so although I'm reluctant to give R-b while the patch is in RFC stage (because I didn't want to take the time to be certain the results are the same amidst so much churn), they are mostly sane.
Quoted 21 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-types.py | 268 +++++++++++++------------------- tests/qapi-schema/qapi-schema-test.json | 4 +- 2 files changed, 114 insertions(+), 158 deletions(-)scripts/qapi-types.py +2
-0@@ -2,88 +2,67 @@ # QAPI types generator # # Copyright IBM, Corp. 2011 +# Copyright (c) 2013-2015 Red Hat Inc. # # Authors: # Anthony Liguori <[email protected]> +# Markus Armbruster <[email protected]> #
scripts/qapi-types.py line 18
Quoted 10 lines
scripts/qapi-types.py +0
-5# This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -from ordereddict import OrderedDict from qapi import * -def generate_fwd_builtin(name): - return mcgen(''' - -typedef struct %(name)sList {
For example, the change of splitting this into:
typedef struct %(name)sList;
struct %(name)sList {
done as a separate patch would make the generated diff of this patch smaller.
scripts/qapi-types.py line 40
Quoted 10 lines
scripts/qapi-types.py +3
-4-def generate_struct_fields(members): +def gen_struct_field(name, typ, optional): ret = '' - for argname, argentry, optional in parse_args(members): - if optional: - ret += mcgen(''' + if optional: + ret += mcgen('''
(sometimes it's annoying that python requires indentation changes when changing control flow; for C files, sometimes I split a patch that merely adds {} and indentation from another patch that changes logic, so that the logic change isn't drowned by the reindentation)
Quoted 17 lines
scripts/qapi-types.py +17
-0+class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): + def __init__(self): + self.decl = None + self.defn = None + self.fwdecl = None + self.fwdefn = None + self.btin = None + def visit_begin(self): + self.decl = '' + self.defn = '' + self.fwdecl = '' + self.fwdefn = '' + self.btin = guardstart('QAPI_TYPES_BUILTIN') + def visit_end(self): + self.decl = self.fwdecl + self.decl + self.fwdecl = None + self.defn = self.fwdefn + self.defn
scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 263
Quoted 10 lines
scripts/qapi-types.py +10
-0+ self.fwdefn = None + # To avoid header dependency hell, we always generate + # declarations for built-in types in our header files and + # simply guard them. + self.btin += guardend('QAPI_TYPES_BUILTIN') + self.decl = self.btin + self.decl + self.btin = None + # Doesn't work for cases where we link in multiple objects + # that have the functions defined, so generate them only with + # option -b (do_builtins).
Does this comment...
scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 276
Quoted 13 lines
scripts/qapi-types.py +13
-0+ def _gen_type_cleanup(self, name): + self.decl += generate_type_cleanup_decl(name) + self.defn += generate_type_cleanup(name) + def visit_enum_type(self, name, info, values): + self.fwdecl += generate_enum(name, values) + self.fwdefn += generate_enum_lookup(name, values) + def visit_array_type(self, name, info, element_type): + if isinstance(element_type, QAPISchemaBuiltinType): + self.btin += gen_fwd_object_or_array(name) + self.btin += gen_array(name, element_type) + self.btin += generate_type_cleanup_decl(name) + if do_builtins: + self.defn += generate_type_cleanup(name)
...fit better here?
scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 282
Quoted 6 lines
scripts/qapi-types.py +6
-0+ else: + self.fwdecl += gen_fwd_object_or_array(name) + self.decl += gen_array(name, element_type) + self._gen_type_cleanup(name) + def visit_object_type(self, name, info, base, members, variants): + if info:
So to make sure I understand, this ensures that implicit types (those handled merely by parameters in a function or as members of a struct) have no declarations required in the -types files.
scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 285
Quoted 3 lines
scripts/qapi-types.py +3
-0+ self.fwdecl += gen_fwd_object_or_array(name) + if variants: + self.decl += gen_union(name, base, variants)
Is it worth 'assert not members' at this point?
scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 295
Quoted 10 lines
scripts/qapi-types.py +10
-0+ else: + self.decl += gen_struct(name, base, members) + self._gen_type_cleanup(name) + def visit_alternate_type(self, name, info, variants): + self.fwdecl += gen_fwd_object_or_array(name) + self.fwdefn += gen_alternate_qtypes(name, variants) + self.decl += gen_union(name, None, variants) + self.decl += gen_alternate_qtypes_decl(name) + self._gen_type_cleanup(name) +
The visitor looks reasonable from inspection review.
scripts/qapi-types.py fdecl.write(mcgen(''' · line 333
Quoted 13 lines
do_builtins= False(
input_file,output_dir,do_c,do_h, prefix, opts) = \scripts/qapi-types.py +0
-6@@ -325,77 +348,10 @@ fdecl.write(mcgen(''' #include <stdint.h> ''')) -exprs = QAPISchema(input_file).get_exprs() - -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_fwd_builtin(typename)) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
This is a place where adding sorting in 25/47 (or a separate patch) would make the impact on generated code from this patch smaller.
Quoted 11 lines
scripts/qapi-types.py +0
-11- -for expr in exprs: - ret = "" - if expr.has_key('struct'): - ret += generate_fwd_struct(expr['struct']) - elif expr.has_key('enum'): - ret += generate_enum(expr['enum'], expr['data']) - ret += generate_fwd_enum_struct(expr['enum']) - fdef.write(generate_enum_lookup(expr['enum'], expr['data'])) - elif expr.has_key('union'): - ret += generate_fwd_struct(expr['union'])
scripts/qapi-types.py fdecl.write(mcgen(''' · line 354
Quoted 10 lines
scripts/qapi-types.py +0
-10- enum_define = discriminator_find_enum_define(expr) - if not enum_define: - ret += generate_enum('%sKind' % expr['union'], expr['data'].keys()) - fdef.write(generate_enum_lookup('%sKind' % expr['union'], - expr['data'].keys())) - elif expr.has_key('alternate'): - ret += generate_fwd_struct(expr['alternate']) - ret += generate_enum('%sKind' % expr['alternate'], expr['data'].keys()) - fdef.write(generate_enum_lookup('%sKind' % expr['alternate'], - expr['data'].keys()))
Tweaking the order of _qtypes[] vs. _lookup[] here in a pre-req patch would also help review of the diff generated by this patch.
Quoted 8 lines
scripts/qapi-types.py +0
-8- fdef.write(generate_alternate_qtypes(expr)) - else: - continue - fdecl.write(ret) - -# to avoid header dependency hell, we always generate declarations -# for built-in types in our header files and simply guard them -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
scripts/qapi-types.py fdecl.write(mcgen(''' · line 372
Quoted 10 lines
scripts/qapi-types.py +0
-10-for typename in builtin_types.keys(): - fdecl.write(generate_type_cleanup_decl(typename + "List")) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) - -# ...this doesn't work for cases where we link in multiple objects that -# have the functions defined, so we use -b option to provide control -# over these cases -if do_builtins: - for typename in builtin_types.keys(): - fdef.write(generate_type_cleanup(typename + "List"))
Another iteration worth sorting in an earlier patch.
scripts/qapi-types.py fdecl.write(mcgen(''' · line 381
Quoted 9 lines
scripts/qapi-types.py +0
-9- -for expr in exprs: - ret = "" - if expr.has_key('struct'): - ret += generate_struct(expr) + "\n" - ret += generate_type_cleanup_decl(expr['struct'] + "List") - fdef.write(generate_type_cleanup(expr['struct'] + "List")) - ret += generate_type_cleanup_decl(expr['struct']) - fdef.write(generate_type_cleanup(expr['struct']))
I think part of the large size of the generated diff comes from whether array fooList types are emitted in a different order via the visitor.
Quoted 31 lines
scripts/qapi-types.py +5
-18- elif expr.has_key('union'): - ret += generate_union(expr, 'union') + "\n" - ret += generate_type_cleanup_decl(expr['union'] + "List") - fdef.write(generate_type_cleanup(expr['union'] + "List")) - ret += generate_type_cleanup_decl(expr['union']) - fdef.write(generate_type_cleanup(expr['union'])) - elif expr.has_key('alternate'): - ret += generate_union(expr, 'alternate') + "\n" - ret += generate_type_cleanup_decl(expr['alternate'] + "List") - fdef.write(generate_type_cleanup(expr['alternate'] + "List")) - ret += generate_type_cleanup_decl(expr['alternate']) - fdef.write(generate_type_cleanup(expr['alternate'])) - elif expr.has_key('enum'): - ret += "\n" + generate_type_cleanup_decl(expr['enum'] + "List") - fdef.write(generate_type_cleanup(expr['enum'] + "List")) - else: - continue - fdecl.write(ret) +schema = QAPISchema(input_file) +gen = QAPISchemaGenTypeVisitor() +schema.visit(gen) +fdef.write(gen.defn) +fdecl.write(gen.decl) close_output(fdef, fdecl)tests/qapi-schema/qapi-schema-test.json +0
-0
tests/qapi-schema/qapi-schema-test.json line 46
Quoted 10 lines
tests/qapi-schema/qapi-schema-test.json +2
-2@@ -39,8 +39,8 @@ 'data': { 'value1' : 'UserDefA', 'value2' : 'UserDefB', 'value3' : 'UserDefB' } } -# FIXME generated struct UserDefFlatUnion has members for direct base -# UserDefUnionBase, but lacks members for indirect base UserDefZero +# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit +# members of indirect base UserDefZero { 'struct': 'UserDefUnionBase', 'base': 'UserDefZero',
Overall, moving in the right direction.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Quoted 9 lines · Eric Blake
(quoted diff) +3
-1-typedef struct int32List { +typedef struct boolList boolList; + +struct boolList {I can understand the minor change in #ifdef name, and even the separation between typedef and struct declaration (even though the old approach of doing it all at once works).
Actually, I don't see any clear-cut policy on the matter (all HACKING says is that "Typedefs are used to eliminate the redundant 'struct' keyword", but does not say how aggressively they must be used). In fact, it looks like most of qemu.git prefers the more compact representation, while your patch goes to the less-common one:
$ git grep 'typedef struct .* {' | wc
1370 5549 71105
$ git grep 'typedef struct .*;' | wc
624 2515 41449However, one thing that your style has going for it, that was not possible with the compact version, is to use:
typedef struct intList intList;
struct intList {
union {
int64_t value;
uint64_t padding;
};- struct intList *next;
+ intList *next;
};
that is, because your typedef is separate, you have the option of being able to drop the use of 'struct' from within the actual intList struct.
Again, if we want that, it would be better to do that as a separate commit.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
In reply to Eric BlakeQuoted 27 lines · Eric Blake
Quoted 6 lines · Markus Armbruster
Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:
Flat union visitors remain broken. They'll be fixed next.
Sadly, the generated files had a huge diffstat, making it very hard to determine if this change is sane:
qapi-types.c | 1412 ++++++------- qapi-types.h | 3705 ++++++++++++++++++++---------------- qga/qapi-generated/qga-qapi-types.c | 110 - qga/qapi-generated/qga-qapi-types.h | 542 +++-- 4 files changed, 3208 insertions(+), 2561 deletions(-)At first, it looks easy, as in:
qapi-types.c:(quoted diff) +1
-1+const int BlockdevRef_qtypes[QTYPE_MAX] = { const char *const BlockdevRefKind_lookup[] = { -const int BlockdevRef_qtypes[QTYPE_MAX] = {
Quoted 2 lines
(that is, the new visitor outputs the two arrays in a different order - I can live with that).
The old code generates enum lookup tables for implicit union or alternate enumerations from within the union or alternate code.
The new code doesn't, because it makes all these enums full-fledged types, so the ordinary enum generator does everything we need.
And that's why generating stuff in alphabetical doesn't help here: the new types get inserted in the sorted sequence, but that's not where the old code generated the implicit enum stuff.
Quoted 8 lines
Then it gets a bit crazy when using normal diff algorithms:
(quoted diff) +1
-1-void qapi_free_int32List(int32List *obj) +void qapi_free_ACPIOSTInfo(ACPIOSTInfo *obj)where declarations are rearranged (so the previous commit, which attempted to sort declarations to make this one easier, did not quite succeed).
I found that quite disappointing myself, but I didn't have better ideas at the time.
Quoted 14 lines
Even in
qapi-types.hthings were rearranged:(quoted diff) +5
-3-#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL -#define QAPI_TYPES_BUILTIN_STRUCT_DECL +#ifndef QAPI_TYPES_BUILTIN +#define QAPI_TYPES_BUILTIN -typedef struct int32List { +typedef struct boolList boolList; + +struct boolList {
Quoted 3 lines
I can understand the minor change in #ifdef name, and even the separation between typedef and struct declaration (even though the old approach of doing it all at once works).
Artifact of my attempt to DRY: I sometimes need the typedef in one place (say .h) and the actual struct in another place (say .c). I make do with the functions generating that even when I want both of them in the same place.
Quoted 9 lines
But the overall diff in the
generated files would be easier to review if done in stages, and either make 25/47 sort closer to what this patch does, or add yet more prerequisite patches that further tweak sorting. Ideally, this patch will be a lot easier to review if the generated code is much closer to the pre-patch version (that is, separating sorting changes from other changes). On the other hand, yes, it's kind of a pain to patch old code to do things differently just before throwing it away with the new code in its place,
Especially when you're throwing away the old code because you can't stand working with it :)
Quoted 2 lines
so it becomes a judgment call of how confident we want to
be that the new implementation isn't breaking things.
Different idea: I hack up a script to split file FOO into one part per top-level syntactical construct (typedef, function, whatever), and store each part in a file named like NUM-NAME-FOO. The script will most likely be hairy, so use diff FOO <(cat *-FOO) to verify it works. Then diff old/*-NAME-FOO new/*-NAME-FOO to see what changed.
Writing the script will take non-trivial effort, but I'll try if you think it's needed.
For what it's worth, I sanity-checked with my "diff sorted" trick mentioned in my reply to your review of PATCH 25.
Quoted 9 lines
I was able to make a bit more sense of things by using git's 'diff patience' algorithm, which showed things more as code motion rather than one-or-two-line changes to lots of common boilerplate, but even that diffstat is still big:
qapi-types1.c | 2030 +++++++++++++++---------------- qapi-types1.h | 3707 +++++++++++++++++++++++++++++++++------------------------ 2 files changed, 3148 insertions(+), 2589 deletions(-)
Quoted 5 lines
That said, I'll still at least review this code by inspection, and things still compile fine, so although I'm reluctant to give R-b while the patch is in RFC stage (because I didn't want to take the time to be certain the results are the same amidst so much churn), they are mostly sane.
Fair enough.
Quoted 33 lines
Quoted 20 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-types.py | 268 +++++++++++++------------------- tests/qapi-schema/qapi-schema-test.json | 4 +- 2 files changed, 114 insertions(+), 158 deletions(-)scripts/qapi-types.py +2
-0@@ -2,88 +2,67 @@ # QAPI types generator # # Copyright IBM, Corp. 2011 +# Copyright (c) 2013-2015 Red Hat Inc. # # Authors: # Anthony Liguori <[email protected]> +# Markus Armbruster <[email protected]> #scripts/qapi-types.py line 18
Quoted 10 lines
scripts/qapi-types.py +0
-5# This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -from ordereddict import OrderedDict from qapi import * -def generate_fwd_builtin(name): - return mcgen(''' - -typedef struct %(name)sList {For example, the change of splitting this into:
Quoted 6 lines
typedef struct %(name)sList;
struct %(name)sList {
done as a separate patch would make the generated diff of this patch smaller.
I'm willing to mess with the old code *locally* to reduce the diffs, if that helps.
Quoted 4 lines
Quoted 4 lines
scripts/qapi-types.py +1
-1-def generate_struct_fields(members): +def gen_struct_field(name, typ, optional): ret = ''
Quoted 10 lines
scripts/qapi-types.py line 40
Quoted 5 lines
scripts/qapi-types.py +2
-3- for argname, argentry, optional in parse_args(members): - if optional: - ret += mcgen(''' + if optional: + ret += mcgen('''(sometimes it's annoying that python requires indentation changes when changing control flow; for C files, sometimes I split a patch that merely adds {} and indentation from another patch that changes logic, so that the logic change isn't drowned by the reindentation)
I sometimes use C-c C-w in diff-mode to view a patch hunk without its whitespace changes.
Quoted 44 lines
Quoted 17 lines
scripts/qapi-types.py +17
-0+class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): + def __init__(self): + self.decl = None + self.defn = None + self.fwdecl = None + self.fwdefn = None + self.btin = None + def visit_begin(self): + self.decl = '' + self.defn = '' + self.fwdecl = '' + self.fwdefn = '' + self.btin = guardstart('QAPI_TYPES_BUILTIN') + def visit_end(self): + self.decl = self.fwdecl + self.decl + self.fwdecl = None + self.defn = self.fwdefn + self.defnscripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 263
Quoted 10 lines
scripts/qapi-types.py +10
-0+ self.fwdefn = None + # To avoid header dependency hell, we always generate + # declarations for built-in types in our header files and + # simply guard them. + self.btin += guardend('QAPI_TYPES_BUILTIN') + self.decl = self.btin + self.decl + self.btin = None + # Doesn't work for cases where we link in multiple objects + # that have the functions defined, so generate them only with + # option -b (do_builtins).Does this comment...
scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 276
Quoted 13 lines
scripts/qapi-types.py +13
-0+ def _gen_type_cleanup(self, name): + self.decl += generate_type_cleanup_decl(name) + self.defn += generate_type_cleanup(name) + def visit_enum_type(self, name, info, values): + self.fwdecl += generate_enum(name, values) + self.fwdefn += generate_enum_lookup(name, values) + def visit_array_type(self, name, info, element_type): + if isinstance(element_type, QAPISchemaBuiltinType): + self.btin += gen_fwd_object_or_array(name) + self.btin += gen_array(name, element_type) + self.btin += generate_type_cleanup_decl(name) + if do_builtins: + self.defn += generate_type_cleanup(name)
Quoted 1 line
...fit better here?
If I rephrase it, probably yes.
Quoted 10 lines
scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 282
Quoted 6 lines
scripts/qapi-types.py +6
-0+ else: + self.fwdecl += gen_fwd_object_or_array(name) + self.decl += gen_array(name, element_type) + self._gen_type_cleanup(name) + def visit_object_type(self, name, info, base, members, variants): + if info:So to make sure I understand, this ensures that implicit types (those handled merely by parameters in a function or as members of a struct) have no declarations required in the -types files.
Yes, (not info) means the object type is implicit. We create implicit object types for command parameters, command returns, and event paramaters that aren't type names.
We don't generate C for these types, they're purely internal. We generate *parameter lists* for them.
Hmm, that's actually not true for command returns. And current master duly bombs if I add such a command to qapi-schema-test.json:
{ 'command': 'user_def_cmd4', 'returns': { 'a': 'int' } }
GEN tests/test-qmp-commands.h
Traceback (most recent call last):
File "/work/armbru/qemu/scripts/qapi-commands.py", line 360, in <module>
ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n"
File "/work/armbru/qemu/scripts/qapi-commands.py", line 29, in generate_command_decl
ret_type=c_type(ret_type), name=c_name(name),
File "/work/armbru/qemu/scripts/qapi.py", line 924, in c_type
assert isinstance(value, str) and value != ""
AssertionError
The easy fix is to outlaw it.
Quoted 5 lines
scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 285
Quoted 3 lines
scripts/qapi-types.py +3
-0+ self.fwdecl += gen_fwd_object_or_array(name) + if variants: + self.decl += gen_union(name, base, variants)Is it worth 'assert not members' at this point?
Yes, with a TODO, because it's an artificial restriction I'd like to lift some day.
Quoted 20 lines
scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 295
Quoted 10 lines
scripts/qapi-types.py +10
-0+ else: + self.decl += gen_struct(name, base, members) + self._gen_type_cleanup(name) + def visit_alternate_type(self, name, info, variants): + self.fwdecl += gen_fwd_object_or_array(name) + self.fwdefn += gen_alternate_qtypes(name, variants) + self.decl += gen_union(name, None, variants) + self.decl += gen_alternate_qtypes_decl(name) + self._gen_type_cleanup(name) +The visitor looks reasonable from inspection review.
Quoted 7 lines
do_builtins= False(
input_file,output_dir,do_c,do_h, prefix, opts) = \scripts/qapi-types.py +0
-0@@ -325,77 +348,10 @@ fdecl.write(mcgen(''' #include <stdint.h> '''))
Quoted 9 lines
scripts/qapi-types.py fdecl.write(mcgen(''' · line 333
Quoted 6 lines
scripts/qapi-types.py +0
-6-exprs = QAPISchema(input_file).get_exprs() - -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_fwd_builtin(typename)) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))This is a place where adding sorting in 25/47 (or a separate patch) would make the impact on generated code from this patch smaller.
You mean iterating over sorted(builtin_types.keys())? I can try and see whether it helps.
Quoted 18 lines
Quoted 18 lines
scripts/qapi-types.py +0
-17- -for expr in exprs: - ret = "" - if expr.has_key('struct'): - ret += generate_fwd_struct(expr['struct']) - elif expr.has_key('enum'): - ret += generate_enum(expr['enum'], expr['data']) - ret += generate_fwd_enum_struct(expr['enum']) - fdef.write(generate_enum_lookup(expr['enum'], expr['data'])) - elif expr.has_key('union'): - ret += generate_fwd_struct(expr['union']) - enum_define = discriminator_find_enum_define(expr) - if not enum_define: - ret += generate_enum('%sKind' % expr['union'], expr['data'].keys()) - fdef.write(generate_enum_lookup('%sKind' % expr['union'], - expr['data'].keys())) - elif expr.has_key('alternate'):
Quoted 8 lines
scripts/qapi-types.py fdecl.write(mcgen(''' · line 354
Quoted 5 lines
scripts/qapi-types.py +0
-4- ret += generate_fwd_struct(expr['alternate']) - ret += generate_enum('%sKind' % expr['alternate'], expr['data'].keys()) - fdef.write(generate_enum_lookup('%sKind' % expr['alternate'], - expr['data'].keys()))Tweaking the order of _qtypes[] vs. _lookup[] here in a pre-req patch would also help review of the diff generated by this patch.
Can try and see.
Quoted 12 lines
Quoted 12 lines
scripts/qapi-types.py +0
-12- fdef.write(generate_alternate_qtypes(expr)) - else: - continue - fdecl.write(ret) - -# to avoid header dependency hell, we always generate declarations -# for built-in types in our header files and simply guard them -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_type_cleanup_decl(typename + "List")) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) -
Quoted 8 lines
scripts/qapi-types.py fdecl.write(mcgen(''' · line 372
Quoted 6 lines
scripts/qapi-types.py +0
-6-# ...this doesn't work for cases where we link in multiple objects that -# have the functions defined, so we use -b option to provide control -# over these cases -if do_builtins: - for typename in builtin_types.keys(): - fdef.write(generate_type_cleanup(typename + "List"))Another iteration worth sorting in an earlier patch.
Can try and see.
Quoted 4 lines
Quoted 4 lines
scripts/qapi-types.py +0
-4- -for expr in exprs: - ret = "" - if expr.has_key('struct'):
Quoted 8 lines
scripts/qapi-types.py fdecl.write(mcgen(''' · line 381
Quoted 5 lines
scripts/qapi-types.py +0
-5- ret += generate_struct(expr) + "\n" - ret += generate_type_cleanup_decl(expr['struct'] + "List") - fdef.write(generate_type_cleanup(expr['struct'] + "List")) - ret += generate_type_cleanup_decl(expr['struct']) - fdef.write(generate_type_cleanup(expr['struct']))I think part of the large size of the generated diff comes from whether array fooList types are emitted in a different order via the visitor.
That's harder to avoid. The old code generates them from within the code generating the element type. The new code doesn't, because it makes all array types full-fledged types, which get visited like any other type.
Quoted 38 lines
Quoted 38 lines
scripts/qapi-types.py +5
-18- elif expr.has_key('union'): - ret += generate_union(expr, 'union') + "\n" - ret += generate_type_cleanup_decl(expr['union'] + "List") - fdef.write(generate_type_cleanup(expr['union'] + "List")) - ret += generate_type_cleanup_decl(expr['union']) - fdef.write(generate_type_cleanup(expr['union'])) - elif expr.has_key('alternate'): - ret += generate_union(expr, 'alternate') + "\n" - ret += generate_type_cleanup_decl(expr['alternate'] + "List") - fdef.write(generate_type_cleanup(expr['alternate'] + "List")) - ret += generate_type_cleanup_decl(expr['alternate']) - fdef.write(generate_type_cleanup(expr['alternate'])) - elif expr.has_key('enum'): - ret += "\n" + generate_type_cleanup_decl(expr['enum'] + "List") - fdef.write(generate_type_cleanup(expr['enum'] + "List")) - else: - continue - fdecl.write(ret) +schema = QAPISchema(input_file) +gen = QAPISchemaGenTypeVisitor() +schema.visit(gen) +fdef.write(gen.defn) +fdecl.write(gen.decl)
close_output(fdef, fdecl)tests/qapi-schema/qapi-schema-test.json +2
-2@@ -39,8 +39,8 @@ 'data': { 'value1' : 'UserDefA', 'value2' : 'UserDefB', 'value3' : 'UserDefB' } } -# FIXME generated struct UserDefFlatUnion has members for direct base -# UserDefUnionBase, but lacks members for indirect base UserDefZero +# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit +# members of indirect base UserDefZero
Quoted 5 lines
tests/qapi-schema/qapi-schema-test.json line 46
Quoted 3 lines
tests/qapi-schema/qapi-schema-test.json +0
-0{ 'struct': 'UserDefUnionBase', 'base': 'UserDefZero',Overall, moving in the right direction.
Thanks!
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
In reply to Markus ArmbrusterQuoted 3 lines · Markus Armbruster
Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:
Okay, I see a cause for part of my confusion.
Quoted 4 lines
(quoted diff) +2
-0+class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): + def visit_end(self):
Review
Quoted 10 lines
(quoted diff) +10
-0+ self.decl = self.fwdecl + self.decl + self.fwdecl = None + self.defn = self.fwdefn + self.defn + self.fwdefn = None + # To avoid header dependency hell, we always generate + # declarations for built-in types in our header files and + # simply guard them. + self.btin += guardend('QAPI_TYPES_BUILTIN') + self.decl = self.btin + self.decl + self.btin = None
The new code goes to great lengths to separate all builtin decls up front. But...
Quoted 6 lines
(quoted diff) +6
-0+ # Doesn't work for cases where we link in multiple objects + # that have the functions defined, so generate them only with + # option -b (do_builtins). + def _gen_type_cleanup(self, name): + self.decl += generate_type_cleanup_decl(name) + self.defn += generate_type_cleanup(name)
Review
Quoted 10 lines
(quoted diff) +10
-0+ def visit_enum_type(self, name, info, values): + self.fwdecl += generate_enum(name, values) + self.fwdefn += generate_enum_lookup(name, values) + def visit_array_type(self, name, info, element_type): + if isinstance(element_type, QAPISchemaBuiltinType): + self.btin += gen_fwd_object_or_array(name) + self.btin += gen_array(name, element_type) + self.btin += generate_type_cleanup_decl(name) + if do_builtins: + self.defn += generate_type_cleanup(name)
...it is still interleaving builtin defns with everything else.
Review
Quoted 6 lines
(quoted diff) +0
-6-exprs = QAPISchema(input_file).get_exprs() - -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_fwd_builtin(typename)) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
Meanwhile, the old code did builtin intList definitions up front...
Review
Quoted 11 lines
(quoted diff) +0
-10- -for expr in exprs: - ret = "" - if expr.has_key('struct'): -# to avoid header dependency hell, we always generate declarations -# for built-in types in our header files and simply guard them -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_type_cleanup_decl(typename + "List")) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
...but declared the cleanup functions at the end (so code motion 1 is that the cleanup functions are now interleaved with the intList declarations)...
Review
Quoted 7 lines
(quoted diff) +0
-7- -# ...this doesn't work for cases where we link in multiple objects that -# have the functions defined, so we use -b option to provide control -# over these cases -if do_builtins: - for typename in builtin_types.keys(): - fdef.write(generate_type_cleanup(typename + "List"))
...and wrote the definitions at the end (so code motion 2 is that the definitions are now interleaved with all other definitions).
Attached a few quick hacks that reduce (but not eliminate) some of the visitor's churn to the generated files, by consolidating the location of the builtins and cleaning up struct declarations as separate cleanups (to be applied prior to 26/47), then updating the visitor to use the same builtin order (to be applied after or squashed with 26/47).
It still doesn't solve everything, but with those hacks, I was able to get to a slightly more manageable:
qapi-types.c | 864 ++++---- qapi-types.h | 3602 ++++++++++++++++++------------------ qga/qapi-generated/qga-qapi-types.c | 110 - qga/qapi-generated/qga-qapi-types.h | 405 ++-- 4 files changed, 2585 insertions(+), 2396 deletions(-)
I'm sure there are further things that could be done, but at this point, I hope you get my picture, and I'll quit focusing on this particular patch.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
From 00df19b3b0d497e0a4670308f971ca95383b20cd Mon Sep 17 00:00:00 2001
From: Eric Blake <[email protected]>
Date: Wed, 22 Jul 2015 13:59:20 -0600
Subject: [PATCH 1/4] qapi-types: sort and consolidate builtins
Just as sorting generated code based on user-defined structs will help analysis of later patches, we also want to sort builtin types rather than letting python randomly dump a hash table. We also want to consolidate the two conditional areas of the generated .h file into one.
--- scripts/qapi-types.py | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-)
scripts/qapi-types.py +5 -9
@@ -327,11 +327,6 @@ fdecl.write(mcgen(''' exprs = QAPISchema(input_file).get_exprs() -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_fwd_builtin(typename)) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL")) - for expr in exprs: ret = "" if expr.has_key('struct'):
@@ -359,16 +354,17 @@ for expr in exprs: # to avoid header dependency hell, we always generate declarations # for built-in types in our header files and simply guard them -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) -for typename in builtin_types.keys(): +fdecl.write(guardstart("QAPI_TYPES_BUILTIN")) +for typename in sorted(builtin_types.keys()): + fdecl.write(generate_fwd_builtin(typename)) fdecl.write(generate_type_cleanup_decl(typename + "List")) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) +fdecl.write(guardend("QAPI_TYPES_BUILTIN")) # ...this doesn't work for cases where we link in multiple objects that # have the functions defined, so we use -b option to provide control # over these cases if do_builtins: - for typename in builtin_types.keys(): + for typename in sorted(builtin_types.keys()): fdef.write(generate_type_cleanup(typename + "List")) for expr in exprs:
Signature · 1 line
2.4.3
From eac8ce42799ecb17271a70aa52185b54173de399 Mon Sep 17 00:00:00 2001
From: Eric Blake <[email protected]>
Date: Wed, 22 Jul 2015 14:00:06 -0600
Subject: [PATCH 2/4] qapi: update typedef usage
Change from:
typedef struct fooList {
...
struct fooList *next;} fooList;
to:
typedef struct fooList fooList;
struct fooList {
...
struct fooList *next;};
If desired, we could further eliminate the now-spurious 'struct' from the 'next' member of the list.
Note that qemu does not have any standard on which of the two forms is preferred, and that more uses tend to go with the typedef and declaration smashed into one.
--- scripts/qapi-types.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
scripts/qapi-types.py +12 -6
@@ -15,13 +15,15 @@ from qapi import * def generate_fwd_builtin(name): return mcgen(''' -typedef struct %(name)sList { +typedef struct %(name)sList %(name)sList; + +struct %(name)sList { union { %(type)s value; uint64_t padding; }; struct %(name)sList *next; -} %(name)sList; +}; ''', type=c_type(name), name=name)
@@ -31,26 +33,30 @@ def generate_fwd_struct(name): typedef struct %(name)s %(name)s; -typedef struct %(name)sList { +typedef struct %(name)sList %(name)sList; + +struct %(name)sList { union { %(name)s *value; uint64_t padding; }; struct %(name)sList *next; -} %(name)sList; +}; ''', name=c_name(name)) def generate_fwd_enum_struct(name): return mcgen(''' -typedef struct %(name)sList { +typedef struct %(name)sList %(name)sList; + +struct %(name)sList { union { %(name)s value; uint64_t padding; }; struct %(name)sList *next; -} %(name)sList; +}; ''', name=c_name(name))
Signature · 1 line
2.4.3
From 348f3d3d3a35afe7dca4b695972677ce03da91ca Mon Sep 17 00:00:00 2001
From: Eric Blake <[email protected]>
Date: Wed, 22 Jul 2015 14:58:25 -0600
Subject: [PATCH 4/4] qapi: squash to visitor
Prior to conversion to a visitor, all code related to lists of builtins were lumped together. Doing so in the visitor code reduces the churn in comparing the generated files, making it easier to prove the conversion to the visitor is correct.
--- scripts/qapi-types.py | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-)
scripts/qapi-types.py +15 -16
@@ -240,27 +240,26 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): self.defn = None self.fwdecl = None self.fwdefn = None - self.btin = None + self.btindecl = None + self.btindefn = None def visit_begin(self): self.decl = '' self.defn = '' self.fwdecl = '' self.fwdefn = '' - self.btin = guardstart('QAPI_TYPES_BUILTIN') + self.btindecl = guardstart('QAPI_TYPES_BUILTIN') + self.btindefn = '' def visit_end(self): - self.decl = self.fwdecl + self.decl - self.fwdecl = None - self.defn = self.fwdefn + self.defn - self.fwdefn = None # To avoid header dependency hell, we always generate # declarations for built-in types in our header files and # simply guard them. - self.btin += guardend('QAPI_TYPES_BUILTIN') - self.decl = self.btin + self.decl - self.btin = None - # Doesn't work for cases where we link in multiple objects - # that have the functions defined, so generate them only with - # option -b (do_builtins). + self.btindecl += guardend('QAPI_TYPES_BUILTIN') + self.decl = self.fwdecl + self.btindecl + self.decl + self.fwdecl = None + self.btindecl = None + self.defn = self.fwdefn + self.btindefn + self.defn + self.fwdefn = None + self.btindefn = None def _gen_type_cleanup(self, name): self.decl += generate_type_cleanup_decl(name) self.defn += generate_type_cleanup(name)
@@ -269,11 +268,11 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): self.fwdefn += generate_enum_lookup(name, values) def visit_array_type(self, name, info, element_type): if isinstance(element_type, QAPISchemaBuiltinType): - self.btin += gen_fwd_object_or_array(name) - self.btin += gen_array(name, element_type) - self.btin += generate_type_cleanup_decl(name) + self.btindecl += gen_fwd_object_or_array(name) + self.btindecl += gen_array(name, element_type) + self.btindecl += generate_type_cleanup_decl(name) if do_builtins: - self.defn += generate_type_cleanup(name) + self.btindefn += generate_type_cleanup(name) else: self.fwdecl += gen_fwd_object_or_array(name) self.decl += gen_array(name, element_type)
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Quoted 9 lines · Eric Blake
Quoted 3 lines · Markus Armbruster
Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:
Okay, I see a cause for part of my confusion.
Quoted 1 line
Quoted 8 lines
...and wrote the definitions at the end (so code motion 2 is that the definitions are now interleaved with all other definitions).
Attached a few quick hacks that reduce (but not eliminate) some of the visitor's churn to the generated files, by consolidating the location of the builtins and cleaning up struct declarations as separate cleanups (to be applied prior to 26/47), then updating the visitor to use the same builtin order (to be applied after or squashed with 26/47).
By the way, I'm perfectly fine if the initial conversion to the visitor keeps .c output separate into two groups to make review easier, then a later patch simplifies things back to a single sorting over the whole file (and undoing the grouping after conversion to visitor will be easier than figuring out how to interleave the builtin array visitors into the right sorted order prior to use of the visitor). I like the final result of everything sorted, it's just that I think it is easier to validate the final result when it is broken into stages with no one step doing too much, even if some of the intermediate stages do more work than needed and get undone by the final stage.
Or put another way, your next spin of the series could include your original 26 + my hack 003 squashed together as one patch (conversion to visitor with minimized generated diff noise), followed by a revert of my hack 003 as the next (no need to group builtins together in the long run).
[and for the record, I mention this because that's the approach I took in my earlier cleanups: commit fd41dd4e included a hack to rename 'type' to 'struct' in the generator, so that commit 895a2a80 could be quite mindless without touching the generator, and then commit 3e391d35 undid my hack. When I originally wrote the conversion in an earlier version of the series, I did not have that hack in place and the resulting single patch was too big; breaking it up made my submission and the review easier.]
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
In reply to Eric BlakeQuoted 22 lines · Eric Blake
Quoted 3 lines · Markus Armbruster
Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:
Okay, I see a cause for part of my confusion.
Quoted 4 lines
(quoted diff) +2
-0+class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): + def visit_end(self):Review
Quoted 10 lines
(quoted diff) +10
-0+ self.decl = self.fwdecl + self.decl + self.fwdecl = None + self.defn = self.fwdefn + self.defn + self.fwdefn = None + # To avoid header dependency hell, we always generate + # declarations for built-in types in our header files and + # simply guard them. + self.btin += guardend('QAPI_TYPES_BUILTIN') + self.decl = self.btin + self.decl + self.btin = None
Quoted 2 lines
The new code goes to great lengths to separate all builtin decls up front. But...
Because they all need to be under the QAPI_TYPES_BUILTIN guard. If I didn't separate them, each of them would need its own guard, which would be ugly.
Quoted 17 lines
Quoted 6 lines
(quoted diff) +6
-0+ # Doesn't work for cases where we link in multiple objects + # that have the functions defined, so generate them only with + # option -b (do_builtins). + def _gen_type_cleanup(self, name): + self.decl += generate_type_cleanup_decl(name) + self.defn += generate_type_cleanup(name)Review
Quoted 10 lines
(quoted diff) +10
-0+ def visit_enum_type(self, name, info, values): + self.fwdecl += generate_enum(name, values) + self.fwdefn += generate_enum_lookup(name, values) + def visit_array_type(self, name, info, element_type): + if isinstance(element_type, QAPISchemaBuiltinType): + self.btin += gen_fwd_object_or_array(name) + self.btin += gen_array(name, element_type) + self.btin += generate_type_cleanup_decl(name) + if do_builtins: + self.defn += generate_type_cleanup(name)
Quoted 1 line
...it is still interleaving builtin defns with everything else.
Because there's no such need for the builtin definitions.
Quoted 42 lines
scripts/qapi-types.py fdecl.write(mcgen(''' · line 333
Quoted 6 lines
scripts/qapi-types.py +0
-6-exprs = QAPISchema(input_file).get_exprs() - -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_fwd_builtin(typename)) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))Meanwhile, the old code did builtin intList definitions up front...
Review
Quoted 11 lines
(quoted diff) +0
-10- -for expr in exprs: - ret = "" - if expr.has_key('struct'): -# to avoid header dependency hell, we always generate declarations -# for built-in types in our header files and simply guard them -fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_type_cleanup_decl(typename + "List")) -fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))...but declared the cleanup functions at the end (so code motion 1 is that the cleanup functions are now interleaved with the intList declarations)...
Review
Quoted 7 lines
(quoted diff) +0
-7- -# ...this doesn't work for cases where we link in multiple objects that -# have the functions defined, so we use -b option to provide control -# over these cases -if do_builtins: - for typename in builtin_types.keys(): - fdef.write(generate_type_cleanup(typename + "List"))...and wrote the definitions at the end (so code motion 2 is that the definitions are now interleaved with all other definitions).
Attached a few quick hacks that reduce (but not eliminate) some of the visitor's churn to the generated files, by consolidating the location of the builtins and cleaning up struct declarations as separate cleanups (to be applied prior to 26/47), then updating the visitor to use the same builtin order (to be applied after or squashed with 26/47).
Quoted 8 lines
It still doesn't solve everything, but with those hacks, I was able to get to a slightly more manageable:
qapi-types.c | 864 ++++---- qapi-types.h | 3602 ++++++++++++++++++------------------ qga/qapi-generated/qga-qapi-types.c | 110 - qga/qapi-generated/qga-qapi-types.h | 405 ++-- 4 files changed, 2585 insertions(+), 2396 deletions(-)
Still plenty bad...
Quoted 2 lines
I'm sure there are further things that could be done, but at this point, I hope you get my picture, and I'll quit focusing on this particular patch.
We need to decide how much code churn to accept just for making the diff of the generated code easier to review.
Thanks!
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Quoted 6 lines · Markus Armbruster
Quoted 3 lines
I'm sure there are further things that could be done, but at this point, I hope you get my picture, and I'll quit focusing on this particular patch.
We need to decide how much code churn to accept just for making the diff of the generated code easier to review.
At this point, I'd be happy with just adding a script or other high-level instructions in the commit message that says how to divide a generated file into pieces (pull out all *List types into one piece, all typedefs into another, etc) and which can be done both pre- and post-patch. With pieces in hand, if you can easily compare that each pair is minimally different, then you have a nice reassurance that the difference in the overall file is due merely to differences in how the pieces are interleaved, and not to added or unintentionally dropped material. As you pointed out, there comes a point of diminishing returns in trying to clean up code that will just be discarded.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Quoted 9 lines · Eric Blake
Quoted 6 lines · Markus Armbruster
Quoted 3 lines
I'm sure there are further things that could be done, but at this point, I hope you get my picture, and I'll quit focusing on this particular patch.
We need to decide how much code churn to accept just for making the diff of the generated code easier to review.
Quoted 10 lines
At this point, I'd be happy with just adding a script or other high-level instructions in the commit message that says how to divide a generated file into pieces (pull out all *List types into one piece, all typedefs into another, etc) and which can be done both pre- and post-patch. With pieces in hand, if you can easily compare that each pair is minimally different, then you have a nice reassurance that the difference in the overall file is due merely to differences in how the pieces are interleaved, and not to added or unintentionally dropped material. As you pointed out, there comes a point of diminishing returns in trying to clean up code that will just be discarded.
I'll give it a try.
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
In reply to Markus ArmbrusterQuoted 17 lines · Markus Armbruster
Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:
scripts/qapi-types.py +6
-3-def generate_alternate_qtypes(expr): +def gen_alternate_qtypes_decl(name): + return mcgen(''' - name = expr['alternate'] - members = expr['data'] +extern const int %(c_name)s_qtypes[]; +''', + c_name=c_name(name)) +def gen_alternate_qtypes(name, variants): ret = mcgen('''
scripts/qapi-types.py typedef enum %(name)s { · line 154
Quoted 10 lines
scripts/qapi-types.py +3
-3const int %(name)s_qtypes[QTYPE_MAX] = { ''', name=c_name(name)) - for key in members: - qtype = find_alternate_member_qtype(members[key]) - assert qtype, "Invalid alternate member" + for var in variants.variants: + qtype = var.type.alternate_qtype() + assert qtype
I think I found a couple more corner case bugs here. We are using C99 initialization of the array; for example:
const int BlockdevRef_qtypes[QTYPE_MAX] = {
[QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION, [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE,
};
but paired with an enum that starts at 0:
typedef enum BlockdevRefKind {
BLOCKDEV_REF_KIND_DEFINITION = 0,
BLOCKDEV_REF_KIND_REFERENCE = 1,
BLOCKDEV_REF_KIND_MAX = 2,
} BlockdevRefKind;
and that means that every QTYPE_ constant that we don't specify in _qtypes[] is also assigned the value 0 (aka BLOCKDEV_REF_KIND_DEFINITION in this example). In operation, calling something like:
{"execute":"blockdev-add","arguments":{"options":
{"driver":"raw","id":"a","file":true}}}
which is invalid per the .json description ("file" must be string or object, not boolean), still manages to get past visit_get_next_type() with success, and fall through to the 0th branch of the switch. If that 0th branch happens to be a struct (as it is for BlockdevRef), then we fortunately catch the error on the very next parse call, where qmp_input_start_struct() complains:
{"error": {"class": "GenericError", "desc": "Invalid parameter type for 'file', expected: QDict"}}
But what happens if the 0th branch is mapped to a different parser, as would be the case if one of the alternate's branches is 'number'? In particular, qmp_input_type_number() accepts BOTH QFloat and QInt types.
So, if we have this qapi:
{ 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
but pass in an integer, visit_get_next_type() will see a qtype of QInt, but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the string parser doesn't like ints) even though the parse should succeed by using the FOO_KIND_B branch.
Interestingly, this means that if we ever write an alternate type that accepts both 'int' and 'number' (we have not attempted that so far), then the number branch will only be taken for inputs that don't also look like ints (normally, 'number' accepts anything numeric). Maybe that means we should document and enforce that 'number' and 'int' cannot be mixed in the same alternate?
So, the bugs are: visit_get_next_type() can't tell the difference between a *_qtypes[] lookup that was explicitly initialized to 0 from one that was accidentally left that way, and therefore can't report failure for an unexpected type (but mostly mitigated by the fact that always returning 0 means the parser will attempt to parse the first branch of the alternate and gracefully fail); and that we don't properly handle QInt for an alternate that accepts 'number' but not 'int'.
I don't think either bug has to be fixed in your series, although you may want to add tests.
The first bug could be resolved by guaranteeing that the _qtypes[] array has non-zero values for the explicitly initialized lookups, and teaching visit_get_next_type() that a lookup that produces 0 meant that an unexpected type was encountered. Perhaps by changing the creation of _qtypes[] in qapi-types.c to list:
const int BlockdevRef_qtypes[QTYPE_MAX] = {
[QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION + 1, [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE + 1,
};
and then having visit_get_next_type() subtract one after verifying a non-zero value was looked up. Or perhaps leave _qtypes alone, and instead change the alternate enum to have a placeholder at 0:
typedef enum BlockdevRefKind {
BLOCKDEV_REF_KIND_INVALID = 0,
BLOCKDEV_REF_KIND_DEFINITION = 1,
BLOCKDEV_REF_KIND_REFERENCE = 2,
BLOCKDEV_REF_KIND_MAX = 3,
} BlockdevRefKind;
and then teaching the generator for visit_type_BlockdevRef() to emit an error if branch 0 is hit.
Fixing the second bug probably entails teaching the generator that if an alternate contains 'number' but not 'int', then we need [QTYPE_QINT] to map to the same lookup value as [QTYPE_QNUMBER].
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Quoted 46 lines · Eric Blake
Quoted 17 lines · Markus Armbruster
Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:
scripts/qapi-types.py +6
-3-def generate_alternate_qtypes(expr): +def gen_alternate_qtypes_decl(name): + return mcgen(''' - name = expr['alternate'] - members = expr['data'] +extern const int %(c_name)s_qtypes[]; +''', + c_name=c_name(name)) +def gen_alternate_qtypes(name, variants): ret = mcgen('''scripts/qapi-types.py typedef enum %(name)s { · line 154
Quoted 10 lines
scripts/qapi-types.py +3
-3const int %(name)s_qtypes[QTYPE_MAX] = { ''', name=c_name(name)) - for key in members: - qtype = find_alternate_member_qtype(members[key]) - assert qtype, "Invalid alternate member" + for var in variants.variants: + qtype = var.type.alternate_qtype() + assert qtypeI think I found a couple more corner case bugs here. We are using C99 initialization of the array; for example:
const int BlockdevRef_qtypes[QTYPE_MAX] = {
[QTYPE_QDICT] =BLOCKDEV_REF_KIND_DEFINITION, [QTYPE_QSTRING] =BLOCKDEV_REF_KIND_REFERENCE,
};but paired with an enum that starts at 0:
typedef enum BlockdevRefKind {
BLOCKDEV_REF_KIND_DEFINITION= 0,
BLOCKDEV_REF_KIND_REFERENCE= 1,
BLOCKDEV_REF_KIND_MAX= 2,
} BlockdevRefKind;
Quoted 2 lines
and that means that every
QTYPE_constant that we don't specify in _qtypes[] is also assigned the value 0 (akaBLOCKDEV_REF_KIND_DEFINITION
I see where this is going...
Quoted 12 lines
in this example). In operation, calling something like:
{"execute":"blockdev-add","arguments":{"options":
{"driver":"raw","id":"a","file":true}}}which is invalid per the .json description ("file" must be string or object, not boolean), still manages to get past
visit_get_next_type()with success, and fall through to the 0th branch of the switch. If that 0th branch happens to be a struct (as it is for BlockdevRef), then we fortunately catch the error on the very next parse call, whereqmp_input_start_struct()complains:
Quoted 2 lines
{"error": {"class": "GenericError", "desc": "Invalid parameter type for 'file', expected: QDict"}}
A lucky case.
Quoted 10 lines
But what happens if the 0th branch is mapped to a different parser, as would be the case if one of the alternate's branches is 'number'? In particular,
qmp_input_type_number()accepts BOTH QFloat and QInt types.
So, if we have this qapi:
{ 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
but pass in an integer,visit_get_next_type()will see a qtype of QInt, but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the string parser doesn't like ints) even though the parse should succeed by using theFOO_KIND_Bbranch.
Yup, bug.
Quoted 6 lines
Interestingly, this means that if we ever write an alternate type that accepts both 'int' and 'number' (we have not attempted that so far), then the number branch will only be taken for inputs that don't also look like ints (normally, 'number' accepts anything numeric). Maybe that means we should document and enforce that 'number' and 'int' cannot be mixed in the same alternate?
Even if we outlaw mixing the two, I'm afraid we still have this bug: an alternate with a 'number' member rejects input that gets parsed as QTYPE_QINT.
Let's simply make alternates behave sanely:
alternate has case selected for
'int' 'number' QTYPE_QINT QTYPE_QFLOAT
no no error error
no yes 'number' 'number'
yes no 'int' error
yes yes 'int' 'number'Quoted 6 lines
So, the bugs are:
visit_get_next_type()can't tell the difference between a *_qtypes[] lookup that was explicitly initialized to 0 from one that was accidentally left that way, and therefore can't report failure for an unexpected type (but mostly mitigated by the fact that always returning 0 means the parser will attempt to parse the first branch of the alternate and gracefully fail);
Yes.
Quoted 2 lines
and that we don't properly
handle QInt for an alternate that accepts 'number' but not 'int'.
Yes.
Quoted 2 lines
I don't think either bug has to be fixed in your series, although you may want to add tests.
Agree.
Quoted 6 lines
The first bug could be resolved by guaranteeing that the _qtypes[] array has non-zero values for the explicitly initialized lookups, and teaching
visit_get_next_type()that a lookup that produces 0 meant that an unexpected type was encountered. Perhaps by changing the creation of _qtypes[] inqapi-types.cto list:
Quoted 7 lines
const int BlockdevRef_qtypes[QTYPE_MAX] = {
[QTYPE_QDICT] =BLOCKDEV_REF_KIND_DEFINITION+ 1, [QTYPE_QSTRING] =BLOCKDEV_REF_KIND_REFERENCE+ 1,
};and then having
visit_get_next_type()subtract one after verifying a non-zero value was looked up.
+ 1 works, because the element type is int, not BlockdevRefKind. It's int so it can serve as argument for visit_get_next_type()'s parameter const int *qtypes.
The + 1, - 1 business could be mildly confusing. We could set all unused elements to -1 instead:
const int BlockdevRef_qtypes[QTYPE_MAX] = {
[QTYPE_NONE] = -1,
[QTYPE_QNULL] = -1,
[QTYPE_QINT] = -1,
[QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE + 1,
[QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION + 1,
[QTYPE_QLIST] = -1,
[QTYPE_QFLOAT] = -1,
[QTYPE_QBOOL] = -1,
};I wouldn't recommend that in hand-written code, but generating it should be fine.
Quoted 10 lines
Or perhaps leave
_qtypesalone, and
instead change the alternate enum to have a placeholder at 0:typedef enum BlockdevRefKind {
BLOCKDEV_REF_KIND_INVALID= 0,
BLOCKDEV_REF_KIND_DEFINITION= 1,
BLOCKDEV_REF_KIND_REFERENCE= 2,
BLOCKDEV_REF_KIND_MAX= 3,
} BlockdevRefKind;
Quoted 2 lines
and then teaching the generator for
visit_type_BlockdevRef()to emit an error if branch 0 is hit.
BLOCKDEV_REF_KIND_INVALID could get in the way elsewhere, e.g. with -Wswitch.
Quoted 3 lines
Fixing the second bug probably entails teaching the generator that if an alternate contains 'number' but not 'int', then we need [
QTYPE_QINT] to map to the same lookup value as [QTYPE_QNUMBER].
Add test eight test cases from my table above, then fix the generator to make them pass.
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Quoted 11 lines · Markus Armbruster
Quoted 10 lines
But what happens if the 0th branch is mapped to a different parser, as would be the case if one of the alternate's branches is 'number'? In particular,
qmp_input_type_number()accepts BOTH QFloat and QInt types.
So, if we have this qapi:
{ 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
but pass in an integer,visit_get_next_type()will see a qtype of QInt, but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the string parser doesn't like ints) even though the parse should succeed by using theFOO_KIND_Bbranch.
Quoted 1 line
Yup, bug.
And it's an order-dependent bug - merely declaring 'b' first makes it appear to work correctly.
Quoted 12 lines
Quoted 7 lines
Interestingly, this means that if we ever write an alternate type that accepts both 'int' and 'number' (we have not attempted that so far), then the number branch will only be taken for inputs that don't also look like ints (normally, 'number' accepts anything numeric). Maybe that means we should document and enforce that 'number' and 'int' cannot be mixed in the same alternate?
Even if we outlaw mixing the two, I'm afraid we still have this bug: an alternate with a 'number' member rejects input that gets parsed as
QTYPE_QINT.
Quoted 8 lines
Let's simply make alternates behave sanely:
alternate has case selected for 'int' 'number' QTYPE_QINT QTYPE_QFLOAT no no error error no yes 'number' 'number' yes no 'int' error yes yes 'int' 'number'
Works for me.
Quoted 7 lines
+ 1 works, because the element type is int, not BlockdevRefKind. It's int so it can serve as argument for visit_get_next_type()'s parameter const int *qtypes.
The + 1, - 1 business could be mildly confusing. We could set all unused elements to -1 instead:
Or, we could ditch the qtypes lookup altogether, and merely create the alternate enum as a non-consecutive QTYPE mapping, for one less level of indirection, as in:
typedef enum BlockdevRefKind {
BLOCKDEV_REF_DEFINITION = QTYPE_QOBJECT,
BLOCKDEV_REF_REFERENCE = QTYPE_QSTRING,
};
then rewrite visit_get_next_type() to directly return the qtype, as well as rewrite the generated switch statement in visit_type_BlockdevRef() to directly inspect the qtypes it cares about. In fact, that's the approach I'm currently playing with.
Quoted 2 lines
Add test eight test cases from my table above, then fix the generator to make them pass.
I hope to post an RFC followup patch along those lines later today.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Quoted 50 lines · Eric Blake
Quoted 11 lines · Markus Armbruster
Quoted 10 lines
But what happens if the 0th branch is mapped to a different parser, as would be the case if one of the alternate's branches is 'number'? In particular,
qmp_input_type_number()accepts BOTH QFloat and QInt types.
So, if we have this qapi:
{ 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
but pass in an integer,visit_get_next_type()will see a qtype of QInt, but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the string parser doesn't like ints) even though the parse should succeed by using theFOO_KIND_Bbranch.Quoted 1 line
Yup, bug.
And it's an order-dependent bug - merely declaring 'b' first makes it appear to work correctly.
Quoted 12 lines
Quoted 7 lines
Interestingly, this means that if we ever write an alternate type that accepts both 'int' and 'number' (we have not attempted that so far), then the number branch will only be taken for inputs that don't also look like ints (normally, 'number' accepts anything numeric). Maybe that means we should document and enforce that 'number' and 'int' cannot be mixed in the same alternate?
Even if we outlaw mixing the two, I'm afraid we still have this bug: an alternate with a 'number' member rejects input that gets parsed as
QTYPE_QINT.Quoted 8 lines
Let's simply make alternates behave sanely:
alternate has case selected for 'int' 'number' QTYPE_QINT QTYPE_QFLOAT no no error error no yes 'number' 'number' yes no 'int' error yes yes 'int' 'number'Works for me.
Quoted 7 lines
+ 1 works, because the element type is int, not BlockdevRefKind. It's int so it can serve as argument for visit_get_next_type()'s parameter const int *qtypes.
The + 1, - 1 business could be mildly confusing. We could set all unused elements to -1 instead:
Quoted 6 lines
Or, we could ditch the qtypes lookup altogether, and merely create the alternate enum as a non-consecutive QTYPE mapping, for one less level of indirection, as in:
typedef enum BlockdevRefKind {
BLOCKDEV_REF_DEFINITION=QTYPE_QOBJECT,
QTYPE_QDICT, but I get what you mean.
Quoted 7 lines
BLOCKDEV_REF_REFERENCE=QTYPE_QSTRING,
};then rewrite
visit_get_next_type()to directly return the qtype, as well as rewrite the generated switch statement invisit_type_BlockdevRef()to directly inspect the qtypes it cares about. In fact, that's the approach I'm currently playing with.
Hmm. The schema currently doesn't let you control enumeration values. qapi-code-gen.txt specifies:
The enumeration values [...] are encoded as C enum integral values in generated code. While the C code starts numbering at 0, it is better to use explicit comparisons to enum values than implicit comparisons to 0; the C code will also include a generated enum member ending in _MAX for tracking the size of the enum, useful when using common functions for converting between strings and enum values.
Strictly speaking, this needn't apply to implicit enums like BlockdevRefKind. But is it a good idea to make them different?
Hmm, your new BlockdevRefKind is basically a subset of qtype_code with the members renamed. Could we simply use qtype_code directly?
Quoted 4 lines
Quoted 2 lines
Add test eight test cases from my table above, then fix the generator to make them pass.
I hope to post an RFC followup patch along those lines later today.
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Quoted 12 lines · Markus Armbruster
Quoted 6 lines
Or, we could ditch the qtypes lookup altogether, and merely create the alternate enum as a non-consecutive QTYPE mapping, for one less level of indirection, as in:
typedef enum BlockdevRefKind {
BLOCKDEV_REF_DEFINITION=QTYPE_QOBJECT,
QTYPE_QDICT, but I get what you mean.Quoted 3 lines
BLOCKDEV_REF_REFERENCE=QTYPE_QSTRING,
};
Quoted 6 lines
Quoted 4 lines
then rewrite
visit_get_next_type()to directly return the qtype, as well as rewrite the generated switch statement invisit_type_BlockdevRef()to directly inspect the qtypes it cares about. In fact, that's the approach I'm currently playing with.Hmm. The schema currently doesn't let you control enumeration values.
For public enums. But the enum for alternate is never public - you never send the name of the branch over the wire, so we don't need to stringize the name anywhere.
Quoted 10 lines
qapi-code-gen.txt specifies:
The enumeration values [...] are encoded as C enum integral values in generated code. While the C code starts numbering at 0, it is better to use explicit comparisons to enum values than implicit comparisons to 0; the C code will also include a generated enum member ending in
_MAXfor tracking the size of the enum, useful when using common functions for converting between strings and enum values.
Quoted 2 lines
Strictly speaking, this needn't apply to implicit enums like BlockdevRefKind. But is it a good idea to make them different?
I think so, but maybe under a different name (maybe BLOCKDEV_REF_REFERENCE_QTYPE) to make it more obvious that this is not the usual generated enum, but special to the alternate.
Quoted 3 lines
Hmm, your new BlockdevRefKind is basically a subset of
qtype_codewith the members renamed. Could we simply useqtype_codedirectly?
We could, except that clients that manipulate the generated struct then have to know the qtype mapping directly; while keeping symbolic names lets them do 'foo->type = BLOCKDEV_REF_REFERENCE; foo->reference = xyz;' as a nice visual indicator of which union member within the struct is being assigned according to the discriminator.
I guess I'll see how much code currently manipulates the generated structs (I already recall from other patches in this series that blockdev played a bit loose by validating that the QMP was okay and then using QDict for everything else rather than the generated struct) and make my decision when posting my RFC patch.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
Quoted 24 lines · Eric Blake
Quoted 7 lines · Markus Armbruster
Quoted 6 lines
Or, we could ditch the qtypes lookup altogether, and merely create the alternate enum as a non-consecutive QTYPE mapping, for one less level of indirection, as in:
typedef enum BlockdevRefKind {
BLOCKDEV_REF_DEFINITION=QTYPE_QOBJECT,Quoted 8 lines
QTYPE_QDICT, but I get what you mean.Quoted 3 lines
BLOCKDEV_REF_REFERENCE=QTYPE_QSTRING,
};Hmm, your new BlockdevRefKind is basically a subset of
qtype_codewith the members renamed. Could we simply useqtype_codedirectly?We could, except that clients that manipulate the generated struct then have to know the qtype mapping directly; while keeping symbolic names lets them do '
foo->type=BLOCKDEV_REF_REFERENCE;foo->reference= xyz;' as a nice visual indicator of which union member within the struct is being assigned according to the discriminator.
Quoted 5 lines
I guess I'll see how much code currently manipulates the generated structs (I already recall from other patches in this series that blockdev played a bit loose by validating that the QMP was okay and then using QDict for everything else rather than the generated struct) and make my decision when posting my RFC patch.
Turns out that using it directly was easier, and less code:
http://thread.gmane.org/gmane.comp.emulators.qemu/353204/focus=354008
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 27/47 qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
In reply to Markus ArmbrusterFixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.
Patch's effect on visit_type_UserDefFlatUnion():
static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp) {
Error *err = NULL;
+ visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err); if (err) {
goto out;
Test cases updated for the bug fix.
Fixes alternates to generate a visitor for their implicit enumeration type. None of them are currently used, obviously. Example:
block-core.json's BlockdevRef now generates visit_type_BlockdevRefKind().
--- scripts/qapi-visit.py | 254 ++++++++++++-------------------- tests/qapi-schema/qapi-schema-test.json | 3 - tests/test-qmp-input-strict.c | 2 +- tests/test-qmp-input-visitor.c | 4 +- 4 files changed, 100 insertions(+), 163 deletions(-)
scripts/qapi-visit.py +97 -157
@@ -12,7 +12,6 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -from ordereddict import OrderedDict from qapi import * import re
@@ -24,13 +23,13 @@ def generate_visit_implicit_struct(type): return '' implicit_structs_seen.add(type) ret = '' - if type not in struct_fields_seen: + if type.name not in struct_fields_seen: # Need a forward declaration ret += mcgen(''' static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp); ''', - c_type=type_name(type)) + c_type=type.c_name()) ret += mcgen('''
@@ -46,7 +45,7 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * error_propagate(errp, err); } ''', - c_type=type_name(type)) + c_type=type.c_name()) return ret def generate_visit_struct_fields(name, members, base = None):
@@ -74,24 +73,24 @@ if (err) { goto out; } ''', - type=type_name(base), c_name=c_name('base')) + type=base.c_name(), c_name=c_name('base')) - for argname, argentry, optional in parse_args(members): - if optional: + for memb in members: + if memb.optional: ret += mcgen(''' visit_optional(m, &(*obj)->has_%(c_name)s, "%(name)s", &err); if (!err && (*obj)->has_%(c_name)s) { ''', - c_name=c_name(argname), name=argname) + c_name=c_name(memb.name), name=memb.name) push_indent() ret += mcgen(''' visit_type_%(type)s(m, &(*obj)->%(c_name)s, "%(name)s", &err); ''', - type=type_name(argentry), c_name=c_name(argname), - name=argname) + type=memb.type.c_name(), c_name=c_name(memb.name), + name=memb.name) - if optional: + if memb.optional: pop_indent() ret += mcgen(''' }
@@ -132,12 +131,7 @@ def generate_visit_struct_body(name): return ret -def generate_visit_struct(expr): - - name = expr['struct'] - members = expr['data'] - base = expr.get('base') - +def gen_visit_struct(name, base, members): ret = generate_visit_struct_fields(name, members, base) ret += mcgen('''
@@ -154,10 +148,10 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e ''') return ret -def generate_visit_list(name): +def gen_visit_list(name, element_type): return mcgen(''' -void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp) +void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp) { Error *err = NULL; GenericList *i, **prev;
@@ -170,8 +164,8 @@ void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, E for (prev = (GenericList **)obj; !err && (i = visit_next_list(m, prev, &err)) != NULL; prev = &i) { - %(name)sList *native_i = (%(name)sList *)i; - visit_type_%(name)s(m, &native_i->value, NULL, &err); + %(name)s *native_i = (%(name)s *)i; + visit_type_%(c_elt_type)s(m, &native_i->value, NULL, &err); } error_propagate(errp, err);
@@ -181,7 +175,8 @@ out: error_propagate(errp, err); } ''', - name=type_name(name)) + name=c_name(name), + c_elt_type=element_type.c_name()) def generate_visit_enum(name): return mcgen('''
@@ -193,7 +188,7 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error ''', c_name=c_name(name), name=name) -def generate_visit_alternate(name, members): +def gen_visit_alternate(name, variants): ret = mcgen(''' void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
@@ -212,25 +207,17 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e ''', name=c_name(name)) - # For alternate, always use the default enum type automatically generated - # as name + 'Kind' - disc_type = c_name(name) + 'Kind' - - for key in members: - assert (members[key] in builtin_types.keys() - or find_struct(members[key]) - or find_union(members[key]) - or find_enum(members[key])), "Invalid alternate member" - - enum_full_value = c_enum_const(disc_type, key) + for var in variants.variants: + enum_full_value = c_enum_const(variants.tag_member.type.name, + var.name) ret += mcgen(''' case %(enum_full_value)s: visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, name, &err); break; ''', enum_full_value = enum_full_value, - c_type = type_name(members[key]), - c_name = c_name(key)) + c_type=var.type.c_name(), + c_name=c_name(var.name)) ret += mcgen(''' default:
@@ -247,35 +234,16 @@ out: return ret - -def generate_visit_union(expr): - - name = expr['union'] - members = expr['data'] - - base = expr.get('base') - discriminator = expr.get('discriminator') - - enum_define = discriminator_find_enum_define(expr) - if enum_define: - # Use the enum type as discriminator - ret = "" - disc_type = c_name(enum_define['enum_name']) - else: - # There will always be a discriminator in the C switch code, by default - # it is an enum type generated silently - ret = generate_visit_enum(name + 'Kind') - disc_type = c_name(name) + 'Kind' +def gen_visit_union(name, base, variants): + ret = '' if base: - assert discriminator - base_fields = find_struct(base)['data'].copy() - del base_fields[discriminator] - ret += generate_visit_struct_fields(name, base_fields) + members = [m for m in base.members if m != variants.tag_member] + ret += generate_visit_struct_fields(name, members) - if discriminator: - for key in members: - ret += generate_visit_implicit_struct(members[key]) + for var in variants.variants: + if var.flat: + ret += generate_visit_implicit_struct(var.type) ret += mcgen('''
@@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error ''', name=c_name(name)) - if not discriminator: - tag = 'kind' - disc_key = "type" - else: - tag = discriminator - disc_key = discriminator + disc_key = variants.tag_member.name + if not variants.tag_name: + # we pointlessly use a different key for simple unions + disc_key = 'type' ret += mcgen(''' - visit_type_%(disc_type)s(m, &(*obj)->%(c_tag)s, "%(disc_key)s", &err); + visit_type_%(disc_type)s(m, &(*obj)->%(c_name)s, "%(disc_key)s", &err); if (err) { goto out_obj; } if (!visit_start_union(m, !!(*obj)->data, &err) || err) { goto out_obj; } - switch ((*obj)->%(c_tag)s) { + switch ((*obj)->%(c_name)s) { ''', - disc_type = disc_type, - c_tag=c_name(tag), + disc_type=variants.tag_member.type.c_name(), + c_name=c_name(variants.tag_member.name), disc_key = disc_key) - for key in members: - if not discriminator: + for var in variants.variants: + if not var.flat: fmt = 'visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, "data", &err);' else: fmt = 'visit_type_implicit_%(c_type)s(m, &(*obj)->%(c_name)s, &err);' - enum_full_value = c_enum_const(disc_type, key) + enum_full_value = c_enum_const(variants.tag_member.type.name, var.name) ret += mcgen(''' case %(enum_full_value)s: ''' + fmt + ''' break; ''', enum_full_value = enum_full_value, - c_type=type_name(members[key]), - c_name=c_name(key)) + c_type=var.type.c_name(), + c_name=c_name(var.name)) ret += mcgen(''' default:
@@ -355,37 +321,57 @@ out: return ret -def generate_declaration(name, builtin_type=False): - ret = "" - if not builtin_type: - name = c_name(name) - ret += mcgen(''' - -void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp); -''', - name=name) - - ret += mcgen(''' -void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp); -''', - name=name) - - return ret - -def generate_enum_declaration(name): - ret = mcgen(''' -void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp); -''', - name=c_name(name)) - - return ret - -def generate_decl_enum(name): +def gen_visit_decl(name, scalar=False): + c_type = c_name(name) + ' *' + if not scalar: + c_type += '*' return mcgen(''' - -void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp); +void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp); ''', - name=c_name(name)) + c_name=c_name(name), c_type=c_type) + +class QAPISchemaGenVisitVisitor(QAPISchemaVisitor): + def __init__(self): + self.decl = None + self.defn = None + self.btin = None + def visit_begin(self): + self.decl = '' + self.defn = '' + self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL') + def visit_end(self): + # to avoid header dependency hell, we always generate + # declarations for built-in types in our header files and + # simply guard them + self.btin += guardend('QAPI_VISIT_BUILTIN_VISITOR_DECL') + self.decl = self.btin + self.decl + self.btin = None + # ...this doesn't work for cases where we link in multiple + # objects that have the functions defined, so we use + # do_builtins (option -b) to provide control + def visit_enum_type(self, name, info, values): + self.decl += gen_visit_decl(name, scalar=True) + self.defn += generate_visit_enum(name) + def visit_array_type(self, name, info, element_type): + decl = gen_visit_decl(name) + defn = gen_visit_list(name, element_type) + if isinstance(element_type, QAPISchemaBuiltinType): + self.btin += decl + if do_builtins: + self.defn += defn + else: + self.decl += decl + self.defn += defn + def visit_object_type(self, name, info, base, members, variants): + if info: + self.decl += gen_visit_decl(name) + if variants: + self.defn += gen_visit_union(name, base, variants) + else: + self.defn += gen_visit_struct(name, base, members) + def visit_alternate_type(self, name, info, variants): + self.decl += gen_visit_decl(name) + self.defn += gen_visit_alternate(name, variants) do_builtins = False
@@ -442,56 +428,10 @@ fdecl.write(mcgen(''' ''', prefix=prefix)) -exprs = QAPISchema(input_file).get_exprs() - -# to avoid header dependency hell, we always generate declarations -# for built-in types in our header files and simply guard them -fdecl.write(guardstart("QAPI_VISIT_BUILTIN_VISITOR_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_declaration(typename, builtin_type=True)) -fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) - -# ...this doesn't work for cases where we link in multiple objects that -# have the functions defined, so we use -b option to provide control -# over these cases -if do_builtins: - for typename in builtin_types.keys(): - fdef.write(generate_visit_list(typename)) - -for expr in exprs: - if expr.has_key('struct'): - ret = generate_visit_struct(expr) - ret += generate_visit_list(expr['struct']) - fdef.write(ret) - - ret = generate_declaration(expr['struct']) - fdecl.write(ret) - elif expr.has_key('union'): - ret = generate_visit_union(expr) - ret += generate_visit_list(expr['union']) - fdef.write(ret) - - enum_define = discriminator_find_enum_define(expr) - ret = "" - if not enum_define: - ret = generate_decl_enum('%sKind' % expr['union']) - ret += generate_declaration(expr['union']) - fdecl.write(ret) - elif expr.has_key('alternate'): - ret = generate_visit_alternate(expr['alternate'], expr['data']) - ret += generate_visit_list(expr['alternate']) - fdef.write(ret) - - ret = generate_decl_enum('%sKind' % expr['alternate']) - ret += generate_declaration(expr['alternate']) - fdecl.write(ret) - elif expr.has_key('enum'): - ret = generate_visit_list(expr['enum']) - ret += generate_visit_enum(expr['enum']) - fdef.write(ret) - - ret = generate_decl_enum(expr['enum']) - ret += generate_enum_declaration(expr['enum']) - fdecl.write(ret) +schema = QAPISchema(input_file) +gen = QAPISchemaGenVisitVisitor() +schema.visit(gen) +fdef.write(gen.defn) +fdecl.write(gen.decl) close_output(fdef, fdecl)
tests/qapi-schema/qapi-schema-test.json +0 -3
@@ -39,8 +39,6 @@ 'data': { 'value1' : 'UserDefA', 'value2' : 'UserDefB', 'value3' : 'UserDefB' } } -# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit -# members of indirect base UserDefZero { 'struct': 'UserDefUnionBase', 'base': 'UserDefZero',
@@ -57,7 +55,6 @@ { 'alternate': 'UserDefAlternate', 'data': { 'uda': 'UserDefA', 's': 'str', 'i': 'int' } } -# FIXME only a declaration of visit_type_UserDefAlternateKind() generated { 'struct': 'UserDefC', 'data': { 'string1': 'str', 'string2': 'str' } }
tests/test-qmp-input-strict.c +1 -1
@@ -167,9 +167,9 @@ static void test_validate_union_flat(TestInputVisitorData *data, v =validate_test_init (data,"{ 'enum1': 'value1', " +"'integer': 41, " "'string': 'str', " "'boolean': true }" ); -/* TODO when generator bug is fixed, add 'integer': 41 */ visit_type_UserDefFlatUnion (v, &tmp,NULL , &err);g_assert (!err);
tests/test-qmp-input-visitor.c +2 -2
@@ -307,15 +307,15 @@ static void test_visitor_in_union_flat(TestInputVisitorData *data, v =visitor_input_test_init (data,"{ 'enum1': 'value1', " +"'integer': 41, " "'string': 'str', " "'boolean': true }" ); -/* TODO when generator bug is fixed, add 'integer': 41 */ visit_type_UserDefFlatUnion (v, &tmp,NULL , &err);g_assert (err ==NULL ); g_assert_cmpint(tmp->enum1, ==, ENUM_ONE_VALUE1); g_assert_cmpstr(tmp->string, ==, "str"); -/* TODO g_assert_cmpint(tmp->integer, ==, 41); */ + g_assert_cmpint(tmp->integer, ==, 41); g_assert_cmpint(tmp->value1->boolean, ==, true);qapi_free_UserDefFlatUnion (tmp); }
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Quoted 24 lines · Markus Armbruster
Fixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.
Patch's effect on
visit_type_UserDefFlatUnion():static void
visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp){
Error *err = NULL;+
visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err); if (err) {
goto out;Test cases updated for the bug fix.
Fixes alternates to generate a visitor for their implicit enumeration type. None of them are currently used, obviously. Example:
block-core.json's BlockdevRef now generatesvisit_type_BlockdevRefKind().
Quoted 7 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-visit.py | 254 ++++++++++++-------------------- tests/qapi-schema/qapi-schema-test.json | 3 - tests/test-qmp-input-strict.c | 2 +- tests/test-qmp-input-visitor.c | 4 +- 4 files changed, 100 insertions(+), 163 deletions(-)
Another conversion that results in a fairly large diffstat to the generated files:
qapi-visit.c | 4542 ++++++++++++++++++------------------ qapi-visit.h | 256 -- qga/qapi-generated/qga-qapi-visit.c | 88 qga/qapi-generated/qga-qapi-visit.h | 36 4 files changed, 2355 insertions(+), 2567 deletions(-)
Same complaints as in 26/47, where splitting some of the cleanups into separate patches would make it easier to validate that the final conversion is correct.
Here, a very common thing in the generated .c file is that you end up swapping the order of visit_type_foo and visit_type_fooList, for any time when foo is an enum. [1]
Quoted 16 lines
scripts/qapi-visit.py +0
-1@@ -12,7 +12,6 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -from ordereddict import OrderedDict from qapi import * import re@@ -24,13 +23,13 @@ def generate_visit_implicit_struct(type): return '' implicit_structs_seen.add(type)
scripts/qapi-visit.py def generate_visit_implicit_struct(type): · line 32
Quoted 10 lines
scripts/qapi-visit.py +2
-2ret = '' - if type not in struct_fields_seen: + if type.name not in struct_fields_seen: # Need a forward declaration ret += mcgen(''' static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp); ''', - c_type=type_name(type)) + c_type=type.c_name())
This looks a little fishy on first read; why are we calling type.c_name() (and not type.c_type()) when assigning to a placeholder named %(c_type)s? But on second thought, it looks correct: since we really do want the name of the type (and not the magic '*' pointer suffix).
Still, it might be nicer to name things %(c_name)s here; and in that case, it's more of a pre-existing cleanup that might be better floating into one of your earlier patches.
scripts/qapi-visit.py static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * · line 48
Quoted 9 lines
scripts/qapi-visit.py +1
-1ret += mcgen('''@@ -46,7 +45,7 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * error_propagate(errp, err); } ''', - c_type=type_name(type)) + c_type=type.c_name())
same here.
scripts/qapi-visit.py if (err) { · line 76
Quoted 9 lines
scripts/qapi-visit.py +1
-1return ret def generate_visit_struct_fields(name, members, base = None):@@ -74,24 +73,24 @@ if (err) { goto out; } ''', - type=type_name(base), c_name=c_name('base')) + type=base.c_name(), c_name=c_name('base'))
And this one's pointless: c_name('base') == 'base'. Pointless since commit 622f557 introduced type inheritance. Why do we even need %(c_name)s if we are always passing a constant string?
Oh, and that means our generator has a collision bug that none of my added tests have exposed yet: you cannot have a base class and simultaneously add a member named 'base':
{ 'struct': 'Base', 'data': { 'i': 'int' } }
{ 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }
because the generated C code is trying to use the name 'base' for its own purposes. I guess that means more pre-req patches to the series to expose the bug, and either tighten the parser to reject things for now (easiest) or update the generator to not collide (harder, and fine for a later series).
By the way, now that we are emitting flat unions in such a way that you can cast to the base class, why don't we change our C code to do likewise? That is, where we now have this generated C:
struct BlockdevOptionsGenericFormat {
BlockdevRef *file;
};struct BlockdevOptionsGenericCOWFormat {
BlockdevOptionsGenericFormat *base;
bool has_backing;
BlockdevRef *backing;
};why can't we instead have an unboxed representation:
struct BlockdevOptionsGenericFormat {
BlockdevRef *file;
};/* This struct can be cast to BlockdevOptionsGenericFormat */
struct BlockdevOptionsGenericCOWFormat {
BlockdevRef *file;
/* end of fields from base class BlockdevOptionsGenericFormat */
bool has_backing;
BlockdevRef *backing;
};where client code that was referring to o->base->file now refers to o->file.
scripts/qapi-visit.py out: · line 242
Quoted 10 lines
scripts/qapi-visit.py +4
-4+def gen_visit_union(name, base, variants): + ret = '' if base: - assert discriminator - base_fields = find_struct(base)['data'].copy() - del base_fields[discriminator] - ret += generate_visit_struct_fields(name, base_fields) + members = [m for m in base.members if m != variants.tag_member] + ret += generate_visit_struct_fields(name, members)
Why not just visit ALL base class members, unconditionally?
scripts/qapi-visit.py out: · line 246
Quoted 7 lines
scripts/qapi-visit.py +3
-3- if discriminator: - for key in members: - ret += generate_visit_implicit_struct(members[key]) + for var in variants.variants: + if var.flat: + ret += generate_visit_implicit_struct(var.type)
Okay, I see where you are using .flat from the initial parse. I still think it is a bit odd that you are defining '.flat' for each 'variant' within 'variants', even though, for a given 'variants', all members will have the same setting of '.flat'. That makes me wonder if '.flat' should belong instead to the top-level 'variants' struct rather than to each 'variant' member.
But again I wonder what would happen if you had instead normalized the input of simple unions into always having an implicit struct (with single member 'data'), so that by the time you get here, you only have to deal with a single representation of unions instead of having to still emit different things for flat vs. simple (since on the wire, we already proved simple is shorthand that can be duplicated by a flat union).
Quoted 6 lines
scripts/qapi-visit.py +0
-0ret += mcgen('''@@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error ''', name=c_name(name))
scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error · line 273
Quoted 10 lines
scripts/qapi-visit.py +3
-6- if not discriminator: - tag = 'kind' - disc_key = "type" - else: - tag = discriminator - disc_key = discriminator + disc_key = variants.tag_member.name + if not variants.tag_name: + # we pointlessly use a different key for simple unions
We could fix that (as a separate patch); wonder how much C code it would affect. A lot of these things that we can alter in generated code are certainly easier to see now that we have a clean generator :)
scripts/qapi-visit.py out: · line 331
Quoted 11 lines
scripts/qapi-visit.py +6
-3+def gen_visit_decl(name, scalar=False): + c_type = c_name(name) + ' *' + if not scalar: + c_type += '*' return mcgen(''' - -void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp); +void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp); ''', - name=c_name(name)) + c_name=c_name(name), c_type=c_type)
Nice way to consolidate several near-identical copies.
Quoted 10 lines
scripts/qapi-visit.py +10
-0+ +class QAPISchemaGenVisitVisitor(QAPISchemaVisitor): + def __init__(self): + self.decl = None + self.defn = None + self.btin = None + def visit_begin(self): + self.decl = '' + self.defn = '' + self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL')
scripts/qapi-visit.py out: · line 351
Quoted 10 lines
scripts/qapi-visit.py +10
-0+ def visit_end(self): + # to avoid header dependency hell, we always generate + # declarations for built-in types in our header files and + # simply guard them + self.btin += guardend('QAPI_VISIT_BUILTIN_VISITOR_DECL') + self.decl = self.btin + self.decl + self.btin = None + # ...this doesn't work for cases where we link in multiple + # objects that have the functions defined, so we use + # do_builtins (option -b) to provide control
And once again, as in 26/47, this floats the .h file to have all builtin representations in one chunk (for continuity with pre-patch), but fails to do the same for the .c code...
scripts/qapi-visit.py out: · line 361
Quoted 10 lines
scripts/qapi-visit.py +10
-0+ def visit_enum_type(self, name, info, values): + self.decl += gen_visit_decl(name, scalar=True) + self.defn += generate_visit_enum(name) + def visit_array_type(self, name, info, element_type): + decl = gen_visit_decl(name) + defn = gen_visit_list(name, element_type) + if isinstance(element_type, QAPISchemaBuiltinType): + self.btin += decl + if do_builtins: + self.defn += defn
...where the builtins are now interleaved with everything else instead of bunched together, making the generated diff larger and more confusing than necessary.
scripts/qapi-visit.py out: · line 369
Quoted 8 lines
scripts/qapi-visit.py +8
-0+ else: + self.decl += decl + self.defn += defn + def visit_object_type(self, name, info, base, members, variants): + if info: + self.decl += gen_visit_decl(name) + if variants: + self.defn += gen_visit_union(name, base, variants)
Worth adding 'assert not members'?
scripts/qapi-visit.py
Quoted 2 lines
scripts/qapi-visit.py +2
-0+ else: + self.defn += gen_visit_struct(name, base, members)
Or maybe we can someday consolidate these two into a single gen_visit_object, that handles all members and variants in a uniform manner, instead of our current differences. I wonder how much C code would be impacted?
Quoted 15 lines
scripts/qapi-visit.py +3
-5+ def visit_alternate_type(self, name, info, variants): + self.decl += gen_visit_decl(name) + self.defn += gen_visit_alternate(name, variants) do_builtins = False@@ -442,56 +428,10 @@ fdecl.write(mcgen(''' ''', prefix=prefix)) -exprs = QAPISchema(input_file).get_exprs() - -# to avoid header dependency hell, we always generate declarations -# for built-in types in our header files and simply guard them -fdecl.write(guardstart("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
scripts/qapi-visit.py fdecl.write(mcgen(''' · line 459
Quoted 10 lines
scripts/qapi-visit.py +0
-10-for typename in builtin_types.keys(): - fdecl.write(generate_declaration(typename, builtin_type=True)) -fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) - -# ...this doesn't work for cases where we link in multiple objects that -# have the functions defined, so we use -b option to provide control -# over these cases -if do_builtins: - for typename in builtin_types.keys(): - fdef.write(generate_visit_list(typename))
Again, a well-placed sorted() over these two loops in a pre-req patch will minimize the churn on the builtins.
Quoted 8 lines
scripts/qapi-visit.py +0
-8- -for expr in exprs: - if expr.has_key('struct'): - ret = generate_visit_struct(expr) - ret += generate_visit_list(expr['struct']) - fdef.write(ret) - - ret = generate_declaration(expr['struct'])
scripts/qapi-visit.py fdecl.write(mcgen(''' · line 477
Quoted 10 lines
scripts/qapi-visit.py +0
-10- fdecl.write(ret) - elif expr.has_key('union'): - ret = generate_visit_union(expr) - ret += generate_visit_list(expr['union']) - fdef.write(ret) - - enum_define = discriminator_find_enum_define(expr) - ret = "" - if not enum_define: - ret = generate_decl_enum('%sKind' % expr['union'])
Nice that the new visitor automatically visits any implicit enum, without us having to special case it.
scripts/qapi-visit.py fdecl.write(mcgen(''' · line 490
Quoted 13 lines
scripts/qapi-visit.py +0
-13- ret += generate_declaration(expr['union']) - fdecl.write(ret) - elif expr.has_key('alternate'): - ret = generate_visit_alternate(expr['alternate'], expr['data']) - ret += generate_visit_list(expr['alternate']) - fdef.write(ret) - - ret = generate_decl_enum('%sKind' % expr['alternate']) - ret += generate_declaration(expr['alternate']) - fdecl.write(ret) - elif expr.has_key('enum'): - ret = generate_visit_list(expr['enum']) - ret += generate_visit_enum(expr['enum'])
[1] swapping these two lines in a pre-req patch will minimize the churn of this conversion.
scripts/qapi-visit.py fdecl.write(mcgen(''' · line 435
Quoted 10 lines
scripts/qapi-visit.py +5
-5- fdef.write(ret) - - ret = generate_decl_enum(expr['enum']) - ret += generate_enum_declaration(expr['enum']) - fdecl.write(ret) +schema = QAPISchema(input_file) +gen = QAPISchemaGenVisitVisitor() +schema.visit(gen) +fdef.write(gen.defn) +fdecl.write(gen.decl)
Again, overall impression is that your series is headed in the right direction.
And nice that the TODOs in the testsuite pointed out what this fixes, for visiting indirect bases.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Quoted 42 lines · Eric Blake
Quoted 24 lines · Markus Armbruster
Fixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.
Patch's effect on
visit_type_UserDefFlatUnion():static void
visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp){
Error *err = NULL;+
visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err); if (err) {
goto out;Test cases updated for the bug fix.
Fixes alternates to generate a visitor for their implicit enumeration type. None of them are currently used, obviously. Example:
block-core.json's BlockdevRef now generatesvisit_type_BlockdevRefKind().Quoted 7 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-visit.py | 254 ++++++++++++-------------------- tests/qapi-schema/qapi-schema-test.json | 3 - tests/test-qmp-input-strict.c | 2 +- tests/test-qmp-input-visitor.c | 4 +- 4 files changed, 100 insertions(+), 163 deletions(-)Another conversion that results in a fairly large diffstat to the generated files:
qapi-visit.c | 4542 ++++++++++++++++++------------------ qapi-visit.h | 256 -- qga/qapi-generated/qga-qapi-visit.c | 88 qga/qapi-generated/qga-qapi-visit.h | 36 4 files changed, 2355 insertions(+), 2567 deletions(-)
Quoted 7 lines
Same complaints as in 26/47, where splitting some of the cleanups into separate patches would make it easier to validate that the final conversion is correct.
Here, a very common thing in the generated .c file is that you end up swapping the order of
visit_type_fooandvisit_type_fooList, for any time when foo is an enum. [1]
Again, this is because the old code generates the array visitor from within the code generating the element type visitor, while the new code makes all array types full-fledged types, which get visited like any other type.
Quoted 22 lines
Quoted 22 lines
scripts/qapi-visit.py +1
-2@@ -12,7 +12,6 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -from ordereddict import OrderedDict from qapi import * import re@@ -24,13 +23,13 @@ def generate_visit_implicit_struct(type): return '' implicit_structs_seen.add(type) ret = '' - if type not in struct_fields_seen: + if type.name not in struct_fields_seen: # Need a forward declaration ret += mcgen('''
Quoted 9 lines
scripts/qapi-visit.py def generate_visit_implicit_struct(type): · line 32
Quoted 4 lines
scripts/qapi-visit.py +1
-1static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp); ''', - c_type=type_name(type)) + c_type=type.c_name())This looks a little fishy on first read; why are we calling
type.c_name()(and nottype.c_type()) when assigning to a placeholder named %(c_type)s? But on second thought, it looks correct: since we really do want the name of the type (and not the magic '*' pointer suffix).
Most of the time, c_type is an identifier naming a type (something like typ.c_name()), but sometimes it's a type expression (something like typ.c_type()). If that turns out to be too confusing, we can use different names for the two. Not exactly looking forward to tracking them down, though...
Quoted 3 lines
Still, it might be nicer to name things %(c_name)s here; and in that case, it's more of a pre-existing cleanup that might be better floating into one of your earlier patches.
This patch tries very hard not to touch lines without need. Cleanup is left for PATCH 33.
I could do some (but not all) cleanup before converting to QAPISchemaVisitor. But I'm afraid that just complicates things even more.
Quoted 12 lines
scripts/qapi-visit.py static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * · line 48
Quoted 9 lines
scripts/qapi-visit.py +1
-1ret += mcgen('''@@ -46,7 +45,7 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * error_propagate(errp, err); } ''', - c_type=type_name(type)) + c_type=type.c_name())same here.
Quoted 4 lines
scripts/qapi-visit.py if (err) {
Quoted 4 lines
scripts/qapi-visit.py +0
-0return ret def generate_visit_struct_fields(name, members, base = None):@@ -74,24 +73,24 @@ if (err) {
ret += mcgen('''
visit_type_implicit_%(c_type)s(m, &(*obj)->%(c_name)s, &err);
if (err) {Quoted 9 lines
scripts/qapi-visit.py if (err) { · line 76
Quoted 5 lines
scripts/qapi-visit.py +1
-1goto out; } ''', - type=type_name(base), c_name=c_name('base')) + type=base.c_name(), c_name=c_name('base'))And this one's pointless:
c_name('base')== 'base'. Pointless since commit 622f557 introduced type inheritance. Why do we even need %(c_name)s if we are always passing a constant string?
My best guess: to keep the code similar to other code generating visit_type_FOO() calls?
Quoted 9 lines
Oh, and that means our generator has a collision bug that none of my added tests have exposed yet: you cannot have a base class and simultaneously add a member named 'base':
{ 'struct': 'Base', 'data': { 'i': 'int' } }
{ 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }because the generated C code is trying to use the name 'base' for its own purposes.
*sigh*
Quoted 4 lines
I guess that means more pre-req patches to the series to
expose the bug, and either tighten the parser to reject things for now (easiest) or update the generator to not collide (harder, and fine for a later series).
Yes.
Life would be easier if the original authors had adopted sane naming conventions from the start.
Quoted 21 lines
By the way, now that we are emitting flat unions in such a way that you can cast to the base class, why don't we change our C code to do likewise? That is, where we now have this generated C:
struct BlockdevOptionsGenericFormat { BlockdevRef *file; };struct BlockdevOptionsGenericCOWFormat { BlockdevOptionsGenericFormat *base; bool has_backing; BlockdevRef *backing; };why can't we instead have an unboxed representation:
struct BlockdevOptionsGenericFormat { BlockdevRef *file; };/* This struct can be cast to BlockdevOptionsGenericFormat */
Quoted 8 lines
struct BlockdevOptionsGenericCOWFormat { BlockdevRef *file; /* end of fields from base class BlockdevOptionsGenericFormat */ bool has_backing; BlockdevRef *backing; };where client code that was referring to
o->base->filenow refers too->file.
That's where I want to go, but not in this series.
Quoted 11 lines
scripts/qapi-visit.py out: · line 242
Quoted 10 lines
scripts/qapi-visit.py +4
-4+def gen_visit_union(name, base, variants): + ret = '' if base: - assert discriminator - base_fields = find_struct(base)['data'].copy() - del base_fields[discriminator] - ret += generate_visit_struct_fields(name, base_fields) + members = [m for m in base.members if m != variants.tag_member] + ret += generate_visit_struct_fields(name, members)
Quoted 1 line
Why not just visit ALL base class members, unconditionally?
This patch converts to QAPISchemaVisitor. It tries hard to keep the generated code the same.
The old code generates the discriminator visit into the visit_type_FOO() and not the the visit_type_FOO_fields() helper (can't tell why), so the new code does the same.
Simply visiting all members makes more sense, but it needs to be done in a followup patch. Since this series is already huge, it'll likely be in a followup series.
Quoted 7 lines
scripts/qapi-visit.py out: · line 246
Quoted 6 lines
scripts/qapi-visit.py +3
-3- if discriminator: - for key in members: - ret += generate_visit_implicit_struct(members[key]) + for var in variants.variants: + if var.flat: + ret += generate_visit_implicit_struct(var.type)
Quoted 6 lines
Okay, I see where you are using .flat from the initial parse. I still think it is a bit odd that you are defining '.flat' for each 'variant' within 'variants', even though, for a given 'variants', all members will have the same setting of '.flat'. That makes me wonder if '.flat' should belong instead to the top-level 'variants' struct rather than to each 'variant' member.
Two reasons for putting flat where it is:
* The philosophical one: from the generator's point of view, there's no fundamental reason why all variants need to be flat or none. The generator really doesn't care.
* The pragmatic one (a.k.a. the real one): there are places where I use v.flat, but don't have the variants owning v handy.
Quoted 6 lines
But again I wonder what would happen if you had instead normalized the input of simple unions into always having an implicit struct (with single member 'data'), so that by the time you get here, you only have to deal with a single representation of unions instead of having to still emit different things for flat vs. simple (since on the wire, we already proved simple is shorthand that can be duplicated by a flat union).
I hope we can get there! But at this stage of the conversion, I want to minimize output change, and .flat makes preserving all its warts much easier.
Quoted 7 lines
Quoted 7 lines
scripts/qapi-visit.py +0
-0ret += mcgen('''@@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error ''', name=c_name(name))
Quoted 13 lines
scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error · line 273
Quoted 9 lines
scripts/qapi-visit.py +3
-6- if not discriminator: - tag = 'kind' - disc_key = "type" - else: - tag = discriminator - disc_key = discriminator + disc_key = variants.tag_member.name + if not variants.tag_name: + # we pointlessly use a different key for simple unionsWe could fix that (as a separate patch); wonder how much C code it would affect. A lot of these things that we can alter in generated code are certainly easier to see now that we have a clean generator :)
Yup, the warts stand out now.
Quoted 30 lines
scripts/qapi-visit.py out: · line 331
Quoted 11 lines
scripts/qapi-visit.py +6
-3+def gen_visit_decl(name, scalar=False): + c_type = c_name(name) + ' *' + if not scalar: + c_type += '*' return mcgen(''' - -void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp); +void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp); ''', - name=c_name(name)) + c_name=c_name(name), c_type=c_type)Nice way to consolidate several near-identical copies.
Quoted 16 lines
scripts/qapi-visit.py +16
-0+ +class QAPISchemaGenVisitVisitor(QAPISchemaVisitor): + def __init__(self): + self.decl = None + self.defn = None + self.btin = None + def visit_begin(self): + self.decl = '' + self.defn = '' + self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL') + def visit_end(self): + # to avoid header dependency hell, we always generate + # declarations for built-in types in our header files and + # simply guard them + self.btin += guardend('QAPI_VISIT_BUILTIN_VISITOR_DECL') + self.decl = self.btin + self.decl
Quoted 8 lines
scripts/qapi-visit.py out: · line 351
Quoted 4 lines
scripts/qapi-visit.py +4
-0+ self.btin = None + # ...this doesn't work for cases where we link in multiple + # objects that have the functions defined, so we use + # do_builtins (option -b) to provide controlAnd once again, as in 26/47, this floats the .h file to have all builtin representations in one chunk (for continuity with pre-patch), but fails to do the same for the .c code...
Yes, because they all need to be under theQAPI_VISIT_BUILTIN_VISITOR_DECL guard.
Quoted 11 lines
scripts/qapi-visit.py out: · line 361
Quoted 10 lines
scripts/qapi-visit.py +10
-0+ def visit_enum_type(self, name, info, values): + self.decl += gen_visit_decl(name, scalar=True) + self.defn += generate_visit_enum(name) + def visit_array_type(self, name, info, element_type): + decl = gen_visit_decl(name) + defn = gen_visit_list(name, element_type) + if isinstance(element_type, QAPISchemaBuiltinType): + self.btin += decl + if do_builtins: + self.defn += defn
Quoted 3 lines
...where the builtins are now interleaved with everything else instead of bunched together, making the generated diff larger and more confusing than necessary.
Yes, because there's no need to separate them out.
Quoted 10 lines
scripts/qapi-visit.py out: · line 369
Quoted 8 lines
scripts/qapi-visit.py +8
-0+ else: + self.decl += decl + self.defn += defn + def visit_object_type(self, name, info, base, members, variants): + if info: + self.decl += gen_visit_decl(name) + if variants: + self.defn += gen_visit_union(name, base, variants)Worth adding 'assert not members'?
Yes, with a TODO, because it's an artificial restriction I'd like to lift some day.
Quoted 7 lines
scripts/qapi-visit.py
Quoted 2 lines
scripts/qapi-visit.py +2
-0+ else: + self.defn += gen_visit_struct(name, base, members)Or maybe we can someday consolidate these two into a single
gen_visit_object, that handles all members and variants in a uniform manner, instead of our current differences. I wonder how much C code would be impacted?
Yes, that's how we'd lift the restriction.
Quoted 20 lines
Quoted 20 lines
scripts/qapi-visit.py +3
-10+ def visit_alternate_type(self, name, info, variants): + self.decl += gen_visit_decl(name) + self.defn += gen_visit_alternate(name, variants) do_builtins = False@@ -442,56 +428,10 @@ fdecl.write(mcgen(''' ''', prefix=prefix)) -exprs = QAPISchema(input_file).get_exprs() - -# to avoid header dependency hell, we always generate declarations -# for built-in types in our header files and simply guard them -fdecl.write(guardstart("QAPI_VISIT_BUILTIN_VISITOR_DECL")) -for typename in builtin_types.keys(): - fdecl.write(generate_declaration(typename, builtin_type=True)) -fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) - -# ...this doesn't work for cases where we link in multiple objects that
Quoted 8 lines
scripts/qapi-visit.py fdecl.write(mcgen(''' · line 459
Quoted 5 lines
scripts/qapi-visit.py +0
-5-# have the functions defined, so we use -b option to provide control -# over these cases -if do_builtins: - for typename in builtin_types.keys(): - fdef.write(generate_visit_list(typename))Again, a well-placed
sorted()over these two loops in a pre-req patch will minimize the churn on the builtins.
Can try and see.
Quoted 13 lines
Quoted 13 lines
scripts/qapi-visit.py +0
-13- -for expr in exprs: - if expr.has_key('struct'): - ret = generate_visit_struct(expr) - ret += generate_visit_list(expr['struct']) - fdef.write(ret) - - ret = generate_declaration(expr['struct']) - fdecl.write(ret) - elif expr.has_key('union'): - ret = generate_visit_union(expr) - ret += generate_visit_list(expr['union']) - fdef.write(ret)
Quoted 8 lines
scripts/qapi-visit.py fdecl.write(mcgen(''' · line 477
Quoted 5 lines
scripts/qapi-visit.py +0
-5- - enum_define = discriminator_find_enum_define(expr) - ret = "" - if not enum_define: - ret = generate_decl_enum('%sKind' % expr['union'])Nice that the new visitor automatically visits any implicit enum, without us having to special case it.
Making implicit stuff explicit in the internal representation kills special cases and de-duplicates code :)
Quoted 14 lines
scripts/qapi-visit.py fdecl.write(mcgen(''' · line 490
Quoted 13 lines
scripts/qapi-visit.py +0
-13- ret += generate_declaration(expr['union']) - fdecl.write(ret) - elif expr.has_key('alternate'): - ret = generate_visit_alternate(expr['alternate'], expr['data']) - ret += generate_visit_list(expr['alternate']) - fdef.write(ret) - - ret = generate_decl_enum('%sKind' % expr['alternate']) - ret += generate_declaration(expr['alternate']) - fdecl.write(ret) - elif expr.has_key('enum'): - ret = generate_visit_list(expr['enum']) - ret += generate_visit_enum(expr['enum'])
Quoted 2 lines
[1] swapping these two lines in a pre-req patch will minimize the churn of this conversion.
Can try and see.
Quoted 14 lines
scripts/qapi-visit.py fdecl.write(mcgen(''' · line 435
Quoted 10 lines
scripts/qapi-visit.py +5
-5- fdef.write(ret) - - ret = generate_decl_enum(expr['enum']) - ret += generate_enum_declaration(expr['enum']) - fdecl.write(ret) +schema = QAPISchema(input_file) +gen = QAPISchemaGenVisitVisitor() +schema.visit(gen) +fdef.write(gen.defn) +fdecl.write(gen.decl)Again, overall impression is that your series is headed in the right direction.
Quoted 2 lines
And nice that the TODOs in the testsuite pointed out what this fixes, for visiting indirect bases.
Thanks!
Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Quoted 17 lines · Markus Armbruster
Quoted 9 lines
Oh, and that means our generator has a collision bug that none of my added tests have exposed yet: you cannot have a base class and simultaneously add a member named 'base':
{ 'struct': 'Base', 'data': { 'i': 'int' } }
{ 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }because the generated C code is trying to use the name 'base' for its own purposes.
*sigh*
Quoted 4 lines
I guess that means more pre-req patches to the series to
expose the bug, and either tighten the parser to reject things for now (easiest) or update the generator to not collide (harder, and fine for a later series).
Quoted 4 lines
Yes.
Life would be easier if the original authors had adopted sane naming conventions from the start.
Like reserving ourselves a namespace based on single _ for internal use.
We practically already have that - all user names either start with a
letter or double underscore, so we could use single (and triple) underscore for internally-generated purposes, freeing up 'base', '*Kind', '*_MAX', and other namespace abuses back to the user. Well, we may also need to reserve mid-name double-underscore (that is, the user can only supply double underscore at the beginning, but not middle, of an identifier). Ah well, food for thought for later patches.
Quoted 7 lines
Quoted 6 lines
Okay, I see where you are using .flat from the initial parse. I still think it is a bit odd that you are defining '.flat' for each 'variant' within 'variants', even though, for a given 'variants', all members will have the same setting of '.flat'. That makes me wonder if '.flat' should belong instead to the top-level 'variants' struct rather than to each 'variant' member.
Quoted 5 lines
Two reasons for putting flat where it is:
* The philosophical one: from the generator's point of view, there's no fundamental reason why all variants need to be flat or none. The generator really doesn't care.
And we may decide to exploit that down the road to allow some sort of qapi syntax for explicitly designating a union branch as flat or boxed, rather than the current approach of the type of union determining the status of all branch members.
Quoted 11 lines
* The pragmatic one (a.k.a. the real one): there are places where I use v.flat, but don't have the variants owning v handy.
Quoted 6 lines
But again I wonder what would happen if you had instead normalized the input of simple unions into always having an implicit struct (with single member 'data'), so that by the time you get here, you only have to deal with a single representation of unions instead of having to still emit different things for flat vs. simple (since on the wire, we already proved simple is shorthand that can be duplicated by a flat union).
Quoted 3 lines
I hope we can get there! But at this stage of the conversion, I want to minimize output change, and .flat makes preserving all its warts much easier.
Agreed. By the end of the series, I was convinced that the use of .flat, at least in this series, makes sense.
Quoted 9 lines
Quoted 7 lines
scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error · line 273
Quoted 3 lines
scripts/qapi-visit.py +3
-0+ disc_key = variants.tag_member.name + if not variants.tag_name: + # we pointlessly use a different key for simple unionsWe could fix that (as a separate patch); wonder how much C code it would affect. A lot of these things that we can alter in generated code are certainly easier to see now that we have a clean generator :)
Yup, the warts stand out now.
And I've already demonstrated what sort of cleanups can be done to attack some of the warts:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05266.html
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Quoted 24 lines · Eric Blake
Quoted 17 lines · Markus Armbruster
Quoted 9 lines
Oh, and that means our generator has a collision bug that none of my added tests have exposed yet: you cannot have a base class and simultaneously add a member named 'base':
{ 'struct': 'Base', 'data': { 'i': 'int' } }
{ 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }because the generated C code is trying to use the name 'base' for its own purposes.
*sigh*
Quoted 4 lines
I guess that means more pre-req patches to the series to
expose the bug, and either tighten the parser to reject things for now (easiest) or update the generator to not collide (harder, and fine for a later series).Quoted 4 lines
Yes.
Life would be easier if the original authors had adopted sane naming conventions from the start.
Quoted 8 lines
Like reserving ourselves a namespace based on single _ for internal use.
We practically already have that - all user names either start with a
letter or double underscore, so we could use single (and triple) underscore for internally-generated purposes, freeing up 'base', '*Kind', '*_MAX', and other namespace abuses back to the user. Well, we may also need to reserve mid-name double-underscore (that is, the user can only supply double underscore at the beginning, but not middle, of an identifier). Ah well, food for thought for later patches.
Another concern: we should take care not to generate reserved identifiers.
* Potential issue with your proposal: identifiers that begin with an underscore and either an uppercase letter or another underscore are always reserved for any use.
* Existing issue: downstream extensions carry a __RFQDN_ prefix in the schema, which map to reserved C identifiers.
Example: qapi-schema-test.json type '__org.qemu_x-Enum' generates
typedef enum __org_qemu_x_Enum {
ORG_QEMU_X_ENUM___ORG_QEMU_X_VALUE = 0,
ORG_QEMU_X_ENUM_MAX = 1,
} __org_qemu_x_Enum;
extern const char *const __org_qemu_x_Enum_lookup[];Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Okay, I see where you are using .flat from the initial parse. I still think it is a bit odd that you are defining '.flat' for each 'variant' within 'variants', even though, for a given 'variants', all members will have the same setting of '.flat'. That makes me wonder if '.flat' should belong instead to the top-level 'variants' struct rather than to each 'variant' member.
Two reasons for putting flat where it is:
Quoted 8 lines
Quoted 3 lines
* The philosophical one: from the generator's point of view, there's no fundamental reason why all variants need to be flat or none. The generator really doesn't care.
And we may decide to exploit that down the road to allow some sort of qapi syntax for explicitly designating a union branch as flat or boxed, rather than the current approach of the type of union determining the status of all branch members.
I can't see a need now, but if one arises, we could do it.
Quoted 10 lines
Quoted 10 lines
* The pragmatic one (a.k.a. the real one): there are places where I use v.flat, but don't have the variants owning v handy.
Quoted 6 lines
But again I wonder what would happen if you had instead normalized the input of simple unions into always having an implicit struct (with single member 'data'), so that by the time you get here, you only have to deal with a single representation of unions instead of having to still emit different things for flat vs. simple (since on the wire, we already proved simple is shorthand that can be duplicated by a flat union).
Quoted 6 lines
Quoted 3 lines
I hope we can get there! But at this stage of the conversion, I want to minimize output change, and .flat makes preserving all its warts much easier.
Agreed. By the end of the series, I was convinced that the use of .flat, at least in this series, makes sense.
Good :)
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error · line 273
Quoted 3 lines
scripts/qapi-visit.py +3
-0+ disc_key = variants.tag_member.name + if not variants.tag_name: + # we pointlessly use a different key for simple unionsWe could fix that (as a separate patch); wonder how much C code it would affect. A lot of these things that we can alter in generated code are certainly easier to see now that we have a clean generator :)
Quoted 5 lines
Quoted 1 line
Yup, the warts stand out now.
And I've already demonstrated what sort of cleanups can be done to attack some of the warts:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05266.html
I only have time for a quick glance now. It looks lovely!
Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Quoted 11 lines · Markus Armbruster
Quoted 8 lines
Like reserving ourselves a namespace based on single _ for internal use.
We practically already have that - all user names either start with a
letter or double underscore, so we could use single (and triple) underscore for internally-generated purposes, freeing up 'base', '*Kind', '*_MAX', and other namespace abuses back to the user. Well, we may also need to reserve mid-name double-underscore (that is, the user can only supply double underscore at the beginning, but not middle, of an identifier). Ah well, food for thought for later patches.Another concern: we should take care not to generate reserved identifiers.
And we do that for a number of identifiers already, by renaming 'default' in qapi to 'q_default' in C.
Quoted 4 lines
* Potential issue with your proposal: identifiers that begin with an underscore and either an uppercase letter or another underscore are always reserved for any use.
True, so only underscore and lower is guaranteed safe. But in practice, we can often get away with more...
Quoted 3 lines
* Existing issue: downstream extensions carry a
__RFQDN_prefix in the schema, which map to reserved C identifiers.
...the whole point of the __RFQDN_ prefix was that it is very unlikely to conflict with any vendor extensions (unless you happen to be the same vendor that writes the compiler and als use __RFQDN_ as part of your compiler implementation). Yes, it's risky, but a risk that hasn't hurt any downstream clients yet :) It's one of those things where if someone reports an actual problem in their environment, we'll fix it; but until then, it's not worth the headache of strict compliance to "fix" something that happens to work in spite of being undefined behavior.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Quoted 17 lines · Eric Blake
Quoted 11 lines · Markus Armbruster
Quoted 8 lines
Like reserving ourselves a namespace based on single _ for internal use.
We practically already have that - all user names either start with a
letter or double underscore, so we could use single (and triple) underscore for internally-generated purposes, freeing up 'base', '*Kind', '*_MAX', and other namespace abuses back to the user. Well, we may also need to reserve mid-name double-underscore (that is, the user can only supply double underscore at the beginning, but not middle, of an identifier). Ah well, food for thought for later patches.Another concern: we should take care not to generate reserved identifiers.
And we do that for a number of identifiers already, by renaming 'default' in qapi to '
q_default' in C.Quoted 1 line
Quoted 5 lines
Quoted 3 lines
* Potential issue with your proposal: identifiers that begin with an underscore and either an uppercase letter or another underscore are always reserved for any use.
True, so only underscore and lower is guaranteed safe.
And we'd get that for most names. Only names we want to shout (macros, enumeration constants) could result in a problematic underscore and uppercase letter.
Note that we *strip* leading underscores from enumeration constants. Example (from qapi-schema-test.json):
{ 'enum': '__org.qemu_x-Enum', 'data': [ '__org.qemu_x-value' ] }generates
typedef enum __org_qemu_x_Enum
{
ORG_QEMU_X_ENUM___ORG_QEMU_X_VALUE = 0,
ORG_QEMU_X_ENUM_MAX = 1,
} __org_qemu_x_Enum;Now add this one:
{ 'enum': 'org_qemu_x-Enum', 'data': [ ] }Just fine according to our documentation. But of course it clashes:
typedef enum org_qemu_x_Enum
{
ORG_QEMU_X_ENUM_MAX = 0,
} org_qemu_x_Enum;Short term, we should note in qapi-code-gen.txt: different names in the schema can nevertheless clash in generated C, and when that happens, you get to pick non-clashing names.
Longer term, we may want to rethink how we map names to C.
Quoted 6 lines
we can often get away with more...
Quoted 3 lines
* Existing issue: downstream extensions carry a
__RFQDN_prefix in the schema, which map to reserved C identifiers.
Quoted 8 lines
...the whole point of the
__RFQDN_prefix was that it is very unlikely to conflict with any vendor extensions (unless you happen to be the same vendor that writes the compiler and als use__RFQDN_as part of your compiler implementation). Yes, it's risky, but a risk that hasn't hurt any downstream clients yet :) It's one of those things where if someone reports an actual problem in their environment, we'll fix it; but until then, it's not worth the headache of strict compliance to "fix" something that happens to work in spite of being undefined behavior.
I'm not religious about reserved names, but this intrusion into the reserved name space was entirely avoidable: we could've just as well picked a non-reserved prefix.
Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
In reply to Markus ArmbrusterQuoted 18 lines · Markus Armbruster
Fixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.
Patch's effect on
visit_type_UserDefFlatUnion():static void
visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp){
Error *err = NULL;+
visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err); if (err) {
goto out;
scripts/qapi-visit.py out: · line 242
Quoted 10 lines
scripts/qapi-visit.py +4
-4+def gen_visit_union(name, base, variants): + ret = '' if base: - assert discriminator - base_fields = find_struct(base)['data'].copy() - del base_fields[discriminator] - ret += generate_visit_struct_fields(name, base_fields) + members = [m for m in base.members if m != variants.tag_member] + ret += generate_visit_struct_fields(name, members)
Ouch. This hurts. If the same class is used as both the base class of a flat union, and the base class of an ordinary struct, then the struct tries to visit the base class, but no longer parses the field that the union was using as its discriminator.
We don't have any code that demonstrates this, but probably should. I ran into it while working up my POC of what it would take to unbox inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)
We want visit_FOO_fields to visit _all_ fields of the struct, no matter who called generate_visit_struct_fields(). So what you must instead do here is use the fact that we've already visited the discriminator...
Quoted 17 lines
scripts/qapi-visit.py +3
-6- if discriminator: - for key in members: - ret += generate_visit_implicit_struct(members[key]) + for var in variants.variants: + if var.flat: + ret += generate_visit_implicit_struct(var.type) ret += mcgen('''@@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error ''', name=c_name(name)) - if not discriminator: - tag = 'kind' - disc_key = "type"
scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error · line 276
Quoted 10 lines
scripts/qapi-visit.py +5
-4- else: - tag = discriminator - disc_key = discriminator + disc_key = variants.tag_member.name + if not variants.tag_name: + # we pointlessly use a different key for simple unions + disc_key = 'type' ret += mcgen(''' - visit_type_%(disc_type)s(m, &(*obj)->%(c_tag)s, "%(disc_key)s", &err); + visit_type_%(disc_type)s(m, &(*obj)->%(c_name)s, "%(disc_key)s", &err);
...and omit this call if the flat union's base class already took care of it.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Quoted 30 lines · Eric Blake
Quoted 18 lines · Markus Armbruster
Fixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.
Patch's effect on
visit_type_UserDefFlatUnion():static void
visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp){
Error *err = NULL;+
visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err); if (err) {
goto out;scripts/qapi-visit.py out: · line 242
Quoted 10 lines
scripts/qapi-visit.py +4
-4+def gen_visit_union(name, base, variants): + ret = '' if base: - assert discriminator - base_fields = find_struct(base)['data'].copy() - del base_fields[discriminator] - ret += generate_visit_struct_fields(name, base_fields) + members = [m for m in base.members if m != variants.tag_member] + ret += generate_visit_struct_fields(name, members)
Quoted 8 lines
Ouch. This hurts. If the same class is used as both the base class of a flat union, and the base class of an ordinary struct, then the struct tries to visit the base class, but no longer parses the field that the union was using as its discriminator.
We don't have any code that demonstrates this, but probably should. I ran into it while working up my POC of what it would take to unbox inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)
Is this broken in master, or do my patches break it?
Got a reproducer?
[...]
Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Quoted 17 lines · Markus Armbruster
Quoted 17 lines
scripts/qapi-visit.py out: · line 242
Quoted 11 lines
scripts/qapi-visit.py +4
-4+def gen_visit_union(name, base, variants): + ret = '' if base: - assert discriminator - base_fields = find_struct(base)['data'].copy() - del base_fields[discriminator] - ret += generate_visit_struct_fields(name, base_fields) + members = [m for m in base.members if m != variants.tag_member] + ret += generate_visit_struct_fields(name, members)Ouch. This hurts. If the same class is used as both the base class of a flat union, and the base class of an ordinary struct, then the struct tries to visit the base class, but no longer parses the field that the union was using as its discriminator.
Quoted 7 lines
Quoted 3 lines
We don't have any code that demonstrates this, but probably should. I ran into it while working up my POC of what it would take to unbox inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)
Is this broken in master, or do my patches break it?
Got a reproducer?
Turns out I'm mistaken; we got lucky. The call to generate_visit_struct_fields() creates a function for 'name' (aka the union name), and not for 'base' (aka the class name that owns the fields). So even if we have Base as a common struct between Child and Union, the code emitted for Child generates visit_Base_fields(), while the code emitted for Union generates visit_Union_fields().
So there is no reproducer, but _only_ as long as we reject unions as a base class for any other object. And there is redundancy: we could reuse visit_Base_fields() for the sake of the union, then avoid (re-)visiting the discriminator, and we would no longer need to emit visit_Union_fields(). But I can do that as part of the followup cleanups; since I don't see anything broken with your patch, we don't have to worry about it during this series.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Quoted 34 lines · Eric Blake
Quoted 17 lines · Markus Armbruster
Quoted 17 lines
scripts/qapi-visit.py out: · line 242
Quoted 11 lines
scripts/qapi-visit.py +4
-4+def gen_visit_union(name, base, variants): + ret = '' if base: - assert discriminator - base_fields = find_struct(base)['data'].copy() - del base_fields[discriminator] - ret += generate_visit_struct_fields(name, base_fields) + members = [m for m in base.members if m != variants.tag_member] + ret += generate_visit_struct_fields(name, members)Ouch. This hurts. If the same class is used as both the base class of a flat union, and the base class of an ordinary struct, then the struct tries to visit the base class, but no longer parses the field that the union was using as its discriminator.
Quoted 7 lines
Quoted 3 lines
We don't have any code that demonstrates this, but probably should. I ran into it while working up my POC of what it would take to unbox inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)
Is this broken in master, or do my patches break it?
Got a reproducer?
Turns out I'm mistaken; we got lucky. The call to
generate_visit_struct_fields()creates a function for 'name' (aka the union name), and not for 'base' (aka the class name that owns the fields). So even if we have Base as a common struct between Child and Union, the code emitted for Child generatesvisit_Base_fields(), while the code emitted for Union generatesvisit_Union_fields().
Quoted 7 lines
So there is no reproducer, but _only_ as long as we reject unions as a base class for any other object. And there is redundancy: we could reuse
visit_Base_fields()for the sake of the union, then avoid (re-)visiting the discriminator, and we would no longer need to emitvisit_Union_fields(). But I can do that as part of the followup cleanups; since I don't see anything broken with your patch, we don't have to worry about it during this series.
Good. Thank you!
Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
Quoted 23 lines · Markus Armbruster
Quoted 15 lines
Quoted 7 lines
Quoted 3 lines
We don't have any code that demonstrates this, but probably should. I ran into it while working up my POC of what it would take to unbox inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)
Is this broken in master, or do my patches break it?
Got a reproducer?
Turns out I'm mistaken; we got lucky. The call to
generate_visit_struct_fields()creates a function for 'name' (aka the union name), and not for 'base' (aka the class name that owns the fields). So even if we have Base as a common struct between Child and Union, the code emitted for Child generatesvisit_Base_fields(), while the code emitted for Union generatesvisit_Union_fields().Quoted 7 lines
So there is no reproducer, but _only_ as long as we reject unions as a base class for any other object. And there is redundancy: we could reuse
visit_Base_fields()for the sake of the union, then avoid (re-)visiting the discriminator, and we would no longer need to emitvisit_Union_fields(). But I can do that as part of the followup cleanups; since I don't see anything broken with your patch, we don't have to worry about it during this series.
Quoted 1 line
Good. Thank you!
And here's a followup patch that cleans it up:
http://thread.gmane.org/gmane.comp.emulators.qemu/353204/focus=353728
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 28/47 qapi-commands: Convert to QAPISchemaVisitor
In reply to Markus ArmbrusterOutput unchanged except for white-space.
--- scripts/qapi-commands.py | 157 ++++++++++++++++++++++++++--------------------- scripts/qapi.py | 2 +- 2 files changed, 87 insertions(+), 72 deletions(-)
scripts/qapi-commands.py +86 -71
@@ -12,21 +12,22 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -from ordereddict import OrderedDict from qapi import * import re def generate_command_decl(name, args, ret_type): arglist="" - for argname, argtype, optional in parse_args(args): - argtype = c_type(argtype, is_param=True) - if optional: - arglist += "bool has_%s, " % c_name(argname) - arglist += "%s %s, " % (argtype, c_name(argname)) + if args: + for memb in args.members: + argtype = memb.type.c_type(is_param=True) + if memb.optional: + arglist += "bool has_%s, " % c_name(memb.name) + arglist += "%s %s, " % (argtype, c_name(memb.name)) return mcgen(''' %(ret_type)s qmp_%(name)s(%(args)sError **errp); ''', - ret_type=c_type(ret_type), name=c_name(name), + ret_type=(ret_type and ret_type.c_type()) or 'void', + name=c_name(name), args=arglist) def gen_err_check(err):
@@ -45,10 +46,11 @@ def gen_sync_call(name, args, ret_type): retval="" if ret_type: retval = "retval = " - for argname, argtype, optional in parse_args(args): - if optional: - arglist += "has_%s, " % c_name(argname) - arglist += "%s, " % (c_name(argname)) + if args: + for memb in args.members: + if memb.optional: + arglist += "has_%s, " % c_name(memb.name) + arglist += "%s, " % c_name(memb.name) push_indent() ret = mcgen(''' %(retval)sqmp_%(name)s(%(args)s&local_err);
@@ -68,7 +70,7 @@ def gen_visitor_input_containers_decl(args): ret = "" push_indent() - if len(args) > 0: + if args: ret += mcgen(''' QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args)); QapiDeallocVisitor *md;
@@ -81,22 +83,26 @@ Visitor *v; def gen_visitor_input_vars_decl(args): ret = "" push_indent() - for argname, argtype, optional in parse_args(args): - if optional: - ret += mcgen(''' + + if args: + for memb in args.members: + if memb.optional: + ret += mcgen(''' bool has_%(argname)s = false; ''', - argname=c_name(argname)) - if is_c_ptr(argtype): - ret += mcgen(''' + argname=c_name(memb.name)) + if is_c_ptr(memb.type.c_type()): + ret += mcgen(''' %(argtype)s %(argname)s = NULL; ''', - argname=c_name(argname), argtype=c_type(argtype)) - else: - ret += mcgen(''' + argname=c_name(memb.name), + argtype=memb.type.c_type()) + else: + ret += mcgen(''' %(argtype)s %(argname)s = {0}; ''', - argname=c_name(argname), argtype=c_type(argtype)) + argname=c_name(memb.name), + argtype=memb.type.c_type()) pop_indent() return ret
@@ -106,7 +112,7 @@ def gen_visitor_input_block(args, dealloc=False): errparg = '&local_err' errarg = 'local_err' - if len(args) == 0: + if not args: return ret push_indent()
@@ -124,25 +130,26 @@ v = qapi_dealloc_get_visitor(md); v = qmp_input_get_visitor(mi); ''') - for argname, argtype, optional in parse_args(args): - if optional: + for memb in args.members: + if memb.optional: ret += mcgen(''' visit_optional(v, &has_%(c_name)s, "%(name)s", %(errp)s); ''', - c_name=c_name(argname), name=argname, errp=errparg) + c_name=c_name(memb.name), name=memb.name, + errp=errparg) ret += gen_err_check(errarg) ret += mcgen(''' if (has_%(c_name)s) { ''', - c_name=c_name(argname)) + c_name=c_name(memb.name)) push_indent() ret += mcgen(''' visit_type_%(visitor)s(v, &%(c_name)s, "%(name)s", %(errp)s); ''', - c_name=c_name(argname), name=argname, argtype=argtype, - visitor=type_name(argtype), errp=errparg) + c_name=c_name(memb.name), name=memb.name, + visitor=memb.type.c_name(), errp=errparg) ret += gen_err_check(errarg) - if optional: + if memb.optional: pop_indent() ret += mcgen(''' }
@@ -160,6 +167,7 @@ def gen_marshal_output(name, ret_type): return "" ret = mcgen(''' + static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_out, Error **errp) { Error *local_err = NULL;
@@ -183,8 +191,8 @@ out: qapi_dealloc_visitor_cleanup(md); } ''', - c_ret_type=c_type(ret_type), c_name=c_name(name), - visitor=type_name(ret_type)) + c_ret_type=ret_type.c_type(), c_name=c_name(name), + visitor=ret_type.c_name()) return ret
@@ -198,6 +206,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode): hdr = gen_marshal_input_decl(name, middle_mode) ret = mcgen(''' + %(header)s { Error *local_err = NULL;
@@ -205,16 +214,16 @@ def gen_marshal_input(name, args, ret_type, middle_mode): header=hdr) if ret_type: - if is_c_ptr(ret_type): - retval = " %s retval = NULL;" % c_type(ret_type) + if is_c_ptr(ret_type.c_type()): + retval = " %s retval = NULL;" % ret_type.c_type() else: - retval = " %s retval;" % c_type(ret_type) + retval = " %s retval;" % ret_type.c_type() ret += mcgen(''' %(retval)s ''', retval=retval) - if len(args) > 0: + if args: ret += gen_visitor_input_containers_decl(args) ret += gen_visitor_input_vars_decl(args) + '\n' ret += gen_visitor_input_block(args) + '\n'
@@ -241,21 +250,23 @@ out: ''') return ret -def gen_registry(commands): - registry="" +def gen_register_command(name, success_response): push_indent() - for cmd in commands: - options = 'QCO_NO_OPTIONS' - if not cmd.get('success-response', True): - options = 'QCO_NO_SUCCESS_RESP' + options = 'QCO_NO_OPTIONS' + if not success_response: + options = 'QCO_NO_SUCCESS_RESP' - registry += mcgen(''' + ret = mcgen(''' qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s); ''', - name=cmd['command'], c_name=c_name(cmd['command']), + name=name, c_name=c_name(name), opts=options) pop_indent() + return ret + +def gen_registry(registry): ret = mcgen(''' + static void qmp_init_marshal(void) { ''')
@@ -267,6 +278,31 @@ qapi_init(qmp_init_marshal); ''') return ret +class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): + def __init__(self): + self.decl = None + self.defn = None + self.regy = None + def visit_begin(self): + self.decl = '' + self.defn = '' + self.regy = '' + def visit_end(self): + if not middle_mode: + self.defn += gen_registry(self.regy) + self.regy = None + def visit_command(self, name, info, args, rets, gen, success_response): + if not gen: + return + self.decl += generate_command_decl(name, args, rets) + if rets: + self.defn += gen_marshal_output(name, rets) + if middle_mode: + self.decl += gen_marshal_input_decl(name, middle_mode) + ';\n' + self.defn += gen_marshal_input(name, args, rets, middle_mode) + if not middle_mode: + self.regy += gen_register_command(name, success_response) + middle_mode = False (input_file, output_dir, do_c, do_h, prefix, opts) = \
@@ -276,10 +312,6 @@ for o, a in opts: if o in ("-m", "--middle"): middle_mode = True -exprs = QAPISchema(input_file).get_exprs() -commands = filter(lambda expr: expr.has_key('command'), exprs) -commands = filter(lambda expr: not expr.has_key('gen'), commands) - c_comment = ''' /* * schema-defined QMP->QAPI command dispatch
@@ -335,29 +367,12 @@ fdecl.write(mcgen(''' #include "qapi/error.h" ''', - prefix=prefix)) + prefix=prefix)) -for cmd in commands: - arglist = [] - ret_type = None - if cmd.has_key('data'): - arglist = cmd['data'] - if cmd.has_key('returns'): - ret_type = cmd['returns'] - ret = generate_command_decl(cmd['command'], arglist, ret_type) - fdecl.write(ret) - if ret_type: - ret = gen_marshal_output(cmd['command'], ret_type) + "\n" - fdef.write(ret) - - if middle_mode: - fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], middle_mode)) - - ret = gen_marshal_input(cmd['command'], arglist, ret_type, middle_mode) + "\n" - fdef.write(ret) - -if not middle_mode: - ret = gen_registry(commands) - fdef.write(ret) +schema = QAPISchema(input_file) +gen = QAPISchemaGenCommandVisitor() +schema.visit(gen) +fdef.write(gen.defn) +fdecl.write(gen.decl) close_output(fdef, fdecl)
scripts/qapi.py +1 -1
@@ -1366,7 +1366,7 @@ def c_type(value, is_param=False): return c_name(value) + pointer_suffix def is_c_ptr(value): - return c_type(value).endswith(pointer_suffix) + return value.endswith(pointer_suffix) def genindent(count): ret = ""
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 28/47] qapi-commands: Convert to QAPISchemaVisitor
Quoted 1 line · Markus Armbruster
Output unchanged except for white-space.
Indeed, and the diffstat shows it was only one blank line:
qga-qmp-marshal.c | 1 + 1 file changed, 1 insertion(+)
MUCH friendlier to review :)
Quoted 7 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 157 ++++++++++++++++++++++++++--------------------- scripts/qapi.py | 2 +- 2 files changed, 87 insertions(+), 72 deletions(-)
Here, I can confidently say:
scripts/qapi.py def c_type(value, is_param=False): · line 1369
Quoted 7 lines
+++
b/scripts/qapi.pyscripts/qapi.py +1
-1@@ -1366,7 +1366,7 @@ def c_type(value, is_param=False): return c_name(value) + pointer_suffix def is_c_ptr(value): - return c_type(value).endswith(pointer_suffix) + return value.endswith(pointer_suffix)
Perhaps this cleanup could be floated earlier in the series?
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 28/47] qapi-commands: Convert to QAPISchemaVisitor
Quoted 19 lines · Eric Blake
Quoted 1 line · Markus Armbruster
Output unchanged except for white-space.
Indeed, and the diffstat shows it was only one blank line:
qga-qmp-marshal.c | 1 + 1 file changed, 1 insertion(+)MUCH friendlier to review :)
Quoted 8 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 157 ++++++++++++++++++++++++++--------------------- scripts/qapi.py | 2 +- 2 files changed, 87 insertions(+), 72 deletions(-)
Quoted 3 lines
Here, I can confidently say:
Reviewed-by Eric Blake <[email protected]>
Thanks!
Quoted 9 lines
scripts/qapi.py def c_type(value, is_param=False): · line 1369
Quoted 7 lines
+++
b/scripts/qapi.pyscripts/qapi.py +1
-1@@ -1366,7 +1366,7 @@ def c_type(value, is_param=False): return c_name(value) + pointer_suffix def is_c_ptr(value): - return c_type(value).endswith(pointer_suffix) + return value.endswith(pointer_suffix)Perhaps this cleanup could be floated earlier in the series?
Before this patch, is_c_ptr() is called with an argument suitable for c_type(). It returns true when c_type() returns a string ending with pointer_suffix.
This patch replaces uses of function c_type() by QAPISchemaType method c_type().
Two ways to do that for the is_c_ptr() calls:
1. Change the argument to a QAPISchemaType object, and make is_c_ptr() call its .c_type().
2. Change the argument to a string, and make the caller compute it by calling the appropriate object's .c_type().
I picked 2. It hardly matters, because is_c_ptr() will go away in the next patch.
I could do a part of 2. in an earlier patch, namely lifting the c_type() call into the callers. But I can't do the conversion from function to method any earlier, because the objects become available only in this patch.
RFC PATCH v2 29/47 qapi: Replace dirty is_c_ptr() by method c_null()
In reply to Markus Armbrusteris_c_ptr() looks whether the end of the C text for the type looks like a pointer. Works, but is fragile.
We now have a better tool: use QAPISchemaType method c_null(). The initializers for non-pointers become prettier: 0, false or the enumeration constant with the value 0 instead of {0}.
One place looks suspicious: we initialize pointers, but not non-pointers. Either the initialization is superfluous and should be deleted, or the non-pointers need it as well, or something subtle is going on and needs a comment. Since I lack the time to figure it out now, mark it FIXME.
--- scripts/qapi-commands.py | 19 +++++++------------ scripts/qapi.py | 3 --- 2 files changed, 7 insertions(+), 15 deletions(-)
scripts/qapi-commands.py +7 -12
@@ -91,18 +91,12 @@ def gen_visitor_input_vars_decl(args): bool has_%(argname)s = false; ''', argname=c_name(memb.name)) - if is_c_ptr(memb.type.c_type()): - ret += mcgen(''' -%(argtype)s %(argname)s = NULL; + ret += mcgen(''' +%(c_type)s %(c_name)s = %(c_null)s; ''', - argname=c_name(memb.name), - argtype=memb.type.c_type()) - else: - ret += mcgen(''' -%(argtype)s %(argname)s = {0}; -''', - argname=c_name(memb.name), - argtype=memb.type.c_type()) + c_name=c_name(memb.name), + c_type=memb.type.c_type(), + c_null=memb.type.c_null()) pop_indent() return ret
@@ -214,7 +208,8 @@ def gen_marshal_input(name, args, ret_type, middle_mode): header=hdr) if ret_type: - if is_c_ptr(ret_type.c_type()): + # FIXME fishy: only pointers are initialized + if ret_type.c_null() == 'NULL': retval = " %s retval = NULL;" % ret_type.c_type() else: retval = " %s retval;" % ret_type.c_type()
scripts/qapi.py +0 -3
@@ -1365,9 +1365,6 @@ def c_type(value, is_param=False): assert isinstance(value, str) and value != "" return c_name(value) + pointer_suffix -def is_c_ptr(value): - return value.endswith(pointer_suffix) - def genindent(count): ret = "" for i in range(count):
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
Quoted 7 lines · Markus Armbruster
is_c_ptr()looks whether the end of the C text for the type looks like a pointer. Works, but is fragile.We now have a better tool: use QAPISchemaType method
c_null(). The initializers for non-pointers become prettier: 0, false or the enumeration constant with the value 0 instead of {0}.
Quoted 5 lines
One place looks suspicious: we initialize pointers, but not non-pointers. Either the initialization is superfluous and should be deleted, or the non-pointers need it as well, or something subtle is going on and needs a comment. Since I lack the time to figure it out now, mark it FIXME.
Commenting on just this part for now (out of time to review this patch proper for today):
scripts/qapi-commands.py def gen_marshal_input(name, args, ret_type, middle_mode): · line 215
Quoted 10 lines
scripts/qapi-commands.py +2
-1@@ -214,7 +208,8 @@ def gen_marshal_input(name, args, ret_type, middle_mode): header=hdr) if ret_type: - if is_c_ptr(ret_type.c_type()): + # FIXME fishy: only pointers are initialized + if ret_type.c_null() == 'NULL': retval = " %s retval = NULL;" % ret_type.c_type() else: retval = " %s retval;" % ret_type.c_type()
Here's an example function impacted by this:
static void qmp_marshal_input_guest_file_open(QDict *args, QObject **ret, Error
**errp)
{
Error *local_err = NULL;
int64_t retval;
QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
QapiDeallocVisitor *md;
Visitor *v;
char *path = NULL;
bool has_mode = false;
char *mode = NULL;
...
retval = qmp_guest_file_open(path, has_mode, mode, &local_err);
if (local_err) {
goto out;
}
qmp_marshal_output_guest_file_open(retval, ret, &local_err);But compare that to any other function that returns a pointer, and you'll see the same pattern (the only use of retval is in the final call to qmp_marshal_output..., right after assigning retval); that is, initializing to NULL is dead code, and you could get away with just always declaring it instead of worrying about c_null() in this code.
Or maybe we have a leak - if the 'if (local_err)' can ever trigger even when a function returned non-NULL, then our out: label is missing a free(retval), but only when retval is a pointer. Or maybe we change the code to assert that retval is NULL if local_err is set after calling the user's function, to prove we don't have a leak.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
Quoted 50 lines · Eric Blake
Quoted 7 lines · Markus Armbruster
is_c_ptr()looks whether the end of the C text for the type looks like a pointer. Works, but is fragile.We now have a better tool: use QAPISchemaType method
c_null(). The initializers for non-pointers become prettier: 0, false or the enumeration constant with the value 0 instead of {0}.Quoted 5 lines
One place looks suspicious: we initialize pointers, but not non-pointers. Either the initialization is superfluous and should be deleted, or the non-pointers need it as well, or something subtle is going on and needs a comment. Since I lack the time to figure it out now, mark it FIXME.
Commenting on just this part for now (out of time to review this patch proper for today):
scripts/qapi-commands.py def gen_marshal_input(name, args, ret_type, middle_mode): · line 215
Quoted 10 lines
scripts/qapi-commands.py +2
-1@@ -214,7 +208,8 @@ def gen_marshal_input(name, args, ret_type, middle_mode): header=hdr) if ret_type: - if is_c_ptr(ret_type.c_type()): + # FIXME fishy: only pointers are initialized + if ret_type.c_null() == 'NULL': retval = " %s retval = NULL;" % ret_type.c_type() else: retval = " %s retval;" % ret_type.c_type()Here's an example function impacted by this:
static void qmp_marshal_input_guest_file_open(QDict *args, QObject **ret, Error
**errp){ Error *local_err = NULL; int64_t retval; QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args)); QapiDeallocVisitor *md; Visitor *v; char *path = NULL; bool has_mode = false; char *mode = NULL; ... retval = qmp_guest_file_open(path, has_mode, mode, &local_err); if (local_err) { goto out; } qmp_marshal_output_guest_file_open(retval, ret, &local_err);
Quoted 3 lines
But compare that to any other function that returns a pointer, and you'll see the same pattern (the only use of retval is in the final call to
qmp_marshal_output..., right after assigning retval); that is,
Correct. Inspection of qapi-commands.py shows retval is only ever read in the call of qmp_marshal_output_FOO().
Quoted 8 lines
initializing to NULL is dead code, and you could get away with just always declaring it instead of worrying about
c_null()in this code.Or maybe we have a leak - if the 'if (
local_err)' can ever trigger even when a function returned non-NULL, then our out: label is missing a free(retval), but only when retval is a pointer. Or maybe we change the code to assert that retval is NULL iflocal_erris set after calling the user's function, to prove we don't have a leak.
Let me rephrase to make sure I understand.
Ignore the (not rets) case, because retval doesn't exist then.
qmp_marshal_output_FOO() visits retval twice. First, with a QMP output visitor to do the actual marshalling. Second, with a QAPI dealloc visitor to destroy it.
If we execute the assignment to retval, we must go on to call qmp_marshal_output_FOO(), or else we have a leak.
If we can reach qmp_marshal_output_FOO() without executing the assignment, we must initialize retval. If we can't, any initialization is unused.
gen_call() generates code of the form
retval = qmp_FOO(... args ..., &local_err);
if (local_err) {
goto out;
}
qmp_marshal_output_FOO(retval, ret, &local_err);Its caller then generates
out:
error_propagate(errp, local_err);and so forth.
Observe:
1. The assignment dominates the only use. Therefore, the initialization is unused. Let's drop it in a separate cleanup patch.
2. We can leak retval only when qmp_FOO() returns non-null and local_err is non-null. This must not happen, because:
a. local_err must be null before the call, and
b. the call must not return non-null when it sets local_err.
We could right after out: assert(!local_err || !retval). Not sure it's worthwhile.
TL;DR: I concur with your analysis.
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
Quoted 7 lines · Markus Armbruster
Let me rephrase to make sure I understand.
Ignore the (not rets) case, because retval doesn't exist then.
qmp_marshal_output_FOO()visits retval twice. First, with a QMP output visitor to do the actual marshalling. Second, with a QAPI dealloc visitor to destroy it.
And the use of the dealloc visitor is buried inside the qmp_marshal_output_FOO() call.
Quoted 23 lines
If we execute the assignment to retval, we must go on to call
qmp_marshal_output_FOO(), or else we have a leak.If we can reach
qmp_marshal_output_FOO()without executing the assignment, we must initialize retval. If we can't, any initialization is unused.
gen_call()generates code of the formretval = qmp_FOO(... args ..., &local_err); if (local_err) { goto out; } qmp_marshal_output_FOO(retval, ret, &local_err);Observe:
1. The assignment dominates the only use. Therefore, the initialization is unused. Let's drop it in a separate cleanup patch.
Quoted 6 lines
2. We can leak retval only when
qmp_FOO()returns non-null andlocal_erris non-null. This must not happen, because:a.
local_errmust be null before the call, andb. the call must not return non-null when it sets
local_err.
We don't state that contract anywhere, but I doubt any of the qmp_FOO() functions violate it, so it is worth making it part of the contract.
Quoted 3 lines
We could right after out:
assert(!local_err || !retval). Not sure it's worthwhile.
I think it IS worthwhile, because it would catch buggy callers. Not sure if after out: is the right place (then you'd need an initializer to cover any other code that jumps to out), but this would do the same
retval = qmp_FOO(...);
if (local_err) {
assert(!retval);
goto out;
}qmp_marshal_output_FOO(retval, ...);
Quoted 2 lines
TL;DR: I concur with your analysis.
Is it worth dropping the dead initializer and adding the assert in the same pre-req cleanup patch? Do you want me to submit it since I did the analysis?
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
Quoted 38 lines · Eric Blake
Quoted 7 lines · Markus Armbruster
Let me rephrase to make sure I understand.
Ignore the (not rets) case, because retval doesn't exist then.
qmp_marshal_output_FOO()visits retval twice. First, with a QMP output visitor to do the actual marshalling. Second, with a QAPI dealloc visitor to destroy it.And the use of the dealloc visitor is buried inside the
qmp_marshal_output_FOO()call.Quoted 26 lines
If we execute the assignment to retval, we must go on to call
qmp_marshal_output_FOO(), or else we have a leak.If we can reach
qmp_marshal_output_FOO()without executing the assignment, we must initialize retval. If we can't, any initialization is unused.
gen_call()generates code of the formretval = qmp_FOO(... args ..., &local_err); if (local_err) { goto out; } qmp_marshal_output_FOO(retval, ret, &local_err);Observe:
1. The assignment dominates the only use. Therefore, the initialization is unused. Let's drop it in a separate cleanup patch.
2. We can leak retval only when
qmp_FOO()returns non-null andlocal_erris non-null. This must not happen, because:
Quoted 6 lines
Quoted 3 lines
a.
local_errmust be null before the call, andb. the call must not return non-null when it sets
local_err.We don't state that contract anywhere, but I doubt any of the
qmp_FOO()functions violate it, so it is worth making it part of the contract.
It's a general Error API rule: set an error exactly on failure. It applies to any function returning errors through an Error **errp parameter, and we generally don't bother to spell it out for the individual functions.
The part that needs to be spelling out is what success and failure mean. A qmp_FOO() returning an object returns null on failure.
Quoted 4 lines
Quoted 2 lines
We could right after out:
assert(!local_err || !retval). Not sure it's worthwhile.I think it IS worthwhile, because it would catch buggy callers. Not
We use the same assumption all over the place, without asserting it.
The boilerplate around calls using the Error API is bad enough as it is. I'm reluctant to make it worse by adding assertions.
Less of an issue in generated code, but violations are even less likely there.
However, I'm reluctant to assert in some places, but not all, because inconsistency breeds confusion.
In general, I prefer a function's pre- and post-conditions to be asserted in the function itself, not in its callers. Asserting selected parts of the post-condition in callers can be occasionally helpful to help human readers and static analyzers lacking insight into the function.
Quoted 9 lines
sure if after out: is the right place (then you'd need an initializer to cover any other code that jumps to out), but this would do the same
retval =
qmp_FOO(...);if (local_err) { assert(!retval); goto out; }
qmp_marshal_output_FOO(retval, ...);
Not quite the same: it doesn't catch the case !local_err && !retval. But it doesn't matter, because that case is bound to die in qmp_marshal_output_FOO().
Quoted 5 lines
Quoted 1 line
TL;DR: I concur with your analysis.
Is it worth dropping the dead initializer and adding the assert in the same pre-req cleanup patch? Do you want me to submit it since I did the analysis?
Let's drop the useless initializer. As explained above, I don't like the assertion for reasons explained above, but if you want it, I'm willing to take it anyway, in a separate follow-up patch.
I'd prefer to drop the initializer myself myself (with you credited in the commit message), because it's certainly less total work, and quite possibly less work for just for me.
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
Quoted 18 lines · Markus Armbruster
Quoted 9 lines
Quoted 6 lines
2. We can leak retval only when
qmp_FOO()returns non-null andlocal_erris non-null. This must not happen, because:a.
local_errmust be null before the call, andb. the call must not return non-null when it sets
local_err.We don't state that contract anywhere, but I doubt any of the
qmp_FOO()functions violate it, so it is worth making it part of the contract.It's a general Error API rule: set an error exactly on failure. It applies to any function returning errors through an Error **errp parameter, and we generally don't bother to spell it out for the individual functions.
The part that needs to be spelling out is what success and failure mean. A
qmp_FOO()returning an object returns null on failure.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
We could right after out:
assert(!local_err || !retval). Not sure it's worthwhile.I think it IS worthwhile, because it would catch buggy callers. Not
We use the same assumption all over the place, without asserting it.
Okay, you've got a point there.
Quoted 8 lines
Let's drop the useless initializer. As explained above, I don't like the assertion for reasons explained above, but if you want it, I'm willing to take it anyway, in a separate follow-up patch.
I'd prefer to drop the initializer myself myself (with you credited in the commit message), because it's certainly less total work, and quite possibly less work for just for me.
No need to add assertions. Maybe worth a patch to add a comment somewhere, maybe as a new section in docs/qapi-code-gen.txt, documenting how to write a handler and hook it into qmp-commands.hx, and what conditions the handler must obey. And I'm fine with you dropping the initializer as part of your v3 series.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
Quoted 38 lines · Eric Blake
Quoted 18 lines · Markus Armbruster
Quoted 9 lines
Quoted 6 lines
2. We can leak retval only when
qmp_FOO()returns non-null andlocal_erris non-null. This must not happen, because:a.
local_errmust be null before the call, andb. the call must not return non-null when it sets
local_err.We don't state that contract anywhere, but I doubt any of the
qmp_FOO()functions violate it, so it is worth making it part of the contract.It's a general Error API rule: set an error exactly on failure. It applies to any function returning errors through an Error **errp parameter, and we generally don't bother to spell it out for the individual functions.
The part that needs to be spelling out is what success and failure mean. A
qmp_FOO()returning an object returns null on failure.Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
We could right after out:
assert(!local_err || !retval). Not sure it's worthwhile.I think it IS worthwhile, because it would catch buggy callers. Not
We use the same assumption all over the place, without asserting it.
Okay, you've got a point there.
Quoted 8 lines
Let's drop the useless initializer. As explained above, I don't like the assertion for reasons explained above, but if you want it, I'm willing to take it anyway, in a separate follow-up patch.
I'd prefer to drop the initializer myself myself (with you credited in the commit message), because it's certainly less total work, and quite possibly less work for just for me.
Quoted 5 lines
No need to add assertions. Maybe worth a patch to add a comment somewhere, maybe as a new section in
docs/qapi-code-gen.txt, documenting how to write a handler and hook it into qmp-commands.hx, and what conditions the handler must obey. And I'm fine with you dropping the initializer as part of your v3 series.
Sounds like a plan. Thanks!
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
Quoted 14 lines · Markus Armbruster
Quoted 12 lines · Eric Blake
Quoted 10 lines · Markus Armbruster
Quoted 9 lines
Quoted 6 lines
2. We can leak retval only when
qmp_FOO()returns non-null andlocal_erris non-null. This must not happen, because:a.
local_errmust be null before the call, andb. the call must not return non-null when it sets
local_err.We don't state that contract anywhere, but I doubt any of the
qmp_FOO()functions violate it, so it is worth making it part of the contract.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
It's a general Error API rule: set an error exactly on failure. It applies to any function returning errors through an Error **errp parameter, and we generally don't bother to spell it out for the individual functions.
The part that needs to be spelling out is what success and failure mean. A
qmp_FOO()returning an object returns null on failure.
For qmp_FOO(), this is a reasonable contract. But our very own generated code does not follow these rules: visit_type_FOO() can assign into *obj even when setting an error, if it encounters a parse error halfway through the struct, leaving the caller responsible to still clean up the mess if it wants to avoid a memory leak.
Maybe that means our generated code needs to be reworked to properly clean up on a failed parse, such that *obj is guaranteed to be NULL if an error is returned. As a separate patch, of course.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
Quoted 20 lines · Eric Blake
Quoted 19 lines · Markus Armbruster
Quoted 17 lines · Eric Blake
Quoted 15 lines · Markus Armbruster
Quoted 9 lines
Quoted 6 lines
2. We can leak retval only when
qmp_FOO()returns non-null andlocal_erris non-null. This must not happen, because:a.
local_errmust be null before the call, andb. the call must not return non-null when it sets
local_err.We don't state that contract anywhere, but I doubt any of the
qmp_FOO()functions violate it, so it is worth making it part of the contract.It's a general Error API rule: set an error exactly on failure. It applies to any function returning errors through an Error **errp parameter, and we generally don't bother to spell it out for the individual functions.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The part that needs to be spelling out is what success and failure mean. A
qmp_FOO()returning an object returns null on failure.For
qmp_FOO(), this is a reasonable contract. But our very own generated code does not follow these rules:visit_type_FOO()can assign into *obj even when setting an error, if it encounters a parse error halfway through the struct, leaving the caller responsible to still clean up the mess if it wants to avoid a memory leak.
Assigning to *obj, then fail is tolerable[*]. Relying on the caller to free it is not. If we do that, it's a bug.
Quoted 3 lines
Maybe that means our generated code needs to be reworked to properly clean up on a failed parse, such that *obj is guaranteed to be NULL if an error is returned. As a separate patch, of course.
Yes. Would you like to propose a FIXME for me to put into this series?
[*] Leaving *obj alone on failure is nicer, but may not always be practical.
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
Quoted 6 lines · Markus Armbruster
Quoted 5 lines
For
qmp_FOO(), this is a reasonable contract. But our very own generated code does not follow these rules:visit_type_FOO()can assign into *obj even when setting an error, if it encounters a parse error halfway through the struct, leaving the caller responsible to still clean up the mess if it wants to avoid a memory leak.
Quoted 8 lines
Assigning to *obj, then fail is tolerable[*]. Relying on the caller to free it is not. If we do that, it's a bug.
Quoted 3 lines
Maybe that means our generated code needs to be reworked to properly clean up on a failed parse, such that *obj is guaranteed to be NULL if an error is returned. As a separate patch, of course.
Yes. Would you like to propose a FIXME for me to put into this series?
Done (see 12.5/47).
Quoted 4 lines
[*] Leaving *obj alone on failure is nicer, but may not always be practical.
I'm wondering if visit_end_struct() should be changed to accept a bool parameter that is true if the struct is ended normally, and false if an error has been detected. We may also need to alter visit_start_struct to return a bool (true if an object was allocated), to help feed our visitor logic on whether to pass true or false to the visit_end_struct().
We did something like that for visit_start_union()/visit_end_union(), but I suspect those visitor interfaces are also a bit screwy. Oh well, I'm not going to try and tweak it today, but am happy with just adding the FIXME.
Also, it would be really nice if we had docs in visitor.h and/or visitor-impl.h, to get some sort of feel for how the visitor is supposed to be used.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
Quoted 29 lines · Eric Blake
Quoted 6 lines · Markus Armbruster
Quoted 5 lines
For
qmp_FOO(), this is a reasonable contract. But our very own generated code does not follow these rules:visit_type_FOO()can assign into *obj even when setting an error, if it encounters a parse error halfway through the struct, leaving the caller responsible to still clean up the mess if it wants to avoid a memory leak.Quoted 8 lines
Assigning to *obj, then fail is tolerable[*]. Relying on the caller to free it is not. If we do that, it's a bug.
Quoted 3 lines
Maybe that means our generated code needs to be reworked to properly clean up on a failed parse, such that *obj is guaranteed to be NULL if an error is returned. As a separate patch, of course.
Yes. Would you like to propose a FIXME for me to put into this series?
Done (see 12.5/47).
Quoted 4 lines
[*] Leaving *obj alone on failure is nicer, but may not always be practical.
I'm wondering if
visit_end_struct()should be changed to accept a bool parameter that is true if the struct is ended normally, and false if an error has been detected. We may also need to altervisit_start_structto return a bool (true if an object was allocated), to help feed our visitor logic on whether to pass true or false to thevisit_end_struct().
Quoted 4 lines
We did something like that for visit_start_union()/visit_end_union(), but I suspect those visitor interfaces are also a bit screwy. Oh well, I'm not going to try and tweak it today, but am happy with just adding the FIXME.
We need to keep the introspection series reasonably focused. I can't afford a detour into visitor semantics right now, so we'll go with your FIXMEs.
Quoted 3 lines
Also, it would be really nice if we had docs in
visitor.hand/orvisitor-impl.h, to get some sort of feel for how the visitor is supposed to be used.
Indeed. We got a grand total of three one-liner comments, and one of them is inaccurate: actual visitors don't obey /* Must be set */.
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
In reply to Markus ArmbrusterQuoted 7 lines · Markus Armbruster
is_c_ptr()looks whether the end of the C text for the type looks like a pointer. Works, but is fragile.We now have a better tool: use QAPISchemaType method
c_null(). The initializers for non-pointers become prettier: 0, false or the enumeration constant with the value 0 instead of {0}.
Quoted 5 lines
One place looks suspicious: we initialize pointers, but not non-pointers. Either the initialization is superfluous and should be deleted, or the non-pointers need it as well, or something subtle is going on and needs a comment. Since I lack the time to figure it out now, mark it FIXME.
I commented on that in another mail - either we have a leak on failure (and need the initializer only for pointers), or we should add an assert (and don't need the initializer at all), depending on what semantics we want to enforce on all handler functions that set their error parameter.
Probably worth cleaning that up in a separate pre-req patch.
Quoted 7 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 19 +++++++------------ scripts/qapi.py | 3 --- 2 files changed, 7 insertions(+), 15 deletions(-)
Nice reduction in size. If the marshal code FIXME disappears due to a separate cleanup, then the rest of this commit is good to go:
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()
Quoted 14 lines · Eric Blake
Quoted 7 lines · Markus Armbruster
is_c_ptr()looks whether the end of the C text for the type looks like a pointer. Works, but is fragile.We now have a better tool: use QAPISchemaType method
c_null(). The initializers for non-pointers become prettier: 0, false or the enumeration constant with the value 0 instead of {0}.Quoted 5 lines
One place looks suspicious: we initialize pointers, but not non-pointers. Either the initialization is superfluous and should be deleted, or the non-pointers need it as well, or something subtle is going on and needs a comment. Since I lack the time to figure it out now, mark it FIXME.
Quoted 5 lines
I commented on that in another mail - either we have a leak on failure (and need the initializer only for pointers), or we should add an assert (and don't need the initializer at all), depending on what semantics we want to enforce on all handler functions that set their error parameter.
Probably worth cleaning that up in a separate pre-req patch.
A function call should either fail completely or succeed completely.
On success, it must not set an error (d'oh!).
On failure, it must set an error (d'oh again!), and should not return stuff for the caller to clean up.
The rules on setting errors together with the rule that a &local_err argument must contain null before the call ensures that afterwards local_err is set exactly when the call failed.
The rule on returning stuff lets us keep the error path simple: no need for cleaning up anything returned by the function. This is pervasive in the code. We generally don't bother to assert() the function actually obeys the rule.
Perhaps I should fix up my pending patch "error: Revamp interface documentation"[*] to cover the "should not return stuff for the caller to clean up" part.
Apply to handler functions returning objects in need of clean up:
retval = qmp_FOO(... args ..., &local_err);
if (local_err) {
goto out;
}
qmp_marshal_output_FOO(retval, ret, &local_err);On success, qmp_FOO() must not touch local_err.
On failure, qmp_FOO() must set local_err, and return null.
Bypassing the cleanup qmp_marshal_output_FOO() is therfore perfectly fine.
Quoted 10 lines
Quoted 6 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 19 +++++++------------ scripts/qapi.py | 3 --- 2 files changed, 7 insertions(+), 15 deletions(-)Nice reduction in size. If the marshal code FIXME disappears due to a separate cleanup, then the rest of this commit is good to go:
Reviewed-by Eric Blake <[email protected]>
Thanks!
[*] https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg04756.html
RFC PATCH v2 30/47 qapi: De-duplicate enum code generation
In reply to Markus ArmbrusterDuplicated in commit 21cd70d. Yes, we can't import qapi-types, but that's no excuse. Move the helpers from qapi-types.py to qapi.py, and replace the duplicates in qapi-event.py.
The generated event enumeration type gains a _MAX member, and its lookup table becomes const-correct (see commit 2e4450f).
--- scripts/qapi-event.py | 67 +++------------------------------------------------ scripts/qapi-types.py | 55 ------------------------------------------ scripts/qapi.py | 55 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 59 insertions(+), 118 deletions(-)
scripts/qapi-event.py +4 -63
@@ -153,63 +153,6 @@ def generate_event_implement(api_name, event_name, params): return ret - -# Following are the functions that generate an enum type for all defined -# events, similar to qapi-types.py. Here we already have enum name and -# values which were generated before and recorded in event_enum_*. It also -# works around the issue that "import qapi-types" can't work. - -def generate_event_enum_decl(event_enum_name, event_enum_values): - lookup_decl = mcgen(''' - -extern const char *%(event_enum_name)s_lookup[]; -''', - event_enum_name = event_enum_name) - - enum_decl = mcgen(''' -typedef enum %(event_enum_name)s { -''', - event_enum_name = event_enum_name) - - # append automatically generated _MAX value - enum_max_value = c_enum_const(event_enum_name, "MAX") - enum_values = event_enum_values + [ enum_max_value ] - - i = 0 - for value in enum_values: - enum_decl += mcgen(''' - %(value)s = %(i)d, -''', - value = value, - i = i) - i += 1 - - enum_decl += mcgen(''' -} %(event_enum_name)s; -''', - event_enum_name = event_enum_name) - - return lookup_decl + enum_decl - -def generate_event_enum_lookup(event_enum_name, event_enum_strings): - ret = mcgen(''' - -const char *%(event_enum_name)s_lookup[] = { -''', - event_enum_name = event_enum_name) - - for string in event_enum_strings: - ret += mcgen(''' - "%(string)s", -''', - string = string) - - ret += mcgen(''' - NULL, -}; -''') - return ret - (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line() c_comment = '''
@@ -266,8 +209,7 @@ fdecl.write(mcgen(''' exprs = QAPISchema(input_file).get_exprs() event_enum_name = c_name(prefix + "QAPIEvent", protect=False) -event_enum_values = [] -event_enum_strings = [] +event_names = [] for expr in exprs: if expr.has_key('event'):
@@ -286,12 +228,11 @@ for expr in exprs: fdef.write(ret) # Record it, and generate enum later - event_enum_values.append(event_enum_value) - event_enum_strings.append(event_name) + event_names.append(event_name) -ret = generate_event_enum_decl(event_enum_name, event_enum_values) +ret = generate_enum(event_enum_name, event_names) fdecl.write(ret) -ret = generate_event_enum_lookup(event_enum_name, event_enum_strings) +ret = generate_enum_lookup(event_enum_name, event_names) fdef.write(ret) close_output(fdef, fdecl)
scripts/qapi-types.py +0 -55
61 lines not on this page · raw message
scripts/qapi.py +55 -0
61 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 30/47] qapi: De-duplicate enum code generation
Quoted 5 lines · Markus Armbruster
Duplicated in commit 21cd70d. Yes, we can't import qapi-types, but that's no excuse. Move the helpers from
qapi-types.pytoqapi.py, and replace the duplicates inqapi-event.py.The generated event enumeration type gains a
_MAXmember,
Not quite; the _MAX member was already there, but done by hand [1] instead of by common code.
Quoted 2 lines
and its
lookup table becomes const-correct (see commit 2e4450f).
Maybe what you meant instead of gaining _MAX was that it also gains C99 initializers:
-const char *QAPIEvent_lookup[] = {
- "ACPI_DEVICE_OST",
- "BALLOON_CHANGE",
...
- "WATCHDOG",
- NULL,
+const char *const QAPIEvent_lookup[] = {
+ [QAPI_EVENT_ACPI_DEVICE_OST] = "ACPI_DEVICE_OST",
+ [QAPI_EVENT_BALLOON_CHANGE] = "BALLOON_CHANGE",
Overall, a good change.
Quoted 7 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-event.py | 67 +++------------------------------------------------ scripts/qapi-types.py | 55 ------------------------------------------ scripts/qapi.py | 55 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 59 insertions(+), 118 deletions(-)
Since only the commit message needs tweaking,
scripts/qapi-event.py def generate_event_implement(api_name, event_name, params): · line 156
Quoted 10 lines
scripts/qapi-event.py +0
-3@@ -153,63 +153,6 @@ def generate_event_implement(api_name, event_name, params): - # append automatically generated _MAX value - enum_max_value = c_enum_const(event_enum_name, "MAX") - enum_values = event_enum_values + [ enum_max_value ]
[1]
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 30/47] qapi: De-duplicate enum code generation
Quoted 9 lines · Eric Blake
Quoted 5 lines · Markus Armbruster
Duplicated in commit 21cd70d. Yes, we can't import qapi-types, but that's no excuse. Move the helpers from
qapi-types.pytoqapi.py, and replace the duplicates inqapi-event.py.The generated event enumeration type gains a
_MAXmember,Not quite; the
_MAXmember was already there, but done by hand [1] instead of by common code.
You're right; I was confused.
Quoted 6 lines
Quoted 2 lines
and its
lookup table becomes const-correct (see commit 2e4450f).Maybe what you meant instead of gaining
_MAXwas that it also gains C99 initializers:
Review
Quoted 9 lines
(quoted diff) +3
-5-const char *QAPIEvent_lookup[] = { - "ACPI_DEVICE_OST", - "BALLOON_CHANGE", - "WATCHDOG", - NULL, +const char *const QAPIEvent_lookup[] = { + [QAPI_EVENT_ACPI_DEVICE_OST] = "ACPI_DEVICE_OST", + [QAPI_EVENT_BALLOON_CHANGE] = "BALLOON_CHANGE",
That's not what I had in mind when I wrote the sentence about _MAX. but mentioning the change in the commit message won't hurt. Note to self:
comes from commit 912ae9c.
Quoted 11 lines
Overall, a good change.
Quoted 8 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-event.py | 67 +++------------------------------------------------ scripts/qapi-types.py | 55 ------------------------------------------ scripts/qapi.py | 55 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 59 insertions(+), 118 deletions(-)
Quoted 2 lines
Since only the commit message needs tweaking,
Reviewed-by Eric Blake <[email protected]>
Thanks!
[...]
RFC PATCH v2 31/47 qapi-event: Eliminate global variable event_enum_value
In reply to Markus Armbruster--- scripts/qapi-event.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi-event.py +1 -2
@@ -137,7 +137,7 @@ def generate_event_implement(api_name, event_name, params): emit(%(event_enum_value)s, qmp, &local_err); """, - event_enum_value = event_enum_value) + event_enum_value = c_enum_const(event_enum_name, event_name)) # step 5: clean up if params:
@@ -223,7 +223,6 @@ for expr in exprs: fdecl.write(ret) # We need an enum value per event - event_enum_value = c_enum_const(event_enum_name, event_name) ret = generate_event_implement(api_name, event_name, params) fdef.write(ret)
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 31/47] qapi-event: Eliminate global variable event_enum_value
Quoted 5 lines · Markus Armbruster
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-event.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
RFC PATCH v2 32/47 qapi-event: Convert to QAPISchemaVisitor, fixing data with base
In reply to Markus ArmbrusterFixes events whose data is struct with base to include the struct's base members. Test case is qapi-schema-test.json's event __org.qemu_x-command:
{ 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }
{ 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
'data': { '__org.qemu_x-member2': 'str' } }
{ 'struct': '__org.qemu_x-Base',
'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }Patch's effect on generated qapi_event_send___org_qemu_x_event():
-void qapi_event_send___org_qemu_x_event(const char *__org_qemu_x_member2,
+void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum __org_qemu_x_member1,
+ const char *__org_qemu_x_member2,
Error **errp)
{
QDict *qmp;
@@ -224,6 +225,10 @@ void qapi_event_send___org_qemu_x_event(
goto clean;
}
+ visit_type___org_qemu_x_Enum(v, &__org_qemu_x_member1, "__org.qemu_x-member1", &local_err);
+ if (local_err) {
+ goto clean;
+ }
visit_type_str(v, (char **)&__org_qemu_x_member2, "__org.qemu_x-member2", &local_err);
if (local_err) {
goto clean;--- scripts/qapi-event.py | 87 ++++++++++++++++----------------- tests/qapi-schema/qapi-schema-test.json | 3 -- 2 files changed, 43 insertions(+), 47 deletions(-)
scripts/qapi-event.py +43 -44
@@ -2,28 +2,29 @@ # QAPI event generator # # Copyright (c) 2014 Wenchao Xia +# Copyright (c) 2013-2015 Red Hat Inc. # # Authors: # Wenchao Xia <[email protected]> +# Markus Armbruster <[email protected]> # # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. -from ordereddict import OrderedDict from qapi import * -def _generate_event_api_name(event_name, params): +def _generate_event_api_name(event_name, data): api_name = "void qapi_event_send_%s(" % c_name(event_name).lower(); l = len(api_name) - if params: - for argname, argentry, optional in parse_args(params): - if optional: - api_name += "bool has_%s,\n" % c_name(argname) + if data: + for m in data.members: + if m.optional: + api_name += "bool has_%s,\n" % c_name(m.name) api_name += "".ljust(l) - api_name += "%s %s,\n" % (c_type(argentry, is_param=True), - c_name(argname)) + api_name += "%s %s,\n" % (m.type.c_type(is_param=True), + c_name(m.name)) api_name += "".ljust(l) api_name += "Error **errp)"
@@ -51,7 +52,7 @@ def generate_event_implement(api_name, event_name, params): """, api_name = api_name) - if params: + if params and params.members: ret += mcgen(""" QmpOutputVisitor *qov; Visitor *v;
@@ -72,7 +73,7 @@ def generate_event_implement(api_name, event_name, params): event_name = event_name) # step 3: visit the params if params != None - if params: + if params and params.members: ret += mcgen(""" qov = qmp_output_visitor_new(); g_assert(qov);
@@ -89,15 +90,15 @@ def generate_event_implement(api_name, event_name, params): """, event_name = event_name) - for argname, argentry, optional in parse_args(params): - if optional: + for memb in params.members: + if memb.optional: ret += mcgen(""" if (has_%(var)s) { """, - var = c_name(argname)) + var=c_name(memb.name)) push_indent() - if argentry == "str": + if memb.type.name == "str": var_type = "(char **)" else: var_type = ""
@@ -109,11 +110,11 @@ def generate_event_implement(api_name, event_name, params): } """, var_type = var_type, - var = c_name(argname), - type = type_name(argentry), - name = argname) + var=c_name(memb.name), + type=memb.type.c_name(), + name=memb.name) - if optional: + if memb.optional: pop_indent() ret += mcgen(""" }
@@ -140,7 +141,7 @@ def generate_event_implement(api_name, event_name, params): event_enum_value = c_enum_const(event_enum_name, event_name)) # step 5: clean up - if params: + if params and params.members: ret += mcgen(""" clean: qmp_output_visitor_cleanup(qov);
@@ -153,6 +154,24 @@ def generate_event_implement(api_name, event_name, params): return ret +class QAPISchemaGenEventVisitor(QAPISchemaVisitor): + def __init__(self): + self.decl = None + self.defn = None + self.event_names = None + def visit_begin(self): + self.decl = '' + self.defn = '' + self.event_names = [] + def visit_end(self): + self.decl += generate_enum(event_enum_name, self.event_names) + self.defn += generate_enum_lookup(event_enum_name, self.event_names) + def visit_event(self, name, info, data): + api_name = _generate_event_api_name(name, data) + self.decl += generate_event_declaration(api_name) + self.defn += generate_event_implement(api_name, name, data) + self.event_names.append(name) + (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line() c_comment = '''
@@ -206,32 +225,12 @@ fdecl.write(mcgen(''' ''', prefix=prefix)) -exprs = QAPISchema(input_file).get_exprs() - event_enum_name = c_name(prefix + "QAPIEvent", protect=False) -event_names = [] -for expr in exprs: - if expr.has_key('event'): - event_name = expr['event'] - params = expr.get('data') - if params and len(params) == 0: - params = None - - api_name = _generate_event_api_name(event_name, params) - ret = generate_event_declaration(api_name) - fdecl.write(ret) - - # We need an enum value per event - ret = generate_event_implement(api_name, event_name, params) - fdef.write(ret) - - # Record it, and generate enum later - event_names.append(event_name) - -ret = generate_enum(event_enum_name, event_names) -fdecl.write(ret) -ret = generate_enum_lookup(event_enum_name, event_names) -fdef.write(ret) +schema = QAPISchema(input_file) +gen = QAPISchemaGenEventVisitor() +schema.visit(gen) +fdef.write(gen.defn) +fdecl.write(gen.decl) close_output(fdef, fdecl)
tests/qapi-schema/qapi-schema-test.json +0 -3
@@ -126,9 +126,6 @@ { 'alternate': '__org.qemu_x-Alt', 'data': { '__org.qemu_x-branch': 'str', 'b': '__org.qemu_x-Base' } } { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' } -# FIXME generated qapi_event_send___org_qemu_x_event() has only a -# parameter for data's member __org_qemu_x_member2, none for its base -# __org.qemu_x-Base's member __org_qemu_x_member1 { 'command': '__org.qemu_x-command', 'data': { 'a': ['__org.qemu_x-Enum'], 'b': ['__org.qemu_x-Struct'], 'c': '__org.qemu_x-Union2', 'd': '__org.qemu_x-Alt' },
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 32/47] qapi-event: Convert to QAPISchemaVisitor, fixing data with base
Quoted 11 lines · Markus Armbruster
Fixes events whose data is struct with base to include the struct's base members. Test case is qapi-schema-test.json's event __org.qemu_x-command:
{ 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' } { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base', 'data': { '__org.qemu_x-member2': 'str' } } { 'struct': '__org.qemu_x-Base', 'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }
No change to generated code in qemu proper, so this is a corner case we are not yet exploiting. But good to have it fixed :)
Quoted 7 lines
Patch's effect on generated
qapi_event_send___org_qemu_x_event():-void qapi_event_send___org_qemu_x_event(const char
*__org_qemu_x_member2, +void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum__org_qemu_x_member1,
+ const char*__org_qemu_x_member2,
Error **errp)
Ouch - I think we have a bug in qapi.py:check_event(). There, we call check_type(... allow_metas=['union', 'struct']) - but it looks like the generated signature requires that we have no variants, which means we cannot have:
{ 'union': 'Un', 'data': ... }
{ 'event': 'EV', 'data': 'Un' }
because it would fail C generation. Sounds like you should add a check to that in 14/47, and a fix for it in 15/47.
Quoted 11 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-event.py | 87 ++++++++++++++++----------------- tests/qapi-schema/qapi-schema-test.json | 3 -- 2 files changed, 43 insertions(+), 47 deletions(-)scripts/qapi-event.py +0
-0
scripts/qapi-event.py line 5
Quoted 4 lines
scripts/qapi-event.py +1
-0@@ -2,28 +2,29 @@ # QAPI event generator # # Copyright (c) 2014 Wenchao Xia +# Copyright (c) 2013-2015 Red Hat Inc.
Umm - the file didn't exist until 2014; and to my knowledge, you aren't adding anything to it that was copied from some other file dating back to 2013. Using the range 2014-2015 might be better.
But that's minor. And assuming that you reject union types for events in an earlier patch, the rest of this is fine:
Quoted 8 lines
scripts/qapi-event.py +2
-2@@ -89,15 +90,15 @@ def generate_event_implement(api_name, event_name, params): """, event_name = event_name) - for argname, argentry, optional in parse_args(params): - if optional: + for memb in params.members: + if memb.optional:
scripts/qapi-event.py def generate_event_implement(api_name, event_name, params): · line 102
Quoted 10 lines
scripts/qapi-event.py +2
-2ret += mcgen(""" if (has_%(var)s) { """, - var = c_name(argname)) + var=c_name(memb.name)) push_indent() - if argentry == "str": + if memb.type.name == "str": var_type = "(char **)"
Our visitors require us to cast away const. Is that something we should consider reworking, so that we don't need to do that? But it's a question for another day.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH RFC v2 32/47] qapi-event: Convert to QAPISchemaVisitor, fixing data with base
Quoted 29 lines · Eric Blake
Quoted 11 lines · Markus Armbruster
Fixes events whose data is struct with base to include the struct's base members. Test case is qapi-schema-test.json's event __org.qemu_x-command:
{ 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' } { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base', 'data': { '__org.qemu_x-member2': 'str' } } { 'struct': '__org.qemu_x-Base', 'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }No change to generated code in qemu proper, so this is a corner case we are not yet exploiting. But good to have it fixed :)
Quoted 7 lines
Patch's effect on generated
qapi_event_send___org_qemu_x_event():-void qapi_event_send___org_qemu_x_event(const char
*__org_qemu_x_member2, +void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum__org_qemu_x_member1,
+ const char*__org_qemu_x_member2,
Error **errp)Ouch - I think we have a bug in qapi.py:check_event(). There, we call check_type(... allow_metas=['union', 'struct']) - but it looks like the generated signature requires that we have no variants, which means we cannot have:
Quoted 5 lines
{ 'union': 'Un', 'data': ... }
{ 'event': 'EV', 'data': 'Un' }because it would fail C generation. Sounds like you should add a check to that in 14/47, and a fix for it in 15/47.
Will do.
Quoted 6 lines
Quoted 6 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-event.py | 87 ++++++++++++++++----------------- tests/qapi-schema/qapi-schema-test.json | 3 -- 2 files changed, 43 insertions(+), 47 deletions(-)
Quoted 13 lines
scripts/qapi-event.py line 5
Quoted 9 lines
scripts/qapi-event.py +1
-0@@ -2,28 +2,29 @@ # QAPI event generator # # Copyright (c) 2014 Wenchao Xia +# Copyright (c) 2013-2015 Red Hat Inc.Umm - the file didn't exist until 2014; and to my knowledge, you aren't adding anything to it that was copied from some other file dating back to 2013. Using the range 2014-2015 might be better.
Pasto, will fix.
Quoted 19 lines
But that's minor. And assuming that you reject union types for events in an earlier patch, the rest of this is fine:
Reviewed-by Eric Blake <[email protected]>Quoted 15 lines
scripts/qapi-event.py +3
-3@@ -89,15 +90,15 @@ def generate_event_implement(api_name, event_name, params): """, event_name = event_name) - for argname, argentry, optional in parse_args(params): - if optional: + for memb in params.members: + if memb.optional: ret += mcgen(""" if (has_%(var)s) { """, - var = c_name(argname)) + var=c_name(memb.name)) push_indent()
Quoted 7 lines
scripts/qapi-event.py def generate_event_implement(api_name, event_name, params): · line 102
Quoted 3 lines
scripts/qapi-event.py +1
-1- if argentry == "str": + if memb.type.name == "str": var_type = "(char **)"Our visitors require us to cast away const. Is that something we should consider reworking, so that we don't need to do that? But it's a question for another day.
Yes, it's annoying, and yes, we need to leave it for later.
Thanks!
RFC PATCH v2 33/47 qapi: Clean up after recent conversions to QAPISchemaVisitor
In reply to Markus ArmbrusterDrop helper functions that are now unused.
Make pylint reasonably happy.
Rename generate_FOO() functions to gen_FOO() for consistency.
Use more consistent and sensible variable names.
Consistently use c_ for mapping keys when their value is a C identifier or type.
Simplify gen_enum() and gen_visit_union()
Consistently use single quotes for C text string literals.
--- scripts/qapi-commands.py | 109 +++++++++++++++++++------------------ scripts/qapi-event.py | 117 +++++++++++++++++++--------------------- scripts/qapi-types.py | 68 ++++++++++++----------- scripts/qapi-visit.py | 121 ++++++++++++++++++++--------------------- scripts/qapi.py | 138 +++++++++-------------------------------------- 5 files changed, 229 insertions(+), 324 deletions(-)
scripts/qapi-commands.py +56 -53
@@ -15,20 +15,20 @@ from qapi import * import re -def generate_command_decl(name, args, ret_type): - arglist="" +def gen_command_decl(name, args, rets): + argstr = '' if args: for memb in args.members: - argtype = memb.type.c_type(is_param=True) if memb.optional: - arglist += "bool has_%s, " % c_name(memb.name) - arglist += "%s %s, " % (argtype, c_name(memb.name)) + argstr += 'bool has_%s, ' % c_name(memb.name) + argstr += '%s %s, ' % (memb.type.c_type(is_param=True), + c_name(memb.name)) return mcgen(''' -%(ret_type)s qmp_%(name)s(%(args)sError **errp); +%(c_type)s qmp_%(c_name)s(%(args)sError **errp); ''', - ret_type=(ret_type and ret_type.c_type()) or 'void', - name=c_name(name), - args=arglist) + c_type=(rets and rets.c_type()) or 'void', + c_name=c_name(name), + args=argstr) def gen_err_check(err): if not err:
@@ -40,34 +40,37 @@ if (%(err)s) { ''', err=err) -def gen_sync_call(name, args, ret_type): - ret = "" - arglist="" - retval="" - if ret_type: - retval = "retval = " +def gen_call(name, args, rets): + ret = '' + + argstr = '' if args: for memb in args.members: if memb.optional: - arglist += "has_%s, " % c_name(memb.name) - arglist += "%s, " % c_name(memb.name) + argstr += 'has_%s, ' % c_name(memb.name) + argstr += '%s, ' % c_name(memb.name) + + lhs = '' + if rets: + lhs = 'retval = ' + push_indent() ret = mcgen(''' -%(retval)sqmp_%(name)s(%(args)s&local_err); +%(lhs)sqmp_%(c_name)s(%(args)s&local_err); ''', - name=c_name(name), args=arglist, retval=retval) - if ret_type: + c_name=c_name(name), args=argstr, lhs=lhs) + if rets: ret += gen_err_check('local_err') ret += mcgen(''' qmp_marshal_output_%(c_name)s(retval, ret, &local_err); ''', - c_name=c_name(name)) + c_name=c_name(name)) pop_indent() return ret def gen_visitor_input_containers_decl(args): - ret = "" + ret = '' push_indent() if args:
@@ -81,16 +84,16 @@ Visitor *v; return ret def gen_visitor_input_vars_decl(args): - ret = "" + ret = '' push_indent() if args: for memb in args.members: if memb.optional: ret += mcgen(''' -bool has_%(argname)s = false; +bool has_%(c_name)s = false; ''', - argname=c_name(memb.name)) + c_name=c_name(memb.name)) ret += mcgen(''' %(c_type)s %(c_name)s = %(c_null)s; ''',
@@ -102,7 +105,7 @@ bool has_%(argname)s = false; return ret def gen_visitor_input_block(args, dealloc=False): - ret = "" + ret = '' errparg = '&local_err' errarg = 'local_err'
@@ -113,7 +116,7 @@ def gen_visitor_input_block(args, dealloc=False): if dealloc: errparg = 'NULL' - errarg = None; + errarg = None ret += mcgen(''' qmp_input_visitor_cleanup(mi); md = qapi_dealloc_visitor_new();
@@ -138,10 +141,10 @@ if (has_%(c_name)s) { c_name=c_name(memb.name)) push_indent() ret += mcgen(''' -visit_type_%(visitor)s(v, &%(c_name)s, "%(name)s", %(errp)s); +visit_type_%(c_type)s(v, &%(c_name)s, "%(name)s", %(errp)s); ''', c_name=c_name(memb.name), name=memb.name, - visitor=memb.type.c_name(), errp=errparg) + c_type=memb.type.c_name(), errp=errparg) ret += gen_err_check(errarg) if memb.optional: pop_indent()
@@ -156,13 +159,13 @@ qapi_dealloc_visitor_cleanup(md); pop_indent() return ret -def gen_marshal_output(name, ret_type): - if not ret_type: - return "" +def gen_marshal_output(name, rets): + if not rets: + return '' ret = mcgen(''' -static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_out, Error **errp) +static void qmp_marshal_output_%(c_cmd_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp) { Error *local_err = NULL; QmpOutputVisitor *mo = qmp_output_visitor_new();
@@ -170,7 +173,7 @@ static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_o Visitor *v; v = qmp_output_get_visitor(mo); - visit_type_%(visitor)s(v, &ret_in, "unused", &local_err); + visit_type_%(c_name)s(v, &ret_in, "unused", &local_err); if (local_err) { goto out; }
@@ -181,23 +184,23 @@ out: qmp_output_visitor_cleanup(mo); md = qapi_dealloc_visitor_new(); v = qapi_dealloc_get_visitor(md); - visit_type_%(visitor)s(v, &ret_in, "unused", NULL); + visit_type_%(c_name)s(v, &ret_in, "unused", NULL); qapi_dealloc_visitor_cleanup(md); } ''', - c_ret_type=ret_type.c_type(), c_name=c_name(name), - visitor=ret_type.c_name()) + c_type=rets.c_type(), c_cmd_name=c_name(name), + c_name=rets.c_name()) return ret -def gen_marshal_input_decl(name, middle_mode): +def gen_marshal_input_decl(name): ret = 'void qmp_marshal_input_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name) if not middle_mode: - ret = "static " + ret + ret = 'static ' + ret return ret -def gen_marshal_input(name, args, ret_type, middle_mode): - hdr = gen_marshal_input_decl(name, middle_mode) +def gen_marshal_input(name, args, rets): + hdr = gen_marshal_input_decl(name) ret = mcgen('''
@@ -207,12 +210,12 @@ def gen_marshal_input(name, args, ret_type, middle_mode): ''', header=hdr) - if ret_type: + if rets: # FIXME fishy: only pointers are initialized - if ret_type.c_null() == 'NULL': - retval = " %s retval = NULL;" % ret_type.c_type() + if rets.c_null() == 'NULL': + retval = ' %s retval = NULL;' % rets.c_type() else: - retval = " %s retval;" % ret_type.c_type() + retval = ' %s retval;' % rets.c_type() ret += mcgen(''' %(retval)s ''',
@@ -229,9 +232,9 @@ def gen_marshal_input(name, args, ret_type, middle_mode): ''') - ret += gen_sync_call(name, args, ret_type) + ret += gen_call(name, args, rets) - if re.search('^ *goto out\\;', ret, re.MULTILINE): + if re.search('^ *goto out;', ret, re.MULTILINE): ret += mcgen(''' out:
@@ -254,8 +257,8 @@ def gen_register_command(name, success_response): ret = mcgen(''' qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s); ''', - name=name, c_name=c_name(name), - opts=options) + name=name, c_name=c_name(name), + opts=options) pop_indent() return ret
@@ -289,12 +292,12 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): def visit_command(self, name, info, args, rets, gen, success_response): if not gen: return - self.decl += generate_command_decl(name, args, rets) + self.decl += gen_command_decl(name, args, rets) if rets: self.defn += gen_marshal_output(name, rets) if middle_mode: - self.decl += gen_marshal_input_decl(name, middle_mode) + ';\n' - self.defn += gen_marshal_input(name, args, rets, middle_mode) + self.decl += gen_marshal_input_decl(name) + ';\n' + self.defn += gen_marshal_input(name, args, rets) if not middle_mode: self.regy += gen_register_command(name, success_response)
@@ -354,7 +357,7 @@ fdef.write(mcgen(''' #include "%(prefix)sqmp-commands.h" ''', - prefix=prefix)) + prefix=prefix)) fdecl.write(mcgen(''' #include "%(prefix)sqapi-types.h"
scripts/qapi-event.py +54 -63
199 lines not on this page · raw message
scripts/qapi-types.py +33 -35
170 lines not on this page · raw message
scripts/qapi-visit.py +61 -60
284 lines not on this page · raw message
scripts/qapi.py +25 -113
191 lines not on this page · raw message
Signature · 1 line
1.9.3
Re: [Qemu-devel] [PATCH RFC v2 33/47] qapi: Clean up after recent conversions to QAPISchemaVisitor
Quoted 6 lines · Markus Armbruster
Drop helper functions that are now unused.
Make pylint reasonably happy.
Rename
generate_FOO()functions togen_FOO()for consistency.
Quoted 8 lines
Use more consistent and sensible variable names.
Consistently use c_ for mapping keys when their value is a C identifier or type.
Simplify
gen_enum()andgen_visit_union()Consistently use single quotes for C text string literals.
Not sure if it is worth splitting this into pieces. Fortunately, there are no changes to generated output.
Quoted 15 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi-commands.py | 109 +++++++++++++++++++------------------ scripts/qapi-event.py | 117 +++++++++++++++++++--------------------- scripts/qapi-types.py | 68 ++++++++++++----------- scripts/qapi-visit.py | 121 ++++++++++++++++++++--------------------- scripts/qapi.py | 138 +++++++++-------------------------------------- 5 files changed, 229 insertions(+), 324 deletions(-)scripts/qapi-commands.py +0
-0
scripts/qapi-commands.py line 18
Quoted 6 lines
scripts/qapi-commands.py +1
-2@@ -15,20 +15,20 @@ from qapi import * import re -def generate_command_decl(name, args, ret_type): - arglist="" +def gen_command_decl(name, args, rets):
I can see how 'args' is plural (even if it is a single string for the name of a type containing the args), but should it be 'ret' instead of 'rets'?
scripts/qapi-commands.py if (%(err)s) { · line 43
Quoted 11 lines
scripts/qapi-commands.py +1
-6@@ -40,34 +40,37 @@ if (%(err)s) { ''', err=err) -def gen_sync_call(name, args, ret_type): - ret = "" - arglist="" - retval="" - if ret_type: - retval = "retval = " +def gen_call(name, args, rets):
At least you're consistent on naming it 'rets',
scripts/qapi-commands.py if (%(err)s) { · line 44
Quoted 1 line
scripts/qapi-commands.py +1
-0+ ret = ''
and the naming lets you distinguish between the parameter (the type describing returned fields) and the local string (the generated C code holding the return information).
scripts/qapi-commands.py def generate_event_implement(api_name, event_name, params): · line 80
Quoted 7 lines
scripts/qapi-commands.py +2
-1@@ -82,45 +76,46 @@ def generate_event_implement(api_name, event_name, params): + # Ugly: need to cast away the const if memb.type.name == "str": - var_type = "(char **)" + cast = "(char **)"
And to think I called it out in a previous patch. So you noticed it too :)
Don't you want to use '(char **)' here, since it is a literal string destined for generated C?
scripts/qapi-commands.py def generate_event_implement(api_name, event_name, params): · line 82
Quoted 3 lines
scripts/qapi-commands.py +1
-1else: - var_type = "" + cast = ""
and '' here?
Quoted 12 lines
scripts/qapi-commands.py +3
-2+++ b/scripts/qapi-types.py@@ -16,22 +16,22 @@ from qapi import * def gen_fwd_object_or_array(name): return mcgen(''' -typedef struct %(name)s %(name)s; +typedef struct %(c_name)s %(c_name)s; ''', - name=c_name(name)) + c_name=c_name(name)) def gen_array(name, element_type):
scripts/qapi-commands.py from qapi import * · line 31
Quoted 10 lines
scripts/qapi-commands.py +2
-2return mcgen(''' -struct %(name)s { +struct %(c_name)s { union { %(c_type)s value; uint64_t padding; }; - struct %(name)s *next; + struct %(c_name)s *next;
May be some churn here if you like my comment earlier in the series that this 'struct' is redundant.
scripts/qapi-commands.py from qapi import * · line 37
Quoted 8 lines
scripts/qapi-commands.py +3
-2+++ b/scripts/qapi-visit.py -def generate_visit_struct_fields(name, members, base = None): +def gen_visit_struct_fields(name, base, members): struct_fields_seen.add(name) - type=base.c_name(), c_name=c_name('base')) + c_type=base.c_name(), c_name=c_name('base'))
Possible churn here based on my earlier comments about c_name(constant) being constant.
Fairly big, but aside from some minor '' quoting issues, I didn't see anything wrong.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org