Skip to contentSkip to conversation

qapi: QMP introspection

RFC PATCH Superseded Markus Armbruster 2 participants 199 messages Jul 1, 2015 – Aug 4, 2015 Kbuild Page 1 of 2

v2 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v2 cover qapi: QMP introspection

Still RFC, because it has a couple of FIXMEs and TODOs.

* PATCH 01-19 are preliminary fixes and cleanups. Not really RFC.

* PATCH 20-38 basically replace the interemediate representation. The replacement isn't complete, but fully functional. See PATCH 21 for rationale and future work.

* PATCH 39-44 replace the '**' type bypass by a proper top type called 'any'.

* PATCH 45 is the introspection RFC.

* PATCH 46-47 are minor variations to hide a bit more detail from clients.

I'm afraid the size of my series will strain reviewers. I did my level best to split things up into reviewable pieces.

Series depends on my "[PATCH for-2.4 0/2] qom: Fix misuse of Error API" to pass tests.

You can also fetch it from my git://repo.or.cz/qemu/armbru.git branch qapi-introspect.

I'll be offline for two weeks starting next Monday.

Bombs away!

v2:
* Old PATCH 01-18 have been merged
* PATCH 19 exploded. Comparing against RFC v1 is almost certainly a waste of time, except perhaps for the QAPI schema introspect.json. The awkward alternate Value got replaced by the proper top type 'any'.
* For even more kaboom, I fixed and cleaned up along the way.

Markus Armbruster (47):
qapi: Clarify docs on including the same file multiple times
qapi: Clean up cgen() and mcgen()
qapi: Simplify guardname()
qapi-event: Clean up how name of enum QAPIEvent is made
qapi: Reject -p arguments that break qapi-event.py
qapi: Drop unused and useless parameters and variables
qapi: Generate a nicer struct for flat unions
qapi-visit: Fix generated code when schema has forward refs
qapi-visit: Replace list implicit_structs by set
qapi-visit: Fix two name arguments passed to visitors tests/qapi-schema: Document alternate's enum lacks visit function tests/qapi-schema: Document events with with base don't work tests/qapi-schema: Restore test case for flat union base bug
qapi-tests: New tests for union, alternate command arguments
qapi: Fix to reject union command arguments
qapi-commands: Fix gen_err_check(e) for e and e != 'local_err'
qapi-commands: Inline gen_marshal_output_call()
qapi-commands: Don't feed output of mcgen() to mcgen() again
qapi: Generated code cleanup
qapi: Rename class QAPISchema to QAPISchemaParser
qapi: New QAPISchema intermediate reperesentation
qapi: QAPISchema code generation helper methods
qapi: New QAPISchemaVisitor
tests/qapi-schema: Convert test harness to QAPISchemaVisitor
qapi: Make generators work on sorted schema expressions
qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
qapi-commands: Convert to QAPISchemaVisitor
qapi: Replace dirty is_c_ptr() by method c_null()
qapi: De-duplicate enum code generation
qapi-event: Eliminate global variable event_enum_value
qapi-event: Convert to QAPISchemaVisitor, fixing data with base
qapi: Clean up after recent conversions to QAPISchemaVisitor
qapi-visit: Rearrange code a bit
qapi-commands: Rearrange code
qapi: Rename qmp_marshal_input_FOO() to qmp_marshal_FOO()
qapi: De-duplicate parameter list generation
qapi-commands: De-duplicate output marshaling functions
qapi: Improve built-in type documentation
qapi: Introduce a first class 'any' type
qom: Don't use 'gen': false for qom-get, qom-set, object-add
qapi-schema: Fix up misleading specification of netdev_add
qmp: Improve netdev_add usage example in the manual
qapi: Pseudo-type '**' is now unused, drop it
qapi: New QMP command query-schema for QMP schema introspection
qapi-introspect: Map all integer types to 'int'
qapi-introspect: Hide type names

 .gitignore                                         |   1 +
 Makefile                                           |   9 +-
 Makefile.objs                                      |   4 +-
 docs/qapi-code-gen.txt                             |  53 +-
 docs/writing-qmp-commands.txt                      |   8 +-
 include/monitor/monitor.h                          |   3 -
 include/qapi/visitor-impl.h                        |   2 +
 include/qapi/visitor.h                             |   1 +
 monitor.c                                          |  17 +-
 qapi-schema.json                                   |  25 +-
 qapi/introspect.json                               |  69 +++
 qapi/qapi-dealloc-visitor.c                        |   9 +
 qapi/qapi-visit-core.c                             |   6 +
 qapi/qmp-input-visitor.c                           |  11 +
 qapi/qmp-output-visitor.c                          |   9 +
 qmp-commands.hx                                    | 266 +++++----
 qmp.c                                              |  20 +-
 scripts/qapi-commands.py                           | 355 ++++++-----
 scripts/qapi-event.py                              | 243 +++-----
 scripts/qapi-introspect.py                         | 189 ++++++
 scripts/qapi-types.py                              | 378 +++++-------
 scripts/qapi-visit.py                              | 368 +++++-------
 scripts/qapi.py                                    | 658 +++++++++++++++++----
 tests/.gitignore                                   |   1 +
 tests/Makefile                                     |  15 +-
 tests/qapi-schema/alternate-good.out               |  16 +-
 tests/qapi-schema/args-alternate.err               |   1 +
 ...type-bypass-no-gen.exit => args-alternate.exit} |   0
 tests/qapi-schema/args-alternate.json              |   4 +
 .../{type-bypass-no-gen.out => args-alternate.out} |   0
 tests/qapi-schema/args-returns-any.err             |   1 +
 tests/qapi-schema/args-returns-any.exit            |   1 +
 tests/qapi-schema/args-returns-any.json            |   6 +
 .../{type-bypass.err => args-returns-any.out}      |   0
 tests/qapi-schema/args-union.err                   |   1 +
 tests/qapi-schema/args-union.exit                  |   1 +
 tests/qapi-schema/args-union.json                  |   4 +
 tests/qapi-schema/args-union.out                   |   0
 tests/qapi-schema/comments.out                     |   5 +-
 tests/qapi-schema/data-member-array.out            |  14 +-
 tests/qapi-schema/empty.out                        |   4 +-
 tests/qapi-schema/enum-empty.out                   |   5 +-
 tests/qapi-schema/event-case.out                   |   5 +-
 tests/qapi-schema/flat-union-base-star.err         |   2 +-
 tests/qapi-schema/flat-union-base-star.json        |   2 +-
 tests/qapi-schema/flat-union-reverse-define.out    |  22 +-
 tests/qapi-schema/ident-with-escape.out            |   8 +-
 tests/qapi-schema/include-relpath.out              |   5 +-
 tests/qapi-schema/include-repetition.out           |   5 +-
 tests/qapi-schema/include-simple.out               |   5 +-
 tests/qapi-schema/indented-expr.out                |   8 +-
 tests/qapi-schema/qapi-schema-test.json            |  33 +-
 tests/qapi-schema/qapi-schema-test.out             | 187 ++++--
 tests/qapi-schema/returns-int.out                  |   6 +-
 tests/qapi-schema/test-qapi.py                     |  37 +-
 tests/qapi-schema/type-bypass-no-gen.err           |   1 -
 tests/qapi-schema/type-bypass-no-gen.json          |   2 -
 tests/qapi-schema/type-bypass.exit                 |   1 -
 tests/qapi-schema/type-bypass.json                 |   2 -
 tests/qapi-schema/type-bypass.out                  |   3 -
 tests/test-qmp-event.c                             |   2 +-
 tests/test-qmp-input-strict.c                      |   2 +-
 tests/test-qmp-input-visitor.c                     |  82 ++-
 tests/test-qmp-output-visitor.c                    |  55 +-
 64 files changed, 2011 insertions(+), 1247 deletions(-)
 create mode 100644 qapi/introspect.json
 create mode 100644 scripts/qapi-introspect.py
 create mode 100644 tests/qapi-schema/args-alternate.err
 rename tests/qapi-schema/{type-bypass-no-gen.exit => args-alternate.exit} (100%)
 create mode 100644 tests/qapi-schema/args-alternate.json
 rename tests/qapi-schema/{type-bypass-no-gen.out => args-alternate.out} (100%)
 create mode 100644 tests/qapi-schema/args-returns-any.err
 create mode 100644 tests/qapi-schema/args-returns-any.exit
 create mode 100644 tests/qapi-schema/args-returns-any.json
 rename tests/qapi-schema/{type-bypass.err => args-returns-any.out} (100%)
 create mode 100644 tests/qapi-schema/args-union.err
 create mode 100644 tests/qapi-schema/args-union.exit
 create mode 100644 tests/qapi-schema/args-union.json
 create mode 100644 tests/qapi-schema/args-union.out
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.err
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.json
 delete mode 100644 tests/qapi-schema/type-bypass.exit
 delete mode 100644 tests/qapi-schema/type-bypass.json
 delete mode 100644 tests/qapi-schema/type-bypass.out
Signature · 1 line
1.9.3
#1 lore raw

RFC PATCH v2 1/47 qapi: Clarify docs on including the same file multiple times

It's idempotent.

---
 docs/qapi-code-gen.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
docs/qapi-code-gen.txt +1 -1
index 61b5be4..17588c1 100644
@@ -163,7 +163,7 @@ The QAPI schema definitions can be modularized using the 'include' directive:
 
 The directive is evaluated recursively, and include paths are relative to the
 file using the directive. Multiple includes of the same file are
-safe.  No other keys should appear in the expression, and the include
+idempotent.  No other keys should appear in the expression, and the include
 value should be a string.
 
 As a matter of style, it is a good idea to have all files be
Signature · 1 line
1.9.3
Reviewed-by on v2 1/47
#2 lore raw

Re: [Qemu-devel] [PATCH RFC v2 01/47] qapi: Clarify docs on including the same file multiple times

Quoted 6 lines · Markus Armbruster

It's idempotent.

---
 docs/qapi-code-gen.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#3 lore raw

RFC PATCH v2 2/47 qapi: Clean up cgen() and mcgen()

In reply to Markus Armbruster

Commit 05dfb26 added eatspace stripping to mcgen(). Move it to cgen(), just in case somebody gets tempted to use cgen() directly instead of via mcgen().

cgen() indents blank lines. No such lines get generated right now, but fix it anyway.

We use triple-quoted strings for program text, like this:

    '''
    Program text
    any number of lines
    '''

Keeps the program text relatively readable, but puts an extra newline at either end. mcgen() "fixes" that by dropping the first and last line outright. Drop only the newlines.

This unmasks a bug in qapi-commands.py: four quotes instead of three. Fix it up.

Output doesn't change

---
 scripts/qapi-commands.py |  2 +-
 scripts/qapi.py          | 19 ++++++++++++-------
 2 files changed, 13 insertions(+), 8 deletions(-)
scripts/qapi-commands.py +1 -1
index ca22acc..ce51408 100644
@@ -56,7 +56,7 @@ def gen_sync_call(name, args, ret_type, indent=0):
                 name=c_name(name), args=arglist, retval=retval).rstrip()
     if ret_type:
         ret += "\n" + gen_err_check('local_err')
-        ret += "\n" + mcgen(''''
+        ret += "\n" + mcgen('''
 %(marshal_output_call)s
 ''',
                             marshal_output_call=gen_marshal_output_call(name, ret_type)).rstrip()
scripts/qapi.py +12 -7
index 06d7fc2..20383ef 100644
@@ -943,16 +943,21 @@ def pop_indent(indent_amount=4):
     global indent_level
     indent_level -= indent_amount
 
+# Generate @code with @kwds interpolated.
+# Obey indent_level, and strip eatspace.
 def cgen(code, **kwds):
-    indent = genindent(indent_level)
-    lines = code.split('\n')
-    lines = map(lambda x: indent + x, lines)
-    return '\n'.join(lines) % kwds + '\n'
-
-def mcgen(code, **kwds):
-    raw = cgen('\n'.join(code.split('\n')[1:-1]), **kwds)
+    raw = code % kwds
+    if indent_level:
+        indent = genindent(indent_level)
+        raw = re.subn("^.", indent + '\g<0>', raw, 0, re.MULTILINE)
+        raw = raw[0]
     return re.sub(re.escape(eatspace) + ' *', '', raw)
 
+def mcgen(code, **kwds):
+    if code[0] == '\n':
+        code = code[1:]
+    return cgen(code, **kwds)
+
 def basename(filename):
     return filename.split("/")[-1]
 
Signature · 1 line
1.9.3
Reviewed-by on v2 2/47
#4 lore raw

Re: [Qemu-devel] [PATCH RFC v2 02/47] qapi: Clean up cgen() and mcgen()

Quoted 14 lines · Markus Armbruster

Commit 05dfb26 added eatspace stripping to mcgen(). Move it to cgen(), just in case somebody gets tempted to use cgen() directly instead of via mcgen().

cgen() indents blank lines. No such lines get generated right now, but fix it anyway.

We use triple-quoted strings for program text, like this:

    '''
    Program text
    any number of lines
    '''
Quoted 8 lines

Keeps the program text relatively readable, but puts an extra newline at either end. mcgen() "fixes" that by dropping the first and last line outright. Drop only the newlines.

This unmasks a bug in qapi-commands.py: four quotes instead of three. Fix it up.

Output doesn't change

Confirmed.

Quoted 7 lines
---
 scripts/qapi-commands.py |  2 +-
 scripts/qapi.py          | 19 ++++++++++++-------
 2 files changed, 13 insertions(+), 8 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#5 lore raw

RFC PATCH v2 3/47 qapi: Simplify guardname()

In reply to Markus Armbruster

The guards around built-in declarations lose their _H. It never made much sense anyway.

---
 scripts/qapi.py | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)
scripts/qapi.py +3 -7
index 20383ef..2bbc8ff 100644
@@ -958,14 +958,9 @@ def mcgen(code, **kwds):
         code = code[1:]
     return cgen(code, **kwds)
 
-def basename(filename):
-    return filename.split("/")[-1]
 
 def guardname(filename):
-    guard = basename(filename).rsplit(".", 1)[0]
-    for substr in [".", " ", "-"]:
-        guard = guard.replace(substr, "_")
-    return guard.upper() + '_H'
+    return c_name(filename, protect=False).upper()
 
 def guardstart(name):
     return mcgen('''
@@ -1035,6 +1030,7 @@ def parse_command_line(extra_options = "", extra_long_options = []):
 
 def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
                 c_comment, h_comment):
+    guard = guardname(prefix + h_file)
     c_file = output_dir + prefix + c_file
     h_file = output_dir + prefix + h_file
 
@@ -1067,7 +1063,7 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
 #define %(guard)s
 
 ''',
-                      comment = h_comment, guard = guardname(h_file)))
+                      comment = h_comment, guard = guard))
 
     return (fdef, fdecl)
 
Signature · 1 line
1.9.3
Reviewed-by on v2 3/47
#6 lore raw

Re: [Qemu-devel] [PATCH RFC v2 03/47] qapi: Simplify guardname()

Quoted 2 lines · Markus Armbruster

The guards around built-in declarations lose their _H. It never made much sense anyway.

Correct - the _H tail on the guard for the overall file makes sense (when the overall file ends in .h), but for an unrelated use in the middle of the file, it has no bearing on any file name and does not need a _H tail.

Quoted 5 lines
---
 scripts/qapi.py | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

For an example of what it changes, qapi-types.h sees changes like this (while the original #ifndef QAPI_TYPES_H guard at the front of the file remains unchanged, which is good):

qapi-types.h renamed from qapi-types.h.old +3 -3
@@ -20,8 +20,8 @@
 #include <stdint.h>
 
 
-#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL_H
-#define QAPI_TYPES_BUILTIN_STRUCT_DECL_H
+#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL
+#define QAPI_TYPES_BUILTIN_STRUCT_DECL
 
 
 typedef struct int32List
@@ -141,7 +141,7 @@
     struct uint32List *next;
 } uint32List;
 
-#endif /* QAPI_TYPES_BUILTIN_STRUCT_DECL_H */
+#endif /* QAPI_TYPES_BUILTIN_STRUCT_DECL */
 
 
 extern const char * const ErrorClass_lookup[];

...

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#7 lore raw

RFC PATCH v2 4/47 qapi-event: Clean up how name of enum QAPIEvent is made

In reply to Markus Armbruster

Use c_name() instead of ad hoc code. Doesn't upcase the -p prefix, which is an improvement in my book. Unbreaks prefix containing '.', but other funny characters remain broken. To be fixed next.

---
 scripts/qapi-event.py  | 2 +-
 tests/test-qmp-event.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
scripts/qapi-event.py +1 -1
index 56bc602..cc74f4d 100644
@@ -267,7 +267,7 @@ fdecl.write(mcgen('''
 
 exprs = parse_schema(input_file)
 
-event_enum_name = prefix.upper().replace('-', '_') + "QAPIEvent"
+event_enum_name = c_name(prefix + "QAPIEvent", protect=False)
 event_enum_values = []
 event_enum_strings = []
 
tests/test-qmp-event.c +1 -1
index 1ee40e1..28f146d 100644
@@ -94,7 +94,7 @@ static bool qdict_cmp_simple(QDict *a, QDict *b)
 
 /* This function is hooked as final emit function, which can verify the
    correctness. */
-static void event_test_emit(TEST_QAPIEvent event, QDict *d, Error **errp)
+static void event_test_emit(test_QAPIEvent event, QDict *d, Error **errp)
 {
     QObject *obj;
     QDict *t;
Signature · 1 line
1.9.3
Reviewed-by on v2 4/47
#8 lore raw

Re: [Qemu-devel] [PATCH RFC v2 04/47] qapi-event: Clean up how name of enum QAPIEvent is made

Quoted 9 lines · Markus Armbruster

Use c_name() instead of ad hoc code. Doesn't upcase the -p prefix, which is an improvement in my book. Unbreaks prefix containing '.', but other funny characters remain broken. To be fixed next.

---
 scripts/qapi-event.py  | 2 +-
 tests/test-qmp-event.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

No change to the generated qapi-event.[ch], so it looks like only the testsuite is affected. [In fact, as far as I can tell, only docs/qapi-code-gen.txt and tests/Makefile even take advantage of the '-p prefix' argument.] Fine by me.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#9 lore raw

RFC PATCH v2 5/47 qapi: Reject -p arguments that break qapi-event.py

In reply to Markus Armbruster

qapi-event.py breaks when you ask for a funny prefix like '@'. Protect it.

---
 scripts/qapi.py | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/qapi.py +6 -0
index 2bbc8ff..ea94ce5 100644
@@ -1003,6 +1003,12 @@ def parse_command_line(extra_options = "", extra_long_options = []):
     for oa in opts:
         o, a = oa
         if o in ("-p", "--prefix"):
+            match = re.match('([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)
+            if match.end() != len(a):
+                print >>sys.stderr, \
+                    "%s: 'funny character '%s' in argument of -prefix" \
+                    % (sys.argv[0], a[match.end()])
+                sys.exit(1)
             prefix = a
         elif o in ("-o", "--output-dir"):
             output_dir = a + "/"
Signature · 1 line
1.9.3
Reviewed-by on v2 5/47
#10 lore raw

Re: [Qemu-devel] [PATCH RFC v2 05/47] qapi: Reject -p arguments that break qapi-event.py

Quoted 2 lines · Markus Armbruster

qapi-event.py breaks when you ask for a funny prefix like '@'. Protect it.

Only possible from the command line (not triggered by our makefiles); but doesn't hurt.

Quoted 13 lines
---
 scripts/qapi.py | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/qapi.py +0 -0
index 2bbc8ff..ea94ce5 100644

scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1006

Quoted 4 lines
scripts/qapi.py +1 -0
@@ -1003,6 +1003,12 @@ def parse_command_line(extra_options = "", extra_long_options = []):
     for oa in opts:
         o, a = oa
         if o in ("-p", "--prefix"):
+            match = re.match('([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)

I can understand allowing a leading _, but why bother allowing a leading '.' or '-'? Those will get normalized to _, but in all honesty, no one should ever be doing that.

I'd be just as happy with the shorter:

match = re.match('([A-Za-z_][A-Za-z0-9_.-]*)?', a)

scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1009

Quoted 3 lines
scripts/qapi.py +3 -0
+            if match.end() != len(a):
+                print >>sys.stderr, \
+                    "%s: 'funny character '%s' in argument of -prefix" \

'qemu' is unusual for accepting -single-dash-long-opts; I don't think python getopts does the same by default. Please spell this error message --prefix.

scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1014

Quoted 6 lines
scripts/qapi.py +2 -0
+                    % (sys.argv[0], a[match.end()])
+                sys.exit(1)
             prefix = a
         elif o in ("-o", "--output-dir"):
             output_dir = a + "/"

With the second spelling fix, and optionally with the shorter regex,

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#11 lore raw

Re: [Qemu-devel] [PATCH RFC v2 05/47] qapi: Reject -p arguments that break qapi-event.py

Quoted 2 lines · Eric Blake

'qemu' is unusual for accepting -single-dash-long-opts; I don't think python getopts does the same by default.

Or to rephrase, qemu acts as though it uses GNU getopt_long_only() (some programs like gcc do likewise), but MOST programs that take long opts behave like they use getopt_long(). It is only the getopt_long_only() variant that accepts long options with a single dash, when it is unambiguous with POSIX getopt() parsing of multiple smashed-together short options.

[well, qemu rolls its own option parser instead of using getopt_long_only(), but you get the picture]

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#12 lore raw

Re: [Qemu-devel] [PATCH RFC v2 05/47] qapi: Reject -p arguments that break qapi-event.py

In reply to Eric Blake
Quoted 15 lines · Eric Blake
Quoted 2 lines · Markus Armbruster

qapi-event.py breaks when you ask for a funny prefix like '@'. Protect it.

Only possible from the command line (not triggered by our makefiles); but doesn't hurt.

Quoted 8 lines
---
 scripts/qapi.py | 6 ++++++
 1 file changed, 6 insertions(+)
Quoted 13 lines

scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1006

Quoted 9 lines
scripts/qapi.py +1 -0
index 2bbc8ff..ea94ce5 100644
@@ -1003,6 +1003,12 @@ def parse_command_line(extra_options = "", extra_long_options = []):
     for oa in opts:
         o, a = oa
         if o in ("-p", "--prefix"):
+            match = re.match('([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)

I can understand allowing a leading _, but why bother allowing a leading '.' or '-'? Those will get normalized to _, but in all honesty, no one should ever be doing that.

My patch rejects exactly the prefixes that won't work.

Quoted 3 lines

I'd be just as happy with the shorter:

match = re.match('([A-Za-z_][A-Za-z0-9_.-]*)?', a)

This additionally rejects a few rather foolish ones.

I have a slight preference for the tool staying out of policing foolish prefixes.

Quoted 7 lines

scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1009

Quoted 3 lines
scripts/qapi.py +3 -0
+            if match.end() != len(a):
+                print >>sys.stderr, \
+                    "%s: 'funny character '%s' in argument of -prefix" \

'qemu' is unusual for accepting -single-dash-long-opts; I don't think python getopts does the same by default. Please spell this error message --prefix.

Typo, will fix.

Quoted 10 lines

scripts/qapi.py def parse_command_line(extra_options = "", extra_long_options = []): · line 1014

Quoted 6 lines
scripts/qapi.py +2 -0
+                    % (sys.argv[0], a[match.end()])
+                sys.exit(1)
             prefix = a
         elif o in ("-o", "--output-dir"):
             output_dir = a + "/"

With the second spelling fix, and optionally with the shorter regex,

Thanks!

#13 lore raw

RFC PATCH v2 6/47 qapi: Drop unused and useless parameters and variables

In reply to Markus Armbruster

gen_sync_call()'s parameter indent is useless: gen_sync_call() uses it only as optional argument for push_indent() and pop_indent(), their default is four, and gen_sync_call()'s only caller passes four.

gen_visitor_input_containers_decl()'s parameter obj is always "QOBJECT(args)".

---
 scripts/qapi-commands.py | 27 +++++++++++++--------------
 scripts/qapi-event.py    |  1 -
 scripts/qapi-types.py    |  1 -
 scripts/qapi-visit.py    | 47 ++++++++++++++++++++++-------------------------
 scripts/qapi.py          |  1 -
 5 files changed, 35 insertions(+), 42 deletions(-)
scripts/qapi-commands.py +13 -14
index ce51408..69029f5 100644
@@ -38,7 +38,7 @@ if (local_err) {
 ''')
     return ''
 
-def gen_sync_call(name, args, ret_type, indent=0):
+def gen_sync_call(name, args, ret_type):
     ret = ""
     arglist=""
     retval=""
@@ -48,7 +48,7 @@ def gen_sync_call(name, args, ret_type, indent=0):
         if optional:
             arglist += "has_%s, " % c_name(argname)
         arglist += "%s, " % (c_name(argname))
-    push_indent(indent)
+    push_indent()
     ret = mcgen('''
 %(retval)sqmp_%(name)s(%(args)s&local_err);
 
@@ -60,7 +60,7 @@ def gen_sync_call(name, args, ret_type, indent=0):
 %(marshal_output_call)s
 ''',
                             marshal_output_call=gen_marshal_output_call(name, ret_type)).rstrip()
-    pop_indent(indent)
+    pop_indent()
     return ret.rstrip()
 
 
@@ -69,17 +69,16 @@ def gen_marshal_output_call(name, ret_type):
         return ""
     return "qmp_marshal_output_%s(retval, ret, &local_err);" % c_name(name)
 
-def gen_visitor_input_containers_decl(args, obj):
+def gen_visitor_input_containers_decl(args):
     ret = ""
 
     push_indent()
     if len(args) > 0:
         ret += mcgen('''
-QmpInputVisitor *mi = qmp_input_visitor_new_strict(%(obj)s);
+QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
 QapiDeallocVisitor *md;
 Visitor *v;
-''',
-                     obj=obj)
+''')
     pop_indent()
 
     return ret.rstrip()
@@ -161,7 +160,7 @@ qapi_dealloc_visitor_cleanup(md);
     pop_indent()
     return ret.rstrip()
 
-def gen_marshal_output(name, args, ret_type, middle_mode):
+def gen_marshal_output(name, ret_type):
     if not ret_type:
         return ""
 
@@ -194,14 +193,14 @@ out:
 
     return ret
 
-def gen_marshal_input_decl(name, args, ret_type, middle_mode):
+def gen_marshal_input_decl(name, middle_mode):
     ret = 'void qmp_marshal_input_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
     if not middle_mode:
         ret = "static " + ret
     return ret
 
 def gen_marshal_input(name, args, ret_type, middle_mode):
-    hdr = gen_marshal_input_decl(name, args, ret_type, middle_mode)
+    hdr = gen_marshal_input_decl(name, middle_mode)
 
     ret = mcgen('''
 %(header)s
@@ -228,7 +227,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
 %(visitor_input_block)s
 
 ''',
-                     visitor_input_containers_decl=gen_visitor_input_containers_decl(args, "QOBJECT(args)"),
+                     visitor_input_containers_decl=gen_visitor_input_containers_decl(args),
                      visitor_input_vars_decl=gen_visitor_input_vars_decl(args),
                      visitor_input_block=gen_visitor_input_block(args))
     else:
@@ -240,7 +239,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
     ret += mcgen('''
 %(sync_call)s
 ''',
-                 sync_call=gen_sync_call(name, args, ret_type, indent=4))
+                 sync_call=gen_sync_call(name, args, ret_type))
     if re.search('^ *goto out\\;', ret, re.MULTILINE):
         ret += mcgen('''
 
@@ -360,11 +359,11 @@ for cmd in commands:
     ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n"
     fdecl.write(ret)
     if ret_type:
-        ret = gen_marshal_output(cmd['command'], arglist, ret_type, middle_mode) + "\n"
+        ret = gen_marshal_output(cmd['command'], ret_type) + "\n"
         fdef.write(ret)
 
     if middle_mode:
-        fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], arglist, ret_type, middle_mode))
+        fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], middle_mode))
 
     ret = gen_marshal_input(cmd['command'], arglist, ret_type, middle_mode) + "\n"
     fdef.write(ret)
scripts/qapi-event.py +0 -1
index cc74f4d..88b0620 100644
@@ -199,7 +199,6 @@ const char *%(event_enum_name)s_lookup[] = {
 ''',
                 event_enum_name = event_enum_name)
 
-    i = 0
     for string in event_enum_strings:
         ret += mcgen('''
     "%(string)s",
scripts/qapi-types.py +0 -1
index e6eb4b6..4902440 100644
@@ -108,7 +108,6 @@ def generate_enum_lookup(name, values):
 const char * const %(name)s_lookup[] = {
 ''',
                 name=c_name(name))
-    i = 0
     for value in values:
         index = c_enum_const(name, value)
         ret += mcgen('''
scripts/qapi-visit.py +22 -25

136 lines not on this page · raw message

scripts/qapi.py +0 -1
index ea94ce5..99e04f6 100644
@@ -507,7 +507,6 @@ def check_command(expr, expr_info):
 def check_event(expr, expr_info):
     global events
     name = expr['event']
-    params = expr.get('data')
 
     if name.upper() == 'MAX':
         raise QAPIExprError(expr_info, "Event name 'MAX' cannot be created")
Signature · 1 line
1.9.3
Reviewed-by on v2 6/47
#14 lore raw

Re: [Qemu-devel] [PATCH RFC v2 06/47] qapi: Drop unused and useless parameters and variables

Quoted 6 lines · Markus Armbruster

gen_sync_call()'s parameter indent is useless: gen_sync_call() uses it only as optional argument for push_indent() and pop_indent(), their default is four, and gen_sync_call()'s only caller passes four.

gen_visitor_input_containers_decl()'s parameter obj is always "QOBJECT(args)".

It might be nice to call out that several other functions are also stripped of unused arguments. I was assuming that only two functions (and their callers) would be modified, but the patch touched more:

Quoted 11 lines
---
 scripts/qapi-commands.py | 27 +++++++++++++--------------
 scripts/qapi-event.py    |  1 -
 scripts/qapi-types.py    |  1 -
 scripts/qapi-visit.py    | 47 ++++++++++++++++++++++-------------------------
 scripts/qapi.py          |  1 -
 5 files changed, 35 insertions(+), 42 deletions(-)

scripts/qapi-commands.py qapi_dealloc_visitor_cleanup(md); · line 165

Quoted 7 lines
scripts/qapi-commands.py +1 -1
@@ -161,7 +160,7 @@ qapi_dealloc_visitor_cleanup(md);
     pop_indent()
     return ret.rstrip()
 
-def gen_marshal_output(name, args, ret_type, middle_mode):
+def gen_marshal_output(name, ret_type):
     if not ret_type:
         return ""

For example, gen_marshal_output() was not mentioned in the commit message.

scripts/qapi-commands.py out: · line 196

Quoted 7 lines
scripts/qapi-commands.py +1 -1
@@ -194,14 +193,14 @@ out:
 
     return ret
 
-def gen_marshal_input_decl(name, args, ret_type, middle_mode):
+def gen_marshal_input_decl(name, middle_mode):

Or gen_marshal_input_decl()

scripts/qapi-event.py const char *%(event_enum_name)s_lookup[] = { · line 202

Quoted 7 lines
scripts/qapi-event.py +1 -1
+++ b/scripts/qapi-event.py
@@ -199,7 +199,6 @@ const char *%(event_enum_name)s_lookup[] = {
 ''',
                 event_enum_name = event_enum_name)
 
-    i = 0
     for string in event_enum_strings:

I guess the subject line covered deletion of unused internal variables.

Quoted 8 lines
scripts/qapi-visit.py +2 -1
+++ b/scripts/qapi-visit.py
 
@@ -441,44 +440,42 @@ fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
 
     elif expr.has_key('union'):
         ret = generate_visit_union(expr)
-        ret += generate_visit_list(expr['union'], expr['data'])
+        ret += generate_visit_list(expr['union'])

scripts/qapi-visit.py fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) · line 450

Quoted 10 lines
scripts/qapi-visit.py +2 -3
         fdef.write(ret)
 
         enum_define = discriminator_find_enum_define(expr)
         ret = ""
         if not enum_define:
-            ret = generate_decl_enum('%sKind' % expr['union'],
-                                     expr['data'].keys())
-        ret += generate_declaration(expr['union'], expr['data'])
+            ret = generate_decl_enum('%sKind' % expr['union'])
+        ret += generate_declaration(expr['union'])

As long as you are touching this, generate_decl_enum(expr['union'] + 'Kind') would read nicer.

scripts/qapi-visit.py fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) · line 469

Quoted 11 lines
scripts/qapi-visit.py +2 -4
         fdecl.write(ret)
     elif expr.has_key('alternate'):
         ret = generate_visit_alternate(expr['alternate'], expr['data'])
-        ret += generate_visit_list(expr['alternate'], expr['data'])
+        ret += generate_visit_list(expr['alternate'])
         fdef.write(ret)
 
-        ret = generate_decl_enum('%sKind' % expr['alternate'],
-                                 expr['data'].keys())
-        ret += generate_declaration(expr['alternate'], expr['data'])
+        ret = generate_decl_enum('%sKind' % expr['alternate'])

Same here.

At any rate, no change to generated output. So assuming you are happy with the commit message, or take my advice to improve it, the code cleanup itself is fine.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#15 lore raw

Re: [Qemu-devel] [PATCH RFC v2 06/47] qapi: Drop unused and useless parameters and variables

Quoted 11 lines · Eric Blake
Quoted 6 lines · Markus Armbruster

gen_sync_call()'s parameter indent is useless: gen_sync_call() uses it only as optional argument for push_indent() and pop_indent(), their default is four, and gen_sync_call()'s only caller passes four.

gen_visitor_input_containers_decl()'s parameter obj is always "QOBJECT(args)".

It might be nice to call out that several other functions are also stripped of unused arguments. I was assuming that only two functions (and their callers) would be modified, but the patch touched more:

I'll rephrase the commit message to make this more obvious.

Quoted 59 lines
Quoted 10 lines
---
 scripts/qapi-commands.py | 27 +++++++++++++--------------
 scripts/qapi-event.py    |  1 -
 scripts/qapi-types.py    |  1 -
 scripts/qapi-visit.py    | 47 ++++++++++++++++++++++-------------------------
 scripts/qapi.py          |  1 -
 5 files changed, 35 insertions(+), 42 deletions(-)

scripts/qapi-commands.py qapi_dealloc_visitor_cleanup(md); · line 165

Quoted 7 lines
scripts/qapi-commands.py +1 -1
@@ -161,7 +160,7 @@ qapi_dealloc_visitor_cleanup(md);
     pop_indent()
     return ret.rstrip()
 
-def gen_marshal_output(name, args, ret_type, middle_mode):
+def gen_marshal_output(name, ret_type):
     if not ret_type:
         return ""

For example, gen_marshal_output() was not mentioned in the commit message.

scripts/qapi-commands.py out: · line 196

Quoted 7 lines
scripts/qapi-commands.py +1 -1
@@ -194,14 +193,14 @@ out:
 
     return ret
 
-def gen_marshal_input_decl(name, args, ret_type, middle_mode):
+def gen_marshal_input_decl(name, middle_mode):

Or gen_marshal_input_decl()

scripts/qapi-event.py const char *%(event_enum_name)s_lookup[] = { · line 202

Quoted 7 lines
scripts/qapi-event.py +1 -1
+++ b/scripts/qapi-event.py
@@ -199,7 +199,6 @@ const char *%(event_enum_name)s_lookup[] = {
 ''',
                 event_enum_name = event_enum_name)
 
-    i = 0
     for string in event_enum_strings:

I guess the subject line covered deletion of unused internal variables.

Quoted 8 lines
scripts/qapi-visit.py +2 -1
+++ b/scripts/qapi-visit.py
 
@@ -441,44 +440,42 @@ fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
 
     elif expr.has_key('union'):
         ret = generate_visit_union(expr)
-        ret += generate_visit_list(expr['union'], expr['data'])
+        ret += generate_visit_list(expr['union'])

scripts/qapi-visit.py fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) · line 450

Quoted 10 lines
scripts/qapi-visit.py +2 -3
         fdef.write(ret)
 
         enum_define = discriminator_find_enum_define(expr)
         ret = ""
         if not enum_define:
-            ret = generate_decl_enum('%sKind' % expr['union'],
-                                     expr['data'].keys())
-        ret += generate_declaration(expr['union'], expr['data'])
+            ret = generate_decl_enum('%sKind' % expr['union'])
+        ret += generate_declaration(expr['union'])
Quoted 2 lines

As long as you are touching this, generate_decl_enum(expr['union'] + 'Kind') would read nicer.

It'll go away in PATCH 27. Let's keep this patch simple.

Quoted 7 lines
Quoted 7 lines
scripts/qapi-visit.py +1 -1
         fdecl.write(ret)
     elif expr.has_key('alternate'):
         ret = generate_visit_alternate(expr['alternate'], expr['data'])
-        ret += generate_visit_list(expr['alternate'], expr['data'])
+        ret += generate_visit_list(expr['alternate'])
         fdef.write(ret)
 
Quoted 6 lines

scripts/qapi-visit.py fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL")) · line 469

Quoted 4 lines
scripts/qapi-visit.py +1 -3
-        ret = generate_decl_enum('%sKind' % expr['alternate'],
-                                 expr['data'].keys())
-        ret += generate_declaration(expr['alternate'], expr['data'])
+        ret = generate_decl_enum('%sKind' % expr['alternate'])

Same here.

Likewise.

Quoted 5 lines

At any rate, no change to generated output. So assuming you are happy with the commit message, or take my advice to improve it, the code cleanup itself is fine.

Thanks!

#16 lore raw

RFC PATCH v2 7/47 qapi: Generate a nicer struct for flat unions

In reply to Markus Armbruster

The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.

Example: qapi-schema-test.json has

    { 'struct': 'UserDefUnionBase',
      'data': { 'string': 'str', 'enum1': 'EnumOne' } }

    { 'union': 'UserDefFlatUnion',
      'base': 'UserDefUnionBase',
      'discriminator': 'enum1',
      'data': { 'value1' : 'UserDefA',
                'value2' : 'UserDefB',
                'value3' : 'UserDefB' } }

We generate:

    struct UserDefFlatUnion
    {
        EnumOne kind;
        union {
            void *data;
            UserDefA *value1;
            UserDefB *value2;
            UserDefB *value3;
        };
        char *string;
    };

Change to put all base members at the beginning, unadulterated. Not only is this easier to understand, it also permits casting the flat union to its base, if that should become useful.

We now generate:

    struct UserDefFlatUnion
    {
        char *string;
        EnumOne enum1;
        /* union tag is EnumOne enum1 */
        union {
            void *data;
            UserDefA *value1;
            UserDefB *value2;
            UserDefB *value3;
        };
    };
---
 scripts/qapi-types.py           | 32 ++++++++++++++++++--------------
 scripts/qapi-visit.py           |  7 +++++--
 tests/test-qmp-input-visitor.c  |  2 +-
 tests/test-qmp-output-visitor.c |  2 +-
 4 files changed, 25 insertions(+), 18 deletions(-)
scripts/qapi-types.py +18 -14
index 4902440..c6c2786 100644
@@ -200,12 +200,27 @@ def generate_union(expr, meta):
     ret = mcgen('''
 struct %(name)s
 {
+''',
+                name=name)
+    if base:
+        base_fields = find_struct(base)['data']
+        ret += generate_struct_fields(base_fields)
+        ret += mcgen('''
+    /* union tag is %(discriminator_type_name)s %(c_name)s */
+''',
+                     discriminator_type_name=c_name(discriminator_type_name),
+                     c_name=c_name(discriminator))
+    else:
+        assert not discriminator
+        ret += mcgen('''
     %(discriminator_type_name)s kind;
+''',
+                     discriminator_type_name=c_name(discriminator_type_name))
+
+    ret += mcgen('''
     union {
         void *data;
-''',
-                name=name,
-                discriminator_type_name=c_name(discriminator_type_name))
+''')
 
     for key in typeinfo:
         ret += mcgen('''
@@ -216,17 +231,6 @@ struct %(name)s
 
     ret += mcgen('''
     };
-''')
-
-    if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_struct_fields(base_fields)
-    else:
-        assert not discriminator
-
-    ret += mcgen('''
 };
 ''')
     if meta == 'alternate':
scripts/qapi-visit.py +5 -2
index e8ee268..4ec79a6 100644
@@ -288,20 +288,23 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
                      name=c_name(name))
 
     if not discriminator:
+        tag = 'kind'
         disc_key = "type"
     else:
+        tag = discriminator
         disc_key = discriminator
     ret += mcgen('''
-        visit_type_%(disc_type)s(m, &(*obj)->kind, "%(disc_key)s", &err);
+        visit_type_%(disc_type)s(m, &(*obj)->%(c_tag)s, "%(disc_key)s", &err);
         if (err) {
             goto out_obj;
         }
         if (!visit_start_union(m, !!(*obj)->data, &err) || err) {
             goto out_obj;
         }
-        switch ((*obj)->kind) {
+        switch ((*obj)->%(c_tag)s) {
 ''',
                  disc_type = disc_type,
+                 c_tag=c_name(tag),
                  disc_key = disc_key)
 
     for key in members:
tests/test-qmp-input-visitor.c +1 -1
index b961953..b7a87ee 100644
@@ -313,7 +313,7 @@ static void test_visitor_in_union_flat(TestInputVisitorData *data,
 
     visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
     g_assert(err == NULL);
-    g_assert_cmpint(tmp->kind, ==, ENUM_ONE_VALUE1);
+    g_assert_cmpint(tmp->enum1, ==, ENUM_ONE_VALUE1);
     g_assert_cmpstr(tmp->string, ==, "str");
     /* TODO g_assert_cmpint(tmp->integer, ==, 41); */
     g_assert_cmpint(tmp->value1->boolean, ==, true);
tests/test-qmp-output-visitor.c +1 -1
index 87ba350..338ada0 100644
@@ -437,7 +437,7 @@ static void test_visitor_out_union_flat(TestOutputVisitorData *data,
     Error *err = NULL;
 
     UserDefFlatUnion *tmp = g_malloc0(sizeof(UserDefFlatUnion));
-    tmp->kind = ENUM_ONE_VALUE1;
+    tmp->enum1 = ENUM_ONE_VALUE1;
     tmp->string = g_strdup("str");
     tmp->value1 = g_malloc0(sizeof(UserDefA));
     /* TODO when generator bug is fixed: tmp->integer = 41; */
Signature · 1 line
1.9.3
Reviewed-by on v2 7/47
#17 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

Quoted 5 lines · Markus Armbruster

The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.

Quoted 8 lines

Change to put all base members at the beginning, unadulterated. Not only is this easier to understand, it also permits casting the flat union to its base, if that should become useful.

We now generate:

    struct UserDefFlatUnion
    {

It might be worth tweaking the generator to output a C comment either here (at the start of the larger struct)...

Quoted 2 lines
        char *string;
        EnumOne enum1;

...or here (at the end of the base struct) mentioning that UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to make it easier when reading the generated code to trace back to that "inheritance" relationship. Right now, there is nothing in the generated UserDefFlatUnion that points you back to the qapi relationship of a base class. But it's not a show-stopper if you don't like my suggestion.

Quoted 8 lines
        /* union tag is EnumOne enum1 */
        union {
            void *data;
            UserDefA *value1;
            UserDefB *value2;
            UserDefB *value3;
        };
    };

Only impact in files generated for qemu was to struct BlockdevOptions, in the files qapi-types.[ch], and I agree that it is an improvement. Oddly enough, it seems none of the C code cared about the field being renamed from 'kind' to 'driver' (I guess that's because a lot of our use of the blockdev code is not directly through the generated C structs, but through QObject dictionary queries).

Quoted 11 lines
---
 scripts/qapi-types.py           | 32 ++++++++++++++++++--------------
 scripts/qapi-visit.py           |  7 +++++--
 tests/test-qmp-input-visitor.c  |  2 +-
 tests/test-qmp-output-visitor.c |  2 +-
 4 files changed, 25 insertions(+), 18 deletions(-)

+''',

scripts/qapi-types.py def generate_union(expr, meta):

Quoted 10 lines
scripts/qapi-types.py +4 -5
+                name=name)
+    if base:
+        base_fields = find_struct(base)['data']
+        ret += generate_struct_fields(base_fields)
 
-    if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_struct_fields(base_fields)

I also like the fact that you no longer modify the base_fields array (because you are no longer floating a single renamed element out-of-order compared to the rest of the base), and therefore avoid the need for a .copy().

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#18 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

Quoted 21 lines · Eric Blake
Quoted 5 lines · Markus Armbruster

The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.

Quoted 8 lines

Change to put all base members at the beginning, unadulterated. Not only is this easier to understand, it also permits casting the flat union to its base, if that should become useful.

We now generate:

    struct UserDefFlatUnion
    {

It might be worth tweaking the generator to output a C comment either here (at the start of the larger struct)...

Quoted 2 lines
        char *string;
        EnumOne enum1;
Quoted 7 lines

...or here (at the end of the base struct) mentioning that UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to make it easier when reading the generated code to trace back to that "inheritance" relationship. Right now, there is nothing in the generated UserDefFlatUnion that points you back to the qapi relationship of a base class. But it's not a show-stopper if you don't like my suggestion.

I do like it. Perhaps something like

    struct UserDefFlatUnion
    {
        /* Members inherited from UserDefUnionBase: */
        char *string;
        EnumOne enum1;
        /* Own members: */
        ...
    };
Quoted 9 lines
Quoted 8 lines
        /* union tag is EnumOne enum1 */
        union {
            void *data;
            UserDefA *value1;
            UserDefB *value2;
            UserDefB *value3;
        };
    };
Quoted 6 lines

Only impact in files generated for qemu was to struct BlockdevOptions, in the files qapi-types.[ch], and I agree that it is an improvement. Oddly enough, it seems none of the C code cared about the field being renamed from 'kind' to 'driver' (I guess that's because a lot of our use of the blockdev code is not directly through the generated C structs, but through QObject dictionary queries).

It surprised me, too.

Quoted 21 lines
Quoted 10 lines
---
 scripts/qapi-types.py           | 32 ++++++++++++++++++--------------
 scripts/qapi-visit.py           |  7 +++++--
 tests/test-qmp-input-visitor.c  |  2 +-
 tests/test-qmp-output-visitor.c |  2 +-
 4 files changed, 25 insertions(+), 18 deletions(-)

+''',

scripts/qapi-types.py def generate_union(expr, meta):

Quoted 10 lines
scripts/qapi-types.py +4 -5
+                name=name)
+    if base:
+        base_fields = find_struct(base)['data']
+        ret += generate_struct_fields(base_fields)
 
-    if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_struct_fields(base_fields)
Quoted 6 lines

I also like the fact that you no longer modify the base_fields array (because you are no longer floating a single renamed element out-of-order compared to the rest of the base), and therefore avoid the need for a .copy().

Thanks!

#19 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

Quoted 16 lines · Markus Armbruster
Quoted 6 lines

It might be worth tweaking the generator to output a C comment either here (at the start of the larger struct)...

Quoted 2 lines
        char *string;
        EnumOne enum1;
Quoted 7 lines

...or here (at the end of the base struct) mentioning that UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to make it easier when reading the generated code to trace back to that "inheritance" relationship. Right now, there is nothing in the generated UserDefFlatUnion that points you back to the qapi relationship of a base class. But it's not a show-stopper if you don't like my suggestion.

I do like it. Perhaps something like

Quoted 8 lines
    struct UserDefFlatUnion
    {
        /* Members inherited from UserDefUnionBase: */
        char *string;
        EnumOne enum1;
        /* Own members: */
        ...
    };

What about nested types? Whatever we pick has to look good when indirect bases are involved. Suppose we have Base -> Mid -> Derived as a struct inheritance. If we generate a comment at both start and end of visiting a base class (and visit base classes recursively), we'd have something like:

struct Derived

{
    /* Members inherited from Mid */
    /* Members inherited from Base */
    int one;
    /* End members inherited from Base */
    int two;
    /* End members inherited from Mid */
    /* own members */
    int three;
    ...
};

Don't know if that helps you think about it more, but now I'm just painting a bikeshed that just adds the tail comment. Here's the resulting RFC:

http://thread.gmane.org/gmane.comp.emulators.qemu/353204

and the resulting generated structs:

struct BlockdevOptionsQcow2 {

-    BlockdevOptionsGenericCOWFormat *base;
+    BlockdevRef *file;
+    /* End fields inherited from BlockdevOptionsGenericFormat. */
+    bool has_backing;
+    BlockdevRef *backing;
+    /* End fields inherited from BlockdevOptionsGenericCOWFormat. */
     bool has_lazy_refcounts;
     bool lazy_refcounts;

...

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#20 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

Quoted 34 lines · Eric Blake
Quoted 16 lines · Markus Armbruster
Quoted 6 lines

It might be worth tweaking the generator to output a C comment either here (at the start of the larger struct)...

Quoted 2 lines
        char *string;
        EnumOne enum1;
Quoted 7 lines

...or here (at the end of the base struct) mentioning that UserDefFlatUnion can be cast into its base struct UserDefUnionBase, to make it easier when reading the generated code to trace back to that "inheritance" relationship. Right now, there is nothing in the generated UserDefFlatUnion that points you back to the qapi relationship of a base class. But it's not a show-stopper if you don't like my suggestion.

I do like it. Perhaps something like

Quoted 8 lines
    struct UserDefFlatUnion
    {
        /* Members inherited from UserDefUnionBase: */
        char *string;
        EnumOne enum1;
        /* Own members: */
        ...
    };

What about nested types? Whatever we pick has to look good when indirect bases are involved. Suppose we have Base -> Mid -> Derived as a struct inheritance. If we generate a comment at both start and end of visiting a base class (and visit base classes recursively), we'd have something like:

struct Derived

Quoted 11 lines
{
    /* Members inherited from Mid */
    /* Members inherited from Base */
    int one;
    /* End members inherited from Base */
    int two;
    /* End members inherited from Mid */
    /* own members */
    int three;
    ...
};

Documenting the direct ancestor in the code is more important than the others, because you can always peruse the direct ancestor to find the next one.

Quoted 9 lines

Don't know if that helps you think about it more, but now I'm just painting a bikeshed that just adds the tail comment. Here's the resulting RFC:

http://thread.gmane.org/gmane.comp.emulators.qemu/353204

and the resulting generated structs:

struct BlockdevOptionsQcow2 {

Review

Quoted 9 lines
(quoted diff) +5 -1
-    BlockdevOptionsGenericCOWFormat *base;
+    BlockdevRef *file;
+    /* End fields inherited from BlockdevOptionsGenericFormat. */
+    bool has_backing;
+    BlockdevRef *backing;
+    /* End fields inherited from BlockdevOptionsGenericCOWFormat. */
     bool has_lazy_refcounts;
     bool lazy_refcounts;
...

This and other followup patches are encouraging. I'll gladly review them when I can spare the cycles. Right now, I need to focus them on introspection.

#21 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

In reply to Markus Armbruster
Quoted 3 lines · Markus Armbruster

The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.

The renaming to 'kind' was a bug waiting to happen. Consider this example, which is broken before your series:

(quoted diff) +0 -0

w/tests/qapi-schema/qapi-schema-test.json
index c7eaa86..12c09e3 100644

tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -37,7 +37,7 @@
   'data': { 'string1': 'str', 'string2': 'str' } }
 
 { 'struct': 'UserDefUnionBase',
-  'data': { 'string': 'str', 'enum1': 'EnumOne' } }
+  'data': { 'kind': 'str', 'enum1': 'EnumOne' } }
 
 { 'union': 'UserDefFlatUnion',
   'base': 'UserDefUnionBase',

leading to this compilation error during 'make check-unit':

In file included from tests/test-qmp-output-visitor.c:17:0:
tests/test-qapi-types.h:617:11: error: duplicate member ‘kind’

     char *kind;
           ^
tests/test-qapi-types.h:631:11: error: duplicate member ‘kind’
     char *kind;
           ^

Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#22 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

Quoted 28 lines · Eric Blake
Quoted 3 lines · Markus Armbruster

The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.

The renaming to 'kind' was a bug waiting to happen. Consider this example, which is broken before your series:

(quoted diff) +0 -0

w/tests/qapi-schema/qapi-schema-test.json
index c7eaa86..12c09e3 100644

tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -37,7 +37,7 @@
   'data': { 'string1': 'str', 'string2': 'str' } }
 
 { 'struct': 'UserDefUnionBase',
-  'data': { 'string': 'str', 'enum1': 'EnumOne' } }
+  'data': { 'kind': 'str', 'enum1': 'EnumOne' } }
 
 { 'union': 'UserDefFlatUnion',
   'base': 'UserDefUnionBase',

leading to this compilation error during 'make check-unit':

In file included from tests/test-qmp-output-visitor.c:17:0:
tests/test-qapi-types.h:617:11: error: duplicate member ‘kind’

Quoted 8 lines
     char *kind;
           ^
tests/test-qapi-types.h:631:11: error: duplicate member ‘kind’
     char *kind;
           ^

Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)

Cool! I'll work (a variation of) this test case into my series.

#23 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

Quoted 8 lines · Markus Armbruster
Quoted 5 lines · Eric Blake
Quoted 3 lines · Markus Armbruster

The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.

Quoted 4 lines
Quoted 2 lines

Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)

Cool! I'll work (a variation of) this test case into my series.

Another name collision bug: our code generates flat unions as:

struct BlockdevOptions {
    BlockdevDriver driver;
...
    /* End fields inherited from BlockdevOptionsBase. */
    /* union tag is BlockdevDriver driver */
    union {
        void *data;
        BlockdevOptionsArchipelago *archipelago;

...

which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '_data'.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#24 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

Quoted 24 lines · Eric Blake
Quoted 8 lines · Markus Armbruster
Quoted 5 lines · Eric Blake
Quoted 3 lines · Markus Armbruster

The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.

Quoted 4 lines
Quoted 2 lines

Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)

Cool! I'll work (a variation of) this test case into my series.

Another name collision bug: our code generates flat unions as:

struct BlockdevOptions {
    BlockdevDriver driver;
...
    /* End fields inherited from BlockdevOptionsBase. */
    /* union tag is BlockdevDriver driver */
    union {
        void *data;
        BlockdevOptionsArchipelago *archipelago;
Quoted 5 lines

...

which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '_data'.

I wonder whether member data is actually used. I'll find out.

#25 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

Quoted 12 lines · Markus Armbruster
Quoted 12 lines

Another name collision bug: our code generates flat unions as:

struct BlockdevOptions {
    BlockdevDriver driver;
...
    /* End fields inherited from BlockdevOptionsBase. */
    /* union tag is BlockdevDriver driver */
    union {
        void *data;
        BlockdevOptionsArchipelago *archipelago;

...

Quoted 5 lines
Quoted 3 lines

which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '_data'.

I wonder whether member data is actually used. I'll find out.

The dealloc visitor uses 'data' being non-null as a flag on whether to deallocate the union even if the tag was invalid for some reason; or more importantly, if parsing consumed the tag but then detected an error while parsing the union, leaving the union branch partially allocated. To avoid a leak, we have to deallocate the branch.

But if the tag was invalid, then why did we ever allocate the union in the first place, and how do we prove we are calling the correct free-ing function? And if the tag is valid, why can't we just guarantee that the union is 0-initialized and that deleting the branch will work through the correct branch type instead of worrying about 'data'?

We still need a dummy member if it is valid to do { 'union':'Foo', 'data':{} } since C doesn't like empty unions, but an empty union seems like something we may want to reject, at which point you are probably right that deleting the data member altogether should work and still let us recover from bad partial parses without a leak.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#26 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

Quoted 26 lines · Eric Blake
Quoted 12 lines · Markus Armbruster
Quoted 12 lines

Another name collision bug: our code generates flat unions as:

struct BlockdevOptions {
    BlockdevDriver driver;
...
    /* End fields inherited from BlockdevOptionsBase. */
    /* union tag is BlockdevDriver driver */
    union {
        void *data;
        BlockdevOptionsArchipelago *archipelago;

...

Quoted 5 lines
Quoted 3 lines

which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '_data'.

I wonder whether member data is actually used. I'll find out.

The dealloc visitor uses 'data' being non-null as a flag on whether to deallocate the union even if the tag was invalid for some reason; or more importantly, if parsing consumed the tag but then detected an error while parsing the union, leaving the union branch partially allocated. To avoid a leak, we have to deallocate the branch.

Quoted 5 lines

But if the tag was invalid, then why did we ever allocate the union in the first place, and how do we prove we are calling the correct free-ing function? And if the tag is valid, why can't we just guarantee that the union is 0-initialized and that deleting the branch will work through the correct branch type instead of worrying about 'data'?

Good questions. Someone will have to review and fix this code. Let's add a FIXME so we don't forget. Care to propose one?

Quoted 5 lines

We still need a dummy member if it is valid to do { 'union':'Foo', 'data':{} } since C doesn't like empty unions, but an empty union seems like something we may want to reject, at which point you are probably right that deleting the data member altogether should work and still let us recover from bad partial parses without a leak.

Either we reject empty unions, or we detect them and add a dummy, similar to what we do for structs since commit 83ecb22. I figure simply rejecting them is easier.

#27 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

Quoted 12 lines · Markus Armbruster
Quoted 12 lines
Quoted 5 lines
Quoted 3 lines

which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '_data'.

I wonder whether member data is actually used. I'll find out.

The dealloc visitor uses 'data' being non-null as a flag on whether to deallocate the union even if the tag was invalid for some reason; or more importantly, if parsing consumed the tag but then detected an error while parsing the union, leaving the union branch partially allocated. To avoid a leak, we have to deallocate the branch.

Quoted 8 lines
Quoted 5 lines

But if the tag was invalid, then why did we ever allocate the union in the first place, and how do we prove we are calling the correct free-ing function? And if the tag is valid, why can't we just guarantee that the union is 0-initialized and that deleting the branch will work through the correct branch type instead of worrying about 'data'?

Good questions. Someone will have to review and fix this code. Let's add a FIXME so we don't forget. Care to propose one?

Sure; see 12.6/47 (since that is close to several other patches adding FIXME comments).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#28 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

In reply to Eric Blake
Quoted 24 lines · Eric Blake
Quoted 8 lines · Markus Armbruster
Quoted 5 lines · Eric Blake
Quoted 3 lines · Markus Armbruster

The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.

Quoted 4 lines
Quoted 2 lines

Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)

Cool! I'll work (a variation of) this test case into my series.

Another name collision bug: our code generates flat unions as:

struct BlockdevOptions {
    BlockdevDriver driver;
...
    /* End fields inherited from BlockdevOptionsBase. */
    /* union tag is BlockdevDriver driver */
    union {
        void *data;
        BlockdevOptionsArchipelago *archipelago;
Quoted 5 lines

...

which means that if we name any of the branches 'data' (that is, if 'data' is a member of the enum discriminator), things fail to compile. We could probably fix that by naming our dummy branch '_data'.

Works, because schema names should not begin with '_', except for downstream extensions, which begin with '__RFQDN_'. We don't enforce that, however. I'll include the appended patch.

Subject: [PATCH] qapi: Document flaws in checking of names

We don't actually enforce our "other than downstream extensions [...], all names should begin with a letter" rule. Add a FIXME.

We should reject names that differ only in '_' vs. '.' vs. '-', because they're liable to clash in generated C. Add a FIXME.

---
 scripts/qapi.py | 4 ++++
 1 file changed, 4 insertions(+)
scripts/qapi.py +4 -0
index 4af47ef..e61db30 100644
@@ -341,6 +341,8 @@ def discriminator_find_enum_define(expr):
 
     return find_enum(discriminator_type)
 
+# FIXME should enforce "other than downstream extensions [...], all
+# names should begin with a letter".
 valid_name = re.compile('^[a-zA-Z_][a-zA-Z0-9_.-]*$')
 def check_name(expr_info, source, name, allow_optional = False,
                enum_member = False):
@@ -367,6 +369,8 @@ def check_name(expr_info, source, name, allow_optional = False,
 def add_name(name, info, meta, implicit = False):
     global all_names
     check_name(info, "'%s'" % meta, name)
+    # FIXME should reject names that differ only in '_' vs. '.'
+    # vs. '-', because they're liable to clash in generated C.
     if name in all_names:
         raise QAPIExprError(info,
                             "%s '%s' is already defined"
Signature · 1 line
2.4.3
#29 lore raw

Re: [Qemu-devel] [PATCH RFC v2 07/47] qapi: Generate a nicer struct for flat unions

In reply to Markus Armbruster
Quoted 36 lines · Markus Armbruster
Quoted 34 lines · Eric Blake
Quoted 3 lines · Markus Armbruster

The struct generated for a flat union is weird: the members of its base are at the end, except for the union tag, which is renamed to 'kind' and put at the beginning.

The renaming to 'kind' was a bug waiting to happen. Consider this example, which is broken before your series:

(quoted diff) +0 -0

w/tests/qapi-schema/qapi-schema-test.json
index c7eaa86..12c09e3 100644

tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -37,7 +37,7 @@
   'data': { 'string1': 'str', 'string2': 'str' } }
 
 { 'struct': 'UserDefUnionBase',
-  'data': { 'string': 'str', 'enum1': 'EnumOne' } }
+  'data': { 'kind': 'str', 'enum1': 'EnumOne' } }
 
 { 'union': 'UserDefFlatUnion',
   'base': 'UserDefUnionBase',

leading to this compilation error during 'make check-unit':

In file included from tests/test-qmp-output-visitor.c:17:0:
tests/test-qapi-types.h:617:11: error: duplicate member ‘kind’

     char *kind;
           ^
tests/test-qapi-types.h:631:11: error: duplicate member ‘kind’
     char *kind;
           ^
Quoted 4 lines
Quoted 2 lines

Therefore, it might be worth mentioning that avoiding the rename to 'kind' is a bug fix, not just a nicer struct :)

Cool! I'll work (a variation of) this test case into my series.

I split the bug fix off this patch. I put the test in the commit message, because I feel it has little value as regression test going forward.

#30 lore raw

RFC PATCH v2 8/47 qapi-visit: Fix generated code when schema has forward refs

In reply to Markus Armbruster

The visit_type_implicit_FOO() are generated on demand, right before their first use. Used by visit_type_STRUCT_fields() when STRUCT has base FOO, and by visit_type_UNION() when flat UNION has member a FOO.

If the schema defines FOO after its first use as struct base or flat union member, visit_type_implicit_FOO() calls visit_type_implicit_FOO() before its definition, which doesn't compile.

Rearrange qapi-schema-test.json to demonstrate the bug.

Fix by generating the necessary forward declaration.

---
 scripts/qapi-visit.py                   | 15 ++++++++++++++-
 tests/qapi-schema/qapi-schema-test.json | 30 +++++++++++++++++-------------
 tests/qapi-schema/qapi-schema-test.out  | 10 +++++-----
 3 files changed, 36 insertions(+), 19 deletions(-)
scripts/qapi-visit.py +14 -1
index 4ec79a6..b3a308f 100644
@@ -17,13 +17,23 @@ from qapi import *
 import re
 
 implicit_structs = []
+struct_fields_seen = set()
 
 def generate_visit_implicit_struct(type):
     global implicit_structs
     if type in implicit_structs:
         return ''
     implicit_structs.append(type)
-    return mcgen('''
+    ret = ''
+    if type not in struct_fields_seen:
+        # Need a forward declaration
+        ret += mcgen('''
+
+static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
+''',
+                     c_type=type_name(type))
+
+    ret += mcgen('''
 
 static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp)
 {
@@ -38,8 +48,11 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
 }
 ''',
                  c_type=type_name(type))
+    return ret
 
 def generate_visit_struct_fields(name, members, base = None):
+    struct_fields_seen.add(name)
+
     ret = ''
 
     if base:
tests/qapi-schema/qapi-schema-test.json +17 -13
index c7eaa86..ccadb13 100644
@@ -7,13 +7,13 @@
   'data': { 'enum1': 'EnumOne', '*enum2': 'EnumOne', 'enum3': 'EnumOne', '*enum4': 'EnumOne' } }
 
 # for testing nested structs
-{ 'struct': 'UserDefZero',
-  'data': { 'integer': 'int' } }
-
 { 'struct': 'UserDefOne',
-  'base': 'UserDefZero',
+  'base': 'UserDefZero',        # intentional forward reference
   'data': { 'string': 'str', '*enum1': 'EnumOne' } }
 
+{ 'struct': 'UserDefZero',
+  'data': { 'integer': 'int' } }
+
 { 'struct': 'UserDefTwoDictDict',
   'data': { 'userdef': 'UserDefOne', 'string': 'str' } }
 
@@ -33,29 +33,33 @@
 { 'struct': 'UserDefB',
   'data': { 'integer': 'int' } }
 
-{ 'struct': 'UserDefC',
-  'data': { 'string1': 'str', 'string2': 'str' } }
-
-{ 'struct': 'UserDefUnionBase',
-  'data': { 'string': 'str', 'enum1': 'EnumOne' } }
-
 { 'union': 'UserDefFlatUnion',
-  'base': 'UserDefUnionBase',
+  'base': 'UserDefUnionBase',   # intentional forward reference
   'discriminator': 'enum1',
-  'data': { 'value1' : 'UserDefA', 'value2' : 'UserDefB', 'value3' : 'UserDefB' } }
+  'data': { 'value1' : 'UserDefA',
+            'value2' : 'UserDefB',
+            'value3' : 'UserDefB' } }
 # FIXME generated struct UserDefFlatUnion has members for direct base
 # UserDefOne, but lacks members for indirect base UserDefZero
 
+{ 'struct': 'UserDefUnionBase',
+  'data': { 'string': 'str', 'enum1': 'EnumOne' } }
+
 # this variant of UserDefFlatUnion defaults to a union that uses fields with
 # allocated types to test corner cases in the cleanup/dealloc visitor
 { 'union': 'UserDefFlatUnion2',
   'base': 'UserDefUnionBase',
   'discriminator': 'enum1',
-  'data': { 'value1' : 'UserDefC', 'value2' : 'UserDefB', 'value3' : 'UserDefA' } }
+  'data': { 'value1' : 'UserDefC', # intentional forward reference
+            'value2' : 'UserDefB',
+            'value3' : 'UserDefA' } }
 
 { 'alternate': 'UserDefAlternate',
   'data': { 'uda': 'UserDefA', 's': 'str', 'i': 'int' } }
 
+{ 'struct': 'UserDefC',
+  'data': { 'string1': 'str', 'string2': 'str' } }
+
 # for testing native lists
 { 'union': 'UserDefNativeListUnion',
   'data': { 'integer': ['int'],
tests/qapi-schema/qapi-schema-test.out +5 -5
index cf0ccc4..a4291db 100644
@@ -1,17 +1,17 @@
 [OrderedDict([('enum', 'EnumOne'), ('data', ['value1', 'value2', 'value3'])]),
  OrderedDict([('struct', 'NestedEnumsOne'), ('data', OrderedDict([('enum1', 'EnumOne'), ('*enum2', 'EnumOne'), ('enum3', 'EnumOne'), ('*enum4', 'EnumOne')]))]),
- OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]),
  OrderedDict([('struct', 'UserDefOne'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('*enum1', 'EnumOne')]))]),
+ OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]),
  OrderedDict([('struct', 'UserDefTwoDictDict'), ('data', OrderedDict([('userdef', 'UserDefOne'), ('string', 'str')]))]),
  OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]),
  OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]),
  OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]),
  OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]),
- OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
- OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
  OrderedDict([('union', 'UserDefFlatUnion'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefA'), ('value2', 'UserDefB'), ('value3', 'UserDefB')]))]),
+ OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
  OrderedDict([('union', 'UserDefFlatUnion2'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefC'), ('value2', 'UserDefB'), ('value3', 'UserDefA')]))]),
  OrderedDict([('alternate', 'UserDefAlternate'), ('data', OrderedDict([('uda', 'UserDefA'), ('s', 'str'), ('i', 'int')]))]),
+ OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
  OrderedDict([('union', 'UserDefNativeListUnion'), ('data', OrderedDict([('integer', ['int']), ('s8', ['int8']), ('s16', ['int16']), ('s32', ['int32']), ('s64', ['int64']), ('u8', ['uint8']), ('u16', ['uint16']), ('u32', ['uint32']), ('u64', ['uint64']), ('number', ['number']), ('boolean', ['bool']), ('string', ['str']), ('sizes', ['size'])]))]),
  OrderedDict([('command', 'user_def_cmd'), ('data', OrderedDict())]),
  OrderedDict([('command', 'user_def_cmd1'), ('data', OrderedDict([('ud1a', 'UserDefOne')]))]),
@@ -39,15 +39,15 @@
  {'enum_name': '__org.qemu_x-Union1Kind', 'enum_values': None},
  {'enum_name': '__org.qemu_x-AltKind', 'enum_values': None}]
 [OrderedDict([('struct', 'NestedEnumsOne'), ('data', OrderedDict([('enum1', 'EnumOne'), ('*enum2', 'EnumOne'), ('enum3', 'EnumOne'), ('*enum4', 'EnumOne')]))]),
- OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]),
  OrderedDict([('struct', 'UserDefOne'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('*enum1', 'EnumOne')]))]),
+ OrderedDict([('struct', 'UserDefZero'), ('data', OrderedDict([('integer', 'int')]))]),
  OrderedDict([('struct', 'UserDefTwoDictDict'), ('data', OrderedDict([('userdef', 'UserDefOne'), ('string', 'str')]))]),
  OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]),
  OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]),
  OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]),
  OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]),
- OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
  OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
+ OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
  OrderedDict([('struct', 'UserDefOptions'), ('data', OrderedDict([('*i64', ['int']), ('*u64', ['uint64']), ('*u16', ['uint16']), ('*i64x', 'int'), ('*u64x', 'uint64')]))]),
  OrderedDict([('struct', 'EventStructOne'), ('data', OrderedDict([('struct1', 'UserDefOne'), ('string', 'str'), ('*enum2', 'EnumOne')]))]),
  OrderedDict([('struct', '__org.qemu_x-Base'), ('data', OrderedDict([('__org.qemu_x-member1', '__org.qemu_x-Enum')]))]),
Signature · 1 line
1.9.3
Reviewed-by on v2 8/47
#31 lore raw

Re: [Qemu-devel] [PATCH RFC v2 08/47] qapi-visit: Fix generated code when schema has forward refs

Quoted 8 lines · Markus Armbruster

The visit_type_implicit_FOO() are generated on demand, right before their first use. Used by visit_type_STRUCT_fields() when STRUCT has base FOO, and by visit_type_UNION() when flat UNION has member a FOO.

If the schema defines FOO after its first use as struct base or flat union member, visit_type_implicit_FOO() calls visit_type_implicit_FOO() before its definition, which doesn't compile.

None of our public qapi .json files currently do this, so no difference to the generated code used in qemu proper.

Quoted 2 lines

Rearrange qapi-schema-test.json to demonstrate the bug.

Indeed, without testsuite exposure, nothing was relying on this fix.

Quoted 24 lines

Fix by generating the necessary forward declaration.

---
 scripts/qapi-visit.py                   | 15 ++++++++++++++-
 tests/qapi-schema/qapi-schema-test.json | 30 +++++++++++++++++-------------
 tests/qapi-schema/qapi-schema-test.out  | 10 +++++-----
 3 files changed, 36 insertions(+), 19 deletions(-)

+++ b/scripts/qapi-visit.py

scripts/qapi-visit.py +1 -1
@@ -17,13 +17,23 @@ from qapi import *
 import re
 
 implicit_structs = []
+struct_fields_seen = set()
 
 def generate_visit_implicit_struct(type):
     global implicit_structs
     if type in implicit_structs:
         return ''
     implicit_structs.append(type)
-    return mcgen('''

...

Quoted 1 line

+ ret += mcgen('''

Oddly enough, the ''' is at the same indentation,...

scripts/qapi-visit.py static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * · line 51

Quoted 8 lines

static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp) {

scripts/qapi-visit.py +1 -0
@@ -38,8 +48,11 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
 }
 ''',
                  c_type=type_name(type))
+    return ret

so nothing needed to be reindented here :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#32 lore raw

Re: [Qemu-devel] [PATCH RFC v2 08/47] qapi-visit: Fix generated code when schema has forward refs

Quoted 5 lines · Eric Blake
Quoted 4 lines · Markus Armbruster

The visit_type_implicit_FOO() are generated on demand, right before their first use. Used by visit_type_STRUCT_fields() when STRUCT has base FOO, and by visit_type_UNION() when flat UNION has member a FOO.

Quoted 7 lines
Quoted 4 lines

If the schema defines FOO after its first use as struct base or flat union member, visit_type_implicit_FOO() calls visit_type_implicit_FOO() before its definition, which doesn't compile.

None of our public qapi .json files currently do this, so no difference to the generated code used in qemu proper.

Yes, we've intentionally avoided forward references.

Nevertheless, the generators either have to catch and reject them, or make them work. Making them work is easy, so I did that.

Quoted 44 lines
Quoted 1 line

Rearrange qapi-schema-test.json to demonstrate the bug.

Indeed, without testsuite exposure, nothing was relying on this fix.

Quoted 24 lines

Fix by generating the necessary forward declaration.

---
 scripts/qapi-visit.py                   | 15 ++++++++++++++-
 tests/qapi-schema/qapi-schema-test.json | 30 +++++++++++++++++-------------
 tests/qapi-schema/qapi-schema-test.out  | 10 +++++-----
 3 files changed, 36 insertions(+), 19 deletions(-)

+++ b/scripts/qapi-visit.py

scripts/qapi-visit.py +1 -1
@@ -17,13 +17,23 @@ from qapi import *
 import re
 
 implicit_structs = []
+struct_fields_seen = set()
 
 def generate_visit_implicit_struct(type):
     global implicit_structs
     if type in implicit_structs:
         return ''
     implicit_structs.append(type)
-    return mcgen('''

...

Quoted 1 line

+ ret += mcgen('''

Oddly enough, the ''' is at the same indentation,...

scripts/qapi-visit.py static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * · line 51

Quoted 8 lines

static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp) {

scripts/qapi-visit.py +1 -0
@@ -38,8 +48,11 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
 }
 ''',
                  c_type=type_name(type))
+    return ret

so nothing needed to be reindented here :)

Quoted 1 line

Thanks!

#33 lore raw

RFC PATCH v2 9/47 qapi-visit: Replace list implicit_structs by set

In reply to Markus Armbruster

Use set because that's what it is. While there, rename to implicit_structs_seen.

---
 scripts/qapi-visit.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
scripts/qapi-visit.py +3 -4
index b3a308f..9fc040e 100644
@@ -16,14 +16,13 @@ from ordereddict import OrderedDict
 from qapi import *
 import re
 
-implicit_structs = []
+implicit_structs_seen = set()
 struct_fields_seen = set()
 
 def generate_visit_implicit_struct(type):
-    global implicit_structs
-    if type in implicit_structs:
+    if type in implicit_structs_seen:
         return ''
-    implicit_structs.append(type)
+    implicit_structs_seen.add(type)
     ret = ''
     if type not in struct_fields_seen:
         # Need a forward declaration
Signature · 1 line
1.9.3
Reviewed-by on v2 9/47
#34 lore raw

Re: [Qemu-devel] [PATCH RFC v2 09/47] qapi-visit: Replace list implicit_structs by set

Quoted 8 lines · Markus Armbruster

Use set because that's what it is. While there, rename to implicit_structs_seen.

---
 scripts/qapi-visit.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

(I have no idea if that makes python more efficient, but it is cleaner)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#35 lore raw

RFC PATCH v2 10/47 qapi-visit: Fix two name arguments passed to visitors

In reply to Markus Armbruster

The generated code passes mangled schema names to visit_type_enum() and union's visit_start_struct(). Fix it to pass the names unadulterated, like we do everywhere else.

Only qapi-schema-test.json actually has names where this makes a
difference: enum __org.qemu_x-Enum, flat union __org.qemu_x-Union2, simple union __org.qemu_x-Union1 and its implicit enum __org.qemu_x-Union1Kind.

---
 scripts/qapi-visit.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
scripts/qapi-visit.py +6 -6
index 9fc040e..73f136f 100644
@@ -185,12 +185,12 @@ out:
 def generate_visit_enum(name):
     return mcgen('''
 
-void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp)
+void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error **errp)
 {
-    visit_type_enum(m, (int *)obj, %(name)s_lookup, "%(name)s", name, errp);
+    visit_type_enum(m, (int *)obj, %(c_name)s_lookup, "%(name)s", name, errp);
 }
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name), name=name)
 
 def generate_visit_alternate(name, members):
     ret = mcgen('''
@@ -278,17 +278,17 @@ def generate_visit_union(expr):
 
     ret += mcgen('''
 
-void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
+void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
 
-    visit_start_struct(m, (void **)obj, "%(name)s", name, sizeof(%(name)s), &err);
+    visit_start_struct(m, (void **)obj, "%(name)s", name, sizeof(%(c_name)s), &err);
     if (err) {
         goto out;
     }
     if (*obj) {
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name), name=name)
 
     if base:
         ret += mcgen('''
Signature · 1 line
1.9.3
Reviewed-by on v2 10/47
#36 lore raw

Re: [Qemu-devel] [PATCH RFC v2 10/47] qapi-visit: Fix two name arguments passed to visitors

Quoted 8 lines · Markus Armbruster

The generated code passes mangled schema names to visit_type_enum() and union's visit_start_struct(). Fix it to pass the names unadulterated, like we do everywhere else.

Only qapi-schema-test.json actually has names where this makes a
difference: enum __org.qemu_x-Enum, flat union __org.qemu_x-Union2, simple union __org.qemu_x-Union1 and its implicit enum __org.qemu_x-Union1Kind.

I'm glad I added that test - it covers things that will affect downstream extensions, but which don't affect mainline.

Quoted 6 lines
---
 scripts/qapi-visit.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#37 lore raw

RFC PATCH v2 11/47 tests/qapi-schema: Document alternate's enum lacks visit function

In reply to Markus Armbruster

We generate a declaration, but no definition.

The QMP schema has two: Qcow2OverlapChecks and BlockdevRef. Neither visit_type_Qcow2OverlapChecksKind() nor visit_type_BlockdevRefKind() is actually used.

---
 tests/qapi-schema/qapi-schema-test.json | 1 +
 1 file changed, 1 insertion(+)
tests/qapi-schema/qapi-schema-test.json +1 -0
index ccadb13..8337ba9 100644
@@ -56,6 +56,7 @@
 
 { 'alternate': 'UserDefAlternate',
   'data': { 'uda': 'UserDefA', 's': 'str', 'i': 'int' } }
+# FIXME only a declaration of visit_type_UserDefAlternateKind() generated
 
 { 'struct': 'UserDefC',
   'data': { 'string1': 'str', 'string2': 'str' } }
Signature · 1 line
1.9.3
Reviewed-by on v2 11/47
#38 lore raw

Re: [Qemu-devel] [PATCH RFC v2 11/47] tests/qapi-schema: Document alternate's enum lacks visit function

Quoted 10 lines · Markus Armbruster

We generate a declaration, but no definition.

The QMP schema has two: Qcow2OverlapChecks and BlockdevRef. Neither visit_type_Qcow2OverlapChecksKind() nor visit_type_BlockdevRefKind() is actually used.

---
 tests/qapi-schema/qapi-schema-test.json | 1 +
 1 file changed, 1 insertion(+)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#39 lore raw

RFC PATCH v2 12/47 tests/qapi-schema: Document events with with base don't work

In reply to Markus Armbruster

When event FOO's 'data' is a struct with a base, we consider only the struct's direct members, and ignore its base. The generated qapi_event_send_foo() doesn't take arguments for base members.

No such events currently exist in the QMP schema.

---
 tests/qapi-schema/qapi-schema-test.json | 3 +++
 1 file changed, 3 insertions(+)
tests/qapi-schema/qapi-schema-test.json +3 -0
index 8337ba9..829dd30 100644
@@ -128,6 +128,9 @@
 { 'alternate': '__org.qemu_x-Alt',
   'data': { '__org.qemu_x-branch': 'str', 'b': '__org.qemu_x-Base' } }
 { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }
+# FIXME generated qapi_event_send___org_qemu_x_event() has only a
+# parameter for data's member __org_qemu_x_member2, none for its base
+# __org.qemu_x-Base's member __org_qemu_x_member1
 { 'command': '__org.qemu_x-command',
   'data': { 'a': ['__org.qemu_x-Enum'], 'b': ['__org.qemu_x-Struct'],
             'c': '__org.qemu_x-Union2', 'd': '__org.qemu_x-Alt' },
Signature · 1 line
1.9.3
Reviewed-by on v2 12/47
#40 lore raw

Re: [Qemu-devel] [PATCH RFC v2 12/47] tests/qapi-schema: Document events with with base don't work

Quoted 10 lines · Markus Armbruster

When event FOO's 'data' is a struct with a base, we consider only the struct's direct members, and ignore its base. The generated qapi_event_send_foo() doesn't take arguments for base members.

No such events currently exist in the QMP schema.

---
 tests/qapi-schema/qapi-schema-test.json | 3 +++
 1 file changed, 3 insertions(+)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#41 lore raw

RFC PATCH 1/1 qapi: Document that input visitor semantics are prone to leaks

In reply to Markus Armbruster

Most functions that can return a pointer or set an Error ** value are decent enough to guarantee a NULL return when reporting an error. Not so with our generated qapi visitor functions. If the caller is not careful to clean up partially-allocated objects on error, then the caller suffers a memory leak.

Properly fixing it is probably complex enough to save for a later day, so merely document it for now.

---

As mentioned elsewhere in the thread (comments on 29/47
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg06107.html)
it would be worth documenting a FIXME for a future series. I'm submitting this as 12.5/47 due to its relation to other similar shortcoming doc patches; and assuming Markus can rebase the rest of the series on top if he wants to fold it into his v3 posting at this spot. Otherwise, I can wait for his v3 and rebase it to be part of my (growing) followup series.

 scripts/qapi-visit.py          | 4 ++++
 tests/test-qmp-input-visitor.c | 2 ++
 2 files changed, 6 insertions(+)
scripts/qapi-visit.py +4 -0
index 73f136f..eec5f1f 100644
@@ -115,6 +115,10 @@ out:
 
 
 def generate_visit_struct_body(name):
+    # FIXME: if *obj is NULL on entry, and visit_start_struct() assigns to
+    # *obj, but then visit_type_FOO_fields() fails, we should clean up *obj
+    # rather than leaving it non-NULL. As currently written, the caller must
+    # call qapi_free_FOO() to avoid a memory leak of the partial FOO.
     ret = mcgen('''
     Error *err = NULL;
 
tests/test-qmp-input-visitor.c +2 -0
index b7a87ee..a5cfefa 100644
@@ -636,6 +636,8 @@ static void test_visitor_in_errors(TestInputVisitorData *data,
 
     visit_type_TestStruct(v, &p, NULL, &err);
     g_assert(err);
+    /* FIXME - a failed parse should not leave a partially-allocated p
+     * for us to clean up; this could cause callers to leak memory. */
     g_assert(p->string == NULL);
 
     error_free(err);
Signature · 1 line
2.4.3
#42 lore raw

Re: [Qemu-devel] [RFC PATCH 12.5/47] qapi: Document that input visitor semantics are prone to leaks

Quoted 10 lines · Eric Blake

Most functions that can return a pointer or set an Error ** value are decent enough to guarantee a NULL return when reporting an error. Not so with our generated qapi visitor functions. If the caller is not careful to clean up partially-allocated objects on error, then the caller suffers a memory leak.

Properly fixing it is probably complex enough to save for a later day, so merely document it for now.

Picked into my series. Thanks!

#43 lore raw

RFC PATCH 1/1 qapi: Document shortcoming with union 'data' branch

In reply to Markus Armbruster

Add a FIXME to remind us to fully audit whether removing the 'void *data' branch of each qapi union type can be done safely.

---

Another potential doc FIXME patch, this time based on 7/47
(https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg06101.html)

 scripts/qapi-types.py | 8 ++++++++
 1 file changed, 8 insertions(+)
scripts/qapi-types.py +8 -0
index c6c2786..b3434b9 100644
@@ -217,6 +217,14 @@ struct %(name)s
 ''',
                      discriminator_type_name=c_name(discriminator_type_name))
 
+    # FIXME: What purpose does data serve, besides preventing a union that
+    # has a branch named 'data'? We use it in qapi-visit.py to decide
+    # whether to bypass the switch statement if visiting the discriminator
+    # failed; but since we 0-initialize structs, and cannot tell what
+    # branch of the union is in use if the discriminator is invalid, there
+    # should not be any data leaks even without a data pointer.  Or, if
+    # 'data' is merely added to guarantee we don't have an empty union,
+    # shouldn't we enforce that at .json parse time?
     ret += mcgen('''
     union {
         void *data;
Signature · 1 line
2.4.3
#44 lore raw

Re: [Qemu-devel] [RFC PATCH 12.6/47] qapi: Document shortcoming with union 'data' branch

Quoted 4 lines · Eric Blake

Add a FIXME to remind us to fully audit whether removing the 'void *data' branch of each qapi union type can be done safely.

Picked into my series. Thanks!

#45 lore raw

RFC PATCH v2 13/47 tests/qapi-schema: Restore test case for flat union base bug

In reply to Markus Armbruster

Test case added in commit 2fc0043, and messed up in commit 5223070.

---
 tests/qapi-schema/qapi-schema-test.json | 5 +++--
 tests/qapi-schema/qapi-schema-test.out  | 8 ++++----
 2 files changed, 7 insertions(+), 6 deletions(-)
tests/qapi-schema/qapi-schema-test.json +3 -2
index 829dd30..a9e5aab 100644
@@ -31,7 +31,7 @@
   'data': { 'boolean': 'bool' } }
 
 { 'struct': 'UserDefB',
-  'data': { 'integer': 'int' } }
+  'data': { 'intb': 'int' } }
 
 { 'union': 'UserDefFlatUnion',
   'base': 'UserDefUnionBase',   # intentional forward reference
@@ -40,9 +40,10 @@
             'value2' : 'UserDefB',
             'value3' : 'UserDefB' } }
 # FIXME generated struct UserDefFlatUnion has members for direct base
-# UserDefOne, but lacks members for indirect base UserDefZero
+# UserDefUnionBase, but lacks members for indirect base UserDefZero
 
 { 'struct': 'UserDefUnionBase',
+  'base': 'UserDefZero',
   'data': { 'string': 'str', 'enum1': 'EnumOne' } }
 
 # this variant of UserDefFlatUnion defaults to a union that uses fields with
tests/qapi-schema/qapi-schema-test.out +4 -4
index a4291db..b0b7187 100644
@@ -6,9 +6,9 @@
  OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]),
  OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]),
  OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]),
- OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]),
+ OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('intb', 'int')]))]),
  OrderedDict([('union', 'UserDefFlatUnion'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefA'), ('value2', 'UserDefB'), ('value3', 'UserDefB')]))]),
- OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
+ OrderedDict([('struct', 'UserDefUnionBase'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
  OrderedDict([('union', 'UserDefFlatUnion2'), ('base', 'UserDefUnionBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'UserDefC'), ('value2', 'UserDefB'), ('value3', 'UserDefA')]))]),
  OrderedDict([('alternate', 'UserDefAlternate'), ('data', OrderedDict([('uda', 'UserDefA'), ('s', 'str'), ('i', 'int')]))]),
  OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
@@ -45,8 +45,8 @@
  OrderedDict([('struct', 'UserDefTwoDict'), ('data', OrderedDict([('string1', 'str'), ('dict2', 'UserDefTwoDictDict'), ('*dict3', 'UserDefTwoDictDict')]))]),
  OrderedDict([('struct', 'UserDefTwo'), ('data', OrderedDict([('string0', 'str'), ('dict1', 'UserDefTwoDict')]))]),
  OrderedDict([('struct', 'UserDefA'), ('data', OrderedDict([('boolean', 'bool')]))]),
- OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('integer', 'int')]))]),
- OrderedDict([('struct', 'UserDefUnionBase'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
+ OrderedDict([('struct', 'UserDefB'), ('data', OrderedDict([('intb', 'int')]))]),
+ OrderedDict([('struct', 'UserDefUnionBase'), ('base', 'UserDefZero'), ('data', OrderedDict([('string', 'str'), ('enum1', 'EnumOne')]))]),
  OrderedDict([('struct', 'UserDefC'), ('data', OrderedDict([('string1', 'str'), ('string2', 'str')]))]),
  OrderedDict([('struct', 'UserDefOptions'), ('data', OrderedDict([('*i64', ['int']), ('*u64', ['uint64']), ('*u16', ['uint16']), ('*i64x', 'int'), ('*u64x', 'uint64')]))]),
  OrderedDict([('struct', 'EventStructOne'), ('data', OrderedDict([('struct1', 'UserDefOne'), ('string', 'str'), ('*enum2', 'EnumOne')]))]),
Signature · 1 line
1.9.3
Reviewed-by on v2 13/47
#46 lore raw

Re: [Qemu-devel] [PATCH RFC v2 13/47] tests/qapi-schema: Restore test case for flat union base bug

Quoted 8 lines · Markus Armbruster

Test case added in commit 2fc0043, and messed up in commit 5223070.

---
 tests/qapi-schema/qapi-schema-test.json | 5 +++--
 tests/qapi-schema/qapi-schema-test.out  | 8 ++++----
 2 files changed, 7 insertions(+), 6 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#47 lore raw

RFC PATCH v2 14/47 qapi-tests: New tests for union, alternate command arguments

In reply to Markus Armbruster

A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.

Existing test case data-int.json covers simple type 'int'. Add test cases for type names referring to union and alternate types.

The latter is caught (good), but the former is not (bug).

---
 tests/Makefile                        | 3 ++-
 tests/qapi-schema/args-alternate.err  | 1 +
 tests/qapi-schema/args-alternate.exit | 1 +
 tests/qapi-schema/args-alternate.json | 4 ++++
 tests/qapi-schema/args-alternate.out  | 0
 tests/qapi-schema/args-union.err      | 0
 tests/qapi-schema/args-union.exit     | 1 +
 tests/qapi-schema/args-union.json     | 4 ++++
 tests/qapi-schema/args-union.out      | 4 ++++
 9 files changed, 17 insertions(+), 1 deletion(-)
 create mode 100644 tests/qapi-schema/args-alternate.err
 create mode 100644 tests/qapi-schema/args-alternate.exit
 create mode 100644 tests/qapi-schema/args-alternate.json
 create mode 100644 tests/qapi-schema/args-alternate.out
 create mode 100644 tests/qapi-schema/args-union.err
 create mode 100644 tests/qapi-schema/args-union.exit
 create mode 100644 tests/qapi-schema/args-union.json
 create mode 100644 tests/qapi-schema/args-union.out
tests/Makefile +2 -1
index eff5e11..5a38748 100644
@@ -225,7 +225,8 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
 	type-bypass.json type-bypass-no-gen.json type-bypass-bad-gen.json \
 	data-array-empty.json data-array-unknown.json data-int.json \
 	data-unknown.json data-member-unknown.json data-member-array.json \
-	data-member-array-bad.json returns-array-bad.json returns-int.json \
+	data-member-array-bad.json args-union.json args-alternate.json \
+	returns-array-bad.json returns-int.json \
 	returns-unknown.json returns-alternate.json returns-whitelist.json \
 	missing-colon.json missing-comma-list.json missing-comma-object.json \
 	nested-struct-data.json nested-struct-returns.json non-objects.json \
tests/qapi-schema/args-alternate.err new file +1 -0
new file mode 100644
index 0000000..fc90a9e
@@ -0,0 +1 @@
+tests/qapi-schema/args-alternate.json:4: 'data' for command 'oops' cannot use alternate type 'Alt'
tests/qapi-schema/args-alternate.exit new file +1 -0
new file mode 100644
index 0000000..d00491f
@@ -0,0 +1 @@
+1
tests/qapi-schema/args-alternate.json new file +4 -0
new file mode 100644
index 0000000..d2f9199
@@ -0,0 +1,4 @@
+# we do not allow alternate arguments
+# TODO should we support this?
+{ 'alternate': 'Alt', 'data': { 'case1': 'int', 'case2': 'str' } }
+{ 'command': 'oops', 'data': 'Alt' }
tests/qapi-schema/args-alternate.out new file +0 -0
new file mode 100644
index 0000000..e69de29
tests/qapi-schema/args-union.err new file +0 -0
new file mode 100644
index 0000000..e69de29
tests/qapi-schema/args-union.exit new file +1 -0
new file mode 100644
index 0000000..573541a
@@ -0,0 +1 @@
+0
tests/qapi-schema/args-union.json new file +4 -0
new file mode 100644
index 0000000..db97ef6
@@ -0,0 +1,4 @@
+# FIXME we should reject union arguments
+# TODO should we support this?
+{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
+{ 'command': 'oops', 'data': 'Uni' }
tests/qapi-schema/args-union.out new file +4 -0
new file mode 100644
index 0000000..907080c
@@ -0,0 +1,4 @@
+[OrderedDict([('union', 'Uni'), ('data', OrderedDict([('case1', 'int'), ('case2', 'str')]))]),
+ OrderedDict([('command', 'oops'), ('data', 'Uni')])]
+[{'enum_name': 'UniKind', 'enum_values': None}]
+[]
Signature · 1 line
1.9.3
Reviewed-by on v2 14/47
#48 lore raw

Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments

Quoted 5 lines · Markus Armbruster

A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.

Existing test case data-int.json covers simple type 'int'. Add test cases for type names referring to union and alternate types.

We could probably relax things to allow a union (which is always a dictionary on the wire), but I agree that allowing an alternate type is not appropriate (the goal here is that we require a dictionary). But it's also easier to be conservative now and relax later.

Quoted 18 lines

The latter is caught (good), but the former is not (bug).

---
 tests/Makefile                        | 3 ++-
 tests/qapi-schema/args-alternate.err  | 1 +
 tests/qapi-schema/args-alternate.exit | 1 +
 tests/qapi-schema/args-alternate.json | 4 ++++
 tests/qapi-schema/args-alternate.out  | 0
 tests/qapi-schema/args-union.err      | 0
 tests/qapi-schema/args-union.exit     | 1 +
 tests/qapi-schema/args-union.json     | 4 ++++
 tests/qapi-schema/args-union.out      | 4 ++++
 9 files changed, 17 insertions(+), 1 deletion(-)

+++ b/tests/qapi-schema/args-alternate.json

tests/qapi-schema/args-alternate.json line 2

Quoted 2 lines
tests/qapi-schema/args-alternate.json new file +2 -0
@@ -0,0 +1,4 @@
+# we do not allow alternate arguments
+# TODO should we support this?

I see no reason to allow a non-dictionary in QMP, so this TODO could be dropped.

tests/qapi-schema/args-union.json line 4

Quoted 6 lines
tests/qapi-schema/args-union.json new file +5 -0
+++ b/tests/qapi-schema/args-union.json
@@ -0,0 +1,4 @@
+# FIXME we should reject union arguments
+# TODO should we support this?
+{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
+{ 'command': 'oops', 'data': 'Uni' }

This, on the other hand, seems valid from the wire format (it will always be a dictionary). I guess the problem is that we generate a C function signature based by calling out each member of the dictionary - but how do you do that for a union? So I see what you are doing:
marking that this test currently passes the parser, but then causes problems for generating C code, so we should either reject it up front, or fix the generator. The FIXME documents what you will do later in the series (reject it up front) and the TODO documents what we can do down the road (fix the generator to allow it).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#49 lore raw

Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments

Quoted 13 lines · Eric Blake
Quoted 5 lines · Markus Armbruster

A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.

Existing test case data-int.json covers simple type 'int'. Add test cases for type names referring to union and alternate types.

We could probably relax things to allow a union (which is always a dictionary on the wire), but I agree that allowing an alternate type is not appropriate (the goal here is that we require a dictionary). But it's also easier to be conservative now and relax later.

Quoted 7 lines

tests/qapi-schema/args-alternate.json line 2

Quoted 4 lines

+++ b/tests/qapi-schema/args-alternate.json

tests/qapi-schema/args-alternate.json new file +2 -0
@@ -0,0 +1,4 @@
+# we do not allow alternate arguments
+# TODO should we support this?

I see no reason to allow a non-dictionary in QMP, so this TODO could be dropped.

Or, to be clear, we document that arguments is always a dictionary, for:
{ "execute":"command", "arguments":{} }. Allowing an alternate would break that, so it is a different level of change to allow an alternate (change the QMP protocol) than what it would be to allow a union (the QMP protocol is unchanged). Not that we can't do it if we ever have a reason, it's just that I don't see it being worth a TODO statement.

Quoted 17 lines

tests/qapi-schema/args-union.json line 4

Quoted 7 lines
tests/qapi-schema/args-union.json new file +5 -0
+++ b/tests/qapi-schema/args-union.json
@@ -0,0 +1,4 @@
+# FIXME we should reject union arguments
+# TODO should we support this?
+{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
+{ 'command': 'oops', 'data': 'Uni' }

This, on the other hand, seems valid from the wire format (it will always be a dictionary). I guess the problem is that we generate a C function signature based by calling out each member of the dictionary - but how do you do that for a union? So I see what you are doing:
marking that this test currently passes the parser, but then causes problems for generating C code, so we should either reject it up front, or fix the generator. The FIXME documents what you will do later in the series (reject it up front) and the TODO documents what we can do down the road (fix the generator to allow it).

See also 32/47 - events have the same problem.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#50 lore raw

Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments

Quoted 5 lines · Eric Blake
Quoted 4 lines · Eric Blake
Quoted 3 lines · Markus Armbruster

A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Existing test case data-int.json covers simple type 'int'. Add test cases for type names referring to union and alternate types.

We could probably relax things to allow a union (which is always a dictionary on the wire), but I agree that allowing an alternate type is not appropriate (the goal here is that we require a dictionary). But it's also easier to be conservative now and relax later.

Yes.

Quoted 8 lines
Quoted 7 lines

tests/qapi-schema/args-alternate.json line 2

Quoted 4 lines

+++ b/tests/qapi-schema/args-alternate.json

tests/qapi-schema/args-alternate.json new file +2 -0
@@ -0,0 +1,4 @@
+# we do not allow alternate arguments
+# TODO should we support this?

I see no reason to allow a non-dictionary in QMP, so this TODO could be dropped.

Quoted 6 lines

Or, to be clear, we document that arguments is always a dictionary, for:
{ "execute":"command", "arguments":{} }. Allowing an alternate would break that, so it is a different level of change to allow an alternate (change the QMP protocol) than what it would be to allow a union (the QMP protocol is unchanged). Not that we can't do it if we ever have a reason, it's just that I don't see it being worth a TODO statement.

I'll drop it.

Quoted 11 lines
Quoted 11 lines

tests/qapi-schema/args-union.json line 4

Quoted 6 lines
tests/qapi-schema/args-union.json new file +5 -0
+++ b/tests/qapi-schema/args-union.json
@@ -0,0 +1,4 @@
+# FIXME we should reject union arguments
+# TODO should we support this?
+{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
+{ 'command': 'oops', 'data': 'Uni' }

This, on the other hand, seems valid from the wire format (it will always be a dictionary). I guess the problem is that we generate a C function signature based by calling out each member of the dictionary - but how do you do that for a union?

Exactly: the problem is neither conceptual nor the wire API, it's the C API we generate.

Quoted 5 lines
Quoted 5 lines

So I see what you are doing:
marking that this test currently passes the parser, but then causes problems for generating C code, so we should either reject it up front, or fix the generator. The FIXME documents what you will do later in the series (reject it up front)

Yes, in PATCH 15.

Quoted 2 lines
Quoted 2 lines

and the TODO documents what we can do down
the road (fix the generator to allow it).

I figure we'd change the C API not to explode the data type into multiple parameters. We can consider that when we have a use for it.

Quoted 1 line

See also 32/47 - events have the same problem.

I'm afraid I don't see the connection to PATCH 32.

Quoted 1 line
Quoted 1 line

Thanks!

#51 lore raw

Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments

Quoted 19 lines · Markus Armbruster
Quoted 4 lines
Quoted 4 lines

This, on the other hand, seems valid from the wire format (it will always be a dictionary). I guess the problem is that we generate a C function signature based by calling out each member of the dictionary - but how do you do that for a union?

Exactly: the problem is neither conceptual nor the wire API, it's the C API we generate.

Quoted 5 lines
Quoted 5 lines

So I see what you are doing:
marking that this test currently passes the parser, but then causes problems for generating C code, so we should either reject it up front, or fix the generator. The FIXME documents what you will do later in the series (reject it up front)

Yes, in PATCH 15.

Quoted 2 lines
Quoted 2 lines

and the TODO documents what we can do down
the road (fix the generator to allow it).

Quoted 6 lines

I figure we'd change the C API not to explode the data type into multiple parameters. We can consider that when we have a use for it.

Quoted 1 line

See also 32/47 - events have the same problem.

I'm afraid I don't see the connection to PATCH 32.

Patch 32 was where I figured out that we have the same problem where the C code we generate for an event will break if an event tries to use a union type as its data dictionary; and therefore, this patch would be an ideal time to add a test for that, and patch 15 would be an ideal time to tweak events to not allow unions (as the simple fix, where someday down the road we may relax things to allow unions in both commands and events).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#52 lore raw

Re: [Qemu-devel] [PATCH RFC v2 14/47] qapi-tests: New tests for union, alternate command arguments

In reply to Eric Blake
Quoted 47 lines · Eric Blake
Quoted 13 lines · Eric Blake
Quoted 5 lines · Markus Armbruster

A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.

Existing test case data-int.json covers simple type 'int'. Add test cases for type names referring to union and alternate types.

We could probably relax things to allow a union (which is always a dictionary on the wire), but I agree that allowing an alternate type is not appropriate (the goal here is that we require a dictionary). But it's also easier to be conservative now and relax later.

Quoted 7 lines

tests/qapi-schema/args-alternate.json line 2

Quoted 4 lines

+++ b/tests/qapi-schema/args-alternate.json

tests/qapi-schema/args-alternate.json new file +2 -0
@@ -0,0 +1,4 @@
+# we do not allow alternate arguments
+# TODO should we support this?

I see no reason to allow a non-dictionary in QMP, so this TODO could be dropped.

Or, to be clear, we document that arguments is always a dictionary, for:
{ "execute":"command", "arguments":{} }. Allowing an alternate would break that, so it is a different level of change to allow an alternate (change the QMP protocol) than what it would be to allow a union (the QMP protocol is unchanged). Not that we can't do it if we ever have a reason, it's just that I don't see it being worth a TODO statement.

Quoted 17 lines

tests/qapi-schema/args-union.json line 4

Quoted 7 lines
tests/qapi-schema/args-union.json new file +5 -0
+++ b/tests/qapi-schema/args-union.json
@@ -0,0 +1,4 @@
+# FIXME we should reject union arguments
+# TODO should we support this?
+{ 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
+{ 'command': 'oops', 'data': 'Uni' }

This, on the other hand, seems valid from the wire format (it will always be a dictionary). I guess the problem is that we generate a C function signature based by calling out each member of the dictionary - but how do you do that for a union? So I see what you are doing:
marking that this test currently passes the parser, but then causes problems for generating C code, so we should either reject it up front, or fix the generator. The FIXME documents what you will do later in the series (reject it up front) and the TODO documents what we can do down the road (fix the generator to allow it).

Quoted 1 line

See also 32/47 - events have the same problem.

I'll amend the next patch to fix them, too.

#53 lore raw

RFC PATCH v2 15/47 qapi: Fix to reject union command arguments

In reply to Markus Armbruster

A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.

Commit dd883c6 tightened the checking there, but not enough: we still accept 'union'. Fix to reject it.

We may want to support union types there, but we'll have to extend qapi-commands.py for it.

---
 scripts/qapi.py                   | 2 +-
 tests/qapi-schema/args-union.err  | 1 +
 tests/qapi-schema/args-union.exit | 2 +-
 tests/qapi-schema/args-union.json | 2 +-
 tests/qapi-schema/args-union.out  | 4 ----
 5 files changed, 4 insertions(+), 7 deletions(-)
scripts/qapi.py +1 -1
index 99e04f6..bebaecc 100644
@@ -495,7 +495,7 @@ def check_command(expr, expr_info):
 
     check_type(expr_info, "'data' for command '%s'" % name,
                expr.get('data'), allow_dict=True, allow_optional=True,
-               allow_metas=['union', 'struct'], allow_star=allow_star)
+               allow_metas=['struct'], allow_star=allow_star)
     returns_meta = ['union', 'struct']
     if name in returns_whitelist:
         returns_meta += ['built-in', 'alternate', 'enum']
tests/qapi-schema/args-union.err +1 -0
index e69de29..1d693d7 100644
@@ -0,0 +1 @@
+tests/qapi-schema/args-union.json:4: 'data' for command 'oops' cannot use union type 'Uni'
tests/qapi-schema/args-union.exit +1 -1
index 573541a..d00491f 100644
@@ -1 +1 @@
-0
+1
tests/qapi-schema/args-union.json +1 -1
index db97ef6..7bdcbb7 100644
@@ -1,4 +1,4 @@
-# FIXME we should reject union arguments
+# we do not allow union arguments
 # TODO should we support this?
 { 'union': 'Uni', 'data': { 'case1': 'int', 'case2': 'str' } }
 { 'command': 'oops', 'data': 'Uni' }
tests/qapi-schema/args-union.out +0 -4
index 907080c..e69de29 100644
@@ -1,4 +0,0 @@
-[OrderedDict([('union', 'Uni'), ('data', OrderedDict([('case1', 'int'), ('case2', 'str')]))]),
- OrderedDict([('command', 'oops'), ('data', 'Uni')])]
-[{'enum_name': 'UniKind', 'enum_values': None}]
-[]
Signature · 1 line
1.9.3
Reviewed-by on v2 15/47
#54 lore raw

Re: [Qemu-devel] [PATCH RFC v2 15/47] qapi: Fix to reject union command arguments

Quoted 5 lines · Markus Armbruster

A command's 'data' must be a struct type, given either as a dictionary, or as struct type name.

Commit dd883c6 tightened the checking there, but not enough: we still accept 'union'. Fix to reject it.

I was thinking that 'all dictionaries are okay' when I wrote that commit, and obviously missed thinking about 'how would we translate unions into a C prototype'.

Quoted 4 lines

We may want to support union types there, but we'll have to extend qapi-commands.py for it.

Quoted 8 lines
---
 scripts/qapi.py                   | 2 +-
 tests/qapi-schema/args-union.err  | 1 +
 tests/qapi-schema/args-union.exit | 2 +-
 tests/qapi-schema/args-union.json | 2 +-
 tests/qapi-schema/args-union.out  | 4 ----
 5 files changed, 4 insertions(+), 7 deletions(-)

scripts/qapi.py def check_command(expr, expr_info): · line 498

Quoted 11 lines
scripts/qapi.py +1 -1
index 99e04f6..bebaecc 100644
@@ -495,7 +495,7 @@ def check_command(expr, expr_info):
 
     check_type(expr_info, "'data' for command '%s'" % name,
                expr.get('data'), allow_dict=True, allow_optional=True,
-               allow_metas=['union', 'struct'], allow_star=allow_star)
+               allow_metas=['struct'], allow_star=allow_star)

I'm still quite pleased at how my earlier work made this a one-liner fix
- the check_type() interface turned out to be rather pleasant.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#55 lore raw

RFC PATCH v2 16/47 qapi-commands: Fix gen_err_check(e) for e and e != 'local_err'

In reply to Markus Armbruster

gen_err_check() hard-codes 'local_err' instead of substituting the argument. Currently harmless, since all callers pass either None or 'local_err'.

---
 scripts/qapi-commands.py | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
scripts/qapi-commands.py +7 -6
index 69029f5..3965ca8 100644
@@ -29,14 +29,15 @@ def generate_command_decl(name, args, ret_type):
                  ret_type=c_type(ret_type), name=c_name(name),
                  args=arglist).strip()
 
-def gen_err_check(errvar):
-    if errvar:
-        return mcgen('''
-if (local_err) {
+def gen_err_check(err):
+    if not err:
+        return ''
+    return mcgen('''
+if (%(err)s) {
     goto out;
 }
-''')
-    return ''
+''',
+                 err=err)
 
 def gen_sync_call(name, args, ret_type):
     ret = ""
Signature · 1 line
1.9.3
Reviewed-by on v2 16/47
#56 lore raw

Re: [Qemu-devel] [PATCH RFC v2 16/47] qapi-commands: Fix gen_err_check(e) for e and e != 'local_err'

Quoted 9 lines · Markus Armbruster

gen_err_check() hard-codes 'local_err' instead of substituting the argument. Currently harmless, since all callers pass either None or 'local_err'.

---
 scripts/qapi-commands.py | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#57 lore raw

RFC PATCH v2 17/47 qapi-commands: Inline gen_marshal_output_call()

In reply to Markus Armbruster
---
 scripts/qapi-commands.py | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
scripts/qapi-commands.py +4 -8
index 3965ca8..6de5229 100644
@@ -57,19 +57,15 @@ def gen_sync_call(name, args, ret_type):
                 name=c_name(name), args=arglist, retval=retval).rstrip()
     if ret_type:
         ret += "\n" + gen_err_check('local_err')
-        ret += "\n" + mcgen('''
-%(marshal_output_call)s
+        ret += mcgen('''
+
+qmp_marshal_output_%(c_name)s(retval, ret, &local_err);
 ''',
-                            marshal_output_call=gen_marshal_output_call(name, ret_type)).rstrip()
+                            c_name=c_name(name))
     pop_indent()
     return ret.rstrip()
 
 
-def gen_marshal_output_call(name, ret_type):
-    if not ret_type:
-        return ""
-    return "qmp_marshal_output_%s(retval, ret, &local_err);" % c_name(name)
-
 def gen_visitor_input_containers_decl(args):
     ret = ""
 
Signature · 1 line
1.9.3
Reviewed-by on v2 17/47
#58 lore raw

Re: [Qemu-devel] [PATCH RFC v2 17/47] qapi-commands: Inline gen_marshal_output_call()

Quoted 5 lines · Markus Armbruster
---
 scripts/qapi-commands.py | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#59 lore raw

RFC PATCH v2 18/47 qapi-commands: Don't feed output of mcgen() to mcgen() again

In reply to Markus Armbruster

Multiple passes through mcgen() is prone to produce unwanted blank lines, which we then combat by sprinkling .rstrip() on top. Just don't do it.

---
 scripts/qapi-commands.py | 52 +++++++++++++++++++-----------------------------
 1 file changed, 21 insertions(+), 31 deletions(-)
scripts/qapi-commands.py +21 -31
index 6de5229..cfbd59c 100644
@@ -27,7 +27,7 @@ def generate_command_decl(name, args, ret_type):
 %(ret_type)s qmp_%(name)s(%(args)sError **errp);
 ''',
                  ret_type=c_type(ret_type), name=c_name(name),
-                 args=arglist).strip()
+                 args=arglist)
 
 def gen_err_check(err):
     if not err:
@@ -52,19 +52,17 @@ def gen_sync_call(name, args, ret_type):
     push_indent()
     ret = mcgen('''
 %(retval)sqmp_%(name)s(%(args)s&local_err);
-
 ''',
-                name=c_name(name), args=arglist, retval=retval).rstrip()
+                name=c_name(name), args=arglist, retval=retval)
     if ret_type:
-        ret += "\n" + gen_err_check('local_err')
+        ret += gen_err_check('local_err')
         ret += mcgen('''
 
 qmp_marshal_output_%(c_name)s(retval, ret, &local_err);
 ''',
                             c_name=c_name(name))
     pop_indent()
-    return ret.rstrip()
-
+    return ret
 
 def gen_visitor_input_containers_decl(args):
     ret = ""
@@ -78,7 +76,7 @@ Visitor *v;
 ''')
     pop_indent()
 
-    return ret.rstrip()
+    return ret
 
 def gen_visitor_input_vars_decl(args):
     ret = ""
@@ -101,7 +99,7 @@ bool has_%(argname)s = false;
                          argname=c_name(argname), argtype=c_type(argtype))
 
     pop_indent()
-    return ret.rstrip()
+    return ret
 
 def gen_visitor_input_block(args, dealloc=False):
     ret = ""
@@ -155,7 +153,7 @@ visit_type_%(visitor)s(v, &%(c_name)s, "%(name)s", %(errp)s);
 qapi_dealloc_visitor_cleanup(md);
 ''')
     pop_indent()
-    return ret.rstrip()
+    return ret
 
 def gen_marshal_output(name, ret_type):
     if not ret_type:
@@ -217,26 +215,17 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
                      retval=retval)
 
     if len(args) > 0:
-        ret += mcgen('''
-%(visitor_input_containers_decl)s
-%(visitor_input_vars_decl)s
-
-%(visitor_input_block)s
-
-''',
-                     visitor_input_containers_decl=gen_visitor_input_containers_decl(args),
-                     visitor_input_vars_decl=gen_visitor_input_vars_decl(args),
-                     visitor_input_block=gen_visitor_input_block(args))
+        ret += gen_visitor_input_containers_decl(args)
+        ret += gen_visitor_input_vars_decl(args) + '\n'
+        ret += gen_visitor_input_block(args) + '\n'
     else:
         ret += mcgen('''
 
     (void)args;
 ''')
 
-    ret += mcgen('''
-%(sync_call)s
-''',
-                 sync_call=gen_sync_call(name, args, ret_type))
+    ret += gen_sync_call(name, args, ret_type)
+
     if re.search('^ *goto out\\;', ret, re.MULTILINE):
         ret += mcgen('''
 
@@ -244,11 +233,11 @@ out:
 ''')
     ret += mcgen('''
     error_propagate(errp, local_err);
-%(visitor_input_block_cleanup)s
+''')
+    ret += gen_visitor_input_block(args, dealloc=True)
+    ret += mcgen('''
 }
-''',
-                 visitor_input_block_cleanup=gen_visitor_input_block(args,
-                                                                     dealloc=True))
+''')
     return ret
 
 def gen_registry(commands):
@@ -268,12 +257,13 @@ qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
     ret = mcgen('''
 static void qmp_init_marshal(void)
 {
-%(registry)s
+''')
+    ret += registry
+    ret += mcgen('''
 }
 
 qapi_init(qmp_init_marshal);
-''',
-                registry=registry.rstrip())
+''')
     return ret
 
 middle_mode = False
@@ -353,7 +343,7 @@ for cmd in commands:
         arglist = cmd['data']
     if cmd.has_key('returns'):
         ret_type = cmd['returns']
-    ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n"
+    ret = generate_command_decl(cmd['command'], arglist, ret_type)
     fdecl.write(ret)
     if ret_type:
         ret = gen_marshal_output(cmd['command'], ret_type) + "\n"
Signature · 1 line
1.9.3
Reviewed-by on v2 18/47
#60 lore raw

Re: [Qemu-devel] [PATCH RFC v2 18/47] qapi-commands: Don't feed output of mcgen() to mcgen() again

Quoted 8 lines · Markus Armbruster

Multiple passes through mcgen() is prone to produce unwanted blank lines, which we then combat by sprinkling .rstrip() on top. Just don't do it.

---
 scripts/qapi-commands.py | 52 +++++++++++++++++++-----------------------------
 1 file changed, 21 insertions(+), 31 deletions(-)

Causes the generated files to be slightly shorter (good), with diffs like the following:

--- qga/qapi-generated/qga-qmp-marshal.c.old 2015-07-20
17:13:06.056297347 -0600
+++ qga/qapi-generated/qga-qmp-marshal.c 2015-07-21
11:16:09.887228043 -0600
@@ -140,7 +140,6 @@

     (void)args;
     qmp_guest_ping(&local_err);
     error_propagate(errp, local_err);

-
}

static void qmp_marshal_output_guest_get_time(int64_t ret_in, QObject
**ret_out, Error **errp)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#61 lore raw

RFC PATCH v2 19/47 qapi: Generated code cleanup

In reply to Markus Armbruster

Clean up white-space, brace placement, and superfluous

---
 scripts/qapi-commands.py |  1 +
 scripts/qapi-event.py    |  3 +--
 scripts/qapi-types.py    | 66 +++++++++++++++++++++++-------------------------
 scripts/qapi-visit.py    |  1 +
 4 files changed, 34 insertions(+), 37 deletions(-)
scripts/qapi-commands.py +1 -0
index cfbd59c..223216d 100644
@@ -222,6 +222,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
         ret += mcgen('''
 
     (void)args;
+
 ''')
 
     ret += gen_sync_call(name, args, ret_type)
scripts/qapi-event.py +1 -2
index 88b0620..7f238df 100644
@@ -167,8 +167,7 @@ extern const char *%(event_enum_name)s_lookup[];
                         event_enum_name = event_enum_name)
 
     enum_decl = mcgen('''
-typedef enum %(event_enum_name)s
-{
+typedef enum %(event_enum_name)s {
 ''',
                       event_enum_name = event_enum_name)
 
scripts/qapi-types.py +31 -35

181 lines not on this page · raw message

scripts/qapi-visit.py +1 -0
index 73f136f..427819a 100644
@@ -62,6 +62,7 @@ def generate_visit_struct_fields(name, members, base = None):
 static void visit_type_%(name)s_fields(Visitor *m, %(name)s **obj, Error **errp)
 {
     Error *err = NULL;
+
 ''',
                  name=c_name(name))
     push_indent()
Signature · 1 line
1.9.3
#62 lore raw

Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup

Quoted 1 line · Markus Armbruster

Clean up white-space, brace placement, and superfluous

Incomplete sentence. I bet it's because your editor line-wrapped, and the rest of your sentence was something like '#ifndef in a .c file.' (see [1] below), then git ate the line thinking it was a comment. I think I'm okay with cramming all of these cleanups into one patch rather than trying to do one style of cleanup per patch (blank lines, { placement, and guard cleanup), but the commit message could do a better job of explaining things.

Quoted 8 lines
---
 scripts/qapi-commands.py |  1 +
 scripts/qapi-event.py    |  3 +--
 scripts/qapi-types.py    | 66 +++++++++++++++++++++++-------------------------
 scripts/qapi-visit.py    |  1 +
 4 files changed, 34 insertions(+), 37 deletions(-)

Missing changes to docs/qapi-code-gen.txt to reflect the improvements. Since having docs that are stale compared to implementation can be misleading, I'd like to wait for v2 before giving my R-b; but see also
[2] below.

Overall, I'm a fan of the cleanups.

I'm going to intersperse diffs of the generated files that were caused by some of these changes:

scripts/qapi-commands.py def gen_marshal_input(name, args, ret_type, middle_mode): · line 226

Quoted 11 lines
scripts/qapi-commands.py +1 -0
index cfbd59c..223216d 100644
@@ -222,6 +222,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
         ret += mcgen('''
 
     (void)args;
+
 ''')

This and similar hunts avoids extra blank lines. Not worth showing a diff to the generated files.

scripts/qapi-event.py extern const char *%(event_enum_name)s_lookup[]; · line 170

Quoted 13 lines
scripts/qapi-commands.py +0 -0
     ret += gen_sync_call(name, args, ret_type)
scripts/qapi-event.py +1 -2
index 88b0620..7f238df 100644
@@ -167,8 +167,7 @@ extern const char *%(event_enum_name)s_lookup[];
                         event_enum_name = event_enum_name)
 
     enum_decl = mcgen('''
-typedef enum %(event_enum_name)s
-{
+typedef enum %(event_enum_name)s {

Several hunks like this; gets us generally closer to qemu style; with generated diffs like:

qapi-types.h:
-typedef struct int32List
-{

+typedef struct int32List {
     union {
         int32_t value;
         uint64_t padding;

scripts/qapi-event.py struct %(name)s · line 107

Quoted 7 lines
scripts/qapi-event.py +2 -1
@@ -105,7 +103,8 @@ struct %(name)s
 
 def generate_enum_lookup(name, values):
     ret = mcgen('''
-const char * const %(name)s_lookup[] = {
+
+const char *const %(name)s_lookup[] = {

[2] generated diffs like this:

qapi-types.c:
-const char * const OnOffAuto_lookup[] = {
+const char *const OnOffAuto_lookup[] = {

Hmm - we already failed to update docs/qapi-code-gen.txt in the past; we added a const in commit 2e4450ff that is missing from the documentation.

scripts/qapi-event.py for typename in builtin_types.keys(): · line 342

Quoted 13 lines
scripts/qapi-event.py +2 -2
@@ -335,22 +333,22 @@ for typename in builtin_types.keys():
 fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
 
 for expr in exprs:
-    ret = "\n"
+    ret = ""
     if expr.has_key('struct'):
         ret += generate_fwd_struct(expr['struct'])
     elif expr.has_key('enum'):
-        ret += generate_enum(expr['enum'], expr['data']) + "\n"
+        ret += generate_enum(expr['enum'], expr['data'])
         ret += generate_fwd_enum_struct(expr['enum'])
         fdef.write(generate_enum_lookup(expr['enum'], expr['data']))

More work at avoiding extra blank lines.

scripts/qapi-event.py fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) · line 376

Quoted 8 lines
scripts/qapi-event.py +0 -2
@@ -370,34 +368,32 @@ fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
 # have the functions defined, so we use -b option to provide control
 # over these cases
 if do_builtins:
-    fdef.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))
     for typename in builtin_types.keys():
         fdef.write(generate_type_cleanup(typename + "List"))
-    fdef.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))

[1] this is the hunk whose description got corrupted in your commit message. It gives a diff like this:

qapi-types.c:

-
-#ifndef QAPI_TYPES_BUILTIN_CLEANUP_DEF
-#define QAPI_TYPES_BUILTIN_CLEANUP_DEF
-
-
void qapi_free_int32List(int32List *obj)

We conditionally declare the functions in the .h, but the .c is not compiled multiple times, so there is no need for a header-style guard.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#63 lore raw

Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup

Quoted 10 lines · Eric Blake
Quoted 1 line · Markus Armbruster

Clean up white-space, brace placement, and superfluous

Incomplete sentence. I bet it's because your editor line-wrapped, and the rest of your sentence was something like '#ifndef in a .c file.' (see [1] below), then git ate the line thinking it was a comment. I think I'm okay with cramming all of these cleanups into one patch rather than trying to do one style of cleanup per patch (blank lines, { placement, and guard cleanup), but the commit message could do a better job of explaining things.

Not throwing away half of it by accident would be a start :)

Quoted 9 lines
Quoted 8 lines
---
 scripts/qapi-commands.py |  1 +
 scripts/qapi-event.py    |  3 +--
 scripts/qapi-types.py | 66
+++++++++++++++++++++++-------------------------
 scripts/qapi-visit.py    |  1 +
 4 files changed, 34 insertions(+), 37 deletions(-)
Quoted 4 lines

Missing changes to docs/qapi-code-gen.txt to reflect the improvements. Since having docs that are stale compared to implementation can be misleading, I'd like to wait for v2 before giving my R-b; but see also
[2] below.

I agree we should keep docs/qapi-code-gen.txt up-to-date.

Since updating it is manual, updating it just once for a complete series can make sense.

I'll double check it's accurate at the end of the series, then decide how to do necessary updates, if any.

Quoted 59 lines

Overall, I'm a fan of the cleanups.

I'm going to intersperse diffs of the generated files that were caused by some of these changes:

scripts/qapi-commands.py def gen_marshal_input(name, args, ret_type, middle_mode): · line 226

Quoted 11 lines
scripts/qapi-commands.py +1 -0
index cfbd59c..223216d 100644
@@ -222,6 +222,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
         ret += mcgen('''
 
     (void)args;
+
 ''')

This and similar hunts avoids extra blank lines. Not worth showing a diff to the generated files.

scripts/qapi-event.py extern const char *%(event_enum_name)s_lookup[]; · line 170

Quoted 13 lines
scripts/qapi-commands.py +0 -0
     ret += gen_sync_call(name, args, ret_type)
scripts/qapi-event.py +1 -2
index 88b0620..7f238df 100644
@@ -167,8 +167,7 @@ extern const char *%(event_enum_name)s_lookup[];
                         event_enum_name = event_enum_name)
 
     enum_decl = mcgen('''
-typedef enum %(event_enum_name)s
-{
+typedef enum %(event_enum_name)s {

Several hunks like this; gets us generally closer to qemu style; with generated diffs like:

qapi-types.h:

scripts/qapi-types.py +1 -2
-typedef struct int32List
-{
+typedef struct int32List {
     union {
         int32_t value;
         uint64_t padding;

scripts/qapi-event.py struct %(name)s · line 107

Quoted 8 lines
scripts/qapi-event.py +2 -1
@@ -105,7 +103,8 @@ struct %(name)s
 
 def generate_enum_lookup(name, values):
     ret = mcgen('''
-const char * const %(name)s_lookup[] = {
+
+const char *const %(name)s_lookup[] = {

[2] generated diffs like this:

qapi-types.c:

(quoted diff) +1 -1
-const char * const OnOffAuto_lookup[] = {
+const char *const OnOffAuto_lookup[] = {
Quoted 2 lines

Hmm - we already failed to update docs/qapi-code-gen.txt in the past; we added a const in commit 2e4450ff that is missing from the documentation.

Minor review fail. Not the first time.

Quoted 37 lines

scripts/qapi-event.py for typename in builtin_types.keys(): · line 342

Quoted 13 lines
scripts/qapi-event.py +2 -2
@@ -335,22 +333,22 @@ for typename in builtin_types.keys():
 fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
 
 for expr in exprs:
-    ret = "\n"
+    ret = ""
     if expr.has_key('struct'):
         ret += generate_fwd_struct(expr['struct'])
     elif expr.has_key('enum'):
-        ret += generate_enum(expr['enum'], expr['data']) + "\n"
+        ret += generate_enum(expr['enum'], expr['data'])
         ret += generate_fwd_enum_struct(expr['enum'])
         fdef.write(generate_enum_lookup(expr['enum'], expr['data']))

More work at avoiding extra blank lines.

scripts/qapi-event.py fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL")) · line 376

Quoted 8 lines
scripts/qapi-event.py +0 -2
@@ -370,34 +368,32 @@ fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
 # have the functions defined, so we use -b option to provide control
 # over these cases
 if do_builtins:
-    fdef.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))
     for typename in builtin_types.keys():
         fdef.write(generate_type_cleanup(typename + "List"))
-    fdef.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DEF"))

[1] this is the hunk whose description got corrupted in your commit message. It gives a diff like this:

qapi-types.c:

(quoted diff) +0 -5
-
-#ifndef QAPI_TYPES_BUILTIN_CLEANUP_DEF
-#define QAPI_TYPES_BUILTIN_CLEANUP_DEF
-
-
 void qapi_free_int32List(int32List *obj)
Quoted 2 lines

We conditionally declare the functions in the .h, but the .c is not compiled multiple times, so there is no need for a header-style guard.

This is going to help me reconstruct my crippled commit message, thanks!

#64 lore raw

Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup

Quoted 3 lines · Markus Armbruster
Quoted 1 line · Eric Blake

On 07/01/2015 02:22 PM, Markus Armbruster wrote:

[...]

Quoted 10 lines
Quoted 10 lines

scripts/qapi-types.py struct %(name)s · line 107

Quoted 7 lines
scripts/qapi-types.py +2 -1
@@ -105,7 +103,8 @@ struct %(name)s
 
 def generate_enum_lookup(name, values):
     ret = mcgen('''
-const char * const %(name)s_lookup[] = {
+
+const char *const %(name)s_lookup[] = {

[2] generated diffs like this:

Quoted 8 lines
Quoted 6 lines

qapi-types.c:

(quoted diff) +1 -1
-const char * const OnOffAuto_lookup[] = {
+const char *const OnOffAuto_lookup[] = {

Hmm - we already failed to update docs/qapi-code-gen.txt in the past; we added a const in commit 2e4450ff that is missing from the documentation.

Minor review fail. Not the first time.

I take that back, it's actually not visible in qapi-code-gen.txt.

[...]

#65 lore raw

Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup

Quoted 20 lines · Markus Armbruster
Quoted 3 lines · Markus Armbruster
Quoted 1 line · Eric Blake

On 07/01/2015 02:22 PM, Markus Armbruster wrote:

[...]

Quoted 14 lines
Quoted 14 lines

scripts/qapi-types.py struct %(name)s · line 107

Quoted 7 lines
scripts/qapi-types.py +2 -1
@@ -105,7 +103,8 @@ struct %(name)s
 
 def generate_enum_lookup(name, values):
     ret = mcgen('''
-const char * const %(name)s_lookup[] = {
+
+const char *const %(name)s_lookup[] = {

[2] generated diffs like this:

qapi-types.c:

(quoted diff) +1 -1
-const char * const OnOffAuto_lookup[] = {
+const char *const OnOffAuto_lookup[] = {
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Hmm - we already failed to update docs/qapi-code-gen.txt in the past; we added a const in commit 2e4450ff that is missing from the documentation.

Minor review fail. Not the first time.

I take that back, it's actually not visible in qapi-code-gen.txt.

Oh, you're right - the only mention of *_lookup[] in the docs file is the array generated for all events, and that one was output with correct spacing (until the rest of your series fixes it to share code rather than duplicate things with slight differences).

Maybe we SHOULD be showing what the generators do for enums, unions, and alternates (by expanding the example-schema.json that is then fed to all the example script usage). But that's fine as a project for another day.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#66 lore raw

Re: [Qemu-devel] [PATCH RFC v2 19/47] qapi: Generated code cleanup

Quoted 28 lines · Eric Blake
Quoted 20 lines · Markus Armbruster
Quoted 3 lines · Markus Armbruster
Quoted 1 line · Eric Blake

On 07/01/2015 02:22 PM, Markus Armbruster wrote:

[...]

Quoted 14 lines
Quoted 14 lines

scripts/qapi-types.py struct %(name)s · line 107

Quoted 7 lines
scripts/qapi-types.py +2 -1
@@ -105,7 +103,8 @@ struct %(name)s
 
 def generate_enum_lookup(name, values):
     ret = mcgen('''
-const char * const %(name)s_lookup[] = {
+
+const char *const %(name)s_lookup[] = {

[2] generated diffs like this:

qapi-types.c:

(quoted diff) +1 -1
-const char * const OnOffAuto_lookup[] = {
+const char *const OnOffAuto_lookup[] = {
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Hmm - we already failed to update docs/qapi-code-gen.txt in the past; we added a const in commit 2e4450ff that is missing from the documentation.

Minor review fail. Not the first time.

I take that back, it's actually not visible in qapi-code-gen.txt.

Quoted 8 lines

Oh, you're right - the only mention of *_lookup[] in the docs file is the array generated for all events, and that one was output with correct spacing (until the rest of your series fixes it to share code rather than duplicate things with slight differences).

Maybe we SHOULD be showing what the generators do for enums, unions, and alternates (by expanding the example-schema.json that is then fed to all the example script usage). But that's fine as a project for another day.

The place where we try to exercise all the schema features is qapi-schema-test.json.

#67 lore raw

RFC PATCH v2 20/47 qapi: Rename class QAPISchema to QAPISchemaParser

In reply to Markus Armbruster

I want to name a new class QAPISchema.

While there, make it a new-style class.

---
 scripts/qapi.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi.py +4 -4
index bebaecc..390ccd0 100644
@@ -103,7 +103,7 @@ class QAPIExprError(Exception):
         return error_path(self.info['parent']) + \
             "%s:%d: %s" % (self.info['file'], self.info['line'], self.msg)
 
-class QAPISchema:
+class QAPISchemaParser(object):
 
     def __init__(self, fp, previously_included = [], incl_info = None):
         abs_fname = os.path.abspath(fp.name)
@@ -149,8 +149,8 @@ class QAPISchema:
                 except IOError, e:
                     raise QAPIExprError(expr_info,
                                         '%s: %s' % (e.strerror, include))
-                exprs_include = QAPISchema(fobj, previously_included,
-                                           expr_info)
+                exprs_include = QAPISchemaParser(fobj, previously_included,
+                                                 expr_info)
                 self.exprs.extend(exprs_include.exprs)
             else:
                 expr_elem = {'expr': expr,
@@ -755,7 +755,7 @@ def check_exprs(exprs):
 
 def parse_schema(fname):
     try:
-        schema = QAPISchema(open(fname, "r"))
+        schema = QAPISchemaParser(open(fname, "r"))
         return check_exprs(schema.exprs)
     except (QAPISchemaError, QAPIExprError), e:
         print >>sys.stderr, e
Signature · 1 line
1.9.3
Reviewed-by on v2 20/47
#68 lore raw

Re: [Qemu-devel] [PATCH RFC v2 20/47] qapi: Rename class QAPISchema to QAPISchemaParser

Quoted 8 lines · Markus Armbruster

I want to name a new class QAPISchema.

While there, make it a new-style class.

---
 scripts/qapi.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#69 lore raw

RFC PATCH v2 21/47 qapi: New QAPISchema intermediate reperesentation

In reply to Markus Armbruster

The QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaties) on the side.

They have clearly outgrown these simple data structures. There's lots of rummaging around in dictionaries, and information is recomputed on the fly. For the work I'm going to do, I want more clearly defined and more convenient interfaces.

Going forward, I also want less coupling between the back-ends and the syntax tree, to make messing with the syntax easier.

Create a bunch of classes to represent QAPI schemata.

Have the QAPISchema initializer call the parser, then walk the syntax tree to create the new internal representation, and finally perform semantic analysis.

Shortcut: the semantic analysis still relies on existing check_exprs() to do the actual semantic checking. All this code needs to move into the classes. Mark as TODO.

We generate array types eagerly, even though most of them aren't used. Mark as TODO.

---
 scripts/qapi-commands.py       |   2 +-
 scripts/qapi-event.py          |   2 +-
 scripts/qapi-types.py          |   2 +-
 scripts/qapi-visit.py          |   2 +-
 scripts/qapi.py                | 351 +++++++++++++++++++++++++++++++++++++++--
 tests/qapi-schema/test-qapi.py |   2 +-
 6 files changed, 347 insertions(+), 14 deletions(-)
scripts/qapi-commands.py +1 -1
index 223216d..3cdbd97 100644
@@ -276,7 +276,7 @@ for o, a in opts:
     if o in ("-m", "--middle"):
         middle_mode = True
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 commands = filter(lambda expr: expr.has_key('command'), exprs)
 commands = filter(lambda expr: not expr.has_key('gen'), commands)
 
scripts/qapi-event.py +1 -1
index 7f238df..aec2d32 100644
@@ -263,7 +263,7 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 
 event_enum_name = c_name(prefix + "QAPIEvent", protect=False)
 event_enum_values = []
scripts/qapi-types.py +1 -1
index 4cc17da..a48ad9c 100644
@@ -325,7 +325,7 @@ fdecl.write(mcgen('''
 #include <stdint.h>
 '''))
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 
 fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
 for typename in builtin_types.keys():
scripts/qapi-visit.py +1 -1
index 427819a..a52a572 100644
@@ -442,7 +442,7 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 
 # to avoid header dependency hell, we always generate declarations
 # for built-in types in our header files and simply guard them
scripts/qapi.py +342 -9

363 lines not on this page · raw message

tests/qapi-schema/test-qapi.py +1 -1
index 634ef2d..461c713 100644
@@ -16,7 +16,7 @@ import os
 import sys
 
 try:
-    exprs = parse_schema(sys.argv[1])
+    exprs = QAPISchema(sys.argv[1]).get_exprs()
 except SystemExit:
     raise
 
Signature · 1 line
1.9.3
#70 lore raw

Re: [Qemu-devel] [PATCH RFC v2 21/47] qapi: New QAPISchema intermediate reperesentation

Quoted 2 lines · Markus Armbruster

The QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaties) on the side.

s/dictionaties/dictionaries/

Quoted 8 lines

They have clearly outgrown these simple data structures. There's lots of rummaging around in dictionaries, and information is recomputed on the fly. For the work I'm going to do, I want more clearly defined and more convenient interfaces.

Going forward, I also want less coupling between the back-ends and the syntax tree, to make messing with the syntax easier.

Indeed - should be a lot easier to add new qapi .json syntax sugar on the front-end, or to tweak generated C layout on the backend, without close coupling between the two. Particularly when adding front-end sugar extensions that still normalize into the same backend structs.

Quoted 7 lines

Create a bunch of classes to represent QAPI schemata.

Have the QAPISchema initializer call the parser, then walk the syntax tree to create the new internal representation, and finally perform semantic analysis.

Quoted 7 lines

Shortcut: the semantic analysis still relies on existing check_exprs() to do the actual semantic checking. All this code needs to move into the classes. Mark as TODO.

We generate array types eagerly, even though most of them aren't used. Mark as TODO.

No change to generated files at this stage in the game (this is mostly additive, then later patches shed their old ways of doing things by using this). Good division of labor between patches.

Quoted 29 lines
---
 scripts/qapi-commands.py       |   2 +-
 scripts/qapi-event.py          |   2 +-
 scripts/qapi-types.py          |   2 +-
 scripts/qapi-visit.py          |   2 +-
 scripts/qapi.py                | 351 +++++++++++++++++++++++++++++++++++++++--
 tests/qapi-schema/test-qapi.py |   2 +-
 6 files changed, 347 insertions(+), 14 deletions(-)

+++ b/scripts/qapi.py

scripts/qapi.py +16 -0
+
+#
+# Schema compiler frontend
+#
+
+class QAPISchemaEntity(object):
+    def __init__(self, name, info):
+        assert isinstance(name, str)
+        self.name = name
+        self.info = info
+    def check(self, schema):
+        pass
+
+class QAPISchemaType(QAPISchemaEntity):
+    pass
+

scripts/qapi.py def check_exprs(exprs): · line 781

Quoted 10 lines
scripts/qapi.py +10 -0
+class QAPISchemaBuiltinType(QAPISchemaType):
+    def __init__(self, name):
+        QAPISchemaType.__init__(self, name, None)
+
+class QAPISchemaEnumType(QAPISchemaType):
+    def __init__(self, name, info, values):
+        QAPISchemaType.__init__(self, name, info)
+        for v in values:
+            assert isinstance(v, str)
+        self.values = values

worth a check() method to ensure values don't collide? Especially since you already do that in QAPISchemaObjectTypeMember.check()

Quoted 19 lines
scripts/qapi.py +19 -0
+
+class QAPISchemaArrayType(QAPISchemaType):
+    def __init__(self, name, info, element_type):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(element_type, str)
+        self.element_type_name = element_type
+        self.element_type = None
+    def check(self, schema):
+        self.element_type = schema.lookup_type(self.element_type_name)
+        assert self.element_type
+
+class QAPISchemaObjectType(QAPISchemaType):
+    def __init__(self, name, info, base, local_members, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert base == None or isinstance(base, str)
+        for m in local_members:
+            assert isinstance(m, QAPISchemaObjectTypeMember)
+        if variants != None:
+            assert isinstance(variants, QAPISchemaObjectTypeVariants)

scripts/qapi.py def check_exprs(exprs): · line 810

Quoted 10 lines
scripts/qapi.py +10 -0
+        self.base_name = base
+        self.base = None
+        self.local_members = local_members
+        self.variants = variants
+        self.members = None
+    def check(self, schema):
+        if self.members:
+            return                      # already checked
+        assert self.members == None     # not running in cycles
+        self.members = False            # mark as being checked

Interesting, but makes sense (since we allow self-referential nesting, populating our own members may require revisiting the same type). Cute that python allows both None and False as distinct non-true settings, so that you end up using the variable as a tri-state.

scripts/qapi.py def check_exprs(exprs): · line 813

Quoted 3 lines
scripts/qapi.py +3 -0
+        if self.base_name:
+            self.base = schema.lookup_type(self.base_name)
+            self.base.check(schema)

Do you need 'assert self.base' here, similar to how you did it in QAPISchemaArrayType.check()? I guess you'll still get a python exception that None.check() doesn't exist if the lookup_type failed, so it just depends on what error message you want.

Quoted 1 line

+ members = list(self.base.members)

For that matter, do you want to assert isinstance(self.base, QAPISchemaObjectType), since lookup_type can return other types, but other child classes of QAPISchemaType do not have a .members?

scripts/qapi.py def check_exprs(exprs): · line 826

Quoted 12 lines
scripts/qapi.py +12 -0
+        else:
+            members = []
+        seen = {}
+        for m in members:
+            seen[m.name] = m
+        for m in self.local_members:
+            m.check(schema, members, seen)
+        if self.variants:
+            self.variants.check(schema, members, seen)
+        self.members = members
+
+class QAPISchemaObjectTypeMember(object):

Is 'object' the right base class, or should it be QAPISchemaEntity?

scripts/qapi.py def check_exprs(exprs): · line 829

Quoted 3 lines
scripts/qapi.py +3 -0
+    def __init__(self, name, typ, optional):
+        assert isinstance(name, str)
+        assert isinstance(typ, str)

I guess that means all callers flatten array types into an unambiguous string first.

scripts/qapi.py def check_exprs(exprs): · line 842

Quoted 13 lines
scripts/qapi.py +13 -0
+        assert isinstance(optional, bool)
+        self.name = name
+        self.type_name = typ
+        self.type = None
+        self.optional = optional
+    def check(self, schema, all_members, seen):
+        assert self.name not in seen
+        self.type = schema.lookup_type(self.type_name)
+        assert self.type
+        all_members.append(self)
+        seen[self.name] = self
+
+class QAPISchemaObjectTypeVariants(object):

Is 'object' the right base class, or should it be QAPISchemaEntity?

scripts/qapi.py def check_exprs(exprs): · line 854

Quoted 12 lines
scripts/qapi.py +12 -0
+    def __init__(self, tag_name, tag_enum, variants):
+        assert tag_name == None or isinstance(tag_name, str)
+        assert tag_enum == None or isinstance(tag_enum, str)
+        for v in variants:
+            assert isinstance(v, QAPISchemaObjectTypeVariant)
+        self.tag_name = tag_name
+        if tag_name:
+            assert not tag_enum
+            self.tag_member = None
+        else:
+            self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum,
+                                                         False)

Nice - the tag is either part of the base struct (flat union) or implicitly created (plain union). By the time we are done with check(), we then have both its type (possibly generated implicitly) and its name.

scripts/qapi.py def check_exprs(exprs): · line 860

Quoted 6 lines
scripts/qapi.py +6 -0
+        self.variants = variants
+    def check(self, schema, members, seen):
+        if self.tag_name:
+            self.tag_member = seen[self.tag_name]
+        else:
+            self.tag_member.check(schema, members, seen)

A little bit tricky here in that flat unions (tag_name was set) must find the tag from the members already loaded from the base class, while plain unions (tag_name is None) add their own implicit member into the union. But it works.

Quoted 6 lines
scripts/qapi.py +6 -0
+        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        for v in self.variants:
+            vseen = dict(seen)
+            v.check(schema, self.tag_member.type, vseen)
+
+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):

scripts/qapi.py def check_exprs(exprs): · line 876

Quoted 10 lines
scripts/qapi.py +10 -0
+    def __init__(self, name, typ, flat):
+        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
+        assert isinstance(flat, bool)
+        self.flat = flat
+    def check(self, schema, tag_type, seen):
+        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        assert self.name in tag_type.values
+        if self.flat:
+            self.type.check(schema)
+            assert isinstance(self.type, QAPISchemaObjectType)

Do we really want to be tracking self.flat for each variant? After all, docs/qapi-code-gen.txt already describes the mapping from simple union into flat union (as if the flat union had a base class with single member 'kind' of the right type, then each branch of the union composed of an implicit object with a lone member 'data' of the correct type). In other words, is it any better to just normalize into that form now, such that each QAPISchemaObjectTypeVariant is merely a (often one-element) list of name:type members being added to the overall QAPISchemaObject? But I guess it remains to be seen how you use self.flat before knowing if it is worth normalizing away from it.

scripts/qapi.py def check_exprs(exprs): · line 884

Quoted 8 lines
scripts/qapi.py +8 -0
+
+class QAPISchemaAlternateType(QAPISchemaType):
+    def __init__(self, name, info, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(variants, QAPISchemaObjectTypeVariants)
+        assert not variants.tag_name
+        for v in variants.variants:
+            assert not v.flat

Hmm. You are using .flat after all.

scripts/qapi.py def check_exprs(exprs): · line 893

Quoted 9 lines
scripts/qapi.py +9 -0
+        self.variants = variants
+    def check(self, schema):
+        self.variants.check(schema, [], {})
+
+class QAPISchemaCommand(QAPISchemaEntity):
+    def __init__(self, name, info, args, rets, gen, success_response):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not args or isinstance(args, str)
+        assert not rets or isinstance(rets, str)

Again, this means that the caller has already had to convert qapi dictionaries into an implicit type and pass in that type name here. Works for me.

Quoted 6 lines
scripts/qapi.py +6 -0
+        self.args_name = args
+        self.args = None
+        self.rets_name = rets
+        self.rets = None
+        self.gen = gen
+        self.success_response = success_response

scripts/qapi.py def check_exprs(exprs): · line 909

Quoted 10 lines
scripts/qapi.py +10 -0
+    def check(self, schema):
+        if self.args_name:
+            self.args = schema.lookup_type(self.args_name)
+            assert isinstance(self.args, QAPISchemaObjectType)
+            assert not self.args.variants # not implemented
+            self.args.check(schema)
+        if self.rets_name:
+            self.rets = schema.lookup_type(self.rets_name)
+            assert isinstance(self.rets, QAPISchemaType)
+            self.rets.check(schema)

Matches our existing difference in requiring a dictionary for arguments, but allowing the return of any type (although we now have a whitelist to prevent too many additions of non-dictionaries).

Quoted 27 lines
scripts/qapi.py +27 -0
+
+class QAPISchemaEvent(QAPISchemaEntity):
+    def __init__(self, name, info, data):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not data or isinstance(data, str)
+        self.data_name = data
+        self.data = None
+    def check(self, schema):
+        if self.data_name:
+            self.data = schema.lookup_type(self.data_name)
+            assert isinstance(self.data, QAPISchemaObjectType)
+            self.data.check(schema)
+
+class QAPISchema(object):
+    def __init__(self, fname):
+        try:
+            self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
+        except (QAPISchemaError, QAPIExprError), err:
+            print >>sys.stderr, err
+            exit(1)
+        self.entity_dict = {}
+        self._def_predefineds()
+        self._def_exprs()
+        self.check()
+
+    def get_exprs(self):
+        return [expr_elem['expr'] for expr_elem in self.exprs]

scripts/qapi.py def check_exprs(exprs): · line 946

Quoted 10 lines
scripts/qapi.py +10 -0
+
+    def _def_entity(self, ent):
+        assert ent.name not in self.entity_dict
+        self.entity_dict[ent.name] = ent
+
+    def lookup_entity(self, name, typ=None):
+        ent = self.entity_dict.get(name)
+        if typ and not isinstance(ent, typ):
+            return None
+        return ent

Ah, so you enforce a shared namespace between types, commands, and events (all three are in self.entity_dict), but can use the typ parameter to allow limiting a lookup to just types.

Quoted 19 lines
scripts/qapi.py +19 -0
+
+    def lookup_type(self, name):
+        return self.lookup_entity(name, QAPISchemaType)
+
+    def _def_builtin_type(self, name):
+        self._def_entity(QAPISchemaBuiltinType(name))
+        if name != '**':
+            self._make_array_type(name) # TODO really needed?
+
+    def _def_predefineds(self):
+        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
+                  'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
+            self._def_builtin_type(t)
+
+    def _make_implicit_enum_type(self, name, values):
+        name = name + 'Kind'
+        self._def_entity(QAPISchemaEnumType(name, None, values))
+        return name
+

scripts/qapi.py def check_exprs(exprs): · line 975

Quoted 10 lines
scripts/qapi.py +10 -0
+    def _make_array_type(self, element_type):
+        name = element_type + 'List'
+        if not self.lookup_type(name):
+            self._def_entity(QAPISchemaArrayType(name, None, element_type))
+        return name
+
+    def _make_implicit_object_type(self, name, role, members):
+        name = ':obj-%s-%s' % (name, role)
+        self._def_entity(QAPISchemaObjectType(name, None, None, members, None))
+        return name

We may extend the qapi .json syntax to allow anonymous types in more places, but so far, this matches that all existing use of anonymous types are for structs, not unions.

Quoted 39 lines
scripts/qapi.py +39 -0
+
+    def _def_enum_type(self, expr, info):
+        name = expr['enum']
+        data = expr['data']
+        self._def_entity(QAPISchemaEnumType(name, info, data))
+        self._make_array_type(name) # TODO really needed?
+
+    def _make_member(self, name, typ):
+        optional = False
+        if name.startswith('*'):
+            name = name[1:]
+            optional = True
+        if isinstance(typ, list):
+            assert len(typ) == 1
+            typ = self._make_array_type(typ[0])
+        return QAPISchemaObjectTypeMember(name, typ, optional)
+
+    def _make_members(self, data):
+        return [self._make_member(key, data[key]) for key in data.keys()]
+
+    def _def_struct_type(self, expr, info):
+        name = expr['struct']
+        base = expr.get('base')
+        data = expr['data']
+        self._def_entity(QAPISchemaObjectType(name, info, base,
+                                              self._make_members(data),
+                                              None))
+        self._make_array_type(name) # TODO really needed?
+
+    def _make_flat_variant(self, case, typ):
+        return QAPISchemaObjectTypeVariant(case, typ, True)
+
+    def _make_flat_variants(self, tag_name, data):
+        variants = [self._make_flat_variant(key, data[key])
+                    for key in data.keys()]
+        return QAPISchemaObjectTypeVariants(tag_name, None, variants)
+
+    def _make_simple_variant(self, case, typ):
+        if isinstance(typ, list):

scripts/qapi.py def check_exprs(exprs): · line 1024

Quoted 10 lines
scripts/qapi.py +10 -0
+            assert len(typ) == 1
+            typ = self._make_array_type(typ[0])
+        return QAPISchemaObjectTypeVariant(case, typ, False)
+
+    def _make_simple_variants(self, type_name, data):
+        variants = [self._make_simple_variant(key, data[key])
+                    for key in data.keys()]
+        enum = self._make_implicit_enum_type(type_name,
+                                             [v.name for v in variants])
+        return QAPISchemaObjectTypeVariants(None, enum, variants)

Again, I wonder if normalizing simple unions into flat unions (to get rid of the need for tracking .flat) would make later use any easier, or if it would just make introspection QMP output more verbose.

Quoted 22 lines
scripts/qapi.py +22 -0
+
+    def _def_union_type(self, expr, info):
+        name = expr['union']
+        data = expr['data']
+        tag_name = expr.get('discriminator')
+        if tag_name:
+            base = expr['base']
+            variants = self._make_flat_variants(tag_name, data)
+        else:
+            base = None
+            variants = self._make_simple_variants(name, data)
+        self._def_entity(QAPISchemaObjectType(name, info, base,
+                                              self._make_members(OrderedDict()),
+                                              variants))
+        self._make_array_type(name) # TODO really needed?
+
+    def _def_alternate_type(self, expr, info):
+        name = expr['alternate']
+        data = expr['data']
+        self._def_entity(QAPISchemaAlternateType(name, info,
+                                    self._make_simple_variants(name, data)))
+        self._make_array_type(name) # TODO really needed?

scripts/qapi.py def check_exprs(exprs): · line 1056

Quoted 10 lines
scripts/qapi.py +10 -0
+
+    def _def_command(self, expr, info):
+        name = expr['command']
+        args = expr.get('data')
+        rets = expr.get('returns')
+        gen = expr.get('gen', True)
+        success_response = expr.get('success-response', True)
+        if args and not isinstance(args, str):
+            args = self._make_implicit_object_type(name, 'args',
+                                                   self._make_members(args))

If I write { 'command':'Foo', 'data':{} }, does that try and create an implicit type, with no members? I'm wondering if it is any simpler to write args = expr.get('data', {}), and have _make_implicit_object_type() do the right thing when presented with an empty list of members.

Quoted 29 lines
scripts/qapi.py +29 -0
+        if rets and isinstance(rets, list):
+            assert len(rets) == 1
+            rets = self._make_array_type(rets[0])
+        elif rets and not isinstance(rets, str):
+            rets = self._make_implicit_object_type(name, 'rets',
+                                                   self._make_members(rets))
+        self._def_entity(QAPISchemaCommand(name, info, args, rets, gen,
+                                           success_response))
+
+    def _def_event(self, expr, info):
+        name = expr['event']
+        data = expr.get('data')
+        if data and not isinstance(data, str):
+            data = self._make_implicit_object_type(name, 'data',
+                                                   self._make_members(data))
+        self._def_entity(QAPISchemaEvent(name, info, data))
+
+    def _def_exprs(self):
+        for expr_elem in self.exprs:
+            expr = expr_elem['expr']
+            info = expr_elem['info']
+            if 'enum' in expr:
+                self._def_enum_type(expr, info)
+            elif 'struct' in expr:
+                self._def_struct_type(expr, info)
+            elif 'union' in expr:
+                self._def_union_type(expr, info)
+            elif 'alternate' in expr:
+                self._def_alternate_type(expr, info)

scripts/qapi.py def check_exprs(exprs): · line 1095

Quoted 10 lines
scripts/qapi.py +10 -0
+            elif 'command' in expr:
+                self._def_command(expr, info)
+            elif 'event' in expr:
+                self._def_event(expr, info)
+            else:
+                assert False
+
+    def check(self):
+        for ent in self.entity_dict.values():
+            ent.check(self)

Overall seems like a very nice layout.

I'll reserve R-b until I see how the rest of the series uses this object hierarchy, but it looks like you won't have to change very much of this patch when you do your next spin of the series.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#71 lore raw

Re: [Qemu-devel] [PATCH RFC v2 21/47] qapi: New QAPISchema intermediate reperesentation

Quoted 5 lines · Eric Blake
Quoted 2 lines · Markus Armbruster

The QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaties) on the side.

s/dictionaties/dictionaries/

Will fix.

Quoted 24 lines
Quoted 7 lines

They have clearly outgrown these simple data structures. There's lots of rummaging around in dictionaries, and information is recomputed on the fly. For the work I'm going to do, I want more clearly defined and more convenient interfaces.

Going forward, I also want less coupling between the back-ends and the syntax tree, to make messing with the syntax easier.

Indeed - should be a lot easier to add new qapi .json syntax sugar on the front-end, or to tweak generated C layout on the backend, without close coupling between the two. Particularly when adding front-end sugar extensions that still normalize into the same backend structs.

Quoted 11 lines

Create a bunch of classes to represent QAPI schemata.

Have the QAPISchema initializer call the parser, then walk the syntax tree to create the new internal representation, and finally perform semantic analysis.

Shortcut: the semantic analysis still relies on existing check_exprs() to do the actual semantic checking. All this code needs to move into the classes. Mark as TODO.

Quoted 7 lines
Quoted 3 lines

We generate array types eagerly, even though most of them aren't used. Mark as TODO.

No change to generated files at this stage in the game (this is mostly additive, then later patches shed their old ways of doing things by using this). Good division of labor between patches.

I'll steal from this paragraph for my commit message.

Quoted 34 lines
Quoted 34 lines
---
 scripts/qapi-commands.py       |   2 +-
 scripts/qapi-event.py          |   2 +-
 scripts/qapi-types.py          |   2 +-
 scripts/qapi-visit.py          |   2 +-
 scripts/qapi.py                | 351 +++++++++++++++++++++++++++++++++++++++--
 tests/qapi-schema/test-qapi.py |   2 +-
 6 files changed, 347 insertions(+), 14 deletions(-)

+++ b/scripts/qapi.py

scripts/qapi.py +21 -0
+
+#
+# Schema compiler frontend
+#
+
+class QAPISchemaEntity(object):
+    def __init__(self, name, info):
+        assert isinstance(name, str)
+        self.name = name
+        self.info = info
+    def check(self, schema):
+        pass
+
+class QAPISchemaType(QAPISchemaEntity):
+    pass
+
+class QAPISchemaBuiltinType(QAPISchemaType):
+    def __init__(self, name):
+        QAPISchemaType.__init__(self, name, None)
+
+class QAPISchemaEnumType(QAPISchemaType):
Quoted 8 lines

scripts/qapi.py def check_exprs(exprs): · line 781

Quoted 5 lines
scripts/qapi.py +5 -0
+    def __init__(self, name, info, values):
+        QAPISchemaType.__init__(self, name, info)
+        for v in values:
+            assert isinstance(v, str)
+        self.values = values

worth a check() method to ensure values don't collide? Especially since you already do that in QAPISchemaObjectTypeMember.check()

Can do.

Aside: everything the check() methods assert must be established by semantic analysis. Moving semantic analysis into the classes as outlined in the commit message will hopefully make that obvious enough to permit dropping the assertions.

Quoted 24 lines
Quoted 24 lines
scripts/qapi.py +24 -0
+
+class QAPISchemaArrayType(QAPISchemaType):
+    def __init__(self, name, info, element_type):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(element_type, str)
+        self.element_type_name = element_type
+        self.element_type = None
+    def check(self, schema):
+        self.element_type = schema.lookup_type(self.element_type_name)
+        assert self.element_type
+
+class QAPISchemaObjectType(QAPISchemaType):
+    def __init__(self, name, info, base, local_members, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert base == None or isinstance(base, str)
+        for m in local_members:
+            assert isinstance(m, QAPISchemaObjectTypeMember)
+        if variants != None:
+            assert isinstance(variants, QAPISchemaObjectTypeVariants)
+        self.base_name = base
+        self.base = None
+        self.local_members = local_members
+        self.variants = variants
+        self.members = None
Quoted 8 lines

scripts/qapi.py def check_exprs(exprs): · line 810

Quoted 5 lines
scripts/qapi.py +5 -0
+    def check(self, schema):
+        if self.members:
+            return                      # already checked
+        assert self.members == None     # not running in cycles
+        self.members = False            # mark as being checked

Interesting, but makes sense (since we allow self-referential nesting, populating our own members may require revisiting the same type).

Yes. Another way to put it: for a recursive type like this, we better check the recursion terminates.

Quoted 3 lines

Cute
that python allows both None and False as distinct non-true settings, so that you end up using the variable as a tri-state.

Yup :)

Quoted 8 lines

scripts/qapi.py def check_exprs(exprs): · line 813

Quoted 3 lines
scripts/qapi.py +3 -0
+        if self.base_name:
+            self.base = schema.lookup_type(self.base_name)
+            self.base.check(schema)

Do you need 'assert self.base' here, similar to how you did it in QAPISchemaArrayType.check()? I guess you'll still get a python exception that None.check() doesn't exist if the lookup_type failed, so it just depends on what error message you want.

I don't see a hard need, but consistency is nice. Since I generally assert elsewhere, I'll add an assertion here.

Quoted 5 lines
Quoted 1 line

+ members = list(self.base.members)

For that matter, do you want to assert isinstance(self.base, QAPISchemaObjectType), since lookup_type can return other types, but other child classes of QAPISchemaType do not have a .members?

Yes, that's the appropriate assertion.

Quoted 6 lines
Quoted 6 lines
scripts/qapi.py +6 -0
+        else:
+            members = []
+        seen = {}
+        for m in members:
+            seen[m.name] = m
+        for m in self.local_members:
Quoted 8 lines

scripts/qapi.py def check_exprs(exprs): · line 826

Quoted 6 lines
scripts/qapi.py +6 -0
+            m.check(schema, members, seen)
+        if self.variants:
+            self.variants.check(schema, members, seen)
+        self.members = members
+
+class QAPISchemaObjectTypeMember(object):

Is 'object' the right base class, or should it be QAPISchemaEntity?

No, it's really object. A schema's entities can be object types, other types, commands and events, but not members. A schema doesn't have "members", only an object type has.

Quoted 6 lines

scripts/qapi.py def check_exprs(exprs): · line 829

Quoted 3 lines
scripts/qapi.py +3 -0
+    def __init__(self, name, typ, optional):
+        assert isinstance(name, str)
+        assert isinstance(typ, str)

I guess that means all callers flatten array types into an unambiguous string first.

Yes.

Conscious design decision: every type in the internal representation has a unique name. If the concrete syntax lets you define an anonymous type, we make up a name in the internal representation. One less special case to worry about.

Quoted 7 lines
Quoted 7 lines
scripts/qapi.py +7 -0
+        assert isinstance(optional, bool)
+        self.name = name
+        self.type_name = typ
+        self.type = None
+        self.optional = optional
+    def check(self, schema, all_members, seen):
+        assert self.name not in seen
Quoted 8 lines

scripts/qapi.py def check_exprs(exprs): · line 842

Quoted 6 lines
scripts/qapi.py +6 -0
+        self.type = schema.lookup_type(self.type_name)
+        assert self.type
+        all_members.append(self)
+        seen[self.name] = self
+
+class QAPISchemaObjectTypeVariants(object):

Is 'object' the right base class, or should it be QAPISchemaEntity?

As for QAPISchemaObjectTypeMember above, object is appropriate.

Quoted 13 lines

scripts/qapi.py def check_exprs(exprs): · line 854

Quoted 12 lines
scripts/qapi.py +12 -0
+    def __init__(self, tag_name, tag_enum, variants):
+        assert tag_name == None or isinstance(tag_name, str)
+        assert tag_enum == None or isinstance(tag_enum, str)
+        for v in variants:
+            assert isinstance(v, QAPISchemaObjectTypeVariant)
+        self.tag_name = tag_name
+        if tag_name:
+            assert not tag_enum
+            self.tag_member = None
+        else:
+            self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum,
+                                                         False)
Quoted 3 lines

Nice - the tag is either part of the base struct (flat union) or implicitly created (plain union). By the time we are done with check(), we then have both its type (possibly generated implicitly) and its name.

... and we don't have to know where they came from anymore.

Quoted 30 lines

scripts/qapi.py def check_exprs(exprs): · line 860

Quoted 6 lines
scripts/qapi.py +6 -0
+        self.variants = variants
+    def check(self, schema, members, seen):
+        if self.tag_name:
+            self.tag_member = seen[self.tag_name]
+        else:
+            self.tag_member.check(schema, members, seen)

A little bit tricky here in that flat unions (tag_name was set) must find the tag from the members already loaded from the base class, while plain unions (tag_name is None) add their own implicit member into the union. But it works.

Quoted 6 lines
scripts/qapi.py +6 -0
+        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        for v in self.variants:
+            vseen = dict(seen)
+            v.check(schema, self.tag_member.type, vseen)
+
+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):

scripts/qapi.py def check_exprs(exprs): · line 876

Quoted 10 lines
scripts/qapi.py +10 -0
+    def __init__(self, name, typ, flat):
+        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
+        assert isinstance(flat, bool)
+        self.flat = flat
+    def check(self, schema, tag_type, seen):
+        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        assert self.name in tag_type.values
+        if self.flat:
+            self.type.check(schema)
+            assert isinstance(self.type, QAPISchemaObjectType)

Do we really want to be tracking self.flat for each variant? After all,

Quoted 8 lines

docs/qapi-code-gen.txt already describes the mapping from simple union into flat union (as if the flat union had a base class with single member 'kind' of the right type, then each branch of the union composed of an implicit object with a lone member 'data' of the correct type). In other words, is it any better to just normalize into that form now, such that each QAPISchemaObjectTypeVariant is merely a (often one-element) list of name:type members being added to the overall QAPISchemaObject?

I tried to do exactly that, but got bogged down in special cases and copped out. Then I went on vacation, and now I don't remember the exact problems anymore %-}

I guess / hope it's just relatively pointless differences in the generated C code I didn't want to get rid of at this time. The series is long and hairy enough as it is...

Quoted 2 lines

But I guess it remains to be seen how you use
self.flat before knowing if it is worth normalizing away from it.

At least introspect.json is oblivious of it.

Quoted 42 lines

scripts/qapi.py def check_exprs(exprs): · line 884

Quoted 8 lines
scripts/qapi.py +8 -0
+
+class QAPISchemaAlternateType(QAPISchemaType):
+    def __init__(self, name, info, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(variants, QAPISchemaObjectTypeVariants)
+        assert not variants.tag_name
+        for v in variants.variants:
+            assert not v.flat

Hmm. You are using .flat after all.

scripts/qapi.py def check_exprs(exprs): · line 893

Quoted 9 lines
scripts/qapi.py +9 -0
+        self.variants = variants
+    def check(self, schema):
+        self.variants.check(schema, [], {})
+
+class QAPISchemaCommand(QAPISchemaEntity):
+    def __init__(self, name, info, args, rets, gen, success_response):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not args or isinstance(args, str)
+        assert not rets or isinstance(rets, str)

Again, this means that the caller has already had to convert qapi dictionaries into an implicit type and pass in that type name here. Works for me.

Quoted 6 lines
scripts/qapi.py +6 -0
+        self.args_name = args
+        self.args = None
+        self.rets_name = rets
+        self.rets = None
+        self.gen = gen
+        self.success_response = success_response

scripts/qapi.py def check_exprs(exprs): · line 909

Quoted 10 lines
scripts/qapi.py +10 -0
+    def check(self, schema):
+        if self.args_name:
+            self.args = schema.lookup_type(self.args_name)
+            assert isinstance(self.args, QAPISchemaObjectType)
+            assert not self.args.variants # not implemented
+            self.args.check(schema)
+        if self.rets_name:
+            self.rets = schema.lookup_type(self.rets_name)
+            assert isinstance(self.rets, QAPISchemaType)
+            self.rets.check(schema)
Quoted 3 lines

Matches our existing difference in requiring a dictionary for arguments, but allowing the return of any type (although we now have a whitelist to prevent too many additions of non-dictionaries).

Yes. In my opinion, the difference between arguments and returns is a design blemish (a function maps from domain to codomain, and both are just sets, so any differences are likely pointless baggage), but we need to work with what we've got.

Quoted 33 lines
Quoted 33 lines
scripts/qapi.py +33 -0
+
+class QAPISchemaEvent(QAPISchemaEntity):
+    def __init__(self, name, info, data):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not data or isinstance(data, str)
+        self.data_name = data
+        self.data = None
+    def check(self, schema):
+        if self.data_name:
+            self.data = schema.lookup_type(self.data_name)
+            assert isinstance(self.data, QAPISchemaObjectType)
+            self.data.check(schema)
+
+class QAPISchema(object):
+    def __init__(self, fname):
+        try:
+ self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
+        except (QAPISchemaError, QAPIExprError), err:
+            print >>sys.stderr, err
+            exit(1)
+        self.entity_dict = {}
+        self._def_predefineds()
+        self._def_exprs()
+        self.check()
+
+    def get_exprs(self):
+        return [expr_elem['expr'] for expr_elem in self.exprs]
+
+    def _def_entity(self, ent):
+        assert ent.name not in self.entity_dict
+        self.entity_dict[ent.name] = ent
+
+    def lookup_entity(self, name, typ=None):
Quoted 8 lines

scripts/qapi.py def check_exprs(exprs): · line 946

Quoted 4 lines
scripts/qapi.py +4 -0
+        ent = self.entity_dict.get(name)
+        if typ and not isinstance(ent, typ):
+            return None
+        return ent

Ah, so you enforce a shared namespace between types, commands, and events (all three are in self.entity_dict), but can use the typ parameter to allow limiting a lookup to just types.

Yes. It's a convenience feature.

Quoted 25 lines
Quoted 25 lines
scripts/qapi.py +25 -0
+
+    def lookup_type(self, name):
+        return self.lookup_entity(name, QAPISchemaType)
+
+    def _def_builtin_type(self, name):
+        self._def_entity(QAPISchemaBuiltinType(name))
+        if name != '**':
+            self._make_array_type(name) # TODO really needed?
+
+    def _def_predefineds(self):
+        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
+ 'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
+            self._def_builtin_type(t)
+
+    def _make_implicit_enum_type(self, name, values):
+        name = name + 'Kind'
+        self._def_entity(QAPISchemaEnumType(name, None, values))
+        return name
+
+    def _make_array_type(self, element_type):
+        name = element_type + 'List'
+        if not self.lookup_type(name):
+            self._def_entity(QAPISchemaArrayType(name, None, element_type))
+        return name
+
Quoted 8 lines

scripts/qapi.py def check_exprs(exprs): · line 975

Quoted 4 lines
scripts/qapi.py +4 -0
+    def _make_implicit_object_type(self, name, role, members):
+        name = ':obj-%s-%s' % (name, role)
+        self._def_entity(QAPISchemaObjectType(name, None, None, members, None))
+        return name

We may extend the qapi .json syntax to allow anonymous types in more places, but so far, this matches that all existing use of anonymous types are for structs, not unions.

Yes.

If we unify struct and union in the concrete syntax like we do in this internal representation, then anonymous unions (really: anonymous objects with variants) may well fall out naturally.

Quoted 45 lines
Quoted 45 lines
scripts/qapi.py +45 -0
+
+    def _def_enum_type(self, expr, info):
+        name = expr['enum']
+        data = expr['data']
+        self._def_entity(QAPISchemaEnumType(name, info, data))
+        self._make_array_type(name) # TODO really needed?
+
+    def _make_member(self, name, typ):
+        optional = False
+        if name.startswith('*'):
+            name = name[1:]
+            optional = True
+        if isinstance(typ, list):
+            assert len(typ) == 1
+            typ = self._make_array_type(typ[0])
+        return QAPISchemaObjectTypeMember(name, typ, optional)
+
+    def _make_members(self, data):
+        return [self._make_member(key, data[key]) for key in data.keys()]
+
+    def _def_struct_type(self, expr, info):
+        name = expr['struct']
+        base = expr.get('base')
+        data = expr['data']
+        self._def_entity(QAPISchemaObjectType(name, info, base,
+                                              self._make_members(data),
+                                              None))
+        self._make_array_type(name) # TODO really needed?
+
+    def _make_flat_variant(self, case, typ):
+        return QAPISchemaObjectTypeVariant(case, typ, True)
+
+    def _make_flat_variants(self, tag_name, data):
+        variants = [self._make_flat_variant(key, data[key])
+                    for key in data.keys()]
+        return QAPISchemaObjectTypeVariants(tag_name, None, variants)
+
+    def _make_simple_variant(self, case, typ):
+        if isinstance(typ, list):
+            assert len(typ) == 1
+            typ = self._make_array_type(typ[0])
+        return QAPISchemaObjectTypeVariant(case, typ, False)
+
+    def _make_simple_variants(self, type_name, data):
+        variants = [self._make_simple_variant(key, data[key])
Quoted 8 lines

scripts/qapi.py def check_exprs(exprs): · line 1024

Quoted 4 lines
scripts/qapi.py +4 -0
+                    for key in data.keys()]
+        enum = self._make_implicit_enum_type(type_name,
+                                             [v.name for v in variants])
+        return QAPISchemaObjectTypeVariants(None, enum, variants)

Again, I wonder if normalizing simple unions into flat unions (to get rid of the need for tracking .flat) would make later use any easier, or if it would just make introspection QMP output more verbose.

They *are* normalized in introspection output!

Quoted 27 lines
Quoted 27 lines
scripts/qapi.py +27 -0
+
+    def _def_union_type(self, expr, info):
+        name = expr['union']
+        data = expr['data']
+        tag_name = expr.get('discriminator')
+        if tag_name:
+            base = expr['base']
+            variants = self._make_flat_variants(tag_name, data)
+        else:
+            base = None
+            variants = self._make_simple_variants(name, data)
+        self._def_entity(QAPISchemaObjectType(name, info, base,
+ self._make_members(OrderedDict()),
+                                              variants))
+        self._make_array_type(name) # TODO really needed?
+
+    def _def_alternate_type(self, expr, info):
+        name = expr['alternate']
+        data = expr['data']
+        self._def_entity(QAPISchemaAlternateType(name, info,
+                                    self._make_simple_variants(name, data)))
+        self._make_array_type(name) # TODO really needed?
+
+    def _def_command(self, expr, info):
+        name = expr['command']
+        args = expr.get('data')
+        rets = expr.get('returns')
Quoted 8 lines

scripts/qapi.py def check_exprs(exprs): · line 1056

Quoted 5 lines
scripts/qapi.py +5 -0
+        gen = expr.get('gen', True)
+        success_response = expr.get('success-response', True)
+        if args and not isinstance(args, str):
+            args = self._make_implicit_object_type(name, 'args',
+                                                   self._make_members(args))

If I write { 'command':'Foo', 'data':{} }, does that try and create an implicit type, with no members?

tests/qapi-schema/qapi-schema-test.json tests this:

    { 'command': 'user_def_cmd', 'data': {} }

args = expr.get('data') gets an empty OrderedDict().

Since an empty dictionary counts as false, we skip calling _make_implicit_object_type(), and pass the empty dictionary to QAPISchemaCommand(). Works, because the class consistently treats any false args the same as None. tests/qapi-schema/qapi-schema-test.out:

    command user_def_cmd None -> None
       gen=True success_response=True

I'll consider cleaning this up a bit.

Quoted 46 lines

I'm wondering if it is any simpler to
write args = expr.get('data', {}), and have _make_implicit_object_type() do the right thing when presented with an empty list of members.

Quoted 29 lines
scripts/qapi.py +29 -0
+        if rets and isinstance(rets, list):
+            assert len(rets) == 1
+            rets = self._make_array_type(rets[0])
+        elif rets and not isinstance(rets, str):
+            rets = self._make_implicit_object_type(name, 'rets',
+                                                   self._make_members(rets))
+        self._def_entity(QAPISchemaCommand(name, info, args, rets, gen,
+                                           success_response))
+
+    def _def_event(self, expr, info):
+        name = expr['event']
+        data = expr.get('data')
+        if data and not isinstance(data, str):
+            data = self._make_implicit_object_type(name, 'data',
+                                                   self._make_members(data))
+        self._def_entity(QAPISchemaEvent(name, info, data))
+
+    def _def_exprs(self):
+        for expr_elem in self.exprs:
+            expr = expr_elem['expr']
+            info = expr_elem['info']
+            if 'enum' in expr:
+                self._def_enum_type(expr, info)
+            elif 'struct' in expr:
+                self._def_struct_type(expr, info)
+            elif 'union' in expr:
+                self._def_union_type(expr, info)
+            elif 'alternate' in expr:
+                self._def_alternate_type(expr, info)

scripts/qapi.py def check_exprs(exprs): · line 1095

Quoted 10 lines
scripts/qapi.py +10 -0
+            elif 'command' in expr:
+                self._def_command(expr, info)
+            elif 'event' in expr:
+                self._def_event(expr, info)
+            else:
+                assert False
+
+    def check(self):
+        for ent in self.entity_dict.values():
+            ent.check(self)

Overall seems like a very nice layout.

Quoted 3 lines

I'll reserve R-b until I see how the rest of the series uses this object hierarchy, but it looks like you won't have to change very much of this patch when you do your next spin of the series.

Thanks!

#72 lore raw

Re: [Qemu-devel] [PATCH RFC v2 21/47] qapi: New QAPISchema intermediate reperesentation

Quoted 34 lines · Markus Armbruster
Quoted 20 lines
Quoted 9 lines

Create a bunch of classes to represent QAPI schemata.

Have the QAPISchema initializer call the parser, then walk the syntax tree to create the new internal representation, and finally perform semantic analysis.

scripts/qapi.py +1 -0
+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):

scripts/qapi.py def check_exprs(exprs): · line 876

Quoted 10 lines
scripts/qapi.py +10 -0
+    def __init__(self, name, typ, flat):
+        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
+        assert isinstance(flat, bool)
+        self.flat = flat
+    def check(self, schema, tag_type, seen):
+        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        assert self.name in tag_type.values
+        if self.flat:
+            self.type.check(schema)
+            assert isinstance(self.type, QAPISchemaObjectType)
Quoted 9 lines

Do we really want to be tracking self.flat for each variant? After all, docs/qapi-code-gen.txt already describes the mapping from simple union into flat union (as if the flat union had a base class with single member 'kind' of the right type, then each branch of the union composed of an implicit object with a lone member 'data' of the correct type). In other words, is it any better to just normalize into that form now, such that each QAPISchemaObjectTypeVariant is merely a (often one-element) list of name:type members being added to the overall QAPISchemaObject?

I tried to do exactly that, but got bogged down in special cases and copped out. Then I went on vacation, and now I don't remember the exact problems anymore %-}

Quoted 8 lines

I guess / hope it's just relatively pointless differences in the generated C code I didn't want to get rid of at this time. The series is long and hairy enough as it is...

Quoted 2 lines

But I guess it remains to be seen how you use
self.flat before knowing if it is worth normalizing away from it.

At least introspect.json is oblivious of it.

Yeah, that was my conclusion by the end of the series - we still had enough special cases where we generate different code for simple unions than for flat unions. It would be possible to merge the generator and simplify things further, but at this point, it is best done as a follow-up series because it will touch lots of C code (anything that uses a simple union would have to convert to the common representation).

And you actually did reference .flat in the patch that first added qapi-introspect.py (good that it did not leak to the introspection output, but it did have to be considered when figuring out what to output); again, something you may want to rework before polishing this into v3, or something you may end up just documenting as a possible cleanup for a future series. But after having reviewed the whole series now, I'm able to live with your use of .flat, if only because it makes the initial conversion faster.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
scripts/qapi.py +6 -0
+
+    def lookup_entity(self, name, typ=None):
+        ent = self.entity_dict.get(name)
+        if typ and not isinstance(ent, typ):
+            return None
+        return ent
Quoted 5 lines
Quoted 3 lines

Ah, so you enforce a shared namespace between types, commands, and events (all three are in self.entity_dict), but can use the typ parameter to allow limiting a lookup to just types.

Yes. It's a convenience feature.

Documentation comments never hurt :) (You did mention in the cover letter that part of the reason this was still RFC was that it was lacking documentation)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#73 lore raw

RFC PATCH v2 22/47 qapi: QAPISchema code generation helper methods

In reply to Markus Armbruster

New methods c_name(), c_type(), c_null(), json_type(), alternate_qtype().

---
 scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 65 insertions(+), 7 deletions(-)
scripts/qapi.py +65 -7
index a80892e..b5fa1be 100644
@@ -763,15 +763,47 @@ class QAPISchemaEntity(object):
         assert isinstance(name, str)
         self.name = name
         self.info = info
+    def c_name(self):
+        return c_name(self.name)
     def check(self, schema):
         pass
 
 class QAPISchemaType(QAPISchemaEntity):
-    pass
+    def c_type(self, is_param=False):
+        return c_name(self.name) + pointer_suffix
+    def c_null(self):
+        return 'NULL'
+    def json_type(self):
+        return None
+    def alternate_qtype(self):
+        json2qtype = {
+            'string':  'QTYPE_QSTRING',
+            'number':  'QTYPE_QFLOAT',
+            'int':     'QTYPE_QINT',
+            'boolean': 'QTYPE_QBOOL',
+            'object':  'QTYPE_QDICT'
+        }
+        return json2qtype.get(self.json_type())
 
 class QAPISchemaBuiltinType(QAPISchemaType):
-    def __init__(self, name):
+    def __init__(self, name, json_type, c_type, c_null):
         QAPISchemaType.__init__(self, name, None)
+        assert not c_type or isinstance(c_type, str)
+        assert json_type in ('string', 'number', 'int', 'boolean', 'null',
+                             'value')
+        self.json_type_name = json_type
+        self.c_type_name = c_type
+        self.c_null_val = c_null
+    def c_name(self):
+        return self.name
+    def c_type(self, is_param=False):
+        if is_param and self.name == 'str':
+            return 'const ' + self.c_type_name
+        return self.c_type_name
+    def c_null(self):
+        return self.c_null_val
+    def json_type(self):
+        return self.json_type_name
 
 class QAPISchemaEnumType(QAPISchemaType):
     def __init__(self, name, info, values):
@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
         for v in values:
             assert isinstance(v, str)
         self.values = values
+    def c_type(self, is_param=False):
+        return c_name(self.name)
+    def c_null(self):
+        return c_enum_const(self.name, self.values[0])
+    def json_type(self):
+        return 'string'
 
 class QAPISchemaArrayType(QAPISchemaType):
     def __init__(self, name, info, element_type):
@@ -822,6 +860,14 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self.variants:
             self.variants.check(schema, members, seen)
         self.members = members
+    def c_name(self):
+        assert self.info
+        return QAPISchemaType.c_name(self)
+    def c_type(self, is_param=False):
+        assert self.info
+        return QAPISchemaType.c_type(self)
+    def json_type(self):
+        return 'object'
 
 class QAPISchemaObjectTypeMember(object):
     def __init__(self, name, typ, optional):
@@ -948,15 +994,27 @@ class QAPISchema(object):
     def lookup_type(self, name):
         return self.lookup_entity(name, QAPISchemaType)
 
-    def _def_builtin_type(self, name):
-        self._def_entity(QAPISchemaBuiltinType(name))
+    def _def_builtin_type(self, name, json_type, c_type, c_null):
+        self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null))
         if name != '**':
             self._make_array_type(name) # TODO really needed?
 
     def _def_predefineds(self):
-        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
-                  'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
-            self._def_builtin_type(t)
+        for t in [('str',    'string',  'char' + pointer_suffix, 'NULL'),
+                  ('number', 'number',  'double',   '0'),
+                  ('int',    'int',     'int64_t',  '0'),
+                  ('int8',   'int',     'int8_t',   '0'),
+                  ('int16',  'int',     'int16_t',  '0'),
+                  ('int32',  'int',     'int32_t',  '0'),
+                  ('int64',  'int',     'int64_t',  '0'),
+                  ('uint8',  'int',     'uint8_t',  '0'),
+                  ('uint16', 'int',     'uint16_t', '0'),
+                  ('uint32', 'int',     'uint32_t', '0'),
+                  ('uint64', 'int',     'uint64_t', '0'),
+                  ('size',   'int',     'uint64_t', '0'),
+                  ('bool',   'boolean', 'bool',     'false'),
+                  ('**',     'value',   None,       None)]:
+            self._def_builtin_type(*t)
 
     def _make_implicit_enum_type(self, name, values):
         name = name + 'Kind'
Signature · 1 line
1.9.3
Reviewed-by on v2 22/47
#74 lore raw

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

Quoted 7 lines · Markus Armbruster

New methods c_name(), c_type(), c_null(), json_type(), alternate_qtype().

---
 scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 65 insertions(+), 7 deletions(-)

Still unused in the rest of the code, so no change to generated code yet; and thanks for separating this apart from the creation of the new object hierarchy.

scripts/qapi.py class QAPISchemaEntity(object): · line 773

Quoted 4 lines

class QAPISchemaType(QAPISchemaEntity):

scripts/qapi.py +2 -1
-    pass
+    def c_type(self, is_param=False):
+        return c_name(self.name) + pointer_suffix

At first, I thought is_param was unused; but reviewing the rest of the patch shows that subclasses are overriding it, so declaring it here is necessary for polymorphic calls to supply the argument and have it properly acted on.

scripts/qapi.py class QAPISchemaEntity(object): · line 785

Quoted 12 lines
scripts/qapi.py +12 -0
+    def c_null(self):
+        return 'NULL'
+    def json_type(self):
+        return None
+    def alternate_qtype(self):
+        json2qtype = {
+            'string':  'QTYPE_QSTRING',
+            'number':  'QTYPE_QFLOAT',
+            'int':     'QTYPE_QINT',
+            'boolean': 'QTYPE_QBOOL',
+            'object':  'QTYPE_QDICT'
+        }

Are there any style rules on whether to include or omit a trailing comma?

Quoted 4 lines
scripts/qapi.py +1 -1
+        return json2qtype.get(self.json_type())
 
 class QAPISchemaBuiltinType(QAPISchemaType):
-    def __init__(self, name):

scripts/qapi.py class QAPISchemaEntity(object): · line 798

Quoted 10 lines
scripts/qapi.py +9 -0
+    def __init__(self, name, json_type, c_type, c_null):
         QAPISchemaType.__init__(self, name, None)
+        assert not c_type or isinstance(c_type, str)
+        assert json_type in ('string', 'number', 'int', 'boolean', 'null',
+                             'value')
+        self.json_type_name = json_type
+        self.c_type_name = c_type
+        self.c_null_val = c_null
+    def c_name(self):
+        return self.name

At first, I thought this was redundant with QAPISchemaEntity.c_name(), until I realized that you are intentionally overriding things here to NOT call global c_name() (so that 'int' does not get munged the 'q_int' in the generated C code). Nice, once I figured it out.

Quoted 12 lines
scripts/qapi.py +8 -0
+    def c_type(self, is_param=False):
+        if is_param and self.name == 'str':
+            return 'const ' + self.c_type_name
+        return self.c_type_name
+    def c_null(self):
+        return self.c_null_val
+    def json_type(self):
+        return self.json_type_name
 
 class QAPISchemaEnumType(QAPISchemaType):
     def __init__(self, name, info, values):

scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 815

Quoted 5 lines
scripts/qapi.py +2 -0
@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
         for v in values:
             assert isinstance(v, str)
         self.values = values
+    def c_type(self, is_param=False):
+        return c_name(self.name)

Probably showing my unfamiliarity with python polymorphism/overriding rules - is it necessary to declare an otherwise-unused optional parameter here if we are overriding a base method where the parameter has a sane default for our needs? That is, would c_type(self) here properly override c_type(self, is_param=False) in the parent class?

Quoted 8 lines
scripts/qapi.py +4 -0
+    def c_null(self):
+        return c_enum_const(self.name, self.values[0])
+    def json_type(self):
+        return 'string'
 
 class QAPISchemaArrayType(QAPISchemaType):
     def __init__(self, name, info, element_type):

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 868

Quoted 9 lines
scripts/qapi.py +6 -0
@@ -822,6 +860,14 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self.variants:
             self.variants.check(schema, members, seen)
         self.members = members
+    def c_name(self):
+        assert self.info
+        return QAPISchemaType.c_name(self)
+    def c_type(self, is_param=False):
+        assert self.info
+        return QAPISchemaType.c_type(self)

So, to make sure I understand correctly, implicit types (those generated from anonymous dictionaries in a command or event) have info=None, and the assertions here are making sure we never try to learn the c_name or c_type of those implicit types in generated code, but rather use them solely in generated code that just enumerates the members (C struct declarations, parameter list to functions that implement commands).

Quoted 12 lines
scripts/qapi.py +3 -2
+    def json_type(self):
+        return 'object'
 
 class QAPISchemaObjectTypeMember(object):
     def __init__(self, name, typ, optional):
@@ -948,15 +994,27 @@ class QAPISchema(object):
     def lookup_type(self, name):
         return self.lookup_entity(name, QAPISchemaType)
 
-    def _def_builtin_type(self, name):
-        self._def_entity(QAPISchemaBuiltinType(name))
+    def _def_builtin_type(self, name, json_type, c_type, c_null):

scripts/qapi.py class QAPISchema(object): · line 1004

Quoted 10 lines
scripts/qapi.py +3 -3
+        self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null))
         if name != '**':
             self._make_array_type(name) # TODO really needed?
 
     def _def_predefineds(self):
-        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
-                  'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
-            self._def_builtin_type(t)
+        for t in [('str',    'string',  'char' + pointer_suffix, 'NULL'),
+                  ('number', 'number',  'double',   '0'),

Do you want '0.0' as the default here (yes, promoting an int 0 to double does the right thing, but I like making it obvious that I know I'm using floating point).

Quoted 7 lines
scripts/qapi.py +6 -0
+                  ('int',    'int',     'int64_t',  '0'),
+                  ('int8',   'int',     'int8_t',   '0'),
+                  ('int16',  'int',     'int16_t',  '0'),
+                  ('int32',  'int',     'int32_t',  '0'),
+                  ('int64',  'int',     'int64_t',  '0'),
+                  ('uint8',  'int',     'uint8_t',  '0'),

scripts/qapi.py class QAPISchema(object): · line 1020

Quoted 10 lines
scripts/qapi.py +7 -0
+                  ('uint16', 'int',     'uint16_t', '0'),
+                  ('uint32', 'int',     'uint32_t', '0'),
+                  ('uint64', 'int',     'uint64_t', '0'),
+                  ('size',   'int',     'uint64_t', '0'),
+                  ('bool',   'boolean', 'bool',     'false'),
+                  ('**',     'value',   None,       None)]:
+            self._def_builtin_type(*t)
 
     def _make_implicit_enum_type(self, name, values):
         name = name + 'Kind'

Tweaks you make (if any) based on the above comments should be minor, so I'm fine with adding:

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#75 lore raw

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

Quoted 19 lines · Eric Blake
Quoted 8 lines · Markus Armbruster

New methods c_name(), c_type(), c_null(), json_type(), alternate_qtype().

---
 scripts/qapi.py | 72
+++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 65 insertions(+), 7 deletions(-)

Still unused in the rest of the code, so no change to generated code yet; and thanks for separating this apart from the creation of the new object hierarchy.

scripts/qapi.py class QAPISchemaEntity(object): · line 773

Quoted 4 lines

class QAPISchemaType(QAPISchemaEntity):

scripts/qapi.py +2 -1
-    pass
+    def c_type(self, is_param=False):
+        return c_name(self.name) + pointer_suffix
Quoted 4 lines

At first, I thought is_param was unused; but reviewing the rest of the patch shows that subclasses are overriding it, so declaring it here is necessary for polymorphic calls to supply the argument and have it properly acted on.

Yup.

Quoted 13 lines

scripts/qapi.py class QAPISchemaEntity(object): · line 785

Quoted 12 lines
scripts/qapi.py +12 -0
+    def c_null(self):
+        return 'NULL'
+    def json_type(self):
+        return None
+    def alternate_qtype(self):
+        json2qtype = {
+            'string':  'QTYPE_QSTRING',
+            'number':  'QTYPE_QFLOAT',
+            'int':     'QTYPE_QINT',
+            'boolean': 'QTYPE_QBOOL',
+            'object':  'QTYPE_QDICT'
+        }
Quoted 1 line

Are there any style rules on whether to include or omit a trailing comma?

PEP8 appears to be silent on it.

Quoted 15 lines
Quoted 4 lines
scripts/qapi.py +1 -1
+        return json2qtype.get(self.json_type())
 
 class QAPISchemaBuiltinType(QAPISchemaType):
-    def __init__(self, name):

scripts/qapi.py class QAPISchemaEntity(object): · line 798

Quoted 10 lines
scripts/qapi.py +9 -0
+    def __init__(self, name, json_type, c_type, c_null):
         QAPISchemaType.__init__(self, name, None)
+        assert not c_type or isinstance(c_type, str)
+        assert json_type in ('string', 'number', 'int', 'boolean', 'null',
+                             'value')
+        self.json_type_name = json_type
+        self.c_type_name = c_type
+        self.c_null_val = c_null
+    def c_name(self):
+        return self.name
Quoted 4 lines

At first, I thought this was redundant with QAPISchemaEntity.c_name(), until I realized that you are intentionally overriding things here to NOT call global c_name() (so that 'int' does not get munged the 'q_int' in the generated C code). Nice, once I figured it out.

I'll consider adding comments to help future readers figure it out.

Quoted 9 lines
Quoted 9 lines
scripts/qapi.py +8 -0
+    def c_type(self, is_param=False):
+        if is_param and self.name == 'str':
+            return 'const ' + self.c_type_name
+        return self.c_type_name
+    def c_null(self):
+        return self.c_null_val
+    def json_type(self):
+        return self.json_type_name
Quoted 14 lines

scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 815

Quoted 8 lines

class QAPISchemaEnumType(QAPISchemaType):

     def __init__(self, name, info, values):
scripts/qapi.py +2 -0
@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
         for v in values:
             assert isinstance(v, str)
         self.values = values
+    def c_type(self, is_param=False):
+        return c_name(self.name)

Probably showing my unfamiliarity with python polymorphism/overriding rules - is it necessary to declare an otherwise-unused optional parameter here if we are overriding a base method where the parameter has a sane default for our needs? That is, would c_type(self) here properly override c_type(self, is_param=False) in the parent class?

I don't know. But the way I wrote it, we don't need to know :)

Quoted 5 lines
Quoted 5 lines
scripts/qapi.py +4 -0
+    def c_null(self):
+        return c_enum_const(self.name, self.values[0])
+    def json_type(self):
+        return 'string'
 
Quoted 15 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 868

Quoted 12 lines
scripts/qapi.py +6 -0
 class QAPISchemaArrayType(QAPISchemaType):
     def __init__(self, name, info, element_type):
@@ -822,6 +860,14 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self.variants:
             self.variants.check(schema, members, seen)
         self.members = members
+    def c_name(self):
+        assert self.info
+        return QAPISchemaType.c_name(self)
+    def c_type(self, is_param=False):
+        assert self.info
+        return QAPISchemaType.c_type(self)

So, to make sure I understand correctly, implicit types (those generated from anonymous dictionaries in a command or event) have info=None,

Correct.

Quoted 5 lines

and
the assertions here are making sure we never try to learn the c_name or c_type of those implicit types in generated code, but rather use them solely in generated code that just enumerates the members (C struct declarations, parameter list to functions that implement commands).

Correct again.

Quoted 16 lines
Quoted 16 lines
scripts/qapi.py +4 -2
+    def json_type(self):
+        return 'object'
 
 class QAPISchemaObjectTypeMember(object):
     def __init__(self, name, typ, optional):
@@ -948,15 +994,27 @@ class QAPISchema(object):
     def lookup_type(self, name):
         return self.lookup_entity(name, QAPISchemaType)
 
-    def _def_builtin_type(self, name):
-        self._def_entity(QAPISchemaBuiltinType(name))
+    def _def_builtin_type(self, name, json_type, c_type, c_null):
+        self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null))
         if name != '**':
             self._make_array_type(name) # TODO really needed?
 
Quoted 10 lines

scripts/qapi.py class QAPISchema(object): · line 1004

Quoted 6 lines
scripts/qapi.py +2 -3
     def _def_predefineds(self):
-        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
-                  'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
-            self._def_builtin_type(t)
+        for t in [('str',    'string',  'char' + pointer_suffix, 'NULL'),
+                  ('number', 'number',  'double',   '0'),

Do you want '0.0' as the default here (yes, promoting an int 0 to double does the right thing, but I like making it obvious that I know I'm using floating point).

The '0' here is used as initializer, i.e. we generate things like

    double d = 0;

Changing it to '0.0' will instead generate

    double d = 0.0;

No strong preference on my part.

Quoted 14 lines
Quoted 14 lines
scripts/qapi.py +13 -0
+                  ('int',    'int',     'int64_t',  '0'),
+                  ('int8',   'int',     'int8_t',   '0'),
+                  ('int16',  'int',     'int16_t',  '0'),
+                  ('int32',  'int',     'int32_t',  '0'),
+                  ('int64',  'int',     'int64_t',  '0'),
+                  ('uint8',  'int',     'uint8_t',  '0'),
+                  ('uint16', 'int',     'uint16_t', '0'),
+                  ('uint32', 'int',     'uint32_t', '0'),
+                  ('uint64', 'int',     'uint64_t', '0'),
+                  ('size',   'int',     'uint64_t', '0'),
+                  ('bool',   'boolean', 'bool',     'false'),
+                  ('**',     'value',   None,       None)]:
+            self._def_builtin_type(*t)
 
Quoted 8 lines

scripts/qapi.py class QAPISchema(object): · line 1020

Quoted 3 lines
scripts/qapi.py +0 -0
     def _make_implicit_enum_type(self, name, values):
         name = name + 'Kind'

Tweaks you make (if any) based on the above comments should be minor, so I'm fine with adding:

Thanks!

#76 lore raw

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

In reply to Markus Armbruster
Quoted 8 lines · Markus Armbruster

New methods c_name(), c_type(), c_null(), json_type(), alternate_qtype().

---
 scripts/qapi.py | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 65 insertions(+), 7 deletions(-)

I just noticed:

scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 817

Quoted 8 lines
scripts/qapi.py +4 -0
@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
         for v in values:
             assert isinstance(v, str)
         self.values = values
+    def c_type(self, is_param=False):
+        return c_name(self.name)
+    def c_null(self):
+        return c_enum_const(self.name, self.values[0])

What does this return for an empty enum, as in { 'enum':'Empty', 'data':[] }? Our testsuite proves we can do that, even if our normal .json code doesn't use it.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#77 lore raw

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

Quoted 13 lines · Eric Blake
Quoted 9 lines · Markus Armbruster

New methods c_name(), c_type(), c_null(), json_type(), alternate_qtype().

---
 scripts/qapi.py | 72
+++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 65 insertions(+), 7 deletions(-)

I just noticed:

Quoted 11 lines

scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 817

Quoted 8 lines
scripts/qapi.py +4 -0
@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
         for v in values:
             assert isinstance(v, str)
         self.values = values
+    def c_type(self, is_param=False):
+        return c_name(self.name)
+    def c_null(self):
+        return c_enum_const(self.name, self.values[0])

What does this return for an empty enum, as in { 'enum':'Empty', 'data':[] }?

I suspect self.values will be [] then, and self.values[0] will bomb.

Possible fixes:

* Outlaw empty enums

* Add the implicit MAX member to self.values[] (other code may have to skip it)

* Catch the special case here, and return the implicit MAX member.

Quoted 2 lines

Our testsuite proves we can do that, even if our normal
.json code doesn't use it.

tests/qapi-schema/enum-empty.json:{ 'enum': 'MyEnum', 'data': [ ] }

#78 lore raw

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

Quoted 29 lines · Markus Armbruster
Quoted 13 lines · Eric Blake
Quoted 9 lines · Markus Armbruster

New methods c_name(), c_type(), c_null(), json_type(), alternate_qtype().

---
 scripts/qapi.py | 72
+++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 65 insertions(+), 7 deletions(-)

I just noticed:

Quoted 11 lines

scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 817

Quoted 8 lines
scripts/qapi.py +4 -0
@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
         for v in values:
             assert isinstance(v, str)
         self.values = values
+    def c_type(self, is_param=False):
+        return c_name(self.name)
+    def c_null(self):
+        return c_enum_const(self.name, self.values[0])

What does this return for an empty enum, as in { 'enum':'Empty', 'data':[] }?

I suspect self.values will be [] then, and self.values[0] will bomb.

Quoted 8 lines

Possible fixes:

* Outlaw empty enums

* Add the implicit MAX member to self.values[] (other code may have to skip it)

* Catch the special case here, and return the implicit MAX member.

I'm leaning toward the third option here.

Quoted 5 lines
Quoted 3 lines

Our testsuite proves we can do that, even if our normal
.json code doesn't use it.

tests/qapi-schema/enum-empty.json:{ 'enum': 'MyEnum', 'data': [ ] }

As I've mentioned elsewhere, most of our tests/qapi-schema/*.json merely cover whether the parser is okay with the input, while tests/qapi-schema/qapi-schema-test.json is the one that also tests that the generated C code works; so sounds like that test should be enhanced to cover some of these corner cases we have been considering in this series (empty enum, command with no arguments and no returns, and so forth).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#79 lore raw

Re: [Qemu-devel] [PATCH RFC v2 22/47] qapi: QAPISchema code generation helper methods

Quoted 32 lines · Eric Blake
Quoted 31 lines · Markus Armbruster
Quoted 13 lines · Eric Blake
Quoted 9 lines · Markus Armbruster

New methods c_name(), c_type(), c_null(), json_type(), alternate_qtype().

---
 scripts/qapi.py | 72
+++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 65 insertions(+), 7 deletions(-)

I just noticed:

Quoted 11 lines

scripts/qapi.py class QAPISchemaEnumType(QAPISchemaType): · line 817

Quoted 8 lines
scripts/qapi.py +4 -0
@@ -779,6 +811,12 @@ class QAPISchemaEnumType(QAPISchemaType):
         for v in values:
             assert isinstance(v, str)
         self.values = values
+    def c_type(self, is_param=False):
+        return c_name(self.name)
+    def c_null(self):
+        return c_enum_const(self.name, self.values[0])

What does this return for an empty enum, as in { 'enum':'Empty', 'data':[] }?

I suspect self.values will be [] then, and self.values[0] will bomb.

Possible fixes:

Quoted 8 lines
Quoted 6 lines

* Outlaw empty enums

* Add the implicit MAX member to self.values[] (other code may have to skip it)

* Catch the special case here, and return the implicit MAX member.

I'm leaning toward the third option here.

Me too now, because it's easy:

        return c_enum_const(self.name, (self.values + [ 'MAX' ])[0])
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines

Our testsuite proves we can do that, even if our normal
.json code doesn't use it.

tests/qapi-schema/enum-empty.json:{ 'enum': 'MyEnum', 'data': [ ] }

As I've mentioned elsewhere, most of our tests/qapi-schema/*.json merely cover whether the parser is okay with the input, while tests/qapi-schema/qapi-schema-test.json is the one that also tests that the generated C code works; so sounds like that test should be enhanced to cover some of these corner cases we have been considering in this series (empty enum, command with no arguments and no returns, and so forth).

Yes, that would be nice, but I feel I have to leave it for another day.

#80 lore raw

RFC PATCH v2 23/47 qapi: New QAPISchemaVisitor

In reply to Markus Armbruster

The visitor will help keeping the code generation code simple and reasonably separated from QAPISchema details.

---
 scripts/qapi.py | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
scripts/qapi.py +44 -0
index b5fa1be..cac7ab5 100644
@@ -767,6 +767,28 @@ class QAPISchemaEntity(object):
         return c_name(self.name)
     def check(self, schema):
         pass
+    def visit(self, visitor):
+        pass
+
+class QAPISchemaVisitor(object):
+    def visit_begin(self):
+        pass
+    def visit_end(self):
+        pass
+    def visit_builtin_type(self, name, info, json_type):
+        pass
+    def visit_enum_type(self, name, info, values):
+        pass
+    def visit_array_type(self, name, info, element_type):
+        pass
+    def visit_object_type(self, name, info, base, members, variants):
+        pass
+    def visit_alternate_type(self, name, info, variants):
+        pass
+    def visit_command(self, name, info, args, rets, gen, success_response):
+        pass
+    def visit_event(self, name, info, data):
+        pass
 
 class QAPISchemaType(QAPISchemaEntity):
     def c_type(self, is_param=False):
@@ -804,6 +826,8 @@ class QAPISchemaBuiltinType(QAPISchemaType):
         return self.c_null_val
     def json_type(self):
         return self.json_type_name
+    def visit(self, visitor):
+        visitor.visit_builtin_type(self.name, self.info, self.json_type())
 
 class QAPISchemaEnumType(QAPISchemaType):
     def __init__(self, name, info, values):
@@ -817,6 +841,8 @@ class QAPISchemaEnumType(QAPISchemaType):
         return c_enum_const(self.name, self.values[0])
     def json_type(self):
         return 'string'
+    def visit(self, visitor):
+        visitor.visit_enum_type(self.name, self.info, self.values)
 
 class QAPISchemaArrayType(QAPISchemaType):
     def __init__(self, name, info, element_type):
@@ -827,6 +853,8 @@ class QAPISchemaArrayType(QAPISchemaType):
     def check(self, schema):
         self.element_type = schema.lookup_type(self.element_type_name)
         assert self.element_type
+    def visit(self, visitor):
+        visitor.visit_array_type(self.name, self.info, self.element_type)
 
 class QAPISchemaObjectType(QAPISchemaType):
     def __init__(self, name, info, base, local_members, variants):
@@ -868,6 +896,9 @@ class QAPISchemaObjectType(QAPISchemaType):
         return QAPISchemaType.c_type(self)
     def json_type(self):
         return 'object'
+    def visit(self, visitor):
+        visitor.visit_object_type(self.name, self.info,
+                                  self.base, self.local_members, self.variants)
 
 class QAPISchemaObjectTypeMember(object):
     def __init__(self, name, typ, optional):
@@ -931,6 +962,8 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
     def check(self, schema):
         self.variants.check(schema, [], {})
+    def visit(self, visitor):
+        visitor.visit_alternate_type(self.name, self.info, self.variants)
 
 class QAPISchemaCommand(QAPISchemaEntity):
     def __init__(self, name, info, args, rets, gen, success_response):
@@ -953,6 +986,9 @@ class QAPISchemaCommand(QAPISchemaEntity):
             self.rets = schema.lookup_type(self.rets_name)
             assert isinstance(self.rets, QAPISchemaType)
             self.rets.check(schema)
+    def visit(self, visitor):
+        visitor.visit_command(self.name, self.info, self.args, self.rets,
+                              self.gen, self.success_response)
 
 class QAPISchemaEvent(QAPISchemaEntity):
     def __init__(self, name, info, data):
@@ -965,6 +1001,8 @@ class QAPISchemaEvent(QAPISchemaEntity):
             self.data = schema.lookup_type(self.data_name)
             assert isinstance(self.data, QAPISchemaObjectType)
             self.data.check(schema)
+    def visit(self, visitor):
+        visitor.visit_event(self.name, self.info, self.data)
 
 class QAPISchema(object):
     def __init__(self, fname):
@@ -1152,6 +1190,12 @@ class QAPISchema(object):
         for ent in self.entity_dict.values():
             ent.check(self)
 
+    def visit(self, visitor):
+        visitor.visit_begin()
+        for name in sorted(self.entity_dict.keys()):
+            self.entity_dict[name].visit(visitor)
+        visitor.visit_end()
+
 #
 # Code generation helpers
 #
Signature · 1 line
1.9.3
Reviewed-by on v2 23/47
#81 lore raw

Re: [Qemu-devel] [PATCH RFC v2 23/47] qapi: New QAPISchemaVisitor

Quoted 2 lines · Markus Armbruster

The visitor will help keeping the code generation code simple and reasonably separated from QAPISchema details.

Still no change to generated code until later patches start using this.

Quoted 6 lines
---
 scripts/qapi.py | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#82 lore raw

RFC PATCH v2 24/47 tests/qapi-schema: Convert test harness to QAPISchemaVisitor

In reply to Markus Armbruster

The old code prints the result of parsing (list of expression dictionaries), and partial results of semantic analysis (list of enum dictionaries, list of struct dictionaries).

The new code prints a trace of a schema visit, i.e. what the back-ends are going to use. Built-in and array types are omitted, because they're boring.

---
 tests/qapi-schema/alternate-good.out            |  15 +-
 tests/qapi-schema/comments.out                  |   4 +-
 tests/qapi-schema/data-member-array.out         |  13 +-
 tests/qapi-schema/empty.out                     |   3 -
 tests/qapi-schema/enum-empty.out                |   4 +-
 tests/qapi-schema/event-case.out                |   4 +-
 tests/qapi-schema/flat-union-reverse-define.out |  21 +--
 tests/qapi-schema/ident-with-escape.out         |   7 +-
 tests/qapi-schema/include-relpath.out           |   4 +-
 tests/qapi-schema/include-repetition.out        |   4 +-
 tests/qapi-schema/include-simple.out            |   4 +-
 tests/qapi-schema/indented-expr.out             |   7 +-
 tests/qapi-schema/qapi-schema-test.out          | 186 +++++++++++++++++-------
 tests/qapi-schema/returns-int.out               |   5 +-
 tests/qapi-schema/test-qapi.py                  |  37 ++++-
 tests/qapi-schema/type-bypass.out               |   7 +-
 16 files changed, 210 insertions(+), 115 deletions(-)
tests/qapi-schema/alternate-good.out +9 -6
index 99848ee..0cbdfa1 100644
@@ -1,6 +1,9 @@
-[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))]),
- OrderedDict([('enum', 'Enum'), ('data', ['hello', 'world'])]),
- OrderedDict([('alternate', 'Alt'), ('data', OrderedDict([('value', 'int'), ('string', 'Enum'), ('struct', 'Data')]))])]
-[{'enum_name': 'Enum', 'enum_values': ['hello', 'world']},
- {'enum_name': 'AltKind', 'enum_values': None}]
-[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))])]
+alternate Alt
+    case value: int flat=False
+    case string: Enum flat=False
+    case struct: Data flat=False
+enum AltKind ['value', 'string', 'struct']
+object Data
+    member number: int optional=True
+    member name: str optional=True
+enum Enum ['hello', 'world']
tests/qapi-schema/comments.out +1 -3
index 4ce3dcf..6161b90 100644
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/data-member-array.out +8 -5
index c39fa25..8911179 100644
@@ -1,5 +1,8 @@
-[OrderedDict([('enum', 'abc'), ('data', ['a', 'b', 'c'])]),
- OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))]),
- OrderedDict([('command', 'okay'), ('data', OrderedDict([('member1', ['int']), ('member2', ['def'])]))])]
-[{'enum_name': 'abc', 'enum_values': ['a', 'b', 'c']}]
-[OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))])]
+object :obj-okay-args
+    member member1: intList optional=False
+    member member2: defList optional=False
+enum abc ['a', 'b', 'c']
+object def
+    member array: abcList optional=False
+command okay :obj-okay-args -> None
+   gen=True success_response=True
tests/qapi-schema/empty.out +0 -3
index b7f89a4..e69de29 100644
@@ -1,3 +0,0 @@
-[]
-[]
-[]
tests/qapi-schema/enum-empty.out +1 -3
index 3b75c16..e09b00f 100644
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'MyEnum'), ('data', [])])]
-[{'enum_name': 'MyEnum', 'enum_values': []}]
-[]
+enum MyEnum []
tests/qapi-schema/event-case.out +1 -3
index 3764bc7..b5ae4c2 100644
@@ -1,3 +1 @@
-[OrderedDict([('event', 'oops')])]
-[]
-[]
+event oops None
tests/qapi-schema/flat-union-reverse-define.out +12 -9
index 1ed7b8a..e156202 100644
@@ -1,9 +1,12 @@
-[OrderedDict([('union', 'TestUnion'), ('base', 'TestBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'TestTypeA'), ('value2', 'TestTypeB')]))]),
- OrderedDict([('struct', 'TestBase'), ('data', OrderedDict([('enum1', 'TestEnum')]))]),
- OrderedDict([('enum', 'TestEnum'), ('data', ['value1', 'value2'])]),
- OrderedDict([('struct', 'TestTypeA'), ('data', OrderedDict([('string', 'str')]))]),
- OrderedDict([('struct', 'TestTypeB'), ('data', OrderedDict([('integer', 'int')]))])]
-[{'enum_name': 'TestEnum', 'enum_values': ['value1', 'value2']}]
-[OrderedDict([('struct', 'TestBase'), ('data', OrderedDict([('enum1', 'TestEnum')]))]),
- OrderedDict([('struct', 'TestTypeA'), ('data', OrderedDict([('string', 'str')]))]),
- OrderedDict([('struct', 'TestTypeB'), ('data', OrderedDict([('integer', 'int')]))])]
+object TestBase
+    member enum1: TestEnum optional=False
+enum TestEnum ['value1', 'value2']
+object TestTypeA
+    member string: str optional=False
+object TestTypeB
+    member integer: int optional=False
+object TestUnion
+    base TestBase
+    tag enum1
+    case value1: TestTypeA flat=True
+    case value2: TestTypeB flat=True
tests/qapi-schema/ident-with-escape.out +4 -3
index 4028430..24abf5c 100644
@@ -1,3 +1,4 @@
-[OrderedDict([('command', 'fooA'), ('data', OrderedDict([('bar1', 'str')]))])]
-[]
-[]
+object :obj-fooA-args
+    member bar1: str optional=False
+command fooA :obj-fooA-args -> None
+   gen=True success_response=True
tests/qapi-schema/include-relpath.out +1 -3
index 4ce3dcf..6161b90 100644
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-repetition.out +1 -3
index 4ce3dcf..6161b90 100644
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-simple.out +1 -3
index 4ce3dcf..6161b90 100644
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/indented-expr.out +4 -3
index b5ce915..c5af55a 100644
@@ -1,3 +1,4 @@
-[OrderedDict([('command', 'eins')]), OrderedDict([('command', 'zwei')])]
-[]
-[]
+command eins None -> None
+   gen=True success_response=True
+command zwei None -> None
+   gen=True success_response=True
tests/qapi-schema/qapi-schema-test.out +131 -55

186 lines not on this page · raw message

tests/qapi-schema/returns-int.out +2 -3
index 70b3ac5..1ac3e1e 100644
@@ -1,3 +1,2 @@
-[OrderedDict([('command', 'guest-get-time'), ('returns', 'int')])]
-[]
-[]
+command guest-get-time None -> int
+   gen=True success_response=True
tests/qapi-schema/test-qapi.py +30 -7
index 461c713..259f515 100644
@@ -15,11 +15,34 @@ from pprint import pprint
 import os
 import sys
 
-try:
-    exprs = QAPISchema(sys.argv[1]).get_exprs()
-except SystemExit:
-    raise
+class QAPISchemaTestVisitor(QAPISchemaVisitor):
+    def visit_enum_type(self, name, info, values):
+        print 'enum %s %s' % (name, values)
+    def visit_object_type(self, name, info, base, members, variants):
+        print 'object %s' % name
+        if base:
+            print '    base %s' % base.name
+        for m in members:
+            print '    member %s: %s optional=%s' % (m.name, m.type.name,
+                                                     m.optional)
+        self._print_variants(variants)
+    def visit_alternate_type(self, name, info, variants):
+        print 'alternate %s' % name
+        self._print_variants(variants)
+    def visit_command(self, name, info, args, rets, gen, success_response):
+        print 'command %s %s -> %s' % (name, (args and args.name),
+                                       (rets and rets.name))
+        print '   gen=%s success_response=%s' % (gen, success_response)
+    def visit_event(self, name, info, data):
+        print 'event %s %s' % (name, data and data.name)
 
-pprint(exprs)
-pprint(enum_types)
-pprint(struct_types)
+    @staticmethod
+    def _print_variants(variants):
+        if variants:
+            if variants.tag_name:
+                print '    tag %s' % variants.tag_name
+            for v in variants.variants:
+                print '    case %s: %s flat=%s' % (v.name, v.type.name, v.flat)
+
+schema = QAPISchema(sys.argv[1])
+schema.visit(QAPISchemaTestVisitor())
tests/qapi-schema/type-bypass.out +4 -3
index eaf20f8..b147864 100644
@@ -1,3 +1,4 @@
-[OrderedDict([('command', 'unsafe'), ('data', OrderedDict([('arg', '**')])), ('returns', '**'), ('gen', False)])]
-[]
-[]
+object :obj-unsafe-args
+    member arg: ** optional=False
+command unsafe :obj-unsafe-args -> **
+   gen=False success_response=True
Signature · 1 line
1.9.3
#83 lore raw

Re: [Qemu-devel] [PATCH RFC v2 24/47] tests/qapi-schema: Convert test harness to QAPISchemaVisitor

Quoted 19 lines · Markus Armbruster

The old code prints the result of parsing (list of expression dictionaries), and partial results of semantic analysis (list of enum dictionaries, list of struct dictionaries).

The new code prints a trace of a schema visit, i.e. what the back-ends are going to use. Built-in and array types are omitted, because they're boring.

---
 tests/qapi-schema/alternate-good.out            |  15 +-
 tests/qapi-schema/comments.out                  |   4 +-
 tests/qapi-schema/data-member-array.out         |  13 +-
 tests/qapi-schema/empty.out                     |   3 -
 tests/qapi-schema/enum-empty.out                |   4 +-
 tests/qapi-schema/event-case.out                |   4 +-
 tests/qapi-schema/flat-union-reverse-define.out |  21 +--
 tests/qapi-schema/ident-with-escape.out         |   7 +-
 tests/qapi-schema/include-relpath.out           |   4 +-
Quoted 8 lines
 tests/qapi-schema/include-repetition.out        |   4 +-
 tests/qapi-schema/include-simple.out            |   4 +-
 tests/qapi-schema/indented-expr.out             |   7 +-
 tests/qapi-schema/qapi-schema-test.out          | 186 +++++++++++++++++-------
 tests/qapi-schema/returns-int.out               |   5 +-
 tests/qapi-schema/test-qapi.py                  |  37 ++++-
 tests/qapi-schema/type-bypass.out               |   7 +-
 16 files changed, 210 insertions(+), 115 deletions(-)

We have a lot more negative than positive tests of the parser (good thing, because that meant fewer .out files to update to the new format).

No change to actual qemu code, and proves that the previous three patches have set up enough of a framework to accurately cover our testsuite.

Quoted 6 lines
tests/qapi-schema/alternate-good.out +0 -0
index 99848ee..0cbdfa1 100644

tests/qapi-schema/alternate-good.out line 4

Quoted 10 lines
tests/qapi-schema/alternate-good.out +4 -6
@@ -1,6 +1,9 @@
-[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))]),
- OrderedDict([('enum', 'Enum'), ('data', ['hello', 'world'])]),
- OrderedDict([('alternate', 'Alt'), ('data', OrderedDict([('value', 'int'), ('string', 'Enum'), ('struct', 'Data')]))])]
-[{'enum_name': 'Enum', 'enum_values': ['hello', 'world']},
- {'enum_name': 'AltKind', 'enum_values': None}]
-[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))])]
+alternate Alt
+    case value: int flat=False
+    case string: Enum flat=False
+    case struct: Data flat=False

I'm still not convinced whether we need .flat exposed through this much detail, or if we should just normalize plain unions into flat unions with implicit structs for each branch. Changing your design will have obvious ripple effects here.

tests/qapi-schema/alternate-good.out line 2

Quoted 9 lines
tests/qapi-schema/alternate-good.out +3 -5
+++ b/tests/qapi-schema/data-member-array.out
@@ -1,5 +1,8 @@
-[OrderedDict([('enum', 'abc'), ('data', ['a', 'b', 'c'])]),
- OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))]),
- OrderedDict([('command', 'okay'), ('data', OrderedDict([('member1', ['int']), ('member2', ['def'])]))])]
-[{'enum_name': 'abc', 'enum_values': ['a', 'b', 'c']}]
-[OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))])]
+object :obj-okay-args
+    member member1: intList optional=False

Took me a moment to realize the object is an implicit one (named ':obj-okay-args') and not a typo for 'object: obj-okay-args' consistent with members being listed 'name: type'. But not worth changing things, as it is sufficiently unambiguous to serve as a valid test.

Quoted 17 lines
tests/qapi-schema/alternate-good.out +17 -0
+object UserDefFlatUnion
+    base UserDefUnionBase
+    tag enum1
+    case value1: UserDefA flat=True
+    case value2: UserDefB flat=True
+    case value3: UserDefB flat=True
+object UserDefFlatUnion2
+    base UserDefUnionBase
+    tag enum1
+    case value1: UserDefC flat=True
+    case value2: UserDefB flat=True
+    case value3: UserDefA flat=True
+object UserDefNativeListUnion
+    case integer: intList flat=False
+    case s8: int8List flat=False
+    case s16: int16List flat=False
+    case s32: int32List flat=False

tests/qapi-schema/alternate-good.out

Quoted 10 lines
tests/qapi-schema/alternate-good.out +10 -0
+    case s64: int64List flat=False
+    case u8: uint8List flat=False
+    case u16: uint16List flat=False
+    case u32: uint32List flat=False
+    case u64: uint64List flat=False
+    case number: numberList flat=False
+    case boolean: boolList flat=False
+    case string: strList flat=False
+    case sizes: sizeList flat=False
+enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32', 's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes']

Hmm. You are dumping the tag name and type of flat unions, but not of simple unions. I would have expected:

object UserDefNativeListUnion
member kind: UserDefNativeListUnionKind
tag kind
case integer: intList flat=False
...

The above was fallout, while below is the meat of the new visitor.

Quoted 10 lines

+++ b/tests/qapi-schema/test-qapi.py

tests/qapi-schema/alternate-good.out +1 -4
@@ -15,11 +15,34 @@ from pprint import pprint
 import os
 import sys
 
-try:
-    exprs = QAPISchema(sys.argv[1]).get_exprs()
-except SystemExit:
-    raise
+class QAPISchemaTestVisitor(QAPISchemaVisitor):

tests/qapi-schema/alternate-good.out from pprint import pprint · line 28

Quoted 10 lines
tests/qapi-schema/alternate-good.out +10 -0
+    def visit_enum_type(self, name, info, values):
+        print 'enum %s %s' % (name, values)
+    def visit_object_type(self, name, info, base, members, variants):
+        print 'object %s' % name
+        if base:
+            print '    base %s' % base.name
+        for m in members:
+            print '    member %s: %s optional=%s' % (m.name, m.type.name,
+                                                     m.optional)
+        self._print_variants(variants)

So here is where information was missing when visiting a simple union. Why didn't 'kind' get injected into members? And...

Quoted 8 lines
tests/qapi-schema/alternate-good.out +8 -0
+    def visit_alternate_type(self, name, info, variants):
+        print 'alternate %s' % name
+        self._print_variants(variants)
+    def visit_command(self, name, info, args, rets, gen, success_response):
+        print 'command %s %s -> %s' % (name, (args and args.name),
+                                       (rets and rets.name))
+        print '   gen=%s success_response=%s' % (gen, success_response)
+    def visit_event(self, name, info, data):

tests/qapi-schema/alternate-good.out from pprint import pprint · line 43

Quoted 10 lines
tests/qapi-schema/alternate-good.out +6 -3
+        print 'event %s %s' % (name, data and data.name)
 
-pprint(exprs)
-pprint(enum_types)
-pprint(struct_types)
+    @staticmethod
+    def _print_variants(variants):
+        if variants:
+            if variants.tag_name:
+                print '    tag %s' % variants.tag_name

...shouldn't a simple union report 'kind' as its tag_name?

tests/qapi-schema/type-bypass.out

Quoted 6 lines
tests/qapi-schema/alternate-good.out +5 -0
+            for v in variants.variants:
+                print '    case %s: %s flat=%s' % (v.name, v.type.name, v.flat)
+
+schema = QAPISchema(sys.argv[1])
+schema.visit(QAPISchemaTestVisitor())
tests/qapi-schema/type-bypass.out +0 -0

Overall, fairly slick, but I have enough questions about the representation of a simple union that I'll wait for the non-RFC respin to see if you change any design.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#84 lore raw

Re: [Qemu-devel] [PATCH RFC v2 24/47] tests/qapi-schema: Convert test harness to QAPISchemaVisitor

Quoted 53 lines · Eric Blake
Quoted 20 lines · Markus Armbruster

The old code prints the result of parsing (list of expression dictionaries), and partial results of semantic analysis (list of enum dictionaries, list of struct dictionaries).

The new code prints a trace of a schema visit, i.e. what the back-ends are going to use. Built-in and array types are omitted, because they're boring.

---
 tests/qapi-schema/alternate-good.out            |  15 +-
 tests/qapi-schema/comments.out                  |   4 +-
 tests/qapi-schema/data-member-array.out         |  13 +-
 tests/qapi-schema/empty.out                     |   3 -
 tests/qapi-schema/enum-empty.out                |   4 +-
 tests/qapi-schema/event-case.out                |   4 +-
 tests/qapi-schema/flat-union-reverse-define.out |  21 +--
 tests/qapi-schema/ident-with-escape.out         |   7 +-
 tests/qapi-schema/include-relpath.out           |   4 +-
 tests/qapi-schema/include-repetition.out        |   4 +-
Quoted 8 lines
 tests/qapi-schema/include-simple.out            |   4 +-
 tests/qapi-schema/indented-expr.out             |   7 +-
 tests/qapi-schema/qapi-schema-test.out | 186
+++++++++++++++++-------
 tests/qapi-schema/returns-int.out               |   5 +-
 tests/qapi-schema/test-qapi.py                  |  37 ++++-
 tests/qapi-schema/type-bypass.out               |   7 +-
 16 files changed, 210 insertions(+), 115 deletions(-)

We have a lot more negative than positive tests of the parser (good thing, because that meant fewer .out files to update to the new format).

No change to actual qemu code, and proves that the previous three patches have set up enough of a framework to accurately cover our testsuite.

Quoted 17 lines
tests/qapi-schema/alternate-good.out +0 -3
index 99848ee..0cbdfa1 100644
@@ -1,6 +1,9 @@
-[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number',
 'int'), ('*name', 'str')]))]),
- OrderedDict([('enum', 'Enum'), ('data', ['hello', 'world'])]),
- OrderedDict([('alternate', 'Alt'), ('data', OrderedDict([('value',

'int'), ('string', 'Enum'), ('struct', 'Data')]))])] -[{'enum_name': 'Enum', 'enum_values': ['hello', 'world']},
- {'enum_name': 'AltKind', 'enum_values': None}] -[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))])]
+alternate Alt

Quoted 8 lines

tests/qapi-schema/alternate-good.out line 4

Quoted 3 lines
tests/qapi-schema/alternate-good.out +3 -0
+    case value: int flat=False
+    case string: Enum flat=False
+    case struct: Data flat=False

I'm still not convinced whether we need .flat exposed through this much detail, or if we should just normalize plain unions into flat unions with implicit structs for each branch. Changing your design will have obvious ripple effects here.

Yes. I think it's okay as long as we keep it out of external interfaces.

Quoted 9 lines
Quoted 9 lines

+++ b/tests/qapi-schema/data-member-array.out

tests/qapi-schema/alternate-good.out +0 -3
@@ -1,5 +1,8 @@
-[OrderedDict([('enum', 'abc'), ('data', ['a', 'b', 'c'])]),
- OrderedDict([('struct', 'def'), ('data', OrderedDict([('array',
 ['abc'])]))]),
- OrderedDict([('command', 'okay'), ('data',

OrderedDict([('member1', ['int']), ('member2', ['def'])]))])] -[{'enum_name': 'abc', 'enum_values': ['a', 'b', 'c']}] -[OrderedDict([('struct', 'def'), ('data', OrderedDict([('array',

Quoted 8 lines

tests/qapi-schema/alternate-good.out

Quoted 3 lines

['abc'])]))])]

tests/qapi-schema/alternate-good.out +2 -0
+object :obj-okay-args
+    member member1: intList optional=False

Took me a moment to realize the object is an implicit one (named ':obj-okay-args') and not a typo for 'object: obj-okay-args' consistent with members being listed 'name: type'. But not worth changing things, as it is sufficiently unambiguous to serve as a valid test.

Perhaps omitting the ':' after member names would be less confusing.

Quoted 33 lines
Quoted 27 lines
tests/qapi-schema/alternate-good.out +27 -0
+object UserDefFlatUnion
+    base UserDefUnionBase
+    tag enum1
+    case value1: UserDefA flat=True
+    case value2: UserDefB flat=True
+    case value3: UserDefB flat=True
+object UserDefFlatUnion2
+    base UserDefUnionBase
+    tag enum1
+    case value1: UserDefC flat=True
+    case value2: UserDefB flat=True
+    case value3: UserDefA flat=True
+object UserDefNativeListUnion
+    case integer: intList flat=False
+    case s8: int8List flat=False
+    case s16: int16List flat=False
+    case s32: int32List flat=False
+    case s64: int64List flat=False
+    case u8: uint8List flat=False
+    case u16: uint16List flat=False
+    case u32: uint32List flat=False
+    case u64: uint64List flat=False
+    case number: numberList flat=False
+    case boolean: boolList flat=False
+    case string: strList flat=False
+    case sizes: sizeList flat=False
+enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32',
Quoted 2 lines

's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes']

Hmm. You are dumping the tag name and type of flat unions, but not of simple unions. I would have expected:

Quoted 5 lines

object UserDefNativeListUnion
member kind: UserDefNativeListUnionKind
tag kind
case integer: intList flat=False
...

See below.

Quoted 17 lines

The above was fallout, while below is the meat of the new visitor.

Quoted 15 lines

+++ b/tests/qapi-schema/test-qapi.py

tests/qapi-schema/alternate-good.out +6 -4
@@ -15,11 +15,34 @@ from pprint import pprint
 import os
 import sys
 
-try:
-    exprs = QAPISchema(sys.argv[1]).get_exprs()
-except SystemExit:
-    raise
+class QAPISchemaTestVisitor(QAPISchemaVisitor):
+    def visit_enum_type(self, name, info, values):
+        print 'enum %s %s' % (name, values)
+    def visit_object_type(self, name, info, base, members, variants):
+        print 'object %s' % name
+        if base:
Quoted 8 lines

tests/qapi-schema/alternate-good.out from pprint import pprint · line 28

Quoted 5 lines
tests/qapi-schema/alternate-good.out +5 -0
+            print '    base %s' % base.name
+        for m in members:
+            print '    member %s: %s optional=%s' % (m.name, m.type.name,
+                                                     m.optional)
+        self._print_variants(variants)

So here is where information was missing when visiting a simple union. Why didn't 'kind' get injected into members? And...

Because members are the explicit members. A simple union's implicit tag member is in QAPISchemaObjectTypeVariants. See further below.

Quoted 10 lines
Quoted 10 lines
tests/qapi-schema/alternate-good.out +9 -0
+    def visit_alternate_type(self, name, info, variants):
+        print 'alternate %s' % name
+        self._print_variants(variants)
+    def visit_command(self, name, info, args, rets, gen, success_response):
+        print 'command %s %s -> %s' % (name, (args and args.name),
+                                       (rets and rets.name))
+        print '   gen=%s success_response=%s' % (gen, success_response)
+    def visit_event(self, name, info, data):
+        print 'event %s %s' % (name, data and data.name)
 
Quoted 10 lines

tests/qapi-schema/alternate-good.out from pprint import pprint · line 43

Quoted 8 lines
tests/qapi-schema/alternate-good.out +5 -3
-pprint(exprs)
-pprint(enum_types)
-pprint(struct_types)
+    @staticmethod
+    def _print_variants(variants):
+        if variants:
+            if variants.tag_name:
+                print '    tag %s' % variants.tag_name

...shouldn't a simple union report 'kind' as its tag_name?

It could. Here's why it currently isn't.

QAPISchemaObjectType has members .base and .local_members, which come straight from the type definition.

These get flattened into .members.

We do create a simple union's implicit member 'kind', and store it in .variants.tag_member. But we don't add it to .local_members, only to .members, similar to how .base's members are only in .members.

We don't print .members here. Instead, we print .base, .local_members and .variants. The part printing .variants doesn't print .variants.tag_member.

We need to draw the line on what to print *somewhere*. Where exactly is of course debatable. My working idea on what to print here is "print everything that isn't derived from other stuff".

Quoted 11 lines

tests/qapi-schema/type-bypass.out

Quoted 7 lines
tests/qapi-schema/alternate-good.out +5 -0
+            for v in variants.variants:
+ print ' case %s: %s flat=%s' % (v.name, v.type.name, v.flat)
+
+schema = QAPISchema(sys.argv[1])
+schema.visit(QAPISchemaTestVisitor())
tests/qapi-schema/type-bypass.out +0 -0

Overall, fairly slick, but I have enough questions about the representation of a simple union that I'll wait for the non-RFC respin to see if you change any design.

Thanks!

#85 lore raw

RFC PATCH v2 25/47 qapi: Make generators work on sorted schema expressions

In reply to Markus Armbruster

Order of expressions doesn't matter. QAPISchema().get_exprs() returns expressions in the order they're parsed.

I'm going to refactor this code. To check refactorings, I want to diff the generated code, and for that I need to preserve its order.

Since I don't feel like preserving parse order, I'm changing get_expr() to return the expressions sorted by name. This order will be trivial to preserve. It also makes the generated code slightly easier to navigate.

---
 scripts/qapi.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
scripts/qapi.py +8 -1
index cac7ab5..20ffdaf 100644
@@ -1017,7 +1017,14 @@ class QAPISchema(object):
         self.check()
 
     def get_exprs(self):
-        return [expr_elem['expr'] for expr_elem in self.exprs]
+        def expr_name(expr):
+            name = expr.get('enum') or expr.get('union') \
+                   or expr.get('alternate') or expr.get('struct') \
+                   or expr.get('command') or expr.get('event')
+            assert name
+            return name
+        return sorted([expr_elem['expr'] for expr_elem in self.exprs],
+                      key=expr_name)
 
     def _def_entity(self, ent):
         assert ent.name not in self.entity_dict
Signature · 1 line
1.9.3
Reviewed-by on v2 25/47
#86 lore raw

Re: [Qemu-devel] [PATCH RFC v2 25/47] qapi: Make generators work on sorted schema expressions

Quoted 10 lines · Markus Armbruster

Order of expressions doesn't matter. QAPISchema().get_exprs() returns expressions in the order they're parsed.

I'm going to refactor this code. To check refactorings, I want to diff the generated code, and for that I need to preserve its order.

Since I don't feel like preserving parse order, I'm changing get_expr() to return the expressions sorted by name. This order will be trivial to preserve. It also makes the generated code slightly easier to navigate.

Huge change to the generated files, but that's to be expected. Diffstat shows it is not a straight 1:1 reshuffle:

 qapi-event.c                          |  890 +--
 qapi-event.h                          |  190
 qapi-types.c                          | 1996 +++----
 qapi-types.h                          | 5420 ++++++++++----------
 qapi-visit.c                          | 9088
+++++++++++++++++-----------------
 qapi-visit.h                          |  746 +-
 qga/qapi-generated/qga-qapi-types.c   |  148
 qga/qapi-generated/qga-qapi-types.h   |  340 -
 qga/qapi-generated/qga-qapi-visit.c   |  446 -
 qga/qapi-generated/qga-qapi-visit.h   |   54
 qga/qapi-generated/qga-qmp-commands.h |   38
 qga/qapi-generated/qga-qmp-marshal.c  |  864 +--
 qmp-commands.h                        |  376 -
 13 files changed, 10308 insertions(+), 10288 deletions(-)

but that's probably because you now emit forward declarations for things that occur alphabetically after their first use (since 8/47 in this series touched forward declarations), whereas before they happened to occurr in topological order from the parse.

Quoted 11 lines
---
 scripts/qapi.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
scripts/qapi.py +0 -0
index cac7ab5..20ffdaf 100644

scripts/qapi.py class QAPISchema(object): · line 1025

Quoted 10 lines
scripts/qapi.py +6 -1
@@ -1017,7 +1017,14 @@ class QAPISchema(object):
         self.check()
 
     def get_exprs(self):
-        return [expr_elem['expr'] for expr_elem in self.exprs]
+        def expr_name(expr):
+            name = expr.get('enum') or expr.get('union') \
+                   or expr.get('alternate') or expr.get('struct') \
+                   or expr.get('command') or expr.get('event')
+            assert name
+            return name

When I was working on this file earlier, I half toyed with the idea of adding expr_elem['name'] holding the entity name, and expr_elem['meta'] holding what meta-type the entity represents; it might have made some of our later 6-way switches simpler. But with your hierarchy of actual objects, I'm not sure my idea helps any more.

scripts/qapi.py class QAPISchema(object): · line 1027

Quoted 2 lines
scripts/qapi.py +2 -0
+        return sorted([expr_elem['expr'] for expr_elem in self.exprs],
+                      key=expr_name)

Python has some nice compact syntactical gems - I'd hate the amount of boilerplate required to write this same filter using straight C code :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#87 lore raw

Re: [Qemu-devel] [PATCH RFC v2 25/47] qapi: Make generators work on sorted schema expressions

Quoted 30 lines · Eric Blake
Quoted 10 lines · Markus Armbruster

Order of expressions doesn't matter. QAPISchema().get_exprs() returns expressions in the order they're parsed.

I'm going to refactor this code. To check refactorings, I want to diff the generated code, and for that I need to preserve its order.

Since I don't feel like preserving parse order, I'm changing get_expr() to return the expressions sorted by name. This order will be trivial to preserve. It also makes the generated code slightly easier to navigate.

Huge change to the generated files, but that's to be expected. Diffstat shows it is not a straight 1:1 reshuffle:

 qapi-event.c                          |  890 +--
 qapi-event.h                          |  190
 qapi-types.c                          | 1996 +++----
 qapi-types.h                          | 5420 ++++++++++----------
 qapi-visit.c                          | 9088 +++++++++++++++++-----------------
 qapi-visit.h                          |  746 +-
 qga/qapi-generated/qga-qapi-types.c   |  148
 qga/qapi-generated/qga-qapi-types.h   |  340 -
 qga/qapi-generated/qga-qapi-visit.c   |  446 -
 qga/qapi-generated/qga-qapi-visit.h   |   54
 qga/qapi-generated/qga-qmp-commands.h |   38
 qga/qapi-generated/qga-qmp-marshal.c  |  864 +--
 qmp-commands.h                        |  376 -
 13 files changed, 10308 insertions(+), 10288 deletions(-)
Quoted 4 lines

but that's probably because you now emit forward declarations for things that occur alphabetically after their first use (since 8/47 in this series touched forward declarations), whereas before they happened to occurr in topological order from the parse.

Here's my cheap trick for sanity checking this commit: compare before and after *sorted*. Results:

* qapi-event.h
- Members of enum QAPIEvent reordered, values changed accordingly (but they shouldn't matter)

* qapi-visit.c
- A bunch of new forward declarations

* test-qapi-visit.c
- One forward declaration gone.

Quoted 12 lines
Quoted 12 lines
---
 scripts/qapi.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
scripts/qapi.py +0 -0
index cac7ab5..20ffdaf 100644
@@ -1017,7 +1017,14 @@ class QAPISchema(object):
         self.check()
 
Quoted 14 lines

scripts/qapi.py class QAPISchema(object): · line 1025

Quoted 8 lines
scripts/qapi.py +6 -1
     def get_exprs(self):
-        return [expr_elem['expr'] for expr_elem in self.exprs]
+        def expr_name(expr):
+            name = expr.get('enum') or expr.get('union') \
+                   or expr.get('alternate') or expr.get('struct') \
+                   or expr.get('command') or expr.get('event')
+            assert name
+            return name

When I was working on this file earlier, I half toyed with the idea of adding expr_elem['name'] holding the entity name, and expr_elem['meta'] holding what meta-type the entity represents; it might have made some of our later 6-way switches simpler. But with your hierarchy of actual objects, I'm not sure my idea helps any more.

The more work we do with the new internal representation instead of the syntax tree, the less it'll help.

Quoted 7 lines

scripts/qapi.py class QAPISchema(object): · line 1027

Quoted 2 lines
scripts/qapi.py +2 -0
+        return sorted([expr_elem['expr'] for expr_elem in self.exprs],
+                      key=expr_name)

Python has some nice compact syntactical gems - I'd hate the amount of boilerplate required to write this same filter using straight C code :)

Thanks!

#88 lore raw

RFC PATCH v2 26/47 qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

In reply to Markus Armbruster

Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:

    { 'union': 'UserDefFlatUnion',
      'base': 'UserDefUnionBase',
      'discriminator': 'enum1',
      'data': { 'value1' : 'UserDefA',
                'value2' : 'UserDefB',
                'value3' : 'UserDefB' } }

    { 'struct': 'UserDefUnionBase',
      'base': 'UserDefZero',
      'data': { 'string': 'str', 'enum1': 'EnumOne' } }

    { 'struct': 'UserDefZero',
      'data': { 'integer': 'int' } }

Patch's effect on UserDefFlatUnion:

     struct UserDefFlatUnion {
    +    int64_t integer;
         char *string;
         EnumOne enum1;
         /* union tag is EnumOne enum1 */
        union {
            void *data;
            UserDefA *value1;
            UserDefB *value2;
            UserDefB *value3;
        };
    };

Flat union visitors remain broken. They'll be fixed next.

---
 scripts/qapi-types.py                   | 268 +++++++++++++-------------------
 tests/qapi-schema/qapi-schema-test.json |   4 +-
 2 files changed, 114 insertions(+), 158 deletions(-)
scripts/qapi-types.py +112 -156
index a48ad9c..d6185c6 100644
@@ -2,88 +2,67 @@
 # QAPI types generator
 #
 # Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2015 Red Hat Inc.
 #
 # Authors:
 #  Anthony Liguori <[email protected]>
+#  Markus Armbruster <[email protected]>
 #
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 
-def generate_fwd_builtin(name):
-    return mcgen('''
-
-typedef struct %(name)sList {
-    union {
-        %(type)s value;
-        uint64_t padding;
-    };
-    struct %(name)sList *next;
-} %(name)sList;
-''',
-                 type=c_type(name),
-                 name=name)
-
-def generate_fwd_struct(name):
+def gen_fwd_object_or_array(name):
     return mcgen('''
 
 typedef struct %(name)s %(name)s;
-
-typedef struct %(name)sList {
-    union {
-        %(name)s *value;
-        uint64_t padding;
-    };
-    struct %(name)sList *next;
-} %(name)sList;
 ''',
                  name=c_name(name))
 
-def generate_fwd_enum_struct(name):
+def gen_array(name, element_type):
     return mcgen('''
 
-typedef struct %(name)sList {
+struct %(name)s {
     union {
-        %(name)s value;
+        %(c_type)s value;
         uint64_t padding;
     };
-    struct %(name)sList *next;
-} %(name)sList;
+    struct %(name)s *next;
+};
 ''',
-                 name=c_name(name))
+                 name=c_name(name), c_type=element_type.c_type())
 
-def generate_struct_fields(members):
+def gen_struct_field(name, typ, optional):
     ret = ''
 
-    for argname, argentry, optional in parse_args(members):
-        if optional:
-            ret += mcgen('''
+    if optional:
+        ret += mcgen('''
     bool has_%(c_name)s;
 ''',
-                         c_name=c_name(argname))
-        ret += mcgen('''
+                     c_name=c_name(name))
+    ret += mcgen('''
     %(c_type)s %(c_name)s;
 ''',
-                     c_type=c_type(argentry), c_name=c_name(argname))
-
+                 c_type=typ.c_type(), c_name=c_name(name))
     return ret
 
-def generate_struct(expr):
+def generate_struct_fields(members):
+    ret = ''
 
-    structname = expr.get('struct', "")
-    members = expr['data']
-    base = expr.get('base')
+    for memb in members:
+        ret += gen_struct_field(memb.name, memb.type, memb.optional)
+    return ret
 
+def gen_struct(name, base, members):
     ret = mcgen('''
 
 struct %(name)s {
 ''',
-          name=c_name(structname))
+                name=c_name(name))
 
     if base:
-        ret += generate_struct_fields({'base': base})
+        ret += gen_struct_field('base', base, False)
 
     ret += generate_struct_fields(members)
 
@@ -156,46 +135,38 @@ typedef enum %(name)s {
 
     return enum_decl + lookup_decl
 
-def generate_alternate_qtypes(expr):
+def gen_alternate_qtypes_decl(name):
+    return mcgen('''
 
-    name = expr['alternate']
-    members = expr['data']
+extern const int %(c_name)s_qtypes[];
+''',
+                 c_name=c_name(name))
 
+def gen_alternate_qtypes(name, variants):
     ret = mcgen('''
 
 const int %(name)s_qtypes[QTYPE_MAX] = {
 ''',
                 name=c_name(name))
 
-    for key in members:
-        qtype = find_alternate_member_qtype(members[key])
-        assert qtype, "Invalid alternate member"
+    for var in variants.variants:
+        qtype = var.type.alternate_qtype()
+        assert qtype
 
         ret += mcgen('''
     [%(qtype)s] = %(enum_const)s,
 ''',
                      qtype = qtype,
-                     enum_const = c_enum_const(name + 'Kind', key))
+                     enum_const=c_enum_const(variants.tag_member.type.name,
+                                             var.name))
 
     ret += mcgen('''
 };
 ''')
     return ret
 
-
-def generate_union(expr, meta):
-
-    name = c_name(expr[meta])
-    typeinfo = expr['data']
-
-    base = expr.get('base')
-    discriminator = expr.get('discriminator')
-
-    enum_define = discriminator_find_enum_define(expr)
-    if enum_define:
-        discriminator_type_name = enum_define['enum_name']
-    else:
-        discriminator_type_name = '%sKind' % (name)
+def gen_union(name, base, variants):
+    name = c_name(name)
 
     ret = mcgen('''
 
@@ -203,56 +174,49 @@ struct %(name)s {
 ''',
                 name=name)
     if base:
-        base_fields = find_struct(base)['data']
-        ret += generate_struct_fields(base_fields)
+        ret += generate_struct_fields(base.members)
         ret += mcgen('''
     /* union tag is %(discriminator_type_name)s %(c_name)s */
 ''',
-                     discriminator_type_name=c_name(discriminator_type_name),
-                     c_name=c_name(discriminator))
+                     discriminator_type_name=c_name(variants.tag_member.type.name),
+                     c_name=c_name(variants.tag_member.name))
     else:
-        assert not discriminator
         ret += mcgen('''
     %(discriminator_type_name)s kind;
 ''',
-                     discriminator_type_name=c_name(discriminator_type_name))
+                     discriminator_type_name=c_name(variants.tag_member.type.name))
 
     ret += mcgen('''
     union {
         void *data;
 ''')
 
-    for key in typeinfo:
+    for var in variants.variants:
         ret += mcgen('''
         %(c_type)s %(c_name)s;
 ''',
-                     c_type=c_type(typeinfo[key]),
-                     c_name=c_name(key))
+                     c_type=var.type.c_type(),
+                     c_name=c_name(var.name))
 
     ret += mcgen('''
     };
 };
 ''')
-    if meta == 'alternate':
-        ret += mcgen('''
-extern const int %(name)s_qtypes[];
-''',
-            name=name)
-
 
     return ret
 
 def generate_type_cleanup_decl(name):
     ret = mcgen('''
-void qapi_free_%(name)s(%(c_type)s obj);
+
+void qapi_free_%(name)s(%(name)s *obj);
 ''',
-                c_type=c_type(name), name=c_name(name))
+                name=c_name(name))
     return ret
 
 def generate_type_cleanup(name):
     ret = mcgen('''
 
-void qapi_free_%(name)s(%(c_type)s obj)
+void qapi_free_%(name)s(%(name)s *obj)
 {
     QapiDeallocVisitor *md;
     Visitor *v;
@@ -267,9 +231,68 @@ void qapi_free_%(name)s(%(c_type)s obj)
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                c_type=c_type(name), name=c_name(name))
+                name=c_name(name))
     return ret
 
+class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.fwdecl = None
+        self.fwdefn = None
+        self.btin = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.fwdecl = ''
+        self.fwdefn = ''
+        self.btin = guardstart('QAPI_TYPES_BUILTIN')
+    def visit_end(self):
+        self.decl = self.fwdecl + self.decl
+        self.fwdecl = None
+        self.defn = self.fwdefn + self.defn
+        self.fwdefn = None
+        # To avoid header dependency hell, we always generate
+        # declarations for built-in types in our header files and
+        # simply guard them.
+        self.btin += guardend('QAPI_TYPES_BUILTIN')
+        self.decl = self.btin + self.decl
+        self.btin = None
+        # Doesn't work for cases where we link in multiple objects
+        # that have the functions defined, so generate them only with
+        # option -b (do_builtins).
+    def _gen_type_cleanup(self, name):
+        self.decl += generate_type_cleanup_decl(name)
+        self.defn += generate_type_cleanup(name)
+    def visit_enum_type(self, name, info, values):
+        self.fwdecl += generate_enum(name, values)
+        self.fwdefn += generate_enum_lookup(name, values)
+    def visit_array_type(self, name, info, element_type):
+        if isinstance(element_type, QAPISchemaBuiltinType):
+            self.btin += gen_fwd_object_or_array(name)
+            self.btin += gen_array(name, element_type)
+            self.btin += generate_type_cleanup_decl(name)
+            if do_builtins:
+                self.defn += generate_type_cleanup(name)
+        else:
+            self.fwdecl += gen_fwd_object_or_array(name)
+            self.decl += gen_array(name, element_type)
+            self._gen_type_cleanup(name)
+    def visit_object_type(self, name, info, base, members, variants):
+        if info:
+            self.fwdecl += gen_fwd_object_or_array(name)
+            if variants:
+                self.decl += gen_union(name, base, variants)
+            else:
+                self.decl += gen_struct(name, base, members)
+            self._gen_type_cleanup(name)
+    def visit_alternate_type(self, name, info, variants):
+        self.fwdecl += gen_fwd_object_or_array(name)
+        self.fwdefn += gen_alternate_qtypes(name, variants)
+        self.decl += gen_union(name, None, variants)
+        self.decl += gen_alternate_qtypes_decl(name)
+        self._gen_type_cleanup(name)
+
 do_builtins = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
@@ -325,77 +348,10 @@ fdecl.write(mcgen('''
 #include <stdint.h>
 '''))
 
-exprs = QAPISchema(input_file).get_exprs()
-
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_fwd_builtin(typename))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-
-for expr in exprs:
-    ret = ""
-    if expr.has_key('struct'):
-        ret += generate_fwd_struct(expr['struct'])
-    elif expr.has_key('enum'):
-        ret += generate_enum(expr['enum'], expr['data'])
-        ret += generate_fwd_enum_struct(expr['enum'])
-        fdef.write(generate_enum_lookup(expr['enum'], expr['data']))
-    elif expr.has_key('union'):
-        ret += generate_fwd_struct(expr['union'])
-        enum_define = discriminator_find_enum_define(expr)
-        if not enum_define:
-            ret += generate_enum('%sKind' % expr['union'], expr['data'].keys())
-            fdef.write(generate_enum_lookup('%sKind' % expr['union'],
-                                            expr['data'].keys()))
-    elif expr.has_key('alternate'):
-        ret += generate_fwd_struct(expr['alternate'])
-        ret += generate_enum('%sKind' % expr['alternate'], expr['data'].keys())
-        fdef.write(generate_enum_lookup('%sKind' % expr['alternate'],
-                                        expr['data'].keys()))
-        fdef.write(generate_alternate_qtypes(expr))
-    else:
-        continue
-    fdecl.write(ret)
-
-# to avoid header dependency hell, we always generate declarations
-# for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_type_cleanup_decl(typename + "List"))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
-
-# ...this doesn't work for cases where we link in multiple objects that
-# have the functions defined, so we use -b option to provide control
-# over these cases
-if do_builtins:
-    for typename in builtin_types.keys():
-        fdef.write(generate_type_cleanup(typename + "List"))
-
-for expr in exprs:
-    ret = ""
-    if expr.has_key('struct'):
-        ret += generate_struct(expr) + "\n"
-        ret += generate_type_cleanup_decl(expr['struct'] + "List")
-        fdef.write(generate_type_cleanup(expr['struct'] + "List"))
-        ret += generate_type_cleanup_decl(expr['struct'])
-        fdef.write(generate_type_cleanup(expr['struct']))
-    elif expr.has_key('union'):
-        ret += generate_union(expr, 'union') + "\n"
-        ret += generate_type_cleanup_decl(expr['union'] + "List")
-        fdef.write(generate_type_cleanup(expr['union'] + "List"))
-        ret += generate_type_cleanup_decl(expr['union'])
-        fdef.write(generate_type_cleanup(expr['union']))
-    elif expr.has_key('alternate'):
-        ret += generate_union(expr, 'alternate') + "\n"
-        ret += generate_type_cleanup_decl(expr['alternate'] + "List")
-        fdef.write(generate_type_cleanup(expr['alternate'] + "List"))
-        ret += generate_type_cleanup_decl(expr['alternate'])
-        fdef.write(generate_type_cleanup(expr['alternate']))
-    elif expr.has_key('enum'):
-        ret += "\n" + generate_type_cleanup_decl(expr['enum'] + "List")
-        fdef.write(generate_type_cleanup(expr['enum'] + "List"))
-    else:
-        continue
-    fdecl.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenTypeVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
tests/qapi-schema/qapi-schema-test.json +2 -2
index a9e5aab..257b4d4 100644
@@ -39,8 +39,8 @@
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
             'value3' : 'UserDefB' } }
-# FIXME generated struct UserDefFlatUnion has members for direct base
-# UserDefUnionBase, but lacks members for indirect base UserDefZero
+# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit
+# members of indirect base UserDefZero
 
 { 'struct': 'UserDefUnionBase',
   'base': 'UserDefZero',
Signature · 1 line
1.9.3
#89 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

Quoted 6 lines · Markus Armbruster

Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:

Flat union visitors remain broken. They'll be fixed next.

Sadly, the generated files had a huge diffstat, making it very hard to determine if this change is sane:

 qapi-types.c                        | 1412 ++++++-------
 qapi-types.h                        | 3705
++++++++++++++++++++----------------
 qga/qapi-generated/qga-qapi-types.c |  110 -
 qga/qapi-generated/qga-qapi-types.h |  542 +++--
 4 files changed, 3208 insertions(+), 2561 deletions(-)

At first, it looks easy, as in:

qapi-types.c:
+const int BlockdevRef_qtypes[QTYPE_MAX] = {
...
const char *const BlockdevRefKind_lookup[] = {
...
-const int BlockdevRef_qtypes[QTYPE_MAX] = {
...

(that is, the new visitor outputs the two arrays in a different order - I can live with that).

Then it gets a bit crazy when using normal diff algorithms:

-void qapi_free_int32List(int32List *obj)
+void qapi_free_ACPIOSTInfo(ACPIOSTInfo *obj)

where declarations are rearranged (so the previous commit, which attempted to sort declarations to make this one easier, did not quite succeed).

Even in qapi-types.h things were rearranged:

-#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL
-#define QAPI_TYPES_BUILTIN_STRUCT_DECL
+#ifndef QAPI_TYPES_BUILTIN
+#define QAPI_TYPES_BUILTIN

-typedef struct int32List {
+typedef struct boolList boolList;
+
+struct boolList {

I can understand the minor change in #ifdef name, and even the separation between typedef and struct declaration (even though the old approach of doing it all at once works). But the overall diff in the generated files would be easier to review if done in stages, and either make 25/47 sort closer to what this patch does, or add yet more prerequisite patches that further tweak sorting. Ideally, this patch will be a lot easier to review if the generated code is much closer to the pre-patch version (that is, separating sorting changes from other changes). On the other hand, yes, it's kind of a pain to patch old code to do things differently just before throwing it away with the new code in its place, so it becomes a judgment call of how confident we want to be that the new implementation isn't breaking things.

I was able to make a bit more sense of things by using git's 'diff patience' algorithm, which showed things more as code motion rather than one-or-two-line changes to lots of common boilerplate, but even that diffstat is still big:

 qapi-types1.c | 2030 +++++++++++++++----------------
 qapi-types1.h | 3707
+++++++++++++++++++++++++++++++++------------------------
 2 files changed, 3148 insertions(+), 2589 deletions(-)

That said, I'll still at least review this code by inspection, and things still compile fine, so although I'm reluctant to give R-b while the patch is in RFC stage (because I didn't want to take the time to be certain the results are the same amidst so much churn), they are mostly sane.

Quoted 21 lines
---
 scripts/qapi-types.py                   | 268 +++++++++++++-------------------
 tests/qapi-schema/qapi-schema-test.json |   4 +-
 2 files changed, 114 insertions(+), 158 deletions(-)
scripts/qapi-types.py +2 -0
index a48ad9c..d6185c6 100644
@@ -2,88 +2,67 @@
 # QAPI types generator
 #
 # Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2015 Red Hat Inc.
 #
 # Authors:
 #  Anthony Liguori <[email protected]>
+#  Markus Armbruster <[email protected]>
 #

scripts/qapi-types.py line 18

Quoted 10 lines
scripts/qapi-types.py +0 -5
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 
-def generate_fwd_builtin(name):
-    return mcgen('''
-
-typedef struct %(name)sList {

For example, the change of splitting this into:

typedef struct %(name)sList;

struct %(name)sList {

done as a separate patch would make the generated diff of this patch smaller.

scripts/qapi-types.py line 40

Quoted 10 lines
scripts/qapi-types.py +3 -4
-def generate_struct_fields(members):
+def gen_struct_field(name, typ, optional):
     ret = ''
 
-    for argname, argentry, optional in parse_args(members):
-        if optional:
-            ret += mcgen('''
+    if optional:
+        ret += mcgen('''

(sometimes it's annoying that python requires indentation changes when changing control flow; for C files, sometimes I split a patch that merely adds {} and indentation from another patch that changes logic, so that the logic change isn't drowned by the reindentation)

Quoted 17 lines
scripts/qapi-types.py +17 -0
+class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.fwdecl = None
+        self.fwdefn = None
+        self.btin = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.fwdecl = ''
+        self.fwdefn = ''
+        self.btin = guardstart('QAPI_TYPES_BUILTIN')
+    def visit_end(self):
+        self.decl = self.fwdecl + self.decl
+        self.fwdecl = None
+        self.defn = self.fwdefn + self.defn

scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 263

Quoted 10 lines
scripts/qapi-types.py +10 -0
+        self.fwdefn = None
+        # To avoid header dependency hell, we always generate
+        # declarations for built-in types in our header files and
+        # simply guard them.
+        self.btin += guardend('QAPI_TYPES_BUILTIN')
+        self.decl = self.btin + self.decl
+        self.btin = None
+        # Doesn't work for cases where we link in multiple objects
+        # that have the functions defined, so generate them only with
+        # option -b (do_builtins).

Does this comment...

scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 276

Quoted 13 lines
scripts/qapi-types.py +13 -0
+    def _gen_type_cleanup(self, name):
+        self.decl += generate_type_cleanup_decl(name)
+        self.defn += generate_type_cleanup(name)
+    def visit_enum_type(self, name, info, values):
+        self.fwdecl += generate_enum(name, values)
+        self.fwdefn += generate_enum_lookup(name, values)
+    def visit_array_type(self, name, info, element_type):
+        if isinstance(element_type, QAPISchemaBuiltinType):
+            self.btin += gen_fwd_object_or_array(name)
+            self.btin += gen_array(name, element_type)
+            self.btin += generate_type_cleanup_decl(name)
+            if do_builtins:
+                self.defn += generate_type_cleanup(name)

...fit better here?

scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 282

Quoted 6 lines
scripts/qapi-types.py +6 -0
+        else:
+            self.fwdecl += gen_fwd_object_or_array(name)
+            self.decl += gen_array(name, element_type)
+            self._gen_type_cleanup(name)
+    def visit_object_type(self, name, info, base, members, variants):
+        if info:

So to make sure I understand, this ensures that implicit types (those handled merely by parameters in a function or as members of a struct) have no declarations required in the -types files.

scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 285

Quoted 3 lines
scripts/qapi-types.py +3 -0
+            self.fwdecl += gen_fwd_object_or_array(name)
+            if variants:
+                self.decl += gen_union(name, base, variants)

Is it worth 'assert not members' at this point?

scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 295

Quoted 10 lines
scripts/qapi-types.py +10 -0
+            else:
+                self.decl += gen_struct(name, base, members)
+            self._gen_type_cleanup(name)
+    def visit_alternate_type(self, name, info, variants):
+        self.fwdecl += gen_fwd_object_or_array(name)
+        self.fwdefn += gen_alternate_qtypes(name, variants)
+        self.decl += gen_union(name, None, variants)
+        self.decl += gen_alternate_qtypes_decl(name)
+        self._gen_type_cleanup(name)
+

The visitor looks reasonable from inspection review.

scripts/qapi-types.py fdecl.write(mcgen(''' · line 333

Quoted 13 lines

do_builtins = False

(input_file, output_dir, do_c, do_h, prefix, opts) = \

scripts/qapi-types.py +0 -6
@@ -325,77 +348,10 @@ fdecl.write(mcgen('''
 #include <stdint.h>
 '''))
 
-exprs = QAPISchema(input_file).get_exprs()
-
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_fwd_builtin(typename))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))

This is a place where adding sorting in 25/47 (or a separate patch) would make the impact on generated code from this patch smaller.

Quoted 11 lines
scripts/qapi-types.py +0 -11
-
-for expr in exprs:
-    ret = ""
-    if expr.has_key('struct'):
-        ret += generate_fwd_struct(expr['struct'])
-    elif expr.has_key('enum'):
-        ret += generate_enum(expr['enum'], expr['data'])
-        ret += generate_fwd_enum_struct(expr['enum'])
-        fdef.write(generate_enum_lookup(expr['enum'], expr['data']))
-    elif expr.has_key('union'):
-        ret += generate_fwd_struct(expr['union'])

scripts/qapi-types.py fdecl.write(mcgen(''' · line 354

Quoted 10 lines
scripts/qapi-types.py +0 -10
-        enum_define = discriminator_find_enum_define(expr)
-        if not enum_define:
-            ret += generate_enum('%sKind' % expr['union'], expr['data'].keys())
-            fdef.write(generate_enum_lookup('%sKind' % expr['union'],
-                                            expr['data'].keys()))
-    elif expr.has_key('alternate'):
-        ret += generate_fwd_struct(expr['alternate'])
-        ret += generate_enum('%sKind' % expr['alternate'], expr['data'].keys())
-        fdef.write(generate_enum_lookup('%sKind' % expr['alternate'],
-                                        expr['data'].keys()))

Tweaking the order of _qtypes[] vs. _lookup[] here in a pre-req patch would also help review of the diff generated by this patch.

Quoted 8 lines
scripts/qapi-types.py +0 -8
-        fdef.write(generate_alternate_qtypes(expr))
-    else:
-        continue
-    fdecl.write(ret)
-
-# to avoid header dependency hell, we always generate declarations
-# for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))

scripts/qapi-types.py fdecl.write(mcgen(''' · line 372

Quoted 10 lines
scripts/qapi-types.py +0 -10
-for typename in builtin_types.keys():
-    fdecl.write(generate_type_cleanup_decl(typename + "List"))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
-
-# ...this doesn't work for cases where we link in multiple objects that
-# have the functions defined, so we use -b option to provide control
-# over these cases
-if do_builtins:
-    for typename in builtin_types.keys():
-        fdef.write(generate_type_cleanup(typename + "List"))

Another iteration worth sorting in an earlier patch.

scripts/qapi-types.py fdecl.write(mcgen(''' · line 381

Quoted 9 lines
scripts/qapi-types.py +0 -9
-
-for expr in exprs:
-    ret = ""
-    if expr.has_key('struct'):
-        ret += generate_struct(expr) + "\n"
-        ret += generate_type_cleanup_decl(expr['struct'] + "List")
-        fdef.write(generate_type_cleanup(expr['struct'] + "List"))
-        ret += generate_type_cleanup_decl(expr['struct'])
-        fdef.write(generate_type_cleanup(expr['struct']))

I think part of the large size of the generated diff comes from whether array fooList types are emitted in a different order via the visitor.

Quoted 31 lines
scripts/qapi-types.py +5 -18
-    elif expr.has_key('union'):
-        ret += generate_union(expr, 'union') + "\n"
-        ret += generate_type_cleanup_decl(expr['union'] + "List")
-        fdef.write(generate_type_cleanup(expr['union'] + "List"))
-        ret += generate_type_cleanup_decl(expr['union'])
-        fdef.write(generate_type_cleanup(expr['union']))
-    elif expr.has_key('alternate'):
-        ret += generate_union(expr, 'alternate') + "\n"
-        ret += generate_type_cleanup_decl(expr['alternate'] + "List")
-        fdef.write(generate_type_cleanup(expr['alternate'] + "List"))
-        ret += generate_type_cleanup_decl(expr['alternate'])
-        fdef.write(generate_type_cleanup(expr['alternate']))
-    elif expr.has_key('enum'):
-        ret += "\n" + generate_type_cleanup_decl(expr['enum'] + "List")
-        fdef.write(generate_type_cleanup(expr['enum'] + "List"))
-    else:
-        continue
-    fdecl.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenTypeVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
tests/qapi-schema/qapi-schema-test.json +0 -0
index a9e5aab..257b4d4 100644

tests/qapi-schema/qapi-schema-test.json line 46

Quoted 10 lines
tests/qapi-schema/qapi-schema-test.json +2 -2
@@ -39,8 +39,8 @@
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
             'value3' : 'UserDefB' } }
-# FIXME generated struct UserDefFlatUnion has members for direct base
-# UserDefUnionBase, but lacks members for indirect base UserDefZero
+# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit
+# members of indirect base UserDefZero
 
 { 'struct': 'UserDefUnionBase',
   'base': 'UserDefZero',

Overall, moving in the right direction.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#90 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

Quoted 9 lines · Eric Blake
(quoted diff) +3 -1
-typedef struct int32List {
+typedef struct boolList boolList;
+
+struct boolList {

I can understand the minor change in #ifdef name, and even the separation between typedef and struct declaration (even though the old approach of doing it all at once works).

Actually, I don't see any clear-cut policy on the matter (all HACKING says is that "Typedefs are used to eliminate the redundant 'struct' keyword", but does not say how aggressively they must be used). In fact, it looks like most of qemu.git prefers the more compact representation, while your patch goes to the less-common one:

$ git grep 'typedef struct .* {' | wc
   1370    5549   71105
$ git grep 'typedef struct .*;' | wc
    624    2515   41449

However, one thing that your style has going for it, that was not possible with the compact version, is to use:

typedef struct intList intList;

 struct intList {
     union {
         int64_t value;
         uint64_t padding;
     };

- struct intList *next;
+ intList *next;
};

that is, because your typedef is separate, you have the option of being able to drop the use of 'struct' from within the actual intList struct.
Again, if we want that, it would be better to do that as a separate commit.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#91 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

In reply to Eric Blake
Quoted 27 lines · Eric Blake
Quoted 6 lines · Markus Armbruster

Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:

Flat union visitors remain broken. They'll be fixed next.

Sadly, the generated files had a huge diffstat, making it very hard to determine if this change is sane:

 qapi-types.c                        | 1412 ++++++-------
 qapi-types.h                        | 3705 ++++++++++++++++++++----------------
 qga/qapi-generated/qga-qapi-types.c |  110 -
 qga/qapi-generated/qga-qapi-types.h |  542 +++--
 4 files changed, 3208 insertions(+), 2561 deletions(-)

At first, it looks easy, as in:

qapi-types.c:

(quoted diff) +1 -1
+const int BlockdevRef_qtypes[QTYPE_MAX] = {
...
 const char *const BlockdevRefKind_lookup[] = {
...
-const int BlockdevRef_qtypes[QTYPE_MAX] = {
...
Quoted 2 lines

(that is, the new visitor outputs the two arrays in a different order - I can live with that).

The old code generates enum lookup tables for implicit union or alternate enumerations from within the union or alternate code.

The new code doesn't, because it makes all these enums full-fledged types, so the ordinary enum generator does everything we need.

And that's why generating stuff in alphabetical doesn't help here: the new types get inserted in the sorted sequence, but that's not where the old code generated the implicit enum stuff.

Quoted 8 lines

Then it gets a bit crazy when using normal diff algorithms:

(quoted diff) +1 -1
-void qapi_free_int32List(int32List *obj)
+void qapi_free_ACPIOSTInfo(ACPIOSTInfo *obj)

where declarations are rearranged (so the previous commit, which attempted to sort declarations to make this one easier, did not quite succeed).

I found that quite disappointing myself, but I didn't have better ideas at the time.

Quoted 14 lines

Even in qapi-types.h things were rearranged:

(quoted diff) +5 -3
-#ifndef QAPI_TYPES_BUILTIN_STRUCT_DECL
-#define QAPI_TYPES_BUILTIN_STRUCT_DECL
+#ifndef QAPI_TYPES_BUILTIN
+#define QAPI_TYPES_BUILTIN
 
 
-typedef struct int32List {
+typedef struct boolList boolList;
+
+struct boolList {
Quoted 3 lines

I can understand the minor change in #ifdef name, and even the separation between typedef and struct declaration (even though the old approach of doing it all at once works).

Artifact of my attempt to DRY: I sometimes need the typedef in one place (say .h) and the actual struct in another place (say .c). I make do with the functions generating that even when I want both of them in the same place.

Quoted 9 lines

But the overall diff in the
generated files would be easier to review if done in stages, and either make 25/47 sort closer to what this patch does, or add yet more prerequisite patches that further tweak sorting. Ideally, this patch will be a lot easier to review if the generated code is much closer to the pre-patch version (that is, separating sorting changes from other changes). On the other hand, yes, it's kind of a pain to patch old code to do things differently just before throwing it away with the new code in its place,

Especially when you're throwing away the old code because you can't stand working with it :)

Quoted 2 lines

so it becomes a judgment call of how confident we want to
be that the new implementation isn't breaking things.

Different idea: I hack up a script to split file FOO into one part per top-level syntactical construct (typedef, function, whatever), and store each part in a file named like NUM-NAME-FOO. The script will most likely be hairy, so use diff FOO <(cat *-FOO) to verify it works. Then diff old/*-NAME-FOO new/*-NAME-FOO to see what changed.

Writing the script will take non-trivial effort, but I'll try if you think it's needed.

For what it's worth, I sanity-checked with my "diff sorted" trick mentioned in my reply to your review of PATCH 25.

Quoted 9 lines

I was able to make a bit more sense of things by using git's 'diff patience' algorithm, which showed things more as code motion rather than one-or-two-line changes to lots of common boilerplate, but even that diffstat is still big:

 qapi-types1.c | 2030 +++++++++++++++----------------
 qapi-types1.h | 3707 +++++++++++++++++++++++++++++++++------------------------
 2 files changed, 3148 insertions(+), 2589 deletions(-)
Quoted 5 lines

That said, I'll still at least review this code by inspection, and things still compile fine, so although I'm reluctant to give R-b while the patch is in RFC stage (because I didn't want to take the time to be certain the results are the same amidst so much churn), they are mostly sane.

Fair enough.

Quoted 33 lines
Quoted 20 lines
---
 scripts/qapi-types.py | 268 +++++++++++++-------------------
 tests/qapi-schema/qapi-schema-test.json |   4 +-
 2 files changed, 114 insertions(+), 158 deletions(-)
scripts/qapi-types.py +2 -0
index a48ad9c..d6185c6 100644
@@ -2,88 +2,67 @@
 # QAPI types generator
 #
 # Copyright IBM, Corp. 2011
+# Copyright (c) 2013-2015 Red Hat Inc.
 #
 # Authors:
 #  Anthony Liguori <[email protected]>
+#  Markus Armbruster <[email protected]>
 #

scripts/qapi-types.py line 18

Quoted 10 lines
scripts/qapi-types.py +0 -5
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 
-def generate_fwd_builtin(name):
-    return mcgen('''
-
-typedef struct %(name)sList {

For example, the change of splitting this into:

Quoted 6 lines

typedef struct %(name)sList;

struct %(name)sList {

done as a separate patch would make the generated diff of this patch smaller.

I'm willing to mess with the old code *locally* to reduce the diffs, if that helps.

Quoted 4 lines
Quoted 4 lines
scripts/qapi-types.py +1 -1
-def generate_struct_fields(members):
+def gen_struct_field(name, typ, optional):
     ret = ''
 
Quoted 10 lines

scripts/qapi-types.py line 40

Quoted 5 lines
scripts/qapi-types.py +2 -3
-    for argname, argentry, optional in parse_args(members):
-        if optional:
-            ret += mcgen('''
+    if optional:
+        ret += mcgen('''

(sometimes it's annoying that python requires indentation changes when changing control flow; for C files, sometimes I split a patch that merely adds {} and indentation from another patch that changes logic, so that the logic change isn't drowned by the reindentation)

I sometimes use C-c C-w in diff-mode to view a patch hunk without its whitespace changes.

Quoted 44 lines
Quoted 17 lines
scripts/qapi-types.py +17 -0
+class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.fwdecl = None
+        self.fwdefn = None
+        self.btin = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.fwdecl = ''
+        self.fwdefn = ''
+        self.btin = guardstart('QAPI_TYPES_BUILTIN')
+    def visit_end(self):
+        self.decl = self.fwdecl + self.decl
+        self.fwdecl = None
+        self.defn = self.fwdefn + self.defn

scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 263

Quoted 10 lines
scripts/qapi-types.py +10 -0
+        self.fwdefn = None
+        # To avoid header dependency hell, we always generate
+        # declarations for built-in types in our header files and
+        # simply guard them.
+        self.btin += guardend('QAPI_TYPES_BUILTIN')
+        self.decl = self.btin + self.decl
+        self.btin = None
+        # Doesn't work for cases where we link in multiple objects
+        # that have the functions defined, so generate them only with
+        # option -b (do_builtins).

Does this comment...

scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 276

Quoted 13 lines
scripts/qapi-types.py +13 -0
+    def _gen_type_cleanup(self, name):
+        self.decl += generate_type_cleanup_decl(name)
+        self.defn += generate_type_cleanup(name)
+    def visit_enum_type(self, name, info, values):
+        self.fwdecl += generate_enum(name, values)
+        self.fwdefn += generate_enum_lookup(name, values)
+    def visit_array_type(self, name, info, element_type):
+        if isinstance(element_type, QAPISchemaBuiltinType):
+            self.btin += gen_fwd_object_or_array(name)
+            self.btin += gen_array(name, element_type)
+            self.btin += generate_type_cleanup_decl(name)
+            if do_builtins:
+                self.defn += generate_type_cleanup(name)
Quoted 1 line

...fit better here?

If I rephrase it, probably yes.

Quoted 10 lines

scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 282

Quoted 6 lines
scripts/qapi-types.py +6 -0
+        else:
+            self.fwdecl += gen_fwd_object_or_array(name)
+            self.decl += gen_array(name, element_type)
+            self._gen_type_cleanup(name)
+    def visit_object_type(self, name, info, base, members, variants):
+        if info:

So to make sure I understand, this ensures that implicit types (those handled merely by parameters in a function or as members of a struct) have no declarations required in the -types files.

Yes, (not info) means the object type is implicit. We create implicit object types for command parameters, command returns, and event paramaters that aren't type names.

We don't generate C for these types, they're purely internal. We generate *parameter lists* for them.

Hmm, that's actually not true for command returns. And current master duly bombs if I add such a command to qapi-schema-test.json:

{ 'command': 'user_def_cmd4', 'returns': { 'a': 'int' } }

GEN tests/test-qmp-commands.h
Traceback (most recent call last):
File "/work/armbru/qemu/scripts/qapi-commands.py", line 360, in <module>
ret = generate_command_decl(cmd['command'], arglist, ret_type) + "\n"
File "/work/armbru/qemu/scripts/qapi-commands.py", line 29, in generate_command_decl
ret_type=c_type(ret_type), name=c_name(name),
File "/work/armbru/qemu/scripts/qapi.py", line 924, in c_type
assert isinstance(value, str) and value != ""
AssertionError

The easy fix is to outlaw it.

Quoted 5 lines

scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 285

Quoted 3 lines
scripts/qapi-types.py +3 -0
+            self.fwdecl += gen_fwd_object_or_array(name)
+            if variants:
+                self.decl += gen_union(name, base, variants)

Is it worth 'assert not members' at this point?

Yes, with a TODO, because it's an artificial restriction I'd like to lift some day.

Quoted 20 lines

scripts/qapi-types.py void qapi_free_%(name)s(%(c_type)s obj) · line 295

Quoted 10 lines
scripts/qapi-types.py +10 -0
+            else:
+                self.decl += gen_struct(name, base, members)
+            self._gen_type_cleanup(name)
+    def visit_alternate_type(self, name, info, variants):
+        self.fwdecl += gen_fwd_object_or_array(name)
+        self.fwdefn += gen_alternate_qtypes(name, variants)
+        self.decl += gen_union(name, None, variants)
+        self.decl += gen_alternate_qtypes_decl(name)
+        self._gen_type_cleanup(name)
+

The visitor looks reasonable from inspection review.

Quoted 7 lines

do_builtins = False

(input_file, output_dir, do_c, do_h, prefix, opts) = \

scripts/qapi-types.py +0 -0
@@ -325,77 +348,10 @@ fdecl.write(mcgen('''
 #include <stdint.h>
 '''))
 
Quoted 9 lines

scripts/qapi-types.py fdecl.write(mcgen(''' · line 333

Quoted 6 lines
scripts/qapi-types.py +0 -6
-exprs = QAPISchema(input_file).get_exprs()
-
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_fwd_builtin(typename))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))

This is a place where adding sorting in 25/47 (or a separate patch) would make the impact on generated code from this patch smaller.

You mean iterating over sorted(builtin_types.keys())? I can try and see whether it helps.

Quoted 18 lines
Quoted 18 lines
scripts/qapi-types.py +0 -17
-
-for expr in exprs:
-    ret = ""
-    if expr.has_key('struct'):
-        ret += generate_fwd_struct(expr['struct'])
-    elif expr.has_key('enum'):
-        ret += generate_enum(expr['enum'], expr['data'])
-        ret += generate_fwd_enum_struct(expr['enum'])
-        fdef.write(generate_enum_lookup(expr['enum'], expr['data']))
-    elif expr.has_key('union'):
-        ret += generate_fwd_struct(expr['union'])
-        enum_define = discriminator_find_enum_define(expr)
-        if not enum_define:
- ret += generate_enum('%sKind' % expr['union'],
 expr['data'].keys())
-            fdef.write(generate_enum_lookup('%sKind' % expr['union'],
-                                            expr['data'].keys()))
-    elif expr.has_key('alternate'):
Quoted 8 lines

scripts/qapi-types.py fdecl.write(mcgen(''' · line 354

Quoted 5 lines
scripts/qapi-types.py +0 -4
-        ret += generate_fwd_struct(expr['alternate'])
- ret += generate_enum('%sKind' % expr['alternate'],
 expr['data'].keys())
-        fdef.write(generate_enum_lookup('%sKind' % expr['alternate'],
-                                        expr['data'].keys()))

Tweaking the order of _qtypes[] vs. _lookup[] here in a pre-req patch would also help review of the diff generated by this patch.

Can try and see.

Quoted 12 lines
Quoted 12 lines
scripts/qapi-types.py +0 -12
-        fdef.write(generate_alternate_qtypes(expr))
-    else:
-        continue
-    fdecl.write(ret)
-
-# to avoid header dependency hell, we always generate declarations
-# for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_type_cleanup_decl(typename + "List"))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
-
Quoted 8 lines

scripts/qapi-types.py fdecl.write(mcgen(''' · line 372

Quoted 6 lines
scripts/qapi-types.py +0 -6
-# ...this doesn't work for cases where we link in multiple objects that
-# have the functions defined, so we use -b option to provide control
-# over these cases
-if do_builtins:
-    for typename in builtin_types.keys():
-        fdef.write(generate_type_cleanup(typename + "List"))

Another iteration worth sorting in an earlier patch.

Can try and see.

Quoted 4 lines
Quoted 4 lines
scripts/qapi-types.py +0 -4
-
-for expr in exprs:
-    ret = ""
-    if expr.has_key('struct'):
Quoted 8 lines

scripts/qapi-types.py fdecl.write(mcgen(''' · line 381

Quoted 5 lines
scripts/qapi-types.py +0 -5
-        ret += generate_struct(expr) + "\n"
-        ret += generate_type_cleanup_decl(expr['struct'] + "List")
-        fdef.write(generate_type_cleanup(expr['struct'] + "List"))
-        ret += generate_type_cleanup_decl(expr['struct'])
-        fdef.write(generate_type_cleanup(expr['struct']))

I think part of the large size of the generated diff comes from whether array fooList types are emitted in a different order via the visitor.

That's harder to avoid. The old code generates them from within the code generating the element type. The new code doesn't, because it makes all array types full-fledged types, which get visited like any other type.

Quoted 38 lines
Quoted 38 lines
scripts/qapi-types.py +5 -18
-    elif expr.has_key('union'):
-        ret += generate_union(expr, 'union') + "\n"
-        ret += generate_type_cleanup_decl(expr['union'] + "List")
-        fdef.write(generate_type_cleanup(expr['union'] + "List"))
-        ret += generate_type_cleanup_decl(expr['union'])
-        fdef.write(generate_type_cleanup(expr['union']))
-    elif expr.has_key('alternate'):
-        ret += generate_union(expr, 'alternate') + "\n"
-        ret += generate_type_cleanup_decl(expr['alternate'] + "List")
-        fdef.write(generate_type_cleanup(expr['alternate'] + "List"))
-        ret += generate_type_cleanup_decl(expr['alternate'])
-        fdef.write(generate_type_cleanup(expr['alternate']))
-    elif expr.has_key('enum'):
-        ret += "\n" + generate_type_cleanup_decl(expr['enum'] + "List")
-        fdef.write(generate_type_cleanup(expr['enum'] + "List"))
-    else:
-        continue
-    fdecl.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenTypeVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)

close_output(fdef, fdecl)

tests/qapi-schema/qapi-schema-test.json +2 -2
index a9e5aab..257b4d4 100644
@@ -39,8 +39,8 @@
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
             'value3' : 'UserDefB' } }
-# FIXME generated struct UserDefFlatUnion has members for direct base
-# UserDefUnionBase, but lacks members for indirect base UserDefZero
+# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit
+# members of indirect base UserDefZero
 
Quoted 5 lines

tests/qapi-schema/qapi-schema-test.json line 46

Quoted 3 lines
tests/qapi-schema/qapi-schema-test.json +0 -0
 { 'struct': 'UserDefUnionBase',
   'base': 'UserDefZero',

Overall, moving in the right direction.

Thanks!

#92 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

In reply to Markus Armbruster
Quoted 3 lines · Markus Armbruster

Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:

Okay, I see a cause for part of my confusion.

Quoted 4 lines
(quoted diff) +2 -0
+class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
+    def visit_end(self):

Review

Quoted 10 lines
(quoted diff) +10 -0
+        self.decl = self.fwdecl + self.decl
+        self.fwdecl = None
+        self.defn = self.fwdefn + self.defn
+        self.fwdefn = None
+        # To avoid header dependency hell, we always generate
+        # declarations for built-in types in our header files and
+        # simply guard them.
+        self.btin += guardend('QAPI_TYPES_BUILTIN')
+        self.decl = self.btin + self.decl
+        self.btin = None

The new code goes to great lengths to separate all builtin decls up front. But...

Quoted 6 lines
(quoted diff) +6 -0
+        # Doesn't work for cases where we link in multiple objects
+        # that have the functions defined, so generate them only with
+        # option -b (do_builtins).
+    def _gen_type_cleanup(self, name):
+        self.decl += generate_type_cleanup_decl(name)
+        self.defn += generate_type_cleanup(name)

Review

Quoted 10 lines
(quoted diff) +10 -0
+    def visit_enum_type(self, name, info, values):
+        self.fwdecl += generate_enum(name, values)
+        self.fwdefn += generate_enum_lookup(name, values)
+    def visit_array_type(self, name, info, element_type):
+        if isinstance(element_type, QAPISchemaBuiltinType):
+            self.btin += gen_fwd_object_or_array(name)
+            self.btin += gen_array(name, element_type)
+            self.btin += generate_type_cleanup_decl(name)
+            if do_builtins:
+                self.defn += generate_type_cleanup(name)

...it is still interleaving builtin defns with everything else.

Review

Quoted 6 lines
(quoted diff) +0 -6
-exprs = QAPISchema(input_file).get_exprs()
-
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_fwd_builtin(typename))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))

Meanwhile, the old code did builtin intList definitions up front...

Review

Quoted 11 lines
(quoted diff) +0 -10
-
-for expr in exprs:
-    ret = ""
-    if expr.has_key('struct'):
 
-# to avoid header dependency hell, we always generate declarations
-# for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_type_cleanup_decl(typename + "List"))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))

...but declared the cleanup functions at the end (so code motion 1 is that the cleanup functions are now interleaved with the intList declarations)...

Review

Quoted 7 lines
(quoted diff) +0 -7
-
-# ...this doesn't work for cases where we link in multiple objects that
-# have the functions defined, so we use -b option to provide control
-# over these cases
-if do_builtins:
-    for typename in builtin_types.keys():
-        fdef.write(generate_type_cleanup(typename + "List"))

...and wrote the definitions at the end (so code motion 2 is that the definitions are now interleaved with all other definitions).

Attached a few quick hacks that reduce (but not eliminate) some of the visitor's churn to the generated files, by consolidating the location of the builtins and cleaning up struct declarations as separate cleanups (to be applied prior to 26/47), then updating the visitor to use the same builtin order (to be applied after or squashed with 26/47).

It still doesn't solve everything, but with those hacks, I was able to get to a slightly more manageable:

 qapi-types.c                        |  864 ++++----
 qapi-types.h                        | 3602
++++++++++++++++++------------------
 qga/qapi-generated/qga-qapi-types.c |  110 -
 qga/qapi-generated/qga-qapi-types.h |  405 ++--
 4 files changed, 2585 insertions(+), 2396 deletions(-)

I'm sure there are further things that could be done, but at this point, I hope you get my picture, and I'll quit focusing on this particular patch.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

From 00df19b3b0d497e0a4670308f971ca95383b20cd Mon Sep 17 00:00:00 2001
From: Eric Blake <[email protected]>
Date: Wed, 22 Jul 2015 13:59:20 -0600
Subject: [PATCH 1/4] qapi-types: sort and consolidate builtins

Just as sorting generated code based on user-defined structs will help analysis of later patches, we also want to sort builtin types rather than letting python randomly dump a hash table. We also want to consolidate the two conditional areas of the generated .h file into one.

---
 scripts/qapi-types.py | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)
scripts/qapi-types.py +5 -9
index a48ad9c..b7028f6 100644
@@ -327,11 +327,6 @@ fdecl.write(mcgen('''
 
 exprs = QAPISchema(input_file).get_exprs()
 
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_fwd_builtin(typename))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-
 for expr in exprs:
     ret = ""
     if expr.has_key('struct'):
@@ -359,16 +354,17 @@ for expr in exprs:
 
 # to avoid header dependency hell, we always generate declarations
 # for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
-for typename in builtin_types.keys():
+fdecl.write(guardstart("QAPI_TYPES_BUILTIN"))
+for typename in sorted(builtin_types.keys()):
+    fdecl.write(generate_fwd_builtin(typename))
     fdecl.write(generate_type_cleanup_decl(typename + "List"))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
+fdecl.write(guardend("QAPI_TYPES_BUILTIN"))
 
 # ...this doesn't work for cases where we link in multiple objects that
 # have the functions defined, so we use -b option to provide control
 # over these cases
 if do_builtins:
-    for typename in builtin_types.keys():
+    for typename in sorted(builtin_types.keys()):
         fdef.write(generate_type_cleanup(typename + "List"))
 
 for expr in exprs:
Signature · 1 line
2.4.3

From eac8ce42799ecb17271a70aa52185b54173de399 Mon Sep 17 00:00:00 2001
From: Eric Blake <[email protected]>
Date: Wed, 22 Jul 2015 14:00:06 -0600
Subject: [PATCH 2/4] qapi: update typedef usage

Change from:

 typedef struct fooList {
     ...
     struct fooList *next;

} fooList;

to:

typedef struct fooList fooList;

 struct fooList {
     ...
     struct fooList *next;

};

If desired, we could further eliminate the now-spurious 'struct' from the 'next' member of the list.

Note that qemu does not have any standard on which of the two forms is preferred, and that more uses tend to go with the typedef and declaration smashed into one.

---
 scripts/qapi-types.py | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
scripts/qapi-types.py +12 -6
index b7028f6..d21381c 100644
@@ -15,13 +15,15 @@ from qapi import *
 def generate_fwd_builtin(name):
     return mcgen('''
 
-typedef struct %(name)sList {
+typedef struct %(name)sList %(name)sList;
+
+struct %(name)sList {
     union {
         %(type)s value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-} %(name)sList;
+};
 ''',
                  type=c_type(name),
                  name=name)
@@ -31,26 +33,30 @@ def generate_fwd_struct(name):
 
 typedef struct %(name)s %(name)s;
 
-typedef struct %(name)sList {
+typedef struct %(name)sList %(name)sList;
+
+struct %(name)sList {
     union {
         %(name)s *value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-} %(name)sList;
+};
 ''',
                  name=c_name(name))
 
 def generate_fwd_enum_struct(name):
     return mcgen('''
 
-typedef struct %(name)sList {
+typedef struct %(name)sList %(name)sList;
+
+struct %(name)sList {
     union {
         %(name)s value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-} %(name)sList;
+};
 ''',
                  name=c_name(name))
 
Signature · 1 line
2.4.3

From 348f3d3d3a35afe7dca4b695972677ce03da91ca Mon Sep 17 00:00:00 2001
From: Eric Blake <[email protected]>
Date: Wed, 22 Jul 2015 14:58:25 -0600
Subject: [PATCH 4/4] qapi: squash to visitor

Prior to conversion to a visitor, all code related to lists of builtins were lumped together. Doing so in the visitor code reduces the churn in comparing the generated files, making it easier to prove the conversion to the visitor is correct.

---
 scripts/qapi-types.py | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)
scripts/qapi-types.py +15 -16
index d6185c6..0817628 100644
@@ -240,27 +240,26 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.defn = None
         self.fwdecl = None
         self.fwdefn = None
-        self.btin = None
+        self.btindecl = None
+        self.btindefn = None
     def visit_begin(self):
         self.decl = ''
         self.defn = ''
         self.fwdecl = ''
         self.fwdefn = ''
-        self.btin = guardstart('QAPI_TYPES_BUILTIN')
+        self.btindecl = guardstart('QAPI_TYPES_BUILTIN')
+        self.btindefn = ''
     def visit_end(self):
-        self.decl = self.fwdecl + self.decl
-        self.fwdecl = None
-        self.defn = self.fwdefn + self.defn
-        self.fwdefn = None
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
         # simply guard them.
-        self.btin += guardend('QAPI_TYPES_BUILTIN')
-        self.decl = self.btin + self.decl
-        self.btin = None
-        # Doesn't work for cases where we link in multiple objects
-        # that have the functions defined, so generate them only with
-        # option -b (do_builtins).
+        self.btindecl += guardend('QAPI_TYPES_BUILTIN')
+        self.decl = self.fwdecl + self.btindecl + self.decl
+        self.fwdecl = None
+        self.btindecl = None
+        self.defn = self.fwdefn + self.btindefn + self.defn
+        self.fwdefn = None
+        self.btindefn = None
     def _gen_type_cleanup(self, name):
         self.decl += generate_type_cleanup_decl(name)
         self.defn += generate_type_cleanup(name)
@@ -269,11 +268,11 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.fwdefn += generate_enum_lookup(name, values)
     def visit_array_type(self, name, info, element_type):
         if isinstance(element_type, QAPISchemaBuiltinType):
-            self.btin += gen_fwd_object_or_array(name)
-            self.btin += gen_array(name, element_type)
-            self.btin += generate_type_cleanup_decl(name)
+            self.btindecl += gen_fwd_object_or_array(name)
+            self.btindecl += gen_array(name, element_type)
+            self.btindecl += generate_type_cleanup_decl(name)
             if do_builtins:
-                self.defn += generate_type_cleanup(name)
+                self.btindefn += generate_type_cleanup(name)
         else:
             self.fwdecl += gen_fwd_object_or_array(name)
             self.decl += gen_array(name, element_type)
Signature · 1 line
2.4.3
#93 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

Quoted 9 lines · Eric Blake
Quoted 3 lines · Markus Armbruster

Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:

Okay, I see a cause for part of my confusion.

Quoted 1 line
Quoted 8 lines

...and wrote the definitions at the end (so code motion 2 is that the definitions are now interleaved with all other definitions).

Attached a few quick hacks that reduce (but not eliminate) some of the visitor's churn to the generated files, by consolidating the location of the builtins and cleaning up struct declarations as separate cleanups (to be applied prior to 26/47), then updating the visitor to use the same builtin order (to be applied after or squashed with 26/47).

By the way, I'm perfectly fine if the initial conversion to the visitor keeps .c output separate into two groups to make review easier, then a later patch simplifies things back to a single sorting over the whole file (and undoing the grouping after conversion to visitor will be easier than figuring out how to interleave the builtin array visitors into the right sorted order prior to use of the visitor). I like the final result of everything sorted, it's just that I think it is easier to validate the final result when it is broken into stages with no one step doing too much, even if some of the intermediate stages do more work than needed and get undone by the final stage.

Or put another way, your next spin of the series could include your original 26 + my hack 003 squashed together as one patch (conversion to visitor with minimized generated diff noise), followed by a revert of my hack 003 as the next (no need to group builtins together in the long run).

[and for the record, I mention this because that's the approach I took in my earlier cleanups: commit fd41dd4e included a hack to rename 'type' to 'struct' in the generator, so that commit 895a2a80 could be quite mindless without touching the generator, and then commit 3e391d35 undid my hack. When I originally wrote the conversion in an earlier version of the series, I did not have that hack in place and the resulting single patch was too big; breaking it up made my submission and the review easier.]

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#94 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

In reply to Eric Blake
Quoted 22 lines · Eric Blake
Quoted 3 lines · Markus Armbruster

Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:

Okay, I see a cause for part of my confusion.

Quoted 4 lines
(quoted diff) +2 -0
+class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
+    def visit_end(self):

Review

Quoted 10 lines
(quoted diff) +10 -0
+        self.decl = self.fwdecl + self.decl
+        self.fwdecl = None
+        self.defn = self.fwdefn + self.defn
+        self.fwdefn = None
+        # To avoid header dependency hell, we always generate
+        # declarations for built-in types in our header files and
+        # simply guard them.
+        self.btin += guardend('QAPI_TYPES_BUILTIN')
+        self.decl = self.btin + self.decl
+        self.btin = None
Quoted 2 lines

The new code goes to great lengths to separate all builtin decls up front. But...

Because they all need to be under the QAPI_TYPES_BUILTIN guard. If I didn't separate them, each of them would need its own guard, which would be ugly.

Quoted 17 lines
Quoted 6 lines
(quoted diff) +6 -0
+        # Doesn't work for cases where we link in multiple objects
+        # that have the functions defined, so generate them only with
+        # option -b (do_builtins).
+    def _gen_type_cleanup(self, name):
+        self.decl += generate_type_cleanup_decl(name)
+        self.defn += generate_type_cleanup(name)

Review

Quoted 10 lines
(quoted diff) +10 -0
+    def visit_enum_type(self, name, info, values):
+        self.fwdecl += generate_enum(name, values)
+        self.fwdefn += generate_enum_lookup(name, values)
+    def visit_array_type(self, name, info, element_type):
+        if isinstance(element_type, QAPISchemaBuiltinType):
+            self.btin += gen_fwd_object_or_array(name)
+            self.btin += gen_array(name, element_type)
+            self.btin += generate_type_cleanup_decl(name)
+            if do_builtins:
+                self.defn += generate_type_cleanup(name)
Quoted 1 line

...it is still interleaving builtin defns with everything else.

Because there's no such need for the builtin definitions.

Quoted 42 lines

scripts/qapi-types.py fdecl.write(mcgen(''' · line 333

Quoted 6 lines
scripts/qapi-types.py +0 -6
-exprs = QAPISchema(input_file).get_exprs()
-
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_fwd_builtin(typename))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_STRUCT_DECL"))

Meanwhile, the old code did builtin intList definitions up front...

Review

Quoted 11 lines
(quoted diff) +0 -10
-
-for expr in exprs:
-    ret = ""
-    if expr.has_key('struct'):
 
-# to avoid header dependency hell, we always generate declarations
-# for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_type_cleanup_decl(typename + "List"))
-fdecl.write(guardend("QAPI_TYPES_BUILTIN_CLEANUP_DECL"))

...but declared the cleanup functions at the end (so code motion 1 is that the cleanup functions are now interleaved with the intList declarations)...

Review

Quoted 7 lines
(quoted diff) +0 -7
-
-# ...this doesn't work for cases where we link in multiple objects that
-# have the functions defined, so we use -b option to provide control
-# over these cases
-if do_builtins:
-    for typename in builtin_types.keys():
-        fdef.write(generate_type_cleanup(typename + "List"))

...and wrote the definitions at the end (so code motion 2 is that the definitions are now interleaved with all other definitions).

Attached a few quick hacks that reduce (but not eliminate) some of the visitor's churn to the generated files, by consolidating the location of the builtins and cleaning up struct declarations as separate cleanups (to be applied prior to 26/47), then updating the visitor to use the same builtin order (to be applied after or squashed with 26/47).

Quoted 8 lines

It still doesn't solve everything, but with those hacks, I was able to get to a slightly more manageable:

 qapi-types.c                        |  864 ++++----
 qapi-types.h                        | 3602 ++++++++++++++++++------------------
 qga/qapi-generated/qga-qapi-types.c |  110 -
 qga/qapi-generated/qga-qapi-types.h |  405 ++--
 4 files changed, 2585 insertions(+), 2396 deletions(-)

Still plenty bad...

Quoted 2 lines

I'm sure there are further things that could be done, but at this point, I hope you get my picture, and I'll quit focusing on this particular patch.

We need to decide how much code churn to accept just for making the diff of the generated code easier to review.

Thanks!

#95 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

Quoted 6 lines · Markus Armbruster
Quoted 3 lines

I'm sure there are further things that could be done, but at this point, I hope you get my picture, and I'll quit focusing on this particular patch.

We need to decide how much code churn to accept just for making the diff of the generated code easier to review.

At this point, I'd be happy with just adding a script or other high-level instructions in the commit message that says how to divide a generated file into pieces (pull out all *List types into one piece, all typedefs into another, etc) and which can be done both pre- and post-patch. With pieces in hand, if you can easily compare that each pair is minimally different, then you have a nice reassurance that the difference in the overall file is due merely to differences in how the pieces are interleaved, and not to added or unintentionally dropped material. As you pointed out, there comes a point of diminishing returns in trying to clean up code that will just be discarded.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#96 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

Quoted 9 lines · Eric Blake
Quoted 6 lines · Markus Armbruster
Quoted 3 lines

I'm sure there are further things that could be done, but at this point, I hope you get my picture, and I'll quit focusing on this particular patch.

We need to decide how much code churn to accept just for making the diff of the generated code easier to review.

Quoted 10 lines

At this point, I'd be happy with just adding a script or other high-level instructions in the commit message that says how to divide a generated file into pieces (pull out all *List types into one piece, all typedefs into another, etc) and which can be done both pre- and post-patch. With pieces in hand, if you can easily compare that each pair is minimally different, then you have a nice reassurance that the difference in the overall file is due merely to differences in how the pieces are interleaved, and not to added or unintentionally dropped material. As you pointed out, there comes a point of diminishing returns in trying to clean up code that will just be discarded.

I'll give it a try.

#97 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

In reply to Markus Armbruster
Quoted 17 lines · Markus Armbruster

Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:

scripts/qapi-types.py +6 -3
-def generate_alternate_qtypes(expr):
+def gen_alternate_qtypes_decl(name):
+    return mcgen('''
 
-    name = expr['alternate']
-    members = expr['data']
+extern const int %(c_name)s_qtypes[];
+''',
+                 c_name=c_name(name))
 
+def gen_alternate_qtypes(name, variants):
     ret = mcgen('''
 

scripts/qapi-types.py typedef enum %(name)s { · line 154

Quoted 10 lines
scripts/qapi-types.py +3 -3
 const int %(name)s_qtypes[QTYPE_MAX] = {
 ''',
                 name=c_name(name))
 
-    for key in members:
-        qtype = find_alternate_member_qtype(members[key])
-        assert qtype, "Invalid alternate member"
+    for var in variants.variants:
+        qtype = var.type.alternate_qtype()
+        assert qtype

I think I found a couple more corner case bugs here. We are using C99 initialization of the array; for example:

const int BlockdevRef_qtypes[QTYPE_MAX] = {
[QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION, [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE,
};

but paired with an enum that starts at 0:

typedef enum BlockdevRefKind {
BLOCKDEV_REF_KIND_DEFINITION = 0,
BLOCKDEV_REF_KIND_REFERENCE = 1,
BLOCKDEV_REF_KIND_MAX = 2,
} BlockdevRefKind;

and that means that every QTYPE_ constant that we don't specify in _qtypes[] is also assigned the value 0 (aka BLOCKDEV_REF_KIND_DEFINITION in this example). In operation, calling something like:

{"execute":"blockdev-add","arguments":{"options":
{"driver":"raw","id":"a","file":true}}}

which is invalid per the .json description ("file" must be string or object, not boolean), still manages to get past visit_get_next_type() with success, and fall through to the 0th branch of the switch. If that 0th branch happens to be a struct (as it is for BlockdevRef), then we fortunately catch the error on the very next parse call, where qmp_input_start_struct() complains:

{"error": {"class": "GenericError", "desc": "Invalid parameter type for 'file', expected: QDict"}}

But what happens if the 0th branch is mapped to a different parser, as would be the case if one of the alternate's branches is 'number'? In particular, qmp_input_type_number() accepts BOTH QFloat and QInt types.
So, if we have this qapi:
{ 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
but pass in an integer, visit_get_next_type() will see a qtype of QInt, but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the string parser doesn't like ints) even though the parse should succeed by using the FOO_KIND_B branch.

Interestingly, this means that if we ever write an alternate type that accepts both 'int' and 'number' (we have not attempted that so far), then the number branch will only be taken for inputs that don't also look like ints (normally, 'number' accepts anything numeric). Maybe that means we should document and enforce that 'number' and 'int' cannot be mixed in the same alternate?

So, the bugs are: visit_get_next_type() can't tell the difference between a *_qtypes[] lookup that was explicitly initialized to 0 from one that was accidentally left that way, and therefore can't report failure for an unexpected type (but mostly mitigated by the fact that always returning 0 means the parser will attempt to parse the first branch of the alternate and gracefully fail); and that we don't properly handle QInt for an alternate that accepts 'number' but not 'int'.

I don't think either bug has to be fixed in your series, although you may want to add tests.

The first bug could be resolved by guaranteeing that the _qtypes[] array has non-zero values for the explicitly initialized lookups, and teaching visit_get_next_type() that a lookup that produces 0 meant that an unexpected type was encountered. Perhaps by changing the creation of _qtypes[] in qapi-types.c to list:

const int BlockdevRef_qtypes[QTYPE_MAX] = {
[QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION + 1, [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE + 1,
};

and then having visit_get_next_type() subtract one after verifying a non-zero value was looked up. Or perhaps leave _qtypes alone, and instead change the alternate enum to have a placeholder at 0:

typedef enum BlockdevRefKind {
BLOCKDEV_REF_KIND_INVALID = 0,
BLOCKDEV_REF_KIND_DEFINITION = 1,
BLOCKDEV_REF_KIND_REFERENCE = 2,
BLOCKDEV_REF_KIND_MAX = 3,
} BlockdevRefKind;

and then teaching the generator for visit_type_BlockdevRef() to emit an error if branch 0 is hit.

Fixing the second bug probably entails teaching the generator that if an alternate contains 'number' but not 'int', then we need [QTYPE_QINT] to map to the same lookup value as [QTYPE_QNUMBER].

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#98 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

Quoted 46 lines · Eric Blake
Quoted 17 lines · Markus Armbruster

Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:

scripts/qapi-types.py +6 -3
-def generate_alternate_qtypes(expr):
+def gen_alternate_qtypes_decl(name):
+    return mcgen('''
 
-    name = expr['alternate']
-    members = expr['data']
+extern const int %(c_name)s_qtypes[];
+''',
+                 c_name=c_name(name))
 
+def gen_alternate_qtypes(name, variants):
     ret = mcgen('''
 

scripts/qapi-types.py typedef enum %(name)s { · line 154

Quoted 10 lines
scripts/qapi-types.py +3 -3
 const int %(name)s_qtypes[QTYPE_MAX] = {
 ''',
                 name=c_name(name))
 
-    for key in members:
-        qtype = find_alternate_member_qtype(members[key])
-        assert qtype, "Invalid alternate member"
+    for var in variants.variants:
+        qtype = var.type.alternate_qtype()
+        assert qtype

I think I found a couple more corner case bugs here. We are using C99 initialization of the array; for example:

const int BlockdevRef_qtypes[QTYPE_MAX] = {
[QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION, [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE,
};

but paired with an enum that starts at 0:

typedef enum BlockdevRefKind {
BLOCKDEV_REF_KIND_DEFINITION = 0,
BLOCKDEV_REF_KIND_REFERENCE = 1,
BLOCKDEV_REF_KIND_MAX = 2,
} BlockdevRefKind;

Quoted 2 lines

and that means that every QTYPE_ constant that we don't specify in _qtypes[] is also assigned the value 0 (aka BLOCKDEV_REF_KIND_DEFINITION

I see where this is going...

Quoted 12 lines

in this example). In operation, calling something like:

{"execute":"blockdev-add","arguments":{"options":
{"driver":"raw","id":"a","file":true}}}

which is invalid per the .json description ("file" must be string or object, not boolean), still manages to get past visit_get_next_type() with success, and fall through to the 0th branch of the switch. If that 0th branch happens to be a struct (as it is for BlockdevRef), then we fortunately catch the error on the very next parse call, where qmp_input_start_struct() complains:

Quoted 2 lines

{"error": {"class": "GenericError", "desc": "Invalid parameter type for 'file', expected: QDict"}}

A lucky case.

Quoted 10 lines

But what happens if the 0th branch is mapped to a different parser, as would be the case if one of the alternate's branches is 'number'? In particular, qmp_input_type_number() accepts BOTH QFloat and QInt types.
So, if we have this qapi:
{ 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
but pass in an integer, visit_get_next_type() will see a qtype of QInt, but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the string parser doesn't like ints) even though the parse should succeed by using the FOO_KIND_B branch.

Yup, bug.

Quoted 6 lines

Interestingly, this means that if we ever write an alternate type that accepts both 'int' and 'number' (we have not attempted that so far), then the number branch will only be taken for inputs that don't also look like ints (normally, 'number' accepts anything numeric). Maybe that means we should document and enforce that 'number' and 'int' cannot be mixed in the same alternate?

Even if we outlaw mixing the two, I'm afraid we still have this bug: an alternate with a 'number' member rejects input that gets parsed as QTYPE_QINT.

Let's simply make alternates behave sanely:

    alternate has      case selected for
    'int'  'number'    QTYPE_QINT  QTYPE_QFLOAT
      no        no     error       error
      no       yes     'number'    'number'
     yes        no     'int'       error
     yes       yes     'int'       'number'
Quoted 6 lines

So, the bugs are: visit_get_next_type() can't tell the difference between a *_qtypes[] lookup that was explicitly initialized to 0 from one that was accidentally left that way, and therefore can't report failure for an unexpected type (but mostly mitigated by the fact that always returning 0 means the parser will attempt to parse the first branch of the alternate and gracefully fail);

Yes.

Quoted 2 lines

and that we don't properly
handle QInt for an alternate that accepts 'number' but not 'int'.

Yes.

Quoted 2 lines

I don't think either bug has to be fixed in your series, although you may want to add tests.

Agree.

Quoted 6 lines

The first bug could be resolved by guaranteeing that the _qtypes[] array has non-zero values for the explicitly initialized lookups, and teaching visit_get_next_type() that a lookup that produces 0 meant that an unexpected type was encountered. Perhaps by changing the creation of _qtypes[] in qapi-types.c to list:

Quoted 7 lines

const int BlockdevRef_qtypes[QTYPE_MAX] = {
[QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION + 1, [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE + 1,
};

and then having visit_get_next_type() subtract one after verifying a non-zero value was looked up.

+ 1 works, because the element type is int, not BlockdevRefKind. It's int so it can serve as argument for visit_get_next_type()'s parameter const int *qtypes.

The + 1, - 1 business could be mildly confusing. We could set all unused elements to -1 instead:

    const int BlockdevRef_qtypes[QTYPE_MAX] = {
        [QTYPE_NONE] = -1,
        [QTYPE_QNULL] = -1,
        [QTYPE_QINT] = -1,
        [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE + 1,
        [QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION + 1,
        [QTYPE_QLIST] = -1,
        [QTYPE_QFLOAT] = -1,
        [QTYPE_QBOOL] = -1,
    };

I wouldn't recommend that in hand-written code, but generating it should be fine.

Quoted 10 lines

Or perhaps leave _qtypes alone, and
instead change the alternate enum to have a placeholder at 0:

typedef enum BlockdevRefKind {
BLOCKDEV_REF_KIND_INVALID = 0,
BLOCKDEV_REF_KIND_DEFINITION = 1,
BLOCKDEV_REF_KIND_REFERENCE = 2,
BLOCKDEV_REF_KIND_MAX = 3,
} BlockdevRefKind;

Quoted 2 lines

and then teaching the generator for visit_type_BlockdevRef() to emit an error if branch 0 is hit.

BLOCKDEV_REF_KIND_INVALID could get in the way elsewhere, e.g. with -Wswitch.

Quoted 3 lines

Fixing the second bug probably entails teaching the generator that if an alternate contains 'number' but not 'int', then we need [QTYPE_QINT] to map to the same lookup value as [QTYPE_QNUMBER].

Add test eight test cases from my table above, then fix the generator to make them pass.

#99 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

Quoted 11 lines · Markus Armbruster
Quoted 10 lines

But what happens if the 0th branch is mapped to a different parser, as would be the case if one of the alternate's branches is 'number'? In particular, qmp_input_type_number() accepts BOTH QFloat and QInt types.
So, if we have this qapi:
{ 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
but pass in an integer, visit_get_next_type() will see a qtype of QInt, but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the string parser doesn't like ints) even though the parse should succeed by using the FOO_KIND_B branch.

Quoted 1 line

Yup, bug.

And it's an order-dependent bug - merely declaring 'b' first makes it appear to work correctly.

Quoted 12 lines
Quoted 7 lines

Interestingly, this means that if we ever write an alternate type that accepts both 'int' and 'number' (we have not attempted that so far), then the number branch will only be taken for inputs that don't also look like ints (normally, 'number' accepts anything numeric). Maybe that means we should document and enforce that 'number' and 'int' cannot be mixed in the same alternate?

Even if we outlaw mixing the two, I'm afraid we still have this bug: an alternate with a 'number' member rejects input that gets parsed as QTYPE_QINT.

Quoted 8 lines

Let's simply make alternates behave sanely:

    alternate has      case selected for
    'int'  'number'    QTYPE_QINT  QTYPE_QFLOAT
      no        no     error       error
      no       yes     'number'    'number'
     yes        no     'int'       error
     yes       yes     'int'       'number'

Works for me.

Quoted 7 lines

+ 1 works, because the element type is int, not BlockdevRefKind. It's int so it can serve as argument for visit_get_next_type()'s parameter const int *qtypes.

The + 1, - 1 business could be mildly confusing. We could set all unused elements to -1 instead:

Or, we could ditch the qtypes lookup altogether, and merely create the alternate enum as a non-consecutive QTYPE mapping, for one less level of indirection, as in:

typedef enum BlockdevRefKind {
BLOCKDEV_REF_DEFINITION = QTYPE_QOBJECT,
BLOCKDEV_REF_REFERENCE = QTYPE_QSTRING,
};

then rewrite visit_get_next_type() to directly return the qtype, as well as rewrite the generated switch statement in visit_type_BlockdevRef() to directly inspect the qtypes it cares about. In fact, that's the approach I'm currently playing with.

Quoted 2 lines

Add test eight test cases from my table above, then fix the generator to make them pass.

I hope to post an RFC followup patch along those lines later today.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#100 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

Quoted 50 lines · Eric Blake
Quoted 11 lines · Markus Armbruster
Quoted 10 lines

But what happens if the 0th branch is mapped to a different parser, as would be the case if one of the alternate's branches is 'number'? In particular, qmp_input_type_number() accepts BOTH QFloat and QInt types.
So, if we have this qapi:
{ 'alternate': 'Foo', 'data': { 'a': 'str', 'b': 'number' } }
but pass in an integer, visit_get_next_type() will see a qtype of QInt, but Foo_qtypes[QTYPE_QINT] will be 0 (due to default initialization) and we will wrongly try to visit the 0th branch (FOO_KIND_A) and fail (the string parser doesn't like ints) even though the parse should succeed by using the FOO_KIND_B branch.

Quoted 1 line

Yup, bug.

And it's an order-dependent bug - merely declaring 'b' first makes it appear to work correctly.

Quoted 12 lines
Quoted 7 lines

Interestingly, this means that if we ever write an alternate type that accepts both 'int' and 'number' (we have not attempted that so far), then the number branch will only be taken for inputs that don't also look like ints (normally, 'number' accepts anything numeric). Maybe that means we should document and enforce that 'number' and 'int' cannot be mixed in the same alternate?

Even if we outlaw mixing the two, I'm afraid we still have this bug: an alternate with a 'number' member rejects input that gets parsed as QTYPE_QINT.

Quoted 8 lines

Let's simply make alternates behave sanely:

    alternate has      case selected for
    'int'  'number'    QTYPE_QINT  QTYPE_QFLOAT
      no        no     error       error
      no       yes     'number'    'number'
     yes        no     'int'       error
     yes       yes     'int'       'number'

Works for me.

Quoted 7 lines

+ 1 works, because the element type is int, not BlockdevRefKind. It's int so it can serve as argument for visit_get_next_type()'s parameter const int *qtypes.

The + 1, - 1 business could be mildly confusing. We could set all unused elements to -1 instead:

Quoted 6 lines

Or, we could ditch the qtypes lookup altogether, and merely create the alternate enum as a non-consecutive QTYPE mapping, for one less level of indirection, as in:

typedef enum BlockdevRefKind {
BLOCKDEV_REF_DEFINITION = QTYPE_QOBJECT,

QTYPE_QDICT, but I get what you mean.

Quoted 7 lines

BLOCKDEV_REF_REFERENCE = QTYPE_QSTRING,
};

then rewrite visit_get_next_type() to directly return the qtype, as well as rewrite the generated switch statement in visit_type_BlockdevRef() to directly inspect the qtypes it cares about. In fact, that's the approach I'm currently playing with.

Hmm. The schema currently doesn't let you control enumeration values. qapi-code-gen.txt specifies:

The enumeration values [...] are encoded as C enum integral values in generated code. While the C code starts numbering at 0, it is better to use explicit comparisons to enum values than implicit comparisons to 0; the C code will also include a generated enum member ending in _MAX for tracking the size of the enum, useful when using common functions for converting between strings and enum values.

Strictly speaking, this needn't apply to implicit enums like BlockdevRefKind. But is it a good idea to make them different?

Hmm, your new BlockdevRefKind is basically a subset of qtype_code with the members renamed. Could we simply use qtype_code directly?

Quoted 4 lines
Quoted 2 lines

Add test eight test cases from my table above, then fix the generator to make them pass.

I hope to post an RFC followup patch along those lines later today.

#101 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

Quoted 12 lines · Markus Armbruster
Quoted 6 lines

Or, we could ditch the qtypes lookup altogether, and merely create the alternate enum as a non-consecutive QTYPE mapping, for one less level of indirection, as in:

typedef enum BlockdevRefKind {
BLOCKDEV_REF_DEFINITION = QTYPE_QOBJECT,

QTYPE_QDICT, but I get what you mean.

Quoted 3 lines

BLOCKDEV_REF_REFERENCE = QTYPE_QSTRING,
};

Quoted 6 lines
Quoted 4 lines

then rewrite visit_get_next_type() to directly return the qtype, as well as rewrite the generated switch statement in visit_type_BlockdevRef() to directly inspect the qtypes it cares about. In fact, that's the approach I'm currently playing with.

Hmm. The schema currently doesn't let you control enumeration values.

For public enums. But the enum for alternate is never public - you never send the name of the branch over the wire, so we don't need to stringize the name anywhere.

Quoted 10 lines

qapi-code-gen.txt specifies:

The enumeration values [...] are encoded as C enum integral values in generated code. While the C code starts numbering at 0, it is better to use explicit comparisons to enum values than implicit comparisons to 0; the C code will also include a generated enum member ending in _MAX for tracking the size of the enum, useful when using common functions for converting between strings and enum values.

Quoted 2 lines

Strictly speaking, this needn't apply to implicit enums like BlockdevRefKind. But is it a good idea to make them different?

I think so, but maybe under a different name (maybe BLOCKDEV_REF_REFERENCE_QTYPE) to make it more obvious that this is not the usual generated enum, but special to the alternate.

Quoted 3 lines

Hmm, your new BlockdevRefKind is basically a subset of qtype_code with the members renamed. Could we simply use qtype_code directly?

We could, except that clients that manipulate the generated struct then have to know the qtype mapping directly; while keeping symbolic names lets them do 'foo->type = BLOCKDEV_REF_REFERENCE; foo->reference = xyz;' as a nice visual indicator of which union member within the struct is being assigned according to the discriminator.

I guess I'll see how much code currently manipulates the generated structs (I already recall from other patches in this series that blockdev played a bit loose by validating that the QMP was okay and then using QDict for everything else rather than the generated struct) and make my decision when posting my RFC patch.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#102 lore raw

Re: [Qemu-devel] [PATCH RFC v2 26/47] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

Quoted 24 lines · Eric Blake
Quoted 7 lines · Markus Armbruster
Quoted 6 lines

Or, we could ditch the qtypes lookup altogether, and merely create the alternate enum as a non-consecutive QTYPE mapping, for one less level of indirection, as in:

typedef enum BlockdevRefKind {
BLOCKDEV_REF_DEFINITION = QTYPE_QOBJECT,

Quoted 8 lines

QTYPE_QDICT, but I get what you mean.

Quoted 3 lines

BLOCKDEV_REF_REFERENCE = QTYPE_QSTRING,
};

Hmm, your new BlockdevRefKind is basically a subset of qtype_code with the members renamed. Could we simply use qtype_code directly?

We could, except that clients that manipulate the generated struct then have to know the qtype mapping directly; while keeping symbolic names lets them do 'foo->type = BLOCKDEV_REF_REFERENCE; foo->reference = xyz;' as a nice visual indicator of which union member within the struct is being assigned according to the discriminator.

Quoted 5 lines

I guess I'll see how much code currently manipulates the generated structs (I already recall from other patches in this series that blockdev played a bit loose by validating that the QMP was okay and then using QDict for everything else rather than the generated struct) and make my decision when posting my RFC patch.

Turns out that using it directly was easier, and less code:
http://thread.gmane.org/gmane.comp.emulators.qemu/353204/focus=354008

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#103 lore raw

RFC PATCH v2 27/47 qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

In reply to Markus Armbruster

Fixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.

Patch's effect on visit_type_UserDefFlatUnion():

static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp) {
Error *err = NULL;

+ visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err); if (err) {
goto out;

Test cases updated for the bug fix.

Fixes alternates to generate a visitor for their implicit enumeration type. None of them are currently used, obviously. Example:
block-core.json's BlockdevRef now generates visit_type_BlockdevRefKind().

---
 scripts/qapi-visit.py                   | 254 ++++++++++++--------------------
 tests/qapi-schema/qapi-schema-test.json |   3 -
 tests/test-qmp-input-strict.c           |   2 +-
 tests/test-qmp-input-visitor.c          |   4 +-
 4 files changed, 100 insertions(+), 163 deletions(-)
scripts/qapi-visit.py +97 -157
index a52a572..135e7c1 100644
@@ -12,7 +12,6 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 import re
 
@@ -24,13 +23,13 @@ def generate_visit_implicit_struct(type):
         return ''
     implicit_structs_seen.add(type)
     ret = ''
-    if type not in struct_fields_seen:
+    if type.name not in struct_fields_seen:
         # Need a forward declaration
         ret += mcgen('''
 
 static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
 ''',
-                     c_type=type_name(type))
+                     c_type=type.c_name())
 
     ret += mcgen('''
 
@@ -46,7 +45,7 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
     error_propagate(errp, err);
 }
 ''',
-                 c_type=type_name(type))
+                 c_type=type.c_name())
     return ret
 
 def generate_visit_struct_fields(name, members, base = None):
@@ -74,24 +73,24 @@ if (err) {
     goto out;
 }
 ''',
-                     type=type_name(base), c_name=c_name('base'))
+                     type=base.c_name(), c_name=c_name('base'))
 
-    for argname, argentry, optional in parse_args(members):
-        if optional:
+    for memb in members:
+        if memb.optional:
             ret += mcgen('''
 visit_optional(m, &(*obj)->has_%(c_name)s, "%(name)s", &err);
 if (!err && (*obj)->has_%(c_name)s) {
 ''',
-                         c_name=c_name(argname), name=argname)
+                         c_name=c_name(memb.name), name=memb.name)
             push_indent()
 
         ret += mcgen('''
 visit_type_%(type)s(m, &(*obj)->%(c_name)s, "%(name)s", &err);
 ''',
-                     type=type_name(argentry), c_name=c_name(argname),
-                     name=argname)
+                     type=memb.type.c_name(), c_name=c_name(memb.name),
+                     name=memb.name)
 
-        if optional:
+        if memb.optional:
             pop_indent()
             ret += mcgen('''
 }
@@ -132,12 +131,7 @@ def generate_visit_struct_body(name):
 
     return ret
 
-def generate_visit_struct(expr):
-
-    name = expr['struct']
-    members = expr['data']
-    base = expr.get('base')
-
+def gen_visit_struct(name, base, members):
     ret = generate_visit_struct_fields(name, members, base)
 
     ret += mcgen('''
@@ -154,10 +148,10 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 ''')
     return ret
 
-def generate_visit_list(name):
+def gen_visit_list(name, element_type):
     return mcgen('''
 
-void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp)
+void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
     GenericList *i, **prev;
@@ -170,8 +164,8 @@ void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, E
     for (prev = (GenericList **)obj;
          !err && (i = visit_next_list(m, prev, &err)) != NULL;
          prev = &i) {
-        %(name)sList *native_i = (%(name)sList *)i;
-        visit_type_%(name)s(m, &native_i->value, NULL, &err);
+        %(name)s *native_i = (%(name)s *)i;
+        visit_type_%(c_elt_type)s(m, &native_i->value, NULL, &err);
     }
 
     error_propagate(errp, err);
@@ -181,7 +175,8 @@ out:
     error_propagate(errp, err);
 }
 ''',
-                name=type_name(name))
+                 name=c_name(name),
+                 c_elt_type=element_type.c_name())
 
 def generate_visit_enum(name):
     return mcgen('''
@@ -193,7 +188,7 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error
 ''',
                  c_name=c_name(name), name=name)
 
-def generate_visit_alternate(name, members):
+def gen_visit_alternate(name, variants):
     ret = mcgen('''
 
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
@@ -212,25 +207,17 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 ''',
                 name=c_name(name))
 
-    # For alternate, always use the default enum type automatically generated
-    # as name + 'Kind'
-    disc_type = c_name(name) + 'Kind'
-
-    for key in members:
-        assert (members[key] in builtin_types.keys()
-            or find_struct(members[key])
-            or find_union(members[key])
-            or find_enum(members[key])), "Invalid alternate member"
-
-        enum_full_value = c_enum_const(disc_type, key)
+    for var in variants.variants:
+        enum_full_value = c_enum_const(variants.tag_member.type.name,
+                                       var.name)
         ret += mcgen('''
     case %(enum_full_value)s:
         visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, name, &err);
         break;
 ''',
                 enum_full_value = enum_full_value,
-                c_type = type_name(members[key]),
-                c_name = c_name(key))
+                c_type=var.type.c_name(),
+                c_name=c_name(var.name))
 
     ret += mcgen('''
     default:
@@ -247,35 +234,16 @@ out:
 
     return ret
 
-
-def generate_visit_union(expr):
-
-    name = expr['union']
-    members = expr['data']
-
-    base = expr.get('base')
-    discriminator = expr.get('discriminator')
-
-    enum_define = discriminator_find_enum_define(expr)
-    if enum_define:
-        # Use the enum type as discriminator
-        ret = ""
-        disc_type = c_name(enum_define['enum_name'])
-    else:
-        # There will always be a discriminator in the C switch code, by default
-        # it is an enum type generated silently
-        ret = generate_visit_enum(name + 'Kind')
-        disc_type = c_name(name) + 'Kind'
+def gen_visit_union(name, base, variants):
+    ret = ''
 
     if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_visit_struct_fields(name, base_fields)
+        members = [m for m in base.members if m != variants.tag_member]
+        ret += generate_visit_struct_fields(name, members)
 
-    if discriminator:
-        for key in members:
-            ret += generate_visit_implicit_struct(members[key])
+    for var in variants.variants:
+        if var.flat:
+            ret += generate_visit_implicit_struct(var.type)
 
     ret += mcgen('''
 
@@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
 ''',
                      name=c_name(name))
 
-    if not discriminator:
-        tag = 'kind'
-        disc_key = "type"
-    else:
-        tag = discriminator
-        disc_key = discriminator
+    disc_key = variants.tag_member.name
+    if not variants.tag_name:
+        # we pointlessly use a different key for simple unions
+        disc_key = 'type'
     ret += mcgen('''
-        visit_type_%(disc_type)s(m, &(*obj)->%(c_tag)s, "%(disc_key)s", &err);
+        visit_type_%(disc_type)s(m, &(*obj)->%(c_name)s, "%(disc_key)s", &err);
         if (err) {
             goto out_obj;
         }
         if (!visit_start_union(m, !!(*obj)->data, &err) || err) {
             goto out_obj;
         }
-        switch ((*obj)->%(c_tag)s) {
+        switch ((*obj)->%(c_name)s) {
 ''',
-                 disc_type = disc_type,
-                 c_tag=c_name(tag),
+                 disc_type=variants.tag_member.type.c_name(),
+                 c_name=c_name(variants.tag_member.name),
                  disc_key = disc_key)
 
-    for key in members:
-        if not discriminator:
+    for var in variants.variants:
+        if not var.flat:
             fmt = 'visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, "data", &err);'
         else:
             fmt = 'visit_type_implicit_%(c_type)s(m, &(*obj)->%(c_name)s, &err);'
 
-        enum_full_value = c_enum_const(disc_type, key)
+        enum_full_value = c_enum_const(variants.tag_member.type.name, var.name)
         ret += mcgen('''
         case %(enum_full_value)s:
             ''' + fmt + '''
             break;
 ''',
                 enum_full_value = enum_full_value,
-                c_type=type_name(members[key]),
-                c_name=c_name(key))
+                c_type=var.type.c_name(),
+                c_name=c_name(var.name))
 
     ret += mcgen('''
         default:
@@ -355,37 +321,57 @@ out:
 
     return ret
 
-def generate_declaration(name, builtin_type=False):
-    ret = ""
-    if not builtin_type:
-        name = c_name(name)
-        ret += mcgen('''
-
-void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp);
-''',
-                     name=name)
-
-    ret += mcgen('''
-void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp);
-''',
-                 name=name)
-
-    return ret
-
-def generate_enum_declaration(name):
-    ret = mcgen('''
-void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp);
-''',
-                name=c_name(name))
-
-    return ret
-
-def generate_decl_enum(name):
+def gen_visit_decl(name, scalar=False):
+    c_type = c_name(name) + ' *'
+    if not scalar:
+        c_type += '*'
     return mcgen('''
-
-void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp);
+void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name), c_type=c_type)
+
+class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.btin = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL')
+    def visit_end(self):
+        # to avoid header dependency hell, we always generate
+        # declarations for built-in types in our header files and
+        # simply guard them
+        self.btin += guardend('QAPI_VISIT_BUILTIN_VISITOR_DECL')
+        self.decl = self.btin + self.decl
+        self.btin = None
+        # ...this doesn't work for cases where we link in multiple
+        # objects that have the functions defined, so we use
+        # do_builtins (option -b) to provide control
+    def visit_enum_type(self, name, info, values):
+        self.decl += gen_visit_decl(name, scalar=True)
+        self.defn += generate_visit_enum(name)
+    def visit_array_type(self, name, info, element_type):
+        decl = gen_visit_decl(name)
+        defn = gen_visit_list(name, element_type)
+        if isinstance(element_type, QAPISchemaBuiltinType):
+            self.btin += decl
+            if do_builtins:
+                self.defn += defn
+        else:
+            self.decl += decl
+            self.defn += defn
+    def visit_object_type(self, name, info, base, members, variants):
+        if info:
+            self.decl += gen_visit_decl(name)
+            if variants:
+                self.defn += gen_visit_union(name, base, variants)
+            else:
+                self.defn += gen_visit_struct(name, base, members)
+    def visit_alternate_type(self, name, info, variants):
+        self.decl += gen_visit_decl(name)
+        self.defn += gen_visit_alternate(name, variants)
 
 do_builtins = False
 
@@ -442,56 +428,10 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = QAPISchema(input_file).get_exprs()
-
-# to avoid header dependency hell, we always generate declarations
-# for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_declaration(typename, builtin_type=True))
-fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
-
-# ...this doesn't work for cases where we link in multiple objects that
-# have the functions defined, so we use -b option to provide control
-# over these cases
-if do_builtins:
-    for typename in builtin_types.keys():
-        fdef.write(generate_visit_list(typename))
-
-for expr in exprs:
-    if expr.has_key('struct'):
-        ret = generate_visit_struct(expr)
-        ret += generate_visit_list(expr['struct'])
-        fdef.write(ret)
-
-        ret = generate_declaration(expr['struct'])
-        fdecl.write(ret)
-    elif expr.has_key('union'):
-        ret = generate_visit_union(expr)
-        ret += generate_visit_list(expr['union'])
-        fdef.write(ret)
-
-        enum_define = discriminator_find_enum_define(expr)
-        ret = ""
-        if not enum_define:
-            ret = generate_decl_enum('%sKind' % expr['union'])
-        ret += generate_declaration(expr['union'])
-        fdecl.write(ret)
-    elif expr.has_key('alternate'):
-        ret = generate_visit_alternate(expr['alternate'], expr['data'])
-        ret += generate_visit_list(expr['alternate'])
-        fdef.write(ret)
-
-        ret = generate_decl_enum('%sKind' % expr['alternate'])
-        ret += generate_declaration(expr['alternate'])
-        fdecl.write(ret)
-    elif expr.has_key('enum'):
-        ret = generate_visit_list(expr['enum'])
-        ret += generate_visit_enum(expr['enum'])
-        fdef.write(ret)
-
-        ret = generate_decl_enum(expr['enum'])
-        ret += generate_enum_declaration(expr['enum'])
-        fdecl.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenVisitVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
tests/qapi-schema/qapi-schema-test.json +0 -3
index 257b4d4..b182174 100644
@@ -39,8 +39,6 @@
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
             'value3' : 'UserDefB' } }
-# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit
-# members of indirect base UserDefZero
 
 { 'struct': 'UserDefUnionBase',
   'base': 'UserDefZero',
@@ -57,7 +55,6 @@
 
 { 'alternate': 'UserDefAlternate',
   'data': { 'uda': 'UserDefA', 's': 'str', 'i': 'int' } }
-# FIXME only a declaration of visit_type_UserDefAlternateKind() generated
 
 { 'struct': 'UserDefC',
   'data': { 'string1': 'str', 'string2': 'str' } }
tests/test-qmp-input-strict.c +1 -1
index 68f855b..00c3e29 100644
@@ -167,9 +167,9 @@ static void test_validate_union_flat(TestInputVisitorData *data,
 
     v = validate_test_init(data,
                            "{ 'enum1': 'value1', "
+                           "'integer': 41, "
                            "'string': 'str', "
                            "'boolean': true }");
-    /* TODO when generator bug is fixed, add 'integer': 41 */
 
     visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
     g_assert(!err);
tests/test-qmp-input-visitor.c +2 -2
index b7a87ee..56da3c6 100644
@@ -307,15 +307,15 @@ static void test_visitor_in_union_flat(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data,
                                 "{ 'enum1': 'value1', "
+                                "'integer': 41, "
                                 "'string': 'str', "
                                 "'boolean': true }");
-    /* TODO when generator bug is fixed, add 'integer': 41 */
 
     visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
     g_assert(err == NULL);
     g_assert_cmpint(tmp->enum1, ==, ENUM_ONE_VALUE1);
     g_assert_cmpstr(tmp->string, ==, "str");
-    /* TODO g_assert_cmpint(tmp->integer, ==, 41); */
+    g_assert_cmpint(tmp->integer, ==, 41);
     g_assert_cmpint(tmp->value1->boolean, ==, true);
     qapi_free_UserDefFlatUnion(tmp);
 }
Signature · 1 line
1.9.3
#104 lore raw

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

Quoted 24 lines · Markus Armbruster

Fixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.

Patch's effect on visit_type_UserDefFlatUnion():

static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp) {
Error *err = NULL;

+ visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err); if (err) {
goto out;

Test cases updated for the bug fix.

Fixes alternates to generate a visitor for their implicit enumeration type. None of them are currently used, obviously. Example:
block-core.json's BlockdevRef now generates visit_type_BlockdevRefKind().

Quoted 7 lines
---
 scripts/qapi-visit.py                   | 254 ++++++++++++--------------------
 tests/qapi-schema/qapi-schema-test.json |   3 -
 tests/test-qmp-input-strict.c           |   2 +-
 tests/test-qmp-input-visitor.c          |   4 +-
 4 files changed, 100 insertions(+), 163 deletions(-)

Another conversion that results in a fairly large diffstat to the generated files:

 qapi-visit.c                        | 4542
++++++++++++++++++------------------
 qapi-visit.h                        |  256 --
 qga/qapi-generated/qga-qapi-visit.c |   88
 qga/qapi-generated/qga-qapi-visit.h |   36
 4 files changed, 2355 insertions(+), 2567 deletions(-)

Same complaints as in 26/47, where splitting some of the cleanups into separate patches would make it easier to validate that the final conversion is correct.

Here, a very common thing in the generated .c file is that you end up swapping the order of visit_type_foo and visit_type_fooList, for any time when foo is an enum. [1]

Quoted 16 lines
scripts/qapi-visit.py +0 -1
index a52a572..135e7c1 100644
@@ -12,7 +12,6 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 import re
 
@@ -24,13 +23,13 @@ def generate_visit_implicit_struct(type):
         return ''
     implicit_structs_seen.add(type)

scripts/qapi-visit.py def generate_visit_implicit_struct(type): · line 32

Quoted 10 lines
scripts/qapi-visit.py +2 -2
     ret = ''
-    if type not in struct_fields_seen:
+    if type.name not in struct_fields_seen:
         # Need a forward declaration
         ret += mcgen('''
 
 static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
 ''',
-                     c_type=type_name(type))
+                     c_type=type.c_name())

This looks a little fishy on first read; why are we calling type.c_name() (and not type.c_type()) when assigning to a placeholder named %(c_type)s? But on second thought, it looks correct: since we really do want the name of the type (and not the magic '*' pointer suffix).

Still, it might be nicer to name things %(c_name)s here; and in that case, it's more of a pre-existing cleanup that might be better floating into one of your earlier patches.

scripts/qapi-visit.py static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * · line 48

Quoted 9 lines
scripts/qapi-visit.py +1 -1
     ret += mcgen('''
 
@@ -46,7 +45,7 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
     error_propagate(errp, err);
 }
 ''',
-                 c_type=type_name(type))
+                 c_type=type.c_name())

same here.

scripts/qapi-visit.py if (err) { · line 76

Quoted 9 lines
scripts/qapi-visit.py +1 -1
     return ret
 
 def generate_visit_struct_fields(name, members, base = None):
@@ -74,24 +73,24 @@ if (err) {
     goto out;
 }
 ''',
-                     type=type_name(base), c_name=c_name('base'))
+                     type=base.c_name(), c_name=c_name('base'))

And this one's pointless: c_name('base') == 'base'. Pointless since commit 622f557 introduced type inheritance. Why do we even need %(c_name)s if we are always passing a constant string?

Oh, and that means our generator has a collision bug that none of my added tests have exposed yet: you cannot have a base class and simultaneously add a member named 'base':

{ 'struct': 'Base', 'data': { 'i': 'int' } }
{ 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }

because the generated C code is trying to use the name 'base' for its own purposes. I guess that means more pre-req patches to the series to expose the bug, and either tighten the parser to reject things for now (easiest) or update the generator to not collide (harder, and fine for a later series).

By the way, now that we are emitting flat unions in such a way that you can cast to the base class, why don't we change our C code to do likewise? That is, where we now have this generated C:

struct BlockdevOptionsGenericFormat {
    BlockdevRef *file;
};
struct BlockdevOptionsGenericCOWFormat {
    BlockdevOptionsGenericFormat *base;
    bool has_backing;
    BlockdevRef *backing;
};

why can't we instead have an unboxed representation:

struct BlockdevOptionsGenericFormat {
    BlockdevRef *file;
};

/* This struct can be cast to BlockdevOptionsGenericFormat */

struct BlockdevOptionsGenericCOWFormat {
    BlockdevRef *file;
    /* end of fields from base class BlockdevOptionsGenericFormat */
    bool has_backing;
    BlockdevRef *backing;
};

where client code that was referring to o->base->file now refers to o->file.

scripts/qapi-visit.py out: · line 242

Quoted 10 lines
scripts/qapi-visit.py +4 -4
+def gen_visit_union(name, base, variants):
+    ret = ''
 
     if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_visit_struct_fields(name, base_fields)
+        members = [m for m in base.members if m != variants.tag_member]
+        ret += generate_visit_struct_fields(name, members)

Why not just visit ALL base class members, unconditionally?

scripts/qapi-visit.py out: · line 246

Quoted 7 lines
scripts/qapi-visit.py +3 -3
-    if discriminator:
-        for key in members:
-            ret += generate_visit_implicit_struct(members[key])
+    for var in variants.variants:
+        if var.flat:
+            ret += generate_visit_implicit_struct(var.type)

Okay, I see where you are using .flat from the initial parse. I still think it is a bit odd that you are defining '.flat' for each 'variant' within 'variants', even though, for a given 'variants', all members will have the same setting of '.flat'. That makes me wonder if '.flat' should belong instead to the top-level 'variants' struct rather than to each 'variant' member.

But again I wonder what would happen if you had instead normalized the input of simple unions into always having an implicit struct (with single member 'data'), so that by the time you get here, you only have to deal with a single representation of unions instead of having to still emit different things for flat vs. simple (since on the wire, we already proved simple is shorthand that can be duplicated by a flat union).

Quoted 6 lines
scripts/qapi-visit.py +0 -0
     ret += mcgen('''
 
@@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
 ''',
                      name=c_name(name))

scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error · line 273

Quoted 10 lines
scripts/qapi-visit.py +3 -6
 
-    if not discriminator:
-        tag = 'kind'
-        disc_key = "type"
-    else:
-        tag = discriminator
-        disc_key = discriminator
+    disc_key = variants.tag_member.name
+    if not variants.tag_name:
+        # we pointlessly use a different key for simple unions

We could fix that (as a separate patch); wonder how much C code it would affect. A lot of these things that we can alter in generated code are certainly easier to see now that we have a clean generator :)

scripts/qapi-visit.py out: · line 331

Quoted 11 lines
scripts/qapi-visit.py +6 -3
+def gen_visit_decl(name, scalar=False):
+    c_type = c_name(name) + ' *'
+    if not scalar:
+        c_type += '*'
     return mcgen('''
-
-void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp);
+void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name), c_type=c_type)

Nice way to consolidate several near-identical copies.

Quoted 10 lines
scripts/qapi-visit.py +10 -0
+
+class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.btin = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL')

scripts/qapi-visit.py out: · line 351

Quoted 10 lines
scripts/qapi-visit.py +10 -0
+    def visit_end(self):
+        # to avoid header dependency hell, we always generate
+        # declarations for built-in types in our header files and
+        # simply guard them
+        self.btin += guardend('QAPI_VISIT_BUILTIN_VISITOR_DECL')
+        self.decl = self.btin + self.decl
+        self.btin = None
+        # ...this doesn't work for cases where we link in multiple
+        # objects that have the functions defined, so we use
+        # do_builtins (option -b) to provide control

And once again, as in 26/47, this floats the .h file to have all builtin representations in one chunk (for continuity with pre-patch), but fails to do the same for the .c code...

scripts/qapi-visit.py out: · line 361

Quoted 10 lines
scripts/qapi-visit.py +10 -0
+    def visit_enum_type(self, name, info, values):
+        self.decl += gen_visit_decl(name, scalar=True)
+        self.defn += generate_visit_enum(name)
+    def visit_array_type(self, name, info, element_type):
+        decl = gen_visit_decl(name)
+        defn = gen_visit_list(name, element_type)
+        if isinstance(element_type, QAPISchemaBuiltinType):
+            self.btin += decl
+            if do_builtins:
+                self.defn += defn

...where the builtins are now interleaved with everything else instead of bunched together, making the generated diff larger and more confusing than necessary.

scripts/qapi-visit.py out: · line 369

Quoted 8 lines
scripts/qapi-visit.py +8 -0
+        else:
+            self.decl += decl
+            self.defn += defn
+    def visit_object_type(self, name, info, base, members, variants):
+        if info:
+            self.decl += gen_visit_decl(name)
+            if variants:
+                self.defn += gen_visit_union(name, base, variants)

Worth adding 'assert not members'?

scripts/qapi-visit.py

Quoted 2 lines
scripts/qapi-visit.py +2 -0
+            else:
+                self.defn += gen_visit_struct(name, base, members)

Or maybe we can someday consolidate these two into a single gen_visit_object, that handles all members and variants in a uniform manner, instead of our current differences. I wonder how much C code would be impacted?

Quoted 15 lines
scripts/qapi-visit.py +3 -5
+    def visit_alternate_type(self, name, info, variants):
+        self.decl += gen_visit_decl(name)
+        self.defn += gen_visit_alternate(name, variants)
 
 do_builtins = False
@@ -442,56 +428,10 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = QAPISchema(input_file).get_exprs()
-
-# to avoid header dependency hell, we always generate declarations
-# for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_VISIT_BUILTIN_VISITOR_DECL"))

scripts/qapi-visit.py fdecl.write(mcgen(''' · line 459

Quoted 10 lines
scripts/qapi-visit.py +0 -10
-for typename in builtin_types.keys():
-    fdecl.write(generate_declaration(typename, builtin_type=True))
-fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
-
-# ...this doesn't work for cases where we link in multiple objects that
-# have the functions defined, so we use -b option to provide control
-# over these cases
-if do_builtins:
-    for typename in builtin_types.keys():
-        fdef.write(generate_visit_list(typename))

Again, a well-placed sorted() over these two loops in a pre-req patch will minimize the churn on the builtins.

Quoted 8 lines
scripts/qapi-visit.py +0 -8
-
-for expr in exprs:
-    if expr.has_key('struct'):
-        ret = generate_visit_struct(expr)
-        ret += generate_visit_list(expr['struct'])
-        fdef.write(ret)
-
-        ret = generate_declaration(expr['struct'])

scripts/qapi-visit.py fdecl.write(mcgen(''' · line 477

Quoted 10 lines
scripts/qapi-visit.py +0 -10
-        fdecl.write(ret)
-    elif expr.has_key('union'):
-        ret = generate_visit_union(expr)
-        ret += generate_visit_list(expr['union'])
-        fdef.write(ret)
-
-        enum_define = discriminator_find_enum_define(expr)
-        ret = ""
-        if not enum_define:
-            ret = generate_decl_enum('%sKind' % expr['union'])

Nice that the new visitor automatically visits any implicit enum, without us having to special case it.

scripts/qapi-visit.py fdecl.write(mcgen(''' · line 490

Quoted 13 lines
scripts/qapi-visit.py +0 -13
-        ret += generate_declaration(expr['union'])
-        fdecl.write(ret)
-    elif expr.has_key('alternate'):
-        ret = generate_visit_alternate(expr['alternate'], expr['data'])
-        ret += generate_visit_list(expr['alternate'])
-        fdef.write(ret)
-
-        ret = generate_decl_enum('%sKind' % expr['alternate'])
-        ret += generate_declaration(expr['alternate'])
-        fdecl.write(ret)
-    elif expr.has_key('enum'):
-        ret = generate_visit_list(expr['enum'])
-        ret += generate_visit_enum(expr['enum'])

[1] swapping these two lines in a pre-req patch will minimize the churn of this conversion.

scripts/qapi-visit.py fdecl.write(mcgen(''' · line 435

Quoted 10 lines
scripts/qapi-visit.py +5 -5
-        fdef.write(ret)
-
-        ret = generate_decl_enum(expr['enum'])
-        ret += generate_enum_declaration(expr['enum'])
-        fdecl.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenVisitVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)

Again, overall impression is that your series is headed in the right direction.

And nice that the TODOs in the testsuite pointed out what this fixes, for visiting indirect bases.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#105 lore raw

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

Quoted 42 lines · Eric Blake
Quoted 24 lines · Markus Armbruster

Fixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.

Patch's effect on visit_type_UserDefFlatUnion():

static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp) {
Error *err = NULL;

+ visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err); if (err) {
goto out;

Test cases updated for the bug fix.

Fixes alternates to generate a visitor for their implicit enumeration type. None of them are currently used, obviously. Example:
block-core.json's BlockdevRef now generates visit_type_BlockdevRefKind().

Quoted 7 lines
---
 scripts/qapi-visit.py                   | 254 ++++++++++++--------------------
 tests/qapi-schema/qapi-schema-test.json |   3 -
 tests/test-qmp-input-strict.c           |   2 +-
 tests/test-qmp-input-visitor.c          |   4 +-
 4 files changed, 100 insertions(+), 163 deletions(-)

Another conversion that results in a fairly large diffstat to the generated files:

 qapi-visit.c                        | 4542 ++++++++++++++++++------------------
 qapi-visit.h                        |  256 --
 qga/qapi-generated/qga-qapi-visit.c |   88
 qga/qapi-generated/qga-qapi-visit.h |   36
 4 files changed, 2355 insertions(+), 2567 deletions(-)
Quoted 7 lines

Same complaints as in 26/47, where splitting some of the cleanups into separate patches would make it easier to validate that the final conversion is correct.

Here, a very common thing in the generated .c file is that you end up swapping the order of visit_type_foo and visit_type_fooList, for any time when foo is an enum. [1]

Again, this is because the old code generates the array visitor from within the code generating the element type visitor, while the new code makes all array types full-fledged types, which get visited like any other type.

Quoted 22 lines
Quoted 22 lines
scripts/qapi-visit.py +1 -2
index a52a572..135e7c1 100644
@@ -12,7 +12,6 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 import re
 
@@ -24,13 +23,13 @@ def generate_visit_implicit_struct(type):
         return ''
     implicit_structs_seen.add(type)
     ret = ''
-    if type not in struct_fields_seen:
+    if type.name not in struct_fields_seen:
         # Need a forward declaration
         ret += mcgen('''
 
Quoted 9 lines

scripts/qapi-visit.py def generate_visit_implicit_struct(type): · line 32

Quoted 4 lines
scripts/qapi-visit.py +1 -1
 static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
 ''',
-                     c_type=type_name(type))
+                     c_type=type.c_name())

This looks a little fishy on first read; why are we calling type.c_name() (and not type.c_type()) when assigning to a placeholder named %(c_type)s? But on second thought, it looks correct: since we really do want the name of the type (and not the magic '*' pointer suffix).

Most of the time, c_type is an identifier naming a type (something like typ.c_name()), but sometimes it's a type expression (something like typ.c_type()). If that turns out to be too confusing, we can use different names for the two. Not exactly looking forward to tracking them down, though...

Quoted 3 lines

Still, it might be nicer to name things %(c_name)s here; and in that case, it's more of a pre-existing cleanup that might be better floating into one of your earlier patches.

This patch tries very hard not to touch lines without need. Cleanup is left for PATCH 33.

I could do some (but not all) cleanup before converting to QAPISchemaVisitor. But I'm afraid that just complicates things even more.

Quoted 12 lines

scripts/qapi-visit.py static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error * · line 48

Quoted 9 lines
scripts/qapi-visit.py +1 -1
     ret += mcgen('''
 
@@ -46,7 +45,7 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
     error_propagate(errp, err);
 }
 ''',
-                 c_type=type_name(type))
+                 c_type=type.c_name())

same here.

Quoted 4 lines

scripts/qapi-visit.py if (err) {

Quoted 4 lines
scripts/qapi-visit.py +0 -0
     return ret
 
 def generate_visit_struct_fields(name, members, base = None):
@@ -74,24 +73,24 @@ if (err) {
            ret += mcgen('''
    visit_type_implicit_%(c_type)s(m, &(*obj)->%(c_name)s, &err);
    if (err) {
Quoted 9 lines

scripts/qapi-visit.py if (err) { · line 76

Quoted 5 lines
scripts/qapi-visit.py +1 -1
     goto out;
 }
 ''',
-                     type=type_name(base), c_name=c_name('base'))
+                     type=base.c_name(), c_name=c_name('base'))

And this one's pointless: c_name('base') == 'base'. Pointless since commit 622f557 introduced type inheritance. Why do we even need %(c_name)s if we are always passing a constant string?

My best guess: to keep the code similar to other code generating visit_type_FOO() calls?

Quoted 9 lines

Oh, and that means our generator has a collision bug that none of my added tests have exposed yet: you cannot have a base class and simultaneously add a member named 'base':

{ 'struct': 'Base', 'data': { 'i': 'int' } }
{ 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }

because the generated C code is trying to use the name 'base' for its own purposes.

*sigh*

Quoted 4 lines

I guess that means more pre-req patches to the series to
expose the bug, and either tighten the parser to reject things for now (easiest) or update the generator to not collide (harder, and fine for a later series).

Yes.

Life would be easier if the original authors had adopted sane naming conventions from the start.

Quoted 21 lines

By the way, now that we are emitting flat unions in such a way that you can cast to the base class, why don't we change our C code to do likewise? That is, where we now have this generated C:

struct BlockdevOptionsGenericFormat {
    BlockdevRef *file;
};
struct BlockdevOptionsGenericCOWFormat {
    BlockdevOptionsGenericFormat *base;
    bool has_backing;
    BlockdevRef *backing;
};

why can't we instead have an unboxed representation:

struct BlockdevOptionsGenericFormat {
    BlockdevRef *file;
};

/* This struct can be cast to BlockdevOptionsGenericFormat */

Quoted 8 lines
struct BlockdevOptionsGenericCOWFormat {
    BlockdevRef *file;
    /* end of fields from base class BlockdevOptionsGenericFormat */
    bool has_backing;
    BlockdevRef *backing;
};

where client code that was referring to o->base->file now refers to o->file.

That's where I want to go, but not in this series.

Quoted 11 lines

scripts/qapi-visit.py out: · line 242

Quoted 10 lines
scripts/qapi-visit.py +4 -4
+def gen_visit_union(name, base, variants):
+    ret = ''
 
     if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_visit_struct_fields(name, base_fields)
+        members = [m for m in base.members if m != variants.tag_member]
+        ret += generate_visit_struct_fields(name, members)
Quoted 1 line

Why not just visit ALL base class members, unconditionally?

This patch converts to QAPISchemaVisitor. It tries hard to keep the generated code the same.

The old code generates the discriminator visit into the visit_type_FOO() and not the the visit_type_FOO_fields() helper (can't tell why), so the new code does the same.

Simply visiting all members makes more sense, but it needs to be done in a followup patch. Since this series is already huge, it'll likely be in a followup series.

Quoted 7 lines

scripts/qapi-visit.py out: · line 246

Quoted 6 lines
scripts/qapi-visit.py +3 -3
-    if discriminator:
-        for key in members:
-            ret += generate_visit_implicit_struct(members[key])
+    for var in variants.variants:
+        if var.flat:
+            ret += generate_visit_implicit_struct(var.type)
Quoted 6 lines

Okay, I see where you are using .flat from the initial parse. I still think it is a bit odd that you are defining '.flat' for each 'variant' within 'variants', even though, for a given 'variants', all members will have the same setting of '.flat'. That makes me wonder if '.flat' should belong instead to the top-level 'variants' struct rather than to each 'variant' member.

Two reasons for putting flat where it is:

* The philosophical one: from the generator's point of view, there's no fundamental reason why all variants need to be flat or none. The generator really doesn't care.

* The pragmatic one (a.k.a. the real one): there are places where I use v.flat, but don't have the variants owning v handy.

Quoted 6 lines

But again I wonder what would happen if you had instead normalized the input of simple unions into always having an implicit struct (with single member 'data'), so that by the time you get here, you only have to deal with a single representation of unions instead of having to still emit different things for flat vs. simple (since on the wire, we already proved simple is shorthand that can be duplicated by a flat union).

I hope we can get there! But at this stage of the conversion, I want to minimize output change, and .flat makes preserving all its warts much easier.

Quoted 7 lines
Quoted 7 lines
scripts/qapi-visit.py +0 -0
     ret += mcgen('''
 
@@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
 ''',
                      name=c_name(name))
 
Quoted 13 lines

scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error · line 273

Quoted 9 lines
scripts/qapi-visit.py +3 -6
-    if not discriminator:
-        tag = 'kind'
-        disc_key = "type"
-    else:
-        tag = discriminator
-        disc_key = discriminator
+    disc_key = variants.tag_member.name
+    if not variants.tag_name:
+        # we pointlessly use a different key for simple unions

We could fix that (as a separate patch); wonder how much C code it would affect. A lot of these things that we can alter in generated code are certainly easier to see now that we have a clean generator :)

Yup, the warts stand out now.

Quoted 30 lines

scripts/qapi-visit.py out: · line 331

Quoted 11 lines
scripts/qapi-visit.py +6 -3
+def gen_visit_decl(name, scalar=False):
+    c_type = c_name(name) + ' *'
+    if not scalar:
+        c_type += '*'
     return mcgen('''
-
-void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp);
+void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name), c_type=c_type)

Nice way to consolidate several near-identical copies.

Quoted 16 lines
scripts/qapi-visit.py +16 -0
+
+class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.btin = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.btin = guardstart('QAPI_VISIT_BUILTIN_VISITOR_DECL')
+    def visit_end(self):
+        # to avoid header dependency hell, we always generate
+        # declarations for built-in types in our header files and
+        # simply guard them
+        self.btin += guardend('QAPI_VISIT_BUILTIN_VISITOR_DECL')
+        self.decl = self.btin + self.decl
Quoted 8 lines

scripts/qapi-visit.py out: · line 351

Quoted 4 lines
scripts/qapi-visit.py +4 -0
+        self.btin = None
+        # ...this doesn't work for cases where we link in multiple
+        # objects that have the functions defined, so we use
+        # do_builtins (option -b) to provide control

And once again, as in 26/47, this floats the .h file to have all builtin representations in one chunk (for continuity with pre-patch), but fails to do the same for the .c code...

Yes, because they all need to be under the
QAPI_VISIT_BUILTIN_VISITOR_DECL guard.

Quoted 11 lines

scripts/qapi-visit.py out: · line 361

Quoted 10 lines
scripts/qapi-visit.py +10 -0
+    def visit_enum_type(self, name, info, values):
+        self.decl += gen_visit_decl(name, scalar=True)
+        self.defn += generate_visit_enum(name)
+    def visit_array_type(self, name, info, element_type):
+        decl = gen_visit_decl(name)
+        defn = gen_visit_list(name, element_type)
+        if isinstance(element_type, QAPISchemaBuiltinType):
+            self.btin += decl
+            if do_builtins:
+                self.defn += defn
Quoted 3 lines

...where the builtins are now interleaved with everything else instead of bunched together, making the generated diff larger and more confusing than necessary.

Yes, because there's no need to separate them out.

Quoted 10 lines

scripts/qapi-visit.py out: · line 369

Quoted 8 lines
scripts/qapi-visit.py +8 -0
+        else:
+            self.decl += decl
+            self.defn += defn
+    def visit_object_type(self, name, info, base, members, variants):
+        if info:
+            self.decl += gen_visit_decl(name)
+            if variants:
+                self.defn += gen_visit_union(name, base, variants)

Worth adding 'assert not members'?

Yes, with a TODO, because it's an artificial restriction I'd like to lift some day.

Quoted 7 lines

scripts/qapi-visit.py

Quoted 2 lines
scripts/qapi-visit.py +2 -0
+            else:
+                self.defn += gen_visit_struct(name, base, members)

Or maybe we can someday consolidate these two into a single gen_visit_object, that handles all members and variants in a uniform manner, instead of our current differences. I wonder how much C code would be impacted?

Yes, that's how we'd lift the restriction.

Quoted 20 lines
Quoted 20 lines
scripts/qapi-visit.py +3 -10
+    def visit_alternate_type(self, name, info, variants):
+        self.decl += gen_visit_decl(name)
+        self.defn += gen_visit_alternate(name, variants)
 
 do_builtins = False
@@ -442,56 +428,10 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = QAPISchema(input_file).get_exprs()
-
-# to avoid header dependency hell, we always generate declarations
-# for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_declaration(typename, builtin_type=True))
-fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
-
-# ...this doesn't work for cases where we link in multiple objects that
Quoted 8 lines

scripts/qapi-visit.py fdecl.write(mcgen(''' · line 459

Quoted 5 lines
scripts/qapi-visit.py +0 -5
-# have the functions defined, so we use -b option to provide control
-# over these cases
-if do_builtins:
-    for typename in builtin_types.keys():
-        fdef.write(generate_visit_list(typename))

Again, a well-placed sorted() over these two loops in a pre-req patch will minimize the churn on the builtins.

Can try and see.

Quoted 13 lines
Quoted 13 lines
scripts/qapi-visit.py +0 -13
-
-for expr in exprs:
-    if expr.has_key('struct'):
-        ret = generate_visit_struct(expr)
-        ret += generate_visit_list(expr['struct'])
-        fdef.write(ret)
-
-        ret = generate_declaration(expr['struct'])
-        fdecl.write(ret)
-    elif expr.has_key('union'):
-        ret = generate_visit_union(expr)
-        ret += generate_visit_list(expr['union'])
-        fdef.write(ret)
Quoted 8 lines

scripts/qapi-visit.py fdecl.write(mcgen(''' · line 477

Quoted 5 lines
scripts/qapi-visit.py +0 -5
-
-        enum_define = discriminator_find_enum_define(expr)
-        ret = ""
-        if not enum_define:
-            ret = generate_decl_enum('%sKind' % expr['union'])

Nice that the new visitor automatically visits any implicit enum, without us having to special case it.

Making implicit stuff explicit in the internal representation kills special cases and de-duplicates code :)

Quoted 14 lines

scripts/qapi-visit.py fdecl.write(mcgen(''' · line 490

Quoted 13 lines
scripts/qapi-visit.py +0 -13
-        ret += generate_declaration(expr['union'])
-        fdecl.write(ret)
-    elif expr.has_key('alternate'):
-        ret = generate_visit_alternate(expr['alternate'], expr['data'])
-        ret += generate_visit_list(expr['alternate'])
-        fdef.write(ret)
-
-        ret = generate_decl_enum('%sKind' % expr['alternate'])
-        ret += generate_declaration(expr['alternate'])
-        fdecl.write(ret)
-    elif expr.has_key('enum'):
-        ret = generate_visit_list(expr['enum'])
-        ret += generate_visit_enum(expr['enum'])
Quoted 2 lines

[1] swapping these two lines in a pre-req patch will minimize the churn of this conversion.

Can try and see.

Quoted 14 lines

scripts/qapi-visit.py fdecl.write(mcgen(''' · line 435

Quoted 10 lines
scripts/qapi-visit.py +5 -5
-        fdef.write(ret)
-
-        ret = generate_decl_enum(expr['enum'])
-        ret += generate_enum_declaration(expr['enum'])
-        fdecl.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenVisitVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)

Again, overall impression is that your series is headed in the right direction.

Quoted 2 lines

And nice that the TODOs in the testsuite pointed out what this fixes, for visiting indirect bases.

Thanks!

#106 lore raw

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

Quoted 17 lines · Markus Armbruster
Quoted 9 lines

Oh, and that means our generator has a collision bug that none of my added tests have exposed yet: you cannot have a base class and simultaneously add a member named 'base':

{ 'struct': 'Base', 'data': { 'i': 'int' } }
{ 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }

because the generated C code is trying to use the name 'base' for its own purposes.

*sigh*

Quoted 4 lines

I guess that means more pre-req patches to the series to
expose the bug, and either tighten the parser to reject things for now (easiest) or update the generator to not collide (harder, and fine for a later series).

Quoted 4 lines

Yes.

Life would be easier if the original authors had adopted sane naming conventions from the start.

Like reserving ourselves a namespace based on single _ for internal use.
We practically already have that - all user names either start with a
letter or double underscore, so we could use single (and triple) underscore for internally-generated purposes, freeing up 'base', '*Kind', '*_MAX', and other namespace abuses back to the user. Well, we may also need to reserve mid-name double-underscore (that is, the user can only supply double underscore at the beginning, but not middle, of an identifier). Ah well, food for thought for later patches.

Quoted 7 lines
Quoted 6 lines

Okay, I see where you are using .flat from the initial parse. I still think it is a bit odd that you are defining '.flat' for each 'variant' within 'variants', even though, for a given 'variants', all members will have the same setting of '.flat'. That makes me wonder if '.flat' should belong instead to the top-level 'variants' struct rather than to each 'variant' member.

Quoted 5 lines

Two reasons for putting flat where it is:

* The philosophical one: from the generator's point of view, there's no fundamental reason why all variants need to be flat or none. The generator really doesn't care.

And we may decide to exploit that down the road to allow some sort of qapi syntax for explicitly designating a union branch as flat or boxed, rather than the current approach of the type of union determining the status of all branch members.

Quoted 11 lines

* The pragmatic one (a.k.a. the real one): there are places where I use v.flat, but don't have the variants owning v handy.

Quoted 6 lines

But again I wonder what would happen if you had instead normalized the input of simple unions into always having an implicit struct (with single member 'data'), so that by the time you get here, you only have to deal with a single representation of unions instead of having to still emit different things for flat vs. simple (since on the wire, we already proved simple is shorthand that can be duplicated by a flat union).

Quoted 3 lines

I hope we can get there! But at this stage of the conversion, I want to minimize output change, and .flat makes preserving all its warts much easier.

Agreed. By the end of the series, I was convinced that the use of .flat, at least in this series, makes sense.

Quoted 9 lines
Quoted 7 lines

scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error · line 273

Quoted 3 lines
scripts/qapi-visit.py +3 -0
+    disc_key = variants.tag_member.name
+    if not variants.tag_name:
+        # we pointlessly use a different key for simple unions

We could fix that (as a separate patch); wonder how much C code it would affect. A lot of these things that we can alter in generated code are certainly easier to see now that we have a clean generator :)

Yup, the warts stand out now.

And I've already demonstrated what sort of cleanups can be done to attack some of the warts:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05266.html

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#107 lore raw

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

Quoted 24 lines · Eric Blake
Quoted 17 lines · Markus Armbruster
Quoted 9 lines

Oh, and that means our generator has a collision bug that none of my added tests have exposed yet: you cannot have a base class and simultaneously add a member named 'base':

{ 'struct': 'Base', 'data': { 'i': 'int' } }
{ 'struct': 'Sub', 'base': 'Base', 'data': { 'base': 'str' } }

because the generated C code is trying to use the name 'base' for its own purposes.

*sigh*

Quoted 4 lines

I guess that means more pre-req patches to the series to
expose the bug, and either tighten the parser to reject things for now (easiest) or update the generator to not collide (harder, and fine for a later series).

Quoted 4 lines

Yes.

Life would be easier if the original authors had adopted sane naming conventions from the start.

Quoted 8 lines

Like reserving ourselves a namespace based on single _ for internal use.
We practically already have that - all user names either start with a
letter or double underscore, so we could use single (and triple) underscore for internally-generated purposes, freeing up 'base', '*Kind', '*_MAX', and other namespace abuses back to the user. Well, we may also need to reserve mid-name double-underscore (that is, the user can only supply double underscore at the beginning, but not middle, of an identifier). Ah well, food for thought for later patches.

Another concern: we should take care not to generate reserved identifiers.

* Potential issue with your proposal: identifiers that begin with an underscore and either an uppercase letter or another underscore are always reserved for any use.

* Existing issue: downstream extensions carry a __RFQDN_ prefix in the schema, which map to reserved C identifiers.

Example: qapi-schema-test.json type '__org.qemu_x-Enum' generates

    typedef enum __org_qemu_x_Enum {
        ORG_QEMU_X_ENUM___ORG_QEMU_X_VALUE = 0,
        ORG_QEMU_X_ENUM_MAX = 1,
    } __org_qemu_x_Enum;

    extern const char *const __org_qemu_x_Enum_lookup[];
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

Okay, I see where you are using .flat from the initial parse. I still think it is a bit odd that you are defining '.flat' for each 'variant' within 'variants', even though, for a given 'variants', all members will have the same setting of '.flat'. That makes me wonder if '.flat' should belong instead to the top-level 'variants' struct rather than to each 'variant' member.

Two reasons for putting flat where it is:

Quoted 8 lines
Quoted 3 lines

* The philosophical one: from the generator's point of view, there's no fundamental reason why all variants need to be flat or none. The generator really doesn't care.

And we may decide to exploit that down the road to allow some sort of qapi syntax for explicitly designating a union branch as flat or boxed, rather than the current approach of the type of union determining the status of all branch members.

I can't see a need now, but if one arises, we could do it.

Quoted 10 lines
Quoted 10 lines

* The pragmatic one (a.k.a. the real one): there are places where I use v.flat, but don't have the variants owning v handy.

Quoted 6 lines

But again I wonder what would happen if you had instead normalized the input of simple unions into always having an implicit struct (with single member 'data'), so that by the time you get here, you only have to deal with a single representation of unions instead of having to still emit different things for flat vs. simple (since on the wire, we already proved simple is shorthand that can be duplicated by a flat union).

Quoted 6 lines
Quoted 3 lines

I hope we can get there! But at this stage of the conversion, I want to minimize output change, and .flat makes preserving all its warts much easier.

Agreed. By the end of the series, I was convinced that the use of .flat, at least in this series, makes sense.

Good :)

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error · line 273

Quoted 3 lines
scripts/qapi-visit.py +3 -0
+    disc_key = variants.tag_member.name
+    if not variants.tag_name:
+        # we pointlessly use a different key for simple unions

We could fix that (as a separate patch); wonder how much C code it would affect. A lot of these things that we can alter in generated code are certainly easier to see now that we have a clean generator :)

Quoted 5 lines
Quoted 1 line

Yup, the warts stand out now.

And I've already demonstrated what sort of cleanups can be done to attack some of the warts:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05266.html

I only have time for a quick glance now. It looks lovely!

#108 lore raw

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

Quoted 11 lines · Markus Armbruster
Quoted 8 lines

Like reserving ourselves a namespace based on single _ for internal use.
We practically already have that - all user names either start with a
letter or double underscore, so we could use single (and triple) underscore for internally-generated purposes, freeing up 'base', '*Kind', '*_MAX', and other namespace abuses back to the user. Well, we may also need to reserve mid-name double-underscore (that is, the user can only supply double underscore at the beginning, but not middle, of an identifier). Ah well, food for thought for later patches.

Another concern: we should take care not to generate reserved identifiers.

And we do that for a number of identifiers already, by renaming 'default' in qapi to 'q_default' in C.

Quoted 4 lines

* Potential issue with your proposal: identifiers that begin with an underscore and either an uppercase letter or another underscore are always reserved for any use.

True, so only underscore and lower is guaranteed safe. But in practice, we can often get away with more...

Quoted 3 lines

* Existing issue: downstream extensions carry a __RFQDN_ prefix in the schema, which map to reserved C identifiers.

...the whole point of the __RFQDN_ prefix was that it is very unlikely to conflict with any vendor extensions (unless you happen to be the same vendor that writes the compiler and als use __RFQDN_ as part of your compiler implementation). Yes, it's risky, but a risk that hasn't hurt any downstream clients yet :) It's one of those things where if someone reports an actual problem in their environment, we'll fix it; but until then, it's not worth the headache of strict compliance to "fix" something that happens to work in spite of being undefined behavior.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#109 lore raw

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

Quoted 17 lines · Eric Blake
Quoted 11 lines · Markus Armbruster
Quoted 8 lines

Like reserving ourselves a namespace based on single _ for internal use.
We practically already have that - all user names either start with a
letter or double underscore, so we could use single (and triple) underscore for internally-generated purposes, freeing up 'base', '*Kind', '*_MAX', and other namespace abuses back to the user. Well, we may also need to reserve mid-name double-underscore (that is, the user can only supply double underscore at the beginning, but not middle, of an identifier). Ah well, food for thought for later patches.

Another concern: we should take care not to generate reserved identifiers.

And we do that for a number of identifiers already, by renaming 'default' in qapi to 'q_default' in C.

Quoted 1 line
Quoted 5 lines
Quoted 3 lines

* Potential issue with your proposal: identifiers that begin with an underscore and either an uppercase letter or another underscore are always reserved for any use.

True, so only underscore and lower is guaranteed safe.

And we'd get that for most names. Only names we want to shout (macros, enumeration constants) could result in a problematic underscore and uppercase letter.

Note that we *strip* leading underscores from enumeration constants. Example (from qapi-schema-test.json):

    { 'enum': '__org.qemu_x-Enum', 'data': [ '__org.qemu_x-value' ] }

generates

    typedef enum __org_qemu_x_Enum
    {
        ORG_QEMU_X_ENUM___ORG_QEMU_X_VALUE = 0,
        ORG_QEMU_X_ENUM_MAX = 1,
    } __org_qemu_x_Enum;

Now add this one:

    { 'enum': 'org_qemu_x-Enum', 'data': [ ] }

Just fine according to our documentation. But of course it clashes:

    typedef enum org_qemu_x_Enum
    {
        ORG_QEMU_X_ENUM_MAX = 0,
    } org_qemu_x_Enum;

Short term, we should note in qapi-code-gen.txt: different names in the schema can nevertheless clash in generated C, and when that happens, you get to pick non-clashing names.

Longer term, we may want to rethink how we map names to C.

Quoted 6 lines

we can often get away with more...

Quoted 3 lines

* Existing issue: downstream extensions carry a __RFQDN_ prefix in the schema, which map to reserved C identifiers.

Quoted 8 lines

...the whole point of the __RFQDN_ prefix was that it is very unlikely to conflict with any vendor extensions (unless you happen to be the same vendor that writes the compiler and als use __RFQDN_ as part of your compiler implementation). Yes, it's risky, but a risk that hasn't hurt any downstream clients yet :) It's one of those things where if someone reports an actual problem in their environment, we'll fix it; but until then, it's not worth the headache of strict compliance to "fix" something that happens to work in spite of being undefined behavior.

I'm not religious about reserved names, but this intrusion into the reserved name space was entirely avoidable: we could've just as well picked a non-reserved prefix.

#110 lore raw

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

In reply to Markus Armbruster
Quoted 18 lines · Markus Armbruster

Fixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.

Patch's effect on visit_type_UserDefFlatUnion():

static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp) {
Error *err = NULL;

+ visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err); if (err) {
goto out;

scripts/qapi-visit.py out: · line 242

Quoted 10 lines
scripts/qapi-visit.py +4 -4
+def gen_visit_union(name, base, variants):
+    ret = ''
 
     if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_visit_struct_fields(name, base_fields)
+        members = [m for m in base.members if m != variants.tag_member]
+        ret += generate_visit_struct_fields(name, members)

Ouch. This hurts. If the same class is used as both the base class of a flat union, and the base class of an ordinary struct, then the struct tries to visit the base class, but no longer parses the field that the union was using as its discriminator.

We don't have any code that demonstrates this, but probably should. I ran into it while working up my POC of what it would take to unbox inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)

We want visit_FOO_fields to visit _all_ fields of the struct, no matter who called generate_visit_struct_fields(). So what you must instead do here is use the fact that we've already visited the discriminator...

Quoted 17 lines
scripts/qapi-visit.py +3 -6
-    if discriminator:
-        for key in members:
-            ret += generate_visit_implicit_struct(members[key])
+    for var in variants.variants:
+        if var.flat:
+            ret += generate_visit_implicit_struct(var.type)
 
     ret += mcgen('''
@@ -300,41 +268,39 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
 ''',
                      name=c_name(name))
 
-    if not discriminator:
-        tag = 'kind'
-        disc_key = "type"

scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error · line 276

Quoted 10 lines
scripts/qapi-visit.py +5 -4
-    else:
-        tag = discriminator
-        disc_key = discriminator
+    disc_key = variants.tag_member.name
+    if not variants.tag_name:
+        # we pointlessly use a different key for simple unions
+        disc_key = 'type'
     ret += mcgen('''
-        visit_type_%(disc_type)s(m, &(*obj)->%(c_tag)s, "%(disc_key)s", &err);
+        visit_type_%(disc_type)s(m, &(*obj)->%(c_name)s, "%(disc_key)s", &err);

...and omit this call if the flat union's base class already took care of it.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#111 lore raw

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

Quoted 30 lines · Eric Blake
Quoted 18 lines · Markus Armbruster

Fixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.

Patch's effect on visit_type_UserDefFlatUnion():

static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp) {
Error *err = NULL;

+ visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err); if (err) {
goto out;

scripts/qapi-visit.py out: · line 242

Quoted 10 lines
scripts/qapi-visit.py +4 -4
+def gen_visit_union(name, base, variants):
+    ret = ''
 
     if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_visit_struct_fields(name, base_fields)
+        members = [m for m in base.members if m != variants.tag_member]
+        ret += generate_visit_struct_fields(name, members)
Quoted 8 lines

Ouch. This hurts. If the same class is used as both the base class of a flat union, and the base class of an ordinary struct, then the struct tries to visit the base class, but no longer parses the field that the union was using as its discriminator.

We don't have any code that demonstrates this, but probably should. I ran into it while working up my POC of what it would take to unbox inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)

Is this broken in master, or do my patches break it?

Got a reproducer?

[...]

#112 lore raw

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

Quoted 17 lines · Markus Armbruster
Quoted 17 lines

scripts/qapi-visit.py out: · line 242

Quoted 11 lines
scripts/qapi-visit.py +4 -4
+def gen_visit_union(name, base, variants):
+    ret = ''
 
     if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_visit_struct_fields(name, base_fields)
+        members = [m for m in base.members if m != variants.tag_member]
+        ret += generate_visit_struct_fields(name, members)

Ouch. This hurts. If the same class is used as both the base class of a flat union, and the base class of an ordinary struct, then the struct tries to visit the base class, but no longer parses the field that the union was using as its discriminator.

Quoted 7 lines
Quoted 3 lines

We don't have any code that demonstrates this, but probably should. I ran into it while working up my POC of what it would take to unbox inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)

Is this broken in master, or do my patches break it?

Got a reproducer?

Turns out I'm mistaken; we got lucky. The call to generate_visit_struct_fields() creates a function for 'name' (aka the union name), and not for 'base' (aka the class name that owns the fields). So even if we have Base as a common struct between Child and Union, the code emitted for Child generates visit_Base_fields(), while the code emitted for Union generates visit_Union_fields().

So there is no reproducer, but _only_ as long as we reject unions as a base class for any other object. And there is redundancy: we could reuse visit_Base_fields() for the sake of the union, then avoid (re-)visiting the discriminator, and we would no longer need to emit visit_Union_fields(). But I can do that as part of the followup cleanups; since I don't see anything broken with your patch, we don't have to worry about it during this series.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#113 lore raw

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

Quoted 34 lines · Eric Blake
Quoted 17 lines · Markus Armbruster
Quoted 17 lines

scripts/qapi-visit.py out: · line 242

Quoted 11 lines
scripts/qapi-visit.py +4 -4
+def gen_visit_union(name, base, variants):
+    ret = ''
 
     if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_visit_struct_fields(name, base_fields)
+        members = [m for m in base.members if m != variants.tag_member]
+        ret += generate_visit_struct_fields(name, members)

Ouch. This hurts. If the same class is used as both the base class of a flat union, and the base class of an ordinary struct, then the struct tries to visit the base class, but no longer parses the field that the union was using as its discriminator.

Quoted 7 lines
Quoted 3 lines

We don't have any code that demonstrates this, but probably should. I ran into it while working up my POC of what it would take to unbox inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)

Is this broken in master, or do my patches break it?

Got a reproducer?

Turns out I'm mistaken; we got lucky. The call to generate_visit_struct_fields() creates a function for 'name' (aka the union name), and not for 'base' (aka the class name that owns the fields). So even if we have Base as a common struct between Child and Union, the code emitted for Child generates visit_Base_fields(), while the code emitted for Union generates visit_Union_fields().

Quoted 7 lines

So there is no reproducer, but _only_ as long as we reject unions as a base class for any other object. And there is redundancy: we could reuse visit_Base_fields() for the sake of the union, then avoid (re-)visiting the discriminator, and we would no longer need to emit visit_Union_fields(). But I can do that as part of the followup cleanups; since I don't see anything broken with your patch, we don't have to worry about it during this series.

Good. Thank you!

#114 lore raw

Re: [Qemu-devel] [PATCH RFC v2 27/47] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

Quoted 23 lines · Markus Armbruster
Quoted 15 lines
Quoted 7 lines
Quoted 3 lines

We don't have any code that demonstrates this, but probably should. I ran into it while working up my POC of what it would take to unbox inherited structs (http://thread.gmane.org/gmane.comp.emulators.qemu/353204)

Is this broken in master, or do my patches break it?

Got a reproducer?

Turns out I'm mistaken; we got lucky. The call to generate_visit_struct_fields() creates a function for 'name' (aka the union name), and not for 'base' (aka the class name that owns the fields). So even if we have Base as a common struct between Child and Union, the code emitted for Child generates visit_Base_fields(), while the code emitted for Union generates visit_Union_fields().

Quoted 7 lines

So there is no reproducer, but _only_ as long as we reject unions as a base class for any other object. And there is redundancy: we could reuse visit_Base_fields() for the sake of the union, then avoid (re-)visiting the discriminator, and we would no longer need to emit visit_Union_fields(). But I can do that as part of the followup cleanups; since I don't see anything broken with your patch, we don't have to worry about it during this series.

Quoted 1 line

Good. Thank you!

And here's a followup patch that cleans it up:
http://thread.gmane.org/gmane.comp.emulators.qemu/353204/focus=353728

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#115 lore raw

RFC PATCH v2 28/47 qapi-commands: Convert to QAPISchemaVisitor

In reply to Markus Armbruster

Output unchanged except for white-space.

---
 scripts/qapi-commands.py | 157 ++++++++++++++++++++++++++---------------------
 scripts/qapi.py          |   2 +-
 2 files changed, 87 insertions(+), 72 deletions(-)
scripts/qapi-commands.py +86 -71
index 3cdbd97..a225bdd 100644
@@ -12,21 +12,22 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 import re
 
 def generate_command_decl(name, args, ret_type):
     arglist=""
-    for argname, argtype, optional in parse_args(args):
-        argtype = c_type(argtype, is_param=True)
-        if optional:
-            arglist += "bool has_%s, " % c_name(argname)
-        arglist += "%s %s, " % (argtype, c_name(argname))
+    if args:
+        for memb in args.members:
+            argtype = memb.type.c_type(is_param=True)
+            if memb.optional:
+                arglist += "bool has_%s, " % c_name(memb.name)
+            arglist += "%s %s, " % (argtype, c_name(memb.name))
     return mcgen('''
 %(ret_type)s qmp_%(name)s(%(args)sError **errp);
 ''',
-                 ret_type=c_type(ret_type), name=c_name(name),
+                 ret_type=(ret_type and ret_type.c_type()) or 'void',
+                 name=c_name(name),
                  args=arglist)
 
 def gen_err_check(err):
@@ -45,10 +46,11 @@ def gen_sync_call(name, args, ret_type):
     retval=""
     if ret_type:
         retval = "retval = "
-    for argname, argtype, optional in parse_args(args):
-        if optional:
-            arglist += "has_%s, " % c_name(argname)
-        arglist += "%s, " % (c_name(argname))
+    if args:
+        for memb in args.members:
+            if memb.optional:
+                arglist += "has_%s, " % c_name(memb.name)
+            arglist += "%s, " % c_name(memb.name)
     push_indent()
     ret = mcgen('''
 %(retval)sqmp_%(name)s(%(args)s&local_err);
@@ -68,7 +70,7 @@ def gen_visitor_input_containers_decl(args):
     ret = ""
 
     push_indent()
-    if len(args) > 0:
+    if args:
         ret += mcgen('''
 QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
 QapiDeallocVisitor *md;
@@ -81,22 +83,26 @@ Visitor *v;
 def gen_visitor_input_vars_decl(args):
     ret = ""
     push_indent()
-    for argname, argtype, optional in parse_args(args):
-        if optional:
-            ret += mcgen('''
+
+    if args:
+        for memb in args.members:
+            if memb.optional:
+                ret += mcgen('''
 bool has_%(argname)s = false;
 ''',
-                         argname=c_name(argname))
-        if is_c_ptr(argtype):
-            ret += mcgen('''
+                             argname=c_name(memb.name))
+            if is_c_ptr(memb.type.c_type()):
+                ret += mcgen('''
 %(argtype)s %(argname)s = NULL;
 ''',
-                         argname=c_name(argname), argtype=c_type(argtype))
-        else:
-            ret += mcgen('''
+                             argname=c_name(memb.name),
+                             argtype=memb.type.c_type())
+            else:
+                ret += mcgen('''
 %(argtype)s %(argname)s = {0};
 ''',
-                         argname=c_name(argname), argtype=c_type(argtype))
+                             argname=c_name(memb.name),
+                             argtype=memb.type.c_type())
 
     pop_indent()
     return ret
@@ -106,7 +112,7 @@ def gen_visitor_input_block(args, dealloc=False):
     errparg = '&local_err'
     errarg = 'local_err'
 
-    if len(args) == 0:
+    if not args:
         return ret
 
     push_indent()
@@ -124,25 +130,26 @@ v = qapi_dealloc_get_visitor(md);
 v = qmp_input_get_visitor(mi);
 ''')
 
-    for argname, argtype, optional in parse_args(args):
-        if optional:
+    for memb in args.members:
+        if memb.optional:
             ret += mcgen('''
 visit_optional(v, &has_%(c_name)s, "%(name)s", %(errp)s);
 ''',
-                         c_name=c_name(argname), name=argname, errp=errparg)
+                         c_name=c_name(memb.name), name=memb.name,
+                         errp=errparg)
             ret += gen_err_check(errarg)
             ret += mcgen('''
 if (has_%(c_name)s) {
 ''',
-                         c_name=c_name(argname))
+                         c_name=c_name(memb.name))
             push_indent()
         ret += mcgen('''
 visit_type_%(visitor)s(v, &%(c_name)s, "%(name)s", %(errp)s);
 ''',
-                     c_name=c_name(argname), name=argname, argtype=argtype,
-                     visitor=type_name(argtype), errp=errparg)
+                     c_name=c_name(memb.name), name=memb.name,
+                     visitor=memb.type.c_name(), errp=errparg)
         ret += gen_err_check(errarg)
-        if optional:
+        if memb.optional:
             pop_indent()
             ret += mcgen('''
 }
@@ -160,6 +167,7 @@ def gen_marshal_output(name, ret_type):
         return ""
 
     ret = mcgen('''
+
 static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_out, Error **errp)
 {
     Error *local_err = NULL;
@@ -183,8 +191,8 @@ out:
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                c_ret_type=c_type(ret_type), c_name=c_name(name),
-                visitor=type_name(ret_type))
+                c_ret_type=ret_type.c_type(), c_name=c_name(name),
+                visitor=ret_type.c_name())
 
     return ret
 
@@ -198,6 +206,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
     hdr = gen_marshal_input_decl(name, middle_mode)
 
     ret = mcgen('''
+
 %(header)s
 {
     Error *local_err = NULL;
@@ -205,16 +214,16 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
                 header=hdr)
 
     if ret_type:
-        if is_c_ptr(ret_type):
-            retval = "    %s retval = NULL;" % c_type(ret_type)
+        if is_c_ptr(ret_type.c_type()):
+            retval = "    %s retval = NULL;" % ret_type.c_type()
         else:
-            retval = "    %s retval;" % c_type(ret_type)
+            retval = "    %s retval;" % ret_type.c_type()
         ret += mcgen('''
 %(retval)s
 ''',
                      retval=retval)
 
-    if len(args) > 0:
+    if args:
         ret += gen_visitor_input_containers_decl(args)
         ret += gen_visitor_input_vars_decl(args) + '\n'
         ret += gen_visitor_input_block(args) + '\n'
@@ -241,21 +250,23 @@ out:
 ''')
     return ret
 
-def gen_registry(commands):
-    registry=""
+def gen_register_command(name, success_response):
     push_indent()
-    for cmd in commands:
-        options = 'QCO_NO_OPTIONS'
-        if not cmd.get('success-response', True):
-            options = 'QCO_NO_SUCCESS_RESP'
+    options = 'QCO_NO_OPTIONS'
+    if not success_response:
+        options = 'QCO_NO_SUCCESS_RESP'
 
-        registry += mcgen('''
+    ret = mcgen('''
 qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
 ''',
-                     name=cmd['command'], c_name=c_name(cmd['command']),
+                     name=name, c_name=c_name(name),
                      opts=options)
     pop_indent()
+    return ret
+
+def gen_registry(registry):
     ret = mcgen('''
+
 static void qmp_init_marshal(void)
 {
 ''')
@@ -267,6 +278,31 @@ qapi_init(qmp_init_marshal);
 ''')
     return ret
 
+class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.regy = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.regy = ''
+    def visit_end(self):
+        if not middle_mode:
+            self.defn += gen_registry(self.regy)
+            self.regy = None
+    def visit_command(self, name, info, args, rets, gen, success_response):
+        if not gen:
+            return
+        self.decl += generate_command_decl(name, args, rets)
+        if rets:
+            self.defn += gen_marshal_output(name, rets)
+        if middle_mode:
+            self.decl += gen_marshal_input_decl(name, middle_mode) + ';\n'
+        self.defn += gen_marshal_input(name, args, rets, middle_mode)
+        if not middle_mode:
+            self.regy += gen_register_command(name, success_response)
+
 middle_mode = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
@@ -276,10 +312,6 @@ for o, a in opts:
     if o in ("-m", "--middle"):
         middle_mode = True
 
-exprs = QAPISchema(input_file).get_exprs()
-commands = filter(lambda expr: expr.has_key('command'), exprs)
-commands = filter(lambda expr: not expr.has_key('gen'), commands)
-
 c_comment = '''
 /*
  * schema-defined QMP->QAPI command dispatch
@@ -335,29 +367,12 @@ fdecl.write(mcgen('''
 #include "qapi/error.h"
 
 ''',
-                 prefix=prefix))
+                  prefix=prefix))
 
-for cmd in commands:
-    arglist = []
-    ret_type = None
-    if cmd.has_key('data'):
-        arglist = cmd['data']
-    if cmd.has_key('returns'):
-        ret_type = cmd['returns']
-    ret = generate_command_decl(cmd['command'], arglist, ret_type)
-    fdecl.write(ret)
-    if ret_type:
-        ret = gen_marshal_output(cmd['command'], ret_type) + "\n"
-        fdef.write(ret)
-
-    if middle_mode:
-        fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], middle_mode))
-
-    ret = gen_marshal_input(cmd['command'], arglist, ret_type, middle_mode) + "\n"
-    fdef.write(ret)
-
-if not middle_mode:
-    ret = gen_registry(commands)
-    fdef.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenCommandVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
scripts/qapi.py +1 -1
index 20ffdaf..98488be 100644
@@ -1366,7 +1366,7 @@ def c_type(value, is_param=False):
         return c_name(value) + pointer_suffix
 
 def is_c_ptr(value):
-    return c_type(value).endswith(pointer_suffix)
+    return value.endswith(pointer_suffix)
 
 def genindent(count):
     ret = ""
Signature · 1 line
1.9.3
Reviewed-by on v2 28/47
#116 lore raw

Re: [Qemu-devel] [PATCH RFC v2 28/47] qapi-commands: Convert to QAPISchemaVisitor

Quoted 1 line · Markus Armbruster

Output unchanged except for white-space.

Indeed, and the diffstat shows it was only one blank line:

 qga-qmp-marshal.c |    1 +
 1 file changed, 1 insertion(+)

MUCH friendlier to review :)

Quoted 7 lines
---
 scripts/qapi-commands.py | 157 ++++++++++++++++++++++++++---------------------
 scripts/qapi.py          |   2 +-
 2 files changed, 87 insertions(+), 72 deletions(-)

Here, I can confidently say:

scripts/qapi.py def c_type(value, is_param=False): · line 1369

Quoted 7 lines

+++ b/scripts/qapi.py

scripts/qapi.py +1 -1
@@ -1366,7 +1366,7 @@ def c_type(value, is_param=False):
         return c_name(value) + pointer_suffix
 
 def is_c_ptr(value):
-    return c_type(value).endswith(pointer_suffix)
+    return value.endswith(pointer_suffix)

Perhaps this cleanup could be floated earlier in the series?

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#117 lore raw

Re: [Qemu-devel] [PATCH RFC v2 28/47] qapi-commands: Convert to QAPISchemaVisitor

Quoted 19 lines · Eric Blake
Quoted 1 line · Markus Armbruster

Output unchanged except for white-space.

Indeed, and the diffstat shows it was only one blank line:

 qga-qmp-marshal.c |    1 +
 1 file changed, 1 insertion(+)

MUCH friendlier to review :)

Quoted 8 lines
---
 scripts/qapi-commands.py | 157
++++++++++++++++++++++++++---------------------
 scripts/qapi.py          |   2 +-
 2 files changed, 87 insertions(+), 72 deletions(-)
Quoted 3 lines

Here, I can confidently say:

Thanks!

Quoted 9 lines

scripts/qapi.py def c_type(value, is_param=False): · line 1369

Quoted 7 lines

+++ b/scripts/qapi.py

scripts/qapi.py +1 -1
@@ -1366,7 +1366,7 @@ def c_type(value, is_param=False):
         return c_name(value) + pointer_suffix
 
 def is_c_ptr(value):
-    return c_type(value).endswith(pointer_suffix)
+    return value.endswith(pointer_suffix)

Perhaps this cleanup could be floated earlier in the series?

Before this patch, is_c_ptr() is called with an argument suitable for c_type(). It returns true when c_type() returns a string ending with pointer_suffix.

This patch replaces uses of function c_type() by QAPISchemaType method c_type().

Two ways to do that for the is_c_ptr() calls:

1. Change the argument to a QAPISchemaType object, and make is_c_ptr() call its .c_type().

2. Change the argument to a string, and make the caller compute it by calling the appropriate object's .c_type().

I picked 2. It hardly matters, because is_c_ptr() will go away in the next patch.

I could do a part of 2. in an earlier patch, namely lifting the c_type() call into the callers. But I can't do the conversion from function to method any earlier, because the objects become available only in this patch.

#118 lore raw

RFC PATCH v2 29/47 qapi: Replace dirty is_c_ptr() by method c_null()

In reply to Markus Armbruster

is_c_ptr() looks whether the end of the C text for the type looks like a pointer. Works, but is fragile.

We now have a better tool: use QAPISchemaType method c_null(). The initializers for non-pointers become prettier: 0, false or the enumeration constant with the value 0 instead of {0}.

One place looks suspicious: we initialize pointers, but not non-pointers. Either the initialization is superfluous and should be deleted, or the non-pointers need it as well, or something subtle is going on and needs a comment. Since I lack the time to figure it out now, mark it FIXME.

---
 scripts/qapi-commands.py | 19 +++++++------------
 scripts/qapi.py          |  3 ---
 2 files changed, 7 insertions(+), 15 deletions(-)
scripts/qapi-commands.py +7 -12
index a225bdd..d3bddb6 100644
@@ -91,18 +91,12 @@ def gen_visitor_input_vars_decl(args):
 bool has_%(argname)s = false;
 ''',
                              argname=c_name(memb.name))
-            if is_c_ptr(memb.type.c_type()):
-                ret += mcgen('''
-%(argtype)s %(argname)s = NULL;
+            ret += mcgen('''
+%(c_type)s %(c_name)s = %(c_null)s;
 ''',
-                             argname=c_name(memb.name),
-                             argtype=memb.type.c_type())
-            else:
-                ret += mcgen('''
-%(argtype)s %(argname)s = {0};
-''',
-                             argname=c_name(memb.name),
-                             argtype=memb.type.c_type())
+                         c_name=c_name(memb.name),
+                         c_type=memb.type.c_type(),
+                         c_null=memb.type.c_null())
 
     pop_indent()
     return ret
@@ -214,7 +208,8 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
                 header=hdr)
 
     if ret_type:
-        if is_c_ptr(ret_type.c_type()):
+        # FIXME fishy: only pointers are initialized
+        if ret_type.c_null() == 'NULL':
             retval = "    %s retval = NULL;" % ret_type.c_type()
         else:
             retval = "    %s retval;" % ret_type.c_type()
scripts/qapi.py +0 -3
index 98488be..dd77a30 100644
@@ -1365,9 +1365,6 @@ def c_type(value, is_param=False):
         assert isinstance(value, str) and value != ""
         return c_name(value) + pointer_suffix
 
-def is_c_ptr(value):
-    return value.endswith(pointer_suffix)
-
 def genindent(count):
     ret = ""
     for i in range(count):
Signature · 1 line
1.9.3
#119 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

Quoted 7 lines · Markus Armbruster

is_c_ptr() looks whether the end of the C text for the type looks like a pointer. Works, but is fragile.

We now have a better tool: use QAPISchemaType method c_null(). The initializers for non-pointers become prettier: 0, false or the enumeration constant with the value 0 instead of {0}.

Quoted 5 lines

One place looks suspicious: we initialize pointers, but not non-pointers. Either the initialization is superfluous and should be deleted, or the non-pointers need it as well, or something subtle is going on and needs a comment. Since I lack the time to figure it out now, mark it FIXME.

Commenting on just this part for now (out of time to review this patch proper for today):

scripts/qapi-commands.py def gen_marshal_input(name, args, ret_type, middle_mode): · line 215

Quoted 10 lines
scripts/qapi-commands.py +2 -1
@@ -214,7 +208,8 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
                 header=hdr)
 
     if ret_type:
-        if is_c_ptr(ret_type.c_type()):
+        # FIXME fishy: only pointers are initialized
+        if ret_type.c_null() == 'NULL':
             retval = "    %s retval = NULL;" % ret_type.c_type()
         else:
             retval = "    %s retval;" % ret_type.c_type()

Here's an example function impacted by this:

static void qmp_marshal_input_guest_file_open(QDict *args, QObject **ret, Error
**errp)

{
    Error *local_err = NULL;
    int64_t retval;
    QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
    QapiDeallocVisitor *md;
    Visitor *v;
    char *path = NULL;
    bool has_mode = false;
    char *mode = NULL;
...
    retval = qmp_guest_file_open(path, has_mode, mode, &local_err);
    if (local_err) {
        goto out;
    }

    qmp_marshal_output_guest_file_open(retval, ret, &local_err);

But compare that to any other function that returns a pointer, and you'll see the same pattern (the only use of retval is in the final call to qmp_marshal_output..., right after assigning retval); that is, initializing to NULL is dead code, and you could get away with just always declaring it instead of worrying about c_null() in this code.

Or maybe we have a leak - if the 'if (local_err)' can ever trigger even when a function returned non-NULL, then our out: label is missing a free(retval), but only when retval is a pointer. Or maybe we change the code to assert that retval is NULL if local_err is set after calling the user's function, to prove we don't have a leak.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#120 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

Quoted 50 lines · Eric Blake
Quoted 7 lines · Markus Armbruster

is_c_ptr() looks whether the end of the C text for the type looks like a pointer. Works, but is fragile.

We now have a better tool: use QAPISchemaType method c_null(). The initializers for non-pointers become prettier: 0, false or the enumeration constant with the value 0 instead of {0}.

Quoted 5 lines

One place looks suspicious: we initialize pointers, but not non-pointers. Either the initialization is superfluous and should be deleted, or the non-pointers need it as well, or something subtle is going on and needs a comment. Since I lack the time to figure it out now, mark it FIXME.

Commenting on just this part for now (out of time to review this patch proper for today):

scripts/qapi-commands.py def gen_marshal_input(name, args, ret_type, middle_mode): · line 215

Quoted 10 lines
scripts/qapi-commands.py +2 -1
@@ -214,7 +208,8 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
                 header=hdr)
 
     if ret_type:
-        if is_c_ptr(ret_type.c_type()):
+        # FIXME fishy: only pointers are initialized
+        if ret_type.c_null() == 'NULL':
             retval = "    %s retval = NULL;" % ret_type.c_type()
         else:
             retval = "    %s retval;" % ret_type.c_type()

Here's an example function impacted by this:

static void qmp_marshal_input_guest_file_open(QDict *args, QObject **ret, Error
**errp)

{
    Error *local_err = NULL;
    int64_t retval;
    QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
    QapiDeallocVisitor *md;
    Visitor *v;
    char *path = NULL;
    bool has_mode = false;
    char *mode = NULL;
...
    retval = qmp_guest_file_open(path, has_mode, mode, &local_err);
    if (local_err) {
        goto out;
    }

    qmp_marshal_output_guest_file_open(retval, ret, &local_err);
Quoted 3 lines

But compare that to any other function that returns a pointer, and you'll see the same pattern (the only use of retval is in the final call to qmp_marshal_output..., right after assigning retval); that is,

Correct. Inspection of qapi-commands.py shows retval is only ever read in the call of qmp_marshal_output_FOO().

Quoted 8 lines

initializing to NULL is dead code, and you could get away with just always declaring it instead of worrying about c_null() in this code.

Or maybe we have a leak - if the 'if (local_err)' can ever trigger even when a function returned non-NULL, then our out: label is missing a free(retval), but only when retval is a pointer. Or maybe we change the code to assert that retval is NULL if local_err is set after calling the user's function, to prove we don't have a leak.

Let me rephrase to make sure I understand.

Ignore the (not rets) case, because retval doesn't exist then.

qmp_marshal_output_FOO() visits retval twice. First, with a QMP output visitor to do the actual marshalling. Second, with a QAPI dealloc visitor to destroy it.

If we execute the assignment to retval, we must go on to call qmp_marshal_output_FOO(), or else we have a leak.

If we can reach qmp_marshal_output_FOO() without executing the assignment, we must initialize retval. If we can't, any initialization is unused.

gen_call() generates code of the form

        retval = qmp_FOO(... args ..., &local_err);
        if (local_err) {
            goto out;
        }

        qmp_marshal_output_FOO(retval, ret, &local_err);

Its caller then generates

    out:
        error_propagate(errp, local_err);

and so forth.

Observe:

1. The assignment dominates the only use. Therefore, the initialization is unused. Let's drop it in a separate cleanup patch.

2. We can leak retval only when qmp_FOO() returns non-null and local_err is non-null. This must not happen, because:

a. local_err must be null before the call, and

b. the call must not return non-null when it sets local_err.

We could right after out: assert(!local_err || !retval). Not sure it's worthwhile.

TL;DR: I concur with your analysis.

#121 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

Quoted 7 lines · Markus Armbruster

Let me rephrase to make sure I understand.

Ignore the (not rets) case, because retval doesn't exist then.

qmp_marshal_output_FOO() visits retval twice. First, with a QMP output visitor to do the actual marshalling. Second, with a QAPI dealloc visitor to destroy it.

And the use of the dealloc visitor is buried inside the qmp_marshal_output_FOO() call.

Quoted 23 lines

If we execute the assignment to retval, we must go on to call qmp_marshal_output_FOO(), or else we have a leak.

If we can reach qmp_marshal_output_FOO() without executing the assignment, we must initialize retval. If we can't, any initialization is unused.

gen_call() generates code of the form

        retval = qmp_FOO(... args ..., &local_err);
        if (local_err) {
            goto out;
        }

        qmp_marshal_output_FOO(retval, ret, &local_err);

Observe:

1. The assignment dominates the only use. Therefore, the initialization is unused. Let's drop it in a separate cleanup patch.

Quoted 6 lines

2. We can leak retval only when qmp_FOO() returns non-null and local_err is non-null. This must not happen, because:

a. local_err must be null before the call, and

b. the call must not return non-null when it sets local_err.

We don't state that contract anywhere, but I doubt any of the qmp_FOO() functions violate it, so it is worth making it part of the contract.

Quoted 3 lines

We could right after out: assert(!local_err || !retval). Not sure it's worthwhile.

I think it IS worthwhile, because it would catch buggy callers. Not sure if after out: is the right place (then you'd need an initializer to cover any other code that jumps to out), but this would do the same

retval = qmp_FOO(...);

if (local_err) {
    assert(!retval);
    goto out;
}

qmp_marshal_output_FOO(retval, ...);

Quoted 2 lines

TL;DR: I concur with your analysis.

Is it worth dropping the dead initializer and adding the assert in the same pre-req cleanup patch? Do you want me to submit it since I did the analysis?

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#122 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

Quoted 38 lines · Eric Blake
Quoted 7 lines · Markus Armbruster

Let me rephrase to make sure I understand.

Ignore the (not rets) case, because retval doesn't exist then.

qmp_marshal_output_FOO() visits retval twice. First, with a QMP output visitor to do the actual marshalling. Second, with a QAPI dealloc visitor to destroy it.

And the use of the dealloc visitor is buried inside the qmp_marshal_output_FOO() call.

Quoted 26 lines

If we execute the assignment to retval, we must go on to call qmp_marshal_output_FOO(), or else we have a leak.

If we can reach qmp_marshal_output_FOO() without executing the assignment, we must initialize retval. If we can't, any initialization is unused.

gen_call() generates code of the form

        retval = qmp_FOO(... args ..., &local_err);
        if (local_err) {
            goto out;
        }

        qmp_marshal_output_FOO(retval, ret, &local_err);

Observe:

1. The assignment dominates the only use. Therefore, the initialization is unused. Let's drop it in a separate cleanup patch.

2. We can leak retval only when qmp_FOO() returns non-null and local_err is non-null. This must not happen, because:

Quoted 6 lines
Quoted 3 lines

a. local_err must be null before the call, and

b. the call must not return non-null when it sets local_err.

We don't state that contract anywhere, but I doubt any of the qmp_FOO() functions violate it, so it is worth making it part of the contract.

It's a general Error API rule: set an error exactly on failure. It applies to any function returning errors through an Error **errp parameter, and we generally don't bother to spell it out for the individual functions.

The part that needs to be spelling out is what success and failure mean. A qmp_FOO() returning an object returns null on failure.

Quoted 4 lines
Quoted 2 lines

We could right after out: assert(!local_err || !retval). Not sure it's worthwhile.

I think it IS worthwhile, because it would catch buggy callers. Not

We use the same assumption all over the place, without asserting it.

The boilerplate around calls using the Error API is bad enough as it is. I'm reluctant to make it worse by adding assertions.

Less of an issue in generated code, but violations are even less likely there.

However, I'm reluctant to assert in some places, but not all, because inconsistency breeds confusion.

In general, I prefer a function's pre- and post-conditions to be asserted in the function itself, not in its callers. Asserting selected parts of the post-condition in callers can be occasionally helpful to help human readers and static analyzers lacking insight into the function.

Quoted 9 lines

sure if after out: is the right place (then you'd need an initializer to cover any other code that jumps to out), but this would do the same

retval = qmp_FOO(...);

if (local_err) {
    assert(!retval);
    goto out;
}

qmp_marshal_output_FOO(retval, ...);

Not quite the same: it doesn't catch the case !local_err && !retval. But it doesn't matter, because that case is bound to die in qmp_marshal_output_FOO().

Quoted 5 lines
Quoted 1 line

TL;DR: I concur with your analysis.

Is it worth dropping the dead initializer and adding the assert in the same pre-req cleanup patch? Do you want me to submit it since I did the analysis?

Let's drop the useless initializer. As explained above, I don't like the assertion for reasons explained above, but if you want it, I'm willing to take it anyway, in a separate follow-up patch.

I'd prefer to drop the initializer myself myself (with you credited in the commit message), because it's certainly less total work, and quite possibly less work for just for me.

#123 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

Quoted 18 lines · Markus Armbruster
Quoted 9 lines
Quoted 6 lines

2. We can leak retval only when qmp_FOO() returns non-null and local_err is non-null. This must not happen, because:

a. local_err must be null before the call, and

b. the call must not return non-null when it sets local_err.

We don't state that contract anywhere, but I doubt any of the qmp_FOO() functions violate it, so it is worth making it part of the contract.

It's a general Error API rule: set an error exactly on failure. It applies to any function returning errors through an Error **errp parameter, and we generally don't bother to spell it out for the individual functions.

The part that needs to be spelling out is what success and failure mean. A qmp_FOO() returning an object returns null on failure.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

We could right after out: assert(!local_err || !retval). Not sure it's worthwhile.

I think it IS worthwhile, because it would catch buggy callers. Not

We use the same assumption all over the place, without asserting it.

Okay, you've got a point there.

Quoted 8 lines

Let's drop the useless initializer. As explained above, I don't like the assertion for reasons explained above, but if you want it, I'm willing to take it anyway, in a separate follow-up patch.

I'd prefer to drop the initializer myself myself (with you credited in the commit message), because it's certainly less total work, and quite possibly less work for just for me.

No need to add assertions. Maybe worth a patch to add a comment somewhere, maybe as a new section in docs/qapi-code-gen.txt, documenting how to write a handler and hook it into qmp-commands.hx, and what conditions the handler must obey. And I'm fine with you dropping the initializer as part of your v3 series.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#124 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

Quoted 38 lines · Eric Blake
Quoted 18 lines · Markus Armbruster
Quoted 9 lines
Quoted 6 lines

2. We can leak retval only when qmp_FOO() returns non-null and local_err is non-null. This must not happen, because:

a. local_err must be null before the call, and

b. the call must not return non-null when it sets local_err.

We don't state that contract anywhere, but I doubt any of the qmp_FOO() functions violate it, so it is worth making it part of the contract.

It's a general Error API rule: set an error exactly on failure. It applies to any function returning errors through an Error **errp parameter, and we generally don't bother to spell it out for the individual functions.

The part that needs to be spelling out is what success and failure mean. A qmp_FOO() returning an object returns null on failure.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

We could right after out: assert(!local_err || !retval). Not sure it's worthwhile.

I think it IS worthwhile, because it would catch buggy callers. Not

We use the same assumption all over the place, without asserting it.

Okay, you've got a point there.

Quoted 8 lines

Let's drop the useless initializer. As explained above, I don't like the assertion for reasons explained above, but if you want it, I'm willing to take it anyway, in a separate follow-up patch.

I'd prefer to drop the initializer myself myself (with you credited in the commit message), because it's certainly less total work, and quite possibly less work for just for me.

Quoted 5 lines

No need to add assertions. Maybe worth a patch to add a comment somewhere, maybe as a new section in docs/qapi-code-gen.txt, documenting how to write a handler and hook it into qmp-commands.hx, and what conditions the handler must obey. And I'm fine with you dropping the initializer as part of your v3 series.

Sounds like a plan. Thanks!

#125 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

Quoted 14 lines · Markus Armbruster
Quoted 12 lines · Eric Blake
Quoted 10 lines · Markus Armbruster
Quoted 9 lines
Quoted 6 lines

2. We can leak retval only when qmp_FOO() returns non-null and local_err is non-null. This must not happen, because:

a. local_err must be null before the call, and

b. the call must not return non-null when it sets local_err.

We don't state that contract anywhere, but I doubt any of the qmp_FOO() functions violate it, so it is worth making it part of the contract.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

It's a general Error API rule: set an error exactly on failure. It applies to any function returning errors through an Error **errp parameter, and we generally don't bother to spell it out for the individual functions.

The part that needs to be spelling out is what success and failure mean. A qmp_FOO() returning an object returns null on failure.

For qmp_FOO(), this is a reasonable contract. But our very own generated code does not follow these rules: visit_type_FOO() can assign into *obj even when setting an error, if it encounters a parse error halfway through the struct, leaving the caller responsible to still clean up the mess if it wants to avoid a memory leak.

Maybe that means our generated code needs to be reworked to properly clean up on a failed parse, such that *obj is guaranteed to be NULL if an error is returned. As a separate patch, of course.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#126 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

Quoted 20 lines · Eric Blake
Quoted 19 lines · Markus Armbruster
Quoted 17 lines · Eric Blake
Quoted 15 lines · Markus Armbruster
Quoted 9 lines
Quoted 6 lines

2. We can leak retval only when qmp_FOO() returns non-null and local_err is non-null. This must not happen, because:

a. local_err must be null before the call, and

b. the call must not return non-null when it sets local_err.

We don't state that contract anywhere, but I doubt any of the qmp_FOO() functions violate it, so it is worth making it part of the contract.

It's a general Error API rule: set an error exactly on failure. It applies to any function returning errors through an Error **errp parameter, and we generally don't bother to spell it out for the individual functions.

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

The part that needs to be spelling out is what success and failure mean. A qmp_FOO() returning an object returns null on failure.

For qmp_FOO(), this is a reasonable contract. But our very own generated code does not follow these rules: visit_type_FOO() can assign into *obj even when setting an error, if it encounters a parse error halfway through the struct, leaving the caller responsible to still clean up the mess if it wants to avoid a memory leak.

Assigning to *obj, then fail is tolerable[*]. Relying on the caller to free it is not. If we do that, it's a bug.

Quoted 3 lines

Maybe that means our generated code needs to be reworked to properly clean up on a failed parse, such that *obj is guaranteed to be NULL if an error is returned. As a separate patch, of course.

Yes. Would you like to propose a FIXME for me to put into this series?

[*] Leaving *obj alone on failure is nicer, but may not always be practical.

#127 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

Quoted 6 lines · Markus Armbruster
Quoted 5 lines

For qmp_FOO(), this is a reasonable contract. But our very own generated code does not follow these rules: visit_type_FOO() can assign into *obj even when setting an error, if it encounters a parse error halfway through the struct, leaving the caller responsible to still clean up the mess if it wants to avoid a memory leak.

Quoted 8 lines

Assigning to *obj, then fail is tolerable[*]. Relying on the caller to free it is not. If we do that, it's a bug.

Quoted 3 lines

Maybe that means our generated code needs to be reworked to properly clean up on a failed parse, such that *obj is guaranteed to be NULL if an error is returned. As a separate patch, of course.

Yes. Would you like to propose a FIXME for me to put into this series?

Done (see 12.5/47).

Quoted 4 lines

[*] Leaving *obj alone on failure is nicer, but may not always be practical.

I'm wondering if visit_end_struct() should be changed to accept a bool parameter that is true if the struct is ended normally, and false if an error has been detected. We may also need to alter visit_start_struct to return a bool (true if an object was allocated), to help feed our visitor logic on whether to pass true or false to the visit_end_struct().

We did something like that for visit_start_union()/visit_end_union(), but I suspect those visitor interfaces are also a bit screwy. Oh well, I'm not going to try and tweak it today, but am happy with just adding the FIXME.

Also, it would be really nice if we had docs in visitor.h and/or visitor-impl.h, to get some sort of feel for how the visitor is supposed to be used.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#128 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

Quoted 29 lines · Eric Blake
Quoted 6 lines · Markus Armbruster
Quoted 5 lines

For qmp_FOO(), this is a reasonable contract. But our very own generated code does not follow these rules: visit_type_FOO() can assign into *obj even when setting an error, if it encounters a parse error halfway through the struct, leaving the caller responsible to still clean up the mess if it wants to avoid a memory leak.

Quoted 8 lines

Assigning to *obj, then fail is tolerable[*]. Relying on the caller to free it is not. If we do that, it's a bug.

Quoted 3 lines

Maybe that means our generated code needs to be reworked to properly clean up on a failed parse, such that *obj is guaranteed to be NULL if an error is returned. As a separate patch, of course.

Yes. Would you like to propose a FIXME for me to put into this series?

Done (see 12.5/47).

Quoted 4 lines

[*] Leaving *obj alone on failure is nicer, but may not always be practical.

I'm wondering if visit_end_struct() should be changed to accept a bool parameter that is true if the struct is ended normally, and false if an error has been detected. We may also need to alter visit_start_struct to return a bool (true if an object was allocated), to help feed our visitor logic on whether to pass true or false to the visit_end_struct().

Quoted 4 lines

We did something like that for visit_start_union()/visit_end_union(), but I suspect those visitor interfaces are also a bit screwy. Oh well, I'm not going to try and tweak it today, but am happy with just adding the FIXME.

We need to keep the introspection series reasonably focused. I can't afford a detour into visitor semantics right now, so we'll go with your FIXMEs.

Quoted 3 lines

Also, it would be really nice if we had docs in visitor.h and/or visitor-impl.h, to get some sort of feel for how the visitor is supposed to be used.

Indeed. We got a grand total of three one-liner comments, and one of them is inaccurate: actual visitors don't obey /* Must be set */.

Reviewed-by on v2 29/47
#129 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

In reply to Markus Armbruster
Quoted 7 lines · Markus Armbruster

is_c_ptr() looks whether the end of the C text for the type looks like a pointer. Works, but is fragile.

We now have a better tool: use QAPISchemaType method c_null(). The initializers for non-pointers become prettier: 0, false or the enumeration constant with the value 0 instead of {0}.

Quoted 5 lines

One place looks suspicious: we initialize pointers, but not non-pointers. Either the initialization is superfluous and should be deleted, or the non-pointers need it as well, or something subtle is going on and needs a comment. Since I lack the time to figure it out now, mark it FIXME.

I commented on that in another mail - either we have a leak on failure (and need the initializer only for pointers), or we should add an assert (and don't need the initializer at all), depending on what semantics we want to enforce on all handler functions that set their error parameter.
Probably worth cleaning that up in a separate pre-req patch.

Quoted 7 lines
---
 scripts/qapi-commands.py | 19 +++++++------------
 scripts/qapi.py          |  3 ---
 2 files changed, 7 insertions(+), 15 deletions(-)

Nice reduction in size. If the marshal code FIXME disappears due to a separate cleanup, then the rest of this commit is good to go:

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#130 lore raw

Re: [Qemu-devel] [PATCH RFC v2 29/47] qapi: Replace dirty is_c_ptr() by method c_null()

Quoted 14 lines · Eric Blake
Quoted 7 lines · Markus Armbruster

is_c_ptr() looks whether the end of the C text for the type looks like a pointer. Works, but is fragile.

We now have a better tool: use QAPISchemaType method c_null(). The initializers for non-pointers become prettier: 0, false or the enumeration constant with the value 0 instead of {0}.

Quoted 5 lines

One place looks suspicious: we initialize pointers, but not non-pointers. Either the initialization is superfluous and should be deleted, or the non-pointers need it as well, or something subtle is going on and needs a comment. Since I lack the time to figure it out now, mark it FIXME.

Quoted 5 lines

I commented on that in another mail - either we have a leak on failure (and need the initializer only for pointers), or we should add an assert (and don't need the initializer at all), depending on what semantics we want to enforce on all handler functions that set their error parameter.
Probably worth cleaning that up in a separate pre-req patch.

A function call should either fail completely or succeed completely.

On success, it must not set an error (d'oh!).

On failure, it must set an error (d'oh again!), and should not return stuff for the caller to clean up.

The rules on setting errors together with the rule that a &local_err argument must contain null before the call ensures that afterwards local_err is set exactly when the call failed.

The rule on returning stuff lets us keep the error path simple: no need for cleaning up anything returned by the function. This is pervasive in the code. We generally don't bother to assert() the function actually obeys the rule.

Perhaps I should fix up my pending patch "error: Revamp interface documentation"[*] to cover the "should not return stuff for the caller to clean up" part.

Apply to handler functions returning objects in need of clean up:

        retval = qmp_FOO(... args ..., &local_err);
        if (local_err) {
            goto out;
        }

        qmp_marshal_output_FOO(retval, ret, &local_err);

On success, qmp_FOO() must not touch local_err.

On failure, qmp_FOO() must set local_err, and return null.

Bypassing the cleanup qmp_marshal_output_FOO() is therfore perfectly fine.

Quoted 10 lines
Quoted 6 lines
---
 scripts/qapi-commands.py | 19 +++++++------------
 scripts/qapi.py          |  3 ---
 2 files changed, 7 insertions(+), 15 deletions(-)

Nice reduction in size. If the marshal code FIXME disappears due to a separate cleanup, then the rest of this commit is good to go:

Thanks!

[*] https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg04756.html

#131 lore raw

RFC PATCH v2 30/47 qapi: De-duplicate enum code generation

In reply to Markus Armbruster

Duplicated in commit 21cd70d. Yes, we can't import qapi-types, but that's no excuse. Move the helpers from qapi-types.py to qapi.py, and replace the duplicates in qapi-event.py.

The generated event enumeration type gains a _MAX member, and its lookup table becomes const-correct (see commit 2e4450f).

---
 scripts/qapi-event.py | 67 +++------------------------------------------------
 scripts/qapi-types.py | 55 ------------------------------------------
 scripts/qapi.py       | 55 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 59 insertions(+), 118 deletions(-)
scripts/qapi-event.py +4 -63
index aec2d32..aed45d6 100644
@@ -153,63 +153,6 @@ def generate_event_implement(api_name, event_name, params):
 
     return ret
 
-
-# Following are the functions that generate an enum type for all defined
-# events, similar to qapi-types.py. Here we already have enum name and
-# values which were generated before and recorded in event_enum_*. It also
-# works around the issue that "import qapi-types" can't work.
-
-def generate_event_enum_decl(event_enum_name, event_enum_values):
-    lookup_decl = mcgen('''
-
-extern const char *%(event_enum_name)s_lookup[];
-''',
-                        event_enum_name = event_enum_name)
-
-    enum_decl = mcgen('''
-typedef enum %(event_enum_name)s {
-''',
-                      event_enum_name = event_enum_name)
-
-    # append automatically generated _MAX value
-    enum_max_value = c_enum_const(event_enum_name, "MAX")
-    enum_values = event_enum_values + [ enum_max_value ]
-
-    i = 0
-    for value in enum_values:
-        enum_decl += mcgen('''
-    %(value)s = %(i)d,
-''',
-                     value = value,
-                     i = i)
-        i += 1
-
-    enum_decl += mcgen('''
-} %(event_enum_name)s;
-''',
-                       event_enum_name = event_enum_name)
-
-    return lookup_decl + enum_decl
-
-def generate_event_enum_lookup(event_enum_name, event_enum_strings):
-    ret = mcgen('''
-
-const char *%(event_enum_name)s_lookup[] = {
-''',
-                event_enum_name = event_enum_name)
-
-    for string in event_enum_strings:
-        ret += mcgen('''
-    "%(string)s",
-''',
-                     string = string)
-
-    ret += mcgen('''
-    NULL,
-};
-''')
-    return ret
-
 (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
 
 c_comment = '''
@@ -266,8 +209,7 @@ fdecl.write(mcgen('''
 exprs = QAPISchema(input_file).get_exprs()
 
 event_enum_name = c_name(prefix + "QAPIEvent", protect=False)
-event_enum_values = []
-event_enum_strings = []
+event_names = []
 
 for expr in exprs:
     if expr.has_key('event'):
@@ -286,12 +228,11 @@ for expr in exprs:
         fdef.write(ret)
 
         # Record it, and generate enum later
-        event_enum_values.append(event_enum_value)
-        event_enum_strings.append(event_name)
+        event_names.append(event_name)
 
-ret = generate_event_enum_decl(event_enum_name, event_enum_values)
+ret = generate_enum(event_enum_name, event_names)
 fdecl.write(ret)
-ret = generate_event_enum_lookup(event_enum_name, event_enum_strings)
+ret = generate_enum_lookup(event_enum_name, event_names)
 fdef.write(ret)
 
 close_output(fdef, fdecl)
scripts/qapi-types.py +0 -55

61 lines not on this page · raw message

scripts/qapi.py +55 -0

61 lines not on this page · raw message

Signature · 1 line
1.9.3
Reviewed-by on v2 30/47
#132 lore raw

Re: [Qemu-devel] [PATCH RFC v2 30/47] qapi: De-duplicate enum code generation

Quoted 5 lines · Markus Armbruster

Duplicated in commit 21cd70d. Yes, we can't import qapi-types, but that's no excuse. Move the helpers from qapi-types.py to qapi.py, and replace the duplicates in qapi-event.py.

The generated event enumeration type gains a _MAX member,

Not quite; the _MAX member was already there, but done by hand [1] instead of by common code.

Quoted 2 lines

and its
lookup table becomes const-correct (see commit 2e4450f).

Maybe what you meant instead of gaining _MAX was that it also gains C99 initializers:

-const char *QAPIEvent_lookup[] = {
- "ACPI_DEVICE_OST",
- "BALLOON_CHANGE",
...
- "WATCHDOG",
- NULL,
+const char *const QAPIEvent_lookup[] = {
+ [QAPI_EVENT_ACPI_DEVICE_OST] = "ACPI_DEVICE_OST",
+ [QAPI_EVENT_BALLOON_CHANGE] = "BALLOON_CHANGE",

Overall, a good change.

Quoted 7 lines
---
 scripts/qapi-event.py | 67 +++------------------------------------------------
 scripts/qapi-types.py | 55 ------------------------------------------
 scripts/qapi.py       | 55 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 59 insertions(+), 118 deletions(-)

Since only the commit message needs tweaking,

scripts/qapi-event.py def generate_event_implement(api_name, event_name, params): · line 156

Quoted 10 lines
scripts/qapi-event.py +0 -3
index aec2d32..aed45d6 100644
@@ -153,63 +153,6 @@ def generate_event_implement(api_name, event_name, params):
 
-    # append automatically generated _MAX value
-    enum_max_value = c_enum_const(event_enum_name, "MAX")
-    enum_values = event_enum_values + [ enum_max_value ]

[1]

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#133 lore raw

Re: [Qemu-devel] [PATCH RFC v2 30/47] qapi: De-duplicate enum code generation

Quoted 9 lines · Eric Blake
Quoted 5 lines · Markus Armbruster

Duplicated in commit 21cd70d. Yes, we can't import qapi-types, but that's no excuse. Move the helpers from qapi-types.py to qapi.py, and replace the duplicates in qapi-event.py.

The generated event enumeration type gains a _MAX member,

Not quite; the _MAX member was already there, but done by hand [1] instead of by common code.

You're right; I was confused.

Quoted 6 lines
Quoted 2 lines

and its
lookup table becomes const-correct (see commit 2e4450f).

Maybe what you meant instead of gaining _MAX was that it also gains C99 initializers:

Review

Quoted 9 lines
(quoted diff) +3 -5
-const char *QAPIEvent_lookup[] = {
-    "ACPI_DEVICE_OST",
-    "BALLOON_CHANGE",
...
-    "WATCHDOG",
-    NULL,
+const char *const QAPIEvent_lookup[] = {
+    [QAPI_EVENT_ACPI_DEVICE_OST] = "ACPI_DEVICE_OST",
+    [QAPI_EVENT_BALLOON_CHANGE] = "BALLOON_CHANGE",

That's not what I had in mind when I wrote the sentence about _MAX. but mentioning the change in the commit message won't hurt. Note to self:
comes from commit 912ae9c.

Quoted 11 lines

Overall, a good change.

Quoted 8 lines
---
 scripts/qapi-event.py | 67
+++------------------------------------------------
 scripts/qapi-types.py | 55 ------------------------------------------
 scripts/qapi.py       | 55 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 59 insertions(+), 118 deletions(-)
Quoted 2 lines

Since only the commit message needs tweaking,

Thanks!

[...]

#134 lore raw

RFC PATCH v2 31/47 qapi-event: Eliminate global variable event_enum_value

In reply to Markus Armbruster
---
 scripts/qapi-event.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi-event.py +1 -2
index aed45d6..537da17 100644
@@ -137,7 +137,7 @@ def generate_event_implement(api_name, event_name, params):
     emit(%(event_enum_value)s, qmp, &local_err);
 
 """,
-                 event_enum_value = event_enum_value)
+                 event_enum_value = c_enum_const(event_enum_name, event_name))
 
     # step 5: clean up
     if params:
@@ -223,7 +223,6 @@ for expr in exprs:
         fdecl.write(ret)
 
         # We need an enum value per event
-        event_enum_value = c_enum_const(event_enum_name, event_name)
         ret = generate_event_implement(api_name, event_name, params)
         fdef.write(ret)
 
Signature · 1 line
1.9.3
Reviewed-by on v2 31/47
#135 lore raw

Re: [Qemu-devel] [PATCH RFC v2 31/47] qapi-event: Eliminate global variable event_enum_value

Quoted 5 lines · Markus Armbruster
---
 scripts/qapi-event.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#136 lore raw

RFC PATCH v2 32/47 qapi-event: Convert to QAPISchemaVisitor, fixing data with base

In reply to Markus Armbruster

Fixes events whose data is struct with base to include the struct's base members. Test case is qapi-schema-test.json's event __org.qemu_x-command:

    { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }

    { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member2': 'str' } }

    { 'struct': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }

Patch's effect on generated qapi_event_send___org_qemu_x_event():

    -void qapi_event_send___org_qemu_x_event(const char *__org_qemu_x_member2,
    +void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum __org_qemu_x_member1,
    +                                        const char *__org_qemu_x_member2,
                                             Error **errp)
     {
         QDict *qmp;
    @@ -224,6 +225,10 @@ void qapi_event_send___org_qemu_x_event(
             goto clean;
         }

    +    visit_type___org_qemu_x_Enum(v, &__org_qemu_x_member1, "__org.qemu_x-member1", &local_err);
    +    if (local_err) {
    +        goto clean;
    +    }
         visit_type_str(v, (char **)&__org_qemu_x_member2, "__org.qemu_x-member2", &local_err);
         if (local_err) {
             goto clean;
---
 scripts/qapi-event.py                   | 87 ++++++++++++++++-----------------
 tests/qapi-schema/qapi-schema-test.json |  3 --
 2 files changed, 43 insertions(+), 47 deletions(-)
scripts/qapi-event.py +43 -44
index 537da17..456e590 100644
@@ -2,28 +2,29 @@
 # QAPI event generator
 #
 # Copyright (c) 2014 Wenchao Xia
+# Copyright (c) 2013-2015 Red Hat Inc.
 #
 # Authors:
 #  Wenchao Xia <[email protected]>
+#  Markus Armbruster <[email protected]>
 #
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 
-def _generate_event_api_name(event_name, params):
+def _generate_event_api_name(event_name, data):
     api_name = "void qapi_event_send_%s(" % c_name(event_name).lower();
     l = len(api_name)
 
-    if params:
-        for argname, argentry, optional in parse_args(params):
-            if optional:
-                api_name += "bool has_%s,\n" % c_name(argname)
+    if data:
+        for m in data.members:
+            if m.optional:
+                api_name += "bool has_%s,\n" % c_name(m.name)
                 api_name += "".ljust(l)
 
-            api_name += "%s %s,\n" % (c_type(argentry, is_param=True),
-                                      c_name(argname))
+            api_name += "%s %s,\n" % (m.type.c_type(is_param=True),
+                                      c_name(m.name))
             api_name += "".ljust(l)
 
     api_name += "Error **errp)"
@@ -51,7 +52,7 @@ def generate_event_implement(api_name, event_name, params):
 """,
                 api_name = api_name)
 
-    if params:
+    if params and params.members:
         ret += mcgen("""
     QmpOutputVisitor *qov;
     Visitor *v;
@@ -72,7 +73,7 @@ def generate_event_implement(api_name, event_name, params):
                  event_name = event_name)
 
     # step 3: visit the params if params != None
-    if params:
+    if params and params.members:
         ret += mcgen("""
     qov = qmp_output_visitor_new();
     g_assert(qov);
@@ -89,15 +90,15 @@ def generate_event_implement(api_name, event_name, params):
 """,
                 event_name = event_name)
 
-        for argname, argentry, optional in parse_args(params):
-            if optional:
+        for memb in params.members:
+            if memb.optional:
                 ret += mcgen("""
     if (has_%(var)s) {
 """,
-                             var = c_name(argname))
+                             var=c_name(memb.name))
                 push_indent()
 
-            if argentry == "str":
+            if memb.type.name == "str":
                 var_type = "(char **)"
             else:
                 var_type = ""
@@ -109,11 +110,11 @@ def generate_event_implement(api_name, event_name, params):
     }
 """,
                          var_type = var_type,
-                         var = c_name(argname),
-                         type = type_name(argentry),
-                         name = argname)
+                         var=c_name(memb.name),
+                         type=memb.type.c_name(),
+                         name=memb.name)
 
-            if optional:
+            if memb.optional:
                 pop_indent()
                 ret += mcgen("""
     }
@@ -140,7 +141,7 @@ def generate_event_implement(api_name, event_name, params):
                  event_enum_value = c_enum_const(event_enum_name, event_name))
 
     # step 5: clean up
-    if params:
+    if params and params.members:
         ret += mcgen("""
  clean:
     qmp_output_visitor_cleanup(qov);
@@ -153,6 +154,24 @@ def generate_event_implement(api_name, event_name, params):
 
     return ret
 
+class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.event_names = None
+    def visit_begin(self):
+        self.decl = ''
+        self.defn = ''
+        self.event_names = []
+    def visit_end(self):
+        self.decl += generate_enum(event_enum_name, self.event_names)
+        self.defn += generate_enum_lookup(event_enum_name, self.event_names)
+    def visit_event(self, name, info, data):
+        api_name = _generate_event_api_name(name, data)
+        self.decl += generate_event_declaration(api_name)
+        self.defn += generate_event_implement(api_name, name, data)
+        self.event_names.append(name)
+
 (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
 
 c_comment = '''
@@ -206,32 +225,12 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = QAPISchema(input_file).get_exprs()
-
 event_enum_name = c_name(prefix + "QAPIEvent", protect=False)
-event_names = []
 
-for expr in exprs:
-    if expr.has_key('event'):
-        event_name = expr['event']
-        params = expr.get('data')
-        if params and len(params) == 0:
-            params = None
-
-        api_name = _generate_event_api_name(event_name, params)
-        ret = generate_event_declaration(api_name)
-        fdecl.write(ret)
-
-        # We need an enum value per event
-        ret = generate_event_implement(api_name, event_name, params)
-        fdef.write(ret)
-
-        # Record it, and generate enum later
-        event_names.append(event_name)
-
-ret = generate_enum(event_enum_name, event_names)
-fdecl.write(ret)
-ret = generate_enum_lookup(event_enum_name, event_names)
-fdef.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenEventVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
tests/qapi-schema/qapi-schema-test.json +0 -3
index b182174..90b4740 100644
@@ -126,9 +126,6 @@
 { 'alternate': '__org.qemu_x-Alt',
   'data': { '__org.qemu_x-branch': 'str', 'b': '__org.qemu_x-Base' } }
 { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }
-# FIXME generated qapi_event_send___org_qemu_x_event() has only a
-# parameter for data's member __org_qemu_x_member2, none for its base
-# __org.qemu_x-Base's member __org_qemu_x_member1
 { 'command': '__org.qemu_x-command',
   'data': { 'a': ['__org.qemu_x-Enum'], 'b': ['__org.qemu_x-Struct'],
             'c': '__org.qemu_x-Union2', 'd': '__org.qemu_x-Alt' },
Signature · 1 line
1.9.3
Reviewed-by on v2 32/47
#137 lore raw

Re: [Qemu-devel] [PATCH RFC v2 32/47] qapi-event: Convert to QAPISchemaVisitor, fixing data with base

Quoted 11 lines · Markus Armbruster

Fixes events whose data is struct with base to include the struct's base members. Test case is qapi-schema-test.json's event __org.qemu_x-command:

    { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }

    { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member2': 'str' } }

    { 'struct': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }

No change to generated code in qemu proper, so this is a corner case we are not yet exploiting. But good to have it fixed :)

Quoted 7 lines

Patch's effect on generated qapi_event_send___org_qemu_x_event():

-void qapi_event_send___org_qemu_x_event(const char *__org_qemu_x_member2, +void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum __org_qemu_x_member1,
+ const char *__org_qemu_x_member2,
Error **errp)

Ouch - I think we have a bug in qapi.py:check_event(). There, we call check_type(... allow_metas=['union', 'struct']) - but it looks like the generated signature requires that we have no variants, which means we cannot have:

{ 'union': 'Un', 'data': ... }
{ 'event': 'EV', 'data': 'Un' }

because it would fail C generation. Sounds like you should add a check to that in 14/47, and a fix for it in 15/47.

Quoted 11 lines
---
 scripts/qapi-event.py                   | 87 ++++++++++++++++-----------------
 tests/qapi-schema/qapi-schema-test.json |  3 --
 2 files changed, 43 insertions(+), 47 deletions(-)
scripts/qapi-event.py +0 -0
index 537da17..456e590 100644

scripts/qapi-event.py line 5

Quoted 4 lines
scripts/qapi-event.py +1 -0
@@ -2,28 +2,29 @@
 # QAPI event generator
 #
 # Copyright (c) 2014 Wenchao Xia
+# Copyright (c) 2013-2015 Red Hat Inc.

Umm - the file didn't exist until 2014; and to my knowledge, you aren't adding anything to it that was copied from some other file dating back to 2013. Using the range 2014-2015 might be better.

But that's minor. And assuming that you reject union types for events in an earlier patch, the rest of this is fine:

Quoted 8 lines
scripts/qapi-event.py +2 -2
@@ -89,15 +90,15 @@ def generate_event_implement(api_name, event_name, params):
 """,
                 event_name = event_name)
 
-        for argname, argentry, optional in parse_args(params):
-            if optional:
+        for memb in params.members:
+            if memb.optional:

scripts/qapi-event.py def generate_event_implement(api_name, event_name, params): · line 102

Quoted 10 lines
scripts/qapi-event.py +2 -2
                 ret += mcgen("""
     if (has_%(var)s) {
 """,
-                             var = c_name(argname))
+                             var=c_name(memb.name))
                 push_indent()
 
-            if argentry == "str":
+            if memb.type.name == "str":
                 var_type = "(char **)"

Our visitors require us to cast away const. Is that something we should consider reworking, so that we don't need to do that? But it's a question for another day.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#138 lore raw

Re: [Qemu-devel] [PATCH RFC v2 32/47] qapi-event: Convert to QAPISchemaVisitor, fixing data with base

Quoted 29 lines · Eric Blake
Quoted 11 lines · Markus Armbruster

Fixes events whose data is struct with base to include the struct's base members. Test case is qapi-schema-test.json's event __org.qemu_x-command:

    { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }

    { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member2': 'str' } }

    { 'struct': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }

No change to generated code in qemu proper, so this is a corner case we are not yet exploiting. But good to have it fixed :)

Quoted 7 lines

Patch's effect on generated qapi_event_send___org_qemu_x_event():

-void qapi_event_send___org_qemu_x_event(const char *__org_qemu_x_member2, +void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum __org_qemu_x_member1,
+ const char *__org_qemu_x_member2,
Error **errp)

Ouch - I think we have a bug in qapi.py:check_event(). There, we call check_type(... allow_metas=['union', 'struct']) - but it looks like the generated signature requires that we have no variants, which means we cannot have:

Quoted 5 lines

{ 'union': 'Un', 'data': ... }
{ 'event': 'EV', 'data': 'Un' }

because it would fail C generation. Sounds like you should add a check to that in 14/47, and a fix for it in 15/47.

Will do.

Quoted 6 lines
Quoted 6 lines
---
 scripts/qapi-event.py | 87 ++++++++++++++++-----------------
 tests/qapi-schema/qapi-schema-test.json |  3 --
 2 files changed, 43 insertions(+), 47 deletions(-)
Quoted 13 lines

scripts/qapi-event.py line 5

Quoted 9 lines
scripts/qapi-event.py +1 -0
index 537da17..456e590 100644
@@ -2,28 +2,29 @@
 # QAPI event generator
 #
 # Copyright (c) 2014 Wenchao Xia
+# Copyright (c) 2013-2015 Red Hat Inc.

Umm - the file didn't exist until 2014; and to my knowledge, you aren't adding anything to it that was copied from some other file dating back to 2013. Using the range 2014-2015 might be better.

Pasto, will fix.

Quoted 19 lines

But that's minor. And assuming that you reject union types for events in an earlier patch, the rest of this is fine:

Quoted 15 lines
scripts/qapi-event.py +3 -3
@@ -89,15 +90,15 @@ def generate_event_implement(api_name, event_name, params):
 """,
                 event_name = event_name)
 
-        for argname, argentry, optional in parse_args(params):
-            if optional:
+        for memb in params.members:
+            if memb.optional:
                 ret += mcgen("""
     if (has_%(var)s) {
 """,
-                             var = c_name(argname))
+                             var=c_name(memb.name))
                 push_indent()
 
Quoted 7 lines

scripts/qapi-event.py def generate_event_implement(api_name, event_name, params): · line 102

Quoted 3 lines
scripts/qapi-event.py +1 -1
-            if argentry == "str":
+            if memb.type.name == "str":
                 var_type = "(char **)"

Our visitors require us to cast away const. Is that something we should consider reworking, so that we don't need to do that? But it's a question for another day.

Yes, it's annoying, and yes, we need to leave it for later.

Thanks!

#139 lore raw

RFC PATCH v2 33/47 qapi: Clean up after recent conversions to QAPISchemaVisitor

In reply to Markus Armbruster

Drop helper functions that are now unused.

Make pylint reasonably happy.

Rename generate_FOO() functions to gen_FOO() for consistency.

Use more consistent and sensible variable names.

Consistently use c_ for mapping keys when their value is a C identifier or type.

Simplify gen_enum() and gen_visit_union()

Consistently use single quotes for C text string literals.

---
 scripts/qapi-commands.py | 109 +++++++++++++++++++------------------
 scripts/qapi-event.py    | 117 +++++++++++++++++++---------------------
 scripts/qapi-types.py    |  68 ++++++++++++-----------
 scripts/qapi-visit.py    | 121 ++++++++++++++++++++---------------------
 scripts/qapi.py          | 138 +++++++++--------------------------------------
 5 files changed, 229 insertions(+), 324 deletions(-)
scripts/qapi-commands.py +56 -53
index d3bddb6..5d11032 100644
@@ -15,20 +15,20 @@
 from qapi import *
 import re
 
-def generate_command_decl(name, args, ret_type):
-    arglist=""
+def gen_command_decl(name, args, rets):
+    argstr = ''
     if args:
         for memb in args.members:
-            argtype = memb.type.c_type(is_param=True)
             if memb.optional:
-                arglist += "bool has_%s, " % c_name(memb.name)
-            arglist += "%s %s, " % (argtype, c_name(memb.name))
+                argstr += 'bool has_%s, ' % c_name(memb.name)
+            argstr += '%s %s, ' % (memb.type.c_type(is_param=True),
+                                   c_name(memb.name))
     return mcgen('''
-%(ret_type)s qmp_%(name)s(%(args)sError **errp);
+%(c_type)s qmp_%(c_name)s(%(args)sError **errp);
 ''',
-                 ret_type=(ret_type and ret_type.c_type()) or 'void',
-                 name=c_name(name),
-                 args=arglist)
+                 c_type=(rets and rets.c_type()) or 'void',
+                 c_name=c_name(name),
+                 args=argstr)
 
 def gen_err_check(err):
     if not err:
@@ -40,34 +40,37 @@ if (%(err)s) {
 ''',
                  err=err)
 
-def gen_sync_call(name, args, ret_type):
-    ret = ""
-    arglist=""
-    retval=""
-    if ret_type:
-        retval = "retval = "
+def gen_call(name, args, rets):
+    ret = ''
+
+    argstr = ''
     if args:
         for memb in args.members:
             if memb.optional:
-                arglist += "has_%s, " % c_name(memb.name)
-            arglist += "%s, " % c_name(memb.name)
+                argstr += 'has_%s, ' % c_name(memb.name)
+            argstr += '%s, ' % c_name(memb.name)
+
+    lhs = ''
+    if rets:
+        lhs = 'retval = '
+
     push_indent()
     ret = mcgen('''
-%(retval)sqmp_%(name)s(%(args)s&local_err);
+%(lhs)sqmp_%(c_name)s(%(args)s&local_err);
 ''',
-                name=c_name(name), args=arglist, retval=retval)
-    if ret_type:
+                c_name=c_name(name), args=argstr, lhs=lhs)
+    if rets:
         ret += gen_err_check('local_err')
         ret += mcgen('''
 
 qmp_marshal_output_%(c_name)s(retval, ret, &local_err);
 ''',
-                            c_name=c_name(name))
+                     c_name=c_name(name))
     pop_indent()
     return ret
 
 def gen_visitor_input_containers_decl(args):
-    ret = ""
+    ret = ''
 
     push_indent()
     if args:
@@ -81,16 +84,16 @@ Visitor *v;
     return ret
 
 def gen_visitor_input_vars_decl(args):
-    ret = ""
+    ret = ''
     push_indent()
 
     if args:
         for memb in args.members:
             if memb.optional:
                 ret += mcgen('''
-bool has_%(argname)s = false;
+bool has_%(c_name)s = false;
 ''',
-                             argname=c_name(memb.name))
+                             c_name=c_name(memb.name))
             ret += mcgen('''
 %(c_type)s %(c_name)s = %(c_null)s;
 ''',
@@ -102,7 +105,7 @@ bool has_%(argname)s = false;
     return ret
 
 def gen_visitor_input_block(args, dealloc=False):
-    ret = ""
+    ret = ''
     errparg = '&local_err'
     errarg = 'local_err'
 
@@ -113,7 +116,7 @@ def gen_visitor_input_block(args, dealloc=False):
 
     if dealloc:
         errparg = 'NULL'
-        errarg = None;
+        errarg = None
         ret += mcgen('''
 qmp_input_visitor_cleanup(mi);
 md = qapi_dealloc_visitor_new();
@@ -138,10 +141,10 @@ if (has_%(c_name)s) {
                          c_name=c_name(memb.name))
             push_indent()
         ret += mcgen('''
-visit_type_%(visitor)s(v, &%(c_name)s, "%(name)s", %(errp)s);
+visit_type_%(c_type)s(v, &%(c_name)s, "%(name)s", %(errp)s);
 ''',
                      c_name=c_name(memb.name), name=memb.name,
-                     visitor=memb.type.c_name(), errp=errparg)
+                     c_type=memb.type.c_name(), errp=errparg)
         ret += gen_err_check(errarg)
         if memb.optional:
             pop_indent()
@@ -156,13 +159,13 @@ qapi_dealloc_visitor_cleanup(md);
     pop_indent()
     return ret
 
-def gen_marshal_output(name, ret_type):
-    if not ret_type:
-        return ""
+def gen_marshal_output(name, rets):
+    if not rets:
+        return ''
 
     ret = mcgen('''
 
-static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_cmd_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
 {
     Error *local_err = NULL;
     QmpOutputVisitor *mo = qmp_output_visitor_new();
@@ -170,7 +173,7 @@ static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_o
     Visitor *v;
 
     v = qmp_output_get_visitor(mo);
-    visit_type_%(visitor)s(v, &ret_in, "unused", &local_err);
+    visit_type_%(c_name)s(v, &ret_in, "unused", &local_err);
     if (local_err) {
         goto out;
     }
@@ -181,23 +184,23 @@ out:
     qmp_output_visitor_cleanup(mo);
     md = qapi_dealloc_visitor_new();
     v = qapi_dealloc_get_visitor(md);
-    visit_type_%(visitor)s(v, &ret_in, "unused", NULL);
+    visit_type_%(c_name)s(v, &ret_in, "unused", NULL);
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                c_ret_type=ret_type.c_type(), c_name=c_name(name),
-                visitor=ret_type.c_name())
+                c_type=rets.c_type(), c_cmd_name=c_name(name),
+                c_name=rets.c_name())
 
     return ret
 
-def gen_marshal_input_decl(name, middle_mode):
+def gen_marshal_input_decl(name):
     ret = 'void qmp_marshal_input_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
     if not middle_mode:
-        ret = "static " + ret
+        ret = 'static ' + ret
     return ret
 
-def gen_marshal_input(name, args, ret_type, middle_mode):
-    hdr = gen_marshal_input_decl(name, middle_mode)
+def gen_marshal_input(name, args, rets):
+    hdr = gen_marshal_input_decl(name)
 
     ret = mcgen('''
 
@@ -207,12 +210,12 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
 ''',
                 header=hdr)
 
-    if ret_type:
+    if rets:
         # FIXME fishy: only pointers are initialized
-        if ret_type.c_null() == 'NULL':
-            retval = "    %s retval = NULL;" % ret_type.c_type()
+        if rets.c_null() == 'NULL':
+            retval = '    %s retval = NULL;' % rets.c_type()
         else:
-            retval = "    %s retval;" % ret_type.c_type()
+            retval = '    %s retval;' % rets.c_type()
         ret += mcgen('''
 %(retval)s
 ''',
@@ -229,9 +232,9 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
 
 ''')
 
-    ret += gen_sync_call(name, args, ret_type)
+    ret += gen_call(name, args, rets)
 
-    if re.search('^ *goto out\\;', ret, re.MULTILINE):
+    if re.search('^ *goto out;', ret, re.MULTILINE):
         ret += mcgen('''
 
 out:
@@ -254,8 +257,8 @@ def gen_register_command(name, success_response):
     ret = mcgen('''
 qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
 ''',
-                     name=name, c_name=c_name(name),
-                     opts=options)
+                name=name, c_name=c_name(name),
+                opts=options)
     pop_indent()
     return ret
 
@@ -289,12 +292,12 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
     def visit_command(self, name, info, args, rets, gen, success_response):
         if not gen:
             return
-        self.decl += generate_command_decl(name, args, rets)
+        self.decl += gen_command_decl(name, args, rets)
         if rets:
             self.defn += gen_marshal_output(name, rets)
         if middle_mode:
-            self.decl += gen_marshal_input_decl(name, middle_mode) + ';\n'
-        self.defn += gen_marshal_input(name, args, rets, middle_mode)
+            self.decl += gen_marshal_input_decl(name) + ';\n'
+        self.defn += gen_marshal_input(name, args, rets)
         if not middle_mode:
             self.regy += gen_register_command(name, success_response)
 
@@ -354,7 +357,7 @@ fdef.write(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                prefix=prefix))
+                 prefix=prefix))
 
 fdecl.write(mcgen('''
 #include "%(prefix)sqapi-types.h"
scripts/qapi-event.py +54 -63

199 lines not on this page · raw message

scripts/qapi-types.py +33 -35

170 lines not on this page · raw message

scripts/qapi-visit.py +61 -60

284 lines not on this page · raw message

scripts/qapi.py +25 -113

191 lines not on this page · raw message

Signature · 1 line
1.9.3
Reviewed-by on v2 33/47
#140 lore raw

Re: [Qemu-devel] [PATCH RFC v2 33/47] qapi: Clean up after recent conversions to QAPISchemaVisitor

Quoted 6 lines · Markus Armbruster

Drop helper functions that are now unused.

Make pylint reasonably happy.

Rename generate_FOO() functions to gen_FOO() for consistency.

Quoted 8 lines

Use more consistent and sensible variable names.

Consistently use c_ for mapping keys when their value is a C identifier or type.

Simplify gen_enum() and gen_visit_union()

Consistently use single quotes for C text string literals.

Not sure if it is worth splitting this into pieces. Fortunately, there are no changes to generated output.

Quoted 15 lines
---
 scripts/qapi-commands.py | 109 +++++++++++++++++++------------------
 scripts/qapi-event.py    | 117 +++++++++++++++++++---------------------
 scripts/qapi-types.py    |  68 ++++++++++++-----------
 scripts/qapi-visit.py    | 121 ++++++++++++++++++++---------------------
 scripts/qapi.py          | 138 +++++++++--------------------------------------
 5 files changed, 229 insertions(+), 324 deletions(-)
scripts/qapi-commands.py +0 -0
index d3bddb6..5d11032 100644

scripts/qapi-commands.py line 18

Quoted 6 lines
scripts/qapi-commands.py +1 -2
@@ -15,20 +15,20 @@
 from qapi import *
 import re
 
-def generate_command_decl(name, args, ret_type):
-    arglist=""
+def gen_command_decl(name, args, rets):

I can see how 'args' is plural (even if it is a single string for the name of a type containing the args), but should it be 'ret' instead of 'rets'?

scripts/qapi-commands.py if (%(err)s) { · line 43

Quoted 11 lines
scripts/qapi-commands.py +1 -6
@@ -40,34 +40,37 @@ if (%(err)s) {
 ''',
                  err=err)
 
-def gen_sync_call(name, args, ret_type):
-    ret = ""
-    arglist=""
-    retval=""
-    if ret_type:
-        retval = "retval = "
+def gen_call(name, args, rets):

At least you're consistent on naming it 'rets',

scripts/qapi-commands.py if (%(err)s) { · line 44

Quoted 1 line
scripts/qapi-commands.py +1 -0
+    ret = ''

and the naming lets you distinguish between the parameter (the type describing returned fields) and the local string (the generated C code holding the return information).

scripts/qapi-commands.py def generate_event_implement(api_name, event_name, params): · line 80

Quoted 7 lines
scripts/qapi-commands.py +2 -1
@@ -82,45 +76,46 @@ def generate_event_implement(api_name, event_name, params):
 
 
+            # Ugly: need to cast away the const
             if memb.type.name == "str":
-                var_type = "(char **)"
+                cast = "(char **)"

And to think I called it out in a previous patch. So you noticed it too :)

Don't you want to use '(char **)' here, since it is a literal string destined for generated C?

scripts/qapi-commands.py def generate_event_implement(api_name, event_name, params): · line 82

Quoted 3 lines
scripts/qapi-commands.py +1 -1
             else:
-                var_type = ""
+                cast = ""

and '' here?

Quoted 12 lines
scripts/qapi-commands.py +3 -2
+++ b/scripts/qapi-types.py
@@ -16,22 +16,22 @@ from qapi import *
 def gen_fwd_object_or_array(name):
     return mcgen('''
 
-typedef struct %(name)s %(name)s;
+typedef struct %(c_name)s %(c_name)s;
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name))
 
 def gen_array(name, element_type):

scripts/qapi-commands.py from qapi import * · line 31

Quoted 10 lines
scripts/qapi-commands.py +2 -2
     return mcgen('''
 
-struct %(name)s {
+struct %(c_name)s {
     union {
         %(c_type)s value;
         uint64_t padding;
     };
-    struct %(name)s *next;
+    struct %(c_name)s *next;

May be some churn here if you like my comment earlier in the series that this 'struct' is redundant.

scripts/qapi-commands.py from qapi import * · line 37

Quoted 8 lines
scripts/qapi-commands.py +3 -2
+++ b/scripts/qapi-visit.py
 
-def generate_visit_struct_fields(name, members, base = None):
+def gen_visit_struct_fields(name, base, members):
     struct_fields_seen.add(name)
 
-                     type=base.c_name(), c_name=c_name('base'))
+                     c_type=base.c_name(), c_name=c_name('base'))

Possible churn here based on my earlier comments about c_name(constant) being constant.

Fairly big, but aside from some minor '' quoting issues, I didn't see anything wrong.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org