Skip to contentSkip to conversation

qapi: QMP introspection

RFC PATCH Superseded Markus Armbruster 5 participants 75 messages Sep 3, 2015 – Sep 7, 2015 Kbuild

v4 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v4 cover qapi: QMP introspection

Still RFC, because we probably want to drop the "just to facilitate review" patches, and maybe squash PATCH 31 or 32 into PATCH 30.

* PATCH 01-24 basically replace the interemediate representation. The replacement isn't complete, but fully functional. See PATCH 02 for rationale and future work.

PATCH 06-09 and 16-17 are temporary scaffolding to ease review. I don't intend to include them in future non-RFC revisions.

* PATCH 25-29 replace the '**' type bypass by a proper top type called 'any'.

* PATCH 30 is the QMP introspection.

* PATCH 31-32 are minor variations to hide a bit more detail from clients.

I'm afraid the size of my series will strain reviewers. I did my level best to split things up into reviewable pieces.

Series is based on "[PATCH 00/26] qapi: Another round of fixes and cleanups" (which actually has 33 patches now). It also needs my "[PATCH for-2.4 0/2] qom: Fix misuse of Error API" to pass tests.

You can fetch it from my git://repo.or.cz/qemu/armbru.git branch qapi-introspect.

v4:
* Straightforward rebase, R-bys retained unless noted otherwise.
* PATCH 02:
- Add a TODO on catching name collisions [Eric]
- Unify how QAPISchemaObjectType.__init__() asserts "either None or isinstance" [Eric]
- Drop bogus "not implemented" comment in simple_union_type() [Eric]
- Use iteritems() where appropriate [Eric]
- Drop dead conditional in _def_command() [Eric]
* PATCH 03:
- Define json_type() always
Just to be neat; it only gets called when it was defined before.
* PATCH 04,10-12,15,30:
- Give visit_begin() parameter schema from the start instead of adding it in PATCH 30 [Eric]
* PATCH 07,16:
- Touch up commit message's hints for reviewers [Eric]
* PATCH 10+11:
- Reword a comment more clearly [Eric]
* PATCH 11:
- Fix test_validate_fail_union_flat_no_discrim() [Eric]
* PATCH 18:
- Drop dead commit message paragraph [Eric]
* PATCH 26:
- Cover ['any'] in qapi-schema-test.json [Eric]
* PATCH 30:
- Rework commit message
- New docs/qapi-code-gen.txt section "Client JSON Protocol introspection"
- Fix docs/qapi-code-gen.txt pastos [Eric]
- Rearrange and document qapi/introspect.json
The forward references are intentional. I picked this order to make the file more readable.
- Fill in proper qmp-commands.hx documentation
- Uniform spacing in generated code
- Put the commit message's TODO on long lines into code, too
- Drop _gen_json()'s useless default argument [Eric]
- Fix name of simple unions' implicit tag [Eric]
- Adjust white-space to make pylint happier
- Fix indentation in test-qmp-input-strict.c [Eric]
* PATCH 31:
- Commit message tweak
- Update docs/qapi-code-gen.txt
- R-by dropped to make sure the doc gets reviewed.
* PATCH 32:
- New option -u to suppress the type name hiding (debugging aid) Easier than generating comments. We can still generate them later.
- Update docs/qapi-code-gen.txt and qapi/introspect.json comments
- Use %d to format integers [Eric]
- R-by dropped to make sure the doc gets reviewed.
* This change log is again long, but compared to last time, the changes are rather local, and most of them are to the last three patches. I think we're converging.

v3:
* Straightforward rebase, R-bys retained unless noted otherwise. Due to my widespread changes, you may prefer to review from scratch anyway.
* Old PATCH 01-19 split off into separate non-RFC series "[PATCH 00/26] qapi: Another round of fixes and cleanups"
* PATCH 02:
- Commit message spelling fix [Eric]
- Clarify patch's non-effect in commit message.
- Assert enumeration constants are unique.
- Assert object's base is an object without variants [Eric].
- Drop a few superfluous check() recursions.
- Use arg_type, ret_type instead of args, ret for commands [Eric], and arg_type instead of data for events.
- Lower simple variants to flat ones as described on qapi-code-gen.txt. Replace QAPISchemaObjectType's .flat by .simple_union_type(), for use by pre-existing code-generation warts.
- _make_implicit_object_type() idempotent (previous item needs

    that).

- Clean up how empty command argument dictionaries work [Eric].
- Same for event.
- No R-by to drop.
* PATCH 03:
- Fix QAPISchemaEnumType.c_null() for empty enums [Eric].
- R-by dropped.
* PATCH 04:
- Use arg_type, ret_type instead of args, ret for commands [Eric], and arg_type instead of data for events.
* PATCH 05:
- Use arg_type, ret_type instead of args, ret for commands [Eric], and arg_type instead of data for events.
- .flat is gone, drop code to print it.
- Expected test output changed due to revision of PATCH 02.
* PATCH 06-09,16-17: New, replacing old PATCH 25 "qapi: Make generators work on sorted schema expressions".
* PATCH 10:
- Update qapi-code-gen.txt with the patch's changes.
- Earlier patches ripple into gen_union().
- Nicer comment on -b [Eric].
- Objects with both members and unions aren't implemented. Add an assertion [Eric].
- No R-by to drop.
* PATCH 11:
- Earlier patches ripple into gen_visit_union().
- Rename guard QAPI_VISIT_BUILTIN_VISITOR_DECL to QAPI_VISIT_BUILTIN.
- Nicer comment on -b [Eric].
- Objects with both members and unions aren't implemented. Add an assertion [Eric].
- No R-by to drop.
* PATCH 12:
- Use arg_type, ret_type instead of args, ret [Eric].
- Zap .regy unconditionally in visit_end (squashed from PATCH 30).
* PATCH 13:
- Commit message fixed and improved [Eric].
- Update qapi-code-gen.txt with the patch's changes.
* PATCH 15
- Copyright notice pasto fixed [Eric].
- Minimize patch by delaying more renames to PATCH 19.
- Use arg_type instead of data.
- Free .event_names in visit_end(), moved here from PATCH 30.
* PATCH 18:
- Moved; was right after PATCH 12 before.
- One hunk dropped due to "[PATCH 25/26] qapi-commands: Drop useless initialization".
* PATCH 19:
- Use arg_type, ret_type instead of args, ret for commands [Eric], and arg_type instead of data or params for events.
- Consistently use single quotes for strings holding generated code

    [Eric].

- Generate just 'FOO' instead of 'struct FOO' when possible [Eric].
- Earlier patches ripple into gen_union() and gen_visit_union().
- R-by dropped (death by a thousand cuts).
* PATCH 20:
- Straightforward conflicts due to new "[PATCH 14/26] qapi: Document that input visitor semantics are prone to leaks" and "[PATCH 15/26] qapi: Document shortcoming with union 'data' branch".
* PATCH 21:
- Use arg_type, ret_type instead of args, ret [Eric].
- Straightforward conflict due to "[PATCH 25/26] qapi-commands: Drop useless initialization".
* PATCH 22:
- Rename in qmp.c's comment, too [Eric].
* PATCH 23:
- Use arg_type, ret_type instead of args, ret for commands [Eric], and arg_type instead of data for events.
- Consistently use single quotes for strings holding generated code

    [Eric].

* PATCH 24:
- Update qapi-code-gen.txt with the patch's changes.
- Use arg_type, ret_type instead of args, ret [Eric].
- Free .visited_ret_types in visit_end(), moved here from PATCH 30.
* PATCH 26:
- Indentation fix [Eric].
- Some test case updates moved here from PATCH 29: move positive test case from args-returns-any.json to qapi-schema-test.json (with new qmp_guest_sync() dummy in test-qmp-commands.c), move negative test case to args-any.json, drop bogus test case.
- Rename flat-union-base-star.* to flat-union-base-any.*.
- Test QMP input visitor's visit_type_any() doesn't add extra members [Eric].
* PATCH 27:
- Drop a TODO obsoleted by the patch [Eric].
* PATCH 28:
- Old PATCH 42+43 squashed together [Eric].
* PATCH 29:
- Documentation update on 'gen' was botched, rephrase [Eric].
- Some test case updates move to PATCH 26.
* PATCH 30:
- success-response isn't implementation detail, add TODO and fix commit message [Eric].
- Cover new qapi-introspect.py in qapi-code-gen.txt [Eric].
- Clarify qmp_query_schema()'s comment.
- Fix introspect.json's comment on default values [Eric].
- Replace SchemaInfoObjectVariant member 'members' by 'type' [Eric].
- New SchemaInfoAlternateMember [Eric].
- Use arg_type, ret_type instead of args, ret for commands [Eric], and arg_type instead of data for events.
- Rewrite actual code generation: instead of generating a list of JSON strings directly, first generate a list of dicts, then format them as JSON.
- Fix to use " instead of ' for JSON strings [Eric].
- Make the generated array const [Eric].
- Fix accidental drop of a line of code [Eric].
- Fix file name in tests/.gitignore [Eric].
- Revision of PATCH 02 ripples into test output diffs.
- Move new test from test-qmp-input-visitor.c to test-qmp-input-strict.c to make it catch extra members.
- R-by dropped.
* PATCH 32:
- Update qapi-code-gen.txt with the patch's changes.
- Rewrite actual code generation: instead of generating a list of JSON strings directly, first generate a list of dicts, then format them as JSON.
- Use plain integers as type names [Eric].
- Don't hide names of builtin types [Eric].
- Put a TODO on type name comments in commit message.

Markus Armbruster (32):
qapi: Rename class QAPISchema to QAPISchemaParser
qapi: New QAPISchema intermediate reperesentation
qapi: QAPISchema code generation helper methods
qapi: New QAPISchemaVisitor
tests/qapi-schema: Convert test harness to QAPISchemaVisitor
qapi: Split up some typedefs to ease review
qapi: Generate comments to simplify splitting for review Revert "qapi: Generate comments to simplify splitting for review" Revert "qapi: Split up some typedefs to ease review"
qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
qapi-commands: Convert to QAPISchemaVisitor
qapi: De-duplicate enum code generation
qapi-event: Eliminate global variable event_enum_value
qapi-event: Convert to QAPISchemaVisitor, fixing data with base
qapi: Generate comments to simplify splitting for review Revert "qapi: Generate comments to simplify splitting for review"
qapi: Replace dirty is_c_ptr() by method c_null()
qapi: Clean up after recent conversions to QAPISchemaVisitor
qapi-visit: Rearrange code a bit
qapi-commands: Rearrange code
qapi: Rename qmp_marshal_input_FOO() to qmp_marshal_FOO()
qapi: De-duplicate parameter list generation
qapi-commands: De-duplicate output marshaling functions
qapi: Improve built-in type documentation
qapi: Introduce a first class 'any' type
qom: Don't use 'gen': false for qom-get, qom-set, object-add
qapi-schema: Fix up misleading specification of netdev_add
qapi: Pseudo-type '**' is now unused, drop it
qapi: New QMP command query-schema for QMP schema introspection
qapi-introspect: Map all integer types to 'int'
qapi-introspect: Hide type names

 .gitignore                                         |   1 +
 Makefile                                           |   9 +-
 Makefile.objs                                      |   4 +-
 docs/qapi-code-gen.txt                             | 349 ++++++++---
 docs/writing-qmp-commands.txt                      |   8 +-
 include/monitor/monitor.h                          |   3 -
 include/qapi/visitor-impl.h                        |   2 +
 include/qapi/visitor.h                             |   1 +
 monitor.c                                          |  17 +-
 qapi-schema.json                                   |  25 +-
 qapi/introspect.json                               | 266 +++++++++
 qapi/qapi-dealloc-visitor.c                        |   9 +
 qapi/qapi-visit-core.c                             |   6 +
 qapi/qmp-input-visitor.c                           |  11 +
 qapi/qmp-output-visitor.c                          |   9 +
 qmp-commands.hx                                    | 267 +++++----
 qmp.c                                              |  27 +-
 scripts/qapi-commands.py                           | 272 ++++-----
 scripts/qapi-event.py                              | 239 +++-----
 scripts/qapi-introspect.py                         | 210 +++++++
 scripts/qapi-types.py                              | 355 +++++-------
 scripts/qapi-visit.py                              | 350 +++++------
 scripts/qapi.py                                    | 637 +++++++++++++++++----
 tests/.gitignore                                   |   1 +
 tests/Makefile                                     |  15 +-
 tests/qapi-schema/alternate-good.out               |  16 +-
 tests/qapi-schema/args-any.err                     |   1 +
 .../{type-bypass-no-gen.exit => args-any.exit}     |   0
 tests/qapi-schema/args-any.json                    |   2 +
 .../qapi-schema/{type-bypass.err => args-any.out}  |   0
 tests/qapi-schema/args-member-array.out            |  14 +-
 tests/qapi-schema/comments.out                     |   5 +-
 tests/qapi-schema/empty.out                        |   4 +-
 tests/qapi-schema/enum-empty.out                   |   5 +-
 tests/qapi-schema/event-case.out                   |   5 +-
 tests/qapi-schema/flat-union-base-any.err          |   1 +
 ...ion-base-star.exit => flat-union-base-any.exit} |   0
 ...ion-base-star.json => flat-union-base-any.json} |   2 +-
 ...e-bypass-no-gen.out => flat-union-base-any.out} |   0
 tests/qapi-schema/flat-union-base-star.err         |   1 -
 tests/qapi-schema/flat-union-base-star.out         |   0
 tests/qapi-schema/flat-union-reverse-define.out    |  22 +-
 tests/qapi-schema/ident-with-escape.out            |   8 +-
 tests/qapi-schema/include-relpath.out              |   5 +-
 tests/qapi-schema/include-repetition.out           |   5 +-
 tests/qapi-schema/include-simple.out               |   5 +-
 tests/qapi-schema/indented-expr.out                |   8 +-
 tests/qapi-schema/qapi-schema-test.json            |  11 +-
 tests/qapi-schema/qapi-schema-test.out             | 222 +++++--
 tests/qapi-schema/returns-int.out                  |   6 +-
 tests/qapi-schema/test-qapi.py                     |  38 +-
 tests/qapi-schema/type-bypass-no-gen.err           |   1 -
 tests/qapi-schema/type-bypass-no-gen.json          |   2 -
 tests/qapi-schema/type-bypass.exit                 |   1 -
 tests/qapi-schema/type-bypass.json                 |   2 -
 tests/qapi-schema/type-bypass.out                  |   3 -
 tests/test-qmp-commands.c                          |   5 +
 tests/test-qmp-input-strict.c                      |  59 +-
 tests/test-qmp-input-visitor.c                     |  49 +-
 tests/test-qmp-output-visitor.c                    |  53 ++
 60 files changed, 2462 insertions(+), 1192 deletions(-)
 create mode 100644 qapi/introspect.json
 create mode 100644 scripts/qapi-introspect.py
 create mode 100644 tests/qapi-schema/args-any.err
 rename tests/qapi-schema/{type-bypass-no-gen.exit => args-any.exit} (100%)
 create mode 100644 tests/qapi-schema/args-any.json
 rename tests/qapi-schema/{type-bypass.err => args-any.out} (100%)
 create mode 100644 tests/qapi-schema/flat-union-base-any.err
 rename tests/qapi-schema/{flat-union-base-star.exit => flat-union-base-any.exit} (100%)
 rename tests/qapi-schema/{flat-union-base-star.json => flat-union-base-any.json} (95%)
 rename tests/qapi-schema/{type-bypass-no-gen.out => flat-union-base-any.out} (100%)
 delete mode 100644 tests/qapi-schema/flat-union-base-star.err
 delete mode 100644 tests/qapi-schema/flat-union-base-star.out
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.err
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.json
 delete mode 100644 tests/qapi-schema/type-bypass.exit
 delete mode 100644 tests/qapi-schema/type-bypass.json
 delete mode 100644 tests/qapi-schema/type-bypass.out
Signature · 1 line
2.4.3
#1 lore raw

RFC PATCH v4 1/32 qapi: Rename class QAPISchema to QAPISchemaParser

I want to name a new class QAPISchema.

While there, make it a new-style class.

---
 scripts/qapi.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi.py +4 -4
index 817d824..a0165dd 100644
@@ -103,7 +103,7 @@ class QAPIExprError(Exception):
         return error_path(self.info['parent']) + \
             "%s:%d: %s" % (self.info['file'], self.info['line'], self.msg)
 
-class QAPISchema:
+class QAPISchemaParser(object):
 
     def __init__(self, fp, previously_included = [], incl_info = None):
         abs_fname = os.path.abspath(fp.name)
@@ -149,8 +149,8 @@ class QAPISchema:
                 except IOError, e:
                     raise QAPIExprError(expr_info,
                                         '%s: %s' % (e.strerror, include))
-                exprs_include = QAPISchema(fobj, previously_included,
-                                           expr_info)
+                exprs_include = QAPISchemaParser(fobj, previously_included,
+                                                 expr_info)
                 self.exprs.extend(exprs_include.exprs)
             else:
                 expr_elem = {'expr': expr,
@@ -751,7 +751,7 @@ def check_exprs(exprs):
 
 def parse_schema(fname):
     try:
-        schema = QAPISchema(open(fname, "r"))
+        schema = QAPISchemaParser(open(fname, "r"))
         return check_exprs(schema.exprs)
     except (QAPISchemaError, QAPIExprError), e:
         print >>sys.stderr, e
Signature · 1 line
2.4.3
Reviewed-by on v4 1/32
#2 lore raw

Re: [Qemu-devel] [PATCH RFC v4 01/32] qapi: Rename class QAPISchema to QAPISchemaParser

Quoted 6 lines · Markus Armbruster

I want to name a new class QAPISchema.

While there, make it a new-style class.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#3 lore raw

RFC PATCH v4 2/32 qapi: New QAPISchema intermediate reperesentation

In reply to Markus Armbruster

The QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaries) on the side.

They have clearly outgrown these simple data structures. There's lots of rummaging around in dictionaries, and information is recomputed on the fly. For the work I'm going to do, I want more clearly defined and more convenient interfaces.

Going forward, I also want less coupling between the back-ends and the syntax tree, to make messing with the syntax easier.

Create a bunch of classes to represent QAPI schemata.

Have the QAPISchema initializer call the parser, then walk the syntax tree to create the new internal representation, and finally perform semantic analysis.

Shortcut: the semantic analysis still relies on existing check_exprs() to do the actual semantic checking. All this code needs to move into the classes. Mark as TODO.

Catching name collisions in generated code would be nice. Mark as TODO.

We generate array types eagerly, even though most of them aren't used. Mark as TODO.

Nothing uses the new intermediate representation just yet, thus no change to generated files.

---
 scripts/qapi-commands.py       |   2 +-
 scripts/qapi-event.py          |   2 +-
 scripts/qapi-types.py          |   2 +-
 scripts/qapi-visit.py          |   2 +-
 scripts/qapi.py                | 360 +++++++++++++++++++++++++++++++++++++++--
 tests/qapi-schema/test-qapi.py |   2 +-
 6 files changed, 356 insertions(+), 14 deletions(-)
scripts/qapi-commands.py +1 -1
index 890ce5d..12bdc4c 100644
@@ -272,7 +272,7 @@ for o, a in opts:
     if o in ("-m", "--middle"):
         middle_mode = True
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 commands = filter(lambda expr: expr.has_key('command'), exprs)
 commands = filter(lambda expr: not expr.has_key('gen'), commands)
 
scripts/qapi-event.py +1 -1
index 7f238df..aec2d32 100644
@@ -263,7 +263,7 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 
 event_enum_name = c_name(prefix + "QAPIEvent", protect=False)
 event_enum_values = []
scripts/qapi-types.py +1 -1
index f2428f3..d162ca2 100644
@@ -336,7 +336,7 @@ fdecl.write(mcgen('''
 #include <stdint.h>
 '''))
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 
 fdecl.write(guardstart("QAPI_TYPES_BUILTIN_STRUCT_DECL"))
 for typename in builtin_types.keys():
scripts/qapi-visit.py +1 -1
index 3cd662b..c493964 100644
@@ -446,7 +446,7 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = parse_schema(input_file)
+exprs = QAPISchema(input_file).get_exprs()
 
 # to avoid header dependency hell, we always generate declarations
 # for built-in types in our header files and simply guard them
scripts/qapi.py +351 -9

372 lines not on this page · raw message

tests/qapi-schema/test-qapi.py +1 -1
index 634ef2d..461c713 100644
@@ -16,7 +16,7 @@ import os
 import sys
 
 try:
-    exprs = parse_schema(sys.argv[1])
+    exprs = QAPISchema(sys.argv[1]).get_exprs()
 except SystemExit:
     raise
 
Signature · 1 line
2.4.3
Reviewed-by on v4 2/32
#4 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02/32] qapi: New QAPISchema intermediate reperesentation

Quoted 24 lines · Markus Armbruster

The QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaries) on the side.

They have clearly outgrown these simple data structures. There's lots of rummaging around in dictionaries, and information is recomputed on the fly. For the work I'm going to do, I want more clearly defined and more convenient interfaces.

Going forward, I also want less coupling between the back-ends and the syntax tree, to make messing with the syntax easier.

Create a bunch of classes to represent QAPI schemata.

Have the QAPISchema initializer call the parser, then walk the syntax tree to create the new internal representation, and finally perform semantic analysis.

Shortcut: the semantic analysis still relies on existing check_exprs() to do the actual semantic checking. All this code needs to move into the classes. Mark as TODO.

Catching name collisions in generated code would be nice. Mark as TODO.

Quoted 8 lines

We generate array types eagerly, even though most of them aren't used. Mark as TODO.

Nothing uses the new intermediate representation just yet, thus no change to generated files.

A few comments inline, but nothing worth blocking on, just style.

Quoted 31 lines
scripts/qapi.py +13 -9
index a0165dd..c42bea1 100644
@@ -747,15 +749,355 @@ def check_exprs(exprs):
         else:
             assert False, 'unexpected meta type'
 
-    return map(lambda expr_elem: expr_elem['expr'], exprs)
-
-def parse_schema(fname):
-    try:
-        schema = QAPISchemaParser(open(fname, "r"))
-        return check_exprs(schema.exprs)
-    except (QAPISchemaError, QAPIExprError), e:
-        print >>sys.stderr, e
-        exit(1)
+    return exprs
+
+#
+# Schema compiler frontend
+#
+
+class QAPISchemaEntity(object):
+    def __init__(self, name, info):
+        assert isinstance(name, str)
+        self.name = name
+        self.info = info
+    def check(self, schema):
+        pass
 
Quoted 1 line

From a style POV, I'd recommend having a single blank

line before each new function def, to make things more readable. I won't nack the patch for that, but if you have need to rebase & post a new version of this series I think it'd be nice to add the the extra blank lines when defining all these classes.

Python has a nice tool called pep8 which can apply a configurable bunch of style checks, it is probably worth someone wiring it up to make check in QEMU, since we have an increasing amount of python code. An exercise for a motivated reader.....

Quoted 152 lines
scripts/qapi.py +152 -0
+
+class QAPISchemaType(QAPISchemaEntity):
+    pass
+
+class QAPISchemaBuiltinType(QAPISchemaType):
+    def __init__(self, name):
+        QAPISchemaType.__init__(self, name, None)
+
+class QAPISchemaEnumType(QAPISchemaType):
+    def __init__(self, name, info, values):
+        QAPISchemaType.__init__(self, name, info)
+        for v in values:
+            assert isinstance(v, str)
+        self.values = values
+    def check(self, schema):
+        assert len(set(self.values)) == len(self.values)
+
+class QAPISchemaArrayType(QAPISchemaType):
+    def __init__(self, name, info, element_type):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(element_type, str)
+        self.element_type_name = element_type
+        self.element_type = None
+    def check(self, schema):
+        self.element_type = schema.lookup_type(self.element_type_name)
+        assert self.element_type
+
+class QAPISchemaObjectType(QAPISchemaType):
+    def __init__(self, name, info, base, local_members, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert base == None or isinstance(base, str)
+        for m in local_members:
+            assert isinstance(m, QAPISchemaObjectTypeMember)
+        assert variants == None \
+            or isinstance(variants, QAPISchemaObjectTypeVariants)
+        self.base_name = base
+        self.base = None
+        self.local_members = local_members
+        self.variants = variants
+        self.members = None
+    def check(self, schema):
+        assert self.members != False    # not running in cycles
+        if self.members:
+            return
+        self.members = False            # mark as being checked
+        if self.base_name:
+            self.base = schema.lookup_type(self.base_name)
+            assert isinstance(self.base, QAPISchemaObjectType)
+            assert not self.base.variants # not implemented
+            self.base.check(schema)
+            members = list(self.base.members)
+        else:
+            members = []
+        seen = {}
+        for m in members:
+            seen[m.name] = m
+        for m in self.local_members:
+            m.check(schema, members, seen)
+        if self.variants:
+            self.variants.check(schema, members, seen)
+        self.members = members
+
+class QAPISchemaObjectTypeMember(object):
+    def __init__(self, name, typ, optional):
+        assert isinstance(name, str)
+        assert isinstance(typ, str)
+        assert isinstance(optional, bool)
+        self.name = name
+        self.type_name = typ
+        self.type = None
+        self.optional = optional
+    def check(self, schema, all_members, seen):
+        assert self.name not in seen
+        self.type = schema.lookup_type(self.type_name)
+        assert self.type
+        all_members.append(self)
+        seen[self.name] = self
+
+class QAPISchemaObjectTypeVariants(object):
+    def __init__(self, tag_name, tag_enum, variants):
+        assert tag_name == None or isinstance(tag_name, str)
+        assert tag_enum == None or isinstance(tag_enum, str)
+        for v in variants:
+            assert isinstance(v, QAPISchemaObjectTypeVariant)
+        self.tag_name = tag_name
+        if tag_name:
+            assert not tag_enum
+            self.tag_member = None
+        else:
+            self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum,
+                                                         False)
+        self.variants = variants
+    def check(self, schema, members, seen):
+        if self.tag_name:
+            self.tag_member = seen[self.tag_name]
+        else:
+            self.tag_member.check(schema, members, seen)
+        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        for v in self.variants:
+            vseen = dict(seen)
+            v.check(schema, self.tag_member.type, vseen)
+
+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
+    def __init__(self, name, typ):
+        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
+    def check(self, schema, tag_type, seen):
+        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        assert self.name in tag_type.values
+    # TODO try to get rid of .simple_union_type()
+    def simple_union_type(self):
+        if isinstance(self.type, QAPISchemaObjectType) and not self.type.info:
+            assert len(self.type.members) == 1
+            assert not self.type.variants
+            return self.type.members[0].type
+        return None
+
+class QAPISchemaAlternateType(QAPISchemaType):
+    def __init__(self, name, info, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(variants, QAPISchemaObjectTypeVariants)
+        assert not variants.tag_name
+        self.variants = variants
+    def check(self, schema):
+        self.variants.check(schema, [], {})
+
+class QAPISchemaCommand(QAPISchemaEntity):
+    def __init__(self, name, info, arg_type, ret_type, gen, success_response):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not arg_type or isinstance(arg_type, str)
+        assert not ret_type or isinstance(ret_type, str)
+        self.arg_type_name = arg_type
+        self.arg_type = None
+        self.ret_type_name = ret_type
+        self.ret_type = None
+        self.gen = gen
+        self.success_response = success_response
+    def check(self, schema):
+        if self.arg_type_name:
+            self.arg_type = schema.lookup_type(self.arg_type_name)
+            assert isinstance(self.arg_type, QAPISchemaObjectType)
+            assert not self.arg_type.variants # not implemented
+        if self.ret_type_name:
+            self.ret_type = schema.lookup_type(self.ret_type_name)
+            assert isinstance(self.ret_type, QAPISchemaType)
+
+class QAPISchemaEvent(QAPISchemaEntity):
+    def __init__(self, name, info, arg_type):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not arg_type or isinstance(arg_type, str)
+        self.arg_type_name = arg_type
+        self.arg_type = None
+    def check(self, schema):

scripts/qapi.py def check_exprs(exprs): · line 926

Quoted 10 lines
scripts/qapi.py +10 -0
+        if self.arg_type_name:
+            self.arg_type = schema.lookup_type(self.arg_type_name)
+            assert isinstance(self.arg_type, QAPISchemaObjectType)
+            assert not self.arg_type.variants # not implemented
+
+class QAPISchema(object):
+    def __init__(self, fname):
+        try:
+            self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
+        except (QAPISchemaError, QAPIExprError), err:

Preferred syntax for exceptions these days is

except Foo as err

instead of

except Foo, err

since the former is portable to Python3, should we need to consider that in future. Again, not a show stopper.

scripts/qapi.py def check_exprs(exprs): · line 928

Quoted 2 lines
scripts/qapi.py +2 -0
+            print >>sys.stderr, err
+            exit(1)

I think it is probably better practice to not have classes print to stderr / exit. I'd just let the error propagate to the caller and have the top level script catch exceptions, print error message and exit. Again since this is just style issue I won't nack it for that but something to consider if needing to re-spin this series again.

Quoted 59 lines
scripts/qapi.py +59 -0
+        self.entity_dict = {}
+        self._def_predefineds()
+        self._def_exprs()
+        self.check()
+
+    def get_exprs(self):
+        return [expr_elem['expr'] for expr_elem in self.exprs]
+
+    def _def_entity(self, ent):
+        assert ent.name not in self.entity_dict
+        self.entity_dict[ent.name] = ent
+
+    def lookup_entity(self, name, typ=None):
+        ent = self.entity_dict.get(name)
+        if typ and not isinstance(ent, typ):
+            return None
+        return ent
+
+    def lookup_type(self, name):
+        return self.lookup_entity(name, QAPISchemaType)
+
+    def _def_builtin_type(self, name):
+        self._def_entity(QAPISchemaBuiltinType(name))
+        if name != '**':
+            self._make_array_type(name) # TODO really needed?
+
+    def _def_predefineds(self):
+        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
+                  'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
+            self._def_builtin_type(t)
+
+    def _make_implicit_enum_type(self, name, values):
+        name = name + 'Kind'
+        self._def_entity(QAPISchemaEnumType(name, None, values))
+        return name
+
+    def _make_array_type(self, element_type):
+        name = element_type + 'List'
+        if not self.lookup_type(name):
+            self._def_entity(QAPISchemaArrayType(name, None, element_type))
+        return name
+
+    def _make_implicit_object_type(self, name, role, members):
+        if not members:
+            return None
+        name = ':obj-%s-%s' % (name, role)
+        if not self.lookup_entity(name, QAPISchemaObjectType):
+            self._def_entity(QAPISchemaObjectType(name, None, None,
+                                                  members, None))
+        return name
+
+    def _def_enum_type(self, expr, info):
+        name = expr['enum']
+        data = expr['data']
+        self._def_entity(QAPISchemaEnumType(name, info, data))
+        self._make_array_type(name) # TODO really needed?
+
+    def _make_member(self, name, typ):
+        optional = False

scripts/qapi.py def check_exprs(exprs): · line 997

Quoted 10 lines
scripts/qapi.py +10 -0
+        if name.startswith('*'):
+            name = name[1:]
+            optional = True
+        if isinstance(typ, list):
+            assert len(typ) == 1
+            typ = self._make_array_type(typ[0])
+        return QAPISchemaObjectTypeMember(name, typ, optional)
+
+    def _make_members(self, data):
+        return [self._make_member(key, value) 

Trailing whitespace at end of this line

Quoted 111 lines
scripts/qapi.py +103 -0
+                for (key, value) in data.iteritems()]
+
+    def _def_struct_type(self, expr, info):
+        name = expr['struct']
+        base = expr.get('base')
+        data = expr['data']
+        self._def_entity(QAPISchemaObjectType(name, info, base,
+                                              self._make_members(data),
+                                              None))
+        self._make_array_type(name) # TODO really needed?
+
+    def _make_variant(self, case, typ):
+        return QAPISchemaObjectTypeVariant(case, typ)
+
+    def _make_simple_variant(self, case, typ):
+        if isinstance(typ, list):
+            assert len(typ) == 1
+            typ = self._make_array_type(typ[0])
+        typ = self._make_implicit_object_type(typ, 'wrapper',
+                                              [self._make_member('data', typ)])
+        return QAPISchemaObjectTypeVariant(case, typ)
+
+    def _make_tag_enum(self, type_name, variants):
+        return self._make_implicit_enum_type(type_name,
+                                             [v.name for v in variants])
+
+    def _def_union_type(self, expr, info):
+        name = expr['union']
+        data = expr['data']
+        base = expr.get('base')
+        tag_name = expr.get('discriminator')
+        tag_enum = None
+        if tag_name:
+            variants = [self._make_variant(key, value)
+                        for (key, value) in data.iteritems()]
+        else:
+            variants = [self._make_simple_variant(key, value)
+                        for (key, value) in data.iteritems()]
+            tag_enum = self._make_tag_enum(name, variants)
+        self._def_entity(QAPISchemaObjectType(name, info, base,
+                                    self._make_members(OrderedDict()),
+                                    QAPISchemaObjectTypeVariants(tag_name,
+                                                                 tag_enum,
+                                                                 variants)))
+        self._make_array_type(name) # TODO really needed?
+
+    def _def_alternate_type(self, expr, info):
+        name = expr['alternate']
+        data = expr['data']
+        variants = [self._make_variant(key, value)
+                    for (key, value) in data.iteritems()]
+        tag_enum = self._make_tag_enum(name, variants)
+        self._def_entity(QAPISchemaAlternateType(name, info,
+                                    QAPISchemaObjectTypeVariants(None,
+                                                                 tag_enum,
+                                                                 variants)))
+        self._make_array_type(name) # TODO really needed?
+
+    def _def_command(self, expr, info):
+        name = expr['command']
+        data = expr.get('data')
+        rets = expr.get('returns')
+        gen = expr.get('gen', True)
+        success_response = expr.get('success-response', True)
+        if isinstance(data, OrderedDict):
+            data = self._make_implicit_object_type(name, 'arg',
+                                                   self._make_members(data))
+        if isinstance(rets, list):
+            assert len(rets) == 1
+            rets = self._make_array_type(rets[0])
+        self._def_entity(QAPISchemaCommand(name, info, data, rets, gen,
+                                           success_response))
+
+    def _def_event(self, expr, info):
+        name = expr['event']
+        data = expr.get('data')
+        if isinstance(data, OrderedDict):
+            data = self._make_implicit_object_type(name, 'arg',
+                                                   self._make_members(data))
+        self._def_entity(QAPISchemaEvent(name, info, data))
+
+    def _def_exprs(self):
+        for expr_elem in self.exprs:
+            expr = expr_elem['expr']
+            info = expr_elem['info']
+            if 'enum' in expr:
+                self._def_enum_type(expr, info)
+            elif 'struct' in expr:
+                self._def_struct_type(expr, info)
+            elif 'union' in expr:
+                self._def_union_type(expr, info)
+            elif 'alternate' in expr:
+                self._def_alternate_type(expr, info)
+            elif 'command' in expr:
+                self._def_command(expr, info)
+            elif 'event' in expr:
+                self._def_event(expr, info)
+            else:
+                assert False
+
+    def check(self):
+        for ent in self.entity_dict.values():
+            ent.check(self)
 
 #
 # Code generation helpers
tests/qapi-schema/test-qapi.py +0 -0
index 634ef2d..461c713 100644

tests/qapi-schema/test-qapi.py import os · line 21

Quoted 7 lines
tests/qapi-schema/test-qapi.py +1 -1
@@ -16,7 +16,7 @@ import os
 import sys
 
 try:
-    exprs = parse_schema(sys.argv[1])
+    exprs = QAPISchema(sys.argv[1]).get_exprs()
 except SystemExit:
 

Following on earlier comment, this could have a block

except Exception as e:
print >>sys.stderr, e
sys.exit(1)

instead of having the object constructor call exit itself.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#5 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02/32] qapi: New QAPISchema intermediate reperesentation

Quoted 68 lines · Daniel P. Berrange
Quoted 24 lines · Markus Armbruster

The QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaries) on the side.

They have clearly outgrown these simple data structures. There's lots of rummaging around in dictionaries, and information is recomputed on the fly. For the work I'm going to do, I want more clearly defined and more convenient interfaces.

Going forward, I also want less coupling between the back-ends and the syntax tree, to make messing with the syntax easier.

Create a bunch of classes to represent QAPI schemata.

Have the QAPISchema initializer call the parser, then walk the syntax tree to create the new internal representation, and finally perform semantic analysis.

Shortcut: the semantic analysis still relies on existing check_exprs() to do the actual semantic checking. All this code needs to move into the classes. Mark as TODO.

Catching name collisions in generated code would be nice. Mark as TODO.

Quoted 8 lines

We generate array types eagerly, even though most of them aren't used. Mark as TODO.

Nothing uses the new intermediate representation just yet, thus no change to generated files.

A few comments inline, but nothing worth blocking on, just style.

Quoted 29 lines
scripts/qapi.py +12 -9
index a0165dd..c42bea1 100644
@@ -747,15 +749,355 @@ def check_exprs(exprs):
         else:
             assert False, 'unexpected meta type'
 
-    return map(lambda expr_elem: expr_elem['expr'], exprs)
-
-def parse_schema(fname):
-    try:
-        schema = QAPISchemaParser(open(fname, "r"))
-        return check_exprs(schema.exprs)
-    except (QAPISchemaError, QAPIExprError), e:
-        print >>sys.stderr, e
-        exit(1)
+    return exprs
+
+#
+# Schema compiler frontend
+#
+
+class QAPISchemaEntity(object):
+    def __init__(self, name, info):
+        assert isinstance(name, str)
+        self.name = name
+        self.info = info
+    def check(self, schema):
Quoted 8 lines

scripts/qapi.py def check_exprs(exprs): · line 764

Quoted 1 line
scripts/qapi.py +1 -0
+        pass

From a style POV, I'd recommend having a single blank line before each new function def, to make things more readable. I won't nack the patch for that, but if you have need to rebase & post a new version of this series I think it'd be nice to add the the extra blank lines when defining all these classes.

I have such blank lines in the more complex stuff. Didn't feel like it in trivial classes like this, but if others want the blank lines, I'm happy to add them.

Quoted 5 lines

Python has a nice tool called pep8 which can apply a configurable bunch of style checks, it is probably worth someone wiring it up to make check in QEMU, since we have an increasing amount of python code. An exercise for a motivated reader.....

I've been using pylint (and dear-oh-dear how much pre-existing lint it finds!), wasn't aware of pep8, can throw it in.

Automatic checking would be nice, but our current code is probably too untidy for that. Besides, I don't have the time and energy to set it up now.

pylint is reasonably happy with the qapi-*.py now. I hope the follow-up work on qapi.py will get it into the same state.

Quoted 167 lines
Quoted 154 lines
scripts/qapi.py +111 -0
+
+class QAPISchemaType(QAPISchemaEntity):
+    pass
+
+class QAPISchemaBuiltinType(QAPISchemaType):
+    def __init__(self, name):
+        QAPISchemaType.__init__(self, name, None)
+
+class QAPISchemaEnumType(QAPISchemaType):
+    def __init__(self, name, info, values):
+        QAPISchemaType.__init__(self, name, info)
+        for v in values:
+            assert isinstance(v, str)
+        self.values = values
+    def check(self, schema):
+        assert len(set(self.values)) == len(self.values)
+
+class QAPISchemaArrayType(QAPISchemaType):
+    def __init__(self, name, info, element_type):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(element_type, str)
+        self.element_type_name = element_type
+        self.element_type = None
+    def check(self, schema):
+        self.element_type = schema.lookup_type(self.element_type_name)
+        assert self.element_type
+
+class QAPISchemaObjectType(QAPISchemaType):
+    def __init__(self, name, info, base, local_members, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert base == None or isinstance(base, str)
+        for m in local_members:
+            assert isinstance(m, QAPISchemaObjectTypeMember)
+        assert variants == None \
+            or isinstance(variants, QAPISchemaObjectTypeVariants)
+        self.base_name = base
+        self.base = None
+        self.local_members = local_members
+        self.variants = variants
+        self.members = None
+    def check(self, schema):
+        assert self.members != False    # not running in cycles
+        if self.members:
+            return
+        self.members = False            # mark as being checked
+        if self.base_name:
+            self.base = schema.lookup_type(self.base_name)
+            assert isinstance(self.base, QAPISchemaObjectType)
+            assert not self.base.variants # not implemented
+            self.base.check(schema)
+            members = list(self.base.members)
+        else:
+            members = []
+        seen = {}
+        for m in members:
+            seen[m.name] = m
+        for m in self.local_members:
+            m.check(schema, members, seen)
+        if self.variants:
+            self.variants.check(schema, members, seen)
+        self.members = members
+
+class QAPISchemaObjectTypeMember(object):
+    def __init__(self, name, typ, optional):
+        assert isinstance(name, str)
+        assert isinstance(typ, str)
+        assert isinstance(optional, bool)
+        self.name = name
+        self.type_name = typ
+        self.type = None
+        self.optional = optional
+    def check(self, schema, all_members, seen):
+        assert self.name not in seen
+        self.type = schema.lookup_type(self.type_name)
+        assert self.type
+        all_members.append(self)
+        seen[self.name] = self
+
+class QAPISchemaObjectTypeVariants(object):
+    def __init__(self, tag_name, tag_enum, variants):
+        assert tag_name == None or isinstance(tag_name, str)
+        assert tag_enum == None or isinstance(tag_enum, str)
+        for v in variants:
+            assert isinstance(v, QAPISchemaObjectTypeVariant)
+        self.tag_name = tag_name
+        if tag_name:
+            assert not tag_enum
+            self.tag_member = None
+        else:
+            self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum,
+                                                         False)
+        self.variants = variants
+    def check(self, schema, members, seen):
+        if self.tag_name:
+            self.tag_member = seen[self.tag_name]
+        else:
+            self.tag_member.check(schema, members, seen)
+        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        for v in self.variants:
+            vseen = dict(seen)
+            v.check(schema, self.tag_member.type, vseen)
+
+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
+    def __init__(self, name, typ):
+        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
+    def check(self, schema, tag_type, seen):
+        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        assert self.name in tag_type.values
+    # TODO try to get rid of .simple_union_type()
+    def simple_union_type(self):
+ if isinstance(self.type, QAPISchemaObjectType) and not

self.type.info:

scripts/qapi.py +16 -0
+            assert len(self.type.members) == 1
+            assert not self.type.variants
+            return self.type.members[0].type
+        return None
+
+class QAPISchemaAlternateType(QAPISchemaType):
+    def __init__(self, name, info, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(variants, QAPISchemaObjectTypeVariants)
+        assert not variants.tag_name
+        self.variants = variants
+    def check(self, schema):
+        self.variants.check(schema, [], {})
+
+class QAPISchemaCommand(QAPISchemaEntity):
+ def __init__(self, name, info, arg_type, ret_type, gen,

success_response):

scripts/qapi.py +25 -0
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not arg_type or isinstance(arg_type, str)
+        assert not ret_type or isinstance(ret_type, str)
+        self.arg_type_name = arg_type
+        self.arg_type = None
+        self.ret_type_name = ret_type
+        self.ret_type = None
+        self.gen = gen
+        self.success_response = success_response
+    def check(self, schema):
+        if self.arg_type_name:
+            self.arg_type = schema.lookup_type(self.arg_type_name)
+            assert isinstance(self.arg_type, QAPISchemaObjectType)
+            assert not self.arg_type.variants # not implemented
+        if self.ret_type_name:
+            self.ret_type = schema.lookup_type(self.ret_type_name)
+            assert isinstance(self.ret_type, QAPISchemaType)
+
+class QAPISchemaEvent(QAPISchemaEntity):
+    def __init__(self, name, info, arg_type):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not arg_type or isinstance(arg_type, str)
+        self.arg_type_name = arg_type
+        self.arg_type = None
+    def check(self, schema):

scripts/qapi.py def check_exprs(exprs): · line 926

Quoted 10 lines
scripts/qapi.py +10 -0
+        if self.arg_type_name:
+            self.arg_type = schema.lookup_type(self.arg_type_name)
+            assert isinstance(self.arg_type, QAPISchemaObjectType)
+            assert not self.arg_type.variants # not implemented
+
+class QAPISchema(object):
+    def __init__(self, fname):
+        try:
+ self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
+        except (QAPISchemaError, QAPIExprError), err:

Preferred syntax for exceptions these days is

Quoted 8 lines

except Foo as err

instead of

except Foo, err

since the former is portable to Python3, should we need to consider that in future. Again, not a show stopper.

Does it work with Python 2.4? That's where we're currently stuck.

Quoted 9 lines

scripts/qapi.py def check_exprs(exprs): · line 928

Quoted 2 lines
scripts/qapi.py +2 -0
+            print >>sys.stderr, err
+            exit(1)

I think it is probably better practice to not have classes print to stderr / exit. I'd just let the error propagate to the caller and have the top level script catch exceptions, print error message and exit. Again since this is just style issue I won't nack it for that but something to consider if needing to re-spin this series again.

Quadruplicates the exception handling...

I think I'll leave this question open for now.

Quoted 63 lines
Quoted 63 lines
scripts/qapi.py +63 -0
+        self.entity_dict = {}
+        self._def_predefineds()
+        self._def_exprs()
+        self.check()
+
+    def get_exprs(self):
+        return [expr_elem['expr'] for expr_elem in self.exprs]
+
+    def _def_entity(self, ent):
+        assert ent.name not in self.entity_dict
+        self.entity_dict[ent.name] = ent
+
+    def lookup_entity(self, name, typ=None):
+        ent = self.entity_dict.get(name)
+        if typ and not isinstance(ent, typ):
+            return None
+        return ent
+
+    def lookup_type(self, name):
+        return self.lookup_entity(name, QAPISchemaType)
+
+    def _def_builtin_type(self, name):
+        self._def_entity(QAPISchemaBuiltinType(name))
+        if name != '**':
+            self._make_array_type(name) # TODO really needed?
+
+    def _def_predefineds(self):
+        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
+ 'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
+            self._def_builtin_type(t)
+
+    def _make_implicit_enum_type(self, name, values):
+        name = name + 'Kind'
+        self._def_entity(QAPISchemaEnumType(name, None, values))
+        return name
+
+    def _make_array_type(self, element_type):
+        name = element_type + 'List'
+        if not self.lookup_type(name):
+            self._def_entity(QAPISchemaArrayType(name, None, element_type))
+        return name
+
+    def _make_implicit_object_type(self, name, role, members):
+        if not members:
+            return None
+        name = ':obj-%s-%s' % (name, role)
+        if not self.lookup_entity(name, QAPISchemaObjectType):
+            self._def_entity(QAPISchemaObjectType(name, None, None,
+                                                  members, None))
+        return name
+
+    def _def_enum_type(self, expr, info):
+        name = expr['enum']
+        data = expr['data']
+        self._def_entity(QAPISchemaEnumType(name, info, data))
+        self._make_array_type(name) # TODO really needed?
+
+    def _make_member(self, name, typ):
+        optional = False
+        if name.startswith('*'):
+            name = name[1:]
+            optional = True
+        if isinstance(typ, list):
Quoted 8 lines

scripts/qapi.py def check_exprs(exprs): · line 997

Quoted 6 lines
scripts/qapi.py +6 -0
+            assert len(typ) == 1
+            typ = self._make_array_type(typ[0])
+        return QAPISchemaObjectTypeMember(name, typ, optional)
+
+    def _make_members(self, data):
+        return [self._make_member(key, value) 

Trailing whitespace at end of this line

Will fix, and search for more.

Quoted 121 lines
Quoted 111 lines
scripts/qapi.py +103 -0
+                for (key, value) in data.iteritems()]
+
+    def _def_struct_type(self, expr, info):
+        name = expr['struct']
+        base = expr.get('base')
+        data = expr['data']
+        self._def_entity(QAPISchemaObjectType(name, info, base,
+                                              self._make_members(data),
+                                              None))
+        self._make_array_type(name) # TODO really needed?
+
+    def _make_variant(self, case, typ):
+        return QAPISchemaObjectTypeVariant(case, typ)
+
+    def _make_simple_variant(self, case, typ):
+        if isinstance(typ, list):
+            assert len(typ) == 1
+            typ = self._make_array_type(typ[0])
+        typ = self._make_implicit_object_type(typ, 'wrapper',
+ [self._make_member('data', typ)])
+        return QAPISchemaObjectTypeVariant(case, typ)
+
+    def _make_tag_enum(self, type_name, variants):
+        return self._make_implicit_enum_type(type_name,
+                                             [v.name for v in variants])
+
+    def _def_union_type(self, expr, info):
+        name = expr['union']
+        data = expr['data']
+        base = expr.get('base')
+        tag_name = expr.get('discriminator')
+        tag_enum = None
+        if tag_name:
+            variants = [self._make_variant(key, value)
+                        for (key, value) in data.iteritems()]
+        else:
+            variants = [self._make_simple_variant(key, value)
+                        for (key, value) in data.iteritems()]
+            tag_enum = self._make_tag_enum(name, variants)
+        self._def_entity(QAPISchemaObjectType(name, info, base,
+                                    self._make_members(OrderedDict()),
+                                    QAPISchemaObjectTypeVariants(tag_name,
+                                                                 tag_enum,
+                                                                 variants)))
+        self._make_array_type(name) # TODO really needed?
+
+    def _def_alternate_type(self, expr, info):
+        name = expr['alternate']
+        data = expr['data']
+        variants = [self._make_variant(key, value)
+                    for (key, value) in data.iteritems()]
+        tag_enum = self._make_tag_enum(name, variants)
+        self._def_entity(QAPISchemaAlternateType(name, info,
+                                    QAPISchemaObjectTypeVariants(None,
+                                                                 tag_enum,
+                                                                 variants)))
+        self._make_array_type(name) # TODO really needed?
+
+    def _def_command(self, expr, info):
+        name = expr['command']
+        data = expr.get('data')
+        rets = expr.get('returns')
+        gen = expr.get('gen', True)
+        success_response = expr.get('success-response', True)
+        if isinstance(data, OrderedDict):
+            data = self._make_implicit_object_type(name, 'arg',
+                                                   self._make_members(data))
+        if isinstance(rets, list):
+            assert len(rets) == 1
+            rets = self._make_array_type(rets[0])
+        self._def_entity(QAPISchemaCommand(name, info, data, rets, gen,
+                                           success_response))
+
+    def _def_event(self, expr, info):
+        name = expr['event']
+        data = expr.get('data')
+        if isinstance(data, OrderedDict):
+            data = self._make_implicit_object_type(name, 'arg',
+                                                   self._make_members(data))
+        self._def_entity(QAPISchemaEvent(name, info, data))
+
+    def _def_exprs(self):
+        for expr_elem in self.exprs:
+            expr = expr_elem['expr']
+            info = expr_elem['info']
+            if 'enum' in expr:
+                self._def_enum_type(expr, info)
+            elif 'struct' in expr:
+                self._def_struct_type(expr, info)
+            elif 'union' in expr:
+                self._def_union_type(expr, info)
+            elif 'alternate' in expr:
+                self._def_alternate_type(expr, info)
+            elif 'command' in expr:
+                self._def_command(expr, info)
+            elif 'event' in expr:
+                self._def_event(expr, info)
+            else:
+                assert False
+
+    def check(self):
+        for ent in self.entity_dict.values():
+            ent.check(self)
 
 #
 # Code generation helpers
tests/qapi-schema/test-qapi.py +0 -0
index 634ef2d..461c713 100644

tests/qapi-schema/test-qapi.py import os · line 21

Quoted 7 lines
tests/qapi-schema/test-qapi.py +1 -1
@@ -16,7 +16,7 @@ import os
 import sys
 
 try:
-    exprs = parse_schema(sys.argv[1])
+    exprs = QAPISchema(sys.argv[1]).get_exprs()
 except SystemExit:
 

Following on earlier comment, this could have a block

Quoted 5 lines

except Exception as e:
print >>sys.stderr, e
sys.exit(1)

instead of having the object constructor call exit itself.

Thanks!

#6 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02/32] qapi: New QAPISchema intermediate reperesentation

Quoted 11 lines · Markus Armbruster
Quoted 6 lines · Daniel P. Berrange

On Thu, Sep 03, 2015 at 04:29:53PM +0200, Markus Armbruster wrote:
Python has a nice tool called pep8 which can apply a configurable bunch of style checks, it is probably worth someone wiring it up to make check in QEMU, since we have an increasing amount of python code. An exercise for a motivated reader.....

I've been using pylint (and dear-oh-dear how much pre-existing lint it finds!), wasn't aware of pep8, can throw it in.

The pep8 tool is design around checking the "official" python style rules as defined by Guido, and seems to be more commonly used than pylint IME.

Quoted 3 lines

Automatic checking would be nice, but our current code is probably too untidy for that. Besides, I don't have the time and energy to set it up now.

Yeah, no big deal.

Quoted 10 lines
Quoted 10 lines

scripts/qapi.py def check_exprs(exprs): · line 926

Quoted 5 lines
scripts/qapi.py +5 -0
+class QAPISchema(object):
+    def __init__(self, fname):
+        try:
+ self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
+        except (QAPISchemaError, QAPIExprError), err:

Preferred syntax for exceptions these days is

except Foo as err

Quoted 8 lines
Quoted 6 lines

instead of

except Foo, err

since the former is portable to Python3, should we need to consider that in future. Again, not a show stopper.

Does it work with Python 2.4? That's where we're currently stuck.

Urgh, that's a shame, what distro requires such an ancient python ? That syntax is only valid from 2.6 onwards.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#7 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02/32] qapi: New QAPISchema intermediate reperesentation

Quoted 10 lines · Daniel P. Berrange
Quoted 9 lines · Markus Armbruster
Quoted 6 lines · Daniel P. Berrange

On Thu, Sep 03, 2015 at 04:29:53PM +0200, Markus Armbruster wrote:
Python has a nice tool called pep8 which can apply a configurable bunch of style checks, it is probably worth someone wiring it up to make check in QEMU, since we have an increasing amount of python code. An exercise for a motivated reader.....

Quoted 6 lines
Quoted 2 lines

I've been using pylint (and dear-oh-dear how much pre-existing lint it finds!), wasn't aware of pep8, can throw it in.

The pep8 tool is design around checking the "official" python style rules as defined by Guido, and seems to be more commonly used than pylint IME.

I'll give the pep8 tool a try.

Quoted 21 lines
Quoted 3 lines

Automatic checking would be nice, but our current code is probably too untidy for that. Besides, I don't have the time and energy to set it up now.

Yeah, no big deal.

Quoted 14 lines
Quoted 14 lines

scripts/qapi.py def check_exprs(exprs): · line 926

Quoted 5 lines
scripts/qapi.py +5 -0
+class QAPISchema(object):
+    def __init__(self, fname):
+        try:
+ self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
+        except (QAPISchemaError, QAPIExprError), err:

Preferred syntax for exceptions these days is

except Foo as err

instead of

except Foo, err

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

since the former is portable to Python3, should we need to consider that in future. Again, not a show stopper.

Does it work with Python 2.4? That's where we're currently stuck.

Urgh, that's a shame, what distro requires such an ancient python ? That syntax is only valid from 2.6 onwards.

We bumped our minimum GLib version to 2.22 in May, which *might* have killed off ancient Python indirectly: GLib 2.22 is from 2009, Python 2.4 is from 2004, and 2.6 if from 2008.

I vaguely remember we went only to 2.22 due to some version of SLES. Anybody knows the oldest version of SLES still at 2.22, and what version of Python it provides?

RHEL-5 is out: nominally glib2-2.12 (and python 2.4.3).

RHEL-6 has glib2-2.28 and python 2.6.6.

#8 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02/32] qapi: New QAPISchema intermediate reperesentation

In reply to Markus Armbruster
Quoted 11 lines · Markus Armbruster

The QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaries) on the side.

---

(quoted diff) +3 -0
+class QAPISchemaObjectTypeVariants(object):
+    def __init__(self, tag_name, tag_enum, variants):
+        assert tag_name == None or isinstance(tag_name, str)

Review

Quoted 10 lines
(quoted diff) +10 -0
+        assert tag_enum == None or isinstance(tag_enum, str)
+        for v in variants:
+            assert isinstance(v, QAPISchemaObjectTypeVariant)
+        self.tag_name = tag_name
+        if tag_name:
+            assert not tag_enum
+            self.tag_member = None
+        else:
+            self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum,
+                                                         False)

See my review on 30/32. Introspection-wise, we want this generated member to be named 'type' to match what is on the wire. This patch, plus two more to qapi-types.py and qapi-visit.py, are sufficient to make no changes to the other generated files, while fixing the bug in qmp-introspect.c:

scripts/qapi.py +1 -1
index d2b6964..b2643af 100644
@@ -928,7 +928,7 @@ class QAPISchemaObjectTypeVariants(object):
             assert not tag_enum
             self.tag_member = None
         else:
-            self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum,
+            self.tag_member = QAPISchemaObjectTypeMember('type', tag_enum,
                                                          False)
         self.variants = variants
     def check(self, schema, members, seen):
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#9 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02/32] qapi: New QAPISchema intermediate reperesentation

In reply to Markus Armbruster

scripts/qapi.py def check_exprs(exprs): · line 787

Quoted 11 lines · Markus Armbruster

The QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaries) on the side.

scripts/qapi.py +7 -0
+
+class QAPISchemaArrayType(QAPISchemaType):
+    def __init__(self, name, info, element_type):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(element_type, str)
+        self.element_type_name = element_type
+        self.element_type = None

Would it make sense to have:

self._element_type_name = element_type
self.element_type = None

scripts/qapi.py def check_exprs(exprs): · line 791

Quoted 4 lines
scripts/qapi.py +4 -0
+    def check(self, schema):
+        self.element_type = schema.lookup_type(self.element_type_name)
+        assert self.element_type
+

and the corresponding
self.element_type = schema.lookup_type(self._element_type_name)

to make it obvious that ._element_type_name is for internal use only, while .element_type is the preferred member for queries/manipulation by qapi-*.py, which should only be using this object after check() has been run?

scripts/qapi.py def check_exprs(exprs): · line 804

Quoted 13 lines
scripts/qapi.py +13 -0
+class QAPISchemaObjectType(QAPISchemaType):
+    def __init__(self, name, info, base, local_members, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert base == None or isinstance(base, str)
+        for m in local_members:
+            assert isinstance(m, QAPISchemaObjectTypeMember)
+        assert variants == None \
+            or isinstance(variants, QAPISchemaObjectTypeVariants)
+        self.base_name = base
+        self.base = None
+        self.local_members = local_members
+        self.variants = variants
+        self.members = None

Similarly for self._base_name

scripts/qapi.py def check_exprs(exprs): · line 835

Quoted 10 lines
scripts/qapi.py +10 -0
+
+class QAPISchemaObjectTypeMember(object):
+    def __init__(self, name, typ, optional):
+        assert isinstance(name, str)
+        assert isinstance(typ, str)
+        assert isinstance(optional, bool)
+        self.name = name
+        self.type_name = typ
+        self.type = None
+        self.optional = optional

and for self._type_name

scripts/qapi.py def check_exprs(exprs): · line 855

Quoted 13 lines
scripts/qapi.py +13 -0
+class QAPISchemaObjectTypeVariants(object):
+    def __init__(self, tag_name, tag_enum, variants):
+        assert tag_name == None or isinstance(tag_name, str)
+        assert tag_enum == None or isinstance(tag_enum, str)
+        for v in variants:
+            assert isinstance(v, QAPISchemaObjectTypeVariant)
+        self.tag_name = tag_name
+        if tag_name:
+            assert not tag_enum
+            self.tag_member = None
+        else:
+            self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum,
+                                                         False)

and eventually for self._tag_name (except that we MUST expose self._tag_name until after we fix the C struct to use 'type' instead of 'kind').

scripts/qapi.py def check_exprs(exprs): · line 900

Quoted 11 lines
scripts/qapi.py +11 -0
+class QAPISchemaCommand(QAPISchemaEntity):
+    def __init__(self, name, info, arg_type, ret_type, gen, success_response):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not arg_type or isinstance(arg_type, str)
+        assert not ret_type or isinstance(ret_type, str)
+        self.arg_type_name = arg_type
+        self.arg_type = None
+        self.ret_type_name = ret_type
+        self.ret_type = None
+        self.gen = gen
+        self.success_response = success_response

and for self._art_type_name and self._ret_type_name

scripts/qapi.py def check_exprs(exprs): · line 915

Quoted 7 lines
scripts/qapi.py +7 -0
+
+class QAPISchemaEvent(QAPISchemaEntity):
+    def __init__(self, name, info, arg_type):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not arg_type or isinstance(arg_type, str)
+        self.arg_type_name = arg_type
+        self.arg_type = None

and for self._arg_type_name

Basically, if I'm understanding the python conventions correctly, naming an instance member with leading underscore implies that it is for private use only; and if nothing else, the rename proves whether any of the other python files are relying on a leaky abstraction.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#10 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02/32] qapi: New QAPISchema intermediate reperesentation

Quoted 74 lines · Eric Blake

scripts/qapi.py def check_exprs(exprs): · line 787

Quoted 11 lines · Markus Armbruster

The QAPI code generators work with a syntax tree (nested dictionaries) plus a few symbol tables (also dictionaries) on the side.

scripts/qapi.py +7 -0
+
+class QAPISchemaArrayType(QAPISchemaType):
+    def __init__(self, name, info, element_type):
+        QAPISchemaType.__init__(self, name, info)
+        assert isinstance(element_type, str)
+        self.element_type_name = element_type
+        self.element_type = None

Would it make sense to have:

self._element_type_name = element_type
self.element_type = None

scripts/qapi.py def check_exprs(exprs): · line 791

Quoted 4 lines
scripts/qapi.py +4 -0
+    def check(self, schema):
+        self.element_type = schema.lookup_type(self.element_type_name)
+        assert self.element_type
+

and the corresponding
self.element_type = schema.lookup_type(self._element_type_name)

to make it obvious that ._element_type_name is for internal use only, while .element_type is the preferred member for queries/manipulation by qapi-*.py, which should only be using this object after check() has been run?

scripts/qapi.py def check_exprs(exprs): · line 804

Quoted 13 lines
scripts/qapi.py +13 -0
+class QAPISchemaObjectType(QAPISchemaType):
+    def __init__(self, name, info, base, local_members, variants):
+        QAPISchemaType.__init__(self, name, info)
+        assert base == None or isinstance(base, str)
+        for m in local_members:
+            assert isinstance(m, QAPISchemaObjectTypeMember)
+        assert variants == None \
+            or isinstance(variants, QAPISchemaObjectTypeVariants)
+        self.base_name = base
+        self.base = None
+        self.local_members = local_members
+        self.variants = variants
+        self.members = None

Similarly for self._base_name

scripts/qapi.py def check_exprs(exprs): · line 835

Quoted 10 lines
scripts/qapi.py +10 -0
+
+class QAPISchemaObjectTypeMember(object):
+    def __init__(self, name, typ, optional):
+        assert isinstance(name, str)
+        assert isinstance(typ, str)
+        assert isinstance(optional, bool)
+        self.name = name
+        self.type_name = typ
+        self.type = None
+        self.optional = optional

and for self._type_name

scripts/qapi.py def check_exprs(exprs): · line 855

Quoted 13 lines
scripts/qapi.py +13 -0
+class QAPISchemaObjectTypeVariants(object):
+    def __init__(self, tag_name, tag_enum, variants):
+        assert tag_name == None or isinstance(tag_name, str)
+        assert tag_enum == None or isinstance(tag_enum, str)
+        for v in variants:
+            assert isinstance(v, QAPISchemaObjectTypeVariant)
+        self.tag_name = tag_name
+        if tag_name:
+            assert not tag_enum
+            self.tag_member = None
+        else:
+            self.tag_member = QAPISchemaObjectTypeMember('kind', tag_enum,
+                                                         False)
Quoted 3 lines

and eventually for self._tag_name (except that we MUST expose self._tag_name until after we fix the C struct to use 'type' instead of 'kind').

Python lets you use variants._tag_name. The leading underscore is advisory (except for wildcard imports, which don't apply here).

Quoted 25 lines

scripts/qapi.py def check_exprs(exprs): · line 900

Quoted 12 lines

+class QAPISchemaCommand(QAPISchemaEntity):
+ def __init__(self, name, info, arg_type, ret_type, gen, success_response):

scripts/qapi.py +9 -0
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not arg_type or isinstance(arg_type, str)
+        assert not ret_type or isinstance(ret_type, str)
+        self.arg_type_name = arg_type
+        self.arg_type = None
+        self.ret_type_name = ret_type
+        self.ret_type = None
+        self.gen = gen
+        self.success_response = success_response

and for self._art_type_name and self._ret_type_name

scripts/qapi.py def check_exprs(exprs): · line 915

Quoted 7 lines
scripts/qapi.py +7 -0
+
+class QAPISchemaEvent(QAPISchemaEntity):
+    def __init__(self, name, info, arg_type):
+        QAPISchemaEntity.__init__(self, name, info)
+        assert not arg_type or isinstance(arg_type, str)
+        self.arg_type_name = arg_type
+        self.arg_type = None

and for self._arg_type_name

Quoted 4 lines

Basically, if I'm understanding the python conventions correctly, naming an instance member with leading underscore implies that it is for private use only; and if nothing else, the rename proves whether any of the other python files are relying on a leaky abstraction.

I followed that convention for methods, but didn't bother for instance variables. Perhaps I should.

#11 lore raw

RFC PATCH v4 1/1 qapi: Hide internal data members of schema objects.

We have a few fields that exist mainly to hold information from __init__() until check() (matching the fact that parsing is two-pass; the first to find type names, the second to associate types together while honoring forward references), or which should only be used through accessor methods. We should not use these fields directly in other files after check() has run, so use the python convention of naming these fields with leading underscore to mark their internal usage, and to check that no one else was using them.

Exception: our crazy handling of simple unions (with a C member 'kind' matching the QMP wire 'type') requires peeking through the hidden field. This leaky abstraction will be cleaned up in a later patch.

---

Technically, I wrote this patch after 32; if you decide to rebase it into the series, you'll have to split it among 2, 10, 11, and 30. Up to you if you want to squash this in during your spin of v5, or if you want me to keep it as a separate patch for inclusion after your series.

 scripts/qapi-introspect.py |  2 +-
 scripts/qapi-types.py      |  2 +-
 scripts/qapi-visit.py      |  4 ++--
 scripts/qapi.py            | 54 +++++++++++++++++++++++-----------------------
 4 files changed, 31 insertions(+), 31 deletions(-)
scripts/qapi-introspect.py +1 -1
index 9886b65..c7a2dfb 100644
@@ -138,7 +138,7 @@ const char %(c_name)s[] = %(c_string)s;
     def visit_object_type_flat(self, name, info, members, variants):
         obj = {'members': [self._gen_member(m) for m in members]}
         if variants:
-            obj.update(self._gen_variants(variants.tag_name,
+            obj.update(self._gen_variants(variants._tag_name,
                                           variants.variants))
         self._gen_json(name, 'object', obj)
 
scripts/qapi-types.py +1 -1
index 83fc421..65c5ed4 100644
@@ -144,7 +144,7 @@ struct %(c_name)s {
     union { /* union tag is @%(c_name)s */
         void *data;
 ''',
-                 c_name=c_name(variants.tag_name or 'kind'))
+                 c_name=c_name(variants._tag_name or 'kind'))
 
     for var in variants.variants:
         # TODO ugly special case for simple union
scripts/qapi-visit.py +2 -2
index 468080e..196f389 100644
@@ -270,7 +270,7 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
                      c_name=c_name(name))
 
     tag_key = variants.tag_member.name
-    if not variants.tag_name:
+    if not variants._tag_name:
         # we pointlessly use a different key for simple unions
         tag_key = 'type'
     ret += mcgen('''
@@ -284,7 +284,7 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
         switch ((*obj)->%(c_name)s) {
 ''',
                  c_type=variants.tag_member.type.c_name(),
-                 c_name=c_name(variants.tag_name or 'kind'),
+                 c_name=c_name(variants._tag_name or 'kind'),
                  name=tag_key)
 
     for var in variants.variants:
scripts/qapi.py +27 -27

135 lines not on this page · raw message

Signature · 1 line
2.4.3
#12 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02.5/32] qapi: Hide internal data members of schema objects.

Quoted 15 lines · Eric Blake

We have a few fields that exist mainly to hold information from __init__() until check() (matching the fact that parsing is two-pass; the first to find type names, the second to associate types together while honoring forward references), or which should only be used through accessor methods. We should not use these fields directly in other files after check() has run, so use the python convention of naming these fields with leading underscore to mark their internal usage, and to check that no one else was using them.

Exception: our crazy handling of simple unions (with a C member 'kind' matching the QMP wire 'type') requires peeking through the hidden field. This leaky abstraction will be cleaned up in a later patch.

Quoted 8 lines

---

Technically, I wrote this patch after 32; if you decide to rebase it into the series, you'll have to split it among 2, 10, 11, and 30. Up to you if you want to squash this in during your spin of v5, or if you want me to keep it as a separate patch for inclusion after your series.

Oh, and tests/qapi-schema/test-qapi.py is annoying - even when it exits non-zero, it does NOT cause a failure in 'make check-qapi-schema'. You'll need to additionally consider squashing this in to 5:

tests/qapi-schema/test-qapi.py +3 -2
index 471f8e1..0fc0622 100644
@@ -40,8 +40,9 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
     @staticmethod
     def _print_variants(variants):
         if variants:
-            if variants.tag_name:
-                print '    tag %s' % variants.tag_name
+            # TODO: fix simple unions so we don't have to look at _tag_name
+            if variants._tag_name:
+                print '    tag %s' % variants.tag_member.name
             for v in variants.variants:
                 print '    case %s: %s' % (v.name, v.type.name)
 
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#13 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02.5/32] qapi: Hide internal data members of schema objects.

Quoted 19 lines · Eric Blake
Quoted 18 lines · Eric Blake

We have a few fields that exist mainly to hold information from __init__() until check() (matching the fact that parsing is two-pass; the first to find type names, the second to associate types together while honoring forward references), or which should only be used through accessor methods. We should not use these fields directly in other files after check() has run, so use the python convention of naming these fields with leading underscore to mark their internal usage, and to check that no one else was using them.

Exception: our crazy handling of simple unions (with a C member 'kind' matching the QMP wire 'type') requires peeking through the hidden field. This leaky abstraction will be cleaned up in a later patch.

---

Quoted 8 lines
Quoted 5 lines

Technically, I wrote this patch after 32; if you decide to rebase it into the series, you'll have to split it among 2, 10, 11, and 30. Up to you if you want to squash this in during your spin of v5, or if you want me to keep it as a separate patch for inclusion after your series.

Oh, and tests/qapi-schema/test-qapi.py is annoying - even when it exits non-zero, it does NOT cause a failure in 'make check-qapi-schema'.

Rather, it does not leave an obvious stack trace or immediate non-zero status; and 'make check-qapi-schema' only fails as a side effect if later output happens to differ from expectations. I would have had a much easier time chasing the problem if I'd had an obvious python stack trace.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#14 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02.5/32] qapi: Hide internal data members of schema objects.

Quoted 9 lines · Eric Blake
Quoted 3 lines

Oh, and tests/qapi-schema/test-qapi.py is annoying - even when it exits non-zero, it does NOT cause a failure in 'make check-qapi-schema'.

Rather, it does not leave an obvious stack trace or immediate non-zero status; and 'make check-qapi-schema' only fails as a side effect if later output happens to differ from expectations. I would have had a much easier time chasing the problem if I'd had an obvious python stack trace.

Oh, I see what happened. It did capture a trace in .err; but since the code checks for differences in .out before .err, I didn't even realize there was a trace to look at.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#15 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02.5/32] qapi: Hide internal data members of schema objects.

Quoted 11 lines · Eric Blake
Quoted 9 lines · Eric Blake
Quoted 3 lines

Oh, and tests/qapi-schema/test-qapi.py is annoying - even when it exits non-zero, it does NOT cause a failure in 'make check-qapi-schema'.

Rather, it does not leave an obvious stack trace or immediate non-zero status; and 'make check-qapi-schema' only fails as a side effect if later output happens to differ from expectations. I would have had a much easier time chasing the problem if I'd had an obvious python stack trace.

Quoted 3 lines

Oh, I see what happened. It did capture a trace in .err; but since the code checks for differences in .out before .err, I didn't even realize there was a trace to look at.

Yes, and it has annoyed me, too. Perhaps we should do something like this (entirely untested):

tests/Makefile +3 -4
index 92d682f..d622356 100644
@@ -519,13 +519,12 @@ check-tests/test-qapi.py: tests/test-qapi.py
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
 		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
-		$^ >$*.test.out 2>$*.test.err; \
+		$^ >$*.test.out 2>$*.test.raw-err; \
 		echo $$? >$*.test.exit, \
 		"  TEST  $*.out")
-	@diff -q $(SRC_PATH)/$*.out $*.test.out
 	@# Sanitize error messages (make them independent of build directory)
-	@perl -p -e 's|\Q$(SRC_PATH)\E/||g' $*.test.err | diff -q $(SRC_PATH)/$*.err -
-	@diff -q $(SRC_PATH)/$*.exit $*.test.exit
+	@perl -p -e 's|\Q$(SRC_PATH)\E/||g' $*.test.raw-err >$*.test.err
+	@diff -q $(SRC_PATH)/$*.out $*.test.out && diff -q $(SRC_PATH)/$*.err $*.test.err && diff -q $(SRC_PATH)/$*.exit $*.test.exit
 
 # Consolidated targets
 
#16 lore raw

Re: [Qemu-devel] [PATCH RFC v4 02.5/32] qapi: Hide internal data members of schema objects.

In reply to Eric Blake
Quoted 10 lines · Eric Blake

We have a few fields that exist mainly to hold information from __init__() until check() (matching the fact that parsing is two-pass; the first to find type names, the second to associate types together while honoring forward references), or which should only be used through accessor methods. We should not use these fields directly in other files after check() has run, so use the python convention of naming these fields with leading underscore to mark their internal usage, and to check that no one else was using them.

Quoted 4 lines

Exception: our crazy handling of simple unions (with a C member 'kind' matching the QMP wire 'type') requires peeking through the hidden field. This leaky abstraction will be cleaned up in a later patch.

There are two more peeks: QAPISchemaAlternateType peeks when it asserts it got simple variants, and test-qapi.py peeks when it prints a flat union's tag. You missed the latter, and neglected to mention the former.

Quoted 8 lines

---

Technically, I wrote this patch after 32; if you decide to rebase it into the series, you'll have to split it among 2, 10, 11, and 30. Up to you if you want to squash this in during your spin of v5, or if you want me to keep it as a separate patch for inclusion after your series.

I doubt the improvement worth the churn as a follow-up patch. Instead, I'll squash it in, less the rename of .tag_name, plus a few more renames elsewhere.

Thanks!

#17 lore raw

RFC PATCH v4 3/32 qapi: QAPISchema code generation helper methods

In reply to Markus Armbruster

New methods c_name(), c_type(), c_null(), json_type(), alternate_qtype().

---
 scripts/qapi.py | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 69 insertions(+), 7 deletions(-)
scripts/qapi.py +69 -7
index c42bea1..cf1436d 100644
@@ -760,15 +760,47 @@ class QAPISchemaEntity(object):
         assert isinstance(name, str)
         self.name = name
         self.info = info
+    def c_name(self):
+        return c_name(self.name)
     def check(self, schema):
         pass
 
 class QAPISchemaType(QAPISchemaEntity):
-    pass
+    def c_type(self, is_param=False):
+        return c_name(self.name) + pointer_suffix
+    def c_null(self):
+        return 'NULL'
+    def json_type(self):
+        pass
+    def alternate_qtype(self):
+        json2qtype = {
+            'string':  'QTYPE_QSTRING',
+            'number':  'QTYPE_QFLOAT',
+            'int':     'QTYPE_QINT',
+            'boolean': 'QTYPE_QBOOL',
+            'object':  'QTYPE_QDICT'
+        }
+        return json2qtype.get(self.json_type())
 
 class QAPISchemaBuiltinType(QAPISchemaType):
-    def __init__(self, name):
+    def __init__(self, name, json_type, c_type, c_null):
         QAPISchemaType.__init__(self, name, None)
+        assert not c_type or isinstance(c_type, str)
+        assert json_type in ('string', 'number', 'int', 'boolean', 'null',
+                             'value')
+        self.json_type_name = json_type
+        self.c_type_name = c_type
+        self.c_null_val = c_null
+    def c_name(self):
+        return self.name
+    def c_type(self, is_param=False):
+        if is_param and self.name == 'str':
+            return 'const ' + self.c_type_name
+        return self.c_type_name
+    def c_null(self):
+        return self.c_null_val
+    def json_type(self):
+        return self.json_type_name
 
 class QAPISchemaEnumType(QAPISchemaType):
     def __init__(self, name, info, values):
@@ -778,6 +810,12 @@ class QAPISchemaEnumType(QAPISchemaType):
         self.values = values
     def check(self, schema):
         assert len(set(self.values)) == len(self.values)
+    def c_type(self, is_param=False):
+        return c_name(self.name)
+    def c_null(self):
+        return c_enum_const(self.name, (self.values + ['MAX'])[0])
+    def json_type(self):
+        return 'string'
 
 class QAPISchemaArrayType(QAPISchemaType):
     def __init__(self, name, info, element_type):
@@ -788,6 +826,8 @@ class QAPISchemaArrayType(QAPISchemaType):
     def check(self, schema):
         self.element_type = schema.lookup_type(self.element_type_name)
         assert self.element_type
+    def json_type(self):
+        return 'array'
 
 class QAPISchemaObjectType(QAPISchemaType):
     def __init__(self, name, info, base, local_members, variants):
@@ -823,6 +863,14 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self.variants:
             self.variants.check(schema, members, seen)
         self.members = members
+    def c_name(self):
+        assert self.info
+        return QAPISchemaType.c_name(self)
+    def c_type(self, is_param=False):
+        assert self.info
+        return QAPISchemaType.c_type(self)
+    def json_type(self):
+        return 'object'
 
 class QAPISchemaObjectTypeMember(object):
     def __init__(self, name, typ, optional):
@@ -886,6 +934,8 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
     def check(self, schema):
         self.variants.check(schema, [], {})
+    def json_type(self):
+        return 'value'
 
 class QAPISchemaCommand(QAPISchemaEntity):
     def __init__(self, name, info, arg_type, ret_type, gen, success_response):
@@ -947,15 +997,27 @@ class QAPISchema(object):
     def lookup_type(self, name):
         return self.lookup_entity(name, QAPISchemaType)
 
-    def _def_builtin_type(self, name):
-        self._def_entity(QAPISchemaBuiltinType(name))
+    def _def_builtin_type(self, name, json_type, c_type, c_null):
+        self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null))
         if name != '**':
             self._make_array_type(name) # TODO really needed?
 
     def _def_predefineds(self):
-        for t in ['str', 'number', 'int', 'int8', 'int16', 'int32', 'int64',
-                  'uint8', 'uint16', 'uint32', 'uint64', 'size', 'bool', '**']:
-            self._def_builtin_type(t)
+        for t in [('str',    'string',  'char' + pointer_suffix, 'NULL'),
+                  ('number', 'number',  'double',   '0'),
+                  ('int',    'int',     'int64_t',  '0'),
+                  ('int8',   'int',     'int8_t',   '0'),
+                  ('int16',  'int',     'int16_t',  '0'),
+                  ('int32',  'int',     'int32_t',  '0'),
+                  ('int64',  'int',     'int64_t',  '0'),
+                  ('uint8',  'int',     'uint8_t',  '0'),
+                  ('uint16', 'int',     'uint16_t', '0'),
+                  ('uint32', 'int',     'uint32_t', '0'),
+                  ('uint64', 'int',     'uint64_t', '0'),
+                  ('size',   'int',     'uint64_t', '0'),
+                  ('bool',   'boolean', 'bool',     'false'),
+                  ('**',     'value',   None,       None)]:
+            self._def_builtin_type(*t)
 
     def _make_implicit_enum_type(self, name, values):
         name = name + 'Kind'
Signature · 1 line
2.4.3
Reviewed-by on v4 3/32
#18 lore raw

Re: [Qemu-devel] [PATCH RFC v4 03/32] qapi: QAPISchema code generation helper methods

Quoted 5 lines · Markus Armbruster

New methods c_name(), c_type(), c_null(), json_type(), alternate_qtype().

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#19 lore raw

RFC PATCH v4 4/32 qapi: New QAPISchemaVisitor

In reply to Markus Armbruster

The visitor will help keeping the code generation code simple and reasonably separated from QAPISchema details.

---
 scripts/qapi.py | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
scripts/qapi.py +46 -0
index cf1436d..3e7c154 100644
@@ -764,6 +764,29 @@ class QAPISchemaEntity(object):
         return c_name(self.name)
     def check(self, schema):
         pass
+    def visit(self, visitor):
+        pass
+
+class QAPISchemaVisitor(object):
+    def visit_begin(self, schema):
+        pass
+    def visit_end(self):
+        pass
+    def visit_builtin_type(self, name, info, json_type):
+        pass
+    def visit_enum_type(self, name, info, values):
+        pass
+    def visit_array_type(self, name, info, element_type):
+        pass
+    def visit_object_type(self, name, info, base, members, variants):
+        pass
+    def visit_alternate_type(self, name, info, variants):
+        pass
+    def visit_command(self, name, info, arg_type, ret_type,
+                      gen, success_response):
+        pass
+    def visit_event(self, name, info, arg_type):
+        pass
 
 class QAPISchemaType(QAPISchemaEntity):
     def c_type(self, is_param=False):
@@ -801,6 +824,8 @@ class QAPISchemaBuiltinType(QAPISchemaType):
         return self.c_null_val
     def json_type(self):
         return self.json_type_name
+    def visit(self, visitor):
+        visitor.visit_builtin_type(self.name, self.info, self.json_type())
 
 class QAPISchemaEnumType(QAPISchemaType):
     def __init__(self, name, info, values):
@@ -816,6 +841,8 @@ class QAPISchemaEnumType(QAPISchemaType):
         return c_enum_const(self.name, (self.values + ['MAX'])[0])
     def json_type(self):
         return 'string'
+    def visit(self, visitor):
+        visitor.visit_enum_type(self.name, self.info, self.values)
 
 class QAPISchemaArrayType(QAPISchemaType):
     def __init__(self, name, info, element_type):
@@ -828,6 +855,8 @@ class QAPISchemaArrayType(QAPISchemaType):
         assert self.element_type
     def json_type(self):
         return 'array'
+    def visit(self, visitor):
+        visitor.visit_array_type(self.name, self.info, self.element_type)
 
 class QAPISchemaObjectType(QAPISchemaType):
     def __init__(self, name, info, base, local_members, variants):
@@ -871,6 +900,9 @@ class QAPISchemaObjectType(QAPISchemaType):
         return QAPISchemaType.c_type(self)
     def json_type(self):
         return 'object'
+    def visit(self, visitor):
+        visitor.visit_object_type(self.name, self.info,
+                                  self.base, self.local_members, self.variants)
 
 class QAPISchemaObjectTypeMember(object):
     def __init__(self, name, typ, optional):
@@ -936,6 +968,8 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants.check(schema, [], {})
     def json_type(self):
         return 'value'
+    def visit(self, visitor):
+        visitor.visit_alternate_type(self.name, self.info, self.variants)
 
 class QAPISchemaCommand(QAPISchemaEntity):
     def __init__(self, name, info, arg_type, ret_type, gen, success_response):
@@ -956,6 +990,10 @@ class QAPISchemaCommand(QAPISchemaEntity):
         if self.ret_type_name:
             self.ret_type = schema.lookup_type(self.ret_type_name)
             assert isinstance(self.ret_type, QAPISchemaType)
+    def visit(self, visitor):
+        visitor.visit_command(self.name, self.info,
+                              self.arg_type, self.ret_type,
+                              self.gen, self.success_response)
 
 class QAPISchemaEvent(QAPISchemaEntity):
     def __init__(self, name, info, arg_type):
@@ -968,6 +1006,8 @@ class QAPISchemaEvent(QAPISchemaEntity):
             self.arg_type = schema.lookup_type(self.arg_type_name)
             assert isinstance(self.arg_type, QAPISchemaObjectType)
             assert not self.arg_type.variants # not implemented
+    def visit(self, visitor):
+        visitor.visit_event(self.name, self.info, self.arg_type)
 
 class QAPISchema(object):
     def __init__(self, fname):
@@ -1161,6 +1201,12 @@ class QAPISchema(object):
         for ent in self.entity_dict.values():
             ent.check(self)
 
+    def visit(self, visitor):
+        visitor.visit_begin(self)
+        for name in sorted(self.entity_dict.keys()):
+            self.entity_dict[name].visit(visitor)
+        visitor.visit_end()
+
 #
 # Code generation helpers
 #
Signature · 1 line
2.4.3
Reviewed-by on v4 4/32
#20 lore raw

Re: [Qemu-devel] [PATCH RFC v4 04/32] qapi: New QAPISchemaVisitor

Quoted 5 lines · Markus Armbruster

The visitor will help keeping the code generation code simple and reasonably separated from QAPISchema details.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#21 lore raw

RFC PATCH v4 5/32 tests/qapi-schema: Convert test harness to QAPISchemaVisitor

In reply to Markus Armbruster

The old code prints the result of parsing (list of expression dictionaries), and partial results of semantic analysis (list of enum dictionaries, list of struct dictionaries).

The new code prints a trace of a schema visit, i.e. what the back-ends are going to use. Built-in and array types are omitted, because they're boring.

---
 tests/qapi-schema/alternate-good.out            |  15 +-
 tests/qapi-schema/args-member-array.out         |  13 +-
 tests/qapi-schema/comments.out                  |   4 +-
 tests/qapi-schema/empty.out                     |   3 -
 tests/qapi-schema/enum-empty.out                |   4 +-
 tests/qapi-schema/event-case.out                |   4 +-
 tests/qapi-schema/flat-union-reverse-define.out |  21 ++-
 tests/qapi-schema/ident-with-escape.out         |   7 +-
 tests/qapi-schema/include-relpath.out           |   4 +-
 tests/qapi-schema/include-repetition.out        |   4 +-
 tests/qapi-schema/include-simple.out            |   4 +-
 tests/qapi-schema/indented-expr.out             |   7 +-
 tests/qapi-schema/qapi-schema-test.out          | 214 ++++++++++++++++++------
 tests/qapi-schema/returns-int.out               |   5 +-
 tests/qapi-schema/test-qapi.py                  |  38 ++++-
 tests/qapi-schema/type-bypass.out               |   7 +-
 16 files changed, 239 insertions(+), 115 deletions(-)
tests/qapi-schema/alternate-good.out +9 -6
index 99848ee..3d765ff 100644
@@ -1,6 +1,9 @@
-[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))]),
- OrderedDict([('enum', 'Enum'), ('data', ['hello', 'world'])]),
- OrderedDict([('alternate', 'Alt'), ('data', OrderedDict([('value', 'int'), ('string', 'Enum'), ('struct', 'Data')]))])]
-[{'enum_name': 'Enum', 'enum_values': ['hello', 'world']},
- {'enum_name': 'AltKind', 'enum_values': None}]
-[OrderedDict([('struct', 'Data'), ('data', OrderedDict([('*number', 'int'), ('*name', 'str')]))])]
+alternate Alt
+    case value: int
+    case string: Enum
+    case struct: Data
+enum AltKind ['value', 'string', 'struct']
+object Data
+    member number: int optional=True
+    member name: str optional=True
+enum Enum ['hello', 'world']
tests/qapi-schema/args-member-array.out +8 -5
index c39fa25..b67384c 100644
@@ -1,5 +1,8 @@
-[OrderedDict([('enum', 'abc'), ('data', ['a', 'b', 'c'])]),
- OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))]),
- OrderedDict([('command', 'okay'), ('data', OrderedDict([('member1', ['int']), ('member2', ['def'])]))])]
-[{'enum_name': 'abc', 'enum_values': ['a', 'b', 'c']}]
-[OrderedDict([('struct', 'def'), ('data', OrderedDict([('array', ['abc'])]))])]
+object :obj-okay-arg
+    member member1: intList optional=False
+    member member2: defList optional=False
+enum abc ['a', 'b', 'c']
+object def
+    member array: abcList optional=False
+command okay :obj-okay-arg -> None
+   gen=True success_response=True
tests/qapi-schema/comments.out +1 -3
index 4ce3dcf..6161b90 100644
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/empty.out +0 -3
index b7f89a4..e69de29 100644
@@ -1,3 +0,0 @@
-[]
-[]
-[]
tests/qapi-schema/enum-empty.out +1 -3
index 3b75c16..e09b00f 100644
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'MyEnum'), ('data', [])])]
-[{'enum_name': 'MyEnum', 'enum_values': []}]
-[]
+enum MyEnum []
tests/qapi-schema/event-case.out +1 -3
index 3764bc7..b5ae4c2 100644
@@ -1,3 +1 @@
-[OrderedDict([('event', 'oops')])]
-[]
-[]
+event oops None
tests/qapi-schema/flat-union-reverse-define.out +12 -9
index 1ed7b8a..477fb31 100644
@@ -1,9 +1,12 @@
-[OrderedDict([('union', 'TestUnion'), ('base', 'TestBase'), ('discriminator', 'enum1'), ('data', OrderedDict([('value1', 'TestTypeA'), ('value2', 'TestTypeB')]))]),
- OrderedDict([('struct', 'TestBase'), ('data', OrderedDict([('enum1', 'TestEnum')]))]),
- OrderedDict([('enum', 'TestEnum'), ('data', ['value1', 'value2'])]),
- OrderedDict([('struct', 'TestTypeA'), ('data', OrderedDict([('string', 'str')]))]),
- OrderedDict([('struct', 'TestTypeB'), ('data', OrderedDict([('integer', 'int')]))])]
-[{'enum_name': 'TestEnum', 'enum_values': ['value1', 'value2']}]
-[OrderedDict([('struct', 'TestBase'), ('data', OrderedDict([('enum1', 'TestEnum')]))]),
- OrderedDict([('struct', 'TestTypeA'), ('data', OrderedDict([('string', 'str')]))]),
- OrderedDict([('struct', 'TestTypeB'), ('data', OrderedDict([('integer', 'int')]))])]
+object TestBase
+    member enum1: TestEnum optional=False
+enum TestEnum ['value1', 'value2']
+object TestTypeA
+    member string: str optional=False
+object TestTypeB
+    member integer: int optional=False
+object TestUnion
+    base TestBase
+    tag enum1
+    case value1: TestTypeA
+    case value2: TestTypeB
tests/qapi-schema/ident-with-escape.out +4 -3
index 4028430..9577d1b 100644
@@ -1,3 +1,4 @@
-[OrderedDict([('command', 'fooA'), ('data', OrderedDict([('bar1', 'str')]))])]
-[]
-[]
+object :obj-fooA-arg
+    member bar1: str optional=False
+command fooA :obj-fooA-arg -> None
+   gen=True success_response=True
tests/qapi-schema/include-relpath.out +1 -3
index 4ce3dcf..6161b90 100644
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-repetition.out +1 -3
index 4ce3dcf..6161b90 100644
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-simple.out +1 -3
index 4ce3dcf..6161b90 100644
@@ -1,3 +1 @@
-[OrderedDict([('enum', 'Status'), ('data', ['good', 'bad', 'ugly'])])]
-[{'enum_name': 'Status', 'enum_values': ['good', 'bad', 'ugly']}]
-[]
+enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/indented-expr.out +4 -3
index b5ce915..c5af55a 100644
@@ -1,3 +1,4 @@
-[OrderedDict([('command', 'eins')]), OrderedDict([('command', 'zwei')])]
-[]
-[]
+command eins None -> None
+   gen=True success_response=True
+command zwei None -> None
+   gen=True success_response=True
tests/qapi-schema/qapi-schema-test.out +159 -55

214 lines not on this page · raw message

tests/qapi-schema/returns-int.out +2 -3
index 70b3ac5..1ac3e1e 100644
@@ -1,3 +1,2 @@
-[OrderedDict([('command', 'guest-get-time'), ('returns', 'int')])]
-[]
-[]
+command guest-get-time None -> int
+   gen=True success_response=True
tests/qapi-schema/test-qapi.py +31 -7
index 461c713..471f8e1 100644
@@ -15,11 +15,35 @@ from pprint import pprint
 import os
 import sys
 
-try:
-    exprs = QAPISchema(sys.argv[1]).get_exprs()
-except SystemExit:
-    raise
+class QAPISchemaTestVisitor(QAPISchemaVisitor):
+    def visit_enum_type(self, name, info, values):
+        print 'enum %s %s' % (name, values)
+    def visit_object_type(self, name, info, base, members, variants):
+        print 'object %s' % name
+        if base:
+            print '    base %s' % base.name
+        for m in members:
+            print '    member %s: %s optional=%s' % \
+                (m.name, m.type.name, m.optional)
+        self._print_variants(variants)
+    def visit_alternate_type(self, name, info, variants):
+        print 'alternate %s' % name
+        self._print_variants(variants)
+    def visit_command(self, name, info, arg_type, ret_type,
+                      gen, success_response):
+        print 'command %s %s -> %s' % \
+            (name, arg_type and arg_type.name, ret_type and ret_type.name)
+        print '   gen=%s success_response=%s' % (gen, success_response)
+    def visit_event(self, name, info, arg_type):
+        print 'event %s %s' % (name, arg_type and arg_type.name)
 
-pprint(exprs)
-pprint(enum_types)
-pprint(struct_types)
+    @staticmethod
+    def _print_variants(variants):
+        if variants:
+            if variants.tag_name:
+                print '    tag %s' % variants.tag_name
+            for v in variants.variants:
+                print '    case %s: %s' % (v.name, v.type.name)
+
+schema = QAPISchema(sys.argv[1])
+schema.visit(QAPISchemaTestVisitor())
tests/qapi-schema/type-bypass.out +4 -3
index eaf20f8..0070d4b 100644
@@ -1,3 +1,4 @@
-[OrderedDict([('command', 'unsafe'), ('data', OrderedDict([('arg', '**')])), ('returns', '**'), ('gen', False)])]
-[]
-[]
+object :obj-unsafe-arg
+    member arg: ** optional=False
+command unsafe :obj-unsafe-arg -> **
+   gen=False success_response=True
Signature · 1 line
2.4.3
Reviewed-by on v4 5/32
#22 lore raw

Re: [Qemu-devel] [PATCH RFC v4 05/32] tests/qapi-schema: Convert test harness to QAPISchemaVisitor

Quoted 10 lines · Markus Armbruster

The old code prints the result of parsing (list of expression dictionaries), and partial results of semantic analysis (list of enum dictionaries, list of struct dictionaries).

The new code prints a trace of a schema visit, i.e. what the back-ends are going to use. Built-in and array types are omitted, because they're boring.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#23 lore raw

Re: [Qemu-devel] [PATCH RFC v4 05/32] tests/qapi-schema: Convert test harness to QAPISchemaVisitor

In reply to Markus Armbruster
Quoted 7 lines · Markus Armbruster

The old code prints the result of parsing (list of expression dictionaries), and partial results of semantic analysis (list of enum dictionaries, list of struct dictionaries).

The new code prints a trace of a schema visit, i.e. what the back-ends are going to use. Built-in and array types are omitted, because they're boring.

Knowing which array types are in memory might not be boring in the future if we address the FIXME to make them generated on an as-needed basis instead of always - but that of course is a matter for the future patches :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#24 lore raw

Re: [Qemu-devel] [PATCH RFC v4 05/32] tests/qapi-schema: Convert test harness to QAPISchemaVisitor

In reply to Markus Armbruster
Quoted 14 lines · Markus Armbruster

The old code prints the result of parsing (list of expression dictionaries), and partial results of semantic analysis (list of enum dictionaries, list of struct dictionaries).

The new code prints a trace of a schema visit, i.e. what the back-ends are going to use. Built-in and array types are omitted, because they're boring.

---

+++ b/tests/qapi-schema/test-qapi.py

tests/qapi-schema/test-qapi.py from pprint import pprint · line 21

Quoted 7 lines
tests/qapi-schema/test-qapi.py +0 -4
@@ -15,11 +15,35 @@ from pprint import pprint
 import os
 import sys
 
-try:
-    exprs = QAPISchema(sys.argv[1]).get_exprs()
-except SystemExit:
-    raise

Don't we still need a try wrapped around...

tests/qapi-schema/test-qapi.py from pprint import pprint · line 49

Quoted 2 lines
tests/qapi-schema/test-qapi.py +2 -0
+schema = QAPISchema(sys.argv[1])
+schema.visit(QAPISchemaTestVisitor())

...this, to guarantee non-zero exit status if an exception caused by refactoring breaks the pretty-printing? (It happened to me when I tried to rename to _tag_name, and my only indication that something was wrong was that qapi-tests/qapi-schema-test.out was truncated due to this script aborting early after trying to access the old spelling tag_name).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#25 lore raw

Re: [Qemu-devel] [PATCH RFC v4 05/32] tests/qapi-schema: Convert test harness to QAPISchemaVisitor

Quoted 25 lines · Eric Blake
Quoted 14 lines · Markus Armbruster

The old code prints the result of parsing (list of expression dictionaries), and partial results of semantic analysis (list of enum dictionaries, list of struct dictionaries).

The new code prints a trace of a schema visit, i.e. what the back-ends are going to use. Built-in and array types are omitted, because they're boring.

---

+++ b/tests/qapi-schema/test-qapi.py

tests/qapi-schema/test-qapi.py from pprint import pprint · line 21

Quoted 7 lines
tests/qapi-schema/test-qapi.py +0 -4
@@ -15,11 +15,35 @@ from pprint import pprint
 import os
 import sys
 
-try:
-    exprs = QAPISchema(sys.argv[1]).get_exprs()
-except SystemExit:
-    raise

Don't we still need a try wrapped around...

Quoted 8 lines

tests/qapi-schema/test-qapi.py from pprint import pprint · line 49

Quoted 2 lines
tests/qapi-schema/test-qapi.py +2 -0
+schema = QAPISchema(sys.argv[1])
+schema.visit(QAPISchemaTestVisitor())

...this, to guarantee non-zero exit status if an exception caused by refactoring breaks the pretty-printing? (It happened to me when I tried to rename to _tag_name, and my only indication that something was wrong was that qapi-tests/qapi-schema-test.out was truncated due to this script aborting early after trying to access the old spelling tag_name).

The new version doesn't catch exceptions, and uncaught exceptions terminate the program unsucsessfully. Anything wrong with that?

The old version hides exceptions other than SystemExit. Why? I believe it's mostly historical accident. Accidents mostly cause by me.

I created the file in commit 9862657. I wrapped parse_schema() in a misguided attempt to keep the .err files concise:

    try:
        exprs = parse_schema(sys.stdin)
    except:
        print >>sys.stderr, "Crashed:", sys.exc_info()[0]
        exit(1)

This misreported errors as crashes, so I amended it in commit 2caba36:

    try:
        exprs = parse_schema(sys.stdin)
    except SystemExit:
        raise
    except:
        print >>sys.stderr, "Crashed:", sys.exc_info()[0]
        exit(1)

Lluís then ripped out the original misguided blanket exception catching, but left in the amend (commit 98c1200):

try:
exprs = parse_schema(sys.stdin)
except SystemExit:
raise

#26 lore raw

RFC PATCH v4 6/32 qapi: Split up some typedefs to ease review

In reply to Markus Armbruster

One of the next patches will among other things generate a separate typedef for some struct types, i.e.

    typedef struct FOO FOO;

    struct FOO {
    ...
    };

instead of

    typedef struct FOO {
    ...
    } FOO;

To make the generated files easier to diff, anticipate the change.

---
 scripts/qapi-types.py | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
scripts/qapi-types.py +12 -6
index d162ca2..124a788 100644
@@ -15,13 +15,15 @@ from qapi import *
 def generate_fwd_builtin(name):
     return mcgen('''
 
-typedef struct %(name)sList {
+typedef struct %(name)sList %(name)sList;
+
+struct %(name)sList {
     union {
         %(type)s value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-} %(name)sList;
+};
 ''',
                  type=c_type(name),
                  name=name)
@@ -31,26 +33,30 @@ def generate_fwd_struct(name):
 
 typedef struct %(name)s %(name)s;
 
-typedef struct %(name)sList {
+typedef struct %(name)sList %(name)sList;
+
+struct %(name)sList {
     union {
         %(name)s *value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-} %(name)sList;
+};
 ''',
                  name=c_name(name))
 
 def generate_fwd_enum_struct(name):
     return mcgen('''
 
-typedef struct %(name)sList {
+typedef struct %(name)sList %(name)sList;
+
+struct %(name)sList {
     union {
         %(name)s value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-} %(name)sList;
+};
 ''',
                  name=c_name(name))
 
Signature · 1 line
2.4.3
Reviewed-by on v4 6/32
#27 lore raw

Re: [Qemu-devel] [PATCH RFC v4 06/32] qapi: Split up some typedefs to ease review

Quoted 11 lines · Markus Armbruster

One of the next patches will among other things generate a separate typedef for some struct types, i.e.

    typedef struct FOO FOO;

    struct FOO {
    ...
    };

instead of

Quoted 7 lines
    typedef struct FOO {
    ...
    } FOO;

To make the generated files easier to diff, anticipate the change.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#28 lore raw

RFC PATCH v4 7/32 qapi: Generate comments to simplify splitting for review

In reply to Markus Armbruster

The effect of the next few patches on generated files is hard to review, because stuff gets generated in different order, which renders diffs of the generated files useless.

To get reviewable diffs, we need to split the generated files into suitable parts: put every top-level declaration in a file named like the thing declared, so we can diff top-level declarations regardless of their order in the generated files.

Since I don't feel like parsing C, simply generate a // comment identifying the declaration right before each top-level declaration. This lets us split with a simple shell loop:

mkdir -p o
for i in q*-{commands,marshal,event,types,visit}.[ch] do
csplit -n 4 -s "$i" '/^\/\//' '{*}'
for j in xx*
do
read h t <$j
[ "$h" = "//" ] || t=""
mv $j o/"$i-${j#xx}-${t/ /-}"
done
done

Splits each file F into F-NUMB-ID, where NUMB counts up from 0001, and ID comes from the // comment.

To check the split's sane, we can run

for i in q*-{commands,marshal,event,types,visit}.[ch] do cat o/$i-* | diff $i -
done

---
 scripts/qapi-commands.py |  7 +++++++
 scripts/qapi-event.py    | 12 ++++++++++++
 scripts/qapi-types.py    | 18 ++++++++++++++++++
 scripts/qapi-visit.py    | 12 ++++++++++++
 scripts/qapi.py          |  5 +++++
 5 files changed, 54 insertions(+)
scripts/qapi-commands.py +7 -0
index 12bdc4c..c3afb64 100644
@@ -24,6 +24,7 @@ def generate_command_decl(name, args, ret_type):
             arglist += "bool has_%s, " % c_name(argname)
         arglist += "%s %s, " % (argtype, c_name(argname))
     return mcgen('''
+// decl qmp_%(name)s
 %(ret_type)s qmp_%(name)s(%(args)sError **errp);
 ''',
                  ret_type=c_type(ret_type), name=c_name(name),
@@ -160,6 +161,7 @@ def gen_marshal_output(name, ret_type):
         return ""
 
     ret = mcgen('''
+// def qmp_marshal_output_%(c_name)s
 static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_out, Error **errp)
 {
     Error *local_err = NULL;
@@ -198,10 +200,12 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
     hdr = gen_marshal_input_decl(name, middle_mode)
 
     ret = mcgen('''
+// def qmp_marshal_input_%(name)s
 %(header)s
 {
     Error *local_err = NULL;
 ''',
+                name=name,
                 header=hdr)
 
     if ret_type:
@@ -252,6 +256,7 @@ qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
                      opts=options)
     pop_indent()
     ret = mcgen('''
+// def qmp_init_marshal
 static void qmp_init_marshal(void)
 {
 ''')
@@ -259,6 +264,7 @@ static void qmp_init_marshal(void)
     ret += mcgen('''
 }
 
+// decl qapi_init
 qapi_init(qmp_init_marshal);
 ''')
     return ret
@@ -347,6 +353,7 @@ for cmd in commands:
         fdef.write(ret)
 
     if middle_mode:
+        fdecl.write('// decl gen_marshal_input_%s\n' % cmd['command'])
         fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], middle_mode))
 
     ret = gen_marshal_input(cmd['command'], arglist, ret_type, middle_mode) + "\n"
scripts/qapi-event.py +12 -0
index aec2d32..e511437 100644
@@ -9,6 +9,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+import re;
 from ordereddict import OrderedDict
 from qapi import *
 
@@ -33,22 +34,30 @@ def _generate_event_api_name(event_name, params):
 # Following are the core functions that generate C APIs to emit event.
 
 def generate_event_declaration(api_name):
+    match = re.match(r'void ([A-Za-z0-9_]+)', api_name)
+    name = match.group(1)
     return mcgen('''
 
+// decl %(name)s
 %(api_name)s;
 ''',
+                 name=name,
                  api_name = api_name)
 
 def generate_event_implement(api_name, event_name, params):
+    match = re.match(r'void ([A-Za-z0-9_]+)', api_name)
+    name = match.group(1)
     # step 1: declare any variables
     ret = mcgen("""
 
+// def %(name)s
 %(api_name)s
 {
     QDict *qmp;
     Error *local_err = NULL;
     QMPEventFuncEmit emit;
 """,
+                name=name,
                 api_name = api_name)
 
     if params:
@@ -162,11 +171,13 @@ def generate_event_implement(api_name, event_name, params):
 def generate_event_enum_decl(event_enum_name, event_enum_values):
     lookup_decl = mcgen('''
 
+// decl %(event_enum_name)s_lookup
 extern const char *%(event_enum_name)s_lookup[];
 ''',
                         event_enum_name = event_enum_name)
 
     enum_decl = mcgen('''
+// typedef %(event_enum_name)s
 typedef enum %(event_enum_name)s {
 ''',
                       event_enum_name = event_enum_name)
@@ -194,6 +205,7 @@ typedef enum %(event_enum_name)s {
 def generate_event_enum_lookup(event_enum_name, event_enum_strings):
     ret = mcgen('''
 
+// def %(event_enum_name)s_lookup
 const char *%(event_enum_name)s_lookup[] = {
 ''',
                 event_enum_name = event_enum_name)
scripts/qapi-types.py +18 -0
index 124a788..7fabcaa 100644
@@ -15,8 +15,10 @@ from qapi import *
 def generate_fwd_builtin(name):
     return mcgen('''
 
+// typedef %(name)sList
 typedef struct %(name)sList %(name)sList;
 
+// struct %(name)sList
 struct %(name)sList {
     union {
         %(type)s value;
@@ -31,10 +33,13 @@ struct %(name)sList {
 def generate_fwd_struct(name):
     return mcgen('''
 
+// typedef %(name)s
 typedef struct %(name)s %(name)s;
 
+// typedef %(name)sList
 typedef struct %(name)sList %(name)sList;
 
+// struct %(name)sList
 struct %(name)sList {
     union {
         %(name)s *value;
@@ -48,8 +53,10 @@ struct %(name)sList {
 def generate_fwd_enum_struct(name):
     return mcgen('''
 
+// typedef %(name)sList
 typedef struct %(name)sList %(name)sList;
 
+// struct %(name)sList
 struct %(name)sList {
     union {
         %(name)s value;
@@ -84,6 +91,7 @@ def generate_struct(expr):
 
     ret = mcgen('''
 
+// struct %(name)s
 struct %(name)s {
 ''',
           name=c_name(structname))
@@ -110,6 +118,7 @@ struct %(name)s {
 def generate_enum_lookup(name, values):
     ret = mcgen('''
 
+// def %(name)s_lookup
 const char *const %(name)s_lookup[] = {
 ''',
                 name=c_name(name))
@@ -132,12 +141,14 @@ def generate_enum(name, values):
     name = c_name(name)
     lookup_decl = mcgen('''
 
+// decl %(name)s_lookup
 extern const char *const %(name)s_lookup[];
 ''',
                 name=name)
 
     enum_decl = mcgen('''
 
+// typedef %(name)s
 typedef enum %(name)s {
 ''',
                 name=name)
@@ -169,6 +180,7 @@ def generate_alternate_qtypes(expr):
 
     ret = mcgen('''
 
+// def %(name)s_qtypes
 const int %(name)s_qtypes[QTYPE_MAX] = {
 ''',
                 name=c_name(name))
@@ -205,6 +217,7 @@ def generate_union(expr, meta):
 
     ret = mcgen('''
 
+// struct %(name)s
 struct %(name)s {
 ''',
                 name=name)
@@ -252,6 +265,8 @@ struct %(name)s {
 ''')
     if meta == 'alternate':
         ret += mcgen('''
+
+// decl %(name)s_qtypes
 extern const int %(name)s_qtypes[];
 ''',
             name=name)
@@ -261,6 +276,8 @@ extern const int %(name)s_qtypes[];
 
 def generate_type_cleanup_decl(name):
     ret = mcgen('''
+
+// decl qapi_free_%(name)s
 void qapi_free_%(name)s(%(c_type)s obj);
 ''',
                 c_type=c_type(name), name=c_name(name))
@@ -269,6 +286,7 @@ void qapi_free_%(name)s(%(c_type)s obj);
 def generate_type_cleanup(name):
     ret = mcgen('''
 
+// def qapi_free_%(name)s
 void qapi_free_%(name)s(%(c_type)s obj)
 {
     QapiDeallocVisitor *md;
scripts/qapi-visit.py +12 -0
index c493964..f0fef55 100644
@@ -28,12 +28,14 @@ def generate_visit_implicit_struct(type):
         # Need a forward declaration
         ret += mcgen('''
 
+// def visit_type_%(c_type)s_fields
 static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
 ''',
                      c_type=type_name(type))
 
     ret += mcgen('''
 
+// def visit_type_implicit_%(c_type)s
 static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp)
 {
     Error *err = NULL;
@@ -59,6 +61,7 @@ def generate_visit_struct_fields(name, members, base = None):
 
     ret += mcgen('''
 
+// def visit_type_%(name)s_fields
 static void visit_type_%(name)s_fields(Visitor *m, %(name)s **obj, Error **errp)
 {
     Error *err = NULL;
@@ -146,6 +149,7 @@ def generate_visit_struct(expr):
 
     ret += mcgen('''
 
+// def visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
 ''',
@@ -161,6 +165,7 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 def generate_visit_list(name):
     return mcgen('''
 
+// def visit_type_%(name)sList
 void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -190,6 +195,7 @@ out:
 def generate_visit_enum(name):
     return mcgen('''
 
+// def visit_type_%(c_name)s
 void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error **errp)
 {
     visit_type_enum(m, (int *)obj, %(c_name)s_lookup, "%(name)s", name, errp);
@@ -200,6 +206,7 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error
 def generate_visit_alternate(name, members):
     ret = mcgen('''
 
+// def visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -283,6 +290,7 @@ def generate_visit_union(expr):
 
     ret += mcgen('''
 
+// def visit_type_%(c_name)s
 void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -365,11 +373,13 @@ def generate_declaration(name, builtin_type=False):
         name = c_name(name)
         ret += mcgen('''
 
+// decl visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp);
 ''',
                      name=name)
 
     ret += mcgen('''
+// decl visit_type_%(name)sList
 void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp);
 ''',
                  name=name)
@@ -378,6 +388,7 @@ void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, E
 
 def generate_enum_declaration(name):
     ret = mcgen('''
+// decl visit_type_%(name)sList
 void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp);
 ''',
                 name=c_name(name))
@@ -387,6 +398,7 @@ void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, E
 def generate_decl_enum(name):
     return mcgen('''
 
+// decl visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp);
 ''',
                  name=c_name(name))
scripts/qapi.py +5 -0
index 3e7c154..2b639a5 100644
@@ -1410,6 +1410,7 @@ def guardname(filename):
 def guardstart(name):
     return mcgen('''
 
+// guardstart %(name)s
 #ifndef %(name)s
 #define %(name)s
 
@@ -1419,6 +1420,7 @@ def guardstart(name):
 def guardend(name):
     return mcgen('''
 
+// guardend %(name)s
 #endif /* %(name)s */
 
 ''',
@@ -1503,12 +1505,14 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
     fdecl = maybe_open(do_h, h_file, 'w')
 
     fdef.write(mcgen('''
+// prologue
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 %(comment)s
 ''',
                      comment = c_comment))
 
     fdecl.write(mcgen('''
+// prologue
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 %(comment)s
 #ifndef %(guard)s
@@ -1521,6 +1525,7 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
 
 def close_output(fdef, fdecl):
     fdecl.write('''
+// epilogue
 #endif
 ''')
     fdecl.close()
Signature · 1 line
2.4.3
Reviewed-by on v4 7/32
#29 lore raw

Re: [Qemu-devel] [PATCH RFC v4 07/32] qapi: Generate comments to simplify splitting for review

Quoted 28 lines · Markus Armbruster

The effect of the next few patches on generated files is hard to review, because stuff gets generated in different order, which renders diffs of the generated files useless.

To get reviewable diffs, we need to split the generated files into suitable parts: put every top-level declaration in a file named like the thing declared, so we can diff top-level declarations regardless of their order in the generated files.

Since I don't feel like parsing C, simply generate a // comment identifying the declaration right before each top-level declaration. This lets us split with a simple shell loop:

mkdir -p o
for i in q*-{commands,marshal,event,types,visit}.[ch] do
csplit -n 4 -s "$i" '/^\/\//' '{*}'
for j in xx*
do
read h t <$j
[ "$h" = "//" ] || t=""
mv $j o/"$i-${j#xx}-${t/ /-}"
done
done

Splits each file F into F-NUMB-ID, where NUMB counts up from 0001, and ID comes from the // comment.

Quoted 7 lines

To check the split's sane, we can run

for i in q*-{commands,marshal,event,types,visit}.[ch] do cat o/$i-* | diff $i -
done

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
#30 lore raw

RFC PATCH v4 8/32 Revert "qapi: Generate comments to simplify splitting for review"

In reply to Markus Armbruster

This reverts commit 22ff4d4ac6cd553f328e400de6913dc879581c4c.

---
 scripts/qapi-commands.py |  7 -------
 scripts/qapi-event.py    | 12 ------------
 scripts/qapi-types.py    | 18 ------------------
 scripts/qapi-visit.py    | 12 ------------
 scripts/qapi.py          |  5 -----
 5 files changed, 54 deletions(-)
scripts/qapi-commands.py +0 -7
index c3afb64..12bdc4c 100644
@@ -24,7 +24,6 @@ def generate_command_decl(name, args, ret_type):
             arglist += "bool has_%s, " % c_name(argname)
         arglist += "%s %s, " % (argtype, c_name(argname))
     return mcgen('''
-// decl qmp_%(name)s
 %(ret_type)s qmp_%(name)s(%(args)sError **errp);
 ''',
                  ret_type=c_type(ret_type), name=c_name(name),
@@ -161,7 +160,6 @@ def gen_marshal_output(name, ret_type):
         return ""
 
     ret = mcgen('''
-// def qmp_marshal_output_%(c_name)s
 static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_out, Error **errp)
 {
     Error *local_err = NULL;
@@ -200,12 +198,10 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
     hdr = gen_marshal_input_decl(name, middle_mode)
 
     ret = mcgen('''
-// def qmp_marshal_input_%(name)s
 %(header)s
 {
     Error *local_err = NULL;
 ''',
-                name=name,
                 header=hdr)
 
     if ret_type:
@@ -256,7 +252,6 @@ qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
                      opts=options)
     pop_indent()
     ret = mcgen('''
-// def qmp_init_marshal
 static void qmp_init_marshal(void)
 {
 ''')
@@ -264,7 +259,6 @@ static void qmp_init_marshal(void)
     ret += mcgen('''
 }
 
-// decl qapi_init
 qapi_init(qmp_init_marshal);
 ''')
     return ret
@@ -353,7 +347,6 @@ for cmd in commands:
         fdef.write(ret)
 
     if middle_mode:
-        fdecl.write('// decl gen_marshal_input_%s\n' % cmd['command'])
         fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], middle_mode))
 
     ret = gen_marshal_input(cmd['command'], arglist, ret_type, middle_mode) + "\n"
scripts/qapi-event.py +0 -12
index e511437..aec2d32 100644
@@ -9,7 +9,6 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-import re;
 from ordereddict import OrderedDict
 from qapi import *
 
@@ -34,30 +33,22 @@ def _generate_event_api_name(event_name, params):
 # Following are the core functions that generate C APIs to emit event.
 
 def generate_event_declaration(api_name):
-    match = re.match(r'void ([A-Za-z0-9_]+)', api_name)
-    name = match.group(1)
     return mcgen('''
 
-// decl %(name)s
 %(api_name)s;
 ''',
-                 name=name,
                  api_name = api_name)
 
 def generate_event_implement(api_name, event_name, params):
-    match = re.match(r'void ([A-Za-z0-9_]+)', api_name)
-    name = match.group(1)
     # step 1: declare any variables
     ret = mcgen("""
 
-// def %(name)s
 %(api_name)s
 {
     QDict *qmp;
     Error *local_err = NULL;
     QMPEventFuncEmit emit;
 """,
-                name=name,
                 api_name = api_name)
 
     if params:
@@ -171,13 +162,11 @@ def generate_event_implement(api_name, event_name, params):
 def generate_event_enum_decl(event_enum_name, event_enum_values):
     lookup_decl = mcgen('''
 
-// decl %(event_enum_name)s_lookup
 extern const char *%(event_enum_name)s_lookup[];
 ''',
                         event_enum_name = event_enum_name)
 
     enum_decl = mcgen('''
-// typedef %(event_enum_name)s
 typedef enum %(event_enum_name)s {
 ''',
                       event_enum_name = event_enum_name)
@@ -205,7 +194,6 @@ typedef enum %(event_enum_name)s {
 def generate_event_enum_lookup(event_enum_name, event_enum_strings):
     ret = mcgen('''
 
-// def %(event_enum_name)s_lookup
 const char *%(event_enum_name)s_lookup[] = {
 ''',
                 event_enum_name = event_enum_name)
scripts/qapi-types.py +0 -18
index 7fabcaa..124a788 100644
@@ -15,10 +15,8 @@ from qapi import *
 def generate_fwd_builtin(name):
     return mcgen('''
 
-// typedef %(name)sList
 typedef struct %(name)sList %(name)sList;
 
-// struct %(name)sList
 struct %(name)sList {
     union {
         %(type)s value;
@@ -33,13 +31,10 @@ struct %(name)sList {
 def generate_fwd_struct(name):
     return mcgen('''
 
-// typedef %(name)s
 typedef struct %(name)s %(name)s;
 
-// typedef %(name)sList
 typedef struct %(name)sList %(name)sList;
 
-// struct %(name)sList
 struct %(name)sList {
     union {
         %(name)s *value;
@@ -53,10 +48,8 @@ struct %(name)sList {
 def generate_fwd_enum_struct(name):
     return mcgen('''
 
-// typedef %(name)sList
 typedef struct %(name)sList %(name)sList;
 
-// struct %(name)sList
 struct %(name)sList {
     union {
         %(name)s value;
@@ -91,7 +84,6 @@ def generate_struct(expr):
 
     ret = mcgen('''
 
-// struct %(name)s
 struct %(name)s {
 ''',
           name=c_name(structname))
@@ -118,7 +110,6 @@ struct %(name)s {
 def generate_enum_lookup(name, values):
     ret = mcgen('''
 
-// def %(name)s_lookup
 const char *const %(name)s_lookup[] = {
 ''',
                 name=c_name(name))
@@ -141,14 +132,12 @@ def generate_enum(name, values):
     name = c_name(name)
     lookup_decl = mcgen('''
 
-// decl %(name)s_lookup
 extern const char *const %(name)s_lookup[];
 ''',
                 name=name)
 
     enum_decl = mcgen('''
 
-// typedef %(name)s
 typedef enum %(name)s {
 ''',
                 name=name)
@@ -180,7 +169,6 @@ def generate_alternate_qtypes(expr):
 
     ret = mcgen('''
 
-// def %(name)s_qtypes
 const int %(name)s_qtypes[QTYPE_MAX] = {
 ''',
                 name=c_name(name))
@@ -217,7 +205,6 @@ def generate_union(expr, meta):
 
     ret = mcgen('''
 
-// struct %(name)s
 struct %(name)s {
 ''',
                 name=name)
@@ -265,8 +252,6 @@ struct %(name)s {
 ''')
     if meta == 'alternate':
         ret += mcgen('''
-
-// decl %(name)s_qtypes
 extern const int %(name)s_qtypes[];
 ''',
             name=name)
@@ -276,8 +261,6 @@ extern const int %(name)s_qtypes[];
 
 def generate_type_cleanup_decl(name):
     ret = mcgen('''
-
-// decl qapi_free_%(name)s
 void qapi_free_%(name)s(%(c_type)s obj);
 ''',
                 c_type=c_type(name), name=c_name(name))
@@ -286,7 +269,6 @@ void qapi_free_%(name)s(%(c_type)s obj);
 def generate_type_cleanup(name):
     ret = mcgen('''
 
-// def qapi_free_%(name)s
 void qapi_free_%(name)s(%(c_type)s obj)
 {
     QapiDeallocVisitor *md;
scripts/qapi-visit.py +0 -12
index f0fef55..c493964 100644
@@ -28,14 +28,12 @@ def generate_visit_implicit_struct(type):
         # Need a forward declaration
         ret += mcgen('''
 
-// def visit_type_%(c_type)s_fields
 static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
 ''',
                      c_type=type_name(type))
 
     ret += mcgen('''
 
-// def visit_type_implicit_%(c_type)s
 static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp)
 {
     Error *err = NULL;
@@ -61,7 +59,6 @@ def generate_visit_struct_fields(name, members, base = None):
 
     ret += mcgen('''
 
-// def visit_type_%(name)s_fields
 static void visit_type_%(name)s_fields(Visitor *m, %(name)s **obj, Error **errp)
 {
     Error *err = NULL;
@@ -149,7 +146,6 @@ def generate_visit_struct(expr):
 
     ret += mcgen('''
 
-// def visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
 ''',
@@ -165,7 +161,6 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 def generate_visit_list(name):
     return mcgen('''
 
-// def visit_type_%(name)sList
 void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -195,7 +190,6 @@ out:
 def generate_visit_enum(name):
     return mcgen('''
 
-// def visit_type_%(c_name)s
 void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error **errp)
 {
     visit_type_enum(m, (int *)obj, %(c_name)s_lookup, "%(name)s", name, errp);
@@ -206,7 +200,6 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error
 def generate_visit_alternate(name, members):
     ret = mcgen('''
 
-// def visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -290,7 +283,6 @@ def generate_visit_union(expr):
 
     ret += mcgen('''
 
-// def visit_type_%(c_name)s
 void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -373,13 +365,11 @@ def generate_declaration(name, builtin_type=False):
         name = c_name(name)
         ret += mcgen('''
 
-// decl visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp);
 ''',
                      name=name)
 
     ret += mcgen('''
-// decl visit_type_%(name)sList
 void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp);
 ''',
                  name=name)
@@ -388,7 +378,6 @@ void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, E
 
 def generate_enum_declaration(name):
     ret = mcgen('''
-// decl visit_type_%(name)sList
 void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp);
 ''',
                 name=c_name(name))
@@ -398,7 +387,6 @@ void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, E
 def generate_decl_enum(name):
     return mcgen('''
 
-// decl visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp);
 ''',
                  name=c_name(name))
scripts/qapi.py +0 -5
index 2b639a5..3e7c154 100644
@@ -1410,7 +1410,6 @@ def guardname(filename):
 def guardstart(name):
     return mcgen('''
 
-// guardstart %(name)s
 #ifndef %(name)s
 #define %(name)s
 
@@ -1420,7 +1419,6 @@ def guardstart(name):
 def guardend(name):
     return mcgen('''
 
-// guardend %(name)s
 #endif /* %(name)s */
 
 ''',
@@ -1505,14 +1503,12 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
     fdecl = maybe_open(do_h, h_file, 'w')
 
     fdef.write(mcgen('''
-// prologue
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 %(comment)s
 ''',
                      comment = c_comment))
 
     fdecl.write(mcgen('''
-// prologue
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 %(comment)s
 #ifndef %(guard)s
@@ -1525,7 +1521,6 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
 
 def close_output(fdef, fdecl):
     fdecl.write('''
-// epilogue
 #endif
 ''')
     fdecl.close()
Signature · 1 line
2.4.3
#31 lore raw

RFC PATCH v4 9/32 Revert "qapi: Split up some typedefs to ease review"

In reply to Markus Armbruster

This reverts commit 6542034f356ebc6f2818c1c6b8c9a143f817bb2a.

---
 scripts/qapi-types.py | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)
scripts/qapi-types.py +6 -12
index 124a788..d162ca2 100644
@@ -15,15 +15,13 @@ from qapi import *
 def generate_fwd_builtin(name):
     return mcgen('''
 
-typedef struct %(name)sList %(name)sList;
-
-struct %(name)sList {
+typedef struct %(name)sList {
     union {
         %(type)s value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-};
+} %(name)sList;
 ''',
                  type=c_type(name),
                  name=name)
@@ -33,30 +31,26 @@ def generate_fwd_struct(name):
 
 typedef struct %(name)s %(name)s;
 
-typedef struct %(name)sList %(name)sList;
-
-struct %(name)sList {
+typedef struct %(name)sList {
     union {
         %(name)s *value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-};
+} %(name)sList;
 ''',
                  name=c_name(name))
 
 def generate_fwd_enum_struct(name):
     return mcgen('''
 
-typedef struct %(name)sList %(name)sList;
-
-struct %(name)sList {
+typedef struct %(name)sList {
     union {
         %(name)s value;
         uint64_t padding;
     };
     struct %(name)sList *next;
-};
+} %(name)sList;
 ''',
                  name=c_name(name))
 
Signature · 1 line
2.4.3
#32 lore raw

RFC PATCH v4 10/32 qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

In reply to Markus Armbruster

Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:

    { 'union': 'UserDefFlatUnion',
      'base': 'UserDefUnionBase',
      'discriminator': 'enum1',
      'data': { 'value1' : 'UserDefA',
                'value2' : 'UserDefB',
                'value3' : 'UserDefB' } }

    { 'struct': 'UserDefUnionBase',
      'base': 'UserDefZero',
      'data': { 'string': 'str', 'enum1': 'EnumOne' } }

    { 'struct': 'UserDefZero',
      'data': { 'integer': 'int' } }

Patch's effect on UserDefFlatUnion:

     struct UserDefFlatUnion {
         /* Members inherited from UserDefUnionBase: */
    +    int64_t integer;
         char *string;
         EnumOne enum1;
         /* Own members: */
         union { /* union tag is @enum1 */
             void *data;
             UserDefA *value1;
             UserDefB *value2;
             UserDefB *value3;
         };
     };

Flat union visitors remain broken. They'll be fixed next.

---
 docs/qapi-code-gen.txt                  |  51 +++---
 scripts/qapi-types.py                   | 273 ++++++++++++++------------------
 tests/qapi-schema/qapi-schema-test.json |   4 +-
 3 files changed, 144 insertions(+), 184 deletions(-)
docs/qapi-code-gen.txt +25 -26
index ff16df2..710defc 100644
@@ -537,22 +537,6 @@ Example:
     $ cat qapi-generated/example-qapi-types.c
 [Uninteresting stuff omitted...]
 
-    void qapi_free_UserDefOneList(UserDefOneList *obj)
-    {
-        QapiDeallocVisitor *md;
-        Visitor *v;
-
-        if (!obj) {
-            return;
-        }
-
-        md = qapi_dealloc_visitor_new();
-        v = qapi_dealloc_get_visitor(md);
-        visit_type_UserDefOneList(v, &obj, NULL, NULL);
-        qapi_dealloc_visitor_cleanup(md);
-    }
-
-
     void qapi_free_UserDefOne(UserDefOne *obj)
     {
         QapiDeallocVisitor *md;
@@ -567,6 +551,21 @@ Example:
         visit_type_UserDefOne(v, &obj, NULL, NULL);
         qapi_dealloc_visitor_cleanup(md);
     }
+
+    void qapi_free_UserDefOneList(UserDefOneList *obj)
+    {
+        QapiDeallocVisitor *md;
+        Visitor *v;
+
+        if (!obj) {
+            return;
+        }
+
+        md = qapi_dealloc_visitor_new();
+        v = qapi_dealloc_get_visitor(md);
+        visit_type_UserDefOneList(v, &obj, NULL, NULL);
+        qapi_dealloc_visitor_cleanup(md);
+    }
     $ cat qapi-generated/example-qapi-types.h
 [Uninteresting stuff omitted...]
 
@@ -577,24 +576,24 @@ Example:
 
     typedef struct UserDefOne UserDefOne;
 
-    typedef struct UserDefOneList {
+    typedef struct UserDefOneList UserDefOneList;
+
+    struct UserDefOne {
+        int64_t integer;
+        char *string;
+    };
+
+    void qapi_free_UserDefOne(UserDefOne *obj);
+
+    struct UserDefOneList {
         union {
             UserDefOne *value;
             uint64_t padding;
         };
         struct UserDefOneList *next;
-    } UserDefOneList;
-
-
-[Functions on built-in types omitted...]
-
-    struct UserDefOne {
-        int64_t integer;
-        char *string;
     };
 
     void qapi_free_UserDefOneList(UserDefOneList *obj);
-    void qapi_free_UserDefOne(UserDefOne *obj);
 
     #endif
 
scripts/qapi-types.py +117 -156

394 lines not on this page · raw message

tests/qapi-schema/qapi-schema-test.json +2 -2
index a9e5aab..257b4d4 100644
@@ -39,8 +39,8 @@
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
             'value3' : 'UserDefB' } }
-# FIXME generated struct UserDefFlatUnion has members for direct base
-# UserDefUnionBase, but lacks members for indirect base UserDefZero
+# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit
+# members of indirect base UserDefZero
 
 { 'struct': 'UserDefUnionBase',
   'base': 'UserDefZero',
Signature · 1 line
2.4.3
#33 lore raw

Re: [Qemu-devel] [PATCH RFC v4 10/32] qapi-types: Convert to QAPISchemaVisitor, fixing flat unions

Quoted 14 lines · Markus Armbruster

Fixes flat unions to get the base's base members. Test case is from commit 2fc0043, in qapi-schema-test.json:

---
 docs/qapi-code-gen.txt                  |  51 +++---
 scripts/qapi-types.py                   | 273 ++++++++++++++------------------
 tests/qapi-schema/qapi-schema-test.json |   4 +-
 3 files changed, 144 insertions(+), 184 deletions(-)

Review struct %(name)s { · line 203

Quoted 5 lines
(quoted diff) +1 -1
@@ -231,39 +200,36 @@ struct %(name)s {
     union { /* union tag is @%(c_name)s */
         void *data;
 ''',
-                 c_name=c_name(discriminator or 'kind'))
+                 c_name=c_name(variants.tag_member.name))

See my review on 30/32. If we change tag_member to match the wire spelling of 'type' in patch 2, then this patch needs a slight alteration. (Of course, I've already posted a potential followup patch, currently for use after your series lands, that fixes the C spelling to quit the pointless rename to 'kind'; maybe we should instead fold that in to the early part of this series? If not, it will be one of our earlier followups).

Here's the diff that got the bug in 30/32 fixed, along with changes to the two other files.

scripts/qapi-types.py +2 -1
index 70a57c0..83fc421 100644
@@ -139,11 +139,12 @@ struct %(c_name)s {
     # should not be any data leaks even without a data pointer.  Or, if
     # 'data' is merely added to guarantee we don't have an empty union,
     # shouldn't we enforce that at .json parse time?
+    # FIXME: Use same tag name in C as on the wire
     ret += mcgen('''
     union { /* union tag is @%(c_name)s */
         void *data;
 ''',
-                 c_name=c_name(variants.tag_member.name))
+                 c_name=c_name(variants.tag_name or 'kind'))
 
     for var in variants.variants:
         # TODO ugly special case for simple union
Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#34 lore raw

RFC PATCH v4 11/32 qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

In reply to Markus Armbruster

Fixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.

Patch's effect on visit_type_UserDefFlatUnion():

static void visit_type_UserDefFlatUnion_fields(Visitor *m, UserDefFlatUnion **obj, Error **errp) {
Error *err = NULL;

+ visit_type_int(m, &(*obj)->integer, "integer", &err);
+ if (err) {
+ goto out;
+ }
visit_type_str(m, &(*obj)->string, "string", &err); if (err) {
goto out;

Test cases updated for the bug fix.

Fixes alternates to generate a visitor for their implicit enumeration type. None of them are currently used, obviously. Example:
block-core.json's BlockdevRef now generates visit_type_BlockdevRefKind().

---
 scripts/qapi-visit.py                   | 260 +++++++++++++-------------------
 tests/qapi-schema/qapi-schema-test.json |   3 -
 tests/test-qmp-input-strict.c           |   4 +-
 tests/test-qmp-input-visitor.c          |   4 +-
 4 files changed, 107 insertions(+), 164 deletions(-)
scripts/qapi-visit.py +103 -157
index c493964..a6ee17f 100644
@@ -12,7 +12,6 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 import re
 
@@ -24,13 +23,13 @@ def generate_visit_implicit_struct(type):
         return ''
     implicit_structs_seen.add(type)
     ret = ''
-    if type not in struct_fields_seen:
+    if type.name not in struct_fields_seen:
         # Need a forward declaration
         ret += mcgen('''
 
 static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
 ''',
-                     c_type=type_name(type))
+                     c_type=type.c_name())
 
     ret += mcgen('''
 
@@ -46,7 +45,7 @@ static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error *
     error_propagate(errp, err);
 }
 ''',
-                 c_type=type_name(type))
+                 c_type=type.c_name())
     return ret
 
 def generate_visit_struct_fields(name, members, base = None):
@@ -74,24 +73,24 @@ if (err) {
     goto out;
 }
 ''',
-                     type=type_name(base), c_name=c_name('base'))
+                     type=base.c_name(), c_name=c_name('base'))
 
-    for argname, argentry, optional in parse_args(members):
-        if optional:
+    for memb in members:
+        if memb.optional:
             ret += mcgen('''
 visit_optional(m, &(*obj)->has_%(c_name)s, "%(name)s", &err);
 if (!err && (*obj)->has_%(c_name)s) {
 ''',
-                         c_name=c_name(argname), name=argname)
+                         c_name=c_name(memb.name), name=memb.name)
             push_indent()
 
         ret += mcgen('''
 visit_type_%(type)s(m, &(*obj)->%(c_name)s, "%(name)s", &err);
 ''',
-                     type=type_name(argentry), c_name=c_name(argname),
-                     name=argname)
+                     type=memb.type.c_name(), c_name=c_name(memb.name),
+                     name=memb.name)
 
-        if optional:
+        if memb.optional:
             pop_indent()
             ret += mcgen('''
 }
@@ -136,12 +135,7 @@ def generate_visit_struct_body(name):
 
     return ret
 
-def generate_visit_struct(expr):
-
-    name = expr['struct']
-    members = expr['data']
-    base = expr.get('base')
-
+def gen_visit_struct(name, base, members):
     ret = generate_visit_struct_fields(name, members, base)
 
     ret += mcgen('''
@@ -158,10 +152,10 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 ''')
     return ret
 
-def generate_visit_list(name):
+def gen_visit_list(name, element_type):
     return mcgen('''
 
-void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp)
+void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
     GenericList *i, **prev;
@@ -174,8 +168,8 @@ void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, E
     for (prev = (GenericList **)obj;
          !err && (i = visit_next_list(m, prev, &err)) != NULL;
          prev = &i) {
-        %(name)sList *native_i = (%(name)sList *)i;
-        visit_type_%(name)s(m, &native_i->value, NULL, &err);
+        %(name)s *native_i = (%(name)s *)i;
+        visit_type_%(c_elt_type)s(m, &native_i->value, NULL, &err);
     }
 
     error_propagate(errp, err);
@@ -185,7 +179,8 @@ out:
     error_propagate(errp, err);
 }
 ''',
-                name=type_name(name))
+                 name=c_name(name),
+                 c_elt_type=element_type.c_name())
 
 def generate_visit_enum(name):
     return mcgen('''
@@ -197,7 +192,7 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error
 ''',
                  c_name=c_name(name), name=name)
 
-def generate_visit_alternate(name, members):
+def gen_visit_alternate(name, variants):
     ret = mcgen('''
 
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
@@ -216,25 +211,17 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 ''',
                 name=c_name(name))
 
-    # For alternate, always use the default enum type automatically generated
-    # as name + 'Kind'
-    disc_type = c_name(name) + 'Kind'
-
-    for key in members:
-        assert (members[key] in builtin_types.keys()
-            or find_struct(members[key])
-            or find_union(members[key])
-            or find_enum(members[key])), "Invalid alternate member"
-
-        enum_full_value = c_enum_const(disc_type, key)
+    for var in variants.variants:
+        enum_full_value = c_enum_const(variants.tag_member.type.name,
+                                       var.name)
         ret += mcgen('''
     case %(enum_full_value)s:
         visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, name, &err);
         break;
 ''',
                 enum_full_value = enum_full_value,
-                c_type = type_name(members[key]),
-                c_name = c_name(key))
+                c_type=var.type.c_name(),
+                c_name=c_name(var.name))
 
     ret += mcgen('''
     default:
@@ -251,35 +238,17 @@ out:
 
     return ret
 
-
-def generate_visit_union(expr):
-
-    name = expr['union']
-    members = expr['data']
-
-    base = expr.get('base')
-    discriminator = expr.get('discriminator')
-
-    enum_define = discriminator_find_enum_define(expr)
-    if enum_define:
-        # Use the enum type as discriminator
-        ret = ""
-        disc_type = c_name(enum_define['enum_name'])
-    else:
-        # There will always be a discriminator in the C switch code, by default
-        # it is an enum type generated silently
-        ret = generate_visit_enum(name + 'Kind')
-        disc_type = c_name(name) + 'Kind'
+def gen_visit_union(name, base, variants):
+    ret = ''
 
     if base:
-        assert discriminator
-        base_fields = find_struct(base)['data'].copy()
-        del base_fields[discriminator]
-        ret += generate_visit_struct_fields(name, base_fields)
+        members = [m for m in base.members if m != variants.tag_member]
+        ret += generate_visit_struct_fields(name, members)
 
-    if discriminator:
-        for key in members:
-            ret += generate_visit_implicit_struct(members[key])
+    for var in variants.variants:
+        # TODO ugly special case for simple union
+        if not var.simple_union_type():
+            ret += generate_visit_implicit_struct(var.type)
 
     ret += mcgen('''
 
@@ -304,41 +273,41 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error
 ''',
                      name=c_name(name))
 
-    if not discriminator:
-        tag = 'kind'
-        disc_key = "type"
-    else:
-        tag = discriminator
-        disc_key = discriminator
+    disc_key = variants.tag_member.name
+    if not variants.tag_name:
+        # we pointlessly use a different key for simple unions
+        disc_key = 'type'
     ret += mcgen('''
-        visit_type_%(disc_type)s(m, &(*obj)->%(c_tag)s, "%(disc_key)s", &err);
+        visit_type_%(disc_type)s(m, &(*obj)->%(c_name)s, "%(disc_key)s", &err);
         if (err) {
             goto out_obj;
         }
         if (!visit_start_union(m, !!(*obj)->data, &err) || err) {
             goto out_obj;
         }
-        switch ((*obj)->%(c_tag)s) {
+        switch ((*obj)->%(c_name)s) {
 ''',
-                 disc_type = disc_type,
-                 c_tag=c_name(tag),
+                 disc_type=variants.tag_member.type.c_name(),
+                 c_name=c_name(variants.tag_member.name),
                  disc_key = disc_key)
 
-    for key in members:
-        if not discriminator:
+    for var in variants.variants:
+        # TODO ugly special case for simple union
+        simple_union_type = var.simple_union_type()
+        if simple_union_type:
             fmt = 'visit_type_%(c_type)s(m, &(*obj)->%(c_name)s, "data", &err);'
         else:
             fmt = 'visit_type_implicit_%(c_type)s(m, &(*obj)->%(c_name)s, &err);'
 
-        enum_full_value = c_enum_const(disc_type, key)
+        enum_full_value = c_enum_const(variants.tag_member.type.name, var.name)
         ret += mcgen('''
         case %(enum_full_value)s:
             ''' + fmt + '''
             break;
 ''',
                 enum_full_value = enum_full_value,
-                c_type=type_name(members[key]),
-                c_name=c_name(key))
+                c_type=(simple_union_type or var.type).c_name(),
+                c_name=c_name(var.name))
 
     ret += mcgen('''
         default:
@@ -359,38 +328,61 @@ out:
 
     return ret
 
-def generate_declaration(name, builtin_type=False):
-    ret = ""
-    if not builtin_type:
-        name = c_name(name)
-        ret += mcgen('''
-
-void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp);
-''',
-                     name=name)
-
-    ret += mcgen('''
-void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp);
-''',
-                 name=name)
-
-    return ret
-
-def generate_enum_declaration(name):
-    ret = mcgen('''
-void visit_type_%(name)sList(Visitor *m, %(name)sList **obj, const char *name, Error **errp);
-''',
-                name=c_name(name))
-
-    return ret
-
-def generate_decl_enum(name):
+def gen_visit_decl(name, scalar=False):
+    c_type = c_name(name) + ' *'
+    if not scalar:
+        c_type += '*'
     return mcgen('''
-
-void visit_type_%(name)s(Visitor *m, %(name)s *obj, const char *name, Error **errp);
+void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
 ''',
-                 name=c_name(name))
+                 c_name=c_name(name), c_type=c_type)
 
+class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.btin = None
+    def visit_begin(self, schema):
+        self.decl = ''
+        self.defn = ''
+        self.btin = guardstart('QAPI_VISIT_BUILTIN')
+    def visit_end(self):
+        # To avoid header dependency hell, we always generate
+        # declarations for built-in types in our header files and
+        # simply guard them.  See also do_builtins (command line
+        # option -b).
+        self.btin += guardend('QAPI_VISIT_BUILTIN')
+        self.decl = self.btin + self.decl
+        self.btin = None
+    def visit_enum_type(self, name, info, values):
+        self.decl += gen_visit_decl(name, scalar=True)
+        self.defn += generate_visit_enum(name)
+    def visit_array_type(self, name, info, element_type):
+        decl = gen_visit_decl(name)
+        defn = gen_visit_list(name, element_type)
+        if isinstance(element_type, QAPISchemaBuiltinType):
+            self.btin += decl
+            if do_builtins:
+                self.defn += defn
+        else:
+            self.decl += decl
+            self.defn += defn
+    def visit_object_type(self, name, info, base, members, variants):
+        if info:
+            self.decl += gen_visit_decl(name)
+            if variants:
+                assert not members # not implemented
+                self.defn += gen_visit_union(name, base, variants)
+            else:
+                self.defn += gen_visit_struct(name, base, members)
+    def visit_alternate_type(self, name, info, variants):
+        self.decl += gen_visit_decl(name)
+        self.defn += gen_visit_alternate(name, variants)
+
+# If you link code generated from multiple schemata, you want only one
+# instance of the code for built-in types.  Generate it only when
+# do_builtins, enabled by command line option -b.  See also
+# QAPISchemaGenVisitVisitor.visit_end().
 do_builtins = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
@@ -446,56 +438,10 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = QAPISchema(input_file).get_exprs()
-
-# to avoid header dependency hell, we always generate declarations
-# for built-in types in our header files and simply guard them
-fdecl.write(guardstart("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
-for typename in builtin_types.keys():
-    fdecl.write(generate_declaration(typename, builtin_type=True))
-fdecl.write(guardend("QAPI_VISIT_BUILTIN_VISITOR_DECL"))
-
-# ...this doesn't work for cases where we link in multiple objects that
-# have the functions defined, so we use -b option to provide control
-# over these cases
-if do_builtins:
-    for typename in builtin_types.keys():
-        fdef.write(generate_visit_list(typename))
-
-for expr in exprs:
-    if expr.has_key('struct'):
-        ret = generate_visit_struct(expr)
-        ret += generate_visit_list(expr['struct'])
-        fdef.write(ret)
-
-        ret = generate_declaration(expr['struct'])
-        fdecl.write(ret)
-    elif expr.has_key('union'):
-        ret = generate_visit_union(expr)
-        ret += generate_visit_list(expr['union'])
-        fdef.write(ret)
-
-        enum_define = discriminator_find_enum_define(expr)
-        ret = ""
-        if not enum_define:
-            ret = generate_decl_enum('%sKind' % expr['union'])
-        ret += generate_declaration(expr['union'])
-        fdecl.write(ret)
-    elif expr.has_key('alternate'):
-        ret = generate_visit_alternate(expr['alternate'], expr['data'])
-        ret += generate_visit_list(expr['alternate'])
-        fdef.write(ret)
-
-        ret = generate_decl_enum('%sKind' % expr['alternate'])
-        ret += generate_declaration(expr['alternate'])
-        fdecl.write(ret)
-    elif expr.has_key('enum'):
-        ret = generate_visit_list(expr['enum'])
-        ret += generate_visit_enum(expr['enum'])
-        fdef.write(ret)
-
-        ret = generate_decl_enum(expr['enum'])
-        ret += generate_enum_declaration(expr['enum'])
-        fdecl.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenVisitVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
tests/qapi-schema/qapi-schema-test.json +0 -3
index 257b4d4..b182174 100644
@@ -39,8 +39,6 @@
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
             'value3' : 'UserDefB' } }
-# FIXME generated visit_type_UserDefFlatUnion_fields() fails to visit
-# members of indirect base UserDefZero
 
 { 'struct': 'UserDefUnionBase',
   'base': 'UserDefZero',
@@ -57,7 +55,6 @@
 
 { 'alternate': 'UserDefAlternate',
   'data': { 'uda': 'UserDefA', 's': 'str', 'i': 'int' } }
-# FIXME only a declaration of visit_type_UserDefAlternateKind() generated
 
 { 'struct': 'UserDefC',
   'data': { 'string1': 'str', 'string2': 'str' } }
tests/test-qmp-input-strict.c +2 -2
index 68f855b..a2ae786 100644
@@ -167,9 +167,9 @@ static void test_validate_union_flat(TestInputVisitorData *data,
 
     v = validate_test_init(data,
                            "{ 'enum1': 'value1', "
+                           "'integer': 41, "
                            "'string': 'str', "
                            "'boolean': true }");
-    /* TODO when generator bug is fixed, add 'integer': 41 */
 
     visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
     g_assert(!err);
@@ -272,7 +272,7 @@ static void test_validate_fail_union_flat_no_discrim(TestInputVisitorData *data,
     Visitor *v;
 
     /* test situation where discriminator field ('enum1' here) is missing */
-    v = validate_test_init(data, "{ 'string': 'c', 'string1': 'd', 'string2': 'e' }");
+    v = validate_test_init(data, "{ 'integer': 42, 'string': 'c', 'string1': 'd', 'string2': 'e' }");
 
     visit_type_UserDefFlatUnion2(v, &tmp, NULL, &err);
     g_assert(err);
tests/test-qmp-input-visitor.c +2 -2
index a5cfefa..508c11a 100644
@@ -307,15 +307,15 @@ static void test_visitor_in_union_flat(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data,
                                 "{ 'enum1': 'value1', "
+                                "'integer': 41, "
                                 "'string': 'str', "
                                 "'boolean': true }");
-    /* TODO when generator bug is fixed, add 'integer': 41 */
 
     visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
     g_assert(err == NULL);
     g_assert_cmpint(tmp->enum1, ==, ENUM_ONE_VALUE1);
     g_assert_cmpstr(tmp->string, ==, "str");
-    /* TODO g_assert_cmpint(tmp->integer, ==, 41); */
+    g_assert_cmpint(tmp->integer, ==, 41);
     g_assert_cmpint(tmp->value1->boolean, ==, true);
     qapi_free_UserDefFlatUnion(tmp);
 }
Signature · 1 line
2.4.3
#35 lore raw

Re: [Qemu-devel] [PATCH RFC v4 11/32] qapi-visit: Convert to QAPISchemaVisitor, fixing bugs

Quoted 31 lines · Markus Armbruster

Fixes flat unions to visit the base's base members (the previous commit merely added them to the struct). Same test case.

---
 scripts/qapi-visit.py                   | 260 +++++++++++++-------------------
 tests/qapi-schema/qapi-schema-test.json |   3 -
 tests/test-qmp-input-strict.c           |   4 +-
 tests/test-qmp-input-visitor.c          |   4 +-
 4 files changed, 107 insertions(+), 164 deletions(-)
(quoted diff) +7 -1
+def gen_visit_union(name, base, variants):
+    ret = ''
 
     if base:
 
+    disc_key = variants.tag_member.name
+    if not variants.tag_name:
+        # we pointlessly use a different key for simple unions
+        disc_key = 'type'
     ret += mcgen('''
-        visit_type_%(disc_type)s(m, &(*obj)->%(c_tag)s, "%(disc_key)s", &err);
+        visit_type_%(disc_type)s(m, &(*obj)->%(c_name)s, "%(disc_key)s", &err);
         if (err) {
             goto out_obj;
         }
         if (!visit_start_union(m, !!(*obj)->data, &err) || err) {

Review

Quoted 10 lines
(quoted diff) +3 -3
             goto out_obj;
         }
-        switch ((*obj)->%(c_tag)s) {
+        switch ((*obj)->%(c_name)s) {
 ''',
-                 disc_type = disc_type,
-                 c_tag=c_name(tag),
+                 disc_type=variants.tag_member.type.c_name(),
+                 c_name=c_name(variants.tag_member.name),
                  disc_key = disc_key)

See my comments on 30/32, and patches in 2 and 10. This is the final place that cannot refer to tag_member.name until after we unify the C and wire spelling.

scripts/qapi-visit.py +0 -0
index 6819866..468080e 100644
@@ -284,7 +284,7 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s

**obj, const char *name, Error

         switch ((*obj)->%(c_name)s) {

''',
c_type=variants.tag_member.type.c_name(),
- c_name=c_name(variants.tag_member.name),
+ c_name=c_name(variants.tag_name or 'kind'),
name=tag_key)

for var in variants.variants:

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#36 lore raw

RFC PATCH v4 12/32 qapi-commands: Convert to QAPISchemaVisitor

In reply to Markus Armbruster

Output unchanged apart from reordering and white-space.

---
 scripts/qapi-commands.py | 154 ++++++++++++++++++++++++++---------------------
 scripts/qapi.py          |   2 +-
 2 files changed, 86 insertions(+), 70 deletions(-)
scripts/qapi-commands.py +85 -69
index 12bdc4c..e95fbb8 100644
@@ -12,21 +12,22 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 import re
 
 def generate_command_decl(name, args, ret_type):
     arglist=""
-    for argname, argtype, optional in parse_args(args):
-        argtype = c_type(argtype, is_param=True)
-        if optional:
-            arglist += "bool has_%s, " % c_name(argname)
-        arglist += "%s %s, " % (argtype, c_name(argname))
+    if args:
+        for memb in args.members:
+            argtype = memb.type.c_type(is_param=True)
+            if memb.optional:
+                arglist += "bool has_%s, " % c_name(memb.name)
+            arglist += "%s %s, " % (argtype, c_name(memb.name))
     return mcgen('''
 %(ret_type)s qmp_%(name)s(%(args)sError **errp);
 ''',
-                 ret_type=c_type(ret_type), name=c_name(name),
+                 ret_type=(ret_type and ret_type.c_type()) or 'void',
+                 name=c_name(name),
                  args=arglist)
 
 def gen_err_check(err):
@@ -45,10 +46,11 @@ def gen_sync_call(name, args, ret_type):
     retval=""
     if ret_type:
         retval = "retval = "
-    for argname, argtype, optional in parse_args(args):
-        if optional:
-            arglist += "has_%s, " % c_name(argname)
-        arglist += "%s, " % (c_name(argname))
+    if args:
+        for memb in args.members:
+            if memb.optional:
+                arglist += "has_%s, " % c_name(memb.name)
+            arglist += "%s, " % c_name(memb.name)
     push_indent()
     ret = mcgen('''
 %(retval)sqmp_%(name)s(%(args)s&local_err);
@@ -68,7 +70,7 @@ def gen_visitor_input_containers_decl(args):
     ret = ""
 
     push_indent()
-    if len(args) > 0:
+    if args:
         ret += mcgen('''
 QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
 QapiDeallocVisitor *md;
@@ -81,22 +83,26 @@ Visitor *v;
 def gen_visitor_input_vars_decl(args):
     ret = ""
     push_indent()
-    for argname, argtype, optional in parse_args(args):
-        if optional:
-            ret += mcgen('''
+
+    if args:
+        for memb in args.members:
+            if memb.optional:
+                ret += mcgen('''
 bool has_%(argname)s = false;
 ''',
-                         argname=c_name(argname))
-        if is_c_ptr(argtype):
-            ret += mcgen('''
+                             argname=c_name(memb.name))
+            if is_c_ptr(memb.type.c_type()):
+                ret += mcgen('''
 %(argtype)s %(argname)s = NULL;
 ''',
-                         argname=c_name(argname), argtype=c_type(argtype))
-        else:
-            ret += mcgen('''
+                             argname=c_name(memb.name),
+                             argtype=memb.type.c_type())
+            else:
+                ret += mcgen('''
 %(argtype)s %(argname)s = {0};
 ''',
-                         argname=c_name(argname), argtype=c_type(argtype))
+                             argname=c_name(memb.name),
+                             argtype=memb.type.c_type())
 
     pop_indent()
     return ret
@@ -106,7 +112,7 @@ def gen_visitor_input_block(args, dealloc=False):
     errparg = '&local_err'
     errarg = 'local_err'
 
-    if len(args) == 0:
+    if not args:
         return ret
 
     push_indent()
@@ -124,25 +130,26 @@ v = qapi_dealloc_get_visitor(md);
 v = qmp_input_get_visitor(mi);
 ''')
 
-    for argname, argtype, optional in parse_args(args):
-        if optional:
+    for memb in args.members:
+        if memb.optional:
             ret += mcgen('''
 visit_optional(v, &has_%(c_name)s, "%(name)s", %(errp)s);
 ''',
-                         c_name=c_name(argname), name=argname, errp=errparg)
+                         c_name=c_name(memb.name), name=memb.name,
+                         errp=errparg)
             ret += gen_err_check(errarg)
             ret += mcgen('''
 if (has_%(c_name)s) {
 ''',
-                         c_name=c_name(argname))
+                         c_name=c_name(memb.name))
             push_indent()
         ret += mcgen('''
 visit_type_%(visitor)s(v, &%(c_name)s, "%(name)s", %(errp)s);
 ''',
-                     c_name=c_name(argname), name=argname, argtype=argtype,
-                     visitor=type_name(argtype), errp=errparg)
+                     c_name=c_name(memb.name), name=memb.name,
+                     visitor=memb.type.c_name(), errp=errparg)
         ret += gen_err_check(errarg)
-        if optional:
+        if memb.optional:
             pop_indent()
             ret += mcgen('''
 }
@@ -160,6 +167,7 @@ def gen_marshal_output(name, ret_type):
         return ""
 
     ret = mcgen('''
+
 static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_out, Error **errp)
 {
     Error *local_err = NULL;
@@ -183,8 +191,8 @@ out:
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                c_ret_type=c_type(ret_type), c_name=c_name(name),
-                visitor=type_name(ret_type))
+                c_ret_type=ret_type.c_type(), c_name=c_name(name),
+                visitor=ret_type.c_name())
 
     return ret
 
@@ -198,6 +206,7 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
     hdr = gen_marshal_input_decl(name, middle_mode)
 
     ret = mcgen('''
+
 %(header)s
 {
     Error *local_err = NULL;
@@ -208,9 +217,9 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
         ret += mcgen('''
     %(c_type)s retval;
 ''',
-                     c_type=c_type(ret_type))
+                     c_type=ret_type.c_type())
 
-    if len(args) > 0:
+    if args:
         ret += gen_visitor_input_containers_decl(args)
         ret += gen_visitor_input_vars_decl(args) + '\n'
         ret += gen_visitor_input_block(args) + '\n'
@@ -237,21 +246,23 @@ out:
 ''')
     return ret
 
-def gen_registry(commands):
-    registry=""
+def gen_register_command(name, success_response):
     push_indent()
-    for cmd in commands:
-        options = 'QCO_NO_OPTIONS'
-        if not cmd.get('success-response', True):
-            options = 'QCO_NO_SUCCESS_RESP'
+    options = 'QCO_NO_OPTIONS'
+    if not success_response:
+        options = 'QCO_NO_SUCCESS_RESP'
 
-        registry += mcgen('''
+    ret = mcgen('''
 qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
 ''',
-                     name=cmd['command'], c_name=c_name(cmd['command']),
+                     name=name, c_name=c_name(name),
                      opts=options)
     pop_indent()
+    return ret
+
+def gen_registry(registry):
     ret = mcgen('''
+
 static void qmp_init_marshal(void)
 {
 ''')
@@ -263,6 +274,32 @@ qapi_init(qmp_init_marshal);
 ''')
     return ret
 
+class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.regy = None
+    def visit_begin(self, schema):
+        self.decl = ''
+        self.defn = ''
+        self.regy = ''
+    def visit_end(self):
+        if not middle_mode:
+            self.defn += gen_registry(self.regy)
+        self.regy = None
+    def visit_command(self, name, info, arg_type, ret_type,
+                      gen, success_response):
+        if not gen:
+            return
+        self.decl += generate_command_decl(name, arg_type, ret_type)
+        if ret_type:
+            self.defn += gen_marshal_output(name, ret_type)
+        if middle_mode:
+            self.decl += gen_marshal_input_decl(name, middle_mode) + ';\n'
+        self.defn += gen_marshal_input(name, arg_type, ret_type, middle_mode)
+        if not middle_mode:
+            self.regy += gen_register_command(name, success_response)
+
 middle_mode = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
@@ -272,10 +309,6 @@ for o, a in opts:
     if o in ("-m", "--middle"):
         middle_mode = True
 
-exprs = QAPISchema(input_file).get_exprs()
-commands = filter(lambda expr: expr.has_key('command'), exprs)
-commands = filter(lambda expr: not expr.has_key('gen'), commands)
-
 c_comment = '''
 /*
  * schema-defined QMP->QAPI command dispatch
@@ -331,29 +364,12 @@ fdecl.write(mcgen('''
 #include "qapi/error.h"
 
 ''',
-                 prefix=prefix))
+                  prefix=prefix))
 
-for cmd in commands:
-    arglist = []
-    ret_type = None
-    if cmd.has_key('data'):
-        arglist = cmd['data']
-    if cmd.has_key('returns'):
-        ret_type = cmd['returns']
-    ret = generate_command_decl(cmd['command'], arglist, ret_type)
-    fdecl.write(ret)
-    if ret_type:
-        ret = gen_marshal_output(cmd['command'], ret_type) + "\n"
-        fdef.write(ret)
-
-    if middle_mode:
-        fdecl.write('%s;\n' % gen_marshal_input_decl(cmd['command'], middle_mode))
-
-    ret = gen_marshal_input(cmd['command'], arglist, ret_type, middle_mode) + "\n"
-    fdef.write(ret)
-
-if not middle_mode:
-    ret = gen_registry(commands)
-    fdef.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenCommandVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
scripts/qapi.py +1 -1
index 3e7c154..923c3ac 100644
@@ -1370,7 +1370,7 @@ def c_type(value, is_param=False):
         return c_name(value) + pointer_suffix
 
 def is_c_ptr(value):
-    return c_type(value).endswith(pointer_suffix)
+    return value.endswith(pointer_suffix)
 
 def genindent(count):
     ret = ""
Signature · 1 line
2.4.3
#37 lore raw

RFC PATCH v4 13/32 qapi: De-duplicate enum code generation

In reply to Markus Armbruster

Duplicated in commit 21cd70d. Yes, we can't import qapi-types, but that's no excuse. Move the helpers from qapi-types.py to qapi.py, and replace the duplicates in qapi-event.py.

The generated event enumeration type's lookup table becomes const-correct (see commit 2e4450f), and uses explicit indexes instead of relying on order (see commit 912ae9c).

---
 docs/qapi-code-gen.txt |  9 ++++---
 scripts/qapi-event.py  | 67 +++-----------------------------------------------
 scripts/qapi-types.py  | 55 -----------------------------------------
 scripts/qapi.py        | 55 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 64 insertions(+), 122 deletions(-)
docs/qapi-code-gen.txt +5 -4
index 710defc..7c500b6 100644
@@ -821,9 +821,9 @@ Example:
         QDECREF(qmp);
     }
 
-    const char *example_QAPIEvent_lookup[] = {
-        "MY_EVENT",
-        NULL,
+    const char *const example_QAPIEvent_lookup[] = {
+        [EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
+        [EXAMPLE_QAPI_EVENT_MAX] = NULL,
     };
     $ cat qapi-generated/example-qapi-event.h
 [Uninteresting stuff omitted...]
@@ -838,10 +838,11 @@ Example:
 
     void qapi_event_send_my_event(Error **errp);
 
-    extern const char *example_QAPIEvent_lookup[];
     typedef enum example_QAPIEvent {
         EXAMPLE_QAPI_EVENT_MY_EVENT = 0,
         EXAMPLE_QAPI_EVENT_MAX = 1,
     } example_QAPIEvent;
 
+    extern const char *const example_QAPIEvent_lookup[];
+
     #endif
scripts/qapi-event.py +4 -63

87 lines not on this page · raw message

scripts/qapi-types.py +0 -55

61 lines not on this page · raw message

scripts/qapi.py +55 -0

61 lines not on this page · raw message

Signature · 1 line
2.4.3
#38 lore raw

RFC PATCH v4 14/32 qapi-event: Eliminate global variable event_enum_value

In reply to Markus Armbruster
---
 scripts/qapi-event.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi-event.py +1 -2
index aed45d6..537da17 100644
@@ -137,7 +137,7 @@ def generate_event_implement(api_name, event_name, params):
     emit(%(event_enum_value)s, qmp, &local_err);
 
 """,
-                 event_enum_value = event_enum_value)
+                 event_enum_value = c_enum_const(event_enum_name, event_name))
 
     # step 5: clean up
     if params:
@@ -223,7 +223,6 @@ for expr in exprs:
         fdecl.write(ret)
 
         # We need an enum value per event
-        event_enum_value = c_enum_const(event_enum_name, event_name)
         ret = generate_event_implement(api_name, event_name, params)
         fdef.write(ret)
 
Signature · 1 line
2.4.3
#39 lore raw

RFC PATCH v4 15/32 qapi-event: Convert to QAPISchemaVisitor, fixing data with base

In reply to Markus Armbruster

Fixes events whose data is struct with base to include the struct's base members. Test case is qapi-schema-test.json's event __org.qemu_x-command:

    { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }

    { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member2': 'str' } }

    { 'struct': '__org.qemu_x-Base',
      'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }

Patch's effect on generated qapi_event_send___org_qemu_x_event():

    -void qapi_event_send___org_qemu_x_event(const char *__org_qemu_x_member2,
    +void qapi_event_send___org_qemu_x_event(__org_qemu_x_Enum __org_qemu_x_member1,
    +                                        const char *__org_qemu_x_member2,
                                             Error **errp)
     {
         QDict *qmp;
    @@ -224,6 +225,10 @@ void qapi_event_send___org_qemu_x_event(
             goto clean;
         }

    +    visit_type___org_qemu_x_Enum(v, &__org_qemu_x_member1, "__org.qemu_x-member1", &local_err);
    +    if (local_err) {
    +        goto clean;
    +    }
         visit_type_str(v, (char **)&__org_qemu_x_member2, "__org.qemu_x-member2", &local_err);
         if (local_err) {
             goto clean;
---
 scripts/qapi-event.py                   | 84 ++++++++++++++++-----------------
 tests/qapi-schema/qapi-schema-test.json |  3 --
 2 files changed, 42 insertions(+), 45 deletions(-)
scripts/qapi-event.py +42 -42
index 537da17..7525bab 100644
@@ -2,14 +2,15 @@
 # QAPI event generator
 #
 # Copyright (c) 2014 Wenchao Xia
+# Copyright (c) 2015 Red Hat Inc.
 #
 # Authors:
 #  Wenchao Xia <[email protected]>
+#  Markus Armbruster <[email protected]>
 #
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-from ordereddict import OrderedDict
 from qapi import *
 
 def _generate_event_api_name(event_name, params):
@@ -17,13 +18,13 @@ def _generate_event_api_name(event_name, params):
     l = len(api_name)
 
     if params:
-        for argname, argentry, optional in parse_args(params):
-            if optional:
-                api_name += "bool has_%s,\n" % c_name(argname)
+        for m in params.members:
+            if m.optional:
+                api_name += "bool has_%s,\n" % c_name(m.name)
                 api_name += "".ljust(l)
 
-            api_name += "%s %s,\n" % (c_type(argentry, is_param=True),
-                                      c_name(argname))
+            api_name += "%s %s,\n" % (m.type.c_type(is_param=True),
+                                      c_name(m.name))
             api_name += "".ljust(l)
 
     api_name += "Error **errp)"
@@ -51,7 +52,7 @@ def generate_event_implement(api_name, event_name, params):
 """,
                 api_name = api_name)
 
-    if params:
+    if params and params.members:
         ret += mcgen("""
     QmpOutputVisitor *qov;
     Visitor *v;
@@ -72,7 +73,7 @@ def generate_event_implement(api_name, event_name, params):
                  event_name = event_name)
 
     # step 3: visit the params if params != None
-    if params:
+    if params and params.members:
         ret += mcgen("""
     qov = qmp_output_visitor_new();
     g_assert(qov);
@@ -89,15 +90,15 @@ def generate_event_implement(api_name, event_name, params):
 """,
                 event_name = event_name)
 
-        for argname, argentry, optional in parse_args(params):
-            if optional:
+        for memb in params.members:
+            if memb.optional:
                 ret += mcgen("""
     if (has_%(var)s) {
 """,
-                             var = c_name(argname))
+                             var=c_name(memb.name))
                 push_indent()
 
-            if argentry == "str":
+            if memb.type.name == "str":
                 var_type = "(char **)"
             else:
                 var_type = ""
@@ -109,11 +110,11 @@ def generate_event_implement(api_name, event_name, params):
     }
 """,
                          var_type = var_type,
-                         var = c_name(argname),
-                         type = type_name(argentry),
-                         name = argname)
+                         var=c_name(memb.name),
+                         type=memb.type.c_name(),
+                         name=memb.name)
 
-            if optional:
+            if memb.optional:
                 pop_indent()
                 ret += mcgen("""
     }
@@ -140,7 +141,7 @@ def generate_event_implement(api_name, event_name, params):
                  event_enum_value = c_enum_const(event_enum_name, event_name))
 
     # step 5: clean up
-    if params:
+    if params and params.members:
         ret += mcgen("""
  clean:
     qmp_output_visitor_cleanup(qov);
@@ -153,6 +154,25 @@ def generate_event_implement(api_name, event_name, params):
 
     return ret
 
+class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.decl = None
+        self.defn = None
+        self.event_names = None
+    def visit_begin(self, schema):
+        self.decl = ''
+        self.defn = ''
+        self.event_names = []
+    def visit_end(self):
+        self.decl += generate_enum(event_enum_name, self.event_names)
+        self.defn += generate_enum_lookup(event_enum_name, self.event_names)
+        self.event_names = None
+    def visit_event(self, name, info, arg_type):
+        api_name = _generate_event_api_name(name, arg_type)
+        self.decl += generate_event_declaration(api_name)
+        self.defn += generate_event_implement(api_name, name, arg_type)
+        self.event_names.append(name)
+
 (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
 
 c_comment = '''
@@ -206,32 +226,12 @@ fdecl.write(mcgen('''
 ''',
                   prefix=prefix))
 
-exprs = QAPISchema(input_file).get_exprs()
-
 event_enum_name = c_name(prefix + "QAPIEvent", protect=False)
-event_names = []
 
-for expr in exprs:
-    if expr.has_key('event'):
-        event_name = expr['event']
-        params = expr.get('data')
-        if params and len(params) == 0:
-            params = None
-
-        api_name = _generate_event_api_name(event_name, params)
-        ret = generate_event_declaration(api_name)
-        fdecl.write(ret)
-
-        # We need an enum value per event
-        ret = generate_event_implement(api_name, event_name, params)
-        fdef.write(ret)
-
-        # Record it, and generate enum later
-        event_names.append(event_name)
-
-ret = generate_enum(event_enum_name, event_names)
-fdecl.write(ret)
-ret = generate_enum_lookup(event_enum_name, event_names)
-fdef.write(ret)
+schema = QAPISchema(input_file)
+gen = QAPISchemaGenEventVisitor()
+schema.visit(gen)
+fdef.write(gen.defn)
+fdecl.write(gen.decl)
 
 close_output(fdef, fdecl)
tests/qapi-schema/qapi-schema-test.json +0 -3
index b182174..90b4740 100644
@@ -126,9 +126,6 @@
 { 'alternate': '__org.qemu_x-Alt',
   'data': { '__org.qemu_x-branch': 'str', 'b': '__org.qemu_x-Base' } }
 { 'event': '__ORG.QEMU_X-EVENT', 'data': '__org.qemu_x-Struct' }
-# FIXME generated qapi_event_send___org_qemu_x_event() has only a
-# parameter for data's member __org_qemu_x_member2, none for its base
-# __org.qemu_x-Base's member __org_qemu_x_member1
 { 'command': '__org.qemu_x-command',
   'data': { 'a': ['__org.qemu_x-Enum'], 'b': ['__org.qemu_x-Struct'],
             'c': '__org.qemu_x-Union2', 'd': '__org.qemu_x-Alt' },
Signature · 1 line
2.4.3
#40 lore raw

RFC PATCH v4 16/32 qapi: Generate comments to simplify splitting for review

In reply to Markus Armbruster

The effect of the previous few patches on generated files is hard to review, because stuff gets generated in different order, which renders diffs of the generated files useless.

To get reviewable diffs, we need to split the generated files into suitable parts: put every top-level declaration in a file named like the thing declared, so we can diff top-level declarations regardless of their order in the generated files.

Since I don't feel like parsing C, simply generate a // comment identifying the declaration right before each top-level declaration. This lets us split with a simple shell loop:

mkdir -p n
for i in q*-{commands,marshal,event,types,visit}.[ch] do
csplit -n 4 -s "$i" '/^\/\//' '{*}'
for j in xx*
do
read h t <$j
[ "$h" = "//" ] || t=""
mv $j n/"$i-${j#xx}-${t/ /-}"
done
done

Splits each file F into F-NUMB-ID, where NUMB counts up from 0001, and ID comes from the // comment.

To check the split's sane, we can run

for i in q*-{commands,marshal,event,types,visit}.[ch] do cat n/$i-* | diff $i -
done

We got a commit similar to this one right before the patches we want to check. If you have that commit's split files in directory o, and this commit's split files in directory n, you can diff them with this loop:

    for i in `(ls o; ls n) | sed -n 's/-[0-9][0-9][0-9][0-9]-/-*-/p' | sort | uniq`
    do
        diff -uBp o/$i n/$i
    done

Complete annotated output:

* QAPIEvent_lookup[] and enum QAPIEvent

Innocent changes due to "De-duplicate enum code generation".

--- o/qapi-event.c-0037-def-QAPIEvent_lookup 2015-08-04 12:47:16.479980075 +0200
+++ n/qapi-event.c-0037-def-QAPIEvent_lookup 2015-08-04 13:02:46.336626925 +0200
@@ -1,39 +1,39 @@
// def QAPIEvent_lookup
-const char *QAPIEvent_lookup[] = {
- "BLOCK_IMAGE_CORRUPTED",
- "BLOCK_IO_ERROR",
- "BLOCK_JOB_COMPLETED",
- "BLOCK_JOB_CANCELLED",
- "BLOCK_JOB_ERROR",
- "BLOCK_JOB_READY",
- "BLOCK_WRITE_THRESHOLD",
- "DEVICE_TRAY_MOVED",
- "SHUTDOWN",
- "POWERDOWN",
- "RESET",
- "STOP",
- "RESUME",
- "SUSPEND",
- "SUSPEND_DISK",
- "WAKEUP",
- "RTC_CHANGE",
- "WATCHDOG",
- "DEVICE_DELETED",
- "NIC_RX_FILTER_CHANGED",
- "VNC_CONNECTED",
- "VNC_INITIALIZED",
- "VNC_DISCONNECTED",
- "SPICE_CONNECTED",
- "SPICE_INITIALIZED",
- "SPICE_DISCONNECTED",
- "SPICE_MIGRATE_COMPLETED",
- "MIGRATION",
- "ACPI_DEVICE_OST",
- "BALLOON_CHANGE",
- "GUEST_PANICKED",
- "QUORUM_FAILURE",
- "QUORUM_REPORT_BAD",
- "VSERPORT_CHANGE",
- "MEM_UNPLUG_ERROR",
- NULL,
+const char *const QAPIEvent_lookup[] = {
+ [QAPI_EVENT_ACPI_DEVICE_OST] = "ACPI_DEVICE_OST",
+ [QAPI_EVENT_BALLOON_CHANGE] = "BALLOON_CHANGE",
+ [QAPI_EVENT_BLOCK_IMAGE_CORRUPTED] = "BLOCK_IMAGE_CORRUPTED",
+ [QAPI_EVENT_BLOCK_IO_ERROR] = "BLOCK_IO_ERROR",
+ [QAPI_EVENT_BLOCK_JOB_CANCELLED] = "BLOCK_JOB_CANCELLED",
+ [QAPI_EVENT_BLOCK_JOB_COMPLETED] = "BLOCK_JOB_COMPLETED",
+ [QAPI_EVENT_BLOCK_JOB_ERROR] = "BLOCK_JOB_ERROR",
+ [QAPI_EVENT_BLOCK_JOB_READY] = "BLOCK_JOB_READY",
+ [QAPI_EVENT_BLOCK_WRITE_THRESHOLD] = "BLOCK_WRITE_THRESHOLD",
+ [QAPI_EVENT_DEVICE_DELETED] = "DEVICE_DELETED",
+ [QAPI_EVENT_DEVICE_TRAY_MOVED] = "DEVICE_TRAY_MOVED",
+ [QAPI_EVENT_GUEST_PANICKED] = "GUEST_PANICKED",
+ [QAPI_EVENT_MEM_UNPLUG_ERROR] = "MEM_UNPLUG_ERROR",
+ [QAPI_EVENT_MIGRATION] = "MIGRATION",
+ [QAPI_EVENT_NIC_RX_FILTER_CHANGED] = "NIC_RX_FILTER_CHANGED",
+ [QAPI_EVENT_POWERDOWN] = "POWERDOWN",
+ [QAPI_EVENT_QUORUM_FAILURE] = "QUORUM_FAILURE",
+ [QAPI_EVENT_QUORUM_REPORT_BAD] = "QUORUM_REPORT_BAD",
+ [QAPI_EVENT_RESET] = "RESET",
+ [QAPI_EVENT_RESUME] = "RESUME",
+ [QAPI_EVENT_RTC_CHANGE] = "RTC_CHANGE",
+ [QAPI_EVENT_SHUTDOWN] = "SHUTDOWN",
+ [QAPI_EVENT_SPICE_CONNECTED] = "SPICE_CONNECTED",
+ [QAPI_EVENT_SPICE_DISCONNECTED] = "SPICE_DISCONNECTED",
+ [QAPI_EVENT_SPICE_INITIALIZED] = "SPICE_INITIALIZED",
+ [QAPI_EVENT_SPICE_MIGRATE_COMPLETED] = "SPICE_MIGRATE_COMPLETED",
+ [QAPI_EVENT_STOP] = "STOP",
+ [QAPI_EVENT_SUSPEND] = "SUSPEND",
+ [QAPI_EVENT_SUSPEND_DISK] = "SUSPEND_DISK",
+ [QAPI_EVENT_VNC_CONNECTED] = "VNC_CONNECTED",
+ [QAPI_EVENT_VNC_DISCONNECTED] = "VNC_DISCONNECTED",
+ [QAPI_EVENT_VNC_INITIALIZED] = "VNC_INITIALIZED",
+ [QAPI_EVENT_VSERPORT_CHANGE] = "VSERPORT_CHANGE",
+ [QAPI_EVENT_WAKEUP] = "WAKEUP",
+ [QAPI_EVENT_WATCHDOG] = "WATCHDOG",
+ [QAPI_EVENT_MAX] = NULL,
};
--- o/qapi-event.h-0037-decl-QAPIEvent_lookup 2015-08-04 12:47:16.480980103 +0200
+++ n/qapi-event.h-0038-decl-QAPIEvent_lookup 2015-08-04 13:02:46.387628338 +0200
@@ -1,2 +1,3 @@
// decl QAPIEvent_lookup
-extern const char *QAPIEvent_lookup[];
+extern const char *const QAPIEvent_lookup[];
+
--- o/qapi-event.h-0038-typedef-QAPIEvent 2015-08-04 12:47:16.480980103 +0200
+++ n/qapi-event.h-0037-typedef-QAPIEvent 2015-08-04 13:02:46.387628338 +0200
@@ -1,40 +1,40 @@
// typedef QAPIEvent
typedef enum QAPIEvent {
- QAPI_EVENT_BLOCK_IMAGE_CORRUPTED = 0,
- QAPI_EVENT_BLOCK_IO_ERROR = 1,
- QAPI_EVENT_BLOCK_JOB_COMPLETED = 2,
- QAPI_EVENT_BLOCK_JOB_CANCELLED = 3,
- QAPI_EVENT_BLOCK_JOB_ERROR = 4,
- QAPI_EVENT_BLOCK_JOB_READY = 5,
- QAPI_EVENT_BLOCK_WRITE_THRESHOLD = 6,
- QAPI_EVENT_DEVICE_TRAY_MOVED = 7,
- QAPI_EVENT_SHUTDOWN = 8,
- QAPI_EVENT_POWERDOWN = 9,
- QAPI_EVENT_RESET = 10,
- QAPI_EVENT_STOP = 11,
- QAPI_EVENT_RESUME = 12,
- QAPI_EVENT_SUSPEND = 13,
- QAPI_EVENT_SUSPEND_DISK = 14,
- QAPI_EVENT_WAKEUP = 15,
- QAPI_EVENT_RTC_CHANGE = 16,
- QAPI_EVENT_WATCHDOG = 17,
- QAPI_EVENT_DEVICE_DELETED = 18,
- QAPI_EVENT_NIC_RX_FILTER_CHANGED = 19,
- QAPI_EVENT_VNC_CONNECTED = 20,
- QAPI_EVENT_VNC_INITIALIZED = 21,
- QAPI_EVENT_VNC_DISCONNECTED = 22,
- QAPI_EVENT_SPICE_CONNECTED = 23,
+ QAPI_EVENT_ACPI_DEVICE_OST = 0,
+ QAPI_EVENT_BALLOON_CHANGE = 1,
+ QAPI_EVENT_BLOCK_IMAGE_CORRUPTED = 2,
+ QAPI_EVENT_BLOCK_IO_ERROR = 3,
+ QAPI_EVENT_BLOCK_JOB_CANCELLED = 4,
+ QAPI_EVENT_BLOCK_JOB_COMPLETED = 5,
+ QAPI_EVENT_BLOCK_JOB_ERROR = 6,
+ QAPI_EVENT_BLOCK_JOB_READY = 7,
+ QAPI_EVENT_BLOCK_WRITE_THRESHOLD = 8,
+ QAPI_EVENT_DEVICE_DELETED = 9,
+ QAPI_EVENT_DEVICE_TRAY_MOVED = 10,
+ QAPI_EVENT_GUEST_PANICKED = 11,
+ QAPI_EVENT_MEM_UNPLUG_ERROR = 12,
+ QAPI_EVENT_MIGRATION = 13,
+ QAPI_EVENT_NIC_RX_FILTER_CHANGED = 14,
+ QAPI_EVENT_POWERDOWN = 15,
+ QAPI_EVENT_QUORUM_FAILURE = 16,
+ QAPI_EVENT_QUORUM_REPORT_BAD = 17,
+ QAPI_EVENT_RESET = 18,
+ QAPI_EVENT_RESUME = 19,
+ QAPI_EVENT_RTC_CHANGE = 20,
+ QAPI_EVENT_SHUTDOWN = 21,
+ QAPI_EVENT_SPICE_CONNECTED = 22,
+ QAPI_EVENT_SPICE_DISCONNECTED = 23,
QAPI_EVENT_SPICE_INITIALIZED = 24,
- QAPI_EVENT_SPICE_DISCONNECTED = 25,
- QAPI_EVENT_SPICE_MIGRATE_COMPLETED = 26,
- QAPI_EVENT_MIGRATION = 27,
- QAPI_EVENT_ACPI_DEVICE_OST = 28,
- QAPI_EVENT_BALLOON_CHANGE = 29,
- QAPI_EVENT_GUEST_PANICKED = 30,
- QAPI_EVENT_QUORUM_FAILURE = 31,
- QAPI_EVENT_QUORUM_REPORT_BAD = 32,
- QAPI_EVENT_VSERPORT_CHANGE = 33,
- QAPI_EVENT_MEM_UNPLUG_ERROR = 34,
+ QAPI_EVENT_SPICE_MIGRATE_COMPLETED = 25,
+ QAPI_EVENT_STOP = 26,
+ QAPI_EVENT_SUSPEND = 27,
+ QAPI_EVENT_SUSPEND_DISK = 28,
+ QAPI_EVENT_VNC_CONNECTED = 29,
+ QAPI_EVENT_VNC_DISCONNECTED = 30,
+ QAPI_EVENT_VNC_INITIALIZED = 31,
+ QAPI_EVENT_VSERPORT_CHANGE = 32,
+ QAPI_EVENT_WAKEUP = 33,
+ QAPI_EVENT_WATCHDOG = 34,
QAPI_EVENT_MAX = 35,
} QAPIEvent;

* Guards QAPI_TYPES_BUILTIN_STRUCT_DECL and QAPI_TYPES_BUILTIN_CLEANUP_DECL replaced by just QAPI_TYPES_BUILTIN

diff: o/qapi-types.h-*-guardend-QAPI_TYPES_BUILTIN: No such file or directory
diff: n/qapi-types.h-*-guardend-QAPI_TYPES_BUILTIN_CLEANUP_DECL: No such file or directory
diff: n/qapi-types.h-*-guardend-QAPI_TYPES_BUILTIN_STRUCT_DECL: No such file or directory
diff: o/qapi-types.h-*-guardstart-QAPI_TYPES_BUILTIN: No such file or directory
diff: n/qapi-types.h-*-guardstart-QAPI_TYPES_BUILTIN_CLEANUP_DECL: No such file or directory
diff: n/qapi-types.h-*-guardstart-QAPI_TYPES_BUILTIN_STRUCT_DECL: No such file or directory

* A bunch of new forward declarations due to different ordering

diff: o/qapi-visit.c-*-decl-visit_type_BlockdevOptionsArchipelago_fields: No such file or directory
diff: o/qapi-visit.c-*-decl-visit_type_BlockdevOptionsBlkdebug_fields: No such file or directory
diff: o/qapi-visit.c-*-decl-visit_type_BlockdevOptionsBlkverify_fields: No such file or directory
diff: o/qapi-visit.c-*-decl-visit_type_BlockdevOptionsFile_fields: No such file or directory
diff: o/qapi-visit.c-*-decl-visit_type_BlockdevOptionsGenericCOWFormat_fields: No such file or directory
diff: o/qapi-visit.c-*-decl-visit_type_BlockdevOptionsGenericFormat_fields: No such file or directory
diff: o/qapi-visit.c-*-decl-visit_type_BlockdevOptionsNull_fields: No such file or directory
diff: o/qapi-visit.c-*-decl-visit_type_BlockdevOptionsQcow2_fields: No such file or directory
diff: o/qapi-visit.c-*-decl-visit_type_BlockdevOptionsQuorum_fields: No such file or directory
diff: o/qapi-visit.c-*-decl-visit_type_BlockdevOptionsVVFAT_fields: No such file or directory

* New visitors due to "qapi-visit: Convert to QAPISchemaVisitor, fixing bugs"

diff: o/qapi-visit.c-*-def-visit_type_BlockdevRefKind: No such file or directory
diff: o/qapi-visit.c-*-def-visit_type_Qcow2OverlapChecksKind: No such file or directory

* Guard QAPI_VISIT_BUILTIN_VISITOR_DECL replaced by QAPI_VISIT_BUILTIN

diff: o/qapi-visit.h-*-guardend-QAPI_VISIT_BUILTIN: No such file or directory
diff: n/qapi-visit.h-*-guardend-QAPI_VISIT_BUILTIN_VISITOR_DECL: No such file or directory
diff: o/qapi-visit.h-*-guardstart-QAPI_VISIT_BUILTIN: No such file or directory
diff: n/qapi-visit.h-*-guardstart-QAPI_VISIT_BUILTIN_VISITOR_DECL: No such file or directory

---
 scripts/qapi-commands.py | 7 +++++++
 scripts/qapi-event.py    | 9 +++++++++
 scripts/qapi-types.py    | 8 ++++++++
 scripts/qapi-visit.py    | 9 +++++++++
 scripts/qapi.py          | 8 ++++++++
 5 files changed, 41 insertions(+)
scripts/qapi-commands.py +7 -0
index e95fbb8..2dd1895 100644
@@ -24,6 +24,7 @@ def generate_command_decl(name, args, ret_type):
                 arglist += "bool has_%s, " % c_name(memb.name)
             arglist += "%s %s, " % (argtype, c_name(memb.name))
     return mcgen('''
+// decl qmp_%(name)s
 %(ret_type)s qmp_%(name)s(%(args)sError **errp);
 ''',
                  ret_type=(ret_type and ret_type.c_type()) or 'void',
@@ -168,6 +169,7 @@ def gen_marshal_output(name, ret_type):
 
     ret = mcgen('''
 
+// def qmp_marshal_output_%(c_name)s
 static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_out, Error **errp)
 {
     Error *local_err = NULL;
@@ -207,10 +209,12 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
 
     ret = mcgen('''
 
+// def qmp_marshal_input_%(name)s
 %(header)s
 {
     Error *local_err = NULL;
 ''',
+                name=name,
                 header=hdr)
 
     if ret_type:
@@ -263,6 +267,7 @@ qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
 def gen_registry(registry):
     ret = mcgen('''
 
+// def qmp_init_marshal
 static void qmp_init_marshal(void)
 {
 ''')
@@ -270,6 +275,7 @@ static void qmp_init_marshal(void)
     ret += mcgen('''
 }
 
+// decl qapi_init
 qapi_init(qmp_init_marshal);
 ''')
     return ret
@@ -295,6 +301,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         if ret_type:
             self.defn += gen_marshal_output(name, ret_type)
         if middle_mode:
+            self.decl += '// decl gen_marshal_input_%s\n' % name
             self.decl += gen_marshal_input_decl(name, middle_mode) + ';\n'
         self.defn += gen_marshal_input(name, arg_type, ret_type, middle_mode)
         if not middle_mode:
scripts/qapi-event.py +9 -0
index 7525bab..5128e9a 100644
@@ -11,6 +11,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
+import re;
 from qapi import *
 
 def _generate_event_api_name(event_name, params):
@@ -34,22 +35,30 @@ def _generate_event_api_name(event_name, params):
 # Following are the core functions that generate C APIs to emit event.
 
 def generate_event_declaration(api_name):
+    match = re.match(r'void ([A-Za-z0-9_]+)', api_name)
+    name = match.group(1)
     return mcgen('''
 
+// decl %(name)s
 %(api_name)s;
 ''',
+                 name=name,
                  api_name = api_name)
 
 def generate_event_implement(api_name, event_name, params):
+    match = re.match(r'void ([A-Za-z0-9_]+)', api_name)
+    name = match.group(1)
     # step 1: declare any variables
     ret = mcgen("""
 
+// def %(name)s
 %(api_name)s
 {
     QDict *qmp;
     Error *local_err = NULL;
     QMPEventFuncEmit emit;
 """,
+                name=name,
                 api_name = api_name)
 
     if params and params.members:
scripts/qapi-types.py +8 -0
index 5ab3b3b..2cce286 100644
@@ -16,6 +16,7 @@ from qapi import *
 def gen_fwd_object_or_array(name):
     return mcgen('''
 
+// typedef %(name)s
 typedef struct %(name)s %(name)s;
 ''',
                  name=c_name(name))
@@ -23,6 +24,7 @@ typedef struct %(name)s %(name)s;
 def gen_array(name, element_type):
     return mcgen('''
 
+// struct %(name)s
 struct %(name)s {
     union {
         %(c_type)s value;
@@ -57,6 +59,7 @@ def generate_struct_fields(members):
 def gen_struct(name, base, members):
     ret = mcgen('''
 
+// struct %(name)s
 struct %(name)s {
 ''',
                 name=c_name(name))
@@ -83,6 +86,7 @@ struct %(name)s {
 def gen_alternate_qtypes_decl(name):
     return mcgen('''
 
+// decl %(c_name)s_qtypes
 extern const int %(c_name)s_qtypes[];
 ''',
                  c_name=c_name(name))
@@ -90,6 +94,7 @@ extern const int %(c_name)s_qtypes[];
 def gen_alternate_qtypes(name, variants):
     ret = mcgen('''
 
+// def %(name)s_qtypes
 const int %(name)s_qtypes[QTYPE_MAX] = {
 ''',
                 name=c_name(name))
@@ -115,6 +120,7 @@ def gen_union(name, base, variants):
 
     ret = mcgen('''
 
+// struct %(name)s
 struct %(name)s {
 ''',
                 name=name)
@@ -166,6 +172,7 @@ struct %(name)s {
 def generate_type_cleanup_decl(name):
     ret = mcgen('''
 
+// decl qapi_free_%(name)s
 void qapi_free_%(name)s(%(name)s *obj);
 ''',
                 name=c_name(name))
@@ -174,6 +181,7 @@ void qapi_free_%(name)s(%(name)s *obj);
 def generate_type_cleanup(name):
     ret = mcgen('''
 
+// def qapi_free_%(name)s
 void qapi_free_%(name)s(%(name)s *obj)
 {
     QapiDeallocVisitor *md;
scripts/qapi-visit.py +9 -0
index a6ee17f..5893540 100644
@@ -27,12 +27,14 @@ def generate_visit_implicit_struct(type):
         # Need a forward declaration
         ret += mcgen('''
 
+// decl visit_type_%(c_type)s_fields
 static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
 ''',
                      c_type=type.c_name())
 
     ret += mcgen('''
 
+// def visit_type_implicit_%(c_type)s
 static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp)
 {
     Error *err = NULL;
@@ -58,6 +60,7 @@ def generate_visit_struct_fields(name, members, base = None):
 
     ret += mcgen('''
 
+// def visit_type_%(name)s_fields
 static void visit_type_%(name)s_fields(Visitor *m, %(name)s **obj, Error **errp)
 {
     Error *err = NULL;
@@ -140,6 +143,7 @@ def gen_visit_struct(name, base, members):
 
     ret += mcgen('''
 
+// def visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
 ''',
@@ -155,6 +159,7 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 def gen_visit_list(name, element_type):
     return mcgen('''
 
+// def visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -185,6 +190,7 @@ out:
 def generate_visit_enum(name):
     return mcgen('''
 
+// def visit_type_%(c_name)s
 void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error **errp)
 {
     visit_type_enum(m, (int *)obj, %(c_name)s_lookup, "%(name)s", name, errp);
@@ -195,6 +201,7 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error
 def gen_visit_alternate(name, variants):
     ret = mcgen('''
 
+// def visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -252,6 +259,7 @@ def gen_visit_union(name, base, variants):
 
     ret += mcgen('''
 
+// def visit_type_%(c_name)s
 void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -333,6 +341,7 @@ def gen_visit_decl(name, scalar=False):
     if not scalar:
         c_type += '*'
     return mcgen('''
+// decl visit_type_%(c_name)s
 void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
 ''',
                  c_name=c_name(name), c_type=c_type)
scripts/qapi.py +8 -0
index 641ea9f..97d0a8e 100644
@@ -1410,6 +1410,7 @@ def guardname(filename):
 def guardstart(name):
     return mcgen('''
 
+// guardstart %(name)s
 #ifndef %(name)s
 #define %(name)s
 
@@ -1419,6 +1420,7 @@ def guardstart(name):
 def guardend(name):
     return mcgen('''
 
+// guardend %(name)s
 #endif /* %(name)s */
 
 ''',
@@ -1427,6 +1429,7 @@ def guardend(name):
 def generate_enum_lookup(name, values):
     ret = mcgen('''
 
+// def %(name)s_lookup
 const char *const %(name)s_lookup[] = {
 ''',
                 name=c_name(name))
@@ -1449,12 +1452,14 @@ def generate_enum(name, values):
     name = c_name(name)
     lookup_decl = mcgen('''
 
+// decl %(name)s_lookup
 extern const char *const %(name)s_lookup[];
 ''',
                 name=name)
 
     enum_decl = mcgen('''
 
+// typedef %(name)s
 typedef enum %(name)s {
 ''',
                 name=name)
@@ -1558,12 +1563,14 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
     fdecl = maybe_open(do_h, h_file, 'w')
 
     fdef.write(mcgen('''
+// prologue
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 %(comment)s
 ''',
                      comment = c_comment))
 
     fdecl.write(mcgen('''
+// prologue
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 %(comment)s
 #ifndef %(guard)s
@@ -1576,6 +1583,7 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
 
 def close_output(fdef, fdecl):
     fdecl.write('''
+// epilogue
 #endif
 ''')
     fdecl.close()
Signature · 1 line
2.4.3
#41 lore raw

RFC PATCH v4 17/32 Revert "qapi: Generate comments to simplify splitting for review"

In reply to Markus Armbruster

This reverts commit ebce6e3cd3fdf3c1d507a060c80822d98b034ba5.

---
 scripts/qapi-commands.py | 7 -------
 scripts/qapi-event.py    | 9 ---------
 scripts/qapi-types.py    | 8 --------
 scripts/qapi-visit.py    | 9 ---------
 scripts/qapi.py          | 8 --------
 5 files changed, 41 deletions(-)
scripts/qapi-commands.py +0 -7
index 2dd1895..e95fbb8 100644
@@ -24,7 +24,6 @@ def generate_command_decl(name, args, ret_type):
                 arglist += "bool has_%s, " % c_name(memb.name)
             arglist += "%s %s, " % (argtype, c_name(memb.name))
     return mcgen('''
-// decl qmp_%(name)s
 %(ret_type)s qmp_%(name)s(%(args)sError **errp);
 ''',
                  ret_type=(ret_type and ret_type.c_type()) or 'void',
@@ -169,7 +168,6 @@ def gen_marshal_output(name, ret_type):
 
     ret = mcgen('''
 
-// def qmp_marshal_output_%(c_name)s
 static void qmp_marshal_output_%(c_name)s(%(c_ret_type)s ret_in, QObject **ret_out, Error **errp)
 {
     Error *local_err = NULL;
@@ -209,12 +207,10 @@ def gen_marshal_input(name, args, ret_type, middle_mode):
 
     ret = mcgen('''
 
-// def qmp_marshal_input_%(name)s
 %(header)s
 {
     Error *local_err = NULL;
 ''',
-                name=name,
                 header=hdr)
 
     if ret_type:
@@ -267,7 +263,6 @@ qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
 def gen_registry(registry):
     ret = mcgen('''
 
-// def qmp_init_marshal
 static void qmp_init_marshal(void)
 {
 ''')
@@ -275,7 +270,6 @@ static void qmp_init_marshal(void)
     ret += mcgen('''
 }
 
-// decl qapi_init
 qapi_init(qmp_init_marshal);
 ''')
     return ret
@@ -301,7 +295,6 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         if ret_type:
             self.defn += gen_marshal_output(name, ret_type)
         if middle_mode:
-            self.decl += '// decl gen_marshal_input_%s\n' % name
             self.decl += gen_marshal_input_decl(name, middle_mode) + ';\n'
         self.defn += gen_marshal_input(name, arg_type, ret_type, middle_mode)
         if not middle_mode:
scripts/qapi-event.py +0 -9
index 5128e9a..7525bab 100644
@@ -11,7 +11,6 @@
 # This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 
-import re;
 from qapi import *
 
 def _generate_event_api_name(event_name, params):
@@ -35,30 +34,22 @@ def _generate_event_api_name(event_name, params):
 # Following are the core functions that generate C APIs to emit event.
 
 def generate_event_declaration(api_name):
-    match = re.match(r'void ([A-Za-z0-9_]+)', api_name)
-    name = match.group(1)
     return mcgen('''
 
-// decl %(name)s
 %(api_name)s;
 ''',
-                 name=name,
                  api_name = api_name)
 
 def generate_event_implement(api_name, event_name, params):
-    match = re.match(r'void ([A-Za-z0-9_]+)', api_name)
-    name = match.group(1)
     # step 1: declare any variables
     ret = mcgen("""
 
-// def %(name)s
 %(api_name)s
 {
     QDict *qmp;
     Error *local_err = NULL;
     QMPEventFuncEmit emit;
 """,
-                name=name,
                 api_name = api_name)
 
     if params and params.members:
scripts/qapi-types.py +0 -8
index 2cce286..5ab3b3b 100644
@@ -16,7 +16,6 @@ from qapi import *
 def gen_fwd_object_or_array(name):
     return mcgen('''
 
-// typedef %(name)s
 typedef struct %(name)s %(name)s;
 ''',
                  name=c_name(name))
@@ -24,7 +23,6 @@ typedef struct %(name)s %(name)s;
 def gen_array(name, element_type):
     return mcgen('''
 
-// struct %(name)s
 struct %(name)s {
     union {
         %(c_type)s value;
@@ -59,7 +57,6 @@ def generate_struct_fields(members):
 def gen_struct(name, base, members):
     ret = mcgen('''
 
-// struct %(name)s
 struct %(name)s {
 ''',
                 name=c_name(name))
@@ -86,7 +83,6 @@ struct %(name)s {
 def gen_alternate_qtypes_decl(name):
     return mcgen('''
 
-// decl %(c_name)s_qtypes
 extern const int %(c_name)s_qtypes[];
 ''',
                  c_name=c_name(name))
@@ -94,7 +90,6 @@ extern const int %(c_name)s_qtypes[];
 def gen_alternate_qtypes(name, variants):
     ret = mcgen('''
 
-// def %(name)s_qtypes
 const int %(name)s_qtypes[QTYPE_MAX] = {
 ''',
                 name=c_name(name))
@@ -120,7 +115,6 @@ def gen_union(name, base, variants):
 
     ret = mcgen('''
 
-// struct %(name)s
 struct %(name)s {
 ''',
                 name=name)
@@ -172,7 +166,6 @@ struct %(name)s {
 def generate_type_cleanup_decl(name):
     ret = mcgen('''
 
-// decl qapi_free_%(name)s
 void qapi_free_%(name)s(%(name)s *obj);
 ''',
                 name=c_name(name))
@@ -181,7 +174,6 @@ void qapi_free_%(name)s(%(name)s *obj);
 def generate_type_cleanup(name):
     ret = mcgen('''
 
-// def qapi_free_%(name)s
 void qapi_free_%(name)s(%(name)s *obj)
 {
     QapiDeallocVisitor *md;
scripts/qapi-visit.py +0 -9
index 5893540..a6ee17f 100644
@@ -27,14 +27,12 @@ def generate_visit_implicit_struct(type):
         # Need a forward declaration
         ret += mcgen('''
 
-// decl visit_type_%(c_type)s_fields
 static void visit_type_%(c_type)s_fields(Visitor *m, %(c_type)s **obj, Error **errp);
 ''',
                      c_type=type.c_name())
 
     ret += mcgen('''
 
-// def visit_type_implicit_%(c_type)s
 static void visit_type_implicit_%(c_type)s(Visitor *m, %(c_type)s **obj, Error **errp)
 {
     Error *err = NULL;
@@ -60,7 +58,6 @@ def generate_visit_struct_fields(name, members, base = None):
 
     ret += mcgen('''
 
-// def visit_type_%(name)s_fields
 static void visit_type_%(name)s_fields(Visitor *m, %(name)s **obj, Error **errp)
 {
     Error *err = NULL;
@@ -143,7 +140,6 @@ def gen_visit_struct(name, base, members):
 
     ret += mcgen('''
 
-// def visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
 ''',
@@ -159,7 +155,6 @@ void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **e
 def gen_visit_list(name, element_type):
     return mcgen('''
 
-// def visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -190,7 +185,6 @@ out:
 def generate_visit_enum(name):
     return mcgen('''
 
-// def visit_type_%(c_name)s
 void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error **errp)
 {
     visit_type_enum(m, (int *)obj, %(c_name)s_lookup, "%(name)s", name, errp);
@@ -201,7 +195,6 @@ void visit_type_%(c_name)s(Visitor *m, %(c_name)s *obj, const char *name, Error
 def gen_visit_alternate(name, variants):
     ret = mcgen('''
 
-// def visit_type_%(name)s
 void visit_type_%(name)s(Visitor *m, %(name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -259,7 +252,6 @@ def gen_visit_union(name, base, variants):
 
     ret += mcgen('''
 
-// def visit_type_%(c_name)s
 void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
 {
     Error *err = NULL;
@@ -341,7 +333,6 @@ def gen_visit_decl(name, scalar=False):
     if not scalar:
         c_type += '*'
     return mcgen('''
-// decl visit_type_%(c_name)s
 void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
 ''',
                  c_name=c_name(name), c_type=c_type)
scripts/qapi.py +0 -8
index 97d0a8e..641ea9f 100644
@@ -1410,7 +1410,6 @@ def guardname(filename):
 def guardstart(name):
     return mcgen('''
 
-// guardstart %(name)s
 #ifndef %(name)s
 #define %(name)s
 
@@ -1420,7 +1419,6 @@ def guardstart(name):
 def guardend(name):
     return mcgen('''
 
-// guardend %(name)s
 #endif /* %(name)s */
 
 ''',
@@ -1429,7 +1427,6 @@ def guardend(name):
 def generate_enum_lookup(name, values):
     ret = mcgen('''
 
-// def %(name)s_lookup
 const char *const %(name)s_lookup[] = {
 ''',
                 name=c_name(name))
@@ -1452,14 +1449,12 @@ def generate_enum(name, values):
     name = c_name(name)
     lookup_decl = mcgen('''
 
-// decl %(name)s_lookup
 extern const char *const %(name)s_lookup[];
 ''',
                 name=name)
 
     enum_decl = mcgen('''
 
-// typedef %(name)s
 typedef enum %(name)s {
 ''',
                 name=name)
@@ -1563,14 +1558,12 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
     fdecl = maybe_open(do_h, h_file, 'w')
 
     fdef.write(mcgen('''
-// prologue
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 %(comment)s
 ''',
                      comment = c_comment))
 
     fdecl.write(mcgen('''
-// prologue
 /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 %(comment)s
 #ifndef %(guard)s
@@ -1583,7 +1576,6 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
 
 def close_output(fdef, fdecl):
     fdecl.write('''
-// epilogue
 #endif
 ''')
     fdecl.close()
Signature · 1 line
2.4.3
#42 lore raw

RFC PATCH v4 18/32 qapi: Replace dirty is_c_ptr() by method c_null()

In reply to Markus Armbruster

is_c_ptr() looks whether the end of the C text for the type looks like a pointer. Works, but is fragile.

We now have a better tool: use QAPISchemaType method c_null(). The initializers for non-pointers become prettier: 0, false or the enumeration constant with the value 0 instead of {0}.

---
 scripts/qapi-commands.py | 16 +++++-----------
 scripts/qapi.py          |  3 ---
 2 files changed, 5 insertions(+), 14 deletions(-)
scripts/qapi-commands.py +5 -11
index e95fbb8..dd207cd 100644
@@ -91,18 +91,12 @@ def gen_visitor_input_vars_decl(args):
 bool has_%(argname)s = false;
 ''',
                              argname=c_name(memb.name))
-            if is_c_ptr(memb.type.c_type()):
-                ret += mcgen('''
-%(argtype)s %(argname)s = NULL;
+            ret += mcgen('''
+%(c_type)s %(c_name)s = %(c_null)s;
 ''',
-                             argname=c_name(memb.name),
-                             argtype=memb.type.c_type())
-            else:
-                ret += mcgen('''
-%(argtype)s %(argname)s = {0};
-''',
-                             argname=c_name(memb.name),
-                             argtype=memb.type.c_type())
+                         c_name=c_name(memb.name),
+                         c_type=memb.type.c_type(),
+                         c_null=memb.type.c_null())
 
     pop_indent()
     return ret
scripts/qapi.py +0 -3
index 641ea9f..c6761f9 100644
@@ -1369,9 +1369,6 @@ def c_type(value, is_param=False):
         assert isinstance(value, str) and value != ""
         return c_name(value) + pointer_suffix
 
-def is_c_ptr(value):
-    return value.endswith(pointer_suffix)
-
 def genindent(count):
     ret = ""
     for i in range(count):
Signature · 1 line
2.4.3
#43 lore raw

RFC PATCH v4 19/32 qapi: Clean up after recent conversions to QAPISchemaVisitor

In reply to Markus Armbruster

Generate just 'FOO' instead of 'struct FOO' when possible.

Drop helper functions that are now unused.

Make pylint reasonably happy.

Rename generate_FOO() functions to gen_FOO() for consistency.

Use more consistent and sensible variable names.

Consistently use c_ for mapping keys when their value is a C identifier or type.

Simplify gen_enum() and gen_visit_union()

Consistently use single quotes for C text string literals.

---
 docs/qapi-code-gen.txt   |   2 +-
 scripts/qapi-commands.py | 129 +++++++++++++++++++++++-----------------------
 scripts/qapi-event.py    | 121 ++++++++++++++++++++-----------------------
 scripts/qapi-types.py    |  64 +++++++++++------------
 scripts/qapi-visit.py    | 119 +++++++++++++++++++++---------------------
 scripts/qapi.py          | 131 +++++++++--------------------------------------
 6 files changed, 239 insertions(+), 327 deletions(-)
docs/qapi-code-gen.txt +1 -1
index 7c500b6..e85f113 100644
@@ -590,7 +590,7 @@ Example:
             UserDefOne *value;
             uint64_t padding;
         };
-        struct UserDefOneList *next;
+        UserDefOneList *next;
     };
 
     void qapi_free_UserDefOneList(UserDefOneList *obj);
scripts/qapi-commands.py +66 -63

270 lines not on this page · raw message

scripts/qapi-event.py +56 -65

206 lines not on this page · raw message

scripts/qapi-types.py +31 -33

170 lines not on this page · raw message

scripts/qapi-visit.py +60 -59

286 lines not on this page · raw message

scripts/qapi.py +25 -106

184 lines not on this page · raw message

Signature · 1 line
2.4.3
#44 lore raw

RFC PATCH v4 20/32 qapi-visit: Rearrange code a bit

In reply to Markus Armbruster

Move gen_visit_decl() to a better place. Inline
generate_visit_struct_body().

---
 scripts/qapi-visit.py | 43 +++++++++++++++++--------------------------
 1 file changed, 17 insertions(+), 26 deletions(-)
scripts/qapi-visit.py +17 -26
index 08504ff..6819866 100644
@@ -18,6 +18,15 @@ import re
 implicit_structs_seen = set()
 struct_fields_seen = set()
 
+def gen_visit_decl(name, scalar=False):
+    c_type = c_name(name) + ' *'
+    if not scalar:
+        c_type += '*'
+    return mcgen('''
+void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
+''',
+                 c_name=c_name(name), c_type=c_type)
+
 def gen_visit_implicit_struct(typ):
     if typ in implicit_structs_seen:
         return ''
@@ -114,12 +123,17 @@ out:
 ''')
     return ret
 
-def gen_visit_struct_body(name):
+def gen_visit_struct(name, base, members):
+    ret = gen_visit_struct_fields(name, base, members)
+
     # FIXME: if *obj is NULL on entry, and visit_start_struct() assigns to
     # *obj, but then visit_type_FOO_fields() fails, we should clean up *obj
     # rather than leaving it non-NULL. As currently written, the caller must
     # call qapi_free_FOO() to avoid a memory leak of the partial FOO.
-    ret = mcgen('''
+    ret += mcgen('''
+
+void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
+{
     Error *err = NULL;
 
     visit_start_struct(m, (void **)obj, "%(name)s", name, sizeof(%(c_name)s), &err);
@@ -130,24 +144,10 @@ def gen_visit_struct_body(name):
         visit_end_struct(m, &err);
     }
     error_propagate(errp, err);
+}
 ''',
                 name=name, c_name=c_name(name))
-    return ret
 
-def gen_visit_struct(name, base, members):
-    ret = gen_visit_struct_fields(name, base, members)
-    ret += mcgen('''
-
-void visit_type_%(c_name)s(Visitor *m, %(c_name)s **obj, const char *name, Error **errp)
-{
-''',
-                 c_name=c_name(name))
-
-    ret += gen_visit_struct_body(name)
-
-    ret += mcgen('''
-}
-''')
     return ret
 
 def gen_visit_list(name, element_type):
@@ -329,15 +329,6 @@ out:
 
     return ret
 
-def gen_visit_decl(name, scalar=False):
-    c_type = c_name(name) + ' *'
-    if not scalar:
-        c_type += '*'
-    return mcgen('''
-void visit_type_%(c_name)s(Visitor *m, %(c_type)sobj, const char *name, Error **errp);
-''',
-                 c_name=c_name(name), c_type=c_type)
-
 class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
     def __init__(self):
         self.decl = None
Signature · 1 line
2.4.3
#45 lore raw

RFC PATCH v4 21/32 qapi-commands: Rearrange code

In reply to Markus Armbruster

Rename gen_marshal_input() to gen_marshal(), because the generated function marshals both arguments and results.

Rename gen_visitor_input_containers_decl() to gen_marshal_vars(), and move the other variable declarations there, too.

Rename gen_visitor_input_block() to gen_marshal_input_visit(), and rearrange its code slightly.

Rename gen_marshal_input_decl() to gen_marshal_proto(), because the result isn't a full declaration, unlike gen_command_decl()'s.

New gen_marshal_decl() actually returns a full declaration.

---
 scripts/qapi-commands.py | 85 ++++++++++++++++++++++--------------------------
 1 file changed, 38 insertions(+), 47 deletions(-)
scripts/qapi-commands.py +38 -47
index 3758dbd..7db30e5 100644
@@ -56,6 +56,7 @@ def gen_call(name, arg_type, ret_type):
 
     push_indent()
     ret = mcgen('''
+
 %(lhs)sqmp_%(c_name)s(%(args)s&local_err);
 ''',
                 c_name=c_name(name), args=argstr, lhs=lhs)
@@ -69,25 +70,26 @@ qmp_marshal_output_%(c_name)s(retval, ret, &local_err);
     pop_indent()
     return ret
 
-def gen_visitor_input_containers_decl(arg_type):
-    ret = ''
+def gen_marshal_vars(arg_type, ret_type):
+    ret = mcgen('''
+    Error *local_err = NULL;
+''')
 
     push_indent()
+
+    if ret_type:
+        ret += mcgen('''
+%(c_type)s retval;
+''',
+                     c_type=ret_type.c_type())
+
     if arg_type:
         ret += mcgen('''
 QmpInputVisitor *mi = qmp_input_visitor_new_strict(QOBJECT(args));
 QapiDeallocVisitor *md;
 Visitor *v;
 ''')
-    pop_indent()
 
-    return ret
-
-def gen_visitor_input_vars_decl(arg_type):
-    ret = ''
-    push_indent()
-
-    if arg_type:
         for memb in arg_type.members:
             if memb.optional:
                 ret += mcgen('''
@@ -100,14 +102,18 @@ bool has_%(c_name)s = false;
                          c_name=c_name(memb.name),
                          c_type=memb.type.c_type(),
                          c_null=memb.type.c_null())
+        ret += '\n'
+    else:
+        ret += mcgen('''
+
+(void)args;
+''')
 
     pop_indent()
     return ret
 
-def gen_visitor_input_block(arg_type, dealloc=False):
+def gen_marshal_input_visit(arg_type, dealloc=False):
     ret = ''
-    errparg = '&local_err'
-    errarg = 'local_err'
 
     if not arg_type:
         return ret
@@ -123,6 +129,8 @@ md = qapi_dealloc_visitor_new();
 v = qapi_dealloc_get_visitor(md);
 ''')
     else:
+        errparg = '&local_err'
+        errarg = 'local_err'
         ret += mcgen('''
 v = qmp_input_get_visitor(mi);
 ''')
@@ -160,10 +168,7 @@ qapi_dealloc_visitor_cleanup(md);
     return ret
 
 def gen_marshal_output(name, ret_type):
-    if not ret_type:
-        return ''
-
-    ret = mcgen('''
+    return mcgen('''
 
 static void qmp_marshal_output_%(c_cmd_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
 {
@@ -188,45 +193,31 @@ out:
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                c_type=ret_type.c_type(), c_cmd_name=c_name(name),
-                c_name=ret_type.c_name())
+                 c_type=ret_type.c_type(), c_cmd_name=c_name(name),
+                 c_name=ret_type.c_name())
 
-    return ret
-
-def gen_marshal_input_decl(name):
+def gen_marshal_proto(name):
     ret = 'void qmp_marshal_input_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
     if not middle_mode:
         ret = 'static ' + ret
     return ret
 
-def gen_marshal_input(name, arg_type, ret_type):
-    hdr = gen_marshal_input_decl(name)
+def gen_marshal_decl(name):
+    return mcgen('''
+%(proto)s;
+''',
+                 proto=gen_marshal_proto(name))
 
+def gen_marshal(name, arg_type, ret_type):
     ret = mcgen('''
 
-%(header)s
+%(proto)s
 {
-    Error *local_err = NULL;
 ''',
-                header=hdr)
-
-    if ret_type:
-        ret += mcgen('''
-    %(c_type)s retval;
-''',
-                     c_type=ret_type.c_type())
-
-    if arg_type:
-        ret += gen_visitor_input_containers_decl(arg_type)
-        ret += gen_visitor_input_vars_decl(arg_type) + '\n'
-        ret += gen_visitor_input_block(arg_type) + '\n'
-    else:
-        ret += mcgen('''
-
-    (void)args;
-
-''')
+                proto=gen_marshal_proto(name))
 
+    ret += gen_marshal_vars(arg_type, ret_type)
+    ret += gen_marshal_input_visit(arg_type)
     ret += gen_call(name, arg_type, ret_type)
 
     if re.search('^ *goto out;', ret, re.MULTILINE):
@@ -237,7 +228,7 @@ out:
     ret += mcgen('''
     error_propagate(errp, local_err);
 ''')
-    ret += gen_visitor_input_block(arg_type, dealloc=True)
+    ret += gen_marshal_input_visit(arg_type, dealloc=True)
     ret += mcgen('''
 }
 ''')
@@ -292,8 +283,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         if ret_type:
             self.defn += gen_marshal_output(name, ret_type)
         if middle_mode:
-            self.decl += gen_marshal_input_decl(name) + ';\n'
-        self.defn += gen_marshal_input(name, arg_type, ret_type)
+            self.decl += gen_marshal_decl(name)
+        self.defn += gen_marshal(name, arg_type, ret_type)
         if not middle_mode:
             self.regy += gen_register_command(name, success_response)
 
Signature · 1 line
2.4.3
#46 lore raw

RFC PATCH v4 22/32 qapi: Rename qmp_marshal_input_FOO() to qmp_marshal_FOO()

In reply to Markus Armbruster

These functions marshal both input and output.

---
 docs/qapi-code-gen.txt        |   4 +-
 docs/writing-qmp-commands.txt |   8 +-
 monitor.c                     |   2 +-
 qmp-commands.hx               | 240 +++++++++++++++++++++---------------------
 qmp.c                         |   6 +-
 scripts/qapi-commands.py      |   4 +-
 6 files changed, 132 insertions(+), 132 deletions(-)
docs/qapi-code-gen.txt +2 -2
index e85f113..09d15dc 100644
@@ -736,7 +736,7 @@ Example:
         qapi_dealloc_visitor_cleanup(md);
     }
 
-    static void qmp_marshal_input_my_command(QDict *args, QObject **ret, Error **errp)
+    static void qmp_marshal_my_command(QDict *args, QObject **ret, Error **errp)
     {
         Error *local_err = NULL;
         UserDefOne *retval;
@@ -769,7 +769,7 @@ Example:
 
     static void qmp_init_marshal(void)
     {
-        qmp_register_command("my-command", qmp_marshal_input_my_command, QCO_NO_OPTIONS);
+        qmp_register_command("my-command", qmp_marshal_my_command, QCO_NO_OPTIONS);
     }
 
     qapi_init(qmp_init_marshal);
docs/writing-qmp-commands.txt +4 -4
index ab1fdd3..c65bdc6 100644
@@ -127,7 +127,7 @@ following in the botton:
     {
         .name       = "hello-world",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_hello_world,
+        .mhandler.cmd_new = qmp_marshal_hello_world,
     },
 
 You're done. Now build qemu, run it as suggested in the "Testing" section,
@@ -179,7 +179,7 @@ The last step is to update the qmp-commands.hx file:
     {
         .name       = "hello-world",
         .args_type  = "message:s?",
-        .mhandler.cmd_new = qmp_marshal_input_hello_world,
+        .mhandler.cmd_new = qmp_marshal_hello_world,
     },
 
 Notice that the "args_type" member got our "message" argument. The character
@@ -461,7 +461,7 @@ The last step is to add the correspoding entry in the qmp-commands.hx file:
     {
         .name       = "query-alarm-clock",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_alarm_clock,
+        .mhandler.cmd_new = qmp_marshal_query_alarm_clock,
     },
 
 Time to test the new command. Build qemu, run it as described in the "Testing"
@@ -607,7 +607,7 @@ To test this you have to add the corresponding qmp-commands.hx entry:
     {
         .name       = "query-alarm-methods",
         .args_type  = "",
-        .mhandler.cmd_new = qmp_marshal_input_query_alarm_methods,
+        .mhandler.cmd_new = qmp_marshal_query_alarm_methods,
     },
 
 Now Build qemu, run it as explained in the "Testing" section and try our new
monitor.c +1 -1
index 641cfae6..f654607 100644
@@ -5109,7 +5109,7 @@ static QObject *get_qmp_greeting(void)
 {
     QObject *ver = NULL;
 
-    qmp_marshal_input_query_version(NULL, &ver, NULL);
+    qmp_marshal_query_version(NULL, &ver, NULL);
     return qobject_from_jsonf("{'QMP':{'version': %p,'capabilities': []}}",ver);
 }
 
qmp-commands.hx +120 -120

935 lines not on this page · raw message

qmp.c +3 -3
index 403805a..f31c6dc 100644
@@ -151,9 +151,9 @@ VncInfo2List *qmp_query_vnc_servers(Error **errp)
  * #ifdef CONFIG_SPICE.  Necessary for an accurate query-commands
  * result.  However, the QAPI schema is blissfully unaware of that,
  * and the QAPI code generator happily generates a dead
- * qmp_marshal_input_query_spice() that calls qmp_query_spice().
- * Provide it one, or else linking fails.
- * FIXME Educate the QAPI schema on CONFIG_SPICE.
+ * qmp_marshal_query_spice() that calls qmp_query_spice().  Provide it
+ * one, or else linking fails.  FIXME Educate the QAPI schema on
+ * CONFIG_SPICE.
  */
 SpiceInfo *qmp_query_spice(Error **errp)
 {
scripts/qapi-commands.py +2 -2
index 7db30e5..18f4088 100644
@@ -197,7 +197,7 @@ out:
                  c_name=ret_type.c_name())
 
 def gen_marshal_proto(name):
-    ret = 'void qmp_marshal_input_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
+    ret = 'void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
     if not middle_mode:
         ret = 'static ' + ret
     return ret
@@ -241,7 +241,7 @@ def gen_register_command(name, success_response):
         options = 'QCO_NO_SUCCESS_RESP'
 
     ret = mcgen('''
-qmp_register_command("%(name)s", qmp_marshal_input_%(c_name)s, %(opts)s);
+qmp_register_command("%(name)s", qmp_marshal_%(c_name)s, %(opts)s);
 ''',
                 name=name, c_name=c_name(name),
                 opts=options)
Signature · 1 line
2.4.3
#47 lore raw

RFC PATCH v4 23/32 qapi: De-duplicate parameter list generation

In reply to Markus Armbruster

Generated qapi-event.[ch] lose line breaks. No change otherwise.

---
 scripts/qapi-commands.py | 11 ++---------
 scripts/qapi-event.py    | 18 +++---------------
 scripts/qapi.py          | 16 ++++++++++++++++
 3 files changed, 21 insertions(+), 24 deletions(-)
scripts/qapi-commands.py +2 -9
index 18f4088..5a63db0 100644
@@ -16,19 +16,12 @@ from qapi import *
 import re
 
 def gen_command_decl(name, arg_type, ret_type):
-    argstr = ''
-    if arg_type:
-        for memb in arg_type.members:
-            if memb.optional:
-                argstr += 'bool has_%s, ' % c_name(memb.name)
-            argstr += '%s %s, ' % (memb.type.c_type(is_param=True),
-                                   c_name(memb.name))
     return mcgen('''
-%(c_type)s qmp_%(c_name)s(%(args)sError **errp);
+%(c_type)s qmp_%(c_name)s(%(params)s);
 ''',
                  c_type=(ret_type and ret_type.c_type()) or 'void',
                  c_name=c_name(name),
-                 args=argstr)
+                 params=gen_params(arg_type, 'Error **errp'))
 
 def gen_err_check(err):
     if not err:
scripts/qapi-event.py +3 -15
index dbcc5a1..d100099 100644
@@ -14,21 +14,9 @@
 from qapi import *
 
 def gen_event_send_proto(name, arg_type):
-    api_name = "void qapi_event_send_%s(" % c_name(name).lower()
-    l = len(api_name)
-
-    if arg_type:
-        for m in arg_type.members:
-            if m.optional:
-                api_name += "bool has_%s,\n" % c_name(m.name)
-                api_name += "".ljust(l)
-
-            api_name += "%s %s,\n" % (m.type.c_type(is_param=True),
-                                      c_name(m.name))
-            api_name += "".ljust(l)
-
-    api_name += "Error **errp)"
-    return api_name
+    return 'void qapi_event_send_%(c_name)s(%(param)s)' % {
+        'c_name': c_name(name.lower()),
+        'param': gen_params(arg_type, 'Error **errp')}
 
 def gen_event_send_decl(name, arg_type):
     return mcgen('''
scripts/qapi.py +16 -0
index d314b13..0a325d5 100644
@@ -1395,6 +1395,22 @@ extern const char *const %(c_name)s_lookup[];
                  c_name=c_name(name))
     return ret
 
+def gen_params(arg_type, extra):
+    if not arg_type:
+        return extra
+    assert not arg_type.variants
+    ret = ''
+    sep = ''
+    for memb in arg_type.members:
+        ret += sep
+        sep = ', '
+        if memb.optional:
+            ret += 'bool has_%s, ' % c_name(memb.name)
+        ret += '%s %s' % (memb.type.c_type(is_param=True), c_name(memb.name))
+    if extra:
+        ret += sep + extra
+    return ret
+
 #
 # Common command line parsing
 #
Signature · 1 line
2.4.3
#48 lore raw

RFC PATCH v4 24/32 qapi-commands: De-duplicate output marshaling functions

In reply to Markus Armbruster

gen_marshal_output() uses its parameter name only for name of the generated function. Name it after the type being marshaled instead of its caller, and drop duplicates.

Saves 7 copies of qmp_marshal_output_int() in qemu-ga, and one copy of qmp_marshal_output_str() in qemu-system-*.

---
 docs/qapi-code-gen.txt   |  4 ++--
 scripts/qapi-commands.py | 17 ++++++++++-------
 2 files changed, 12 insertions(+), 9 deletions(-)
docs/qapi-code-gen.txt +2 -2
index 09d15dc..147638f 100644
@@ -713,7 +713,7 @@ Example:
     $ cat qapi-generated/example-qmp-marshal.c
 [Uninteresting stuff omitted...]
 
-    static void qmp_marshal_output_my_command(UserDefOne *ret_in, QObject **ret_out, Error **errp)
+    static void qmp_marshal_output_UserDefOne(UserDefOne *ret_in, QObject **ret_out, Error **errp)
     {
         Error *local_err = NULL;
         QmpOutputVisitor *mo = qmp_output_visitor_new();
@@ -756,7 +756,7 @@ Example:
             goto out;
         }
 
-        qmp_marshal_output_my_command(retval, ret, &local_err);
+        qmp_marshal_output_UserDefOne(retval, ret, &local_err);
 
     out:
         error_propagate(errp, local_err);
scripts/qapi-commands.py +10 -7
index 5a63db0..7ff7c31 100644
@@ -59,7 +59,7 @@ def gen_call(name, arg_type, ret_type):
 
 qmp_marshal_output_%(c_name)s(retval, ret, &local_err);
 ''',
-                     c_name=c_name(name))
+                     c_name=ret_type.c_name())
     pop_indent()
     return ret
 
@@ -160,10 +160,10 @@ qapi_dealloc_visitor_cleanup(md);
     pop_indent()
     return ret
 
-def gen_marshal_output(name, ret_type):
+def gen_marshal_output(ret_type):
     return mcgen('''
 
-static void qmp_marshal_output_%(c_cmd_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
+static void qmp_marshal_output_%(c_name)s(%(c_type)s ret_in, QObject **ret_out, Error **errp)
 {
     Error *local_err = NULL;
     QmpOutputVisitor *mo = qmp_output_visitor_new();
@@ -186,8 +186,7 @@ out:
     qapi_dealloc_visitor_cleanup(md);
 }
 ''',
-                 c_type=ret_type.c_type(), c_cmd_name=c_name(name),
-                 c_name=ret_type.c_name())
+                 c_type=ret_type.c_type(), c_name=ret_type.c_name())
 
 def gen_marshal_proto(name):
     ret = 'void qmp_marshal_%s(QDict *args, QObject **ret, Error **errp)' % c_name(name)
@@ -260,21 +259,25 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self.decl = None
         self.defn = None
         self.regy = None
+        self.visited_ret_types = None
     def visit_begin(self, schema):
         self.decl = ''
         self.defn = ''
         self.regy = ''
+        self.visited_ret_types = set()
     def visit_end(self):
         if not middle_mode:
             self.defn += gen_registry(self.regy)
         self.regy = None
+        self.visited_ret_types = None
     def visit_command(self, name, info, arg_type, ret_type,
                       gen, success_response):
         if not gen:
             return
         self.decl += gen_command_decl(name, arg_type, ret_type)
-        if ret_type:
-            self.defn += gen_marshal_output(name, ret_type)
+        if ret_type and ret_type not in self.visited_ret_types:
+            self.visited_ret_types.add(ret_type)
+            self.defn += gen_marshal_output(ret_type)
         if middle_mode:
             self.decl += gen_marshal_decl(name)
         self.defn += gen_marshal(name, arg_type, ret_type)
Signature · 1 line
2.4.3
#49 lore raw

RFC PATCH v4 25/32 qapi: Improve built-in type documentation

In reply to Markus Armbruster

Clarify how they map to JSON. Add how they map to C. Fix the reference to StringInputVisitor.

---
 docs/qapi-code-gen.txt | 29 ++++++++++++++++++-----------
 1 file changed, 18 insertions(+), 11 deletions(-)
docs/qapi-code-gen.txt +18 -11
index 147638f..35a4a0f 100644
@@ -140,17 +140,24 @@ must have a value that forms a struct name.
 
 === Built-in Types ===
 
-The following types are built-in to the parser:
-  'str' - arbitrary UTF-8 string
-  'int' - 64-bit signed integer (although the C code may place further
-          restrictions on acceptable range)
-  'number' - floating point number
-  'bool' - JSON value of true or false
-  'int8', 'int16', 'int32', 'int64' - like 'int', but enforce maximum
-                                      bit size
-  'uint8', 'uint16', 'uint32', 'uint64' - unsigned counterparts
-  'size' - like 'uint64', but allows scaled suffix from command line
-           visitor
+The following types are predefined, and map to C as follows:
+
+  Schema    C          JSON
+  str       char *     any JSON string, UTF-8
+  number    double     any JSON number
+  int       int64_t    a JSON number without fractional part
+                       that fits into the C integer type
+  int8      int8_t     likewise
+  int16     int16_t    likewise
+  int32     int32_t    likewise
+  int64     int64_t    likewise
+  uint8     uint8_t    likewise
+  uint16    uint16_t   likewise
+  uint32    uint32_t   likewise
+  uint64    uint64_t   likewise
+  size      uint64_t   like uint64_t, except StringInputVisitor
+                       accepts size suffixes
+  bool      bool       JSON true or false
 
 
 === Includes ===
Signature · 1 line
2.4.3
#50 lore raw

RFC PATCH v4 26/32 qapi: Introduce a first class 'any' type

In reply to Markus Armbruster

It's first class, because unlike '**', it actually works, i.e. doesn't require 'gen': false.

'**' will go away next.

---
 docs/qapi-code-gen.txt                             |  1 +
 include/qapi/visitor-impl.h                        |  2 +
 include/qapi/visitor.h                             |  1 +
 qapi/qapi-dealloc-visitor.c                        |  9 ++++
 qapi/qapi-visit-core.c                             |  6 +++
 qapi/qmp-input-visitor.c                           | 11 +++++
 qapi/qmp-output-visitor.c                          |  9 ++++
 scripts/qapi-types.py                              |  1 +
 scripts/qapi.py                                    |  9 ++--
 tests/Makefile                                     |  3 +-
 tests/qapi-schema/args-any.err                     |  1 +
 .../{flat-union-base-star.exit => args-any.exit}   |  0
 tests/qapi-schema/args-any.json                    |  2 +
 .../{flat-union-base-star.out => args-any.out}     |  0
 tests/qapi-schema/flat-union-base-any.err          |  1 +
 tests/qapi-schema/flat-union-base-any.exit         |  1 +
 ...ion-base-star.json => flat-union-base-any.json} |  2 +-
 tests/qapi-schema/flat-union-base-any.out          |  0
 tests/qapi-schema/flat-union-base-star.err         |  1 -
 tests/qapi-schema/qapi-schema-test.json            |  5 +-
 tests/qapi-schema/qapi-schema-test.out             |  9 +++-
 tests/qapi-schema/type-bypass.out                  |  4 +-
 tests/test-qmp-commands.c                          |  5 ++
 tests/test-qmp-input-visitor.c                     | 45 ++++++++++++++++++
 tests/test-qmp-output-visitor.c                    | 53 ++++++++++++++++++++++
 25 files changed, 171 insertions(+), 10 deletions(-)
 create mode 100644 tests/qapi-schema/args-any.err
 rename tests/qapi-schema/{flat-union-base-star.exit => args-any.exit} (100%)
 create mode 100644 tests/qapi-schema/args-any.json
 rename tests/qapi-schema/{flat-union-base-star.out => args-any.out} (100%)
 create mode 100644 tests/qapi-schema/flat-union-base-any.err
 create mode 100644 tests/qapi-schema/flat-union-base-any.exit
 rename tests/qapi-schema/{flat-union-base-star.json => flat-union-base-any.json} (95%)
 create mode 100644 tests/qapi-schema/flat-union-base-any.out
 delete mode 100644 tests/qapi-schema/flat-union-base-star.err
docs/qapi-code-gen.txt +1 -0
index 35a4a0f..c713e3a 100644
@@ -158,6 +158,7 @@ The following types are predefined, and map to C as follows:
   size      uint64_t   like uint64_t, except StringInputVisitor
                        accepts size suffixes
   bool      bool       JSON true or false
+  any       QObject *  any JSON value
 
 
 === Includes ===
include/qapi/visitor-impl.h +2 -0
index f4a2f74..8c0ba57 100644
@@ -40,6 +40,8 @@ struct Visitor
     void (*type_str)(Visitor *v, char **obj, const char *name, Error **errp);
     void (*type_number)(Visitor *v, double *obj, const char *name,
                         Error **errp);
+    void (*type_any)(Visitor *v, QObject **obj, const char *name,
+                     Error **errp);
 
     /* May be NULL */
     void (*optional)(Visitor *v, bool *present, const char *name,
include/qapi/visitor.h +1 -0
index 00ba104..cfc19a6 100644
@@ -58,6 +58,7 @@ void visit_type_size(Visitor *v, uint64_t *obj, const char *name, Error **errp);
 void visit_type_bool(Visitor *v, bool *obj, const char *name, Error **errp);
 void visit_type_str(Visitor *v, char **obj, const char *name, Error **errp);
 void visit_type_number(Visitor *v, double *obj, const char *name, Error **errp);
+void visit_type_any(Visitor *v, QObject **obj, const char *name, Error **errp);
 bool visit_start_union(Visitor *v, bool data_present, Error **errp);
 void visit_end_union(Visitor *v, bool data_present, Error **errp);
 
qapi/qapi-dealloc-visitor.c +9 -0
index d7f92c5..737deab 100644
@@ -151,6 +151,14 @@ static void qapi_dealloc_type_number(Visitor *v, double *obj, const char *name,
 {
 }
 
+static void qapi_dealloc_type_anything(Visitor *v, QObject **obj,
+                                       const char *name, Error **errp)
+{
+    if (obj) {
+        qobject_decref(*obj);
+    }
+}
+
 static void qapi_dealloc_type_size(Visitor *v, uint64_t *obj, const char *name,
                                    Error **errp)
 {
@@ -216,6 +224,7 @@ QapiDeallocVisitor *qapi_dealloc_visitor_new(void)
     v->visitor.type_bool = qapi_dealloc_type_bool;
     v->visitor.type_str = qapi_dealloc_type_str;
     v->visitor.type_number = qapi_dealloc_type_number;
+    v->visitor.type_any = qapi_dealloc_type_anything;
     v->visitor.type_size = qapi_dealloc_type_size;
     v->visitor.start_union = qapi_dealloc_start_union;
 
qapi/qapi-visit-core.c +6 -0
index 5a7c900..59ed506 100644
@@ -260,6 +260,12 @@ void visit_type_number(Visitor *v, double *obj, const char *name, Error **errp)
     v->type_number(v, obj, name, errp);
 }
 
+void visit_type_any(Visitor *v, QObject **obj, const char *name,
+                    Error **errp)
+{
+    v->type_any(v, obj, name, errp);
+}
+
 void output_type_enum(Visitor *v, int *obj, const char * const strings[],
                       const char *kind, const char *name,
                       Error **errp)
qapi/qmp-input-visitor.c +11 -0
index e97b8a4..5dd9ed5 100644
@@ -286,6 +286,16 @@ static void qmp_input_type_number(Visitor *v, double *obj, const char *name,
     }
 }
 
+static void qmp_input_type_any(Visitor *v, QObject **obj, const char *name,
+                               Error **errp)
+{
+    QmpInputVisitor *qiv = to_qiv(v);
+    QObject *qobj = qmp_input_get_object(qiv, name, true);
+
+    qobject_incref(qobj);
+    *obj = qobj;
+}
+
 static void qmp_input_optional(Visitor *v, bool *present, const char *name,
                                Error **errp)
 {
@@ -329,6 +339,7 @@ QmpInputVisitor *qmp_input_visitor_new(QObject *obj)
     v->visitor.type_bool = qmp_input_type_bool;
     v->visitor.type_str = qmp_input_type_str;
     v->visitor.type_number = qmp_input_type_number;
+    v->visitor.type_any = qmp_input_type_any;
     v->visitor.optional = qmp_input_optional;
     v->visitor.get_next_type = qmp_input_get_next_type;
 
qapi/qmp-output-visitor.c +9 -0
index efc19d5..1e36a7a 100644
@@ -186,6 +186,14 @@ static void qmp_output_type_number(Visitor *v, double *obj, const char *name,
     qmp_output_add(qov, name, qfloat_from_double(*obj));
 }
 
+static void qmp_output_type_any(Visitor *v, QObject **obj, const char *name,
+                                Error **errp)
+{
+    QmpOutputVisitor *qov = to_qov(v);
+    qobject_incref(*obj);
+    qmp_output_add_obj(qov, name, *obj);
+}
+
 QObject *qmp_output_get_qobject(QmpOutputVisitor *qov)
 {
     QObject *obj = qmp_output_first(qov);
@@ -233,6 +241,7 @@ QmpOutputVisitor *qmp_output_visitor_new(void)
     v->visitor.type_bool = qmp_output_type_bool;
     v->visitor.type_str = qmp_output_type_str;
     v->visitor.type_number = qmp_output_type_number;
+    v->visitor.type_any = qmp_output_type_any;
 
     QTAILQ_INIT(&v->stack);
 
scripts/qapi-types.py +1 -0
index b5ebeee..70a57c0 100644
@@ -305,6 +305,7 @@ fdef.write(mcgen('''
 fdecl.write(mcgen('''
 #include <stdbool.h>
 #include <stdint.h>
+#include "qapi/qmp/qobject.h"
 '''))
 
 schema = QAPISchema(input_file)
scripts/qapi.py +6 -3
index 0a325d5..a3e4b66 100644
@@ -33,6 +33,7 @@ builtin_types = {
     'uint32':   'QTYPE_QINT',
     'uint64':   'QTYPE_QINT',
     'size':     'QTYPE_QINT',
+    'any':      None,           # any qtype_code possible, actually
 }
 
 # Whitelist of commands allowed to return a non-dictionary
@@ -1036,8 +1037,7 @@ class QAPISchema(object):
 
     def _def_builtin_type(self, name, json_type, c_type, c_null):
         self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type, c_null))
-        if name != '**':
-            self._make_array_type(name) # TODO really needed?
+        self._make_array_type(name) # TODO really needed?
 
     def _def_predefineds(self):
         for t in [('str',    'string',  'char' + pointer_suffix, 'NULL'),
@@ -1053,8 +1053,9 @@ class QAPISchema(object):
                   ('uint64', 'int',     'uint64_t', '0'),
                   ('size',   'int',     'uint64_t', '0'),
                   ('bool',   'boolean', 'bool',     'false'),
-                  ('**',     'value',   None,       None)]:
+                  ('any',    'value',   'QObject' + pointer_suffix , 'NULL')]:
             self._def_builtin_type(*t)
+        self.entity_dict['**'] = self.lookup_type('any') # TODO drop this alias
 
     def _make_implicit_enum_type(self, name, values):
         name = name + 'Kind'
@@ -1201,6 +1202,8 @@ class QAPISchema(object):
     def visit(self, visitor):
         visitor.visit_begin(self)
         for name in sorted(self.entity_dict.keys()):
+            if self.entity_dict[name].name != name:
+                continue        # ignore alias TODO drop alias and remove
             self.entity_dict[name].visit(visitor)
         visitor.visit_end()
 
tests/Makefile +2 -1
index b128e28..fc6169a 100644
@@ -233,6 +233,7 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
 	args-array-empty.json args-array-unknown.json args-int.json \
 	args-unknown.json args-member-unknown.json args-member-array.json \
 	args-member-array-bad.json args-alternate.json args-union.json \
+	args-any.json \
 	returns-array-bad.json returns-int.json returns-dict.json \
 	returns-unknown.json returns-alternate.json returns-whitelist.json \
 	missing-colon.json missing-comma-list.json missing-comma-object.json \
@@ -249,7 +250,7 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
 	flat-union-invalid-branch-key.json flat-union-reverse-define.json \
 	flat-union-string-discriminator.json union-base-no-discriminator.json \
 	flat-union-bad-discriminator.json flat-union-bad-base.json \
-	flat-union-base-star.json \
+	flat-union-base-any.json \
 	flat-union-array-branch.json flat-union-int-branch.json \
 	flat-union-base-union.json flat-union-branch-clash.json \
 	alternate-nested.json alternate-unknown.json alternate-clash.json \
tests/qapi-schema/args-any.err new file +1 -0
new file mode 100644
index 0000000..bf9b5e0
@@ -0,0 +1 @@
+tests/qapi-schema/args-any.json:2: 'data' for command 'oops' cannot use built-in type 'any'
tests/qapi-schema/args-any.exit renamed from tests/qapi-schema/flat-union-base-star.exit +0 -0
similarity index 100%
rename from tests/qapi-schema/flat-union-base-star.exit
rename to tests/qapi-schema/args-any.exit
tests/qapi-schema/args-any.json new file +2 -0
new file mode 100644
index 0000000..58fe5e4
@@ -0,0 +1,2 @@
+# we do not allow an 'any' argument
+{ 'command': 'oops', 'data': 'any' }
tests/qapi-schema/args-any.out renamed from tests/qapi-schema/flat-union-base-star.out +0 -0
similarity index 100%
rename from tests/qapi-schema/flat-union-base-star.out
rename to tests/qapi-schema/args-any.out
tests/qapi-schema/flat-union-base-any.err new file +1 -0
new file mode 100644
index 0000000..ad4d629
@@ -0,0 +1 @@
+tests/qapi-schema/flat-union-base-any.json:8: Base 'any' is not a valid struct
tests/qapi-schema/flat-union-base-any.exit new file +1 -0
new file mode 100644
index 0000000..d00491f
@@ -0,0 +1 @@
+1
tests/qapi-schema/flat-union-base-any.json renamed from tests/qapi-schema/flat-union-base-star.json +1 -1
similarity index 95%
rename from tests/qapi-schema/flat-union-base-star.json
rename to tests/qapi-schema/flat-union-base-any.json
index 5099439..fe66b71 100644
@@ -6,7 +6,7 @@
 { 'struct': 'TestTypeB',
   'data': { 'integer': 'int' } }
 { 'union': 'TestUnion',
-  'base': '**',
+  'base': 'any',
   'discriminator': 'enum1',
   'data': { 'value1': 'TestTypeA',
             'value2': 'TestTypeB' } }
tests/qapi-schema/flat-union-base-any.out new file +0 -0
new file mode 100644
index 0000000..e69de29
tests/qapi-schema/flat-union-base-star.err deleted +0 -1
deleted file mode 100644
index b7748f0..0000000
@@ -1 +0,0 @@
-tests/qapi-schema/flat-union-base-star.json:8: Base '**' is not a valid struct
tests/qapi-schema/qapi-schema-test.json +4 -1
index 90b4740..e855018 100644
@@ -73,7 +73,8 @@
             'number': ['number'],
             'boolean': ['bool'],
             'string': ['str'],
-            'sizes': ['size'] } }
+            'sizes': ['size'],
+            'any': ['any'] } }
 
 # testing commands
 { 'command': 'user_def_cmd', 'data': {} }
@@ -83,6 +84,8 @@
   'returns': 'UserDefTwo' }
 { 'command': 'user_def_cmd3', 'data': {'a': 'int', '*b': 'int' },
   'returns': 'int' }
+# note: command name 'guest-sync' chosen to avoid "cannot use built-in" error
+{ 'command': 'guest-sync', 'data': { 'arg': 'any' }, 'returns': 'any' }
 
 # For testing integer range flattening in opts-visitor. The following schema
 # corresponds to the option format:
tests/qapi-schema/qapi-schema-test.out +8 -1
index b30bccc..fbb590f 100644
@@ -12,8 +12,12 @@ object :obj-__org.qemu_x-command-arg
     member b: __org.qemu_x-StructList optional=False
     member c: __org.qemu_x-Union2 optional=False
     member d: __org.qemu_x-Alt optional=False
+object :obj-anyList-wrapper
+    member data: anyList optional=False
 object :obj-boolList-wrapper
     member data: boolList optional=False
+object :obj-guest-sync-arg
+    member arg: any optional=False
 object :obj-int16List-wrapper
     member data: int16List optional=False
 object :obj-int32List-wrapper
@@ -100,7 +104,8 @@ object UserDefNativeListUnion
     case boolean: :obj-boolList-wrapper
     case string: :obj-strList-wrapper
     case sizes: :obj-sizeList-wrapper
-enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32', 's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes']
+    case any: :obj-anyList-wrapper
+enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32', 's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes', 'any']
 object UserDefOne
     base UserDefZero
     member string: str optional=False
@@ -149,6 +154,8 @@ object __org.qemu_x-Union2
     case __org.qemu_x-value: __org.qemu_x-Struct2
 command __org.qemu_x-command :obj-__org.qemu_x-command-arg -> __org.qemu_x-Union1
    gen=True success_response=True
+command guest-sync :obj-guest-sync-arg -> any
+   gen=True success_response=True
 command user_def_cmd None -> None
    gen=True success_response=True
 command user_def_cmd1 :obj-user_def_cmd1-arg -> None
tests/qapi-schema/type-bypass.out +2 -2
index 0070d4b..db2a4e6 100644
@@ -1,4 +1,4 @@
 object :obj-unsafe-arg
-    member arg: ** optional=False
-command unsafe :obj-unsafe-arg -> **
+    member arg: any optional=False
+command unsafe :obj-unsafe-arg -> any
    gen=False success_response=True
tests/test-qmp-commands.c +5 -0
index 9918f23..8d5249e 100644
@@ -51,6 +51,11 @@ int64_t qmp_user_def_cmd3(int64_t a, bool has_b, int64_t b, Error **errp)
     return a + (has_b ? b : 0);
 }
 
+QObject *qmp_guest_sync(QObject *arg, Error **errp)
+{
+    return arg;
+}
+
 __org_qemu_x_Union1 *qmp___org_qemu_x_command(__org_qemu_x_EnumList *a,
                                               __org_qemu_x_StructList *b,
                                               __org_qemu_x_Union2 *c,
tests/test-qmp-input-visitor.c +45 -0

57 lines not on this page · raw message

tests/test-qmp-output-visitor.c +53 -0

65 lines not on this page · raw message

Signature · 1 line
2.4.3
#51 lore raw

RFC PATCH v4 27/32 qom: Don't use 'gen': false for qom-get, qom-set, object-add

In reply to Markus Armbruster

With the previous commit, the generated marshalers just work, and save us a bit of handwritten code.

---
 include/monitor/monitor.h |  3 ---
 qapi-schema.json          |  9 +++------
 qmp-commands.hx           |  6 +++---
 qmp.c                     | 21 +++++++--------------
 scripts/qapi.py           |  1 +
 5 files changed, 14 insertions(+), 26 deletions(-)
include/monitor/monitor.h +0 -3
index 9aff47e..bc6cb6d 100644
@@ -42,9 +42,6 @@ void monitor_read_command(Monitor *mon, int show_prompt);
 int monitor_read_password(Monitor *mon, ReadLineFunc *readline_func,
                           void *opaque);
 
-void qmp_qom_set(QDict *qdict, QObject **ret, Error **errp);
-void qmp_qom_get(QDict *qdict, QObject **ret, Error **errp);
-void qmp_object_add(QDict *qdict, QObject **ret, Error **errp);
 void object_add(const char *type, const char *id, const QDict *qdict,
                 Visitor *v, Error **errp);
 
qapi-schema.json +3 -6
index 4342a08..17d2e60 100644
@@ -1698,8 +1698,7 @@
 ##
 { 'command': 'qom-get',
   'data': { 'path': 'str', 'property': 'str' },
-  'returns': '**',
-  'gen': false }
+  'returns': 'any' }
 
 ##
 # @qom-set:
@@ -1716,8 +1715,7 @@
 # Since: 1.2
 ##
 { 'command': 'qom-set',
-  'data': { 'path': 'str', 'property': 'str', 'value': '**' },
-  'gen': false }
+  'data': { 'path': 'str', 'property': 'str', 'value': 'any' } }
 
 ##
 # @set_password:
@@ -2113,8 +2111,7 @@
 # Since: 2.0
 ##
 { 'command': 'object-add',
-  'data': {'qom-type': 'str', 'id': 'str', '*props': '**'},
-  'gen': false }
+  'data': {'qom-type': 'str', 'id': 'str', '*props': 'any'} }
 
 ##
 # @object-del:
qmp-commands.hx +3 -3
index 41a39fc..76942a6 100644
@@ -928,7 +928,7 @@ EQMP
     {
         .name       = "object-add",
         .args_type  = "qom-type:s,id:s,props:q?",
-        .mhandler.cmd_new = qmp_object_add,
+        .mhandler.cmd_new = qmp_marshal_object_add,
     },
 
 SQMP
@@ -3550,13 +3550,13 @@ EQMP
     {
         .name       = "qom-set",
 	.args_type  = "path:s,property:s,value:q",
-	.mhandler.cmd_new = qmp_qom_set,
+	.mhandler.cmd_new = qmp_marshal_qom_set,
     },
 
     {
         .name       = "qom-get",
 	.args_type  = "path:s,property:s",
-	.mhandler.cmd_new = qmp_qom_get,
+	.mhandler.cmd_new = qmp_marshal_qom_get,
     },
 
     {
qmp.c +7 -14
index f31c6dc..b8b8b5b 100644
@@ -228,12 +228,9 @@ ObjectPropertyInfoList *qmp_qom_list(const char *path, Error **errp)
     return props;
 }
 
-/* FIXME: teach qapi about how to pass through Visitors */
-void qmp_qom_set(QDict *qdict, QObject **ret, Error **errp)
+void qmp_qom_set(const char *path, const char *property, QObject *value,
+                 Error **errp)
 {
-    const char *path = qdict_get_str(qdict, "path");
-    const char *property = qdict_get_str(qdict, "property");
-    QObject *value = qdict_get(qdict, "value");
     Object *obj;
 
     obj = object_resolve_path(path, NULL);
@@ -246,20 +243,18 @@ void qmp_qom_set(QDict *qdict, QObject **ret, Error **errp)
     object_property_set_qobject(obj, value, property, errp);
 }
 
-void qmp_qom_get(QDict *qdict, QObject **ret, Error **errp)
+QObject *qmp_qom_get(const char *path, const char *property, Error **errp)
 {
-    const char *path = qdict_get_str(qdict, "path");
-    const char *property = qdict_get_str(qdict, "property");
     Object *obj;
 
     obj = object_resolve_path(path, NULL);
     if (!obj) {
         error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
                   "Device '%s' not found", path);
-        return;
+        return NULL;
     }
 
-    *ret = object_property_get_qobject(obj, property, errp);
+    return object_property_get_qobject(obj, property, errp);
 }
 
 void qmp_set_password(const char *protocol, const char *password,
@@ -655,11 +650,9 @@ out:
     object_unref(obj);
 }
 
-void qmp_object_add(QDict *qdict, QObject **ret, Error **errp)
+void qmp_object_add(const char *type, const char *id,
+                    bool has_props, QObject *props, Error **errp)
 {
-    const char *type = qdict_get_str(qdict, "qom-type");
-    const char *id = qdict_get_str(qdict, "id");
-    QObject *props = qdict_get(qdict, "props");
     const QDict *pdict = NULL;
     QmpInputVisitor *qiv;
 
scripts/qapi.py +1 -0
index a3e4b66..ba1d6ec 100644
@@ -40,6 +40,7 @@ builtin_types = {
 returns_whitelist = [
     # From QMP:
     'human-monitor-command',
+    'qom-get',
     'query-migrate-cache-size',
     'query-tpm-models',
     'query-tpm-types',
Signature · 1 line
2.4.3
#52 lore raw

RFC PATCH v4 28/32 qapi-schema: Fix up misleading specification of netdev_add

In reply to Markus Armbruster

It doesn't take a 'props' argument, let alone one in the format "NAME=VALUE,..."

The bogus arguments specification doesn't matter due to 'gen': false. Clean it up to be incomplete rather than wrong, and document the incompleteness.

While there, improve netdev_add usage example in the manual: add a device option to show how it's done.

---
 docs/qapi-code-gen.txt |  2 +-
 qapi-schema.json       | 13 +++++++------
 qmp-commands.hx        |  4 +++-
 3 files changed, 11 insertions(+), 8 deletions(-)
docs/qapi-code-gen.txt +1 -1
index c713e3a..a5fccd4 100644
@@ -463,7 +463,7 @@ try to avoid adding new commands that rely on this, and instead use
 type-safe unions.  For an example of bypass usage:
 
  { 'command': 'netdev_add',
-   'data': {'type': 'str', 'id': 'str', '*props': '**'},
+   'data': {'type': 'str', 'id': 'str'},
    'gen': false }
 
 Normally, the QAPI schema is used to describe synchronous exchanges,
qapi-schema.json +7 -6
index 17d2e60..e91e3b9 100644
@@ -2065,11 +2065,12 @@
 #
 # @id: the name of the new network backend
 #
-# @props: #optional a list of properties to be passed to the backend in
-#         the format 'name=value', like 'ifname=tap0,script=no'
+# Additional arguments depend on the type.
 #
-# Notes: The semantics of @props is not well defined.  Future commands will be
-#        introduced that provide stronger typing for backend creation.
+# TODO This command effectively bypasses QAPI completely due to its
+# "additional arguments" business.  It shouldn't have been added to
+# the schema in this form.  It should be qapified properly, or
+# replaced by a properly qapified command.
 #
 # Since: 0.14.0
 #
@@ -2077,8 +2078,8 @@
 #          If @type is not a valid network backend, DeviceNotFound
 ##
 { 'command': 'netdev_add',
-  'data': {'type': 'str', 'id': 'str', '*props': '**'},
-  'gen': false }
+  'data': {'type': 'str', 'id': 'str'},
+  'gen': false }                # so we can get the additional arguments
 
 ##
 # @netdev_del:
qmp-commands.hx +3 -1
index 76942a6..b06d74c 100644
@@ -892,7 +892,9 @@ Arguments:
 
 Example:
 
--> { "execute": "netdev_add", "arguments": { "type": "user", "id": "netdev1" } }
+-> { "execute": "netdev_add",
+     "arguments": { "type": "user", "id": "netdev1",
+                    "dnssearch": "example.org" } }
 <- { "return": {} }
 
 Note: The supported device options are the same ones supported by the '-netdev'
Signature · 1 line
2.4.3
#53 lore raw

RFC PATCH v4 29/32 qapi: Pseudo-type '**' is now unused, drop it

In reply to Markus Armbruster

'gen': false needs to stay for now, because netdev_add is still using it.

---
 docs/qapi-code-gen.txt                    | 18 ++++++------------
 scripts/qapi.py                           | 20 ++++----------------
 tests/Makefile                            |  2 +-
 tests/qapi-schema/type-bypass-no-gen.err  |  1 -
 tests/qapi-schema/type-bypass-no-gen.exit |  1 -
 tests/qapi-schema/type-bypass-no-gen.json |  2 --
 tests/qapi-schema/type-bypass-no-gen.out  |  0
 tests/qapi-schema/type-bypass.err         |  0
 tests/qapi-schema/type-bypass.exit        |  1 -
 tests/qapi-schema/type-bypass.json        |  2 --
 tests/qapi-schema/type-bypass.out         |  4 ----
 11 files changed, 11 insertions(+), 40 deletions(-)
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.err
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.exit
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.json
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.out
 delete mode 100644 tests/qapi-schema/type-bypass.err
 delete mode 100644 tests/qapi-schema/type-bypass.exit
 delete mode 100644 tests/qapi-schema/type-bypass.json
 delete mode 100644 tests/qapi-schema/type-bypass.out
docs/qapi-code-gen.txt +6 -12
index a5fccd4..ce32d74 100644
@@ -111,10 +111,7 @@ and field names within a type, should be all lower case with words
 separated by a hyphen.  However, some existing older commands and
 complex types use underscore; when extending such expressions,
 consistency is preferred over blindly avoiding underscore.  Event
-names should be ALL_CAPS with words separated by underscore.  The
-special string '**' appears for some commands that manually perform
-their own type checking rather than relying on the type-safe code
-produced by the qapi code generators.
+names should be ALL_CAPS with words separated by underscore.
 
 Any name (command, event, type, field, or enum value) beginning with
 "x-" is marked experimental, and may be withdrawn or changed
@@ -453,14 +450,11 @@ which would validate this Client JSON Protocol transaction:
  <= { "return": [ { "value": "one" }, { } ] }
 
 In rare cases, QAPI cannot express a type-safe representation of a
-corresponding Client JSON Protocol command.  In these cases, if the
-command expression includes the key 'gen' with boolean value false,
-then the 'data' or 'returns' member that intends to bypass generated
-type-safety and do its own manual validation should use an inline
-dictionary definition, with a value of '**' rather than a valid type
-name for the keys that the generated code will not validate.  Please
-try to avoid adding new commands that rely on this, and instead use
-type-safe unions.  For an example of bypass usage:
+corresponding Client JSON Protocol command.  You then have to suppress
+generation of a marshalling function by including a key 'gen' with
+boolean value false, and instead write your own function.  Please try
+to avoid adding new commands that rely on this, and instead use
+type-safe unions.  For an example of this usage:
 
  { 'command': 'netdev_add',
    'data': {'type': 'str', 'id': 'str'},
scripts/qapi.py +4 -16
index ba1d6ec..7fe9f30 100644
@@ -428,15 +428,12 @@ def is_enum(name):
 
 def check_type(expr_info, source, value, allow_array = False,
                allow_dict = False, allow_optional = False,
-               allow_star = False, allow_metas = []):
+               allow_metas = []):
     global all_names
 
     if value is None:
         return
 
-    if allow_star and value == '**':
-        return
-
     # Check if array type for value is okay
     if isinstance(value, list):
         if not allow_array:
@@ -450,10 +447,6 @@ def check_type(expr_info, source, value, allow_array = False,
 
     # Check if type name for value is okay
     if isinstance(value, str):
-        if value == '**':
-            raise QAPIExprError(expr_info,
-                                "%s uses '**' but did not request 'gen':false"
-                                % source)
         if not value in all_names:
             raise QAPIExprError(expr_info,
                                 "%s uses unknown type '%s'"
@@ -479,7 +472,7 @@ def check_type(expr_info, source, value, allow_array = False,
         # Todo: allow dictionaries to represent default values of
         # an optional argument.
         check_type(expr_info, "Member '%s' of %s" % (key, source), arg,
-                   allow_array=True, allow_star=allow_star,
+                   allow_array=True,
                    allow_metas=['built-in', 'union', 'alternate', 'struct',
                                 'enum'])
 
@@ -499,18 +492,16 @@ def check_member_clash(expr_info, base_name, data, source = ""):
 
 def check_command(expr, expr_info):
     name = expr['command']
-    allow_star = expr.has_key('gen')
 
     check_type(expr_info, "'data' for command '%s'" % name,
                expr.get('data'), allow_dict=True, allow_optional=True,
-               allow_metas=['struct'], allow_star=allow_star)
+               allow_metas=['struct'])
     returns_meta = ['union', 'struct']
     if name in returns_whitelist:
         returns_meta += ['built-in', 'alternate', 'enum']
     check_type(expr_info, "'returns' for command '%s'" % name,
                expr.get('returns'), allow_array=True,
-               allow_optional=True, allow_metas=returns_meta,
-               allow_star=allow_star)
+               allow_optional=True, allow_metas=returns_meta)
 
 def check_event(expr, expr_info):
     global events
@@ -1056,7 +1047,6 @@ class QAPISchema(object):
                   ('bool',   'boolean', 'bool',     'false'),
                   ('any',    'value',   'QObject' + pointer_suffix , 'NULL')]:
             self._def_builtin_type(*t)
-        self.entity_dict['**'] = self.lookup_type('any') # TODO drop this alias
 
     def _make_implicit_enum_type(self, name, values):
         name = name + 'Kind'
@@ -1203,8 +1193,6 @@ class QAPISchema(object):
     def visit(self, visitor):
         visitor.visit_begin(self)
         for name in sorted(self.entity_dict.keys()):
-            if self.entity_dict[name].name != name:
-                continue        # ignore alias TODO drop alias and remove
             self.entity_dict[name].visit(visitor)
         visitor.visit_end()
 
tests/Makefile +1 -1
index fc6169a..68adad4 100644
@@ -228,7 +228,7 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
 	bad-type-dict.json double-data.json unknown-expr-key.json \
 	redefined-type.json redefined-command.json redefined-builtin.json \
 	redefined-event.json command-int.json bad-data.json event-max.json \
-	type-bypass.json type-bypass-no-gen.json type-bypass-bad-gen.json \
+	type-bypass-bad-gen.json \
 	args-invalid.json \
 	args-array-empty.json args-array-unknown.json args-int.json \
 	args-unknown.json args-member-unknown.json args-member-array.json \
tests/qapi-schema/type-bypass-no-gen.err deleted +0 -1
deleted file mode 100644
index 20cef0a..0000000
@@ -1 +0,0 @@
-tests/qapi-schema/type-bypass-no-gen.json:2: Member 'arg' of 'data' for command 'unsafe' uses '**' but did not request 'gen':false
tests/qapi-schema/type-bypass-no-gen.exit deleted +0 -1
deleted file mode 100644
index d00491f..0000000
@@ -1 +0,0 @@
-1
tests/qapi-schema/type-bypass-no-gen.json deleted +0 -2
deleted file mode 100644
index 4feae37..0000000
@@ -1,2 +0,0 @@
-# type bypass only works with 'gen':false
-{ 'command': 'unsafe', 'data': { 'arg': '**' }, 'returns': '**' }
tests/qapi-schema/type-bypass-no-gen.out deleted +0 -0
deleted file mode 100644
index e69de29..0000000
tests/qapi-schema/type-bypass.err deleted +0 -0
deleted file mode 100644
index e69de29..0000000
tests/qapi-schema/type-bypass.exit deleted +0 -1
deleted file mode 100644
index 573541a..0000000
@@ -1 +0,0 @@
-0
tests/qapi-schema/type-bypass.json deleted +0 -2
deleted file mode 100644
index 48b2137..0000000
@@ -1,2 +0,0 @@
-# Use of 'gen':false allows bypassing type system
-{ 'command': 'unsafe', 'data': { 'arg': '**' }, 'returns': '**', 'gen': false }
tests/qapi-schema/type-bypass.out deleted +0 -4
deleted file mode 100644
index db2a4e6..0000000
@@ -1,4 +0,0 @@
-object :obj-unsafe-arg
-    member arg: any optional=False
-command unsafe :obj-unsafe-arg -> any
-   gen=False success_response=True
Signature · 1 line
2.4.3
#54 lore raw

Re: [Qemu-devel] [PATCH RFC v4 29/32] qapi: Pseudo-type '**' is now unused, drop it

Quoted 9 lines · Markus Armbruster

'gen': false needs to stay for now, because netdev_add is still using it.

---

+++ b/tests/Makefile

tests/Makefile check-qapi-schema-y := $(addprefix tests/qapi-schema/, \ · line 234

Quoted 8 lines
tests/Makefile +1 -1
@@ -228,7 +228,7 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
 	bad-type-dict.json double-data.json unknown-expr-key.json \
 	redefined-type.json redefined-command.json redefined-builtin.json \
 	redefined-event.json command-int.json bad-data.json event-max.json \
-	type-bypass.json type-bypass-no-gen.json type-bypass-bad-gen.json \
+	type-bypass-bad-gen.json \
 	args-invalid.json \
 	args-array-empty.json args-array-unknown.json args-int.json \
 	args-unknown.json args-member-unknown.json args-member-array.json \

Not for this patch, but we aren't very consistent on any form of sorting or line length in this section. It might be nicer if it were one test per line (lots more line continuations) and/or alphabetical order. If that sounds nice, then it's a trivial patch to add in as one of the followups after this series lands.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#55 lore raw

Re: [Qemu-devel] [PATCH RFC v4 29/32] qapi: Pseudo-type '**' is now unused, drop it

Quoted 8 lines · Eric Blake
Quoted 6 lines · Markus Armbruster

'gen': false needs to stay for now, because netdev_add is still using it.

---

Quoted 16 lines

tests/Makefile check-qapi-schema-y := $(addprefix tests/qapi-schema/, \ · line 234

Quoted 10 lines

+++ b/tests/Makefile

tests/Makefile +1 -1
@@ -228,7 +228,7 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
 	bad-type-dict.json double-data.json unknown-expr-key.json \
 	redefined-type.json redefined-command.json redefined-builtin.json \
 	redefined-event.json command-int.json bad-data.json event-max.json \
-	type-bypass.json type-bypass-no-gen.json type-bypass-bad-gen.json \
+	type-bypass-bad-gen.json \
 	args-invalid.json \
 	args-array-empty.json args-array-unknown.json args-int.json \
 	args-unknown.json args-member-unknown.json args-member-array.json \

Not for this patch, but we aren't very consistent on any form of sorting or line length in this section. It might be nicer if it were one test per line (lots more line continuations) and/or alphabetical order. If that sounds nice, then it's a trivial patch to add in as one of the followups after this series lands.

Sorting: yes, please! One per line: meh.

#56 lore raw

RFC PATCH v4 30/32 qapi: New QMP command query-schema for QMP schema introspection

In reply to Markus Armbruster

qapi/introspect.json defines the introspection schema. It's designed for QMP introspection, but should do for similar uses, such as QGA.

The introspection schema does not reflect all the rules and restrictions that apply to QAPI schemata. A valid QAPI schema has an introspection value conforming to the introspection schema, but the converse is not true.

Introspection lowers away a number of schema details, and makes implicit things explicit:

* The built-in types are declared with their JSON type.

TODO Should we map all the integer types to just int?

* Implicit type definitions are made explicit, and given auto-generated names:

- Array types, named by appending "List" to the name of their element type, like in generated C.

- The enumeration types implicitly defined by simple union types, named by appending "Kind" to the name of their simple union type, like in generated C.

- Types that don't occur in generated C. Their names start with ':' so they don't clash with the user's names.

* All type references are by name.

* The struct and union types are generalized into an object type.

* Base types are flattened.

* Commands take a single argument and return a single result.

Dictionary argument or list result is an implicit type definition.

The empty object type is used when a command takes no arguments or produces no results.

The argument is always of object type, but the introspection schema doesn't reflect that.

The 'gen': false directive is omitted as implementation detail.

The 'success-response' directive is omitted as well for now, even though it's not an implementation detail, because it's not used by QMP.

* Events carry a single data value.

Implicit type definition and empty object type use, just like for commands.

The value is of object type, but the introspection schema doesn't reflect that.

* Types not used by commands or events are omitted.

Indirect use counts as use.

* Optional members have a default, which can only be null right now

Instead of a mandatory "optional" flag, we have an optional default. No default means mandatory, default null means optional without default value. Non-null is available for optional with default (possible future extension).

* Clients should *not* look up types by name, because type names are not ABI. Look up the command or event you're interested in, then follow the references.

TODO Should we hide the type names to eliminate the temptation?

New generator scripts/qapi-introspect.py computes an introspection value for its input, and generates a C variable holding it.

It can generate awfully long lines. Marked TODO.

A new test-qmp-input-visitor test case feeds its result for both tests/qapi-schema/qapi-schema-test.json and qapi-schema.json to a QmpInputVisitor to verify it actually conforms to the schema.

New QMP command query-schema takes its return value from that variable. Its reply is some 85KiBytes for me right now.

If this turns out to be too much, we have a couple of options:

* We can use shorter names in the JSON. Not the QMP style.

* Optionally return the sub-schema for commands and events given as arguments.

Right now qmp_query_schema() sends the string literal computed by qmp-introspect.py. To compute sub-schema at run time, we'd have to duplicate parts of qapi-introspect.py in C. Unattractive.

* Let clients cache the output of query-schema.

It changes only on QEMU upgrades, i.e. rarely. Provide a command query-schema-hash. Clients can have a cache indexed by hash, and re-query the schema only when they don't have it cached. Even
simpler: put the hash in the QMP greeting.

---
 .gitignore                                      |   1 +
 Makefile                                        |   9 +-
 Makefile.objs                                   |   4 +-
 docs/qapi-code-gen.txt                          | 226 +++++++++++++++++++-
 monitor.c                                       |  15 ++
 qapi-schema.json                                |   3 +
 qapi/introspect.json                            | 271 ++++++++++++++++++++++++
 qmp-commands.hx                                 |  17 ++
 scripts/qapi-introspect.py                      | 174 +++++++++++++++
 scripts/qapi.py                                 |  12 +-
 tests/.gitignore                                |   1 +
 tests/Makefile                                  |  10 +-
 tests/qapi-schema/alternate-good.out            |   1 +
 tests/qapi-schema/args-member-array.out         |   1 +
 tests/qapi-schema/comments.out                  |   1 +
 tests/qapi-schema/empty.out                     |   1 +
 tests/qapi-schema/enum-empty.out                |   1 +
 tests/qapi-schema/event-case.out                |   1 +
 tests/qapi-schema/flat-union-reverse-define.out |   1 +
 tests/qapi-schema/ident-with-escape.out         |   1 +
 tests/qapi-schema/include-relpath.out           |   1 +
 tests/qapi-schema/include-repetition.out        |   1 +
 tests/qapi-schema/include-simple.out            |   1 +
 tests/qapi-schema/indented-expr.out             |   1 +
 tests/qapi-schema/qapi-schema-test.out          |   1 +
 tests/qapi-schema/returns-int.out               |   1 +
 tests/test-qmp-input-strict.c                   |  55 +++++
 27 files changed, 801 insertions(+), 11 deletions(-)
 create mode 100644 qapi/introspect.json
 create mode 100644 scripts/qapi-introspect.py
.gitignore +1 -0
index cb4b8ec..ce02e3c 100644
@@ -34,6 +34,7 @@
 /qapi-visit.[ch]
 /qapi-event.[ch]
 /qmp-commands.h
+/qmp-introspect.[ch]
 /qmp-marshal.c
 /qemu-doc.html
 /qemu-tech.html
Makefile +8 -1
index 9ce3972..b053195 100644
@@ -52,6 +52,8 @@ endif
 GENERATED_HEADERS = config-host.h qemu-options.def
 GENERATED_HEADERS += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
 GENERATED_SOURCES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
+GENERATED_HEADERS += qmp-introspect.h
+GENERATED_SOURCES += qmp-introspect.c
 
 GENERATED_HEADERS += trace/generated-events.h
 GENERATED_SOURCES += trace/generated-events.c
@@ -264,7 +266,7 @@ $(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
 
 qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/block.json $(SRC_PATH)/qapi/block-core.json \
-               $(SRC_PATH)/qapi/event.json
+               $(SRC_PATH)/qapi/event.json $(SRC_PATH)/qapi/introspect.json
 
 qapi-types.c qapi-types.h :\
 $(qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
@@ -286,6 +288,11 @@ $(qapi-modules) $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
 	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
 		$(gen-out-type) -o "." -m $<, \
 		"  GEN   $@")
+qmp-introspect.h qmp-introspect.c :\
+$(qapi-modules) $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
+		$(gen-out-type) -o "." $<, \
+		"  GEN   $@")
 
 QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qmp-commands.h)
 $(qga-obj-y) qemu-ga.o: $(QGALIB_GEN)
Makefile.objs +3 -1
index f094eff..b5a9e99 100644
@@ -1,7 +1,8 @@
 #######################################################################
 # Common libraries for tools and emulators
 stub-obj-y = stubs/
-util-obj-y = util/ qobject/ qapi/ qapi-types.o qapi-visit.o qapi-event.o
+util-obj-y = util/ qobject/ qapi/
+util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o
 util-obj-y += crypto/
 
 #######################################################################
@@ -83,6 +84,7 @@ common-obj-$(CONFIG_FDT) += device_tree.o
 # qapi
 
 common-obj-y += qmp-marshal.o
+common-obj-y += qmp-introspect.o
 common-obj-y += qmp.o hmp.o
 endif
 
docs/qapi-code-gen.txt +221 -5

237 lines not on this page · raw message

monitor.c +15 -0
index f654607..47494c2 100644
@@ -74,6 +74,7 @@
 #include "block/qapi.h"
 #include "qapi/qmp-event.h"
 #include "qapi-event.h"
+#include "qmp-introspect.h"
 #include "sysemu/block-backend.h"
 
 /* for hmp_info_irq/pic */
@@ -924,6 +925,20 @@ EventInfoList *qmp_query_events(Error **errp)
     return ev_list;
 }
 
+/*
+ * Minor hack: generated marshalling suppressed for this command
+ * ('gen': false in the schema) so we can parse the JSON string
+ * directly into QObject instead of first parsing it with
+ * visit_type_SchemaInfoList() into a SchemaInfoList, then marshal it
+ * to QObject with generated output marshallers, every time.  Instead,
+ * we do it in test-qmp-input-visitor.c, just to make sure
+ * qapi-introspect.py's output actually conforms to the schema.
+ */
+static void qmp_query_schema(QDict *qdict, QObject **ret_data, Error **errp)
+{
+    *ret_data = qobject_from_json(qmp_schema_json);
+}
+
 /* set the current CPU defined by the user */
 int monitor_set_cpu(int cpu_index)
 {
qapi-schema.json +3 -0
index e91e3b9..af19810 100644
@@ -14,6 +14,9 @@
 # Tracing commands
 { 'include': 'qapi/trace.json' }
 
+# QAPI introspection
+{ 'include': 'qapi/introspect.json' }
+
 ##
 # @LostTickPolicy:
 #
qapi/introspect.json new file +271 -0

271 lines not on this page · raw message

qmp-commands.hx +17 -0
index b06d74c..6232ca0 100644
@@ -2168,6 +2168,23 @@ EQMP
     },
 
 SQMP
+query-schema
+------------
+
+Return the QMP wire schema.  The returned value is a json-array of
+named schema entities.  Entities are commands, events and various
+types.  See docs/qapi-code-gen.txt for information on their structure
+and intended use.
+
+EQMP
+
+    {
+        .name       = "query-schema",
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_query_schema,
+    },
+
+SQMP
 query-chardev
 -------------
 
scripts/qapi-introspect.py new file +174 -0

174 lines not on this page · raw message

scripts/qapi.py +10 -2
index 7fe9f30..8c651f9 100644
@@ -773,6 +773,8 @@ class QAPISchemaVisitor(object):
         pass
     def visit_object_type(self, name, info, base, members, variants):
         pass
+    def visit_object_type_flat(self, name, info, members, variants):
+        pass
     def visit_alternate_type(self, name, info, variants):
         pass
     def visit_command(self, name, info, arg_type, ret_type,
@@ -896,6 +898,8 @@ class QAPISchemaObjectType(QAPISchemaType):
     def visit(self, visitor):
         visitor.visit_object_type(self.name, self.info,
                                   self.base, self.local_members, self.variants)
+        visitor.visit_object_type_flat(self.name, self.info,
+                                       self.members, self.variants)
 
 class QAPISchemaObjectTypeMember(object):
     def __init__(self, name, typ, optional):
@@ -1047,6 +1051,9 @@ class QAPISchema(object):
                   ('bool',   'boolean', 'bool',     'false'),
                   ('any',    'value',   'QObject' + pointer_suffix , 'NULL')]:
             self._def_builtin_type(*t)
+        self.the_empty_object_type = QAPISchemaObjectType(':empty', None, None,
+                                                          [], None)
+        self._def_entity(self.the_empty_object_type)
 
     def _make_implicit_enum_type(self, name, values):
         name = name + 'Kind'
@@ -1191,9 +1198,10 @@ class QAPISchema(object):
             ent.check(self)
 
     def visit(self, visitor):
-        visitor.visit_begin(self)
+        ignore = visitor.visit_begin(self)
         for name in sorted(self.entity_dict.keys()):
-            self.entity_dict[name].visit(visitor)
+            if not ignore or not isinstance(self.entity_dict[name], ignore):
+                self.entity_dict[name].visit(visitor)
         visitor.visit_end()
 
 #
tests/.gitignore +1 -0
index ccc92e4..e476d52 100644
@@ -28,6 +28,7 @@ test-qmp-commands.h
 test-qmp-event
 test-qmp-input-strict
 test-qmp-input-visitor
+test-qmp-introspect.[ch]
 test-qmp-marshal.c
 test-qmp-output-visitor
 test-rcu-list
tests/Makefile +8 -2
index 68adad4..92d682f 100644
@@ -263,7 +263,8 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \
 	struct-base-clash.json struct-base-clash-deep.json )
 
 GENERATED_HEADERS += tests/test-qapi-types.h tests/test-qapi-visit.h \
-		     tests/test-qmp-commands.h tests/test-qapi-event.h
+	tests/test-qmp-commands.h tests/test-qapi-event.h \
+	tests/test-qmp-introspect.h
 
 test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \
 	tests/check-qlist.o tests/check-qfloat.o tests/check-qjson.o \
@@ -276,7 +277,7 @@ test-obj-y = tests/check-qint.o tests/check-qstring.o tests/check-qdict.o \
 	tests/rcutorture.o tests/test-rcu-list.o
 
 test-qapi-obj-y = tests/test-qapi-visit.o tests/test-qapi-types.o \
-		  tests/test-qapi-event.o
+		  tests/test-qapi-event.o tests/test-qmp-introspect.o
 
 $(test-obj-y): QEMU_INCLUDES += -Itests
 QEMU_CFLAGS += -I$(SRC_PATH)/tests
@@ -337,6 +338,11 @@ $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-eve
 	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \
 		$(gen-out-type) -o tests -p "test-" $<, \
 		"  GEN   $@")
+tests/test-qmp-introspect.c tests/test-qmp-introspect.h :\
+$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
+		$(gen-out-type) -o tests -p "test-" $<, \
+		"  GEN   $@")
 
 tests/test-string-output-visitor$(EXESUF): tests/test-string-output-visitor.o $(test-qapi-obj-y) libqemuutil.a libqemustub.a
 tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(test-qapi-obj-y) libqemuutil.a libqemustub.a
tests/qapi-schema/alternate-good.out +1 -0
index 3d765ff..65af727 100644
@@ -1,3 +1,4 @@
+object :empty
 alternate Alt
     case value: int
     case string: Enum
tests/qapi-schema/args-member-array.out +1 -0
index b67384c..b3b92df 100644
@@ -1,3 +1,4 @@
+object :empty
 object :obj-okay-arg
     member member1: intList optional=False
     member member2: defList optional=False
tests/qapi-schema/comments.out +1 -0
index 6161b90..9e2c656 100644
@@ -1 +1,2 @@
+object :empty
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/empty.out +1 -0
index e69de29..272b161 100644
@@ -0,0 +1 @@
+object :empty
tests/qapi-schema/enum-empty.out +1 -0
index e09b00f..a449d45 100644
@@ -1 +1,2 @@
+object :empty
 enum MyEnum []
tests/qapi-schema/event-case.out +1 -0
index b5ae4c2..cdfd264 100644
@@ -1 +1,2 @@
+object :empty
 event oops None
tests/qapi-schema/flat-union-reverse-define.out +1 -0
index 477fb31..a5a9134 100644
@@ -1,3 +1,4 @@
+object :empty
 object TestBase
     member enum1: TestEnum optional=False
 enum TestEnum ['value1', 'value2']
tests/qapi-schema/ident-with-escape.out +1 -0
index 9577d1b..f4542b1 100644
@@ -1,3 +1,4 @@
+object :empty
 object :obj-fooA-arg
     member bar1: str optional=False
 command fooA :obj-fooA-arg -> None
tests/qapi-schema/include-relpath.out +1 -0
index 6161b90..9e2c656 100644
@@ -1 +1,2 @@
+object :empty
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-repetition.out +1 -0
index 6161b90..9e2c656 100644
@@ -1 +1,2 @@
+object :empty
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-simple.out +1 -0
index 6161b90..9e2c656 100644
@@ -1 +1,2 @@
+object :empty
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/indented-expr.out +1 -0
index c5af55a..226d300 100644
@@ -1,3 +1,4 @@
+object :empty
 command eins None -> None
    gen=True success_response=True
 command zwei None -> None
tests/qapi-schema/qapi-schema-test.out +1 -0
index fbb590f..a9c87a0 100644
@@ -1,3 +1,4 @@
+object :empty
 object :obj-EVENT_C-arg
     member a: int optional=True
     member b: UserDefOne optional=True
tests/qapi-schema/returns-int.out +1 -0
index 1ac3e1e..a2da259 100644
@@ -1,2 +1,3 @@
+object :empty
 command guest-get-time None -> int
    gen=True success_response=True
tests/test-qmp-input-strict.c +55 -0

79 lines not on this page · raw message

Signature · 1 line
2.4.3
#57 lore raw

Re: [Qemu-devel] [PATCH RFC v4 30/32] qapi: New QMP command query-schema for QMP schema introspection

Quoted 2 lines · Markus Armbruster

qapi/introspect.json defines the introspection schema. It's designed for QMP introspection, but should do for similar uses, such as QGA.

[review to follow in separate message; I'm using this message to focus on one point for easier tracking of the sub-thread]

Quoted 6 lines

The introspection schema does not reflect all the rules and restrictions that apply to QAPI schemata. A valid QAPI schema has an introspection value conforming to the introspection schema, but the converse is not true.

Quoted 6 lines

Introspection lowers away a number of schema details, and makes implicit things explicit:

* The built-in types are declared with their JSON type.

TODO Should we map all the integer types to just int?

So, given the discussion on v3, I think we will have consensus if v5 does the following:

- Merge 30 and 31 into a single patch, and drop this TODO statement (I think we have agreement that exposing QMP [that is, JSON number] wire form, with no additional constraints, as a single 'int', rather than the underlying generated C types, is the way to go), provided that...
- Document in the qapi side that the schema intentionally lacks details on ranges, and that consulting the qapi and/or C code may be required to see what actual values are valid anywhere introspection merely says 'int',
- leave 32 as a separate patch, as it is complex enough, and still may have debate on whether the types '123' and 'int' should have corresponding array types '[123]' and '[int]' rather than the v3 patch munging of '456' and '789' (I haven't yet reviewed whether v4 changed that).

Any further arguments on whether exposing just 'int' in the introspection for all integral types, and/or whether patches 30 and 31 should be merged, are best made in response to this mail.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#58 lore raw

Re: [Qemu-devel] [PATCH RFC v4 30/32] qapi: New QMP command query-schema for QMP schema introspection

Quoting Eric Blake (2015-09-03 15:50:16)

Quoted 23 lines · Markus Armbruster
Quoted 2 lines

qapi/introspect.json defines the introspection schema. It's designed for QMP introspection, but should do for similar uses, such as QGA.

[review to follow in separate message; I'm using this message to focus on one point for easier tracking of the sub-thread]

Quoted 6 lines

The introspection schema does not reflect all the rules and restrictions that apply to QAPI schemata. A valid QAPI schema has an introspection value conforming to the introspection schema, but the converse is not true.

Quoted 6 lines

Introspection lowers away a number of schema details, and makes implicit things explicit:

* The built-in types are declared with their JSON type.

TODO Should we map all the integer types to just int?

So, given the discussion on v3, I think we will have consensus if v5 does the following:

Quoted 11 lines

- Merge 30 and 31 into a single patch, and drop this TODO statement (I think we have agreement that exposing QMP [that is, JSON number] wire form, with no additional constraints, as a single 'int', rather than the underlying generated C types, is the way to go), provided that...
- Document in the qapi side that the schema intentionally lacks details on ranges, and that consulting the qapi and/or C code may be required to see what actual values are valid anywhere introspection merely says 'int',
- leave 32 as a separate patch, as it is complex enough, and still may have debate on whether the types '123' and 'int' should have corresponding array types '[123]' and '[int]' rather than the v3 patch munging of '456' and '789' (I haven't yet reviewed whether v4 changed that).

Agreed on all 3

Quoted 4 lines

Any further arguments on whether exposing just 'int' in the introspection for all integral types, and/or whether patches 30 and 31 should be merged, are best made in response to this mail.

Given the above, I don't think I have any further objections.

Quoted 5 lines
Signature · 3 lines
-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#59 lore raw

Re: [Qemu-devel] [PATCH RFC v4 30/32] qapi: New QMP command query-schema for QMP schema introspection

Quoted 31 lines · Michael Roth

Quoting Eric Blake (2015-09-03 15:50:16)

Quoted 30 lines · Markus Armbruster
Quoted 2 lines

qapi/introspect.json defines the introspection schema. It's designed for QMP introspection, but should do for similar uses, such as QGA.

[review to follow in separate message; I'm using this message to focus on one point for easier tracking of the sub-thread]

Quoted 6 lines

The introspection schema does not reflect all the rules and restrictions that apply to QAPI schemata. A valid QAPI schema has an introspection value conforming to the introspection schema, but the converse is not true.

Quoted 6 lines

Introspection lowers away a number of schema details, and makes implicit things explicit:

* The built-in types are declared with their JSON type.

TODO Should we map all the integer types to just int?

So, given the discussion on v3, I think we will have consensus if v5 does the following:

- Merge 30 and 31 into a single patch, and drop this TODO statement (I think we have agreement that exposing QMP [that is, JSON number] wire form, with no additional constraints, as a single 'int', rather than the underlying generated C types, is the way to go), provided that...
- Document in the qapi side that the schema intentionally lacks details on ranges, and that consulting the qapi and/or C code may be required to see what actual values are valid anywhere introspection merely says 'int',

Please review docs/qapi-code-gen.txt in this patch for this aspect.

Quoted 6 lines
Quoted 4 lines

- leave 32 as a separate patch, as it is complex enough, and still may have debate on whether the types '123' and 'int' should have corresponding array types '[123]' and '[int]' rather than the v3 patch munging of '456' and '789' (I haven't yet reviewed whether v4 changed that).

Agreed on all 3

Excellent.

Quoted 5 lines
Quoted 3 lines

Any further arguments on whether exposing just 'int' in the introspection for all integral types, and/or whether patches 30 and 31 should be merged, are best made in response to this mail.

Given the above, I don't think I have any further objections.

Thanks!

#60 lore raw

Re: [Qemu-devel] [PATCH RFC v4 30/32] qapi: New QMP command query-schema for QMP schema introspection

In reply to Markus Armbruster

Quoting Markus Armbruster (2015-09-03 09:30:21)

Quoted 138 lines

qapi/introspect.json defines the introspection schema. It's designed for QMP introspection, but should do for similar uses, such as QGA.

The introspection schema does not reflect all the rules and restrictions that apply to QAPI schemata. A valid QAPI schema has an introspection value conforming to the introspection schema, but the converse is not true.

Introspection lowers away a number of schema details, and makes implicit things explicit:

* The built-in types are declared with their JSON type.

TODO Should we map all the integer types to just int?

* Implicit type definitions are made explicit, and given auto-generated names:

- Array types, named by appending "List" to the name of their element type, like in generated C.

- The enumeration types implicitly defined by simple union types, named by appending "Kind" to the name of their simple union type, like in generated C.

- Types that don't occur in generated C. Their names start with ':' so they don't clash with the user's names.

* All type references are by name.

* The struct and union types are generalized into an object type.

* Base types are flattened.

* Commands take a single argument and return a single result.

Dictionary argument or list result is an implicit type definition.

The empty object type is used when a command takes no arguments or produces no results.

The argument is always of object type, but the introspection schema doesn't reflect that.

The 'gen': false directive is omitted as implementation detail.

The 'success-response' directive is omitted as well for now, even though it's not an implementation detail, because it's not used by QMP.

* Events carry a single data value.

Implicit type definition and empty object type use, just like for commands.

The value is of object type, but the introspection schema doesn't reflect that.

* Types not used by commands or events are omitted.

Indirect use counts as use.

* Optional members have a default, which can only be null right now

Instead of a mandatory "optional" flag, we have an optional default. No default means mandatory, default null means optional without default value. Non-null is available for optional with default (possible future extension).

* Clients should *not* look up types by name, because type names are not ABI. Look up the command or event you're interested in, then follow the references.

TODO Should we hide the type names to eliminate the temptation?

New generator scripts/qapi-introspect.py computes an introspection value for its input, and generates a C variable holding it.

It can generate awfully long lines. Marked TODO.

A new test-qmp-input-visitor test case feeds its result for both tests/qapi-schema/qapi-schema-test.json and qapi-schema.json to a QmpInputVisitor to verify it actually conforms to the schema.

New QMP command query-schema takes its return value from that variable. Its reply is some 85KiBytes for me right now.

If this turns out to be too much, we have a couple of options:

* We can use shorter names in the JSON. Not the QMP style.

* Optionally return the sub-schema for commands and events given as arguments.

Right now qmp_query_schema() sends the string literal computed by qmp-introspect.py. To compute sub-schema at run time, we'd have to duplicate parts of qapi-introspect.py in C. Unattractive.

* Let clients cache the output of query-schema.

It changes only on QEMU upgrades, i.e. rarely. Provide a command query-schema-hash. Clients can have a cache indexed by hash, and re-query the schema only when they don't have it cached. Even
simpler: put the hash in the QMP greeting.

---
 .gitignore                                      |   1 +
 Makefile                                        |   9 +-
 Makefile.objs                                   |   4 +-
 docs/qapi-code-gen.txt                          | 226 +++++++++++++++++++-
 monitor.c                                       |  15 ++
 qapi-schema.json                                |   3 +
 qapi/introspect.json                            | 271 ++++++++++++++++++++++++
 qmp-commands.hx                                 |  17 ++
 scripts/qapi-introspect.py                      | 174 +++++++++++++++
 scripts/qapi.py                                 |  12 +-
 tests/.gitignore                                |   1 +
 tests/Makefile                                  |  10 +-
 tests/qapi-schema/alternate-good.out            |   1 +
 tests/qapi-schema/args-member-array.out         |   1 +
 tests/qapi-schema/comments.out                  |   1 +
 tests/qapi-schema/empty.out                     |   1 +
 tests/qapi-schema/enum-empty.out                |   1 +
 tests/qapi-schema/event-case.out                |   1 +
 tests/qapi-schema/flat-union-reverse-define.out |   1 +
 tests/qapi-schema/ident-with-escape.out         |   1 +
 tests/qapi-schema/include-relpath.out           |   1 +
 tests/qapi-schema/include-repetition.out        |   1 +
 tests/qapi-schema/include-simple.out            |   1 +
 tests/qapi-schema/indented-expr.out             |   1 +
 tests/qapi-schema/qapi-schema-test.out          |   1 +
 tests/qapi-schema/returns-int.out               |   1 +
 tests/test-qmp-input-strict.c                   |  55 +++++
 27 files changed, 801 insertions(+), 11 deletions(-)
 create mode 100644 qapi/introspect.json
 create mode 100644 scripts/qapi-introspect.py

<snip>

Quoted 35 lines
qapi-schema.json +3 -0
index e91e3b9..af19810 100644
@@ -14,6 +14,9 @@
 # Tracing commands
 { 'include': 'qapi/trace.json' }
 
+# QAPI introspection
+{ 'include': 'qapi/introspect.json' }
+
 ##
 # @LostTickPolicy:
 #
qapi/introspect.json new file +15 -0
new file mode 100644
index 0000000..171baba
@@ -0,0 +1,271 @@
+# -*- Mode: Python -*-
+#
+# QAPI/QMP introspection
+#
+# Copyright (C) 2015 Red Hat, Inc.
+#
+# Authors:
+#  Markus Armbruster <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+##
+# @query-schema
+#

qapi/introspect.json line 25

Quoted 10 lines
qapi/introspect.json new file +10 -0
+# Command query-schema exposes the QMP wire ABI as an array of
+# SchemaInfo.  This lets QMP clients figure out what commands and
+# events are available in this QEMU, and their parameters and results.
+#
+# Returns: array of @SchemaInfo, where each element describes an
+# entity in the ABI: command, event, type, ...
+#
+# Note: the QAPI schema is also used to help define *internal*
+# interfaces, by defining QAPI types.  These are not part of the QMP
+# wire ABI, and therefore not returned by this command.

Maybe add something like:

"These internal interfaces may place additional restrictions on the values of individual fields, so users should reference the QAPI schema to avoid unexpected behavior resulting from invalid field values."

<snip>

Quoted 5 lines
qmp-commands.hx +0 -0
index b06d74c..6232ca0 100644

qmp-commands.hx EQMP · line 2177

Quoted 10 lines
qmp-commands.hx +7 -0
@@ -2168,6 +2168,23 @@ EQMP
     },
 
 SQMP
+query-schema
+------------
+
+Return the QMP wire schema.  The returned value is a json-array of
+named schema entities.  Entities are commands, events and various
+types.  See docs/qapi-code-gen.txt for information on their structure
+and intended use.

We should probably duplicate any notes from QAPI schema description here as well, or at least direct users to reference them.

Quoted 68 lines
qmp-commands.hx +10 -0
+
+EQMP
+
+    {
+        .name       = "query-schema",
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_query_schema,
+    },
+
+SQMP
 query-chardev
 -------------
 
scripts/qapi-introspect.py new file +49 -0
new file mode 100644
index 0000000..4bcffc4
@@ -0,0 +1,174 @@
+#
+# QAPI introspection generator
+#
+# Copyright (C) 2015 Red Hat, Inc.
+#
+# Authors:
+#  Markus Armbruster <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from qapi import *
+import string
+
+# Caveman's json.dumps() replacement (we're stuck at 2.4)
+# TODO try to use json.dumps() once we get unstuck
+def to_json(obj, level=0):
+    if obj == None:
+        ret = 'null'
+    elif isinstance(obj, str):
+        ret = '"' + obj.replace('"', r'\"') + '"'
+    elif isinstance(obj, list):
+        elts = [to_json(elt, level + 1)
+                for elt in obj]
+        ret = '[' + ', '.join(elts) + ']'
+    elif isinstance(obj, dict):
+        elts = ['"%s": %s' % (key, to_json(obj[key], level + 1))
+                for key in sorted(obj.keys())]
+        ret = '{' + ', '.join(elts) + '}'
+    else:
+        assert False                # not implemented
+    if level == 1:
+        ret = '\n' + ret
+    return ret
+
+def to_c_string(string):
+    return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"'
+
+class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.defn = None
+        self.decl = None
+        self.schema = None
+        self.jsons = None
+        self.used_types = None
+
+    def visit_begin(self, schema):
+        self.schema = schema
+        self.jsons = []

scripts/qapi-introspect.py line 59

Quoted 10 lines
scripts/qapi-introspect.py new file +10 -0
+        self.used_types = []
+        return QAPISchemaType   # don't visit types for now
+
+    def visit_end(self):
+        # visit the types that are actually used
+        for typ in self.used_types:
+            typ.visit(self)
+        self.jsons.sort()
+        # generate C
+        # TODO can generate awfully long lines

Not sure if this is planned for this series, but a multi-line representation in the generated files would make things a bit easier to review/check. If we went a little further and made them pretty-printed it might make some of the docs referencing the output a bit more clear as well. Not a huge deal, but don't think it would take much code. Would suggest a JSON library or something but that would probably mangle the ordering, which wouldn't help with readability.

Quoted 15 lines
scripts/qapi-introspect.py new file +15 -0
+        name = prefix + 'qmp_schema_json'
+        self.decl = mcgen('''
+extern const char %(c_name)s[];
+''',
+                          c_name=c_name(name))
+        lines = to_json(self.jsons).split('\n')
+        c_string = '\n    '.join([to_c_string(line) for line in lines])
+        self.defn = mcgen('''
+const char %(c_name)s[] = %(c_string)s;
+''',
+                          c_name=c_name(name),
+                          c_string=c_string)
+        self.schema = None
+        self.jsons = None
+        self.used_types = None
#61 lore raw

Re: [Qemu-devel] [PATCH RFC v4 30/32] qapi: New QMP command query-schema for QMP schema introspection

Quoted 190 lines · Michael Roth

Quoting Markus Armbruster (2015-09-03 09:30:21)

Quoted 138 lines

qapi/introspect.json defines the introspection schema. It's designed for QMP introspection, but should do for similar uses, such as QGA.

The introspection schema does not reflect all the rules and restrictions that apply to QAPI schemata. A valid QAPI schema has an introspection value conforming to the introspection schema, but the converse is not true.

Introspection lowers away a number of schema details, and makes implicit things explicit:

* The built-in types are declared with their JSON type.

TODO Should we map all the integer types to just int?

* Implicit type definitions are made explicit, and given auto-generated names:

- Array types, named by appending "List" to the name of their element type, like in generated C.

- The enumeration types implicitly defined by simple union types, named by appending "Kind" to the name of their simple union type, like in generated C.

- Types that don't occur in generated C. Their names start with ':' so they don't clash with the user's names.

* All type references are by name.

* The struct and union types are generalized into an object type.

* Base types are flattened.

* Commands take a single argument and return a single result.

Dictionary argument or list result is an implicit type definition.

The empty object type is used when a command takes no arguments or produces no results.

The argument is always of object type, but the introspection schema doesn't reflect that.

The 'gen': false directive is omitted as implementation detail.

The 'success-response' directive is omitted as well for now, even though it's not an implementation detail, because it's not used by QMP.

* Events carry a single data value.

Implicit type definition and empty object type use, just like for commands.

The value is of object type, but the introspection schema doesn't reflect that.

* Types not used by commands or events are omitted.

Indirect use counts as use.

* Optional members have a default, which can only be null right now

Instead of a mandatory "optional" flag, we have an optional default. No default means mandatory, default null means optional without default value. Non-null is available for optional with default (possible future extension).

* Clients should *not* look up types by name, because type names are not ABI. Look up the command or event you're interested in, then follow the references.

TODO Should we hide the type names to eliminate the temptation?

New generator scripts/qapi-introspect.py computes an introspection value for its input, and generates a C variable holding it.

It can generate awfully long lines. Marked TODO.

A new test-qmp-input-visitor test case feeds its result for both tests/qapi-schema/qapi-schema-test.json and qapi-schema.json to a QmpInputVisitor to verify it actually conforms to the schema.

New QMP command query-schema takes its return value from that variable. Its reply is some 85KiBytes for me right now.

If this turns out to be too much, we have a couple of options:

* We can use shorter names in the JSON. Not the QMP style.

* Optionally return the sub-schema for commands and events given as arguments.

Right now qmp_query_schema() sends the string literal computed by qmp-introspect.py. To compute sub-schema at run time, we'd have to duplicate parts of qapi-introspect.py in C. Unattractive.

* Let clients cache the output of query-schema.

It changes only on QEMU upgrades, i.e. rarely. Provide a command query-schema-hash. Clients can have a cache indexed by hash, and re-query the schema only when they don't have it cached. Even
simpler: put the hash in the QMP greeting.

---
 .gitignore                                      |   1 +
 Makefile                                        |   9 +-
 Makefile.objs                                   |   4 +-
 docs/qapi-code-gen.txt                          | 226 +++++++++++++++++++-
 monitor.c                                       |  15 ++
 qapi-schema.json                                |   3 +
 qapi/introspect.json                            | 271 ++++++++++++++++++++++++
 qmp-commands.hx                                 |  17 ++
 scripts/qapi-introspect.py                      | 174 +++++++++++++++
 scripts/qapi.py                                 |  12 +-
 tests/.gitignore                                |   1 +
 tests/Makefile                                  |  10 +-
 tests/qapi-schema/alternate-good.out            |   1 +
 tests/qapi-schema/args-member-array.out         |   1 +
 tests/qapi-schema/comments.out                  |   1 +
 tests/qapi-schema/empty.out                     |   1 +
 tests/qapi-schema/enum-empty.out                |   1 +
 tests/qapi-schema/event-case.out                |   1 +
 tests/qapi-schema/flat-union-reverse-define.out |   1 +
 tests/qapi-schema/ident-with-escape.out         |   1 +
 tests/qapi-schema/include-relpath.out           |   1 +
 tests/qapi-schema/include-repetition.out        |   1 +
 tests/qapi-schema/include-simple.out            |   1 +
 tests/qapi-schema/indented-expr.out             |   1 +
 tests/qapi-schema/qapi-schema-test.out          |   1 +
 tests/qapi-schema/returns-int.out               |   1 +
 tests/test-qmp-input-strict.c                   |  55 +++++
 27 files changed, 801 insertions(+), 11 deletions(-)
 create mode 100644 qapi/introspect.json
 create mode 100644 scripts/qapi-introspect.py

<snip>

Quoted 35 lines
qapi-schema.json +3 -0
index e91e3b9..af19810 100644
@@ -14,6 +14,9 @@
 # Tracing commands
 { 'include': 'qapi/trace.json' }
 
+# QAPI introspection
+{ 'include': 'qapi/introspect.json' }
+
 ##
 # @LostTickPolicy:
 #
qapi/introspect.json new file +15 -0
new file mode 100644
index 0000000..171baba
@@ -0,0 +1,271 @@
+# -*- Mode: Python -*-
+#
+# QAPI/QMP introspection
+#
+# Copyright (C) 2015 Red Hat, Inc.
+#
+# Authors:
+#  Markus Armbruster <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+##
+# @query-schema
+#

qapi/introspect.json line 25

Quoted 10 lines
qapi/introspect.json new file +10 -0
+# Command query-schema exposes the QMP wire ABI as an array of
+# SchemaInfo.  This lets QMP clients figure out what commands and
+# events are available in this QEMU, and their parameters and results.
+#
+# Returns: array of @SchemaInfo, where each element describes an
+# entity in the ABI: command, event, type, ...
+#
+# Note: the QAPI schema is also used to help define *internal*
+# interfaces, by defining QAPI types.  These are not part of the QMP
+# wire ABI, and therefore not returned by this command.

Maybe add something like:

Quoted 3 lines

"These internal interfaces may place additional restrictions on the values of individual fields, so users should reference the QAPI schema to avoid unexpected behavior resulting from invalid field values."

Hmm, my paragraph tries to explain something else, namely that the QAPI schema defines purely internal interfaces in addition to QMP interfaces. The latter consist of the QMP wire interface and the internal QMP interface. Generated code translates between the two.

Furthermore, "additional restrictions" apply not only to "values of individual fields", but also to structure. For instance, SchemaInfoObject members tag and variants must both be present or both be absent.

What about having this somewhere:

The SchemaInfo can't reflect all the rules and restrictions that apply to QMP. It's interface introspection (figuring out what's there), not interface specification. The specification is in the QAPI schema, and users should peruse it to understand how QMP is to be used.

Then continue with an example showing a restriction that's spelled out in the QAPI schema, but can't be expressed in SchemaInfo.

Quoted 9 lines

<snip>

Quoted 7 lines
qmp-commands.hx +0 -0
index b06d74c..6232ca0 100644
@@ -2168,6 +2168,23 @@ EQMP
     },
 
Quoted 11 lines

qmp-commands.hx EQMP · line 2177

Quoted 8 lines
qmp-commands.hx +7 -0
 SQMP
+query-schema
+------------
+
+Return the QMP wire schema.  The returned value is a json-array of
+named schema entities.  Entities are commands, events and various
+types.  See docs/qapi-code-gen.txt for information on their structure
+and intended use.

We should probably duplicate any notes from QAPI schema description here as well, or at least direct users to reference them.

qmp-commands.hx needs to die.

Until it's dead, some information needs to be duplicated there. It already points to docs/qapi-code-gen.txt, which in turn points to the QAPI schema. I can add a direct pointer if you think it'll help.

Quoted 79 lines
Quoted 68 lines
qmp-commands.hx +10 -0
+
+EQMP
+
+    {
+        .name       = "query-schema",
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_query_schema,
+    },
+
+SQMP
 query-chardev
 -------------
 
scripts/qapi-introspect.py new file +49 -0
new file mode 100644
index 0000000..4bcffc4
@@ -0,0 +1,174 @@
+#
+# QAPI introspection generator
+#
+# Copyright (C) 2015 Red Hat, Inc.
+#
+# Authors:
+#  Markus Armbruster <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.
+# See the COPYING file in the top-level directory.
+
+from qapi import *
+import string
+
+# Caveman's json.dumps() replacement (we're stuck at 2.4)
+# TODO try to use json.dumps() once we get unstuck
+def to_json(obj, level=0):
+    if obj == None:
+        ret = 'null'
+    elif isinstance(obj, str):
+        ret = '"' + obj.replace('"', r'\"') + '"'
+    elif isinstance(obj, list):
+        elts = [to_json(elt, level + 1)
+                for elt in obj]
+        ret = '[' + ', '.join(elts) + ']'
+    elif isinstance(obj, dict):
+        elts = ['"%s": %s' % (key, to_json(obj[key], level + 1))
+                for key in sorted(obj.keys())]
+        ret = '{' + ', '.join(elts) + '}'
+    else:
+        assert False                # not implemented
+    if level == 1:
+        ret = '\n' + ret
+    return ret
+
+def to_c_string(string):
+    return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"'
+
+class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
+    def __init__(self):
+        self.defn = None
+        self.decl = None
+        self.schema = None
+        self.jsons = None
+        self.used_types = None
+
+    def visit_begin(self, schema):
+        self.schema = schema
+        self.jsons = []

scripts/qapi-introspect.py line 59

Quoted 10 lines
scripts/qapi-introspect.py new file +10 -0
+        self.used_types = []
+        return QAPISchemaType   # don't visit types for now
+
+    def visit_end(self):
+        # visit the types that are actually used
+        for typ in self.used_types:
+            typ.visit(self)
+        self.jsons.sort()
+        # generate C
+        # TODO can generate awfully long lines
Quoted 8 lines

Not sure if this is planned for this series, but a multi-line representation in the generated files would make things a bit easier to review/check. If we went a little further and made them pretty-printed it might make some of the docs referencing the output a bit more clear as well. Not a huge deal, but don't think it would take much code. Would suggest a JSON library or something but that would probably mangle the ordering, which wouldn't help with readability.

The obvious Python JSON library is "json — JSON encoder and decoder"[*], which provides pretty-printing, but it's 2.6, and we're stuck in 2.4.

I tried to structure the code in a way that makes dropping in the library relatively painless, see function to_json() above.

Quoted 7 lines
Quoted 7 lines
scripts/qapi-introspect.py new file +7 -0
+        name = prefix + 'qmp_schema_json'
+        self.decl = mcgen('''
+extern const char %(c_name)s[];
+''',
+                          c_name=c_name(name))
+        lines = to_json(self.jsons).split('\n')
+        c_string = '\n    '.join([to_c_string(line) for line in lines])
Quoted 8 lines

scripts/qapi-introspect.py line 74

Quoted 8 lines
scripts/qapi-introspect.py new file +8 -0
+        self.defn = mcgen('''
+const char %(c_name)s[] = %(c_string)s;
+''',
+                          c_name=c_name(name),
+                          c_string=c_string)
+        self.schema = None
+        self.jsons = None
+        self.used_types = None

[*] https://docs.python.org/2.7/library/json.html

Reviewed-by on v4 30/32
#62 lore raw

Re: [Qemu-devel] [PATCH RFC v4 30/32] qapi: New QMP command query-schema for QMP schema introspection

In reply to Markus Armbruster
Quoted 2 lines · Markus Armbruster

qapi/introspect.json defines the introspection schema. It's designed for QMP introspection, but should do for similar uses, such as QGA.

[review in this sub-thread; for comments on 'int' munging or other followups, see other subthread]

There is at least one definite bug (see multiple references below to [1]), and several ideas for cleanups, but in general, I think that the remaining changes are going to be small enough that I'd probably be okay if v5 started life with:

(Of course, you have every right to not add the R-b, especially if you want me to do another close review :)

Quoted 9 lines

The introspection schema does not reflect all the rules and restrictions that apply to QAPI schemata. A valid QAPI schema has an introspection value conforming to the introspection schema, but the converse is not true.

The empty object type is used when a command takes no arguments or produces no results.

We had discussed whether to expose the empty type in a separate patch, to reduce the size of this patch; but at this point in the game, I'm happy to keep it as-is (the faster we get v5 out for review and into the tree, the faster we can process the backlog of followup patches).

Quoted 7 lines

---

+++ b/docs/qapi-code-gen.txt

docs/qapi-code-gen.txt Resulting in this JSON object: · line 502

Quoted 9 lines
docs/qapi-code-gen.txt +6 -0
@@ -494,13 +494,195 @@ Resulting in this JSON object:
   "timestamp": { "seconds": 1267020223, "microseconds": 435656 } }
 
 
+== Client JSON Protocol introspection ==
+
+Clients of a Client JSON Protocol commonly need to figure out what
+exactly the server (QEMU) supports.
+
+For this purpose, QMP provides introspection via command query-schema.

Will need tweaking to the agreed-on name query-qmp-schema...

docs/qapi-code-gen.txt Resulting in this JSON object: · line 503

Quoted 1 line
docs/qapi-code-gen.txt +1 -0
+QGA currently doesn't support introspection.

... (and we'd obviously want qga to eventually gain a matching counterpart, possibly 'guest-schema' or 'guest-get-schema', to match prevailing naming in qga/qapi-schema.json).

docs/qapi-code-gen.txt Resulting in this JSON object: · line 506

Quoted 3 lines
docs/qapi-code-gen.txt +3 -0
+
+query-schema returns a JSON array of SchemaInfo objects.  These
+objects together describe the wire ABI, as defined in the QAPI schema.

Maybe an additional sentence here, perhaps along the lines of:

Note that the wire format cannot express everything; it is not designed to show semantic constraints (such as when exactly one of a pair of mutually-exclusive optional members must be present, or when an integral value has specific limitations on valid values). Introspection allows an application to know if a feature is present, but the application must obey the qapi documentation to properly interact with that feature.

Quoted 5 lines
docs/qapi-code-gen.txt +5 -0
+
+Like any other command, query-schema is itself defined in the QAPI
+schema, along with the SchemaInfo type.  This text attempts to give an
+overview how things work.  For details you need to consult the QAPI
+schema.

docs/qapi-code-gen.txt Resulting in this JSON object: · line 521

Quoted 10 lines
docs/qapi-code-gen.txt +10 -0
+
+SchemaInfo objects have common members "name" and "meta-type", and
+additional variant members depending on the value of meta-type.
+
+Each SchemaInfo object describes a wire ABI entity of a certain
+meta-type: a command, event or one of several kinds of type.
+
+SchemaInfo for entities defined in the QAPI schema have the same name
+as in the schema.  This is the case for all commands and events, and
+most types.

And hopefully patch 32 munges this text to match the simplification made there (I haven't checked yet).

docs/qapi-code-gen.txt Resulting in this JSON object: · line 530

Quoted 9 lines
docs/qapi-code-gen.txt +9 -0
+
+Command and event names are part of the wire ABI, but type names are
+not.  Therefore, looking up a type by its name in the QAPI schema is
+wrong.  Look up the command or event, then follow references by name.
+
+QAPI schema definitions not reachable that way are omitted.
+
+The SchemaInfo for a command has meta-type "command", and variant
+members "arg-type" and "ret-type".  The arguments member clients pass

I found it easier to read with:
s/arguments/arguments that/

docs/qapi-code-gen.txt Resulting in this JSON object: · line 532

Quoted 2 lines
docs/qapi-code-gen.txt +2 -0
+with a command on the wire must conform to the type named by
+"arg-type", which is always an object type.  The return value the

Likewise s/value/value that/

docs/qapi-code-gen.txt Resulting in this JSON object: · line 535

Quoted 3 lines
docs/qapi-code-gen.txt +3 -0
+server passes in a success response conforms to the type named by
+"ret-type".
+

[not for this patch. Thinking aloud: I wonder if QGA's success-response could be coded in by having 'ret-type':null as a way of saying the command will have no response if it is successful; perhaps a bit nicer than exposing an additional boolean flag member]

Quoted 4 lines
docs/qapi-code-gen.txt +4 -0
+If the command takes no arguments, "arg-type" names an object type
+without members.  Likewise, if the command returns nothing, "ret-type"
+names an object type without members.
+

docs/qapi-code-gen.txt Resulting in this JSON object: · line 549

Quoted 10 lines
docs/qapi-code-gen.txt +10 -0
+Example: the SchemaInfo for command query-schema
+
+    { "name": "query-schema", "meta-type": "command",
+      "arg-type": ":empty", "ret-type": "SchemaInfoList" }
+
+    Type ":empty" is an object type without members, and type
+    "SchemaInfoList" is the array of SchemaInfo type.
+
+The SchemaInfo for an event has meta-type "event", and variant member
+"arg-type".  The data member the server passes with an event conforms

again, reads a little better with s/member/member that/

Quoted 11 lines
docs/qapi-code-gen.txt +11 -0
+to the type named by "arg-type".  It is always an object type.
+
+If the event carries no additional information, "arg-type" names an
+object type without members.  The event may not have a data member on
+the wire then.
+
+Each command or event defined with dictionary-valued 'data' in the
+QAPI schema implicitly defines an object type called ":obj-NAME-arg",
+where NAME is the command or event's name.
+
+Example: the SchemaInfo for EVENT_C from section Events

docs/qapi-code-gen.txt Resulting in this JSON object: · line 570

Quoted 10 lines
docs/qapi-code-gen.txt +10 -0
+
+    { "name": "EVENT_C", "meta-type": "event",
+      "arg-type": ":obj-EVENT_C-arg" }
+
+    Type ":obj-EVENT_C-arg" is an implicitly defined object type with
+    the two members from the event's definition.
+
+The SchemaInfo for struct and union types has meta-type "object".
+
+The SchemaInfo for a struct type has variant member "members".

Worth mentioning that "members" may be empty, for an empty struct?

docs/qapi-code-gen.txt Resulting in this JSON object: · line 573

Quoted 3 lines
docs/qapi-code-gen.txt +3 -0
+
+The SchemaInfo for a union type additionally has variant members "tag"
+and "variants".

Worth mentioning that "members" will never be empty, because it contains at least the member also referenced in "tag"?

Quoted 13 lines
docs/qapi-code-gen.txt +13 -0
+
+"members" is a JSON array describing the object's common members.
+Each element is a JSON object with members "name" (the member's name),
+"type" (the name of its type), and optionally "default".  The member
+is optional if "default" is present.  Currently, "default" can only
+have value null.  Other values are reserved for future extensions.
+
+Example: the SchemaInfo for MyType from section Struct types
+
+    { "name": "MyType", "meta-type": "object",
+      "members": [
+          { "name": "member1", "type": "str" },
+          { "name": "member2", "type": "int" },

docs/qapi-code-gen.txt Resulting in this JSON object: · line 596

Quoted 10 lines
docs/qapi-code-gen.txt +10 -0
+          { "name": "member3", "type": "str", "default": null } ] }
+
+"tag" is the name of the common member serving as type tag.
+"variants" is a JSON array describing the object's variant members.
+Each element is a JSON object with members "case" (the value of type
+tag this element applies to) and "type" (the name of an object type
+that provides the variant members for this type tag value).
+
+Example: the SchemaInfo for flat union BlockdevOptions from section
+Union types

Hmm, the "Union types" section has two mentions of BlockdevOptions; and this example matches the second. It may help readability if we rename one of the two examples to be distinct (preferably the first, since it doesn't match actual QMP). I guess the phrase "flat union BlockdevOptions" is sufficient to make it obvious that we are referring to the second usage, but it is subtle.

Quoted 14 lines
docs/qapi-code-gen.txt +14 -0
+
+    { "name": "BlockdevOptions", "meta-type": "object",
+      "members": [
+          { "name": "driver", "type": "BlockdevDriver" },
+          { "name": "readonly", "type": "bool"} ],
+      "tag": "driver",
+      "variants": [
+          { "case": "file", "type": "FileOptions" },
+          { "case": "qcow2", "type": "Qcow2Options" } ] }
+
+Note that base types are "flattened": its members are included in the
+"members" array.
+
+A simple union implicitly defines an enumeration type for its implicit

docs/qapi-code-gen.txt Resulting in this JSON object: · line 620

Quoted 10 lines
docs/qapi-code-gen.txt +10 -0
+discriminator (called "type" on the wire, see section Union types).
+Such a type's name is made by appending "Kind" to the simple union's
+name.
+
+A simple union implicitly defines an object type for each of its
+variants.  The type's name is ":obj-NAME-wrapper", where NAME is the
+name of the name of the variant's type.
+
+Example: the SchemaInfo for simple union BlockdevOptions from section
+Union types

Ah, and here you refer to the other BlockdevOptions. So the point about a judicious rename may still be warranted.

docs/qapi-code-gen.txt Resulting in this JSON object: · line 624

Quoted 4 lines
docs/qapi-code-gen.txt +4 -0
+
+    { "name": "BlockdevOptions", "meta-type": "object",
+      "members": [
+          { "name": "kind", "type": "BlockdevOptionsKind" } ],

[1] Ouch. That should be "name": "type". I pointed out the bug in v3, but it looks like it still hasn't been fixed. Or rather, that our attempt to fix it wasn't correct.

Quoted 41 lines
docs/qapi-code-gen.txt +41 -0
+      "tag": "type",
+      "variants": [
+          { "case": "file", "type": ":obj-FileOptions-wrapper" },
+          { "case": "qcow2", "type": ":obj-Qcow2Options-wrapper" } ] }
+
+    Enumeration type "BlockdevOptionsKind" and the object types
+    ":obj-FileOptions-wrapper", ":obj-Qcow2Options-wrapper" are
+    implicitly defined.
+
+The SchemaInfo for an alternate type has meta-type "alternate", and
+variant member "members".  "members" is a JSON array.  Each element is
+a JSON object with member "type", which names a type.  Values of the
+alternate type conform to exactly one of its member types.
+
+Example: the SchemaInfo for BlockRef from section Alternate types
+
+    { "name": "BlockRef", "meta-type": "alternate",
+      "members": [
+          { "type": "BlockdevOptions" },
+          { "type": "str" } ] }
+
+The SchemaInfo for an array type has meta-type "array", and variant
+member "element-type", which names the array's element type.  Array
+types are implicitly defined.  An array type's name is made by
+appending "List" to its element type's name.
+
+Example: the SchemaInfo for ['str']
+
+    { "name": "strList", "meta-type": "array",
+      "element-type": "str" }
+
+The SchemaInfo for an enumeration type has meta-type "enum" and
+variant member "values".
+
+Example: the SchemaInfo for MyEnum from section Enumeration types
+
+    { "name": "MyEnum", "meta-type": "enum",
+      "values": [ "value1", "value2", "value3" ] }
+
+The SchemaInfo for a built-in type has the same name as the type in
+the QAPI schema (see section Built-in Types).  It has variant member

docs/qapi-code-gen.txt Resulting in this JSON object: · line 675

Quoted 10 lines
docs/qapi-code-gen.txt +10 -0
+"json-type" that shows how values of this type are encoded on the
+wire.
+
+Example: the SchemaInfo for str
+
+    { "name": "str", "meta-type": "builtin", "json-type": "string" }
+
+As explained above, type names are not part of the wire ABI.  Not even
+the names of built-in types.  Clients should examine member
+"json-type" instead of hard-coding names of built-in types.

Good point (although we may still end up with clients that disobey this, I will certainly make sure to do this in libvirt's interactions).

Quoted 5 lines
docs/qapi-code-gen.txt +2 -1
+
+
 == Code generation ==
 
-Schemas are fed into 3 scripts to generate all the code/files that, paired

docs/qapi-code-gen.txt Resulting in this JSON object: · line 685

Quoted 10 lines
docs/qapi-code-gen.txt +6 -4
-with the core QAPI libraries, comprise everything required to take JSON
-commands read in by a Client JSON Protocol server, unmarshal the arguments into
-the underlying C types, call into the corresponding C function, and map the
-response back to a Client JSON Protocol response to be returned to the user.
+Schemas are fed into four scripts to generate all the code/files that,
+paired with the core QAPI libraries, comprise everything required to
+take JSON commands read in by a Client JSON Protocol server, unmarshal
+the arguments into the underlying C types, call into the corresponding
+C function, and map the response back to a Client JSON Protocol
+response to be returned to the user.

Doesn't mention introspection; but then again, it also doesn't mention that it generates the C interface for emitting events.

Quoted 31 lines
docs/qapi-code-gen.txt +24 -0
 As an example, we'll use the following schema, which describes a single
 complex user-defined type (which will produce a C struct, along with a list
@@ -848,3 +1030,37 @@ Example:
     extern const char *const example_QAPIEvent_lookup[];
 
     #endif
+
+=== scripts/qapi-introspect.py ===
+
+Used to generate the introspection C code for a schema. The following
+files are created:
+
+$(prefix)qmp-introspect.c - Defines a string holding a JSON
+                            description of the schema.
+$(prefix)qmp-introspect.h - Declares the above string.
+
+Example:
+
+    $ python scripts/qapi-introspect.py --output-dir="qapi-generated"
+    --prefix="example-" example-schema.json
+    $ cat qapi-generated/example-qmp-introspect.c
+[Uninteresting stuff omitted...]
+
+    const char example_qmp_schema_json[] = "["
+        "{\"arg-type\": \":empty\", \"meta-type\": \"event\", \"name\": \"MY_EVENT\"}, "
+        "{\"json-type\": \"int\", \"meta-type\": \"builtin\", \"name\": \"int\"}, "
+        "{\"json-type\": \"string\", \"meta-type\": \"builtin\", \"name\": \"str\"}, "
+        "{\"members\": [], \"meta-type\": \"object\", \"name\": \":empty\"}, "
+        "{\"members\": [{\"name\": \"arg1\", \"type\": \"UserDefOne\"}], \"meta-type\": \"object\", \"name\": \":obj-my-command-arg\"}, "
+        "{\"members\": [{\"name\": \"integer\", \"type\": \"int\"}, {\"name\": \"string\", \"type\": \"str\"}], \"meta-type\": \"object\", \"name\": \"UserDefOne\"}, "

docs/qapi-code-gen.txt Example: · line 1066

Quoted 10 lines
docs/qapi-code-gen.txt +10 -0
+        "{\"arg-type\": \":obj-my-command-arg\", \"meta-type\": \"command\", \"name\": \"my-command\", \"ret-type\": \"UserDefOne\"}]";
+    $ cat qapi-generated/example-qmp-introspect.h
+[Uninteresting stuff omitted...]
+
+    #ifndef EXAMPLE_QMP_INTROSPECT_H
+    #define EXAMPLE_QMP_INTROSPECT_H
+
+    extern const char example_qmp_schema_json[];
+
+    #endif

Overall, a nice documentation addition, and solves one of my missing checkboxes against v3 :)

qapi-schema.json line 18

Quoted 7 lines

+++ b/qapi-schema.json

qapi-schema.json +2 -0
@@ -14,6 +14,9 @@
 # Tracing commands
 { 'include': 'qapi/trace.json' }
 
+# QAPI introspection
+{ 'include': 'qapi/introspect.json' }

When we add QGA introspection, we'll want qapi/introspect.json to contain JUST types, and move the 'command' into qapi-schema.json proper (that way, qga/qapi-schema.json can also include the same types, then add its variation on the command). As it won't matter until we actually do get to QGA, I'm okay whether you do the hoisting of the 'command' now or save it for later.

Quoted 7 lines
qapi/introspect.json new file +5 -0
+++ b/qapi/introspect.json
@@ -0,0 +1,271 @@
 
+
+##
+# @query-schema
+#

qapi/introspect.json line 25

Quoted 10 lines
qapi/introspect.json new file +10 -0
+# Command query-schema exposes the QMP wire ABI as an array of
+# SchemaInfo.  This lets QMP clients figure out what commands and
+# events are available in this QEMU, and their parameters and results.
+#
+# Returns: array of @SchemaInfo, where each element describes an
+# entity in the ABI: command, event, type, ...
+#
+# Note: the QAPI schema is also used to help define *internal*
+# interfaces, by defining QAPI types.  These are not part of the QMP
+# wire ABI, and therefore not returned by this command.

Same as for the docs above: may be worth a paragraph explicitly mentioning that the wire ABI cannot express everything, such as integer ranges, or such as semantics where exactly one of two mutually-exclusive optional members must be present.

...

Quoted 4 lines
qapi/introspect.json new file +4 -0
+##
+# @SchemaInfoBase
+#
+# Members common to any @SchemaInfo.

qapi/introspect.json line 72

Quoted 10 lines
qapi/introspect.json new file +10 -0
+#
+# Since: 2.5
+##
+{ 'struct': 'SchemaInfoBase',
+  'data': { 'name': 'str', 'meta-type': 'SchemaMetaType' } }
+
+##
+# @SchemaInfo
+#
+# @name: the entity's name, inherited from @base.

Interesting way to document @name (I probably would have documented @base as part of SchemaInfoBase); but it worked out well. No need to change it :)

Quoted 25 lines
qapi/introspect.json new file +25 -0
+# Entities defined in the QAPI schema have the name defined in the schema.
+# Implicitly defined entities have generated names.  See
+# docs/qapi-code-gen.txt section "Client JSON Protocol introspection"
+# for details.
+#
+# All references to other SchemaInfo are by name.
+#
+# Command and event names are part of the wire ABI, but type names are
+# not.  Therefore, looking up a type by "well-known" name is wrong.
+# Look up the command or event, then follow the references.
+#
+# @meta-type: the entity's meta type, inherited from @base.
+#
+# Additional members depend on the value of @meta-type.
+#
+# Since: 2.5
+##
+{ 'union': 'SchemaInfo',
+  'base': 'SchemaInfoBase',
+  'discriminator': 'meta-type',
+  'data': {
+      'builtin': 'SchemaInfoBuiltin',
+      'enum': 'SchemaInfoEnum',
+      'array': 'SchemaInfoArray',
+      'object': 'SchemaInfoObject',

qapi/introspect.json line 107

Quoted 10 lines
qapi/introspect.json new file +10 -0
+      'alternate': 'SchemaInfoAlternate',
+      'command': 'SchemaInfoCommand',
+      'event': 'SchemaInfoEvent' } }
+
+##
+# @SchemaInfoBuiltin
+#
+# Additional SchemaInfo members for meta-type 'builtin'.
+#
+# @json-type: the JSON type used for this type on the wire.

Might be worth repeating the caveat in the earlier docs that @name is not guaranteed to be stable, so clients should check @json-type.

(especially true if we later decide to expose range information by adding more builtins or user-defined subtypes of builtins to represent those ranged integers).

...

qapi/introspect.json line 134

Quoted 9 lines
qapi/introspect.json new file +9 -0
+
+##
+# @SchemaInfoEnum
+#
+# Additional SchemaInfo members for meta-type 'enum'.
+#
+# @values: the enumeration type's values.
+#
+# Values of this type are JSON string on the wire.

I'm assuming that if we add sorting of elements in a later patch, we can better document that guarantee here.

Quoted 4 lines
qapi/introspect.json new file +4 -0
+#
+# Since: 2.5
+##
+{ 'struct': 'SchemaInfoEnum',

qapi/introspect.json line 148

Quoted 10 lines
qapi/introspect.json new file +10 -0
+  'data': { 'values': ['str'] } }
+
+##
+# @SchemaInfoArray
+#
+# Additional SchemaInfo members for meta-type 'array'.
+#
+# @element-type: the array type's element type.
+#
+# Values of this type are JSON array on the wire.

Is it worth adding either of these clarifications?

As required by RFC 7159, the order of individual elements with in the array sent over the wire is assumed to be significant unless the documented semantics in qapi state otherwise.

While RFC 7159 permits an array to have elements of different types, all elements of a QMP array should have the same type.

Quoted 11 lines
qapi/introspect.json new file +11 -0
+#
+# Since: 2.5
+##
+{ 'struct': 'SchemaInfoArray',
+  'data': { 'element-type': 'str' } }
+
+##
+# @SchemaInfoObject
+#
+# Additional SchemaInfo members for meta-type 'object'.
+#

qapi/introspect.json line 169

Quoted 10 lines
qapi/introspect.json new file +10 -0
+# @members: the object type's (non-variant) members.
+#
+# @tag: #optional the name of the member serving as type tag.  An
+# element of @members with this name must exist.
+#
+# @variants: #optional variant members, i.e. additional members that
+# depend on the type tag's value.  Present exactly when @tag is
+# present.
+#
+# Values of this type are JSON object on the wire.

Worth adding this clarification?

As documented by RFC 7159, the wire format allows members of the JSON object to be sent in any order, and requires that members must not be duplicated.

Quoted 10 lines
qapi/introspect.json new file +10 -0
+#
+# Since: 2.5
+##
+{ 'struct': 'SchemaInfoObject',
+  'data': { 'members': [ 'SchemaInfoObjectMember' ],
+            '*tag': 'str',
+            '*variants': [ 'SchemaInfoObjectVariant' ] } }
+
+##
+# @SchemaInfoObjectMember

qapi/introspect.json line 189

Quoted 10 lines
qapi/introspect.json new file +10 -0
+#
+# An object member.
+#
+# @name: the member's name, as defined in the QAPI schema.
+#
+# @type: the name of the member's type.
+#
+# @default: #optional default when used as command parameter.
+#           If absent, the parameter is mandatory.
+#           If present, the value must be null.  The parameter is

Maybe:

s/The parameter/On the wire, the parameter/

qapi/introspect.json line 193

Quoted 4 lines
qapi/introspect.json new file +4 -0
+#           optional, and behavior when it's missing is not specified
+#           here.
+#           Future extension: if present and non-null, the parameter
+#	    is optional, and defaults to this value.

TAB damage.

qapi/introspect.json line 199

Quoted 6 lines
qapi/introspect.json new file +6 -0
+#
+# Since: 2.5
+##
+{ 'struct': 'SchemaInfoObjectMember',
+  'data': { 'name': 'str', 'type': 'str', '*default': 'any' } }
+# @default's type must be null or match @type

Is this line still worth keeping, given the more complete explanation above? But we do need to use 'any' here, as that is currently the only way for qapi-generated code to allow for JSON null.

Quoted 5 lines
qapi/introspect.json new file +5 -0
+
+##
+# @SchemaInfoObjectVariant
+#
+# The variant members for a value of the type tag.

qapi/introspect.json line 214

Quoted 10 lines
qapi/introspect.json new file +10 -0
+#
+# @case: a value of the type tag.
+#
+# @type: the name of the object type that provides the variant members
+# when the type tag has value @case.
+#
+# Since: 2.5
+##
+{ 'struct': 'SchemaInfoObjectVariant',
+  'data': { 'case': 'str', 'type': 'str' } }

Do we want to document whether all case values of the tag are guaranteed to be covered (happens to be true for all current qapi, but is not yet enforced; although that's one of my planned followup patches is to start enforcing it - so I guess I can document it at that time if we don't do it here)

qapi/introspect.json line 253

Quoted 11 lines
qapi/introspect.json new file +11 -0
+##
+# @SchemaInfoCommand
+#
+# Additional SchemaInfo members for meta-type 'command'.
+#
+# @arg-type: the name of the object type that provides the command's
+# parameters.
+#
+# @ret-type: the name of the command's result type.
+#
+# TODO @success-response (currently irrelevant, because it's QGA, not QMP)

and my idea of using 'ret-type':null may obviate the need for an actual 'success-response' parameter.

scripts/qapi-introspect.py line 9

Quoted 11 lines

+++ b/scripts/qapi-introspect.py

scripts/qapi-introspect.py new file +9 -0
@@ -0,0 +1,174 @@
+#
+# QAPI introspection generator
+#
+# Copyright (C) 2015 Red Hat, Inc.
+#
+# Authors:
+#  Markus Armbruster <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.

No 'or later' clause? Oh, because your hands are tied: scripts/qapi.py is GPLv2-only. Yuck. I wonder if we can fix that; although it may be hard getting Anthony to explain his original choice. Separate cleanup, if at all. [2]

scripts/qapi-introspect.py line 15

Quoted 6 lines
scripts/qapi-introspect.py new file +6 -0
+# See the COPYING file in the top-level directory.
+
+from qapi import *
+import string
+
+# Caveman's json.dumps() replacement (we're stuck at 2.4)

s/2.4/python 2.4/

scripts/qapi-introspect.py line 27

Quoted 12 lines
scripts/qapi-introspect.py new file +12 -0
+# TODO try to use json.dumps() once we get unstuck
+def to_json(obj, level=0):
+    if obj == None:
+        ret = 'null'
+    elif isinstance(obj, str):
+        ret = '"' + obj.replace('"', r'\"') + '"'
+    elif isinstance(obj, list):
+        elts = [to_json(elt, level + 1)
+                for elt in obj]
+        ret = '[' + ', '.join(elts) + ']'
+    elif isinstance(obj, dict):
+        elts = ['"%s": %s' % (key, to_json(obj[key], level + 1))

This relies on 'key' never having any embedded ". It happens to work given our naming conventions of what we pass through this dumper, but might be safer if you used key.replace('"', r'\"') to handle generic dumping.

scripts/qapi-introspect.py line 93

Quoted 9 lines
scripts/qapi-introspect.py new file +9 -0
+
+    def _gen_member(self, member):
+        ret = {'name': member.name, 'type': self._use_type(member.type)}
+        if member.optional:
+            ret['default'] = None
+        return ret
+
+    def _gen_variants(self, tag_name, variants):
+        return {'tag': tag_name or 'type',

[1] Ouch. Why are we still printing 'kind' as the name for simple unions? Oh, it's because tag_name is ALWAYS provided by the visitor, so the "or 'type'" clause never fires. I have a pending patch to change the C code to generate 'type' as the C code name to match the wire ABI; but until that patch is in, I think a hack solution might be to fix the visitor to supply None instead of a tag_name for simple unions. I'll respond to the appropriate earlier patch.

scripts/qapi-introspect.py line 155

Quoted 10 lines
scripts/qapi-introspect.py new file +10 -0
+h_comment = '''
+/*
+ * QAPI/QMP schema introspection
+ *
+ * Copyright (C) 2015 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */

[2] It's interesting that you are following the pattern of scripts/qapi-commands.py (commit c17d9908); even there, the script itself was GPLv2-only, while the generated output was LGPLv2+.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#63 lore raw

Re: [Qemu-devel] [PATCH RFC v4 30/32] qapi: New QMP command query-schema for QMP schema introspection

Quoted 27 lines · Eric Blake
Quoted 2 lines · Markus Armbruster

qapi/introspect.json defines the introspection schema. It's designed for QMP introspection, but should do for similar uses, such as QGA.

[review in this sub-thread; for comments on 'int' munging or other followups, see other subthread]

There is at least one definite bug (see multiple references below to [1]), and several ideas for cleanups, but in general, I think that the remaining changes are going to be small enough that I'd probably be okay if v5 started life with:

(Of course, you have every right to not add the R-b, especially if you want me to do another close review :)

docs/qapi-code-gen.txt Resulting in this JSON object: · line 624

Quoted 4 lines
docs/qapi-code-gen.txt +4 -0
+
+    { "name": "BlockdevOptions", "meta-type": "object",
+      "members": [
+          { "name": "kind", "type": "BlockdevOptionsKind" } ],

[1] Ouch. That should be "name": "type". I pointed out the bug in v3, but it looks like it still hasn't been fixed. Or rather, that our attempt to fix it wasn't correct.

Quoted 6 lines

scripts/qapi-introspect.py line 93

Quoted 2 lines
scripts/qapi-introspect.py new file +2 -0
+    def _gen_variants(self, tag_name, variants):
+        return {'tag': tag_name or 'type',

[1] Ouch. Why are we still printing 'kind' as the name for simple unions? Oh, it's because tag_name is ALWAYS provided by the visitor, so the "or 'type'" clause never fires.

Not quite the right comment. 'tag' is being output correctly (tag_name was indeed None), what was wrong was the 'members':[] array (which was assuming the member was named 'kind').

Quoted 5 lines

I have a pending patch to change
the C code to generate 'type' as the C code name to match the wire ABI; but until that patch is in, I think a hack solution might be to fix the visitor to supply None instead of a tag_name for simple unions. I'll respond to the appropriate earlier patch.

Rather, 3 earlier patches. See my comments on 2, 10, and 11.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#64 lore raw

Re: [Qemu-devel] [PATCH RFC v4 30/32] qapi: New QMP command query-schema for QMP schema introspection

In reply to Eric Blake
Quoted 4 lines · Eric Blake
Quoted 2 lines · Markus Armbruster

qapi/introspect.json defines the introspection schema. It's designed for QMP introspection, but should do for similar uses, such as QGA.

Quoted 8 lines

[review in this sub-thread; for comments on 'int' munging or other followups, see other subthread]

There is at least one definite bug (see multiple references below to [1]), and several ideas for cleanups, but in general, I think that the remaining changes are going to be small enough that I'd probably be okay if v5 started life with:

Thanks!

Quoted 2 lines

(Of course, you have every right to not add the R-b, especially if you want me to do another close review :)

Judgement call, as always :)

Quoted 11 lines
Quoted 8 lines

The introspection schema does not reflect all the rules and restrictions that apply to QAPI schemata. A valid QAPI schema has an introspection value conforming to the introspection schema, but the converse is not true.

The empty object type is used when a command takes no arguments or produces no results.

We had discussed whether to expose the empty type in a separate patch, to reduce the size of this patch;

I decided against it, because I feel the churn between revisions of the series would outweigh the relatively small reduction in this patch's size, and...

Quoted 3 lines

but at this point in the game, I'm
happy to keep it as-is (the faster we get v5 out for review and into the tree, the faster we can process the backlog of followup patches).

... I also feel we should get this series wrapped as soon as possible.

Quoted 9 lines
Quoted 9 lines

---

+++ b/docs/qapi-code-gen.txt

docs/qapi-code-gen.txt +0 -0
@@ -494,13 +494,195 @@ Resulting in this JSON object:
   "timestamp": { "seconds": 1267020223, "microseconds": 435656 } }
 
 
Quoted 8 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 502

Quoted 6 lines
docs/qapi-code-gen.txt +6 -0
+== Client JSON Protocol introspection ==
+
+Clients of a Client JSON Protocol commonly need to figure out what
+exactly the server (QEMU) supports.
+
+For this purpose, QMP provides introspection via command query-schema.

Will need tweaking to the agreed-on name query-qmp-schema...

Yup.

Quoted 5 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 503

Quoted 1 line
docs/qapi-code-gen.txt +1 -0
+QGA currently doesn't support introspection.

... (and we'd obviously want qga to eventually gain a matching counterpart, possibly 'guest-schema' or 'guest-get-schema', to match prevailing naming in qga/qapi-schema.json).

Think so, but I don't wish to tell the QGA maintainers what they need, so I chose neutral language.

Quoted 6 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 506

Quoted 3 lines
docs/qapi-code-gen.txt +3 -0
+
+query-schema returns a JSON array of SchemaInfo objects.  These
+objects together describe the wire ABI, as defined in the QAPI schema.

Maybe an additional sentence here, perhaps along the lines of:

Quoted 6 lines

Note that the wire format cannot express everything; it is not designed to show semantic constraints (such as when exactly one of a pair of mutually-exclusive optional members must be present, or when an integral value has specific limitations on valid values). Introspection allows an application to know if a feature is present, but the application must obey the qapi documentation to properly interact with that feature.

Michael raised the same point, and I drafted a paragraph in my reply to him:

The SchemaInfo can't reflect all the rules and restrictions that apply to QMP. It's interface introspection (figuring out what's there), not interface specification. The specification is in the QAPI schema, and users should peruse it to understand how QMP is to be used.

Then continue with an example showing a restriction that's spelled out in the QAPI schema, but can't be expressed in SchemaInfo.

Use mine as is? Merge our two proposals somehow?

Quoted 10 lines
Quoted 10 lines
docs/qapi-code-gen.txt +10 -0
+
+Like any other command, query-schema is itself defined in the QAPI
+schema, along with the SchemaInfo type.  This text attempts to give an
+overview how things work.  For details you need to consult the QAPI
+schema.
+
+SchemaInfo objects have common members "name" and "meta-type", and
+additional variant members depending on the value of meta-type.
+
+Each SchemaInfo object describes a wire ABI entity of a certain
Quoted 8 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 521

Quoted 5 lines
docs/qapi-code-gen.txt +5 -0
+meta-type: a command, event or one of several kinds of type.
+
+SchemaInfo for entities defined in the QAPI schema have the same name
+as in the schema.  This is the case for all commands and events, and
+most types.

And hopefully patch 32 munges this text to match the simplification made there (I haven't checked yet).

It does :)

Quoted 11 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 530

Quoted 8 lines
docs/qapi-code-gen.txt +8 -0
+Command and event names are part of the wire ABI, but type names are
+not.  Therefore, looking up a type by its name in the QAPI schema is
+wrong.  Look up the command or event, then follow references by name.
+
+QAPI schema definitions not reachable that way are omitted.
+
+The SchemaInfo for a command has meta-type "command", and variant
+members "arg-type" and "ret-type".  The arguments member clients pass

I found it easier to read with:
s/arguments/arguments that/

Easier to read, but no longer emphasizes that the specific "arguments" member of the { "execute" ... } form needs to conform. I'll use yours anyway, unless you have a better idea.

Quoted 4 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 532

Quoted 2 lines
docs/qapi-code-gen.txt +2 -0
+with a command on the wire must conform to the type named by
+"arg-type", which is always an object type.  The return value the

Likewise s/value/value that/

Likewise.

Quoted 8 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 535

Quoted 3 lines
docs/qapi-code-gen.txt +3 -0
+server passes in a success response conforms to the type named by
+"ret-type".
+

[not for this patch. Thinking aloud: I wonder if QGA's success-response could be coded in by having 'ret-type':null as a way of saying the command will have no response if it is successful; perhaps a bit nicer than exposing an additional boolean flag member]

The QAPI schema could also do it this way. What do you think?

Quoted 8 lines
Quoted 8 lines
docs/qapi-code-gen.txt +8 -0
+If the command takes no arguments, "arg-type" names an object type
+without members.  Likewise, if the command returns nothing, "ret-type"
+names an object type without members.
+
+Example: the SchemaInfo for command query-schema
+
+    { "name": "query-schema", "meta-type": "command",
+      "arg-type": ":empty", "ret-type": "SchemaInfoList" }
Quoted 8 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 549

Quoted 6 lines
docs/qapi-code-gen.txt +6 -0
+
+    Type ":empty" is an object type without members, and type
+    "SchemaInfoList" is the array of SchemaInfo type.
+
+The SchemaInfo for an event has meta-type "event", and variant member
+"arg-type".  The data member the server passes with an event conforms

again, reads a little better with s/member/member that/

My remark on command arguments applies.

Quoted 15 lines
Quoted 15 lines
docs/qapi-code-gen.txt +15 -0
+to the type named by "arg-type".  It is always an object type.
+
+If the event carries no additional information, "arg-type" names an
+object type without members.  The event may not have a data member on
+the wire then.
+
+Each command or event defined with dictionary-valued 'data' in the
+QAPI schema implicitly defines an object type called ":obj-NAME-arg",
+where NAME is the command or event's name.
+
+Example: the SchemaInfo for EVENT_C from section Events
+
+    { "name": "EVENT_C", "meta-type": "event",
+      "arg-type": ":obj-EVENT_C-arg" }
+
Quoted 8 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 570

Quoted 6 lines
docs/qapi-code-gen.txt +6 -0
+    Type ":obj-EVENT_C-arg" is an implicitly defined object type with
+    the two members from the event's definition.
+
+The SchemaInfo for struct and union types has meta-type "object".
+
+The SchemaInfo for a struct type has variant member "members".

Worth mentioning that "members" may be empty, for an empty struct?

Its value gets explained below. I feel discussing the special case "empty" here would be premature.

Quoted 6 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 573

Quoted 3 lines
docs/qapi-code-gen.txt +3 -0
+
+The SchemaInfo for a union type additionally has variant members "tag"
+and "variants".

Worth mentioning that "members" will never be empty, because it contains at least the member also referenced in "tag"?

Likewise.

Quoted 2 lines

Review Resulting in this JSON object: · line 574

Quoted 2 lines
(quoted diff) +2 -0
+
+"members" is a JSON array describing the object's common members.

Could say "the object's common members, if any." What do you think?

Quoted 6 lines
Quoted 6 lines
docs/qapi-code-gen.txt +6 -0
+Each element is a JSON object with members "name" (the member's name),
+"type" (the name of its type), and optionally "default".  The member
+is optional if "default" is present.  Currently, "default" can only
+have value null.  Other values are reserved for future extensions.
+
+Example: the SchemaInfo for MyType from section Struct types
Quoted 8 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 589

Quoted 8 lines
docs/qapi-code-gen.txt +8 -0
+
+    { "name": "MyType", "meta-type": "object",
+      "members": [
+          { "name": "member1", "type": "str" },
+          { "name": "member2", "type": "int" },
+          { "name": "member3", "type": "str", "default": null } ] }
+
+"tag" is the name of the common member serving as type tag.

This implies that this member must exist in "members". Want me to repeat that explicitly?

Quoted 6 lines
Quoted 6 lines
docs/qapi-code-gen.txt +6 -0
+"variants" is a JSON array describing the object's variant members.
+Each element is a JSON object with members "case" (the value of type
+tag this element applies to) and "type" (the name of an object type
+that provides the variant members for this type tag value).
+
+Example: the SchemaInfo for flat union BlockdevOptions from section
Quoted 8 lines

Review Resulting in this JSON object: · line 595

Quoted 1 line
(quoted diff) +1 -0
+Union types

Hmm, the "Union types" section has two mentions of BlockdevOptions; and this example matches the second. It may help readability if we rename one of the two examples to be distinct (preferably the first, since it doesn't match actual QMP). I guess the phrase "flat union BlockdevOptions" is sufficient to make it obvious that we are referring to the second usage, but it is subtle.

Follow-up patch?

Quoted 28 lines
Quoted 14 lines
docs/qapi-code-gen.txt +14 -0
+
+    { "name": "BlockdevOptions", "meta-type": "object",
+      "members": [
+          { "name": "driver", "type": "BlockdevDriver" },
+          { "name": "readonly", "type": "bool"} ],
+      "tag": "driver",
+      "variants": [
+          { "case": "file", "type": "FileOptions" },
+          { "case": "qcow2", "type": "Qcow2Options" } ] }
+
+Note that base types are "flattened": its members are included in the
+"members" array.
+
+A simple union implicitly defines an enumeration type for its implicit

docs/qapi-code-gen.txt Resulting in this JSON object: · line 620

Quoted 10 lines
docs/qapi-code-gen.txt +10 -0
+discriminator (called "type" on the wire, see section Union types).
+Such a type's name is made by appending "Kind" to the simple union's
+name.
+
+A simple union implicitly defines an object type for each of its
+variants.  The type's name is ":obj-NAME-wrapper", where NAME is the
+name of the name of the variant's type.
+
+Example: the SchemaInfo for simple union BlockdevOptions from section
+Union types

Ah, and here you refer to the other BlockdevOptions. So the point about a judicious rename may still be warranted.

Quoted 8 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 624

Quoted 4 lines
docs/qapi-code-gen.txt +4 -0
+
+    { "name": "BlockdevOptions", "meta-type": "object",
+      "members": [
+          { "name": "kind", "type": "BlockdevOptionsKind" } ],

[1] Ouch. That should be "name": "type". I pointed out the bug in v3, but it looks like it still hasn't been fixed. Or rather, that our attempt to fix it wasn't correct.

I'll double-check.

Quoted 46 lines
Quoted 46 lines
docs/qapi-code-gen.txt +46 -0
+      "tag": "type",
+      "variants": [
+          { "case": "file", "type": ":obj-FileOptions-wrapper" },
+          { "case": "qcow2", "type": ":obj-Qcow2Options-wrapper" } ] }
+
+    Enumeration type "BlockdevOptionsKind" and the object types
+    ":obj-FileOptions-wrapper", ":obj-Qcow2Options-wrapper" are
+    implicitly defined.
+
+The SchemaInfo for an alternate type has meta-type "alternate", and
+variant member "members".  "members" is a JSON array.  Each element is
+a JSON object with member "type", which names a type.  Values of the
+alternate type conform to exactly one of its member types.
+
+Example: the SchemaInfo for BlockRef from section Alternate types
+
+    { "name": "BlockRef", "meta-type": "alternate",
+      "members": [
+          { "type": "BlockdevOptions" },
+          { "type": "str" } ] }
+
+The SchemaInfo for an array type has meta-type "array", and variant
+member "element-type", which names the array's element type.  Array
+types are implicitly defined.  An array type's name is made by
+appending "List" to its element type's name.
+
+Example: the SchemaInfo for ['str']
+
+    { "name": "strList", "meta-type": "array",
+      "element-type": "str" }
+
+The SchemaInfo for an enumeration type has meta-type "enum" and
+variant member "values".
+
+Example: the SchemaInfo for MyEnum from section Enumeration types
+
+    { "name": "MyEnum", "meta-type": "enum",
+      "values": [ "value1", "value2", "value3" ] }
+
+The SchemaInfo for a built-in type has the same name as the type in
+the QAPI schema (see section Built-in Types).  It has variant member
+"json-type" that shows how values of this type are encoded on the
+wire.
+
+Example: the SchemaInfo for str
+
Quoted 8 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 675

Quoted 5 lines
docs/qapi-code-gen.txt +5 -0
+    { "name": "str", "meta-type": "builtin", "json-type": "string" }
+
+As explained above, type names are not part of the wire ABI.  Not even
+the names of built-in types.  Clients should examine member
+"json-type" instead of hard-coding names of built-in types.

Good point (although we may still end up with clients that disobey this, I will certainly make sure to do this in libvirt's interactions).

We can't fully prevent misuse of our interfaces. Where we can't, we can still try to make correct use as easy and obvious as we can.

PATCH 32 prevents misuse of most type names by hiding them. RFC v2 hid all type names. In review, we decided not to hide the names of built-in types. Tradeoff: makes the introspection value easier to read for humans (good), but enables misuse by machines (bad).

Either way works for me, hide non-built-in type names (and have docs tell users not to rely on them), or hide them all. Got a preference?

Quoted 4 lines
Quoted 4 lines
docs/qapi-code-gen.txt +2 -0
+
+
 == Code generation ==
 
Quoted 14 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 685

Quoted 11 lines
docs/qapi-code-gen.txt +6 -5
-Schemas are fed into 3 scripts to generate all the code/files that, paired
-with the core QAPI libraries, comprise everything required to take JSON
-commands read in by a Client JSON Protocol server, unmarshal the arguments into
-the underlying C types, call into the corresponding C function, and map the
-response back to a Client JSON Protocol response to be returned to the user.
+Schemas are fed into four scripts to generate all the code/files that,
+paired with the core QAPI libraries, comprise everything required to
+take JSON commands read in by a Client JSON Protocol server, unmarshal
+the arguments into the underlying C types, call into the corresponding
+C function, and map the response back to a Client JSON Protocol
+response to be returned to the user.

Doesn't mention introspection; but then again, it also doesn't mention that it generates the C interface for emitting events.

Good enough for me now, but I'm of course happy to consider proposed improvements.

Quoted 36 lines
Quoted 36 lines
docs/qapi-code-gen.txt +29 -0
 As an example, we'll use the following schema, which describes a single
 complex user-defined type (which will produce a C struct, along with a list
@@ -848,3 +1030,37 @@ Example:
     extern const char *const example_QAPIEvent_lookup[];
 
     #endif
+
+=== scripts/qapi-introspect.py ===
+
+Used to generate the introspection C code for a schema. The following
+files are created:
+
+$(prefix)qmp-introspect.c - Defines a string holding a JSON
+                            description of the schema.
+$(prefix)qmp-introspect.h - Declares the above string.
+
+Example:
+
+    $ python scripts/qapi-introspect.py --output-dir="qapi-generated"
+    --prefix="example-" example-schema.json
+    $ cat qapi-generated/example-qmp-introspect.c
+[Uninteresting stuff omitted...]
+
+    const char example_qmp_schema_json[] = "["
+ "{\"arg-type\": \":empty\", \"meta-type\": \"event\", \"name\": \"MY_EVENT\"}, "
+ "{\"json-type\": \"int\", \"meta-type\": \"builtin\", \"name\": \"int\"}, "
+ "{\"json-type\": \"string\", \"meta-type\": \"builtin\", \"name\": \"str\"}, "
+ "{\"members\": [], \"meta-type\": \"object\", \"name\": \":empty\"}, "
+ "{\"members\": [{\"name\": \"arg1\", \"type\": \"UserDefOne\"}], \"meta-type\": \"object\", \"name\": \":obj-my-command-arg\"}, "
+ "{\"members\": [{\"name\": \"integer\", \"type\": \"int\"}, {\"name\": \"string\", \"type\": \"str\"}], \"meta-type\": \"object\", \"name\": \"UserDefOne\"}, "
+ "{\"arg-type\": \":obj-my-command-arg\", \"meta-type\": \"command\", \"name\": \"my-command\", \"ret-type\": \"UserDefOne\"}]";
+    $ cat qapi-generated/example-qmp-introspect.h
+[Uninteresting stuff omitted...]
+
+    #ifndef EXAMPLE_QMP_INTROSPECT_H
Quoted 8 lines

docs/qapi-code-gen.txt Example: · line 1066

Quoted 5 lines
docs/qapi-code-gen.txt +5 -0
+    #define EXAMPLE_QMP_INTROSPECT_H
+
+    extern const char example_qmp_schema_json[];
+
+    #endif

Overall, a nice documentation addition, and solves one of my missing checkboxes against v3 :)

Thanks!

Quoted 5 lines
Quoted 5 lines

+++ b/qapi-schema.json

qapi-schema.json +0 -0
@@ -14,6 +14,9 @@
 # Tracing commands
 { 'include': 'qapi/trace.json' }
 
Quoted 9 lines

qapi-schema.json line 18

Quoted 2 lines
qapi-schema.json +2 -0
+# QAPI introspection
+{ 'include': 'qapi/introspect.json' }

When we add QGA introspection, we'll want qapi/introspect.json to contain JUST types, and move the 'command' into qapi-schema.json proper (that way, qga/qapi-schema.json can also include the same types, then add its variation on the command). As it won't matter until we actually do get to QGA, I'm okay whether you do the hoisting of the 'command' now or save it for later.

Yes, let's keep things simple now, and complicate them only as needed.

Quoted 14 lines
Quoted 14 lines
qapi/introspect.json new file +12 -0
+++ b/qapi/introspect.json
@@ -0,0 +1,271 @@
 
+
+##
+# @query-schema
+#
+# Command query-schema exposes the QMP wire ABI as an array of
+# SchemaInfo.  This lets QMP clients figure out what commands and
+# events are available in this QEMU, and their parameters and results.
+#
+# Returns: array of @SchemaInfo, where each element describes an
+# entity in the ABI: command, event, type, ...
+#
Quoted 8 lines

qapi/introspect.json line 25

Quoted 3 lines
qapi/introspect.json new file +3 -0
+# Note: the QAPI schema is also used to help define *internal*
+# interfaces, by defining QAPI types.  These are not part of the QMP
+# wire ABI, and therefore not returned by this command.

Same as for the docs above: may be worth a paragraph explicitly mentioning that the wire ABI cannot express everything, such as integer ranges, or such as semantics where exactly one of two mutually-exclusive optional members must be present.

Here's where Michael commented.

Quoted 11 lines

...

Quoted 10 lines
qapi/introspect.json new file +10 -0
+##
+# @SchemaInfoBase
+#
+# Members common to any @SchemaInfo.
+#
+# Since: 2.5
+##
+{ 'struct': 'SchemaInfoBase',
+  'data': { 'name': 'str', 'meta-type': 'SchemaMetaType' } }
+
Quoted 8 lines

qapi/introspect.json line 72

Quoted 4 lines
qapi/introspect.json new file +4 -0
+##
+# @SchemaInfo
+#
+# @name: the entity's name, inherited from @base.

Interesting way to document @name (I probably would have documented @base as part of SchemaInfoBase); but it worked out well. No need to change it :)

Okay, thanks :)

Quoted 30 lines
Quoted 30 lines
qapi/introspect.json new file +30 -0
+# Entities defined in the QAPI schema have the name defined in the schema.
+# Implicitly defined entities have generated names.  See
+# docs/qapi-code-gen.txt section "Client JSON Protocol introspection"
+# for details.
+#
+# All references to other SchemaInfo are by name.
+#
+# Command and event names are part of the wire ABI, but type names are
+# not.  Therefore, looking up a type by "well-known" name is wrong.
+# Look up the command or event, then follow the references.
+#
+# @meta-type: the entity's meta type, inherited from @base.
+#
+# Additional members depend on the value of @meta-type.
+#
+# Since: 2.5
+##
+{ 'union': 'SchemaInfo',
+  'base': 'SchemaInfoBase',
+  'discriminator': 'meta-type',
+  'data': {
+      'builtin': 'SchemaInfoBuiltin',
+      'enum': 'SchemaInfoEnum',
+      'array': 'SchemaInfoArray',
+      'object': 'SchemaInfoObject',
+      'alternate': 'SchemaInfoAlternate',
+      'command': 'SchemaInfoCommand',
+      'event': 'SchemaInfoEvent' } }
+
+##
Quoted 8 lines

qapi/introspect.json line 107

Quoted 5 lines
qapi/introspect.json new file +5 -0
+# @SchemaInfoBuiltin
+#
+# Additional SchemaInfo members for meta-type 'builtin'.
+#
+# @json-type: the JSON type used for this type on the wire.

Might be worth repeating the caveat in the earlier docs that @name is not guaranteed to be stable, so clients should check @json-type.

Yes, the QAPI schema should contain all the important information. Terse is okay.

"Earlier docs" = docs/qapi-code-gen.txt. That one should be less terse, but doesn't have to cover all the details.

Quoted 5 lines

(especially true if we later decide to expose range information by adding more builtins or user-defined subtypes of builtins to represent those ranged integers).

...

Quoted 12 lines

qapi/introspect.json line 134

Quoted 9 lines
qapi/introspect.json new file +9 -0
+
+##
+# @SchemaInfoEnum
+#
+# Additional SchemaInfo members for meta-type 'enum'.
+#
+# @values: the enumeration type's values.
+#
+# Values of this type are JSON string on the wire.

I'm assuming that if we add sorting of elements in a later patch, we can better document that guarantee here.

Yes.

Quoted 17 lines
Quoted 4 lines
qapi/introspect.json new file +4 -0
+#
+# Since: 2.5
+##
+{ 'struct': 'SchemaInfoEnum',

qapi/introspect.json line 148

Quoted 10 lines
qapi/introspect.json new file +10 -0
+  'data': { 'values': ['str'] } }
+
+##
+# @SchemaInfoArray
+#
+# Additional SchemaInfo members for meta-type 'array'.
+#
+# @element-type: the array type's element type.
+#
+# Values of this type are JSON array on the wire.

Is it worth adding either of these clarifications?

Quoted 3 lines

As required by RFC 7159, the order of individual elements with in the array sent over the wire is assumed to be significant unless the documented semantics in qapi state otherwise.

Got an example where insignificance of order actually matters?

Need to check for overlap with docs/qmp/qmp-spec.txt.

Quoted 2 lines

While RFC 7159 permits an array to have elements of different types, all elements of a QMP array should have the same type.

Implied by having an element-type, but making it explicit won't hurt. The new 'any' type lets us do inhomogeneous arrays, though: ['any']. If I can think of a concise way to clarify all that, I'll add it. If not, we can add it later.

Quoted 24 lines
Quoted 11 lines
qapi/introspect.json new file +11 -0
+#
+# Since: 2.5
+##
+{ 'struct': 'SchemaInfoArray',
+  'data': { 'element-type': 'str' } }
+
+##
+# @SchemaInfoObject
+#
+# Additional SchemaInfo members for meta-type 'object'.
+#

qapi/introspect.json line 169

Quoted 10 lines
qapi/introspect.json new file +10 -0
+# @members: the object type's (non-variant) members.
+#
+# @tag: #optional the name of the member serving as type tag.  An
+# element of @members with this name must exist.
+#
+# @variants: #optional variant members, i.e. additional members that
+# depend on the type tag's value.  Present exactly when @tag is
+# present.
+#
+# Values of this type are JSON object on the wire.

Worth adding this clarification?

Quoted 3 lines

As documented by RFC 7159, the wire format allows members of the JSON object to be sent in any order, and requires that members must not be duplicated.

Need to check for overlap with docs/qmp/qmp-spec.txt.

Quoted 23 lines
Quoted 10 lines
qapi/introspect.json new file +10 -0
+#
+# Since: 2.5
+##
+{ 'struct': 'SchemaInfoObject',
+  'data': { 'members': [ 'SchemaInfoObjectMember' ],
+            '*tag': 'str',
+            '*variants': [ 'SchemaInfoObjectVariant' ] } }
+
+##
+# @SchemaInfoObjectMember

qapi/introspect.json line 189

Quoted 10 lines
qapi/introspect.json new file +10 -0
+#
+# An object member.
+#
+# @name: the member's name, as defined in the QAPI schema.
+#
+# @type: the name of the member's type.
+#
+# @default: #optional default when used as command parameter.
+#           If absent, the parameter is mandatory.
+#           If present, the value must be null.  The parameter is

Maybe:

Quoted 1 line

s/The parameter/On the wire, the parameter/

Okay.

Quoted 6 lines

qapi/introspect.json line 193

Quoted 4 lines
qapi/introspect.json new file +4 -0
+#           optional, and behavior when it's missing is not specified
+#           here.
+#           Future extension: if present and non-null, the parameter
+#	    is optional, and defaults to this value.

TAB damage.

Will fix, and search for more.

Quoted 10 lines

qapi/introspect.json line 199

Quoted 6 lines
qapi/introspect.json new file +6 -0
+#
+# Since: 2.5
+##
+{ 'struct': 'SchemaInfoObjectMember',
+  'data': { 'name': 'str', 'type': 'str', '*default': 'any' } }
+# @default's type must be null or match @type

Is this line still worth keeping, given the more complete explanation above? But we do need to use 'any' here, as that is currently the only way for qapi-generated code to allow for JSON null.

I'll drop it. We'll add it back with support for non-null values (assuming we get around to that).

Quoted 13 lines
Quoted 13 lines
qapi/introspect.json new file +13 -0
+
+##
+# @SchemaInfoObjectVariant
+#
+# The variant members for a value of the type tag.
+#
+# @case: a value of the type tag.
+#
+# @type: the name of the object type that provides the variant members
+# when the type tag has value @case.
+#
+# Since: 2.5
+##
Quoted 8 lines

qapi/introspect.json line 214

Quoted 2 lines
qapi/introspect.json new file +2 -0
+{ 'struct': 'SchemaInfoObjectVariant',
+  'data': { 'case': 'str', 'type': 'str' } }

Do we want to document whether all case values of the tag are guaranteed to be covered (happens to be true for all current qapi, but is not yet enforced; although that's one of my planned followup patches is to start enforcing it - so I guess I can document it at that time if we don't do it here)

Requiring all cases to be covered explicitly may have value in the QAPI schema (more verbose, but you get a reminder to update the union after you extended the tag enum), but it's pretty useless in introspection, isn't it?

Quoted 6 lines
Quoted 6 lines
qapi/introspect.json new file +6 -0
+##
+# @SchemaInfoCommand
+#
+# Additional SchemaInfo members for meta-type 'command'.
+#
+# @arg-type: the name of the object type that provides the command's
Quoted 8 lines

qapi/introspect.json line 253

Quoted 5 lines
qapi/introspect.json new file +5 -0
+# parameters.
+#
+# @ret-type: the name of the command's result type.
+#
+# TODO @success-response (currently irrelevant, because it's QGA, not QMP)

and my idea of using 'ret-type':null may obviate the need for an actual 'success-response' parameter.

So let's read the TODO as "need to cover @success-response somehow for QGA" :)

Quoted 6 lines
Quoted 6 lines

+++ b/scripts/qapi-introspect.py

scripts/qapi-introspect.py new file +4 -0
@@ -0,0 +1,174 @@
+#
+# QAPI introspection generator
+#
+# Copyright (C) 2015 Red Hat, Inc.
Quoted 8 lines

scripts/qapi-introspect.py line 9

Quoted 5 lines
scripts/qapi-introspect.py new file +5 -0
+#
+# Authors:
+#  Markus Armbruster <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2.

No 'or later' clause? Oh, because your hands are tied: scripts/qapi.py is GPLv2-only. Yuck.

Since I wrote this file from scratch, and even the qapi.py interface it uses is pretty much mine, I guess I could use "or later" here.

Quoted 3 lines

I wonder if we can fix that; although it may be
hard getting Anthony to explain his original choice. Separate cleanup, if at all. [2]

I truly wish we'd simply told all contributors from the start "this is our license, take it or leave it".

Quoted 8 lines

scripts/qapi-introspect.py line 15

Quoted 6 lines
scripts/qapi-introspect.py new file +6 -0
+# See the COPYING file in the top-level directory.
+
+from qapi import *
+import string
+
+# Caveman's json.dumps() replacement (we're stuck at 2.4)

s/2.4/python 2.4/

Okay.

Quoted 9 lines
Quoted 9 lines
scripts/qapi-introspect.py new file +9 -0
+# TODO try to use json.dumps() once we get unstuck
+def to_json(obj, level=0):
+    if obj == None:
+        ret = 'null'
+    elif isinstance(obj, str):
+        ret = '"' + obj.replace('"', r'\"') + '"'
+    elif isinstance(obj, list):
+        elts = [to_json(elt, level + 1)
+                for elt in obj]
Quoted 8 lines

scripts/qapi-introspect.py line 27

Quoted 3 lines
scripts/qapi-introspect.py new file +3 -0
+        ret = '[' + ', '.join(elts) + ']'
+    elif isinstance(obj, dict):
+        elts = ['"%s": %s' % (key, to_json(obj[key], level + 1))

This relies on 'key' never having any embedded ". It happens to work given our naming conventions of what we pass through this dumper, but might be safer if you used key.replace('"', r'\"') to handle generic dumping.

Either that, or assert.

Quoted 10 lines

scripts/qapi-introspect.py line 93

Quoted 9 lines
scripts/qapi-introspect.py new file +9 -0
+
+    def _gen_member(self, member):
+        ret = {'name': member.name, 'type': self._use_type(member.type)}
+        if member.optional:
+            ret['default'] = None
+        return ret
+
+    def _gen_variants(self, tag_name, variants):
+        return {'tag': tag_name or 'type',
Quoted 7 lines

[1] Ouch. Why are we still printing 'kind' as the name for simple unions? Oh, it's because tag_name is ALWAYS provided by the visitor, so the "or 'type'" clause never fires. I have a pending patch to change the C code to generate 'type' as the C code name to match the wire ABI; but until that patch is in, I think a hack solution might be to fix the visitor to supply None instead of a tag_name for simple unions. I'll respond to the appropriate earlier patch.

You followed up with patches, and I'll peruse them after lunch.

Quoted 15 lines

scripts/qapi-introspect.py

Quoted 7 lines
scripts/qapi-introspect.py new file +7 -0
+h_comment = '''
+/*
+ * QAPI/QMP schema introspection
+ *
+ * Copyright (C) 2015 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU LGPL, version

2.1 or later.

scripts/qapi-introspect.py line 144

Quoted 3 lines
scripts/qapi-introspect.py new file +3 -0
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */

[2] It's interesting that you are following the pattern of scripts/qapi-commands.py (commit c17d9908); even there, the script itself was GPLv2-only, while the generated output was LGPLv2+.

#65 lore raw

Re: [Qemu-devel] [PATCH RFC v4 30/32] qapi: New QMP command query-schema for QMP schema introspection

Quoted 22 lines · Markus Armbruster
Quoted 6 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 506

Quoted 3 lines
docs/qapi-code-gen.txt +3 -0
+
+query-schema returns a JSON array of SchemaInfo objects.  These
+objects together describe the wire ABI, as defined in the QAPI schema.

Maybe an additional sentence here, perhaps along the lines of:

Quoted 6 lines

Note that the wire format cannot express everything; it is not designed to show semantic constraints (such as when exactly one of a pair of mutually-exclusive optional members must be present, or when an integral value has specific limitations on valid values). Introspection allows an application to know if a feature is present, but the application must obey the qapi documentation to properly interact with that feature.

Michael raised the same point, and I drafted a paragraph in my reply to him:

The SchemaInfo can't reflect all the rules and restrictions that apply to QMP. It's interface introspection (figuring out what's there), not interface specification. The specification is in the QAPI schema, and users should peruse it to understand how QMP is to be used.

Quoted 4 lines

Then continue with an example showing a restriction that's spelled out in the QAPI schema, but can't be expressed in SchemaInfo.

Use mine as is? Merge our two proposals somehow?

Your wording works okay for me, but if you want to merge any elements of my wording in, go for it.

Quoted 10 lines
Quoted 5 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 530

Quoted 2 lines
docs/qapi-code-gen.txt +2 -0
+The SchemaInfo for a command has meta-type "command", and variant
+members "arg-type" and "ret-type".  The arguments member clients pass

I found it easier to read with:
s/arguments/arguments that/

Easier to read, but no longer emphasizes that the specific "arguments" member of the { "execute" ... } form needs to conform. I'll use yours anyway, unless you have a better idea.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

+with a command on the wire must conform to the type named by +"arg-type", which is always an object type. The return value the

Likewise s/value/value that/

Likewise.

Quoted 2 lines
Quoted 2 lines

+server passes in a success response conforms to the type named by +"ret-type".

Maybe:

On the wire, the "arguments" key of a client's "execute" command must conform to the type named by "arg-type", which is always an object type. The "return" key that the server passes in a success response will always conform to the type named by "ret-type".

Up to you; we'll see how it looks in v5, and at that point, any further word-smithing can be done in followup patches rather than delaying this series further.

Quoted 6 lines
Quoted 4 lines

[not for this patch. Thinking aloud: I wonder if QGA's success-response could be coded in by having 'ret-type':null as a way of saying the command will have no response if it is successful; perhaps a bit nicer than exposing an additional boolean flag member]

The QAPI schema could also do it this way. What do you think?

Sounds like I have another trivial followup patch added to my work queue :)

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

+The SchemaInfo for an event has meta-type "event", and variant member +"arg-type". The data member the server passes with an event conforms

again, reads a little better with s/member/member that/

My remark on command arguments applies.

Quoted 1 line
Quoted 1 line

+to the type named by "arg-type". It is always an object type.

Maybe:

On the wire, the "data" member included in any event passed by the server will conform to the type named by "arg-type".

I'm not sure if we need to document whether it is always an object type; command arguments are user-supplied, while event data is server-supplied. And there's the question of any possible duplication with the QMP spec file. I guess I'm fine as long as we aren't stating conflicting things between the two places.

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

+
+"members" is a JSON array describing the object's common members.

Could say "the object's common members, if any." What do you think?

Yes, adding the 'if any' helps.

Quoted 9 lines
Quoted 7 lines

Hmm, the "Union types" section has two mentions of BlockdevOptions; and this example matches the second. It may help readability if we rename one of the two examples to be distinct (preferably the first, since it doesn't match actual QMP). I guess the phrase "flat union BlockdevOptions" is sufficient to make it obvious that we are referring to the second usage, but it is subtle.

Follow-up patch?

Yes.

Quoted 6 lines
Quoted 4 lines

[1] Ouch. That should be "name": "type". I pointed out the bug in v3, but it looks like it still hasn't been fixed. Or rather, that our attempt to fix it wasn't correct.

I'll double-check.

As you noted later, I've already done that legwork.

Quoted 11 lines
Quoted 7 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 675

Quoted 4 lines
docs/qapi-code-gen.txt +4 -0
+
+As explained above, type names are not part of the wire ABI.  Not even
+the names of built-in types.  Clients should examine member
+"json-type" instead of hard-coding names of built-in types.

Good point (although we may still end up with clients that disobey this, I will certainly make sure to do this in libvirt's interactions).

We can't fully prevent misuse of our interfaces. Where we can't, we can still try to make correct use as easy and obvious as we can.

Quoted 8 lines

PATCH 32 prevents misuse of most type names by hiding them. RFC v2 hid all type names. In review, we decided not to hide the names of built-in types. Tradeoff: makes the introspection value easier to read for humans (good), but enables misuse by machines (bad).

Either way works for me, hide non-built-in type names (and have docs tell users not to rely on them), or hide them all. Got a preference?

I'm still leaning towards exposing built-in type names directly, and furthermore exposing arrays via '[123]' naming. As you say, correct clients shouldn't be abusing this, but it IS easier to see, and I'm having a hard time seeing how it could paint us into corners in the future for having exposed too much information up front (while we DO rename object types and don't want that to affect ABI, I don't see us trying to rename builtin types - at most, we might add subtypes with range or other restrictions, but those would be new types and still leave the basic 'int' and 'str' unchanged). So leaving it as just the documentation on what proper clients should do is sufficient; don't go back to the v2 munging of builtin names.

Quoted 23 lines
Quoted 14 lines
Quoted 14 lines

+++ b/qapi/introspect.json

qapi/introspect.json new file +11 -0
@@ -0,0 +1,271 @@
 
+
+##
+# @query-schema
+#
+# Command query-schema exposes the QMP wire ABI as an array of
+# SchemaInfo.  This lets QMP clients figure out what commands and
+# events are available in this QEMU, and their parameters and results.
+#
+# Returns: array of @SchemaInfo, where each element describes an
+# entity in the ABI: command, event, type, ...
+#
Quoted 8 lines

qapi/introspect.json line 25

Quoted 3 lines
qapi/introspect.json new file +3 -0
+# Note: the QAPI schema is also used to help define *internal*
+# interfaces, by defining QAPI types.  These are not part of the QMP
+# wire ABI, and therefore not returned by this command.

Same as for the docs above: may be worth a paragraph explicitly mentioning that the wire ABI cannot express everything, such as integer ranges, or such as semantics where exactly one of two mutually-exclusive optional members must be present.

Quoted 1 line

Here's where Michael commented.

Actually, he mentioned the .hx file, which I didn't. I don't know how many duplicate places need to refer to the same information, or just abbreviate by pointing to other places. But I'll be happy as long as the information is present at least somewhere, and that somewhere is easy to find (perhaps by good references rather than duplications in the other places).

Quoted 6 lines
Quoted 4 lines

As required by RFC 7159, the order of individual elements with in the array sent over the wire is assumed to be significant unless the documented semantics in qapi state otherwise.

Got an example where insignificance of order actually matters?

migrate-set-capabilities does not care what order the 'capabilities' array is in, for example.

Quoted 9 lines
Quoted 8 lines

qapi/introspect.json line 214

Quoted 2 lines
qapi/introspect.json new file +2 -0
+{ 'struct': 'SchemaInfoObjectVariant',
+  'data': { 'case': 'str', 'type': 'str' } }

Do we want to document whether all case values of the tag are guaranteed to be covered (happens to be true for all current qapi, but is not yet enforced; although that's one of my planned followup patches is to start enforcing it - so I guess I can document it at that time if we don't do it here)

Quoted 4 lines

Requiring all cases to be covered explicitly may have value in the QAPI schema (more verbose, but you get a reminder to update the union after you extended the tag enum), but it's pretty useless in introspection, isn't it?

At first glance, I wouldn't call it useless. But thinking about it
more: Right now, if not all members of the enum are listed, the generated code abort()s when passing the uncovered enum value. But ideally we want semantics where an omitted case in the qapi (or maybe explicit 'case':null or 'case':{} notation) results in adding no further variant members to the overall object. In that case, knowing that the tag enum value is valid and maps to ':empty' on the wire is worth advertising, even if we choose to allow omitting a case in the qapi .json as the syntax for that action.

So we can touch up the wording at the same time we fix code to not abort().

Quoted 6 lines
Quoted 4 lines

scripts/qapi-introspect.py line 15

Quoted 2 lines
scripts/qapi-introspect.py new file +2 -0
+
+# Caveman's json.dumps() replacement (we're stuck at 2.4)

s/2.4/python 2.4/

Okay.

Of course, if Dan's query proves we can bump our minimum to python 2.6 anyways, this may go away soon.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#66 lore raw

Re: [Qemu-devel] [PATCH RFC v4 30/32] qapi: New QMP command query-schema for QMP schema introspection

Quoted 24 lines · Eric Blake
Quoted 22 lines · Markus Armbruster
Quoted 6 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 506

Quoted 3 lines
docs/qapi-code-gen.txt +3 -0
+
+query-schema returns a JSON array of SchemaInfo objects.  These
+objects together describe the wire ABI, as defined in the QAPI schema.

Maybe an additional sentence here, perhaps along the lines of:

Quoted 6 lines

Note that the wire format cannot express everything; it is not designed to show semantic constraints (such as when exactly one of a pair of mutually-exclusive optional members must be present, or when an integral value has specific limitations on valid values). Introspection allows an application to know if a feature is present, but the application must obey the qapi documentation to properly interact with that feature.

Michael raised the same point, and I drafted a paragraph in my reply to him:

The SchemaInfo can't reflect all the rules and restrictions that apply to QMP. It's interface introspection (figuring out what's there), not interface specification. The specification is in the QAPI schema, and users should peruse it to understand how QMP is to be used.

Quoted 7 lines
Quoted 4 lines

Then continue with an example showing a restriction that's spelled out in the QAPI schema, but can't be expressed in SchemaInfo.

Use mine as is? Merge our two proposals somehow?

Your wording works okay for me, but if you want to merge any elements of my wording in, go for it.

I'll probably use mine, but I'll look at it once more when I respin this patch.

Quoted 22 lines
Quoted 10 lines
Quoted 5 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 530

Quoted 2 lines
docs/qapi-code-gen.txt +2 -0
+The SchemaInfo for a command has meta-type "command", and variant
+members "arg-type" and "ret-type".  The arguments member clients pass

I found it easier to read with:
s/arguments/arguments that/

Easier to read, but no longer emphasizes that the specific "arguments" member of the { "execute" ... } form needs to conform. I'll use yours anyway, unless you have a better idea.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

+with a command on the wire must conform to the type named by +"arg-type", which is always an object type. The return value the

Likewise s/value/value that/

Likewise.

Quoted 2 lines
Quoted 2 lines

+server passes in a success response conforms to the type named by +"ret-type".

Maybe:

Quoted 8 lines

On the wire, the "arguments" key of a client's "execute" command must conform to the type named by "arg-type", which is always an object type. The "return" key that the server passes in a success response will always conform to the type named by "ret-type".

Up to you; we'll see how it looks in v5, and at that point, any further word-smithing can be done in followup patches rather than delaying this series further.

Sounds nice, I'll probably adopt it.

Quoted 21 lines
Quoted 6 lines
Quoted 4 lines

[not for this patch. Thinking aloud: I wonder if QGA's success-response could be coded in by having 'ret-type':null as a way of saying the command will have no response if it is successful; perhaps a bit nicer than exposing an additional boolean flag member]

The QAPI schema could also do it this way. What do you think?

Sounds like I have another trivial followup patch added to my work queue :)

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

+The SchemaInfo for an event has meta-type "event", and variant member +"arg-type". The data member the server passes with an event conforms

again, reads a little better with s/member/member that/

My remark on command arguments applies.

Quoted 1 line
Quoted 1 line

+to the type named by "arg-type". It is always an object type.

Maybe:

Quoted 8 lines

On the wire, the "data" member included in any event passed by the server will conform to the type named by "arg-type".

I'm not sure if we need to document whether it is always an object type; command arguments are user-supplied, while event data is server-supplied. And there's the question of any possible duplication with the QMP spec file. I guess I'm fine as long as we aren't stating conflicting things between the two places.

I feel duplication is okay when it makes qapi-code-gen.txt easier to understand.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

+
+"members" is a JSON array describing the object's common members.

Could say "the object's common members, if any." What do you think?

Yes, adding the 'if any' helps.

Sold.

Quoted 13 lines
Quoted 9 lines
Quoted 7 lines

Hmm, the "Union types" section has two mentions of BlockdevOptions; and this example matches the second. It may help readability if we rename one of the two examples to be distinct (preferably the first, since it doesn't match actual QMP). I guess the phrase "flat union BlockdevOptions" is sufficient to make it obvious that we are referring to the second usage, but it is subtle.

Follow-up patch?

Yes.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

[1] Ouch. That should be "name": "type". I pointed out the bug in v3, but it looks like it still hasn't been fixed. Or rather, that our attempt to fix it wasn't correct.

I'll double-check.

As you noted later, I've already done that legwork.

Thanks a lot for that!

Quoted 20 lines
Quoted 11 lines
Quoted 7 lines

docs/qapi-code-gen.txt Resulting in this JSON object: · line 675

Quoted 4 lines
docs/qapi-code-gen.txt +4 -0
+
+As explained above, type names are not part of the wire ABI.  Not even
+the names of built-in types.  Clients should examine member
+"json-type" instead of hard-coding names of built-in types.

Good point (although we may still end up with clients that disobey this, I will certainly make sure to do this in libvirt's interactions).

We can't fully prevent misuse of our interfaces. Where we can't, we can still try to make correct use as easy and obvious as we can.

Quoted 8 lines

PATCH 32 prevents misuse of most type names by hiding them. RFC v2 hid all type names. In review, we decided not to hide the names of built-in types. Tradeoff: makes the introspection value easier to read for humans (good), but enables misuse by machines (bad).

Either way works for me, hide non-built-in type names (and have docs tell users not to rely on them), or hide them all. Got a preference?

Quoted 11 lines

I'm still leaning towards exposing built-in type names directly, and furthermore exposing arrays via '[123]' naming. As you say, correct clients shouldn't be abusing this, but it IS easier to see, and I'm having a hard time seeing how it could paint us into corners in the future for having exposed too much information up front (while we DO rename object types and don't want that to affect ABI, I don't see us trying to rename builtin types - at most, we might add subtypes with range or other restrictions, but those would be new types and still leave the basic 'int' and 'str' unchanged). So leaving it as just the documentation on what proper clients should do is sufficient; don't go back to the v2 munging of builtin names.

Okay, no change then. We can explore arrays in follow-up patches.

Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 7 lines

+++ b/qapi/introspect.json

qapi/introspect.json new file +4 -0
@@ -0,0 +1,271 @@
 
+
+##
+# @query-schema
+#

qapi/introspect.json line 25

Quoted 10 lines
qapi/introspect.json new file +10 -0
+# Command query-schema exposes the QMP wire ABI as an array of
+# SchemaInfo.  This lets QMP clients figure out what commands and
+# events are available in this QEMU, and their parameters and results.
+#
+# Returns: array of @SchemaInfo, where each element describes an
+# entity in the ABI: command, event, type, ...
+#
+# Note: the QAPI schema is also used to help define *internal*
+# interfaces, by defining QAPI types.  These are not part of the QMP
+# wire ABI, and therefore not returned by this command.

Same as for the docs above: may be worth a paragraph explicitly mentioning that the wire ABI cannot express everything, such as integer ranges, or such as semantics where exactly one of two mutually-exclusive optional members must be present.

Quoted 8 lines
Quoted 1 line

Here's where Michael commented.

Actually, he mentioned the .hx file, which I didn't. I don't know how many duplicate places need to refer to the same information, or just abbreviate by pointing to other places. But I'll be happy as long as the information is present at least somewhere, and that somewhere is easy to find (perhaps by good references rather than duplications in the other places).

Judging from how I look for information, the QAPI schema should have the complete reference information, while pointers suffice for qmp-commands.hx. docs/qapi-code-gen.txt is introduction / big picture, so it should *explain* things, but it doesn't have to be exhaustively complete.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

As required by RFC 7159, the order of individual elements with in the array sent over the wire is assumed to be significant unless the documented semantics in qapi state otherwise.

Got an example where insignificance of order actually matters?

migrate-set-capabilities does not care what order the 'capabilities' array is in, for example.

Correct, but I'm not sure it matters enough to justify repeating basic JSON semantics here once more, so let's leave things as they are. Could be just me getting worn out by this series (I really, really want it finished). Perhaps I'll be more amenable to a follow-up patch than to changing it now.

Quoted 24 lines
Quoted 9 lines
Quoted 8 lines

qapi/introspect.json line 214

Quoted 2 lines
qapi/introspect.json new file +2 -0
+{ 'struct': 'SchemaInfoObjectVariant',
+  'data': { 'case': 'str', 'type': 'str' } }

Do we want to document whether all case values of the tag are guaranteed to be covered (happens to be true for all current qapi, but is not yet enforced; although that's one of my planned followup patches is to start enforcing it - so I guess I can document it at that time if we don't do it here)

Quoted 4 lines

Requiring all cases to be covered explicitly may have value in the QAPI schema (more verbose, but you get a reminder to update the union after you extended the tag enum), but it's pretty useless in introspection, isn't it?

At first glance, I wouldn't call it useless. But thinking about it
more: Right now, if not all members of the enum are listed, the generated code abort()s when passing the uncovered enum value. But ideally we want semantics where an omitted case in the qapi (or maybe explicit 'case':null or 'case':{} notation) results in adding no further variant members to the overall object. In that case, knowing that the tag enum value is valid and maps to ':empty' on the wire is worth advertising, even if we choose to allow omitting a case in the qapi .json as the syntax for that action.

Quoted 1 line

So we can touch up the wording at the same time we fix code to not abort().

SchemaInfoObject needs to show the complete set of accepted type tag values, and for each value, the variant members.

Currently, SchemaInfoObject.tag always names a member of enumeration type (I guess I should add that to the docs). That type is the set of accepted type tag values.

SchemaInfoObject.variants neither adds nor removes from that set. All it does is show variant members.

If the object has none for a certain type tag value, then a natural and compact way to encode that is not to have it in .variants.

Another way is to have a rather pointless .variants member that explicitly states "no members", basically {"case": "FOO", "type":
":empty"}.

Only if we permitted the type tags type to have more values than object actually accepts as type tags, the presence of a type tag value in .variants becomes meaningful even when it adds no variant members.

Why would we ever want that?

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

scripts/qapi-introspect.py line 15

Quoted 2 lines
scripts/qapi-introspect.py new file +2 -0
+
+# Caveman's json.dumps() replacement (we're stuck at 2.4)

s/2.4/python 2.4/

Okay.

Of course, if Dan's query proves we can bump our minimum to python 2.6 anyways, this may go away soon.

I'll start a thread.

#67 lore raw

RFC PATCH v4 31/32 qapi-introspect: Map all integer types to 'int'

In reply to Markus Armbruster

How many bits we use internally is an implementation detail. It could be pressed into external interface service as very approximate range information, but that's probably a bad idea. If we need range information, we better do it properly.

Reduces output of query-schema by a negligible 0.5 out of 85KiB.

---
 docs/qapi-code-gen.txt     | 11 ++++++++---
 scripts/qapi-introspect.py |  7 +++++++
 2 files changed, 15 insertions(+), 3 deletions(-)
docs/qapi-code-gen.txt +8 -3
index 1f8f891..a079b51 100644
@@ -662,14 +662,19 @@ Example: the SchemaInfo for MyEnum from section Enumeration types
       "values": [ "value1", "value2", "value3" ] }
 
 The SchemaInfo for a built-in type has the same name as the type in
-the QAPI schema (see section Built-in Types).  It has variant member
-"json-type" that shows how values of this type are encoded on the
-wire.
+the QAPI schema (see section Built-in Types), with one exception
+detailed below.  It has variant member "json-type" that shows how
+values of this type are encoded on the wire.
 
 Example: the SchemaInfo for str
 
     { "name": "str", "meta-type": "builtin", "json-type": "string" }
 
+The QAPI schema supports a number of integer types that only differ in
+how they map to C.  They are identical as far as SchemaInfo is
+concerned.  Therefore, they get all mapped to a single type "int" in
+SchemaInfo.
+
 As explained above, type names are not part of the wire ABI.  Not even
 the names of built-in types.  Clients should examine member
 "json-type" instead of hard-coding names of built-in types.
scripts/qapi-introspect.py +7 -0
index 4bcffc4..2f6a4c6 100644
@@ -74,6 +74,13 @@ const char %(c_name)s[] = %(c_string)s;
         self.used_types = None
 
     def _use_type(self, typ):
+        # Map the various integer types to plain int
+        if typ.json_type() == 'int':
+            typ = self.schema.lookup_type('int')
+        elif isinstance(typ, QAPISchemaArrayType) \
+             and typ.element_type.json_type() == 'int':
+            typ = self.schema.lookup_type('intList')
+        # Add type to work queue if new
         if typ not in self.used_types:
             self.used_types.append(typ)
         return typ.name
Signature · 1 line
2.4.3
Reviewed-by on v4 31/32
#68 lore raw

Re: [Qemu-devel] [PATCH RFC v4 31/32] qapi-introspect: Map all integer types to 'int'

Quoted 5 lines · Markus Armbruster

How many bits we use internally is an implementation detail. It could be pressed into external interface service as very approximate range information, but that's probably a bad idea. If we need range information, we better do it properly.

Quoted 7 lines

Reduces output of query-schema by a negligible 0.5 out of 85KiB.

---
 docs/qapi-code-gen.txt     | 11 ++++++++---
 scripts/qapi-introspect.py |  7 +++++++
 2 files changed, 15 insertions(+), 3 deletions(-)

Wording is nice; and as discussed this will be squashed into 30 for v5.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#69 lore raw

RFC PATCH v4 32/32 qapi-introspect: Hide type names

In reply to Markus Armbruster

To eliminate the temptation for clients to look up types by name (which are not ABI), replace all type names by meaningless strings.

Reduces output of query-schema by 13 out of 85KiB.

As a debugging aid, provide option -u to suppress the hiding.

---
 docs/qapi-code-gen.txt     | 36 +++++++++++++++++-------------------
 qapi/introspect.json       | 11 +++--------
 scripts/qapi-introspect.py | 41 +++++++++++++++++++++++++++++++++++------
 3 files changed, 55 insertions(+), 33 deletions(-)
docs/qapi-code-gen.txt +17 -19
index a079b51..94fc296 100644
@@ -516,13 +516,16 @@ additional variant members depending on the value of meta-type.
 Each SchemaInfo object describes a wire ABI entity of a certain
 meta-type: a command, event or one of several kinds of type.
 
-SchemaInfo for entities defined in the QAPI schema have the same name
-as in the schema.  This is the case for all commands and events, and
-most types.
+SchemaInfo for commands and events have the same name as in the QAPI
+schema
 
 Command and event names are part of the wire ABI, but type names are
-not.  Therefore, looking up a type by its name in the QAPI schema is
-wrong.  Look up the command or event, then follow references by name.
+not.  Therefore, the SchemaInfo for types have auto-generated
+meaningless names.  For readability, the examples in this document use
+meaningful type names instead.
+
+To examine a type, start with a the command or event using it, then
+follow references by name.
 
 QAPI schema definitions not reachable that way are omitted.
 
@@ -554,8 +557,7 @@ object type without members.  The event may not have a data member on
 the wire then.
 
 Each command or event defined with dictionary-valued 'data' in the
-QAPI schema implicitly defines an object type called ":obj-NAME-arg",
-where NAME is the command or event's name.
+QAPI schema implicitly defines an object type.
 
 Example: the SchemaInfo for EVENT_C from section Events
 
@@ -609,12 +611,9 @@ Note that base types are "flattened": its members are included in the
 
 A simple union implicitly defines an enumeration type for its implicit
 discriminator (called "type" on the wire, see section Union types).
-Such a type's name is made by appending "Kind" to the simple union's
-name.
 
 A simple union implicitly defines an object type for each of its
-variants.  The type's name is ":obj-NAME-wrapper", where NAME is the
-name of the name of the variant's type.
+variants.
 
 Example: the SchemaInfo for simple union BlockdevOptions from section
 Union types
@@ -645,8 +644,7 @@ Example: the SchemaInfo for BlockRef from section Alternate types
 
 The SchemaInfo for an array type has meta-type "array", and variant
 member "element-type", which names the array's element type.  Array
-types are implicitly defined.  An array type's name is made by
-appending "List" to its element type's name.
+types are implicitly defined.
 
 Example: the SchemaInfo for ['str']
 
@@ -1053,13 +1051,13 @@ Example:
 [Uninteresting stuff omitted...]
 
     const char example_qmp_schema_json[] = "["
-        "{\"arg-type\": \":empty\", \"meta-type\": \"event\", \"name\": \"MY_EVENT\"}, "
+        "{\"arg-type\": \"0\", \"meta-type\": \"event\", \"name\": \"MY_EVENT\"}, "
+        "{\"arg-type\": \"1\", \"meta-type\": \"command\", \"name\": \"my-command\", \"ret-type\": \"2\"}, "
+        "{\"members\": [], \"meta-type\": \"object\", \"name\": \"0\"}, "
+        "{\"members\": [{\"name\": \"arg1\", \"type\": \"2\"}], \"meta-type\": \"object\", \"name\": \"1\"}, "
+        "{\"members\": [{\"name\": \"integer\", \"type\": \"int\"}, {\"name\": \"string\", \"type\": \"str\"}], \"meta-type\": \"object\", \"name\": \"2\"}, "
         "{\"json-type\": \"int\", \"meta-type\": \"builtin\", \"name\": \"int\"}, "
-        "{\"json-type\": \"string\", \"meta-type\": \"builtin\", \"name\": \"str\"}, "
-        "{\"members\": [], \"meta-type\": \"object\", \"name\": \":empty\"}, "
-        "{\"members\": [{\"name\": \"arg1\", \"type\": \"UserDefOne\"}], \"meta-type\": \"object\", \"name\": \":obj-my-command-arg\"}, "
-        "{\"members\": [{\"name\": \"integer\", \"type\": \"int\"}, {\"name\": \"string\", \"type\": \"str\"}], \"meta-type\": \"object\", \"name\": \"UserDefOne\"}, "
-        "{\"arg-type\": \":obj-my-command-arg\", \"meta-type\": \"command\", \"name\": \"my-command\", \"ret-type\": \"UserDefOne\"}]";
+        "{\"json-type\": \"string\", \"meta-type\": \"builtin\", \"name\": \"str\"}]";
     $ cat qapi-generated/example-qmp-introspect.h
 [Uninteresting stuff omitted...]
 
qapi/introspect.json +3 -8
index 171baba..46e78a1 100644
@@ -70,17 +70,12 @@
 # @SchemaInfo
 #
 # @name: the entity's name, inherited from @base.
-# Entities defined in the QAPI schema have the name defined in the schema.
-# Implicitly defined entities have generated names.  See
-# docs/qapi-code-gen.txt section "Client JSON Protocol introspection"
-# for details.
+# Commands and events have the name defined in the QAPI schema.
+# Unlike command and event names, type names are not part of the wire
+# ABI.  Consequently, type names are meaningless strings here.
 #
 # All references to other SchemaInfo are by name.
 #
-# Command and event names are part of the wire ABI, but type names are
-# not.  Therefore, looking up a type by "well-known" name is wrong.
-# Look up the command or event, then follow the references.
-#
 # @meta-type: the entity's meta type, inherited from @base.
 #
 # Additional members depend on the value of @meta-type.
scripts/qapi-introspect.py +35 -6

96 lines not on this page · raw message

Signature · 1 line
2.4.3
Reviewed-by on v4 32/32
#70 lore raw

Re: [Qemu-devel] [PATCH RFC v4 32/32] qapi-introspect: Hide type names

Quoted 19 lines · Markus Armbruster

To eliminate the temptation for clients to look up types by name (which are not ABI), replace all type names by meaningless strings.

Reduces output of query-schema by 13 out of 85KiB.

As a debugging aid, provide option -u to suppress the hiding.

---
 docs/qapi-code-gen.txt     | 36 +++++++++++++++++-------------------
 qapi/introspect.json       | 11 +++--------
 scripts/qapi-introspect.py | 41 +++++++++++++++++++++++++++++++++++------
 3 files changed, 55 insertions(+), 33 deletions(-)
docs/qapi-code-gen.txt +0 -0
index a079b51..94fc296 100644

docs/qapi-code-gen.txt additional variant members depending on the value of meta-type. · line 520

Quoted 8 lines
docs/qapi-code-gen.txt +2 -3
@@ -516,13 +516,16 @@ additional variant members depending on the value of meta-type.
 Each SchemaInfo object describes a wire ABI entity of a certain
 meta-type: a command, event or one of several kinds of type.
 
-SchemaInfo for entities defined in the QAPI schema have the same name
-as in the schema.  This is the case for all commands and events, and
-most types.
+SchemaInfo for commands and events have the same name as in the QAPI
+schema

Missing a trailing '.'

docs/qapi-code-gen.txt additional variant members depending on the value of meta-type. · line 527

Quoted 9 lines
docs/qapi-code-gen.txt +5 -2
 Command and event names are part of the wire ABI, but type names are
-not.  Therefore, looking up a type by its name in the QAPI schema is
-wrong.  Look up the command or event, then follow references by name.
+not.  Therefore, the SchemaInfo for types have auto-generated
+meaningless names.  For readability, the examples in this document use
+meaningful type names instead.
+
+To examine a type, start with a the command or event using it, then

s/a the/the/

docs/qapi-code-gen.txt Example: · line 1055

Quoted 10 lines
docs/qapi-code-gen.txt +3 -1
+follow references by name.
 
 
@@ -1053,13 +1051,13 @@ Example:
 [Uninteresting stuff omitted...]
 
     const char example_qmp_schema_json[] = "["
-        "{\"arg-type\": \":empty\", \"meta-type\": \"event\", \"name\": \"MY_EVENT\"}, "
+        "{\"arg-type\": \"0\", \"meta-type\": \"event\", \"name\": \"MY_EVENT\"}, "
+        "{\"arg-type\": \"1\", \"meta-type\": \"command\", \"name\": \"my-command\", \"ret-type\": \"2\"}, "

I thought you said that this document would use the meaningful type names? Oh, I see - the earlier examples (not touched by this commit) DO use meaningful names; while _this_ example uses the munged names. I don't know if any additional wording would help.

Would it be worth documenting -u as a usage argument for qapi-introspect.py? [For that matter, should we document what -m for middle-mode does for qapi-commands.py? But as a separate patch]

I still think that masking arrays as "[123]" is nicer, but that can be a followup patch (especially since I've already posted it in one of my numerous RFC followups, which I'll have to rebase anyways)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#71 lore raw

Re: [Qemu-devel] [PATCH RFC v4 32/32] qapi-introspect: Hide type names

Quoted 24 lines · Eric Blake
Quoted 23 lines · Markus Armbruster

To eliminate the temptation for clients to look up types by name (which are not ABI), replace all type names by meaningless strings.

Reduces output of query-schema by 13 out of 85KiB.

As a debugging aid, provide option -u to suppress the hiding.

---
 docs/qapi-code-gen.txt     | 36 +++++++++++++++++-------------------
 qapi/introspect.json       | 11 +++--------
 scripts/qapi-introspect.py | 41 +++++++++++++++++++++++++++++++++++------
 3 files changed, 55 insertions(+), 33 deletions(-)
docs/qapi-code-gen.txt +0 -0
index a079b51..94fc296 100644
@@ -516,13 +516,16 @@ additional variant members depending on the

value of meta-type.
Each SchemaInfo object describes a wire ABI entity of a certain
meta-type: a command, event or one of several kinds of type.

Quoted 7 lines

docs/qapi-code-gen.txt additional variant members depending on the value of meta-type. · line 520

Quoted 5 lines
docs/qapi-code-gen.txt +2 -3
-SchemaInfo for entities defined in the QAPI schema have the same name
-as in the schema.  This is the case for all commands and events, and
-most types.
+SchemaInfo for commands and events have the same name as in the QAPI
+schema

Missing a trailing '.'

Will fix.

Quoted 10 lines

docs/qapi-code-gen.txt additional variant members depending on the value of meta-type. · line 527

Quoted 8 lines

Command and event names are part of the wire ABI, but type names are

docs/qapi-code-gen.txt +5 -2
-not.  Therefore, looking up a type by its name in the QAPI schema is
-wrong.  Look up the command or event, then follow references by name.
+not.  Therefore, the SchemaInfo for types have auto-generated
+meaningless names.  For readability, the examples in this document use
+meaningful type names instead.
+
+To examine a type, start with a the command or event using it, then

s/a the/the/

Will fix.

Quoted 6 lines
Quoted 6 lines
docs/qapi-code-gen.txt +1 -0
+follow references by name.
@@ -1053,13 +1051,13 @@ Example:
 [Uninteresting stuff omitted...]
 
Quoted 9 lines

docs/qapi-code-gen.txt Example: · line 1055

Quoted 4 lines
docs/qapi-code-gen.txt +2 -1
     const char example_qmp_schema_json[] = "["
-        "{\"arg-type\": \":empty\", \"meta-type\": \"event\", \"name\": \"MY_EVENT\"}, "
+        "{\"arg-type\": \"0\", \"meta-type\": \"event\", \"name\": \"MY_EVENT\"}, "
+        "{\"arg-type\": \"1\", \"meta-type\": \"command\", \"name\": \"my-command\", \"ret-type\": \"2\"}, "

I thought you said that this document would use the meaningful type names? Oh, I see - the earlier examples (not touched by this commit) DO use meaningful names; while _this_ example uses the munged names. I don't know if any additional wording would help.

Different section. I'll change the wording there

meaningless names. For readability, the examples in this document use meaningful type names instead.

to say "this section" instead of "this document".

Quoted 3 lines

Would it be worth documenting -u as a usage argument for qapi-introspect.py? [For that matter, should we document what -m for middle-mode does for qapi-commands.py? But as a separate patch]

None of the options are documented outside the source, and even the source doesn't really explain. It's just a development tool. I guess adding --help would be nice. Beyond that, returns diminish rapidly.

Quoted 5 lines

I still think that masking arrays as "[123]" is nicer, but that can be a followup patch (especially since I've already posted it in one of my numerous RFC followups, which I'll have to rebase anyways)

Thanks!

#72 lore raw
In reply to Markus Armbruster

Hi Markus

Quoted 6 lines · Markus Armbruster

Series is based on "[PATCH 00/26] qapi: Another round of fixes and cleanups" (which actually has 33 patches now). It also needs my "[PATCH for-2.4 0/2] qom: Fix misuse of Error API" to pass tests.

You can fetch it from my git://repo.or.cz/qemu/armbru.git branch qapi-introspect.

For the records, I have also rebased my qapi-related patches on top of your branch. It adds: varargs support, doc generation, use of qmp_dispatch() & async support.

https://github.com/elmarco/qemu branch qapi

cheers

Signature · 1 line
Marc-André Lureau
#73 lore raw
Quoted 7 lines · Marc-André Lureau

Hi Markus

Quoted 4 lines · Markus Armbruster

Series is based on "[PATCH 00/26] qapi: Another round of fixes and cleanups" (which actually has 33 patches now). It also needs my "[PATCH for-2.4 0/2] qom: Fix misuse of Error API" to pass tests.

Quoted 9 lines
Quoted 2 lines

You can fetch it from my git://repo.or.cz/qemu/armbru.git branch qapi-introspect.

For the records, I have also rebased my qapi-related patches on top of your branch. It adds: varargs support, doc generation, use of qmp_dispatch() & async support.

https://github.com/elmarco/qemu branch qapi

I don't think we want varargs; you added it for 'netdev_add', but the right fix for that is to use a flat union type and fix 'command' to allow a boxed union argument, rather than adding an untyped varargs.

I guess I need to rebase my pending patches, and get them posted, since it touches some of the same things. We'll get netdev_add fixed soon enough :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#74 lore raw
In reply to Marc-André Lureau
Quoted 10 lines · Marc-André Lureau

Hi Markus

Quoted 6 lines · Markus Armbruster

Series is based on "[PATCH 00/26] qapi: Another round of fixes and cleanups" (which actually has 33 patches now). It also needs my "[PATCH for-2.4 0/2] qom: Fix misuse of Error API" to pass tests.

You can fetch it from my git://repo.or.cz/qemu/armbru.git branch qapi-introspect.

Quoted 7 lines

For the records, I have also rebased my qapi-related patches on top of your branch. It adds: varargs support, doc generation, use of qmp_dispatch() & async support.

https://github.com/elmarco/qemu branch qapi

cheers

By now we have quite a line of patches queuing up behind this series, most of them in need of review. Let's remember "perfect is the enemy of good", wrap up my series, and get the queue moving.