Skip to contentSkip to conversation

qapi: Code generation fixes

PATCH In review Markus Armbruster 2 participants 17 messages Mar 1, 2019 – Mar 4, 2019

lore
Share & tools
lore
#0 lore raw

cover qapi: Code generation fixes

Markus Armbruster (7):
tests/qapi-schema: Make test-qapi.py print arrays tests/qapi-schema: Cover conditional arrays
qapi: Pass file name to QAPIGen constructor instead of methods
qapi: Fix code generation for sub-modules in other directories
tests: Rename UserDefNativeListUnion to UserDefListUnion tests/qapi-schema: Cover forward reference to sub-module
qapi: Fix array first used in a different module

 scripts/qapi/commands.py                   |   2 +-
 scripts/qapi/common.py                     | 106 ++++++-----
 scripts/qapi/doc.py                        |   4 +-
 tests/.gitignore                           |   8 +
 tests/Makefile.include                     |  44 ++++-
 tests/qapi-schema/include-relpath-sub.json |   2 -
 tests/qapi-schema/include-relpath.err      |   0
 tests/qapi-schema/include-relpath.exit     |   1 -
 tests/qapi-schema/include-relpath.json     |   1 -
 tests/qapi-schema/include-relpath.out      |  20 ---
 tests/qapi-schema/include/relpath.json     |   1 -
 tests/qapi-schema/include/sub-module.json  |   7 +
 tests/qapi-schema/qapi-schema-test.json    |  14 +-
 tests/qapi-schema/qapi-schema-test.out     |  37 +++-
 tests/qapi-schema/sub-sub-module.json      |   6 +
 tests/qapi-schema/test-qapi.py             |   6 +
 tests/test-clone-visitor.c                 |  16 +-
 tests/test-qmp-cmds.c                      |   2 +-
 tests/test-qobject-input-visitor.c         | 196 ++++++++++-----------
 tests/test-qobject-output-visitor.c        | 190 ++++++++++----------
 20 files changed, 371 insertions(+), 292 deletions(-)
 delete mode 100644 tests/qapi-schema/include-relpath-sub.json
 delete mode 100644 tests/qapi-schema/include-relpath.err
 delete mode 100644 tests/qapi-schema/include-relpath.exit
 delete mode 100644 tests/qapi-schema/include-relpath.json
 delete mode 100644 tests/qapi-schema/include-relpath.out
 delete mode 100644 tests/qapi-schema/include/relpath.json
 create mode 100644 tests/qapi-schema/include/sub-module.json
 create mode 100644 tests/qapi-schema/sub-sub-module.json
Signature · 1 line
2.17.2
#1 lore raw

PATCH 1/7 tests/qapi-schema: Make test-qapi.py print arrays

---
 tests/qapi-schema/qapi-schema-test.out | 6 ++++++
 tests/qapi-schema/test-qapi.py         | 6 ++++++
 2 files changed, 12 insertions(+)
tests/qapi-schema/qapi-schema-test.out +6 -0
index d8aec17115..b59f4ca5a4 100644
@@ -52,6 +52,9 @@ object UserDefTwo
     member dict1: UserDefTwoDict optional=False
 object UserDefThree
     member string0: str optional=False
+array UserDefOneList UserDefOne
+array UserDefTwoList UserDefTwo
+array TestStructList TestStruct
 object ForceArrays
     member unused1: UserDefOneList optional=False
     member unused2: UserDefTwoList optional=False
@@ -248,6 +251,7 @@ object __org.qemu_x-Union1
     member type: __org.qemu_x-Union1Kind optional=False
     tag type
     case __org.qemu_x-branch: q_obj_str-wrapper
+array __org.qemu_x-Union1List __org.qemu_x-Union1
 object __org.qemu_x-Struct2
     member array: __org.qemu_x-Union1List optional=False
 object __org.qemu_x-Union2
@@ -260,6 +264,8 @@ alternate __org.qemu_x-Alt
     case b: __org.qemu_x-Base
 event __ORG.QEMU_X-EVENT __org.qemu_x-Struct
    boxed=False
+array __org.qemu_x-EnumList __org.qemu_x-Enum
+array __org.qemu_x-StructList __org.qemu_x-Struct
 object q_obj___org.qemu_x-command-arg
     member a: __org.qemu_x-EnumList optional=False
     member b: __org.qemu_x-StructList optional=False
tests/qapi-schema/test-qapi.py +6 -0
index d592854601..d21fca01fc 100644
@@ -32,6 +32,12 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
             self._print_if(m.ifcond, indent=8)
         self._print_if(ifcond)
 
+    def visit_array_type(self, name, info, ifcond, element_type):
+        if not info:
+            return              # suppress built-in arrays
+        print('array %s %s' % (name, element_type.name))
+        self._print_if(ifcond)
+
     def visit_object_type(self, name, info, ifcond, base, members, variants):
         print('object %s' % name)
         if base:
Signature · 1 line
2.17.2
Reviewed-by on v1 1/7
#2 lore raw

Re: [Qemu-devel] [PATCH 1/7] tests/qapi-schema: Make test-qapi.py print arrays

Quoted 2 lines · Markus Armbruster

---

Sparse on the why, but I can guess - you intend for the rest of the series to fix the bug where we don't generate arrays in the same module as their underlying type when their only use is in a different module, but to prove that works you need them listed in dump output.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#3 lore raw

Re: [Qemu-devel] [PATCH 1/7] tests/qapi-schema: Make test-qapi.py print arrays

Quoted 8 lines · Eric Blake
Quoted 2 lines · Markus Armbruster

---

Sparse on the why, but I can guess - you intend for the rest of the series to fix the bug where we don't generate arrays in the same module as their underlying type when their only use is in a different module, but to prove that works you need them listed in dump output.

Could amend like this:

tests/qapi-schema: Make test-qapi.py print arrays

The next few commits mess with array types, and having the changes exposed in output of test-qapi.py will be useful.

Quoted 1 line

Thanks!

#4 lore raw

PATCH 2/7 tests/qapi-schema: Cover conditional arrays

In reply to Markus Armbruster

Commit 967c885108f neglected to cover arrays of conditional types. Do that now.

---
 tests/qapi-schema/qapi-schema-test.json | 2 +-
 tests/qapi-schema/qapi-schema-test.out  | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)
tests/qapi-schema/qapi-schema-test.json +1 -1
index cb0857df52..70612f394e 100644
@@ -236,5 +236,5 @@
 
 { 'event': 'TestIfEvent', 'data':
   { 'foo': 'TestIfStruct',
-    'bar': { 'type': 'TestIfEnum', 'if': 'defined(TEST_IF_EVT_BAR)' } },
+    'bar': { 'type': ['TestIfEnum'], 'if': 'defined(TEST_IF_EVT_BAR)' } },
   'if': 'defined(TEST_IF_EVT) && defined(TEST_IF_STRUCT)' }
tests/qapi-schema/qapi-schema-test.out +3 -1
index b59f4ca5a4..97f671ca5f 100644
@@ -325,9 +325,11 @@ command TestIfCmd q_obj_TestIfCmd-arg -> UserDefThree
     if ['defined(TEST_IF_CMD)', 'defined(TEST_IF_STRUCT)']
 command TestCmdReturnDefThree None -> UserDefThree
    gen=True success_response=True boxed=False oob=False preconfig=False
+array TestIfEnumList TestIfEnum
+    if ['defined(TEST_IF_ENUM)']
 object q_obj_TestIfEvent-arg
     member foo: TestIfStruct optional=False
-    member bar: TestIfEnum optional=False
+    member bar: TestIfEnumList optional=False
         if ['defined(TEST_IF_EVT_BAR)']
     if ['defined(TEST_IF_EVT) && defined(TEST_IF_STRUCT)']
 event TestIfEvent q_obj_TestIfEvent-arg
Signature · 1 line
2.17.2
Reviewed-by on v1 2/7
#5 lore raw

Re: [Qemu-devel] [PATCH 2/7] tests/qapi-schema: Cover conditional arrays

Quoted 9 lines · Markus Armbruster

Commit 967c885108f neglected to cover arrays of conditional types. Do that now.

---
 tests/qapi-schema/qapi-schema-test.json | 2 +-
 tests/qapi-schema/qapi-schema-test.out  | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#6 lore raw

PATCH 3/7 qapi: Pass file name to QAPIGen constructor instead of methods

In reply to Markus Armbruster

Not much of an improvement now, but the next commit will profit.

---
 scripts/qapi/commands.py |  2 +-
 scripts/qapi/common.py   | 68 +++++++++++++++++++++-------------------
 scripts/qapi/doc.py      |  4 +--
 3 files changed, 38 insertions(+), 36 deletions(-)
scripts/qapi/commands.py +1 -1
index ebf488953d..6d66bf6aa3 100644
@@ -239,7 +239,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
         QAPISchemaModularCVisitor.__init__(
             self, prefix, 'qapi-commands',
             ' * Schema-defined QAPI/QMP commands', __doc__)
-        self._regy = QAPIGenCCode()
+        self._regy = QAPIGenCCode(None)
         self._visited_ret_types = {}
 
     def _begin_user_module(self, name):
scripts/qapi/common.py +35 -33

153 lines not on this page · raw message

scripts/qapi/doc.py +2 -2
index c03b690161..5c8c136899 100755
@@ -207,11 +207,11 @@ def texi_entity(doc, what, ifcond, base=None, variants=None,
 class QAPISchemaGenDocVisitor(qapi.common.QAPISchemaVisitor):
     def __init__(self, prefix):
         self._prefix = prefix
-        self._gen = qapi.common.QAPIGenDoc()
+        self._gen = qapi.common.QAPIGenDoc(self._prefix + 'qapi-doc.texi')
         self.cur_doc = None
 
     def write(self, output_dir):
-        self._gen.write(output_dir, self._prefix + 'qapi-doc.texi')
+        self._gen.write(output_dir)
 
     def visit_enum_type(self, name, info, ifcond, members, prefix):
         doc = self.cur_doc
Signature · 1 line
2.17.2
Reviewed-by on v1 3/7
#7 lore raw

Re: [Qemu-devel] [PATCH 3/7] qapi: Pass file name to QAPIGen constructor instead of methods

Quoted 9 lines · Markus Armbruster

Not much of an improvement now, but the next commit will profit.

---
 scripts/qapi/commands.py |  2 +-
 scripts/qapi/common.py   | 68 +++++++++++++++++++++-------------------
 scripts/qapi/doc.py      |  4 +--
 3 files changed, 38 insertions(+), 36 deletions(-)

Looks like a straightforward refactoring, and I can indeed guess how it becomes useful later.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#8 lore raw

PATCH 4/7 qapi: Fix code generation for sub-modules in other directories

In reply to Markus Armbruster

The #include directives to pull in sub-modules use file names relative to the main module. Works only when all modules are in the same directory, or the main module's output directory is in the compiler's include path. Use relative file names instead.

The dummy variable we generate to avoid empty .o files has an invalid name for sub-modules in other directories. Fix that.

Both messed up in commit 252dc3105fc "qapi: Generate separate .h, .c for each module". Escaped testing because tests/qapi-schema-test.json doesn't cover sub-modules in other directories, only tests/qapi-schema/include-relpath.json does, and we generate and compile C code only for the former, not the latter. Fold the latter into the former. This would have caught the mistakes fixed in this commit.

---
 scripts/qapi/common.py                     | 33 ++++++++++------
 tests/.gitignore                           |  8 ++++
 tests/Makefile.include                     | 44 +++++++++++++++++++---
 tests/qapi-schema/include-relpath-sub.json |  2 -
 tests/qapi-schema/include-relpath.err      |  0
 tests/qapi-schema/include-relpath.exit     |  1 -
 tests/qapi-schema/include-relpath.json     |  1 -
 tests/qapi-schema/include-relpath.out      | 20 ----------
 tests/qapi-schema/include/relpath.json     |  1 -
 tests/qapi-schema/include/sub-module.json  |  5 +++
 tests/qapi-schema/qapi-schema-test.json    |  3 ++
 tests/qapi-schema/qapi-schema-test.out     |  9 +++++
 tests/qapi-schema/sub-sub-module.json      |  6 +++
 13 files changed, 91 insertions(+), 42 deletions(-)
 delete mode 100644 tests/qapi-schema/include-relpath-sub.json
 delete mode 100644 tests/qapi-schema/include-relpath.err
 delete mode 100644 tests/qapi-schema/include-relpath.exit
 delete mode 100644 tests/qapi-schema/include-relpath.json
 delete mode 100644 tests/qapi-schema/include-relpath.out
 delete mode 100644 tests/qapi-schema/include/relpath.json
 create mode 100644 tests/qapi-schema/include/sub-module.json
 create mode 100644 tests/qapi-schema/sub-sub-module.json
scripts/qapi/common.py +21 -12
index 8512cac427..f51948364c 100644
@@ -2017,8 +2017,8 @@ def mcgen(code, **kwds):
     return cgen(code, **kwds)
 
 
-def guardname(filename):
-    return re.sub(r'[^A-Za-z0-9_]', '_', filename).upper()
+def c_fname(filename):
+    return re.sub(r'[^A-Za-z0-9_]', '_', filename)
 
 
 def guardstart(name):
@@ -2027,7 +2027,7 @@ def guardstart(name):
 #define %(name)s
 
 ''',
-                 name=guardname(name))
+                 name=c_fname(name).upper())
 
 
 def guardend(name):
@@ -2035,7 +2035,7 @@ def guardend(name):
 
 #endif /* %(name)s */
 ''',
-                 name=guardname(name))
+                 name=c_fname(name).upper())
 
 
 def gen_if(ifcond):
@@ -2281,9 +2281,9 @@ class QAPIGenC(QAPIGenCCode):
         return mcgen('''
 
 /* Dummy declaration to prevent empty .o file */
-char dummy_%(name)s;
+char qapi_dummy_%(name)s;
 ''',
-                     name=c_name(self.fname))
+                     name=c_fname(self.fname))
 
 
 class QAPIGenH(QAPIGenC):
@@ -2337,21 +2337,29 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor):
     def _is_builtin_module(name):
         return not name
 
+    def _module_dirname(self, what, name):
+        if self._is_user_module(name):
+            return os.path.dirname(name)
+        return ''
+
     def _module_basename(self, what, name):
         ret = '' if self._is_builtin_module(name) else self._prefix
         if self._is_user_module(name):
-            dirname, basename = os.path.split(name)
+            basename = os.path.basename(name)
             ret += what
             if name != self._main_module:
                 ret += '-' + os.path.splitext(basename)[0]
-            ret = os.path.join(dirname, ret)
         else:
             name = name[2:] if name else 'builtin'
             ret += re.sub(r'-', '-' + name + '-', what)
         return ret
 
+    def _module_filename(self, what, name):
+        return os.path.join(self._module_dirname(what, name),
+                            self._module_basename(what, name))
+
     def _add_module(self, name, blurb):
-        basename = self._module_basename(self._what, name)
+        basename = self._module_filename(self._what, name)
         genc = QAPIGenC(basename + '.c', blurb, self._pydoc)
         genh = QAPIGenH(basename + '.h', blurb, self._pydoc)
         self._module[name] = (genc, genh)
@@ -2393,8 +2401,9 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor):
             self._begin_user_module(name)
 
     def visit_include(self, name, info):
-        basename = self._module_basename(self._what, name)
+        relname = os.path.relpath(self._module_filename(self._what, name),
+                                  os.path.dirname(self._genh.fname))
         self._genh.preamble_add(mcgen('''
-#include "%(basename)s.h"
+#include "%(relname)s.h"
 ''',
-                                      basename=basename))
+                                      relname=relname))
tests/.gitignore +8 -0
index 72c18aaab0..b7b61ffb38 100644
@@ -11,9 +11,17 @@ test-*
 !test-*.c
 !docker/test-*
 test-qapi-commands.[ch]
+include/test-qapi-commands-sub-module.[ch]
+test-qapi-commands-sub-sub-module.[ch]
 test-qapi-events.[ch]
+include/test-qapi-events-sub-module.[ch]
+test-qapi-events-sub-sub-module.[ch]
 test-qapi-types.[ch]
+include/test-qapi-types-sub-module.[ch]
+test-qapi-types-sub-sub-module.[ch]
 test-qapi-visit.[ch]
+include/test-qapi-visit-sub-module.[ch]
+test-qapi-visit-sub-sub-module.[ch]
 test-qapi-introspect.[ch]
 *-test
 qapi-schema/*.test.*
tests/Makefile.include +39 -5

73 lines not on this page · raw message

tests/qapi-schema/include-relpath-sub.json deleted +0 -2
deleted file mode 100644
index 4bd4af4162..0000000000
@@ -1,2 +0,0 @@
-{ 'enum': 'Status',
-  'data': [ 'good', 'bad', 'ugly' ] }
tests/qapi-schema/include-relpath.err deleted +0 -0
deleted file mode 100644
index e69de29bb2..0000000000
tests/qapi-schema/include-relpath.exit deleted +0 -1
deleted file mode 100644
index 573541ac97..0000000000
@@ -1 +0,0 @@
-0
tests/qapi-schema/include-relpath.json deleted +0 -1
deleted file mode 100644
index 05018f3908..0000000000
@@ -1 +0,0 @@
-{ 'include': 'include/relpath.json' }
tests/qapi-schema/include-relpath.out deleted +0 -20
deleted file mode 100644
index cf8636b78f..0000000000
@@ -1,20 +0,0 @@
-module None
-object q_empty
-enum QType
-    prefix QTYPE
-    member none
-    member qnull
-    member qnum
-    member qstring
-    member qdict
-    member qlist
-    member qbool
-module include-relpath.json
-include include/relpath.json
-module include/relpath.json
-include include-relpath-sub.json
-module include-relpath-sub.json
-enum Status
-    member good
-    member bad
-    member ugly
tests/qapi-schema/include/relpath.json deleted +0 -1
deleted file mode 100644
index 45dee24704..0000000000
@@ -1 +0,0 @@
-{ 'include': '../include-relpath-sub.json' }
tests/qapi-schema/include/sub-module.json new file +5 -0
new file mode 100644
index 0000000000..f2bdbd3990
@@ -0,0 +1,5 @@
+# *-*- Mode: Python -*-*
+
+# Sub-module of ../qapi-schema-test.json
+
+{ 'include': '../sub-sub-module.json' }
tests/qapi-schema/qapi-schema-test.json +3 -0
index 70612f394e..1f130a0491 100644
@@ -130,6 +130,9 @@
             'sizes': ['size'],
             'any': ['any'] } }
 
+# for testing sub-modules
+{ 'include': 'include/sub-module.json' }
+
 # testing commands
 { 'command': 'user_def_cmd', 'data': {} }
 { 'command': 'user_def_cmd1', 'data': {'ud1a': 'UserDefOne'} }
tests/qapi-schema/qapi-schema-test.out +9 -0
index 97f671ca5f..baba4c8384 100644
@@ -176,6 +176,15 @@ object UserDefNativeListUnion
     case string: q_obj_strList-wrapper
     case sizes: q_obj_sizeList-wrapper
     case any: q_obj_anyList-wrapper
+include include/sub-module.json
+module include/sub-module.json
+include sub-sub-module.json
+module sub-sub-module.json
+enum Status
+    member good
+    member bad
+    member ugly
+module qapi-schema-test.json
 command user_def_cmd None -> None
    gen=True success_response=True boxed=False oob=False preconfig=False
 object q_obj_user_def_cmd1-arg
tests/qapi-schema/sub-sub-module.json new file +6 -0
new file mode 100644
index 0000000000..524ef9b83f
@@ -0,0 +1,6 @@
+# *-*- Mode: Python -*-*
+
+# Sub-module of sub-module include/sub-module.json of qapi-schema-test.json
+
+{ 'enum': 'Status',
+  'data': [ 'good', 'bad', 'ugly' ] }
Signature · 1 line
2.17.2
Reviewed-by on v1 4/7
#9 lore raw

Re: [Qemu-devel] [PATCH 4/7] qapi: Fix code generation for sub-modules in other directories

Quoted 27 lines · Markus Armbruster

The #include directives to pull in sub-modules use file names relative to the main module. Works only when all modules are in the same directory, or the main module's output directory is in the compiler's include path. Use relative file names instead.

The dummy variable we generate to avoid empty .o files has an invalid name for sub-modules in other directories. Fix that.

Both messed up in commit 252dc3105fc "qapi: Generate separate .h, .c for each module". Escaped testing because tests/qapi-schema-test.json doesn't cover sub-modules in other directories, only tests/qapi-schema/include-relpath.json does, and we generate and compile C code only for the former, not the latter. Fold the latter into the former. This would have caught the mistakes fixed in this commit.

---

+++ b/tests/.gitignore

tests/.gitignore +2 -0
@@ -11,9 +11,17 @@ test-*
 !test-*.c
 !docker/test-*
 test-qapi-commands.[ch]
+include/test-qapi-commands-sub-module.[ch]
+test-qapi-commands-sub-sub-module.[ch]
 test-qapi-events.[ch]

tests/.gitignore test-* · line 26

Quoted 10 lines
tests/.gitignore +6 -0
+include/test-qapi-events-sub-module.[ch]
+test-qapi-events-sub-sub-module.[ch]
 test-qapi-types.[ch]
+include/test-qapi-types-sub-module.[ch]
+test-qapi-types-sub-sub-module.[ch]
 test-qapi-visit.[ch]
+include/test-qapi-visit-sub-module.[ch]
+test-qapi-visit-sub-sub-module.[ch]
 test-qapi-introspect.[ch]
 *-test

Messes with sorting, and makes me wonder if any globs could compress it further, but I can live with it (as I don't have any concrete suggestions on how better to write it).

Quoted 9 lines
tests/Makefile.include +3 -2
@@ -505,8 +504,18 @@ qapi-schema += unknown-expr-key.json
 
 check-qapi-schema-y := $(addprefix tests/qapi-schema/, $(qapi-schema))
 
-GENERATED_FILES += tests/test-qapi-types.h tests/test-qapi-visit.h \
-	tests/test-qapi-commands.h tests/test-qapi-events.h \
+GENERATED_FILES += tests/test-qapi-types.h \
+	tests/include/test-qapi-types-sub-module.h \
+	tests/test-qapi-types-sub-sub-module.h \

tests/Makefile.include qapi-schema += unknown-expr-key.json · line 519

Quoted 10 lines
tests/Makefile.include +9 -0
+	tests/test-qapi-visit.h \
+	tests/include/test-qapi-visit-sub-module.h \
+	tests/test-qapi-visit-sub-sub-module.h \
+	tests/test-qapi-commands.h \
+	tests/include/test-qapi-commands-sub-module.h \
+	tests/test-qapi-commands-sub-sub-module.h \
+	tests/test-qapi-events.h \
+	tests/include/test-qapi-events-sub-module.h \
+	tests/test-qapi-events-sub-sub-module.h \
 	tests/test-qapi-introspect.h

Worth declaring a good portion of this list in a separate variable, then appending that variable to GENERATED FILES as well as...

Quoted 14 lines
tests/Makefile.include +8 -0
@@ -609,12 +623,32 @@ tests/test-replication$(EXESUF): tests/test-replication.o $(test-util-obj-y) \
 	$(test-block-obj-y)
 
 tests/test-qapi-types.c tests/test-qapi-types.h \
+tests/include/test-qapi-types-sub-module.c \
+tests/include/test-qapi-types-sub-module.h \
+tests/test-qapi-types-sub-sub-module.c \
+tests/test-qapi-types-sub-sub-module.h \
 tests/test-qapi-visit.c tests/test-qapi-visit.h \
+tests/include/test-qapi-visit-sub-module.c \
+tests/include/test-qapi-visit-sub-module.h \
+tests/test-qapi-visit-sub-sub-module.c \
+tests/test-qapi-visit-sub-sub-module.h \
 tests/test-qapi-commands.h tests/test-qapi-commands.c \

tests/Makefile.include tests/test-replication$(EXESUF): tests/test-replication.o $(test-util-obj-y) \ · line 645

Quoted 10 lines
tests/Makefile.include +8 -0
+tests/include/test-qapi-commands-sub-module.h \
+tests/include/test-qapi-commands-sub-module.c \
+tests/test-qapi-commands-sub-sub-module.h \
+tests/test-qapi-commands-sub-sub-module.c \
 tests/test-qapi-events.c tests/test-qapi-events.h \
+tests/include/test-qapi-events-sub-module.c \
+tests/include/test-qapi-events-sub-module.h \
+tests/test-qapi-events-sub-sub-module.c \
+tests/test-qapi-events-sub-sub-module.h \
 tests/test-qapi-introspect.c tests/test-qapi-introspect.h: \

...reusing it here for less duplication?

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#10 lore raw

Re: [Qemu-devel] [PATCH 4/7] qapi: Fix code generation for sub-modules in other directories

Quoted 34 lines · Eric Blake
Quoted 33 lines · Markus Armbruster

The #include directives to pull in sub-modules use file names relative to the main module. Works only when all modules are in the same directory, or the main module's output directory is in the compiler's include path. Use relative file names instead.

The dummy variable we generate to avoid empty .o files has an invalid name for sub-modules in other directories. Fix that.

Both messed up in commit 252dc3105fc "qapi: Generate separate .h, .c for each module". Escaped testing because tests/qapi-schema-test.json doesn't cover sub-modules in other directories, only tests/qapi-schema/include-relpath.json does, and we generate and compile C code only for the former, not the latter. Fold the latter into the former. This would have caught the mistakes fixed in this commit.

---

+++ b/tests/.gitignore

tests/.gitignore +6 -0
@@ -11,9 +11,17 @@ test-*
 !test-*.c
 !docker/test-*
 test-qapi-commands.[ch]
+include/test-qapi-commands-sub-module.[ch]
+test-qapi-commands-sub-sub-module.[ch]
 test-qapi-events.[ch]
+include/test-qapi-events-sub-module.[ch]
+test-qapi-events-sub-sub-module.[ch]
 test-qapi-types.[ch]
+include/test-qapi-types-sub-module.[ch]
+test-qapi-types-sub-sub-module.[ch]
 test-qapi-visit.[ch]
Quoted 8 lines

tests/.gitignore test-* · line 26

Quoted 4 lines
tests/.gitignore +2 -0
+include/test-qapi-visit-sub-module.[ch]
+test-qapi-visit-sub-sub-module.[ch]
 test-qapi-introspect.[ch]
 *-test

Messes with sorting, and makes me wonder if any globs could compress it further, but I can live with it (as I don't have any concrete suggestions on how better to write it).

I do: support only out-of-tree builds ;-P

Quoted 42 lines
Quoted 9 lines
tests/Makefile.include +3 -2
@@ -505,8 +504,18 @@ qapi-schema += unknown-expr-key.json
 
 check-qapi-schema-y := $(addprefix tests/qapi-schema/, $(qapi-schema))
 
-GENERATED_FILES += tests/test-qapi-types.h tests/test-qapi-visit.h \
-	tests/test-qapi-commands.h tests/test-qapi-events.h \
+GENERATED_FILES += tests/test-qapi-types.h \
+	tests/include/test-qapi-types-sub-module.h \
+	tests/test-qapi-types-sub-sub-module.h \

tests/Makefile.include qapi-schema += unknown-expr-key.json · line 519

Quoted 10 lines
tests/Makefile.include +9 -0
+	tests/test-qapi-visit.h \
+	tests/include/test-qapi-visit-sub-module.h \
+	tests/test-qapi-visit-sub-sub-module.h \
+	tests/test-qapi-commands.h \
+	tests/include/test-qapi-commands-sub-module.h \
+	tests/test-qapi-commands-sub-sub-module.h \
+	tests/test-qapi-events.h \
+	tests/include/test-qapi-events-sub-module.h \
+	tests/test-qapi-events-sub-sub-module.h \
 	tests/test-qapi-introspect.h

Worth declaring a good portion of this list in a separate variable, then appending that variable to GENERATED FILES as well as...

Quoted 18 lines
tests/Makefile.include +12 -0
@@ -609,12 +623,32 @@ tests/test-replication$(EXESUF): tests/test-replication.o $(test-util-obj-y) \
 	$(test-block-obj-y)
 
 tests/test-qapi-types.c tests/test-qapi-types.h \
+tests/include/test-qapi-types-sub-module.c \
+tests/include/test-qapi-types-sub-module.h \
+tests/test-qapi-types-sub-sub-module.c \
+tests/test-qapi-types-sub-sub-module.h \
 tests/test-qapi-visit.c tests/test-qapi-visit.h \
+tests/include/test-qapi-visit-sub-module.c \
+tests/include/test-qapi-visit-sub-module.h \
+tests/test-qapi-visit-sub-sub-module.c \
+tests/test-qapi-visit-sub-sub-module.h \
 tests/test-qapi-commands.h tests/test-qapi-commands.c \
+tests/include/test-qapi-commands-sub-module.h \
+tests/include/test-qapi-commands-sub-module.c \
+tests/test-qapi-commands-sub-sub-module.h \
+tests/test-qapi-commands-sub-sub-module.c \
Quoted 8 lines

tests/Makefile.include tests/test-replication$(EXESUF): tests/test-replication.o $(test-util-obj-y) \ · line 645

Quoted 6 lines
tests/Makefile.include +4 -0
 tests/test-qapi-events.c tests/test-qapi-events.h \
+tests/include/test-qapi-events-sub-module.c \
+tests/include/test-qapi-events-sub-module.h \
+tests/test-qapi-events-sub-sub-module.c \
+tests/test-qapi-events-sub-sub-module.h \
 tests/test-qapi-introspect.c tests/test-qapi-introspect.h: \

...reusing it here for less duplication?

We compute stuff from ${QAPI_MODULES} for qapi/qapi-schema.json. Reduces verbosity somewhat, for a price in complexity. Verbosity increases with number of modules, complexity stays flat. qapi/qapi-schema.json has 20 sub-modules, tests/qapi-schema/qapi-schema-test.json has two. I'd prefer to stick with stupid for now.

Quoted 1 line

Thanks!

#11 lore raw

PATCH 5/7 tests: Rename UserDefNativeListUnion to UserDefListUnion

In reply to Markus Armbruster

The lists in UserDefNativeListUnion aren't "native", they're lists of built-in types. The next commit will add a list of a user-defined type. Drop "Native", and adjust the tests using the type.

---
 tests/qapi-schema/qapi-schema-test.json |   6 +-
 tests/qapi-schema/qapi-schema-test.out  |   8 +-
 tests/test-clone-visitor.c              |  16 +-
 tests/test-qmp-cmds.c                   |   2 +-
 tests/test-qobject-input-visitor.c      | 196 ++++++++++++------------
 tests/test-qobject-output-visitor.c     | 190 +++++++++++------------
 6 files changed, 209 insertions(+), 209 deletions(-)
tests/qapi-schema/qapi-schema-test.json +3 -3
index 1f130a0491..908f718027 100644
@@ -113,8 +113,8 @@
 # for testing use of 'str' within alternates
 { 'alternate': 'AltStrObj', 'data': { 's': 'str', 'o': 'TestStruct' } }
 
-# for testing native lists
-{ 'union': 'UserDefNativeListUnion',
+# for testing lists
+{ 'union': 'UserDefListUnion',
   'data': { 'integer': ['int'],
             's8': ['int8'],
             's16': ['int16'],
@@ -147,7 +147,7 @@
   'returns': 'int' }
 { 'command': 'guest-sync', 'data': { 'arg': 'any' }, 'returns': 'any' }
 { 'command': 'boxed-struct', 'boxed': true, 'data': 'UserDefZero' }
-{ 'command': 'boxed-union', 'data': 'UserDefNativeListUnion', 'boxed': true }
+{ 'command': 'boxed-union', 'data': 'UserDefListUnion', 'boxed': true }
 
 # Smoke test on out-of-band and allow-preconfig-test
 { 'command': 'test-flags-command', 'allow-oob': true, 'allow-preconfig': true }
tests/qapi-schema/qapi-schema-test.out +4 -4
index baba4c8384..cecf270ec3 100644
@@ -144,7 +144,7 @@ object q_obj_sizeList-wrapper
     member data: sizeList optional=False
 object q_obj_anyList-wrapper
     member data: anyList optional=False
-enum UserDefNativeListUnionKind
+enum UserDefListUnionKind
     member integer
     member s8
     member s16
@@ -159,8 +159,8 @@ enum UserDefNativeListUnionKind
     member string
     member sizes
     member any
-object UserDefNativeListUnion
-    member type: UserDefNativeListUnionKind optional=False
+object UserDefListUnion
+    member type: UserDefListUnionKind optional=False
     tag type
     case integer: q_obj_intList-wrapper
     case s8: q_obj_int8List-wrapper
@@ -209,7 +209,7 @@ command guest-sync q_obj_guest-sync-arg -> any
    gen=True success_response=True boxed=False oob=False preconfig=False
 command boxed-struct UserDefZero -> None
    gen=True success_response=True boxed=True oob=False preconfig=False
-command boxed-union UserDefNativeListUnion -> None
+command boxed-union UserDefListUnion -> None
    gen=True success_response=True boxed=True oob=False preconfig=False
 command test-flags-command None -> None
    gen=True success_response=True boxed=False oob=True preconfig=True
tests/test-clone-visitor.c +8 -8
index 9aeaf86a07..5e1e8b2f5e 100644
@@ -63,7 +63,7 @@ static void test_clone_alternate(void)
     qapi_free_AltEnumBool(s_dst);
 }
 
-static void test_clone_native_list(void)
+static void test_clone_list_union(void)
 {
     uint8List *src, *dst;
     uint8List *tmp = NULL;
@@ -102,18 +102,18 @@ static void test_clone_empty(void)
 
 static void test_clone_complex1(void)
 {
-    UserDefNativeListUnion *src, *dst;
+    UserDefListUnion *src, *dst;
 
-    src = g_new0(UserDefNativeListUnion, 1);
-    src->type = USER_DEF_NATIVE_LIST_UNION_KIND_STRING;
+    src = g_new0(UserDefListUnion, 1);
+    src->type = USER_DEF_LIST_UNION_KIND_STRING;
 
-    dst = QAPI_CLONE(UserDefNativeListUnion, src);
+    dst = QAPI_CLONE(UserDefListUnion, src);
     g_assert(dst);
     g_assert_cmpint(dst->type, ==, src->type);
     g_assert(!dst->u.string.data);
 
-    qapi_free_UserDefNativeListUnion(src);
-    qapi_free_UserDefNativeListUnion(dst);
+    qapi_free_UserDefListUnion(src);
+    qapi_free_UserDefListUnion(dst);
 }
 
 static void test_clone_complex2(void)
@@ -192,7 +192,7 @@ int main(int argc, char **argv)
 
     g_test_add_func("/visitor/clone/struct", test_clone_struct);
     g_test_add_func("/visitor/clone/alternate", test_clone_alternate);
-    g_test_add_func("/visitor/clone/native_list", test_clone_native_list);
+    g_test_add_func("/visitor/clone/list_union", test_clone_list_union);
     g_test_add_func("/visitor/clone/empty", test_clone_empty);
     g_test_add_func("/visitor/clone/complex1", test_clone_complex1);
     g_test_add_func("/visitor/clone/complex2", test_clone_complex2);
tests/test-qmp-cmds.c +1 -1
index 481cb069ca..630b1b9bac 100644
@@ -87,7 +87,7 @@ void qmp_boxed_struct(UserDefZero *arg, Error **errp)
 {
 }
 
-void qmp_boxed_union(UserDefNativeListUnion *arg, Error **errp)
+void qmp_boxed_union(UserDefListUnion *arg, Error **errp)
 {
 }
 
tests/test-qobject-input-visitor.c +98 -98

374 lines not on this page · raw message

tests/test-qobject-output-visitor.c +95 -95

373 lines not on this page · raw message

Signature · 1 line
2.17.2
Reviewed-by on v1 5/7
#12 lore raw

Re: [Qemu-devel] [PATCH 5/7] tests: Rename UserDefNativeListUnion to UserDefListUnion

Quoted 6 lines · Markus Armbruster

The lists in UserDefNativeListUnion aren't "native", they're lists of built-in types. The next commit will add a list of a user-defined type. Drop "Native", and adjust the tests using the type.

---

Looks mechanical, and makes sense.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#13 lore raw

PATCH 6/7 tests/qapi-schema: Cover forward reference to sub-module

In reply to Markus Armbruster

The forward reference from the main module to the sub-module works fine, except for an issue visible in qapi-schema-test.out: the array type wrapped around the forward reference ends up in the main module, not the sub-module. The next commit will explain why that's bad, and fix it.

---
 tests/qapi-schema/qapi-schema-test.json | 3 ++-
 tests/qapi-schema/qapi-schema-test.out  | 5 +++++
 2 files changed, 7 insertions(+), 1 deletion(-)
tests/qapi-schema/qapi-schema-test.json +2 -1
index 908f718027..0952c68734 100644
@@ -128,7 +128,8 @@
             'boolean': ['bool'],
             'string': ['str'],
             'sizes': ['size'],
-            'any': ['any'] } }
+            'any': ['any'],
+            'user': ['Status'] } } # intentional forward ref. to sub-module
 
 # for testing sub-modules
 { 'include': 'include/sub-module.json' }
tests/qapi-schema/qapi-schema-test.out +5 -0
index cecf270ec3..38c1de70d8 100644
@@ -144,6 +144,9 @@ object q_obj_sizeList-wrapper
     member data: sizeList optional=False
 object q_obj_anyList-wrapper
     member data: anyList optional=False
+array StatusList Status
+object q_obj_StatusList-wrapper
+    member data: StatusList optional=False
 enum UserDefListUnionKind
     member integer
     member s8
@@ -159,6 +162,7 @@ enum UserDefListUnionKind
     member string
     member sizes
     member any
+    member user
 object UserDefListUnion
     member type: UserDefListUnionKind optional=False
     tag type
@@ -176,6 +180,7 @@ object UserDefListUnion
     case string: q_obj_strList-wrapper
     case sizes: q_obj_sizeList-wrapper
     case any: q_obj_anyList-wrapper
+    case user: q_obj_StatusList-wrapper
 include include/sub-module.json
 module include/sub-module.json
 include sub-sub-module.json
Signature · 1 line
2.17.2
Reviewed-by on v1 6/7
#14 lore raw

Re: [Qemu-devel] [PATCH 6/7] tests/qapi-schema: Cover forward reference to sub-module

Quoted 6 lines · Markus Armbruster

The forward reference from the main module to the sub-module works fine, except for an issue visible in qapi-schema-test.out: the array type wrapped around the forward reference ends up in the main module, not the sub-module. The next commit will explain why that's bad, and fix it.

Quoted 6 lines
---
 tests/qapi-schema/qapi-schema-test.json | 3 ++-
 tests/qapi-schema/qapi-schema-test.out  | 5 +++++
 2 files changed, 7 insertions(+), 1 deletion(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#15 lore raw

PATCH 7/7 qapi: Fix array first used in a different module

In reply to Markus Armbruster

We generally put implicitly defined types in whatever module triggered their definition. This is wrong for array types, as the included test case demonstrates. Let's have a closer look at it.

Type 'Status' is defined sub-sub-module.json. Array type ['Status'] occurs in main module qapi-schema-test.json and in include/sub-module.json. The main module's use is first, so the array type gets put into the main module.

The generated C headers define StatusList in qapi-types.h. But include/qapi-types-sub-module.h uses it without including qapi-types.h. Oops.

To fix that, put the array type into its element type's module.

Now StatusList gets generated into qapi-types-sub-module.h, which all its users include.

---
 scripts/qapi/common.py                    | 9 +++++----
 tests/qapi-schema/include/sub-module.json | 2 ++
 tests/qapi-schema/qapi-schema-test.out    | 5 +++++
 3 files changed, 12 insertions(+), 4 deletions(-)
scripts/qapi/common.py +5 -4
index f51948364c..f07869ec73 100644
@@ -1089,6 +1089,9 @@ class QAPISchemaEntity(object):
             self.ifcond = typ.ifcond
         else:
             self.ifcond = listify_cond(self._ifcond)
+        if self.info:
+            self.module = os.path.relpath(self.info['file'],
+                                          os.path.dirname(schema.fname))
 
     def is_implicit(self):
         return not self.info
@@ -1262,6 +1265,7 @@ class QAPISchemaArrayType(QAPISchemaType):
         self.element_type = schema.lookup_type(self._element_type_name)
         assert self.element_type
         self.element_type.check(schema)
+        self.module = self.element_type.module
         self.ifcond = self.element_type.ifcond
 
     def is_implicit(self):
@@ -1603,7 +1607,7 @@ class QAPISchemaEvent(QAPISchemaEntity):
 
 class QAPISchema(object):
     def __init__(self, fname):
-        self._fname = fname
+        self.fname = fname
         if sys.version_info[0] >= 3:
             f = open(fname, 'r', encoding='utf-8')
         else:
@@ -1626,9 +1630,6 @@ class QAPISchema(object):
         self._entity_list.append(ent)
         if ent.name is not None:
             self._entity_dict[ent.name] = ent
-        if ent.info:
-            ent.module = os.path.relpath(ent.info['file'],
-                                         os.path.dirname(self._fname))
 
     def lookup_entity(self, name, typ=None):
         ent = self._entity_dict.get(name)
tests/qapi-schema/include/sub-module.json +2 -0
index f2bdbd3990..afdb267228 100644
@@ -3,3 +3,5 @@
 # Sub-module of ../qapi-schema-test.json
 
 { 'include': '../sub-sub-module.json' }
+
+{ 'struct': 'SecondArrayRef', 'data': { 's': ['Status'] } }
tests/qapi-schema/qapi-schema-test.out +5 -0
index 38c1de70d8..77fb1e1aa9 100644
@@ -144,7 +144,9 @@ object q_obj_sizeList-wrapper
     member data: sizeList optional=False
 object q_obj_anyList-wrapper
     member data: anyList optional=False
+module sub-sub-module.json
 array StatusList Status
+module qapi-schema-test.json
 object q_obj_StatusList-wrapper
     member data: StatusList optional=False
 enum UserDefListUnionKind
@@ -189,6 +191,9 @@ enum Status
     member good
     member bad
     member ugly
+module include/sub-module.json
+object SecondArrayRef
+    member s: StatusList optional=False
 module qapi-schema-test.json
 command user_def_cmd None -> None
    gen=True success_response=True boxed=False oob=False preconfig=False
Signature · 1 line
2.17.2
Reviewed-by on v1 7/7
#16 lore raw

Re: [Qemu-devel] [PATCH 7/7] qapi: Fix array first used in a different module

Quoted 13 lines · Markus Armbruster

We generally put implicitly defined types in whatever module triggered their definition. This is wrong for array types, as the included test case demonstrates. Let's have a closer look at it.

Type 'Status' is defined sub-sub-module.json. Array type ['Status'] occurs in main module qapi-schema-test.json and in include/sub-module.json. The main module's use is first, so the array type gets put into the main module.

The generated C headers define StatusList in qapi-types.h. But include/qapi-types-sub-module.h uses it without including qapi-types.h. Oops.

Quoted 7 lines

To fix that, put the array type into its element type's module.

Now StatusList gets generated into qapi-types-sub-module.h, which all its users include.

---

Yay! One less hack in .qapi files where we had to use dummy structs to work around awkward declaration locations.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org