Skip to contentSkip to conversation

Modularize generated QAPI code

RFC PATCH Superseded Markus Armbruster 5 participants 87 messages Feb 2, 2018 – Feb 8, 2018 Page 1 of 2

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Modularize generated QAPI code

Our qapi-schema.json is composed of modules connected by include directives, but the generated code is monolithic all the same: one qapi-types.h with all the types, one qapi-visit.h with all the visitors, and so forth. These monolithic headers get included all over the place. In my "build everyhing" tree, adding a QAPI type recompiles about 4500 out of 4800 objects.

Nobody would write such monolithic headers by hand. It stands to reason that one shouldn't generate them, either.

This series' basic idea is to split up generated headers to mirror the schema's modular structure: one header per module. That way, you can include just what you need.

The series is RFC for a number of reasons:

* The split is implemented only for qapi-types.h. That one should provide the biggest benefits, though.

* There's a bit of code duplication.

* I haven't re-read my patches, yet.

Even in this incomplete state, the compile-time improvements can be massive. Before this series, any QAPI schema change recompiles some 4500 out of 4800 objects in my "build everything" tree. Afterwards, adding a type to qapi/migration.json recompiles less than 400, adding a QMP event recompiles less than 200, and a documentation change no longer recompiles anything.

Related: Marc-André's 'unit' pragma proposal. That's a different way to split off parts of the generated code, motivated by the desire to use poisoned identifiers such as TARGET_I386. I noted in my review of v3 that I "can either accept it, or come up with a better solution." This is my attempt at a better solution. It's a bit more ambitious, and thus more useful (I hope). The pragma has one theoretical advantage, though: you can modularize the generated output in different ways than the input. The patches using don't do that, however.

Based-on: <[email protected]> [PATCH v3 00/19] Clean up includes to reduce compile time

Markus Armbruster (21):
qapi: Streamline boilerplate comment generation
qapi: Generate up-to-date copyright notice
qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc
qapi: Reduce use of global variables in generators some
qapi: Turn generators into modules
qapi-gen: New common driver for code and doc generators
qapi: Move parse_command_line() next to its only use
qapi: Touch generated files only when they change
qapi: Don't absolutize include file name in error messages qapi/common: Eliminate QAPISchema.exprs
qapi: Lift error reporting from QAPISchema.__init__() to callers
qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()
qapi: Record 'include' directives in parse tree
qapi: Generate in source order
qapi: Record 'include' directives in intermediate representation qapi/types qapi/visit: Make visitors use QAPIGen more qapi/types qapi/visit: Generate built-in stuff into separate files qapi/common: Fix guardname() for funny filenames qapi/types: Generate separate .h, .c for each module Include less of qapi-types.h
qapi: Empty out qapi-schema.json

 Makefile                                           |  131 +-
 Makefile.objs                                      |   20 +-
 crypto/cipherpriv.h                                |    2 +-
 hw/block/block.c                                   |    1 +
 hw/block/hd-geometry.c                             |    1 +
 hw/net/rocker/rocker_fp.c                          |    2 +-
 include/block/block.h                              |    2 +-
 include/block/dirty-bitmap.h                       |    2 +-
 include/chardev/char.h                             |    1 +
 include/crypto/cipher.h                            |    2 +-
 include/crypto/hash.h                              |    2 +-
 include/crypto/hmac.h                              |    2 +-
 include/crypto/secret.h                            |    1 +
 include/crypto/tlscreds.h                          |    1 +
 include/hw/block/block.h                           |    2 +-
 include/hw/block/fdc.h                             |    2 +-
 include/hw/ppc/spapr_drc.h                         |    1 +
 include/hw/qdev-properties.h                       |    2 +
 include/io/dns-resolver.h                          |    1 +
 include/migration/colo.h                           |    2 +-
 include/migration/failover.h                       |    2 +-
 include/migration/global_state.h                   |    1 +
 include/monitor/monitor.h                          |    1 +
 include/net/filter.h                               |    1 +
 include/net/net.h                                  |    2 +-
 include/qapi/error.h                               |    2 +-
 include/qapi/qmp/qobject.h                         |    2 +-
 include/qapi/visitor.h                             |    2 +-
 include/qemu/sockets.h                             |    2 +-
 include/qemu/throttle.h                            |    2 +-
 include/qom/cpu.h                                  |    1 +
 include/qom/object.h                               |    2 +-
 include/sysemu/arch_init.h                         |    2 +-
 include/sysemu/balloon.h                           |    2 +-
 include/sysemu/dump.h                              |    2 +
 include/sysemu/hostmem.h                           |    1 +
 include/sysemu/replay.h                            |    2 +
 include/sysemu/sysemu.h                            |    1 +
 include/sysemu/tpm.h                               |    1 +
 include/sysemu/watchdog.h                          |    2 +-
 include/ui/input.h                                 |    2 +-
 migration/migration.h                              |    1 +
 migration/ram.h                                    |    2 +-
 net/tap_int.h                                      |    2 +-
 qapi-schema.json                                   | 3098 +-------------------
 qapi/misc.json                                     | 3090 +++++++++++++++++++
 qapi/run-state.json                                |   10 +
 replication.h                                      |    1 +
 scripts/qapi-gen.py                                |   95 +
 scripts/qapi/__init__.py                           |    0
 scripts/{qapi-commands.py => qapi/commands.py}     |   98 +-
 scripts/{qapi.py => qapi/common.py}                |  290 +-
 scripts/{qapi2texi.py => qapi/doc.py}              |   28 +-
 scripts/{qapi-event.py => qapi/events.py}          |  101 +-
 scripts/{qapi-introspect.py => qapi/introspect.py} |   87 +-
 scripts/{qapi-types.py => qapi/types.py}           |  218 +-
 scripts/{qapi-visit.py => qapi/visit.py}           |  208 +-
 tests/Makefile.include                             |   56 +-
 tests/qapi-schema/comments.out                     |    3 +-
 tests/qapi-schema/doc-bad-section.out              |    5 +-
 tests/qapi-schema/doc-good.out                     |   33 +-
 tests/qapi-schema/doc-good.texi                    |    3 +-
 tests/qapi-schema/empty.out                        |    2 +-
 tests/qapi-schema/event-case.out                   |    3 +-
 tests/qapi-schema/ident-with-escape.out            |    7 +-
 tests/qapi-schema/include-no-file.err              |    2 +-
 tests/qapi-schema/include-relpath.out              |    7 +-
 tests/qapi-schema/include-repetition.out           |   12 +-
 tests/qapi-schema/include-simple.out               |    5 +-
 tests/qapi-schema/indented-expr.out                |    3 +-
 tests/qapi-schema/qapi-schema-test.out             |  321 +-
 tests/qapi-schema/test-qapi.py                     |   17 +-
 ui/vnc.h                                           |    1 +
 73 files changed, 4040 insertions(+), 3984 deletions(-)
 create mode 100644 qapi/misc.json
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (78%)
 rename scripts/{qapi.py => qapi/common.py} (93%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (93%)
 mode change 100755 => 100644
 rename scripts/{qapi-event.py => qapi/events.py} (69%)
 rename scripts/{qapi-introspect.py => qapi/introspect.py} (78%)
 rename scripts/{qapi-types.py => qapi/types.py} (54%)
 rename scripts/{qapi-visit.py => qapi/visit.py} (66%)
Signature · 1 line
2.13.6
#1 lore raw
Quoted 14 lines · Markus Armbruster

Our qapi-schema.json is composed of modules connected by include directives, but the generated code is monolithic all the same: one qapi-types.h with all the types, one qapi-visit.h with all the visitors, and so forth. These monolithic headers get included all over the place. In my "build everyhing" tree, adding a QAPI type recompiles about 4500 out of 4800 objects.

Nobody would write such monolithic headers by hand. It stands to reason that one shouldn't generate them, either.

This series' basic idea is to split up generated headers to mirror the schema's modular structure: one header per module. That way, you can include just what you need.

Quoted 8 lines

The series is RFC for a number of reasons:

* The split is implemented only for qapi-types.h. That one should provide the biggest benefits, though.

* There's a bit of code duplication.

* I haven't re-read my patches, yet.

And of course

* Needs a doc update

[...]

#2 lore raw

RFC PATCH 1/21 qapi: Streamline boilerplate comment generation

In reply to Markus Armbruster

Every generator has separate boilerplate for .h and .c, and their differences are boring. All of them repeat the license note.

Reduce the repetition as follows. Move common text like the license note to common open_output(), next to the existintg common text there. For each generator, replace the two separate descriptions by a single one.

While there, emit an "automatically generated" note into generated documentation, too.

---
 scripts/qapi-commands.py        | 26 +++-----------------------
 scripts/qapi-event.py           | 26 +++-----------------------
 scripts/qapi-introspect.py      | 21 ++-------------------
 scripts/qapi-types.py           | 26 +++-----------------------
 scripts/qapi-visit.py           | 26 +++-----------------------
 scripts/qapi.py                 | 31 ++++++++++++++++++-------------
 scripts/qapi2texi.py            |  3 ++-
 tests/qapi-schema/doc-good.texi |  3 ++-
 8 files changed, 36 insertions(+), 126 deletions(-)
scripts/qapi-commands.py +3 -23
index 26c56c5062..25ac52503a 100644
@@ -255,38 +255,18 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
 
-c_comment = '''
-/*
- * schema-defined QMP->QAPI command dispatch
+blurb = '''
+ * Schema-defined QAPI/QMP commands
  *
  * Copyright IBM, Corp. 2011
  *
  * Authors:
  *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI function prototypes
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-marshal.c', 'qmp-commands.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 
scripts/qapi-event.py +3 -23
index 9d7134658d..31faedc689 100644
@@ -171,38 +171,18 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 
 (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
 
-c_comment = '''
-/*
- * schema-defined QAPI event functions
+blurb = '''
+ * Schema-defined QAPI/QMP events
  *
  * Copyright (c) 2014 Wenchao Xia
  *
  * Authors:
  *  Wenchao Xia   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI event functions
- *
- * Copyright (c) 2014 Wenchao Xia
- *
- * Authors:
- *  Wenchao Xia  <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-event.c', 'qapi-event.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-introspect.py +2 -19
index 032bcea491..83da2bdb94 100644
@@ -176,32 +176,15 @@ for o, a in opts:
     if o in ('-u', '--unmask-non-abi-names'):
         opt_unmask = True
 
-c_comment = '''
-/*
+blurb = '''
  * QAPI/QMP schema introspection
  *
  * Copyright (C) 2015 Red Hat, Inc.
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * QAPI/QMP schema introspection
- *
- * Copyright (C) 2015 Red Hat, Inc.
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-introspect.c', 'qmp-introspect.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-types.py +3 -23
index 7e3051dbb9..86afc57f92 100644
@@ -250,39 +250,19 @@ for o, a in opts:
     if o in ('-b', '--builtins'):
         do_builtins = True
 
-c_comment = '''
-/*
- * deallocation functions for schema-defined QAPI types
+blurb = '''
+ * Schema-defined QAPI types
  *
  * Copyright IBM, Corp. 2011
  *
  * Authors:
  *  Anthony Liguori   <[email protected]>
  *  Michael Roth      <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI types
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-types.c', 'qapi-types.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-visit.py +3 -23
index 7e1cfc13f0..9f7127e548 100644
@@ -334,38 +334,18 @@ for o, a in opts:
     if o in ('-b', '--builtins'):
         do_builtins = True
 
-c_comment = '''
-/*
- * schema-defined QAPI visitor functions
+blurb = '''
+ * Schema-defined QAPI visitors
  *
  * Copyright IBM, Corp. 2011
  *
  * Authors:
  *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI visitor functions
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-visit.c', 'qapi-visit.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi.py +18 -13
index 43a54bf40f..2e4b2bc9eb 100644
@@ -1975,11 +1975,21 @@ def parse_command_line(extra_options='', extra_long_options=[]):
 #
 
 
-def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
-                c_comment, h_comment):
+def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb):
     guard = guardname(prefix + h_file)
     c_file = output_dir + prefix + c_file
     h_file = output_dir + prefix + h_file
+    comment = mcgen('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+
+/*
+%(blurb)s
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ */
+
+''',
+                    blurb=blurb.strip('\n'))
 
     if output_dir:
         try:
@@ -1998,27 +2008,22 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
     fdef = maybe_open(do_c, c_file, 'w')
     fdecl = maybe_open(do_h, h_file, 'w')
 
-    fdef.write(mcgen('''
-/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
-%(comment)s
-''',
-                     comment=c_comment))
-
+    fdef.write(comment)
+    fdecl.write(comment)
     fdecl.write(mcgen('''
-/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
-%(comment)s
 #ifndef %(guard)s
 #define %(guard)s
 
 ''',
-                      comment=h_comment, guard=guard))
+                      guard=guard))
 
     return (fdef, fdecl)
 
 
 def close_output(fdef, fdecl):
-    fdecl.write('''
+    fdecl.write(mcgen('''
+
 #endif
-''')
+'''))
     fdecl.close()
     fdef.close()
scripts/qapi2texi.py +2 -1
index 92e2af2cd6..924b374cd3 100755
@@ -282,7 +282,8 @@ def main(argv):
         print >>sys.stderr, ("%s: need pragma 'doc-required' "
                              "to generate documentation" % argv[0])
         sys.exit(1)
-    print texi_schema(schema)
+    print '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n'
+    print texi_schema(schema),
 
 
 if __name__ == '__main__':
tests/qapi-schema/doc-good.texi +2 -1
index 1778312581..0aed2300a5 100644
@@ -1,3 +1,5 @@
+@c AUTOMATICALLY GENERATED, DO NOT MODIFY
+
 @section Section
 
 @subsection Subsection
@@ -231,4 +233,3 @@ If you're bored enough to read this, go see a video of boxed cats
 
 @end deftypefn
 
-
Signature · 1 line
2.13.6
Reviewed-by on v1 1/21
#3 lore raw

Re: [Qemu-devel] [PATCH RFC 01/21] qapi: Streamline boilerplate comment generation

Quoted 5 lines · Markus Armbruster

Every generator has separate boilerplate for .h and .c, and their differences are boring. All of them repeat the license note.

Reduce the repetition as follows. Move common text like the license note to common open_output(), next to the existintg common text there.

s/existintg/existing/

Quoted 11 lines

For each generator, replace the two separate descriptions by a single one.

While there, emit an "automatically generated" note into generated documentation, too.

---

+++ b/scripts/qapi2texi.py

scripts/qapi2texi.py def main(argv): · line 286

Quoted 6 lines
scripts/qapi2texi.py +2 -1
@@ -282,7 +282,8 @@ def main(argv):
         print >>sys.stderr, ("%s: need pragma 'doc-required' "
                              "to generate documentation" % argv[0])
         sys.exit(1)
-    print texi_schema(schema)
+    print '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n'
+    print texi_schema(schema),

Spurious addition of trailing comma. Otherwise,

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#4 lore raw

Re: [Qemu-devel] [PATCH RFC 01/21] qapi: Streamline boilerplate comment generation

Quoted 8 lines · Eric Blake
Quoted 5 lines · Markus Armbruster

Every generator has separate boilerplate for .h and .c, and their differences are boring. All of them repeat the license note.

Reduce the repetition as follows. Move common text like the license note to common open_output(), next to the existintg common text there.

s/existintg/existing/

Fixing...

Quoted 9 lines
Quoted 8 lines

For each generator, replace the two separate descriptions by a single one.

While there, emit an "automatically generated" note into generated documentation, too.

---

Quoted 10 lines

scripts/qapi2texi.py def main(argv): · line 286

Quoted 8 lines

+++ b/scripts/qapi2texi.py

scripts/qapi2texi.py +2 -1
@@ -282,7 +282,8 @@ def main(argv):
         print >>sys.stderr, ("%s: need pragma 'doc-required' "
                              "to generate documentation" % argv[0])
         sys.exit(1)
-    print texi_schema(schema)
+    print '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n'
+    print texi_schema(schema),

Spurious addition of trailing comma. Otherwise,

This is Python's (rather unobvious) way to print without adding a newline.

Quoted 1 line

Thanks!

Reviewed-by Marc-André Lureau on v1 1/21
#5 lore raw

Re: [Qemu-devel] [PATCH RFC 01/21] qapi: Streamline boilerplate comment generation

In reply to Markus Armbruster
Quoted 8 lines · Markus Armbruster

Every generator has separate boilerplate for .h and .c, and their differences are boring. All of them repeat the license note.

Reduce the repetition as follows. Move common text like the license note to common open_output(), next to the existintg common text there. For each generator, replace the two separate descriptions by a single one.

Quoted 4 lines

While there, emit an "automatically generated" note into generated documentation, too.

Looks good,

Quoted 329 lines
---
 scripts/qapi-commands.py        | 26 +++-----------------------
 scripts/qapi-event.py           | 26 +++-----------------------
 scripts/qapi-introspect.py      | 21 ++-------------------
 scripts/qapi-types.py           | 26 +++-----------------------
 scripts/qapi-visit.py           | 26 +++-----------------------
 scripts/qapi.py                 | 31 ++++++++++++++++++-------------
 scripts/qapi2texi.py            |  3 ++-
 tests/qapi-schema/doc-good.texi |  3 ++-
 8 files changed, 36 insertions(+), 126 deletions(-)
scripts/qapi-commands.py +3 -23
index 26c56c5062..25ac52503a 100644
@@ -255,38 +255,18 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
 
-c_comment = '''
-/*
- * schema-defined QMP->QAPI command dispatch
+blurb = '''
+ * Schema-defined QAPI/QMP commands
  *
  * Copyright IBM, Corp. 2011
  *
  * Authors:
  *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI function prototypes
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-marshal.c', 'qmp-commands.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 
scripts/qapi-event.py +3 -23
index 9d7134658d..31faedc689 100644
@@ -171,38 +171,18 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 
 (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
 
-c_comment = '''
-/*
- * schema-defined QAPI event functions
+blurb = '''
+ * Schema-defined QAPI/QMP events
  *
  * Copyright (c) 2014 Wenchao Xia
  *
  * Authors:
  *  Wenchao Xia   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI event functions
- *
- * Copyright (c) 2014 Wenchao Xia
- *
- * Authors:
- *  Wenchao Xia  <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-event.c', 'qapi-event.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-introspect.py +2 -19
index 032bcea491..83da2bdb94 100644
@@ -176,32 +176,15 @@ for o, a in opts:
     if o in ('-u', '--unmask-non-abi-names'):
         opt_unmask = True
 
-c_comment = '''
-/*
+blurb = '''
  * QAPI/QMP schema introspection
  *
  * Copyright (C) 2015 Red Hat, Inc.
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * QAPI/QMP schema introspection
- *
- * Copyright (C) 2015 Red Hat, Inc.
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-introspect.c', 'qmp-introspect.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-types.py +3 -23
index 7e3051dbb9..86afc57f92 100644
@@ -250,39 +250,19 @@ for o, a in opts:
     if o in ('-b', '--builtins'):
         do_builtins = True
 
-c_comment = '''
-/*
- * deallocation functions for schema-defined QAPI types
+blurb = '''
+ * Schema-defined QAPI types
  *
  * Copyright IBM, Corp. 2011
  *
  * Authors:
  *  Anthony Liguori   <[email protected]>
  *  Michael Roth      <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI types
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-types.c', 'qapi-types.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-visit.py +3 -23
index 7e1cfc13f0..9f7127e548 100644
@@ -334,38 +334,18 @@ for o, a in opts:
     if o in ('-b', '--builtins'):
         do_builtins = True
 
-c_comment = '''
-/*
- * schema-defined QAPI visitor functions
+blurb = '''
+ * Schema-defined QAPI visitors
  *
  * Copyright IBM, Corp. 2011
  *
  * Authors:
  *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI visitor functions
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-visit.c', 'qapi-visit.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi.py +18 -13
index 43a54bf40f..2e4b2bc9eb 100644
@@ -1975,11 +1975,21 @@ def parse_command_line(extra_options='', extra_long_options=[]):
 #
 
 
-def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
-                c_comment, h_comment):
+def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb):
     guard = guardname(prefix + h_file)
     c_file = output_dir + prefix + c_file
     h_file = output_dir + prefix + h_file
+    comment = mcgen('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+
+/*
+%(blurb)s
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ */
+
+''',
+                    blurb=blurb.strip('\n'))
 
     if output_dir:
         try:
@@ -1998,27 +2008,22 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
     fdef = maybe_open(do_c, c_file, 'w')
     fdecl = maybe_open(do_h, h_file, 'w')
 
-    fdef.write(mcgen('''
-/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
-%(comment)s
-''',
-                     comment=c_comment))
-
+    fdef.write(comment)
+    fdecl.write(comment)
     fdecl.write(mcgen('''
-/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
-%(comment)s
 #ifndef %(guard)s
 #define %(guard)s
 
 ''',
-                      comment=h_comment, guard=guard))
+                      guard=guard))
 
     return (fdef, fdecl)
 
 
 def close_output(fdef, fdecl):
-    fdecl.write('''
+    fdecl.write(mcgen('''
+
 #endif
-''')
+'''))
     fdecl.close()
     fdef.close()
scripts/qapi2texi.py +2 -1
index 92e2af2cd6..924b374cd3 100755
@@ -282,7 +282,8 @@ def main(argv):
         print >>sys.stderr, ("%s: need pragma 'doc-required' "
                              "to generate documentation" % argv[0])
         sys.exit(1)
-    print texi_schema(schema)
+    print '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n'
+    print texi_schema(schema),
 
 
 if __name__ == '__main__':
tests/qapi-schema/doc-good.texi +2 -2
index 1778312581..0aed2300a5 100644
@@ -1,3 +1,5 @@
+@c AUTOMATICALLY GENERATED, DO NOT MODIFY
+
 @section Section
 
 @subsection Subsection
@@ -231,4 +233,3 @@ If you're bored enough to read this, go see a video of boxed cats
 
 @end deftypefn
 
-
--

2.13.6

#6 lore raw

RFC PATCH 2/21 qapi: Generate up-to-date copyright notice

In reply to Markus Armbruster

Each generator carries a copyright notice for the generator itself, and another one for the files it generates. Only the former have been updated along the way, the latter have not, and are all out of date.

Fix by copying the generator's copyright notice to the generated files instead.

---
 scripts/qapi-commands.py   | 34 +++++++++++++++-------------------
 scripts/qapi-event.py      | 32 ++++++++++++++------------------
 scripts/qapi-introspect.py | 25 ++++++++++++-------------
 scripts/qapi-types.py      | 32 ++++++++++++++------------------
 scripts/qapi-visit.py      | 34 +++++++++++++++-------------------
 scripts/qapi.py            |  7 +++++--
 6 files changed, 75 insertions(+), 89 deletions(-)
scripts/qapi-commands.py +15 -19
index 25ac52503a..a861ac52e7 100644
@@ -1,16 +1,17 @@
-#
-# QAPI command marshaller generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (C) 2014-2016 Red Hat, Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Michael Roth    <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI command marshaller generator
+
+Copyright IBM, Corp. 2011
+Copyright (C) 2014-2018 Red Hat, Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -257,16 +258,11 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 
 blurb = '''
  * Schema-defined QAPI/QMP commands
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-marshal.c', 'qmp-commands.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 
scripts/qapi-event.py +14 -18
index 31faedc689..b1d611c5ea 100644
@@ -1,15 +1,16 @@
-#
-# QAPI event generator
-#
-# Copyright (c) 2014 Wenchao Xia
-# Copyright (c) 2015-2016 Red Hat Inc.
-#
-# Authors:
-#  Wenchao Xia <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI event generator
+
+Copyright (c) 2014 Wenchao Xia
+Copyright (c) 2015-2018 Red Hat Inc.
+
+Authors:
+ Wenchao Xia <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -173,16 +174,11 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 
 blurb = '''
  * Schema-defined QAPI/QMP events
- *
- * Copyright (c) 2014 Wenchao Xia
- *
- * Authors:
- *  Wenchao Xia   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-event.c', 'qapi-event.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-introspect.py +12 -13
index 83da2bdb94..bd9253a172 100644
@@ -1,13 +1,14 @@
-#
-# QAPI introspection generator
-#
-# Copyright (C) 2015-2016 Red Hat, Inc.
-#
-# Authors:
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI introspection generator
+
+Copyright (C) 2015-2018 Red Hat, Inc.
+
+Authors:
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -178,13 +179,11 @@ for o, a in opts:
 
 blurb = '''
  * QAPI/QMP schema introspection
- *
- * Copyright (C) 2015 Red Hat, Inc.
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-introspect.c', 'qmp-introspect.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-types.py +14 -18
index 86afc57f92..1103dbda2d 100644
@@ -1,15 +1,17 @@
-#
-# QAPI types generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (c) 2013-2016 Red Hat Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
+"""
+QAPI types generator
+
+Copyright IBM, Corp. 2011
+Copyright (c) 2013-2018 Red Hat Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -252,17 +254,11 @@ for o, a in opts:
 
 blurb = '''
  * Schema-defined QAPI types
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *  Michael Roth      <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-types.c', 'qapi-types.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-visit.py +15 -19
index 9f7127e548..5231f89c36 100644
@@ -1,16 +1,17 @@
-#
-# QAPI visitor generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (C) 2014-2016 Red Hat, Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Michael Roth    <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI visitor generator
+
+Copyright IBM, Corp. 2011
+Copyright (C) 2014-2018 Red Hat, Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth    <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -336,16 +337,11 @@ for o, a in opts:
 
 blurb = '''
  * Schema-defined QAPI visitors
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-visit.c', 'qapi-visit.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi.py +5 -2
index 2e4b2bc9eb..d0816f7479 100644
@@ -1975,21 +1975,24 @@ def parse_command_line(extra_options='', extra_long_options=[]):
 #
 
 
-def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb):
+def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb, doc):
     guard = guardname(prefix + h_file)
     c_file = output_dir + prefix + c_file
     h_file = output_dir + prefix + h_file
+    copyright = '\n * '.join(re.findall(r'^Copyright .*', doc, re.MULTILINE))
     comment = mcgen('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
 /*
 %(blurb)s
  *
+ * %(copyright)s
+ *
  * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
  * See the COPYING.LIB file in the top-level directory.
  */
 
 ''',
-                    blurb=blurb.strip('\n'))
+                    blurb=blurb.strip('\n'), copyright=copyright)
 
     if output_dir:
         try:
Signature · 1 line
2.13.6
Reviewed-by on v1 2/21
#7 lore raw

Re: [Qemu-devel] [PATCH RFC 02/21] qapi: Generate up-to-date copyright notice

Quoted 29 lines · Markus Armbruster

Each generator carries a copyright notice for the generator itself, and another one for the files it generates. Only the former have been updated along the way, the latter have not, and are all out of date.

Fix by copying the generator's copyright notice to the generated files instead.

---

+++ b/scripts/qapi-commands.py

scripts/qapi-commands.py +4 -13
@@ -1,16 +1,17 @@
-#
-# QAPI command marshaller generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (C) 2014-2016 Red Hat, Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Michael Roth    <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI command marshaller generator
+
+Copyright IBM, Corp. 2011

scripts/qapi-commands.py line 14

Quoted 10 lines
scripts/qapi-commands.py +10 -0
+Copyright (C) 2014-2018 Red Hat, Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""

So python lets you start a file with a string constant that is not associated with any variable name?

scripts/qapi-commands.py class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): · line 265

Quoted 5 lines
scripts/qapi-commands.py +1 -1
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-marshal.c', 'qmp-commands.h',
-                            blurb)
+                            blurb, __doc__)

Ah, and there's what I was missing - python auto-assigns such an initial string to a magic automatic variable. Cool language feature I hadn't seen before!

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#8 lore raw

Re: [Qemu-devel] [PATCH RFC 02/21] qapi: Generate up-to-date copyright notice

Quoted 35 lines · Eric Blake
Quoted 34 lines · Markus Armbruster

Each generator carries a copyright notice for the generator itself, and another one for the files it generates. Only the former have been updated along the way, the latter have not, and are all out of date.

Fix by copying the generator's copyright notice to the generated files instead.

---

+++ b/scripts/qapi-commands.py

scripts/qapi-commands.py +9 -13
@@ -1,16 +1,17 @@
-#
-# QAPI command marshaller generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (C) 2014-2016 Red Hat, Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Michael Roth    <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI command marshaller generator
+
+Copyright IBM, Corp. 2011
+Copyright (C) 2014-2018 Red Hat, Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth <[email protected]>
Quoted 8 lines

scripts/qapi-commands.py line 14

Quoted 5 lines
scripts/qapi-commands.py +5 -0
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""

So python lets you start a file with a string constant that is not associated with any variable name?

It's the module's doc string, done the same way as function doc strings.

https://www.python.org/dev/peps/pep-0257/#what-is-a-docstring

Quoted 10 lines

scripts/qapi-commands.py class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): · line 265

Quoted 4 lines
scripts/qapi-commands.py +1 -1
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-marshal.c', 'qmp-commands.h',
-                            blurb)
+                            blurb, __doc__)

Ah, and there's what I was missing - python auto-assigns such an initial string to a magic automatic variable. Cool language feature I hadn't seen before!

Thanks!

Reviewed-by Marc-André Lureau on v1 2/21
#9 lore raw

Re: [Qemu-devel] [PATCH RFC 02/21] qapi: Generate up-to-date copyright notice

In reply to Markus Armbruster

Hi

Quoted 6 lines · Markus Armbruster

Each generator carries a copyright notice for the generator itself, and another one for the files it generates. Only the former have been updated along the way, the latter have not, and are all out of date.

Fix by copying the generator's copyright notice to the generated files instead.

That makes sense, but we loose the Author lines in the generated files. Not a big deal I guess, but worth to point out, no?

Quoted 2 lines

Other than that,

Quoted 296 lines
---
 scripts/qapi-commands.py   | 34 +++++++++++++++-------------------
 scripts/qapi-event.py      | 32 ++++++++++++++------------------
 scripts/qapi-introspect.py | 25 ++++++++++++-------------
 scripts/qapi-types.py      | 32 ++++++++++++++------------------
 scripts/qapi-visit.py      | 34 +++++++++++++++-------------------
 scripts/qapi.py            |  7 +++++--
 6 files changed, 75 insertions(+), 89 deletions(-)
scripts/qapi-commands.py +15 -19
index 25ac52503a..a861ac52e7 100644
@@ -1,16 +1,17 @@
-#
-# QAPI command marshaller generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (C) 2014-2016 Red Hat, Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Michael Roth    <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI command marshaller generator
+
+Copyright IBM, Corp. 2011
+Copyright (C) 2014-2018 Red Hat, Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -257,16 +258,11 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 
 blurb = '''
  * Schema-defined QAPI/QMP commands
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-marshal.c', 'qmp-commands.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 
scripts/qapi-event.py +14 -18
index 31faedc689..b1d611c5ea 100644
@@ -1,15 +1,16 @@
-#
-# QAPI event generator
-#
-# Copyright (c) 2014 Wenchao Xia
-# Copyright (c) 2015-2016 Red Hat Inc.
-#
-# Authors:
-#  Wenchao Xia <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI event generator
+
+Copyright (c) 2014 Wenchao Xia
+Copyright (c) 2015-2018 Red Hat Inc.
+
+Authors:
+ Wenchao Xia <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -173,16 +174,11 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 
 blurb = '''
  * Schema-defined QAPI/QMP events
- *
- * Copyright (c) 2014 Wenchao Xia
- *
- * Authors:
- *  Wenchao Xia   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-event.c', 'qapi-event.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-introspect.py +12 -13
index 83da2bdb94..bd9253a172 100644
@@ -1,13 +1,14 @@
-#
-# QAPI introspection generator
-#
-# Copyright (C) 2015-2016 Red Hat, Inc.
-#
-# Authors:
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI introspection generator
+
+Copyright (C) 2015-2018 Red Hat, Inc.
+
+Authors:
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -178,13 +179,11 @@ for o, a in opts:
 
 blurb = '''
  * QAPI/QMP schema introspection
- *
- * Copyright (C) 2015 Red Hat, Inc.
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-introspect.c', 'qmp-introspect.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-types.py +14 -18
index 86afc57f92..1103dbda2d 100644
@@ -1,15 +1,17 @@
-#
-# QAPI types generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (c) 2013-2016 Red Hat Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
+"""
+QAPI types generator
+
+Copyright IBM, Corp. 2011
+Copyright (c) 2013-2018 Red Hat Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -252,17 +254,11 @@ for o, a in opts:
 
 blurb = '''
  * Schema-defined QAPI types
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *  Michael Roth      <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-types.c', 'qapi-types.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-visit.py +15 -19
index 9f7127e548..5231f89c36 100644
@@ -1,16 +1,17 @@
-#
-# QAPI visitor generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (C) 2014-2016 Red Hat, Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Michael Roth    <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI visitor generator
+
+Copyright IBM, Corp. 2011
+Copyright (C) 2014-2018 Red Hat, Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth    <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -336,16 +337,11 @@ for o, a in opts:
 
 blurb = '''
  * Schema-defined QAPI visitors
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-visit.c', 'qapi-visit.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi.py +5 -3
index 2e4b2bc9eb..d0816f7479 100644
@@ -1975,21 +1975,24 @@ def parse_command_line(extra_options='', extra_long_options=[]):
 #
 
 
-def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb):
+def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb, doc):
     guard = guardname(prefix + h_file)
     c_file = output_dir + prefix + c_file
     h_file = output_dir + prefix + h_file
+    copyright = '\n * '.join(re.findall(r'^Copyright .*', doc, re.MULTILINE))
     comment = mcgen('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
 /*
 %(blurb)s
  *
+ * %(copyright)s
+ *
  * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
  * See the COPYING.LIB file in the top-level directory.
  */
 
 ''',
-                    blurb=blurb.strip('\n'))
+                    blurb=blurb.strip('\n'), copyright=copyright)
 
     if output_dir:
         try:
--

2.13.6

#10 lore raw

Re: [Qemu-devel] [PATCH RFC 02/21] qapi: Generate up-to-date copyright notice

Quoted 7 lines · Marc-Andre Lureau

Hi

Quoted 4 lines · Markus Armbruster

Each generator carries a copyright notice for the generator itself, and another one for the files it generates. Only the former have been updated along the way, the latter have not, and are all out of date.

Quoted 5 lines
Quoted 2 lines

Fix by copying the generator's copyright notice to the generated files instead.

That makes sense, but we loose the Author lines in the generated files. Not a big deal I guess, but worth to point out, no?

We do. They've become just as outdated as the copyright notices. Giving them the same treatment doesn't seem to be worth the trouble. I'm happy to explain this in the commit message.

Quoted 5 lines
Quoted 2 lines

Other than that,

Thanks!

#11 lore raw

Re: [Qemu-devel] [PATCH RFC 02/21] qapi: Generate up-to-date copyright notice

In reply to Marc-Andre Lureau
Quoted 7 lines · Marc-Andre Lureau

Hi

Quoted 4 lines · Markus Armbruster

Each generator carries a copyright notice for the generator itself, and another one for the files it generates. Only the former have been updated along the way, the latter have not, and are all out of date.

Quoted 5 lines
Quoted 2 lines

Fix by copying the generator's copyright notice to the generated files instead.

That makes sense, but we loose the Author lines in the generated files. Not a big deal I guess, but worth to point out, no?

I don't see too much use for author lines as long as you have a 'this file is generated' line; you can use git history to learn the authors of the generator code. But yeah, a commit message blurb can't hurt.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#12 lore raw

RFC PATCH 3/21 qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc

In reply to Markus Armbruster

These classes encapsulate accumulating and writing output.

Convert C code generation to QAPIGenC and QAPIGenH. The conversion is rather shallow: most of the output accumulation is not converted. Left for later.

The indentation machinery uses a single global variable indent_level, even though we generally interleave creation of a .c and its .h. It should become instance variable of QAPIGenC. Also left for later.

Documentation generation isn't converted, and QAPIGenDoc isn't used. This will change shortly.

---
 scripts/qapi-commands.py   | 27 ++++++-------
 scripts/qapi-event.py      | 26 +++++++------
 scripts/qapi-introspect.py | 22 ++++++-----
 scripts/qapi-types.py      | 26 +++++++------
 scripts/qapi-visit.py      | 26 +++++++------
 scripts/qapi.py            | 96 ++++++++++++++++++++++++++--------------------
 6 files changed, 122 insertions(+), 101 deletions(-)
scripts/qapi-commands.py +14 -13
index a861ac52e7..4be7dbc482 100644
@@ -260,12 +260,10 @@ blurb = '''
  * Schema-defined QAPI/QMP commands
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-marshal.c', 'qmp-commands.h',
-                            blurb, __doc__)
-
-fdef.write(mcgen('''
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -279,21 +277,24 @@ fdef.write(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-                  prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+                prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenCommandVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenCommandVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-marshal.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-commands.h')
scripts/qapi-event.py +14 -12
index b1d611c5ea..da3de17c76 100644
@@ -176,11 +176,10 @@ blurb = '''
  * Schema-defined QAPI/QMP events
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-event.c', 'qapi-event.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "%(prefix)sqapi-event.h"
@@ -190,22 +189,25 @@ fdef.write(mcgen('''
 #include "qapi/qmp-event.h"
 
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "qapi/util.h"
 #include "qapi/qmp/qdict.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                  prefix=prefix))
+                prefix=prefix))
 
 event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenEventVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenEventVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-event.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-event.h')
scripts/qapi-introspect.py +12 -10
index bd9253a172..c654f8fa94 100644
@@ -181,21 +181,23 @@ blurb = '''
  * QAPI/QMP schema introspection
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-introspect.c', 'qmp-introspect.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "%(prefix)sqmp-introspect.h"
 
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenIntrospectVisitor(opt_unmask)
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-introspect.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-introspect.h')
scripts/qapi-types.py +14 -12
index 1103dbda2d..97406b3368 100644
@@ -180,7 +180,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.decl = ''
         self.defn = ''
         self._fwdecl = ''
-        self._btin = guardstart('QAPI_TYPES_BUILTIN')
+        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
 
     def visit_end(self):
         self.decl = self._fwdecl + self.decl
@@ -256,26 +256,28 @@ blurb = '''
  * Schema-defined QAPI types
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-types.c', 'qapi-types.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
 #include "%(prefix)sqapi-types.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "qapi/util.h"
 '''))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenTypeVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenTypeVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-types.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-types.h')
scripts/qapi-visit.py +14 -12
index 5231f89c36..d1b25daf0d 100644
@@ -339,30 +339,32 @@ blurb = '''
  * Schema-defined QAPI visitors
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-visit.c', 'qapi-visit.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qapi/error.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "qapi/visitor.h"
 #include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                  prefix=prefix))
+                prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenVisitVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenVisitVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-visit.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-visit.h')
scripts/qapi.py +54 -42

133 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v1 3/21
#13 lore raw

Re: [Qemu-devel] [PATCH RFC 03/21] qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc

Quoted 14 lines · Markus Armbruster

These classes encapsulate accumulating and writing output.

Convert C code generation to QAPIGenC and QAPIGenH. The conversion is rather shallow: most of the output accumulation is not converted. Left for later.

The indentation machinery uses a single global variable indent_level, even though we generally interleave creation of a .c and its .h. It should become instance variable of QAPIGenC. Also left for later.

Documentation generation isn't converted, and QAPIGenDoc isn't used. This will change shortly.

Quoted 9 lines
---
 scripts/qapi-commands.py   | 27 ++++++-------
 scripts/qapi-event.py      | 26 +++++++------
 scripts/qapi-introspect.py | 22 ++++++-----
 scripts/qapi-types.py      | 26 +++++++------
 scripts/qapi-visit.py      | 26 +++++++------
 scripts/qapi.py            | 96 ++++++++++++++++++++++++++--------------------
 6 files changed, 122 insertions(+), 101 deletions(-)

A little bit longer due to more structure, but reasonable diffstat in that it shows the conversion is fairly straightforward and opens the doors for later patches to use the new structures more effectively.

scripts/qapi-event.py fdef.write(mcgen(''' · line 208

Quoted 10 lines

schema = QAPISchema(input_file)

scripts/qapi-event.py +4 -4
-gen = QAPISchemaGenEventVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenEventVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)

I don't know if it is worth a sentence in the commit message that the visitor variable is renamed from 'gen' to 'vis' for less confusion with the new class instances 'genc' and 'genh'.

scripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 183

Quoted 7 lines

+++ b/scripts/qapi-types.py

scripts/qapi-types.py +1 -1
@@ -180,7 +180,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.decl = ''
         self.defn = ''
         self._fwdecl = ''
-        self._btin = guardstart('QAPI_TYPES_BUILTIN')
+        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')

Tweaks like this means you were paying attention to still producing identical generated files; always a good sign.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#14 lore raw

Re: [Qemu-devel] [PATCH RFC 03/21] qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc

Quoted 30 lines · Eric Blake
Quoted 14 lines · Markus Armbruster

These classes encapsulate accumulating and writing output.

Convert C code generation to QAPIGenC and QAPIGenH. The conversion is rather shallow: most of the output accumulation is not converted. Left for later.

The indentation machinery uses a single global variable indent_level, even though we generally interleave creation of a .c and its .h. It should become instance variable of QAPIGenC. Also left for later.

Documentation generation isn't converted, and QAPIGenDoc isn't used. This will change shortly.

Quoted 9 lines
---
 scripts/qapi-commands.py   | 27 ++++++-------
 scripts/qapi-event.py      | 26 +++++++------
 scripts/qapi-introspect.py | 22 ++++++-----
 scripts/qapi-types.py      | 26 +++++++------
 scripts/qapi-visit.py      | 26 +++++++------
 scripts/qapi.py            | 96 ++++++++++++++++++++++++++--------------------
 6 files changed, 122 insertions(+), 101 deletions(-)

A little bit longer due to more structure, but reasonable diffstat in that it shows the conversion is fairly straightforward and opens the doors for later patches to use the new structures more effectively.

Quoted 1 line
Quoted 13 lines

scripts/qapi-event.py fdef.write(mcgen(''' · line 208

Quoted 9 lines

schema = QAPISchema(input_file)

scripts/qapi-event.py +4 -4
-gen = QAPISchemaGenEventVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenEventVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)

I don't know if it is worth a sentence in the commit message that the visitor variable is renamed from 'gen' to 'vis' for less confusion with the new class instances 'genc' and 'genh'.

Did the rename give you pause when reviewing?

Quoted 11 lines

scripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 183

Quoted 7 lines

+++ b/scripts/qapi-types.py

scripts/qapi-types.py +1 -1
@@ -180,7 +180,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.decl = ''
         self.defn = ''
         self._fwdecl = ''
-        self._btin = guardstart('QAPI_TYPES_BUILTIN')
+        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')

Tweaks like this means you were paying attention to still producing identical generated files; always a good sign.

Quoted 1 line

Thanks!

#15 lore raw

Re: [Qemu-devel] [PATCH RFC 03/21] qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc

Quoted 31 lines · Markus Armbruster
Quoted 15 lines · Eric Blake
Quoted 13 lines · Markus Armbruster

These classes encapsulate accumulating and writing output.

Convert C code generation to QAPIGenC and QAPIGenH. The conversion is rather shallow: most of the output accumulation is not converted. Left for later.

The indentation machinery uses a single global variable indent_level, even though we generally interleave creation of a .c and its .h. It should become instance variable of QAPIGenC. Also left for later.

Documentation generation isn't converted, and QAPIGenDoc isn't used. This will change shortly.

Quoted 13 lines

scripts/qapi-event.py fdef.write(mcgen(''' · line 208

Quoted 9 lines

schema = QAPISchema(input_file)

scripts/qapi-event.py +4 -4
-gen = QAPISchemaGenEventVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenEventVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)

I don't know if it is worth a sentence in the commit message that the visitor variable is renamed from 'gen' to 'vis' for less confusion with the new class instances 'genc' and 'genh'.

Quoted 1 line

Did the rename give you pause when reviewing?

Enough to question whether it was intentional, since it wasn't mentioned in the commit message (I obviously figured out that it was intentional and useful, but the fact that I even pointed it out meant that I did pause during the review).

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#16 lore raw

Re: [Qemu-devel] [PATCH RFC 03/21] qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc

Quoted 32 lines · Eric Blake
Quoted 31 lines · Markus Armbruster
Quoted 28 lines · Eric Blake
Quoted 15 lines · Markus Armbruster

These classes encapsulate accumulating and writing output.

Convert C code generation to QAPIGenC and QAPIGenH. The conversion is rather shallow: most of the output accumulation is not converted. Left for later.

The indentation machinery uses a single global variable indent_level, even though we generally interleave creation of a .c and its .h. It should become instance variable of QAPIGenC. Also left for later.

Documentation generation isn't converted, and QAPIGenDoc isn't used. This will change shortly.

schema = QAPISchema(input_file)

scripts/qapi-event.py fdef.write(mcgen(''' · line 208

Quoted 8 lines
scripts/qapi-event.py +4 -4
-gen = QAPISchemaGenEventVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenEventVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)

I don't know if it is worth a sentence in the commit message that the visitor variable is renamed from 'gen' to 'vis' for less confusion with the new class instances 'genc' and 'genh'.

Quoted 6 lines
Quoted 1 line

Did the rename give you pause when reviewing?

Enough to question whether it was intentional, since it wasn't mentioned in the commit message (I obviously figured out that it was intentional and useful, but the fact that I even pointed it out meant that I did pause during the review).

Okay, I'll try to make it clearer.

Reviewed-by Marc-André Lureau on v1 3/21
#17 lore raw

Re: [Qemu-devel] [PATCH RFC 03/21] qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc

In reply to Markus Armbruster

Hi

Quoted 316 lines · Markus Armbruster

These classes encapsulate accumulating and writing output.

Convert C code generation to QAPIGenC and QAPIGenH. The conversion is rather shallow: most of the output accumulation is not converted. Left for later.

The indentation machinery uses a single global variable indent_level, even though we generally interleave creation of a .c and its .h. It should become instance variable of QAPIGenC. Also left for later.

Documentation generation isn't converted, and QAPIGenDoc isn't used. This will change shortly.

---
 scripts/qapi-commands.py   | 27 ++++++-------
 scripts/qapi-event.py      | 26 +++++++------
 scripts/qapi-introspect.py | 22 ++++++-----
 scripts/qapi-types.py      | 26 +++++++------
 scripts/qapi-visit.py      | 26 +++++++------
 scripts/qapi.py            | 96 ++++++++++++++++++++++++++--------------------
 6 files changed, 122 insertions(+), 101 deletions(-)
scripts/qapi-commands.py +14 -13
index a861ac52e7..4be7dbc482 100644
@@ -260,12 +260,10 @@ blurb = '''
  * Schema-defined QAPI/QMP commands
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-marshal.c', 'qmp-commands.h',
-                            blurb, __doc__)
-
-fdef.write(mcgen('''
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -279,21 +277,24 @@ fdef.write(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-                  prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+                prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenCommandVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenCommandVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-marshal.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-commands.h')
scripts/qapi-event.py +14 -12
index b1d611c5ea..da3de17c76 100644
@@ -176,11 +176,10 @@ blurb = '''
  * Schema-defined QAPI/QMP events
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-event.c', 'qapi-event.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "%(prefix)sqapi-event.h"
@@ -190,22 +189,25 @@ fdef.write(mcgen('''
 #include "qapi/qmp-event.h"
 
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "qapi/util.h"
 #include "qapi/qmp/qdict.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                  prefix=prefix))
+                prefix=prefix))
 
 event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenEventVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenEventVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-event.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-event.h')
scripts/qapi-introspect.py +12 -10
index bd9253a172..c654f8fa94 100644
@@ -181,21 +181,23 @@ blurb = '''
  * QAPI/QMP schema introspection
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-introspect.c', 'qmp-introspect.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "%(prefix)sqmp-introspect.h"
 
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenIntrospectVisitor(opt_unmask)
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-introspect.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-introspect.h')
scripts/qapi-types.py +14 -12
index 1103dbda2d..97406b3368 100644
@@ -180,7 +180,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.decl = ''
         self.defn = ''
         self._fwdecl = ''
-        self._btin = guardstart('QAPI_TYPES_BUILTIN')
+        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
 
     def visit_end(self):
         self.decl = self._fwdecl + self.decl
@@ -256,26 +256,28 @@ blurb = '''
  * Schema-defined QAPI types
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-types.c', 'qapi-types.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
 #include "%(prefix)sqapi-types.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "qapi/util.h"
 '''))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenTypeVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenTypeVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-types.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-types.h')
scripts/qapi-visit.py +14 -12
index 5231f89c36..d1b25daf0d 100644
@@ -339,30 +339,32 @@ blurb = '''
  * Schema-defined QAPI visitors
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-visit.c', 'qapi-visit.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qapi/error.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "qapi/visitor.h"
 #include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                  prefix=prefix))
+                prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenVisitVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenVisitVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-visit.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-visit.h')
scripts/qapi.py +11 -4
index d0816f7479..d73ef618e2 100644
@@ -2,7 +2,7 @@
 # QAPI helper library
 #
 # Copyright IBM, Corp. 2011
-# Copyright (c) 2013-2016 Red Hat Inc.
+# Copyright (c) 2013-2018 Red Hat Inc.
 #
 # Authors:
 #  Anthony Liguori <[email protected]>
@@ -1820,7 +1820,6 @@ def guardname(filename):
 
 def guardstart(name):
     return mcgen('''
-
 #ifndef %(name)s
 #define %(name)s
 
@@ -1832,7 +1831,6 @@ def guardend(name):
     return mcgen('''
 
 #endif /* %(name)s */
-
 ''',
                  name=guardname(name))
 
@@ -1970,17 +1968,53 @@ def parse_command_line(extra_options='', extra_long_options=[]):
 
     return (fname, output_dir, do_c, do_h, prefix, extra_opts)
 
+
 #
-# Generate output files with boilerplate
+# Accumulate and write output
 #
 
+class QAPIGen(object):
+
+    def __init__(self):
+        self._preamble = ''
+        self._body = ''
+
+    def preamble(self, text):
+        self._preamble += text

scripts/qapi.py def parse_command_line(extra_options='', extra_long_options=[]): · line 1993

Quoted 10 lines
scripts/qapi.py +10 -0
+
+    def body(self, text):
+        self._body += text
+
+    def top(self, fname):
+        return ''
+
+    def bottom(self, fname):
+        return ''
+

Some methods appends, other return. That's a bit confusing. Why not name them accordingly? add_preamble, add_body, get_top...?

Quoted 86 lines
scripts/qapi.py +33 -39
+    def write(self, output_dir, fname):
+        if output_dir:
+            try:
+                os.makedirs(output_dir)
+            except os.error as e:
+                if e.errno != errno.EEXIST:
+                    raise
+        f = open(os.path.join(output_dir, fname), 'w')
+        f.write(self.top(fname) + self._preamble + self._body
+                + self.bottom(fname))
+        f.close()
+
+
+class QAPIGenC(QAPIGen):
+
+    def __init__(self, blurb, pydoc):
+        QAPIGen.__init__(self)
+        self._blurb = blurb.strip('\n')
+        self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
+                                                  re.MULTILINE))
 
-def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb, doc):
-    guard = guardname(prefix + h_file)
-    c_file = output_dir + prefix + c_file
-    h_file = output_dir + prefix + h_file
-    copyright = '\n * '.join(re.findall(r'^Copyright .*', doc, re.MULTILINE))
-    comment = mcgen('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+    def top(self, fname):
+        return mcgen('''
+/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
 /*
 %(blurb)s
@@ -1992,41 +2026,19 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb, doc):
  */
 
 ''',
-                    blurb=blurb.strip('\n'), copyright=copyright)
+                     blurb=self._blurb, copyright=self._copyright)
 
-    if output_dir:
-        try:
-            os.makedirs(output_dir)
-        except os.error as e:
-            if e.errno != errno.EEXIST:
-                raise
 
-    def maybe_open(really, name, opt):
-        if really:
-            return open(name, opt)
-        else:
-            import StringIO
-            return StringIO.StringIO()
+class QAPIGenH(QAPIGenC):
 
-    fdef = maybe_open(do_c, c_file, 'w')
-    fdecl = maybe_open(do_h, h_file, 'w')
+    def top(self, fname):
+        return QAPIGenC.top(self, fname) + guardstart(fname)
 
-    fdef.write(comment)
-    fdecl.write(comment)
-    fdecl.write(mcgen('''
-#ifndef %(guard)s
-#define %(guard)s
+    def bottom(self, fname):
+        return guardend(fname)
 
-''',
-                      guard=guard))
 
-    return (fdef, fdecl)
-
-
-def close_output(fdef, fdecl):
-    fdecl.write(mcgen('''
-
-#endif
-'''))
-    fdecl.close()
-    fdef.close()
+class QAPIGenDoc(QAPIGen):
+    def top(self, fname):
+        return (QAPIGen.top(self, fname)
+                + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')
--
Quoted 2 lines

2.13.6

#18 lore raw

Re: [Qemu-devel] [PATCH RFC 03/21] qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc

Quoted 324 lines · Marc-Andre Lureau

Hi

Quoted 321 lines · Markus Armbruster

These classes encapsulate accumulating and writing output.

Convert C code generation to QAPIGenC and QAPIGenH. The conversion is rather shallow: most of the output accumulation is not converted. Left for later.

The indentation machinery uses a single global variable indent_level, even though we generally interleave creation of a .c and its .h. It should become instance variable of QAPIGenC. Also left for later.

Documentation generation isn't converted, and QAPIGenDoc isn't used. This will change shortly.

---
 scripts/qapi-commands.py   | 27 ++++++-------
 scripts/qapi-event.py      | 26 +++++++------
 scripts/qapi-introspect.py | 22 ++++++-----
 scripts/qapi-types.py      | 26 +++++++------
 scripts/qapi-visit.py      | 26 +++++++------
 scripts/qapi.py            | 96 ++++++++++++++++++++++++++--------------------
 6 files changed, 122 insertions(+), 101 deletions(-)
scripts/qapi-commands.py +14 -13
index a861ac52e7..4be7dbc482 100644
@@ -260,12 +260,10 @@ blurb = '''
  * Schema-defined QAPI/QMP commands
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-marshal.c', 'qmp-commands.h',
-                            blurb, __doc__)
-
-fdef.write(mcgen('''
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -279,21 +277,24 @@ fdef.write(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-                  prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+                prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenCommandVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenCommandVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-marshal.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-commands.h')
scripts/qapi-event.py +14 -12
index b1d611c5ea..da3de17c76 100644
@@ -176,11 +176,10 @@ blurb = '''
  * Schema-defined QAPI/QMP events
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-event.c', 'qapi-event.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "%(prefix)sqapi-event.h"
@@ -190,22 +189,25 @@ fdef.write(mcgen('''
 #include "qapi/qmp-event.h"
 
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "qapi/util.h"
 #include "qapi/qmp/qdict.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                  prefix=prefix))
+                prefix=prefix))
 
 event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenEventVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenEventVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-event.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-event.h')
scripts/qapi-introspect.py +12 -10
index bd9253a172..c654f8fa94 100644
@@ -181,21 +181,23 @@ blurb = '''
  * QAPI/QMP schema introspection
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-introspect.c', 'qmp-introspect.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "%(prefix)sqmp-introspect.h"
 
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenIntrospectVisitor(opt_unmask)
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-introspect.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-introspect.h')
scripts/qapi-types.py +14 -12
index 1103dbda2d..97406b3368 100644
@@ -180,7 +180,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.decl = ''
         self.defn = ''
         self._fwdecl = ''
-        self._btin = guardstart('QAPI_TYPES_BUILTIN')
+        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
 
     def visit_end(self):
         self.decl = self._fwdecl + self.decl
@@ -256,26 +256,28 @@ blurb = '''
  * Schema-defined QAPI types
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-types.c', 'qapi-types.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
 #include "%(prefix)sqapi-types.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "qapi/util.h"
 '''))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenTypeVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenTypeVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-types.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-types.h')
scripts/qapi-visit.py +14 -12
index 5231f89c36..d1b25daf0d 100644
@@ -339,30 +339,32 @@ blurb = '''
  * Schema-defined QAPI visitors
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-visit.c', 'qapi-visit.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qapi/error.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                 prefix=prefix))
+                prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.body(mcgen('''
 #include "qapi/visitor.h"
 #include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                  prefix=prefix))
+                prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenVisitVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenVisitVisitor()
+schema.visit(vis)
+genc.body(vis.defn)
+genh.body(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-visit.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-visit.h')
scripts/qapi.py +16 -4
index d0816f7479..d73ef618e2 100644
@@ -2,7 +2,7 @@
 # QAPI helper library
 #
 # Copyright IBM, Corp. 2011
-# Copyright (c) 2013-2016 Red Hat Inc.
+# Copyright (c) 2013-2018 Red Hat Inc.
 #
 # Authors:
 #  Anthony Liguori <[email protected]>
@@ -1820,7 +1820,6 @@ def guardname(filename):
 
 def guardstart(name):
     return mcgen('''
-
 #ifndef %(name)s
 #define %(name)s
 
@@ -1832,7 +1831,6 @@ def guardend(name):
     return mcgen('''
 
 #endif /* %(name)s */
-
 ''',
                  name=guardname(name))
 
@@ -1970,17 +1968,53 @@ def parse_command_line(extra_options='', extra_long_options=[]):
 
     return (fname, output_dir, do_c, do_h, prefix, extra_opts)
 
+
 #
-# Generate output files with boilerplate
+# Accumulate and write output
 #
 
+class QAPIGen(object):
+
+    def __init__(self):
+        self._preamble = ''
+        self._body = ''
+
+    def preamble(self, text):
+        self._preamble += text
+
+    def body(self, text):
+        self._body += text
+
+    def top(self, fname):
Quoted 8 lines

scripts/qapi.py def parse_command_line(extra_options='', extra_long_options=[]): · line 1993

Quoted 5 lines
scripts/qapi.py +5 -0
+        return ''
+
+    def bottom(self, fname):
+        return ''
+

Some methods appends, other return. That's a bit confusing. Why not name them accordingly? add_preamble, add_body, get_top...?

You're right, the functions called for side effects have less than obvious names. I think I'll keep the noun names for the pure functions.

Quoted 82 lines
Quoted 82 lines
scripts/qapi.py +30 -38
+    def write(self, output_dir, fname):
+        if output_dir:
+            try:
+                os.makedirs(output_dir)
+            except os.error as e:
+                if e.errno != errno.EEXIST:
+                    raise
+        f = open(os.path.join(output_dir, fname), 'w')
+        f.write(self.top(fname) + self._preamble + self._body
+                + self.bottom(fname))
+        f.close()
+
+
+class QAPIGenC(QAPIGen):
+
+    def __init__(self, blurb, pydoc):
+        QAPIGen.__init__(self)
+        self._blurb = blurb.strip('\n')
+        self._copyright = '\n * '.join(re.findall(r'^Copyright .*', pydoc,
+                                                  re.MULTILINE))
 
-def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb, doc):
-    guard = guardname(prefix + h_file)
-    c_file = output_dir + prefix + c_file
-    h_file = output_dir + prefix + h_file
-    copyright = '\n * '.join(re.findall(r'^Copyright .*', doc, re.MULTILINE))
-    comment = mcgen('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+    def top(self, fname):
+        return mcgen('''
+/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
 /*
 %(blurb)s
@@ -1992,41 +2026,19 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb, doc):
  */
 
 ''',
-                    blurb=blurb.strip('\n'), copyright=copyright)
+                     blurb=self._blurb, copyright=self._copyright)
 
-    if output_dir:
-        try:
-            os.makedirs(output_dir)
-        except os.error as e:
-            if e.errno != errno.EEXIST:
-                raise
 
-    def maybe_open(really, name, opt):
-        if really:
-            return open(name, opt)
-        else:
-            import StringIO
-            return StringIO.StringIO()
+class QAPIGenH(QAPIGenC):
 
-    fdef = maybe_open(do_c, c_file, 'w')
-    fdecl = maybe_open(do_h, h_file, 'w')
+    def top(self, fname):
+        return QAPIGenC.top(self, fname) + guardstart(fname)
 
-    fdef.write(comment)
-    fdecl.write(comment)
-    fdecl.write(mcgen('''
-#ifndef %(guard)s
-#define %(guard)s
+    def bottom(self, fname):
+        return guardend(fname)
 
-''',
-                      guard=guard))
 
-    return (fdef, fdecl)
-
-
-def close_output(fdef, fdecl):
-    fdecl.write(mcgen('''
-
-#endif
-'''))
-    fdecl.close()
-    fdef.close()
+class QAPIGenDoc(QAPIGen):
Quoted 8 lines
Quoted 6 lines
scripts/qapi.py +3 -1
+    def top(self, fname):
+        return (QAPIGen.top(self, fname)
+                + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')
--

2.13.6

Thanks!

#19 lore raw

RFC PATCH 4/21 qapi: Reduce use of global variables in generators some

In reply to Markus Armbruster

In preparation of the next commit, which will turn the generators into modules. These global variables will become local to main() then.

---
 scripts/qapi-commands.py   |  9 +++++----
 scripts/qapi-event.py      | 15 +++++++--------
 scripts/qapi-introspect.py |  7 ++++---
 scripts/qapi-types.py      | 17 +++++++++--------
 scripts/qapi-visit.py      | 17 +++++++++--------
 5 files changed, 34 insertions(+), 31 deletions(-)
scripts/qapi-commands.py +5 -4
index 4be7dbc482..d229537659 100644
@@ -207,7 +207,7 @@ def gen_register_command(name, success_response):
     return ret
 
 
-def gen_registry(registry):
+def gen_registry(registry, prefix):
     ret = mcgen('''
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -224,7 +224,8 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
 
 
 class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, prefix):
+        self._prefix = prefix
         self.decl = None
         self.defn = None
         self._regy = None
@@ -237,7 +238,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._visited_ret_types = set()
 
     def visit_end(self):
-        self.defn += gen_registry(self._regy)
+        self.defn += gen_registry(self._regy, self._prefix)
         self._regy = None
         self._visited_ret_types = None
 
@@ -289,7 +290,7 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
                 prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenCommandVisitor()
+vis = QAPISchemaGenCommandVisitor(prefix)
 schema.visit(vis)
 genc.body(vis.defn)
 genh.body(vis.decl)
scripts/qapi-event.py +7 -8
index da3de17c76..1af21b580a 100644
@@ -58,7 +58,7 @@ def gen_param_var(typ):
     return ret
 
 
-def gen_event_send(name, arg_type, boxed):
+def gen_event_send(name, arg_type, boxed, event_enum_name):
     # FIXME: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
@@ -149,7 +149,8 @@ out:
 
 
 class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, prefix):
+        self._enum_name = c_name(prefix + 'QAPIEvent', protect=False)
         self.decl = None
         self.defn = None
         self._event_names = None
@@ -160,13 +161,13 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self._event_names = []
 
     def visit_end(self):
-        self.decl += gen_enum(event_enum_name, self._event_names)
-        self.defn += gen_enum_lookup(event_enum_name, self._event_names)
+        self.decl += gen_enum(self._enum_name, self._event_names)
+        self.defn += gen_enum_lookup(self._enum_name, self._event_names)
         self._event_names = None
 
     def visit_event(self, name, info, arg_type, boxed):
         self.decl += gen_event_send_decl(name, arg_type, boxed)
-        self.defn += gen_event_send(name, arg_type, boxed)
+        self.defn += gen_event_send(name, arg_type, boxed, self._enum_name)
         self._event_names.append(name)
 
 
@@ -199,10 +200,8 @@ genh.body(mcgen('''
 ''',
                 prefix=prefix))
 
-event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenEventVisitor()
+vis = QAPISchemaGenEventVisitor(prefix)
 schema.visit(vis)
 genc.body(vis.defn)
 genh.body(vis.decl)
scripts/qapi-introspect.py +4 -3
index c654f8fa94..8d4e3c1c3a 100644
@@ -41,7 +41,8 @@ def to_c_string(string):
 
 
 class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
-    def __init__(self, unmask):
+    def __init__(self, prefix, unmask):
+        self._prefix = prefix
         self._unmask = unmask
         self.defn = None
         self.decl = None
@@ -65,7 +66,7 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
         # generate C
         # TODO can generate awfully long lines
         jsons.extend(self._jsons)
-        name = c_name(prefix, protect=False) + 'qmp_schema_json'
+        name = c_name(self._prefix, protect=False) + 'qmp_schema_json'
         self.decl = mcgen('''
 extern const char %(c_name)s[];
 ''',
@@ -192,7 +193,7 @@ genc.body(mcgen('''
                 prefix=prefix))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
+vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
 schema.visit(vis)
 genc.body(vis.defn)
 genh.body(vis.decl)
scripts/qapi-types.py +9 -8
index 97406b3368..2d711b137b 100644
@@ -168,7 +168,8 @@ void qapi_free_%(c_name)s(%(c_name)s *obj)
 
 
 class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, opt_builtins):
+        self._opt_builtins = opt_builtins
         self.decl = None
         self.defn = None
         self._fwdecl = None
@@ -187,7 +188,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self._fwdecl = None
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
-        # simply guard them.  See also do_builtins (command line
+        # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_TYPES_BUILTIN')
         self.decl = self._btin + self.decl
@@ -202,7 +203,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         # TODO use something cleaner than existence of info
         if not info:
             self._btin += gen_enum(name, values, prefix)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_enum_lookup(name, values, prefix)
         else:
             self._fwdecl += gen_enum(name, values, prefix)
@@ -213,7 +214,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
             self._btin += gen_fwd_object_or_array(name)
             self._btin += gen_array(name, element_type)
             self._btin += gen_type_cleanup_decl(name)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_type_cleanup(name)
         else:
             self._fwdecl += gen_fwd_object_or_array(name)
@@ -241,16 +242,16 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
 # If you link code generated from multiple schemata, you want only one
 # instance of the code for built-in types.  Generate it only when
-# do_builtins, enabled by command line option -b.  See also
+# opt_builtins, enabled by command line option -b.  See also
 # QAPISchemaGenTypeVisitor.visit_end().
-do_builtins = False
+opt_builtins = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
     parse_command_line('b', ['builtins'])
 
 for o, a in opts:
     if o in ('-b', '--builtins'):
-        do_builtins = True
+        opt_builtins = True
 
 blurb = '''
  * Schema-defined QAPI types
@@ -272,7 +273,7 @@ genh.body(mcgen('''
 '''))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenTypeVisitor()
+vis = QAPISchemaGenTypeVisitor(opt_builtins)
 schema.visit(vis)
 genc.body(vis.defn)
 genh.body(vis.decl)
scripts/qapi-visit.py +9 -8
index d1b25daf0d..79dc6cae48 100644
@@ -264,7 +264,8 @@ out:
 
 
 class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, opt_builtins):
+        self._opt_builtins = opt_builtins
         self.decl = None
         self.defn = None
         self._btin = None
@@ -277,7 +278,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
     def visit_end(self):
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
-        # simply guard them.  See also do_builtins (command line
+        # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_VISIT_BUILTIN')
         self.decl = self._btin + self.decl
@@ -288,7 +289,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         # TODO use something cleaner than existence of info
         if not info:
             self._btin += gen_visit_decl(name, scalar=True)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_visit_enum(name)
         else:
             self.decl += gen_visit_decl(name, scalar=True)
@@ -299,7 +300,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         defn = gen_visit_list(name, element_type)
         if isinstance(element_type, QAPISchemaBuiltinType):
             self._btin += decl
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += defn
         else:
             self.decl += decl
@@ -324,16 +325,16 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
 
 # If you link code generated from multiple schemata, you want only one
 # instance of the code for built-in types.  Generate it only when
-# do_builtins, enabled by command line option -b.  See also
+# opt_builtins, enabled by command line option -b.  See also
 # QAPISchemaGenVisitVisitor.visit_end().
-do_builtins = False
+opt_builtins = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
     parse_command_line('b', ['builtins'])
 
 for o, a in opts:
     if o in ('-b', '--builtins'):
-        do_builtins = True
+        opt_builtins = True
 
 blurb = '''
  * Schema-defined QAPI visitors
@@ -359,7 +360,7 @@ genh.body(mcgen('''
                 prefix=prefix))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenVisitVisitor()
+vis = QAPISchemaGenVisitVisitor(opt_builtins)
 schema.visit(vis)
 genc.body(vis.defn)
 genh.body(vis.decl)
Signature · 1 line
2.13.6
Reviewed-by on v1 4/21
#20 lore raw

Re: [Qemu-devel] [PATCH RFC 04/21] qapi: Reduce use of global variables in generators some

Quoted 5 lines · Markus Armbruster

In preparation of the next commit, which will turn the generators into modules. These global variables will become local to main() then.

---

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by Marc-André Lureau on v1 4/21
#21 lore raw

Re: [Qemu-devel] [PATCH RFC 04/21] qapi: Reduce use of global variables in generators some

In reply to Markus Armbruster
Quoted 4 lines · Markus Armbruster

In preparation of the next commit, which will turn the generators into modules. These global variables will become local to main() then.

Quoted 276 lines
---
 scripts/qapi-commands.py   |  9 +++++----
 scripts/qapi-event.py      | 15 +++++++--------
 scripts/qapi-introspect.py |  7 ++++---
 scripts/qapi-types.py      | 17 +++++++++--------
 scripts/qapi-visit.py      | 17 +++++++++--------
 5 files changed, 34 insertions(+), 31 deletions(-)
scripts/qapi-commands.py +5 -4
index 4be7dbc482..d229537659 100644
@@ -207,7 +207,7 @@ def gen_register_command(name, success_response):
     return ret
 
 
-def gen_registry(registry):
+def gen_registry(registry, prefix):
     ret = mcgen('''
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -224,7 +224,8 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
 
 
 class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, prefix):
+        self._prefix = prefix
         self.decl = None
         self.defn = None
         self._regy = None
@@ -237,7 +238,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._visited_ret_types = set()
 
     def visit_end(self):
-        self.defn += gen_registry(self._regy)
+        self.defn += gen_registry(self._regy, self._prefix)
         self._regy = None
         self._visited_ret_types = None
 
@@ -289,7 +290,7 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
                 prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenCommandVisitor()
+vis = QAPISchemaGenCommandVisitor(prefix)
 schema.visit(vis)
 genc.body(vis.defn)
 genh.body(vis.decl)
scripts/qapi-event.py +7 -8
index da3de17c76..1af21b580a 100644
@@ -58,7 +58,7 @@ def gen_param_var(typ):
     return ret
 
 
-def gen_event_send(name, arg_type, boxed):
+def gen_event_send(name, arg_type, boxed, event_enum_name):
     # FIXME: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
@@ -149,7 +149,8 @@ out:
 
 
 class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, prefix):
+        self._enum_name = c_name(prefix + 'QAPIEvent', protect=False)
         self.decl = None
         self.defn = None
         self._event_names = None
@@ -160,13 +161,13 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self._event_names = []
 
     def visit_end(self):
-        self.decl += gen_enum(event_enum_name, self._event_names)
-        self.defn += gen_enum_lookup(event_enum_name, self._event_names)
+        self.decl += gen_enum(self._enum_name, self._event_names)
+        self.defn += gen_enum_lookup(self._enum_name, self._event_names)
         self._event_names = None
 
     def visit_event(self, name, info, arg_type, boxed):
         self.decl += gen_event_send_decl(name, arg_type, boxed)
-        self.defn += gen_event_send(name, arg_type, boxed)
+        self.defn += gen_event_send(name, arg_type, boxed, self._enum_name)
         self._event_names.append(name)
 
 
@@ -199,10 +200,8 @@ genh.body(mcgen('''
 ''',
                 prefix=prefix))
 
-event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenEventVisitor()
+vis = QAPISchemaGenEventVisitor(prefix)
 schema.visit(vis)
 genc.body(vis.defn)
 genh.body(vis.decl)
scripts/qapi-introspect.py +4 -3
index c654f8fa94..8d4e3c1c3a 100644
@@ -41,7 +41,8 @@ def to_c_string(string):
 
 
 class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
-    def __init__(self, unmask):
+    def __init__(self, prefix, unmask):
+        self._prefix = prefix
         self._unmask = unmask
         self.defn = None
         self.decl = None
@@ -65,7 +66,7 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
         # generate C
         # TODO can generate awfully long lines
         jsons.extend(self._jsons)
-        name = c_name(prefix, protect=False) + 'qmp_schema_json'
+        name = c_name(self._prefix, protect=False) + 'qmp_schema_json'
         self.decl = mcgen('''
 extern const char %(c_name)s[];
 ''',
@@ -192,7 +193,7 @@ genc.body(mcgen('''
                 prefix=prefix))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
+vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
 schema.visit(vis)
 genc.body(vis.defn)
 genh.body(vis.decl)
scripts/qapi-types.py +9 -8
index 97406b3368..2d711b137b 100644
@@ -168,7 +168,8 @@ void qapi_free_%(c_name)s(%(c_name)s *obj)
 
 
 class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, opt_builtins):
+        self._opt_builtins = opt_builtins
         self.decl = None
         self.defn = None
         self._fwdecl = None
@@ -187,7 +188,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self._fwdecl = None
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
-        # simply guard them.  See also do_builtins (command line
+        # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_TYPES_BUILTIN')
         self.decl = self._btin + self.decl
@@ -202,7 +203,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         # TODO use something cleaner than existence of info
         if not info:
             self._btin += gen_enum(name, values, prefix)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_enum_lookup(name, values, prefix)
         else:
             self._fwdecl += gen_enum(name, values, prefix)
@@ -213,7 +214,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
             self._btin += gen_fwd_object_or_array(name)
             self._btin += gen_array(name, element_type)
             self._btin += gen_type_cleanup_decl(name)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_type_cleanup(name)
         else:
             self._fwdecl += gen_fwd_object_or_array(name)
@@ -241,16 +242,16 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
 # If you link code generated from multiple schemata, you want only one
 # instance of the code for built-in types.  Generate it only when
-# do_builtins, enabled by command line option -b.  See also
+# opt_builtins, enabled by command line option -b.  See also
 # QAPISchemaGenTypeVisitor.visit_end().
-do_builtins = False
+opt_builtins = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
     parse_command_line('b', ['builtins'])
 
 for o, a in opts:
     if o in ('-b', '--builtins'):
-        do_builtins = True
+        opt_builtins = True
 
 blurb = '''
  * Schema-defined QAPI types
@@ -272,7 +273,7 @@ genh.body(mcgen('''
 '''))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenTypeVisitor()
+vis = QAPISchemaGenTypeVisitor(opt_builtins)
 schema.visit(vis)
 genc.body(vis.defn)
 genh.body(vis.decl)
scripts/qapi-visit.py +9 -9
index d1b25daf0d..79dc6cae48 100644
@@ -264,7 +264,8 @@ out:
 
 
 class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, opt_builtins):
+        self._opt_builtins = opt_builtins
         self.decl = None
         self.defn = None
         self._btin = None
@@ -277,7 +278,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
     def visit_end(self):
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
-        # simply guard them.  See also do_builtins (command line
+        # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_VISIT_BUILTIN')
         self.decl = self._btin + self.decl
@@ -288,7 +289,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         # TODO use something cleaner than existence of info
         if not info:
             self._btin += gen_visit_decl(name, scalar=True)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_visit_enum(name)
         else:
             self.decl += gen_visit_decl(name, scalar=True)
@@ -299,7 +300,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         defn = gen_visit_list(name, element_type)
         if isinstance(element_type, QAPISchemaBuiltinType):
             self._btin += decl
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += defn
         else:
             self.decl += decl
@@ -324,16 +325,16 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
 
 # If you link code generated from multiple schemata, you want only one
 # instance of the code for built-in types.  Generate it only when
-# do_builtins, enabled by command line option -b.  See also
+# opt_builtins, enabled by command line option -b.  See also
 # QAPISchemaGenVisitVisitor.visit_end().
-do_builtins = False
+opt_builtins = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
     parse_command_line('b', ['builtins'])
 
 for o, a in opts:
     if o in ('-b', '--builtins'):
-        do_builtins = True
+        opt_builtins = True
 
 blurb = '''
  * Schema-defined QAPI visitors
@@ -359,7 +360,7 @@ genh.body(mcgen('''
                 prefix=prefix))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenVisitVisitor()
+vis = QAPISchemaGenVisitVisitor(opt_builtins)
 schema.visit(vis)
 genc.body(vis.defn)
 genh.body(vis.decl)
--

2.13.6

#22 lore raw

RFC PATCH 5/21 qapi: Turn generators into modules

In reply to Markus Armbruster

The next commit will introduce a common driver program for all generators. The generators need to be modules for that. qapi2texi.py already is. Make the other generators follow suit.

The changes are actually trivial. Obvious in the diffs once you view them with whitespace changes ignored.

---
 scripts/qapi-commands.py   | 43 ++++++++++++++++++--------------
 scripts/qapi-event.py      | 43 ++++++++++++++++++--------------
 scripts/qapi-introspect.py | 54 ++++++++++++++++++++++------------------
 scripts/qapi-types.py      | 56 ++++++++++++++++++++++-------------------
 scripts/qapi-visit.py      | 62 +++++++++++++++++++++++++---------------------
 5 files changed, 143 insertions(+), 115 deletions(-)
scripts/qapi-commands.py +24 -19
index d229537659..331b58670e 100644
@@ -255,16 +255,17 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._regy += gen_register_command(name, success_response)
 
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
+def main(argv):
+    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
 
-blurb = '''
+    blurb = '''
  * Schema-defined QAPI/QMP commands
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.body(mcgen('''
+    genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -278,24 +279,28 @@ genc.body(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                prefix=prefix))
+                    prefix=prefix))
 
-genh.body(mcgen('''
+    genh.body(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-                prefix=prefix, c_prefix=c_name(prefix, protect=False)))
-
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenCommandVisitor(prefix)
-schema.visit(vis)
-genc.body(vis.defn)
-genh.body(vis.decl)
-
-if do_c:
-    genc.write(output_dir, prefix + 'qmp-marshal.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qmp-commands.h')
+                    prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenCommandVisitor(prefix)
+    schema.visit(vis)
+    genc.body(vis.defn)
+    genh.body(vis.decl)
+
+    if do_c:
+        genc.write(output_dir, prefix + 'qmp-marshal.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qmp-commands.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi-event.py +24 -19

63 lines not on this page · raw message

scripts/qapi-introspect.py +30 -24

68 lines not on this page · raw message

scripts/qapi-types.py +31 -25

76 lines not on this page · raw message

scripts/qapi-visit.py +34 -28

83 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v1 5/21
#23 lore raw

Re: [Qemu-devel] [PATCH RFC 05/21] qapi: Turn generators into modules

Quoted 7 lines · Markus Armbruster

The next commit will introduce a common driver program for all generators. The generators need to be modules for that. qapi2texi.py already is. Make the other generators follow suit.

The changes are actually trivial. Obvious in the diffs once you view them with whitespace changes ignored.

Quoted 9 lines
---
 scripts/qapi-commands.py   | 43 ++++++++++++++++++--------------
 scripts/qapi-event.py      | 43 ++++++++++++++++++--------------
 scripts/qapi-introspect.py | 54 ++++++++++++++++++++++------------------
 scripts/qapi-types.py      | 56 ++++++++++++++++++++++-------------------
 scripts/qapi-visit.py      | 62 +++++++++++++++++++++++++---------------------
 5 files changed, 143 insertions(+), 115 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by Marc-André Lureau on v1 5/21
#24 lore raw

Re: [Qemu-devel] [PATCH RFC 05/21] qapi: Turn generators into modules

In reply to Markus Armbruster
Quoted 8 lines · Markus Armbruster

The next commit will introduce a common driver program for all generators. The generators need to be modules for that. qapi2texi.py already is. Make the other generators follow suit.

The changes are actually trivial. Obvious in the diffs once you view them with whitespace changes ignored.

Quoted 392 lines
---
 scripts/qapi-commands.py   | 43 ++++++++++++++++++--------------
 scripts/qapi-event.py      | 43 ++++++++++++++++++--------------
 scripts/qapi-introspect.py | 54 ++++++++++++++++++++++------------------
 scripts/qapi-types.py      | 56 ++++++++++++++++++++++-------------------
 scripts/qapi-visit.py      | 62 +++++++++++++++++++++++++---------------------
 5 files changed, 143 insertions(+), 115 deletions(-)
scripts/qapi-commands.py +24 -19
index d229537659..331b58670e 100644
@@ -255,16 +255,17 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._regy += gen_register_command(name, success_response)
 
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
+def main(argv):
+    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
 
-blurb = '''
+    blurb = '''
  * Schema-defined QAPI/QMP commands
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.body(mcgen('''
+    genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -278,24 +279,28 @@ genc.body(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                prefix=prefix))
+                    prefix=prefix))
 
-genh.body(mcgen('''
+    genh.body(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-                prefix=prefix, c_prefix=c_name(prefix, protect=False)))
-
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenCommandVisitor(prefix)
-schema.visit(vis)
-genc.body(vis.defn)
-genh.body(vis.decl)
-
-if do_c:
-    genc.write(output_dir, prefix + 'qmp-marshal.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qmp-commands.h')
+                    prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenCommandVisitor(prefix)
+    schema.visit(vis)
+    genc.body(vis.defn)
+    genh.body(vis.decl)
+
+    if do_c:
+        genc.write(output_dir, prefix + 'qmp-marshal.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qmp-commands.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi-event.py +24 -19
index 1af21b580a..5b33c694d4 100644
@@ -171,16 +171,17 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self._event_names.append(name)
 
 
-(input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
+def main(argv):
+    (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
 
-blurb = '''
+    blurb = '''
  * Schema-defined QAPI/QMP events
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.body(mcgen('''
+    genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "%(prefix)sqapi-event.h"
@@ -190,23 +191,27 @@ genc.body(mcgen('''
 #include "qapi/qmp-event.h"
 
 ''',
-                prefix=prefix))
+                    prefix=prefix))
 
-genh.body(mcgen('''
+    genh.body(mcgen('''
 #include "qapi/util.h"
 #include "qapi/qmp/qdict.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                prefix=prefix))
-
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenEventVisitor(prefix)
-schema.visit(vis)
-genc.body(vis.defn)
-genh.body(vis.decl)
-
-if do_c:
-    genc.write(output_dir, prefix + 'qapi-event.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qapi-event.h')
+                    prefix=prefix))
+
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenEventVisitor(prefix)
+    schema.visit(vis)
+    genc.body(vis.defn)
+    genh.body(vis.decl)
+
+    if do_c:
+        genc.write(output_dir, prefix + 'qapi-event.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qapi-event.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi-introspect.py +30 -24
index 8d4e3c1c3a..09e7d1f140 100644
@@ -167,38 +167,44 @@ const char %(c_name)s[] = %(c_string)s;
         arg_type = arg_type or self._schema.the_empty_object_type
         self._gen_json(name, 'event', {'arg-type': self._use_type(arg_type)})
 
-# Debugging aid: unmask QAPI schema's type names
-# We normally mask them, because they're not QMP wire ABI
-opt_unmask = False
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = \
-    parse_command_line('u', ['unmask-non-abi-names'])
+def main(argv):
+    # Debugging aid: unmask QAPI schema's type names
+    # We normally mask them, because they're not QMP wire ABI
+    opt_unmask = False
 
-for o, a in opts:
-    if o in ('-u', '--unmask-non-abi-names'):
-        opt_unmask = True
+    (input_file, output_dir, do_c, do_h, prefix, opts) = \
+        parse_command_line('u', ['unmask-non-abi-names'])
 
-blurb = '''
+    for o, a in opts:
+        if o in ('-u', '--unmask-non-abi-names'):
+            opt_unmask = True
+
+    blurb = '''
  * QAPI/QMP schema introspection
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.body(mcgen('''
+    genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "%(prefix)sqmp-introspect.h"
 
 ''',
-                prefix=prefix))
-
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
-schema.visit(vis)
-genc.body(vis.defn)
-genh.body(vis.decl)
-
-if do_c:
-    genc.write(output_dir, prefix + 'qmp-introspect.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qmp-introspect.h')
+                    prefix=prefix))
+
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
+    schema.visit(vis)
+    genc.body(vis.defn)
+    genh.body(vis.decl)
+
+    if do_c:
+        genc.write(output_dir, prefix + 'qmp-introspect.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qmp-introspect.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi-types.py +31 -25
index 2d711b137b..f2ddde94b1 100644
@@ -240,45 +240,51 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.decl += gen_object(name, None, [variants.tag_member], variants)
         self._gen_type_cleanup(name)
 
-# If you link code generated from multiple schemata, you want only one
-# instance of the code for built-in types.  Generate it only when
-# opt_builtins, enabled by command line option -b.  See also
-# QAPISchemaGenTypeVisitor.visit_end().
-opt_builtins = False
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = \
-    parse_command_line('b', ['builtins'])
+def main(argv):
+    # If you link code generated from multiple schemata, you want only one
+    # instance of the code for built-in types.  Generate it only when
+    # opt_builtins, enabled by command line option -b.  See also
+    # QAPISchemaGenTypeVisitor.visit_end().
+    opt_builtins = False
 
-for o, a in opts:
-    if o in ('-b', '--builtins'):
-        opt_builtins = True
+    (input_file, output_dir, do_c, do_h, prefix, opts) = \
+        parse_command_line('b', ['builtins'])
 
-blurb = '''
+    for o, a in opts:
+        if o in ('-b', '--builtins'):
+            opt_builtins = True
+
+    blurb = '''
  * Schema-defined QAPI types
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.body(mcgen('''
+    genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
 #include "%(prefix)sqapi-types.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                prefix=prefix))
+                    prefix=prefix))
 
-genh.body(mcgen('''
+    genh.body(mcgen('''
 #include "qapi/util.h"
 '''))
 
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenTypeVisitor(opt_builtins)
-schema.visit(vis)
-genc.body(vis.defn)
-genh.body(vis.decl)
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenTypeVisitor(opt_builtins)
+    schema.visit(vis)
+    genc.body(vis.defn)
+    genh.body(vis.decl)
 
-if do_c:
-    genc.write(output_dir, prefix + 'qapi-types.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qapi-types.h')
+    if do_c:
+        genc.write(output_dir, prefix + 'qapi-types.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qapi-types.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi-visit.py +34 -29
index 79dc6cae48..473fa72574 100644
@@ -323,49 +323,55 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         self.decl += gen_visit_decl(name)
         self.defn += gen_visit_alternate(name, variants)
 
-# If you link code generated from multiple schemata, you want only one
-# instance of the code for built-in types.  Generate it only when
-# opt_builtins, enabled by command line option -b.  See also
-# QAPISchemaGenVisitVisitor.visit_end().
-opt_builtins = False
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = \
-    parse_command_line('b', ['builtins'])
+def main(argv):
+    # If you link code generated from multiple schemata, you want only one
+    # instance of the code for built-in types.  Generate it only when
+    # opt_builtins, enabled by command line option -b.  See also
+    # QAPISchemaGenVisitVisitor.visit_end().
+    opt_builtins = False
 
-for o, a in opts:
-    if o in ('-b', '--builtins'):
-        opt_builtins = True
+    (input_file, output_dir, do_c, do_h, prefix, opts) = \
+        parse_command_line('b', ['builtins'])
 
-blurb = '''
+    for o, a in opts:
+        if o in ('-b', '--builtins'):
+            opt_builtins = True
+
+    blurb = '''
  * Schema-defined QAPI visitors
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.body(mcgen('''
+    genc.body(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qapi/error.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                prefix=prefix))
+                    prefix=prefix))
 
-genh.body(mcgen('''
+    genh.body(mcgen('''
 #include "qapi/visitor.h"
 #include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                prefix=prefix))
-
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenVisitVisitor(opt_builtins)
-schema.visit(vis)
-genc.body(vis.defn)
-genh.body(vis.decl)
-
-if do_c:
-    genc.write(output_dir, prefix + 'qapi-visit.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qapi-visit.h')
+                    prefix=prefix))
+
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenVisitVisitor(opt_builtins)
+    schema.visit(vis)
+    genc.body(vis.defn)
+    genh.body(vis.decl)
+
+    if do_c:
+        genc.write(output_dir, prefix + 'qapi-visit.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qapi-visit.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
--

2.13.6

#25 lore raw

RFC PATCH 6/21 qapi-gen: New common driver for code and doc generators

In reply to Markus Armbruster

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. This is silly. Replace the six programs by a single program that spits out all eleven files.

---
 Makefile                                           | 86 ++++++++++------------
 scripts/qapi-gen.py                                | 41 +++++++++++
 scripts/qapi/__init__.py                           |  0
 scripts/{qapi-commands.py => qapi/commands.py}     | 23 ++----
 scripts/{qapi.py => qapi/common.py}                |  0
 scripts/{qapi2texi.py => qapi/doc.py}              | 29 ++------
 scripts/{qapi-event.py => qapi/events.py}          | 23 ++----
 scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++------
 scripts/{qapi-types.py => qapi/types.py}           | 34 ++-------
 scripts/{qapi-visit.py => qapi/visit.py}           | 34 ++-------
 tests/Makefile.include                             | 56 +++++++-------
 tests/qapi-schema/test-qapi.py                     |  2 +-
 12 files changed, 140 insertions(+), 220 deletions(-)
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (94%)
 rename scripts/{qapi.py => qapi/common.py} (100%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
 mode change 100755 => 100644
 rename scripts/{qapi-event.py => qapi/events.py} (92%)
 rename scripts/{qapi-introspect.py => qapi/introspect.py} (90%)
 rename scripts/{qapi-types.py => qapi/types.py} (90%)
 rename scripts/{qapi-visit.py => qapi/visit.py} (92%)
Makefile +37 -49
index af31e8981f..e02f0c13ef 100644
@@ -92,6 +92,7 @@ GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
 GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
 GENERATED_FILES += qmp-introspect.h
 GENERATED_FILES += qmp-introspect.c
+GENERATED_FILES += qapi.texi
 
 GENERATED_FILES += trace/generated-tcg-tracers.h
 
@@ -477,25 +478,26 @@ qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated
 qemu-keymap$(EXESUF): LIBS += $(XKBCOMMON_LIBS)
 qemu-keymap$(EXESUF): QEMU_CFLAGS += $(XKBCOMMON_CFLAGS)
 
-gen-out-type = $(subst .,-,$(suffix $@))
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py
 
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
-
-qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \
-		$(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-		"GEN","$@")
-qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \
-		$(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-		"GEN","$@")
-qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
-		$(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-		"GEN","$@")
+qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
+qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qapi.texi: \
+qga/qapi-generated/qapi-gen-timestamp ;
+qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+		-o qga/qapi-generated -p "qga-" $<, \
+		"GEN","$(@:%-timestamp=%)")
+	@>$@
 
 qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/block.json $(SRC_PATH)/qapi/block-core.json \
@@ -512,31 +514,18 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/transaction.json \
                $(SRC_PATH)/qapi/ui.json
 
-qapi-types.c qapi-types.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \
-		$(gen-out-type) -o "." -b $<, \
-		"GEN","$@")
-qapi-visit.c qapi-visit.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \
-		$(gen-out-type) -o "." -b $<, \
-		"GEN","$@")
-qapi-event.c qapi-event.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-event.py $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \
-		$(gen-out-type) -o "." $<, \
-		"GEN","$@")
-qmp-commands.h qmp-marshal.c :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
-		$(gen-out-type) -o "." $<, \
-		"GEN","$@")
-qmp-introspect.h qmp-introspect.c :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
-		$(gen-out-type) -o "." $<, \
-		"GEN","$@")
+qapi-types.c qapi-types.h \
+qapi-visit.c qapi-visit.h \
+qmp-commands.h qmp-marshal.c \
+qapi-event.c qapi-event.h \
+qmp-introspect.h qmp-introspect.c \
+qapi.texi: \
+qapi-gen-timestamp ;
+qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+		-o "." -b $<, \
+		"GEN","$(@:%-timestamp=%)")
+	@>$@
 
 QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qmp-commands.h)
 $(qga-obj-y): $(QGALIB_GEN)
@@ -596,6 +585,7 @@ clean:
 	rm -f trace/generated-tracers-dtrace.dtrace*
 	rm -f trace/generated-tracers-dtrace.h*
 	rm -f $(foreach f,$(GENERATED_FILES),$(f) $(f)-timestamp)
+	rm -f qapi-gen-timestamp
 	rm -rf qapi-generated
 	rm -rf qga/qapi-generated
 	for d in $(ALL_SUBDIRS); do \
@@ -803,13 +793,11 @@ qemu-monitor-info.texi: $(SRC_PATH)/hmp-commands-info.hx $(SRC_PATH)/scripts/hxt
 qemu-img-cmds.texi: $(SRC_PATH)/qemu-img-cmds.hx $(SRC_PATH)/scripts/hxtool
 	$(call quiet-command,sh $(SRC_PATH)/scripts/hxtool -t < $< > $@,"GEN","$@")
 
-docs/interop/qemu-qmp-qapi.texi docs/interop/qemu-ga-qapi.texi: $(SRC_PATH)/scripts/qapi2texi.py $(qapi-py)
+docs/interop/qemu-qmp-qapi.texi: qapi.texi
+	@cp -p $< $@
 
-docs/interop/qemu-qmp-qapi.texi: $(qapi-modules)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@")
-
-docs/interop/qemu-ga-qapi.texi: $(SRC_PATH)/qga/qapi-schema.json
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@")
+docs/interop/qemu-ga-qapi.texi: qga/qapi-generated/qga-qapi.texi
+	@cp -p $< $@
 
 qemu.1: qemu-doc.texi qemu-options.texi qemu-monitor.texi qemu-monitor-info.texi
 qemu.1: qemu-option-trace.texi
scripts/qapi-gen.py new file +41 -0

41 lines not on this page · raw message

scripts/qapi/__init__.py new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +5 -18
similarity index 94%
rename from scripts/qapi-commands.py
rename to scripts/qapi/commands.py
index 331b58670e..383a4dd426 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
@@ -255,13 +255,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._regy += gen_register_command(name, success_response)
 
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP commands
-'''
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -290,17 +285,9 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
                     prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenCommandVisitor(prefix)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qmp-marshal.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qmp-commands.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qmp-marshal.c')
+    genh.write(output_dir, prefix + 'qmp-commands.h')
scripts/qapi/common.py renamed from scripts/qapi.py +0 -0
similarity index 100%
rename from scripts/qapi.py
rename to scripts/qapi/common.py
scripts/qapi/doc.py renamed from scripts/qapi2texi.py +8 -21
old mode 100755
new mode 100644
similarity index 92%
rename from scripts/qapi2texi.py
rename to scripts/qapi/doc.py
index 924b374cd3..1f57f6e1c2
@@ -4,10 +4,9 @@
 # This work is licensed under the terms of the GNU LGPL, version 2+.
 # See the COPYING file in the top-level directory.
 """This script produces the documentation of a qapi schema in texinfo format"""
+
 import re
-import sys
-
-import qapi
+import qapi.common
 
 MSG_FMT = """
 @deftypefn {type} {{}} {name}
@@ -196,7 +195,7 @@ def texi_entity(doc, what, base=None, variants=None,
             + texi_sections(doc))
 
 
-class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
+class QAPISchemaGenDocVisitor(qapi.common.QAPISchemaVisitor):
     def __init__(self):
         self.out = None
         self.cur_doc = None
@@ -271,20 +270,8 @@ def texi_schema(schema):
     return gen.out
 
 
-def main(argv):
-    """Takes schema argument, prints result to stdout"""
-    if len(argv) != 2:
-        print >>sys.stderr, "%s: need exactly 1 argument: SCHEMA" % argv[0]
-        sys.exit(1)
-
-    schema = qapi.QAPISchema(argv[1])
-    if not qapi.doc_required:
-        print >>sys.stderr, ("%s: need pragma 'doc-required' "
-                             "to generate documentation" % argv[0])
-        sys.exit(1)
-    print '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n'
-    print texi_schema(schema),
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+def gen_doc(schema, output_dir, prefix):
+    if qapi.common.doc_required:
+        gen = qapi.common.QAPIGenDoc()
+        gen.body(texi_schema(schema))
+        gen.write(output_dir, prefix + 'qapi.texi')
scripts/qapi/events.py renamed from scripts/qapi-event.py +5 -18
similarity index 92%
rename from scripts/qapi-event.py
rename to scripts/qapi/events.py
index 5b33c694d4..1f267686db 100644
@@ -12,7 +12,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def build_event_send_proto(name, arg_type, boxed):
@@ -171,13 +171,8 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self._event_names.append(name)
 
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP events
-'''
-
+def gen_events(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP events'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -201,17 +196,9 @@ def main(argv):
 ''',
                     prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenEventVisitor(prefix)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-event.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-event.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-event.c')
+    genh.write(output_dir, prefix + 'qapi-event.h')
scripts/qapi/introspect.py renamed from scripts/qapi-introspect.py +5 -27
similarity index 90%
rename from scripts/qapi-introspect.py
rename to scripts/qapi/introspect.py
index 09e7d1f140..2153060f2c 100644
@@ -10,7 +10,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # Caveman's json.dumps() replacement (we're stuck at Python 2.4)
@@ -168,22 +168,8 @@ const char %(c_name)s[] = %(c_string)s;
         self._gen_json(name, 'event', {'arg-type': self._use_type(arg_type)})
 
 
-def main(argv):
-    # Debugging aid: unmask QAPI schema's type names
-    # We normally mask them, because they're not QMP wire ABI
-    opt_unmask = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('u', ['unmask-non-abi-names'])
-
-    for o, a in opts:
-        if o in ('-u', '--unmask-non-abi-names'):
-            opt_unmask = True
-
-    blurb = '''
- * QAPI/QMP schema introspection
-'''
-
+def gen_introspect(schema, output_dir, prefix, opt_unmask):
+    blurb = ' * QAPI/QMP schema introspection'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -194,17 +180,9 @@ def main(argv):
 ''',
                     prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qmp-introspect.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qmp-introspect.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qmp-introspect.c')
+    genh.write(output_dir, prefix + 'qmp-introspect.h')
scripts/qapi/types.py renamed from scripts/qapi-types.py +5 -29
similarity index 90%
rename from scripts/qapi-types.py
rename to scripts/qapi/types.py
index f2ddde94b1..b2095120e0 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # variants must be emitted before their container; track what has already
@@ -241,24 +241,8 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self._gen_type_cleanup(name)
 
 
-def main(argv):
-    # If you link code generated from multiple schemata, you want only one
-    # instance of the code for built-in types.  Generate it only when
-    # opt_builtins, enabled by command line option -b.  See also
-    # QAPISchemaGenTypeVisitor.visit_end().
-    opt_builtins = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('b', ['builtins'])
-
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-
-    blurb = '''
- * Schema-defined QAPI types
-'''
-
+def gen_types(schema, output_dir, prefix, opt_builtins):
+    blurb = ' * Schema-defined QAPI types'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -274,17 +258,9 @@ def main(argv):
 #include "qapi/util.h"
 '''))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenTypeVisitor(opt_builtins)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-types.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-types.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-types.c')
+    genh.write(output_dir, prefix + 'qapi-types.h')
scripts/qapi/visit.py renamed from scripts/qapi-visit.py +5 -29
similarity index 92%
rename from scripts/qapi-visit.py
rename to scripts/qapi/visit.py
index 473fa72574..80c0b85f8c 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def gen_visit_decl(name, scalar=False):
@@ -324,24 +324,8 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         self.defn += gen_visit_alternate(name, variants)
 
 
-def main(argv):
-    # If you link code generated from multiple schemata, you want only one
-    # instance of the code for built-in types.  Generate it only when
-    # opt_builtins, enabled by command line option -b.  See also
-    # QAPISchemaGenVisitVisitor.visit_end().
-    opt_builtins = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('b', ['builtins'])
-
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-
-    blurb = '''
- * Schema-defined QAPI visitors
-'''
-
+def gen_visit(schema, output_dir, prefix, opt_builtins):
+    blurb = ' * Schema-defined QAPI visitors'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -361,17 +345,9 @@ def main(argv):
 ''',
                     prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenVisitVisitor(opt_builtins)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-visit.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-visit.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-visit.c')
+    genh.write(output_dir, prefix + 'qapi-visit.h')
tests/Makefile.include +28 -28

75 lines not on this page · raw message

tests/qapi-schema/test-qapi.py +1 -1
index fe0ca08d78..7772d09919 100644
@@ -10,7 +10,7 @@
 # See the COPYING file in the top-level directory.
 #
 
-from qapi import *
+from qapi.common import *
 from pprint import pprint
 import os
 import sys
Signature · 1 line
2.13.6
#26 lore raw

Re: [Qemu-devel] [PATCH RFC 06/21] qapi-gen: New common driver for code and doc generators

Quoted 3 lines · Markus Armbruster

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. This is silly. Replace the six programs by a single program that spits out all eleven files.

Yay, about time!

One program, but still invoked multiple times:

Quoted 13 lines
---
 Makefile                                           | 86 ++++++++++------------
 scripts/qapi-gen.py                                | 41 +++++++++++
 scripts/qapi/__init__.py                           |  0
 scripts/{qapi-commands.py => qapi/commands.py}     | 23 ++----
 scripts/{qapi.py => qapi/common.py}                |  0
 scripts/{qapi2texi.py => qapi/doc.py}              | 29 ++------
 scripts/{qapi-event.py => qapi/events.py}          | 23 ++----
 scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++------
 scripts/{qapi-types.py => qapi/types.py}           | 34 ++-------
 scripts/{qapi-visit.py => qapi/visit.py}           | 34 ++-------

Maybe the commit message should mention:

This requires moving some files around for proper use in python.

Quoted 9 lines
 tests/Makefile.include                             | 56 +++++++-------
 tests/qapi-schema/test-qapi.py                     |  2 +-
 12 files changed, 140 insertions(+), 220 deletions(-)
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (94%)
 rename scripts/{qapi.py => qapi/common.py} (100%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
 mode change 100755 => 100644

Unintinentional? We're inconsistent on which of our *.py files are executable in git. Does it matter whether a file that is designed for use as a module is marked executable (if so, perhaps 5/21 should be adding the x attribute on other files, rather than this patch removing it on the doc generator).

Makefile qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated · line 500

Quoted 12 lines

+++ b/Makefile

Makefile +10 -0
+qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
+qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qapi.texi: \
+qga/qapi-generated/qapi-gen-timestamp ;
+qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+		-o qga/qapi-generated -p "qga-" $<, \
+		"GEN","$(@:%-timestamp=%)")
+	@>$@

once for qga,

Makefile qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \ · line 528

Quoted 12 lines
Makefile +12 -0
+qapi-types.c qapi-types.h \
+qapi-visit.c qapi-visit.h \
+qmp-commands.h qmp-marshal.c \
+qapi-event.c qapi-event.h \
+qmp-introspect.h qmp-introspect.c \
+qapi.texi: \
+qapi-gen-timestamp ;
+qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+		-o "." -b $<, \
+		"GEN","$(@:%-timestamp=%)")
+	@>$@

and again for the main level. Still, a definite improvement!

Quoted 13 lines

+++ b/scripts/qapi-gen.py

scripts/qapi-gen.py new file +11 -0
+def main(argv):
+    (input_file, output_dir, do_c, do_h, prefix, opts) = \
+        parse_command_line('bu', ['builtins', 'unmask-non-abi-names'])
+
+    opt_builtins = False
+    opt_unmask = False
+
+    for o, a in opts:
+        if o in ('-b', '--builtins'):
+            opt_builtins = True
+        if o in ('-u', '--unmask-non-abi-names'):

scripts/qapi-gen.py line 37

Quoted 10 lines
scripts/qapi-gen.py new file +10 -0
+            opt_unmask = True
+
+    schema = QAPISchema(input_file)
+
+    gen_types(schema, output_dir, prefix, opt_builtins)
+    gen_visit(schema, output_dir, prefix, opt_builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, opt_unmask)
+    gen_doc(schema, output_dir, prefix)

Rather simple, but definitely nicer all in one python file than as a series of make rules.

scripts/qapi/commands.py class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): · line 259

Quoted 13 lines
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +2 -7
@@ -255,13 +255,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._regy += gen_register_command(name, success_response)
 
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP commands
-'''
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'

Some churn on the definition of blurb; worth tidying this in the introduction earlier in the series?

Quoted 14 lines

rename to scripts/qapi/introspect.py
index 09e7d1f140..2153060f2c 100644

scripts/qapi/introspect.py renamed from scripts/qapi-introspect.py +1 -1
@@ -10,7 +10,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # Caveman's json.dumps() replacement (we're stuck at Python 2.4)

scripts/qapi/introspect.py const char %(c_name)s[] = %(c_string)s; · line 177

Quoted 10 lines
scripts/qapi/introspect.py renamed from scripts/qapi-introspect.py +0 -7
@@ -168,22 +168,8 @@ const char %(c_name)s[] = %(c_string)s;
         self._gen_json(name, 'event', {'arg-type': self._use_type(arg_type)})
 
 
-def main(argv):
-    # Debugging aid: unmask QAPI schema's type names
-    # We normally mask them, because they're not QMP wire ABI
-    opt_unmask = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('u', ['unmask-non-abi-names'])

Hmm - we didn't have any docs about this hidden command line option; I see you still preserved it, but it may be worth mentioning in your pending doc updates for the series respin.

Quoted 7 lines
tests/Makefile.include +0 -1
@@ -23,7 +23,16 @@ check-help:
 ifneq ($(wildcard config-host.mak),)
 export SRC_PATH
 
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py

tests/Makefile.include check-help: · line 35

Quoted 10 lines
tests/Makefile.include +10 -0
+# TODO don't duplicate $(SRC_PATH)/Makefile's qapi-py here
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py

So, were you counting these in the 11 generated files mentioned in the commit message? :)

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#27 lore raw

Re: [Qemu-devel] [PATCH RFC 06/21] qapi-gen: New common driver for code and doc generators

Quoted 23 lines · Eric Blake
Quoted 3 lines · Markus Armbruster

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. This is silly. Replace the six programs by a single program that spits out all eleven files.

Yay, about time!

One program, but still invoked multiple times:

Quoted 13 lines
---
 Makefile                                           | 86 ++++++++++------------
 scripts/qapi-gen.py                                | 41 +++++++++++
 scripts/qapi/__init__.py                           |  0
 scripts/{qapi-commands.py => qapi/commands.py}     | 23 ++----
 scripts/{qapi.py => qapi/common.py}                |  0
 scripts/{qapi2texi.py => qapi/doc.py}              | 29 ++------
 scripts/{qapi-event.py => qapi/events.py}          | 23 ++----
 scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++------
 scripts/{qapi-types.py => qapi/types.py}           | 34 ++-------
 scripts/{qapi-visit.py => qapi/visit.py}           | 34 ++-------
Quoted 3 lines

Maybe the commit message should mention:

This requires moving some files around for proper use in python.

Yes, I should mention that I wrap the QAPI modules in a Python package.

Quoted 10 lines
Quoted 9 lines
 tests/Makefile.include                             | 56 +++++++-------
 tests/qapi-schema/test-qapi.py                     |  2 +-
 12 files changed, 140 insertions(+), 220 deletions(-)
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (94%)
 rename scripts/{qapi.py => qapi/common.py} (100%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
 mode change 100755 => 100644
Quoted 5 lines

Unintinentional? We're inconsistent on which of our *.py files are executable in git. Does it matter whether a file that is designed for use as a module is marked executable (if so, perhaps 5/21 should be adding the x attribute on other files, rather than this patch removing it on the doc generator).

qapi2texi.py is no longer runnable as a standalone program after this patch.

So are qapi-{commands,events,introspect,types,visit}.py, but they never had the executable bit set.

Quoted 14 lines

Makefile qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated · line 500

Quoted 12 lines

+++ b/Makefile

Makefile +10 -0
+qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
+qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qapi.texi: \
+qga/qapi-generated/qapi-gen-timestamp ;
+qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+		-o qga/qapi-generated -p "qga-" $<, \
+		"GEN","$(@:%-timestamp=%)")
+	@>$@

once for qga,

That's the QAPI/QGA schema. The commit message talks about the QAPI/QMP schema. I wish the two weren't named the same.

Modularization might make fusing them possible. Whether that's a good idea I don't know.

Quoted 6 lines
Quoted 6 lines
Makefile +6 -0
+qapi-types.c qapi-types.h \
+qapi-visit.c qapi-visit.h \
+qmp-commands.h qmp-marshal.c \
+qapi-event.c qapi-event.h \
+qmp-introspect.h qmp-introspect.c \
+qapi.texi: \
Quoted 8 lines

Makefile qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \ · line 528

Quoted 6 lines
Makefile +6 -0
+qapi-gen-timestamp ;
+qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+		-o "." -b $<, \
+		"GEN","$(@:%-timestamp=%)")
+	@>$@

and again for the main level. Still, a definite improvement!

Perhaps I can find a way to clarify the commit message.

Quoted 31 lines
Quoted 13 lines

+++ b/scripts/qapi-gen.py

scripts/qapi-gen.py new file +11 -0
+def main(argv):
+    (input_file, output_dir, do_c, do_h, prefix, opts) = \
+        parse_command_line('bu', ['builtins', 'unmask-non-abi-names'])
+
+    opt_builtins = False
+    opt_unmask = False
+
+    for o, a in opts:
+        if o in ('-b', '--builtins'):
+            opt_builtins = True
+        if o in ('-u', '--unmask-non-abi-names'):

scripts/qapi-gen.py line 37

Quoted 10 lines
scripts/qapi-gen.py new file +10 -0
+            opt_unmask = True
+
+    schema = QAPISchema(input_file)
+
+    gen_types(schema, output_dir, prefix, opt_builtins)
+    gen_visit(schema, output_dir, prefix, opt_builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, opt_unmask)
+    gen_doc(schema, output_dir, prefix)

Rather simple, but definitely nicer all in one python file than as a series of make rules.

Quoted 4 lines
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +0 -0
@@ -255,13 +255,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._regy += gen_register_command(name, success_response)
 
 
Quoted 12 lines

scripts/qapi/commands.py class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): · line 259

Quoted 9 lines
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +2 -7
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP commands
-'''
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'

Some churn on the definition of blurb; worth tidying this in the introduction earlier in the series?

Doesn't seem worth a separate patch, but I can try to reshuffle things to reduce churn.

Quoted 17 lines
Quoted 17 lines

rename to scripts/qapi/introspect.py
index 09e7d1f140..2153060f2c 100644

scripts/qapi/introspect.py renamed from scripts/qapi-introspect.py +1 -1
@@ -10,7 +10,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # Caveman's json.dumps() replacement (we're stuck at Python 2.4)
@@ -168,22 +168,8 @@ const char %(c_name)s[] = %(c_string)s;
         self._gen_json(name, 'event', {'arg-type': self._use_type(arg_type)})
 
 
Quoted 11 lines

scripts/qapi/introspect.py const char %(c_name)s[] = %(c_string)s; · line 177

Quoted 7 lines
scripts/qapi/introspect.py renamed from scripts/qapi-introspect.py +0 -7
-def main(argv):
-    # Debugging aid: unmask QAPI schema's type names
-    # We normally mask them, because they're not QMP wire ABI
-    opt_unmask = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('u', ['unmask-non-abi-names'])

Hmm - we didn't have any docs about this hidden command line option; I see you still preserved it, but it may be worth mentioning in your pending doc updates for the series respin.

Maybe.

Providing --help would probably be more useful. We should convert qapi-gen.py to argparse.

Quoted 6 lines
Quoted 6 lines
tests/Makefile.include +0 -0
@@ -23,7 +23,16 @@ check-help:
 ifneq ($(wildcard config-host.mak),)
 export SRC_PATH
 
Quoted 14 lines

tests/Makefile.include check-help: · line 35

Quoted 11 lines
tests/Makefile.include +10 -1
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
+# TODO don't duplicate $(SRC_PATH)/Makefile's qapi-py here
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py

So, were you counting these in the 11 generated files mentioned in the commit message? :)

I'm not sure I understand what you mean here...

#28 lore raw

Re: [Qemu-devel] [PATCH RFC 06/21] qapi-gen: New common driver for code and doc generators

Quoted 21 lines · Markus Armbruster
Quoted 10 lines · Eric Blake
Quoted 3 lines · Markus Armbruster

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. This is silly. Replace the six programs by a single program that spits out all eleven files.

Yay, about time!

One program, but still invoked multiple times:

Quoted 8 lines
Quoted 2 lines
 rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
 mode change 100755 => 100644

Unintinentional? We're inconsistent on which of our *.py files are executable in git. Does it matter whether a file that is designed for use as a module is marked executable (if so, perhaps 5/21 should be adding the x attribute on other files, rather than this patch removing it on the doc generator).

Quoted 5 lines

qapi2texi.py is no longer runnable as a standalone program after this patch.

So are qapi-{commands,events,introspect,types,visit}.py, but they never had the executable bit set.

Okay, so dropping the bit consistently makes sense; still, a mention in the commit message wouldn't hurt.

Quoted 16 lines
Quoted 15 lines

Makefile qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated · line 500

Quoted 13 lines

+++ b/Makefile

Makefile +10 -0
+qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
+qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qapi.texi: \
+qga/qapi-generated/qapi-gen-timestamp ;
+qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+		-o qga/qapi-generated -p "qga-" $<, \
+		"GEN","$(@:%-timestamp=%)")
+	@>$@

once for qga,

Quoted 2 lines

That's the QAPI/QGA schema. The commit message talks about the QAPI/QMP schema. I wish the two weren't named the same.

7 files here,...

Quoted 18 lines

Modularization might make fusing them possible. Whether that's a good idea I don't know.

Quoted 14 lines

Makefile qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \ · line 528

Quoted 12 lines
Makefile +12 -0
+qapi-types.c qapi-types.h \
+qapi-visit.c qapi-visit.h \
+qmp-commands.h qmp-marshal.c \
+qapi-event.c qapi-event.h \
+qmp-introspect.h qmp-introspect.c \
+qapi.texi: \
+qapi-gen-timestamp ;
+qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+		-o "." -b $<, \
+		"GEN","$(@:%-timestamp=%)")
+	@>$@

and again for the main level. Still, a definite improvement!

11 files here,...

Quoted 3 lines

Perhaps I can find a way to clarify the commit message.

[1]

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +0 -4
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
Quoted 8 lines

scripts/qapi/commands.py class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): · line 259

Quoted 5 lines

- * Schema-defined QAPI/QMP commands
-'''

scripts/qapi/commands.py renamed from scripts/qapi-commands.py +2 -1
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'

Some churn on the definition of blurb; worth tidying this in the introduction earlier in the series?

Quoted 2 lines

Doesn't seem worth a separate patch, but I can try to reshuffle things to reduce churn.

Yeah, my question was more about the conversion between multiline '''...''' to single-line '...'; why not just use the single-line construct earlier in the series when introducing the blurb variable. You are right that creating blurb didn't need a separate patch, just less churn over the series.

Quoted 16 lines
Quoted 15 lines

tests/Makefile.include check-help: · line 35

Quoted 12 lines
tests/Makefile.include +10 -1
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
+# TODO don't duplicate $(SRC_PATH)/Makefile's qapi-py here
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py

So, were you counting these in the 11 generated files mentioned in the commit message? :)

Quoted 1 line

I'm not sure I understand what you mean here...

[1] and 9 more files. So, the commit message only mentioned the 11 QMP files, rather than all 27 (if I counted right) QAPI generated files. My comments were trying to point out that you simplified more than just the QMP code generation into fewer script invocations, and the counts looked off since out of the three spots converted, only one of the spots had 11 files.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#29 lore raw

Re: [Qemu-devel] [PATCH RFC 06/21] qapi-gen: New common driver for code and doc generators

Quoted 22 lines · Eric Blake
Quoted 21 lines · Markus Armbruster
Quoted 18 lines · Eric Blake
Quoted 3 lines · Markus Armbruster

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. This is silly. Replace the six programs by a single program that spits out all eleven files.

Yay, about time!

One program, but still invoked multiple times:

Quoted 2 lines
 rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
 mode change 100755 => 100644

Unintinentional? We're inconsistent on which of our *.py files are executable in git. Does it matter whether a file that is designed for use as a module is marked executable (if so, perhaps 5/21 should be adding the x attribute on other files, rather than this patch removing it on the doc generator).

Quoted 8 lines
Quoted 5 lines

qapi2texi.py is no longer runnable as a standalone program after this patch.

So are qapi-{commands,events,introspect,types,visit}.py, but they never had the executable bit set.

Okay, so dropping the bit consistently makes sense; still, a mention in the commit message wouldn't hurt.

Can do.

Quoted 61 lines
Quoted 15 lines
Quoted 14 lines

Makefile qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated · line 500

Quoted 12 lines

+++ b/Makefile

Makefile +10 -0
+qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
+qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qapi.texi: \
+qga/qapi-generated/qapi-gen-timestamp ;
+qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+		-o qga/qapi-generated -p "qga-" $<, \
+		"GEN","$(@:%-timestamp=%)")
+	@>$@

once for qga,

Quoted 2 lines

That's the QAPI/QGA schema. The commit message talks about the QAPI/QMP schema. I wish the two weren't named the same.

7 files here,...

Quoted 18 lines

Modularization might make fusing them possible. Whether that's a good idea I don't know.

Quoted 14 lines

Makefile qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \ · line 528

Quoted 12 lines
Makefile +12 -0
+qapi-types.c qapi-types.h \
+qapi-visit.c qapi-visit.h \
+qmp-commands.h qmp-marshal.c \
+qapi-event.c qapi-event.h \
+qmp-introspect.h qmp-introspect.c \
+qapi.texi: \
+qapi-gen-timestamp ;
+qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+		-o "." -b $<, \
+		"GEN","$(@:%-timestamp=%)")
+	@>$@

and again for the main level. Still, a definite improvement!

11 files here,...

Quoted 3 lines

Perhaps I can find a way to clarify the commit message.

[1]

Quoted 13 lines
Quoted 12 lines

scripts/qapi/commands.py class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): · line 259

Quoted 9 lines
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +2 -7
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP commands
-'''
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'

Some churn on the definition of blurb; worth tidying this in the introduction earlier in the series?

Quoted 6 lines
Quoted 2 lines

Doesn't seem worth a separate patch, but I can try to reshuffle things to reduce churn.

Yeah, my question was more about the conversion between multiline '''...''' to single-line '...'; why not just use the single-line construct earlier in the series when introducing the blurb variable.

Introduced in PATCH 01:

-c_comment = '''
-/*
- * schema-defined QMP->QAPI command dispatch
+blurb = '''
+ * Schema-defined QAPI/QMP commands
*
* Copyright IBM, Corp. 2011
*
* Authors:
* Anthony Liguori <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''

Shortened in PATCH 02:

blurb = '''
* Schema-defined QAPI/QMP commands
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- * Anthony Liguori <[email protected]>
'''

Reformatted in PATCH 06 (see above).

Moved in PATCH 16 to visitor's __init__ for types and visits (the rest aren't implemented, yet):

class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
- def __init__(self, opt_builtins):
+ def __init__(self, opt_builtins, prefix):
self._opt_builtins = opt_builtins
- self.decl = None
- self.defn = None
- self._fwdecl = None
- self._btin = None
+ self._prefix = prefix
+ blurb = ' * Schema-defined QAPI types'
+ self._genc = QAPIGenC(blurb, __doc__)
+ self._genh = QAPIGenH(blurb, __doc__)

Variable eliminated there in PATCH 17:

    -        blurb = ' * Schema-defined QAPI types'
    -        self._genc = QAPIGenC(blurb, __doc__)
    -        self._genh = QAPIGenH(blurb, __doc__)
    +        self._module = {}
    +        self._add_module(None, ' * Built-in QAPI types')

I could delay the reformatting until PATCH 16, or do it in PATCH 02. Feels like a wash to me, but if you have a preference...

Quoted 19 lines

You are right that creating blurb didn't need a separate patch, just less churn over the series.

Quoted 16 lines
Quoted 15 lines

tests/Makefile.include check-help: · line 35

Quoted 12 lines
tests/Makefile.include +10 -1
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
+# TODO don't duplicate $(SRC_PATH)/Makefile's qapi-py here
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py

So, were you counting these in the 11 generated files mentioned in the commit message? :)

Quoted 8 lines
Quoted 1 line

I'm not sure I understand what you mean here...

[1] and 9 more files. So, the commit message only mentioned the 11 QMP files, rather than all 27 (if I counted right) QAPI generated files. My comments were trying to point out that you simplified more than just the QMP code generation into fewer script invocations, and the counts looked off since out of the three spots converted, only one of the spots had 11 files.

The commit message talks only about the QAPI/QMP schema. It's confusing because our poor taste in file names creates ambiguity: does qapi-schema.json refer to ./qapi-schema.json (i.e. the QAPI/QMP one), qga/qapi-schema.json, or both?

Perhaps I should rename qapi-schema.json to qapi/schema.json.

The commit message needs a note along the lines of "same for qga/qapi-schema.json".

#30 lore raw

Re: [Qemu-devel] [PATCH RFC 06/21] qapi-gen: New common driver for code and doc generators

Quoted 22 lines · Markus Armbruster
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

scripts/qapi/commands.py class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): · line 259

Quoted 9 lines
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +2 -7
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP commands
-'''
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'

Some churn on the definition of blurb; worth tidying this in the introduction earlier in the series?

Quoted 6 lines
Quoted 2 lines

Doesn't seem worth a separate patch, but I can try to reshuffle things to reduce churn.

Yeah, my question was more about the conversion between multiline '''...''' to single-line '...'; why not just use the single-line construct earlier in the series when introducing the blurb variable.

Introduced in PATCH 01:

Quoted 7 lines
     -c_comment = '''
     -/*
     - * schema-defined QMP->QAPI command dispatch
     +blurb = '''
     + * Schema-defined QAPI/QMP commands
       *
       * Copyright IBM, Corp. 2011

Multiline made sense here.

Quoted 9 lines

Shortened in PATCH 02:

blurb = '''
* Schema-defined QAPI/QMP commands
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- * Anthony Liguori <[email protected]>

       '''
Quoted 1 line

Where it is now a single line...

Quoted 7 lines

Variable eliminated there in PATCH 17:

     -        blurb = ' * Schema-defined QAPI types'
     -        self._genc = QAPIGenC(blurb, __doc__)
     -        self._genh = QAPIGenH(blurb, __doc__)
     +        self._module = {}
     +        self._add_module(None, ' * Built-in QAPI types')

Oh, I didn't even notice that!

Quoted 3 lines

I could delay the reformatting until PATCH 16, or do it in PATCH 02. Feels like a wash to me, but if you have a preference...

I guess my preference is to reformat to single line in PATCH 02, then patch 6 doesn't have to touch it, and patch 16 can still get rid of it. But, as it disappears by the end of the series anyways, I'm also okay if you don't bother (churn is not necessarily bad, if it is still easy to review).

Quoted 6 lines

The commit message talks only about the QAPI/QMP schema. It's confusing because our poor taste in file names creates ambiguity: does qapi-schema.json refer to ./qapi-schema.json (i.e. the QAPI/QMP one), qga/qapi-schema.json, or both?

Perhaps I should rename qapi-schema.json to qapi/schema.json.

I'd be in favor of such a change. We'd also rename qga/qapi-schema.json to qga/schema.json?

Quoted 4 lines

The commit message needs a note along the lines of "same for qga/qapi-schema.json".

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by Marc-André Lureau on v1 6/21
#31 lore raw

Re: [Qemu-devel] [PATCH RFC 06/21] qapi-gen: New common driver for code and doc generators

In reply to Markus Armbruster
Quoted 4 lines · Markus Armbruster

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. This is silly. Replace the six programs by a single program that spits out all eleven files.

Now we will need documentation update.

Also greping for the renamed files:

git grep -E 'qapi-commands|qapi2texi|qapi-event.py|qapi-intros|qapi-types.py|qapi-visit.py' docs/devel/qapi-code-gen.txt:=== scripts/qapi-types.py ===
docs/devel/qapi-code-gen.txt: $ python scripts/qapi-types.py
--output-dir="qapi-generated" \
docs/devel/qapi-code-gen.txt:=== scripts/qapi-visit.py ===
docs/devel/qapi-code-gen.txt: $ python scripts/qapi-visit.py
--output-dir="qapi-generated"
docs/devel/qapi-code-gen.txt:=== scripts/qapi-commands.py ===
docs/devel/qapi-code-gen.txt: generated by
qapi-visit.py are used to
docs/devel/qapi-code-gen.txt: $ python scripts/qapi-commands.py
--output-dir="qapi-generated"
docs/devel/qapi-code-gen.txt:=== scripts/qapi-event.py ===
docs/devel/qapi-code-gen.txt: $ python scripts/qapi-event.py
--output-dir="qapi-generated"
docs/devel/qapi-code-gen.txt:=== scripts/qapi-introspect.py ===
docs/devel/qapi-code-gen.txt: $ python scripts/qapi-introspect.py
--output-dir="qapi-generated"
monitor.c: * qapi-introspect.py's output actually conforms to the schema. qapi-schema.json:# Documentation generated with qapi2texi.py is in
source order, with

Quoted 282 lines
---
 Makefile                                           | 86 ++++++++++------------
 scripts/qapi-gen.py                                | 41 +++++++++++
 scripts/qapi/__init__.py                           |  0
 scripts/{qapi-commands.py => qapi/commands.py}     | 23 ++----
 scripts/{qapi.py => qapi/common.py}                |  0
 scripts/{qapi2texi.py => qapi/doc.py}              | 29 ++------
 scripts/{qapi-event.py => qapi/events.py}          | 23 ++----
 scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++------
 scripts/{qapi-types.py => qapi/types.py}           | 34 ++-------
 scripts/{qapi-visit.py => qapi/visit.py}           | 34 ++-------
 tests/Makefile.include                             | 56 +++++++-------
 tests/qapi-schema/test-qapi.py                     |  2 +-
 12 files changed, 140 insertions(+), 220 deletions(-)
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (94%)
 rename scripts/{qapi.py => qapi/common.py} (100%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
 mode change 100755 => 100644
 rename scripts/{qapi-event.py => qapi/events.py} (92%)
 rename scripts/{qapi-introspect.py => qapi/introspect.py} (90%)
 rename scripts/{qapi-types.py => qapi/types.py} (90%)
 rename scripts/{qapi-visit.py => qapi/visit.py} (92%)
Makefile +37 -49
index af31e8981f..e02f0c13ef 100644
@@ -92,6 +92,7 @@ GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
 GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
 GENERATED_FILES += qmp-introspect.h
 GENERATED_FILES += qmp-introspect.c
+GENERATED_FILES += qapi.texi
 
 GENERATED_FILES += trace/generated-tcg-tracers.h
 
@@ -477,25 +478,26 @@ qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated
 qemu-keymap$(EXESUF): LIBS += $(XKBCOMMON_LIBS)
 qemu-keymap$(EXESUF): QEMU_CFLAGS += $(XKBCOMMON_CFLAGS)
 
-gen-out-type = $(subst .,-,$(suffix $@))
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py
 
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
-
-qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \
-               $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-               "GEN","$@")
-qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \
-               $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-               "GEN","$@")
-qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
-               $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-               "GEN","$@")
+qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
+qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qapi.texi: \
+qga/qapi-generated/qapi-gen-timestamp ;
+qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
+       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o qga/qapi-generated -p "qga-" $<, \
+               "GEN","$(@:%-timestamp=%)")
+       @>$@
 
 qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/block.json $(SRC_PATH)/qapi/block-core.json \
@@ -512,31 +514,18 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/transaction.json \
                $(SRC_PATH)/qapi/ui.json
 
-qapi-types.c qapi-types.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \
-               $(gen-out-type) -o "." -b $<, \
-               "GEN","$@")
-qapi-visit.c qapi-visit.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \
-               $(gen-out-type) -o "." -b $<, \
-               "GEN","$@")
-qapi-event.c qapi-event.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-event.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \
-               $(gen-out-type) -o "." $<, \
-               "GEN","$@")
-qmp-commands.h qmp-marshal.c :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
-               $(gen-out-type) -o "." $<, \
-               "GEN","$@")
-qmp-introspect.h qmp-introspect.c :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
-               $(gen-out-type) -o "." $<, \
-               "GEN","$@")
+qapi-types.c qapi-types.h \
+qapi-visit.c qapi-visit.h \
+qmp-commands.h qmp-marshal.c \
+qapi-event.c qapi-event.h \
+qmp-introspect.h qmp-introspect.c \
+qapi.texi: \
+qapi-gen-timestamp ;
+qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
+       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o "." -b $<, \
+               "GEN","$(@:%-timestamp=%)")
+       @>$@
 
 QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qmp-commands.h)
 $(qga-obj-y): $(QGALIB_GEN)
@@ -596,6 +585,7 @@ clean:
        rm -f trace/generated-tracers-dtrace.dtrace*
        rm -f trace/generated-tracers-dtrace.h*
        rm -f $(foreach f,$(GENERATED_FILES),$(f) $(f)-timestamp)
+       rm -f qapi-gen-timestamp
        rm -rf qapi-generated
        rm -rf qga/qapi-generated
        for d in $(ALL_SUBDIRS); do \
@@ -803,13 +793,11 @@ qemu-monitor-info.texi: $(SRC_PATH)/hmp-commands-info.hx $(SRC_PATH)/scripts/hxt
 qemu-img-cmds.texi: $(SRC_PATH)/qemu-img-cmds.hx $(SRC_PATH)/scripts/hxtool
        $(call quiet-command,sh $(SRC_PATH)/scripts/hxtool -t < $< > $@,"GEN","$@")
 
-docs/interop/qemu-qmp-qapi.texi docs/interop/qemu-ga-qapi.texi: $(SRC_PATH)/scripts/qapi2texi.py $(qapi-py)
+docs/interop/qemu-qmp-qapi.texi: qapi.texi
+       @cp -p $< $@
 
-docs/interop/qemu-qmp-qapi.texi: $(qapi-modules)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@")
-
-docs/interop/qemu-ga-qapi.texi: $(SRC_PATH)/qga/qapi-schema.json
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@")
+docs/interop/qemu-ga-qapi.texi: qga/qapi-generated/qga-qapi.texi
+       @cp -p $< $@
 
 qemu.1: qemu-doc.texi qemu-options.texi qemu-monitor.texi qemu-monitor-info.texi
 qemu.1: qemu-option-trace.texi
scripts/qapi-gen.py new file +41 -0
new file mode 100755
index 0000000000..575c938a1b
@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+# QAPI generator
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+import sys
+from qapi.common import parse_command_line, QAPISchema
+from qapi.types import gen_types
+from qapi.visit import gen_visit
+from qapi.commands import gen_commands
+from qapi.events import gen_events
+from qapi.introspect import gen_introspect
+from qapi.doc import gen_doc
+
+
+def main(argv):
+    (input_file, output_dir, do_c, do_h, prefix, opts) = \
+        parse_command_line('bu', ['builtins', 'unmask-non-abi-names'])
+
+    opt_builtins = False
+    opt_unmask = False
+
+    for o, a in opts:
+        if o in ('-b', '--builtins'):
+            opt_builtins = True
+        if o in ('-u', '--unmask-non-abi-names'):
+            opt_unmask = True
+
+    schema = QAPISchema(input_file)
+
+    gen_types(schema, output_dir, prefix, opt_builtins)
+    gen_visit(schema, output_dir, prefix, opt_builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, opt_unmask)
+    gen_doc(schema, output_dir, prefix)
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi/__init__.py new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +5 -18
similarity index 94%
rename from scripts/qapi-commands.py
rename to scripts/qapi/commands.py
index 331b58670e..383a4dd426 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
@@ -255,13 +255,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._regy += gen_register_command(name, success_response)
 
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP commands
-'''
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -290,17 +285,9 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
                     prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenCommandVisitor(prefix)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qmp-marshal.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qmp-commands.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qmp-marshal.c')
+    genh.write(output_dir, prefix + 'qmp-commands.h')
scripts/qapi/common.py renamed from scripts/qapi.py +0 -0
similarity index 100%
rename from scripts/qapi.py
rename to scripts/qapi/common.py
scripts/qapi/doc.py renamed from scripts/qapi2texi.py +2 -3
old mode 100755
new mode 100644
similarity index 92%
rename from scripts/qapi2texi.py
rename to scripts/qapi/doc.py
index 924b374cd3..1f57f6e1c2
@@ -4,10 +4,9 @@
 # This work is licensed under the terms of the GNU LGPL, version 2+.
 # See the COPYING file in the top-level directory.
 """This script produces the documentation of a qapi schema in texinfo format"""
+
 import re
-import sys
-
-import qapi
+import qapi.common
 
 MSG_FMT = """
 @deftypefn {type} {{}} {name}

scripts/qapi/doc.py def texi_entity(doc, what, base=None, variants=None, · line 198

Quoted 5 lines
scripts/qapi/doc.py renamed from scripts/qapi2texi.py +1 -1
@@ -196,7 +195,7 @@ def texi_entity(doc, what, base=None, variants=None,
             + texi_sections(doc))
 
 
-class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
+class QAPISchemaGenDocVisitor(qapi.common.QAPISchemaVisitor):

Would be a bit easier to read and more consitent with a top-level "from qapi.common import QAPISchemaVisitor"

Quoted 273 lines
scripts/qapi/doc.py renamed from scripts/qapi2texi.py +5 -17
     def __init__(self):
         self.out = None
         self.cur_doc = None
@@ -271,20 +270,8 @@ def texi_schema(schema):
     return gen.out
 
 
-def main(argv):
-    """Takes schema argument, prints result to stdout"""
-    if len(argv) != 2:
-        print >>sys.stderr, "%s: need exactly 1 argument: SCHEMA" % argv[0]
-        sys.exit(1)
-
-    schema = qapi.QAPISchema(argv[1])
-    if not qapi.doc_required:
-        print >>sys.stderr, ("%s: need pragma 'doc-required' "
-                             "to generate documentation" % argv[0])
-        sys.exit(1)
-    print '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n'
-    print texi_schema(schema),
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+def gen_doc(schema, output_dir, prefix):
+    if qapi.common.doc_required:
+        gen = qapi.common.QAPIGenDoc()
+        gen.body(texi_schema(schema))
+        gen.write(output_dir, prefix + 'qapi.texi')
scripts/qapi/events.py renamed from scripts/qapi-event.py +5 -18
similarity index 92%
rename from scripts/qapi-event.py
rename to scripts/qapi/events.py
index 5b33c694d4..1f267686db 100644
@@ -12,7 +12,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def build_event_send_proto(name, arg_type, boxed):
@@ -171,13 +171,8 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self._event_names.append(name)
 
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP events
-'''
-
+def gen_events(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP events'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -201,17 +196,9 @@ def main(argv):
 ''',
                     prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenEventVisitor(prefix)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-event.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-event.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-event.c')
+    genh.write(output_dir, prefix + 'qapi-event.h')
scripts/qapi/introspect.py renamed from scripts/qapi-introspect.py +5 -27
similarity index 90%
rename from scripts/qapi-introspect.py
rename to scripts/qapi/introspect.py
index 09e7d1f140..2153060f2c 100644
@@ -10,7 +10,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # Caveman's json.dumps() replacement (we're stuck at Python 2.4)
@@ -168,22 +168,8 @@ const char %(c_name)s[] = %(c_string)s;
         self._gen_json(name, 'event', {'arg-type': self._use_type(arg_type)})
 
 
-def main(argv):
-    # Debugging aid: unmask QAPI schema's type names
-    # We normally mask them, because they're not QMP wire ABI
-    opt_unmask = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('u', ['unmask-non-abi-names'])
-
-    for o, a in opts:
-        if o in ('-u', '--unmask-non-abi-names'):
-            opt_unmask = True
-
-    blurb = '''
- * QAPI/QMP schema introspection
-'''
-
+def gen_introspect(schema, output_dir, prefix, opt_unmask):
+    blurb = ' * QAPI/QMP schema introspection'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -194,17 +180,9 @@ def main(argv):
 ''',
                     prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qmp-introspect.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qmp-introspect.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qmp-introspect.c')
+    genh.write(output_dir, prefix + 'qmp-introspect.h')
scripts/qapi/types.py renamed from scripts/qapi-types.py +5 -29
similarity index 90%
rename from scripts/qapi-types.py
rename to scripts/qapi/types.py
index f2ddde94b1..b2095120e0 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # variants must be emitted before their container; track what has already
@@ -241,24 +241,8 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self._gen_type_cleanup(name)
 
 
-def main(argv):
-    # If you link code generated from multiple schemata, you want only one
-    # instance of the code for built-in types.  Generate it only when
-    # opt_builtins, enabled by command line option -b.  See also
-    # QAPISchemaGenTypeVisitor.visit_end().
-    opt_builtins = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('b', ['builtins'])
-
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-
-    blurb = '''
- * Schema-defined QAPI types
-'''
-
+def gen_types(schema, output_dir, prefix, opt_builtins):
+    blurb = ' * Schema-defined QAPI types'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -274,17 +258,9 @@ def main(argv):
 #include "qapi/util.h"
 '''))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenTypeVisitor(opt_builtins)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-types.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-types.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-types.c')
+    genh.write(output_dir, prefix + 'qapi-types.h')
scripts/qapi/visit.py renamed from scripts/qapi-visit.py +5 -29
similarity index 92%
rename from scripts/qapi-visit.py
rename to scripts/qapi/visit.py
index 473fa72574..80c0b85f8c 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def gen_visit_decl(name, scalar=False):
@@ -324,24 +324,8 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         self.defn += gen_visit_alternate(name, variants)
 
 
-def main(argv):
-    # If you link code generated from multiple schemata, you want only one
-    # instance of the code for built-in types.  Generate it only when
-    # opt_builtins, enabled by command line option -b.  See also
-    # QAPISchemaGenVisitVisitor.visit_end().
-    opt_builtins = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('b', ['builtins'])
-
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-
-    blurb = '''
- * Schema-defined QAPI visitors
-'''
-
+def gen_visit(schema, output_dir, prefix, opt_builtins):
+    blurb = ' * Schema-defined QAPI visitors'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -361,17 +345,9 @@ def main(argv):
 ''',
                     prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenVisitVisitor(opt_builtins)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-visit.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-visit.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-visit.c')
+    genh.write(output_dir, prefix + 'qapi-visit.h')
tests/Makefile.include +0 -0
index 851aafe9d1..768655a810 100644

tests/Makefile.include check-help: · line 26

Quoted 5 lines
tests/Makefile.include +1 -1
@@ -23,7 +23,16 @@ check-help:
 ifneq ($(wildcard config-host.mak),)
 export SRC_PATH
 
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
+# TODO don't duplicate $(SRC_PATH)/Makefile's qapi-py here

hmm, it doesn't look necessary, since the makefile is included from there. Right?

Quoted 86 lines
tests/Makefile.include +27 -27
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py
 
 # Get the list of all supported sysemu targets
 SYSEMU_TARGET_LIST := $(subst -softmmu.mak,,$(notdir \
@@ -642,34 +651,24 @@ tests/test-logging$(EXESUF): tests/test-logging.o $(test-util-obj-y)
 tests/test-replication$(EXESUF): tests/test-replication.o $(test-util-obj-y) \
        $(test-block-obj-y)
 
-tests/test-qapi-types.c tests/test-qapi-types.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qapi-visit.c tests/test-qapi-visit.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qmp-commands.h tests/test-qmp-marshal.c :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qapi-event.c tests/test-qapi-event.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-event.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qmp-introspect.c tests/test-qmp-introspect.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
+tests/test-qapi-types.c tests/test-qapi-types.h \
+tests/test-qapi-visit.c tests/test-qapi-visit.h \
+tests/test-qmp-commands.h tests/test-qmp-marshal.c \
+tests/test-qapi-event.c tests/test-qapi-event.h \
+tests/test-qmp-introspect.c tests/test-qmp-introspect.h: \
+tests/test-qapi-gen-timestamp ;
+tests/test-qapi-gen-timestamp: $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(qapi-py)
+       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o tests -p "test-" $<, \
+               "GEN","$(@:%-timestamp=%)")
+       @>$@
 
-tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-good.json $(SRC_PATH)/scripts/qapi2texi.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@")
+tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-good.json $(qapi-py)
+       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o tests/qapi-schema -p "doc-good-" $<, \
+               "GEN","$@")
+       @mv tests/qapi-schema/doc-good-qapi.texi $@
+       @rm -f tests/qapi-schema/doc-good-qapi-*.[ch] tests/qapi-schema/doc-good-qmp-*.[ch]
 
 tests/test-string-output-visitor$(EXESUF): tests/test-string-output-visitor.o $(test-qapi-obj-y)
 tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(test-qapi-obj-y)
@@ -937,6 +936,7 @@ check-clean:
        $(MAKE) -C tests/tcg clean
        rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
        rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
+       rm -f tests/test-qapi-gen-timestamp
 
 clean: check-clean
 
tests/qapi-schema/test-qapi.py +1 -2
index fe0ca08d78..7772d09919 100644
@@ -10,7 +10,7 @@
 # See the COPYING file in the top-level directory.
 #
 
-from qapi import *
+from qapi.common import *
 from pprint import pprint
 import os
 import sys
--
Quoted 2 lines

2.13.6

good to me otherwise,

#32 lore raw

Re: [Qemu-devel] [PATCH RFC 06/21] qapi-gen: New common driver for code and doc generators

Quoted 7 lines · Marc-Andre Lureau
Quoted 4 lines · Markus Armbruster

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. This is silly. Replace the six programs by a single program that spits out all eleven files.

Now we will need documentation update.

Absolutely! One reason this is RFC.

Quoted 309 lines

Also greping for the renamed files:

git grep -E 'qapi-commands|qapi2texi|qapi-event.py|qapi-intros|qapi-types.py|qapi-visit.py' docs/devel/qapi-code-gen.txt:=== scripts/qapi-types.py ===
docs/devel/qapi-code-gen.txt: $ python scripts/qapi-types.py
--output-dir="qapi-generated" \
docs/devel/qapi-code-gen.txt:=== scripts/qapi-visit.py ===
docs/devel/qapi-code-gen.txt: $ python scripts/qapi-visit.py
--output-dir="qapi-generated"
docs/devel/qapi-code-gen.txt:=== scripts/qapi-commands.py ===
docs/devel/qapi-code-gen.txt: generated by
qapi-visit.py are used to
docs/devel/qapi-code-gen.txt: $ python scripts/qapi-commands.py
--output-dir="qapi-generated"
docs/devel/qapi-code-gen.txt:=== scripts/qapi-event.py ===
docs/devel/qapi-code-gen.txt: $ python scripts/qapi-event.py
--output-dir="qapi-generated"
docs/devel/qapi-code-gen.txt:=== scripts/qapi-introspect.py ===
docs/devel/qapi-code-gen.txt: $ python scripts/qapi-introspect.py
--output-dir="qapi-generated"
monitor.c: * qapi-introspect.py's output actually conforms to the schema. qapi-schema.json:# Documentation generated with qapi2texi.py is in
source order, with

Quoted 285 lines
---
 Makefile                                           | 86 ++++++++++------------
 scripts/qapi-gen.py                                | 41 +++++++++++
 scripts/qapi/__init__.py                           |  0
 scripts/{qapi-commands.py => qapi/commands.py}     | 23 ++----
 scripts/{qapi.py => qapi/common.py}                |  0
 scripts/{qapi2texi.py => qapi/doc.py}              | 29 ++------
 scripts/{qapi-event.py => qapi/events.py}          | 23 ++----
 scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++------
 scripts/{qapi-types.py => qapi/types.py}           | 34 ++-------
 scripts/{qapi-visit.py => qapi/visit.py}           | 34 ++-------
 tests/Makefile.include                             | 56 +++++++-------
 tests/qapi-schema/test-qapi.py                     |  2 +-
 12 files changed, 140 insertions(+), 220 deletions(-)
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (94%)
 rename scripts/{qapi.py => qapi/common.py} (100%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
 mode change 100755 => 100644
 rename scripts/{qapi-event.py => qapi/events.py} (92%)
 rename scripts/{qapi-introspect.py => qapi/introspect.py} (90%)
 rename scripts/{qapi-types.py => qapi/types.py} (90%)
 rename scripts/{qapi-visit.py => qapi/visit.py} (92%)
Makefile +37 -49
index af31e8981f..e02f0c13ef 100644
@@ -92,6 +92,7 @@ GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
 GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
 GENERATED_FILES += qmp-introspect.h
 GENERATED_FILES += qmp-introspect.c
+GENERATED_FILES += qapi.texi
 
 GENERATED_FILES += trace/generated-tcg-tracers.h
 
@@ -477,25 +478,26 @@ qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated
 qemu-keymap$(EXESUF): LIBS += $(XKBCOMMON_LIBS)
 qemu-keymap$(EXESUF): QEMU_CFLAGS += $(XKBCOMMON_CFLAGS)
 
-gen-out-type = $(subst .,-,$(suffix $@))
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py
 
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
-
-qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \
-               $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-               "GEN","$@")
-qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \
-               $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-               "GEN","$@")
-qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
-               $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-               "GEN","$@")
+qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
+qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qapi.texi: \
+qga/qapi-generated/qapi-gen-timestamp ;
+qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
+       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o qga/qapi-generated -p "qga-" $<, \
+               "GEN","$(@:%-timestamp=%)")
+       @>$@
 
 qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/block.json $(SRC_PATH)/qapi/block-core.json \
@@ -512,31 +514,18 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/transaction.json \
                $(SRC_PATH)/qapi/ui.json
 
-qapi-types.c qapi-types.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \
-               $(gen-out-type) -o "." -b $<, \
-               "GEN","$@")
-qapi-visit.c qapi-visit.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \
-               $(gen-out-type) -o "." -b $<, \
-               "GEN","$@")
-qapi-event.c qapi-event.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-event.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \
-               $(gen-out-type) -o "." $<, \
-               "GEN","$@")
-qmp-commands.h qmp-marshal.c :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
-               $(gen-out-type) -o "." $<, \
-               "GEN","$@")
-qmp-introspect.h qmp-introspect.c :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
-               $(gen-out-type) -o "." $<, \
-               "GEN","$@")
+qapi-types.c qapi-types.h \
+qapi-visit.c qapi-visit.h \
+qmp-commands.h qmp-marshal.c \
+qapi-event.c qapi-event.h \
+qmp-introspect.h qmp-introspect.c \
+qapi.texi: \
+qapi-gen-timestamp ;
+qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
+       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o "." -b $<, \
+               "GEN","$(@:%-timestamp=%)")
+       @>$@
 
 QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qmp-commands.h)
 $(qga-obj-y): $(QGALIB_GEN)
@@ -596,6 +585,7 @@ clean:
        rm -f trace/generated-tracers-dtrace.dtrace*
        rm -f trace/generated-tracers-dtrace.h*
        rm -f $(foreach f,$(GENERATED_FILES),$(f) $(f)-timestamp)
+       rm -f qapi-gen-timestamp
        rm -rf qapi-generated
        rm -rf qga/qapi-generated
        for d in $(ALL_SUBDIRS); do \
@@ -803,13 +793,11 @@ qemu-monitor-info.texi: $(SRC_PATH)/hmp-commands-info.hx $(SRC_PATH)/scripts/hxt
 qemu-img-cmds.texi: $(SRC_PATH)/qemu-img-cmds.hx $(SRC_PATH)/scripts/hxtool
        $(call quiet-command,sh $(SRC_PATH)/scripts/hxtool -t < $< > $@,"GEN","$@")
 
-docs/interop/qemu-qmp-qapi.texi docs/interop/qemu-ga-qapi.texi: $(SRC_PATH)/scripts/qapi2texi.py $(qapi-py)
+docs/interop/qemu-qmp-qapi.texi: qapi.texi
+       @cp -p $< $@
 
-docs/interop/qemu-qmp-qapi.texi: $(qapi-modules)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@")
-
-docs/interop/qemu-ga-qapi.texi: $(SRC_PATH)/qga/qapi-schema.json
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@")
+docs/interop/qemu-ga-qapi.texi: qga/qapi-generated/qga-qapi.texi
+       @cp -p $< $@
 
 qemu.1: qemu-doc.texi qemu-options.texi qemu-monitor.texi qemu-monitor-info.texi
 qemu.1: qemu-option-trace.texi
scripts/qapi-gen.py new file +41 -0
new file mode 100755
index 0000000000..575c938a1b
@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+# QAPI generator
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+import sys
+from qapi.common import parse_command_line, QAPISchema
+from qapi.types import gen_types
+from qapi.visit import gen_visit
+from qapi.commands import gen_commands
+from qapi.events import gen_events
+from qapi.introspect import gen_introspect
+from qapi.doc import gen_doc
+
+
+def main(argv):
+    (input_file, output_dir, do_c, do_h, prefix, opts) = \
+        parse_command_line('bu', ['builtins', 'unmask-non-abi-names'])
+
+    opt_builtins = False
+    opt_unmask = False
+
+    for o, a in opts:
+        if o in ('-b', '--builtins'):
+            opt_builtins = True
+        if o in ('-u', '--unmask-non-abi-names'):
+            opt_unmask = True
+
+    schema = QAPISchema(input_file)
+
+    gen_types(schema, output_dir, prefix, opt_builtins)
+    gen_visit(schema, output_dir, prefix, opt_builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, opt_unmask)
+    gen_doc(schema, output_dir, prefix)
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi/__init__.py new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +5 -18
similarity index 94%
rename from scripts/qapi-commands.py
rename to scripts/qapi/commands.py
index 331b58670e..383a4dd426 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
@@ -255,13 +255,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._regy += gen_register_command(name, success_response)
 
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP commands
-'''
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -290,17 +285,9 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
                     prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenCommandVisitor(prefix)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qmp-marshal.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qmp-commands.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qmp-marshal.c')
+    genh.write(output_dir, prefix + 'qmp-commands.h')
scripts/qapi/common.py renamed from scripts/qapi.py +0 -0
similarity index 100%
rename from scripts/qapi.py
rename to scripts/qapi/common.py
scripts/qapi/doc.py renamed from scripts/qapi2texi.py +2 -3
old mode 100755
new mode 100644
similarity index 92%
rename from scripts/qapi2texi.py
rename to scripts/qapi/doc.py
index 924b374cd3..1f57f6e1c2
@@ -4,10 +4,9 @@
 # This work is licensed under the terms of the GNU LGPL, version 2+.
 # See the COPYING file in the top-level directory.
 """This script produces the documentation of a qapi schema in texinfo format"""
+
 import re
-import sys
-
-import qapi
+import qapi.common
 
 MSG_FMT = """
 @deftypefn {type} {{}} {name}
@@ -196,7 +195,7 @@ def texi_entity(doc, what, base=None, variants=None,
             + texi_sections(doc))
 
 
Quoted 5 lines

scripts/qapi/doc.py def texi_entity(doc, what, base=None, variants=None, · line 198

Quoted 2 lines
scripts/qapi/doc.py renamed from scripts/qapi2texi.py +1 -1
-class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
+class QAPISchemaGenDocVisitor(qapi.common.QAPISchemaVisitor):

Would be a bit easier to read and more consitent with a top-level "from qapi.common import QAPISchemaVisitor"

Can do.

Quoted 276 lines
Quoted 276 lines
scripts/qapi/doc.py renamed from scripts/qapi2texi.py +5 -17
     def __init__(self):
         self.out = None
         self.cur_doc = None
@@ -271,20 +270,8 @@ def texi_schema(schema):
     return gen.out
 
 
-def main(argv):
-    """Takes schema argument, prints result to stdout"""
-    if len(argv) != 2:
-        print >>sys.stderr, "%s: need exactly 1 argument: SCHEMA" % argv[0]
-        sys.exit(1)
-
-    schema = qapi.QAPISchema(argv[1])
-    if not qapi.doc_required:
-        print >>sys.stderr, ("%s: need pragma 'doc-required' "
-                             "to generate documentation" % argv[0])
-        sys.exit(1)
-    print '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n'
-    print texi_schema(schema),
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+def gen_doc(schema, output_dir, prefix):
+    if qapi.common.doc_required:
+        gen = qapi.common.QAPIGenDoc()
+        gen.body(texi_schema(schema))
+        gen.write(output_dir, prefix + 'qapi.texi')
scripts/qapi/events.py renamed from scripts/qapi-event.py +5 -18
similarity index 92%
rename from scripts/qapi-event.py
rename to scripts/qapi/events.py
index 5b33c694d4..1f267686db 100644
@@ -12,7 +12,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def build_event_send_proto(name, arg_type, boxed):
@@ -171,13 +171,8 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self._event_names.append(name)
 
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP events
-'''
-
+def gen_events(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP events'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -201,17 +196,9 @@ def main(argv):
 ''',
                     prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenEventVisitor(prefix)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-event.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-event.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-event.c')
+    genh.write(output_dir, prefix + 'qapi-event.h')
scripts/qapi/introspect.py renamed from scripts/qapi-introspect.py +5 -27
similarity index 90%
rename from scripts/qapi-introspect.py
rename to scripts/qapi/introspect.py
index 09e7d1f140..2153060f2c 100644
@@ -10,7 +10,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # Caveman's json.dumps() replacement (we're stuck at Python 2.4)
@@ -168,22 +168,8 @@ const char %(c_name)s[] = %(c_string)s;
         self._gen_json(name, 'event', {'arg-type': self._use_type(arg_type)})
 
 
-def main(argv):
-    # Debugging aid: unmask QAPI schema's type names
-    # We normally mask them, because they're not QMP wire ABI
-    opt_unmask = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('u', ['unmask-non-abi-names'])
-
-    for o, a in opts:
-        if o in ('-u', '--unmask-non-abi-names'):
-            opt_unmask = True
-
-    blurb = '''
- * QAPI/QMP schema introspection
-'''
-
+def gen_introspect(schema, output_dir, prefix, opt_unmask):
+    blurb = ' * QAPI/QMP schema introspection'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -194,17 +180,9 @@ def main(argv):
 ''',
                     prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qmp-introspect.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qmp-introspect.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qmp-introspect.c')
+    genh.write(output_dir, prefix + 'qmp-introspect.h')
scripts/qapi/types.py renamed from scripts/qapi-types.py +5 -29
similarity index 90%
rename from scripts/qapi-types.py
rename to scripts/qapi/types.py
index f2ddde94b1..b2095120e0 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # variants must be emitted before their container; track what has already
@@ -241,24 +241,8 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self._gen_type_cleanup(name)
 
 
-def main(argv):
-    # If you link code generated from multiple schemata, you want only one
-    # instance of the code for built-in types.  Generate it only when
-    # opt_builtins, enabled by command line option -b.  See also
-    # QAPISchemaGenTypeVisitor.visit_end().
-    opt_builtins = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('b', ['builtins'])
-
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-
-    blurb = '''
- * Schema-defined QAPI types
-'''
-
+def gen_types(schema, output_dir, prefix, opt_builtins):
+    blurb = ' * Schema-defined QAPI types'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -274,17 +258,9 @@ def main(argv):
 #include "qapi/util.h"
 '''))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenTypeVisitor(opt_builtins)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-types.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-types.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-types.c')
+    genh.write(output_dir, prefix + 'qapi-types.h')
scripts/qapi/visit.py renamed from scripts/qapi-visit.py +5 -29
similarity index 92%
rename from scripts/qapi-visit.py
rename to scripts/qapi/visit.py
index 473fa72574..80c0b85f8c 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def gen_visit_decl(name, scalar=False):
@@ -324,24 +324,8 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         self.defn += gen_visit_alternate(name, variants)
 
 
-def main(argv):
-    # If you link code generated from multiple schemata, you want only one
-    # instance of the code for built-in types.  Generate it only when
-    # opt_builtins, enabled by command line option -b.  See also
-    # QAPISchemaGenVisitVisitor.visit_end().
-    opt_builtins = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('b', ['builtins'])
-
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-
-    blurb = '''
- * Schema-defined QAPI visitors
-'''
-
+def gen_visit(schema, output_dir, prefix, opt_builtins):
+    blurb = ' * Schema-defined QAPI visitors'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -361,17 +345,9 @@ def main(argv):
 ''',
                     prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenVisitVisitor(opt_builtins)
     schema.visit(vis)
     genc.body(vis.defn)
     genh.body(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-visit.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-visit.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-visit.c')
+    genh.write(output_dir, prefix + 'qapi-visit.h')
tests/Makefile.include +0 -0
index 851aafe9d1..768655a810 100644
@@ -23,7 +23,16 @@ check-help:
 ifneq ($(wildcard config-host.mak),)
 export SRC_PATH
 
Quoted 5 lines

tests/Makefile.include check-help: · line 26

Quoted 2 lines
tests/Makefile.include +1 -1
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
+# TODO don't duplicate $(SRC_PATH)/Makefile's qapi-py here

hmm, it doesn't look necessary, since the makefile is included from there. Right?

That's what I thought, so I deleted it, breaking my build. I still don't quite understand what's happening here.

Quoted 90 lines
Quoted 86 lines
tests/Makefile.include +27 -27
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py
 
 # Get the list of all supported sysemu targets
 SYSEMU_TARGET_LIST := $(subst -softmmu.mak,,$(notdir \
@@ -642,34 +651,24 @@ tests/test-logging$(EXESUF): tests/test-logging.o $(test-util-obj-y)
 tests/test-replication$(EXESUF): tests/test-replication.o $(test-util-obj-y) \
        $(test-block-obj-y)
 
-tests/test-qapi-types.c tests/test-qapi-types.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qapi-visit.c tests/test-qapi-visit.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qmp-commands.h tests/test-qmp-marshal.c :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qapi-event.c tests/test-qapi-event.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-event.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qmp-introspect.c tests/test-qmp-introspect.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
+tests/test-qapi-types.c tests/test-qapi-types.h \
+tests/test-qapi-visit.c tests/test-qapi-visit.h \
+tests/test-qmp-commands.h tests/test-qmp-marshal.c \
+tests/test-qapi-event.c tests/test-qapi-event.h \
+tests/test-qmp-introspect.c tests/test-qmp-introspect.h: \
+tests/test-qapi-gen-timestamp ;
+tests/test-qapi-gen-timestamp: $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(qapi-py)
+       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o tests -p "test-" $<, \
+               "GEN","$(@:%-timestamp=%)")
+       @>$@
 
-tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-good.json $(SRC_PATH)/scripts/qapi2texi.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@")
+tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-good.json $(qapi-py)
+       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o tests/qapi-schema -p "doc-good-" $<, \
+               "GEN","$@")
+       @mv tests/qapi-schema/doc-good-qapi.texi $@
+       @rm -f tests/qapi-schema/doc-good-qapi-*.[ch] tests/qapi-schema/doc-good-qmp-*.[ch]
 
 tests/test-string-output-visitor$(EXESUF): tests/test-string-output-visitor.o $(test-qapi-obj-y)
 tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(test-qapi-obj-y)
@@ -937,6 +936,7 @@ check-clean:
        $(MAKE) -C tests/tcg clean
        rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
        rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
+       rm -f tests/test-qapi-gen-timestamp
 
 clean: check-clean
 
tests/qapi-schema/test-qapi.py +1 -2
index fe0ca08d78..7772d09919 100644
@@ -10,7 +10,7 @@
 # See the COPYING file in the top-level directory.
 #
 
-from qapi import *
+from qapi.common import *
 from pprint import pprint
 import os
 import sys
--
Quoted 2 lines

2.13.6

good to me otherwise,

Quoted 1 line

Thanks!

#33 lore raw

Re: [Qemu-devel] [PATCH RFC 06/21] qapi-gen: New common driver for code and doc generators

Quoted 3 lines · Markus Armbruster
Quoted 1 line · Marc-Andre Lureau

On Fri, Feb 2, 2018 at 2:03 PM, Markus Armbruster <[email protected]> wrote:

[...]

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
scripts/qapi/doc.py renamed from scripts/qapi2texi.py +2 -3
old mode 100755
new mode 100644
similarity index 92%
rename from scripts/qapi2texi.py
rename to scripts/qapi/doc.py
index 924b374cd3..1f57f6e1c2
@@ -4,10 +4,9 @@
 # This work is licensed under the terms of the GNU LGPL, version 2+.
 # See the COPYING file in the top-level directory.
 """This script produces the documentation of a qapi schema in texinfo format"""
+
 import re
-import sys
-
-import qapi
+import qapi.common
 
 MSG_FMT = """
 @deftypefn {type} {{}} {name}
@@ -196,7 +195,7 @@ def texi_entity(doc, what, base=None, variants=None,
             + texi_sections(doc))
 
 
Quoted 7 lines
Quoted 5 lines

scripts/qapi/doc.py def texi_entity(doc, what, base=None, variants=None, · line 198

Quoted 2 lines
scripts/qapi/doc.py renamed from scripts/qapi2texi.py +1 -1
-class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
+class QAPISchemaGenDocVisitor(qapi.common.QAPISchemaVisitor):

Would be a bit easier to read and more consitent with a top-level "from qapi.common import QAPISchemaVisitor"

Can do.

The obvious patch (appended) doesn't work, because doc_required is always False in gen_doc(). WTF?!?

[...]

scripts/qapi/doc.py +4 -4
index 4027722032..919e77b79e 100644
@@ -7,7 +7,7 @@
 
 from __future__ import print_function
 import re
-import qapi.common
+from qapi.common import doc_required, QAPIGenDoc, QAPISchemaVisitor
 
 MSG_FMT = """
 @deftypefn {type} {{}} {name}
@@ -196,7 +196,7 @@ def texi_entity(doc, what, base=None, variants=None,
             + texi_sections(doc))
 
 
-class QAPISchemaGenDocVisitor(qapi.common.QAPISchemaVisitor):
+class QAPISchemaGenDocVisitor(QAPISchemaVisitor):
     def __init__(self):
         self.out = None
         self.cur_doc = None
@@ -272,7 +272,7 @@ def texi_schema(schema):
 
 
 def gen_doc(schema, output_dir, prefix):
-    if qapi.common.doc_required:
-        gen = qapi.common.QAPIGenDoc()
+    if doc_required:
+        gen = QAPIGenDoc()
         gen.add(texi_schema(schema))
         gen.write(output_dir, prefix + 'qapi.texi')
#34 lore raw

RFC PATCH 7/21 qapi: Move parse_command_line() next to its only use

In reply to Markus Armbruster
---
 scripts/qapi-gen.py    | 52 +++++++++++++++++++++++++++++++++++++++++++++++-
 scripts/qapi/common.py | 54 --------------------------------------------------
 2 files changed, 51 insertions(+), 55 deletions(-)
scripts/qapi-gen.py +51 -1
index 575c938a1b..6302fd0d55 100755
@@ -4,8 +4,10 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+import getopt
+import re
 import sys
-from qapi.common import parse_command_line, QAPISchema
+from qapi.common import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 from qapi.commands import gen_commands
@@ -14,6 +16,54 @@ from qapi.introspect import gen_introspect
 from qapi.doc import gen_doc
 
 
+def parse_command_line(extra_options='', extra_long_options=[]):
+
+    try:
+        opts, args = getopt.gnu_getopt(sys.argv[1:],
+                                       'chp:o:' + extra_options,
+                                       ['source', 'header', 'prefix=',
+                                        'output-dir='] + extra_long_options)
+    except getopt.GetoptError as err:
+        print >>sys.stderr, "%s: %s" % (sys.argv[0], str(err))
+        sys.exit(1)
+
+    output_dir = ''
+    prefix = ''
+    do_c = False
+    do_h = False
+    extra_opts = []
+
+    for oa in opts:
+        o, a = oa
+        if o in ('-p', '--prefix'):
+            match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)
+            if match.end() != len(a):
+                print >>sys.stderr, \
+                    "%s: 'funny character '%s' in argument of --prefix" \
+                    % (sys.argv[0], a[match.end()])
+                sys.exit(1)
+            prefix = a
+        elif o in ('-o', '--output-dir'):
+            output_dir = a + '/'
+        elif o in ('-c', '--source'):
+            do_c = True
+        elif o in ('-h', '--header'):
+            do_h = True
+        else:
+            extra_opts.append(oa)
+
+    if not do_c and not do_h:
+        do_c = True
+        do_h = True
+
+    if len(args) != 1:
+        print >>sys.stderr, "%s: need exactly one argument" % sys.argv[0]
+        sys.exit(1)
+    fname = args[0]
+
+    return (fname, output_dir, do_c, do_h, prefix, extra_opts)
+
+
 def main(argv):
     (input_file, output_dir, do_c, do_h, prefix, opts) = \
         parse_command_line('bu', ['builtins', 'unmask-non-abi-names'])
scripts/qapi/common.py +0 -54

66 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v1 7/21
#35 lore raw

Re: [Qemu-devel] [PATCH RFC 07/21] qapi: Move parse_command_line() next to its only use

Quoted 6 lines · Markus Armbruster
---
 scripts/qapi-gen.py    | 52 +++++++++++++++++++++++++++++++++++++++++++++++-
 scripts/qapi/common.py | 54 --------------------------------------------------
 2 files changed, 51 insertions(+), 55 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by Marc-André Lureau on v1 7/21
#36 lore raw

Re: [Qemu-devel] [PATCH RFC 07/21] qapi: Move parse_command_line() next to its only use

In reply to Markus Armbruster
Quoted 1 line · Markus Armbruster
Quoted 151 lines
---
 scripts/qapi-gen.py    | 52 +++++++++++++++++++++++++++++++++++++++++++++++-
 scripts/qapi/common.py | 54 --------------------------------------------------
 2 files changed, 51 insertions(+), 55 deletions(-)
scripts/qapi-gen.py +51 -1
index 575c938a1b..6302fd0d55 100755
@@ -4,8 +4,10 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+import getopt
+import re
 import sys
-from qapi.common import parse_command_line, QAPISchema
+from qapi.common import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 from qapi.commands import gen_commands
@@ -14,6 +16,54 @@ from qapi.introspect import gen_introspect
 from qapi.doc import gen_doc
 
 
+def parse_command_line(extra_options='', extra_long_options=[]):
+
+    try:
+        opts, args = getopt.gnu_getopt(sys.argv[1:],
+                                       'chp:o:' + extra_options,
+                                       ['source', 'header', 'prefix=',
+                                        'output-dir='] + extra_long_options)
+    except getopt.GetoptError as err:
+        print >>sys.stderr, "%s: %s" % (sys.argv[0], str(err))
+        sys.exit(1)
+
+    output_dir = ''
+    prefix = ''
+    do_c = False
+    do_h = False
+    extra_opts = []
+
+    for oa in opts:
+        o, a = oa
+        if o in ('-p', '--prefix'):
+            match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)
+            if match.end() != len(a):
+                print >>sys.stderr, \
+                    "%s: 'funny character '%s' in argument of --prefix" \
+                    % (sys.argv[0], a[match.end()])
+                sys.exit(1)
+            prefix = a
+        elif o in ('-o', '--output-dir'):
+            output_dir = a + '/'
+        elif o in ('-c', '--source'):
+            do_c = True
+        elif o in ('-h', '--header'):
+            do_h = True
+        else:
+            extra_opts.append(oa)
+
+    if not do_c and not do_h:
+        do_c = True
+        do_h = True
+
+    if len(args) != 1:
+        print >>sys.stderr, "%s: need exactly one argument" % sys.argv[0]
+        sys.exit(1)
+    fname = args[0]
+
+    return (fname, output_dir, do_c, do_h, prefix, extra_opts)
+
+
 def main(argv):
     (input_file, output_dir, do_c, do_h, prefix, opts) = \
         parse_command_line('bu', ['builtins', 'unmask-non-abi-names'])
scripts/qapi/common.py +0 -55
index d73ef618e2..cfa2671ca3 100644
@@ -12,7 +12,6 @@
 # See the COPYING file in the top-level directory.
 
 import errno
-import getopt
 import os
 import re
 import string
@@ -1917,59 +1916,6 @@ def build_params(arg_type, boxed, extra):
 
 
 #
-# Common command line parsing
-#
-
-
-def parse_command_line(extra_options='', extra_long_options=[]):
-
-    try:
-        opts, args = getopt.gnu_getopt(sys.argv[1:],
-                                       'chp:o:' + extra_options,
-                                       ['source', 'header', 'prefix=',
-                                        'output-dir='] + extra_long_options)
-    except getopt.GetoptError as err:
-        print >>sys.stderr, "%s: %s" % (sys.argv[0], str(err))
-        sys.exit(1)
-
-    output_dir = ''
-    prefix = ''
-    do_c = False
-    do_h = False
-    extra_opts = []
-
-    for oa in opts:
-        o, a = oa
-        if o in ('-p', '--prefix'):
-            match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)
-            if match.end() != len(a):
-                print >>sys.stderr, \
-                    "%s: 'funny character '%s' in argument of --prefix" \
-                    % (sys.argv[0], a[match.end()])
-                sys.exit(1)
-            prefix = a
-        elif o in ('-o', '--output-dir'):
-            output_dir = a + '/'
-        elif o in ('-c', '--source'):
-            do_c = True
-        elif o in ('-h', '--header'):
-            do_h = True
-        else:
-            extra_opts.append(oa)
-
-    if not do_c and not do_h:
-        do_c = True
-        do_h = True
-
-    if len(args) != 1:
-        print >>sys.stderr, "%s: need exactly one argument" % sys.argv[0]
-        sys.exit(1)
-    fname = args[0]
-
-    return (fname, output_dir, do_c, do_h, prefix, extra_opts)
-
-
-#
 # Accumulate and write output
 #
 
--

2.13.6

#37 lore raw

RFC PATCH 8/21 qapi: Touch generated files only when they change

In reply to Markus Armbruster

A massive number of objects depends on QAPI-generated headers. In my "build everything" tree, it's roughly 4500 out of 4800. This is particularly annoying when only some of the generated files change, say for a doc fix.

Improve qapi-gen.py to touch its output files only if they actually change. Rebuild time for a QAPI doc fix drops from many minutes to a few seconds. Rebuilds get faster for certain code changes, too. For instance, adding a simple QMP event now recompiles less than 200 instead of 4500 objects. But adding a QAPI type is as bad as ever; we clearly got more work to do.

---
 scripts/qapi/common.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
scripts/qapi/common.py +9 -2
index cfa2671ca3..be0fcc548a 100644
@@ -1944,9 +1944,16 @@ class QAPIGen(object):
             except os.error as e:
                 if e.errno != errno.EEXIST:
                     raise
-        f = open(os.path.join(output_dir, fname), 'w')
-        f.write(self.top(fname) + self._preamble + self._body
+        fd = os.open(os.path.join(output_dir, fname),
+                     os.O_RDWR | os.O_CREAT, 0666)
+        f = os.fdopen(fd, 'r+')
+        text = (self.top(fname) + self._preamble + self._body
                 + self.bottom(fname))
+        oldtext = f.read(len(text) + 1)
+        if text != oldtext:
+            f.seek(0)
+            f.truncate(0)
+            f.write(text)
         f.close()
 
 
Signature · 1 line
2.13.6
Reviewed-by on v1 8/21
#38 lore raw

Re: [Qemu-devel] [PATCH RFC 08/21] qapi: Touch generated files only when they change

Quoted 11 lines · Markus Armbruster

A massive number of objects depends on QAPI-generated headers. In my "build everything" tree, it's roughly 4500 out of 4800. This is particularly annoying when only some of the generated files change, say for a doc fix.

Improve qapi-gen.py to touch its output files only if they actually change. Rebuild time for a QAPI doc fix drops from many minutes to a few seconds. Rebuilds get faster for certain code changes, too. For instance, adding a simple QMP event now recompiles less than 200 instead of 4500 objects. But adding a QAPI type is as bad as ever; we clearly got more work to do.

The last phrase sounds quite colloquial. It may have been intentional; but if not, s/we/we've/

Quoted 16 lines
---
 scripts/qapi/common.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
scripts/qapi/common.py +0 -2
index cfa2671ca3..be0fcc548a 100644
@@ -1944,9 +1944,16 @@ class QAPIGen(object):
             except os.error as e:
                 if e.errno != errno.EEXIST:
                     raise
-        f = open(os.path.join(output_dir, fname), 'w')
-        f.write(self.top(fname) + self._preamble + self._body

scripts/qapi/common.py class QAPIGen(object): · line 1956

Quoted 10 lines
scripts/qapi/common.py +9 -0
+        fd = os.open(os.path.join(output_dir, fname),
+                     os.O_RDWR | os.O_CREAT, 0666)
+        f = os.fdopen(fd, 'r+')
+        text = (self.top(fname) + self._preamble + self._body
                 + self.bottom(fname))
+        oldtext = f.read(len(text) + 1)
+        if text != oldtext:
+            f.seek(0)
+            f.truncate(0)
+            f.write(text)

In-memory rewrite rather than playing games with a secondary file combined with the rename(2) syscall. It means you're not atomic (an external reader has a window of time where they can see an incomplete version of the file). But I think make dependency rules mean we don't care - even in a parallel rule, as long as all clients depend on the timestamp file, and the timestamp file isn't updated until all our rewrites (if any) have settled, then we don't create any of those external readers that can find the window of incomplete contents.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#39 lore raw

Re: [Qemu-devel] [PATCH RFC 08/21] qapi: Touch generated files only when they change

Quoted 13 lines · Eric Blake
Quoted 11 lines · Markus Armbruster

A massive number of objects depends on QAPI-generated headers. In my "build everything" tree, it's roughly 4500 out of 4800. This is particularly annoying when only some of the generated files change, say for a doc fix.

Improve qapi-gen.py to touch its output files only if they actually change. Rebuild time for a QAPI doc fix drops from many minutes to a few seconds. Rebuilds get faster for certain code changes, too. For instance, adding a simple QMP event now recompiles less than 200 instead of 4500 objects. But adding a QAPI type is as bad as ever; we clearly got more work to do.

Quoted 2 lines

The last phrase sounds quite colloquial. It may have been intentional; but if not, s/we/we've/

I'll change it.

Quoted 22 lines
Quoted 22 lines
---
 scripts/qapi/common.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
scripts/qapi/common.py +6 -2
index cfa2671ca3..be0fcc548a 100644
@@ -1944,9 +1944,16 @@ class QAPIGen(object):
             except os.error as e:
                 if e.errno != errno.EEXIST:
                     raise
-        f = open(os.path.join(output_dir, fname), 'w')
-        f.write(self.top(fname) + self._preamble + self._body
+        fd = os.open(os.path.join(output_dir, fname),
+                     os.O_RDWR | os.O_CREAT, 0666)
+        f = os.fdopen(fd, 'r+')
+        text = (self.top(fname) + self._preamble + self._body
                 + self.bottom(fname))
+        oldtext = f.read(len(text) + 1)
+        if text != oldtext:
Quoted 8 lines

scripts/qapi/common.py class QAPIGen(object): · line 1956

Quoted 3 lines
scripts/qapi/common.py +3 -0
+            f.seek(0)
+            f.truncate(0)
+            f.write(text)

In-memory rewrite rather than playing games with a secondary file combined with the rename(2) syscall. It means you're not atomic (an external reader has a window of time where they can see an incomplete version of the file).

Same as before.

Quoted 7 lines

But I think make dependency rules mean we don't
care - even in a parallel rule, as long as all clients depend on the timestamp file, and the timestamp file isn't updated until all our rewrites (if any) have settled, then we don't create any of those external readers that can find the window of incomplete contents.

Thanks!

#40 lore raw

RFC PATCH 9/21 qapi: Don't absolutize include file name in error messages

In reply to Markus Armbruster

Error messages print absolute filenames of included files even gave a relative one on the command line:

PYTHONPATH=scripts python -B tests/qapi-schema/test-qapi.py tests/qapi-schema/include-cycle.json
In file included from tests/qapi-schema/include-cycle.json:1:
In file included from /work/armbru/qemu/tests/qapi-schema/include-cycle-b.json:1:
/work/armbru/qemu/tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Improve this to

In file included from tests/qapi-schema/include-cycle.json:1:
In file included from tests/qapi-schema/include-cycle-b.json:1:
tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

---
 scripts/qapi/common.py                | 12 ++++++------
 tests/qapi-schema/include-no-file.err |  2 +-
 2 files changed, 7 insertions(+), 7 deletions(-)
scripts/qapi/common.py +6 -6
index be0fcc548a..6c6962a364 100644
@@ -255,9 +255,8 @@ class QAPIDoc(object):
 class QAPISchemaParser(object):
 
     def __init__(self, fp, previously_included=[], incl_info=None):
-        abs_fname = os.path.abspath(fp.name)
         self.fname = fp.name
-        previously_included.append(abs_fname)
+        previously_included.append(os.path.abspath(fp.name))
         self.incl_info = incl_info
         self.src = fp.read()
         if self.src == '' or self.src[-1] != '\n':
@@ -288,7 +287,7 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                self._include(include, info, os.path.dirname(abs_fname),
+                self._include(include, info, os.path.dirname(self.fname),
                               previously_included)
             elif "pragma" in expr:
                 self.reject_expr_doc(cur_doc)
@@ -321,7 +320,8 @@ class QAPISchemaParser(object):
                 % doc.symbol)
 
     def _include(self, include, info, base_dir, previously_included):
-        incl_abs_fname = os.path.join(base_dir, include)
+        incl_fname = os.path.join(base_dir, include)
+        incl_abs_fname = os.path.abspath(incl_fname)
         # catch inclusion cycle
         inf = info
         while inf:
@@ -333,9 +333,9 @@ class QAPISchemaParser(object):
         if incl_abs_fname in previously_included:
             return
         try:
-            fobj = open(incl_abs_fname, 'r')
+            fobj = open(incl_fname, 'r')
         except IOError as e:
-            raise QAPISemError(info, '%s: %s' % (e.strerror, include))
+            raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname))
         exprs_include = QAPISchemaParser(fobj, previously_included, info)
         self.exprs.extend(exprs_include.exprs)
         self.docs.extend(exprs_include.docs)
tests/qapi-schema/include-no-file.err +1 -1
index d5b9b22d85..e42bcf4bc1 100644
@@ -1 +1 @@
-tests/qapi-schema/include-no-file.json:1: No such file or directory: include-no-file-sub.json
+tests/qapi-schema/include-no-file.json:1: No such file or directory: tests/qapi-schema/include-no-file-sub.json
Signature · 1 line
2.13.6
Reviewed-by on v1 9/21
#41 lore raw

Re: [Qemu-devel] [PATCH RFC 09/21] qapi: Don't absolutize include file name in error messages

Quoted 1 line · Markus Armbruster

Error messages print absolute filenames of included files even gave a

s/even gave/even when given/

Quoted 7 lines

relative one on the command line:

PYTHONPATH=scripts python -B tests/qapi-schema/test-qapi.py tests/qapi-schema/include-cycle.json
In file included from tests/qapi-schema/include-cycle.json:1:
In file included from /work/armbru/qemu/tests/qapi-schema/include-cycle-b.json:1:
/work/armbru/qemu/tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Quoted 5 lines

Improve this to

In file included from tests/qapi-schema/include-cycle.json:1:
In file included from tests/qapi-schema/include-cycle-b.json:1:
tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Nice, and makes developing new qapi tests a little less painful since it's less modification to qapi-schema/*.err additions.

Quoted 7 lines
---
 scripts/qapi/common.py                | 12 ++++++------
 tests/qapi-schema/include-no-file.err |  2 +-
 2 files changed, 7 insertions(+), 7 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#42 lore raw

Re: [Qemu-devel] [PATCH RFC 09/21] qapi: Don't absolutize include file name in error messages

Quoted 4 lines · Eric Blake
Quoted 1 line · Markus Armbruster

Error messages print absolute filenames of included files even gave a

s/even gave/even when given/

I meant to write "even if the user gave". Is that okay?

Quoted 7 lines
Quoted 7 lines

relative one on the command line:

PYTHONPATH=scripts python -B tests/qapi-schema/test-qapi.py tests/qapi-schema/include-cycle.json
In file included from tests/qapi-schema/include-cycle.json:1:
In file included from /work/armbru/qemu/tests/qapi-schema/include-cycle-b.json:1:
/work/armbru/qemu/tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Quoted 8 lines
Quoted 5 lines

Improve this to

In file included from tests/qapi-schema/include-cycle.json:1:
In file included from tests/qapi-schema/include-cycle-b.json:1:
tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Nice, and makes developing new qapi tests a little less painful since it's less modification to qapi-schema/*.err additions.

Probably not, as our make rule strips off $(SRC_PATH):

	@perl -p -e 's|\Q$(SRC_PATH)\E/||g' $*.test.err | diff -q $(SRC_PATH)/$*.err -

I've kept that, because it might also occur in stack backtraces. I think.

Quoted 8 lines
Quoted 6 lines
---
 scripts/qapi/common.py                | 12 ++++++------
 tests/qapi-schema/include-no-file.err |  2 +-
 2 files changed, 7 insertions(+), 7 deletions(-)

Thanks!

#43 lore raw

Re: [Qemu-devel] [PATCH RFC 09/21] qapi: Don't absolutize include file name in error messages

Quoted 8 lines · Markus Armbruster
Quoted 4 lines · Eric Blake
Quoted 1 line · Markus Armbruster

Error messages print absolute filenames of included files even gave a

s/even gave/even when given/

I meant to write "even if the user gave". Is that okay?

Yes, that works too.

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines

relative one on the command line:

PYTHONPATH=scripts python -B tests/qapi-schema/test-qapi.py tests/qapi-schema/include-cycle.json
In file included from tests/qapi-schema/include-cycle.json:1:
In file included from /work/armbru/qemu/tests/qapi-schema/include-cycle-b.json:1:
/work/armbru/qemu/tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Quoted 8 lines
Quoted 5 lines

Improve this to

In file included from tests/qapi-schema/include-cycle.json:1:
In file included from tests/qapi-schema/include-cycle-b.json:1:
tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Nice, and makes developing new qapi tests a little less painful since it's less modification to qapi-schema/*.err additions.

Quoted 6 lines

Probably not, as our make rule strips off $(SRC_PATH):

	@perl -p -e 's|\Q$(SRC_PATH)\E/||g' $*.test.err | diff -q $(SRC_PATH)/$*.err -

I've kept that, because it might also occur in stack backtraces. I think.

I still recall having to hand-edit .err files when doing a naive "run the test to get the failure, then 'mv' the bad files into the expected filenames, then rerun the tests"; so I'm not sure if our make rule was properly munging absolute names out of the file in the first place. I'm not too fussed about it, though, as adding new tests is less frequent and it's still fairly easy to learn if you did it right or not by whether 'make check' succeeds.

Quoted 12 lines
Quoted 9 lines
Quoted 7 lines
---
 scripts/qapi/common.py                | 12 ++++++------
 tests/qapi-schema/include-no-file.err |  2 +-
 2 files changed, 7 insertions(+), 7 deletions(-)

Thanks!

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#44 lore raw

Re: [Qemu-devel] [PATCH RFC 09/21] qapi: Don't absolutize include file name in error messages

Quoted 31 lines · Eric Blake
Quoted 8 lines · Markus Armbruster
Quoted 4 lines · Eric Blake
Quoted 1 line · Markus Armbruster

Error messages print absolute filenames of included files even gave a

s/even gave/even when given/

I meant to write "even if the user gave". Is that okay?

Yes, that works too.

Quoted 19 lines
Quoted 8 lines
Quoted 8 lines

relative one on the command line:

PYTHONPATH=scripts python -B tests/qapi-schema/test-qapi.py tests/qapi-schema/include-cycle.json
In file included from tests/qapi-schema/include-cycle.json:1:
In file included from /work/armbru/qemu/tests/qapi-schema/include-cycle-b.json:1:
/work/armbru/qemu/tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Quoted 8 lines
Quoted 5 lines

Improve this to

In file included from tests/qapi-schema/include-cycle.json:1:
In file included from tests/qapi-schema/include-cycle-b.json:1:
tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Nice, and makes developing new qapi tests a little less painful since it's less modification to qapi-schema/*.err additions.

Probably not, as our make rule strips off $(SRC_PATH):

Quoted 8 lines
Quoted 4 lines
	@perl -p -e 's|\Q$(SRC_PATH)\E/||g' $*.test.err | diff -q $(SRC_PATH)/$*.err -

I've kept that, because it might also occur in stack backtraces. I think.

I still recall having to hand-edit .err files when doing a naive "run the test to get the failure, then 'mv' the bad files into the expected filenames, then rerun the tests";

Whenever the Perl script does something, you can't simply move the .test.err to .err. Annoying.

A make target check-accept could automate the work.

However, with this patch, the Perl script should do something less often. I guess that's your point.

Sorry for being slow on the update :)

Quoted 17 lines

so I'm not sure if our make rule was
properly munging absolute names out of the file in the first place. I'm not too fussed about it, though, as adding new tests is less frequent and it's still fairly easy to learn if you did it right or not by whether 'make check' succeeds.

Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
---
 scripts/qapi/common.py                | 12 ++++++------
 tests/qapi-schema/include-no-file.err |  2 +-
 2 files changed, 7 insertions(+), 7 deletions(-)

Thanks!

#45 lore raw

Re: [Qemu-devel] [PATCH RFC 09/21] qapi: Don't absolutize include file name in error messages

Quoted 6 lines · Markus Armbruster
Quoted 6 lines
Quoted 6 lines

Probably not, as our make rule strips off $(SRC_PATH):

	@perl -p -e 's|\Q$(SRC_PATH)\E/||g' $*.test.err | diff -q $(SRC_PATH)/$*.err -

I've kept that, because it might also occur in stack backtraces. I think.

Ideally, we don't have any stack backtraces ;) But we have indeed historically checked in .err files with a backtrace that a later patch in the series cleans up, so that part of the munging does indeed play a role in a clean 'make check' for documenting an expected weakness in the generator.

Quoted 7 lines
Quoted 4 lines

I still recall having to hand-edit .err files when doing a naive "run the test to get the failure, then 'mv' the bad files into the expected filenames, then rerun the tests";

Whenever the Perl script does something, you can't simply move the .test.err to .err. Annoying.

We could also munge the original output prior to creating .test.err; but that also has drawbacks if it eats something that would have been useful for debugging the test.

Quoted 2 lines

A make target check-accept could automate the work.

Yeah, but I'm not sure how often we'd need to invoke it, or even remember that it exists. But it would at least avoid the issue of having to debug a munged .test.err.

Quoted 5 lines

However, with this patch, the Perl script should do something less often. I guess that's your point.

Sorry for being slow on the update :)

No problem; I figure you've got a lot on your mind as you are trying to tie down loose ends before some much deserved time off.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by Marc-André Lureau on v1 9/21
#46 lore raw

Re: [Qemu-devel] [PATCH RFC 09/21] qapi: Don't absolutize include file name in error messages

In reply to Markus Armbruster
Quoted 8 lines · Markus Armbruster

Error messages print absolute filenames of included files even gave a relative one on the command line:

PYTHONPATH=scripts python -B tests/qapi-schema/test-qapi.py tests/qapi-schema/include-cycle.json
In file included from tests/qapi-schema/include-cycle.json:1:
In file included from /work/armbru/qemu/tests/qapi-schema/include-cycle-b.json:1:
/work/armbru/qemu/tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Quoted 7 lines

Improve this to

In file included from tests/qapi-schema/include-cycle.json:1:
In file included from tests/qapi-schema/include-cycle-b.json:1:
tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

most of the necessary refactoring/split is done in previous patch,

Quoted 61 lines
---
 scripts/qapi/common.py                | 12 ++++++------
 tests/qapi-schema/include-no-file.err |  2 +-
 2 files changed, 7 insertions(+), 7 deletions(-)
scripts/qapi/common.py +6 -6
index be0fcc548a..6c6962a364 100644
@@ -255,9 +255,8 @@ class QAPIDoc(object):
 class QAPISchemaParser(object):
 
     def __init__(self, fp, previously_included=[], incl_info=None):
-        abs_fname = os.path.abspath(fp.name)
         self.fname = fp.name
-        previously_included.append(abs_fname)
+        previously_included.append(os.path.abspath(fp.name))
         self.incl_info = incl_info
         self.src = fp.read()
         if self.src == '' or self.src[-1] != '\n':
@@ -288,7 +287,7 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                self._include(include, info, os.path.dirname(abs_fname),
+                self._include(include, info, os.path.dirname(self.fname),
                               previously_included)
             elif "pragma" in expr:
                 self.reject_expr_doc(cur_doc)
@@ -321,7 +320,8 @@ class QAPISchemaParser(object):
                 % doc.symbol)
 
     def _include(self, include, info, base_dir, previously_included):
-        incl_abs_fname = os.path.join(base_dir, include)
+        incl_fname = os.path.join(base_dir, include)
+        incl_abs_fname = os.path.abspath(incl_fname)
         # catch inclusion cycle
         inf = info
         while inf:
@@ -333,9 +333,9 @@ class QAPISchemaParser(object):
         if incl_abs_fname in previously_included:
             return
         try:
-            fobj = open(incl_abs_fname, 'r')
+            fobj = open(incl_fname, 'r')
         except IOError as e:
-            raise QAPISemError(info, '%s: %s' % (e.strerror, include))
+            raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname))
         exprs_include = QAPISchemaParser(fobj, previously_included, info)
         self.exprs.extend(exprs_include.exprs)
         self.docs.extend(exprs_include.docs)
tests/qapi-schema/include-no-file.err +1 -2
index d5b9b22d85..e42bcf4bc1 100644
@@ -1 +1 @@
-tests/qapi-schema/include-no-file.json:1: No such file or directory: include-no-file-sub.json
+tests/qapi-schema/include-no-file.json:1: No such file or directory: tests/qapi-schema/include-no-file-sub.json
--

2.13.6

#47 lore raw

RFC PATCH 10/21 qapi/common: Eliminate QAPISchema.exprs

In reply to Markus Armbruster
---
 scripts/qapi/common.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +4 -4
index 6c6962a364..78e960d07c 100644
@@ -1457,13 +1457,13 @@ class QAPISchema(object):
     def __init__(self, fname):
         try:
             parser = QAPISchemaParser(open(fname, 'r'))
-            self.exprs = check_exprs(parser.exprs)
+            exprs = check_exprs(parser.exprs)
             self.docs = parser.docs
             self._entity_dict = {}
             self._predefining = True
             self._def_predefineds()
             self._predefining = False
-            self._def_exprs()
+            self._def_exprs(exprs)
             self.check()
         except QAPIError as err:
             print >>sys.stderr, err
@@ -1648,8 +1648,8 @@ class QAPISchema(object):
                 name, info, doc, 'arg', self._make_members(data, info))
         self._def_entity(QAPISchemaEvent(name, info, doc, data, boxed))
 
-    def _def_exprs(self):
-        for expr_elem in self.exprs:
+    def _def_exprs(self, exprs):
+        for expr_elem in exprs:
             expr = expr_elem['expr']
             info = expr_elem['info']
             doc = expr_elem.get('doc')
Signature · 1 line
2.13.6
Reviewed-by on v1 10/21
#48 lore raw

Re: [Qemu-devel] [PATCH RFC 10/21] qapi/common: Eliminate QAPISchema.exprs

Quoted 5 lines · Markus Armbruster
---
 scripts/qapi/common.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by Marc-André Lureau on v1 10/21
#49 lore raw

Re: [Qemu-devel] [PATCH RFC 10/21] qapi/common: Eliminate QAPISchema.exprs

In reply to Markus Armbruster
Quoted 1 line · Markus Armbruster
Quoted 38 lines
---
 scripts/qapi/common.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +4 -5
index 6c6962a364..78e960d07c 100644
@@ -1457,13 +1457,13 @@ class QAPISchema(object):
     def __init__(self, fname):
         try:
             parser = QAPISchemaParser(open(fname, 'r'))
-            self.exprs = check_exprs(parser.exprs)
+            exprs = check_exprs(parser.exprs)
             self.docs = parser.docs
             self._entity_dict = {}
             self._predefining = True
             self._def_predefineds()
             self._predefining = False
-            self._def_exprs()
+            self._def_exprs(exprs)
             self.check()
         except QAPIError as err:
             print >>sys.stderr, err
@@ -1648,8 +1648,8 @@ class QAPISchema(object):
                 name, info, doc, 'arg', self._make_members(data, info))
         self._def_entity(QAPISchemaEvent(name, info, doc, data, boxed))
 
-    def _def_exprs(self):
-        for expr_elem in self.exprs:
+    def _def_exprs(self, exprs):
+        for expr_elem in exprs:
             expr = expr_elem['expr']
             info = expr_elem['info']
             doc = expr_elem.get('doc')
--

2.13.6

#50 lore raw

RFC PATCH 11/21 qapi: Lift error reporting from QAPISchema.__init__() to callers

In reply to Markus Armbruster
---
 scripts/qapi-gen.py            |  8 ++++++--
 scripts/qapi/common.py         | 23 +++++++++--------------
 tests/qapi-schema/test-qapi.py |  8 +++++++-
 3 files changed, 22 insertions(+), 17 deletions(-)
scripts/qapi-gen.py +6 -2
index 6302fd0d55..ba82ca92cc 100755
@@ -7,7 +7,7 @@
 import getopt
 import re
 import sys
-from qapi.common import QAPISchema
+from qapi.common import QAPIError, QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 from qapi.commands import gen_commands
@@ -77,7 +77,11 @@ def main(argv):
         if o in ('-u', '--unmask-non-abi-names'):
             opt_unmask = True
 
-    schema = QAPISchema(input_file)
+    try:
+        schema = QAPISchema(input_file)
+    except QAPIError as err:
+        print >>sys.stderr, err
+        exit(1)
 
     gen_types(schema, output_dir, prefix, opt_builtins)
     gen_visit(schema, output_dir, prefix, opt_builtins)
scripts/qapi/common.py +9 -14
index 78e960d07c..d334e1db5a 100644
@@ -15,7 +15,6 @@ import errno
 import os
 import re
 import string
-import sys
 from ordereddict import OrderedDict
 
 builtin_types = {
@@ -1455,19 +1454,15 @@ class QAPISchemaEvent(QAPISchemaEntity):
 
 class QAPISchema(object):
     def __init__(self, fname):
-        try:
-            parser = QAPISchemaParser(open(fname, 'r'))
-            exprs = check_exprs(parser.exprs)
-            self.docs = parser.docs
-            self._entity_dict = {}
-            self._predefining = True
-            self._def_predefineds()
-            self._predefining = False
-            self._def_exprs(exprs)
-            self.check()
-        except QAPIError as err:
-            print >>sys.stderr, err
-            exit(1)
+        parser = QAPISchemaParser(open(fname, 'r'))
+        exprs = check_exprs(parser.exprs)
+        self.docs = parser.docs
+        self._entity_dict = {}
+        self._predefining = True
+        self._def_predefineds()
+        self._predefining = False
+        self._def_exprs(exprs)
+        self.check()
 
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
tests/qapi-schema/test-qapi.py +7 -1
index 7772d09919..d6bb8ec6a4 100644
@@ -53,7 +53,13 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
             for v in variants.variants:
                 print '    case %s: %s' % (v.name, v.type.name)
 
-schema = QAPISchema(sys.argv[1])
+
+try:
+    schema = QAPISchema(sys.argv[1])
+except QAPIError as err:
+    print >>sys.stderr, err
+    exit(1)
+
 schema.visit(QAPISchemaTestVisitor())
 
 for doc in schema.docs:
Signature · 1 line
2.13.6
Reviewed-by Marc-André Lureau on v1 11/21
#51 lore raw

Re: [Qemu-devel] [PATCH RFC 11/21] qapi: Lift error reporting from QAPISchema.__init__() to callers

Quoted 1 line · Markus Armbruster
Quoted 95 lines
---
 scripts/qapi-gen.py            |  8 ++++++--
 scripts/qapi/common.py         | 23 +++++++++--------------
 tests/qapi-schema/test-qapi.py |  8 +++++++-
 3 files changed, 22 insertions(+), 17 deletions(-)
scripts/qapi-gen.py +6 -2
index 6302fd0d55..ba82ca92cc 100755
@@ -7,7 +7,7 @@
 import getopt
 import re
 import sys
-from qapi.common import QAPISchema
+from qapi.common import QAPIError, QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 from qapi.commands import gen_commands
@@ -77,7 +77,11 @@ def main(argv):
         if o in ('-u', '--unmask-non-abi-names'):
             opt_unmask = True
 
-    schema = QAPISchema(input_file)
+    try:
+        schema = QAPISchema(input_file)
+    except QAPIError as err:
+        print >>sys.stderr, err
+        exit(1)
 
     gen_types(schema, output_dir, prefix, opt_builtins)
     gen_visit(schema, output_dir, prefix, opt_builtins)
scripts/qapi/common.py +9 -14
index 78e960d07c..d334e1db5a 100644
@@ -15,7 +15,6 @@ import errno
 import os
 import re
 import string
-import sys
 from ordereddict import OrderedDict
 
 builtin_types = {
@@ -1455,19 +1454,15 @@ class QAPISchemaEvent(QAPISchemaEntity):
 
 class QAPISchema(object):
     def __init__(self, fname):
-        try:
-            parser = QAPISchemaParser(open(fname, 'r'))
-            exprs = check_exprs(parser.exprs)
-            self.docs = parser.docs
-            self._entity_dict = {}
-            self._predefining = True
-            self._def_predefineds()
-            self._predefining = False
-            self._def_exprs(exprs)
-            self.check()
-        except QAPIError as err:
-            print >>sys.stderr, err
-            exit(1)
+        parser = QAPISchemaParser(open(fname, 'r'))
+        exprs = check_exprs(parser.exprs)
+        self.docs = parser.docs
+        self._entity_dict = {}
+        self._predefining = True
+        self._def_predefineds()
+        self._predefining = False
+        self._def_exprs(exprs)
+        self.check()
 
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
tests/qapi-schema/test-qapi.py +7 -2
index 7772d09919..d6bb8ec6a4 100644
@@ -53,7 +53,13 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
             for v in variants.variants:
                 print '    case %s: %s' % (v.name, v.type.name)
 
-schema = QAPISchema(sys.argv[1])
+
+try:
+    schema = QAPISchema(sys.argv[1])
+except QAPIError as err:
+    print >>sys.stderr, err
+    exit(1)
+
 schema.visit(QAPISchemaTestVisitor())
 
 for doc in schema.docs:
--

2.13.6

#52 lore raw

RFC PATCH 12/21 qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()

In reply to Markus Armbruster
---
 scripts/qapi/common.py | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
scripts/qapi/common.py +9 -6
index d334e1db5a..7a327bfe9f 100644
@@ -286,8 +286,12 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                self._include(include, info, os.path.dirname(self.fname),
-                              previously_included)
+                exprs_include = self._include(include, info,
+                                              os.path.dirname(self.fname),
+                                              previously_included)
+                if exprs_include:
+                    self.exprs.extend(exprs_include.exprs)
+                    self.docs.extend(exprs_include.docs)
             elif "pragma" in expr:
                 self.reject_expr_doc(cur_doc)
                 if len(expr) != 1:
@@ -330,14 +334,13 @@ class QAPISchemaParser(object):
 
         # skip multiple include of the same file
         if incl_abs_fname in previously_included:
-            return
+            return None
+
         try:
             fobj = open(incl_fname, 'r')
         except IOError as e:
             raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname))
-        exprs_include = QAPISchemaParser(fobj, previously_included, info)
-        self.exprs.extend(exprs_include.exprs)
-        self.docs.extend(exprs_include.docs)
+        return QAPISchemaParser(fobj, previously_included, info)
 
     def _pragma(self, name, value, info):
         global doc_required, returns_whitelist, name_case_whitelist
Signature · 1 line
2.13.6
Reviewed-by Marc-André Lureau on v1 12/21
#53 lore raw

Re: [Qemu-devel] [PATCH RFC 12/21] qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()

Quoted 1 line · Markus Armbruster

It's not obvious the motivation behind this change (beside behind more elegant), but

Quoted 44 lines
---
 scripts/qapi/common.py | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
scripts/qapi/common.py +9 -7
index d334e1db5a..7a327bfe9f 100644
@@ -286,8 +286,12 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                self._include(include, info, os.path.dirname(self.fname),
-                              previously_included)
+                exprs_include = self._include(include, info,
+                                              os.path.dirname(self.fname),
+                                              previously_included)
+                if exprs_include:
+                    self.exprs.extend(exprs_include.exprs)
+                    self.docs.extend(exprs_include.docs)
             elif "pragma" in expr:
                 self.reject_expr_doc(cur_doc)
                 if len(expr) != 1:
@@ -330,14 +334,13 @@ class QAPISchemaParser(object):
 
         # skip multiple include of the same file
         if incl_abs_fname in previously_included:
-            return
+            return None
+
         try:
             fobj = open(incl_fname, 'r')
         except IOError as e:
             raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname))
-        exprs_include = QAPISchemaParser(fobj, previously_included, info)
-        self.exprs.extend(exprs_include.exprs)
-        self.docs.extend(exprs_include.docs)
+        return QAPISchemaParser(fobj, previously_included, info)
 
     def _pragma(self, name, value, info):
         global doc_required, returns_whitelist, name_case_whitelist
--

2.13.6

#54 lore raw

Re: [Qemu-devel] [PATCH RFC 12/21] qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()

Quoted 5 lines · Marc-Andre Lureau
Quoted 1 line · Markus Armbruster

It's not obvious the motivation behind this change (beside behind more elegant), but

At one point during my search for a good way to record includes in the parse tree, updating .exprs in two places got in the way, so I cleaned it up. I guess it's not in the way of the solution I eventually chose, but I consider it a worthwhile little cleanup on its own.

Quoted 1 line

Thanks!

#55 lore raw

RFC PATCH 13/21 qapi: Record 'include' directives in parse tree

In reply to Markus Armbruster

The parse tree is a list of expressions. Except include expressions currently get replaced by the included file's parse tree.

Instead of throwing away the include expression, keep it with the file name expanded so you don't have to track the including file's directory to make sense of it.

A future commit will put this include expression to use.

---
 scripts/qapi/common.py | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
scripts/qapi/common.py +17 -4
index 7a327bfe9f..d5b93e7381 100644
@@ -286,8 +286,11 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                exprs_include = self._include(include, info,
-                                              os.path.dirname(self.fname),
+                incl_fname = os.path.join(os.path.dirname(self.fname),
+                                          include)
+                self.exprs.append({'expr': {'include': incl_fname},
+                                   'info': info})
+                exprs_include = self._include(include, info, incl_fname,
                                               previously_included)
                 if exprs_include:
                     self.exprs.extend(exprs_include.exprs)
@@ -322,8 +325,7 @@ class QAPISchemaParser(object):
                 "Documentation for '%s' is not followed by the definition"
                 % doc.symbol)
 
-    def _include(self, include, info, base_dir, previously_included):
-        incl_fname = os.path.join(base_dir, include)
+    def _include(self, include, info, incl_fname, previously_included):
         incl_abs_fname = os.path.abspath(incl_fname)
         # catch inclusion cycle
         inf = info
@@ -889,6 +891,9 @@ def check_exprs(exprs):
         info = expr_elem['info']
         doc = expr_elem.get('doc')
 
+        if 'include' in expr:
+            continue
+
         if not doc and doc_required:
             raise QAPISemError(info,
                                "Expression missing documentation comment")
@@ -927,6 +932,9 @@ def check_exprs(exprs):
 
     # Try again for hidden UnionKind enum
     for expr_elem in exprs:
+        if 'include' in expr:
+            continue
+
         expr = expr_elem['expr']
         if 'union' in expr and not discriminator_find_enum_define(expr):
             name = '%sKind' % expr['union']
@@ -939,6 +947,9 @@ def check_exprs(exprs):
 
     # Validate that exprs make sense
     for expr_elem in exprs:
+        if 'include' in expr:
+            continue
+
         expr = expr_elem['expr']
         info = expr_elem['info']
         doc = expr_elem.get('doc')
@@ -1663,6 +1674,8 @@ class QAPISchema(object):
                 self._def_command(expr, info, doc)
             elif 'event' in expr:
                 self._def_event(expr, info, doc)
+            elif 'include' in expr:
+                pass
             else:
                 assert False
 
Signature · 1 line
2.13.6
Reviewed-by Marc-André Lureau on v1 13/21
#56 lore raw

Re: [Qemu-devel] [PATCH RFC 13/21] qapi: Record 'include' directives in parse tree

Quoted 10 lines · Markus Armbruster

The parse tree is a list of expressions. Except include expressions currently get replaced by the included file's parse tree.

Instead of throwing away the include expression, keep it with the file name expanded so you don't have to track the including file's directory to make sense of it.

A future commit will put this include expression to use.

Quoted 74 lines
---
 scripts/qapi/common.py | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
scripts/qapi/common.py +17 -5
index 7a327bfe9f..d5b93e7381 100644
@@ -286,8 +286,11 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                exprs_include = self._include(include, info,
-                                              os.path.dirname(self.fname),
+                incl_fname = os.path.join(os.path.dirname(self.fname),
+                                          include)
+                self.exprs.append({'expr': {'include': incl_fname},
+                                   'info': info})
+                exprs_include = self._include(include, info, incl_fname,
                                               previously_included)
                 if exprs_include:
                     self.exprs.extend(exprs_include.exprs)
@@ -322,8 +325,7 @@ class QAPISchemaParser(object):
                 "Documentation for '%s' is not followed by the definition"
                 % doc.symbol)
 
-    def _include(self, include, info, base_dir, previously_included):
-        incl_fname = os.path.join(base_dir, include)
+    def _include(self, include, info, incl_fname, previously_included):
         incl_abs_fname = os.path.abspath(incl_fname)
         # catch inclusion cycle
         inf = info
@@ -889,6 +891,9 @@ def check_exprs(exprs):
         info = expr_elem['info']
         doc = expr_elem.get('doc')
 
+        if 'include' in expr:
+            continue
+
         if not doc and doc_required:
             raise QAPISemError(info,
                                "Expression missing documentation comment")
@@ -927,6 +932,9 @@ def check_exprs(exprs):
 
     # Try again for hidden UnionKind enum
     for expr_elem in exprs:
+        if 'include' in expr:
+            continue
+
         expr = expr_elem['expr']
         if 'union' in expr and not discriminator_find_enum_define(expr):
             name = '%sKind' % expr['union']
@@ -939,6 +947,9 @@ def check_exprs(exprs):
 
     # Validate that exprs make sense
     for expr_elem in exprs:
+        if 'include' in expr:
+            continue
+
         expr = expr_elem['expr']
         info = expr_elem['info']
         doc = expr_elem.get('doc')
@@ -1663,6 +1674,8 @@ class QAPISchema(object):
                 self._def_command(expr, info, doc)
             elif 'event' in expr:
                 self._def_event(expr, info, doc)
+            elif 'include' in expr:
+                pass
             else:
                 assert False
 
--

2.13.6

#57 lore raw

RFC PATCH 14/21 qapi: Generate in source order

In reply to Markus Armbruster

The generators' conversion to visitors (merge commit 9e72681d16) changed the processing order of entities from source order to alphabetical order. The next commit needs source order, so change it back.

---
 scripts/qapi/common.py                   |   4 +-
 tests/qapi-schema/comments.out           |   2 +-
 tests/qapi-schema/doc-bad-section.out    |   4 +-
 tests/qapi-schema/doc-good.out           |  32 ++--
 tests/qapi-schema/empty.out              |   2 +-
 tests/qapi-schema/event-case.out         |   2 +-
 tests/qapi-schema/ident-with-escape.out  |   6 +-
 tests/qapi-schema/include-relpath.out    |   2 +-
 tests/qapi-schema/include-repetition.out |   2 +-
 tests/qapi-schema/include-simple.out     |   2 +-
 tests/qapi-schema/indented-expr.out      |   2 +-
 tests/qapi-schema/qapi-schema-test.out   | 320 +++++++++++++++----------------
 12 files changed, 191 insertions(+), 189 deletions(-)
scripts/qapi/common.py +3 -1
index d5b93e7381..3b97bf8702 100644
@@ -1471,6 +1471,7 @@ class QAPISchema(object):
         parser = QAPISchemaParser(open(fname, 'r'))
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
+        self._entity_list = []
         self._entity_dict = {}
         self._predefining = True
         self._def_predefineds()
@@ -1482,6 +1483,7 @@ class QAPISchema(object):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
         assert ent.name not in self._entity_dict
+        self._entity_list.append(ent)
         self._entity_dict[ent.name] = ent
 
     def lookup_entity(self, name, typ=None):
@@ -1685,7 +1687,7 @@ class QAPISchema(object):
 
     def visit(self, visitor):
         visitor.visit_begin(self)
-        for (name, entity) in sorted(self._entity_dict.items()):
+        for entity in self._entity_list:
             if visitor.visit_needed(entity):
                 entity.visit(visitor)
         visitor.visit_end()
tests/qapi-schema/comments.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/doc-bad-section.out +2 -2
index 089bde1381..23bf8c71ab 100644
@@ -1,7 +1,7 @@
-enum Enum ['one', 'two']
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-object q_empty
+enum Enum ['one', 'two']
 doc symbol=Enum
     body=
 == Produces *invalid* texinfo
tests/qapi-schema/doc-good.out +16 -16
index 1d2c250527..0c07301f07 100644
@@ -1,35 +1,35 @@
+object q_empty
+enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+    prefix QTYPE
+enum Enum ['one', 'two']
 object Base
     member base1: Enum optional=False
-enum Enum ['one', 'two']
+object Variant1
+    member var1: str optional=False
+object Variant2
 object Object
     base Base
     tag base1
     case one: Variant1
     case two: Variant2
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
-    prefix QTYPE
+object q_obj_Variant1-wrapper
+    member data: Variant1 optional=False
+object q_obj_Variant2-wrapper
+    member data: Variant2 optional=False
+enum SugaredUnionKind ['one', 'two']
 object SugaredUnion
     member type: SugaredUnionKind optional=False
     tag type
     case one: q_obj_Variant1-wrapper
     case two: q_obj_Variant2-wrapper
-enum SugaredUnionKind ['one', 'two']
-object Variant1
-    member var1: str optional=False
-object Variant2
-command cmd q_obj_cmd-arg -> Object
-   gen=True success_response=True boxed=False
-command cmd-boxed Object -> None
-   gen=True success_response=True boxed=True
-object q_empty
-object q_obj_Variant1-wrapper
-    member data: Variant1 optional=False
-object q_obj_Variant2-wrapper
-    member data: Variant2 optional=False
 object q_obj_cmd-arg
     member arg1: int optional=False
     member arg2: str optional=True
     member arg3: bool optional=False
+command cmd q_obj_cmd-arg -> Object
+   gen=True success_response=True boxed=False
+command cmd-boxed Object -> None
+   gen=True success_response=True boxed=True
 doc freeform
     body=
 = Section
tests/qapi-schema/empty.out +1 -1
index 40b886ddae..0ec234eec4 100644
@@ -1,3 +1,3 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-object q_empty
tests/qapi-schema/event-case.out +1 -1
index 313c0fe7be..110571b793 100644
@@ -1,5 +1,5 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 event oops None
    boxed=False
-object q_empty
tests/qapi-schema/ident-with-escape.out +3 -3
index b5637cb2e0..8336aa7629 100644
@@ -1,7 +1,7 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-command fooA q_obj_fooA-arg -> None
-   gen=True success_response=True boxed=False
-object q_empty
 object q_obj_fooA-arg
     member bar1: str optional=False
+command fooA q_obj_fooA-arg -> None
+   gen=True success_response=True boxed=False
tests/qapi-schema/include-relpath.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/include-repetition.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/include-simple.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/indented-expr.out +1 -1
index 586795f44d..34de8be426 100644
@@ -1,7 +1,7 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 command eins None -> None
    gen=True success_response=True boxed=False
-object q_empty
 command zwei None -> None
    gen=True success_response=True boxed=False
tests/qapi-schema/qapi-schema-test.out +160 -160

380 lines not on this page · raw message

Signature · 1 line
2.13.6
#58 lore raw

Re: [Qemu-devel] [PATCH RFC 14/21] qapi: Generate in source order

Quoted 34 lines · Markus Armbruster

The generators' conversion to visitors (merge commit 9e72681d16) changed the processing order of entities from source order to alphabetical order. The next commit needs source order, so change it back.

---
 scripts/qapi/common.py                   |   4 +-
 tests/qapi-schema/comments.out           |   2 +-
 tests/qapi-schema/doc-bad-section.out    |   4 +-
 tests/qapi-schema/doc-good.out           |  32 ++--
 tests/qapi-schema/empty.out              |   2 +-
 tests/qapi-schema/event-case.out         |   2 +-
 tests/qapi-schema/ident-with-escape.out  |   6 +-
 tests/qapi-schema/include-relpath.out    |   2 +-
 tests/qapi-schema/include-repetition.out |   2 +-
 tests/qapi-schema/include-simple.out     |   2 +-
 tests/qapi-schema/indented-expr.out      |   2 +-
 tests/qapi-schema/qapi-schema-test.out   | 320 +++++++++++++++----------------
 12 files changed, 191 insertions(+), 189 deletions(-)
scripts/qapi/common.py +1 -0
index d5b93e7381..3b97bf8702 100644
@@ -1471,6 +1471,7 @@ class QAPISchema(object):
         parser = QAPISchemaParser(open(fname, 'r'))
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
+        self._entity_list = []
         self._entity_dict = {}
         self._predefining = True
         self._def_predefineds()

scripts/qapi/common.py class QAPISchema(object): · line 1487

Quoted 5 lines
scripts/qapi/common.py +1 -0
@@ -1482,6 +1483,7 @@ class QAPISchema(object):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
         assert ent.name not in self._entity_dict
+        self._entity_list.append(ent)
         self._entity_dict[ent.name] = ent

Why not use the OrderedDict instead?

Quoted 558 lines
scripts/qapi/common.py +1 -1
     def lookup_entity(self, name, typ=None):
@@ -1685,7 +1687,7 @@ class QAPISchema(object):
 
     def visit(self, visitor):
         visitor.visit_begin(self)
-        for (name, entity) in sorted(self._entity_dict.items()):
+        for entity in self._entity_list:
             if visitor.visit_needed(entity):
                 entity.visit(visitor)
         visitor.visit_end()
tests/qapi-schema/comments.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/doc-bad-section.out +2 -2
index 089bde1381..23bf8c71ab 100644
@@ -1,7 +1,7 @@
-enum Enum ['one', 'two']
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-object q_empty
+enum Enum ['one', 'two']
 doc symbol=Enum
     body=
 == Produces *invalid* texinfo
tests/qapi-schema/doc-good.out +16 -16
index 1d2c250527..0c07301f07 100644
@@ -1,35 +1,35 @@
+object q_empty
+enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+    prefix QTYPE
+enum Enum ['one', 'two']
 object Base
     member base1: Enum optional=False
-enum Enum ['one', 'two']
+object Variant1
+    member var1: str optional=False
+object Variant2
 object Object
     base Base
     tag base1
     case one: Variant1
     case two: Variant2
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
-    prefix QTYPE
+object q_obj_Variant1-wrapper
+    member data: Variant1 optional=False
+object q_obj_Variant2-wrapper
+    member data: Variant2 optional=False
+enum SugaredUnionKind ['one', 'two']
 object SugaredUnion
     member type: SugaredUnionKind optional=False
     tag type
     case one: q_obj_Variant1-wrapper
     case two: q_obj_Variant2-wrapper
-enum SugaredUnionKind ['one', 'two']
-object Variant1
-    member var1: str optional=False
-object Variant2
-command cmd q_obj_cmd-arg -> Object
-   gen=True success_response=True boxed=False
-command cmd-boxed Object -> None
-   gen=True success_response=True boxed=True
-object q_empty
-object q_obj_Variant1-wrapper
-    member data: Variant1 optional=False
-object q_obj_Variant2-wrapper
-    member data: Variant2 optional=False
 object q_obj_cmd-arg
     member arg1: int optional=False
     member arg2: str optional=True
     member arg3: bool optional=False
+command cmd q_obj_cmd-arg -> Object
+   gen=True success_response=True boxed=False
+command cmd-boxed Object -> None
+   gen=True success_response=True boxed=True
 doc freeform
     body=
 = Section
tests/qapi-schema/empty.out +1 -1
index 40b886ddae..0ec234eec4 100644
@@ -1,3 +1,3 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-object q_empty
tests/qapi-schema/event-case.out +1 -1
index 313c0fe7be..110571b793 100644
@@ -1,5 +1,5 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 event oops None
    boxed=False
-object q_empty
tests/qapi-schema/ident-with-escape.out +3 -3
index b5637cb2e0..8336aa7629 100644
@@ -1,7 +1,7 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-command fooA q_obj_fooA-arg -> None
-   gen=True success_response=True boxed=False
-object q_empty
 object q_obj_fooA-arg
     member bar1: str optional=False
+command fooA q_obj_fooA-arg -> None
+   gen=True success_response=True boxed=False
tests/qapi-schema/include-relpath.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/include-repetition.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/include-simple.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/indented-expr.out +1 -1
index 586795f44d..34de8be426 100644
@@ -1,7 +1,7 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 command eins None -> None
    gen=True success_response=True boxed=False
-object q_empty
 command zwei None -> None
    gen=True success_response=True boxed=False
tests/qapi-schema/qapi-schema-test.out +160 -161
index 3b1e9082d3..50706b0136 100644
@@ -1,87 +1,129 @@
-alternate AltEnumBool
-    tag type
-    case e: EnumOne
-    case b: bool
-alternate AltEnumInt
-    tag type
-    case e: EnumOne
-    case i: int
-alternate AltEnumNum
-    tag type
-    case e: EnumOne
-    case n: number
-alternate AltNumEnum
-    tag type
-    case n: number
-    case e: EnumOne
-alternate AltStrObj
-    tag type
-    case s: str
-    case o: TestStruct
-event EVENT_A None
-   boxed=False
-event EVENT_B None
-   boxed=False
-event EVENT_C q_obj_EVENT_C-arg
-   boxed=False
-event EVENT_D q_obj_EVENT_D-arg
-   boxed=False
-event EVENT_E UserDefZero
-   boxed=True
-event EVENT_F UserDefAlternate
-   boxed=True
+object q_empty
+enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+    prefix QTYPE
+object TestStruct
+    member integer: int optional=False
+    member boolean: bool optional=False
+    member string: str optional=False
+object NestedEnumsOne
+    member enum1: EnumOne optional=False
+    member enum2: EnumOne optional=True
+    member enum3: EnumOne optional=False
+    member enum4: EnumOne optional=True
+enum MyEnum []
 object Empty1
 object Empty2
     base Empty1
+command user_def_cmd0 Empty2 -> Empty2
+   gen=True success_response=True boxed=False
+enum QEnumTwo ['value1', 'value2']
+    prefix QENUM_TWO
+object UserDefOne
+    base UserDefZero
+    member string: str optional=False
+    member enum1: EnumOne optional=True
 enum EnumOne ['value1', 'value2', 'value3']
-object EventStructOne
-    member struct1: UserDefOne optional=False
+object UserDefZero
+    member integer: int optional=False
+object UserDefTwoDictDict
+    member userdef: UserDefOne optional=False
     member string: str optional=False
-    member enum2: EnumOne optional=True
+object UserDefTwoDict
+    member string1: str optional=False
+    member dict2: UserDefTwoDictDict optional=False
+    member dict3: UserDefTwoDictDict optional=True
+object UserDefTwo
+    member string0: str optional=False
+    member dict1: UserDefTwoDict optional=False
 object ForceArrays
     member unused1: UserDefOneList optional=False
     member unused2: UserDefTwoList optional=False
     member unused3: TestStructList optional=False
-enum MyEnum []
-object NestedEnumsOne
-    member enum1: EnumOne optional=False
-    member enum2: EnumOne optional=True
-    member enum3: EnumOne optional=False
-    member enum4: EnumOne optional=True
-enum QEnumTwo ['value1', 'value2']
-    prefix QENUM_TWO
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
-    prefix QTYPE
-object TestStruct
-    member integer: int optional=False
-    member boolean: bool optional=False
-    member string: str optional=False
 object UserDefA
     member boolean: bool optional=False
     member a_b: int optional=True
-alternate UserDefAlternate
-    tag type
-    case udfu: UserDefFlatUnion
-    case e: EnumOne
-    case i: int
-    case n: null
 object UserDefB
     member intb: int optional=False
     member a-b: bool optional=True
-object UserDefC
-    member string1: str optional=False
-    member string2: str optional=False
 object UserDefFlatUnion
     base UserDefUnionBase
     tag enum1
     case value1: UserDefA
     case value2: UserDefB
     case value3: UserDefB
+object UserDefUnionBase
+    base UserDefZero
+    member string: str optional=False
+    member enum1: EnumOne optional=False
+object q_obj_UserDefFlatUnion2-base
+    member integer: int optional=True
+    member string: str optional=False
+    member enum1: QEnumTwo optional=False
 object UserDefFlatUnion2
     base q_obj_UserDefFlatUnion2-base
     tag enum1
     case value1: UserDefC
     case value2: UserDefB
+object WrapAlternate
+    member alt: UserDefAlternate optional=False
+alternate UserDefAlternate
+    tag type
+    case udfu: UserDefFlatUnion
+    case e: EnumOne
+    case i: int
+    case n: null
+object UserDefC
+    member string1: str optional=False
+    member string2: str optional=False
+alternate AltEnumBool
+    tag type
+    case e: EnumOne
+    case b: bool
+alternate AltEnumNum
+    tag type
+    case e: EnumOne
+    case n: number
+alternate AltNumEnum
+    tag type
+    case n: number
+    case e: EnumOne
+alternate AltEnumInt
+    tag type
+    case e: EnumOne
+    case i: int
+alternate AltStrObj
+    tag type
+    case s: str
+    case o: TestStruct
+object q_obj_intList-wrapper
+    member data: intList optional=False
+object q_obj_int8List-wrapper
+    member data: int8List optional=False
+object q_obj_int16List-wrapper
+    member data: int16List optional=False
+object q_obj_int32List-wrapper
+    member data: int32List optional=False
+object q_obj_int64List-wrapper
+    member data: int64List optional=False
+object q_obj_uint8List-wrapper
+    member data: uint8List optional=False
+object q_obj_uint16List-wrapper
+    member data: uint16List optional=False
+object q_obj_uint32List-wrapper
+    member data: uint32List optional=False
+object q_obj_uint64List-wrapper
+    member data: uint64List optional=False
+object q_obj_numberList-wrapper
+    member data: numberList optional=False
+object q_obj_boolList-wrapper
+    member data: boolList optional=False
+object q_obj_strList-wrapper
+    member data: strList optional=False
+object q_obj_sizeList-wrapper
+    member data: sizeList optional=False
+object q_obj_anyList-wrapper
+    member data: anyList optional=False
+enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32', 's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes', 'any']
 object UserDefNativeListUnion
     member type: UserDefNativeListUnionKind optional=False
     tag type
@@ -99,133 +141,91 @@ object UserDefNativeListUnion
     case string: q_obj_strList-wrapper
     case sizes: q_obj_sizeList-wrapper
     case any: q_obj_anyList-wrapper
-enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32', 's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes', 'any']
-object UserDefOne
-    base UserDefZero
-    member string: str optional=False
-    member enum1: EnumOne optional=True
+command user_def_cmd None -> None
+   gen=True success_response=True boxed=False
+object q_obj_user_def_cmd1-arg
+    member ud1a: UserDefOne optional=False
+command user_def_cmd1 q_obj_user_def_cmd1-arg -> None
+   gen=True success_response=True boxed=False
+object q_obj_user_def_cmd2-arg
+    member ud1a: UserDefOne optional=False
+    member ud1b: UserDefOne optional=True
+command user_def_cmd2 q_obj_user_def_cmd2-arg -> UserDefTwo
+   gen=True success_response=True boxed=False
+object q_obj_guest-get-time-arg
+    member a: int optional=False
+    member b: int optional=True
+command guest-get-time q_obj_guest-get-time-arg -> int
+   gen=True success_response=True boxed=False
+object q_obj_guest-sync-arg
+    member arg: any optional=False
+command guest-sync q_obj_guest-sync-arg -> any
+   gen=True success_response=True boxed=False
+command boxed-struct UserDefZero -> None
+   gen=True success_response=True boxed=True
+command boxed-union UserDefNativeListUnion -> None
+   gen=True success_response=True boxed=True
 object UserDefOptions
     member i64: intList optional=True
     member u64: uint64List optional=True
     member u16: uint16List optional=True
     member i64x: int optional=True
     member u64x: uint64 optional=True
-object UserDefTwo
-    member string0: str optional=False
-    member dict1: UserDefTwoDict optional=False
-object UserDefTwoDict
-    member string1: str optional=False
-    member dict2: UserDefTwoDictDict optional=False
-    member dict3: UserDefTwoDictDict optional=True
-object UserDefTwoDictDict
-    member userdef: UserDefOne optional=False
+object EventStructOne
+    member struct1: UserDefOne optional=False
     member string: str optional=False
-object UserDefUnionBase
-    base UserDefZero
-    member string: str optional=False
-    member enum1: EnumOne optional=False
-object UserDefZero
-    member integer: int optional=False
-object WrapAlternate
-    member alt: UserDefAlternate optional=False
-event __ORG.QEMU_X-EVENT __org.qemu_x-Struct
+    member enum2: EnumOne optional=True
+event EVENT_A None
+   boxed=False
+event EVENT_B None
+   boxed=False
+object q_obj_EVENT_C-arg
+    member a: int optional=True
+    member b: UserDefOne optional=True
+    member c: str optional=False
+event EVENT_C q_obj_EVENT_C-arg
+   boxed=False
+object q_obj_EVENT_D-arg
+    member a: EventStructOne optional=False
+    member b: str optional=False
+    member c: str optional=True
+    member enum3: EnumOne optional=True
+event EVENT_D q_obj_EVENT_D-arg
    boxed=False
-alternate __org.qemu_x-Alt
-    tag type
-    case __org.qemu_x-branch: str
-    case b: __org.qemu_x-Base
+event EVENT_E UserDefZero
+   boxed=True
+event EVENT_F UserDefAlternate
+   boxed=True
+enum __org.qemu_x-Enum ['__org.qemu_x-value']
 object __org.qemu_x-Base
     member __org.qemu_x-member1: __org.qemu_x-Enum optional=False
-enum __org.qemu_x-Enum ['__org.qemu_x-value']
 object __org.qemu_x-Struct
     base __org.qemu_x-Base
     member __org.qemu_x-member2: str optional=False
     member wchar-t: int optional=True
-object __org.qemu_x-Struct2
-    member array: __org.qemu_x-Union1List optional=False
+object q_obj_str-wrapper
+    member data: str optional=False
+enum __org.qemu_x-Union1Kind ['__org.qemu_x-branch']
 object __org.qemu_x-Union1
     member type: __org.qemu_x-Union1Kind optional=False
     tag type
     case __org.qemu_x-branch: q_obj_str-wrapper
-enum __org.qemu_x-Union1Kind ['__org.qemu_x-branch']
+object __org.qemu_x-Struct2
+    member array: __org.qemu_x-Union1List optional=False
 object __org.qemu_x-Union2
     base __org.qemu_x-Base
     tag __org.qemu_x-member1
     case __org.qemu_x-value: __org.qemu_x-Struct2
-command __org.qemu_x-command q_obj___org.qemu_x-command-arg -> __org.qemu_x-Union1
-   gen=True success_response=True boxed=False
-command boxed-struct UserDefZero -> None
-   gen=True success_response=True boxed=True
-command boxed-union UserDefNativeListUnion -> None
-   gen=True success_response=True boxed=True
-command guest-get-time q_obj_guest-get-time-arg -> int
-   gen=True success_response=True boxed=False
-command guest-sync q_obj_guest-sync-arg -> any
-   gen=True success_response=True boxed=False
-object q_empty
-object q_obj_EVENT_C-arg
-    member a: int optional=True
-    member b: UserDefOne optional=True
-    member c: str optional=False
-object q_obj_EVENT_D-arg
-    member a: EventStructOne optional=False
-    member b: str optional=False
-    member c: str optional=True
-    member enum3: EnumOne optional=True
-object q_obj_UserDefFlatUnion2-base
-    member integer: int optional=True
-    member string: str optional=False
-    member enum1: QEnumTwo optional=False
+alternate __org.qemu_x-Alt
+    tag type
+    case __org.qemu_x-branch: str
+    case b: __org.qemu_x-Base
+event __ORG.QEMU_X-EVENT __org.qemu_x-Struct
+   boxed=False
 object q_obj___org.qemu_x-command-arg
     member a: __org.qemu_x-EnumList optional=False
     member b: __org.qemu_x-StructList optional=False
     member c: __org.qemu_x-Union2 optional=False
     member d: __org.qemu_x-Alt optional=False
-object q_obj_anyList-wrapper
-    member data: anyList optional=False
-object q_obj_boolList-wrapper
-    member data: boolList optional=False
-object q_obj_guest-get-time-arg
-    member a: int optional=False
-    member b: int optional=True
-object q_obj_guest-sync-arg
-    member arg: any optional=False
-object q_obj_int16List-wrapper
-    member data: int16List optional=False
-object q_obj_int32List-wrapper
-    member data: int32List optional=False
-object q_obj_int64List-wrapper
-    member data: int64List optional=False
-object q_obj_int8List-wrapper
-    member data: int8List optional=False
-object q_obj_intList-wrapper
-    member data: intList optional=False
-object q_obj_numberList-wrapper
-    member data: numberList optional=False
-object q_obj_sizeList-wrapper
-    member data: sizeList optional=False
-object q_obj_str-wrapper
-    member data: str optional=False
-object q_obj_strList-wrapper
-    member data: strList optional=False
-object q_obj_uint16List-wrapper
-    member data: uint16List optional=False
-object q_obj_uint32List-wrapper
-    member data: uint32List optional=False
-object q_obj_uint64List-wrapper
-    member data: uint64List optional=False
-object q_obj_uint8List-wrapper
-    member data: uint8List optional=False
-object q_obj_user_def_cmd1-arg
-    member ud1a: UserDefOne optional=False
-object q_obj_user_def_cmd2-arg
-    member ud1a: UserDefOne optional=False
-    member ud1b: UserDefOne optional=True
-command user_def_cmd None -> None
-   gen=True success_response=True boxed=False
-command user_def_cmd0 Empty2 -> Empty2
-   gen=True success_response=True boxed=False
-command user_def_cmd1 q_obj_user_def_cmd1-arg -> None
-   gen=True success_response=True boxed=False
-command user_def_cmd2 q_obj_user_def_cmd2-arg -> UserDefTwo
+command __org.qemu_x-command q_obj___org.qemu_x-command-arg -> __org.qemu_x-Union1
    gen=True success_response=True boxed=False
--

2.13.6

#59 lore raw

Re: [Qemu-devel] [PATCH RFC 14/21] qapi: Generate in source order

Quoted 34 lines · Marc-Andre Lureau
Quoted 33 lines · Markus Armbruster

The generators' conversion to visitors (merge commit 9e72681d16) changed the processing order of entities from source order to alphabetical order. The next commit needs source order, so change it back.

---
 scripts/qapi/common.py                   |   4 +-
 tests/qapi-schema/comments.out           |   2 +-
 tests/qapi-schema/doc-bad-section.out    |   4 +-
 tests/qapi-schema/doc-good.out           |  32 ++--
 tests/qapi-schema/empty.out              |   2 +-
 tests/qapi-schema/event-case.out         |   2 +-
 tests/qapi-schema/ident-with-escape.out  |   6 +-
 tests/qapi-schema/include-relpath.out    |   2 +-
 tests/qapi-schema/include-repetition.out |   2 +-
 tests/qapi-schema/include-simple.out     |   2 +-
 tests/qapi-schema/indented-expr.out      |   2 +-
 tests/qapi-schema/qapi-schema-test.out   | 320 +++++++++++++++----------------
 12 files changed, 191 insertions(+), 189 deletions(-)
scripts/qapi/common.py +1 -0
index d5b93e7381..3b97bf8702 100644
@@ -1471,6 +1471,7 @@ class QAPISchema(object):
         parser = QAPISchemaParser(open(fname, 'r'))
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
+        self._entity_list = []
         self._entity_dict = {}
         self._predefining = True
         self._def_predefineds()
Quoted 8 lines

scripts/qapi/common.py class QAPISchema(object): · line 1487

Quoted 6 lines
scripts/qapi/common.py +1 -0
@@ -1482,6 +1483,7 @@ class QAPISchema(object):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
         assert ent.name not in self._entity_dict
+        self._entity_list.append(ent)
         self._entity_dict[ent.name] = ent

Why not use the OrderedDict instead?

Fair question! However, the next patch will create anonymous entities, which get added only to ._entity_list, not _entity_dict.

[...]

Reviewed-by Marc-André Lureau
#60 lore raw

Re: [Qemu-devel] [PATCH RFC 14/21] qapi: Generate in source order

Quoted 45 lines · Markus Armbruster
Quoted 34 lines · Marc-Andre Lureau
Quoted 33 lines · Markus Armbruster

The generators' conversion to visitors (merge commit 9e72681d16) changed the processing order of entities from source order to alphabetical order. The next commit needs source order, so change it back.

---
 scripts/qapi/common.py                   |   4 +-
 tests/qapi-schema/comments.out           |   2 +-
 tests/qapi-schema/doc-bad-section.out    |   4 +-
 tests/qapi-schema/doc-good.out           |  32 ++--
 tests/qapi-schema/empty.out              |   2 +-
 tests/qapi-schema/event-case.out         |   2 +-
 tests/qapi-schema/ident-with-escape.out  |   6 +-
 tests/qapi-schema/include-relpath.out    |   2 +-
 tests/qapi-schema/include-repetition.out |   2 +-
 tests/qapi-schema/include-simple.out     |   2 +-
 tests/qapi-schema/indented-expr.out      |   2 +-
 tests/qapi-schema/qapi-schema-test.out   | 320 +++++++++++++++----------------
 12 files changed, 191 insertions(+), 189 deletions(-)
scripts/qapi/common.py +1 -0
index d5b93e7381..3b97bf8702 100644
@@ -1471,6 +1471,7 @@ class QAPISchema(object):
         parser = QAPISchemaParser(open(fname, 'r'))
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
+        self._entity_list = []
         self._entity_dict = {}
         self._predefining = True
         self._def_predefineds()
Quoted 8 lines

scripts/qapi/common.py class QAPISchema(object): · line 1487

Quoted 6 lines
scripts/qapi/common.py +1 -0
@@ -1482,6 +1483,7 @@ class QAPISchema(object):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
         assert ent.name not in self._entity_dict
+        self._entity_list.append(ent)
         self._entity_dict[ent.name] = ent

Why not use the OrderedDict instead?

Quoted 2 lines

Fair question! However, the next patch will create anonymous entities, which get added only to ._entity_list, not _entity_dict.

I see, something we could improve or clarify, but that's ok to me as is:

Quoted 2 lines

[...]

#61 lore raw

Re: [Qemu-devel] [PATCH RFC 14/21] qapi: Generate in source order

In reply to Markus Armbruster
Quoted 45 lines · Markus Armbruster
Quoted 34 lines · Marc-Andre Lureau
Quoted 33 lines · Markus Armbruster

The generators' conversion to visitors (merge commit 9e72681d16) changed the processing order of entities from source order to alphabetical order. The next commit needs source order, so change it back.

---
 scripts/qapi/common.py                   |   4 +-
 tests/qapi-schema/comments.out           |   2 +-
 tests/qapi-schema/doc-bad-section.out    |   4 +-
 tests/qapi-schema/doc-good.out           |  32 ++--
 tests/qapi-schema/empty.out              |   2 +-
 tests/qapi-schema/event-case.out         |   2 +-
 tests/qapi-schema/ident-with-escape.out  |   6 +-
 tests/qapi-schema/include-relpath.out    |   2 +-
 tests/qapi-schema/include-repetition.out |   2 +-
 tests/qapi-schema/include-simple.out     |   2 +-
 tests/qapi-schema/indented-expr.out      |   2 +-
 tests/qapi-schema/qapi-schema-test.out   | 320 +++++++++++++++----------------
 12 files changed, 191 insertions(+), 189 deletions(-)
scripts/qapi/common.py +1 -0
index d5b93e7381..3b97bf8702 100644
@@ -1471,6 +1471,7 @@ class QAPISchema(object):
         parser = QAPISchemaParser(open(fname, 'r'))
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
+        self._entity_list = []
         self._entity_dict = {}
         self._predefining = True
         self._def_predefineds()
Quoted 8 lines

scripts/qapi/common.py class QAPISchema(object): · line 1487

Quoted 6 lines
scripts/qapi/common.py +1 -0
@@ -1482,6 +1483,7 @@ class QAPISchema(object):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
         assert ent.name not in self._entity_dict
+        self._entity_list.append(ent)
         self._entity_dict[ent.name] = ent

Why not use the OrderedDict instead?

Quoted 4 lines

Fair question! However, the next patch will create anonymous entities, which get added only to ._entity_list, not _entity_dict.

[...]

Flaw in QAPISchema.check():

for ent in self._entity_dict.values():
ent.check(self)

Need to iterate over entity_list, or else we fail to check anonymous entities. Currently harmless, as QAPISchemaInclude.check() does nothing. I'll fix it.

#62 lore raw

RFC PATCH 15/21 qapi: Record 'include' directives in intermediate representation

In reply to Markus Armbruster

The include directive permits modular QAPI schemata, but the generated code is monolithic all the same. To permit generating modular code, the front end needs to pass more information on inclusions to the back ends. The commit before last added the necessary information to the parse tree. This commit adds it to the intermediate representation and its QAPISchemaVisitor. A later commit will use this to to generate modular code.

New entity QAPISchemaInclude represents inclusions. Call new visitor method visit_include() for it, so visitors can see the sub-modules a module includes.

New QAPISchemaEntity attribute @module names the entity's source file. Call new visitor method visit_module() when it changes during a visit, so visitors can keep track of the module being visited.

---
 scripts/qapi/common.py                   | 44 ++++++++++++++++++++++++++++----
 tests/qapi-schema/comments.out           |  1 +
 tests/qapi-schema/doc-bad-section.out    |  1 +
 tests/qapi-schema/doc-good.out           |  1 +
 tests/qapi-schema/event-case.out         |  1 +
 tests/qapi-schema/ident-with-escape.out  |  1 +
 tests/qapi-schema/include-relpath.out    |  5 ++++
 tests/qapi-schema/include-repetition.out | 10 ++++++++
 tests/qapi-schema/include-simple.out     |  3 +++
 tests/qapi-schema/indented-expr.out      |  1 +
 tests/qapi-schema/qapi-schema-test.out   |  1 +
 tests/qapi-schema/test-qapi.py           |  7 +++++
 12 files changed, 71 insertions(+), 5 deletions(-)
scripts/qapi/common.py +39 -5
index 3b97bf8702..f4e9ebbb53 100644
@@ -981,8 +981,9 @@ def check_exprs(exprs):
 
 class QAPISchemaEntity(object):
     def __init__(self, name, info, doc):
-        assert isinstance(name, str)
+        assert name is None or isinstance(name, str)
         self.name = name
+        self.module = None
         # For explicitly defined entities, info points to the (explicit)
         # definition.  For builtins (and their arrays), info is None.
         # For implicitly defined entities, info points to a place that
@@ -1011,10 +1012,16 @@ class QAPISchemaVisitor(object):
     def visit_end(self):
         pass
 
+    def visit_module(self, fname):
+        pass
+
     def visit_needed(self, entity):
         # Default to visiting everything
         return True
 
+    def visit_include(self, fname, info):
+        pass
+
     def visit_builtin_type(self, name, info, json_type):
         pass
 
@@ -1041,6 +1048,16 @@ class QAPISchemaVisitor(object):
         pass
 
 
+class QAPISchemaInclude(QAPISchemaEntity):
+
+    def __init__(self, fname, info):
+        QAPISchemaEntity.__init__(self, None, info, None)
+        self.fname = fname
+
+    def visit(self, visitor):
+        visitor.visit_include(self.fname, self.info)
+
+
 class QAPISchemaType(QAPISchemaEntity):
     # Return the C type for common use.
     # For the types we commonly box, this is a pointer type.
@@ -1468,6 +1485,7 @@ class QAPISchemaEvent(QAPISchemaEntity):
 
 class QAPISchema(object):
     def __init__(self, fname):
+        self._fname = fname
         parser = QAPISchemaParser(open(fname, 'r'))
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
@@ -1475,16 +1493,19 @@ class QAPISchema(object):
         self._entity_dict = {}
         self._predefining = True
         self._def_predefineds()
-        self._predefining = False
         self._def_exprs(exprs)
         self.check()
 
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
-        assert ent.name not in self._entity_dict
+        assert ent.name is None or ent.name not in self._entity_dict
         self._entity_list.append(ent)
-        self._entity_dict[ent.name] = ent
+        if ent.name is not None:
+            self._entity_dict[ent.name] = ent
+        if ent.info:
+            ent.module = os.path.relpath(ent.info['file'],
+                                         os.path.dirname(self._fname))
 
     def lookup_entity(self, name, typ=None):
         ent = self._entity_dict.get(name)
@@ -1495,6 +1516,15 @@ class QAPISchema(object):
     def lookup_type(self, name):
         return self.lookup_entity(name, QAPISchemaType)
 
+    def _def_include(self, expr, info, doc):
+        include = expr['include']
+        assert doc is None
+        main_info = info
+        while main_info['parent']:
+            main_info = main_info['parent']
+        fname = os.path.relpath(include, os.path.dirname(main_info['file']))
+        self._def_entity(QAPISchemaInclude(fname, info))
+
     def _def_builtin_type(self, name, json_type, c_type):
         self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
         # TODO As long as we have QAPI_TYPES_BUILTIN to share multiple
@@ -1677,7 +1707,7 @@ class QAPISchema(object):
             elif 'event' in expr:
                 self._def_event(expr, info, doc)
             elif 'include' in expr:
-                pass
+                self._def_include(expr, info, doc)
             else:
                 assert False
 
@@ -1687,8 +1717,12 @@ class QAPISchema(object):
 
     def visit(self, visitor):
         visitor.visit_begin(self)
+        module = None
         for entity in self._entity_list:
             if visitor.visit_needed(entity):
+                if entity.module != module:
+                    module = entity.module
+                    visitor.visit_module(module)
                 entity.visit(visitor)
         visitor.visit_end()
 
tests/qapi-schema/comments.out +1 -0
index 0261ddf202..8d2f1ce8a2 100644
@@ -1,4 +1,5 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module comments.json
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/doc-bad-section.out +1 -0
index 23bf8c71ab..cd28721568 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module doc-bad-section.json
 enum Enum ['one', 'two']
 doc symbol=Enum
     body=
tests/qapi-schema/doc-good.out +1 -0
index 0c07301f07..430b5a87db 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module doc-good.json
 enum Enum ['one', 'two']
 object Base
     member base1: Enum optional=False
tests/qapi-schema/event-case.out +1 -0
index 110571b793..88c0964917 100644
@@ -1,5 +1,6 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module event-case.json
 event oops None
    boxed=False
tests/qapi-schema/ident-with-escape.out +1 -0
index 8336aa7629..ee3b34e623 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module ident-with-escape.json
 object q_obj_fooA-arg
     member bar1: str optional=False
 command fooA q_obj_fooA-arg -> None
tests/qapi-schema/include-relpath.out +5 -0
index 0261ddf202..ebbabd7a18 100644
@@ -1,4 +1,9 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module include-relpath.json
+include include/relpath.json
+module include/relpath.json
+include include-relpath-sub.json
+module include-relpath-sub.json
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-repetition.out +10 -0
index 0261ddf202..7235e055bc 100644
@@ -1,4 +1,14 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module include-repetition.json
+include comments.json
+module comments.json
 enum Status ['good', 'bad', 'ugly']
+module include-repetition.json
+include include-repetition-sub.json
+module include-repetition-sub.json
+include comments.json
+include comments.json
+module include-repetition.json
+include comments.json
tests/qapi-schema/include-simple.out +3 -0
index 0261ddf202..006f723eeb 100644
@@ -1,4 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module include-simple.json
+include include-simple-sub.json
+module include-simple-sub.json
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/indented-expr.out +1 -0
index 34de8be426..a79935e8c3 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module indented-expr.json
 command eins None -> None
    gen=True success_response=True boxed=False
 command zwei None -> None
tests/qapi-schema/qapi-schema-test.out +1 -0
index 50706b0136..012e7fc06a 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module qapi-schema-test.json
 object TestStruct
     member integer: int optional=False
     member boolean: bool optional=False
tests/qapi-schema/test-qapi.py +7 -0
index d6bb8ec6a4..2027eef3e5 100644
@@ -17,6 +17,13 @@ import sys
 
 
 class QAPISchemaTestVisitor(QAPISchemaVisitor):
+
+    def visit_module(self, name):
+        print 'module %s' % name
+
+    def visit_include(self, name, info):
+        print 'include %s' % name
+
     def visit_enum_type(self, name, info, values, prefix):
         print 'enum %s %s' % (name, values)
         if prefix:
Signature · 1 line
2.13.6
Reviewed-by Marc-André Lureau on v1 15/21
#63 lore raw

Re: [Qemu-devel] [PATCH RFC 15/21] qapi: Record 'include' directives in intermediate representation

Quoted 12 lines · Markus Armbruster

The include directive permits modular QAPI schemata, but the generated code is monolithic all the same. To permit generating modular code, the front end needs to pass more information on inclusions to the back ends. The commit before last added the necessary information to the parse tree. This commit adds it to the intermediate representation and its QAPISchemaVisitor. A later commit will use this to to generate modular code.

New entity QAPISchemaInclude represents inclusions. Call new visitor method visit_include() for it, so visitors can see the sub-modules a module includes.

Quoted 5 lines

New QAPISchemaEntity attribute @module names the entity's source file. Call new visitor method visit_module() when it changes during a visit, so visitors can keep track of the module being visited.

Quoted 280 lines
---
 scripts/qapi/common.py                   | 44 ++++++++++++++++++++++++++++----
 tests/qapi-schema/comments.out           |  1 +
 tests/qapi-schema/doc-bad-section.out    |  1 +
 tests/qapi-schema/doc-good.out           |  1 +
 tests/qapi-schema/event-case.out         |  1 +
 tests/qapi-schema/ident-with-escape.out  |  1 +
 tests/qapi-schema/include-relpath.out    |  5 ++++
 tests/qapi-schema/include-repetition.out | 10 ++++++++
 tests/qapi-schema/include-simple.out     |  3 +++
 tests/qapi-schema/indented-expr.out      |  1 +
 tests/qapi-schema/qapi-schema-test.out   |  1 +
 tests/qapi-schema/test-qapi.py           |  7 +++++
 12 files changed, 71 insertions(+), 5 deletions(-)
scripts/qapi/common.py +39 -5
index 3b97bf8702..f4e9ebbb53 100644
@@ -981,8 +981,9 @@ def check_exprs(exprs):
 
 class QAPISchemaEntity(object):
     def __init__(self, name, info, doc):
-        assert isinstance(name, str)
+        assert name is None or isinstance(name, str)
         self.name = name
+        self.module = None
         # For explicitly defined entities, info points to the (explicit)
         # definition.  For builtins (and their arrays), info is None.
         # For implicitly defined entities, info points to a place that
@@ -1011,10 +1012,16 @@ class QAPISchemaVisitor(object):
     def visit_end(self):
         pass
 
+    def visit_module(self, fname):
+        pass
+
     def visit_needed(self, entity):
         # Default to visiting everything
         return True
 
+    def visit_include(self, fname, info):
+        pass
+
     def visit_builtin_type(self, name, info, json_type):
         pass
 
@@ -1041,6 +1048,16 @@ class QAPISchemaVisitor(object):
         pass
 
 
+class QAPISchemaInclude(QAPISchemaEntity):
+
+    def __init__(self, fname, info):
+        QAPISchemaEntity.__init__(self, None, info, None)
+        self.fname = fname
+
+    def visit(self, visitor):
+        visitor.visit_include(self.fname, self.info)
+
+
 class QAPISchemaType(QAPISchemaEntity):
     # Return the C type for common use.
     # For the types we commonly box, this is a pointer type.
@@ -1468,6 +1485,7 @@ class QAPISchemaEvent(QAPISchemaEntity):
 
 class QAPISchema(object):
     def __init__(self, fname):
+        self._fname = fname
         parser = QAPISchemaParser(open(fname, 'r'))
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
@@ -1475,16 +1493,19 @@ class QAPISchema(object):
         self._entity_dict = {}
         self._predefining = True
         self._def_predefineds()
-        self._predefining = False
         self._def_exprs(exprs)
         self.check()
 
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
-        assert ent.name not in self._entity_dict
+        assert ent.name is None or ent.name not in self._entity_dict
         self._entity_list.append(ent)
-        self._entity_dict[ent.name] = ent
+        if ent.name is not None:
+            self._entity_dict[ent.name] = ent
+        if ent.info:
+            ent.module = os.path.relpath(ent.info['file'],
+                                         os.path.dirname(self._fname))
 
     def lookup_entity(self, name, typ=None):
         ent = self._entity_dict.get(name)
@@ -1495,6 +1516,15 @@ class QAPISchema(object):
     def lookup_type(self, name):
         return self.lookup_entity(name, QAPISchemaType)
 
+    def _def_include(self, expr, info, doc):
+        include = expr['include']
+        assert doc is None
+        main_info = info
+        while main_info['parent']:
+            main_info = main_info['parent']
+        fname = os.path.relpath(include, os.path.dirname(main_info['file']))
+        self._def_entity(QAPISchemaInclude(fname, info))
+
     def _def_builtin_type(self, name, json_type, c_type):
         self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
         # TODO As long as we have QAPI_TYPES_BUILTIN to share multiple
@@ -1677,7 +1707,7 @@ class QAPISchema(object):
             elif 'event' in expr:
                 self._def_event(expr, info, doc)
             elif 'include' in expr:
-                pass
+                self._def_include(expr, info, doc)
             else:
                 assert False
 
@@ -1687,8 +1717,12 @@ class QAPISchema(object):
 
     def visit(self, visitor):
         visitor.visit_begin(self)
+        module = None
         for entity in self._entity_list:
             if visitor.visit_needed(entity):
+                if entity.module != module:
+                    module = entity.module
+                    visitor.visit_module(module)
                 entity.visit(visitor)
         visitor.visit_end()
 
tests/qapi-schema/comments.out +1 -0
index 0261ddf202..8d2f1ce8a2 100644
@@ -1,4 +1,5 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module comments.json
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/doc-bad-section.out +1 -0
index 23bf8c71ab..cd28721568 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module doc-bad-section.json
 enum Enum ['one', 'two']
 doc symbol=Enum
     body=
tests/qapi-schema/doc-good.out +1 -0
index 0c07301f07..430b5a87db 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module doc-good.json
 enum Enum ['one', 'two']
 object Base
     member base1: Enum optional=False
tests/qapi-schema/event-case.out +1 -0
index 110571b793..88c0964917 100644
@@ -1,5 +1,6 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module event-case.json
 event oops None
    boxed=False
tests/qapi-schema/ident-with-escape.out +1 -0
index 8336aa7629..ee3b34e623 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module ident-with-escape.json
 object q_obj_fooA-arg
     member bar1: str optional=False
 command fooA q_obj_fooA-arg -> None
tests/qapi-schema/include-relpath.out +5 -0
index 0261ddf202..ebbabd7a18 100644
@@ -1,4 +1,9 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module include-relpath.json
+include include/relpath.json
+module include/relpath.json
+include include-relpath-sub.json
+module include-relpath-sub.json
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-repetition.out +10 -0
index 0261ddf202..7235e055bc 100644
@@ -1,4 +1,14 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module include-repetition.json
+include comments.json
+module comments.json
 enum Status ['good', 'bad', 'ugly']
+module include-repetition.json
+include include-repetition-sub.json
+module include-repetition-sub.json
+include comments.json
+include comments.json
+module include-repetition.json
+include comments.json
tests/qapi-schema/include-simple.out +3 -0
index 0261ddf202..006f723eeb 100644
@@ -1,4 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module include-simple.json
+include include-simple-sub.json
+module include-simple-sub.json
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/indented-expr.out +1 -0
index 34de8be426..a79935e8c3 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module indented-expr.json
 command eins None -> None
    gen=True success_response=True boxed=False
 command zwei None -> None
tests/qapi-schema/qapi-schema-test.out +1 -0
index 50706b0136..012e7fc06a 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module qapi-schema-test.json
 object TestStruct
     member integer: int optional=False
     member boolean: bool optional=False
tests/qapi-schema/test-qapi.py +7 -1
index d6bb8ec6a4..2027eef3e5 100644
@@ -17,6 +17,13 @@ import sys
 
 
 class QAPISchemaTestVisitor(QAPISchemaVisitor):
+
+    def visit_module(self, name):
+        print 'module %s' % name
+
+    def visit_include(self, name, info):
+        print 'include %s' % name
+
     def visit_enum_type(self, name, info, values, prefix):
         print 'enum %s %s' % (name, values)
         if prefix:
--

2.13.6

#64 lore raw

RFC PATCH 16/21 qapi/types qapi/visit: Make visitors use QAPIGen more

In reply to Markus Armbruster

The conversion is rather shallow so far: most of the output accumulation is not converted. Take the next step: convert output accumulation in QAPISchemaGenTypeVisitor and QAPISchemaGenVisitVisitor. Helper functions outside these classes are not converted.

---
 scripts/qapi/types.py | 83 +++++++++++++++++++++++----------------------------
 scripts/qapi/visit.py | 83 +++++++++++++++++++++++----------------------------
 2 files changed, 74 insertions(+), 92 deletions(-)
scripts/qapi/types.py +37 -46
index b2095120e0..eeffcbf32c 100644
@@ -168,35 +168,44 @@ void qapi_free_%(c_name)s(%(c_name)s *obj)
 
 
 class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
-    def __init__(self, opt_builtins):
+    def __init__(self, opt_builtins, prefix):
         self._opt_builtins = opt_builtins
-        self.decl = None
-        self.defn = None
-        self._fwdecl = None
-        self._btin = None
+        self._prefix = prefix
+        blurb = ' * Schema-defined QAPI types'
+        self._genc = QAPIGenC(blurb, __doc__)
+        self._genh = QAPIGenH(blurb, __doc__)
+        self._genc.preamble(mcgen('''
+#include "qemu/osdep.h"
+#include "qapi/dealloc-visitor.h"
+#include "%(prefix)sqapi-types.h"
+#include "%(prefix)sqapi-visit.h"
+''',
+                                  prefix=prefix))
+        self._genh.preamble(mcgen('''
+#include "qapi/util.h"
+'''))
+        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
+
+    def write(self, output_dir):
+        self._genc.write(output_dir, self._prefix + 'qapi-types.c')
+        self._genh.write(output_dir, self._prefix + 'qapi-types.h')
 
     def visit_begin(self, schema):
         # gen_object() is recursive, ensure it doesn't visit the empty type
         objects_seen.add(schema.the_empty_object_type.name)
-        self.decl = ''
-        self.defn = ''
-        self._fwdecl = ''
-        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
 
     def visit_end(self):
-        self.decl = self._fwdecl + self.decl
-        self._fwdecl = None
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
         # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_TYPES_BUILTIN')
-        self.decl = self._btin + self.decl
+        self._genh.preamble(self._btin)
         self._btin = None
 
     def _gen_type_cleanup(self, name):
-        self.decl += gen_type_cleanup_decl(name)
-        self.defn += gen_type_cleanup(name)
+        self._genh.body(gen_type_cleanup_decl(name))
+        self._genc.body(gen_type_cleanup(name))
 
     def visit_enum_type(self, name, info, values, prefix):
         # Special case for our lone builtin enum type
@@ -204,10 +213,10 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         if not info:
             self._btin += gen_enum(name, values, prefix)
             if self._opt_builtins:
-                self.defn += gen_enum_lookup(name, values, prefix)
+                self._genc.body(gen_enum_lookup(name, values, prefix))
         else:
-            self._fwdecl += gen_enum(name, values, prefix)
-            self.defn += gen_enum_lookup(name, values, prefix)
+            self._genh.preamble(gen_enum(name, values, prefix))
+            self._genc.body(gen_enum_lookup(name, values, prefix))
 
     def visit_array_type(self, name, info, element_type):
         if isinstance(element_type, QAPISchemaBuiltinType):
@@ -215,20 +224,20 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
             self._btin += gen_array(name, element_type)
             self._btin += gen_type_cleanup_decl(name)
             if self._opt_builtins:
-                self.defn += gen_type_cleanup(name)
+                self._genc.body(gen_type_cleanup(name))
         else:
-            self._fwdecl += gen_fwd_object_or_array(name)
-            self.decl += gen_array(name, element_type)
+            self._genh.preamble(gen_fwd_object_or_array(name))
+            self._genh.body(gen_array(name, element_type))
             self._gen_type_cleanup(name)
 
     def visit_object_type(self, name, info, base, members, variants):
         # Nothing to do for the special empty builtin
         if name == 'q_empty':
             return
-        self._fwdecl += gen_fwd_object_or_array(name)
-        self.decl += gen_object(name, base, members, variants)
+        self._genh.preamble(gen_fwd_object_or_array(name))
+        self._genh.body(gen_object(name, base, members, variants))
         if base and not base.is_implicit():
-            self.decl += gen_upcast(name, base)
+            self._genh.body(gen_upcast(name, base))
         # TODO Worth changing the visitor signature, so we could
         # directly use rather than repeat type.is_implicit()?
         if not name.startswith('q_'):
@@ -236,31 +245,13 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
             self._gen_type_cleanup(name)
 
     def visit_alternate_type(self, name, info, variants):
-        self._fwdecl += gen_fwd_object_or_array(name)
-        self.decl += gen_object(name, None, [variants.tag_member], variants)
+        self._genh.preamble(gen_fwd_object_or_array(name))
+        self._genh.body(gen_object(name, None,
+                                   [variants.tag_member], variants))
         self._gen_type_cleanup(name)
 
 
 def gen_types(schema, output_dir, prefix, opt_builtins):
-    blurb = ' * Schema-defined QAPI types'
-    genc = QAPIGenC(blurb, __doc__)
-    genh = QAPIGenH(blurb, __doc__)
-
-    genc.body(mcgen('''
-#include "qemu/osdep.h"
-#include "qapi/dealloc-visitor.h"
-#include "%(prefix)sqapi-types.h"
-#include "%(prefix)sqapi-visit.h"
-''',
-                    prefix=prefix))
-
-    genh.body(mcgen('''
-#include "qapi/util.h"
-'''))
-
-    vis = QAPISchemaGenTypeVisitor(opt_builtins)
+    vis = QAPISchemaGenTypeVisitor(opt_builtins, prefix)
     schema.visit(vis)
-    genc.body(vis.defn)
-    genh.body(vis.decl)
-    genc.write(output_dir, prefix + 'qapi-types.c')
-    genh.write(output_dir, prefix + 'qapi-types.h')
+    vis.write(output_dir)
scripts/qapi/visit.py +37 -46

124 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by Marc-André Lureau on v1 16/21
#65 lore raw

Re: [Qemu-devel] [PATCH RFC 16/21] qapi/types qapi/visit: Make visitors use QAPIGen more

Quoted 7 lines · Markus Armbruster

The conversion is rather shallow so far: most of the output accumulation is not converted. Take the next step: convert output accumulation in QAPISchemaGenTypeVisitor and QAPISchemaGenVisitVisitor. Helper functions outside these classes are not converted.

Quoted 280 lines
---
 scripts/qapi/types.py | 83 +++++++++++++++++++++++----------------------------
 scripts/qapi/visit.py | 83 +++++++++++++++++++++++----------------------------
 2 files changed, 74 insertions(+), 92 deletions(-)
scripts/qapi/types.py +37 -46
index b2095120e0..eeffcbf32c 100644
@@ -168,35 +168,44 @@ void qapi_free_%(c_name)s(%(c_name)s *obj)
 
 
 class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
-    def __init__(self, opt_builtins):
+    def __init__(self, opt_builtins, prefix):
         self._opt_builtins = opt_builtins
-        self.decl = None
-        self.defn = None
-        self._fwdecl = None
-        self._btin = None
+        self._prefix = prefix
+        blurb = ' * Schema-defined QAPI types'
+        self._genc = QAPIGenC(blurb, __doc__)
+        self._genh = QAPIGenH(blurb, __doc__)
+        self._genc.preamble(mcgen('''
+#include "qemu/osdep.h"
+#include "qapi/dealloc-visitor.h"
+#include "%(prefix)sqapi-types.h"
+#include "%(prefix)sqapi-visit.h"
+''',
+                                  prefix=prefix))
+        self._genh.preamble(mcgen('''
+#include "qapi/util.h"
+'''))
+        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
+
+    def write(self, output_dir):
+        self._genc.write(output_dir, self._prefix + 'qapi-types.c')
+        self._genh.write(output_dir, self._prefix + 'qapi-types.h')
 
     def visit_begin(self, schema):
         # gen_object() is recursive, ensure it doesn't visit the empty type
         objects_seen.add(schema.the_empty_object_type.name)
-        self.decl = ''
-        self.defn = ''
-        self._fwdecl = ''
-        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
 
     def visit_end(self):
-        self.decl = self._fwdecl + self.decl
-        self._fwdecl = None
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
         # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_TYPES_BUILTIN')
-        self.decl = self._btin + self.decl
+        self._genh.preamble(self._btin)
         self._btin = None
 
     def _gen_type_cleanup(self, name):
-        self.decl += gen_type_cleanup_decl(name)
-        self.defn += gen_type_cleanup(name)
+        self._genh.body(gen_type_cleanup_decl(name))
+        self._genc.body(gen_type_cleanup(name))
 
     def visit_enum_type(self, name, info, values, prefix):
         # Special case for our lone builtin enum type
@@ -204,10 +213,10 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         if not info:
             self._btin += gen_enum(name, values, prefix)
             if self._opt_builtins:
-                self.defn += gen_enum_lookup(name, values, prefix)
+                self._genc.body(gen_enum_lookup(name, values, prefix))
         else:
-            self._fwdecl += gen_enum(name, values, prefix)
-            self.defn += gen_enum_lookup(name, values, prefix)
+            self._genh.preamble(gen_enum(name, values, prefix))
+            self._genc.body(gen_enum_lookup(name, values, prefix))
 
     def visit_array_type(self, name, info, element_type):
         if isinstance(element_type, QAPISchemaBuiltinType):
@@ -215,20 +224,20 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
             self._btin += gen_array(name, element_type)
             self._btin += gen_type_cleanup_decl(name)
             if self._opt_builtins:
-                self.defn += gen_type_cleanup(name)
+                self._genc.body(gen_type_cleanup(name))
         else:
-            self._fwdecl += gen_fwd_object_or_array(name)
-            self.decl += gen_array(name, element_type)
+            self._genh.preamble(gen_fwd_object_or_array(name))
+            self._genh.body(gen_array(name, element_type))
             self._gen_type_cleanup(name)
 
     def visit_object_type(self, name, info, base, members, variants):
         # Nothing to do for the special empty builtin
         if name == 'q_empty':
             return
-        self._fwdecl += gen_fwd_object_or_array(name)
-        self.decl += gen_object(name, base, members, variants)
+        self._genh.preamble(gen_fwd_object_or_array(name))
+        self._genh.body(gen_object(name, base, members, variants))
         if base and not base.is_implicit():
-            self.decl += gen_upcast(name, base)
+            self._genh.body(gen_upcast(name, base))
         # TODO Worth changing the visitor signature, so we could
         # directly use rather than repeat type.is_implicit()?
         if not name.startswith('q_'):
@@ -236,31 +245,13 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
             self._gen_type_cleanup(name)
 
     def visit_alternate_type(self, name, info, variants):
-        self._fwdecl += gen_fwd_object_or_array(name)
-        self.decl += gen_object(name, None, [variants.tag_member], variants)
+        self._genh.preamble(gen_fwd_object_or_array(name))
+        self._genh.body(gen_object(name, None,
+                                   [variants.tag_member], variants))
         self._gen_type_cleanup(name)
 
 
 def gen_types(schema, output_dir, prefix, opt_builtins):
-    blurb = ' * Schema-defined QAPI types'
-    genc = QAPIGenC(blurb, __doc__)
-    genh = QAPIGenH(blurb, __doc__)
-
-    genc.body(mcgen('''
-#include "qemu/osdep.h"
-#include "qapi/dealloc-visitor.h"
-#include "%(prefix)sqapi-types.h"
-#include "%(prefix)sqapi-visit.h"
-''',
-                    prefix=prefix))
-
-    genh.body(mcgen('''
-#include "qapi/util.h"
-'''))
-
-    vis = QAPISchemaGenTypeVisitor(opt_builtins)
+    vis = QAPISchemaGenTypeVisitor(opt_builtins, prefix)
     schema.visit(vis)
-    genc.body(vis.defn)
-    genh.body(vis.decl)
-    genc.write(output_dir, prefix + 'qapi-types.c')
-    genh.write(output_dir, prefix + 'qapi-types.h')
+    vis.write(output_dir)
scripts/qapi/visit.py +37 -47
index 80c0b85f8c..ee1a849574 100644
@@ -264,24 +264,38 @@ out:
 
 
 class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
-    def __init__(self, opt_builtins):
+    def __init__(self, opt_builtins, prefix):
         self._opt_builtins = opt_builtins
-        self.decl = None
-        self.defn = None
-        self._btin = None
-
-    def visit_begin(self, schema):
-        self.decl = ''
-        self.defn = ''
+        self._prefix = prefix
+        blurb = ' * Schema-defined QAPI visitors'
+        self._genc = QAPIGenC(blurb, __doc__)
+        self._genh = QAPIGenH(blurb, __doc__)
+        self._genc.preamble(mcgen('''
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "qapi/error.h"
+#include "%(prefix)sqapi-visit.h"
+''',
+                                  prefix=prefix))
+        self._genh.preamble(mcgen('''
+#include "qapi/visitor.h"
+#include "qapi/qmp/qerror.h"
+#include "%(prefix)sqapi-types.h"
+''',
+                                  prefix=prefix))
         self._btin = guardstart('QAPI_VISIT_BUILTIN')
 
+    def write(self, output_dir):
+        self._genc.write(output_dir, self._prefix + 'qapi-visit.c')
+        self._genh.write(output_dir, self._prefix + 'qapi-visit.h')
+
     def visit_end(self):
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
         # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_VISIT_BUILTIN')
-        self.decl = self._btin + self.decl
+        self._genh.preamble(self._btin)
         self._btin = None
 
     def visit_enum_type(self, name, info, values, prefix):
@@ -290,10 +304,10 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         if not info:
             self._btin += gen_visit_decl(name, scalar=True)
             if self._opt_builtins:
-                self.defn += gen_visit_enum(name)
+                self._genc.body(gen_visit_enum(name))
         else:
-            self.decl += gen_visit_decl(name, scalar=True)
-            self.defn += gen_visit_enum(name)
+            self._genh.body(gen_visit_decl(name, scalar=True))
+            self._genc.body(gen_visit_enum(name))
 
     def visit_array_type(self, name, info, element_type):
         decl = gen_visit_decl(name)
@@ -301,53 +315,30 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         if isinstance(element_type, QAPISchemaBuiltinType):
             self._btin += decl
             if self._opt_builtins:
-                self.defn += defn
+                self._genc.body(defn)
         else:
-            self.decl += decl
-            self.defn += defn
+            self._genh.body(decl)
+            self._genc.body(defn)
 
     def visit_object_type(self, name, info, base, members, variants):
         # Nothing to do for the special empty builtin
         if name == 'q_empty':
             return
-        self.decl += gen_visit_members_decl(name)
-        self.defn += gen_visit_object_members(name, base, members, variants)
+        self._genh.body(gen_visit_members_decl(name))
+        self._genc.body(gen_visit_object_members(name, base, members, variants))
         # TODO Worth changing the visitor signature, so we could
         # directly use rather than repeat type.is_implicit()?
         if not name.startswith('q_'):
             # only explicit types need an allocating visit
-            self.decl += gen_visit_decl(name)
-            self.defn += gen_visit_object(name, base, members, variants)
+            self._genh.body(gen_visit_decl(name))
+            self._genc.body(gen_visit_object(name, base, members, variants))
 
     def visit_alternate_type(self, name, info, variants):
-        self.decl += gen_visit_decl(name)
-        self.defn += gen_visit_alternate(name, variants)
+        self._genh.body(gen_visit_decl(name))
+        self._genc.body(gen_visit_alternate(name, variants))
 
 
 def gen_visit(schema, output_dir, prefix, opt_builtins):
-    blurb = ' * Schema-defined QAPI visitors'
-    genc = QAPIGenC(blurb, __doc__)
-    genh = QAPIGenH(blurb, __doc__)
-
-    genc.body(mcgen('''
-#include "qemu/osdep.h"
-#include "qemu-common.h"
-#include "qapi/error.h"
-#include "%(prefix)sqapi-visit.h"
-''',
-                    prefix=prefix))
-
-    genh.body(mcgen('''
-#include "qapi/visitor.h"
-#include "qapi/qmp/qerror.h"
-#include "%(prefix)sqapi-types.h"
-
-''',
-                    prefix=prefix))
-
-    vis = QAPISchemaGenVisitVisitor(opt_builtins)
+    vis = QAPISchemaGenVisitVisitor(opt_builtins, prefix)
     schema.visit(vis)
-    genc.body(vis.defn)
-    genh.body(vis.decl)
-    genc.write(output_dir, prefix + 'qapi-visit.c')
-    genh.write(output_dir, prefix + 'qapi-visit.h')
+    vis.write(output_dir)
--

2.13.6

#66 lore raw

RFC PATCH 17/21 qapi/types qapi/visit: Generate built-in stuff into separate files

In reply to Markus Armbruster

Linking code from multiple separate QAPI schemata into the same program is possible, but involves some weirdness around built-in types:

* We generate code for built-in types into .c only with option --builtins. The user is responsible to generate code for exactly one QAPI schema per program with --builtins.

* We generate code for them it into .h regardless of --builtins, guarded by #ifndef QAPI_VISIT_BUILTIN. Because the code for built-in types is exactly the same in all of them, including any combination of these headers works.

Replace this contraption by something more conventional: generate code for built-in types into their very own files: qapi-builtin-types.c, qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but only with --builtins. Obey --output-dir, but ignore --prefix for them.

Make qapi-types.h include qapi-builtin-types.h. With multiple schemata you now have multiple qapi-types.[ch], but only one qapi-builtin-types.[ch]. Same for qapi-visit.[ch] and qapi-builtin-visit.[ch].

Bonus: if all you need is built-in stuff, you can include a much smaller header. To be exploited shortly.

---
 Makefile               | 13 +++++---
 Makefile.objs          |  1 +
 scripts/qapi/common.py | 18 +++++------
 scripts/qapi/types.py  | 82 ++++++++++++++++++++++++++----------------------
 scripts/qapi/visit.py  | 84 ++++++++++++++++++++++++++++----------------------
 5 files changed, 111 insertions(+), 87 deletions(-)
Makefile +9 -4
index e02f0c13ef..f9b7900330 100644
@@ -88,10 +88,13 @@ endif
 include $(SRC_PATH)/rules.mak
 
 GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
-GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
-GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
-GENERATED_FILES += qmp-introspect.h
-GENERATED_FILES += qmp-introspect.c
+GENERATED_FILES += qmp-commands.h qmp-marshal.c
+GENERATED_FILES += qapi-builtin-types.h qapi-builtin-types.c
+GENERATED_FILES += qapi-types.h qapi-types.c
+GENERATED_FILES += qapi-builtin-visit.h qapi-builtin-visit.c
+GENERATED_FILES += qapi-visit.h qapi-visit.c
+GENERATED_FILES += qapi-event.h qapi-event.c
+GENERATED_FILES += qmp-introspect.c qmp-introspect.h
 GENERATED_FILES += qapi.texi
 
 GENERATED_FILES += trace/generated-tcg-tracers.h
@@ -514,7 +517,9 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/transaction.json \
                $(SRC_PATH)/qapi/ui.json
 
+qapi-builtin-types.c qapi-builtin-types.h \
 qapi-types.c qapi-types.h \
+qapi-builtin-visit.c qapi-builtin-visit.h \
 qapi-visit.c qapi-visit.h \
 qmp-commands.h qmp-marshal.c \
 qapi-event.c qapi-event.h \
Makefile.objs +1 -0
index 323ef12384..f16cca06e7 100644
@@ -2,6 +2,7 @@
 # Common libraries for tools and emulators
 stub-obj-y = stubs/ crypto/
 util-obj-y = util/ qobject/ qapi/
+util-obj-y += qapi-builtin-types.o qapi-builtin-visit.o
 util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o
 
 chardev-obj-y = chardev/
scripts/qapi/common.py +9 -9
index f4e9ebbb53..7ffffc78d9 100644
@@ -1527,11 +1527,10 @@ class QAPISchema(object):
 
     def _def_builtin_type(self, name, json_type, c_type):
         self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
-        # TODO As long as we have QAPI_TYPES_BUILTIN to share multiple
-        # qapi-types.h from a single .c, all arrays of builtins must be
-        # declared in the first file whether or not they are used.  Nicer
-        # would be to use lazy instantiation, while figuring out how to
-        # avoid compilation issues with multiple qapi-types.h.
+        # Instantiating only the arrays that are actually used would
+        # be nice, but we can't as long as their generated code
+        # (qapi-builtin-types.[ch]) may be shared by some other
+        # schema.
         self._make_array_type(name, None)
 
     def _def_predefineds(self):
@@ -1985,14 +1984,15 @@ class QAPIGen(object):
         return ''
 
     def write(self, output_dir, fname):
-        if output_dir:
+        pathname = os.path.join(output_dir, fname)
+        dir = os.path.dirname(pathname)
+        if dir:
             try:
-                os.makedirs(output_dir)
+                os.makedirs(dir)
             except os.error as e:
                 if e.errno != errno.EEXIST:
                     raise
-        fd = os.open(os.path.join(output_dir, fname),
-                     os.O_RDWR | os.O_CREAT, 0666)
+        fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0666)
         f = os.fdopen(fd, 'r+')
         text = (self.top(fname) + self._preamble + self._body
                 + self.bottom(fname))
scripts/qapi/types.py +45 -37

109 lines not on this page · raw message

scripts/qapi/visit.py +47 -37

105 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by Marc-André Lureau on v1 17/21
#67 lore raw

Re: [Qemu-devel] [PATCH RFC 17/21] qapi/types qapi/visit: Generate built-in stuff into separate files

Quoted 24 lines · Markus Armbruster

Linking code from multiple separate QAPI schemata into the same program is possible, but involves some weirdness around built-in types:

* We generate code for built-in types into .c only with option --builtins. The user is responsible to generate code for exactly one QAPI schema per program with --builtins.

* We generate code for them it into .h regardless of --builtins, guarded by #ifndef QAPI_VISIT_BUILTIN. Because the code for built-in types is exactly the same in all of them, including any combination of these headers works.

Replace this contraption by something more conventional: generate code for built-in types into their very own files: qapi-builtin-types.c, qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but only with --builtins. Obey --output-dir, but ignore --prefix for them.

Make qapi-types.h include qapi-builtin-types.h. With multiple schemata you now have multiple qapi-types.[ch], but only one qapi-builtin-types.[ch]. Same for qapi-visit.[ch] and qapi-builtin-visit.[ch].

Quoted 4 lines

Bonus: if all you need is built-in stuff, you can include a much smaller header. To be exploited shortly.

Quoted 319 lines
---
 Makefile               | 13 +++++---
 Makefile.objs          |  1 +
 scripts/qapi/common.py | 18 +++++------
 scripts/qapi/types.py  | 82 ++++++++++++++++++++++++++----------------------
 scripts/qapi/visit.py  | 84 ++++++++++++++++++++++++++++----------------------
 5 files changed, 111 insertions(+), 87 deletions(-)
Makefile +9 -4
index e02f0c13ef..f9b7900330 100644
@@ -88,10 +88,13 @@ endif
 include $(SRC_PATH)/rules.mak
 
 GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
-GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
-GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
-GENERATED_FILES += qmp-introspect.h
-GENERATED_FILES += qmp-introspect.c
+GENERATED_FILES += qmp-commands.h qmp-marshal.c
+GENERATED_FILES += qapi-builtin-types.h qapi-builtin-types.c
+GENERATED_FILES += qapi-types.h qapi-types.c
+GENERATED_FILES += qapi-builtin-visit.h qapi-builtin-visit.c
+GENERATED_FILES += qapi-visit.h qapi-visit.c
+GENERATED_FILES += qapi-event.h qapi-event.c
+GENERATED_FILES += qmp-introspect.c qmp-introspect.h
 GENERATED_FILES += qapi.texi
 
 GENERATED_FILES += trace/generated-tcg-tracers.h
@@ -514,7 +517,9 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/transaction.json \
                $(SRC_PATH)/qapi/ui.json
 
+qapi-builtin-types.c qapi-builtin-types.h \
 qapi-types.c qapi-types.h \
+qapi-builtin-visit.c qapi-builtin-visit.h \
 qapi-visit.c qapi-visit.h \
 qmp-commands.h qmp-marshal.c \
 qapi-event.c qapi-event.h \
Makefile.objs +1 -0
index 323ef12384..f16cca06e7 100644
@@ -2,6 +2,7 @@
 # Common libraries for tools and emulators
 stub-obj-y = stubs/ crypto/
 util-obj-y = util/ qobject/ qapi/
+util-obj-y += qapi-builtin-types.o qapi-builtin-visit.o
 util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o
 
 chardev-obj-y = chardev/
scripts/qapi/common.py +9 -9
index f4e9ebbb53..7ffffc78d9 100644
@@ -1527,11 +1527,10 @@ class QAPISchema(object):
 
     def _def_builtin_type(self, name, json_type, c_type):
         self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
-        # TODO As long as we have QAPI_TYPES_BUILTIN to share multiple
-        # qapi-types.h from a single .c, all arrays of builtins must be
-        # declared in the first file whether or not they are used.  Nicer
-        # would be to use lazy instantiation, while figuring out how to
-        # avoid compilation issues with multiple qapi-types.h.
+        # Instantiating only the arrays that are actually used would
+        # be nice, but we can't as long as their generated code
+        # (qapi-builtin-types.[ch]) may be shared by some other
+        # schema.
         self._make_array_type(name, None)
 
     def _def_predefineds(self):
@@ -1985,14 +1984,15 @@ class QAPIGen(object):
         return ''
 
     def write(self, output_dir, fname):
-        if output_dir:
+        pathname = os.path.join(output_dir, fname)
+        dir = os.path.dirname(pathname)
+        if dir:
             try:
-                os.makedirs(output_dir)
+                os.makedirs(dir)
             except os.error as e:
                 if e.errno != errno.EEXIST:
                     raise
-        fd = os.open(os.path.join(output_dir, fname),
-                     os.O_RDWR | os.O_CREAT, 0666)
+        fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0666)
         f = os.fdopen(fd, 'r+')
         text = (self.top(fname) + self._preamble + self._body
                 + self.bottom(fname))
scripts/qapi/types.py +45 -37
index eeffcbf32c..f84ed17960 100644
@@ -171,64 +171,72 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def __init__(self, opt_builtins, prefix):
         self._opt_builtins = opt_builtins
         self._prefix = prefix
-        blurb = ' * Schema-defined QAPI types'
-        self._genc = QAPIGenC(blurb, __doc__)
-        self._genh = QAPIGenH(blurb, __doc__)
+        self._module = {}
+        self._add_module(None, ' * Built-in QAPI types')
         self._genc.preamble(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
-#include "%(prefix)sqapi-types.h"
-#include "%(prefix)sqapi-visit.h"
-''',
-                                  prefix=prefix))
+#include "qapi-builtin-types.h"
+#include "qapi-builtin-visit.h"
+'''))
         self._genh.preamble(mcgen('''
 #include "qapi/util.h"
 '''))
-        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
+
+    def _module_basename(self, name):
+        if name is None:
+            return 'qapi-builtin-types'
+        return self._prefix + 'qapi-types'
+
+    def _add_module(self, name, blurb):
+        genc = QAPIGenC(blurb, __doc__)
+        genh = QAPIGenH(blurb, __doc__)
+        self._module[name] = (genc, genh)
+        self._set_module(name)
+
+    def _set_module(self, name):
+        self._genc, self._genh = self._module[name]
 
     def write(self, output_dir):
-        self._genc.write(output_dir, self._prefix + 'qapi-types.c')
-        self._genh.write(output_dir, self._prefix + 'qapi-types.h')
+        for name in self._module:
+            if name is None and not self._opt_builtins:
+                continue
+            basename = self._module_basename(name)
+            (genc, genh) = self._module[name]
+            genc.write(output_dir, basename + '.c')
+            genh.write(output_dir, basename + '.h')
 
     def visit_begin(self, schema):
         # gen_object() is recursive, ensure it doesn't visit the empty type
         objects_seen.add(schema.the_empty_object_type.name)
 
-    def visit_end(self):
-        # To avoid header dependency hell, we always generate
-        # declarations for built-in types in our header files and
-        # simply guard them.  See also opt_builtins (command line
-        # option -b).
-        self._btin += guardend('QAPI_TYPES_BUILTIN')
-        self._genh.preamble(self._btin)
-        self._btin = None
+    def visit_module(self, name):
+        if len(self._module) != 1:
+            return
+        self._add_module(name, ' * Schema-defined QAPI types')
+        self._genc.preamble(mcgen('''
+#include "qemu/osdep.h"
+#include "qapi/dealloc-visitor.h"
+#include "%(prefix)sqapi-types.h"
+#include "%(prefix)sqapi-visit.h"
+''',
+                                  prefix=self._prefix))
+        self._genh.preamble(mcgen('''
+#include "qapi-builtin-types.h"
+'''))
 
     def _gen_type_cleanup(self, name):
         self._genh.body(gen_type_cleanup_decl(name))
         self._genc.body(gen_type_cleanup(name))
 
     def visit_enum_type(self, name, info, values, prefix):
-        # Special case for our lone builtin enum type
-        # TODO use something cleaner than existence of info
-        if not info:
-            self._btin += gen_enum(name, values, prefix)
-            if self._opt_builtins:
-                self._genc.body(gen_enum_lookup(name, values, prefix))
-        else:
-            self._genh.preamble(gen_enum(name, values, prefix))
-            self._genc.body(gen_enum_lookup(name, values, prefix))
+        self._genh.preamble(gen_enum(name, values, prefix))
+        self._genc.body(gen_enum_lookup(name, values, prefix))
 
     def visit_array_type(self, name, info, element_type):
-        if isinstance(element_type, QAPISchemaBuiltinType):
-            self._btin += gen_fwd_object_or_array(name)
-            self._btin += gen_array(name, element_type)
-            self._btin += gen_type_cleanup_decl(name)
-            if self._opt_builtins:
-                self._genc.body(gen_type_cleanup(name))
-        else:
-            self._genh.preamble(gen_fwd_object_or_array(name))
-            self._genh.body(gen_array(name, element_type))
-            self._gen_type_cleanup(name)
+        self._genh.preamble(gen_fwd_object_or_array(name))
+        self._genh.body(gen_array(name, element_type))
+        self._gen_type_cleanup(name)
 
     def visit_object_type(self, name, info, base, members, variants):
         # Nothing to do for the special empty builtin
scripts/qapi/visit.py +47 -38
index ee1a849574..f23e71bb9a 100644
@@ -267,58 +267,68 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
     def __init__(self, opt_builtins, prefix):
         self._opt_builtins = opt_builtins
         self._prefix = prefix
-        blurb = ' * Schema-defined QAPI visitors'
-        self._genc = QAPIGenC(blurb, __doc__)
-        self._genh = QAPIGenH(blurb, __doc__)
+        self._module = {}
+        self._add_module(None, ' * Built-in QAPI visitors')
         self._genc.preamble(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qapi/error.h"
-#include "%(prefix)sqapi-visit.h"
-''',
-                                  prefix=prefix))
+#include "qapi-builtin-visit.h"
+'''))
         self._genh.preamble(mcgen('''
 #include "qapi/visitor.h"
 #include "qapi/qmp/qerror.h"
-#include "%(prefix)sqapi-types.h"
+#include "qapi-builtin-types.h"
 ''',
-                                  prefix=prefix))
-        self._btin = guardstart('QAPI_VISIT_BUILTIN')
+                              prefix=prefix))
+
+    def _module_basename(self, name):
+        if name is None:
+            return 'qapi-builtin-visit'
+        return self._prefix + 'qapi-visit'
+
+    def _add_module(self, name, blurb):
+        genc = QAPIGenC(blurb, __doc__)
+        genh = QAPIGenH(blurb, __doc__)
+        self._module[name] = (genc, genh)
+        self._set_module(name)
+
+    def _set_module(self, name):
+        self._genc, self._genh = self._module[name]
 
     def write(self, output_dir):
-        self._genc.write(output_dir, self._prefix + 'qapi-visit.c')
-        self._genh.write(output_dir, self._prefix + 'qapi-visit.h')
+        for name in self._module:
+            if name is None and not self._opt_builtins:
+                continue
+            basename = self._module_basename(name)
+            (genc, genh) = self._module[name]
+            genc.write(output_dir, basename + '.c')
+            genh.write(output_dir, basename + '.h')
 
-    def visit_end(self):
-        # To avoid header dependency hell, we always generate
-        # declarations for built-in types in our header files and
-        # simply guard them.  See also opt_builtins (command line
-        # option -b).
-        self._btin += guardend('QAPI_VISIT_BUILTIN')
-        self._genh.preamble(self._btin)
-        self._btin = None
+    def visit_module(self, name):
+        if len(self._module) != 1:
+            return
+        self._add_module(name, ' * Schema-defined QAPI visitors')
+        self._genc.preamble(mcgen('''
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "qapi/error.h"
+#include "%(prefix)sqapi-visit.h"
+''',
+                                  prefix=self._prefix))
+        self._genh.preamble(mcgen('''
+#include "qapi-builtin-visit.h"
+#include "%(prefix)sqapi-types.h"
+''',
+                                  prefix=self._prefix))
 
     def visit_enum_type(self, name, info, values, prefix):
-        # Special case for our lone builtin enum type
-        # TODO use something cleaner than existence of info
-        if not info:
-            self._btin += gen_visit_decl(name, scalar=True)
-            if self._opt_builtins:
-                self._genc.body(gen_visit_enum(name))
-        else:
-            self._genh.body(gen_visit_decl(name, scalar=True))
-            self._genc.body(gen_visit_enum(name))
+        self._genh.body(gen_visit_decl(name, scalar=True))
+        self._genc.body(gen_visit_enum(name))
 
     def visit_array_type(self, name, info, element_type):
-        decl = gen_visit_decl(name)
-        defn = gen_visit_list(name, element_type)
-        if isinstance(element_type, QAPISchemaBuiltinType):
-            self._btin += decl
-            if self._opt_builtins:
-                self._genc.body(defn)
-        else:
-            self._genh.body(decl)
-            self._genc.body(defn)
+        self._genh.body(gen_visit_decl(name))
+        self._genc.body(gen_visit_list(name, element_type))
 
     def visit_object_type(self, name, info, base, members, variants):
         # Nothing to do for the special empty builtin
--

2.13.6

#68 lore raw

Re: [Qemu-devel] [PATCH RFC 17/21] qapi/types qapi/visit: Generate built-in stuff into separate files

In reply to Markus Armbruster
Quoted 9 lines · Markus Armbruster

Linking code from multiple separate QAPI schemata into the same program is possible, but involves some weirdness around built-in types:

* We generate code for built-in types into .c only with option --builtins. The user is responsible to generate code for exactly one QAPI schema per program with --builtins.

* We generate code for them it into .h regardless of --builtins,

s/them it/them/

Quoted 20 lines

guarded by #ifndef QAPI_VISIT_BUILTIN. Because the code for built-in types is exactly the same in all of them, including any combination of these headers works.

Replace this contraption by something more conventional: generate code for built-in types into their very own files: qapi-builtin-types.c, qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but only with --builtins. Obey --output-dir, but ignore --prefix for them.

Make qapi-types.h include qapi-builtin-types.h. With multiple schemata you now have multiple qapi-types.[ch], but only one qapi-builtin-types.[ch]. Same for qapi-visit.[ch] and qapi-builtin-visit.[ch].

Bonus: if all you need is built-in stuff, you can include a much smaller header. To be exploited shortly.

---

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#69 lore raw

RFC PATCH 18/21 qapi/common: Fix guardname() for funny filenames

In reply to Markus Armbruster

guardname() fails to return a valid C identifier for arguments containing anything but [A-Za-z0-9_.-']. Fix that.

---
 scripts/qapi/common.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/common.py +1 -1
index 7ffffc78d9..7d497b5b17 100644
@@ -1860,7 +1860,7 @@ def mcgen(code, **kwds):
 
 
 def guardname(filename):
-    return c_name(filename, protect=False).upper()
+    return re.sub(r'[^A-Za-z0-9_]', '_', filename).upper()
 
 
 def guardstart(name):
Signature · 1 line
2.13.6
Reviewed-by Marc-André Lureau on v1 18/21
#70 lore raw

Re: [Qemu-devel] [PATCH RFC 18/21] qapi/common: Fix guardname() for funny filenames

Quoted 4 lines · Markus Armbruster

guardname() fails to return a valid C identifier for arguments containing anything but [A-Za-z0-9_.-']. Fix that.

Quoted 20 lines
---
 scripts/qapi/common.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/common.py +1 -2
index 7ffffc78d9..7d497b5b17 100644
@@ -1860,7 +1860,7 @@ def mcgen(code, **kwds):
 
 
 def guardname(filename):
-    return c_name(filename, protect=False).upper()
+    return re.sub(r'[^A-Za-z0-9_]', '_', filename).upper()
 
 
 def guardstart(name):
--

2.13.6

Reviewed-by on v1 18/21
#71 lore raw

Re: [Qemu-devel] [PATCH RFC 18/21] qapi/common: Fix guardname() for funny filenames

In reply to Markus Armbruster
Quoted 13 lines · Markus Armbruster

guardname() fails to return a valid C identifier for arguments containing anything but [A-Za-z0-9_.-']. Fix that.

---
  scripts/qapi/common.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/common.py +0 -0
index 7ffffc78d9..7d497b5b17 100644

scripts/qapi/common.py def mcgen(code, **kwds): · line 1863

Quoted 5 lines
scripts/qapi/common.py +1 -1
@@ -1860,7 +1860,7 @@ def mcgen(code, **kwds):
 
 
  def guardname(filename):
-    return c_name(filename, protect=False).upper()
+    return re.sub(r'[^A-Za-z0-9_]', '_', filename).upper()

For some choices of filename, the old code prefixes a q_ (via c_name) which gets turned into Q_ in the final guard name. The new code does not. Then again, all of the names protected by c_name() all contain lower case, while guard names are all upper case; so we aren't protecting ourselves from defining a reserved word; the main use for c_name() is to protect ourselves where we are not changing case (for example, _BOOL is no better than Q__BOOL as a guard name for a file named _Bool).

Might be worth mentioning this design consideration in the commit message, but the change itself is reasonable.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#72 lore raw

RFC PATCH 19/21 qapi/types: Generate separate .h, .c for each module

In reply to Markus Armbruster

Our qapi-schema.json is composed of modules connected by include directives, but the generated code is monolithic all the same: one qapi-types.h with all the types, one qapi-visit.h with all the visitors, and so forth. These monolithic headers get included all over the place. In my "build everything" tree, adding a QAPI type recompiles about 4500 out of 4800 objects.

Nobody would write such monolithic headers by hand. It stands to reason that one shouldn't generate them, either.

Split up generated qapi-types.h to mirror the schema's modular
structure: one header per module. Name the main module's header qapi-types.h, and sub-module D/B.json's header D/qapi-types-B.h.

Mirror the schema's includes in the headers, so that qapi-types.h gets you everything exactly as before. If you need less, you can include one or more of the sub-module headers. To be exploited shortly.

Split up qapi-types.c similarly.

---
 Makefile              | 30 ++++++++++++++++++++++++++++++
 Makefile.objs         | 18 +++++++++++++++++-
 scripts/qapi/types.py | 18 ++++++++++++++++--
 3 files changed, 63 insertions(+), 3 deletions(-)
Makefile +30 -0
index f9b7900330..f1b68dca9b 100644
@@ -91,6 +91,21 @@ GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
 GENERATED_FILES += qmp-commands.h qmp-marshal.c
 GENERATED_FILES += qapi-builtin-types.h qapi-builtin-types.c
 GENERATED_FILES += qapi-types.h qapi-types.c
+GENERATED_FILES += qapi/qapi-types-block-core.h qapi/qapi-types-block-core.c
+GENERATED_FILES += qapi/qapi-types-block.h qapi/qapi-types-block.c
+GENERATED_FILES += qapi/qapi-types-char.h qapi/qapi-types-char.c
+GENERATED_FILES += qapi/qapi-types-common.h qapi/qapi-types-common.c
+GENERATED_FILES += qapi/qapi-types-crypto.h qapi/qapi-types-crypto.c
+GENERATED_FILES += qapi/qapi-types-introspect.h qapi/qapi-types-introspect.c
+GENERATED_FILES += qapi/qapi-types-migration.h qapi/qapi-types-migration.c
+GENERATED_FILES += qapi/qapi-types-net.h qapi/qapi-types-net.c
+GENERATED_FILES += qapi/qapi-types-rocker.h qapi/qapi-types-rocker.c
+GENERATED_FILES += qapi/qapi-types-run-state.h qapi/qapi-types-run-state.c
+GENERATED_FILES += qapi/qapi-types-sockets.h qapi/qapi-types-sockets.c
+GENERATED_FILES += qapi/qapi-types-tpm.h qapi/qapi-types-tpm.c
+GENERATED_FILES += qapi/qapi-types-trace.h qapi/qapi-types-trace.c
+GENERATED_FILES += qapi/qapi-types-transaction.h qapi/qapi-types-transaction.c
+GENERATED_FILES += qapi/qapi-types-ui.h qapi/qapi-types-ui.c
 GENERATED_FILES += qapi-builtin-visit.h qapi-builtin-visit.c
 GENERATED_FILES += qapi-visit.h qapi-visit.c
 GENERATED_FILES += qapi-event.h qapi-event.c
@@ -519,6 +534,21 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
 
 qapi-builtin-types.c qapi-builtin-types.h \
 qapi-types.c qapi-types.h \
+qapi/qapi-types-block-core.c qapi/qapi-types-block-core.h \
+qapi/qapi-types-block.c qapi/qapi-types-block.h \
+qapi/qapi-types-char.c qapi/qapi-types-char.h \
+qapi/qapi-types-common.c qapi/qapi-types-common.h \
+qapi/qapi-types-crypto.c qapi/qapi-types-crypto.h \
+qapi/qapi-types-introspect.c qapi/qapi-types-introspect.h \
+qapi/qapi-types-migration.c qapi/qapi-types-migration.h \
+qapi/qapi-types-net.c qapi/qapi-types-net.h \
+qapi/qapi-types-rocker.c qapi/qapi-types-rocker.h \
+qapi/qapi-types-run-state.c qapi/qapi-types-run-state.h \
+qapi/qapi-types-sockets.c qapi/qapi-types-sockets.h \
+qapi/qapi-types-tpm.c qapi/qapi-types-tpm.h \
+qapi/qapi-types-trace.c qapi/qapi-types-trace.h \
+qapi/qapi-types-transaction.c qapi/qapi-types-transaction.h \
+qapi/qapi-types-ui.c qapi/qapi-types-ui.h \
 qapi-builtin-visit.c qapi-builtin-visit.h \
 qapi-visit.c qapi-visit.h \
 qmp-commands.h qmp-marshal.c \
Makefile.objs +17 -1
index f16cca06e7..e7411a2658 100644
@@ -3,7 +3,23 @@
 stub-obj-y = stubs/ crypto/
 util-obj-y = util/ qobject/ qapi/
 util-obj-y += qapi-builtin-types.o qapi-builtin-visit.o
-util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o
+util-obj-y += qapi-types.o
+util-obj-y += qapi/qapi-types-block-core.o
+util-obj-y += qapi/qapi-types-block.o
+util-obj-y += qapi/qapi-types-char.o
+util-obj-y += qapi/qapi-types-common.o
+util-obj-y += qapi/qapi-types-crypto.o
+util-obj-y += qapi/qapi-types-introspect.o
+util-obj-y += qapi/qapi-types-migration.o
+util-obj-y += qapi/qapi-types-net.o
+util-obj-y += qapi/qapi-types-rocker.o
+util-obj-y += qapi/qapi-types-run-state.o
+util-obj-y += qapi/qapi-types-sockets.o
+util-obj-y += qapi/qapi-types-tpm.o
+util-obj-y += qapi/qapi-types-trace.o
+util-obj-y += qapi/qapi-types-transaction.o
+util-obj-y += qapi/qapi-types-ui.o
+util-obj-y += qmp-introspect.o qapi-visit.o qapi-event.o
 
 chardev-obj-y = chardev/
 
scripts/qapi/types.py +16 -2
index f84ed17960..7bd8e1a978 100644
@@ -172,6 +172,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self._opt_builtins = opt_builtins
         self._prefix = prefix
         self._module = {}
+        self._main_module = None
         self._add_module(None, ' * Built-in QAPI types')
         self._genc.preamble(mcgen('''
 #include "qemu/osdep.h"
@@ -186,7 +187,11 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def _module_basename(self, name):
         if name is None:
             return 'qapi-builtin-types'
-        return self._prefix + 'qapi-types'
+        basename = os.path.join(os.path.dirname(name),
+                                self._prefix + 'qapi-types')
+        if name == self._main_module:
+            return basename
+        return basename + '-' + os.path.splitext(os.path.basename(name))[0]
 
     def _add_module(self, name, blurb):
         genc = QAPIGenC(blurb, __doc__)
@@ -211,7 +216,10 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         objects_seen.add(schema.the_empty_object_type.name)
 
     def visit_module(self, name):
-        if len(self._module) != 1:
+        if self._main_module is None:
+            self._main_module = name
+        if name in self._module:
+            self._set_module(name)
             return
         self._add_module(name, ' * Schema-defined QAPI types')
         self._genc.preamble(mcgen('''
@@ -225,6 +233,12 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 #include "qapi-builtin-types.h"
 '''))
 
+    def visit_include(self, name, info):
+        self._genh.preamble(mcgen('''
+#include "%(basename)s.h"
+''',
+                                  basename=self._module_basename(name)))
+
     def _gen_type_cleanup(self, name):
         self._genh.body(gen_type_cleanup_decl(name))
         self._genc.body(gen_type_cleanup(name))
Signature · 1 line
2.13.6
Reviewed-by Marc-André Lureau on v1 19/21
#73 lore raw

Re: [Qemu-devel] [PATCH RFC 19/21] qapi/types: Generate separate .h, .c for each module

Quoted 14 lines · Markus Armbruster

Our qapi-schema.json is composed of modules connected by include directives, but the generated code is monolithic all the same: one qapi-types.h with all the types, one qapi-visit.h with all the visitors, and so forth. These monolithic headers get included all over the place. In my "build everything" tree, adding a QAPI type recompiles about 4500 out of 4800 objects.

Nobody would write such monolithic headers by hand. It stands to reason that one shouldn't generate them, either.

Split up generated qapi-types.h to mirror the schema's modular
structure: one header per module. Name the main module's header qapi-types.h, and sub-module D/B.json's header D/qapi-types-B.h.

Quoted 7 lines

Mirror the schema's includes in the headers, so that qapi-types.h gets you everything exactly as before. If you need less, you can include one or more of the sub-module headers. To be exploited shortly.

Split up qapi-types.c similarly.

Most of the necessary split work is done in previous patch, this enables it:

Quoted 136 lines
---
 Makefile              | 30 ++++++++++++++++++++++++++++++
 Makefile.objs         | 18 +++++++++++++++++-
 scripts/qapi/types.py | 18 ++++++++++++++++--
 3 files changed, 63 insertions(+), 3 deletions(-)
Makefile +30 -0
index f9b7900330..f1b68dca9b 100644
@@ -91,6 +91,21 @@ GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
 GENERATED_FILES += qmp-commands.h qmp-marshal.c
 GENERATED_FILES += qapi-builtin-types.h qapi-builtin-types.c
 GENERATED_FILES += qapi-types.h qapi-types.c
+GENERATED_FILES += qapi/qapi-types-block-core.h qapi/qapi-types-block-core.c
+GENERATED_FILES += qapi/qapi-types-block.h qapi/qapi-types-block.c
+GENERATED_FILES += qapi/qapi-types-char.h qapi/qapi-types-char.c
+GENERATED_FILES += qapi/qapi-types-common.h qapi/qapi-types-common.c
+GENERATED_FILES += qapi/qapi-types-crypto.h qapi/qapi-types-crypto.c
+GENERATED_FILES += qapi/qapi-types-introspect.h qapi/qapi-types-introspect.c
+GENERATED_FILES += qapi/qapi-types-migration.h qapi/qapi-types-migration.c
+GENERATED_FILES += qapi/qapi-types-net.h qapi/qapi-types-net.c
+GENERATED_FILES += qapi/qapi-types-rocker.h qapi/qapi-types-rocker.c
+GENERATED_FILES += qapi/qapi-types-run-state.h qapi/qapi-types-run-state.c
+GENERATED_FILES += qapi/qapi-types-sockets.h qapi/qapi-types-sockets.c
+GENERATED_FILES += qapi/qapi-types-tpm.h qapi/qapi-types-tpm.c
+GENERATED_FILES += qapi/qapi-types-trace.h qapi/qapi-types-trace.c
+GENERATED_FILES += qapi/qapi-types-transaction.h qapi/qapi-types-transaction.c
+GENERATED_FILES += qapi/qapi-types-ui.h qapi/qapi-types-ui.c
 GENERATED_FILES += qapi-builtin-visit.h qapi-builtin-visit.c
 GENERATED_FILES += qapi-visit.h qapi-visit.c
 GENERATED_FILES += qapi-event.h qapi-event.c
@@ -519,6 +534,21 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
 
 qapi-builtin-types.c qapi-builtin-types.h \
 qapi-types.c qapi-types.h \
+qapi/qapi-types-block-core.c qapi/qapi-types-block-core.h \
+qapi/qapi-types-block.c qapi/qapi-types-block.h \
+qapi/qapi-types-char.c qapi/qapi-types-char.h \
+qapi/qapi-types-common.c qapi/qapi-types-common.h \
+qapi/qapi-types-crypto.c qapi/qapi-types-crypto.h \
+qapi/qapi-types-introspect.c qapi/qapi-types-introspect.h \
+qapi/qapi-types-migration.c qapi/qapi-types-migration.h \
+qapi/qapi-types-net.c qapi/qapi-types-net.h \
+qapi/qapi-types-rocker.c qapi/qapi-types-rocker.h \
+qapi/qapi-types-run-state.c qapi/qapi-types-run-state.h \
+qapi/qapi-types-sockets.c qapi/qapi-types-sockets.h \
+qapi/qapi-types-tpm.c qapi/qapi-types-tpm.h \
+qapi/qapi-types-trace.c qapi/qapi-types-trace.h \
+qapi/qapi-types-transaction.c qapi/qapi-types-transaction.h \
+qapi/qapi-types-ui.c qapi/qapi-types-ui.h \
 qapi-builtin-visit.c qapi-builtin-visit.h \
 qapi-visit.c qapi-visit.h \
 qmp-commands.h qmp-marshal.c \
Makefile.objs +17 -1
index f16cca06e7..e7411a2658 100644
@@ -3,7 +3,23 @@
 stub-obj-y = stubs/ crypto/
 util-obj-y = util/ qobject/ qapi/
 util-obj-y += qapi-builtin-types.o qapi-builtin-visit.o
-util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o
+util-obj-y += qapi-types.o
+util-obj-y += qapi/qapi-types-block-core.o
+util-obj-y += qapi/qapi-types-block.o
+util-obj-y += qapi/qapi-types-char.o
+util-obj-y += qapi/qapi-types-common.o
+util-obj-y += qapi/qapi-types-crypto.o
+util-obj-y += qapi/qapi-types-introspect.o
+util-obj-y += qapi/qapi-types-migration.o
+util-obj-y += qapi/qapi-types-net.o
+util-obj-y += qapi/qapi-types-rocker.o
+util-obj-y += qapi/qapi-types-run-state.o
+util-obj-y += qapi/qapi-types-sockets.o
+util-obj-y += qapi/qapi-types-tpm.o
+util-obj-y += qapi/qapi-types-trace.o
+util-obj-y += qapi/qapi-types-transaction.o
+util-obj-y += qapi/qapi-types-ui.o
+util-obj-y += qmp-introspect.o qapi-visit.o qapi-event.o
 
 chardev-obj-y = chardev/
 
scripts/qapi/types.py +16 -3
index f84ed17960..7bd8e1a978 100644
@@ -172,6 +172,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self._opt_builtins = opt_builtins
         self._prefix = prefix
         self._module = {}
+        self._main_module = None
         self._add_module(None, ' * Built-in QAPI types')
         self._genc.preamble(mcgen('''
 #include "qemu/osdep.h"
@@ -186,7 +187,11 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def _module_basename(self, name):
         if name is None:
             return 'qapi-builtin-types'
-        return self._prefix + 'qapi-types'
+        basename = os.path.join(os.path.dirname(name),
+                                self._prefix + 'qapi-types')
+        if name == self._main_module:
+            return basename
+        return basename + '-' + os.path.splitext(os.path.basename(name))[0]
 
     def _add_module(self, name, blurb):
         genc = QAPIGenC(blurb, __doc__)
@@ -211,7 +216,10 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         objects_seen.add(schema.the_empty_object_type.name)
 
     def visit_module(self, name):
-        if len(self._module) != 1:
+        if self._main_module is None:
+            self._main_module = name
+        if name in self._module:
+            self._set_module(name)
             return
         self._add_module(name, ' * Schema-defined QAPI types')
         self._genc.preamble(mcgen('''
@@ -225,6 +233,12 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 #include "qapi-builtin-types.h"
 '''))
 
+    def visit_include(self, name, info):
+        self._genh.preamble(mcgen('''
+#include "%(basename)s.h"
+''',
+                                  basename=self._module_basename(name)))
+
     def _gen_type_cleanup(self, name):
         self._genh.body(gen_type_cleanup_decl(name))
         self._genc.body(gen_type_cleanup(name))
--

2.13.6

#74 lore raw

RFC PATCH 20/21 Include less of qapi-types.h

In reply to Markus Armbruster

In my "build everything" tree, a change to the types in qapi-schema.json triggers a recompile of about 4500 out of 4800 objects.

The previous commit split up the generated qapi-types.h. Replace includes of qapi-types.h (i.e. all types) by includes of parts where possible.

To illustrate the benefits: adding a type to qapi/migration.json now recompiles some 2300 instead of 4500 objects. The next commit will improve it further.

---
 crypto/cipherpriv.h              | 2 +-
 hw/block/block.c                 | 1 +
 hw/block/hd-geometry.c           | 1 +
 hw/net/rocker/rocker_fp.c        | 2 +-
 include/block/block.h            | 2 +-
 include/block/dirty-bitmap.h     | 2 +-
 include/chardev/char.h           | 1 +
 include/crypto/cipher.h          | 2 +-
 include/crypto/hash.h            | 2 +-
 include/crypto/hmac.h            | 2 +-
 include/crypto/secret.h          | 1 +
 include/crypto/tlscreds.h        | 1 +
 include/hw/block/block.h         | 2 +-
 include/hw/block/fdc.h           | 2 +-
 include/hw/ppc/spapr_drc.h       | 1 +
 include/hw/qdev-properties.h     | 1 +
 include/io/dns-resolver.h        | 1 +
 include/migration/colo.h         | 2 +-
 include/migration/failover.h     | 2 +-
 include/migration/global_state.h | 1 +
 include/monitor/monitor.h        | 1 +
 include/net/filter.h             | 1 +
 include/net/net.h                | 2 +-
 include/qapi/error.h             | 2 +-
 include/qapi/qmp/qobject.h       | 2 +-
 include/qapi/visitor.h           | 2 +-
 include/qemu/sockets.h           | 2 +-
 include/qemu/throttle.h          | 2 +-
 include/qom/cpu.h                | 1 +
 include/qom/object.h             | 2 +-
 include/sysemu/dump.h            | 2 ++
 include/sysemu/hostmem.h         | 1 +
 include/sysemu/replay.h          | 1 +
 include/sysemu/sysemu.h          | 1 +
 include/sysemu/tpm.h             | 1 +
 include/sysemu/watchdog.h        | 2 +-
 include/ui/input.h               | 2 +-
 migration/migration.h            | 1 +
 migration/ram.h                  | 2 +-
 net/tap_int.h                    | 2 +-
 replication.h                    | 1 +
 ui/vnc.h                         | 1 +
 42 files changed, 43 insertions(+), 22 deletions(-)
crypto/cipherpriv.h +1 -1
index 77da4c2f32..0823239f41 100644
@@ -15,7 +15,7 @@
 #ifndef QCRYPTO_CIPHERPRIV_H
 #define QCRYPTO_CIPHERPRIV_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 typedef struct QCryptoCipherDriver QCryptoCipherDriver;
 
hw/block/block.c +1 -0
index b0269c857f..b91e2b6d7e 100644
@@ -12,6 +12,7 @@
 #include "sysemu/block-backend.h"
 #include "hw/block/block.h"
 #include "qapi/error.h"
+#include "qapi/qapi-types-block.h"
 #include "qemu/error-report.h"
 
 void blkconf_serial(BlockConf *conf, char **serial)
hw/block/hd-geometry.c +1 -0
index 57ad5012a7..79384a2b0a 100644
@@ -32,6 +32,7 @@
 
 #include "qemu/osdep.h"
 #include "sysemu/block-backend.h"
+#include "qapi/qapi-types-block.h"
 #include "qemu/bswap.h"
 #include "hw/block/block.h"
 #include "trace.h"
hw/net/rocker/rocker_fp.c +1 -1
index 4b3c9847db..27b17c890f 100644
@@ -16,7 +16,7 @@
 
 #include "qemu/osdep.h"
 #include "net/clients.h"
-
+#include "qapi/qapi-types-rocker.h"
 #include "rocker.h"
 #include "rocker_hw.h"
 #include "rocker_fp.h"
include/block/block.h +1 -1
index ae1517f32d..70b90cd767 100644
@@ -2,7 +2,7 @@
 #define BLOCK_H
 
 #include "block/aio.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/iov.h"
 #include "qemu/coroutine.h"
 #include "block/accounting.h"
include/block/dirty-bitmap.h +1 -1
index 3da8486ab1..1454be358d 100644
@@ -2,7 +2,7 @@
 #define BLOCK_DIRTY_BITMAP_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/hbitmap.h"
 
 BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
include/chardev/char.h +1 -0
index a381dc3df8..ebf1e0ba04 100644
@@ -1,6 +1,7 @@
 #ifndef QEMU_CHAR_H
 #define QEMU_CHAR_H
 
+#include "qapi/qapi-types-char.h"
 #include "qemu/main-loop.h"
 #include "qemu/bitmap.h"
 #include "qom/object.h"
include/crypto/cipher.h +1 -1
index 984fb8243f..bce2d4c8e4 100644
@@ -21,7 +21,7 @@
 #ifndef QCRYPTO_CIPHER_H
 #define QCRYPTO_CIPHER_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 typedef struct QCryptoCipher QCryptoCipher;
 
include/crypto/hash.h +1 -1
index ca3267f3df..077ac7bea0 100644
@@ -21,7 +21,7 @@
 #ifndef QCRYPTO_HASH_H
 #define QCRYPTO_HASH_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 /* See also "QCryptoHashAlgorithm" defined in qapi/crypto.json */
 
include/crypto/hmac.h +1 -1
index 5e88905989..aa3c97a2ff 100644
@@ -12,7 +12,7 @@
 #ifndef QCRYPTO_HMAC_H
 #define QCRYPTO_HMAC_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 typedef struct QCryptoHmac QCryptoHmac;
 struct QCryptoHmac {
include/crypto/secret.h +1 -0
index 07a963e794..edd0e13236 100644
@@ -21,6 +21,7 @@
 #ifndef QCRYPTO_SECRET_H
 #define QCRYPTO_SECRET_H
 
+#include "qapi/qapi-types-crypto.h"
 #include "qom/object.h"
 
 #define TYPE_QCRYPTO_SECRET "secret"
include/crypto/tlscreds.h +1 -0
index ad47d88be7..6b011e1dbc 100644
@@ -21,6 +21,7 @@
 #ifndef QCRYPTO_TLSCREDS_H
 #define QCRYPTO_TLSCREDS_H
 
+#include "qapi/qapi-types-crypto.h"
 #include "qom/object.h"
 
 #ifdef CONFIG_GNUTLS
include/hw/block/block.h +1 -1
index f532d10e35..d4f4dfffab 100644
@@ -12,7 +12,7 @@
 #define HW_BLOCK_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 
 /* Configuration */
 
include/hw/block/fdc.h +1 -1
index 68a0c904ea..3b813c7f7d 100644
@@ -2,7 +2,7 @@
 #define HW_FDC_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block.h"
 
 /* fdc.c */
 #define MAX_FD 2
include/hw/ppc/spapr_drc.h +1 -0
index f8d9f5b231..f6ff32e7e2 100644
@@ -14,6 +14,7 @@
 #define HW_SPAPR_DRC_H
 
 #include <libfdt.h>
+#include "qapi/qapi-types-run-state.h"
 #include "qom/object.h"
 #include "sysemu/sysemu.h"
 #include "hw/qdev.h"
include/hw/qdev-properties.h +1 -0
index 5bbfec634b..c5d1b1bd63 100644
@@ -1,6 +1,7 @@
 #ifndef QEMU_QDEV_PROPERTIES_H
 #define QEMU_QDEV_PROPERTIES_H
 
+#include "qapi-types.h"
 #include "hw/qdev-core.h"
 
 /*** qdev-properties.c ***/
include/io/dns-resolver.h +1 -0
index 2f69c08c13..1a162185cc 100644
@@ -22,6 +22,7 @@
 #define QIO_DNS_RESOLVER_H
 
 #include "qemu-common.h"
+#include "qapi/qapi-types-sockets.h"
 #include "qom/object.h"
 #include "io/task.h"
 
include/migration/colo.h +1 -1
index 50ace16205..2fe48ad353 100644
@@ -14,7 +14,7 @@
 #define QEMU_COLO_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-migration.h"
 
 void colo_info_init(void);
 
include/migration/failover.h +1 -1
index ad91ef2381..4c37218dcc 100644
@@ -14,7 +14,7 @@
 #define QEMU_FAILOVER_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-migration.h"
 
 void failover_init_state(void);
 FailoverStatus failover_set_state(FailoverStatus old_state,
include/migration/global_state.h +1 -0
index d307de8350..fd22dd3034 100644
@@ -13,6 +13,7 @@
 #ifndef QEMU_MIGRATION_GLOBAL_STATE_H
 #define QEMU_MIGRATION_GLOBAL_STATE_H
 
+#include "qapi/qapi-types-run-state.h"
 #include "sysemu/sysemu.h"
 
 void register_global_state(void);
include/monitor/monitor.h +1 -0
index ad64ad8e68..50f7cea057 100644
@@ -3,6 +3,7 @@
 
 #include "qemu-common.h"
 #include "block/block.h"
+#include "qapi-types.h"
 #include "qemu/readline.h"
 
 extern Monitor *cur_mon;
include/net/filter.h +1 -0
index 0c4a2ea6c9..435acd6f82 100644
@@ -9,6 +9,7 @@
 #ifndef QEMU_NET_FILTER_H
 #define QEMU_NET_FILTER_H
 
+#include "qapi/qapi-types-net.h"
 #include "qom/object.h"
 #include "qemu-common.h"
 #include "net/queue.h"
include/net/net.h +1 -1
index 3fc48e4f51..727643032c 100644
@@ -2,7 +2,7 @@
 #define QEMU_NET_H
 
 #include "qemu/queue.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-net.h"
 #include "net/queue.h"
 #include "migration/vmstate.h"
 
include/qapi/error.h +1 -1
index 341b229066..36065da35d 100644
@@ -115,7 +115,7 @@
 #ifndef ERROR_H
 #define ERROR_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-common.h"
 
 /*
  * Overall category of an error.
include/qapi/qmp/qobject.h +1 -1
index 38ac68845c..a2964fbf25 100644
@@ -32,7 +32,7 @@
 #ifndef QOBJECT_H
 #define QOBJECT_H
 
-#include "qapi-types.h"
+#include "qapi-builtin-types.h"
 
 struct QObject {
     QType type;
include/qapi/visitor.h +1 -1
index ecff296c11..9e57508446 100644
@@ -15,7 +15,7 @@
 #ifndef QAPI_VISITOR_H
 #define QAPI_VISITOR_H
 
-#include "qapi-types.h"
+#include "qapi-builtin-types.h"
 
 /*
  * The QAPI schema defines both a set of C data types, and a QMP wire
include/qemu/sockets.h +1 -1
index 8889bcb1ec..e88d4c37ab 100644
@@ -9,7 +9,7 @@ int inet_aton(const char *cp, struct in_addr *ia);
 
 #endif /* !_WIN32 */
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-sockets.h"
 
 /* misc helpers */
 int qemu_socket(int domain, int type, int protocol);
include/qemu/throttle.h +1 -1
index 03d45f44f8..abeb886d93 100644
@@ -26,7 +26,7 @@
 #define THROTTLE_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/timer.h"
 
 #define THROTTLE_VALUE_MAX 1000000000000000LL
include/qom/cpu.h +1 -0
index aff88fa16f..dc6d4956a8 100644
@@ -24,6 +24,7 @@
 #include "disas/bfd.h"
 #include "exec/hwaddr.h"
 #include "exec/memattrs.h"
+#include "qapi/qapi-types-run-state.h"
 #include "qemu/bitmap.h"
 #include "qemu/queue.h"
 #include "qemu/thread.h"
include/qom/object.h +1 -1
index dc73d59660..5b5c016d8f 100644
@@ -14,7 +14,7 @@
 #ifndef QEMU_OBJECT_H
 #define QEMU_OBJECT_H
 
-#include "qapi-types.h"
+#include "qapi-builtin-types.h"
 #include "qemu/queue.h"
 
 struct TypeImpl;
include/sysemu/dump.h +2 -0
index c14bcfe8c6..2424e31425 100644
@@ -14,6 +14,8 @@
 #ifndef DUMP_H
 #define DUMP_H
 
+#include "qapi-types.h"
+
 #define MAKEDUMPFILE_SIGNATURE      "makedumpfile"
 #define MAX_SIZE_MDF_HEADER         (4096) /* max size of makedumpfile_header */
 #define TYPE_FLAT_HEADER            (1)    /* type of flattened format */
include/sysemu/hostmem.h +1 -0
index 621a3f9d42..6424f96df9 100644
@@ -14,6 +14,7 @@
 #define SYSEMU_HOSTMEM_H
 
 #include "sysemu/sysemu.h" /* for MAX_NODES */
+#include "qapi-types.h"
 #include "qom/object.h"
 #include "exec/memory.h"
 #include "qemu/bitmap.h"
include/sysemu/replay.h +1 -0
index dc8ae7b6b1..fb533ed9b6 100644
@@ -13,6 +13,7 @@
  */
 
 #include "sysemu.h"
+#include "qapi-types.h"
 
 /* replay clock kinds */
 enum ReplayClockKind {
include/sysemu/sysemu.h +1 -0
index 77bb3da582..bfbef9e69c 100644
@@ -2,6 +2,7 @@
 #define SYSEMU_H
 /* Misc. things related to the system emulator.  */
 
+#include "qapi/qapi-types-run-state.h"
 #include "qemu/queue.h"
 #include "qemu/timer.h"
 #include "qemu/notify.h"
include/sysemu/tpm.h +1 -0
index 00be220248..20ac1b2993 100644
@@ -12,6 +12,7 @@
 #ifndef QEMU_TPM_H
 #define QEMU_TPM_H
 
+#include "qapi/qapi-types-tpm.h"
 #include "qom/object.h"
 
 int tpm_config_parse(QemuOptsList *opts_list, const char *optarg);
include/sysemu/watchdog.h +1 -1
index 677ace3945..a08d16380d 100644
@@ -23,7 +23,7 @@
 #define QEMU_WATCHDOG_H
 
 #include "qemu/queue.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-run-state.h"
 
 struct WatchdogTimerModel {
     QLIST_ENTRY(WatchdogTimerModel) entry;
include/ui/input.h +1 -1
index 05aab2db5c..fc70f503fb 100644
@@ -1,7 +1,7 @@
 #ifndef INPUT_H
 #define INPUT_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-ui.h"
 
 #define INPUT_EVENT_MASK_KEY   (1<<INPUT_EVENT_KIND_KEY)
 #define INPUT_EVENT_MASK_BTN   (1<<INPUT_EVENT_KIND_BTN)
migration/migration.h +1 -0
index 7e3dbed057..87503f03bc 100644
@@ -15,6 +15,7 @@
 #define QEMU_MIGRATION_H
 
 #include "qemu-common.h"
+#include "qapi/qapi-types-migration.h"
 #include "qemu/thread.h"
 #include "exec/cpu-common.h"
 #include "qemu/coroutine_int.h"
migration/ram.h +1 -1
index f3a227b4fc..53f0021c51 100644
@@ -30,7 +30,7 @@
 #define QEMU_MIGRATION_RAM_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-migration.h"
 #include "exec/cpu-common.h"
 
 extern MigrationStats ram_counters;
net/tap_int.h +1 -1
index ae6888f74a..9f931d52d6 100644
@@ -27,7 +27,7 @@
 #define NET_TAP_INT_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-net.h"
 
 int tap_open(char *ifname, int ifname_size, int *vnet_hdr,
              int vnet_hdr_required, int mq_required, Error **errp);
replication.h +1 -0
index ece6ca6133..8faefe005f 100644
@@ -15,6 +15,7 @@
 #ifndef REPLICATION_H
 #define REPLICATION_H
 
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/queue.h"
 
 typedef struct ReplicationOps ReplicationOps;
ui/vnc.h +1 -0
index cfc3d0a570..da8495c0c8 100644
@@ -28,6 +28,7 @@
 #define QEMU_VNC_H
 
 #include "qemu-common.h"
+#include "qapi/qapi-types-ui.h"
 #include "qemu/queue.h"
 #include "qemu/thread.h"
 #include "ui/console.h"
Signature · 1 line
2.13.6
Reviewed-by Marc-André Lureau on v1 20/21
#75 lore raw

Re: [Qemu-devel] [PATCH RFC 20/21] Include less of qapi-types.h

Quoted 8 lines · Markus Armbruster

In my "build everything" tree, a change to the types in qapi-schema.json triggers a recompile of about 4500 out of 4800 objects.

The previous commit split up the generated qapi-types.h. Replace includes of qapi-types.h (i.e. all types) by includes of parts where possible.

Quoted 5 lines

To illustrate the benefits: adding a type to qapi/migration.json now recompiles some 2300 instead of 4500 objects. The next commit will improve it further.

for the record, what was your methodology?

Quoted 575 lines
---
 crypto/cipherpriv.h              | 2 +-
 hw/block/block.c                 | 1 +
 hw/block/hd-geometry.c           | 1 +
 hw/net/rocker/rocker_fp.c        | 2 +-
 include/block/block.h            | 2 +-
 include/block/dirty-bitmap.h     | 2 +-
 include/chardev/char.h           | 1 +
 include/crypto/cipher.h          | 2 +-
 include/crypto/hash.h            | 2 +-
 include/crypto/hmac.h            | 2 +-
 include/crypto/secret.h          | 1 +
 include/crypto/tlscreds.h        | 1 +
 include/hw/block/block.h         | 2 +-
 include/hw/block/fdc.h           | 2 +-
 include/hw/ppc/spapr_drc.h       | 1 +
 include/hw/qdev-properties.h     | 1 +
 include/io/dns-resolver.h        | 1 +
 include/migration/colo.h         | 2 +-
 include/migration/failover.h     | 2 +-
 include/migration/global_state.h | 1 +
 include/monitor/monitor.h        | 1 +
 include/net/filter.h             | 1 +
 include/net/net.h                | 2 +-
 include/qapi/error.h             | 2 +-
 include/qapi/qmp/qobject.h       | 2 +-
 include/qapi/visitor.h           | 2 +-
 include/qemu/sockets.h           | 2 +-
 include/qemu/throttle.h          | 2 +-
 include/qom/cpu.h                | 1 +
 include/qom/object.h             | 2 +-
 include/sysemu/dump.h            | 2 ++
 include/sysemu/hostmem.h         | 1 +
 include/sysemu/replay.h          | 1 +
 include/sysemu/sysemu.h          | 1 +
 include/sysemu/tpm.h             | 1 +
 include/sysemu/watchdog.h        | 2 +-
 include/ui/input.h               | 2 +-
 migration/migration.h            | 1 +
 migration/ram.h                  | 2 +-
 net/tap_int.h                    | 2 +-
 replication.h                    | 1 +
 ui/vnc.h                         | 1 +
 42 files changed, 43 insertions(+), 22 deletions(-)
crypto/cipherpriv.h +1 -1
index 77da4c2f32..0823239f41 100644
@@ -15,7 +15,7 @@
 #ifndef QCRYPTO_CIPHERPRIV_H
 #define QCRYPTO_CIPHERPRIV_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 typedef struct QCryptoCipherDriver QCryptoCipherDriver;
 
hw/block/block.c +1 -0
index b0269c857f..b91e2b6d7e 100644
@@ -12,6 +12,7 @@
 #include "sysemu/block-backend.h"
 #include "hw/block/block.h"
 #include "qapi/error.h"
+#include "qapi/qapi-types-block.h"
 #include "qemu/error-report.h"
 
 void blkconf_serial(BlockConf *conf, char **serial)
hw/block/hd-geometry.c +1 -0
index 57ad5012a7..79384a2b0a 100644
@@ -32,6 +32,7 @@
 
 #include "qemu/osdep.h"
 #include "sysemu/block-backend.h"
+#include "qapi/qapi-types-block.h"
 #include "qemu/bswap.h"
 #include "hw/block/block.h"
 #include "trace.h"
hw/net/rocker/rocker_fp.c +1 -1
index 4b3c9847db..27b17c890f 100644
@@ -16,7 +16,7 @@
 
 #include "qemu/osdep.h"
 #include "net/clients.h"
-
+#include "qapi/qapi-types-rocker.h"
 #include "rocker.h"
 #include "rocker_hw.h"
 #include "rocker_fp.h"
include/block/block.h +1 -1
index ae1517f32d..70b90cd767 100644
@@ -2,7 +2,7 @@
 #define BLOCK_H
 
 #include "block/aio.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/iov.h"
 #include "qemu/coroutine.h"
 #include "block/accounting.h"
include/block/dirty-bitmap.h +1 -1
index 3da8486ab1..1454be358d 100644
@@ -2,7 +2,7 @@
 #define BLOCK_DIRTY_BITMAP_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/hbitmap.h"
 
 BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
include/chardev/char.h +1 -0
index a381dc3df8..ebf1e0ba04 100644
@@ -1,6 +1,7 @@
 #ifndef QEMU_CHAR_H
 #define QEMU_CHAR_H
 
+#include "qapi/qapi-types-char.h"
 #include "qemu/main-loop.h"
 #include "qemu/bitmap.h"
 #include "qom/object.h"
include/crypto/cipher.h +1 -1
index 984fb8243f..bce2d4c8e4 100644
@@ -21,7 +21,7 @@
 #ifndef QCRYPTO_CIPHER_H
 #define QCRYPTO_CIPHER_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 typedef struct QCryptoCipher QCryptoCipher;
 
include/crypto/hash.h +1 -1
index ca3267f3df..077ac7bea0 100644
@@ -21,7 +21,7 @@
 #ifndef QCRYPTO_HASH_H
 #define QCRYPTO_HASH_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 /* See also "QCryptoHashAlgorithm" defined in qapi/crypto.json */
 
include/crypto/hmac.h +1 -1
index 5e88905989..aa3c97a2ff 100644
@@ -12,7 +12,7 @@
 #ifndef QCRYPTO_HMAC_H
 #define QCRYPTO_HMAC_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 typedef struct QCryptoHmac QCryptoHmac;
 struct QCryptoHmac {
include/crypto/secret.h +1 -0
index 07a963e794..edd0e13236 100644
@@ -21,6 +21,7 @@
 #ifndef QCRYPTO_SECRET_H
 #define QCRYPTO_SECRET_H
 
+#include "qapi/qapi-types-crypto.h"
 #include "qom/object.h"
 
 #define TYPE_QCRYPTO_SECRET "secret"
include/crypto/tlscreds.h +1 -0
index ad47d88be7..6b011e1dbc 100644
@@ -21,6 +21,7 @@
 #ifndef QCRYPTO_TLSCREDS_H
 #define QCRYPTO_TLSCREDS_H
 
+#include "qapi/qapi-types-crypto.h"
 #include "qom/object.h"
 
 #ifdef CONFIG_GNUTLS
include/hw/block/block.h +1 -1
index f532d10e35..d4f4dfffab 100644
@@ -12,7 +12,7 @@
 #define HW_BLOCK_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 
 /* Configuration */
 
include/hw/block/fdc.h +1 -1
index 68a0c904ea..3b813c7f7d 100644
@@ -2,7 +2,7 @@
 #define HW_FDC_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block.h"
 
 /* fdc.c */
 #define MAX_FD 2
include/hw/ppc/spapr_drc.h +1 -0
index f8d9f5b231..f6ff32e7e2 100644
@@ -14,6 +14,7 @@
 #define HW_SPAPR_DRC_H
 
 #include <libfdt.h>
+#include "qapi/qapi-types-run-state.h"
 #include "qom/object.h"
 #include "sysemu/sysemu.h"
 #include "hw/qdev.h"
include/hw/qdev-properties.h +1 -0
index 5bbfec634b..c5d1b1bd63 100644
@@ -1,6 +1,7 @@
 #ifndef QEMU_QDEV_PROPERTIES_H
 #define QEMU_QDEV_PROPERTIES_H
 
+#include "qapi-types.h"
 #include "hw/qdev-core.h"
 
 /*** qdev-properties.c ***/
include/io/dns-resolver.h +1 -0
index 2f69c08c13..1a162185cc 100644
@@ -22,6 +22,7 @@
 #define QIO_DNS_RESOLVER_H
 
 #include "qemu-common.h"
+#include "qapi/qapi-types-sockets.h"
 #include "qom/object.h"
 #include "io/task.h"
 
include/migration/colo.h +1 -1
index 50ace16205..2fe48ad353 100644
@@ -14,7 +14,7 @@
 #define QEMU_COLO_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-migration.h"
 
 void colo_info_init(void);
 
include/migration/failover.h +1 -1
index ad91ef2381..4c37218dcc 100644
@@ -14,7 +14,7 @@
 #define QEMU_FAILOVER_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-migration.h"
 
 void failover_init_state(void);
 FailoverStatus failover_set_state(FailoverStatus old_state,
include/migration/global_state.h +1 -0
index d307de8350..fd22dd3034 100644
@@ -13,6 +13,7 @@
 #ifndef QEMU_MIGRATION_GLOBAL_STATE_H
 #define QEMU_MIGRATION_GLOBAL_STATE_H
 
+#include "qapi/qapi-types-run-state.h"
 #include "sysemu/sysemu.h"
 
 void register_global_state(void);
include/monitor/monitor.h +1 -0
index ad64ad8e68..50f7cea057 100644
@@ -3,6 +3,7 @@
 
 #include "qemu-common.h"
 #include "block/block.h"
+#include "qapi-types.h"
 #include "qemu/readline.h"
 
 extern Monitor *cur_mon;
include/net/filter.h +1 -0
index 0c4a2ea6c9..435acd6f82 100644
@@ -9,6 +9,7 @@
 #ifndef QEMU_NET_FILTER_H
 #define QEMU_NET_FILTER_H
 
+#include "qapi/qapi-types-net.h"
 #include "qom/object.h"
 #include "qemu-common.h"
 #include "net/queue.h"
include/net/net.h +1 -1
index 3fc48e4f51..727643032c 100644
@@ -2,7 +2,7 @@
 #define QEMU_NET_H
 
 #include "qemu/queue.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-net.h"
 #include "net/queue.h"
 #include "migration/vmstate.h"
 
include/qapi/error.h +1 -1
index 341b229066..36065da35d 100644
@@ -115,7 +115,7 @@
 #ifndef ERROR_H
 #define ERROR_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-common.h"
 
 /*
  * Overall category of an error.
include/qapi/qmp/qobject.h +1 -1
index 38ac68845c..a2964fbf25 100644
@@ -32,7 +32,7 @@
 #ifndef QOBJECT_H
 #define QOBJECT_H
 
-#include "qapi-types.h"
+#include "qapi-builtin-types.h"
 
 struct QObject {
     QType type;
include/qapi/visitor.h +1 -1
index ecff296c11..9e57508446 100644
@@ -15,7 +15,7 @@
 #ifndef QAPI_VISITOR_H
 #define QAPI_VISITOR_H
 
-#include "qapi-types.h"
+#include "qapi-builtin-types.h"
 
 /*
  * The QAPI schema defines both a set of C data types, and a QMP wire
include/qemu/sockets.h +1 -1
index 8889bcb1ec..e88d4c37ab 100644
@@ -9,7 +9,7 @@ int inet_aton(const char *cp, struct in_addr *ia);
 
 #endif /* !_WIN32 */
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-sockets.h"
 
 /* misc helpers */
 int qemu_socket(int domain, int type, int protocol);
include/qemu/throttle.h +1 -1
index 03d45f44f8..abeb886d93 100644
@@ -26,7 +26,7 @@
 #define THROTTLE_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/timer.h"
 
 #define THROTTLE_VALUE_MAX 1000000000000000LL
include/qom/cpu.h +1 -0
index aff88fa16f..dc6d4956a8 100644
@@ -24,6 +24,7 @@
 #include "disas/bfd.h"
 #include "exec/hwaddr.h"
 #include "exec/memattrs.h"
+#include "qapi/qapi-types-run-state.h"
 #include "qemu/bitmap.h"
 #include "qemu/queue.h"
 #include "qemu/thread.h"
include/qom/object.h +1 -1
index dc73d59660..5b5c016d8f 100644
@@ -14,7 +14,7 @@
 #ifndef QEMU_OBJECT_H
 #define QEMU_OBJECT_H
 
-#include "qapi-types.h"
+#include "qapi-builtin-types.h"
 #include "qemu/queue.h"
 
 struct TypeImpl;
include/sysemu/dump.h +2 -0
index c14bcfe8c6..2424e31425 100644
@@ -14,6 +14,8 @@
 #ifndef DUMP_H
 #define DUMP_H
 
+#include "qapi-types.h"
+
 #define MAKEDUMPFILE_SIGNATURE      "makedumpfile"
 #define MAX_SIZE_MDF_HEADER         (4096) /* max size of makedumpfile_header */
 #define TYPE_FLAT_HEADER            (1)    /* type of flattened format */
include/sysemu/hostmem.h +1 -0
index 621a3f9d42..6424f96df9 100644
@@ -14,6 +14,7 @@
 #define SYSEMU_HOSTMEM_H
 
 #include "sysemu/sysemu.h" /* for MAX_NODES */
+#include "qapi-types.h"
 #include "qom/object.h"
 #include "exec/memory.h"
 #include "qemu/bitmap.h"
include/sysemu/replay.h +1 -0
index dc8ae7b6b1..fb533ed9b6 100644
@@ -13,6 +13,7 @@
  */
 
 #include "sysemu.h"
+#include "qapi-types.h"
 
 /* replay clock kinds */
 enum ReplayClockKind {
include/sysemu/sysemu.h +1 -0
index 77bb3da582..bfbef9e69c 100644
@@ -2,6 +2,7 @@
 #define SYSEMU_H
 /* Misc. things related to the system emulator.  */
 
+#include "qapi/qapi-types-run-state.h"
 #include "qemu/queue.h"
 #include "qemu/timer.h"
 #include "qemu/notify.h"
include/sysemu/tpm.h +1 -0
index 00be220248..20ac1b2993 100644
@@ -12,6 +12,7 @@
 #ifndef QEMU_TPM_H
 #define QEMU_TPM_H
 
+#include "qapi/qapi-types-tpm.h"
 #include "qom/object.h"
 
 int tpm_config_parse(QemuOptsList *opts_list, const char *optarg);
include/sysemu/watchdog.h +1 -1
index 677ace3945..a08d16380d 100644
@@ -23,7 +23,7 @@
 #define QEMU_WATCHDOG_H
 
 #include "qemu/queue.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-run-state.h"
 
 struct WatchdogTimerModel {
     QLIST_ENTRY(WatchdogTimerModel) entry;
include/ui/input.h +1 -1
index 05aab2db5c..fc70f503fb 100644
@@ -1,7 +1,7 @@
 #ifndef INPUT_H
 #define INPUT_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-ui.h"
 
 #define INPUT_EVENT_MASK_KEY   (1<<INPUT_EVENT_KIND_KEY)
 #define INPUT_EVENT_MASK_BTN   (1<<INPUT_EVENT_KIND_BTN)
migration/migration.h +1 -0
index 7e3dbed057..87503f03bc 100644
@@ -15,6 +15,7 @@
 #define QEMU_MIGRATION_H
 
 #include "qemu-common.h"
+#include "qapi/qapi-types-migration.h"
 #include "qemu/thread.h"
 #include "exec/cpu-common.h"
 #include "qemu/coroutine_int.h"
migration/ram.h +1 -1
index f3a227b4fc..53f0021c51 100644
@@ -30,7 +30,7 @@
 #define QEMU_MIGRATION_RAM_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-migration.h"
 #include "exec/cpu-common.h"
 
 extern MigrationStats ram_counters;
net/tap_int.h +1 -1
index ae6888f74a..9f931d52d6 100644
@@ -27,7 +27,7 @@
 #define NET_TAP_INT_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-net.h"
 
 int tap_open(char *ifname, int ifname_size, int *vnet_hdr,
              int vnet_hdr_required, int mq_required, Error **errp);
replication.h +1 -0
index ece6ca6133..8faefe005f 100644
@@ -15,6 +15,7 @@
 #ifndef REPLICATION_H
 #define REPLICATION_H
 
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/queue.h"
 
 typedef struct ReplicationOps ReplicationOps;
ui/vnc.h +1 -1
index cfc3d0a570..da8495c0c8 100644
@@ -28,6 +28,7 @@
 #define QEMU_VNC_H
 
 #include "qemu-common.h"
+#include "qapi/qapi-types-ui.h"
 #include "qemu/queue.h"
 #include "qemu/thread.h"
 #include "ui/console.h"
--

2.13.6

#76 lore raw

Re: [Qemu-devel] [PATCH RFC 20/21] Include less of qapi-types.h

Quoted 9 lines · Marc-Andre Lureau
Quoted 8 lines · Markus Armbruster

In my "build everything" tree, a change to the types in qapi-schema.json triggers a recompile of about 4500 out of 4800 objects.

The previous commit split up the generated qapi-types.h. Replace includes of qapi-types.h (i.e. all types) by includes of parts where possible.

Quoted 7 lines
Quoted 5 lines

To illustrate the benefits: adding a type to qapi/migration.json now recompiles some 2300 instead of 4500 objects. The next commit will improve it further.

for the record, what was your methodology?

I counted compiles like this:

    $ make 2>&1 | awk '/^  [A-Z]/ { h[$1]++ } END { n = asorti(h, x); for (i = 1; i <=n; i++) print x[i], h[x[i]] }'
Quoted 1 line

Thanks!

#77 lore raw

RFC PATCH 21/21 qapi: Empty out qapi-schema.json

In reply to Markus Armbruster

Types defined directly in qapi-schema.json end up in qapi-types.h, and including that pulls in everything else.

Move everything but include directives from qapi-schema.json to new sub-module qapi/misc.json, and replace the remaining includes of qapi-types.h.

Adding a type to qapi/migration.json now recompiles less than 200 instead of 2300 out of 4500 objects.

---
 Makefile                     |    2 +
 Makefile.objs                |    1 +
 include/hw/qdev-properties.h |    3 +-
 include/monitor/monitor.h    |    2 +-
 include/sysemu/arch_init.h   |    2 +-
 include/sysemu/balloon.h     |    2 +-
 include/sysemu/dump.h        |    2 +-
 include/sysemu/hostmem.h     |    2 +-
 include/sysemu/replay.h      |    3 +-
 qapi-schema.json             | 3098 +-----------------------------------------
 qapi/misc.json               | 3090 +++++++++++++++++++++++++++++++++++++++++
 qapi/run-state.json          |   10 +
 12 files changed, 3113 insertions(+), 3104 deletions(-)
 create mode 100644 qapi/misc.json
Makefile +2 -0
index f1b68dca9b..0cf2a9caba 100644
@@ -98,6 +98,7 @@ GENERATED_FILES += qapi/qapi-types-common.h qapi/qapi-types-common.c
 GENERATED_FILES += qapi/qapi-types-crypto.h qapi/qapi-types-crypto.c
 GENERATED_FILES += qapi/qapi-types-introspect.h qapi/qapi-types-introspect.c
 GENERATED_FILES += qapi/qapi-types-migration.h qapi/qapi-types-migration.c
+GENERATED_FILES += qapi/qapi-types-misc.h qapi/qapi-types-misc.c
 GENERATED_FILES += qapi/qapi-types-net.h qapi/qapi-types-net.c
 GENERATED_FILES += qapi/qapi-types-rocker.h qapi/qapi-types-rocker.c
 GENERATED_FILES += qapi/qapi-types-run-state.h qapi/qapi-types-run-state.c
@@ -541,6 +542,7 @@ qapi/qapi-types-common.c qapi/qapi-types-common.h \
 qapi/qapi-types-crypto.c qapi/qapi-types-crypto.h \
 qapi/qapi-types-introspect.c qapi/qapi-types-introspect.h \
 qapi/qapi-types-migration.c qapi/qapi-types-migration.h \
+qapi/qapi-types-misc.c qapi/qapi-types-misc.h \
 qapi/qapi-types-net.c qapi/qapi-types-net.h \
 qapi/qapi-types-rocker.c qapi/qapi-types-rocker.h \
 qapi/qapi-types-run-state.c qapi/qapi-types-run-state.h \
Makefile.objs +1 -0
index e7411a2658..efef7f095b 100644
@@ -11,6 +11,7 @@ util-obj-y += qapi/qapi-types-common.o
 util-obj-y += qapi/qapi-types-crypto.o
 util-obj-y += qapi/qapi-types-introspect.o
 util-obj-y += qapi/qapi-types-migration.o
+util-obj-y += qapi/qapi-types-misc.o
 util-obj-y += qapi/qapi-types-net.o
 util-obj-y += qapi/qapi-types-rocker.o
 util-obj-y += qapi/qapi-types-run-state.o
include/hw/qdev-properties.h +2 -1
index c5d1b1bd63..2f6bd7d73d 100644
@@ -1,7 +1,8 @@
 #ifndef QEMU_QDEV_PROPERTIES_H
 #define QEMU_QDEV_PROPERTIES_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-block.h"
+#include "qapi/qapi-types-misc.h"
 #include "hw/qdev-core.h"
 
 /*** qdev-properties.c ***/
include/monitor/monitor.h +1 -1
index 50f7cea057..d1024d4bdc 100644
@@ -3,7 +3,7 @@
 
 #include "qemu-common.h"
 #include "block/block.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-misc.h"
 #include "qemu/readline.h"
 
 extern Monitor *cur_mon;
include/sysemu/arch_init.h +1 -1
index ced67c7842..979fd022d0 100644
@@ -1,7 +1,7 @@
 #ifndef QEMU_ARCH_INIT_H
 #define QEMU_ARCH_INIT_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-misc.h"
 
 enum {
     QEMU_ARCH_ALL = -1,
include/sysemu/balloon.h +1 -1
index af49e19c78..66543ae8f4 100644
@@ -14,7 +14,7 @@
 #ifndef QEMU_BALLOON_H
 #define QEMU_BALLOON_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-misc.h"
 
 typedef void (QEMUBalloonEvent)(void *opaque, ram_addr_t target);
 typedef void (QEMUBalloonStatus)(void *opaque, BalloonInfo *info);
include/sysemu/dump.h +1 -1
index 2424e31425..d824bc0941 100644
@@ -14,7 +14,7 @@
 #ifndef DUMP_H
 #define DUMP_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-misc.h"
 
 #define MAKEDUMPFILE_SIGNATURE      "makedumpfile"
 #define MAX_SIZE_MDF_HEADER         (4096) /* max size of makedumpfile_header */
include/sysemu/hostmem.h +1 -1
index 6424f96df9..dc3ac3d301 100644
@@ -14,7 +14,7 @@
 #define SYSEMU_HOSTMEM_H
 
 #include "sysemu/sysemu.h" /* for MAX_NODES */
-#include "qapi-types.h"
+#include "qapi/qapi-types-misc.h"
 #include "qom/object.h"
 #include "exec/memory.h"
 #include "qemu/bitmap.h"
include/sysemu/replay.h +2 -1
index fb533ed9b6..c0204e641c 100644
@@ -13,7 +13,8 @@
  */
 
 #include "sysemu.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-misc.h"
+#include "qapi/qapi-types-ui.h"
 
 /* replay clock kinds */
 enum ReplayClockKind {
qapi-schema.json +1 -3097

2,921 lines not on this page · raw message

qapi/misc.json new file +3090 -0

3,090 lines not on this page · raw message

qapi/run-state.json +10 -0
index bca46a8785..a27c3c2a93 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.13.6

Part of the diff is not shown here. View the raw message.