Skip to contentSkip to conversation

Modularize generated QAPI code

PATCH In review Markus Armbruster 6 participants 110 messages Feb 11, 2018 – Apr 13, 2018 Page 1 of 2

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v2 cover Modularize generated QAPI code

Our qapi-schema.json is composed of modules connected by include directives, but the generated code is monolithic all the same: one qapi-types.h with all the types, one qapi-visit.h with all the visitors, and so forth. These monolithic headers get included all over the place. In my "build everything" tree, adding a QAPI type recompiles about 4800 out of 5100 objects.

We wouldn't write such monolithic headers by hand. It stands to reason that we shouldn't generate them, either.

This series' basic idea is to split up generated headers to mirror the schema's modular structure: one header per module. That way, you can include just what you need.

The compile-time improvements can be massive. Before this series, any QAPI schema change recompiles some 4800 out of 5100 objects in my "build everything" tree. Afterwards, adding a type to qapi/migration.json recompiles a bit over 100, and a documentation change no longer recompiles anything.

Related: Marc-André's 'unit' pragma proposal. That's a different way to split off parts of the generated code, motivated by the desire to use poisoned identifiers such as TARGET_I386. I noted in my review of v3 that I "can either accept it, or come up with a better solution." This is my attempt at a better solution. It's a bit more ambitious, and thus more useful (I hope). The pragma has one theoretical advantage, though: you can modularize the generated output in different ways than the input. The patches using don't do that, however.

v2:
* Rebased, annoying conflicts due to the Python 2 rigmarole
* PATCH 01: New
* PATCH 02-03,08,10-11,17,19-24: Commit messages improved [Eric, Marc-André]
* PATCH 04: Split off the next patch [Eric]
* PATCH 05,07-08,10,19: QAPIGen methods renamed [Marc-André]
* PATCH 08,20,22: fix missing .gitignore updates
* PATCH 08:
- generated doc renamed to qapi-doc.texi
- qapi-code-gen.txt updated
- useless options -c, -h dropped
* PATCH 09: New, replacing old PATCH 07
* PATCH 16: Latent bug in QAPISchema.check() fixed
* PATCH 18: New
* PATCH 19-20,22-24:
- Common code factored out into QAPISchemaMonolithicCVisitor and QAPISchemaModularCVisitor
- All code-generating visitors covered
- Consistent parameter order prefix, opt_builtins
- R-bys dropped
* PATCH 25-29: New

Markus Armbruster (29):
Include qapi/qmp/qerror.h exactly where needed
qapi: Streamline boilerplate comment generation
qapi: Generate up-to-date copyright notice
qapi: Rename variable holding the QAPISchemaGenFOOVisitor
qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc
qapi: Reduce use of global variables in generators some
qapi: Turn generators into modules
qapi-gen: New common driver for code and doc generators
qapi-gen: Convert from getopt to argparse
qapi: Touch generated files only when they change
qapi: Improve include file name reporting in error messages qapi/common: Eliminate QAPISchema.exprs
qapi: Lift error reporting from QAPISchema.__init__() to callers
qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()
qapi: Record 'include' directives in parse tree
qapi: Generate in source order
qapi: Record 'include' directives in intermediate representation
qapi: Rename generated qmp-marshal.c to qmp-commands.c
qapi: Make code-generating visitors use QAPIGen more qapi/types qapi/visit: Generate built-in stuff into separate files qapi/common: Fix guardname() for funny filenames
qapi: Generate separate .h, .c for each module
Include less of the generated modular QAPI headers
qapi: Empty out qapi-schema.json docs/devel/writing-qmp-commands: Update for modular QAPI
docs: Correct outdated information on QAPI
qapi: Move qapi-schema.json to qapi/, rename generated files Fix up dangling references to qmp-commands.* in comment and doc
qapi: Don't create useless directory qapi-generated

 .gitignore                                         |  80 ++++-
 Makefile                                           | 233 ++++++++++----
 Makefile.objs                                      |  80 ++++-
 arch_init.c                                        |   2 +-
 backends/cryptodev.c                               |   1 -
 backends/hostmem.c                                 |   3 +-
 balloon.c                                          |   2 +-
 block.c                                            |   2 -
 block/block-backend.c                              |   2 +-
 block/crypto.c                                     |   2 +-
 block/iscsi.c                                      |   2 +-
 block/nbd.c                                        |   2 +-
 block/nfs.c                                        |   2 +-
 block/qapi.c                                       |   4 +-
 block/qcow2.c                                      |   4 +-
 block/quorum.c                                     |   2 +-
 block/sheepdog.c                                   |   2 +-
 block/ssh.c                                        |   2 +-
 block/throttle-groups.c                            |   2 +-
 block/write-threshold.c                            |   4 +-
 blockdev-nbd.c                                     |   2 +-
 blockdev.c                                         |   5 +-
 blockjob.c                                         |   2 +-
 chardev/char-fe.c                                  |   2 +-
 chardev/char-ringbuf.c                             |   2 +-
 chardev/char-socket.c                              |   1 +
 chardev/char.c                                     |   4 +-
 configure                                          |   1 -
 cpus.c                                             |   4 +-
 crypto/cipherpriv.h                                |   2 +-
 docs/devel/qapi-code-gen.txt                       | 119 ++++---
 docs/devel/writing-qmp-commands.txt                |  39 +--
 docs/interop/qmp-intro.txt                         |   3 +-
 docs/xen-save-devices-state.txt                    |   3 +-
 dump.c                                             |   4 +-
 hmp.c                                              |  12 +-
 hw/acpi/core.c                                     |   4 +-
 hw/acpi/cpu.c                                      |   2 +-
 hw/acpi/memory_hotplug.c                           |   2 +-
 hw/acpi/vmgenid.c                                  |   2 +-
 hw/block/block.c                                   |   1 +
 hw/block/hd-geometry.c                             |   1 +
 hw/char/virtio-console.c                           |   2 +-
 hw/core/machine.c                                  |   2 +-
 hw/core/qdev.c                                     |   2 +-
 hw/i386/pc.c                                       |   2 +-
 hw/i386/xen/xen-hvm.c                              |   2 +-
 hw/ipmi/ipmi.c                                     |   2 +-
 hw/mem/nvdimm.c                                    |   1 -
 hw/net/rocker/qmp-norocker.c                       |   2 +-
 hw/net/rocker/rocker.c                             |   2 +-
 hw/net/rocker/rocker_fp.c                          |   2 +-
 hw/net/rocker/rocker_of_dpa.c                      |   2 +-
 hw/net/virtio-net.c                                |   2 +-
 hw/pci/pci-stub.c                                  |   2 +-
 hw/pci/pci.c                                       |   2 +-
 hw/ppc/spapr_rtas.c                                |   1 -
 hw/ppc/spapr_rtc.c                                 |   2 +-
 hw/s390x/s390-skeys.c                              |   2 +-
 hw/timer/mc146818rtc.c                             |   4 +-
 hw/tpm/tpm_emulator.c                              |   1 +
 hw/tpm/tpm_passthrough.c                           |   1 +
 hw/virtio/virtio-balloon.c                         |   2 +-
 hw/watchdog/watchdog.c                             |   4 +-
 include/block/block.h                              |   2 +-
 include/block/dirty-bitmap.h                       |   2 +-
 include/block/nbd.h                                |   2 +-
 include/chardev/char.h                             |   1 +
 include/crypto/cipher.h                            |   2 +-
 include/crypto/hash.h                              |   2 +-
 include/crypto/hmac.h                              |   2 +-
 include/crypto/secret.h                            |   1 +
 include/crypto/tlscreds.h                          |   1 +
 include/hw/block/block.h                           |   2 +-
 include/hw/block/fdc.h                             |   2 +-
 include/hw/ppc/spapr_drc.h                         |   1 +
 include/hw/qdev-properties.h                       |   2 +
 include/io/dns-resolver.h                          |   1 +
 include/migration/colo.h                           |   2 +-
 include/migration/failover.h                       |   2 +-
 include/migration/global_state.h                   |   1 +
 include/monitor/monitor.h                          |   1 +
 include/net/filter.h                               |   1 +
 include/net/net.h                                  |   2 +-
 include/qapi/clone-visitor.h                       |   1 -
 include/qapi/error.h                               |   2 +-
 include/qapi/qmp/qobject.h                         |   2 +-
 include/qapi/visitor.h                             |   2 +-
 include/qemu/sockets.h                             |   2 +-
 include/qemu/throttle.h                            |   2 +-
 include/qom/cpu.h                                  |   1 +
 include/qom/object.h                               |   2 +-
 include/sysemu/arch_init.h                         |   2 +-
 include/sysemu/balloon.h                           |   2 +-
 include/sysemu/dump.h                              |   2 +
 include/sysemu/hostmem.h                           |   1 +
 include/sysemu/replay.h                            |   2 +
 include/sysemu/sysemu.h                            |   1 +
 include/sysemu/tpm.h                               |   1 +
 include/sysemu/watchdog.h                          |   2 +-
 include/ui/input.h                                 |   2 +-
 io/channel-socket.c                                |   1 +
 io/dns-resolver.c                                  |   1 +
 iothread.c                                         |   2 +-
 migration/colo-failover.c                          |   2 +-
 migration/colo.c                                   |   2 +-
 migration/migration.c                              |   4 +-
 migration/migration.h                              |   1 +
 migration/ram.c                                    |   2 +-
 migration/ram.h                                    |   2 +-
 migration/savevm.c                                 |   3 +-
 monitor.c                                          |   8 +-
 net/colo-compare.c                                 |   1 -
 net/filter-buffer.c                                |   2 +-
 net/filter-mirror.c                                |   1 -
 net/filter-rewriter.c                              |   1 -
 net/net.c                                          |   4 +-
 net/tap_int.h                                      |   2 +-
 net/vhost-user.c                                   |   2 +-
 numa.c                                             |   4 +-
 qapi-schema.json => qapi/misc.json                 | 105 +-----
 qapi/qapi-schema.json                              |  95 ++++++
 qapi/run-state.json                                |  10 +
 qdev-monitor.c                                     |   2 +-
 qemu-img.c                                         |   2 +-
 qga/Makefile.objs                                  |   2 +-
 qga/commands-posix.c                               |   2 +-
 qga/commands-win32.c                               |   2 +-
 qga/commands.c                                     |   2 +-
 qga/main.c                                         |   2 +-
 qmp.c                                              |  18 +-
 qom/object.c                                       |   2 +-
 qom/object_interfaces.c                            |   2 +-
 replay/replay-input.c                              |   1 +
 replication.h                                      |   1 +
 scripts/qapi-gen.py                                |  57 ++++
 scripts/qapi/__init__.py                           |   0
 scripts/{qapi-commands.py => qapi/commands.py}     | 155 ++++-----
 scripts/{qapi.py => qapi/common.py}                | 355 +++++++++++++--------
 scripts/{qapi2texi.py => qapi/doc.py}              |  92 +++---
 scripts/{qapi-event.py => qapi/events.py}          | 128 +++-----
 scripts/{qapi-introspect.py => qapi/introspect.py} | 123 +++----
 scripts/{qapi-types.py => qapi/types.py}           | 185 ++++-------
 scripts/{qapi-visit.py => qapi/visit.py}           | 189 ++++-------
 stubs/tpm.c                                        |   3 +-
 stubs/uuid.c                                       |   2 +-
 stubs/vmgenid.c                                    |   2 +-
 stubs/xen-hvm.c                                    |   2 +-
 target/arm/monitor.c                               |   3 +-
 target/i386/cpu.c                                  |   4 +-
 target/s390x/kvm.c                                 |   1 -
 tests/.gitignore                                   |   9 +-
 tests/Makefile.include                             |  70 ++--
 tests/qapi-schema/comments.out                     |   3 +-
 tests/qapi-schema/doc-bad-section.out              |   5 +-
 tests/qapi-schema/doc-good.out                     |  33 +-
 tests/qapi-schema/doc-good.texi                    |   3 +-
 tests/qapi-schema/empty.out                        |   2 +-
 tests/qapi-schema/event-case.out                   |   3 +-
 tests/qapi-schema/ident-with-escape.out            |   7 +-
 tests/qapi-schema/include-no-file.err              |   2 +-
 tests/qapi-schema/include-relpath.out              |   7 +-
 tests/qapi-schema/include-repetition.out           |  12 +-
 tests/qapi-schema/include-simple.out               |   5 +-
 tests/qapi-schema/indented-expr.out                |   3 +-
 tests/qapi-schema/qapi-schema-test.out             | 321 +++++++++----------
 tests/qapi-schema/test-qapi.py                     |  19 +-
 tests/qmp-test.c                                   |   3 +-
 tests/test-char.c                                  |   2 +-
 tests/{test-qmp-commands.c => test-qmp-cmds.c}     |   2 +-
 tests/test-qmp-event.c                             |   3 +-
 tests/test-qobject-input-visitor.c                 |   8 +-
 tests/test-visitor-serialization.c                 |   1 -
 tpm.c                                              |   3 +-
 trace/qmp.c                                        |   2 +-
 ui/cocoa.m                                         |   2 +-
 ui/console.c                                       |   2 +-
 ui/gtk.c                                           |   2 +-
 ui/input-legacy.c                                  |   2 +-
 ui/input.c                                         |   2 +-
 ui/spice-core.c                                    |   4 +-
 ui/vnc.c                                           |   4 +-
 ui/vnc.h                                           |   1 +
 util/qemu-config.c                                 |   2 +-
 util/qemu-sockets.c                                |   2 +-
 vl.c                                               |   8 +-
 186 files changed, 1556 insertions(+), 1352 deletions(-)
 rename qapi-schema.json => qapi/misc.json (95%)
 create mode 100644 qapi/qapi-schema.json
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (67%)
 rename scripts/{qapi.py => qapi/common.py} (90%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (78%)
 mode change 100755 => 100644
 rename scripts/{qapi-event.py => qapi/events.py} (61%)
 rename scripts/{qapi-introspect.py => qapi/introspect.py} (71%)
 rename scripts/{qapi-types.py => qapi/types.py} (53%)
 rename scripts/{qapi-visit.py => qapi/visit.py} (64%)
 rename tests/{test-qmp-commands.c => test-qmp-cmds.c} (99%)
Signature · 1 line
2.13.6
#1 lore raw

PATCH v2 1/29 Include qapi/qmp/qerror.h exactly where needed

---
 block.c                 | 1 -
 block/qcow2.c           | 1 -
 chardev/char-fe.c       | 1 +
 chardev/char.c          | 1 +
 qom/object_interfaces.c | 1 +
 scripts/qapi-visit.py   | 2 +-
 6 files changed, 4 insertions(+), 3 deletions(-)
block.c +0 -1
index f94585b230..05a484b4b8 100644
@@ -32,7 +32,6 @@
 #include "qemu/module.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qdict.h"
-#include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qjson.h"
 #include "qapi/qmp/qstring.h"
 #include "sysemu/block-backend.h"
block/qcow2.c +0 -1
index a64a572785..9245deac19 100644
@@ -30,7 +30,6 @@
 #include "block/qcow2.h"
 #include "qemu/error-report.h"
 #include "qapi/error.h"
-#include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
 #include "qapi-event.h"
chardev/char-fe.c +1 -0
index c611b3fa3e..e5f870e4d2 100644
@@ -24,6 +24,7 @@
 #include "qemu/osdep.h"
 #include "qemu/error-report.h"
 #include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
 #include "qapi-visit.h"
 #include "sysemu/replay.h"
 
chardev/char.c +1 -0
index 01d979a1da..c9a4da5516 100644
@@ -32,6 +32,7 @@
 #include "qmp-commands.h"
 #include "qapi-visit.h"
 #include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
 #include "sysemu/replay.h"
 #include "qemu/help_option.h"
 #include "qemu/option.h"
qom/object_interfaces.c +1 -0
index 80d09139be..43d9aa0946 100644
@@ -1,6 +1,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qerror.h"
 #include "qom/object_interfaces.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
scripts/qapi-visit.py +1 -1
index 7e1cfc13f0..bc2b8b581a 100644
@@ -371,13 +371,13 @@ fdef.write(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
                  prefix=prefix))
 
 fdecl.write(mcgen('''
 #include "qapi/visitor.h"
-#include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
Signature · 1 line
2.13.6
Reviewed-by on v2 1/29
#2 lore raw

Re: [Qemu-devel] [PATCH v2 01/29] Include qapi/qmp/qerror.h exactly where needed

Quoted 9 lines · Markus Armbruster
---
  block.c                 | 1 -
  block/qcow2.c           | 1 -
  chardev/char-fe.c       | 1 +
  chardev/char.c          | 1 +
  qom/object_interfaces.c | 1 +
  scripts/qapi-visit.py   | 2 +-
  6 files changed, 4 insertions(+), 3 deletions(-)

(May get stale again soon; we'll have fun with last-minute rebasing at the time this is ready for a pull request)

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by Marc-André Lureau on v2 1/29
#3 lore raw

Re: [Qemu-devel] [PATCH v2 01/29] Include qapi/qmp/qerror.h exactly where needed

In reply to Markus Armbruster
Quoted 1 line · Markus Armbruster
Quoted 91 lines
---
 block.c                 | 1 -
 block/qcow2.c           | 1 -
 chardev/char-fe.c       | 1 +
 chardev/char.c          | 1 +
 qom/object_interfaces.c | 1 +
 scripts/qapi-visit.py   | 2 +-
 6 files changed, 4 insertions(+), 3 deletions(-)
block.c +0 -1
index f94585b230..05a484b4b8 100644
@@ -32,7 +32,6 @@
 #include "qemu/module.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qdict.h"
-#include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qjson.h"
 #include "qapi/qmp/qstring.h"
 #include "sysemu/block-backend.h"
block/qcow2.c +0 -1
index a64a572785..9245deac19 100644
@@ -30,7 +30,6 @@
 #include "block/qcow2.h"
 #include "qemu/error-report.h"
 #include "qapi/error.h"
-#include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
 #include "qapi-event.h"
chardev/char-fe.c +1 -0
index c611b3fa3e..e5f870e4d2 100644
@@ -24,6 +24,7 @@
 #include "qemu/osdep.h"
 #include "qemu/error-report.h"
 #include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
 #include "qapi-visit.h"
 #include "sysemu/replay.h"
 
chardev/char.c +1 -0
index 01d979a1da..c9a4da5516 100644
@@ -32,6 +32,7 @@
 #include "qmp-commands.h"
 #include "qapi-visit.h"
 #include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
 #include "sysemu/replay.h"
 #include "qemu/help_option.h"
 #include "qemu/option.h"
qom/object_interfaces.c +1 -0
index 80d09139be..43d9aa0946 100644
@@ -1,6 +1,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qerror.h"
 #include "qom/object_interfaces.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
scripts/qapi-visit.py +1 -2
index 7e1cfc13f0..bc2b8b581a 100644
@@ -371,13 +371,13 @@ fdef.write(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
                  prefix=prefix))
 
 fdecl.write(mcgen('''
 #include "qapi/visitor.h"
-#include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
--

2.13.6

#4 lore raw

PATCH v2 2/29 qapi: Streamline boilerplate comment generation

In reply to Markus Armbruster

Every generator has separate boilerplate for .h and .c, and their differences are boring. All of them repeat the license note.

Reduce the repetition as follows. Move common text like the license note to common open_output(), next to the existing common text there. For each generator, replace the two separate descriptions by a single one.

While there, emit an "automatically generated" note into generated documentation, too.

---
 scripts/qapi-commands.py        | 26 +++-----------------------
 scripts/qapi-event.py           | 26 +++-----------------------
 scripts/qapi-introspect.py      | 21 ++-------------------
 scripts/qapi-types.py           | 26 +++-----------------------
 scripts/qapi-visit.py           | 26 +++-----------------------
 scripts/qapi.py                 | 31 ++++++++++++++++++-------------
 scripts/qapi2texi.py            |  3 ++-
 tests/qapi-schema/doc-good.texi |  3 ++-
 8 files changed, 36 insertions(+), 126 deletions(-)
scripts/qapi-commands.py +3 -23
index f89d748ba4..8e8da7c796 100644
@@ -255,38 +255,18 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
 
-c_comment = '''
-/*
- * schema-defined QMP->QAPI command dispatch
+blurb = '''
+ * Schema-defined QAPI/QMP commands
  *
  * Copyright IBM, Corp. 2011
  *
  * Authors:
  *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI function prototypes
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-marshal.c', 'qmp-commands.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 
scripts/qapi-event.py +3 -23
index c710968dc2..2b7d720c08 100644
@@ -171,38 +171,18 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 
 (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
 
-c_comment = '''
-/*
- * schema-defined QAPI event functions
+blurb = '''
+ * Schema-defined QAPI/QMP events
  *
  * Copyright (c) 2014 Wenchao Xia
  *
  * Authors:
  *  Wenchao Xia   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI event functions
- *
- * Copyright (c) 2014 Wenchao Xia
- *
- * Authors:
- *  Wenchao Xia  <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-event.c', 'qapi-event.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-introspect.py +2 -19
index 032bcea491..83da2bdb94 100644
@@ -176,32 +176,15 @@ for o, a in opts:
     if o in ('-u', '--unmask-non-abi-names'):
         opt_unmask = True
 
-c_comment = '''
-/*
+blurb = '''
  * QAPI/QMP schema introspection
  *
  * Copyright (C) 2015 Red Hat, Inc.
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * QAPI/QMP schema introspection
- *
- * Copyright (C) 2015 Red Hat, Inc.
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-introspect.c', 'qmp-introspect.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-types.py +3 -23
index 7e3051dbb9..86afc57f92 100644
@@ -250,39 +250,19 @@ for o, a in opts:
     if o in ('-b', '--builtins'):
         do_builtins = True
 
-c_comment = '''
-/*
- * deallocation functions for schema-defined QAPI types
+blurb = '''
+ * Schema-defined QAPI types
  *
  * Copyright IBM, Corp. 2011
  *
  * Authors:
  *  Anthony Liguori   <[email protected]>
  *  Michael Roth      <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI types
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-types.c', 'qapi-types.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-visit.py +3 -23
index bc2b8b581a..0a367072fb 100644
@@ -334,38 +334,18 @@ for o, a in opts:
     if o in ('-b', '--builtins'):
         do_builtins = True
 
-c_comment = '''
-/*
- * schema-defined QAPI visitor functions
+blurb = '''
+ * Schema-defined QAPI visitors
  *
  * Copyright IBM, Corp. 2011
  *
  * Authors:
  *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-'''
-h_comment = '''
-/*
- * schema-defined QAPI visitor functions
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-visit.c', 'qapi-visit.h',
-                            c_comment, h_comment)
+                            blurb)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi.py +18 -13
index 58f995b07f..2e5a1ce6a8 100644
@@ -1985,11 +1985,21 @@ def parse_command_line(extra_options='', extra_long_options=[]):
 #
 
 
-def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
-                c_comment, h_comment):
+def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb):
     guard = guardname(prefix + h_file)
     c_file = output_dir + prefix + c_file
     h_file = output_dir + prefix + h_file
+    comment = mcgen('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
+
+/*
+%(blurb)s
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ */
+
+''',
+                    blurb=blurb.strip('\n'))
 
     if output_dir:
         try:
@@ -2007,27 +2017,22 @@ def open_output(output_dir, do_c, do_h, prefix, c_file, h_file,
     fdef = maybe_open(do_c, c_file, 'w')
     fdecl = maybe_open(do_h, h_file, 'w')
 
-    fdef.write(mcgen('''
-/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
-%(comment)s
-''',
-                     comment=c_comment))
-
+    fdef.write(comment)
+    fdecl.write(comment)
     fdecl.write(mcgen('''
-/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
-%(comment)s
 #ifndef %(guard)s
 #define %(guard)s
 
 ''',
-                      comment=h_comment, guard=guard))
+                      guard=guard))
 
     return (fdef, fdecl)
 
 
 def close_output(fdef, fdecl):
-    fdecl.write('''
+    fdecl.write(mcgen('''
+
 #endif
-''')
+'''))
     fdecl.close()
     fdef.close()
scripts/qapi2texi.py +2 -1
index bf1c57b2e2..8a604d86a6 100755
@@ -283,7 +283,8 @@ def main(argv):
         print("%s: need pragma 'doc-required' "
                "to generate documentation" % argv[0], file=sys.stderr)
         sys.exit(1)
-    print(texi_schema(schema))
+    print('@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n')
+    print(texi_schema(schema), end='')
 
 
 if __name__ == '__main__':
tests/qapi-schema/doc-good.texi +2 -1
index 1778312581..0aed2300a5 100644
@@ -1,3 +1,5 @@
+@c AUTOMATICALLY GENERATED, DO NOT MODIFY
+
 @section Section
 
 @subsection Subsection
@@ -231,4 +233,3 @@ If you're bored enough to read this, go see a video of boxed cats
 
 @end deftypefn
 
-
Signature · 1 line
2.13.6
#5 lore raw

PATCH v2 3/29 qapi: Generate up-to-date copyright notice

In reply to Markus Armbruster

Each generator carries a copyright notice for the generator itself, and another one for the files it generates. Only the former have been updated along the way, the latter have not, and are all out of date.

Fix by copying the generator's copyright notice to the generated files instead. Note that the fix doesn't copy the "Authors:" part; the generated files' outdated Authors list goes away without replacement.

---
 scripts/qapi-commands.py   | 34 +++++++++++++++-------------------
 scripts/qapi-event.py      | 32 ++++++++++++++------------------
 scripts/qapi-introspect.py | 25 ++++++++++++-------------
 scripts/qapi-types.py      | 32 ++++++++++++++------------------
 scripts/qapi-visit.py      | 34 +++++++++++++++-------------------
 scripts/qapi.py            |  7 +++++--
 6 files changed, 75 insertions(+), 89 deletions(-)
scripts/qapi-commands.py +15 -19
index 8e8da7c796..84a980d882 100644
@@ -1,16 +1,17 @@
-#
-# QAPI command marshaller generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (C) 2014-2016 Red Hat, Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Michael Roth    <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI command marshaller generator
+
+Copyright IBM, Corp. 2011
+Copyright (C) 2014-2018 Red Hat, Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -257,16 +258,11 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 
 blurb = '''
  * Schema-defined QAPI/QMP commands
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-marshal.c', 'qmp-commands.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 
scripts/qapi-event.py +14 -18
index 2b7d720c08..0a098803e2 100644
@@ -1,15 +1,16 @@
-#
-# QAPI event generator
-#
-# Copyright (c) 2014 Wenchao Xia
-# Copyright (c) 2015-2016 Red Hat Inc.
-#
-# Authors:
-#  Wenchao Xia <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI event generator
+
+Copyright (c) 2014 Wenchao Xia
+Copyright (c) 2015-2018 Red Hat Inc.
+
+Authors:
+ Wenchao Xia <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -173,16 +174,11 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 
 blurb = '''
  * Schema-defined QAPI/QMP events
- *
- * Copyright (c) 2014 Wenchao Xia
- *
- * Authors:
- *  Wenchao Xia   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-event.c', 'qapi-event.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-introspect.py +12 -13
index 83da2bdb94..bd9253a172 100644
@@ -1,13 +1,14 @@
-#
-# QAPI introspection generator
-#
-# Copyright (C) 2015-2016 Red Hat, Inc.
-#
-# Authors:
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI introspection generator
+
+Copyright (C) 2015-2018 Red Hat, Inc.
+
+Authors:
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -178,13 +179,11 @@ for o, a in opts:
 
 blurb = '''
  * QAPI/QMP schema introspection
- *
- * Copyright (C) 2015 Red Hat, Inc.
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-introspect.c', 'qmp-introspect.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-types.py +14 -18
index 86afc57f92..1103dbda2d 100644
@@ -1,15 +1,17 @@
-#
-# QAPI types generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (c) 2013-2016 Red Hat Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
+"""
+QAPI types generator
+
+Copyright IBM, Corp. 2011
+Copyright (c) 2013-2018 Red Hat Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -252,17 +254,11 @@ for o, a in opts:
 
 blurb = '''
  * Schema-defined QAPI types
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *  Michael Roth      <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-types.c', 'qapi-types.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-visit.py +15 -19
index 0a367072fb..e56b3c1256 100644
@@ -1,16 +1,17 @@
-#
-# QAPI visitor generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (C) 2014-2016 Red Hat, Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Michael Roth    <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI visitor generator
+
+Copyright IBM, Corp. 2011
+Copyright (C) 2014-2018 Red Hat, Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth    <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -336,16 +337,11 @@ for o, a in opts:
 
 blurb = '''
  * Schema-defined QAPI visitors
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-visit.c', 'qapi-visit.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi.py +5 -2
index 2e5a1ce6a8..26c45cee34 100644
@@ -1985,21 +1985,24 @@ def parse_command_line(extra_options='', extra_long_options=[]):
 #
 
 
-def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb):
+def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb, doc):
     guard = guardname(prefix + h_file)
     c_file = output_dir + prefix + c_file
     h_file = output_dir + prefix + h_file
+    copyright = '\n * '.join(re.findall(r'^Copyright .*', doc, re.MULTILINE))
     comment = mcgen('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
 /*
 %(blurb)s
  *
+ * %(copyright)s
+ *
  * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
  * See the COPYING.LIB file in the top-level directory.
  */
 
 ''',
-                    blurb=blurb.strip('\n'))
+                    blurb=blurb.strip('\n'), copyright=copyright)
 
     if output_dir:
         try:
Signature · 1 line
2.13.6
#6 lore raw

Re: [Qemu-devel] [PATCH v2 03/29] qapi: Generate up-to-date copyright notice

Quoted 16 lines · Markus Armbruster

Each generator carries a copyright notice for the generator itself, and another one for the files it generates. Only the former have been updated along the way, the latter have not, and are all out of date.

Fix by copying the generator's copyright notice to the generated files instead. Note that the fix doesn't copy the "Authors:" part; the generated files' outdated Authors list goes away without replacement.

---

+++ b/scripts/qapi-commands.py

scripts/qapi-commands.py class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): · line 261

Quoted 9 lines
scripts/qapi-commands.py +0 -5
@@ -257,16 +258,11 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 
  blurb = '''
   * Schema-defined QAPI/QMP commands
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
  '''

This is where we could shorten blurb to a one-line assignment instead of a multiline '''...''' string, if desired. R-b stands either way, though.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v2 3/29
#7 lore raw

Re: [Qemu-devel] [PATCH v2 03/29] qapi: Generate up-to-date copyright notice

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:41)

Quoted 11 lines

Each generator carries a copyright notice for the generator itself, and another one for the files it generates. Only the former have been updated along the way, the latter have not, and are all out of date.

Fix by copying the generator's copyright notice to the generated files instead. Note that the fix doesn't copy the "Authors:" part; the generated files' outdated Authors list goes away without replacement.

Quoted 296 lines
---
 scripts/qapi-commands.py   | 34 +++++++++++++++-------------------
 scripts/qapi-event.py      | 32 ++++++++++++++------------------
 scripts/qapi-introspect.py | 25 ++++++++++++-------------
 scripts/qapi-types.py      | 32 ++++++++++++++------------------
 scripts/qapi-visit.py      | 34 +++++++++++++++-------------------
 scripts/qapi.py            |  7 +++++--
 6 files changed, 75 insertions(+), 89 deletions(-)
scripts/qapi-commands.py +15 -19
index 8e8da7c796..84a980d882 100644
@@ -1,16 +1,17 @@
-#
-# QAPI command marshaller generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (C) 2014-2016 Red Hat, Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Michael Roth    <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI command marshaller generator
+
+Copyright IBM, Corp. 2011
+Copyright (C) 2014-2018 Red Hat, Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -257,16 +258,11 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 
 blurb = '''
  * Schema-defined QAPI/QMP commands
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-marshal.c', 'qmp-commands.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 
scripts/qapi-event.py +14 -18
index 2b7d720c08..0a098803e2 100644
@@ -1,15 +1,16 @@
-#
-# QAPI event generator
-#
-# Copyright (c) 2014 Wenchao Xia
-# Copyright (c) 2015-2016 Red Hat Inc.
-#
-# Authors:
-#  Wenchao Xia <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI event generator
+
+Copyright (c) 2014 Wenchao Xia
+Copyright (c) 2015-2018 Red Hat Inc.
+
+Authors:
+ Wenchao Xia <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -173,16 +174,11 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 
 blurb = '''
  * Schema-defined QAPI/QMP events
- *
- * Copyright (c) 2014 Wenchao Xia
- *
- * Authors:
- *  Wenchao Xia   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-event.c', 'qapi-event.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-introspect.py +12 -13
index 83da2bdb94..bd9253a172 100644
@@ -1,13 +1,14 @@
-#
-# QAPI introspection generator
-#
-# Copyright (C) 2015-2016 Red Hat, Inc.
-#
-# Authors:
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI introspection generator
+
+Copyright (C) 2015-2018 Red Hat, Inc.
+
+Authors:
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -178,13 +179,11 @@ for o, a in opts:
 
 blurb = '''
  * QAPI/QMP schema introspection
- *
- * Copyright (C) 2015 Red Hat, Inc.
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qmp-introspect.c', 'qmp-introspect.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-types.py +14 -18
index 86afc57f92..1103dbda2d 100644
@@ -1,15 +1,17 @@
-#
-# QAPI types generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (c) 2013-2016 Red Hat Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
+"""
+QAPI types generator
+
+Copyright IBM, Corp. 2011
+Copyright (c) 2013-2018 Red Hat Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -252,17 +254,11 @@ for o, a in opts:
 
 blurb = '''
  * Schema-defined QAPI types
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
- *  Michael Roth      <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-types.c', 'qapi-types.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi-visit.py +15 -19
index 0a367072fb..e56b3c1256 100644
@@ -1,16 +1,17 @@
-#
-# QAPI visitor generator
-#
-# Copyright IBM, Corp. 2011
-# Copyright (C) 2014-2016 Red Hat, Inc.
-#
-# Authors:
-#  Anthony Liguori <[email protected]>
-#  Michael Roth    <[email protected]>
-#  Markus Armbruster <[email protected]>
-#
-# This work is licensed under the terms of the GNU GPL, version 2.
-# See the COPYING file in the top-level directory.
+"""
+QAPI visitor generator
+
+Copyright IBM, Corp. 2011
+Copyright (C) 2014-2018 Red Hat, Inc.
+
+Authors:
+ Anthony Liguori <[email protected]>
+ Michael Roth    <[email protected]>
+ Markus Armbruster <[email protected]>
+
+This work is licensed under the terms of the GNU GPL, version 2.
+See the COPYING file in the top-level directory.
+"""
 
 from qapi import *
 
@@ -336,16 +337,11 @@ for o, a in opts:
 
 blurb = '''
  * Schema-defined QAPI visitors
- *
- * Copyright IBM, Corp. 2011
- *
- * Authors:
- *  Anthony Liguori   <[email protected]>
 '''
 
 (fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
                             'qapi-visit.c', 'qapi-visit.h',
-                            blurb)
+                            blurb, __doc__)
 
 fdef.write(mcgen('''
 #include "qemu/osdep.h"
scripts/qapi.py +5 -2
index 2e5a1ce6a8..26c45cee34 100644
@@ -1985,21 +1985,24 @@ def parse_command_line(extra_options='', extra_long_options=[]):
 #
 
 
-def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb):
+def open_output(output_dir, do_c, do_h, prefix, c_file, h_file, blurb, doc):
     guard = guardname(prefix + h_file)
     c_file = output_dir + prefix + c_file
     h_file = output_dir + prefix + h_file
+    copyright = '\n * '.join(re.findall(r'^Copyright .*', doc, re.MULTILINE))
     comment = mcgen('''/* AUTOMATICALLY GENERATED, DO NOT MODIFY */
 
 /*
 %(blurb)s
  *
+ * %(copyright)s
+ *
  * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
  * See the COPYING.LIB file in the top-level directory.
  */
 
 ''',
-                    blurb=blurb.strip('\n'))
+                    blurb=blurb.strip('\n'), copyright=copyright)
 
     if output_dir:
         try:
Signature · 1 line
2.13.6
#8 lore raw

PATCH v2 4/29 qapi: Rename variable holding the QAPISchemaGenFOOVisitor

In reply to Markus Armbruster

Rename the variable holding the QAPISchemaGenFOOVisitor from gen to vis, to avoid confusion in the next commit.

---
 scripts/qapi-commands.py   | 8 ++++----
 scripts/qapi-event.py      | 8 ++++----
 scripts/qapi-introspect.py | 8 ++++----
 scripts/qapi-types.py      | 8 ++++----
 scripts/qapi-visit.py      | 8 ++++----
 5 files changed, 20 insertions(+), 20 deletions(-)
scripts/qapi-commands.py +4 -4
index 84a980d882..c3aa52fce1 100644
@@ -291,9 +291,9 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
                   prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenCommandVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenCommandVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-event.py +4 -4
index 0a098803e2..edb9ddb650 100644
@@ -203,9 +203,9 @@ fdecl.write(mcgen('''
 event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenEventVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenEventVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-introspect.py +4 -4
index bd9253a172..ebe8706f41 100644
@@ -193,9 +193,9 @@ fdef.write(mcgen('''
                  prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenIntrospectVisitor(opt_unmask)
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-types.py +4 -4
index 1103dbda2d..4db8424da1 100644
@@ -273,9 +273,9 @@ fdecl.write(mcgen('''
 '''))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenTypeVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenTypeVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-visit.py +4 -4
index e56b3c1256..3c1a0e2544 100644
@@ -360,9 +360,9 @@ fdecl.write(mcgen('''
                   prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenVisitVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenVisitVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
Signature · 1 line
2.13.6
Reviewed-by on v2 4/29
#9 lore raw

Re: [Qemu-devel] [PATCH v2 04/29] qapi: Rename variable holding the QAPISchemaGenFOOVisitor

Quoted 11 lines · Markus Armbruster

Rename the variable holding the QAPISchemaGenFOOVisitor from gen to vis, to avoid confusion in the next commit.

---
  scripts/qapi-commands.py   | 8 ++++----
  scripts/qapi-event.py      | 8 ++++----
  scripts/qapi-introspect.py | 8 ++++----
  scripts/qapi-types.py      | 8 ++++----
  scripts/qapi-visit.py      | 8 ++++----
  5 files changed, 20 insertions(+), 20 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by Marc-André Lureau on v2 4/29
#10 lore raw

Re: [Qemu-devel] [PATCH v2 04/29] qapi: Rename variable holding the QAPISchemaGenFOOVisitor

In reply to Markus Armbruster
Quoted 4 lines · Markus Armbruster

Rename the variable holding the QAPISchemaGenFOOVisitor from gen to vis, to avoid confusion in the next commit.

Quoted 101 lines
---
 scripts/qapi-commands.py   | 8 ++++----
 scripts/qapi-event.py      | 8 ++++----
 scripts/qapi-introspect.py | 8 ++++----
 scripts/qapi-types.py      | 8 ++++----
 scripts/qapi-visit.py      | 8 ++++----
 5 files changed, 20 insertions(+), 20 deletions(-)
scripts/qapi-commands.py +4 -4
index 84a980d882..c3aa52fce1 100644
@@ -291,9 +291,9 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
                   prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenCommandVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenCommandVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-event.py +4 -4
index 0a098803e2..edb9ddb650 100644
@@ -203,9 +203,9 @@ fdecl.write(mcgen('''
 event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenEventVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenEventVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-introspect.py +4 -4
index bd9253a172..ebe8706f41 100644
@@ -193,9 +193,9 @@ fdef.write(mcgen('''
                  prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenIntrospectVisitor(opt_unmask)
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-types.py +4 -4
index 1103dbda2d..4db8424da1 100644
@@ -273,9 +273,9 @@ fdecl.write(mcgen('''
 '''))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenTypeVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenTypeVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-visit.py +4 -5
index e56b3c1256..3c1a0e2544 100644
@@ -360,9 +360,9 @@ fdecl.write(mcgen('''
                   prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenVisitVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenVisitVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
--

2.13.6

Reviewed-by on v2 4/29
#11 lore raw

Re: [Qemu-devel] [PATCH v2 04/29] qapi: Rename variable holding the QAPISchemaGenFOOVisitor

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:42)

Quoted 4 lines

Rename the variable holding the QAPISchemaGenFOOVisitor from gen to vis, to avoid confusion in the next commit.

Quoted 101 lines
---
 scripts/qapi-commands.py   | 8 ++++----
 scripts/qapi-event.py      | 8 ++++----
 scripts/qapi-introspect.py | 8 ++++----
 scripts/qapi-types.py      | 8 ++++----
 scripts/qapi-visit.py      | 8 ++++----
 5 files changed, 20 insertions(+), 20 deletions(-)
scripts/qapi-commands.py +4 -4
index 84a980d882..c3aa52fce1 100644
@@ -291,9 +291,9 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
                   prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenCommandVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenCommandVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-event.py +4 -4
index 0a098803e2..edb9ddb650 100644
@@ -203,9 +203,9 @@ fdecl.write(mcgen('''
 event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenEventVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenEventVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-introspect.py +4 -4
index bd9253a172..ebe8706f41 100644
@@ -193,9 +193,9 @@ fdef.write(mcgen('''
                  prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenIntrospectVisitor(opt_unmask)
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-types.py +4 -4
index 1103dbda2d..4db8424da1 100644
@@ -273,9 +273,9 @@ fdecl.write(mcgen('''
 '''))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenTypeVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenTypeVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
scripts/qapi-visit.py +4 -4
index e56b3c1256..3c1a0e2544 100644
@@ -360,9 +360,9 @@ fdecl.write(mcgen('''
                   prefix=prefix))
 
 schema = QAPISchema(input_file)
-gen = QAPISchemaGenVisitVisitor()
-schema.visit(gen)
-fdef.write(gen.defn)
-fdecl.write(gen.decl)
+vis = QAPISchemaGenVisitVisitor()
+schema.visit(vis)
+fdef.write(vis.defn)
+fdecl.write(vis.decl)
 
 close_output(fdef, fdecl)
Signature · 1 line
2.13.6
#12 lore raw

PATCH v2 5/29 qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc

In reply to Markus Armbruster

These classes encapsulate accumulating and writing output.

Convert C code generation to QAPIGenC and QAPIGenH. The conversion is rather shallow: most of the output accumulation is not converted. Left for later.

The indentation machinery uses a single global variable indent_level, even though we generally interleave creation of a .c and its .h. It should become instance variable of QAPIGenC. Also left for later.

Documentation generation isn't converted, and QAPIGenDoc isn't used. This will change shortly.

---
 scripts/qapi-commands.py   | 23 +++++------
 scripts/qapi-event.py      | 22 ++++++-----
 scripts/qapi-introspect.py | 18 +++++----
 scripts/qapi-types.py      | 22 ++++++-----
 scripts/qapi-visit.py      | 22 ++++++-----
 scripts/qapi.py            | 99 +++++++++++++++++++++++++---------------------
 6 files changed, 112 insertions(+), 94 deletions(-)
scripts/qapi-commands.py +12 -11
index c3aa52fce1..8d38ade076 100644
@@ -260,12 +260,10 @@ blurb = '''
  * Schema-defined QAPI/QMP commands
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-marshal.c', 'qmp-commands.h',
-                            blurb, __doc__)
-
-fdef.write(mcgen('''
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
+genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -280,20 +278,23 @@ fdef.write(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                 prefix=prefix))
+               prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.add(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-                  prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+               prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
 vis = QAPISchemaGenCommandVisitor()
 schema.visit(vis)
-fdef.write(vis.defn)
-fdecl.write(vis.decl)
+genc.add(vis.defn)
+genh.add(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-marshal.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-commands.h')
scripts/qapi-event.py +12 -10
index edb9ddb650..bd7a9be3dc 100644
@@ -176,11 +176,10 @@ blurb = '''
  * Schema-defined QAPI/QMP events
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-event.c', 'qapi-event.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "%(prefix)sqapi-event.h"
@@ -191,21 +190,24 @@ fdef.write(mcgen('''
 #include "qapi/qmp-event.h"
 
 ''',
-                 prefix=prefix))
+               prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.add(mcgen('''
 #include "qapi/util.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                  prefix=prefix))
+               prefix=prefix))
 
 event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
 
 schema = QAPISchema(input_file)
 vis = QAPISchemaGenEventVisitor()
 schema.visit(vis)
-fdef.write(vis.defn)
-fdecl.write(vis.decl)
+genc.add(vis.defn)
+genh.add(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-event.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-event.h')
scripts/qapi-introspect.py +10 -8
index ebe8706f41..3d65690fe3 100644
@@ -181,21 +181,23 @@ blurb = '''
  * QAPI/QMP schema introspection
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-introspect.c', 'qmp-introspect.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "%(prefix)sqmp-introspect.h"
 
 ''',
-                 prefix=prefix))
+               prefix=prefix))
 
 schema = QAPISchema(input_file)
 vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
 schema.visit(vis)
-fdef.write(vis.defn)
-fdecl.write(vis.decl)
+genc.add(vis.defn)
+genh.add(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-introspect.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-introspect.h')
scripts/qapi-types.py +12 -10
index 4db8424da1..c0ac879beb 100644
@@ -180,7 +180,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.decl = ''
         self.defn = ''
         self._fwdecl = ''
-        self._btin = guardstart('QAPI_TYPES_BUILTIN')
+        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
 
     def visit_end(self):
         self.decl = self._fwdecl + self.decl
@@ -256,26 +256,28 @@ blurb = '''
  * Schema-defined QAPI types
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-types.c', 'qapi-types.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
 #include "%(prefix)sqapi-types.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                 prefix=prefix))
+               prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.add(mcgen('''
 #include "qapi/util.h"
 '''))
 
 schema = QAPISchema(input_file)
 vis = QAPISchemaGenTypeVisitor()
 schema.visit(vis)
-fdef.write(vis.defn)
-fdecl.write(vis.decl)
+genc.add(vis.defn)
+genh.add(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-types.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-types.h')
scripts/qapi-visit.py +12 -10
index 3c1a0e2544..888c686303 100644
@@ -339,30 +339,32 @@ blurb = '''
  * Schema-defined QAPI visitors
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-visit.c', 'qapi-visit.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                 prefix=prefix))
+               prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.add(mcgen('''
 #include "qapi/visitor.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                  prefix=prefix))
+               prefix=prefix))
 
 schema = QAPISchema(input_file)
 vis = QAPISchemaGenVisitVisitor()
 schema.visit(vis)
-fdef.write(vis.defn)
-fdecl.write(vis.decl)
+genc.add(vis.defn)
+genh.add(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-visit.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-visit.h')
scripts/qapi.py +54 -45

142 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v2 5/29
#13 lore raw

Re: [Qemu-devel] [PATCH v2 05/29] qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc

Quoting Markus Armbruster (2018-02-11 03:35:43)

Quoted 10 lines

These classes encapsulate accumulating and writing output.

Convert C code generation to QAPIGenC and QAPIGenH. The conversion is rather shallow: most of the output accumulation is not converted. Left for later.

The indentation machinery uses a single global variable indent_level, even though we generally interleave creation of a .c and its .h. It should become instance variable of QAPIGenC. Also left for later.

Quoted 6 lines

Documentation generation isn't converted, and QAPIGenDoc isn't used. This will change shortly.

2 minor nits below, but in any case:

Quoted 150 lines
---
 scripts/qapi-commands.py   | 23 +++++------
 scripts/qapi-event.py      | 22 ++++++-----
 scripts/qapi-introspect.py | 18 +++++----
 scripts/qapi-types.py      | 22 ++++++-----
 scripts/qapi-visit.py      | 22 ++++++-----
 scripts/qapi.py            | 99 +++++++++++++++++++++++++---------------------
 6 files changed, 112 insertions(+), 94 deletions(-)
scripts/qapi-commands.py +12 -11
index c3aa52fce1..8d38ade076 100644
@@ -260,12 +260,10 @@ blurb = '''
  * Schema-defined QAPI/QMP commands
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-marshal.c', 'qmp-commands.h',
-                            blurb, __doc__)
-
-fdef.write(mcgen('''
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
+genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -280,20 +278,23 @@ fdef.write(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                 prefix=prefix))
+               prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.add(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-                  prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+               prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
 vis = QAPISchemaGenCommandVisitor()
 schema.visit(vis)
-fdef.write(vis.defn)
-fdecl.write(vis.decl)
+genc.add(vis.defn)
+genh.add(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-marshal.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-commands.h')
scripts/qapi-event.py +12 -10
index edb9ddb650..bd7a9be3dc 100644
@@ -176,11 +176,10 @@ blurb = '''
  * Schema-defined QAPI/QMP events
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-event.c', 'qapi-event.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "%(prefix)sqapi-event.h"
@@ -191,21 +190,24 @@ fdef.write(mcgen('''
 #include "qapi/qmp-event.h"
 
 ''',
-                 prefix=prefix))
+               prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.add(mcgen('''
 #include "qapi/util.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                  prefix=prefix))
+               prefix=prefix))
 
 event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
 
 schema = QAPISchema(input_file)
 vis = QAPISchemaGenEventVisitor()
 schema.visit(vis)
-fdef.write(vis.defn)
-fdecl.write(vis.decl)
+genc.add(vis.defn)
+genh.add(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-event.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-event.h')
scripts/qapi-introspect.py +10 -8
index ebe8706f41..3d65690fe3 100644
@@ -181,21 +181,23 @@ blurb = '''
  * QAPI/QMP schema introspection
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-introspect.c', 'qmp-introspect.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "%(prefix)sqmp-introspect.h"
 
 ''',
-                 prefix=prefix))
+               prefix=prefix))
 
 schema = QAPISchema(input_file)
 vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
 schema.visit(vis)
-fdef.write(vis.defn)
-fdecl.write(vis.decl)
+genc.add(vis.defn)
+genh.add(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-introspect.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-introspect.h')
scripts/qapi-types.py +0 -0
index 4db8424da1..c0ac879beb 100644

scripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 183

Quoted 5 lines
scripts/qapi-types.py +1 -1
@@ -180,7 +180,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.decl = ''
         self.defn = ''
         self._fwdecl = ''
-        self._btin = guardstart('QAPI_TYPES_BUILTIN')
+        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')

Minor nit, but if we compensate for guardstart() change here, shouldn't we do the same in QAPISchemaGenVisitVisitor? Both are cosmetic (though Visit is in less need since it has extra an extra newline already, but changing one and not the other to compensate here makes the patch appear less mechanical, and the resulting formatting fix-up gets dropped later in the series anyway)

Quoted 3 lines
scripts/qapi-types.py +0 -0
     def visit_end(self):
         self.decl = self._fwdecl + self.decl

<snip>

scripts/qapi-types.py

Quoted 4 lines
scripts/qapi-types.py +4 -0
+class QAPIGenDoc(QAPIGen):
+    def _top(self, fname):
+        return (QAPIGen._top(self, fname)
+                + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')

The whitespace change in
"qapi/types qapi/visit: Generate built-in stuff into separate files" should probably be squashed in here:

class QAPIGenDoc(QAPIGen):
+
def _top(self, fname):
return (QAPIGen._top(self, fname)
+ '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')

Quoted 3 lines
Signature · 1 line
2.13.6
#14 lore raw

Re: [Qemu-devel] [PATCH v2 05/29] qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc

Quoted 176 lines · Michael Roth

Quoting Markus Armbruster (2018-02-11 03:35:43)

Quoted 10 lines

These classes encapsulate accumulating and writing output.

Convert C code generation to QAPIGenC and QAPIGenH. The conversion is rather shallow: most of the output accumulation is not converted. Left for later.

The indentation machinery uses a single global variable indent_level, even though we generally interleave creation of a .c and its .h. It should become instance variable of QAPIGenC. Also left for later.

Quoted 6 lines

Documentation generation isn't converted, and QAPIGenDoc isn't used. This will change shortly.

2 minor nits below, but in any case:

Quoted 154 lines
---
 scripts/qapi-commands.py   | 23 +++++------
 scripts/qapi-event.py      | 22 ++++++-----
 scripts/qapi-introspect.py | 18 +++++----
 scripts/qapi-types.py      | 22 ++++++-----
 scripts/qapi-visit.py      | 22 ++++++-----
 scripts/qapi.py            | 99 +++++++++++++++++++++++++---------------------
 6 files changed, 112 insertions(+), 94 deletions(-)
scripts/qapi-commands.py +12 -11
index c3aa52fce1..8d38ade076 100644
@@ -260,12 +260,10 @@ blurb = '''
  * Schema-defined QAPI/QMP commands
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-marshal.c', 'qmp-commands.h',
-                            blurb, __doc__)
-
-fdef.write(mcgen('''
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
+genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -280,20 +278,23 @@ fdef.write(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                 prefix=prefix))
+               prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.add(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-                  prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+               prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
 vis = QAPISchemaGenCommandVisitor()
 schema.visit(vis)
-fdef.write(vis.defn)
-fdecl.write(vis.decl)
+genc.add(vis.defn)
+genh.add(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-marshal.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-commands.h')
scripts/qapi-event.py +12 -10
index edb9ddb650..bd7a9be3dc 100644
@@ -176,11 +176,10 @@ blurb = '''
  * Schema-defined QAPI/QMP events
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qapi-event.c', 'qapi-event.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "%(prefix)sqapi-event.h"
@@ -191,21 +190,24 @@ fdef.write(mcgen('''
 #include "qapi/qmp-event.h"
 
 ''',
-                 prefix=prefix))
+               prefix=prefix))
 
-fdecl.write(mcgen('''
+genh.add(mcgen('''
 #include "qapi/util.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                  prefix=prefix))
+               prefix=prefix))
 
 event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
 
 schema = QAPISchema(input_file)
 vis = QAPISchemaGenEventVisitor()
 schema.visit(vis)
-fdef.write(vis.defn)
-fdecl.write(vis.decl)
+genc.add(vis.defn)
+genh.add(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qapi-event.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qapi-event.h')
scripts/qapi-introspect.py +10 -8
index ebe8706f41..3d65690fe3 100644
@@ -181,21 +181,23 @@ blurb = '''
  * QAPI/QMP schema introspection
 '''
 
-(fdef, fdecl) = open_output(output_dir, do_c, do_h, prefix,
-                            'qmp-introspect.c', 'qmp-introspect.h',
-                            blurb, __doc__)
+genc = QAPIGenC(blurb, __doc__)
+genh = QAPIGenH(blurb, __doc__)
 
-fdef.write(mcgen('''
+genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "%(prefix)sqmp-introspect.h"
 
 ''',
-                 prefix=prefix))
+               prefix=prefix))
 
 schema = QAPISchema(input_file)
 vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
 schema.visit(vis)
-fdef.write(vis.defn)
-fdecl.write(vis.decl)
+genc.add(vis.defn)
+genh.add(vis.decl)
 
-close_output(fdef, fdecl)
+if do_c:
+    genc.write(output_dir, prefix + 'qmp-introspect.c')
+if do_h:
+    genh.write(output_dir, prefix + 'qmp-introspect.h')
scripts/qapi-types.py +0 -1
index 4db8424da1..c0ac879beb 100644
@@ -180,7 +180,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.decl = ''
         self.defn = ''
         self._fwdecl = ''
-        self._btin = guardstart('QAPI_TYPES_BUILTIN')
Quoted 8 lines

scripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 183

Quoted 1 line
scripts/qapi-types.py +1 -0
+        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')

Minor nit, but if we compensate for guardstart() change here, shouldn't we do the same in QAPISchemaGenVisitVisitor? Both are cosmetic (though Visit is in less need since it has extra an extra newline already, but changing one and not the other to compensate here makes the patch appear less mechanical, and the resulting formatting fix-up gets dropped later in the series anyway)

Yes, it's better to keep the two consistent.

Quoted 14 lines
Quoted 2 lines
scripts/qapi-types.py +0 -0
     def visit_end(self):
         self.decl = self._fwdecl + self.decl

<snip>

scripts/qapi-types.py

Quoted 4 lines
scripts/qapi-types.py +4 -0
+class QAPIGenDoc(QAPIGen):
+    def _top(self, fname):
+        return (QAPIGen._top(self, fname)
+                + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')

The whitespace change in
"qapi/types qapi/visit: Generate built-in stuff into separate files" should probably be squashed in here:

Quoted 5 lines

class QAPIGenDoc(QAPIGen):
+
def _top(self, fname):
return (QAPIGen._top(self, fname)
+ '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')

Yes.

#15 lore raw

PATCH v2 6/29 qapi: Reduce use of global variables in generators some

In reply to Markus Armbruster

In preparation of the next commit, which will turn the generators into modules. These global variables will become local to main() then.

---
 scripts/qapi-commands.py   |  9 +++++----
 scripts/qapi-event.py      | 15 +++++++--------
 scripts/qapi-introspect.py |  7 ++++---
 scripts/qapi-types.py      | 17 +++++++++--------
 scripts/qapi-visit.py      | 17 +++++++++--------
 5 files changed, 34 insertions(+), 31 deletions(-)
scripts/qapi-commands.py +5 -4
index 8d38ade076..e97e16e828 100644
@@ -207,7 +207,7 @@ def gen_register_command(name, success_response):
     return ret
 
 
-def gen_registry(registry):
+def gen_registry(registry, prefix):
     ret = mcgen('''
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -224,7 +224,8 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
 
 
 class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, prefix):
+        self._prefix = prefix
         self.decl = None
         self.defn = None
         self._regy = None
@@ -237,7 +238,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._visited_ret_types = set()
 
     def visit_end(self):
-        self.defn += gen_registry(self._regy)
+        self.defn += gen_registry(self._regy, self._prefix)
         self._regy = None
         self._visited_ret_types = None
 
@@ -289,7 +290,7 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
                prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenCommandVisitor()
+vis = QAPISchemaGenCommandVisitor(prefix)
 schema.visit(vis)
 genc.add(vis.defn)
 genh.add(vis.decl)
scripts/qapi-event.py +7 -8
index bd7a9be3dc..3f98e2491a 100644
@@ -58,7 +58,7 @@ def gen_param_var(typ):
     return ret
 
 
-def gen_event_send(name, arg_type, boxed):
+def gen_event_send(name, arg_type, boxed, event_enum_name):
     # FIXME: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
@@ -149,7 +149,8 @@ out:
 
 
 class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, prefix):
+        self._enum_name = c_name(prefix + 'QAPIEvent', protect=False)
         self.decl = None
         self.defn = None
         self._event_names = None
@@ -160,13 +161,13 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self._event_names = []
 
     def visit_end(self):
-        self.decl += gen_enum(event_enum_name, self._event_names)
-        self.defn += gen_enum_lookup(event_enum_name, self._event_names)
+        self.decl += gen_enum(self._enum_name, self._event_names)
+        self.defn += gen_enum_lookup(self._enum_name, self._event_names)
         self._event_names = None
 
     def visit_event(self, name, info, arg_type, boxed):
         self.decl += gen_event_send_decl(name, arg_type, boxed)
-        self.defn += gen_event_send(name, arg_type, boxed)
+        self.defn += gen_event_send(name, arg_type, boxed, self._enum_name)
         self._event_names.append(name)
 
 
@@ -199,10 +200,8 @@ genh.add(mcgen('''
 ''',
                prefix=prefix))
 
-event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenEventVisitor()
+vis = QAPISchemaGenEventVisitor(prefix)
 schema.visit(vis)
 genc.add(vis.defn)
 genh.add(vis.decl)
scripts/qapi-introspect.py +4 -3
index 3d65690fe3..2418b80a82 100644
@@ -41,7 +41,8 @@ def to_c_string(string):
 
 
 class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
-    def __init__(self, unmask):
+    def __init__(self, prefix, unmask):
+        self._prefix = prefix
         self._unmask = unmask
         self.defn = None
         self.decl = None
@@ -65,7 +66,7 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
         # generate C
         # TODO can generate awfully long lines
         jsons.extend(self._jsons)
-        name = c_name(prefix, protect=False) + 'qmp_schema_json'
+        name = c_name(self._prefix, protect=False) + 'qmp_schema_json'
         self.decl = mcgen('''
 extern const char %(c_name)s[];
 ''',
@@ -192,7 +193,7 @@ genc.add(mcgen('''
                prefix=prefix))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
+vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
 schema.visit(vis)
 genc.add(vis.defn)
 genh.add(vis.decl)
scripts/qapi-types.py +9 -8
index c0ac879beb..5ff2bfcf41 100644
@@ -168,7 +168,8 @@ void qapi_free_%(c_name)s(%(c_name)s *obj)
 
 
 class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, opt_builtins):
+        self._opt_builtins = opt_builtins
         self.decl = None
         self.defn = None
         self._fwdecl = None
@@ -187,7 +188,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self._fwdecl = None
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
-        # simply guard them.  See also do_builtins (command line
+        # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_TYPES_BUILTIN')
         self.decl = self._btin + self.decl
@@ -202,7 +203,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         # TODO use something cleaner than existence of info
         if not info:
             self._btin += gen_enum(name, values, prefix)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_enum_lookup(name, values, prefix)
         else:
             self._fwdecl += gen_enum(name, values, prefix)
@@ -213,7 +214,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
             self._btin += gen_fwd_object_or_array(name)
             self._btin += gen_array(name, element_type)
             self._btin += gen_type_cleanup_decl(name)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_type_cleanup(name)
         else:
             self._fwdecl += gen_fwd_object_or_array(name)
@@ -241,16 +242,16 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
 # If you link code generated from multiple schemata, you want only one
 # instance of the code for built-in types.  Generate it only when
-# do_builtins, enabled by command line option -b.  See also
+# opt_builtins, enabled by command line option -b.  See also
 # QAPISchemaGenTypeVisitor.visit_end().
-do_builtins = False
+opt_builtins = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
     parse_command_line('b', ['builtins'])
 
 for o, a in opts:
     if o in ('-b', '--builtins'):
-        do_builtins = True
+        opt_builtins = True
 
 blurb = '''
  * Schema-defined QAPI types
@@ -272,7 +273,7 @@ genh.add(mcgen('''
 '''))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenTypeVisitor()
+vis = QAPISchemaGenTypeVisitor(opt_builtins)
 schema.visit(vis)
 genc.add(vis.defn)
 genh.add(vis.decl)
scripts/qapi-visit.py +9 -8
index 888c686303..0a1eff7736 100644
@@ -264,7 +264,8 @@ out:
 
 
 class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, opt_builtins):
+        self._opt_builtins = opt_builtins
         self.decl = None
         self.defn = None
         self._btin = None
@@ -277,7 +278,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
     def visit_end(self):
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
-        # simply guard them.  See also do_builtins (command line
+        # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_VISIT_BUILTIN')
         self.decl = self._btin + self.decl
@@ -288,7 +289,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         # TODO use something cleaner than existence of info
         if not info:
             self._btin += gen_visit_decl(name, scalar=True)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_visit_enum(name)
         else:
             self.decl += gen_visit_decl(name, scalar=True)
@@ -299,7 +300,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         defn = gen_visit_list(name, element_type)
         if isinstance(element_type, QAPISchemaBuiltinType):
             self._btin += decl
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += defn
         else:
             self.decl += decl
@@ -324,16 +325,16 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
 
 # If you link code generated from multiple schemata, you want only one
 # instance of the code for built-in types.  Generate it only when
-# do_builtins, enabled by command line option -b.  See also
+# opt_builtins, enabled by command line option -b.  See also
 # QAPISchemaGenVisitVisitor.visit_end().
-do_builtins = False
+opt_builtins = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
     parse_command_line('b', ['builtins'])
 
 for o, a in opts:
     if o in ('-b', '--builtins'):
-        do_builtins = True
+        opt_builtins = True
 
 blurb = '''
  * Schema-defined QAPI visitors
@@ -359,7 +360,7 @@ genh.add(mcgen('''
                prefix=prefix))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenVisitVisitor()
+vis = QAPISchemaGenVisitVisitor(opt_builtins)
 schema.visit(vis)
 genc.add(vis.defn)
 genh.add(vis.decl)
Signature · 1 line
2.13.6
Reviewed-by on v2 6/29
#16 lore raw

Re: [Qemu-devel] [PATCH v2 06/29] qapi: Reduce use of global variables in generators some

Quoting Markus Armbruster (2018-02-11 03:35:44)

Quoted 6 lines

In preparation of the next commit, which will turn the generators into modules. These global variables will become local to main() then.

Quoted 276 lines
---
 scripts/qapi-commands.py   |  9 +++++----
 scripts/qapi-event.py      | 15 +++++++--------
 scripts/qapi-introspect.py |  7 ++++---
 scripts/qapi-types.py      | 17 +++++++++--------
 scripts/qapi-visit.py      | 17 +++++++++--------
 5 files changed, 34 insertions(+), 31 deletions(-)
scripts/qapi-commands.py +5 -4
index 8d38ade076..e97e16e828 100644
@@ -207,7 +207,7 @@ def gen_register_command(name, success_response):
     return ret
 
 
-def gen_registry(registry):
+def gen_registry(registry, prefix):
     ret = mcgen('''
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
@@ -224,7 +224,8 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
 
 
 class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, prefix):
+        self._prefix = prefix
         self.decl = None
         self.defn = None
         self._regy = None
@@ -237,7 +238,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._visited_ret_types = set()
 
     def visit_end(self):
-        self.defn += gen_registry(self._regy)
+        self.defn += gen_registry(self._regy, self._prefix)
         self._regy = None
         self._visited_ret_types = None
 
@@ -289,7 +290,7 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
                prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenCommandVisitor()
+vis = QAPISchemaGenCommandVisitor(prefix)
 schema.visit(vis)
 genc.add(vis.defn)
 genh.add(vis.decl)
scripts/qapi-event.py +7 -8
index bd7a9be3dc..3f98e2491a 100644
@@ -58,7 +58,7 @@ def gen_param_var(typ):
     return ret
 
 
-def gen_event_send(name, arg_type, boxed):
+def gen_event_send(name, arg_type, boxed, event_enum_name):
     # FIXME: Our declaration of local variables (and of 'errp' in the
     # parameter list) can collide with exploded members of the event's
     # data type passed in as parameters.  If this collision ever hits in
@@ -149,7 +149,8 @@ out:
 
 
 class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, prefix):
+        self._enum_name = c_name(prefix + 'QAPIEvent', protect=False)
         self.decl = None
         self.defn = None
         self._event_names = None
@@ -160,13 +161,13 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self._event_names = []
 
     def visit_end(self):
-        self.decl += gen_enum(event_enum_name, self._event_names)
-        self.defn += gen_enum_lookup(event_enum_name, self._event_names)
+        self.decl += gen_enum(self._enum_name, self._event_names)
+        self.defn += gen_enum_lookup(self._enum_name, self._event_names)
         self._event_names = None
 
     def visit_event(self, name, info, arg_type, boxed):
         self.decl += gen_event_send_decl(name, arg_type, boxed)
-        self.defn += gen_event_send(name, arg_type, boxed)
+        self.defn += gen_event_send(name, arg_type, boxed, self._enum_name)
         self._event_names.append(name)
 
 
@@ -199,10 +200,8 @@ genh.add(mcgen('''
 ''',
                prefix=prefix))
 
-event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenEventVisitor()
+vis = QAPISchemaGenEventVisitor(prefix)
 schema.visit(vis)
 genc.add(vis.defn)
 genh.add(vis.decl)
scripts/qapi-introspect.py +4 -3
index 3d65690fe3..2418b80a82 100644
@@ -41,7 +41,8 @@ def to_c_string(string):
 
 
 class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
-    def __init__(self, unmask):
+    def __init__(self, prefix, unmask):
+        self._prefix = prefix
         self._unmask = unmask
         self.defn = None
         self.decl = None
@@ -65,7 +66,7 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
         # generate C
         # TODO can generate awfully long lines
         jsons.extend(self._jsons)
-        name = c_name(prefix, protect=False) + 'qmp_schema_json'
+        name = c_name(self._prefix, protect=False) + 'qmp_schema_json'
         self.decl = mcgen('''
 extern const char %(c_name)s[];
 ''',
@@ -192,7 +193,7 @@ genc.add(mcgen('''
                prefix=prefix))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenIntrospectVisitor(opt_unmask)
+vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
 schema.visit(vis)
 genc.add(vis.defn)
 genh.add(vis.decl)
scripts/qapi-types.py +9 -8
index c0ac879beb..5ff2bfcf41 100644
@@ -168,7 +168,8 @@ void qapi_free_%(c_name)s(%(c_name)s *obj)
 
 
 class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, opt_builtins):
+        self._opt_builtins = opt_builtins
         self.decl = None
         self.defn = None
         self._fwdecl = None
@@ -187,7 +188,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self._fwdecl = None
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
-        # simply guard them.  See also do_builtins (command line
+        # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_TYPES_BUILTIN')
         self.decl = self._btin + self.decl
@@ -202,7 +203,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         # TODO use something cleaner than existence of info
         if not info:
             self._btin += gen_enum(name, values, prefix)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_enum_lookup(name, values, prefix)
         else:
             self._fwdecl += gen_enum(name, values, prefix)
@@ -213,7 +214,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
             self._btin += gen_fwd_object_or_array(name)
             self._btin += gen_array(name, element_type)
             self._btin += gen_type_cleanup_decl(name)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_type_cleanup(name)
         else:
             self._fwdecl += gen_fwd_object_or_array(name)
@@ -241,16 +242,16 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
 # If you link code generated from multiple schemata, you want only one
 # instance of the code for built-in types.  Generate it only when
-# do_builtins, enabled by command line option -b.  See also
+# opt_builtins, enabled by command line option -b.  See also
 # QAPISchemaGenTypeVisitor.visit_end().
-do_builtins = False
+opt_builtins = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
     parse_command_line('b', ['builtins'])
 
 for o, a in opts:
     if o in ('-b', '--builtins'):
-        do_builtins = True
+        opt_builtins = True
 
 blurb = '''
  * Schema-defined QAPI types
@@ -272,7 +273,7 @@ genh.add(mcgen('''
 '''))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenTypeVisitor()
+vis = QAPISchemaGenTypeVisitor(opt_builtins)
 schema.visit(vis)
 genc.add(vis.defn)
 genh.add(vis.decl)
scripts/qapi-visit.py +9 -8
index 888c686303..0a1eff7736 100644
@@ -264,7 +264,8 @@ out:
 
 
 class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
-    def __init__(self):
+    def __init__(self, opt_builtins):
+        self._opt_builtins = opt_builtins
         self.decl = None
         self.defn = None
         self._btin = None
@@ -277,7 +278,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
     def visit_end(self):
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
-        # simply guard them.  See also do_builtins (command line
+        # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_VISIT_BUILTIN')
         self.decl = self._btin + self.decl
@@ -288,7 +289,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         # TODO use something cleaner than existence of info
         if not info:
             self._btin += gen_visit_decl(name, scalar=True)
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += gen_visit_enum(name)
         else:
             self.decl += gen_visit_decl(name, scalar=True)
@@ -299,7 +300,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         defn = gen_visit_list(name, element_type)
         if isinstance(element_type, QAPISchemaBuiltinType):
             self._btin += decl
-            if do_builtins:
+            if self._opt_builtins:
                 self.defn += defn
         else:
             self.decl += decl
@@ -324,16 +325,16 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
 
 # If you link code generated from multiple schemata, you want only one
 # instance of the code for built-in types.  Generate it only when
-# do_builtins, enabled by command line option -b.  See also
+# opt_builtins, enabled by command line option -b.  See also
 # QAPISchemaGenVisitVisitor.visit_end().
-do_builtins = False
+opt_builtins = False
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = \
     parse_command_line('b', ['builtins'])
 
 for o, a in opts:
     if o in ('-b', '--builtins'):
-        do_builtins = True
+        opt_builtins = True
 
 blurb = '''
  * Schema-defined QAPI visitors
@@ -359,7 +360,7 @@ genh.add(mcgen('''
                prefix=prefix))
 
 schema = QAPISchema(input_file)
-vis = QAPISchemaGenVisitVisitor()
+vis = QAPISchemaGenVisitVisitor(opt_builtins)
 schema.visit(vis)
 genc.add(vis.defn)
 genh.add(vis.decl)
Signature · 1 line
2.13.6
#17 lore raw

PATCH v2 7/29 qapi: Turn generators into modules

In reply to Markus Armbruster

The next commit will introduce a common driver program for all generators. The generators need to be modules for that. qapi2texi.py already is. Make the other generators follow suit.

The changes are actually trivial. Obvious in the diffs once you view them with whitespace changes ignored.

---
 scripts/qapi-commands.py   | 43 ++++++++++++++++++--------------
 scripts/qapi-event.py      | 43 ++++++++++++++++++--------------
 scripts/qapi-introspect.py | 54 ++++++++++++++++++++++------------------
 scripts/qapi-types.py      | 56 ++++++++++++++++++++++-------------------
 scripts/qapi-visit.py      | 62 +++++++++++++++++++++++++---------------------
 5 files changed, 143 insertions(+), 115 deletions(-)
scripts/qapi-commands.py +24 -19
index e97e16e828..8584cb5873 100644
@@ -255,16 +255,17 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._regy += gen_register_command(name, success_response)
 
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
+def main(argv):
+    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
 
-blurb = '''
+    blurb = '''
  * Schema-defined QAPI/QMP commands
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.add(mcgen('''
+    genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -279,23 +280,27 @@ genc.add(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-               prefix=prefix))
+                   prefix=prefix))
 
-genh.add(mcgen('''
+    genh.add(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-               prefix=prefix, c_prefix=c_name(prefix, protect=False)))
-
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenCommandVisitor(prefix)
-schema.visit(vis)
-genc.add(vis.defn)
-genh.add(vis.decl)
-
-if do_c:
-    genc.write(output_dir, prefix + 'qmp-marshal.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qmp-commands.h')
+                   prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenCommandVisitor(prefix)
+    schema.visit(vis)
+    genc.add(vis.defn)
+    genh.add(vis.decl)
+
+    if do_c:
+        genc.write(output_dir, prefix + 'qmp-marshal.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qmp-commands.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi-event.py +24 -19

62 lines not on this page · raw message

scripts/qapi-introspect.py +30 -24

68 lines not on this page · raw message

scripts/qapi-types.py +31 -25

76 lines not on this page · raw message

scripts/qapi-visit.py +34 -28

83 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v2 7/29
#18 lore raw

Re: [Qemu-devel] [PATCH v2 07/29] qapi: Turn generators into modules

Quoting Markus Armbruster (2018-02-11 03:35:45)

Quoted 10 lines

The next commit will introduce a common driver program for all generators. The generators need to be modules for that. qapi2texi.py already is. Make the other generators follow suit.

The changes are actually trivial. Obvious in the diffs once you view them with whitespace changes ignored.

Quoted 390 lines
---
 scripts/qapi-commands.py   | 43 ++++++++++++++++++--------------
 scripts/qapi-event.py      | 43 ++++++++++++++++++--------------
 scripts/qapi-introspect.py | 54 ++++++++++++++++++++++------------------
 scripts/qapi-types.py      | 56 ++++++++++++++++++++++-------------------
 scripts/qapi-visit.py      | 62 +++++++++++++++++++++++++---------------------
 5 files changed, 143 insertions(+), 115 deletions(-)
scripts/qapi-commands.py +24 -19
index e97e16e828..8584cb5873 100644
@@ -255,16 +255,17 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._regy += gen_register_command(name, success_response)
 
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
+def main(argv):
+    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
 
-blurb = '''
+    blurb = '''
  * Schema-defined QAPI/QMP commands
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.add(mcgen('''
+    genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -279,23 +280,27 @@ genc.add(mcgen('''
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-               prefix=prefix))
+                   prefix=prefix))
 
-genh.add(mcgen('''
+    genh.add(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-               prefix=prefix, c_prefix=c_name(prefix, protect=False)))
-
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenCommandVisitor(prefix)
-schema.visit(vis)
-genc.add(vis.defn)
-genh.add(vis.decl)
-
-if do_c:
-    genc.write(output_dir, prefix + 'qmp-marshal.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qmp-commands.h')
+                   prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenCommandVisitor(prefix)
+    schema.visit(vis)
+    genc.add(vis.defn)
+    genh.add(vis.decl)
+
+    if do_c:
+        genc.write(output_dir, prefix + 'qmp-marshal.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qmp-commands.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi-event.py +24 -19
index 3f98e2491a..e7e07f0055 100644
@@ -171,16 +171,17 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self._event_names.append(name)
 
 
-(input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
+def main(argv):
+    (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
 
-blurb = '''
+    blurb = '''
  * Schema-defined QAPI/QMP events
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.add(mcgen('''
+    genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "%(prefix)sqapi-event.h"
@@ -191,22 +192,26 @@ genc.add(mcgen('''
 #include "qapi/qmp-event.h"
 
 ''',
-               prefix=prefix))
+                   prefix=prefix))
 
-genh.add(mcgen('''
+    genh.add(mcgen('''
 #include "qapi/util.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-               prefix=prefix))
-
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenEventVisitor(prefix)
-schema.visit(vis)
-genc.add(vis.defn)
-genh.add(vis.decl)
-
-if do_c:
-    genc.write(output_dir, prefix + 'qapi-event.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qapi-event.h')
+                   prefix=prefix))
+
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenEventVisitor(prefix)
+    schema.visit(vis)
+    genc.add(vis.defn)
+    genh.add(vis.decl)
+
+    if do_c:
+        genc.write(output_dir, prefix + 'qapi-event.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qapi-event.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi-introspect.py +30 -24
index 2418b80a82..b098b95053 100644
@@ -167,38 +167,44 @@ const char %(c_name)s[] = %(c_string)s;
         arg_type = arg_type or self._schema.the_empty_object_type
         self._gen_json(name, 'event', {'arg-type': self._use_type(arg_type)})
 
-# Debugging aid: unmask QAPI schema's type names
-# We normally mask them, because they're not QMP wire ABI
-opt_unmask = False
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = \
-    parse_command_line('u', ['unmask-non-abi-names'])
+def main(argv):
+    # Debugging aid: unmask QAPI schema's type names
+    # We normally mask them, because they're not QMP wire ABI
+    opt_unmask = False
 
-for o, a in opts:
-    if o in ('-u', '--unmask-non-abi-names'):
-        opt_unmask = True
+    (input_file, output_dir, do_c, do_h, prefix, opts) = \
+        parse_command_line('u', ['unmask-non-abi-names'])
 
-blurb = '''
+    for o, a in opts:
+        if o in ('-u', '--unmask-non-abi-names'):
+            opt_unmask = True
+
+    blurb = '''
  * QAPI/QMP schema introspection
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.add(mcgen('''
+    genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "%(prefix)sqmp-introspect.h"
 
 ''',
-               prefix=prefix))
-
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
-schema.visit(vis)
-genc.add(vis.defn)
-genh.add(vis.decl)
-
-if do_c:
-    genc.write(output_dir, prefix + 'qmp-introspect.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qmp-introspect.h')
+                   prefix=prefix))
+
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
+    schema.visit(vis)
+    genc.add(vis.defn)
+    genh.add(vis.decl)
+
+    if do_c:
+        genc.write(output_dir, prefix + 'qmp-introspect.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qmp-introspect.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi-types.py +31 -25
index 5ff2bfcf41..10955d1c01 100644
@@ -240,45 +240,51 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.decl += gen_object(name, None, [variants.tag_member], variants)
         self._gen_type_cleanup(name)
 
-# If you link code generated from multiple schemata, you want only one
-# instance of the code for built-in types.  Generate it only when
-# opt_builtins, enabled by command line option -b.  See also
-# QAPISchemaGenTypeVisitor.visit_end().
-opt_builtins = False
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = \
-    parse_command_line('b', ['builtins'])
+def main(argv):
+    # If you link code generated from multiple schemata, you want only one
+    # instance of the code for built-in types.  Generate it only when
+    # opt_builtins, enabled by command line option -b.  See also
+    # QAPISchemaGenTypeVisitor.visit_end().
+    opt_builtins = False
 
-for o, a in opts:
-    if o in ('-b', '--builtins'):
-        opt_builtins = True
+    (input_file, output_dir, do_c, do_h, prefix, opts) = \
+        parse_command_line('b', ['builtins'])
 
-blurb = '''
+    for o, a in opts:
+        if o in ('-b', '--builtins'):
+            opt_builtins = True
+
+    blurb = '''
  * Schema-defined QAPI types
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.add(mcgen('''
+    genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
 #include "%(prefix)sqapi-types.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-               prefix=prefix))
+                   prefix=prefix))
 
-genh.add(mcgen('''
+    genh.add(mcgen('''
 #include "qapi/util.h"
 '''))
 
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenTypeVisitor(opt_builtins)
-schema.visit(vis)
-genc.add(vis.defn)
-genh.add(vis.decl)
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenTypeVisitor(opt_builtins)
+    schema.visit(vis)
+    genc.add(vis.defn)
+    genh.add(vis.decl)
 
-if do_c:
-    genc.write(output_dir, prefix + 'qapi-types.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qapi-types.h')
+    if do_c:
+        genc.write(output_dir, prefix + 'qapi-types.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qapi-types.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi-visit.py +34 -28
index 0a1eff7736..6d829c4d1d 100644
@@ -323,49 +323,55 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         self.decl += gen_visit_decl(name)
         self.defn += gen_visit_alternate(name, variants)
 
-# If you link code generated from multiple schemata, you want only one
-# instance of the code for built-in types.  Generate it only when
-# opt_builtins, enabled by command line option -b.  See also
-# QAPISchemaGenVisitVisitor.visit_end().
-opt_builtins = False
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = \
-    parse_command_line('b', ['builtins'])
+def main(argv):
+    # If you link code generated from multiple schemata, you want only one
+    # instance of the code for built-in types.  Generate it only when
+    # opt_builtins, enabled by command line option -b.  See also
+    # QAPISchemaGenVisitVisitor.visit_end().
+    opt_builtins = False
 
-for o, a in opts:
-    if o in ('-b', '--builtins'):
-        opt_builtins = True
+    (input_file, output_dir, do_c, do_h, prefix, opts) = \
+        parse_command_line('b', ['builtins'])
 
-blurb = '''
+    for o, a in opts:
+        if o in ('-b', '--builtins'):
+            opt_builtins = True
+
+    blurb = '''
  * Schema-defined QAPI visitors
 '''
 
-genc = QAPIGenC(blurb, __doc__)
-genh = QAPIGenH(blurb, __doc__)
+    genc = QAPIGenC(blurb, __doc__)
+    genh = QAPIGenH(blurb, __doc__)
 
-genc.add(mcgen('''
+    genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-               prefix=prefix))
+                   prefix=prefix))
 
-genh.add(mcgen('''
+    genh.add(mcgen('''
 #include "qapi/visitor.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-               prefix=prefix))
-
-schema = QAPISchema(input_file)
-vis = QAPISchemaGenVisitVisitor(opt_builtins)
-schema.visit(vis)
-genc.add(vis.defn)
-genh.add(vis.decl)
-
-if do_c:
-    genc.write(output_dir, prefix + 'qapi-visit.c')
-if do_h:
-    genh.write(output_dir, prefix + 'qapi-visit.h')
+                   prefix=prefix))
+
+    schema = QAPISchema(input_file)
+    vis = QAPISchemaGenVisitVisitor(opt_builtins)
+    schema.visit(vis)
+    genc.add(vis.defn)
+    genh.add(vis.decl)
+
+    if do_c:
+        genc.write(output_dir, prefix + 'qapi-visit.c')
+    if do_h:
+        genh.write(output_dir, prefix + 'qapi-visit.h')
+
+
+if __name__ == '__main__':
+    main(sys.argv)
Signature · 1 line
2.13.6
#19 lore raw

Re: [Qemu-devel] [PATCH v2 07/29] qapi: Turn generators into modules

In reply to Markus Armbruster
Quoted 10 lines · Markus Armbruster

The next commit will introduce a common driver program for all generators. The generators need to be modules for that. qapi2texi.py already is. Make the other generators follow suit.

The changes are actually trivial. Obvious in the diffs once you view them with whitespace changes ignored.

Quoted 7 lines
---
  scripts/qapi-commands.py   | 43 ++++++++++++++++++--------------
  scripts/qapi-event.py      | 43 ++++++++++++++++++--------------
  scripts/qapi-introspect.py | 54 ++++++++++++++++++++++------------------
  scripts/qapi-types.py      | 56 ++++++++++++++++++++++-------------------
  scripts/qapi-visit.py      | 62 +++++++++++++++++++++++++---------------------
  5 files changed, 143 insertions(+), 115 deletions(-)

Urgh. One annoyance of this patch is that if you switch to another branch without the patch, leftover .pyc files in the source tree can cause weird errors until you 'rm -f scripts/qapi/*.pyc':

$ make -j3
  GEN     config-host.h
  GEN     qmp-commands.h
  GEN     qapi-types.h
Traceback (most recent call last):
  File "/home/eblake/qemu/scripts/qapi-commands.py", line 225, in <module>
    class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
NameError: name 'QAPISchemaVisitor' is not defined
Traceback (most recent call last):
  File "/home/eblake/qemu/scripts/qapi-types.py", line 168, in <module>
    class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
NameError: name 'QAPISchemaVisitor' is not defined
make: *** [Makefile:522: qapi-types.h] Error 1
make: *** Waiting for unfinished jobs....
make: *** [Makefile:537: qmp-commands.h] Error 1

Not a show-stopper, but annoying when toggling between branches.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#20 lore raw

Re: [Qemu-devel] [PATCH v2 07/29] qapi: Turn generators into modules

Quoted 19 lines · Eric Blake
Quoted 10 lines · Markus Armbruster

The next commit will introduce a common driver program for all generators. The generators need to be modules for that. qapi2texi.py already is. Make the other generators follow suit.

The changes are actually trivial. Obvious in the diffs once you view them with whitespace changes ignored.

Quoted 7 lines
---
  scripts/qapi-commands.py   | 43 ++++++++++++++++++--------------
  scripts/qapi-event.py      | 43 ++++++++++++++++++--------------
  scripts/qapi-introspect.py | 54 ++++++++++++++++++++++------------------
  scripts/qapi-types.py      | 56 ++++++++++++++++++++++-------------------
  scripts/qapi-visit.py      | 62 +++++++++++++++++++++++++---------------------
  5 files changed, 143 insertions(+), 115 deletions(-)
Quoted 3 lines

Urgh. One annoyance of this patch is that if you switch to another branch without the patch, leftover .pyc files in the source tree can cause weird errors until you 'rm -f scripts/qapi/*.pyc':

Pehaps make our build system set PYTHONDONTWRITEBYTECODE=1 env variable

https://docs.python.org/3/using/cmdline.html#envvar-PYTHONDONTWRITEBYTECODE

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#21 lore raw

Re: [Qemu-devel] [PATCH v2 07/29] qapi: Turn generators into modules

Quoted 20 lines · Daniel P. Berrangé
Quoted 19 lines · Eric Blake
Quoted 17 lines · Markus Armbruster

The next commit will introduce a common driver program for all generators. The generators need to be modules for that. qapi2texi.py already is. Make the other generators follow suit.

The changes are actually trivial. Obvious in the diffs once you view them with whitespace changes ignored.

---
  scripts/qapi-commands.py   | 43 ++++++++++++++++++--------------
  scripts/qapi-event.py      | 43 ++++++++++++++++++--------------
  scripts/qapi-introspect.py | 54 ++++++++++++++++++++++------------------
  scripts/qapi-types.py      | 56 ++++++++++++++++++++++-------------------
  scripts/qapi-visit.py      | 62 +++++++++++++++++++++++++---------------------
  5 files changed, 143 insertions(+), 115 deletions(-)
Quoted 7 lines
Quoted 3 lines

Urgh. One annoyance of this patch is that if you switch to another branch without the patch, leftover .pyc files in the source tree can cause weird errors until you 'rm -f scripts/qapi/*.pyc':

Pehaps make our build system set PYTHONDONTWRITEBYTECODE=1 env variable

https://docs.python.org/3/using/cmdline.html#envvar-PYTHONDONTWRITEBYTECODE

We already have configure put PYTHON=python -B into config-host.mak.

I still manage to litter my tree with .pyc files...

#22 lore raw

PATCH v2 8/29 qapi-gen: New common driver for code and doc generators

In reply to Markus Armbruster

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. Similar for qga/qapi-schema.json. This is silly. Replace the six programs by a single program that spits out all eleven files.

The programs become modules in new Python package qapi, along with the helper library. This requires moving them to scripts/qapi/.

---
 .gitignore                                         |  2 +
 Makefile                                           | 86 +++++++++----------
 docs/devel/qapi-code-gen.txt                       | 97 ++++++++++------------
 monitor.c                                          |  2 +-
 qapi-schema.json                                   |  2 +-
 scripts/qapi-gen.py                                | 41 +++++++++
 scripts/qapi/__init__.py                           |  0
 scripts/{qapi-commands.py => qapi/commands.py}     | 23 ++---
 scripts/{qapi.py => qapi/common.py}                | 18 +---
 scripts/{qapi2texi.py => qapi/doc.py}              | 29 ++-----
 scripts/{qapi-event.py => qapi/events.py}          | 23 ++---
 scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++-----
 scripts/{qapi-types.py => qapi/types.py}           | 34 ++------
 scripts/{qapi-visit.py => qapi/visit.py}           | 34 ++------
 tests/Makefile.include                             | 56 ++++++-------
 tests/qapi-schema/test-qapi.py                     |  4 +-
 16 files changed, 193 insertions(+), 290 deletions(-)
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (94%)
 rename scripts/{qapi.py => qapi/common.py} (99%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
 mode change 100755 => 100644
 rename scripts/{qapi-event.py => qapi/events.py} (92%)
 rename scripts/{qapi-introspect.py => qapi/introspect.py} (90%)
 rename scripts/{qapi-types.py => qapi/types.py} (90%)
 rename scripts/{qapi-visit.py => qapi/visit.py} (92%)
.gitignore +2 -0
index 704b22285d..2f9a92f6cc 100644
@@ -28,9 +28,11 @@
 /linux-headers/asm
 /qga/qapi-generated
 /qapi-generated
+/qapi-gen-timestamp
 /qapi-types.[ch]
 /qapi-visit.[ch]
 /qapi-event.[ch]
+/qapi-doc.texi
 /qmp-commands.h
 /qmp-introspect.[ch]
 /qmp-marshal.c
Makefile +37 -49

118 lines not on this page · raw message

docs/devel/qapi-code-gen.txt +45 -52

184 lines not on this page · raw message

monitor.c +1 -1
index f4992505b1..df670f3e15 100644
@@ -951,7 +951,7 @@ EventInfoList *qmp_query_events(Error **errp)
  * visit_type_SchemaInfoList() into a SchemaInfoList, then marshal it
  * to QObject with generated output marshallers, every time.  Instead,
  * we do it in test-qobject-input-visitor.c, just to make sure
- * qapi-introspect.py's output actually conforms to the schema.
+ * qapi-gen.py's output actually conforms to the schema.
  */
 static void qmp_query_qmp_schema(QDict *qdict, QObject **ret_data,
                                  Error **errp)
qapi-schema.json +1 -1
index 5c06745c79..0d651c8399 100644
@@ -72,7 +72,7 @@
         'q_obj_CpuInfo-base'    # CPU, visible through query-cpu
     ] } }
 
-# Documentation generated with qapi2texi.py is in source order, with
+# Documentation generated with qapi-gen.py is in source order, with
 # included sub-schemas inserted at the first include directive
 # (subsequent include directives have no effect).  To get a sane and
 # stable order, it's best to include each sub-schema just once, or
scripts/qapi-gen.py new file +41 -0

41 lines not on this page · raw message

scripts/qapi/__init__.py new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +5 -18
similarity index 94%
rename from scripts/qapi-commands.py
rename to scripts/qapi/commands.py
index 8584cb5873..a744611d58 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
@@ -255,13 +255,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._regy += gen_register_command(name, success_response)
 
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP commands
-'''
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -290,17 +285,9 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
                    prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenCommandVisitor(prefix)
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qmp-marshal.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qmp-commands.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qmp-marshal.c')
+    genh.write(output_dir, prefix + 'qmp-commands.h')
scripts/qapi/common.py renamed from scripts/qapi.py +4 -14
similarity index 99%
rename from scripts/qapi.py
rename to scripts/qapi/common.py
index 9693fd1851..868ec25deb 100644
@@ -1932,17 +1932,15 @@ def parse_command_line(extra_options='', extra_long_options=[]):
 
     try:
         opts, args = getopt.gnu_getopt(sys.argv[1:],
-                                       'chp:o:' + extra_options,
-                                       ['source', 'header', 'prefix=',
-                                        'output-dir='] + extra_long_options)
+                                       'p:o:' + extra_options,
+                                       ['prefix=', 'output-dir=']
+                                       + extra_long_options)
     except getopt.GetoptError as err:
         print("%s: %s" % (sys.argv[0], str(err)), file=sys.stderr)
         sys.exit(1)
 
     output_dir = ''
     prefix = ''
-    do_c = False
-    do_h = False
     extra_opts = []
 
     for oa in opts:
@@ -1956,23 +1954,15 @@ def parse_command_line(extra_options='', extra_long_options=[]):
             prefix = a
         elif o in ('-o', '--output-dir'):
             output_dir = a + '/'
-        elif o in ('-c', '--source'):
-            do_c = True
-        elif o in ('-h', '--header'):
-            do_h = True
         else:
             extra_opts.append(oa)
 
-    if not do_c and not do_h:
-        do_c = True
-        do_h = True
-
     if len(args) != 1:
         print("%s: need exactly one argument" % sys.argv[0], file=sys.stderr)
         sys.exit(1)
     fname = args[0]
 
-    return (fname, output_dir, do_c, do_h, prefix, extra_opts)
+    return (fname, output_dir, prefix, extra_opts)
 
 
 #
scripts/qapi/doc.py renamed from scripts/qapi2texi.py +8 -21
old mode 100755
new mode 100644
similarity index 92%
rename from scripts/qapi2texi.py
rename to scripts/qapi/doc.py
index 8a604d86a6..cc4d5a43fb
@@ -4,11 +4,10 @@
 # This work is licensed under the terms of the GNU LGPL, version 2+.
 # See the COPYING file in the top-level directory.
 """This script produces the documentation of a qapi schema in texinfo format"""
+
 from __future__ import print_function
 import re
-import sys
-
-import qapi
+import qapi.common
 
 MSG_FMT = """
 @deftypefn {type} {{}} {name}
@@ -197,7 +196,7 @@ def texi_entity(doc, what, base=None, variants=None,
             + texi_sections(doc))
 
 
-class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
+class QAPISchemaGenDocVisitor(qapi.common.QAPISchemaVisitor):
     def __init__(self):
         self.out = None
         self.cur_doc = None
@@ -272,20 +271,8 @@ def texi_schema(schema):
     return gen.out
 
 
-def main(argv):
-    """Takes schema argument, prints result to stdout"""
-    if len(argv) != 2:
-        print("%s: need exactly 1 argument: SCHEMA" % argv[0], file=sys.stderr)
-        sys.exit(1)
-
-    schema = qapi.QAPISchema(argv[1])
-    if not qapi.doc_required:
-        print("%s: need pragma 'doc-required' "
-               "to generate documentation" % argv[0], file=sys.stderr)
-        sys.exit(1)
-    print('@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n')
-    print(texi_schema(schema), end='')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+def gen_doc(schema, output_dir, prefix):
+    if qapi.common.doc_required:
+        gen = qapi.common.QAPIGenDoc()
+        gen.add(texi_schema(schema))
+        gen.write(output_dir, prefix + 'qapi-doc.texi')
scripts/qapi/events.py renamed from scripts/qapi-event.py +5 -18
similarity index 92%
rename from scripts/qapi-event.py
rename to scripts/qapi/events.py
index e7e07f0055..b7dc82004f 100644
@@ -12,7 +12,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def build_event_send_proto(name, arg_type, boxed):
@@ -171,13 +171,8 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self._event_names.append(name)
 
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP events
-'''
-
+def gen_events(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP events'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -201,17 +196,9 @@ def main(argv):
 ''',
                    prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenEventVisitor(prefix)
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-event.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-event.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-event.c')
+    genh.write(output_dir, prefix + 'qapi-event.h')
scripts/qapi/introspect.py renamed from scripts/qapi-introspect.py +5 -27
similarity index 90%
rename from scripts/qapi-introspect.py
rename to scripts/qapi/introspect.py
index b098b95053..1e4f065164 100644
@@ -10,7 +10,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # Caveman's json.dumps() replacement (we're stuck at Python 2.4)
@@ -168,22 +168,8 @@ const char %(c_name)s[] = %(c_string)s;
         self._gen_json(name, 'event', {'arg-type': self._use_type(arg_type)})
 
 
-def main(argv):
-    # Debugging aid: unmask QAPI schema's type names
-    # We normally mask them, because they're not QMP wire ABI
-    opt_unmask = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('u', ['unmask-non-abi-names'])
-
-    for o, a in opts:
-        if o in ('-u', '--unmask-non-abi-names'):
-            opt_unmask = True
-
-    blurb = '''
- * QAPI/QMP schema introspection
-'''
-
+def gen_introspect(schema, output_dir, prefix, opt_unmask):
+    blurb = ' * QAPI/QMP schema introspection'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -194,17 +180,9 @@ def main(argv):
 ''',
                    prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qmp-introspect.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qmp-introspect.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qmp-introspect.c')
+    genh.write(output_dir, prefix + 'qmp-introspect.h')
scripts/qapi/types.py renamed from scripts/qapi-types.py +5 -29
similarity index 90%
rename from scripts/qapi-types.py
rename to scripts/qapi/types.py
index 10955d1c01..aa3c01e750 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # variants must be emitted before their container; track what has already
@@ -241,24 +241,8 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self._gen_type_cleanup(name)
 
 
-def main(argv):
-    # If you link code generated from multiple schemata, you want only one
-    # instance of the code for built-in types.  Generate it only when
-    # opt_builtins, enabled by command line option -b.  See also
-    # QAPISchemaGenTypeVisitor.visit_end().
-    opt_builtins = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('b', ['builtins'])
-
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-
-    blurb = '''
- * Schema-defined QAPI types
-'''
-
+def gen_types(schema, output_dir, prefix, opt_builtins):
+    blurb = ' * Schema-defined QAPI types'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -274,17 +258,9 @@ def main(argv):
 #include "qapi/util.h"
 '''))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenTypeVisitor(opt_builtins)
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-types.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-types.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-types.c')
+    genh.write(output_dir, prefix + 'qapi-types.h')
scripts/qapi/visit.py renamed from scripts/qapi-visit.py +5 -29
similarity index 92%
rename from scripts/qapi-visit.py
rename to scripts/qapi/visit.py
index 6d829c4d1d..f966e7975f 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def gen_visit_decl(name, scalar=False):
@@ -324,24 +324,8 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         self.defn += gen_visit_alternate(name, variants)
 
 
-def main(argv):
-    # If you link code generated from multiple schemata, you want only one
-    # instance of the code for built-in types.  Generate it only when
-    # opt_builtins, enabled by command line option -b.  See also
-    # QAPISchemaGenVisitVisitor.visit_end().
-    opt_builtins = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('b', ['builtins'])
-
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-
-    blurb = '''
- * Schema-defined QAPI visitors
-'''
-
+def gen_visit(schema, output_dir, prefix, opt_builtins):
+    blurb = ' * Schema-defined QAPI visitors'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -361,17 +345,9 @@ def main(argv):
 ''',
                    prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenVisitVisitor(opt_builtins)
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-visit.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-visit.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-visit.c')
+    genh.write(output_dir, prefix + 'qapi-visit.h')
tests/Makefile.include +28 -28

75 lines not on this page · raw message

tests/qapi-schema/test-qapi.py +1 -3
index ac43d3458e..bb1b6dd297 100644
@@ -11,10 +11,8 @@
 #
 
 from __future__ import print_function
-from qapi import *
-from pprint import pprint
-import os
 import sys
+from qapi.common import QAPISchema, QAPISchemaVisitor
 
 
 class QAPISchemaTestVisitor(QAPISchemaVisitor):
Signature · 1 line
2.13.6
Reviewed-by on v2 8/29
#23 lore raw

Re: [Qemu-devel] [PATCH v2 08/29] qapi-gen: New common driver for code and doc generators

Quoted 28 lines · Markus Armbruster

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. Similar for qga/qapi-schema.json. This is silly. Replace the six programs by a single program that spits out all eleven files.

The programs become modules in new Python package qapi, along with the helper library. This requires moving them to scripts/qapi/.

---
  .gitignore                                         |  2 +
  Makefile                                           | 86 +++++++++----------
  docs/devel/qapi-code-gen.txt                       | 97 ++++++++++------------
  monitor.c                                          |  2 +-
  qapi-schema.json                                   |  2 +-
  scripts/qapi-gen.py                                | 41 +++++++++
  scripts/qapi/__init__.py                           |  0
  scripts/{qapi-commands.py => qapi/commands.py}     | 23 ++---
  scripts/{qapi.py => qapi/common.py}                | 18 +---
  scripts/{qapi2texi.py => qapi/doc.py}              | 29 ++-----
  scripts/{qapi-event.py => qapi/events.py}          | 23 ++---
  scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++-----
  scripts/{qapi-types.py => qapi/types.py}           | 34 ++------
  scripts/{qapi-visit.py => qapi/visit.py}           | 34 ++------
  tests/Makefile.include                             | 56 ++++++-------
  tests/qapi-schema/test-qapi.py                     |  4 +-
  16 files changed, 193 insertions(+), 290 deletions(-)
Quoted 6 lines

create mode 100755 scripts/qapi-gen.py
create mode 100644 scripts/qapi/__init__.py rename scripts/{qapi-commands.py => qapi/commands.py} (94%) rename scripts/{qapi.py => qapi/common.py} (99%) rename scripts/{qapi2texi.py => qapi/doc.py} (92%) mode change 100755 => 100644

Still forgot mention that the mode bit change was intentional, but not worth a respin for just that.

Quoted 4 lines

rename scripts/{qapi-event.py => qapi/events.py} (92%) rename scripts/{qapi-introspect.py => qapi/introspect.py} (90%) rename scripts/{qapi-types.py => qapi/types.py} (90%) rename scripts/{qapi-visit.py => qapi/visit.py} (92%)

Quoted 8 lines

+++ b/docs/devel/qapi-code-gen.txt

docs/devel/qapi-code-gen.txt +0 -2
-    $ python scripts/qapi-event.py --output-dir="qapi-generated"
-    --prefix="example-" example-schema.json
      $ cat qapi-generated/example-qapi-event.h
  [Uninteresting stuff omitted...]

docs/devel/qapi-code-gen.txt Example: · line 1302

Quoted 8 lines
docs/devel/qapi-code-gen.txt +2 -1
@@ -1302,23 +1296,22 @@ Example:
      }
 
      const char *const example_QAPIEvent_lookup[] = {
-        [EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
+
+[EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
          [EXAMPLE_QAPI_EVENT__MAX] = NULL,
      };

Looks like our generated code indentation has slightly regressed from what we would write by hand, but it's still okay for generated code (and the commit message in 5/29 did call that out)

Quoted 15 lines
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +3 -1
+++ b/scripts/qapi-gen.py
 
+++ b/scripts/qapi/commands.py
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
  See the COPYING file in the top-level directory.
  """
 
-from qapi import *
+from qapi.common import *
 
 
  def gen_command_decl(name, arg_type, boxed, ret_type):
@@ -255,13 +255,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
          self._regy += gen_register_command(name, success_response)
 

scripts/qapi/commands.py class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): · line 259

Quoted 10 lines
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +2 -7
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP commands
-'''
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'

We discussed whether to make the assignment to blurb be a one-liner in an earlier patch, but I'm also fine with the churn you have over the course of the series.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#24 lore raw

Re: [Qemu-devel] [PATCH v2 08/29] qapi-gen: New common driver for code and doc generators

Quoted 61 lines · Eric Blake
Quoted 28 lines · Markus Armbruster

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. Similar for qga/qapi-schema.json. This is silly. Replace the six programs by a single program that spits out all eleven files.

The programs become modules in new Python package qapi, along with the helper library. This requires moving them to scripts/qapi/.

---
  .gitignore                                         |  2 +
  Makefile                                           | 86 +++++++++----------
  docs/devel/qapi-code-gen.txt                       | 97 ++++++++++------------
  monitor.c                                          |  2 +-
  qapi-schema.json                                   |  2 +-
  scripts/qapi-gen.py                                | 41 +++++++++
  scripts/qapi/__init__.py                           |  0
  scripts/{qapi-commands.py => qapi/commands.py}     | 23 ++---
  scripts/{qapi.py => qapi/common.py}                | 18 +---
  scripts/{qapi2texi.py => qapi/doc.py}              | 29 ++-----
  scripts/{qapi-event.py => qapi/events.py}          | 23 ++---
  scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++-----
  scripts/{qapi-types.py => qapi/types.py}           | 34 ++------
  scripts/{qapi-visit.py => qapi/visit.py}           | 34 ++------
  tests/Makefile.include                             | 56 ++++++-------
  tests/qapi-schema/test-qapi.py                     |  4 +-
  16 files changed, 193 insertions(+), 290 deletions(-)
Quoted 6 lines

create mode 100755 scripts/qapi-gen.py
create mode 100644 scripts/qapi/__init__.py rename scripts/{qapi-commands.py => qapi/commands.py} (94%) rename scripts/{qapi.py => qapi/common.py} (99%) rename scripts/{qapi2texi.py => qapi/doc.py} (92%) mode change 100755 => 100644

Still forgot mention that the mode bit change was intentional, but not worth a respin for just that.

Quoted 4 lines

rename scripts/{qapi-event.py => qapi/events.py} (92%) rename scripts/{qapi-introspect.py => qapi/introspect.py} (90%) rename scripts/{qapi-types.py => qapi/types.py} (90%) rename scripts/{qapi-visit.py => qapi/visit.py} (92%)

Quoted 4 lines

+++ b/docs/devel/qapi-code-gen.txt

docs/devel/qapi-code-gen.txt +0 -2
-    $ python scripts/qapi-event.py --output-dir="qapi-generated"
-    --prefix="example-" example-schema.json

docs/devel/qapi-code-gen.txt Example: · line 1302

Quoted 10 lines
docs/devel/qapi-code-gen.txt +2 -1
      $ cat qapi-generated/example-qapi-event.h
  [Uninteresting stuff omitted...]
  @@ -1302,23 +1296,22 @@ Example:
      }
        const char *const example_QAPIEvent_lookup[] = {
-        [EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
+
+[EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
          [EXAMPLE_QAPI_EVENT__MAX] = NULL,
      };
Quoted 3 lines

Looks like our generated code indentation has slightly regressed from what we would write by hand, but it's still okay for generated code (and the commit message in 5/29 did call that out)

I'd prefer to have this tidied up.

Quoted 16 lines
Quoted 16 lines

+++ b/scripts/qapi-gen.py

+++ b/scripts/qapi/commands.py

scripts/qapi/commands.py renamed from scripts/qapi-commands.py +1 -4
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
  See the COPYING file in the top-level directory.
  """
  -from qapi import *
+from qapi.common import *
      def gen_command_decl(name, arg_type, boxed, ret_type):
@@ -255,13 +255,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
          self._regy += gen_register_command(name, success_response)
    -def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP commands
Quoted 8 lines

scripts/qapi/commands.py class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): · line 259

Quoted 4 lines
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +2 -2
-'''
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'

We discussed whether to make the assignment to blurb be a one-liner in an earlier patch, but I'm also fine with the churn you have over the course of the series.

I ran out of time, and decided to leave this one as is.

#25 lore raw

Re: [Qemu-devel] [PATCH v2 08/29] qapi-gen: New common driver for code and doc generators

Quoted 26 lines · Markus Armbruster
Quoted 24 lines · Eric Blake
Quoted 22 lines · Markus Armbruster

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. Similar for qga/qapi-schema.json. This is silly. Replace the six programs by a single program that spits out all eleven files.

The programs become modules in new Python package qapi, along with the helper library. This requires moving them to scripts/qapi/.

---

+++ b/docs/devel/qapi-code-gen.txt

   @@ -1302,23 +1296,22 @@ Example:
       }
         const char *const example_QAPIEvent_lookup[] = {
docs/devel/qapi-code-gen.txt +2 -1
-        [EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
+
+[EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
           [EXAMPLE_QAPI_EVENT__MAX] = NULL,
       };
Quoted 5 lines
Quoted 3 lines

Looks like our generated code indentation has slightly regressed from what we would write by hand, but it's still okay for generated code (and the commit message in 5/29 did call that out)

I'd prefer to have this tidied up.

Actually, we currently output with good indentation, but with different text altogether:

const QEnumLookup example_QAPIEvent_lookup = {
    .array = (const char *const[]) {
        [EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
    },
    .size = EXAMPLE_QAPI_EVENT__MAX
};

I'm fixing that up as part of prepping this series for PULL.

Quoted 8 lines
Quoted 6 lines

scripts/qapi/commands.py class QAPISchemaGenCommandVisitor(QAPISchemaVisitor): · line 259

Quoted 2 lines
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +2 -0
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'

We discussed whether to make the assignment to blurb be a one-liner in an earlier patch, but I'm also fine with the churn you have over the course of the series.

I ran out of time, and decided to leave this one as is.

It wasn't too difficult (although the ripple effects hit 3 other patches), so I did this as well.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v2 8/29
#26 lore raw

Re: [Qemu-devel] [PATCH v2 08/29] qapi-gen: New common driver for code and doc generators

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:46)

Quoted 10 lines

Whenever qapi-schema.json changes, we run six programs eleven times to update eleven files. Similar for qga/qapi-schema.json. This is silly. Replace the six programs by a single program that spits out all eleven files.

The programs become modules in new Python package qapi, along with the helper library. This requires moving them to scripts/qapi/.

Quoted 946 lines
---
 .gitignore                                         |  2 +
 Makefile                                           | 86 +++++++++----------
 docs/devel/qapi-code-gen.txt                       | 97 ++++++++++------------
 monitor.c                                          |  2 +-
 qapi-schema.json                                   |  2 +-
 scripts/qapi-gen.py                                | 41 +++++++++
 scripts/qapi/__init__.py                           |  0
 scripts/{qapi-commands.py => qapi/commands.py}     | 23 ++---
 scripts/{qapi.py => qapi/common.py}                | 18 +---
 scripts/{qapi2texi.py => qapi/doc.py}              | 29 ++-----
 scripts/{qapi-event.py => qapi/events.py}          | 23 ++---
 scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++-----
 scripts/{qapi-types.py => qapi/types.py}           | 34 ++------
 scripts/{qapi-visit.py => qapi/visit.py}           | 34 ++------
 tests/Makefile.include                             | 56 ++++++-------
 tests/qapi-schema/test-qapi.py                     |  4 +-
 16 files changed, 193 insertions(+), 290 deletions(-)
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (94%)
 rename scripts/{qapi.py => qapi/common.py} (99%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
 mode change 100755 => 100644
 rename scripts/{qapi-event.py => qapi/events.py} (92%)
 rename scripts/{qapi-introspect.py => qapi/introspect.py} (90%)
 rename scripts/{qapi-types.py => qapi/types.py} (90%)
 rename scripts/{qapi-visit.py => qapi/visit.py} (92%)
.gitignore +2 -0
index 704b22285d..2f9a92f6cc 100644
@@ -28,9 +28,11 @@
 /linux-headers/asm
 /qga/qapi-generated
 /qapi-generated
+/qapi-gen-timestamp
 /qapi-types.[ch]
 /qapi-visit.[ch]
 /qapi-event.[ch]
+/qapi-doc.texi
 /qmp-commands.h
 /qmp-introspect.[ch]
 /qmp-marshal.c
Makefile +37 -49
index 4ec7a3cb82..bd781c6aad 100644
@@ -94,6 +94,7 @@ GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
 GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
 GENERATED_FILES += qmp-introspect.h
 GENERATED_FILES += qmp-introspect.c
+GENERATED_FILES += qapi-doc.texi
 
 GENERATED_FILES += trace/generated-tcg-tracers.h
 
@@ -483,25 +484,26 @@ qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated
 qemu-keymap$(EXESUF): LIBS += $(XKBCOMMON_LIBS)
 qemu-keymap$(EXESUF): QEMU_CFLAGS += $(XKBCOMMON_CFLAGS)
 
-gen-out-type = $(subst .,-,$(suffix $@))
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py
 
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
-
-qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-types.py \
-               $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-               "GEN","$@")
-qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-visit.py \
-               $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-               "GEN","$@")
-qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
-       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-commands.py \
-               $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-               "GEN","$@")
+qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
+qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qapi-doc.texi: \
+qga/qapi-generated/qapi-gen-timestamp ;
+qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
+       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o qga/qapi-generated -p "qga-" $<, \
+               "GEN","$(@:%-timestamp=%)")
+       @>$@
 
 qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/block.json $(SRC_PATH)/qapi/block-core.json \
@@ -518,31 +520,18 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/transaction.json \
                $(SRC_PATH)/qapi/ui.json
 
-qapi-types.c qapi-types.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-types.py \
-               $(gen-out-type) -o "." -b $<, \
-               "GEN","$@")
-qapi-visit.c qapi-visit.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-visit.py \
-               $(gen-out-type) -o "." -b $<, \
-               "GEN","$@")
-qapi-event.c qapi-event.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-event.py $(qapi-py)
-       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-event.py \
-               $(gen-out-type) -o "." $<, \
-               "GEN","$@")
-qmp-commands.h qmp-marshal.c :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
-       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-commands.py \
-               $(gen-out-type) -o "." $<, \
-               "GEN","$@")
-qmp-introspect.h qmp-introspect.c :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
-       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-introspect.py \
-               $(gen-out-type) -o "." $<, \
-               "GEN","$@")
+qapi-types.c qapi-types.h \
+qapi-visit.c qapi-visit.h \
+qmp-commands.h qmp-marshal.c \
+qapi-event.c qapi-event.h \
+qmp-introspect.h qmp-introspect.c \
+qapi-doc.texi: \
+qapi-gen-timestamp ;
+qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
+       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o "." -b $<, \
+               "GEN","$(@:%-timestamp=%)")
+       @>$@
 
 QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qmp-commands.h)
 $(qga-obj-y): $(QGALIB_GEN)
@@ -602,6 +591,7 @@ clean:
        rm -f trace/generated-tracers-dtrace.dtrace*
        rm -f trace/generated-tracers-dtrace.h*
        rm -f $(foreach f,$(GENERATED_FILES),$(f) $(f)-timestamp)
+       rm -f qapi-gen-timestamp
        rm -rf qapi-generated
        rm -rf qga/qapi-generated
        for d in $(ALL_SUBDIRS); do \
@@ -810,13 +800,11 @@ qemu-monitor-info.texi: $(SRC_PATH)/hmp-commands-info.hx $(SRC_PATH)/scripts/hxt
 qemu-img-cmds.texi: $(SRC_PATH)/qemu-img-cmds.hx $(SRC_PATH)/scripts/hxtool
        $(call quiet-command,sh $(SRC_PATH)/scripts/hxtool -t < $< > $@,"GEN","$@")
 
-docs/interop/qemu-qmp-qapi.texi docs/interop/qemu-ga-qapi.texi: $(SRC_PATH)/scripts/qapi2texi.py $(qapi-py)
+docs/interop/qemu-qmp-qapi.texi: qapi-doc.texi
+       @cp -p $< $@
 
-docs/interop/qemu-qmp-qapi.texi: $(qapi-modules)
-       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@")
-
-docs/interop/qemu-ga-qapi.texi: $(SRC_PATH)/qga/qapi-schema.json
-       $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@")
+docs/interop/qemu-ga-qapi.texi: qga/qapi-generated/qga-qapi-doc.texi
+       @cp -p $< $@
 
 qemu.1: qemu-doc.texi qemu-options.texi qemu-monitor.texi qemu-monitor-info.texi
 qemu.1: qemu-option-trace.texi
docs/devel/qapi-code-gen.txt +45 -52
index 5900b39b91..1a1cbaea7b 100644
@@ -899,12 +899,13 @@ the names of built-in types.  Clients should examine member
 
 == Code generation ==
 
-Schemas are fed into five scripts to generate all the code/files that,
-paired with the core QAPI libraries, comprise everything required to
-take JSON commands read in by a Client JSON Protocol server, unmarshal
-the arguments into the underlying C types, call into the corresponding
-C function, map the response back to a Client JSON Protocol response
-to be returned to the user, and introspect the commands.
+The QAPI code generator qapi-gen.py generates code and documentation
+from the schema.  Together with the core QAPI libraries, this code
+provides everything required to take JSON commands read in by a Client
+JSON Protocol server, unmarshal the arguments into the underlying C
+types, call into the corresponding C function, map the response back
+to a Client JSON Protocol response to be returned to the user, and
+introspect the commands.
 
 As an example, we'll use the following schema, which describes a
 single complex user-defined type, along with command which takes a
@@ -922,18 +923,23 @@ qmp_my_command(); everything else is produced by the generator.
 
     { 'event': 'MY_EVENT' }
 
+We run qapi-gen.py like this:
+
+    $ python scripts/qapi-gen.py --output-dir="qapi-generated" \
+    --prefix="example-" example-schema.json
+
 For a more thorough look at generated code, the testsuite includes
 tests/qapi-schema/qapi-schema-tests.json that covers more examples of
 what the generator will accept, and compiles the resulting C code as
 part of 'make check-unit'.
 
-=== scripts/qapi-types.py ===
+=== Code generated for QAPI types ===
 
-Used to generate the C types defined by a schema, along with
-supporting code. The following files are created:
+The following files are created:
 
 $(prefix)qapi-types.h - C types corresponding to types defined in
-                        the schema you pass in
+                        the schema
+
 $(prefix)qapi-types.c - Cleanup functions for the above C types
 
 The $(prefix) is an optional parameter used as a namespace to keep the
@@ -943,8 +949,6 @@ created code.
 
 Example:
 
-    $ python scripts/qapi-types.py --output-dir="qapi-generated" \
-    --prefix="example-" example-schema.json
     $ cat qapi-generated/example-qapi-types.h
 [Uninteresting stuff omitted...]
 
@@ -1008,28 +1012,26 @@ Example:
         visit_free(v);
     }
 
-=== scripts/qapi-visit.py ===
+=== Code generated for visiting QAPI types ===
 
-Used to generate the visitor functions used to walk through and
-convert between a native QAPI C data structure and some other format
-(such as QObject); the generated functions are named visit_type_FOO()
-and visit_type_FOO_members().
+These are the visitor functions used to walk through and convert
+between a native QAPI C data structure and some other format (such as
+QObject); the generated functions are named visit_type_FOO() and
+visit_type_FOO_members().
 
 The following files are generated:
 
-$(prefix)qapi-visit.c: visitor function for a particular C type, used
+$(prefix)qapi-visit.c: Visitor function for a particular C type, used
                        to automagically convert QObjects into the
                        corresponding C type and vice-versa, as well
                        as for deallocating memory for an existing C
                        type
 
-$(prefix)qapi-visit.h: declarations for previously mentioned visitor
+$(prefix)qapi-visit.h: Declarations for previously mentioned visitor
                        functions
 
 Example:
 
-    $ python scripts/qapi-visit.py --output-dir="qapi-generated"
-    --prefix="example-" example-schema.json
     $ cat qapi-generated/example-qapi-visit.h
 [Uninteresting stuff omitted...]
 
@@ -1137,30 +1139,22 @@ Example:
         error_propagate(errp, err);
     }
 
-=== scripts/qapi-commands.py ===
+=== Code generated for commands ===
 
-Used to generate the marshaling/dispatch functions for the commands
-defined in the schema. The generated code implements
-qmp_marshal_COMMAND() (registered automatically), and declares
-qmp_COMMAND() that the user must implement.  The following files are
-generated:
+These are the marshaling/dispatch functions for the commands defined
+in the schema.  The generated code provides qmp_marshal_COMMAND(), and
+declares qmp_COMMAND() that the user must implement.
 
-$(prefix)qmp-marshal.c: command marshal/dispatch functions for each
-                        QMP command defined in the schema. Functions
-                        generated by qapi-visit.py are used to
-                        convert QObjects received from the wire into
-                        function parameters, and uses the same
-                        visitor functions to convert native C return
-                        values to QObjects from transmission back
-                        over the wire.
+The following files are generated:
+
+$(prefix)qmp-marshal.c: Command marshal/dispatch functions for each
+                        QMP command defined in the schema
 
 $(prefix)qmp-commands.h: Function prototypes for the QMP commands
-                         specified in the schema.
+                         specified in the schema
 
 Example:
 
-    $ python scripts/qapi-commands.py --output-dir="qapi-generated"
-    --prefix="example-" example-schema.json
     $ cat qapi-generated/example-qmp-commands.h
 [Uninteresting stuff omitted...]
 
@@ -1242,20 +1236,20 @@ Example:
                              qmp_marshal_my_command, QCO_NO_OPTIONS);
     }
 
-=== scripts/qapi-event.py ===
+=== Code generated for events ===
 
-Used to generate the event-related C code defined by a schema, with
-implementations for qapi_event_send_FOO(). The following files are
-created:
+This is the code related to events defined in the schema, providing
+qapi_event_send_EVENT().
+
+The following files are created:
 
 $(prefix)qapi-event.h - Function prototypes for each event type, plus an
                         enumeration of all event names
+
 $(prefix)qapi-event.c - Implementation of functions to send an event
 
 Example:
 
-    $ python scripts/qapi-event.py --output-dir="qapi-generated"
-    --prefix="example-" example-schema.json
     $ cat qapi-generated/example-qapi-event.h
 [Uninteresting stuff omitted...]
 
@@ -1302,23 +1296,22 @@ Example:
     }
 
     const char *const example_QAPIEvent_lookup[] = {
-        [EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
+
+[EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
         [EXAMPLE_QAPI_EVENT__MAX] = NULL,
     };
 
-=== scripts/qapi-introspect.py ===
+=== Code generated for introspection ===
 
-Used to generate the introspection C code for a schema. The following
-files are created:
+The following files are created:
 
 $(prefix)qmp-introspect.c - Defines a string holding a JSON
-                            description of the schema.
-$(prefix)qmp-introspect.h - Declares the above string.
+                            description of the schema
+
+$(prefix)qmp-introspect.h - Declares the above string
 
 Example:
 
-    $ python scripts/qapi-introspect.py --output-dir="qapi-generated"
-    --prefix="example-" example-schema.json
     $ cat qapi-generated/example-qmp-introspect.h
 [Uninteresting stuff omitted...]
 
monitor.c +1 -1
index f4992505b1..df670f3e15 100644
@@ -951,7 +951,7 @@ EventInfoList *qmp_query_events(Error **errp)
  * visit_type_SchemaInfoList() into a SchemaInfoList, then marshal it
  * to QObject with generated output marshallers, every time.  Instead,
  * we do it in test-qobject-input-visitor.c, just to make sure
- * qapi-introspect.py's output actually conforms to the schema.
+ * qapi-gen.py's output actually conforms to the schema.
  */
 static void qmp_query_qmp_schema(QDict *qdict, QObject **ret_data,
                                  Error **errp)
qapi-schema.json +1 -1
index 5c06745c79..0d651c8399 100644
@@ -72,7 +72,7 @@
         'q_obj_CpuInfo-base'    # CPU, visible through query-cpu
     ] } }
 
-# Documentation generated with qapi2texi.py is in source order, with
+# Documentation generated with qapi-gen.py is in source order, with
 # included sub-schemas inserted at the first include directive
 # (subsequent include directives have no effect).  To get a sane and
 # stable order, it's best to include each sub-schema just once, or
scripts/qapi-gen.py new file +41 -0
new file mode 100755
index 0000000000..2100ca1145
@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+# QAPI generator
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+import sys
+from qapi.common import parse_command_line, QAPISchema
+from qapi.types import gen_types
+from qapi.visit import gen_visit
+from qapi.commands import gen_commands
+from qapi.events import gen_events
+from qapi.introspect import gen_introspect
+from qapi.doc import gen_doc
+
+
+def main(argv):
+    (input_file, output_dir, prefix, opts) = \
+        parse_command_line('bu', ['builtins', 'unmask-non-abi-names'])
+
+    opt_builtins = False
+    opt_unmask = False
+
+    for o, a in opts:
+        if o in ('-b', '--builtins'):
+            opt_builtins = True
+        if o in ('-u', '--unmask-non-abi-names'):
+            opt_unmask = True
+
+    schema = QAPISchema(input_file)
+
+    gen_types(schema, output_dir, prefix, opt_builtins)
+    gen_visit(schema, output_dir, prefix, opt_builtins)
+    gen_commands(schema, output_dir, prefix)
+    gen_events(schema, output_dir, prefix)
+    gen_introspect(schema, output_dir, prefix, opt_unmask)
+    gen_doc(schema, output_dir, prefix)
+
+
+if __name__ == '__main__':
+    main(sys.argv)
scripts/qapi/__init__.py new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
scripts/qapi/commands.py renamed from scripts/qapi-commands.py +5 -18
similarity index 94%
rename from scripts/qapi-commands.py
rename to scripts/qapi/commands.py
index 8584cb5873..a744611d58 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def gen_command_decl(name, arg_type, boxed, ret_type):
@@ -255,13 +255,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._regy += gen_register_command(name, success_response)
 
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP commands
-'''
-
+def gen_commands(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP commands'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -290,17 +285,9 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
                    prefix=prefix, c_prefix=c_name(prefix, protect=False)))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenCommandVisitor(prefix)
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qmp-marshal.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qmp-commands.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qmp-marshal.c')
+    genh.write(output_dir, prefix + 'qmp-commands.h')
scripts/qapi/common.py renamed from scripts/qapi.py +4 -14
similarity index 99%
rename from scripts/qapi.py
rename to scripts/qapi/common.py
index 9693fd1851..868ec25deb 100644
@@ -1932,17 +1932,15 @@ def parse_command_line(extra_options='', extra_long_options=[]):
 
     try:
         opts, args = getopt.gnu_getopt(sys.argv[1:],
-                                       'chp:o:' + extra_options,
-                                       ['source', 'header', 'prefix=',
-                                        'output-dir='] + extra_long_options)
+                                       'p:o:' + extra_options,
+                                       ['prefix=', 'output-dir=']
+                                       + extra_long_options)
     except getopt.GetoptError as err:
         print("%s: %s" % (sys.argv[0], str(err)), file=sys.stderr)
         sys.exit(1)
 
     output_dir = ''
     prefix = ''
-    do_c = False
-    do_h = False
     extra_opts = []
 
     for oa in opts:
@@ -1956,23 +1954,15 @@ def parse_command_line(extra_options='', extra_long_options=[]):
             prefix = a
         elif o in ('-o', '--output-dir'):
             output_dir = a + '/'
-        elif o in ('-c', '--source'):
-            do_c = True
-        elif o in ('-h', '--header'):
-            do_h = True
         else:
             extra_opts.append(oa)
 
-    if not do_c and not do_h:
-        do_c = True
-        do_h = True
-
     if len(args) != 1:
         print("%s: need exactly one argument" % sys.argv[0], file=sys.stderr)
         sys.exit(1)
     fname = args[0]
 
-    return (fname, output_dir, do_c, do_h, prefix, extra_opts)
+    return (fname, output_dir, prefix, extra_opts)
 
 
 #
scripts/qapi/doc.py renamed from scripts/qapi2texi.py +8 -21
old mode 100755
new mode 100644
similarity index 92%
rename from scripts/qapi2texi.py
rename to scripts/qapi/doc.py
index 8a604d86a6..cc4d5a43fb
@@ -4,11 +4,10 @@
 # This work is licensed under the terms of the GNU LGPL, version 2+.
 # See the COPYING file in the top-level directory.
 """This script produces the documentation of a qapi schema in texinfo format"""
+
 from __future__ import print_function
 import re
-import sys
-
-import qapi
+import qapi.common
 
 MSG_FMT = """
 @deftypefn {type} {{}} {name}
@@ -197,7 +196,7 @@ def texi_entity(doc, what, base=None, variants=None,
             + texi_sections(doc))
 
 
-class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
+class QAPISchemaGenDocVisitor(qapi.common.QAPISchemaVisitor):
     def __init__(self):
         self.out = None
         self.cur_doc = None
@@ -272,20 +271,8 @@ def texi_schema(schema):
     return gen.out
 
 
-def main(argv):
-    """Takes schema argument, prints result to stdout"""
-    if len(argv) != 2:
-        print("%s: need exactly 1 argument: SCHEMA" % argv[0], file=sys.stderr)
-        sys.exit(1)
-
-    schema = qapi.QAPISchema(argv[1])
-    if not qapi.doc_required:
-        print("%s: need pragma 'doc-required' "
-               "to generate documentation" % argv[0], file=sys.stderr)
-        sys.exit(1)
-    print('@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n')
-    print(texi_schema(schema), end='')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+def gen_doc(schema, output_dir, prefix):
+    if qapi.common.doc_required:
+        gen = qapi.common.QAPIGenDoc()
+        gen.add(texi_schema(schema))
+        gen.write(output_dir, prefix + 'qapi-doc.texi')
scripts/qapi/events.py renamed from scripts/qapi-event.py +5 -18
similarity index 92%
rename from scripts/qapi-event.py
rename to scripts/qapi/events.py
index e7e07f0055..b7dc82004f 100644
@@ -12,7 +12,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def build_event_send_proto(name, arg_type, boxed):
@@ -171,13 +171,8 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
         self._event_names.append(name)
 
 
-def main(argv):
-    (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
-
-    blurb = '''
- * Schema-defined QAPI/QMP events
-'''
-
+def gen_events(schema, output_dir, prefix):
+    blurb = ' * Schema-defined QAPI/QMP events'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -201,17 +196,9 @@ def main(argv):
 ''',
                    prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenEventVisitor(prefix)
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-event.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-event.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-event.c')
+    genh.write(output_dir, prefix + 'qapi-event.h')
scripts/qapi/introspect.py renamed from scripts/qapi-introspect.py +5 -27
similarity index 90%
rename from scripts/qapi-introspect.py
rename to scripts/qapi/introspect.py
index b098b95053..1e4f065164 100644
@@ -10,7 +10,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # Caveman's json.dumps() replacement (we're stuck at Python 2.4)
@@ -168,22 +168,8 @@ const char %(c_name)s[] = %(c_string)s;
         self._gen_json(name, 'event', {'arg-type': self._use_type(arg_type)})
 
 
-def main(argv):
-    # Debugging aid: unmask QAPI schema's type names
-    # We normally mask them, because they're not QMP wire ABI
-    opt_unmask = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('u', ['unmask-non-abi-names'])
-
-    for o, a in opts:
-        if o in ('-u', '--unmask-non-abi-names'):
-            opt_unmask = True
-
-    blurb = '''
- * QAPI/QMP schema introspection
-'''
-
+def gen_introspect(schema, output_dir, prefix, opt_unmask):
+    blurb = ' * QAPI/QMP schema introspection'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -194,17 +180,9 @@ def main(argv):
 ''',
                    prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qmp-introspect.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qmp-introspect.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qmp-introspect.c')
+    genh.write(output_dir, prefix + 'qmp-introspect.h')
scripts/qapi/types.py renamed from scripts/qapi-types.py +5 -29
similarity index 90%
rename from scripts/qapi-types.py
rename to scripts/qapi/types.py
index 10955d1c01..aa3c01e750 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 # See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 # variants must be emitted before their container; track what has already
@@ -241,24 +241,8 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self._gen_type_cleanup(name)
 
 
-def main(argv):
-    # If you link code generated from multiple schemata, you want only one
-    # instance of the code for built-in types.  Generate it only when
-    # opt_builtins, enabled by command line option -b.  See also
-    # QAPISchemaGenTypeVisitor.visit_end().
-    opt_builtins = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('b', ['builtins'])
-
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-
-    blurb = '''
- * Schema-defined QAPI types
-'''
-
+def gen_types(schema, output_dir, prefix, opt_builtins):
+    blurb = ' * Schema-defined QAPI types'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -274,17 +258,9 @@ def main(argv):
 #include "qapi/util.h"
 '''))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenTypeVisitor(opt_builtins)
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-types.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-types.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-types.c')
+    genh.write(output_dir, prefix + 'qapi-types.h')
scripts/qapi/visit.py renamed from scripts/qapi-visit.py +5 -29
similarity index 92%
rename from scripts/qapi-visit.py
rename to scripts/qapi/visit.py
index 6d829c4d1d..f966e7975f 100644
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 2.
 See the COPYING file in the top-level directory.
 """
 
-from qapi import *
+from qapi.common import *
 
 
 def gen_visit_decl(name, scalar=False):
@@ -324,24 +324,8 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         self.defn += gen_visit_alternate(name, variants)
 
 
-def main(argv):
-    # If you link code generated from multiple schemata, you want only one
-    # instance of the code for built-in types.  Generate it only when
-    # opt_builtins, enabled by command line option -b.  See also
-    # QAPISchemaGenVisitVisitor.visit_end().
-    opt_builtins = False
-
-    (input_file, output_dir, do_c, do_h, prefix, opts) = \
-        parse_command_line('b', ['builtins'])
-
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-
-    blurb = '''
- * Schema-defined QAPI visitors
-'''
-
+def gen_visit(schema, output_dir, prefix, opt_builtins):
+    blurb = ' * Schema-defined QAPI visitors'
     genc = QAPIGenC(blurb, __doc__)
     genh = QAPIGenH(blurb, __doc__)
 
@@ -361,17 +345,9 @@ def main(argv):
 ''',
                    prefix=prefix))
 
-    schema = QAPISchema(input_file)
     vis = QAPISchemaGenVisitVisitor(opt_builtins)
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-
-    if do_c:
-        genc.write(output_dir, prefix + 'qapi-visit.c')
-    if do_h:
-        genh.write(output_dir, prefix + 'qapi-visit.h')
-
-
-if __name__ == '__main__':
-    main(sys.argv)
+    genc.write(output_dir, prefix + 'qapi-visit.c')
+    genh.write(output_dir, prefix + 'qapi-visit.h')
tests/Makefile.include +28 -28
index f41da235ae..b406531b34 100644
@@ -23,7 +23,16 @@ check-help:
 ifneq ($(wildcard config-host.mak),)
 export SRC_PATH
 
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
+# TODO don't duplicate $(SRC_PATH)/Makefile's qapi-py here
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py
 
 # Get the list of all supported sysemu targets
 SYSEMU_TARGET_LIST := $(subst -softmmu.mak,,$(notdir \
@@ -645,34 +654,24 @@ tests/test-logging$(EXESUF): tests/test-logging.o $(test-util-obj-y)
 tests/test-replication$(EXESUF): tests/test-replication.o $(test-util-obj-y) \
        $(test-block-obj-y)
 
-tests/test-qapi-types.c tests/test-qapi-types.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qapi-visit.c tests/test-qapi-visit.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qmp-commands.h tests/test-qmp-marshal.c :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qapi-event.c tests/test-qapi-event.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-event.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
-tests/test-qmp-introspect.c tests/test-qmp-introspect.h :\
-$(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \
-               $(gen-out-type) -o tests -p "test-" $<, \
-               "GEN","$@")
+tests/test-qapi-types.c tests/test-qapi-types.h \
+tests/test-qapi-visit.c tests/test-qapi-visit.h \
+tests/test-qmp-commands.h tests/test-qmp-marshal.c \
+tests/test-qapi-event.c tests/test-qapi-event.h \
+tests/test-qmp-introspect.c tests/test-qmp-introspect.h: \
+tests/test-qapi-gen-timestamp ;
+tests/test-qapi-gen-timestamp: $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(qapi-py)
+       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o tests -p "test-" $<, \
+               "GEN","$(@:%-timestamp=%)")
+       @>$@
 
-tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-good.json $(SRC_PATH)/scripts/qapi2texi.py $(qapi-py)
-       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"GEN","$@")
+tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-good.json $(qapi-py)
+       $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+               -o tests/qapi-schema -p "doc-good-" $<, \
+               "GEN","$@")
+       @mv tests/qapi-schema/doc-good-qapi-doc.texi $@
+       @rm -f tests/qapi-schema/doc-good-qapi-*.[ch] tests/qapi-schema/doc-good-qmp-*.[ch]
 
 tests/test-string-output-visitor$(EXESUF): tests/test-string-output-visitor.o $(test-qapi-obj-y)
 tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(test-qapi-obj-y)
@@ -941,6 +940,7 @@ check-clean:
        $(MAKE) -C tests/tcg clean
        rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
        rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
+       rm -f tests/test-qapi-gen-timestamp
 
 clean: check-clean
 
tests/qapi-schema/test-qapi.py +1 -3
index ac43d3458e..bb1b6dd297 100644
@@ -11,10 +11,8 @@
 #
 
 from __future__ import print_function
-from qapi import *
-from pprint import pprint
-import os
 import sys
+from qapi.common import QAPISchema, QAPISchemaVisitor
 
 
 class QAPISchemaTestVisitor(QAPISchemaVisitor):
Signature · 1 line
2.13.6
#27 lore raw

PATCH v2 9/29 qapi-gen: Convert from getopt to argparse

In reply to Markus Armbruster

argparse is nicer to use than getopt, and gives us --help almost for free.

---
 scripts/qapi-gen.py    | 48 ++++++++++++++++++++++++++++++------------------
 scripts/qapi/common.py | 43 -------------------------------------------
 2 files changed, 30 insertions(+), 61 deletions(-)
scripts/qapi-gen.py +30 -18
index 2100ca1145..e5be484e3e 100755
@@ -4,8 +4,11 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+from __future__ import print_function
+import argparse
+import re
 import sys
-from qapi.common import parse_command_line, QAPISchema
+from qapi.common import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 from qapi.commands import gen_commands
@@ -15,26 +18,35 @@ from qapi.doc import gen_doc
 
 
 def main(argv):
-    (input_file, output_dir, prefix, opts) = \
-        parse_command_line('bu', ['builtins', 'unmask-non-abi-names'])
+    parser = argparse.ArgumentParser(
+        description='Generate code from a QAPI schema')
+    parser.add_argument('-b', '--builtins', action='store_true',
+                        help="generate code for built-in types")
+    parser.add_argument('-o', '--output_dir', action='store', default='',
+                        help="write output to directory OUTPUT_DIR")
+    parser.add_argument('-p', '--prefix', action='store', default='',
+                        help="prefix for symbols")
+    parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
+                        dest='unmask',
+                        help="expose non-ABI names in introspection")
+    parser.add_argument('schema', action='store')
+    args = parser.parse_args()
 
-    opt_builtins = False
-    opt_unmask = False
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
+    if match.end() != len(args.prefix):
+        print("%s: 'funny character '%s' in argument of --prefix"
+              % (sys.argv[0], args.prefix[match.end()]),
+              file=sys.stderr)
+        sys.exit(1)
 
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-        if o in ('-u', '--unmask-non-abi-names'):
-            opt_unmask = True
+    schema = QAPISchema(args.schema)
 
-    schema = QAPISchema(input_file)
-
-    gen_types(schema, output_dir, prefix, opt_builtins)
-    gen_visit(schema, output_dir, prefix, opt_builtins)
-    gen_commands(schema, output_dir, prefix)
-    gen_events(schema, output_dir, prefix)
-    gen_introspect(schema, output_dir, prefix, opt_unmask)
-    gen_doc(schema, output_dir, prefix)
+    gen_types(schema, args.output_dir, args.prefix, args.builtins)
+    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
+    gen_commands(schema, args.output_dir, args.prefix)
+    gen_events(schema, args.output_dir, args.prefix)
+    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+    gen_doc(schema, args.output_dir, args.prefix)
 
 
 if __name__ == '__main__':
scripts/qapi/common.py +0 -43

55 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v2 9/29
#28 lore raw

Re: [Qemu-devel] [PATCH v2 09/29] qapi-gen: Convert from getopt to argparse

Quoted 9 lines · Markus Armbruster

argparse is nicer to use than getopt, and gives us --help almost for free.

---
  scripts/qapi-gen.py    | 48 ++++++++++++++++++++++++++++++------------------
  scripts/qapi/common.py | 43 -------------------------------------------
  2 files changed, 30 insertions(+), 61 deletions(-)

More functionality in fewer lines of code - always a win ;)

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by Marc-André Lureau on v2 9/29
#29 lore raw

Re: [Qemu-devel] [PATCH v2 09/29] qapi-gen: Convert from getopt to argparse

In reply to Markus Armbruster
Quoted 4 lines · Markus Armbruster

argparse is nicer to use than getopt, and gives us --help almost for free.

nice,

Quoted 139 lines
---
 scripts/qapi-gen.py    | 48 ++++++++++++++++++++++++++++++------------------
 scripts/qapi/common.py | 43 -------------------------------------------
 2 files changed, 30 insertions(+), 61 deletions(-)
scripts/qapi-gen.py +30 -18
index 2100ca1145..e5be484e3e 100755
@@ -4,8 +4,11 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+from __future__ import print_function
+import argparse
+import re
 import sys
-from qapi.common import parse_command_line, QAPISchema
+from qapi.common import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 from qapi.commands import gen_commands
@@ -15,26 +18,35 @@ from qapi.doc import gen_doc
 
 
 def main(argv):
-    (input_file, output_dir, prefix, opts) = \
-        parse_command_line('bu', ['builtins', 'unmask-non-abi-names'])
+    parser = argparse.ArgumentParser(
+        description='Generate code from a QAPI schema')
+    parser.add_argument('-b', '--builtins', action='store_true',
+                        help="generate code for built-in types")
+    parser.add_argument('-o', '--output_dir', action='store', default='',
+                        help="write output to directory OUTPUT_DIR")
+    parser.add_argument('-p', '--prefix', action='store', default='',
+                        help="prefix for symbols")
+    parser.add_argument('-u', '--unmask-non-abi-names', action='store_true',
+                        dest='unmask',
+                        help="expose non-ABI names in introspection")
+    parser.add_argument('schema', action='store')
+    args = parser.parse_args()
 
-    opt_builtins = False
-    opt_unmask = False
+    match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', args.prefix)
+    if match.end() != len(args.prefix):
+        print("%s: 'funny character '%s' in argument of --prefix"
+              % (sys.argv[0], args.prefix[match.end()]),
+              file=sys.stderr)
+        sys.exit(1)
 
-    for o, a in opts:
-        if o in ('-b', '--builtins'):
-            opt_builtins = True
-        if o in ('-u', '--unmask-non-abi-names'):
-            opt_unmask = True
+    schema = QAPISchema(args.schema)
 
-    schema = QAPISchema(input_file)
-
-    gen_types(schema, output_dir, prefix, opt_builtins)
-    gen_visit(schema, output_dir, prefix, opt_builtins)
-    gen_commands(schema, output_dir, prefix)
-    gen_events(schema, output_dir, prefix)
-    gen_introspect(schema, output_dir, prefix, opt_unmask)
-    gen_doc(schema, output_dir, prefix)
+    gen_types(schema, args.output_dir, args.prefix, args.builtins)
+    gen_visit(schema, args.output_dir, args.prefix, args.builtins)
+    gen_commands(schema, args.output_dir, args.prefix)
+    gen_events(schema, args.output_dir, args.prefix)
+    gen_introspect(schema, args.output_dir, args.prefix, args.unmask)
+    gen_doc(schema, args.output_dir, args.prefix)
 
 
 if __name__ == '__main__':
scripts/qapi/common.py +0 -44
index 868ec25deb..8290795dc1 100644
@@ -13,7 +13,6 @@
 
 from __future__ import print_function
 import errno
-import getopt
 import os
 import re
 import string
@@ -1924,48 +1923,6 @@ def build_params(arg_type, boxed, extra):
 
 
 #
-# Common command line parsing
-#
-
-
-def parse_command_line(extra_options='', extra_long_options=[]):
-
-    try:
-        opts, args = getopt.gnu_getopt(sys.argv[1:],
-                                       'p:o:' + extra_options,
-                                       ['prefix=', 'output-dir=']
-                                       + extra_long_options)
-    except getopt.GetoptError as err:
-        print("%s: %s" % (sys.argv[0], str(err)), file=sys.stderr)
-        sys.exit(1)
-
-    output_dir = ''
-    prefix = ''
-    extra_opts = []
-
-    for oa in opts:
-        o, a = oa
-        if o in ('-p', '--prefix'):
-            match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', a)
-            if match.end() != len(a):
-                print("%s: 'funny character '%s' in argument of --prefix" \
-                      % (sys.argv[0], a[match.end()]), file=sys.stderr)
-                sys.exit(1)
-            prefix = a
-        elif o in ('-o', '--output-dir'):
-            output_dir = a + '/'
-        else:
-            extra_opts.append(oa)
-
-    if len(args) != 1:
-        print("%s: need exactly one argument" % sys.argv[0], file=sys.stderr)
-        sys.exit(1)
-    fname = args[0]
-
-    return (fname, output_dir, prefix, extra_opts)
-
-
-#
 # Accumulate and write output
 #
 
--

2.13.6

#30 lore raw

Re: [Qemu-devel] [PATCH v2 09/29] qapi-gen: Convert from getopt to argparse

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:47)

Quoted 27 lines

argparse is nicer to use than getopt, and gives us --help almost for free.

---
 scripts/qapi-gen.py    | 48 ++++++++++++++++++++++++++++++------------------
 scripts/qapi/common.py | 43 -------------------------------------------
 2 files changed, 30 insertions(+), 61 deletions(-)
scripts/qapi-gen.py +4 -1
index 2100ca1145..e5be484e3e 100755
@@ -4,8 +4,11 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+from __future__ import print_function
+import argparse
+import re
 import sys
-from qapi.common import parse_command_line, QAPISchema
+from qapi.common import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 from qapi.commands import gen_commands

scripts/qapi-gen.py from qapi.doc import gen_doc · line 25

Quoted 10 lines
scripts/qapi-gen.py +5 -2
@@ -15,26 +18,35 @@ from qapi.doc import gen_doc
 
 
 def main(argv):
-    (input_file, output_dir, prefix, opts) = \
-        parse_command_line('bu', ['builtins', 'unmask-non-abi-names'])
+    parser = argparse.ArgumentParser(
+        description='Generate code from a QAPI schema')
+    parser.add_argument('-b', '--builtins', action='store_true',
+                        help="generate code for built-in types")
+    parser.add_argument('-o', '--output_dir', action='store', default='',

Was the change from --output-dir to --output_dir intentional? The former seems more consistent.

#31 lore raw

Re: [Qemu-devel] [PATCH v2 09/29] qapi-gen: Convert from getopt to argparse

Quoted 30 lines · Michael Roth

Quoting Markus Armbruster (2018-02-11 03:35:47)

Quoted 29 lines

argparse is nicer to use than getopt, and gives us --help almost for free.

---
 scripts/qapi-gen.py    | 48 ++++++++++++++++++++++++++++++------------------
 scripts/qapi/common.py | 43 -------------------------------------------
 2 files changed, 30 insertions(+), 61 deletions(-)
scripts/qapi-gen.py +4 -1
index 2100ca1145..e5be484e3e 100755
@@ -4,8 +4,11 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or later.
 # See the COPYING file in the top-level directory.
 
+from __future__ import print_function
+import argparse
+import re
 import sys
-from qapi.common import parse_command_line, QAPISchema
+from qapi.common import QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 from qapi.commands import gen_commands
@@ -15,26 +18,35 @@ from qapi.doc import gen_doc
 
 
Quoted 11 lines

scripts/qapi-gen.py from qapi.doc import gen_doc · line 25

Quoted 8 lines
scripts/qapi-gen.py +5 -2
 def main(argv):
-    (input_file, output_dir, prefix, opts) = \
-        parse_command_line('bu', ['builtins', 'unmask-non-abi-names'])
+    parser = argparse.ArgumentParser(
+        description='Generate code from a QAPI schema')
+    parser.add_argument('-b', '--builtins', action='store_true',
+                        help="generate code for built-in types")
+    parser.add_argument('-o', '--output_dir', action='store', default='',

Was the change from --output-dir to --output_dir intentional? The former seems more consistent.

Editing accident, good catch!

#32 lore raw

PATCH v2 10/29 qapi: Touch generated files only when they change

In reply to Markus Armbruster

A massive number of objects depends on QAPI-generated headers. In my "build everything" tree, it's roughly 4800 out of 5100. This is particularly annoying when only some of the generated files change, say for a doc fix.

Improve qapi-gen.py to touch its output files only if they actually change. Rebuild time for a QAPI doc fix drops from many minutes to a few seconds. Rebuilds get faster for certain code changes, too. For instance, adding a simple QMP event now recompiles less than 200 instead of 4800 objects. But adding a QAPI type is as bad as ever; we've clearly got more work to do.

---
 scripts/qapi/common.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
scripts/qapi/common.py +9 -2
index 8290795dc1..2e58573a39 100644
@@ -1951,9 +1951,16 @@ class QAPIGen(object):
             except os.error as e:
                 if e.errno != errno.EEXIST:
                     raise
-        f = open(os.path.join(output_dir, fname), 'w')
-        f.write(self._top(fname) + self._preamble + self._body
+        fd = os.open(os.path.join(output_dir, fname),
+                     os.O_RDWR | os.O_CREAT, 0666)
+        f = os.fdopen(fd, 'r+')
+        text = (self._top(fname) + self._preamble + self._body
                 + self._bottom(fname))
+        oldtext = f.read(len(text) + 1)
+        if text != oldtext:
+            f.seek(0)
+            f.truncate(0)
+            f.write(text)
         f.close()
 
 
Signature · 1 line
2.13.6
#33 lore raw

Re: [Qemu-devel] [PATCH v2 10/29] qapi: Touch generated files only when they change

Quoted 23 lines · Markus Armbruster

A massive number of objects depends on QAPI-generated headers. In my "build everything" tree, it's roughly 4800 out of 5100. This is particularly annoying when only some of the generated files change, say for a doc fix.

Improve qapi-gen.py to touch its output files only if they actually change. Rebuild time for a QAPI doc fix drops from many minutes to a few seconds. Rebuilds get faster for certain code changes, too. For instance, adding a simple QMP event now recompiles less than 200 instead of 4800 objects. But adding a QAPI type is as bad as ever; we've clearly got more work to do.

---
  scripts/qapi/common.py | 11 +++++++++--
  1 file changed, 9 insertions(+), 2 deletions(-)
scripts/qapi/common.py +0 -0
index 8290795dc1..2e58573a39 100644

scripts/qapi/common.py class QAPIGen(object): · line 1955

Quoted 7 lines
scripts/qapi/common.py +2 -2
@@ -1951,9 +1951,16 @@ class QAPIGen(object):
              except os.error as e:
                  if e.errno != errno.EEXIST:
                      raise
-        f = open(os.path.join(output_dir, fname), 'w')
-        f.write(self._top(fname) + self._preamble + self._body
+        fd = os.open(os.path.join(output_dir, fname),
+                     os.O_RDWR | os.O_CREAT, 0666)

patchew complained here for mingw; I'm not sure why.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#34 lore raw

Re: [Qemu-devel] [PATCH v2 10/29] qapi: Touch generated files only when they change

Hi

Quoted 20 lines · Eric Blake
Quoted 19 lines · Markus Armbruster

A massive number of objects depends on QAPI-generated headers. In my "build everything" tree, it's roughly 4800 out of 5100. This is particularly annoying when only some of the generated files change, say for a doc fix.

Improve qapi-gen.py to touch its output files only if they actually change. Rebuild time for a QAPI doc fix drops from many minutes to a few seconds. Rebuilds get faster for certain code changes, too. For instance, adding a simple QMP event now recompiles less than 200 instead of 4800 objects. But adding a QAPI type is as bad as ever; we've clearly got more work to do.

---
  scripts/qapi/common.py | 11 +++++++++--
  1 file changed, 9 insertions(+), 2 deletions(-)
Quoted 15 lines

scripts/qapi/common.py class QAPIGen(object): · line 1955

Quoted 12 lines
scripts/qapi/common.py +2 -2
index 8290795dc1..2e58573a39 100644
@@ -1951,9 +1951,16 @@ class QAPIGen(object):
              except os.error as e:
                  if e.errno != errno.EEXIST:
                      raise
-        f = open(os.path.join(output_dir, fname), 'w')
-        f.write(self._top(fname) + self._preamble + self._body
+        fd = os.open(os.path.join(output_dir, fname),
+                     os.O_RDWR | os.O_CREAT, 0666)

patchew complained here for mingw; I'm not sure why.

python3 syntax error.
https://stackoverflow.com/questions/1837874/invalid-token-when-using-octal-numbers

Quoted 5 lines

--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org

#35 lore raw

Re: [Qemu-devel] [PATCH v2 10/29] qapi: Touch generated files only when they change

Quoted 39 lines · Marc-Andre Lureau

Hi

Quoted 20 lines · Eric Blake
Quoted 19 lines · Markus Armbruster

A massive number of objects depends on QAPI-generated headers. In my "build everything" tree, it's roughly 4800 out of 5100. This is particularly annoying when only some of the generated files change, say for a doc fix.

Improve qapi-gen.py to touch its output files only if they actually change. Rebuild time for a QAPI doc fix drops from many minutes to a few seconds. Rebuilds get faster for certain code changes, too. For instance, adding a simple QMP event now recompiles less than 200 instead of 4800 objects. But adding a QAPI type is as bad as ever; we've clearly got more work to do.

---
  scripts/qapi/common.py | 11 +++++++++--
  1 file changed, 9 insertions(+), 2 deletions(-)
Quoted 15 lines

scripts/qapi/common.py class QAPIGen(object): · line 1955

Quoted 12 lines
scripts/qapi/common.py +2 -2
index 8290795dc1..2e58573a39 100644
@@ -1951,9 +1951,16 @@ class QAPIGen(object):
              except os.error as e:
                  if e.errno != errno.EEXIST:
                      raise
-        f = open(os.path.join(output_dir, fname), 'w')
-        f.write(self._top(fname) + self._preamble + self._body
+        fd = os.open(os.path.join(output_dir, fname),
+                     os.O_RDWR | os.O_CREAT, 0666)

patchew complained here for mingw; I'm not sure why.

Quoted 2 lines

python3 syntax error.
https://stackoverflow.com/questions/1837874/invalid-token-when-using-octal-numbers

Yes, we need to spell the mode 0o666.

Reviewed-by on v2 10/29
#36 lore raw

Re: [Qemu-devel] [PATCH v2 10/29] qapi: Touch generated files only when they change

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:48)

Quoted 12 lines

A massive number of objects depends on QAPI-generated headers. In my "build everything" tree, it's roughly 4800 out of 5100. This is particularly annoying when only some of the generated files change, say for a doc fix.

Improve qapi-gen.py to touch its output files only if they actually change. Rebuild time for a QAPI doc fix drops from many minutes to a few seconds. Rebuilds get faster for certain code changes, too. For instance, adding a simple QMP event now recompiles less than 200 instead of 4800 objects. But adding a QAPI type is as bad as ever; we've clearly got more work to do.

Quoted 2 lines
Quoted 30 lines
---
 scripts/qapi/common.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
scripts/qapi/common.py +9 -2
index 8290795dc1..2e58573a39 100644
@@ -1951,9 +1951,16 @@ class QAPIGen(object):
             except os.error as e:
                 if e.errno != errno.EEXIST:
                     raise
-        f = open(os.path.join(output_dir, fname), 'w')
-        f.write(self._top(fname) + self._preamble + self._body
+        fd = os.open(os.path.join(output_dir, fname),
+                     os.O_RDWR | os.O_CREAT, 0666)
+        f = os.fdopen(fd, 'r+')
+        text = (self._top(fname) + self._preamble + self._body
                 + self._bottom(fname))
+        oldtext = f.read(len(text) + 1)
+        if text != oldtext:
+            f.seek(0)
+            f.truncate(0)
+            f.write(text)
         f.close()
 
 
Signature · 1 line
2.13.6
#37 lore raw

PATCH v2 11/29 qapi: Improve include file name reporting in error messages

In reply to Markus Armbruster

Error messages print absolute file names of included files even if the user gave a relative one on the command line:

    $ PYTHONPATH=scripts python -B tests/qapi-schema/test-qapi.py tests/qapi-schema/include-cycle.json
    In file included from tests/qapi-schema/include-cycle.json:1:
    In file included from /work/armbru/qemu/tests/qapi-schema/include-cycle-b.json:1:
    /work/armbru/qemu/tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Improve this to

In file included from tests/qapi-schema/include-cycle.json:1:
In file included from tests/qapi-schema/include-cycle-b.json:1:
tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

The error message when an include file can't be opened prints the include directive's file name, which is relative to the including file. Change this to print the file name relative to the working directory. Visible in tests/qapi-schema/include-no-file.err.

---
 scripts/qapi/common.py                | 12 ++++++------
 tests/qapi-schema/include-no-file.err |  2 +-
 2 files changed, 7 insertions(+), 7 deletions(-)
scripts/qapi/common.py +6 -6
index 2e58573a39..3e92b38ade 100644
@@ -259,9 +259,8 @@ class QAPIDoc(object):
 class QAPISchemaParser(object):
 
     def __init__(self, fp, previously_included=[], incl_info=None):
-        abs_fname = os.path.abspath(fp.name)
         self.fname = fp.name
-        previously_included.append(abs_fname)
+        previously_included.append(os.path.abspath(fp.name))
         self.incl_info = incl_info
         self.src = fp.read()
         if self.src == '' or self.src[-1] != '\n':
@@ -292,7 +291,7 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                self._include(include, info, os.path.dirname(abs_fname),
+                self._include(include, info, os.path.dirname(self.fname),
                               previously_included)
             elif "pragma" in expr:
                 self.reject_expr_doc(cur_doc)
@@ -325,7 +324,8 @@ class QAPISchemaParser(object):
                 % doc.symbol)
 
     def _include(self, include, info, base_dir, previously_included):
-        incl_abs_fname = os.path.join(base_dir, include)
+        incl_fname = os.path.join(base_dir, include)
+        incl_abs_fname = os.path.abspath(incl_fname)
         # catch inclusion cycle
         inf = info
         while inf:
@@ -337,9 +337,9 @@ class QAPISchemaParser(object):
         if incl_abs_fname in previously_included:
             return
         try:
-            fobj = open(incl_abs_fname, 'r')
+            fobj = open(incl_fname, 'r')
         except IOError as e:
-            raise QAPISemError(info, '%s: %s' % (e.strerror, include))
+            raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname))
         exprs_include = QAPISchemaParser(fobj, previously_included, info)
         self.exprs.extend(exprs_include.exprs)
         self.docs.extend(exprs_include.docs)
tests/qapi-schema/include-no-file.err +1 -1
index d5b9b22d85..e42bcf4bc1 100644
@@ -1 +1 @@
-tests/qapi-schema/include-no-file.json:1: No such file or directory: include-no-file-sub.json
+tests/qapi-schema/include-no-file.json:1: No such file or directory: tests/qapi-schema/include-no-file-sub.json
Signature · 1 line
2.13.6
Reviewed-by on v2 11/29
#38 lore raw

Re: [Qemu-devel] [PATCH v2 11/29] qapi: Improve include file name reporting in error messages

Quoting Markus Armbruster (2018-02-11 03:35:49)

Quoted 14 lines

Error messages print absolute file names of included files even if the user gave a relative one on the command line:

    $ PYTHONPATH=scripts python -B tests/qapi-schema/test-qapi.py tests/qapi-schema/include-cycle.json
    In file included from tests/qapi-schema/include-cycle.json:1:
    In file included from /work/armbru/qemu/tests/qapi-schema/include-cycle-b.json:1:
    /work/armbru/qemu/tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Improve this to

In file included from tests/qapi-schema/include-cycle.json:1:
In file included from tests/qapi-schema/include-cycle-b.json:1:
tests/qapi-schema/include-cycle-c.json:1: Inclusion loop for include-cycle.json

Quoted 8 lines

The error message when an include file can't be opened prints the include directive's file name, which is relative to the including file. Change this to print the file name relative to the working directory. Visible in tests/qapi-schema/include-no-file.err.

Quoted 61 lines
---
 scripts/qapi/common.py                | 12 ++++++------
 tests/qapi-schema/include-no-file.err |  2 +-
 2 files changed, 7 insertions(+), 7 deletions(-)
scripts/qapi/common.py +6 -6
index 2e58573a39..3e92b38ade 100644
@@ -259,9 +259,8 @@ class QAPIDoc(object):
 class QAPISchemaParser(object):
 
     def __init__(self, fp, previously_included=[], incl_info=None):
-        abs_fname = os.path.abspath(fp.name)
         self.fname = fp.name
-        previously_included.append(abs_fname)
+        previously_included.append(os.path.abspath(fp.name))
         self.incl_info = incl_info
         self.src = fp.read()
         if self.src == '' or self.src[-1] != '\n':
@@ -292,7 +291,7 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                self._include(include, info, os.path.dirname(abs_fname),
+                self._include(include, info, os.path.dirname(self.fname),
                               previously_included)
             elif "pragma" in expr:
                 self.reject_expr_doc(cur_doc)
@@ -325,7 +324,8 @@ class QAPISchemaParser(object):
                 % doc.symbol)
 
     def _include(self, include, info, base_dir, previously_included):
-        incl_abs_fname = os.path.join(base_dir, include)
+        incl_fname = os.path.join(base_dir, include)
+        incl_abs_fname = os.path.abspath(incl_fname)
         # catch inclusion cycle
         inf = info
         while inf:
@@ -337,9 +337,9 @@ class QAPISchemaParser(object):
         if incl_abs_fname in previously_included:
             return
         try:
-            fobj = open(incl_abs_fname, 'r')
+            fobj = open(incl_fname, 'r')
         except IOError as e:
-            raise QAPISemError(info, '%s: %s' % (e.strerror, include))
+            raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname))
         exprs_include = QAPISchemaParser(fobj, previously_included, info)
         self.exprs.extend(exprs_include.exprs)
         self.docs.extend(exprs_include.docs)
tests/qapi-schema/include-no-file.err +1 -1
index d5b9b22d85..e42bcf4bc1 100644
@@ -1 +1 @@
-tests/qapi-schema/include-no-file.json:1: No such file or directory: include-no-file-sub.json
+tests/qapi-schema/include-no-file.json:1: No such file or directory: tests/qapi-schema/include-no-file-sub.json
Signature · 1 line
2.13.6
#39 lore raw

PATCH v2 12/29 qapi/common: Eliminate QAPISchema.exprs

In reply to Markus Armbruster
---
 scripts/qapi/common.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +4 -4
index 3e92b38ade..d72c339ad5 100644
@@ -1461,13 +1461,13 @@ class QAPISchema(object):
     def __init__(self, fname):
         try:
             parser = QAPISchemaParser(open(fname, 'r'))
-            self.exprs = check_exprs(parser.exprs)
+            exprs = check_exprs(parser.exprs)
             self.docs = parser.docs
             self._entity_dict = {}
             self._predefining = True
             self._def_predefineds()
             self._predefining = False
-            self._def_exprs()
+            self._def_exprs(exprs)
             self.check()
         except QAPIError as err:
             print(err, file=sys.stderr)
@@ -1652,8 +1652,8 @@ class QAPISchema(object):
                 name, info, doc, 'arg', self._make_members(data, info))
         self._def_entity(QAPISchemaEvent(name, info, doc, data, boxed))
 
-    def _def_exprs(self):
-        for expr_elem in self.exprs:
+    def _def_exprs(self, exprs):
+        for expr_elem in exprs:
             expr = expr_elem['expr']
             info = expr_elem['info']
             doc = expr_elem.get('doc')
Signature · 1 line
2.13.6
Reviewed-by on v2 12/29
#40 lore raw

Re: [Qemu-devel] [PATCH v2 12/29] qapi/common: Eliminate QAPISchema.exprs

Quoting Markus Armbruster (2018-02-11 03:35:50)

Quoted 3 lines
Quoted 38 lines
---
 scripts/qapi/common.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/qapi/common.py +4 -4
index 3e92b38ade..d72c339ad5 100644
@@ -1461,13 +1461,13 @@ class QAPISchema(object):
     def __init__(self, fname):
         try:
             parser = QAPISchemaParser(open(fname, 'r'))
-            self.exprs = check_exprs(parser.exprs)
+            exprs = check_exprs(parser.exprs)
             self.docs = parser.docs
             self._entity_dict = {}
             self._predefining = True
             self._def_predefineds()
             self._predefining = False
-            self._def_exprs()
+            self._def_exprs(exprs)
             self.check()
         except QAPIError as err:
             print(err, file=sys.stderr)
@@ -1652,8 +1652,8 @@ class QAPISchema(object):
                 name, info, doc, 'arg', self._make_members(data, info))
         self._def_entity(QAPISchemaEvent(name, info, doc, data, boxed))
 
-    def _def_exprs(self):
-        for expr_elem in self.exprs:
+    def _def_exprs(self, exprs):
+        for expr_elem in exprs:
             expr = expr_elem['expr']
             info = expr_elem['info']
             doc = expr_elem.get('doc')
Signature · 1 line
2.13.6
#41 lore raw

PATCH v2 13/29 qapi: Lift error reporting from QAPISchema.__init__() to callers

In reply to Markus Armbruster
---
 scripts/qapi-gen.py            |  8 ++++++--
 scripts/qapi/common.py         | 23 +++++++++--------------
 tests/qapi-schema/test-qapi.py | 10 ++++++++--
 3 files changed, 23 insertions(+), 18 deletions(-)
scripts/qapi-gen.py +6 -2
index e5be484e3e..fea7d4589b 100755
@@ -8,7 +8,7 @@ from __future__ import print_function
 import argparse
 import re
 import sys
-from qapi.common import QAPISchema
+from qapi.common import QAPIError, QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 from qapi.commands import gen_commands
@@ -39,7 +39,11 @@ def main(argv):
               file=sys.stderr)
         sys.exit(1)
 
-    schema = QAPISchema(args.schema)
+    try:
+        schema = QAPISchema(args.schema)
+    except QAPIError as err:
+        print(err, file=sys.stderr)
+        exit(1)
 
     gen_types(schema, args.output_dir, args.prefix, args.builtins)
     gen_visit(schema, args.output_dir, args.prefix, args.builtins)
scripts/qapi/common.py +9 -14
index d72c339ad5..dce289ae21 100644
@@ -16,7 +16,6 @@ import errno
 import os
 import re
 import string
-import sys
 try:
     from collections import OrderedDict
 except:
@@ -1459,19 +1458,15 @@ class QAPISchemaEvent(QAPISchemaEntity):
 
 class QAPISchema(object):
     def __init__(self, fname):
-        try:
-            parser = QAPISchemaParser(open(fname, 'r'))
-            exprs = check_exprs(parser.exprs)
-            self.docs = parser.docs
-            self._entity_dict = {}
-            self._predefining = True
-            self._def_predefineds()
-            self._predefining = False
-            self._def_exprs(exprs)
-            self.check()
-        except QAPIError as err:
-            print(err, file=sys.stderr)
-            exit(1)
+        parser = QAPISchemaParser(open(fname, 'r'))
+        exprs = check_exprs(parser.exprs)
+        self.docs = parser.docs
+        self._entity_dict = {}
+        self._predefining = True
+        self._def_predefineds()
+        self._predefining = False
+        self._def_exprs(exprs)
+        self.check()
 
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
tests/qapi-schema/test-qapi.py +8 -2
index bb1b6dd297..4da14b43af 100644
@@ -12,7 +12,7 @@
 
 from __future__ import print_function
 import sys
-from qapi.common import QAPISchema, QAPISchemaVisitor
+from qapi.common import QAPIError, QAPISchema, QAPISchemaVisitor
 
 
 class QAPISchemaTestVisitor(QAPISchemaVisitor):
@@ -52,7 +52,13 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
             for v in variants.variants:
                 print('    case %s: %s' % (v.name, v.type.name))
 
-schema = QAPISchema(sys.argv[1])
+
+try:
+    schema = QAPISchema(sys.argv[1])
+except QAPIError as err:
+    print(err, file=sys.stderr)
+    exit(1)
+
 schema.visit(QAPISchemaTestVisitor())
 
 for doc in schema.docs:
Signature · 1 line
2.13.6
Reviewed-by on v2 13/29
#42 lore raw

Re: [Qemu-devel] [PATCH v2 13/29] qapi: Lift error reporting from QAPISchema.__init__() to callers

Quoted 7 lines · Markus Armbruster
---
  scripts/qapi-gen.py            |  8 ++++++--
  scripts/qapi/common.py         | 23 +++++++++--------------
  tests/qapi-schema/test-qapi.py | 10 ++++++++--
  3 files changed, 23 insertions(+), 18 deletions(-)

The commit message didn't say why, but the change is reasonable.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v2 13/29
#43 lore raw

Re: [Qemu-devel] [PATCH v2 13/29] qapi: Lift error reporting from QAPISchema.__init__() to callers

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:51)

Quoted 2 lines
Quoted 104 lines
---
 scripts/qapi-gen.py            |  8 ++++++--
 scripts/qapi/common.py         | 23 +++++++++--------------
 tests/qapi-schema/test-qapi.py | 10 ++++++++--
 3 files changed, 23 insertions(+), 18 deletions(-)
scripts/qapi-gen.py +6 -2
index e5be484e3e..fea7d4589b 100755
@@ -8,7 +8,7 @@ from __future__ import print_function
 import argparse
 import re
 import sys
-from qapi.common import QAPISchema
+from qapi.common import QAPIError, QAPISchema
 from qapi.types import gen_types
 from qapi.visit import gen_visit
 from qapi.commands import gen_commands
@@ -39,7 +39,11 @@ def main(argv):
               file=sys.stderr)
         sys.exit(1)
 
-    schema = QAPISchema(args.schema)
+    try:
+        schema = QAPISchema(args.schema)
+    except QAPIError as err:
+        print(err, file=sys.stderr)
+        exit(1)
 
     gen_types(schema, args.output_dir, args.prefix, args.builtins)
     gen_visit(schema, args.output_dir, args.prefix, args.builtins)
scripts/qapi/common.py +9 -14
index d72c339ad5..dce289ae21 100644
@@ -16,7 +16,6 @@ import errno
 import os
 import re
 import string
-import sys
 try:
     from collections import OrderedDict
 except:
@@ -1459,19 +1458,15 @@ class QAPISchemaEvent(QAPISchemaEntity):
 
 class QAPISchema(object):
     def __init__(self, fname):
-        try:
-            parser = QAPISchemaParser(open(fname, 'r'))
-            exprs = check_exprs(parser.exprs)
-            self.docs = parser.docs
-            self._entity_dict = {}
-            self._predefining = True
-            self._def_predefineds()
-            self._predefining = False
-            self._def_exprs(exprs)
-            self.check()
-        except QAPIError as err:
-            print(err, file=sys.stderr)
-            exit(1)
+        parser = QAPISchemaParser(open(fname, 'r'))
+        exprs = check_exprs(parser.exprs)
+        self.docs = parser.docs
+        self._entity_dict = {}
+        self._predefining = True
+        self._def_predefineds()
+        self._predefining = False
+        self._def_exprs(exprs)
+        self.check()
 
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
tests/qapi-schema/test-qapi.py +8 -2
index bb1b6dd297..4da14b43af 100644
@@ -12,7 +12,7 @@
 
 from __future__ import print_function
 import sys
-from qapi.common import QAPISchema, QAPISchemaVisitor
+from qapi.common import QAPIError, QAPISchema, QAPISchemaVisitor
 
 
 class QAPISchemaTestVisitor(QAPISchemaVisitor):
@@ -52,7 +52,13 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
             for v in variants.variants:
                 print('    case %s: %s' % (v.name, v.type.name))
 
-schema = QAPISchema(sys.argv[1])
+
+try:
+    schema = QAPISchema(sys.argv[1])
+except QAPIError as err:
+    print(err, file=sys.stderr)
+    exit(1)
+
 schema.visit(QAPISchemaTestVisitor())
 
 for doc in schema.docs:
Signature · 1 line
2.13.6
#44 lore raw

PATCH v2 14/29 qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()

In reply to Markus Armbruster
---
 scripts/qapi/common.py | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
scripts/qapi/common.py +9 -6
index dce289ae21..cc5a5941dd 100644
@@ -290,8 +290,12 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                self._include(include, info, os.path.dirname(self.fname),
-                              previously_included)
+                exprs_include = self._include(include, info,
+                                              os.path.dirname(self.fname),
+                                              previously_included)
+                if exprs_include:
+                    self.exprs.extend(exprs_include.exprs)
+                    self.docs.extend(exprs_include.docs)
             elif "pragma" in expr:
                 self.reject_expr_doc(cur_doc)
                 if len(expr) != 1:
@@ -334,14 +338,13 @@ class QAPISchemaParser(object):
 
         # skip multiple include of the same file
         if incl_abs_fname in previously_included:
-            return
+            return None
+
         try:
             fobj = open(incl_fname, 'r')
         except IOError as e:
             raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname))
-        exprs_include = QAPISchemaParser(fobj, previously_included, info)
-        self.exprs.extend(exprs_include.exprs)
-        self.docs.extend(exprs_include.docs)
+        return QAPISchemaParser(fobj, previously_included, info)
 
     def _pragma(self, name, value, info):
         global doc_required, returns_whitelist, name_case_whitelist
Signature · 1 line
2.13.6
Reviewed-by on v2 14/29
#45 lore raw

Re: [Qemu-devel] [PATCH v2 14/29] qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()

Quoted 5 lines · Markus Armbruster
---
  scripts/qapi/common.py | 15 +++++++++------
  1 file changed, 9 insertions(+), 6 deletions(-)

Again, not much mention of 'why' in the commit message. But centralizing the initialization operations in one place rather than having to chase down multiple places is good.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v2 14/29
#46 lore raw

Re: [Qemu-devel] [PATCH v2 14/29] qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:52)

Quoted 29 lines
---
 scripts/qapi/common.py | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
scripts/qapi/common.py +6 -2
index dce289ae21..cc5a5941dd 100644
@@ -290,8 +290,12 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                self._include(include, info, os.path.dirname(self.fname),
-                              previously_included)
+                exprs_include = self._include(include, info,
+                                              os.path.dirname(self.fname),
+                                              previously_included)
+                if exprs_include:
+                    self.exprs.extend(exprs_include.exprs)
+                    self.docs.extend(exprs_include.docs)
             elif "pragma" in expr:
                 self.reject_expr_doc(cur_doc)
                 if len(expr) != 1:
@@ -334,14 +338,13 @@ class QAPISchemaParser(object):
 
         # skip multiple include of the same file
         if incl_abs_fname in previously_included:

scripts/qapi/common.py class QAPISchemaParser(object): · line 344

Quoted 10 lines
scripts/qapi/common.py +2 -4
-            return
+            return None
+
         try:
             fobj = open(incl_fname, 'r')
         except IOError as e:
             raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname))
-        exprs_include = QAPISchemaParser(fobj, previously_included, info)
-        self.exprs.extend(exprs_include.exprs)
-        self.docs.extend(exprs_include.docs)

minor nit, but the function of _include() seems more appropriately described now as _parse_include() or _parse_if_new() or something similar. Maybe it would be more readable to just move the previously_included checks into init() and just drop _include() entirely?

Quoted 7 lines
scripts/qapi/common.py +1 -0
+        return QAPISchemaParser(fobj, previously_included, info)
 
     def _pragma(self, name, value, info):
         global doc_required, returns_whitelist, name_case_whitelist
Signature · 1 line
2.13.6
#47 lore raw

Re: [Qemu-devel] [PATCH v2 14/29] qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()

Quoted 28 lines · Michael Roth

Quoting Markus Armbruster (2018-02-11 03:35:52)

Quoted 27 lines
---
 scripts/qapi/common.py | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
scripts/qapi/common.py +6 -2
index dce289ae21..cc5a5941dd 100644
@@ -290,8 +290,12 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                self._include(include, info, os.path.dirname(self.fname),
-                              previously_included)
+                exprs_include = self._include(include, info,
+                                              os.path.dirname(self.fname),
+                                              previously_included)
+                if exprs_include:
+                    self.exprs.extend(exprs_include.exprs)
+                    self.docs.extend(exprs_include.docs)
             elif "pragma" in expr:
                 self.reject_expr_doc(cur_doc)
                 if len(expr) != 1:
@@ -334,14 +338,13 @@ class QAPISchemaParser(object):
 
Quoted 15 lines

scripts/qapi/common.py class QAPISchemaParser(object): · line 344

Quoted 12 lines
scripts/qapi/common.py +2 -4
         # skip multiple include of the same file
         if incl_abs_fname in previously_included:
-            return
+            return None
+
         try:
             fobj = open(incl_fname, 'r')
         except IOError as e:
             raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname))
-        exprs_include = QAPISchemaParser(fobj, previously_included, info)
-        self.exprs.extend(exprs_include.exprs)
-        self.docs.extend(exprs_include.docs)

minor nit, but the function of _include() seems more appropriately described now as _parse_include() or _parse_if_new() or something similar.

I'm open to renaming, but "parse" doesn't really fit. _include()'s caller checks syntax, then calls _include() to check semantic constraints and evaluate the include, then splices in the evaluated result.

Quoted 2 lines

Maybe it would be more readable to just move the previously_included checks into init() and just drop _include() entirely?

Maybe. But the conditional gets rather long then.

Quoted 1 line

Thanks!

#48 lore raw

PATCH v2 15/29 qapi: Record 'include' directives in parse tree

In reply to Markus Armbruster

The parse tree is a list of expressions. Except include expressions currently get replaced by the included file's parse tree.

Instead of throwing away the include expression, keep it with the file name expanded so you don't have to track the including file's directory to make sense of it.

A future commit will put this include expression to use.

---
 scripts/qapi/common.py | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
scripts/qapi/common.py +17 -4
index cc5a5941dd..6d49709784 100644
@@ -290,8 +290,11 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                exprs_include = self._include(include, info,
-                                              os.path.dirname(self.fname),
+                incl_fname = os.path.join(os.path.dirname(self.fname),
+                                          include)
+                self.exprs.append({'expr': {'include': incl_fname},
+                                   'info': info})
+                exprs_include = self._include(include, info, incl_fname,
                                               previously_included)
                 if exprs_include:
                     self.exprs.extend(exprs_include.exprs)
@@ -326,8 +329,7 @@ class QAPISchemaParser(object):
                 "Documentation for '%s' is not followed by the definition"
                 % doc.symbol)
 
-    def _include(self, include, info, base_dir, previously_included):
-        incl_fname = os.path.join(base_dir, include)
+    def _include(self, include, info, incl_fname, previously_included):
         incl_abs_fname = os.path.abspath(incl_fname)
         # catch inclusion cycle
         inf = info
@@ -893,6 +895,9 @@ def check_exprs(exprs):
         info = expr_elem['info']
         doc = expr_elem.get('doc')
 
+        if 'include' in expr:
+            continue
+
         if not doc and doc_required:
             raise QAPISemError(info,
                                "Expression missing documentation comment")
@@ -931,6 +936,9 @@ def check_exprs(exprs):
 
     # Try again for hidden UnionKind enum
     for expr_elem in exprs:
+        if 'include' in expr:
+            continue
+
         expr = expr_elem['expr']
         if 'union' in expr and not discriminator_find_enum_define(expr):
             name = '%sKind' % expr['union']
@@ -943,6 +951,9 @@ def check_exprs(exprs):
 
     # Validate that exprs make sense
     for expr_elem in exprs:
+        if 'include' in expr:
+            continue
+
         expr = expr_elem['expr']
         info = expr_elem['info']
         doc = expr_elem.get('doc')
@@ -1667,6 +1678,8 @@ class QAPISchema(object):
                 self._def_command(expr, info, doc)
             elif 'event' in expr:
                 self._def_event(expr, info, doc)
+            elif 'include' in expr:
+                pass
             else:
                 assert False
 
Signature · 1 line
2.13.6
Reviewed-by on v2 15/29
#49 lore raw

Re: [Qemu-devel] [PATCH v2 15/29] qapi: Record 'include' directives in parse tree

Quoted 7 lines · Markus Armbruster

The parse tree is a list of expressions. Except include expressions currently get replaced by the included file's parse tree.

Instead of throwing away the include expression, keep it with the file name expanded so you don't have to track the including file's directory to make sense of it.

Quoted 8 lines

A future commit will put this include expression to use.

---
  scripts/qapi/common.py | 21 +++++++++++++++++----
  1 file changed, 17 insertions(+), 4 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#50 lore raw

Re: [Qemu-devel] [PATCH v2 15/29] qapi: Record 'include' directives in parse tree

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:53)

Quoted 54 lines

The parse tree is a list of expressions. Except include expressions currently get replaced by the included file's parse tree.

Instead of throwing away the include expression, keep it with the file name expanded so you don't have to track the including file's directory to make sense of it.

A future commit will put this include expression to use.

---
 scripts/qapi/common.py | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
scripts/qapi/common.py +9 -4
index cc5a5941dd..6d49709784 100644
@@ -290,8 +290,11 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                exprs_include = self._include(include, info,
-                                              os.path.dirname(self.fname),
+                incl_fname = os.path.join(os.path.dirname(self.fname),
+                                          include)
+                self.exprs.append({'expr': {'include': incl_fname},
+                                   'info': info})
+                exprs_include = self._include(include, info, incl_fname,
                                               previously_included)
                 if exprs_include:
                     self.exprs.extend(exprs_include.exprs)
@@ -326,8 +329,7 @@ class QAPISchemaParser(object):
                 "Documentation for '%s' is not followed by the definition"
                 % doc.symbol)
 
-    def _include(self, include, info, base_dir, previously_included):
-        incl_fname = os.path.join(base_dir, include)
+    def _include(self, include, info, incl_fname, previously_included):
         incl_abs_fname = os.path.abspath(incl_fname)
         # catch inclusion cycle
         inf = info
@@ -893,6 +895,9 @@ def check_exprs(exprs):
         info = expr_elem['info']
         doc = expr_elem.get('doc')
 
+        if 'include' in expr:
+            continue
+
         if not doc and doc_required:
             raise QAPISemError(info,
                                "Expression missing documentation comment")

scripts/qapi/common.py def check_exprs(exprs): · line 941

Quoted 6 lines
scripts/qapi/common.py +3 -0
@@ -931,6 +936,9 @@ def check_exprs(exprs):
 
     # Try again for hidden UnionKind enum
     for expr_elem in exprs:
+        if 'include' in expr:
+            continue
+

Wouldn't this ^

scripts/qapi/common.py def check_exprs(exprs): · line 942

Quoted 1 line
scripts/qapi/common.py +0 -0
         expr = expr_elem['expr']

Need to come after this?

scripts/qapi/common.py def check_exprs(exprs): · line 957

Quoted 10 lines
scripts/qapi/common.py +3 -0
         if 'union' in expr and not discriminator_find_enum_define(expr):
             name = '%sKind' % expr['union']
@@ -943,6 +951,9 @@ def check_exprs(exprs):
 
     # Validate that exprs make sense
     for expr_elem in exprs:
+        if 'include' in expr:
+            continue
+
         expr = expr_elem['expr']

And here.

Quoted 14 lines
scripts/qapi/common.py +2 -0
         info = expr_elem['info']
         doc = expr_elem.get('doc')
@@ -1667,6 +1678,8 @@ class QAPISchema(object):
                 self._def_command(expr, info, doc)
             elif 'event' in expr:
                 self._def_event(expr, info, doc)
+            elif 'include' in expr:
+                pass
             else:
                 assert False
 
Signature · 1 line
2.13.6
#51 lore raw

Re: [Qemu-devel] [PATCH v2 15/29] qapi: Record 'include' directives in parse tree

Quoted 64 lines · Michael Roth

Quoting Markus Armbruster (2018-02-11 03:35:53)

Quoted 54 lines

The parse tree is a list of expressions. Except include expressions currently get replaced by the included file's parse tree.

Instead of throwing away the include expression, keep it with the file name expanded so you don't have to track the including file's directory to make sense of it.

A future commit will put this include expression to use.

---
 scripts/qapi/common.py | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
scripts/qapi/common.py +9 -4
index cc5a5941dd..6d49709784 100644
@@ -290,8 +290,11 @@ class QAPISchemaParser(object):
                 if not isinstance(include, str):
                     raise QAPISemError(info,
                                        "Value of 'include' must be a string")
-                exprs_include = self._include(include, info,
-                                              os.path.dirname(self.fname),
+                incl_fname = os.path.join(os.path.dirname(self.fname),
+                                          include)
+                self.exprs.append({'expr': {'include': incl_fname},
+                                   'info': info})
+                exprs_include = self._include(include, info, incl_fname,
                                               previously_included)
                 if exprs_include:
                     self.exprs.extend(exprs_include.exprs)
@@ -326,8 +329,7 @@ class QAPISchemaParser(object):
                 "Documentation for '%s' is not followed by the definition"
                 % doc.symbol)
 
-    def _include(self, include, info, base_dir, previously_included):
-        incl_fname = os.path.join(base_dir, include)
+    def _include(self, include, info, incl_fname, previously_included):
         incl_abs_fname = os.path.abspath(incl_fname)
         # catch inclusion cycle
         inf = info
@@ -893,6 +895,9 @@ def check_exprs(exprs):
         info = expr_elem['info']
         doc = expr_elem.get('doc')
 
+        if 'include' in expr:
+            continue
+
         if not doc and doc_required:
             raise QAPISemError(info,
                                "Expression missing documentation comment")

scripts/qapi/common.py def check_exprs(exprs): · line 941

Quoted 6 lines
scripts/qapi/common.py +3 -0
@@ -931,6 +936,9 @@ def check_exprs(exprs):
 
     # Try again for hidden UnionKind enum
     for expr_elem in exprs:
+        if 'include' in expr:
+            continue
+

Wouldn't this ^

Quoted 3 lines

scripts/qapi/common.py def check_exprs(exprs): · line 942

Quoted 1 line
scripts/qapi/common.py +0 -0
         expr = expr_elem['expr']

Need to come after this?

D'oh!

Quoted 4 lines
Quoted 4 lines
scripts/qapi/common.py +0 -0
         if 'union' in expr and not discriminator_find_enum_define(expr):
             name = '%sKind' % expr['union']
@@ -943,6 +951,9 @@ def check_exprs(exprs):
 
Quoted 8 lines

scripts/qapi/common.py def check_exprs(exprs): · line 957

Quoted 6 lines
scripts/qapi/common.py +3 -0
     # Validate that exprs make sense
     for expr_elem in exprs:
+        if 'include' in expr:
+            continue
+
         expr = expr_elem['expr']

And here.

Right. I wonder how this passes tests.

Quoted 14 lines
Quoted 14 lines
scripts/qapi/common.py +2 -0
         info = expr_elem['info']
         doc = expr_elem.get('doc')
@@ -1667,6 +1678,8 @@ class QAPISchema(object):
                 self._def_command(expr, info, doc)
             elif 'event' in expr:
                 self._def_event(expr, info, doc)
+            elif 'include' in expr:
+                pass
             else:
                 assert False
 
Signature · 1 line
2.13.6
#52 lore raw

PATCH v2 16/29 qapi: Generate in source order

In reply to Markus Armbruster

The generators' conversion to visitors (merge commit 9e72681d16) changed the processing order of entities from source order to alphabetical order. The next commit needs source order, so change it back.

---
 scripts/qapi/common.py                   |   6 +-
 tests/qapi-schema/comments.out           |   2 +-
 tests/qapi-schema/doc-bad-section.out    |   4 +-
 tests/qapi-schema/doc-good.out           |  32 ++--
 tests/qapi-schema/empty.out              |   2 +-
 tests/qapi-schema/event-case.out         |   2 +-
 tests/qapi-schema/ident-with-escape.out  |   6 +-
 tests/qapi-schema/include-relpath.out    |   2 +-
 tests/qapi-schema/include-repetition.out |   2 +-
 tests/qapi-schema/include-simple.out     |   2 +-
 tests/qapi-schema/indented-expr.out      |   2 +-
 tests/qapi-schema/qapi-schema-test.out   | 320 +++++++++++++++----------------
 12 files changed, 192 insertions(+), 190 deletions(-)
scripts/qapi/common.py +4 -2
index 6d49709784..b531ab519f 100644
@@ -1475,6 +1475,7 @@ class QAPISchema(object):
         parser = QAPISchemaParser(open(fname, 'r'))
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
+        self._entity_list = []
         self._entity_dict = {}
         self._predefining = True
         self._def_predefineds()
@@ -1486,6 +1487,7 @@ class QAPISchema(object):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
         assert ent.name not in self._entity_dict
+        self._entity_list.append(ent)
         self._entity_dict[ent.name] = ent
 
     def lookup_entity(self, name, typ=None):
@@ -1684,12 +1686,12 @@ class QAPISchema(object):
                 assert False
 
     def check(self):
-        for (name, ent) in sorted(self._entity_dict.items()):
+        for ent in self._entity_list:
             ent.check(self)
 
     def visit(self, visitor):
         visitor.visit_begin(self)
-        for (name, entity) in sorted(self._entity_dict.items()):
+        for entity in self._entity_list:
             if visitor.visit_needed(entity):
                 entity.visit(visitor)
         visitor.visit_end()
tests/qapi-schema/comments.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/doc-bad-section.out +2 -2
index 089bde1381..23bf8c71ab 100644
@@ -1,7 +1,7 @@
-enum Enum ['one', 'two']
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-object q_empty
+enum Enum ['one', 'two']
 doc symbol=Enum
     body=
 == Produces *invalid* texinfo
tests/qapi-schema/doc-good.out +16 -16
index 1d2c250527..0c07301f07 100644
@@ -1,35 +1,35 @@
+object q_empty
+enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+    prefix QTYPE
+enum Enum ['one', 'two']
 object Base
     member base1: Enum optional=False
-enum Enum ['one', 'two']
+object Variant1
+    member var1: str optional=False
+object Variant2
 object Object
     base Base
     tag base1
     case one: Variant1
     case two: Variant2
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
-    prefix QTYPE
+object q_obj_Variant1-wrapper
+    member data: Variant1 optional=False
+object q_obj_Variant2-wrapper
+    member data: Variant2 optional=False
+enum SugaredUnionKind ['one', 'two']
 object SugaredUnion
     member type: SugaredUnionKind optional=False
     tag type
     case one: q_obj_Variant1-wrapper
     case two: q_obj_Variant2-wrapper
-enum SugaredUnionKind ['one', 'two']
-object Variant1
-    member var1: str optional=False
-object Variant2
-command cmd q_obj_cmd-arg -> Object
-   gen=True success_response=True boxed=False
-command cmd-boxed Object -> None
-   gen=True success_response=True boxed=True
-object q_empty
-object q_obj_Variant1-wrapper
-    member data: Variant1 optional=False
-object q_obj_Variant2-wrapper
-    member data: Variant2 optional=False
 object q_obj_cmd-arg
     member arg1: int optional=False
     member arg2: str optional=True
     member arg3: bool optional=False
+command cmd q_obj_cmd-arg -> Object
+   gen=True success_response=True boxed=False
+command cmd-boxed Object -> None
+   gen=True success_response=True boxed=True
 doc freeform
     body=
 = Section
tests/qapi-schema/empty.out +1 -1
index 40b886ddae..0ec234eec4 100644
@@ -1,3 +1,3 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-object q_empty
tests/qapi-schema/event-case.out +1 -1
index 313c0fe7be..110571b793 100644
@@ -1,5 +1,5 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 event oops None
    boxed=False
-object q_empty
tests/qapi-schema/ident-with-escape.out +3 -3
index b5637cb2e0..8336aa7629 100644
@@ -1,7 +1,7 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-command fooA q_obj_fooA-arg -> None
-   gen=True success_response=True boxed=False
-object q_empty
 object q_obj_fooA-arg
     member bar1: str optional=False
+command fooA q_obj_fooA-arg -> None
+   gen=True success_response=True boxed=False
tests/qapi-schema/include-relpath.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/include-repetition.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/include-simple.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/indented-expr.out +1 -1
index 586795f44d..34de8be426 100644
@@ -1,7 +1,7 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 command eins None -> None
    gen=True success_response=True boxed=False
-object q_empty
 command zwei None -> None
    gen=True success_response=True boxed=False
tests/qapi-schema/qapi-schema-test.out +160 -160

380 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v2 16/29
#53 lore raw

Re: [Qemu-devel] [PATCH v2 16/29] qapi: Generate in source order

Quoted 8 lines · Markus Armbruster

The generators' conversion to visitors (merge commit 9e72681d16) changed the processing order of entities from source order to alphabetical order. The next commit needs source order, so change it back.

---

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v2 16/29
#54 lore raw

Re: [Qemu-devel] [PATCH v2 16/29] qapi: Generate in source order

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:54)

Quoted 7 lines

The generators' conversion to visitors (merge commit 9e72681d16) changed the processing order of entities from source order to alphabetical order. The next commit needs source order, so change it back.

Quoted 597 lines
---
 scripts/qapi/common.py                   |   6 +-
 tests/qapi-schema/comments.out           |   2 +-
 tests/qapi-schema/doc-bad-section.out    |   4 +-
 tests/qapi-schema/doc-good.out           |  32 ++--
 tests/qapi-schema/empty.out              |   2 +-
 tests/qapi-schema/event-case.out         |   2 +-
 tests/qapi-schema/ident-with-escape.out  |   6 +-
 tests/qapi-schema/include-relpath.out    |   2 +-
 tests/qapi-schema/include-repetition.out |   2 +-
 tests/qapi-schema/include-simple.out     |   2 +-
 tests/qapi-schema/indented-expr.out      |   2 +-
 tests/qapi-schema/qapi-schema-test.out   | 320 +++++++++++++++----------------
 12 files changed, 192 insertions(+), 190 deletions(-)
scripts/qapi/common.py +4 -2
index 6d49709784..b531ab519f 100644
@@ -1475,6 +1475,7 @@ class QAPISchema(object):
         parser = QAPISchemaParser(open(fname, 'r'))
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
+        self._entity_list = []
         self._entity_dict = {}
         self._predefining = True
         self._def_predefineds()
@@ -1486,6 +1487,7 @@ class QAPISchema(object):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
         assert ent.name not in self._entity_dict
+        self._entity_list.append(ent)
         self._entity_dict[ent.name] = ent
 
     def lookup_entity(self, name, typ=None):
@@ -1684,12 +1686,12 @@ class QAPISchema(object):
                 assert False
 
     def check(self):
-        for (name, ent) in sorted(self._entity_dict.items()):
+        for ent in self._entity_list:
             ent.check(self)
 
     def visit(self, visitor):
         visitor.visit_begin(self)
-        for (name, entity) in sorted(self._entity_dict.items()):
+        for entity in self._entity_list:
             if visitor.visit_needed(entity):
                 entity.visit(visitor)
         visitor.visit_end()
tests/qapi-schema/comments.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/doc-bad-section.out +2 -2
index 089bde1381..23bf8c71ab 100644
@@ -1,7 +1,7 @@
-enum Enum ['one', 'two']
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-object q_empty
+enum Enum ['one', 'two']
 doc symbol=Enum
     body=
 == Produces *invalid* texinfo
tests/qapi-schema/doc-good.out +16 -16
index 1d2c250527..0c07301f07 100644
@@ -1,35 +1,35 @@
+object q_empty
+enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+    prefix QTYPE
+enum Enum ['one', 'two']
 object Base
     member base1: Enum optional=False
-enum Enum ['one', 'two']
+object Variant1
+    member var1: str optional=False
+object Variant2
 object Object
     base Base
     tag base1
     case one: Variant1
     case two: Variant2
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
-    prefix QTYPE
+object q_obj_Variant1-wrapper
+    member data: Variant1 optional=False
+object q_obj_Variant2-wrapper
+    member data: Variant2 optional=False
+enum SugaredUnionKind ['one', 'two']
 object SugaredUnion
     member type: SugaredUnionKind optional=False
     tag type
     case one: q_obj_Variant1-wrapper
     case two: q_obj_Variant2-wrapper
-enum SugaredUnionKind ['one', 'two']
-object Variant1
-    member var1: str optional=False
-object Variant2
-command cmd q_obj_cmd-arg -> Object
-   gen=True success_response=True boxed=False
-command cmd-boxed Object -> None
-   gen=True success_response=True boxed=True
-object q_empty
-object q_obj_Variant1-wrapper
-    member data: Variant1 optional=False
-object q_obj_Variant2-wrapper
-    member data: Variant2 optional=False
 object q_obj_cmd-arg
     member arg1: int optional=False
     member arg2: str optional=True
     member arg3: bool optional=False
+command cmd q_obj_cmd-arg -> Object
+   gen=True success_response=True boxed=False
+command cmd-boxed Object -> None
+   gen=True success_response=True boxed=True
 doc freeform
     body=
 = Section
tests/qapi-schema/empty.out +1 -1
index 40b886ddae..0ec234eec4 100644
@@ -1,3 +1,3 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-object q_empty
tests/qapi-schema/event-case.out +1 -1
index 313c0fe7be..110571b793 100644
@@ -1,5 +1,5 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 event oops None
    boxed=False
-object q_empty
tests/qapi-schema/ident-with-escape.out +3 -3
index b5637cb2e0..8336aa7629 100644
@@ -1,7 +1,7 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
-command fooA q_obj_fooA-arg -> None
-   gen=True success_response=True boxed=False
-object q_empty
 object q_obj_fooA-arg
     member bar1: str optional=False
+command fooA q_obj_fooA-arg -> None
+   gen=True success_response=True boxed=False
tests/qapi-schema/include-relpath.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/include-repetition.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/include-simple.out +1 -1
index 17e652535c..0261ddf202 100644
@@ -1,4 +1,4 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
-object q_empty
tests/qapi-schema/indented-expr.out +1 -1
index 586795f44d..34de8be426 100644
@@ -1,7 +1,7 @@
+object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
 command eins None -> None
    gen=True success_response=True boxed=False
-object q_empty
 command zwei None -> None
    gen=True success_response=True boxed=False
tests/qapi-schema/qapi-schema-test.out +160 -160
index 3b1e9082d3..50706b0136 100644
@@ -1,87 +1,129 @@
-alternate AltEnumBool
-    tag type
-    case e: EnumOne
-    case b: bool
-alternate AltEnumInt
-    tag type
-    case e: EnumOne
-    case i: int
-alternate AltEnumNum
-    tag type
-    case e: EnumOne
-    case n: number
-alternate AltNumEnum
-    tag type
-    case n: number
-    case e: EnumOne
-alternate AltStrObj
-    tag type
-    case s: str
-    case o: TestStruct
-event EVENT_A None
-   boxed=False
-event EVENT_B None
-   boxed=False
-event EVENT_C q_obj_EVENT_C-arg
-   boxed=False
-event EVENT_D q_obj_EVENT_D-arg
-   boxed=False
-event EVENT_E UserDefZero
-   boxed=True
-event EVENT_F UserDefAlternate
-   boxed=True
+object q_empty
+enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+    prefix QTYPE
+object TestStruct
+    member integer: int optional=False
+    member boolean: bool optional=False
+    member string: str optional=False
+object NestedEnumsOne
+    member enum1: EnumOne optional=False
+    member enum2: EnumOne optional=True
+    member enum3: EnumOne optional=False
+    member enum4: EnumOne optional=True
+enum MyEnum []
 object Empty1
 object Empty2
     base Empty1
+command user_def_cmd0 Empty2 -> Empty2
+   gen=True success_response=True boxed=False
+enum QEnumTwo ['value1', 'value2']
+    prefix QENUM_TWO
+object UserDefOne
+    base UserDefZero
+    member string: str optional=False
+    member enum1: EnumOne optional=True
 enum EnumOne ['value1', 'value2', 'value3']
-object EventStructOne
-    member struct1: UserDefOne optional=False
+object UserDefZero
+    member integer: int optional=False
+object UserDefTwoDictDict
+    member userdef: UserDefOne optional=False
     member string: str optional=False
-    member enum2: EnumOne optional=True
+object UserDefTwoDict
+    member string1: str optional=False
+    member dict2: UserDefTwoDictDict optional=False
+    member dict3: UserDefTwoDictDict optional=True
+object UserDefTwo
+    member string0: str optional=False
+    member dict1: UserDefTwoDict optional=False
 object ForceArrays
     member unused1: UserDefOneList optional=False
     member unused2: UserDefTwoList optional=False
     member unused3: TestStructList optional=False
-enum MyEnum []
-object NestedEnumsOne
-    member enum1: EnumOne optional=False
-    member enum2: EnumOne optional=True
-    member enum3: EnumOne optional=False
-    member enum4: EnumOne optional=True
-enum QEnumTwo ['value1', 'value2']
-    prefix QENUM_TWO
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
-    prefix QTYPE
-object TestStruct
-    member integer: int optional=False
-    member boolean: bool optional=False
-    member string: str optional=False
 object UserDefA
     member boolean: bool optional=False
     member a_b: int optional=True
-alternate UserDefAlternate
-    tag type
-    case udfu: UserDefFlatUnion
-    case e: EnumOne
-    case i: int
-    case n: null
 object UserDefB
     member intb: int optional=False
     member a-b: bool optional=True
-object UserDefC
-    member string1: str optional=False
-    member string2: str optional=False
 object UserDefFlatUnion
     base UserDefUnionBase
     tag enum1
     case value1: UserDefA
     case value2: UserDefB
     case value3: UserDefB
+object UserDefUnionBase
+    base UserDefZero
+    member string: str optional=False
+    member enum1: EnumOne optional=False
+object q_obj_UserDefFlatUnion2-base
+    member integer: int optional=True
+    member string: str optional=False
+    member enum1: QEnumTwo optional=False
 object UserDefFlatUnion2
     base q_obj_UserDefFlatUnion2-base
     tag enum1
     case value1: UserDefC
     case value2: UserDefB
+object WrapAlternate
+    member alt: UserDefAlternate optional=False
+alternate UserDefAlternate
+    tag type
+    case udfu: UserDefFlatUnion
+    case e: EnumOne
+    case i: int
+    case n: null
+object UserDefC
+    member string1: str optional=False
+    member string2: str optional=False
+alternate AltEnumBool
+    tag type
+    case e: EnumOne
+    case b: bool
+alternate AltEnumNum
+    tag type
+    case e: EnumOne
+    case n: number
+alternate AltNumEnum
+    tag type
+    case n: number
+    case e: EnumOne
+alternate AltEnumInt
+    tag type
+    case e: EnumOne
+    case i: int
+alternate AltStrObj
+    tag type
+    case s: str
+    case o: TestStruct
+object q_obj_intList-wrapper
+    member data: intList optional=False
+object q_obj_int8List-wrapper
+    member data: int8List optional=False
+object q_obj_int16List-wrapper
+    member data: int16List optional=False
+object q_obj_int32List-wrapper
+    member data: int32List optional=False
+object q_obj_int64List-wrapper
+    member data: int64List optional=False
+object q_obj_uint8List-wrapper
+    member data: uint8List optional=False
+object q_obj_uint16List-wrapper
+    member data: uint16List optional=False
+object q_obj_uint32List-wrapper
+    member data: uint32List optional=False
+object q_obj_uint64List-wrapper
+    member data: uint64List optional=False
+object q_obj_numberList-wrapper
+    member data: numberList optional=False
+object q_obj_boolList-wrapper
+    member data: boolList optional=False
+object q_obj_strList-wrapper
+    member data: strList optional=False
+object q_obj_sizeList-wrapper
+    member data: sizeList optional=False
+object q_obj_anyList-wrapper
+    member data: anyList optional=False
+enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32', 's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes', 'any']
 object UserDefNativeListUnion
     member type: UserDefNativeListUnionKind optional=False
     tag type
@@ -99,133 +141,91 @@ object UserDefNativeListUnion
     case string: q_obj_strList-wrapper
     case sizes: q_obj_sizeList-wrapper
     case any: q_obj_anyList-wrapper
-enum UserDefNativeListUnionKind ['integer', 's8', 's16', 's32', 's64', 'u8', 'u16', 'u32', 'u64', 'number', 'boolean', 'string', 'sizes', 'any']
-object UserDefOne
-    base UserDefZero
-    member string: str optional=False
-    member enum1: EnumOne optional=True
+command user_def_cmd None -> None
+   gen=True success_response=True boxed=False
+object q_obj_user_def_cmd1-arg
+    member ud1a: UserDefOne optional=False
+command user_def_cmd1 q_obj_user_def_cmd1-arg -> None
+   gen=True success_response=True boxed=False
+object q_obj_user_def_cmd2-arg
+    member ud1a: UserDefOne optional=False
+    member ud1b: UserDefOne optional=True
+command user_def_cmd2 q_obj_user_def_cmd2-arg -> UserDefTwo
+   gen=True success_response=True boxed=False
+object q_obj_guest-get-time-arg
+    member a: int optional=False
+    member b: int optional=True
+command guest-get-time q_obj_guest-get-time-arg -> int
+   gen=True success_response=True boxed=False
+object q_obj_guest-sync-arg
+    member arg: any optional=False
+command guest-sync q_obj_guest-sync-arg -> any
+   gen=True success_response=True boxed=False
+command boxed-struct UserDefZero -> None
+   gen=True success_response=True boxed=True
+command boxed-union UserDefNativeListUnion -> None
+   gen=True success_response=True boxed=True
 object UserDefOptions
     member i64: intList optional=True
     member u64: uint64List optional=True
     member u16: uint16List optional=True
     member i64x: int optional=True
     member u64x: uint64 optional=True
-object UserDefTwo
-    member string0: str optional=False
-    member dict1: UserDefTwoDict optional=False
-object UserDefTwoDict
-    member string1: str optional=False
-    member dict2: UserDefTwoDictDict optional=False
-    member dict3: UserDefTwoDictDict optional=True
-object UserDefTwoDictDict
-    member userdef: UserDefOne optional=False
+object EventStructOne
+    member struct1: UserDefOne optional=False
     member string: str optional=False
-object UserDefUnionBase
-    base UserDefZero
-    member string: str optional=False
-    member enum1: EnumOne optional=False
-object UserDefZero
-    member integer: int optional=False
-object WrapAlternate
-    member alt: UserDefAlternate optional=False
-event __ORG.QEMU_X-EVENT __org.qemu_x-Struct
+    member enum2: EnumOne optional=True
+event EVENT_A None
+   boxed=False
+event EVENT_B None
+   boxed=False
+object q_obj_EVENT_C-arg
+    member a: int optional=True
+    member b: UserDefOne optional=True
+    member c: str optional=False
+event EVENT_C q_obj_EVENT_C-arg
+   boxed=False
+object q_obj_EVENT_D-arg
+    member a: EventStructOne optional=False
+    member b: str optional=False
+    member c: str optional=True
+    member enum3: EnumOne optional=True
+event EVENT_D q_obj_EVENT_D-arg
    boxed=False
-alternate __org.qemu_x-Alt
-    tag type
-    case __org.qemu_x-branch: str
-    case b: __org.qemu_x-Base
+event EVENT_E UserDefZero
+   boxed=True
+event EVENT_F UserDefAlternate
+   boxed=True
+enum __org.qemu_x-Enum ['__org.qemu_x-value']
 object __org.qemu_x-Base
     member __org.qemu_x-member1: __org.qemu_x-Enum optional=False
-enum __org.qemu_x-Enum ['__org.qemu_x-value']
 object __org.qemu_x-Struct
     base __org.qemu_x-Base
     member __org.qemu_x-member2: str optional=False
     member wchar-t: int optional=True
-object __org.qemu_x-Struct2
-    member array: __org.qemu_x-Union1List optional=False
+object q_obj_str-wrapper
+    member data: str optional=False
+enum __org.qemu_x-Union1Kind ['__org.qemu_x-branch']
 object __org.qemu_x-Union1
     member type: __org.qemu_x-Union1Kind optional=False
     tag type
     case __org.qemu_x-branch: q_obj_str-wrapper
-enum __org.qemu_x-Union1Kind ['__org.qemu_x-branch']
+object __org.qemu_x-Struct2
+    member array: __org.qemu_x-Union1List optional=False
 object __org.qemu_x-Union2
     base __org.qemu_x-Base
     tag __org.qemu_x-member1
     case __org.qemu_x-value: __org.qemu_x-Struct2
-command __org.qemu_x-command q_obj___org.qemu_x-command-arg -> __org.qemu_x-Union1
-   gen=True success_response=True boxed=False
-command boxed-struct UserDefZero -> None
-   gen=True success_response=True boxed=True
-command boxed-union UserDefNativeListUnion -> None
-   gen=True success_response=True boxed=True
-command guest-get-time q_obj_guest-get-time-arg -> int
-   gen=True success_response=True boxed=False
-command guest-sync q_obj_guest-sync-arg -> any
-   gen=True success_response=True boxed=False
-object q_empty
-object q_obj_EVENT_C-arg
-    member a: int optional=True
-    member b: UserDefOne optional=True
-    member c: str optional=False
-object q_obj_EVENT_D-arg
-    member a: EventStructOne optional=False
-    member b: str optional=False
-    member c: str optional=True
-    member enum3: EnumOne optional=True
-object q_obj_UserDefFlatUnion2-base
-    member integer: int optional=True
-    member string: str optional=False
-    member enum1: QEnumTwo optional=False
+alternate __org.qemu_x-Alt
+    tag type
+    case __org.qemu_x-branch: str
+    case b: __org.qemu_x-Base
+event __ORG.QEMU_X-EVENT __org.qemu_x-Struct
+   boxed=False
 object q_obj___org.qemu_x-command-arg
     member a: __org.qemu_x-EnumList optional=False
     member b: __org.qemu_x-StructList optional=False
     member c: __org.qemu_x-Union2 optional=False
     member d: __org.qemu_x-Alt optional=False
-object q_obj_anyList-wrapper
-    member data: anyList optional=False
-object q_obj_boolList-wrapper
-    member data: boolList optional=False
-object q_obj_guest-get-time-arg
-    member a: int optional=False
-    member b: int optional=True
-object q_obj_guest-sync-arg
-    member arg: any optional=False
-object q_obj_int16List-wrapper
-    member data: int16List optional=False
-object q_obj_int32List-wrapper
-    member data: int32List optional=False
-object q_obj_int64List-wrapper
-    member data: int64List optional=False
-object q_obj_int8List-wrapper
-    member data: int8List optional=False
-object q_obj_intList-wrapper
-    member data: intList optional=False
-object q_obj_numberList-wrapper
-    member data: numberList optional=False
-object q_obj_sizeList-wrapper
-    member data: sizeList optional=False
-object q_obj_str-wrapper
-    member data: str optional=False
-object q_obj_strList-wrapper
-    member data: strList optional=False
-object q_obj_uint16List-wrapper
-    member data: uint16List optional=False
-object q_obj_uint32List-wrapper
-    member data: uint32List optional=False
-object q_obj_uint64List-wrapper
-    member data: uint64List optional=False
-object q_obj_uint8List-wrapper
-    member data: uint8List optional=False
-object q_obj_user_def_cmd1-arg
-    member ud1a: UserDefOne optional=False
-object q_obj_user_def_cmd2-arg
-    member ud1a: UserDefOne optional=False
-    member ud1b: UserDefOne optional=True
-command user_def_cmd None -> None
-   gen=True success_response=True boxed=False
-command user_def_cmd0 Empty2 -> Empty2
-   gen=True success_response=True boxed=False
-command user_def_cmd1 q_obj_user_def_cmd1-arg -> None
-   gen=True success_response=True boxed=False
-command user_def_cmd2 q_obj_user_def_cmd2-arg -> UserDefTwo
+command __org.qemu_x-command q_obj___org.qemu_x-command-arg -> __org.qemu_x-Union1
    gen=True success_response=True boxed=False
Signature · 1 line
2.13.6
#55 lore raw

PATCH v2 17/29 qapi: Record 'include' directives in intermediate representation

In reply to Markus Armbruster

The include directive permits modular QAPI schemata, but the generated code is monolithic all the same. To permit generating modular code, the front end needs to pass more information on inclusions to the back ends. The commit before last added the necessary information to the parse tree. This commit adds it to the intermediate representation and its QAPISchemaVisitor. A later commit will use this to to generate modular code.

New entity QAPISchemaInclude represents inclusions. Call new visitor method visit_include() for it, so visitors can see the sub-modules a module includes.

Note that unlike other entities, QAPISchemaInclude has no name, and is therefore not added to entity_dict.

New QAPISchemaEntity attribute @module names the entity's source file. Call new visitor method visit_module() when it changes during a visit, so visitors can keep track of the module being visited.

---
 scripts/qapi/common.py                   | 44 ++++++++++++++++++++++++++++----
 tests/qapi-schema/comments.out           |  1 +
 tests/qapi-schema/doc-bad-section.out    |  1 +
 tests/qapi-schema/doc-good.out           |  1 +
 tests/qapi-schema/event-case.out         |  1 +
 tests/qapi-schema/ident-with-escape.out  |  1 +
 tests/qapi-schema/include-relpath.out    |  5 ++++
 tests/qapi-schema/include-repetition.out | 10 ++++++++
 tests/qapi-schema/include-simple.out     |  3 +++
 tests/qapi-schema/indented-expr.out      |  1 +
 tests/qapi-schema/qapi-schema-test.out   |  1 +
 tests/qapi-schema/test-qapi.py           |  7 +++++
 12 files changed, 71 insertions(+), 5 deletions(-)
scripts/qapi/common.py +39 -5
index b531ab519f..29d98ca934 100644
@@ -985,8 +985,9 @@ def check_exprs(exprs):
 
 class QAPISchemaEntity(object):
     def __init__(self, name, info, doc):
-        assert isinstance(name, str)
+        assert name is None or isinstance(name, str)
         self.name = name
+        self.module = None
         # For explicitly defined entities, info points to the (explicit)
         # definition.  For builtins (and their arrays), info is None.
         # For implicitly defined entities, info points to a place that
@@ -1015,10 +1016,16 @@ class QAPISchemaVisitor(object):
     def visit_end(self):
         pass
 
+    def visit_module(self, fname):
+        pass
+
     def visit_needed(self, entity):
         # Default to visiting everything
         return True
 
+    def visit_include(self, fname, info):
+        pass
+
     def visit_builtin_type(self, name, info, json_type):
         pass
 
@@ -1045,6 +1052,16 @@ class QAPISchemaVisitor(object):
         pass
 
 
+class QAPISchemaInclude(QAPISchemaEntity):
+
+    def __init__(self, fname, info):
+        QAPISchemaEntity.__init__(self, None, info, None)
+        self.fname = fname
+
+    def visit(self, visitor):
+        visitor.visit_include(self.fname, self.info)
+
+
 class QAPISchemaType(QAPISchemaEntity):
     # Return the C type for common use.
     # For the types we commonly box, this is a pointer type.
@@ -1472,6 +1489,7 @@ class QAPISchemaEvent(QAPISchemaEntity):
 
 class QAPISchema(object):
     def __init__(self, fname):
+        self._fname = fname
         parser = QAPISchemaParser(open(fname, 'r'))
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
@@ -1479,16 +1497,19 @@ class QAPISchema(object):
         self._entity_dict = {}
         self._predefining = True
         self._def_predefineds()
-        self._predefining = False
         self._def_exprs(exprs)
         self.check()
 
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
-        assert ent.name not in self._entity_dict
+        assert ent.name is None or ent.name not in self._entity_dict
         self._entity_list.append(ent)
-        self._entity_dict[ent.name] = ent
+        if ent.name is not None:
+            self._entity_dict[ent.name] = ent
+        if ent.info:
+            ent.module = os.path.relpath(ent.info['file'],
+                                         os.path.dirname(self._fname))
 
     def lookup_entity(self, name, typ=None):
         ent = self._entity_dict.get(name)
@@ -1499,6 +1520,15 @@ class QAPISchema(object):
     def lookup_type(self, name):
         return self.lookup_entity(name, QAPISchemaType)
 
+    def _def_include(self, expr, info, doc):
+        include = expr['include']
+        assert doc is None
+        main_info = info
+        while main_info['parent']:
+            main_info = main_info['parent']
+        fname = os.path.relpath(include, os.path.dirname(main_info['file']))
+        self._def_entity(QAPISchemaInclude(fname, info))
+
     def _def_builtin_type(self, name, json_type, c_type):
         self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
         # TODO As long as we have QAPI_TYPES_BUILTIN to share multiple
@@ -1681,7 +1711,7 @@ class QAPISchema(object):
             elif 'event' in expr:
                 self._def_event(expr, info, doc)
             elif 'include' in expr:
-                pass
+                self._def_include(expr, info, doc)
             else:
                 assert False
 
@@ -1691,8 +1721,12 @@ class QAPISchema(object):
 
     def visit(self, visitor):
         visitor.visit_begin(self)
+        module = None
         for entity in self._entity_list:
             if visitor.visit_needed(entity):
+                if entity.module != module:
+                    module = entity.module
+                    visitor.visit_module(module)
                 entity.visit(visitor)
         visitor.visit_end()
 
tests/qapi-schema/comments.out +1 -0
index 0261ddf202..8d2f1ce8a2 100644
@@ -1,4 +1,5 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module comments.json
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/doc-bad-section.out +1 -0
index 23bf8c71ab..cd28721568 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module doc-bad-section.json
 enum Enum ['one', 'two']
 doc symbol=Enum
     body=
tests/qapi-schema/doc-good.out +1 -0
index 0c07301f07..430b5a87db 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module doc-good.json
 enum Enum ['one', 'two']
 object Base
     member base1: Enum optional=False
tests/qapi-schema/event-case.out +1 -0
index 110571b793..88c0964917 100644
@@ -1,5 +1,6 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module event-case.json
 event oops None
    boxed=False
tests/qapi-schema/ident-with-escape.out +1 -0
index 8336aa7629..ee3b34e623 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module ident-with-escape.json
 object q_obj_fooA-arg
     member bar1: str optional=False
 command fooA q_obj_fooA-arg -> None
tests/qapi-schema/include-relpath.out +5 -0
index 0261ddf202..ebbabd7a18 100644
@@ -1,4 +1,9 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module include-relpath.json
+include include/relpath.json
+module include/relpath.json
+include include-relpath-sub.json
+module include-relpath-sub.json
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-repetition.out +10 -0
index 0261ddf202..7235e055bc 100644
@@ -1,4 +1,14 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module include-repetition.json
+include comments.json
+module comments.json
 enum Status ['good', 'bad', 'ugly']
+module include-repetition.json
+include include-repetition-sub.json
+module include-repetition-sub.json
+include comments.json
+include comments.json
+module include-repetition.json
+include comments.json
tests/qapi-schema/include-simple.out +3 -0
index 0261ddf202..006f723eeb 100644
@@ -1,4 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module include-simple.json
+include include-simple-sub.json
+module include-simple-sub.json
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/indented-expr.out +1 -0
index 34de8be426..a79935e8c3 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module indented-expr.json
 command eins None -> None
    gen=True success_response=True boxed=False
 command zwei None -> None
tests/qapi-schema/qapi-schema-test.out +1 -0
index 50706b0136..012e7fc06a 100644
@@ -1,6 +1,7 @@
 object q_empty
 enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
     prefix QTYPE
+module qapi-schema-test.json
 object TestStruct
     member integer: int optional=False
     member boolean: bool optional=False
tests/qapi-schema/test-qapi.py +7 -0
index 4da14b43af..67e417e298 100644
@@ -16,6 +16,13 @@ from qapi.common import QAPIError, QAPISchema, QAPISchemaVisitor
 
 
 class QAPISchemaTestVisitor(QAPISchemaVisitor):
+
+    def visit_module(self, name):
+        print('module %s' % name)
+
+    def visit_include(self, name, info):
+        print('include %s' % name)
+
     def visit_enum_type(self, name, info, values, prefix):
         print('enum %s %s' % (name, values))
         if prefix:
Signature · 1 line
2.13.6
#56 lore raw

Re: [Qemu-devel] [PATCH v2 17/29] qapi: Record 'include' directives in intermediate representation

Quoted 23 lines · Markus Armbruster

The include directive permits modular QAPI schemata, but the generated code is monolithic all the same. To permit generating modular code, the front end needs to pass more information on inclusions to the back ends. The commit before last added the necessary information to the parse tree. This commit adds it to the intermediate representation and its QAPISchemaVisitor. A later commit will use this to to generate modular code.

New entity QAPISchemaInclude represents inclusions. Call new visitor method visit_include() for it, so visitors can see the sub-modules a module includes.

Note that unlike other entities, QAPISchemaInclude has no name, and is therefore not added to entity_dict.

New QAPISchemaEntity attribute @module names the entity's source file. Call new visitor method visit_module() when it changes during a visit, so visitors can keep track of the module being visited.

---

scripts/qapi/common.py class QAPISchema(object): · line 1501

Quoted 6 lines
scripts/qapi/common.py +0 -1
@@ -1479,16 +1497,19 @@ class QAPISchema(object):
          self._entity_dict = {}
          self._predefining = True
          self._def_predefineds()
-        self._predefining = False
          self._def_exprs(exprs)
          self.check()

Why does self._predfining not need to be toggled anymore? Do we even need this variable any more...

scripts/qapi/common.py class QAPISchema(object): · line 1489

Quoted 5 lines
scripts/qapi/common.py +0 -1
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
-        assert ent.name not in self._entity_dict

...and/or is this assert now worthless?

tests/qapi-schema/comments.out line 5

Quoted 7 lines
tests/qapi-schema/comments.out +2 -0
+++ b/tests/qapi-schema/comments.out
@@ -1,4 +1,5 @@
  object q_empty
  enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
      prefix QTYPE
+module comments.json
  enum Status ['good', 'bad', 'ugly']

Based on the generated output, it looks like you can tell whether you are in the predefining stage by not having any module at all; the first visit_module call is what flips the switch that everything else is defined by a module and must therefore have associated info.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#57 lore raw

Re: [Qemu-devel] [PATCH v2 17/29] qapi: Record 'include' directives in intermediate representation

Quoted 43 lines · Eric Blake
Quoted 23 lines · Markus Armbruster

The include directive permits modular QAPI schemata, but the generated code is monolithic all the same. To permit generating modular code, the front end needs to pass more information on inclusions to the back ends. The commit before last added the necessary information to the parse tree. This commit adds it to the intermediate representation and its QAPISchemaVisitor. A later commit will use this to to generate modular code.

New entity QAPISchemaInclude represents inclusions. Call new visitor method visit_include() for it, so visitors can see the sub-modules a module includes.

Note that unlike other entities, QAPISchemaInclude has no name, and is therefore not added to entity_dict.

New QAPISchemaEntity attribute @module names the entity's source file. Call new visitor method visit_module() when it changes during a visit, so visitors can keep track of the module being visited.

---

scripts/qapi/common.py class QAPISchema(object): · line 1501

Quoted 6 lines
scripts/qapi/common.py +0 -1
@@ -1479,16 +1497,19 @@ class QAPISchema(object):
          self._entity_dict = {}
          self._predefining = True
          self._def_predefineds()
-        self._predefining = False
          self._def_exprs(exprs)
          self.check()

Why does self._predfining not need to be toggled anymore? Do we even need this variable any more...

scripts/qapi/common.py class QAPISchema(object): · line 1489

Quoted 5 lines
scripts/qapi/common.py +0 -1
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
-        assert ent.name not in self._entity_dict

...and/or is this assert now worthless?

Quoted 12 lines

tests/qapi-schema/comments.out line 5

Quoted 7 lines
tests/qapi-schema/comments.out +2 -0
+++ b/tests/qapi-schema/comments.out
@@ -1,4 +1,5 @@
  object q_empty
  enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
      prefix QTYPE
+module comments.json
  enum Status ['good', 'bad', 'ugly']

Based on the generated output, it looks like you can tell whether you are in the predefining stage by not having any module at all; the first visit_module call is what flips the switch that everything else is defined by a module and must therefore have associated info.

Editing accident. I started down the road you described, decided I lack the time to reach its end, then failed to back out completely.

#58 lore raw

PATCH v2 18/29 qapi: Rename generated qmp-marshal.c to qmp-commands.c

In reply to Markus Armbruster

All generated .c are named like their .h, except for qmp-marshal.c and qmp-commands.h. To add to the confusion, tests-qmp-commands.c falsely matches generated test-qmp-commands.h.

Get rid of this unnecessary complication.

---
 .gitignore                                     |  3 +--
 Makefile                                       |  6 +++---
 Makefile.objs                                  |  2 +-
 docs/devel/qapi-code-gen.txt                   |  6 +++---
 qga/Makefile.objs                              |  2 +-
 scripts/qapi/commands.py                       |  2 +-
 tests/.gitignore                               |  5 ++---
 tests/Makefile.include                         | 10 +++++-----
 tests/{test-qmp-commands.c => test-qmp-cmds.c} |  0
 9 files changed, 17 insertions(+), 19 deletions(-)
 rename tests/{test-qmp-commands.c => test-qmp-cmds.c} (100%)
.gitignore +1 -2
index 2f9a92f6cc..7d783e6e66 100644
@@ -33,9 +33,8 @@
 /qapi-visit.[ch]
 /qapi-event.[ch]
 /qapi-doc.texi
-/qmp-commands.h
+/qmp-commands.[ch]
 /qmp-introspect.[ch]
-/qmp-marshal.c
 /qemu-doc.html
 /qemu-doc.info
 /qemu-doc.txt
Makefile +3 -3
index bd781c6aad..164a38578e 100644
@@ -91,7 +91,7 @@ include $(SRC_PATH)/rules.mak
 
 GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
 GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
-GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
+GENERATED_FILES += qmp-commands.c qapi-types.c qapi-visit.c qapi-event.c
 GENERATED_FILES += qmp-introspect.h
 GENERATED_FILES += qmp-introspect.c
 GENERATED_FILES += qapi-doc.texi
@@ -496,7 +496,7 @@ $(SRC_PATH)/scripts/qapi-gen.py
 
 qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
 qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
-qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-commands.c \
 qga/qapi-generated/qga-qapi-doc.texi: \
 qga/qapi-generated/qapi-gen-timestamp ;
 qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
@@ -522,7 +522,7 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
 
 qapi-types.c qapi-types.h \
 qapi-visit.c qapi-visit.h \
-qmp-commands.h qmp-marshal.c \
+qmp-commands.h qmp-commands.c \
 qapi-event.c qapi-event.h \
 qmp-introspect.h qmp-introspect.c \
 qapi-doc.texi: \
Makefile.objs +1 -1
index 2efba6d768..d255aaf194 100644
@@ -78,7 +78,7 @@ common-obj-$(CONFIG_FDT) += device_tree.o
 ######################################################################
 # qapi
 
-common-obj-y += qmp-marshal.o
+common-obj-y += qmp-commands.o
 common-obj-y += qmp-introspect.o
 common-obj-y += qmp.o hmp.o
 endif
docs/devel/qapi-code-gen.txt +3 -3
index 1a1cbaea7b..ba1dc73298 100644
@@ -1147,8 +1147,8 @@ declares qmp_COMMAND() that the user must implement.
 
 The following files are generated:
 
-$(prefix)qmp-marshal.c: Command marshal/dispatch functions for each
-                        QMP command defined in the schema
+$(prefix)qmp-commands.c: Command marshal/dispatch functions for each
+                         QMP command defined in the schema
 
 $(prefix)qmp-commands.h: Function prototypes for the QMP commands
                          specified in the schema
@@ -1170,7 +1170,7 @@ Example:
     void qmp_marshal_my_command(QDict *args, QObject **ret, Error **errp);
 
     #endif
-    $ cat qapi-generated/example-qmp-marshal.c
+    $ cat qapi-generated/example-qmp-commands.c
 [Uninteresting stuff omitted...]
 
     static void qmp_marshal_output_UserDefOne(UserDefOne *ret_in, QObject **ret_out, Error **errp)
qga/Makefile.objs +1 -1
index 1c5986c0bb..6151378ae4 100644
@@ -3,6 +3,6 @@ qga-obj-$(CONFIG_POSIX) += commands-posix.o channel-posix.o
 qga-obj-$(CONFIG_WIN32) += commands-win32.o channel-win32.o service-win32.o
 qga-obj-$(CONFIG_WIN32) += vss-win32.o
 qga-obj-y += qapi-generated/qga-qapi-types.o qapi-generated/qga-qapi-visit.o
-qga-obj-y += qapi-generated/qga-qmp-marshal.o
+qga-obj-y += qapi-generated/qga-qmp-commands.o
 
 qga-vss-dll-obj-$(CONFIG_QGA_VSS) += vss-win32/
scripts/qapi/commands.py +1 -1
index a744611d58..05fe33a03b 100644
@@ -289,5 +289,5 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-    genc.write(output_dir, prefix + 'qmp-marshal.c')
+    genc.write(output_dir, prefix + 'qmp-commands.c')
     genh.write(output_dir, prefix + 'qmp-commands.h')
tests/.gitignore +2 -3
index e5c744b7ed..2629cfc2f9 100644
@@ -70,13 +70,12 @@ test-qdist
 test-qga
 test-qht
 test-qht-par
-test-qmp-commands
-test-qmp-commands.h
+test-qmp-cmds
+test-qmp-commands.[ch]
 test-qmp-event
 test-qobject-input-strict
 test-qobject-input-visitor
 test-qmp-introspect.[ch]
-test-qmp-marshal.c
 test-qobject-output-visitor
 test-rcu-list
 test-replication
tests/Makefile.include +5 -5
index b406531b34..9a21ed7ee9 100644
@@ -61,8 +61,8 @@ check-unit-y += tests/test-clone-visitor$(EXESUF)
 gcov-files-test-clone-visitor-y = qapi/qapi-clone-visitor.c
 check-unit-y += tests/test-qobject-input-visitor$(EXESUF)
 gcov-files-test-qobject-input-visitor-y = qapi/qobject-input-visitor.c
-check-unit-y += tests/test-qmp-commands$(EXESUF)
-gcov-files-test-qmp-commands-y = qapi/qmp-dispatch.c
+check-unit-y += tests/test-qmp-cmds$(EXESUF)
+gcov-files-test-qmp-cmds-y = qapi/qmp-dispatch.c
 check-unit-y += tests/test-string-input-visitor$(EXESUF)
 gcov-files-test-string-input-visitor-y = qapi/string-input-visitor.c
 check-unit-y += tests/test-string-output-visitor$(EXESUF)
@@ -576,7 +576,7 @@ test-obj-y = tests/check-qnum.o tests/check-qstring.o tests/check-qdict.o \
 	tests/test-string-input-visitor.o tests/test-qobject-output-visitor.o \
 	tests/test-clone-visitor.o \
 	tests/test-qobject-input-visitor.o \
-	tests/test-qmp-commands.o tests/test-visitor-serialization.o \
+	tests/test-qmp-cmds.o tests/test-visitor-serialization.o \
 	tests/test-x86-cpuid.o tests/test-mul64.o tests/test-int128.o \
 	tests/test-opts-visitor.o tests/test-qmp-event.o \
 	tests/rcutorture.o tests/test-rcu-list.o \
@@ -656,7 +656,7 @@ tests/test-replication$(EXESUF): tests/test-replication.o $(test-util-obj-y) \
 
 tests/test-qapi-types.c tests/test-qapi-types.h \
 tests/test-qapi-visit.c tests/test-qapi-visit.h \
-tests/test-qmp-commands.h tests/test-qmp-marshal.c \
+tests/test-qmp-commands.h tests/test-qmp-commands.c \
 tests/test-qapi-event.c tests/test-qapi-event.h \
 tests/test-qmp-introspect.c tests/test-qmp-introspect.h: \
 tests/test-qapi-gen-timestamp ;
@@ -679,7 +679,7 @@ tests/test-qmp-event$(EXESUF): tests/test-qmp-event.o $(test-qapi-obj-y)
 tests/test-qobject-output-visitor$(EXESUF): tests/test-qobject-output-visitor.o $(test-qapi-obj-y)
 tests/test-clone-visitor$(EXESUF): tests/test-clone-visitor.o $(test-qapi-obj-y)
 tests/test-qobject-input-visitor$(EXESUF): tests/test-qobject-input-visitor.o $(test-qapi-obj-y)
-tests/test-qmp-commands$(EXESUF): tests/test-qmp-commands.o tests/test-qmp-marshal.o $(test-qapi-obj-y)
+tests/test-qmp-cmds$(EXESUF): tests/test-qmp-cmds.o tests/test-qmp-commands.o $(test-qapi-obj-y)
 tests/test-visitor-serialization$(EXESUF): tests/test-visitor-serialization.o $(test-qapi-obj-y)
 tests/test-opts-visitor$(EXESUF): tests/test-opts-visitor.o $(test-qapi-obj-y)
 
tests/test-qmp-cmds.c renamed from tests/test-qmp-commands.c +0 -0
similarity index 100%
rename from tests/test-qmp-commands.c
rename to tests/test-qmp-cmds.c
Signature · 1 line
2.13.6
Reviewed-by on v2 18/29
#59 lore raw

Re: [Qemu-devel] [PATCH v2 18/29] qapi: Rename generated qmp-marshal.c to qmp-commands.c

Quoted 5 lines · Markus Armbruster

All generated .c are named like their .h, except for qmp-marshal.c and qmp-commands.h. To add to the confusion, tests-qmp-commands.c falsely matches generated test-qmp-commands.h.

Get rid of this unnecessary complication.

Yay for saner naming.

tests/test-qmp-cmds.c

Quoted 21 lines
---
  .gitignore                                     |  3 +--
  Makefile                                       |  6 +++---
  Makefile.objs                                  |  2 +-
  docs/devel/qapi-code-gen.txt                   |  6 +++---
  qga/Makefile.objs                              |  2 +-
  scripts/qapi/commands.py                       |  2 +-
  tests/.gitignore                               |  5 ++---
  tests/Makefile.include                         | 10 +++++-----
  tests/{test-qmp-commands.c => test-qmp-cmds.c} |  0
  9 files changed, 17 insertions(+), 19 deletions(-)

rename tests/{test-qmp-commands.c => test-qmp-cmds.c} (100%)

tests/test-qmp-cmds.c renamed from tests/test-qmp-commands.c +0 -0
similarity index 100%
rename from tests/test-qmp-commands.c
rename to tests/test-qmp-cmds.c

Noticed while renaming this file - it's non-trivial in length, but bears no copyright or license clause. Perhaps we should add that (but can be separate patch).

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by Marc-André Lureau on v2 18/29
#60 lore raw

Re: [Qemu-devel] [PATCH v2 18/29] qapi: Rename generated qmp-marshal.c to qmp-commands.c

In reply to Markus Armbruster
Quoted 7 lines · Markus Armbruster

All generated .c are named like their .h, except for qmp-marshal.c and qmp-commands.h. To add to the confusion, tests-qmp-commands.c falsely matches generated test-qmp-commands.h.

Get rid of this unnecessary complication.

that makes more sense,

Quoted 188 lines
---
 .gitignore                                     |  3 +--
 Makefile                                       |  6 +++---
 Makefile.objs                                  |  2 +-
 docs/devel/qapi-code-gen.txt                   |  6 +++---
 qga/Makefile.objs                              |  2 +-
 scripts/qapi/commands.py                       |  2 +-
 tests/.gitignore                               |  5 ++---
 tests/Makefile.include                         | 10 +++++-----
 tests/{test-qmp-commands.c => test-qmp-cmds.c} |  0
 9 files changed, 17 insertions(+), 19 deletions(-)
 rename tests/{test-qmp-commands.c => test-qmp-cmds.c} (100%)
.gitignore +1 -2
index 2f9a92f6cc..7d783e6e66 100644
@@ -33,9 +33,8 @@
 /qapi-visit.[ch]
 /qapi-event.[ch]
 /qapi-doc.texi
-/qmp-commands.h
+/qmp-commands.[ch]
 /qmp-introspect.[ch]
-/qmp-marshal.c
 /qemu-doc.html
 /qemu-doc.info
 /qemu-doc.txt
Makefile +3 -3
index bd781c6aad..164a38578e 100644
@@ -91,7 +91,7 @@ include $(SRC_PATH)/rules.mak
 
 GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
 GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
-GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
+GENERATED_FILES += qmp-commands.c qapi-types.c qapi-visit.c qapi-event.c
 GENERATED_FILES += qmp-introspect.h
 GENERATED_FILES += qmp-introspect.c
 GENERATED_FILES += qapi-doc.texi
@@ -496,7 +496,7 @@ $(SRC_PATH)/scripts/qapi-gen.py
 
 qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
 qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
-qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-commands.c \
 qga/qapi-generated/qga-qapi-doc.texi: \
 qga/qapi-generated/qapi-gen-timestamp ;
 qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
@@ -522,7 +522,7 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
 
 qapi-types.c qapi-types.h \
 qapi-visit.c qapi-visit.h \
-qmp-commands.h qmp-marshal.c \
+qmp-commands.h qmp-commands.c \
 qapi-event.c qapi-event.h \
 qmp-introspect.h qmp-introspect.c \
 qapi-doc.texi: \
Makefile.objs +1 -1
index 2efba6d768..d255aaf194 100644
@@ -78,7 +78,7 @@ common-obj-$(CONFIG_FDT) += device_tree.o
 ######################################################################
 # qapi
 
-common-obj-y += qmp-marshal.o
+common-obj-y += qmp-commands.o
 common-obj-y += qmp-introspect.o
 common-obj-y += qmp.o hmp.o
 endif
docs/devel/qapi-code-gen.txt +3 -3
index 1a1cbaea7b..ba1dc73298 100644
@@ -1147,8 +1147,8 @@ declares qmp_COMMAND() that the user must implement.
 
 The following files are generated:
 
-$(prefix)qmp-marshal.c: Command marshal/dispatch functions for each
-                        QMP command defined in the schema
+$(prefix)qmp-commands.c: Command marshal/dispatch functions for each
+                         QMP command defined in the schema
 
 $(prefix)qmp-commands.h: Function prototypes for the QMP commands
                          specified in the schema
@@ -1170,7 +1170,7 @@ Example:
     void qmp_marshal_my_command(QDict *args, QObject **ret, Error **errp);
 
     #endif
-    $ cat qapi-generated/example-qmp-marshal.c
+    $ cat qapi-generated/example-qmp-commands.c
 [Uninteresting stuff omitted...]
 
     static void qmp_marshal_output_UserDefOne(UserDefOne *ret_in, QObject **ret_out, Error **errp)
qga/Makefile.objs +1 -1
index 1c5986c0bb..6151378ae4 100644
@@ -3,6 +3,6 @@ qga-obj-$(CONFIG_POSIX) += commands-posix.o channel-posix.o
 qga-obj-$(CONFIG_WIN32) += commands-win32.o channel-win32.o service-win32.o
 qga-obj-$(CONFIG_WIN32) += vss-win32.o
 qga-obj-y += qapi-generated/qga-qapi-types.o qapi-generated/qga-qapi-visit.o
-qga-obj-y += qapi-generated/qga-qmp-marshal.o
+qga-obj-y += qapi-generated/qga-qmp-commands.o
 
 qga-vss-dll-obj-$(CONFIG_QGA_VSS) += vss-win32/
scripts/qapi/commands.py +1 -1
index a744611d58..05fe33a03b 100644
@@ -289,5 +289,5 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-    genc.write(output_dir, prefix + 'qmp-marshal.c')
+    genc.write(output_dir, prefix + 'qmp-commands.c')
     genh.write(output_dir, prefix + 'qmp-commands.h')
tests/.gitignore +2 -3
index e5c744b7ed..2629cfc2f9 100644
@@ -70,13 +70,12 @@ test-qdist
 test-qga
 test-qht
 test-qht-par
-test-qmp-commands
-test-qmp-commands.h
+test-qmp-cmds
+test-qmp-commands.[ch]
 test-qmp-event
 test-qobject-input-strict
 test-qobject-input-visitor
 test-qmp-introspect.[ch]
-test-qmp-marshal.c
 test-qobject-output-visitor
 test-rcu-list
 test-replication
tests/Makefile.include +5 -5
index b406531b34..9a21ed7ee9 100644
@@ -61,8 +61,8 @@ check-unit-y += tests/test-clone-visitor$(EXESUF)
 gcov-files-test-clone-visitor-y = qapi/qapi-clone-visitor.c
 check-unit-y += tests/test-qobject-input-visitor$(EXESUF)
 gcov-files-test-qobject-input-visitor-y = qapi/qobject-input-visitor.c
-check-unit-y += tests/test-qmp-commands$(EXESUF)
-gcov-files-test-qmp-commands-y = qapi/qmp-dispatch.c
+check-unit-y += tests/test-qmp-cmds$(EXESUF)
+gcov-files-test-qmp-cmds-y = qapi/qmp-dispatch.c
 check-unit-y += tests/test-string-input-visitor$(EXESUF)
 gcov-files-test-string-input-visitor-y = qapi/string-input-visitor.c
 check-unit-y += tests/test-string-output-visitor$(EXESUF)
@@ -576,7 +576,7 @@ test-obj-y = tests/check-qnum.o tests/check-qstring.o tests/check-qdict.o \
        tests/test-string-input-visitor.o tests/test-qobject-output-visitor.o \
        tests/test-clone-visitor.o \
        tests/test-qobject-input-visitor.o \
-       tests/test-qmp-commands.o tests/test-visitor-serialization.o \
+       tests/test-qmp-cmds.o tests/test-visitor-serialization.o \
        tests/test-x86-cpuid.o tests/test-mul64.o tests/test-int128.o \
        tests/test-opts-visitor.o tests/test-qmp-event.o \
        tests/rcutorture.o tests/test-rcu-list.o \
@@ -656,7 +656,7 @@ tests/test-replication$(EXESUF): tests/test-replication.o $(test-util-obj-y) \
 
 tests/test-qapi-types.c tests/test-qapi-types.h \
 tests/test-qapi-visit.c tests/test-qapi-visit.h \
-tests/test-qmp-commands.h tests/test-qmp-marshal.c \
+tests/test-qmp-commands.h tests/test-qmp-commands.c \
 tests/test-qapi-event.c tests/test-qapi-event.h \
 tests/test-qmp-introspect.c tests/test-qmp-introspect.h: \
 tests/test-qapi-gen-timestamp ;
@@ -679,7 +679,7 @@ tests/test-qmp-event$(EXESUF): tests/test-qmp-event.o $(test-qapi-obj-y)
 tests/test-qobject-output-visitor$(EXESUF): tests/test-qobject-output-visitor.o $(test-qapi-obj-y)
 tests/test-clone-visitor$(EXESUF): tests/test-clone-visitor.o $(test-qapi-obj-y)
 tests/test-qobject-input-visitor$(EXESUF): tests/test-qobject-input-visitor.o $(test-qapi-obj-y)
-tests/test-qmp-commands$(EXESUF): tests/test-qmp-commands.o tests/test-qmp-marshal.o $(test-qapi-obj-y)
+tests/test-qmp-cmds$(EXESUF): tests/test-qmp-cmds.o tests/test-qmp-commands.o $(test-qapi-obj-y)
 tests/test-visitor-serialization$(EXESUF): tests/test-visitor-serialization.o $(test-qapi-obj-y)
 tests/test-opts-visitor$(EXESUF): tests/test-opts-visitor.o $(test-qapi-obj-y)
 
tests/test-qmp-cmds.c renamed from tests/test-qmp-commands.c +0 -1
similarity index 100%
rename from tests/test-qmp-commands.c
rename to tests/test-qmp-cmds.c
--

2.13.6

Reviewed-by on v2 18/29
#61 lore raw

Re: [Qemu-devel] [PATCH v2 18/29] qapi: Rename generated qmp-marshal.c to qmp-commands.c

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:56)

Quoted 7 lines

All generated .c are named like their .h, except for qmp-marshal.c and qmp-commands.h. To add to the confusion, tests-qmp-commands.c falsely matches generated test-qmp-commands.h.

Get rid of this unnecessary complication.

Quoted 188 lines
---
 .gitignore                                     |  3 +--
 Makefile                                       |  6 +++---
 Makefile.objs                                  |  2 +-
 docs/devel/qapi-code-gen.txt                   |  6 +++---
 qga/Makefile.objs                              |  2 +-
 scripts/qapi/commands.py                       |  2 +-
 tests/.gitignore                               |  5 ++---
 tests/Makefile.include                         | 10 +++++-----
 tests/{test-qmp-commands.c => test-qmp-cmds.c} |  0
 9 files changed, 17 insertions(+), 19 deletions(-)
 rename tests/{test-qmp-commands.c => test-qmp-cmds.c} (100%)
.gitignore +1 -2
index 2f9a92f6cc..7d783e6e66 100644
@@ -33,9 +33,8 @@
 /qapi-visit.[ch]
 /qapi-event.[ch]
 /qapi-doc.texi
-/qmp-commands.h
+/qmp-commands.[ch]
 /qmp-introspect.[ch]
-/qmp-marshal.c
 /qemu-doc.html
 /qemu-doc.info
 /qemu-doc.txt
Makefile +3 -3
index bd781c6aad..164a38578e 100644
@@ -91,7 +91,7 @@ include $(SRC_PATH)/rules.mak
 
 GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
 GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
-GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
+GENERATED_FILES += qmp-commands.c qapi-types.c qapi-visit.c qapi-event.c
 GENERATED_FILES += qmp-introspect.h
 GENERATED_FILES += qmp-introspect.c
 GENERATED_FILES += qapi-doc.texi
@@ -496,7 +496,7 @@ $(SRC_PATH)/scripts/qapi-gen.py
 
 qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
 qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
-qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-commands.c \
 qga/qapi-generated/qga-qapi-doc.texi: \
 qga/qapi-generated/qapi-gen-timestamp ;
 qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
@@ -522,7 +522,7 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
 
 qapi-types.c qapi-types.h \
 qapi-visit.c qapi-visit.h \
-qmp-commands.h qmp-marshal.c \
+qmp-commands.h qmp-commands.c \
 qapi-event.c qapi-event.h \
 qmp-introspect.h qmp-introspect.c \
 qapi-doc.texi: \
Makefile.objs +1 -1
index 2efba6d768..d255aaf194 100644
@@ -78,7 +78,7 @@ common-obj-$(CONFIG_FDT) += device_tree.o
 ######################################################################
 # qapi
 
-common-obj-y += qmp-marshal.o
+common-obj-y += qmp-commands.o
 common-obj-y += qmp-introspect.o
 common-obj-y += qmp.o hmp.o
 endif
docs/devel/qapi-code-gen.txt +3 -3
index 1a1cbaea7b..ba1dc73298 100644
@@ -1147,8 +1147,8 @@ declares qmp_COMMAND() that the user must implement.
 
 The following files are generated:
 
-$(prefix)qmp-marshal.c: Command marshal/dispatch functions for each
-                        QMP command defined in the schema
+$(prefix)qmp-commands.c: Command marshal/dispatch functions for each
+                         QMP command defined in the schema
 
 $(prefix)qmp-commands.h: Function prototypes for the QMP commands
                          specified in the schema
@@ -1170,7 +1170,7 @@ Example:
     void qmp_marshal_my_command(QDict *args, QObject **ret, Error **errp);
 
     #endif
-    $ cat qapi-generated/example-qmp-marshal.c
+    $ cat qapi-generated/example-qmp-commands.c
 [Uninteresting stuff omitted...]
 
     static void qmp_marshal_output_UserDefOne(UserDefOne *ret_in, QObject **ret_out, Error **errp)
qga/Makefile.objs +1 -1
index 1c5986c0bb..6151378ae4 100644
@@ -3,6 +3,6 @@ qga-obj-$(CONFIG_POSIX) += commands-posix.o channel-posix.o
 qga-obj-$(CONFIG_WIN32) += commands-win32.o channel-win32.o service-win32.o
 qga-obj-$(CONFIG_WIN32) += vss-win32.o
 qga-obj-y += qapi-generated/qga-qapi-types.o qapi-generated/qga-qapi-visit.o
-qga-obj-y += qapi-generated/qga-qmp-marshal.o
+qga-obj-y += qapi-generated/qga-qmp-commands.o
 
 qga-vss-dll-obj-$(CONFIG_QGA_VSS) += vss-win32/
scripts/qapi/commands.py +1 -1
index a744611d58..05fe33a03b 100644
@@ -289,5 +289,5 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
     schema.visit(vis)
     genc.add(vis.defn)
     genh.add(vis.decl)
-    genc.write(output_dir, prefix + 'qmp-marshal.c')
+    genc.write(output_dir, prefix + 'qmp-commands.c')
     genh.write(output_dir, prefix + 'qmp-commands.h')
tests/.gitignore +2 -3
index e5c744b7ed..2629cfc2f9 100644
@@ -70,13 +70,12 @@ test-qdist
 test-qga
 test-qht
 test-qht-par
-test-qmp-commands
-test-qmp-commands.h
+test-qmp-cmds
+test-qmp-commands.[ch]
 test-qmp-event
 test-qobject-input-strict
 test-qobject-input-visitor
 test-qmp-introspect.[ch]
-test-qmp-marshal.c
 test-qobject-output-visitor
 test-rcu-list
 test-replication
tests/Makefile.include +5 -5
index b406531b34..9a21ed7ee9 100644
@@ -61,8 +61,8 @@ check-unit-y += tests/test-clone-visitor$(EXESUF)
 gcov-files-test-clone-visitor-y = qapi/qapi-clone-visitor.c
 check-unit-y += tests/test-qobject-input-visitor$(EXESUF)
 gcov-files-test-qobject-input-visitor-y = qapi/qobject-input-visitor.c
-check-unit-y += tests/test-qmp-commands$(EXESUF)
-gcov-files-test-qmp-commands-y = qapi/qmp-dispatch.c
+check-unit-y += tests/test-qmp-cmds$(EXESUF)
+gcov-files-test-qmp-cmds-y = qapi/qmp-dispatch.c
 check-unit-y += tests/test-string-input-visitor$(EXESUF)
 gcov-files-test-string-input-visitor-y = qapi/string-input-visitor.c
 check-unit-y += tests/test-string-output-visitor$(EXESUF)
@@ -576,7 +576,7 @@ test-obj-y = tests/check-qnum.o tests/check-qstring.o tests/check-qdict.o \
        tests/test-string-input-visitor.o tests/test-qobject-output-visitor.o \
        tests/test-clone-visitor.o \
        tests/test-qobject-input-visitor.o \
-       tests/test-qmp-commands.o tests/test-visitor-serialization.o \
+       tests/test-qmp-cmds.o tests/test-visitor-serialization.o \
        tests/test-x86-cpuid.o tests/test-mul64.o tests/test-int128.o \
        tests/test-opts-visitor.o tests/test-qmp-event.o \
        tests/rcutorture.o tests/test-rcu-list.o \
@@ -656,7 +656,7 @@ tests/test-replication$(EXESUF): tests/test-replication.o $(test-util-obj-y) \
 
 tests/test-qapi-types.c tests/test-qapi-types.h \
 tests/test-qapi-visit.c tests/test-qapi-visit.h \
-tests/test-qmp-commands.h tests/test-qmp-marshal.c \
+tests/test-qmp-commands.h tests/test-qmp-commands.c \
 tests/test-qapi-event.c tests/test-qapi-event.h \
 tests/test-qmp-introspect.c tests/test-qmp-introspect.h: \
 tests/test-qapi-gen-timestamp ;
@@ -679,7 +679,7 @@ tests/test-qmp-event$(EXESUF): tests/test-qmp-event.o $(test-qapi-obj-y)
 tests/test-qobject-output-visitor$(EXESUF): tests/test-qobject-output-visitor.o $(test-qapi-obj-y)
 tests/test-clone-visitor$(EXESUF): tests/test-clone-visitor.o $(test-qapi-obj-y)
 tests/test-qobject-input-visitor$(EXESUF): tests/test-qobject-input-visitor.o $(test-qapi-obj-y)
-tests/test-qmp-commands$(EXESUF): tests/test-qmp-commands.o tests/test-qmp-marshal.o $(test-qapi-obj-y)
+tests/test-qmp-cmds$(EXESUF): tests/test-qmp-cmds.o tests/test-qmp-commands.o $(test-qapi-obj-y)
 tests/test-visitor-serialization$(EXESUF): tests/test-visitor-serialization.o $(test-qapi-obj-y)
 tests/test-opts-visitor$(EXESUF): tests/test-opts-visitor.o $(test-qapi-obj-y)
 
tests/test-qmp-cmds.c renamed from tests/test-qmp-commands.c +0 -0
similarity index 100%
rename from tests/test-qmp-commands.c
rename to tests/test-qmp-cmds.c
Signature · 1 line
2.13.6
#62 lore raw

PATCH v2 19/29 qapi: Make code-generating visitors use QAPIGen more

In reply to Markus Armbruster

The use of QAPIGen is rather shallow so far: most of the output accumulation is not converted. Take the next step: convert output accumulation in the code-generating visitor classes. Helper functions outside these classes are not converted.

---
 scripts/qapi/commands.py   | 71 ++++++++++++++++------------------------
 scripts/qapi/common.py     | 13 ++++++++
 scripts/qapi/doc.py        | 74 ++++++++++++++++++++----------------------
 scripts/qapi/events.py     | 55 ++++++++++++-------------------
 scripts/qapi/introspect.py | 56 +++++++++++++-------------------
 scripts/qapi/types.py      | 81 +++++++++++++++++++---------------------------
 scripts/qapi/visit.py      | 80 +++++++++++++++++++--------------------------
 7 files changed, 188 insertions(+), 242 deletions(-)
scripts/qapi/commands.py +28 -43
index 05fe33a03b..46757db771 100644
@@ -223,44 +223,15 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
     return ret
 
 
-class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
+class QAPISchemaGenCommandVisitor(QAPISchemaMonolithicCVisitor):
+
     def __init__(self, prefix):
-        self._prefix = prefix
-        self.decl = None
-        self.defn = None
-        self._regy = None
-        self._visited_ret_types = None
-
-    def visit_begin(self, schema):
-        self.decl = ''
-        self.defn = ''
+        QAPISchemaMonolithicCVisitor.__init__(
+            self, prefix, 'qmp-commands',
+            ' * Schema-defined QAPI/QMP commands', __doc__)
         self._regy = ''
         self._visited_ret_types = set()
-
-    def visit_end(self):
-        self.defn += gen_registry(self._regy, self._prefix)
-        self._regy = None
-        self._visited_ret_types = None
-
-    def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
-        if not gen:
-            return
-        self.decl += gen_command_decl(name, arg_type, boxed, ret_type)
-        if ret_type and ret_type not in self._visited_ret_types:
-            self._visited_ret_types.add(ret_type)
-            self.defn += gen_marshal_output(ret_type)
-        self.decl += gen_marshal_decl(name)
-        self.defn += gen_marshal(name, arg_type, boxed, ret_type)
-        self._regy += gen_register_command(name, success_response)
-
-
-def gen_commands(schema, output_dir, prefix):
-    blurb = ' * Schema-defined QAPI/QMP commands'
-    genc = QAPIGenC(blurb, __doc__)
-    genh = QAPIGenH(blurb, __doc__)
-
-    genc.add(mcgen('''
+        self._genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -275,19 +246,33 @@ def gen_commands(schema, output_dir, prefix):
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                   prefix=prefix))
-
-    genh.add(mcgen('''
+                             prefix=prefix))
+        self._genh.add(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-                   prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+                             prefix=prefix,
+                             c_prefix=c_name(prefix, protect=False)))
 
+    def visit_end(self):
+        self._genc.add(gen_registry(self._regy, self._prefix))
+
+    def visit_command(self, name, info, arg_type, ret_type,
+                      gen, success_response, boxed):
+        if not gen:
+            return
+        self._genh.add(gen_command_decl(name, arg_type, boxed, ret_type))
+        if ret_type and ret_type not in self._visited_ret_types:
+            self._visited_ret_types.add(ret_type)
+            self._genc.add(gen_marshal_output(ret_type))
+        self._genh.add(gen_marshal_decl(name))
+        self._genc.add(gen_marshal(name, arg_type, boxed, ret_type))
+        self._regy += gen_register_command(name, success_response)
+
+
+def gen_commands(schema, output_dir, prefix):
     vis = QAPISchemaGenCommandVisitor(prefix)
     schema.visit(vis)
-    genc.add(vis.defn)
-    genh.add(vis.decl)
-    genc.write(output_dir, prefix + 'qmp-commands.c')
-    genh.write(output_dir, prefix + 'qmp-commands.h')
+    vis.write(output_dir)
scripts/qapi/common.py +13 -0
index 29d98ca934..31d2f73e7e 100644
@@ -2049,3 +2049,16 @@ class QAPIGenDoc(QAPIGen):
     def _top(self, fname):
         return (QAPIGen._top(self, fname)
                 + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')
+
+
+class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
+
+    def __init__(self, prefix, what, blurb, pydoc):
+        self._prefix = prefix
+        self._what = what
+        self._genc = QAPIGenC(blurb, pydoc)
+        self._genh = QAPIGenH(blurb, pydoc)
+
+    def write(self, output_dir):
+        self._genc.write(output_dir, self._prefix + self._what + '.c')
+        self._genh.write(output_dir, self._prefix + self._what + '.h')
scripts/qapi/doc.py +35 -39

112 lines not on this page · raw message

scripts/qapi/events.py +21 -34

74 lines not on this page · raw message

scripts/qapi/introspect.py +23 -33

86 lines not on this page · raw message

scripts/qapi/types.py +34 -47

131 lines not on this page · raw message

scripts/qapi/visit.py +34 -46

120 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v2 19/29
#63 lore raw

Re: [Qemu-devel] [PATCH v2 19/29] qapi: Make code-generating visitors use QAPIGen more

Quoted 6 lines · Markus Armbruster

The use of QAPIGen is rather shallow so far: most of the output accumulation is not converted. Take the next step: convert output accumulation in the code-generating visitor classes. Helper functions outside these classes are not converted.

Quoted 10 lines
---
  scripts/qapi/commands.py   | 71 ++++++++++++++++------------------------
  scripts/qapi/common.py     | 13 ++++++++
  scripts/qapi/doc.py        | 74 ++++++++++++++++++++----------------------
  scripts/qapi/events.py     | 55 ++++++++++++-------------------
  scripts/qapi/introspect.py | 56 +++++++++++++-------------------
  scripts/qapi/types.py      | 81 +++++++++++++++++++---------------------------
  scripts/qapi/visit.py      | 80 +++++++++++++++++++--------------------------
  7 files changed, 188 insertions(+), 242 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Reviewed-by Marc-André Lureau on v2 19/29
#64 lore raw

Re: [Qemu-devel] [PATCH v2 19/29] qapi: Make code-generating visitors use QAPIGen more

In reply to Markus Armbruster
Quoted 6 lines · Markus Armbruster

The use of QAPIGen is rather shallow so far: most of the output accumulation is not converted. Take the next step: convert output accumulation in the code-generating visitor classes. Helper functions outside these classes are not converted.

Quoted 689 lines
---
 scripts/qapi/commands.py   | 71 ++++++++++++++++------------------------
 scripts/qapi/common.py     | 13 ++++++++
 scripts/qapi/doc.py        | 74 ++++++++++++++++++++----------------------
 scripts/qapi/events.py     | 55 ++++++++++++-------------------
 scripts/qapi/introspect.py | 56 +++++++++++++-------------------
 scripts/qapi/types.py      | 81 +++++++++++++++++++---------------------------
 scripts/qapi/visit.py      | 80 +++++++++++++++++++--------------------------
 7 files changed, 188 insertions(+), 242 deletions(-)
scripts/qapi/commands.py +28 -43
index 05fe33a03b..46757db771 100644
@@ -223,44 +223,15 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
     return ret
 
 
-class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
+class QAPISchemaGenCommandVisitor(QAPISchemaMonolithicCVisitor):
+
     def __init__(self, prefix):
-        self._prefix = prefix
-        self.decl = None
-        self.defn = None
-        self._regy = None
-        self._visited_ret_types = None
-
-    def visit_begin(self, schema):
-        self.decl = ''
-        self.defn = ''
+        QAPISchemaMonolithicCVisitor.__init__(
+            self, prefix, 'qmp-commands',
+            ' * Schema-defined QAPI/QMP commands', __doc__)
         self._regy = ''
         self._visited_ret_types = set()
-
-    def visit_end(self):
-        self.defn += gen_registry(self._regy, self._prefix)
-        self._regy = None
-        self._visited_ret_types = None
-
-    def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
-        if not gen:
-            return
-        self.decl += gen_command_decl(name, arg_type, boxed, ret_type)
-        if ret_type and ret_type not in self._visited_ret_types:
-            self._visited_ret_types.add(ret_type)
-            self.defn += gen_marshal_output(ret_type)
-        self.decl += gen_marshal_decl(name)
-        self.defn += gen_marshal(name, arg_type, boxed, ret_type)
-        self._regy += gen_register_command(name, success_response)
-
-
-def gen_commands(schema, output_dir, prefix):
-    blurb = ' * Schema-defined QAPI/QMP commands'
-    genc = QAPIGenC(blurb, __doc__)
-    genh = QAPIGenH(blurb, __doc__)
-
-    genc.add(mcgen('''
+        self._genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qemu/module.h"
@@ -275,19 +246,33 @@ def gen_commands(schema, output_dir, prefix):
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                   prefix=prefix))
-
-    genh.add(mcgen('''
+                             prefix=prefix))
+        self._genh.add(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/dispatch.h"
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-                   prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+                             prefix=prefix,
+                             c_prefix=c_name(prefix, protect=False)))
 
+    def visit_end(self):
+        self._genc.add(gen_registry(self._regy, self._prefix))
+
+    def visit_command(self, name, info, arg_type, ret_type,
+                      gen, success_response, boxed):
+        if not gen:
+            return
+        self._genh.add(gen_command_decl(name, arg_type, boxed, ret_type))
+        if ret_type and ret_type not in self._visited_ret_types:
+            self._visited_ret_types.add(ret_type)
+            self._genc.add(gen_marshal_output(ret_type))
+        self._genh.add(gen_marshal_decl(name))
+        self._genc.add(gen_marshal(name, arg_type, boxed, ret_type))
+        self._regy += gen_register_command(name, success_response)
+
+
+def gen_commands(schema, output_dir, prefix):
     vis = QAPISchemaGenCommandVisitor(prefix)
     schema.visit(vis)
-    genc.add(vis.defn)
-    genh.add(vis.decl)
-    genc.write(output_dir, prefix + 'qmp-commands.c')
-    genh.write(output_dir, prefix + 'qmp-commands.h')
+    vis.write(output_dir)
scripts/qapi/common.py +13 -0
index 29d98ca934..31d2f73e7e 100644
@@ -2049,3 +2049,16 @@ class QAPIGenDoc(QAPIGen):
     def _top(self, fname):
         return (QAPIGen._top(self, fname)
                 + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')
+
+
+class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
+
+    def __init__(self, prefix, what, blurb, pydoc):
+        self._prefix = prefix
+        self._what = what
+        self._genc = QAPIGenC(blurb, pydoc)
+        self._genh = QAPIGenH(blurb, pydoc)
+
+    def write(self, output_dir):
+        self._genc.write(output_dir, self._prefix + self._what + '.c')
+        self._genh.write(output_dir, self._prefix + self._what + '.h')
scripts/qapi/doc.py +35 -39
index cc4d5a43fb..0ea68bf813 100644
@@ -197,33 +197,35 @@ def texi_entity(doc, what, base=None, variants=None,
 
 
 class QAPISchemaGenDocVisitor(qapi.common.QAPISchemaVisitor):
-    def __init__(self):
-        self.out = None
+    def __init__(self, prefix):
+        self._prefix = prefix
+        self._gen = qapi.common.QAPIGenDoc()
         self.cur_doc = None
 
-    def visit_begin(self, schema):
-        self.out = ''
+    def write(self, output_dir):
+        self._gen.write(output_dir, self._prefix + 'qapi-doc.texi')
 
     def visit_enum_type(self, name, info, values, prefix):
         doc = self.cur_doc
-        self.out += TYPE_FMT(type='Enum',
-                             name=doc.symbol,
-                             body=texi_entity(doc, 'Values',
-                                              member_func=texi_enum_value))
+        self._gen.add(TYPE_FMT(type='Enum',
+                               name=doc.symbol,
+                               body=texi_entity(doc, 'Values',
+                                                member_func=texi_enum_value)))
 
     def visit_object_type(self, name, info, base, members, variants):
         doc = self.cur_doc
         if base and base.is_implicit():
             base = None
-        self.out += TYPE_FMT(type='Object',
-                             name=doc.symbol,
-                             body=texi_entity(doc, 'Members', base, variants))
+        self._gen.add(TYPE_FMT(type='Object',
+                               name=doc.symbol,
+                               body=texi_entity(doc, 'Members',
+                                                base, variants)))
 
     def visit_alternate_type(self, name, info, variants):
         doc = self.cur_doc
-        self.out += TYPE_FMT(type='Alternate',
-                             name=doc.symbol,
-                             body=texi_entity(doc, 'Members'))
+        self._gen.add(TYPE_FMT(type='Alternate',
+                               name=doc.symbol,
+                               body=texi_entity(doc, 'Members')))
 
     def visit_command(self, name, info, arg_type, ret_type,
                       gen, success_response, boxed):
@@ -235,44 +237,38 @@ class QAPISchemaGenDocVisitor(qapi.common.QAPISchemaVisitor):
             body += texi_sections(doc)
         else:
             body = texi_entity(doc, 'Arguments')
-        self.out += MSG_FMT(type='Command',
-                            name=doc.symbol,
-                            body=body)
+        self._gen.add(MSG_FMT(type='Command',
+                              name=doc.symbol,
+                              body=body))
 
     def visit_event(self, name, info, arg_type, boxed):
         doc = self.cur_doc
-        self.out += MSG_FMT(type='Event',
-                            name=doc.symbol,
-                            body=texi_entity(doc, 'Arguments'))
+        self._gen.add(MSG_FMT(type='Event',
+                              name=doc.symbol,
+                              body=texi_entity(doc, 'Arguments')))
 
     def symbol(self, doc, entity):
-        if self.out:
-            self.out += '\n'
+        if self._gen._body:
+            self._gen.add('\n')
         self.cur_doc = doc
         entity.visit(self)
         self.cur_doc = None
 
     def freeform(self, doc):
         assert not doc.args
-        if self.out:
-            self.out += '\n'
-        self.out += texi_body(doc) + texi_sections(doc)
+        if self._gen._body:
+            self._gen.add('\n')
+        self._gen.add(texi_body(doc) + texi_sections(doc))
 
 
-def texi_schema(schema):
-    """Convert QAPI schema documentation to Texinfo"""
-    gen = QAPISchemaGenDocVisitor()
-    gen.visit_begin(schema)
+def gen_doc(schema, output_dir, prefix):
+    if not qapi.common.doc_required:
+        return
+    vis = QAPISchemaGenDocVisitor(prefix)
+    vis.visit_begin(schema)
     for doc in schema.docs:
         if doc.symbol:
-            gen.symbol(doc, schema.lookup_entity(doc.symbol))
+            vis.symbol(doc, schema.lookup_entity(doc.symbol))
         else:
-            gen.freeform(doc)
-    return gen.out
-
-
-def gen_doc(schema, output_dir, prefix):
-    if qapi.common.doc_required:
-        gen = qapi.common.QAPIGenDoc()
-        gen.add(texi_schema(schema))
-        gen.write(output_dir, prefix + 'qapi-doc.texi')
+            vis.freeform(doc)
+    vis.write(output_dir)
scripts/qapi/events.py +21 -34
index b7dc82004f..81ab3abb30 100644
@@ -148,35 +148,15 @@ out:
     return ret
 
 
-class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
+class QAPISchemaGenEventVisitor(QAPISchemaMonolithicCVisitor):
+
     def __init__(self, prefix):
+        QAPISchemaMonolithicCVisitor.__init__(
+            self, prefix, 'qapi-event',
+            ' * Schema-defined QAPI/QMP events', __doc__)
         self._enum_name = c_name(prefix + 'QAPIEvent', protect=False)
-        self.decl = None
-        self.defn = None
-        self._event_names = None
-
-    def visit_begin(self, schema):
-        self.decl = ''
-        self.defn = ''
         self._event_names = []
-
-    def visit_end(self):
-        self.decl += gen_enum(self._enum_name, self._event_names)
-        self.defn += gen_enum_lookup(self._enum_name, self._event_names)
-        self._event_names = None
-
-    def visit_event(self, name, info, arg_type, boxed):
-        self.decl += gen_event_send_decl(name, arg_type, boxed)
-        self.defn += gen_event_send(name, arg_type, boxed, self._enum_name)
-        self._event_names.append(name)
-
-
-def gen_events(schema, output_dir, prefix):
-    blurb = ' * Schema-defined QAPI/QMP events'
-    genc = QAPIGenC(blurb, __doc__)
-    genh = QAPIGenH(blurb, __doc__)
-
-    genc.add(mcgen('''
+        self._genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "%(prefix)sqapi-event.h"
@@ -187,18 +167,25 @@ def gen_events(schema, output_dir, prefix):
 #include "qapi/qmp-event.h"
 
 ''',
-                   prefix=prefix))
-
-    genh.add(mcgen('''
+                             prefix=prefix))
+        self._genh.add(mcgen('''
 #include "qapi/util.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                   prefix=prefix))
+                             prefix=prefix))
 
+    def visit_end(self):
+        self._genh.add(gen_enum(self._enum_name, self._event_names))
+        self._genc.add(gen_enum_lookup(self._enum_name, self._event_names))
+
+    def visit_event(self, name, info, arg_type, boxed):
+        self._genh.add(gen_event_send_decl(name, arg_type, boxed))
+        self._genc.add(gen_event_send(name, arg_type, boxed, self._enum_name))
+        self._event_names.append(name)
+
+
+def gen_events(schema, output_dir, prefix):
     vis = QAPISchemaGenEventVisitor(prefix)
     schema.visit(vis)
-    genc.add(vis.defn)
-    genh.add(vis.decl)
-    genc.write(output_dir, prefix + 'qapi-event.c')
-    genh.write(output_dir, prefix + 'qapi-event.h')
+    vis.write(output_dir)
scripts/qapi/introspect.py +23 -33
index 1e4f065164..f571cc134c 100644
@@ -40,22 +40,26 @@ def to_c_string(string):
     return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"'
 
 
-class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
+class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor):
+
     def __init__(self, prefix, unmask):
-        self._prefix = prefix
+        QAPISchemaMonolithicCVisitor.__init__(
+            self, prefix, 'qmp-introspect',
+            ' * QAPI/QMP schema introspection', __doc__)
         self._unmask = unmask
-        self.defn = None
-        self.decl = None
         self._schema = None
-        self._jsons = None
-        self._used_types = None
-        self._name_map = None
+        self._jsons = []
+        self._used_types = []
+        self._name_map = {}
+        self._genc.add(mcgen('''
+#include "qemu/osdep.h"
+#include "%(prefix)sqmp-introspect.h"
+
+''',
+                             prefix=prefix))
 
     def visit_begin(self, schema):
         self._schema = schema
-        self._jsons = []
-        self._used_types = []
-        self._name_map = {}
 
     def visit_end(self):
         # visit the types that are actually used
@@ -67,21 +71,21 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
         # TODO can generate awfully long lines
         jsons.extend(self._jsons)
         name = c_name(self._prefix, protect=False) + 'qmp_schema_json'
-        self.decl = mcgen('''
+        self._genh.add(mcgen('''
 extern const char %(c_name)s[];
 ''',
-                          c_name=c_name(name))
+                             c_name=c_name(name)))
         lines = to_json(jsons).split('\n')
         c_string = '\n    '.join([to_c_string(line) for line in lines])
-        self.defn = mcgen('''
+        self._genc.add(mcgen('''
 const char %(c_name)s[] = %(c_string)s;
 ''',
-                          c_name=c_name(name),
-                          c_string=c_string)
+                             c_name=c_name(name),
+                             c_string=c_string))
         self._schema = None
-        self._jsons = None
-        self._used_types = None
-        self._name_map = None
+        self._jsons = []
+        self._used_types = []
+        self._name_map = {}
 
     def visit_needed(self, entity):
         # Ignore types on first pass; visit_end() will pick up used types
@@ -169,20 +173,6 @@ const char %(c_name)s[] = %(c_string)s;
 
 
 def gen_introspect(schema, output_dir, prefix, opt_unmask):
-    blurb = ' * QAPI/QMP schema introspection'
-    genc = QAPIGenC(blurb, __doc__)
-    genh = QAPIGenH(blurb, __doc__)
-
-    genc.add(mcgen('''
-#include "qemu/osdep.h"
-#include "%(prefix)sqmp-introspect.h"
-
-''',
-                   prefix=prefix))
-
     vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask)
     schema.visit(vis)
-    genc.add(vis.defn)
-    genh.add(vis.decl)
-    genc.write(output_dir, prefix + 'qmp-introspect.c')
-    genh.write(output_dir, prefix + 'qmp-introspect.h')
+    vis.write(output_dir)
scripts/qapi/types.py +34 -47
index aa3c01e750..d2b8423479 100644
@@ -167,36 +167,41 @@ void qapi_free_%(c_name)s(%(c_name)s *obj)
     return ret
 
 
-class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
-    def __init__(self, opt_builtins):
+class QAPISchemaGenTypeVisitor(QAPISchemaMonolithicCVisitor):
+
+    def __init__(self, prefix, opt_builtins):
+        QAPISchemaMonolithicCVisitor.__init__(
+            self, prefix, 'qapi-types', ' * Schema-defined QAPI types',
+            __doc__)
         self._opt_builtins = opt_builtins
-        self.decl = None
-        self.defn = None
-        self._fwdecl = None
-        self._btin = None
+        self._genc.preamble_add(mcgen('''
+#include "qemu/osdep.h"
+#include "qapi/dealloc-visitor.h"
+#include "%(prefix)sqapi-types.h"
+#include "%(prefix)sqapi-visit.h"
+''',
+                                      prefix=prefix))
+        self._genh.preamble_add(mcgen('''
+#include "qapi/util.h"
+'''))
+        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
 
     def visit_begin(self, schema):
         # gen_object() is recursive, ensure it doesn't visit the empty type
         objects_seen.add(schema.the_empty_object_type.name)
-        self.decl = ''
-        self.defn = ''
-        self._fwdecl = ''
-        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
 
     def visit_end(self):
-        self.decl = self._fwdecl + self.decl
-        self._fwdecl = None
         # To avoid header dependency hell, we always generate
         # declarations for built-in types in our header files and
         # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_TYPES_BUILTIN')
-        self.decl = self._btin + self.decl
+        self._genh.preamble_add(self._btin)
         self._btin = None
 
     def _gen_type_cleanup(self, name):
-        self.decl += gen_type_cleanup_decl(name)
-        self.defn += gen_type_cleanup(name)
+        self._genh.add(gen_type_cleanup_decl(name))
+        self._genc.add(gen_type_cleanup(name))
 
     def visit_enum_type(self, name, info, values, prefix):
         # Special case for our lone builtin enum type
@@ -204,10 +209,10 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         if not info:
             self._btin += gen_enum(name, values, prefix)
             if self._opt_builtins:
-                self.defn += gen_enum_lookup(name, values, prefix)
+                self._genc.add(gen_enum_lookup(name, values, prefix))
         else:
-            self._fwdecl += gen_enum(name, values, prefix)
-            self.defn += gen_enum_lookup(name, values, prefix)
+            self._genh.preamble_add(gen_enum(name, values, prefix))
+            self._genc.add(gen_enum_lookup(name, values, prefix))
 
     def visit_array_type(self, name, info, element_type):
         if isinstance(element_type, QAPISchemaBuiltinType):
@@ -215,20 +220,20 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
             self._btin += gen_array(name, element_type)
             self._btin += gen_type_cleanup_decl(name)
             if self._opt_builtins:
-                self.defn += gen_type_cleanup(name)
+                self._genc.add(gen_type_cleanup(name))
         else:
-            self._fwdecl += gen_fwd_object_or_array(name)
-            self.decl += gen_array(name, element_type)
+            self._genh.preamble_add(gen_fwd_object_or_array(name))
+            self._genh.add(gen_array(name, element_type))
             self._gen_type_cleanup(name)
 
     def visit_object_type(self, name, info, base, members, variants):
         # Nothing to do for the special empty builtin
         if name == 'q_empty':
             return
-        self._fwdecl += gen_fwd_object_or_array(name)
-        self.decl += gen_object(name, base, members, variants)
+        self._genh.preamble_add(gen_fwd_object_or_array(name))
+        self._genh.add(gen_object(name, base, members, variants))
         if base and not base.is_implicit():
-            self.decl += gen_upcast(name, base)
+            self._genh.add(gen_upcast(name, base))
         # TODO Worth changing the visitor signature, so we could
         # directly use rather than repeat type.is_implicit()?
         if not name.startswith('q_'):
@@ -236,31 +241,13 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
             self._gen_type_cleanup(name)
 
     def visit_alternate_type(self, name, info, variants):
-        self._fwdecl += gen_fwd_object_or_array(name)
-        self.decl += gen_object(name, None, [variants.tag_member], variants)
+        self._genh.preamble_add(gen_fwd_object_or_array(name))
+        self._genh.add(gen_object(name, None,
+                                  [variants.tag_member], variants))
         self._gen_type_cleanup(name)
 
 
 def gen_types(schema, output_dir, prefix, opt_builtins):
-    blurb = ' * Schema-defined QAPI types'
-    genc = QAPIGenC(blurb, __doc__)
-    genh = QAPIGenH(blurb, __doc__)
-
-    genc.add(mcgen('''
-#include "qemu/osdep.h"
-#include "qapi/dealloc-visitor.h"
-#include "%(prefix)sqapi-types.h"
-#include "%(prefix)sqapi-visit.h"
-''',
-                   prefix=prefix))
-
-    genh.add(mcgen('''
-#include "qapi/util.h"
-'''))
-
-    vis = QAPISchemaGenTypeVisitor(opt_builtins)
+    vis = QAPISchemaGenTypeVisitor(prefix, opt_builtins)
     schema.visit(vis)
-    genc.add(vis.defn)
-    genh.add(vis.decl)
-    genc.write(output_dir, prefix + 'qapi-types.c')
-    genh.write(output_dir, prefix + 'qapi-types.h')
+    vis.write(output_dir)
scripts/qapi/visit.py +34 -47
index f966e7975f..3d09d44265 100644
@@ -263,16 +263,27 @@ out:
                  c_name=c_name(name))
 
 
-class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
-    def __init__(self, opt_builtins):
+class QAPISchemaGenVisitVisitor(QAPISchemaMonolithicCVisitor):
+
+    def __init__(self, prefix, opt_builtins):
+        QAPISchemaMonolithicCVisitor.__init__(
+            self, prefix, 'qapi-visit', ' * Schema-defined QAPI visitors',
+            __doc__)
         self._opt_builtins = opt_builtins
-        self.decl = None
-        self.defn = None
-        self._btin = None
+        self._genc.preamble_add(mcgen('''
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
+#include "%(prefix)sqapi-visit.h"
+''',
+                                      prefix=prefix))
+        self._genh.preamble_add(mcgen('''
+#include "qapi/visitor.h"
+#include "%(prefix)sqapi-types.h"
 
-    def visit_begin(self, schema):
-        self.decl = ''
-        self.defn = ''
+''',
+                                      prefix=prefix))
         self._btin = guardstart('QAPI_VISIT_BUILTIN')
 
     def visit_end(self):
@@ -281,7 +292,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         # simply guard them.  See also opt_builtins (command line
         # option -b).
         self._btin += guardend('QAPI_VISIT_BUILTIN')
-        self.decl = self._btin + self.decl
+        self._genh.preamble_add(self._btin)
         self._btin = None
 
     def visit_enum_type(self, name, info, values, prefix):
@@ -290,10 +301,10 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         if not info:
             self._btin += gen_visit_decl(name, scalar=True)
             if self._opt_builtins:
-                self.defn += gen_visit_enum(name)
+                self._genc.add(gen_visit_enum(name))
         else:
-            self.decl += gen_visit_decl(name, scalar=True)
-            self.defn += gen_visit_enum(name)
+            self._genh.add(gen_visit_decl(name, scalar=True))
+            self._genc.add(gen_visit_enum(name))
 
     def visit_array_type(self, name, info, element_type):
         decl = gen_visit_decl(name)
@@ -301,53 +312,30 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
         if isinstance(element_type, QAPISchemaBuiltinType):
             self._btin += decl
             if self._opt_builtins:
-                self.defn += defn
+                self._genc.add(defn)
         else:
-            self.decl += decl
-            self.defn += defn
+            self._genh.add(decl)
+            self._genc.add(defn)
 
     def visit_object_type(self, name, info, base, members, variants):
         # Nothing to do for the special empty builtin
         if name == 'q_empty':
             return
-        self.decl += gen_visit_members_decl(name)
-        self.defn += gen_visit_object_members(name, base, members, variants)
+        self._genh.add(gen_visit_members_decl(name))
+        self._genc.add(gen_visit_object_members(name, base, members, variants))
         # TODO Worth changing the visitor signature, so we could
         # directly use rather than repeat type.is_implicit()?
         if not name.startswith('q_'):
             # only explicit types need an allocating visit
-            self.decl += gen_visit_decl(name)
-            self.defn += gen_visit_object(name, base, members, variants)
+            self._genh.add(gen_visit_decl(name))
+            self._genc.add(gen_visit_object(name, base, members, variants))
 
     def visit_alternate_type(self, name, info, variants):
-        self.decl += gen_visit_decl(name)
-        self.defn += gen_visit_alternate(name, variants)
+        self._genh.add(gen_visit_decl(name))
+        self._genc.add(gen_visit_alternate(name, variants))
 
 
 def gen_visit(schema, output_dir, prefix, opt_builtins):
-    blurb = ' * Schema-defined QAPI visitors'
-    genc = QAPIGenC(blurb, __doc__)
-    genh = QAPIGenH(blurb, __doc__)
-
-    genc.add(mcgen('''
-#include "qemu/osdep.h"
-#include "qemu-common.h"
-#include "qapi/error.h"
-#include "qapi/qmp/qerror.h"
-#include "%(prefix)sqapi-visit.h"
-''',
-                   prefix=prefix))
-
-    genh.add(mcgen('''
-#include "qapi/visitor.h"
-#include "%(prefix)sqapi-types.h"
-
-''',
-                   prefix=prefix))
-
-    vis = QAPISchemaGenVisitVisitor(opt_builtins)
+    vis = QAPISchemaGenVisitVisitor(prefix, opt_builtins)
     schema.visit(vis)
-    genc.add(vis.defn)
-    genh.add(vis.decl)
-    genc.write(output_dir, prefix + 'qapi-visit.c')
-    genh.write(output_dir, prefix + 'qapi-visit.h')
+    vis.write(output_dir)
--

2.13.6

Reviewed-by on v2 19/29
#65 lore raw

Re: [Qemu-devel] [PATCH v2 19/29] qapi: Make code-generating visitors use QAPIGen more

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:57)

Quoted 16 lines

The use of QAPIGen is rather shallow so far: most of the output accumulation is not converted. Take the next step: convert output accumulation in the code-generating visitor classes. Helper functions outside these classes are not converted.

---
 scripts/qapi/commands.py   | 71 ++++++++++++++++------------------------
 scripts/qapi/common.py     | 13 ++++++++
 scripts/qapi/doc.py        | 74 ++++++++++++++++++++----------------------
 scripts/qapi/events.py     | 55 ++++++++++++-------------------
 scripts/qapi/introspect.py | 56 +++++++++++++-------------------
 scripts/qapi/types.py      | 81 +++++++++++++++++++---------------------------
 scripts/qapi/visit.py      | 80 +++++++++++++++++++--------------------------
 7 files changed, 188 insertions(+), 242 deletions(-)

<snip>

Quoted 11 lines
scripts/qapi/common.py +3 -0
index 29d98ca934..31d2f73e7e 100644
@@ -2049,3 +2049,16 @@ class QAPIGenDoc(QAPIGen):
     def _top(self, fname):
         return (QAPIGen._top(self, fname)
                 + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')
+
+
+class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):

scripts/qapi/common.py class QAPIGenDoc(QAPIGen): · line 2064

Quoted 10 lines
scripts/qapi/common.py +10 -0
+
+    def __init__(self, prefix, what, blurb, pydoc):
+        self._prefix = prefix
+        self._what = what
+        self._genc = QAPIGenC(blurb, pydoc)
+        self._genh = QAPIGenH(blurb, pydoc)
+
+    def write(self, output_dir):
+        self._genc.write(output_dir, self._prefix + self._what + '.c')
+        self._genh.write(output_dir, self._prefix + self._what + '.h')

minor nit: since subclasses of QAPISchemaVisitor and QAPISchemaMonolithicCVisitor all rely on .write() now, should we declare it in the abstract QAPISchemaVisitor?

#66 lore raw

Re: [Qemu-devel] [PATCH v2 19/29] qapi: Make code-generating visitors use QAPIGen more

Quoted 37 lines · Michael Roth

Quoting Markus Armbruster (2018-02-11 03:35:57)

Quoted 16 lines

The use of QAPIGen is rather shallow so far: most of the output accumulation is not converted. Take the next step: convert output accumulation in the code-generating visitor classes. Helper functions outside these classes are not converted.

---
 scripts/qapi/commands.py   | 71 ++++++++++++++++------------------------
 scripts/qapi/common.py     | 13 ++++++++
 scripts/qapi/doc.py        | 74 ++++++++++++++++++++----------------------
 scripts/qapi/events.py     | 55 ++++++++++++-------------------
 scripts/qapi/introspect.py | 56 +++++++++++++-------------------
 scripts/qapi/types.py      | 81 +++++++++++++++++++---------------------------
 scripts/qapi/visit.py      | 80 +++++++++++++++++++--------------------------
 7 files changed, 188 insertions(+), 242 deletions(-)

<snip>

Quoted 17 lines
scripts/qapi/common.py +9 -0
index 29d98ca934..31d2f73e7e 100644
@@ -2049,3 +2049,16 @@ class QAPIGenDoc(QAPIGen):
     def _top(self, fname):
         return (QAPIGen._top(self, fname)
                 + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')
+
+
+class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
+
+    def __init__(self, prefix, what, blurb, pydoc):
+        self._prefix = prefix
+        self._what = what
+        self._genc = QAPIGenC(blurb, pydoc)
+        self._genh = QAPIGenH(blurb, pydoc)
Quoted 8 lines

scripts/qapi/common.py class QAPIGenDoc(QAPIGen): · line 2064

Quoted 4 lines
scripts/qapi/common.py +4 -0
+
+    def write(self, output_dir):
+        self._genc.write(output_dir, self._prefix + self._what + '.c')
+        self._genh.write(output_dir, self._prefix + self._what + '.h')

minor nit: since subclasses of QAPISchemaVisitor and QAPISchemaMonolithicCVisitor all rely on .write() now, should we declare it in the abstract QAPISchemaVisitor?

Perhaps. If they had more in common, the case for an abstract super class would be clearer.

Quoted 1 line

Thanks!

#67 lore raw

PATCH v2 20/29 qapi/types qapi/visit: Generate built-in stuff into separate files

In reply to Markus Armbruster

Linking code from multiple separate QAPI schemata into the same program is possible, but involves some weirdness around built-in types:

* We generate code for built-in types into .c only with option --builtins. The user is responsible for generating code for exactly one QAPI schema per program with --builtins.

* We generate code for built-in types into .h regardless of --builtins, but guarded by #ifndef QAPI_VISIT_BUILTIN. Because all copies of this code are exactly the same, including any combination of these headers works.

Replace this contraption by something more conventional: generate code for built-in types into their very own files: qapi-builtin-types.c, qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but only with --builtins. Obey --output-dir, but ignore --prefix for them.

Make qapi-types.h include qapi-builtin-types.h. With multiple schemata you now have multiple qapi-types.[ch], but only one qapi-builtin-types.[ch]. Same for qapi-visit.[ch] and qapi-builtin-visit.[ch].

Bonus: if all you need is built-in stuff, you can include a much smaller header. To be exploited shortly.

---
 .gitignore             |  2 ++
 Makefile               | 13 ++++++----
 Makefile.objs          |  2 ++
 scripts/qapi/common.py | 61 ++++++++++++++++++++++++++++++++++++++++-------
 scripts/qapi/types.py  | 61 +++++++++++++++++++----------------------------
 scripts/qapi/visit.py  | 64 +++++++++++++++++++++-----------------------------
 6 files changed, 116 insertions(+), 87 deletions(-)
.gitignore +2 -0
index 7d783e6e66..9477a08b6b 100644
@@ -29,6 +29,8 @@
 /qga/qapi-generated
 /qapi-generated
 /qapi-gen-timestamp
+/qapi-builtin-types.[ch]
+/qapi-builtin-visit.[ch]
 /qapi-types.[ch]
 /qapi-visit.[ch]
 /qapi-event.[ch]
Makefile +9 -4
index 164a38578e..60ddc9c945 100644
@@ -90,10 +90,13 @@ endif
 include $(SRC_PATH)/rules.mak
 
 GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
-GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
-GENERATED_FILES += qmp-commands.c qapi-types.c qapi-visit.c qapi-event.c
-GENERATED_FILES += qmp-introspect.h
-GENERATED_FILES += qmp-introspect.c
+GENERATED_FILES += qapi-builtin-types.h qapi-builtin-types.c
+GENERATED_FILES += qapi-types.h qapi-types.c
+GENERATED_FILES += qapi-builtin-visit.h qapi-builtin-visit.c
+GENERATED_FILES += qapi-visit.h qapi-visit.c
+GENERATED_FILES += qmp-commands.h qmp-commands.c
+GENERATED_FILES += qapi-event.h qapi-event.c
+GENERATED_FILES += qmp-introspect.c qmp-introspect.h
 GENERATED_FILES += qapi-doc.texi
 
 GENERATED_FILES += trace/generated-tcg-tracers.h
@@ -520,7 +523,9 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/transaction.json \
                $(SRC_PATH)/qapi/ui.json
 
+qapi-builtin-types.c qapi-builtin-types.h \
 qapi-types.c qapi-types.h \
+qapi-builtin-visit.c qapi-builtin-visit.h \
 qapi-visit.c qapi-visit.h \
 qmp-commands.h qmp-commands.c \
 qapi-event.c qapi-event.h \
Makefile.objs +2 -0
index d255aaf194..2813e984fd 100644
@@ -2,6 +2,8 @@
 # Common libraries for tools and emulators
 stub-obj-y = stubs/ crypto/
 util-obj-y = util/ qobject/ qapi/
+util-obj-y += qapi-builtin-types.o
+util-obj-y += qapi-builtin-visit.o
 util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o
 
 chardev-obj-y = chardev/
scripts/qapi/common.py +52 -9

86 lines not on this page · raw message

scripts/qapi/types.py +24 -37

94 lines not on this page · raw message

scripts/qapi/visit.py +27 -37

91 lines not on this page · raw message

Signature · 1 line
2.13.6
Reviewed-by on v2 20/29
#68 lore raw

Re: [Qemu-devel] [PATCH v2 20/29] qapi/types qapi/visit: Generate built-in stuff into separate files

Quoted 30 lines · Markus Armbruster

Linking code from multiple separate QAPI schemata into the same program is possible, but involves some weirdness around built-in types:

* We generate code for built-in types into .c only with option --builtins. The user is responsible for generating code for exactly one QAPI schema per program with --builtins.

* We generate code for built-in types into .h regardless of --builtins, but guarded by #ifndef QAPI_VISIT_BUILTIN. Because all copies of this code are exactly the same, including any combination of these headers works.

Replace this contraption by something more conventional: generate code for built-in types into their very own files: qapi-builtin-types.c, qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but only with --builtins. Obey --output-dir, but ignore --prefix for them.

Make qapi-types.h include qapi-builtin-types.h. With multiple schemata you now have multiple qapi-types.[ch], but only one qapi-builtin-types.[ch]. Same for qapi-visit.[ch] and qapi-builtin-visit.[ch].

Bonus: if all you need is built-in stuff, you can include a much smaller header. To be exploited shortly.

---

scripts/qapi/common.py class QAPIGenH(QAPIGenC): · line 2052

Quoted 7 lines
scripts/qapi/common.py +1 -0
@@ -2046,6 +2046,7 @@ class QAPIGenH(QAPIGenC):
 
 
  class QAPIGenDoc(QAPIGen):
+
      def _top(self, fname):
          return (QAPIGen._top(self, fname)
                  + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')

Does this hunk belong in an earlier patch?

Otherwise,

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#69 lore raw

Re: [Qemu-devel] [PATCH v2 20/29] qapi/types qapi/visit: Generate built-in stuff into separate files

Quoted 28 lines · Eric Blake
Quoted 27 lines · Markus Armbruster

Linking code from multiple separate QAPI schemata into the same program is possible, but involves some weirdness around built-in types:

* We generate code for built-in types into .c only with option --builtins. The user is responsible for generating code for exactly one QAPI schema per program with --builtins.

* We generate code for built-in types into .h regardless of --builtins, but guarded by #ifndef QAPI_VISIT_BUILTIN. Because all copies of this code are exactly the same, including any combination of these headers works.

Replace this contraption by something more conventional: generate code for built-in types into their very own files: qapi-builtin-types.c, qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but only with --builtins. Obey --output-dir, but ignore --prefix for them.

Make qapi-types.h include qapi-builtin-types.h. With multiple schemata you now have multiple qapi-types.[ch], but only one qapi-builtin-types.[ch]. Same for qapi-visit.[ch] and qapi-builtin-visit.[ch].

Bonus: if all you need is built-in stuff, you can include a much smaller header. To be exploited shortly.

Quoted 10 lines

scripts/qapi/common.py class QAPIGenH(QAPIGenC): · line 2050

Quoted 8 lines

---

scripts/qapi/common.py +1 -0
@@ -2046,6 +2046,7 @@ class QAPIGenH(QAPIGenC):
      class QAPIGenDoc(QAPIGen):
+
      def _top(self, fname):
          return (QAPIGen._top(self, fname)
                  + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')

Does this hunk belong in an earlier patch?

Yes: PATCH 05.

Quoted 2 lines

Otherwise,

Thanks!

Reviewed-by Marc-André Lureau on v2 20/29
#70 lore raw

Re: [Qemu-devel] [PATCH v2 20/29] qapi/types qapi/visit: Generate built-in stuff into separate files

In reply to Markus Armbruster

Hi

Quoted 24 lines · Markus Armbruster

Linking code from multiple separate QAPI schemata into the same program is possible, but involves some weirdness around built-in types:

* We generate code for built-in types into .c only with option --builtins. The user is responsible for generating code for exactly one QAPI schema per program with --builtins.

* We generate code for built-in types into .h regardless of --builtins, but guarded by #ifndef QAPI_VISIT_BUILTIN. Because all copies of this code are exactly the same, including any combination of these headers works.

Replace this contraption by something more conventional: generate code for built-in types into their very own files: qapi-builtin-types.c, qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but only with --builtins. Obey --output-dir, but ignore --prefix for them.

Make qapi-types.h include qapi-builtin-types.h. With multiple schemata you now have multiple qapi-types.[ch], but only one qapi-builtin-types.[ch]. Same for qapi-visit.[ch] and qapi-builtin-visit.[ch].

Quoted 4 lines

Bonus: if all you need is built-in stuff, you can include a much smaller header. To be exploited shortly.

Quoted 362 lines
---
 .gitignore             |  2 ++
 Makefile               | 13 ++++++----
 Makefile.objs          |  2 ++
 scripts/qapi/common.py | 61 ++++++++++++++++++++++++++++++++++++++++-------
 scripts/qapi/types.py  | 61 +++++++++++++++++++----------------------------
 scripts/qapi/visit.py  | 64 +++++++++++++++++++++-----------------------------
 6 files changed, 116 insertions(+), 87 deletions(-)
.gitignore +2 -0
index 7d783e6e66..9477a08b6b 100644
@@ -29,6 +29,8 @@
 /qga/qapi-generated
 /qapi-generated
 /qapi-gen-timestamp
+/qapi-builtin-types.[ch]
+/qapi-builtin-visit.[ch]
 /qapi-types.[ch]
 /qapi-visit.[ch]
 /qapi-event.[ch]
Makefile +9 -4
index 164a38578e..60ddc9c945 100644
@@ -90,10 +90,13 @@ endif
 include $(SRC_PATH)/rules.mak
 
 GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
-GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
-GENERATED_FILES += qmp-commands.c qapi-types.c qapi-visit.c qapi-event.c
-GENERATED_FILES += qmp-introspect.h
-GENERATED_FILES += qmp-introspect.c
+GENERATED_FILES += qapi-builtin-types.h qapi-builtin-types.c
+GENERATED_FILES += qapi-types.h qapi-types.c
+GENERATED_FILES += qapi-builtin-visit.h qapi-builtin-visit.c
+GENERATED_FILES += qapi-visit.h qapi-visit.c
+GENERATED_FILES += qmp-commands.h qmp-commands.c
+GENERATED_FILES += qapi-event.h qapi-event.c
+GENERATED_FILES += qmp-introspect.c qmp-introspect.h
 GENERATED_FILES += qapi-doc.texi
 
 GENERATED_FILES += trace/generated-tcg-tracers.h
@@ -520,7 +523,9 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/transaction.json \
                $(SRC_PATH)/qapi/ui.json
 
+qapi-builtin-types.c qapi-builtin-types.h \
 qapi-types.c qapi-types.h \
+qapi-builtin-visit.c qapi-builtin-visit.h \
 qapi-visit.c qapi-visit.h \
 qmp-commands.h qmp-commands.c \
 qapi-event.c qapi-event.h \
Makefile.objs +2 -0
index d255aaf194..2813e984fd 100644
@@ -2,6 +2,8 @@
 # Common libraries for tools and emulators
 stub-obj-y = stubs/ crypto/
 util-obj-y = util/ qobject/ qapi/
+util-obj-y += qapi-builtin-types.o
+util-obj-y += qapi-builtin-visit.o
 util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o
 
 chardev-obj-y = chardev/
scripts/qapi/common.py +52 -9
index 31d2f73e7e..de12f8469a 100644
@@ -1531,11 +1531,10 @@ class QAPISchema(object):
 
     def _def_builtin_type(self, name, json_type, c_type):
         self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
-        # TODO As long as we have QAPI_TYPES_BUILTIN to share multiple
-        # qapi-types.h from a single .c, all arrays of builtins must be
-        # declared in the first file whether or not they are used.  Nicer
-        # would be to use lazy instantiation, while figuring out how to
-        # avoid compilation issues with multiple qapi-types.h.
+        # Instantiating only the arrays that are actually used would
+        # be nice, but we can't as long as their generated code
+        # (qapi-builtin-types.[ch]) may be shared by some other
+        # schema.
         self._make_array_type(name, None)
 
     def _def_predefineds(self):
@@ -1992,14 +1991,15 @@ class QAPIGen(object):
         return ''
 
     def write(self, output_dir, fname):
-        if output_dir:
+        pathname = os.path.join(output_dir, fname)
+        dir = os.path.dirname(pathname)
+        if dir:
             try:
-                os.makedirs(output_dir)
+                os.makedirs(dir)
             except os.error as e:
                 if e.errno != errno.EEXIST:
                     raise
-        fd = os.open(os.path.join(output_dir, fname),
-                     os.O_RDWR | os.O_CREAT, 0666)
+        fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0666)
         f = os.fdopen(fd, 'r+')
         text = (self._top(fname) + self._preamble + self._body
                 + self._bottom(fname))
@@ -2046,6 +2046,7 @@ class QAPIGenH(QAPIGenC):
 
 
 class QAPIGenDoc(QAPIGen):
+
     def _top(self, fname):
         return (QAPIGen._top(self, fname)
                 + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')
@@ -2062,3 +2063,45 @@ class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
     def write(self, output_dir):
         self._genc.write(output_dir, self._prefix + self._what + '.c')
         self._genh.write(output_dir, self._prefix + self._what + '.h')
+
+
+class QAPISchemaModularCVisitor(QAPISchemaVisitor):
+
+    def __init__(self, prefix, what, blurb, pydoc):
+        self._prefix = prefix
+        self._what = what
+        self._blurb = blurb
+        self._pydoc = pydoc
+        self._module = {}
+
+    def _module_basename(self, what, name):
+        if name is None:
+            return re.sub(r'-', '-builtin-', what)
+        return self._prefix + what
+
+    def _add_module(self, name, blurb):
+        genc = QAPIGenC(blurb, self._pydoc)
+        genh = QAPIGenH(blurb, self._pydoc)
+        self._module[name] = (genc, genh)
+        self._set_module(name)
+
+    def _set_module(self, name):
+        self._genc, self._genh = self._module[name]
+
+    def write(self, output_dir, opt_builtins):
+        for name in self._module:
+            if name is None and not opt_builtins:
+                continue
+            basename = self._module_basename(self._what, name)
+            (genc, genh) = self._module[name]
+            genc.write(output_dir, basename + '.c')
+            genh.write(output_dir, basename + '.h')
+
+    def _begin_module(self, name):
+        pass
+
+    def visit_module(self, name):
+        if len(self._module) != 1:
+            return
+        self._add_module(name, self._blurb)
+        self._begin_module(name)
scripts/qapi/types.py +24 -37
index d2b8423479..59826b1162 100644
@@ -167,64 +167,51 @@ void qapi_free_%(c_name)s(%(c_name)s *obj)
     return ret
 
 
-class QAPISchemaGenTypeVisitor(QAPISchemaMonolithicCVisitor):
+class QAPISchemaGenTypeVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix, opt_builtins):
-        QAPISchemaMonolithicCVisitor.__init__(
+    def __init__(self, prefix):
+        QAPISchemaModularCVisitor.__init__(
             self, prefix, 'qapi-types', ' * Schema-defined QAPI types',
             __doc__)
-        self._opt_builtins = opt_builtins
+        self._add_module(None, ' * Built-in QAPI types')
+        self._genc.preamble_add(mcgen('''
+#include "qemu/osdep.h"
+#include "qapi/dealloc-visitor.h"
+#include "qapi-builtin-types.h"
+#include "qapi-builtin-visit.h"
+'''))
+        self._genh.preamble_add(mcgen('''
+#include "qapi/util.h"
+'''))
+
+    def _begin_module(self, name):
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
 #include "%(prefix)sqapi-types.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                                      prefix=prefix))
+                                      prefix=self._prefix))
         self._genh.preamble_add(mcgen('''
-#include "qapi/util.h"
+#include "qapi-builtin-types.h"
 '''))
-        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
 
     def visit_begin(self, schema):
         # gen_object() is recursive, ensure it doesn't visit the empty type
         objects_seen.add(schema.the_empty_object_type.name)
 
-    def visit_end(self):
-        # To avoid header dependency hell, we always generate
-        # declarations for built-in types in our header files and
-        # simply guard them.  See also opt_builtins (command line
-        # option -b).
-        self._btin += guardend('QAPI_TYPES_BUILTIN')
-        self._genh.preamble_add(self._btin)
-        self._btin = None
-
     def _gen_type_cleanup(self, name):
         self._genh.add(gen_type_cleanup_decl(name))
         self._genc.add(gen_type_cleanup(name))
 
     def visit_enum_type(self, name, info, values, prefix):
-        # Special case for our lone builtin enum type
-        # TODO use something cleaner than existence of info
-        if not info:
-            self._btin += gen_enum(name, values, prefix)
-            if self._opt_builtins:
-                self._genc.add(gen_enum_lookup(name, values, prefix))
-        else:
-            self._genh.preamble_add(gen_enum(name, values, prefix))
-            self._genc.add(gen_enum_lookup(name, values, prefix))
+        self._genh.preamble_add(gen_enum(name, values, prefix))
+        self._genc.add(gen_enum_lookup(name, values, prefix))
 
     def visit_array_type(self, name, info, element_type):
-        if isinstance(element_type, QAPISchemaBuiltinType):
-            self._btin += gen_fwd_object_or_array(name)
-            self._btin += gen_array(name, element_type)
-            self._btin += gen_type_cleanup_decl(name)
-            if self._opt_builtins:
-                self._genc.add(gen_type_cleanup(name))
-        else:
-            self._genh.preamble_add(gen_fwd_object_or_array(name))
-            self._genh.add(gen_array(name, element_type))
-            self._gen_type_cleanup(name)
+        self._genh.preamble_add(gen_fwd_object_or_array(name))
+        self._genh.add(gen_array(name, element_type))
+        self._gen_type_cleanup(name)
 
     def visit_object_type(self, name, info, base, members, variants):
         # Nothing to do for the special empty builtin
@@ -248,6 +235,6 @@ class QAPISchemaGenTypeVisitor(QAPISchemaMonolithicCVisitor):
 
 
 def gen_types(schema, output_dir, prefix, opt_builtins):
-    vis = QAPISchemaGenTypeVisitor(prefix, opt_builtins)
+    vis = QAPISchemaGenTypeVisitor(prefix)
     schema.visit(vis)
-    vis.write(output_dir)
+    vis.write(output_dir, opt_builtins)
scripts/qapi/visit.py +27 -38
index 3d09d44265..9b678e7263 100644
@@ -263,13 +263,27 @@ out:
                  c_name=c_name(name))
 
 
-class QAPISchemaGenVisitVisitor(QAPISchemaMonolithicCVisitor):
+class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix, opt_builtins):
-        QAPISchemaMonolithicCVisitor.__init__(
+    def __init__(self, prefix):
+        QAPISchemaModularCVisitor.__init__(
             self, prefix, 'qapi-visit', ' * Schema-defined QAPI visitors',
             __doc__)
-        self._opt_builtins = opt_builtins
+        self._add_module(None, ' * Built-in QAPI visitors')
+        self._genc.preamble_add(mcgen('''
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "qapi/error.h"
+#include "qapi-builtin-visit.h"
+'''))
+        self._genh.preamble_add(mcgen('''
+#include "qapi/visitor.h"
+#include "qapi-builtin-types.h"
+
+''',
+                                      prefix=prefix))
+
+    def _begin_module(self, name):
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
@@ -277,45 +291,21 @@ class QAPISchemaGenVisitVisitor(QAPISchemaMonolithicCVisitor):
 #include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                                      prefix=prefix))
+                                      prefix=self._prefix))
         self._genh.preamble_add(mcgen('''
-#include "qapi/visitor.h"
+#include "qapi-builtin-visit.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                                      prefix=prefix))
-        self._btin = guardstart('QAPI_VISIT_BUILTIN')
-
-    def visit_end(self):
-        # To avoid header dependency hell, we always generate
-        # declarations for built-in types in our header files and
-        # simply guard them.  See also opt_builtins (command line
-        # option -b).
-        self._btin += guardend('QAPI_VISIT_BUILTIN')
-        self._genh.preamble_add(self._btin)
-        self._btin = None
+                                      prefix=self._prefix))
 
     def visit_enum_type(self, name, info, values, prefix):
-        # Special case for our lone builtin enum type
-        # TODO use something cleaner than existence of info
-        if not info:
-            self._btin += gen_visit_decl(name, scalar=True)
-            if self._opt_builtins:
-                self._genc.add(gen_visit_enum(name))
-        else:
-            self._genh.add(gen_visit_decl(name, scalar=True))
-            self._genc.add(gen_visit_enum(name))
+        self._genh.add(gen_visit_decl(name, scalar=True))
+        self._genc.add(gen_visit_enum(name))
 
     def visit_array_type(self, name, info, element_type):
-        decl = gen_visit_decl(name)
-        defn = gen_visit_list(name, element_type)
-        if isinstance(element_type, QAPISchemaBuiltinType):
-            self._btin += decl
-            if self._opt_builtins:
-                self._genc.add(defn)
-        else:
-            self._genh.add(decl)
-            self._genc.add(defn)
+        self._genh.add(gen_visit_decl(name))
+        self._genc.add(gen_visit_list(name, element_type))
 
     def visit_object_type(self, name, info, base, members, variants):
         # Nothing to do for the special empty builtin
@@ -336,6 +326,6 @@ class QAPISchemaGenVisitVisitor(QAPISchemaMonolithicCVisitor):
 
 
 def gen_visit(schema, output_dir, prefix, opt_builtins):
-    vis = QAPISchemaGenVisitVisitor(prefix, opt_builtins)
+    vis = QAPISchemaGenVisitVisitor(prefix)
     schema.visit(vis)
-    vis.write(output_dir)
+    vis.write(output_dir, opt_builtins)
--

2.13.6

Reviewed-by on v2 20/29
#71 lore raw

Re: [Qemu-devel] [PATCH v2 20/29] qapi/types qapi/visit: Generate built-in stuff into separate files

In reply to Markus Armbruster

Quoting Markus Armbruster (2018-02-11 03:35:58)

Quoted 24 lines

Linking code from multiple separate QAPI schemata into the same program is possible, but involves some weirdness around built-in types:

* We generate code for built-in types into .c only with option --builtins. The user is responsible for generating code for exactly one QAPI schema per program with --builtins.

* We generate code for built-in types into .h regardless of --builtins, but guarded by #ifndef QAPI_VISIT_BUILTIN. Because all copies of this code are exactly the same, including any combination of these headers works.

Replace this contraption by something more conventional: generate code for built-in types into their very own files: qapi-builtin-types.c, qapi-builtin-visit.c, qapi-builtin-types.h, qapi-builtin-visit.h, but only with --builtins. Obey --output-dir, but ignore --prefix for them.

Make qapi-types.h include qapi-builtin-types.h. With multiple schemata you now have multiple qapi-types.[ch], but only one qapi-builtin-types.[ch]. Same for qapi-visit.[ch] and qapi-builtin-visit.[ch].

Quoted 4 lines

Bonus: if all you need is built-in stuff, you can include a much smaller header. To be exploited shortly.

Quoted 362 lines
---
 .gitignore             |  2 ++
 Makefile               | 13 ++++++----
 Makefile.objs          |  2 ++
 scripts/qapi/common.py | 61 ++++++++++++++++++++++++++++++++++++++++-------
 scripts/qapi/types.py  | 61 +++++++++++++++++++----------------------------
 scripts/qapi/visit.py  | 64 +++++++++++++++++++++-----------------------------
 6 files changed, 116 insertions(+), 87 deletions(-)
.gitignore +2 -0
index 7d783e6e66..9477a08b6b 100644
@@ -29,6 +29,8 @@
 /qga/qapi-generated
 /qapi-generated
 /qapi-gen-timestamp
+/qapi-builtin-types.[ch]
+/qapi-builtin-visit.[ch]
 /qapi-types.[ch]
 /qapi-visit.[ch]
 /qapi-event.[ch]
Makefile +9 -4
index 164a38578e..60ddc9c945 100644
@@ -90,10 +90,13 @@ endif
 include $(SRC_PATH)/rules.mak
 
 GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
-GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
-GENERATED_FILES += qmp-commands.c qapi-types.c qapi-visit.c qapi-event.c
-GENERATED_FILES += qmp-introspect.h
-GENERATED_FILES += qmp-introspect.c
+GENERATED_FILES += qapi-builtin-types.h qapi-builtin-types.c
+GENERATED_FILES += qapi-types.h qapi-types.c
+GENERATED_FILES += qapi-builtin-visit.h qapi-builtin-visit.c
+GENERATED_FILES += qapi-visit.h qapi-visit.c
+GENERATED_FILES += qmp-commands.h qmp-commands.c
+GENERATED_FILES += qapi-event.h qapi-event.c
+GENERATED_FILES += qmp-introspect.c qmp-introspect.h
 GENERATED_FILES += qapi-doc.texi
 
 GENERATED_FILES += trace/generated-tcg-tracers.h
@@ -520,7 +523,9 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
                $(SRC_PATH)/qapi/transaction.json \
                $(SRC_PATH)/qapi/ui.json
 
+qapi-builtin-types.c qapi-builtin-types.h \
 qapi-types.c qapi-types.h \
+qapi-builtin-visit.c qapi-builtin-visit.h \
 qapi-visit.c qapi-visit.h \
 qmp-commands.h qmp-commands.c \
 qapi-event.c qapi-event.h \
Makefile.objs +2 -0
index d255aaf194..2813e984fd 100644
@@ -2,6 +2,8 @@
 # Common libraries for tools and emulators
 stub-obj-y = stubs/ crypto/
 util-obj-y = util/ qobject/ qapi/
+util-obj-y += qapi-builtin-types.o
+util-obj-y += qapi-builtin-visit.o
 util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o
 
 chardev-obj-y = chardev/
scripts/qapi/common.py +52 -9
index 31d2f73e7e..de12f8469a 100644
@@ -1531,11 +1531,10 @@ class QAPISchema(object):
 
     def _def_builtin_type(self, name, json_type, c_type):
         self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
-        # TODO As long as we have QAPI_TYPES_BUILTIN to share multiple
-        # qapi-types.h from a single .c, all arrays of builtins must be
-        # declared in the first file whether or not they are used.  Nicer
-        # would be to use lazy instantiation, while figuring out how to
-        # avoid compilation issues with multiple qapi-types.h.
+        # Instantiating only the arrays that are actually used would
+        # be nice, but we can't as long as their generated code
+        # (qapi-builtin-types.[ch]) may be shared by some other
+        # schema.
         self._make_array_type(name, None)
 
     def _def_predefineds(self):
@@ -1992,14 +1991,15 @@ class QAPIGen(object):
         return ''
 
     def write(self, output_dir, fname):
-        if output_dir:
+        pathname = os.path.join(output_dir, fname)
+        dir = os.path.dirname(pathname)
+        if dir:
             try:
-                os.makedirs(output_dir)
+                os.makedirs(dir)
             except os.error as e:
                 if e.errno != errno.EEXIST:
                     raise
-        fd = os.open(os.path.join(output_dir, fname),
-                     os.O_RDWR | os.O_CREAT, 0666)
+        fd = os.open(pathname, os.O_RDWR | os.O_CREAT, 0666)
         f = os.fdopen(fd, 'r+')
         text = (self._top(fname) + self._preamble + self._body
                 + self._bottom(fname))
@@ -2046,6 +2046,7 @@ class QAPIGenH(QAPIGenC):
 
 
 class QAPIGenDoc(QAPIGen):
+
     def _top(self, fname):
         return (QAPIGen._top(self, fname)
                 + '@c AUTOMATICALLY GENERATED, DO NOT MODIFY\n\n')
@@ -2062,3 +2063,45 @@ class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor):
     def write(self, output_dir):
         self._genc.write(output_dir, self._prefix + self._what + '.c')
         self._genh.write(output_dir, self._prefix + self._what + '.h')
+
+
+class QAPISchemaModularCVisitor(QAPISchemaVisitor):
+
+    def __init__(self, prefix, what, blurb, pydoc):
+        self._prefix = prefix
+        self._what = what
+        self._blurb = blurb
+        self._pydoc = pydoc
+        self._module = {}
+
+    def _module_basename(self, what, name):
+        if name is None:
+            return re.sub(r'-', '-builtin-', what)
+        return self._prefix + what
+
+    def _add_module(self, name, blurb):
+        genc = QAPIGenC(blurb, self._pydoc)
+        genh = QAPIGenH(blurb, self._pydoc)
+        self._module[name] = (genc, genh)
+        self._set_module(name)
+
+    def _set_module(self, name):
+        self._genc, self._genh = self._module[name]
+
+    def write(self, output_dir, opt_builtins):
+        for name in self._module:
+            if name is None and not opt_builtins:
+                continue
+            basename = self._module_basename(self._what, name)
+            (genc, genh) = self._module[name]
+            genc.write(output_dir, basename + '.c')
+            genh.write(output_dir, basename + '.h')
+
+    def _begin_module(self, name):
+        pass
+
+    def visit_module(self, name):
+        if len(self._module) != 1:
+            return
+        self._add_module(name, self._blurb)
+        self._begin_module(name)
scripts/qapi/types.py +24 -37
index d2b8423479..59826b1162 100644
@@ -167,64 +167,51 @@ void qapi_free_%(c_name)s(%(c_name)s *obj)
     return ret
 
 
-class QAPISchemaGenTypeVisitor(QAPISchemaMonolithicCVisitor):
+class QAPISchemaGenTypeVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix, opt_builtins):
-        QAPISchemaMonolithicCVisitor.__init__(
+    def __init__(self, prefix):
+        QAPISchemaModularCVisitor.__init__(
             self, prefix, 'qapi-types', ' * Schema-defined QAPI types',
             __doc__)
-        self._opt_builtins = opt_builtins
+        self._add_module(None, ' * Built-in QAPI types')
+        self._genc.preamble_add(mcgen('''
+#include "qemu/osdep.h"
+#include "qapi/dealloc-visitor.h"
+#include "qapi-builtin-types.h"
+#include "qapi-builtin-visit.h"
+'''))
+        self._genh.preamble_add(mcgen('''
+#include "qapi/util.h"
+'''))
+
+    def _begin_module(self, name):
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
 #include "%(prefix)sqapi-types.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                                      prefix=prefix))
+                                      prefix=self._prefix))
         self._genh.preamble_add(mcgen('''
-#include "qapi/util.h"
+#include "qapi-builtin-types.h"
 '''))
-        self._btin = '\n' + guardstart('QAPI_TYPES_BUILTIN')
 
     def visit_begin(self, schema):
         # gen_object() is recursive, ensure it doesn't visit the empty type
         objects_seen.add(schema.the_empty_object_type.name)
 
-    def visit_end(self):
-        # To avoid header dependency hell, we always generate
-        # declarations for built-in types in our header files and
-        # simply guard them.  See also opt_builtins (command line
-        # option -b).
-        self._btin += guardend('QAPI_TYPES_BUILTIN')
-        self._genh.preamble_add(self._btin)
-        self._btin = None
-
     def _gen_type_cleanup(self, name):
         self._genh.add(gen_type_cleanup_decl(name))
         self._genc.add(gen_type_cleanup(name))
 
     def visit_enum_type(self, name, info, values, prefix):
-        # Special case for our lone builtin enum type
-        # TODO use something cleaner than existence of info
-        if not info:
-            self._btin += gen_enum(name, values, prefix)
-            if self._opt_builtins:
-                self._genc.add(gen_enum_lookup(name, values, prefix))
-        else:
-            self._genh.preamble_add(gen_enum(name, values, prefix))
-            self._genc.add(gen_enum_lookup(name, values, prefix))
+        self._genh.preamble_add(gen_enum(name, values, prefix))
+        self._genc.add(gen_enum_lookup(name, values, prefix))
 
     def visit_array_type(self, name, info, element_type):
-        if isinstance(element_type, QAPISchemaBuiltinType):
-            self._btin += gen_fwd_object_or_array(name)
-            self._btin += gen_array(name, element_type)
-            self._btin += gen_type_cleanup_decl(name)
-            if self._opt_builtins:
-                self._genc.add(gen_type_cleanup(name))
-        else:
-            self._genh.preamble_add(gen_fwd_object_or_array(name))
-            self._genh.add(gen_array(name, element_type))
-            self._gen_type_cleanup(name)
+        self._genh.preamble_add(gen_fwd_object_or_array(name))
+        self._genh.add(gen_array(name, element_type))
+        self._gen_type_cleanup(name)
 
     def visit_object_type(self, name, info, base, members, variants):
         # Nothing to do for the special empty builtin
@@ -248,6 +235,6 @@ class QAPISchemaGenTypeVisitor(QAPISchemaMonolithicCVisitor):
 
 
 def gen_types(schema, output_dir, prefix, opt_builtins):
-    vis = QAPISchemaGenTypeVisitor(prefix, opt_builtins)
+    vis = QAPISchemaGenTypeVisitor(prefix)
     schema.visit(vis)
-    vis.write(output_dir)
+    vis.write(output_dir, opt_builtins)
scripts/qapi/visit.py +27 -37
index 3d09d44265..9b678e7263 100644
@@ -263,13 +263,27 @@ out:
                  c_name=c_name(name))
 
 
-class QAPISchemaGenVisitVisitor(QAPISchemaMonolithicCVisitor):
+class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
 
-    def __init__(self, prefix, opt_builtins):
-        QAPISchemaMonolithicCVisitor.__init__(
+    def __init__(self, prefix):
+        QAPISchemaModularCVisitor.__init__(
             self, prefix, 'qapi-visit', ' * Schema-defined QAPI visitors',
             __doc__)
-        self._opt_builtins = opt_builtins
+        self._add_module(None, ' * Built-in QAPI visitors')
+        self._genc.preamble_add(mcgen('''
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "qapi/error.h"
+#include "qapi-builtin-visit.h"
+'''))
+        self._genh.preamble_add(mcgen('''
+#include "qapi/visitor.h"
+#include "qapi-builtin-types.h"
+
+''',
+                                      prefix=prefix))
+
+    def _begin_module(self, name):
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
@@ -277,45 +291,21 @@ class QAPISchemaGenVisitVisitor(QAPISchemaMonolithicCVisitor):
 #include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-visit.h"
 ''',
-                                      prefix=prefix))
+                                      prefix=self._prefix))
         self._genh.preamble_add(mcgen('''
-#include "qapi/visitor.h"
+#include "qapi-builtin-visit.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                                      prefix=prefix))
-        self._btin = guardstart('QAPI_VISIT_BUILTIN')
-
-    def visit_end(self):
-        # To avoid header dependency hell, we always generate
-        # declarations for built-in types in our header files and
-        # simply guard them.  See also opt_builtins (command line
-        # option -b).
-        self._btin += guardend('QAPI_VISIT_BUILTIN')
-        self._genh.preamble_add(self._btin)
-        self._btin = None
+                                      prefix=self._prefix))
 
     def visit_enum_type(self, name, info, values, prefix):
-        # Special case for our lone builtin enum type
-        # TODO use something cleaner than existence of info
-        if not info:
-            self._btin += gen_visit_decl(name, scalar=True)
-            if self._opt_builtins:
-                self._genc.add(gen_visit_enum(name))
-        else:
-            self._genh.add(gen_visit_decl(name, scalar=True))
-            self._genc.add(gen_visit_enum(name))
+        self._genh.add(gen_visit_decl(name, scalar=True))
+        self._genc.add(gen_visit_enum(name))
 
     def visit_array_type(self, name, info, element_type):
-        decl = gen_visit_decl(name)
-        defn = gen_visit_list(name, element_type)
-        if isinstance(element_type, QAPISchemaBuiltinType):
-            self._btin += decl
-            if self._opt_builtins:
-                self._genc.add(defn)
-        else:
-            self._genh.add(decl)
-            self._genc.add(defn)
+        self._genh.add(gen_visit_decl(name))
+        self._genc.add(gen_visit_list(name, element_type))
 
     def visit_object_type(self, name, info, base, members, variants):
         # Nothing to do for the special empty builtin
@@ -336,6 +326,6 @@ class QAPISchemaGenVisitVisitor(QAPISchemaMonolithicCVisitor):
 
 
 def gen_visit(schema, output_dir, prefix, opt_builtins):
-    vis = QAPISchemaGenVisitVisitor(prefix, opt_builtins)
+    vis = QAPISchemaGenVisitVisitor(prefix)
     schema.visit(vis)
-    vis.write(output_dir)
+    vis.write(output_dir, opt_builtins)
Signature · 1 line
2.13.6
#72 lore raw

PATCH v2 21/29 qapi/common: Fix guardname() for funny filenames

In reply to Markus Armbruster

guardname() fails to return a valid C identifier for arguments containing anything but [A-Za-z0-9_.-']. Fix that. Don't bother protecting ticklish identifiers; header guards are all-caps, and no ticklish identifiers are.

---
 scripts/qapi/common.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/common.py +1 -1
index de12f8469a..6e5152b173 100644
@@ -1867,7 +1867,7 @@ def mcgen(code, **kwds):
 
 
 def guardname(filename):
-    return c_name(filename, protect=False).upper()
+    return re.sub(r'[^A-Za-z0-9_]', '_', filename).upper()
 
 
 def guardstart(name):
Signature · 1 line
2.13.6
Reviewed-by on v2 21/29
#73 lore raw

Re: [Qemu-devel] [PATCH v2 21/29] qapi/common: Fix guardname() for funny filenames

Quoting Markus Armbruster (2018-02-11 03:35:59)

Quoted 8 lines

guardname() fails to return a valid C identifier for arguments containing anything but [A-Za-z0-9_.-']. Fix that. Don't bother protecting ticklish identifiers; header guards are all-caps, and no ticklish identifiers are.

Quoted 20 lines
---
 scripts/qapi/common.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi/common.py +1 -1
index de12f8469a..6e5152b173 100644
@@ -1867,7 +1867,7 @@ def mcgen(code, **kwds):
 
 
 def guardname(filename):
-    return c_name(filename, protect=False).upper()
+    return re.sub(r'[^A-Za-z0-9_]', '_', filename).upper()
 
 
 def guardstart(name):
Signature · 1 line
2.13.6
#74 lore raw

PATCH v2 22/29 qapi: Generate separate .h, .c for each module

In reply to Markus Armbruster

Our qapi-schema.json is composed of modules connected by include directives, but the generated code is monolithic all the same: one qapi-types.h with all the types, one qapi-visit.h with all the visitors, and so forth. These monolithic headers get included all over the place. In my "build everything" tree, adding a QAPI type recompiles about 4800 out of 5100 objects.

We wouldn't write such monolithic headers by hand. It stands to reason that we shouldn't generate them, either.

Split up generated qapi-types.h to mirror the schema's modular
structure: one header per module. Name the main module's header qapi-types.h, and sub-module D/B.json's header D/qapi-types-B.h.

Mirror the schema's includes in the headers, so that qapi-types.h gets you everything exactly as before. If you need less, you can include one or more of the sub-module headers. To be exploited shortly.

Split up qapi-types.c, qapi-visit.h, qapi-visit.c, qmp-commands.h, qmp-commands.c, qapi-event.h, qapi-event.c the same way. qmp-introspect.h, qmp-introspect.c and qapi.texi remain monolithic.

The split of qmp-commands.c duplicates static helper function qmp_marshal_output_str() in qapi-commands-char.c and qapi-commands-misc.c. This happens when commands returning the same type occur in multiple modules. Not worth avoiding.

Since I'm going to rename qapi-event.[ch] to qapi-events.[ch], and qmp-commands.[ch] to qapi-commands.[ch], name the shards that way already, to reduce churn. This requires temporary hacks in commands.py and events.py. They'll go away with the rename.

---
 .gitignore               |  60 ++++++++++++++++++++++++
 Makefile                 | 120 +++++++++++++++++++++++++++++++++++++++++++++++
 Makefile.objs            |  65 ++++++++++++++++++++++++-
 scripts/qapi/commands.py |  35 +++++++++-----
 scripts/qapi/common.py   |  21 +++++++--
 scripts/qapi/events.py   |  19 ++++++--
 6 files changed, 300 insertions(+), 20 deletions(-)
.gitignore +60 -0
index 9477a08b6b..42c57998fd 100644
@@ -31,7 +31,67 @@
 /qapi-gen-timestamp
 /qapi-builtin-types.[ch]
 /qapi-builtin-visit.[ch]
+/qapi/qapi-commands-block-core.[ch]
+/qapi/qapi-commands-block.[ch]
+/qapi/qapi-commands-char.[ch]
+/qapi/qapi-commands-common.[ch]
+/qapi/qapi-commands-crypto.[ch]
+/qapi/qapi-commands-introspect.[ch]
+/qapi/qapi-commands-migration.[ch]
+/qapi/qapi-commands-net.[ch]
+/qapi/qapi-commands-rocker.[ch]
+/qapi/qapi-commands-run-state.[ch]
+/qapi/qapi-commands-sockets.[ch]
+/qapi/qapi-commands-tpm.[ch]
+/qapi/qapi-commands-trace.[ch]
+/qapi/qapi-commands-transaction.[ch]
+/qapi/qapi-commands-ui.[ch]
+/qapi/qapi-events-block-core.[ch]
+/qapi/qapi-events-block.[ch]
+/qapi/qapi-events-char.[ch]
+/qapi/qapi-events-common.[ch]
+/qapi/qapi-events-crypto.[ch]
+/qapi/qapi-events-introspect.[ch]
+/qapi/qapi-events-migration.[ch]
+/qapi/qapi-events-net.[ch]
+/qapi/qapi-events-rocker.[ch]
+/qapi/qapi-events-run-state.[ch]
+/qapi/qapi-events-sockets.[ch]
+/qapi/qapi-events-tpm.[ch]
+/qapi/qapi-events-trace.[ch]
+/qapi/qapi-events-transaction.[ch]
+/qapi/qapi-events-ui.[ch]
+/qapi/qapi-types-block-core.[ch]
+/qapi/qapi-types-block.[ch]
+/qapi/qapi-types-char.[ch]
+/qapi/qapi-types-common.[ch]
+/qapi/qapi-types-crypto.[ch]
+/qapi/qapi-types-introspect.[ch]
+/qapi/qapi-types-migration.[ch]
+/qapi/qapi-types-net.[ch]
+/qapi/qapi-types-rocker.[ch]
+/qapi/qapi-types-run-state.[ch]
+/qapi/qapi-types-sockets.[ch]
+/qapi/qapi-types-tpm.[ch]
+/qapi/qapi-types-trace.[ch]
+/qapi/qapi-types-transaction.[ch]
+/qapi/qapi-types-ui.[ch]
 /qapi-types.[ch]
+/qapi/qapi-visit-block-core.[ch]
+/qapi/qapi-visit-block.[ch]
+/qapi/qapi-visit-char.[ch]
+/qapi/qapi-visit-common.[ch]
+/qapi/qapi-visit-crypto.[ch]
+/qapi/qapi-visit-introspect.[ch]
+/qapi/qapi-visit-migration.[ch]
+/qapi/qapi-visit-net.[ch]
+/qapi/qapi-visit-rocker.[ch]
+/qapi/qapi-visit-run-state.[ch]
+/qapi/qapi-visit-sockets.[ch]
+/qapi/qapi-visit-tpm.[ch]
+/qapi/qapi-visit-trace.[ch]
+/qapi/qapi-visit-transaction.[ch]
+/qapi/qapi-visit-ui.[ch]
 /qapi-visit.[ch]
 /qapi-event.[ch]
 /qapi-doc.texi
Makefile +120 -0

140 lines not on this page · raw message

Makefile.objs +64 -1

78 lines not on this page · raw message

scripts/qapi/commands.py +24 -11
index 46757db771..a43bccb190 100644
@@ -223,14 +223,24 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
     return ret
 
 
-class QAPISchemaGenCommandVisitor(QAPISchemaMonolithicCVisitor):
+class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
 
     def __init__(self, prefix):
-        QAPISchemaMonolithicCVisitor.__init__(
-            self, prefix, 'qmp-commands',
+        QAPISchemaModularCVisitor.__init__(
+            self, prefix, 'qapi-commands',
             ' * Schema-defined QAPI/QMP commands', __doc__)
         self._regy = ''
-        self._visited_ret_types = set()
+        self._visited_ret_types = {}
+
+    # Temporary HACK:
+    def _module_basename(self, what, name):
+        basename = QAPISchemaModularCVisitor._module_basename(self, what, name)
+        if name == self._main_module:
+            return re.sub(r'qapi-commands', 'qmp-commands', basename)
+        return basename
+
+    def _begin_module(self, name):
+        self._visited_ret_types[self._genc] = set()
         self._genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
@@ -246,26 +256,29 @@ class QAPISchemaGenCommandVisitor(QAPISchemaMonolithicCVisitor):
 #include "%(prefix)sqmp-commands.h"
 
 ''',
-                             prefix=prefix))
+                             prefix=self._prefix))
         self._genh.add(mcgen('''
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/dispatch.h"
 
-void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-                             prefix=prefix,
-                             c_prefix=c_name(prefix, protect=False)))
+                             prefix=self._prefix))
 
     def visit_end(self):
-        self._genc.add(gen_registry(self._regy, self._prefix))
+        (genc, genh) = self._module[self._main_module]
+        genh.add(mcgen('''
+void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
+''',
+                       c_prefix=c_name(self._prefix, protect=False)))
+        genc.add(gen_registry(self._regy, self._prefix))
 
     def visit_command(self, name, info, arg_type, ret_type,
                       gen, success_response, boxed):
         if not gen:
             return
         self._genh.add(gen_command_decl(name, arg_type, boxed, ret_type))
-        if ret_type and ret_type not in self._visited_ret_types:
-            self._visited_ret_types.add(ret_type)
+        if ret_type and ret_type not in self._visited_ret_types[self._genc]:
+            self._visited_ret_types[self._genc].add(ret_type)
             self._genc.add(gen_marshal_output(ret_type))
         self._genh.add(gen_marshal_decl(name))
         self._genc.add(gen_marshal(name, arg_type, boxed, ret_type))
scripts/qapi/common.py +18 -3
index 6e5152b173..b9a52e820d 100644
@@ -2073,13 +2073,20 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor):
         self._blurb = blurb
         self._pydoc = pydoc
         self._module = {}
+        self._main_module = None
 
     def _module_basename(self, what, name):
         if name is None:
             return re.sub(r'-', '-builtin-', what)
-        return self._prefix + what
+        basename = os.path.join(os.path.dirname(name),
+                                self._prefix + what)
+        if name == self._main_module:
+            return basename
+        return basename + '-' + os.path.splitext(os.path.basename(name))[0]
 
     def _add_module(self, name, blurb):
+        if self._main_module is None and name is not None:
+            self._main_module = name
         genc = QAPIGenC(blurb, self._pydoc)
         genh = QAPIGenH(blurb, self._pydoc)
         self._module[name] = (genc, genh)
@@ -2088,7 +2095,7 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor):
     def _set_module(self, name):
         self._genc, self._genh = self._module[name]
 
-    def write(self, output_dir, opt_builtins):
+    def write(self, output_dir, opt_builtins=False):
         for name in self._module:
             if name is None and not opt_builtins:
                 continue
@@ -2101,7 +2108,15 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor):
         pass
 
     def visit_module(self, name):
-        if len(self._module) != 1:
+        if name in self._module:
+            self._set_module(name)
             return
         self._add_module(name, self._blurb)
         self._begin_module(name)
+
+    def visit_include(self, name, info):
+        basename = self._module_basename(self._what, name)
+        self._genh.preamble_add(mcgen('''
+#include "%(basename)s.h"
+''',
+                                      basename=basename))
scripts/qapi/events.py +14 -5
index 81ab3abb30..1e0b990f35 100644
@@ -148,14 +148,23 @@ out:
     return ret
 
 
-class QAPISchemaGenEventVisitor(QAPISchemaMonolithicCVisitor):
+class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
 
     def __init__(self, prefix):
-        QAPISchemaMonolithicCVisitor.__init__(
-            self, prefix, 'qapi-event',
+        QAPISchemaModularCVisitor.__init__(
+            self, prefix, 'qapi-events',
             ' * Schema-defined QAPI/QMP events', __doc__)
         self._enum_name = c_name(prefix + 'QAPIEvent', protect=False)
         self._event_names = []
+
+    # Temporary HACK:
+    def _module_basename(self, what, name):
+        basename = QAPISchemaModularCVisitor._module_basename(self, what, name)
+        if name == self._main_module:
+            return re.sub(r'qapi-events', 'qapi-event', basename)
+        return basename
+
+    def _begin_module(self, name):
         self._genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
@@ -167,13 +176,13 @@ class QAPISchemaGenEventVisitor(QAPISchemaMonolithicCVisitor):
 #include "qapi/qmp-event.h"
 
 ''',
-                             prefix=prefix))
+                             prefix=self._prefix))
         self._genh.add(mcgen('''
 #include "qapi/util.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-                             prefix=prefix))
+                             prefix=self._prefix))
 
     def visit_end(self):
         self._genh.add(gen_enum(self._enum_name, self._event_names))
Signature · 1 line
2.13.6
#75 lore raw

Re: [Qemu-devel] [PATCH v2 22/29] qapi: Generate separate .h, .c for each module

Quoted 14 lines · Markus Armbruster

Our qapi-schema.json is composed of modules connected by include directives, but the generated code is monolithic all the same: one qapi-types.h with all the types, one qapi-visit.h with all the visitors, and so forth. These monolithic headers get included all over the place. In my "build everything" tree, adding a QAPI type recompiles about 4800 out of 5100 objects.

We wouldn't write such monolithic headers by hand. It stands to reason that we shouldn't generate them, either.

Split up generated qapi-types.h to mirror the schema's modular
structure: one header per module. Name the main module's header qapi-types.h, and sub-module D/B.json's header D/qapi-types-B.h.

Quoted 7 lines

Mirror the schema's includes in the headers, so that qapi-types.h gets you everything exactly as before. If you need less, you can include one or more of the sub-module headers. To be exploited shortly.

Split up qapi-types.c, qapi-visit.h, qapi-visit.c, qmp-commands.h, qmp-commands.c, qapi-event.h, qapi-event.c the same way. qmp-introspect.h, qmp-introspect.c and qapi.texi remain monolithic.

Make sense.

Quoted 5 lines

The split of qmp-commands.c duplicates static helper function qmp_marshal_output_str() in qapi-commands-char.c and qapi-commands-misc.c. This happens when commands returning the same type occur in multiple modules. Not worth avoiding.

As long as it is static, and neither .c file includes the other, we're fine (gdb may have a harder time figuring out which copy to put a breakpoint on, but that's not too terrible).

Quoted 5 lines

Since I'm going to rename qapi-event.[ch] to qapi-events.[ch], and qmp-commands.[ch] to qapi-commands.[ch], name the shards that way already, to reduce churn. This requires temporary hacks in commands.py and events.py. They'll go away with the rename.

The planned rename makes sense; prepping now is fine.

Quoted 16 lines
---
  .gitignore               |  60 ++++++++++++++++++++++++
  Makefile                 | 120 +++++++++++++++++++++++++++++++++++++++++++++++
  Makefile.objs            |  65 ++++++++++++++++++++++++-
  scripts/qapi/commands.py |  35 +++++++++-----
  scripts/qapi/common.py   |  21 +++++++--
  scripts/qapi/events.py   |  19 ++++++--
  6 files changed, 300 insertions(+), 20 deletions(-)
.gitignore +0 -0
index 9477a08b6b..42c57998fd 100644

.gitignore line 35

Quoted 5 lines
.gitignore +2 -0
@@ -31,7 +31,67 @@
  /qapi-gen-timestamp
  /qapi-builtin-types.[ch]
  /qapi-builtin-visit.[ch]
+/qapi/qapi-commands-block-core.[ch]
+/qapi/qapi-commands-block.[ch]

Is it worth using a glob instead of specific patterns, as in:

/qapi/qapi-commands-*.[ch]

Maybe being precise doesn't hurt, if we aren't adding sub-modules all that often; but being generous with a glob makes it less likely that a future module addition will forget to update .gitignore.

.gitignore include $(SRC_PATH)/rules.mak · line 95

Quoted 6 lines
.gitignore +2 -0
+++ b/Makefile
@@ -92,10 +92,70 @@ include $(SRC_PATH)/rules.mak
  GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
  GENERATED_FILES += qapi-builtin-types.h qapi-builtin-types.c
  GENERATED_FILES += qapi-types.h qapi-types.c
+GENERATED_FILES += qapi/qapi-types-block-core.h qapi/qapi-types-block-core.c

For the makefile, I'd suggest using an intermediate list of module names, and then using make string operations to expand it into larger lists, to cut down on the repetition. Something like (untested):

QAPI_MODULES = block-core block char common ...
GENERATED_FILES += $(addprefix qapi/qapi-types-,$(addsuffix .c,$(QAPI_MODULES)) GENERATED_FILES += $(addprefix qapi/qapi-types-,$(addsuffix .h,$(QAPI_MODULES))

.gitignore qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \ · line 589

Quoted 6 lines
.gitignore +2 -0
@@ -525,10 +585,70 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
 
  qapi-builtin-types.c qapi-builtin-types.h \
  qapi-types.c qapi-types.h \
+qapi/qapi-types-block-core.c qapi/qapi-types-block-core.h \
+qapi/qapi-types-block.c qapi/qapi-types-block.h \

And repeating the list of filenames; again, an intermediate variable would let you do:

QAPI_GENERATED_FILES = ...
GENERATED_FILES += $(QAPI_GENERATED_FILES)

$(QAPI_GENERATED_FILES): ...

.gitignore qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \ · line 592

Quoted 3 lines
.gitignore +0 -0
  qmp-introspect.h qmp-introspect.c \
  qapi-doc.texi: \
  qapi-gen-timestamp ;

Not only does it cut down on the repetition, but it will make adding a future module easier.

Makefile.objs line 7

Quoted 10 lines
Makefile.objs +2 -0
index 2813e984fd..7a55d45669 100644
@@ -3,8 +3,56 @@
  stub-obj-y = stubs/ crypto/
  util-obj-y = util/ qobject/ qapi/
  util-obj-y += qapi-builtin-types.o
+util-obj-y += qapi-types.o
+util-obj-y += qapi/qapi-types-block-core.o

Perhaps this can also exploit make text manipulation?

Otherwise looks good.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#76 lore raw

Re: [Qemu-devel] [PATCH v2 22/29] qapi: Generate separate .h, .c for each module

Quoted 67 lines · Eric Blake
Quoted 14 lines · Markus Armbruster

Our qapi-schema.json is composed of modules connected by include directives, but the generated code is monolithic all the same: one qapi-types.h with all the types, one qapi-visit.h with all the visitors, and so forth. These monolithic headers get included all over the place. In my "build everything" tree, adding a QAPI type recompiles about 4800 out of 5100 objects.

We wouldn't write such monolithic headers by hand. It stands to reason that we shouldn't generate them, either.

Split up generated qapi-types.h to mirror the schema's modular
structure: one header per module. Name the main module's header qapi-types.h, and sub-module D/B.json's header D/qapi-types-B.h.

Quoted 7 lines

Mirror the schema's includes in the headers, so that qapi-types.h gets you everything exactly as before. If you need less, you can include one or more of the sub-module headers. To be exploited shortly.

Split up qapi-types.c, qapi-visit.h, qapi-visit.c, qmp-commands.h, qmp-commands.c, qapi-event.h, qapi-event.c the same way. qmp-introspect.h, qmp-introspect.c and qapi.texi remain monolithic.

Make sense.

Quoted 5 lines

The split of qmp-commands.c duplicates static helper function qmp_marshal_output_str() in qapi-commands-char.c and qapi-commands-misc.c. This happens when commands returning the same type occur in multiple modules. Not worth avoiding.

As long as it is static, and neither .c file includes the other, we're fine (gdb may have a harder time figuring out which copy to put a breakpoint on, but that's not too terrible).

Quoted 5 lines

Since I'm going to rename qapi-event.[ch] to qapi-events.[ch], and qmp-commands.[ch] to qapi-commands.[ch], name the shards that way already, to reduce churn. This requires temporary hacks in commands.py and events.py. They'll go away with the rename.

The planned rename makes sense; prepping now is fine.

Quoted 16 lines
---
  .gitignore               |  60 ++++++++++++++++++++++++
  Makefile                 | 120 +++++++++++++++++++++++++++++++++++++++++++++++
  Makefile.objs            |  65 ++++++++++++++++++++++++-
  scripts/qapi/commands.py |  35 +++++++++-----
  scripts/qapi/common.py   |  21 +++++++--
  scripts/qapi/events.py   |  19 ++++++--
  6 files changed, 300 insertions(+), 20 deletions(-)
.gitignore +0 -0
index 9477a08b6b..42c57998fd 100644

.gitignore line 35

Quoted 5 lines
.gitignore +2 -0
@@ -31,7 +31,67 @@
  /qapi-gen-timestamp
  /qapi-builtin-types.[ch]
  /qapi-builtin-visit.[ch]
+/qapi/qapi-commands-block-core.[ch]
+/qapi/qapi-commands-block.[ch]

Is it worth using a glob instead of specific patterns, as in:

Quoted 5 lines

/qapi/qapi-commands-*.[ch]

Maybe being precise doesn't hurt, if we aren't adding sub-modules all that often; but being generous with a glob makes it less likely that a future module addition will forget to update .gitignore.

We should simply ditch building in-tree. I wanted to post patches for that for quite some time.

Quoted 11 lines

.gitignore include $(SRC_PATH)/rules.mak · line 95

Quoted 6 lines
.gitignore +2 -0
+++ b/Makefile
@@ -92,10 +92,70 @@ include $(SRC_PATH)/rules.mak
  GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
  GENERATED_FILES += qapi-builtin-types.h qapi-builtin-types.c
  GENERATED_FILES += qapi-types.h qapi-types.c
+GENERATED_FILES += qapi/qapi-types-block-core.h qapi/qapi-types-block-core.c

For the makefile, I'd suggest using an intermediate list of module names, and then using make string operations to expand it into larger lists, to cut down on the repetition. Something like (untested):

Quoted 5 lines

QAPI_MODULES = block-core block char common ... GENERATED_FILES += $(addprefix qapi/qapi-types-,$(addsuffix .c,$(QAPI_MODULES))
GENERATED_FILES += $(addprefix qapi/qapi-types-,$(addsuffix .h,$(QAPI_MODULES))

I did it the stupid way to save time. Improvements welcome :)

Quoted 34 lines

.gitignore qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \ · line 588

Quoted 5 lines
.gitignore +2 -0
@@ -525,10 +585,70 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
    qapi-builtin-types.c qapi-builtin-types.h \
  qapi-types.c qapi-types.h \
+qapi/qapi-types-block-core.c qapi/qapi-types-block-core.h \
+qapi/qapi-types-block.c qapi/qapi-types-block.h \

And repeating the list of filenames; again, an intermediate variable would let you do:

QAPI_GENERATED_FILES = ...
GENERATED_FILES += $(QAPI_GENERATED_FILES)

$(QAPI_GENERATED_FILES): ...

.gitignore qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \ · line 591

Quoted 3 lines
.gitignore +0 -0
  qmp-introspect.h qmp-introspect.c \
  qapi-doc.texi: \
  qapi-gen-timestamp ;

Not only does it cut down on the repetition, but it will make adding a future module easier.

Makefile.objs line 7

Quoted 10 lines
Makefile.objs +2 -0
index 2813e984fd..7a55d45669 100644
@@ -3,8 +3,56 @@
  stub-obj-y = stubs/ crypto/
  util-obj-y = util/ qobject/ qapi/
  util-obj-y += qapi-builtin-types.o
+util-obj-y += qapi-types.o
+util-obj-y += qapi/qapi-types-block-core.o

Perhaps this can also exploit make text manipulation?

Quoted 1 line

Otherwise looks good.

Thanks!

Reviewed-by
#77 lore raw

Re: [Qemu-devel] [PATCH v2 22/29] qapi: Generate separate .h, .c for each module

Quoted 30 lines · Markus Armbruster
Quoted 11 lines

Makefile include $(SRC_PATH)/rules.mak · line 95

Quoted 6 lines

+++ b/Makefile

Makefile +1 -0
@@ -92,10 +92,70 @@ include $(SRC_PATH)/rules.mak
   GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
   GENERATED_FILES += qapi-builtin-types.h qapi-builtin-types.c
   GENERATED_FILES += qapi-types.h qapi-types.c
+GENERATED_FILES += qapi/qapi-types-block-core.h qapi/qapi-types-block-core.c

For the makefile, I'd suggest using an intermediate list of module names, and then using make string operations to expand it into larger lists, to cut down on the repetition. Something like (untested):

Quoted 5 lines

QAPI_MODULES = block-core block char common ... GENERATED_FILES += $(addprefix qapi/qapi-types-,$(addsuffix .c,$(QAPI_MODULES))
GENERATED_FILES += $(addprefix qapi/qapi-types-,$(addsuffix .h,$(QAPI_MODULES))

I did it the stupid way to save time. Improvements welcome :)

Quoted 10 lines

Makefile.objs line 7

Quoted 7 lines
Makefile.objs +3 -0
+++ b/Makefile.objs
@@ -3,8 +3,56 @@
   stub-obj-y = stubs/ crypto/
   util-obj-y = util/ qobject/ qapi/
   util-obj-y += qapi-builtin-types.o
+util-obj-y += qapi-types.o
+util-obj-y += qapi/qapi-types-block-core.o

Perhaps this can also exploit make text manipulation?

Quoted 3 lines
Quoted 1 line

Otherwise looks good.

Thanks!

At this point, I'll take your patch as-is, and worry about improvements as followups.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#78 lore raw

PATCH v2 23/29 Include less of the generated modular QAPI headers

In reply to Markus Armbruster

In my "build everything" tree, a change to the types in qapi-schema.json triggers a recompile of about 4800 out of 5100 objects.

The previous commit split up qmp-commands.h, qmp-event.h, qmp-visit.h, qapi-types.h. Each of these headers still includes all its shards. Reduce compile time by including just the shards we actually need.

To illustrate the benefits: adding a type to qapi/migration.json now recompiles some 2300 instead of 4800 objects. The next commit will improve it further.

---
 backends/cryptodev.c             |  1 -
 backends/hostmem.c               |  3 ++-
 block.c                          |  1 -
 block/block-backend.c            |  2 +-
 block/crypto.c                   |  2 +-
 block/nbd.c                      |  2 +-
 block/nfs.c                      |  2 +-
 block/qapi.c                     |  4 ++--
 block/qcow2.c                    |  3 +--
 block/quorum.c                   |  2 +-
 block/sheepdog.c                 |  2 +-
 block/ssh.c                      |  2 +-
 block/throttle-groups.c          |  2 +-
 block/write-threshold.c          |  4 ++--
 blockdev-nbd.c                   |  2 +-
 blockdev.c                       |  5 +++--
 blockjob.c                       |  2 +-
 chardev/char-fe.c                |  1 -
 chardev/char-ringbuf.c           |  2 +-
 chardev/char-socket.c            |  1 +
 chardev/char.c                   |  3 +--
 cpus.c                           |  2 +-
 crypto/cipherpriv.h              |  2 +-
 hmp.c                            |  2 +-
 hw/acpi/core.c                   |  2 +-
 hw/block/block.c                 |  1 +
 hw/block/hd-geometry.c           |  1 +
 hw/char/virtio-console.c         |  2 +-
 hw/core/machine.c                |  2 +-
 hw/i386/pc.c                     |  2 +-
 hw/mem/nvdimm.c                  |  1 -
 hw/net/rocker/qmp-norocker.c     |  2 +-
 hw/net/rocker/rocker.c           |  2 +-
 hw/net/rocker/rocker_fp.c        |  2 +-
 hw/net/rocker/rocker_of_dpa.c    |  2 +-
 hw/net/virtio-net.c              |  2 +-
 hw/ppc/spapr_rtas.c              |  1 -
 hw/tpm/tpm_emulator.c            |  1 +
 hw/tpm/tpm_passthrough.c         |  1 +
 hw/watchdog/watchdog.c           |  2 +-
 include/block/block.h            |  2 +-
 include/block/dirty-bitmap.h     |  2 +-
 include/block/nbd.h              |  2 +-
 include/chardev/char.h           |  1 +
 include/crypto/cipher.h          |  2 +-
 include/crypto/hash.h            |  2 +-
 include/crypto/hmac.h            |  2 +-
 include/crypto/secret.h          |  1 +
 include/crypto/tlscreds.h        |  1 +
 include/hw/block/block.h         |  2 +-
 include/hw/block/fdc.h           |  2 +-
 include/hw/ppc/spapr_drc.h       |  1 +
 include/hw/qdev-properties.h     |  1 +
 include/io/dns-resolver.h        |  1 +
 include/migration/colo.h         |  2 +-
 include/migration/failover.h     |  2 +-
 include/migration/global_state.h |  1 +
 include/monitor/monitor.h        |  1 +
 include/net/filter.h             |  1 +
 include/net/net.h                |  2 +-
 include/qapi/clone-visitor.h     |  1 -
 include/qapi/error.h             |  2 +-
 include/qapi/qmp/qobject.h       |  2 +-
 include/qapi/visitor.h           |  2 +-
 include/qemu/sockets.h           |  2 +-
 include/qemu/throttle.h          |  2 +-
 include/qom/cpu.h                |  1 +
 include/qom/object.h             |  2 +-
 include/sysemu/dump.h            |  2 ++
 include/sysemu/hostmem.h         |  1 +
 include/sysemu/replay.h          |  1 +
 include/sysemu/sysemu.h          |  1 +
 include/sysemu/tpm.h             |  1 +
 include/sysemu/watchdog.h        |  2 +-
 include/ui/input.h               |  2 +-
 io/channel-socket.c              |  1 +
 io/dns-resolver.c                |  1 +
 migration/colo-failover.c        |  2 +-
 migration/colo.c                 |  2 +-
 migration/migration.c            |  4 ++--
 migration/migration.h            |  1 +
 migration/ram.c                  |  2 +-
 migration/ram.h                  |  2 +-
 net/colo-compare.c               |  1 -
 net/filter-buffer.c              |  2 +-
 net/filter-mirror.c              |  1 -
 net/filter-rewriter.c            |  1 -
 net/net.c                        |  4 ++--
 net/tap_int.h                    |  2 +-
 net/vhost-user.c                 |  2 +-
 qemu-img.c                       |  2 +-
 qom/object.c                     |  2 +-
 qom/object_interfaces.c          |  1 -
 replay/replay-input.c            |  1 +
 replication.h                    |  1 +
 scripts/qapi/commands.py         | 14 ++++++++------
 scripts/qapi/events.py           | 10 ++++++----
 scripts/qapi/types.py            |  8 +++++---
 scripts/qapi/visit.py            | 10 ++++++----
 stubs/tpm.c                      |  3 ++-
 target/s390x/kvm.c               |  1 -
 tests/test-char.c                |  2 +-
 tests/test-qmp-event.c           |  1 -
 tpm.c                            |  2 +-
 trace/qmp.c                      |  2 +-
 ui/console.c                     |  2 +-
 ui/input-legacy.c                |  2 +-
 ui/input.c                       |  2 +-
 ui/spice-core.c                  |  4 ++--
 ui/vnc.c                         |  2 +-
 ui/vnc.h                         |  1 +
 util/qemu-sockets.c              |  2 +-
 vl.c                             |  4 ++--
 113 files changed, 133 insertions(+), 109 deletions(-)
backends/cryptodev.c +0 -1
index d0dff1a463..f35be377ef 100644
@@ -26,7 +26,6 @@
 #include "hw/boards.h"
 #include "qapi/error.h"
 #include "qapi/visitor.h"
-#include "qapi-visit.h"
 #include "qemu/config-file.h"
 #include "qom/object_interfaces.h"
 #include "hw/virtio/virtio-crypto.h"
backends/hostmem.c +2 -1
index 81d14554a7..0afaf7160a 100644
@@ -9,12 +9,13 @@
  * This work is licensed under the terms of the GNU GPL, version 2 or later.
  * See the COPYING file in the top-level directory.
  */
+
 #include "qemu/osdep.h"
 #include "sysemu/hostmem.h"
 #include "hw/boards.h"
 #include "qapi/error.h"
+#include "qapi-builtin-visit.h"
 #include "qapi/visitor.h"
-#include "qapi-visit.h"
 #include "qemu/config-file.h"
 #include "qom/object_interfaces.h"
 
block.c +0 -1
index 05a484b4b8..0b2784f322 100644
@@ -41,7 +41,6 @@
 #include "qemu/coroutine.h"
 #include "block/qapi.h"
 #include "qemu/timer.h"
-#include "qapi-event.h"
 #include "qemu/cutils.h"
 #include "qemu/id.h"
 
block/block-backend.c +1 -1
index 0266ac990b..94ffbb6a60 100644
@@ -17,8 +17,8 @@
 #include "block/throttle-groups.h"
 #include "sysemu/blockdev.h"
 #include "sysemu/sysemu.h"
-#include "qapi-event.h"
 #include "qapi/error.h"
+#include "qapi/qapi-events-block.h"
 #include "qemu/id.h"
 #include "qemu/option.h"
 #include "trace.h"
block/crypto.c +1 -1
index 70e3691cd8..c87abd5250 100644
@@ -24,9 +24,9 @@
 #include "sysemu/block-backend.h"
 #include "crypto/block.h"
 #include "qapi/opts-visitor.h"
+#include "qapi/qapi-visit-crypto.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qobject-input-visitor.h"
-#include "qapi-visit.h"
 #include "qapi/error.h"
 #include "qemu/option.h"
 #include "block/crypto.h"
block/nbd.c +1 -1
index 411eeb42a7..6c0e766b6f 100644
@@ -33,7 +33,7 @@
 #include "block/block_int.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
-#include "qapi-visit.h"
+#include "qapi/qapi-visit-sockets.h"
 #include "qapi/qobject-input-visitor.h"
 #include "qapi/qobject-output-visitor.h"
 #include "qapi/qmp/qdict.h"
block/nfs.c +1 -1
index 6576a73d6e..bbdb4fadad 100644
@@ -35,9 +35,9 @@
 #include "qemu/uri.h"
 #include "qemu/cutils.h"
 #include "sysemu/sysemu.h"
+#include "qapi/qapi-visit-block-core.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
-#include "qapi-visit.h"
 #include "qapi/qobject-input-visitor.h"
 #include "qapi/qobject-output-visitor.h"
 #include <nfsc/libnfs.h>
block/qapi.c +2 -2
index 1fdeb1ef2f..4c9923d262 100644
@@ -27,10 +27,10 @@
 #include "block/block_int.h"
 #include "block/throttle-groups.h"
 #include "block/write-threshold.h"
-#include "qmp-commands.h"
-#include "qapi-visit.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-block-core.h"
 #include "qapi/qobject-output-visitor.h"
+#include "qapi/qapi-visit-block-core.h"
 #include "qapi/qmp/qbool.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qlist.h"
block/qcow2.c +1 -2
index 9245deac19..2fc58415bb 100644
@@ -30,15 +30,14 @@
 #include "block/qcow2.h"
 #include "qemu/error-report.h"
 #include "qapi/error.h"
+#include "qapi/qapi-events-block-core.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
-#include "qapi-event.h"
 #include "trace.h"
 #include "qemu/option_int.h"
 #include "qemu/cutils.h"
 #include "qemu/bswap.h"
 #include "qapi/opts-visitor.h"
-#include "qapi-visit.h"
 #include "block/crypto.h"
 
 /*
block/quorum.c +1 -1
index 19f1c34425..14333c18aa 100644
@@ -18,11 +18,11 @@
 #include "qemu/option.h"
 #include "block/block_int.h"
 #include "qapi/error.h"
+#include "qapi/qapi-events-block.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qlist.h"
 #include "qapi/qmp/qstring.h"
-#include "qapi-event.h"
 #include "crypto/hash.h"
 
 #define HASH_LENGTH 32
block/sheepdog.c +1 -1
index af125a2c8d..4c20fcda04 100644
@@ -13,8 +13,8 @@
  */
 
 #include "qemu/osdep.h"
-#include "qapi-visit.h"
 #include "qapi/error.h"
+#include "qapi/qapi-visit-sockets.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qobject-input-visitor.h"
 #include "qemu/uri.h"
block/ssh.c +1 -1
index b63addcf94..b11d4c5e86 100644
@@ -34,7 +34,7 @@
 #include "qemu/cutils.h"
 #include "qemu/sockets.h"
 #include "qemu/uri.h"
-#include "qapi-visit.h"
+#include "qapi/qapi-visit-sockets.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
 #include "qapi/qobject-input-visitor.h"
block/throttle-groups.c +1 -1
index f26bcb5eee..36cc0430c3 100644
@@ -30,7 +30,7 @@
 #include "qemu/thread.h"
 #include "sysemu/qtest.h"
 #include "qapi/error.h"
-#include "qapi-visit.h"
+#include "qapi/qapi-visit-block-core.h"
 #include "qom/object.h"
 #include "qom/object_interfaces.h"
 
block/write-threshold.c +2 -2
index db3de0fa6d..1d48fc2077 100644
@@ -15,9 +15,9 @@
 #include "qemu/coroutine.h"
 #include "block/write-threshold.h"
 #include "qemu/notify.h"
-#include "qapi-event.h"
 #include "qapi/error.h"
-#include "qmp-commands.h"
+#include "qapi/qapi-commands-block-core.h"
+#include "qapi/qapi-events-block-core.h"
 
 uint64_t bdrv_write_threshold_get(const BlockDriverState *bs)
 {
blockdev-nbd.c +1 -1
index 3a5479bdad..65a84739ed 100644
@@ -14,8 +14,8 @@
 #include "sysemu/block-backend.h"
 #include "hw/block/block.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-block.h"
 #include "sysemu/sysemu.h"
-#include "qmp-commands.h"
 #include "block/nbd.h"
 #include "io/channel-socket.h"
 #include "io/net-listener.h"
blockdev.c +3 -2
index bdbdeae7e4..85e011191b 100644
@@ -40,10 +40,12 @@
 #include "qemu/error-report.h"
 #include "qemu/option.h"
 #include "qemu/config-file.h"
+#include "qapi/qapi-commands-block.h"
+#include "qapi/qapi-commands-transaction.h"
+#include "qapi/qapi-visit-block-core.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qnum.h"
 #include "qapi/qmp/qstring.h"
-#include "qapi-visit.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qlist.h"
@@ -51,7 +53,6 @@
 #include "sysemu/sysemu.h"
 #include "sysemu/iothread.h"
 #include "block/block_int.h"
-#include "qmp-commands.h"
 #include "block/trace.h"
 #include "sysemu/arch_init.h"
 #include "sysemu/qtest.h"
blockjob.c +1 -1
index 3f52f29f75..801d29d849 100644
@@ -30,11 +30,11 @@
 #include "block/block_int.h"
 #include "sysemu/block-backend.h"
 #include "qapi/error.h"
+#include "qapi/qapi-events-block-core.h"
 #include "qapi/qmp/qerror.h"
 #include "qemu/coroutine.h"
 #include "qemu/id.h"
 #include "qemu/timer.h"
-#include "qapi-event.h"
 
 /* Right now, this mutex is only needed to synchronize accesses to job->busy
  * and job->sleep_timer, such as concurrent calls to block_job_do_yield and
chardev/char-fe.c +0 -1
index e5f870e4d2..392db78b13 100644
@@ -25,7 +25,6 @@
 #include "qemu/error-report.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qerror.h"
-#include "qapi-visit.h"
 #include "sysemu/replay.h"
 
 #include "chardev/char-fe.h"
chardev/char-ringbuf.c +1 -1
index 679afaa4fd..87832e2792 100644
@@ -24,8 +24,8 @@
 
 #include "qemu/osdep.h"
 #include "chardev/char.h"
-#include "qmp-commands.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-char.h"
 #include "qemu/base64.h"
 #include "qemu/option.h"
 
chardev/char-socket.c +1 -0
index bdd6cff5f6..22f65971a1 100644
@@ -31,6 +31,7 @@
 #include "qemu/option.h"
 #include "qapi/error.h"
 #include "qapi/clone-visitor.h"
+#include "qapi/qapi-visit-sockets.h"
 
 #include "chardev/char-io.h"
 
chardev/char.c +1 -2
index c9a4da5516..5d7b079ef0 100644
@@ -29,9 +29,8 @@
 #include "qemu/config-file.h"
 #include "qemu/error-report.h"
 #include "chardev/char.h"
-#include "qmp-commands.h"
-#include "qapi-visit.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-char.h"
 #include "qapi/qmp/qerror.h"
 #include "sysemu/replay.h"
 #include "qemu/help_option.h"
cpus.c +1 -1
index f298b659f4..ddb7f9b811 100644
@@ -27,6 +27,7 @@
 #include "cpu.h"
 #include "monitor/monitor.h"
 #include "qapi/error.h"
+#include "qapi/qapi-events-run-state.h"
 #include "qapi/qmp/qerror.h"
 #include "qemu/error-report.h"
 #include "sysemu/sysemu.h"
@@ -49,7 +50,6 @@
 #include "qemu/bitmap.h"
 #include "qemu/seqlock.h"
 #include "tcg.h"
-#include "qapi-event.h"
 #include "hw/nmi.h"
 #include "sysemu/replay.h"
 #include "hw/boards.h"
crypto/cipherpriv.h +1 -1
index 77da4c2f32..0823239f41 100644
@@ -15,7 +15,7 @@
 #ifndef QCRYPTO_CIPHERPRIV_H
 #define QCRYPTO_CIPHERPRIV_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 typedef struct QCryptoCipherDriver QCryptoCipherDriver;
 
hmp.c +1 -1
index 7870d6a300..a49f4c126e 100644
@@ -29,11 +29,11 @@
 #include "monitor/qdev.h"
 #include "qapi/error.h"
 #include "qapi/opts-visitor.h"
+#include "qapi-builtin-visit.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/string-input-visitor.h"
 #include "qapi/string-output-visitor.h"
-#include "qapi-visit.h"
 #include "qom/object_interfaces.h"
 #include "ui/console.h"
 #include "block/nbd.h"
hw/acpi/core.c +1 -1
index b50b3ca772..5d4ce925dc 100644
@@ -27,8 +27,8 @@
 #include "qemu/config-file.h"
 #include "qapi/error.h"
 #include "qapi/opts-visitor.h"
+#include "qapi/qapi-events-run-state.h"
 #include "qapi-visit.h"
-#include "qapi-event.h"
 #include "qemu/error-report.h"
 #include "qemu/option.h"
 
hw/block/block.c +1 -0
index b0269c857f..b91e2b6d7e 100644
@@ -12,6 +12,7 @@
 #include "sysemu/block-backend.h"
 #include "hw/block/block.h"
 #include "qapi/error.h"
+#include "qapi/qapi-types-block.h"
 #include "qemu/error-report.h"
 
 void blkconf_serial(BlockConf *conf, char **serial)
hw/block/hd-geometry.c +1 -0
index 57ad5012a7..79384a2b0a 100644
@@ -32,6 +32,7 @@
 
 #include "qemu/osdep.h"
 #include "sysemu/block-backend.h"
+#include "qapi/qapi-types-block.h"
 #include "qemu/bswap.h"
 #include "hw/block/block.h"
 #include "trace.h"
hw/char/virtio-console.c +1 -1
index 4be5d4ee52..679a824888 100644
@@ -15,8 +15,8 @@
 #include "qemu/error-report.h"
 #include "trace.h"
 #include "hw/virtio/virtio-serial.h"
-#include "qapi-event.h"
 #include "qapi/error.h"
+#include "qapi/qapi-events-char.h"
 
 #define TYPE_VIRTIO_CONSOLE_SERIAL_PORT "virtserialport"
 #define VIRTIO_CONSOLE(obj) \
hw/core/machine.c +1 -1
index cdc1163dc6..f35075bff6 100644
@@ -13,7 +13,7 @@
 #include "qemu/osdep.h"
 #include "hw/boards.h"
 #include "qapi/error.h"
-#include "qapi-visit.h"
+#include "qapi/qapi-visit-common.h"
 #include "qapi/visitor.h"
 #include "hw/sysbus.h"
 #include "sysemu/sysemu.h"
hw/i386/pc.c +1 -1
index 55e69d66fe..94cfd40ef2 100644
@@ -67,8 +67,8 @@
 #include "acpi-build.h"
 #include "hw/mem/pc-dimm.h"
 #include "qapi/error.h"
+#include "qapi/qapi-visit-common.h"
 #include "qapi/visitor.h"
-#include "qapi-visit.h"
 #include "qom/cpu.h"
 #include "hw/nmi.h"
 #include "hw/i386/intel_iommu.h"
hw/mem/nvdimm.c +0 -1
index 61e677f92f..acb656b672 100644
@@ -25,7 +25,6 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qapi/visitor.h"
-#include "qapi-visit.h"
 #include "hw/mem/nvdimm.h"
 
 static void nvdimm_get_label_size(Object *obj, Visitor *v, const char *name,
hw/net/rocker/qmp-norocker.c +1 -1
index 94c1e480ae..0d60513c01 100644
@@ -17,8 +17,8 @@
 
 #include "qemu/osdep.h"
 #include "qemu-common.h"
-#include "qmp-commands.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-rocker.h"
 #include "qapi/qmp/qerror.h"
 
 RockerSwitch *qmp_query_rocker(const char *name, Error **errp)
hw/net/rocker/rocker.c +1 -1
index a2a76c2a74..c02cbefece 100644
@@ -22,9 +22,9 @@
 #include "net/net.h"
 #include "net/eth.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-rocker.h"
 #include "qemu/iov.h"
 #include "qemu/bitops.h"
-#include "qmp-commands.h"
 
 #include "rocker.h"
 #include "rocker_hw.h"
hw/net/rocker/rocker_fp.c +1 -1
index 4b3c9847db..27b17c890f 100644
@@ -16,7 +16,7 @@
 
 #include "qemu/osdep.h"
 #include "net/clients.h"
-
+#include "qapi/qapi-types-rocker.h"
 #include "rocker.h"
 #include "rocker_hw.h"
 #include "rocker_fp.h"
hw/net/rocker/rocker_of_dpa.c +1 -1
index 9339df2d09..60046720a5 100644
@@ -17,9 +17,9 @@
 #include "qemu/osdep.h"
 #include "net/eth.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-rocker.h"
 #include "qemu/iov.h"
 #include "qemu/timer.h"
-#include "qmp-commands.h"
 
 #include "rocker.h"
 #include "rocker_hw.h"
hw/net/virtio-net.c +1 -1
index 369d40b378..188744e17d 100644
@@ -23,7 +23,7 @@
 #include "net/vhost_net.h"
 #include "hw/virtio/virtio-bus.h"
 #include "qapi/error.h"
-#include "qapi-event.h"
+#include "qapi/qapi-events-net.h"
 #include "hw/virtio/virtio-access.h"
 #include "migration/misc.h"
 
hw/ppc/spapr_rtas.c +0 -1
index 4bb939d3d1..0ec5fa4cfe 100644
@@ -38,7 +38,6 @@
 #include "hw/ppc/spapr_vio.h"
 #include "hw/ppc/spapr_rtas.h"
 #include "hw/ppc/ppc.h"
-#include "qapi-event.h"
 #include "hw/boards.h"
 
 #include <libfdt.h>
hw/tpm/tpm_emulator.c +1 -0
index b787aee13b..f187a72c10 100644
@@ -38,6 +38,7 @@
 #include "migration/blocker.h"
 #include "qapi/error.h"
 #include "qapi/clone-visitor.h"
+#include "qapi/qapi-visit-tpm.h"
 #include "chardev/char-fe.h"
 
 #define DEBUG_TPM 0
hw/tpm/tpm_passthrough.c +1 -0
index a495fe07f4..211df3191c 100644
@@ -30,6 +30,7 @@
 #include "tpm_int.h"
 #include "hw/hw.h"
 #include "qapi/clone-visitor.h"
+#include "qapi/qapi-visit-tpm.h"
 #include "tpm_util.h"
 
 #define DEBUG_TPM 0
hw/watchdog/watchdog.c +1 -1
index 98a5dd6689..c7843d5748 100644
@@ -24,9 +24,9 @@
 #include "qemu/config-file.h"
 #include "qemu/queue.h"
 #include "qapi/error.h"
+#include "qapi/qapi-events-run-state.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/watchdog.h"
-#include "qapi-event.h"
 #include "hw/nmi.h"
 #include "qemu/help_option.h"
 #include "qmp-commands.h"
include/block/block.h +1 -1
index 24ef816960..a1936996db 100644
@@ -2,7 +2,7 @@
 #define BLOCK_H
 
 #include "block/aio.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/iov.h"
 #include "qemu/coroutine.h"
 #include "block/accounting.h"
include/block/dirty-bitmap.h +1 -1
index 3da8486ab1..1454be358d 100644
@@ -2,7 +2,7 @@
 #define BLOCK_DIRTY_BITMAP_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/hbitmap.h"
 
 BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
include/block/nbd.h +1 -1
index fc50003003..631290dac9 100644
@@ -20,7 +20,7 @@
 #ifndef NBD_H
 #define NBD_H
 
-
+#include "qapi/qapi-types-block.h"
 #include "io/channel-socket.h"
 #include "crypto/tlscreds.h"
 
include/chardev/char.h +1 -0
index a381dc3df8..ebf1e0ba04 100644
@@ -1,6 +1,7 @@
 #ifndef QEMU_CHAR_H
 #define QEMU_CHAR_H
 
+#include "qapi/qapi-types-char.h"
 #include "qemu/main-loop.h"
 #include "qemu/bitmap.h"
 #include "qom/object.h"
include/crypto/cipher.h +1 -1
index 984fb8243f..bce2d4c8e4 100644
@@ -21,7 +21,7 @@
 #ifndef QCRYPTO_CIPHER_H
 #define QCRYPTO_CIPHER_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 typedef struct QCryptoCipher QCryptoCipher;
 
include/crypto/hash.h +1 -1
index ca3267f3df..077ac7bea0 100644
@@ -21,7 +21,7 @@
 #ifndef QCRYPTO_HASH_H
 #define QCRYPTO_HASH_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 /* See also "QCryptoHashAlgorithm" defined in qapi/crypto.json */
 
include/crypto/hmac.h +1 -1
index 5e88905989..aa3c97a2ff 100644
@@ -12,7 +12,7 @@
 #ifndef QCRYPTO_HMAC_H
 #define QCRYPTO_HMAC_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-crypto.h"
 
 typedef struct QCryptoHmac QCryptoHmac;
 struct QCryptoHmac {
include/crypto/secret.h +1 -0
index 07a963e794..edd0e13236 100644
@@ -21,6 +21,7 @@
 #ifndef QCRYPTO_SECRET_H
 #define QCRYPTO_SECRET_H
 
+#include "qapi/qapi-types-crypto.h"
 #include "qom/object.h"
 
 #define TYPE_QCRYPTO_SECRET "secret"
include/crypto/tlscreds.h +1 -0
index ad47d88be7..6b011e1dbc 100644
@@ -21,6 +21,7 @@
 #ifndef QCRYPTO_TLSCREDS_H
 #define QCRYPTO_TLSCREDS_H
 
+#include "qapi/qapi-types-crypto.h"
 #include "qom/object.h"
 
 #ifdef CONFIG_GNUTLS
include/hw/block/block.h +1 -1
index f532d10e35..d4f4dfffab 100644
@@ -12,7 +12,7 @@
 #define HW_BLOCK_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 
 /* Configuration */
 
include/hw/block/fdc.h +1 -1
index 68a0c904ea..3b813c7f7d 100644
@@ -2,7 +2,7 @@
 #define HW_FDC_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block.h"
 
 /* fdc.c */
 #define MAX_FD 2
include/hw/ppc/spapr_drc.h +1 -0
index f8d9f5b231..f6ff32e7e2 100644
@@ -14,6 +14,7 @@
 #define HW_SPAPR_DRC_H
 
 #include <libfdt.h>
+#include "qapi/qapi-types-run-state.h"
 #include "qom/object.h"
 #include "sysemu/sysemu.h"
 #include "hw/qdev.h"
include/hw/qdev-properties.h +1 -0
index 1d61a35108..40c2c8acb1 100644
@@ -1,6 +1,7 @@
 #ifndef QEMU_QDEV_PROPERTIES_H
 #define QEMU_QDEV_PROPERTIES_H
 
+#include "qapi-types.h"
 #include "hw/qdev-core.h"
 
 /*** qdev-properties.c ***/
include/io/dns-resolver.h +1 -0
index 2f69c08c13..1a162185cc 100644
@@ -22,6 +22,7 @@
 #define QIO_DNS_RESOLVER_H
 
 #include "qemu-common.h"
+#include "qapi/qapi-types-sockets.h"
 #include "qom/object.h"
 #include "io/task.h"
 
include/migration/colo.h +1 -1
index 50ace16205..2fe48ad353 100644
@@ -14,7 +14,7 @@
 #define QEMU_COLO_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-migration.h"
 
 void colo_info_init(void);
 
include/migration/failover.h +1 -1
index ad91ef2381..4c37218dcc 100644
@@ -14,7 +14,7 @@
 #define QEMU_FAILOVER_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-migration.h"
 
 void failover_init_state(void);
 FailoverStatus failover_set_state(FailoverStatus old_state,
include/migration/global_state.h +1 -0
index d307de8350..fd22dd3034 100644
@@ -13,6 +13,7 @@
 #ifndef QEMU_MIGRATION_GLOBAL_STATE_H
 #define QEMU_MIGRATION_GLOBAL_STATE_H
 
+#include "qapi/qapi-types-run-state.h"
 #include "sysemu/sysemu.h"
 
 void register_global_state(void);
include/monitor/monitor.h +1 -0
index ad64ad8e68..50f7cea057 100644
@@ -3,6 +3,7 @@
 
 #include "qemu-common.h"
 #include "block/block.h"
+#include "qapi-types.h"
 #include "qemu/readline.h"
 
 extern Monitor *cur_mon;
include/net/filter.h +1 -0
index 0c4a2ea6c9..435acd6f82 100644
@@ -9,6 +9,7 @@
 #ifndef QEMU_NET_FILTER_H
 #define QEMU_NET_FILTER_H
 
+#include "qapi/qapi-types-net.h"
 #include "qom/object.h"
 #include "qemu-common.h"
 #include "net/queue.h"
include/net/net.h +1 -1
index 3fc48e4f51..727643032c 100644
@@ -2,7 +2,7 @@
 #define QEMU_NET_H
 
 #include "qemu/queue.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-net.h"
 #include "net/queue.h"
 #include "migration/vmstate.h"
 
include/qapi/clone-visitor.h +0 -1
index b119d3daa9..5b665ee38c 100644
@@ -12,7 +12,6 @@
 #define QAPI_CLONE_VISITOR_H
 
 #include "qapi/visitor.h"
-#include "qapi-visit.h"
 
 /*
  * The clone visitor is for direct use only by the QAPI_CLONE() macro;
include/qapi/error.h +1 -1
index c2115a6a4a..bcb86a79f5 100644
@@ -115,7 +115,7 @@
 #ifndef ERROR_H
 #define ERROR_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-common.h"
 
 /*
  * Overall category of an error.
include/qapi/qmp/qobject.h +1 -1
index 38ac68845c..a2964fbf25 100644
@@ -32,7 +32,7 @@
 #ifndef QOBJECT_H
 #define QOBJECT_H
 
-#include "qapi-types.h"
+#include "qapi-builtin-types.h"
 
 struct QObject {
     QType type;
include/qapi/visitor.h +1 -1
index ecff296c11..9e57508446 100644
@@ -15,7 +15,7 @@
 #ifndef QAPI_VISITOR_H
 #define QAPI_VISITOR_H
 
-#include "qapi-types.h"
+#include "qapi-builtin-types.h"
 
 /*
  * The QAPI schema defines both a set of C data types, and a QMP wire
include/qemu/sockets.h +1 -1
index 8889bcb1ec..e88d4c37ab 100644
@@ -9,7 +9,7 @@ int inet_aton(const char *cp, struct in_addr *ia);
 
 #endif /* !_WIN32 */
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-sockets.h"
 
 /* misc helpers */
 int qemu_socket(int domain, int type, int protocol);
include/qemu/throttle.h +1 -1
index 03d45f44f8..abeb886d93 100644
@@ -26,7 +26,7 @@
 #define THROTTLE_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/timer.h"
 
 #define THROTTLE_VALUE_MAX 1000000000000000LL
include/qom/cpu.h +1 -0
index aff88fa16f..dc6d4956a8 100644
@@ -24,6 +24,7 @@
 #include "disas/bfd.h"
 #include "exec/hwaddr.h"
 #include "exec/memattrs.h"
+#include "qapi/qapi-types-run-state.h"
 #include "qemu/bitmap.h"
 #include "qemu/queue.h"
 #include "qemu/thread.h"
include/qom/object.h +1 -1
index dc73d59660..5b5c016d8f 100644
@@ -14,7 +14,7 @@
 #ifndef QEMU_OBJECT_H
 #define QEMU_OBJECT_H
 
-#include "qapi-types.h"
+#include "qapi-builtin-types.h"
 #include "qemu/queue.h"
 
 struct TypeImpl;
include/sysemu/dump.h +2 -0
index c14bcfe8c6..2424e31425 100644
@@ -14,6 +14,8 @@
 #ifndef DUMP_H
 #define DUMP_H
 
+#include "qapi-types.h"
+
 #define MAKEDUMPFILE_SIGNATURE      "makedumpfile"
 #define MAX_SIZE_MDF_HEADER         (4096) /* max size of makedumpfile_header */
 #define TYPE_FLAT_HEADER            (1)    /* type of flattened format */
include/sysemu/hostmem.h +1 -0
index 621a3f9d42..6424f96df9 100644
@@ -14,6 +14,7 @@
 #define SYSEMU_HOSTMEM_H
 
 #include "sysemu/sysemu.h" /* for MAX_NODES */
+#include "qapi-types.h"
 #include "qom/object.h"
 #include "exec/memory.h"
 #include "qemu/bitmap.h"
include/sysemu/replay.h +1 -0
index dc8ae7b6b1..fb533ed9b6 100644
@@ -13,6 +13,7 @@
  */
 
 #include "sysemu.h"
+#include "qapi-types.h"
 
 /* replay clock kinds */
 enum ReplayClockKind {
include/sysemu/sysemu.h +1 -0
index 77bb3da582..bfbef9e69c 100644
@@ -2,6 +2,7 @@
 #define SYSEMU_H
 /* Misc. things related to the system emulator.  */
 
+#include "qapi/qapi-types-run-state.h"
 #include "qemu/queue.h"
 #include "qemu/timer.h"
 #include "qemu/notify.h"
include/sysemu/tpm.h +1 -0
index 32b753d4f3..9ae1ab6da3 100644
@@ -12,6 +12,7 @@
 #ifndef QEMU_TPM_H
 #define QEMU_TPM_H
 
+#include "qapi/qapi-types-tpm.h"
 #include "qom/object.h"
 
 int tpm_config_parse(QemuOptsList *opts_list, const char *optarg);
include/sysemu/watchdog.h +1 -1
index 677ace3945..a08d16380d 100644
@@ -23,7 +23,7 @@
 #define QEMU_WATCHDOG_H
 
 #include "qemu/queue.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-run-state.h"
 
 struct WatchdogTimerModel {
     QLIST_ENTRY(WatchdogTimerModel) entry;
include/ui/input.h +1 -1
index ceb22b8eef..16395ab8f2 100644
@@ -1,7 +1,7 @@
 #ifndef INPUT_H
 #define INPUT_H
 
-#include "qapi-types.h"
+#include "qapi/qapi-types-ui.h"
 
 #define INPUT_EVENT_MASK_KEY   (1<<INPUT_EVENT_KIND_KEY)
 #define INPUT_EVENT_MASK_BTN   (1<<INPUT_EVENT_KIND_BTN)
io/channel-socket.c +1 -0
index 563e297357..8359b6683a 100644
@@ -20,6 +20,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "qapi/qapi-visit-sockets.h"
 #include "io/channel-socket.h"
 #include "io/channel-watch.h"
 #include "trace.h"
io/dns-resolver.c +1 -0
index c072d121c3..8c924071c4 100644
@@ -21,6 +21,7 @@
 #include "qemu/osdep.h"
 #include "io/dns-resolver.h"
 #include "qapi/clone-visitor.h"
+#include "qapi/qapi-visit-sockets.h"
 #include "qemu/sockets.h"
 #include "qapi/error.h"
 #include "qemu/cutils.h"
migration/colo-failover.c +1 -1
index 891785cb63..0ae0c41221 100644
@@ -15,8 +15,8 @@
 #include "migration/failover.h"
 #include "qemu/main-loop.h"
 #include "migration.h"
-#include "qmp-commands.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-migration.h"
 #include "qapi/qmp/qerror.h"
 #include "qemu/error-report.h"
 #include "trace.h"
migration/colo.c +1 -1
index 245a46d59d..4381067ed4 100644
@@ -13,6 +13,7 @@
 #include "qemu/osdep.h"
 #include "sysemu/sysemu.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-migration.h"
 #include "qemu-file-channel.h"
 #include "migration.h"
 #include "qemu-file.h"
@@ -24,7 +25,6 @@
 #include "qemu/error-report.h"
 #include "migration/failover.h"
 #include "replication.h"
-#include "qmp-commands.h"
 
 static bool vmstate_loading;
 
migration/migration.c +2 -2
index 86d69120a6..d26c5999de 100644
@@ -31,15 +31,15 @@
 #include "migration/vmstate.h"
 #include "block/block.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-migration.h"
+#include "qapi/qapi-events-migration.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qnull.h"
 #include "qemu/rcu.h"
 #include "block.h"
 #include "postcopy-ram.h"
 #include "qemu/thread.h"
-#include "qmp-commands.h"
 #include "trace.h"
-#include "qapi-event.h"
 #include "exec/target_page.h"
 #include "io/channel-buffer.h"
 #include "migration/colo.h"
migration/migration.h +1 -0
index 848f638a20..e2f18256c9 100644
@@ -15,6 +15,7 @@
 #define QEMU_MIGRATION_H
 
 #include "qemu-common.h"
+#include "qapi/qapi-types-migration.h"
 #include "qemu/thread.h"
 #include "exec/cpu-common.h"
 #include "qemu/coroutine_int.h"
migration/ram.c +1 -1
index 8333d8e35e..1b30305faf 100644
@@ -29,7 +29,6 @@
 #include "qemu/osdep.h"
 #include "cpu.h"
 #include <zlib.h>
-#include "qapi-event.h"
 #include "qemu/cutils.h"
 #include "qemu/bitops.h"
 #include "qemu/bitmap.h"
@@ -44,6 +43,7 @@
 #include "migration/page_cache.h"
 #include "qemu/error-report.h"
 #include "qapi/error.h"
+#include "qapi/qapi-events-migration.h"
 #include "qapi/qmp/qerror.h"
 #include "trace.h"
 #include "exec/ram_addr.h"
migration/ram.h +1 -1
index f3a227b4fc..53f0021c51 100644
@@ -30,7 +30,7 @@
 #define QEMU_MIGRATION_RAM_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-migration.h"
 #include "exec/cpu-common.h"
 
 extern MigrationStats ram_counters;
net/colo-compare.c +0 -1
index 76e03fdb14..23b2d2c4cc 100644
@@ -25,7 +25,6 @@
 #include "net/queue.h"
 #include "chardev/char-fe.h"
 #include "qemu/sockets.h"
-#include "qapi-visit.h"
 #include "net/colo.h"
 #include "sysemu/iothread.h"
 
net/filter-buffer.c +1 -1
index 9ce96aaa35..7c487629f9 100644
@@ -13,8 +13,8 @@
 #include "qemu-common.h"
 #include "qemu/timer.h"
 #include "qemu/iov.h"
+#include "qapi-builtin-visit.h"
 #include "qapi/qmp/qerror.h"
-#include "qapi-visit.h"
 #include "qom/object.h"
 
 #define TYPE_FILTER_BUFFER "filter-buffer"
net/filter-mirror.c +0 -1
index bd78e25d12..3a61cf21e8 100644
@@ -14,7 +14,6 @@
 #include "net/net.h"
 #include "qemu-common.h"
 #include "qapi/error.h"
-#include "qapi-visit.h"
 #include "qom/object.h"
 #include "qemu/main-loop.h"
 #include "qemu/error-report.h"
net/filter-rewriter.c +0 -1
index 6201494ceb..62dad2d773 100644
@@ -16,7 +16,6 @@
 #include "net/net.h"
 #include "qemu-common.h"
 #include "qemu/error-report.h"
-#include "qapi-visit.h"
 #include "qom/object.h"
 #include "qemu/main-loop.h"
 #include "qemu/iov.h"
net/net.c +2 -2
index 7d42925258..547c499110 100644
@@ -33,18 +33,18 @@
 
 #include "monitor/monitor.h"
 #include "qemu/help_option.h"
+#include "qapi/qapi-commands-net.h"
+#include "qapi/qapi-visit-net.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qerror.h"
 #include "qemu/error-report.h"
 #include "qemu/sockets.h"
 #include "qemu/cutils.h"
 #include "qemu/config-file.h"
-#include "qmp-commands.h"
 #include "hw/qdev.h"
 #include "qemu/iov.h"
 #include "qemu/main-loop.h"
 #include "qemu/option.h"
-#include "qapi-visit.h"
 #include "qapi/error.h"
 #include "qapi/opts-visitor.h"
 #include "sysemu/sysemu.h"
net/tap_int.h +1 -1
index ae6888f74a..9f931d52d6 100644
@@ -27,7 +27,7 @@
 #define NET_TAP_INT_H
 
 #include "qemu-common.h"
-#include "qapi-types.h"
+#include "qapi/qapi-types-net.h"
 
 int tap_open(char *ifname, int ifname_size, int *vnet_hdr,
              int vnet_hdr_required, int mq_required, Error **errp);
net/vhost-user.c +1 -1
index cb45512506..98f3601049 100644
@@ -14,10 +14,10 @@
 #include "net/vhost-user.h"
 #include "chardev/char-fe.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-net.h"
 #include "qemu/config-file.h"
 #include "qemu/error-report.h"
 #include "qemu/option.h"
-#include "qmp-commands.h"
 #include "trace.h"
 
 typedef struct VhostUserState {
qemu-img.c +1 -1
index 56edc15218..40bf7aa7d1 100644
@@ -27,7 +27,7 @@
 
 #include "qemu-version.h"
 #include "qapi/error.h"
-#include "qapi-visit.h"
+#include "qapi/qapi-visit-block-core.h"
 #include "qapi/qobject-output-visitor.h"
 #include "qapi/qmp/qjson.h"
 #include "qapi/qmp/qdict.h"
qom/object.c +1 -1
index 5dcee4683c..81b4f7ac48 100644
@@ -16,9 +16,9 @@
 #include "qom/object_interfaces.h"
 #include "qemu/cutils.h"
 #include "qapi/visitor.h"
-#include "qapi-visit.h"
 #include "qapi/string-input-visitor.h"
 #include "qapi/string-output-visitor.h"
+#include "qapi-builtin-visit.h"
 #include "qapi/qmp/qerror.h"
 #include "trace.h"
 
qom/object_interfaces.c +0 -1
index 43d9aa0946..2f76e1f36d 100644
@@ -5,7 +5,6 @@
 #include "qom/object_interfaces.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
-#include "qapi-visit.h"
 #include "qapi/opts-visitor.h"
 #include "qemu/config-file.h"
 
replay/replay-input.c +1 -0
index 3ab1536bf7..6ee8b5f8db 100644
@@ -16,6 +16,7 @@
 #include "qemu/notify.h"
 #include "ui/input.h"
 #include "qapi/clone-visitor.h"
+#include "qapi/qapi-visit-ui.h"
 
 void replay_save_input_event(InputEvent *evt)
 {
replication.h +1 -0
index ece6ca6133..8faefe005f 100644
@@ -15,6 +15,7 @@
 #ifndef REPLICATION_H
 #define REPLICATION_H
 
+#include "qapi/qapi-types-block-core.h"
 #include "qemu/queue.h"
 
 typedef struct ReplicationOps ReplicationOps;
scripts/qapi/commands.py +8 -6
index a43bccb190..953449171b 100644
@@ -241,6 +241,9 @@ class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
 
     def _begin_module(self, name):
         self._visited_ret_types[self._genc] = set()
+        commands = self._module_basename('qapi-commands', name)
+        types = self._module_basename('qapi-types', name)
+        visit = self._module_basename('qapi-visit', name)
         self._genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
@@ -251,18 +254,17 @@ class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
 #include "qapi/qobject-input-visitor.h"
 #include "qapi/dealloc-visitor.h"
 #include "qapi/error.h"
-#include "%(prefix)sqapi-types.h"
-#include "%(prefix)sqapi-visit.h"
-#include "%(prefix)sqmp-commands.h"
+#include "%(visit)s.h"
+#include "%(commands)s.h"
 
 ''',
-                             prefix=self._prefix))
+                             commands=commands, visit=visit))
         self._genh.add(mcgen('''
-#include "%(prefix)sqapi-types.h"
+#include "%(types)s.h"
 #include "qapi/qmp/dispatch.h"
 
 ''',
-                             prefix=self._prefix))
+                             types=types))
 
     def visit_end(self):
         (genc, genh) = self._module[self._main_module]
scripts/qapi/events.py +6 -4
index 1e0b990f35..5ad6708491 100644
@@ -165,24 +165,26 @@ class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor):
         return basename
 
     def _begin_module(self, name):
+        types = self._module_basename('qapi-types', name)
+        visit = self._module_basename('qapi-visit', name)
         self._genc.add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "%(prefix)sqapi-event.h"
-#include "%(prefix)sqapi-visit.h"
+#include "%(visit)s.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qobject-output-visitor.h"
 #include "qapi/qmp-event.h"
 
 ''',
-                             prefix=self._prefix))
+                             visit=visit, prefix=self._prefix))
         self._genh.add(mcgen('''
 #include "qapi/util.h"
-#include "%(prefix)sqapi-types.h"
+#include "%(types)s.h"
 
 ''',
-                             prefix=self._prefix))
+                             types=types))
 
     def visit_end(self):
         self._genh.add(gen_enum(self._enum_name, self._event_names))
scripts/qapi/types.py +5 -3
index 59826b1162..2a3c502cf6 100644
@@ -185,13 +185,15 @@ class QAPISchemaGenTypeVisitor(QAPISchemaModularCVisitor):
 '''))
 
     def _begin_module(self, name):
+        types = self._module_basename('qapi-types', name)
+        visit = self._module_basename('qapi-visit', name)
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qapi/dealloc-visitor.h"
-#include "%(prefix)sqapi-types.h"
-#include "%(prefix)sqapi-visit.h"
+#include "%(types)s.h"
+#include "%(visit)s.h"
 ''',
-                                      prefix=self._prefix))
+                                      types=types, visit=visit))
         self._genh.preamble_add(mcgen('''
 #include "qapi-builtin-types.h"
 '''))
scripts/qapi/visit.py +6 -4
index 9b678e7263..de09966643 100644
@@ -284,20 +284,22 @@ class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor):
                                       prefix=prefix))
 
     def _begin_module(self, name):
+        types = self._module_basename('qapi-types', name)
+        visit = self._module_basename('qapi-visit', name)
         self._genc.preamble_add(mcgen('''
 #include "qemu/osdep.h"
 #include "qemu-common.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qerror.h"
-#include "%(prefix)sqapi-visit.h"
+#include "%(visit)s.h"
 ''',
-                                      prefix=self._prefix))
+                                      visit=visit, prefix=self._prefix))
         self._genh.preamble_add(mcgen('''
 #include "qapi-builtin-visit.h"
-#include "%(prefix)sqapi-types.h"
+#include "%(types)s.h"
 
 ''',
-                                      prefix=self._prefix))
+                                      types=types))
 
     def visit_enum_type(self, name, info, values, prefix):
         self._genh.add(gen_visit_decl(name, scalar=True))
stubs/tpm.c +2 -1
index c18aac1c73..6729bc8517 100644
@@ -4,9 +4,10 @@
  * This work is licensed under the terms of the GNU GPL, version 2 or later.
  * See the COPYING file in the top-level directory.
  */
+
 #include "qemu/osdep.h"
+#include "qapi/qapi-commands-tpm.h"
 #include "sysemu/tpm.h"
-#include "qmp-commands.h"
 
 int tpm_init(void)
 {
target/s390x/kvm.c +0 -1
index 0301e9d519..00db81f438 100644
@@ -41,7 +41,6 @@
 #include "exec/gdbstub.h"
 #include "exec/address-spaces.h"
 #include "trace.h"
-#include "qapi-event.h"
 #include "hw/s390x/s390-pci-inst.h"
 #include "hw/s390x/s390-pci-bus.h"
 #include "hw/s390x/ipl.h"
tests/test-char.c +1 -1
index b358620911..b3a77af085 100644
@@ -8,9 +8,9 @@
 #include "chardev/char-mux.h"
 #include "sysemu/sysemu.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-char.h"
 #include "qapi/qmp/qdict.h"
 #include "qom/qom-qobject.h"
-#include "qmp-commands.h"
 
 static bool quit;
 
tests/test-qmp-event.c +0 -1
index 8012341343..5fbe7e551f 100644
@@ -14,7 +14,6 @@
 #include "qemu/osdep.h"
 
 #include "qemu-common.h"
-#include "test-qapi-visit.h"
 #include "test-qapi-event.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qbool.h"
tpm.c +1 -1
index d11b10bed8..2db03a09b2 100644
@@ -15,12 +15,12 @@
 #include "qemu/osdep.h"
 
 #include "qapi/error.h"
+#include "qapi/qapi-commands-tpm.h"
 #include "qapi/qmp/qerror.h"
 #include "sysemu/tpm_backend.h"
 #include "sysemu/tpm.h"
 #include "qemu/config-file.h"
 #include "qemu/error-report.h"
-#include "qmp-commands.h"
 
 static QLIST_HEAD(, TPMBackend) tpm_backends =
     QLIST_HEAD_INITIALIZER(tpm_backends);
trace/qmp.c +1 -1
index ccd35cd840..756086c79f 100644
@@ -9,7 +9,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
-#include "qmp-commands.h"
+#include "qapi/qapi-commands-trace.h"
 #include "trace/control.h"
 
 
ui/console.c +1 -1
index 36584d039e..a789b8c408 100644
@@ -26,9 +26,9 @@
 #include "ui/console.h"
 #include "hw/qdev-core.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-ui.h"
 #include "qemu/option.h"
 #include "qemu/timer.h"
-#include "qmp-commands.h"
 #include "chardev/char-fe.h"
 #include "trace.h"
 #include "exec/memory.h"
ui/input-legacy.c +1 -1
index 92b37ccb90..e5d4db1d97 100644
@@ -23,9 +23,9 @@
  */
 
 #include "qemu/osdep.h"
+#include "qapi/qapi-commands-ui.h"
 #include "sysemu/sysemu.h"
 #include "ui/console.h"
-#include "qmp-commands.h"
 #include "ui/keymaps.h"
 #include "ui/input.h"
 
ui/input.c +1 -1
index 8bef0fb038..51b1019252 100644
@@ -1,9 +1,9 @@
 #include "qemu/osdep.h"
 #include "sysemu/sysemu.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-ui.h"
 #include "qapi/qmp/qdict.h"
 #include "qemu/error-report.h"
-#include "qmp-commands.h"
 #include "trace.h"
 #include "ui/input.h"
 #include "ui/console.h"
ui/spice-core.c +2 -2
index e449172fe9..ae8921a201 100644
@@ -28,14 +28,14 @@
 #include "qemu/queue.h"
 #include "qemu-x509.h"
 #include "qemu/sockets.h"
-#include "qmp-commands.h"
 #include "qapi/error.h"
+#include "qapi/qapi-commands-ui.h"
+#include "qapi/qapi-events-ui.h"
 #include "qemu/notify.h"
 #include "qemu/option.h"
 #include "migration/misc.h"
 #include "hw/hw.h"
 #include "ui/spice-display.h"
-#include "qapi-event.h"
 
 /* core bits */
 
ui/vnc.c +1 -1
index c715bae1cf..d6329c8bd5 100644
@@ -36,7 +36,7 @@
 #include "qemu/acl.h"
 #include "qemu/config-file.h"
 #include "qapi/error.h"
-#include "qmp-commands.h"
+#include "qapi/qapi-commands-ui.h"
 #include "ui/input.h"
 #include "qapi-event.h"
 #include "crypto/hash.h"
ui/vnc.h +1 -0
index 1ca062f332..7b29def77d 100644
@@ -28,6 +28,7 @@
 #define QEMU_VNC_H
 
 #include "qemu-common.h"
+#include "qapi/qapi-types-ui.h"
 #include "qemu/queue.h"
 #include "qemu/thread.h"
 #include "ui/console.h"
util/qemu-sockets.c +1 -1
index fbbef69f62..7f13e8a338 100644
@@ -24,11 +24,11 @@
 #include "monitor/monitor.h"
 #include "qapi/clone-visitor.h"
 #include "qapi/error.h"
+#include "qapi/qapi-visit-sockets.h"
 #include "qemu/sockets.h"
 #include "qemu/main-loop.h"
 #include "qapi/qobject-input-visitor.h"
 #include "qapi/qobject-output-visitor.h"
-#include "qapi-visit.h"
 #include "qemu/cutils.h"
 
 #ifndef AI_ADDRCONFIG
vl.c +2 -2
index 21878496ec..4e8287c507 100644
@@ -97,7 +97,6 @@ int main(int argc, char **argv)
 #include "sysemu/kvm.h"
 #include "sysemu/hax.h"
 #include "qapi/qobject-input-visitor.h"
-#include "qapi-visit.h"
 #include "qemu/option.h"
 #include "qemu/config-file.h"
 #include "qemu-options.h"
@@ -122,10 +121,11 @@ int main(int argc, char **argv)
 #include "qapi/string-input-visitor.h"
 #include "qapi/opts-visitor.h"
 #include "qom/object_interfaces.h"
-#include "qapi-event.h"
 #include "exec/semihost.h"
 #include "crypto/init.h"
 #include "sysemu/replay.h"
+#include "qapi/qapi-events-run-state.h"
+#include "qapi/qapi-visit-block-core.h"
 #include "qapi/qmp/qerror.h"
 #include "sysemu/iothread.h"
 
Signature · 1 line
2.13.6
Reviewed-by on v2 23/29
#79 lore raw

Re: [Qemu-devel] [PATCH v2 23/29] Include less of the generated modular QAPI headers

Quoted 8 lines · Markus Armbruster

In my "build everything" tree, a change to the types in qapi-schema.json triggers a recompile of about 4800 out of 5100 objects.

The previous commit split up qmp-commands.h, qmp-event.h, qmp-visit.h, qapi-types.h. Each of these headers still includes all its shards. Reduce compile time by including just the shards we actually need.

Quoted 8 lines

To illustrate the benefits: adding a type to qapi/migration.json now recompiles some 2300 instead of 4800 objects. The next commit will improve it further.

---
  backends/cryptodev.c             |  1 -
  backends/hostmem.c               |  3 ++-

How did you determine which shards to include where? Remove all shards, try compiling, and see what fails, then add back in the right includes? Or was it scripted somehow? (Trying to figure out, in case I have to rebase this patch)

block/crypto.c line 30

Quoted 10 lines

+++ b/block/crypto.c

block/crypto.c +1 -1
@@ -24,9 +24,9 @@
  #include "sysemu/block-backend.h"
  #include "crypto/block.h"
  #include "qapi/opts-visitor.h"
+#include "qapi/qapi-visit-crypto.h"
  #include "qapi/qmp/qdict.h"
  #include "qapi/qobject-input-visitor.h"
-#include "qapi-visit.h"
  #include "qapi/error.h"

Any rhyme or reason to the resulting include order that I should be looking for (we aren't alphabetical in general, but if your changes were trying to honor a particular pattern among the few affected lines, that's useful to know).

Review line 31

Quoted 1 line
(quoted diff) +1 -0
+++ b/scripts/qapi/commands.py

The non-mechanical portion of the patch :)

Quoted 21 lines
scripts/qapi/commands.py +5 -3
@@ -241,6 +241,9 @@ class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
 
      def _begin_module(self, name):
          self._visited_ret_types[self._genc] = set()
+        commands = self._module_basename('qapi-commands', name)
+        types = self._module_basename('qapi-types', name)
+        visit = self._module_basename('qapi-visit', name)
          self._genc.add(mcgen('''
  #include "qemu/osdep.h"
  #include "qemu-common.h"
@@ -251,18 +254,17 @@ class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
  #include "qapi/qobject-input-visitor.h"
  #include "qapi/dealloc-visitor.h"
  #include "qapi/error.h"
-#include "%(prefix)sqapi-types.h"
-#include "%(prefix)sqapi-visit.h"
-#include "%(prefix)sqmp-commands.h"
+#include "%(visit)s.h"
+#include "%(commands)s.h"
 
  ''',

scripts/qapi/commands.py class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor): · line 267

Quoted 10 lines
scripts/qapi/commands.py +3 -3
-                             prefix=self._prefix))
+                             commands=commands, visit=visit))
          self._genh.add(mcgen('''
-#include "%(prefix)sqapi-types.h"
+#include "%(types)s.h"
  #include "qapi/qmp/dispatch.h"
 
  ''',
-                             prefix=self._prefix))
+                             types=types))

Makes sense.

The compiler will catch anything we did wrong.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org