qapi: Module fixes and cleanups
cover qapi: Module fixes and cleanups
Kevin recently posted a minimally invasive fix for empty QAPI modules[*]. This is my attempt at a fix that also addresses the design weakness that led to the bug.
Markus Armbruster (6):
qapi: Tweak "command returns a nice type" check for clarity tests/Makefile.include: Fix missing test-qapi-emit-events.[ch]
qapi: Generate command registration stuff into separate files
qapi: Proper intermediate representation for modules
qapi: Fix code generation for empty modules
qapi: Simplify QAPISchemaModularCVisitor
docs/devel/qapi-code-gen.txt | 19 ++++- Makefile | 4 +- monitor/misc.c | 7 +- qga/main.c | 2 +- tests/test-qmp-cmds.c | 1 + .gitignore | 1 + qapi/Makefile.objs | 1 + qga/Makefile.objs | 1 + scripts/qapi/commands.py | 17 +++-- scripts/qapi/events.py | 2 +- scripts/qapi/gen.py | 28 ++++---- scripts/qapi/schema.py | 92 +++++++++++++++--------- scripts/qapi/types.py | 5 +- scripts/qapi/visit.py | 8 +-- tests/.gitignore | 1 + tests/Makefile.include | 9 ++- tests/qapi-schema/empty.out | 1 + tests/qapi-schema/include-repetition.out | 6 +- tests/qapi-schema/qapi-schema-test.out | 24 +++---- 19 files changed, 144 insertions(+), 85 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/6 qapi: Tweak "command returns a nice type" check for clarity
--- scripts/qapi/schema.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
scripts/qapi/schema.py +5 -4
@@ -711,10 +711,11 @@ class QAPISchemaCommand(QAPISchemaEntity): self.ret_type = schema.resolve_type( self._ret_type_name, self.info, "command's 'returns'") if self.name not in self.info.pragma.returns_whitelist: - if not (isinstance(self.ret_type, QAPISchemaObjectType) - or (isinstance(self.ret_type, QAPISchemaArrayType) - and isinstance(self.ret_type.element_type, - QAPISchemaObjectType))): + typ = self.ret_type + if isinstance(typ, QAPISchemaArrayType): + typ = self.ret_type.element_type + assert typ + if not isinstance(typ, QAPISchemaObjectType): raise QAPISemError( self.info, "command's 'returns' cannot take %s"
Signature · 1 line
2.21.0
Re: [PATCH 1/6] qapi: Tweak "command returns a nice type" check for clarity
Quoted 4 lines · Markus Armbruster
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi/schema.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
Quoted 22 lines
scripts/qapi/schema.py +5
-4@@ -711,10 +711,11 @@ class QAPISchemaCommand(QAPISchemaEntity): self.ret_type = schema.resolve_type( self._ret_type_name, self.info, "command's 'returns'") if self.name not in self.info.pragma.returns_whitelist: - if not (isinstance(self.ret_type, QAPISchemaObjectType) - or (isinstance(self.ret_type, QAPISchemaArrayType) - and isinstance(self.ret_type.element_type, - QAPISchemaObjectType))): + typ = self.ret_type + if isinstance(typ, QAPISchemaArrayType): + typ = self.ret_type.element_type + assert typ + if not isinstance(typ, QAPISchemaObjectType): raise QAPISemError( self.info, "command's 'returns' cannot take %s"
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH 2/6 tests/Makefile.include: Fix missing test-qapi-emit-events.[ch]
In reply to Markus ArmbrusterCommit 5d75648b56 "qapi: Generate QAPIEvent stuff into separate files" added tests/test-qapi-emit-events.[ch] to the set of generated files, but neglected to update tests/.gitignore and tests/Makefile.include. Commit a0af8cee3c "tests/.gitignore: ignore test-qapi-emit-events.[ch] for in-tree builds" fixed the former. Now fix the latter.
--- tests/Makefile.include | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tests/Makefile.include +3 -1
@@ -503,6 +503,7 @@ generated-files-y += tests/test-qapi-visit-sub-sub-module.h generated-files-y += tests/test-qapi-commands.h generated-files-y += tests/include/test-qapi-commands-sub-module.h generated-files-y += tests/test-qapi-commands-sub-sub-module.h +generated-files-y += tests/test-qapi-emit-events.h generated-files-y += tests/test-qapi-events.h generated-files-y += tests/include/test-qapi-events-sub-module.h generated-files-y += tests/test-qapi-events-sub-sub-module.h
@@ -610,6 +611,7 @@ tests/include/test-qapi-commands-sub-module.h \ tests/include/test-qapi-commands-sub-module.c \ tests/test-qapi-commands-sub-sub-module.h \ tests/test-qapi-commands-sub-sub-module.c \ +tests/test-qapi-emit-events.c tests/test-qapi-emit-events.h \ tests/test-qapi-events.c tests/test-qapi-events.h \ tests/include/test-qapi-events-sub-module.c \ tests/include/test-qapi-events-sub-module.h \
@@ -637,7 +639,7 @@ tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-good.jso tests/test-string-output-visitor$(EXESUF): tests/test-string-output-visitor.o $(test-qapi-obj-y) tests/test-string-input-visitor$(EXESUF): tests/test-string-input-visitor.o $(test-qapi-obj-y) -tests/test-qmp-event$(EXESUF): tests/test-qmp-event.o $(test-qapi-obj-y) tests/test-qapi-events.o +tests/test-qmp-event$(EXESUF): tests/test-qmp-event.o $(test-qapi-obj-y) tests/test-qapi-emit-events.o tests/test-qapi-events.o tests/test-qobject-output-visitor$(EXESUF): tests/test-qobject-output-visitor.o $(test-qapi-obj-y) tests/test-clone-visitor$(EXESUF): tests/test-clone-visitor.o $(test-qapi-obj-y) tests/test-qobject-input-visitor$(EXESUF): tests/test-qobject-input-visitor.o $(test-qapi-obj-y)
Signature · 1 line
2.21.0
Re: [PATCH 2/6] tests/Makefile.include: Fix missing test-qapi-emit-events.[ch]
Quoted 11 lines · Markus Armbruster
Commit
5d75648b56"qapi: Generate QAPIEvent stuff into separate files" added tests/test-qapi-emit-events.[ch] to the set of generated files, but neglected to update tests/.gitignore and tests/Makefile.include. Commita0af8cee3c"tests/.gitignore: ignore test-qapi-emit-events.[ch] for in-tree builds" fixed the former. Now fix the latter.Signed-off-by Markus Armbruster <[email protected]>--- tests/Makefile.include | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH 3/6 qapi: Generate command registration stuff into separate files
In reply to Markus ArmbrusterHaving to include qapi-commands.h just for qmp_init_marshal() is suboptimal. Generate it into separate files. This lets monitor/misc.c, qga/main.c, and the generated qapi-commands-FOO.h include less.
--- docs/devel/qapi-code-gen.txt | 19 ++++++++++++++++--- Makefile | 4 +++- monitor/misc.c | 7 ++++++- qga/main.c | 2 +- tests/test-qmp-cmds.c | 1 + .gitignore | 1 + qapi/Makefile.objs | 1 + qga/Makefile.objs | 1 + scripts/qapi/commands.py | 15 +++++++++++---- tests/.gitignore | 1 + tests/Makefile.include | 5 ++++- 11 files changed, 46 insertions(+), 11 deletions(-)
docs/devel/qapi-code-gen.txt +16 -3
@@ -1493,6 +1493,10 @@ $(prefix)qapi-commands.c: Command marshal/dispatch functions for each $(prefix)qapi-commands.h: Function prototypes for the QMP commands specified in the schema +$(prefix)qapi-init-commands.h - Command initialization prototype + +$(prefix)qapi-init-commands.h - Command initialization code + Example: $ cat qapi-generated/example-qapi-commands.h
@@ -1502,11 +1506,9 @@ Example: #define EXAMPLE_QAPI_COMMANDS_H #include "example-qapi-types.h" - #include "qapi/qmp/dispatch.h" UserDefOne *qmp_my_command(UserDefOneList *arg1, Error **errp); void qmp_marshal_my_command(QDict *args, QObject **ret, Error **errp); - void example_qmp_init_marshal(QmpCommandList *cmds); #endif /* EXAMPLE_QAPI_COMMANDS_H */ $ cat qapi-generated/example-qapi-commands.c
@@ -1566,7 +1568,19 @@ Example: visit_end_struct(v, NULL); visit_free(v); } +[Uninteresting stuff omitted...] + $ cat qapi-generated/example-qapi-init-commands.h +[Uninteresting stuff omitted...] + #ifndef EXAMPLE_QAPI_INIT_COMMANDS_H + #define EXAMPLE_QAPI_INIT_COMMANDS_H + #include "qapi/qmp/dispatch.h" + + void example_qmp_init_marshal(QmpCommandList *cmds); + + #endif /* EXAMPLE_QAPI_INIT_COMMANDS_H */ + $ cat qapi-generated/example-qapi-init-commands. +[Uninteresting stuff omitted...] void example_qmp_init_marshal(QmpCommandList *cmds) { QTAILQ_INIT(cmds);
@@ -1574,7 +1588,6 @@ Example: qmp_register_command(cmds, "my-command", qmp_marshal_my_command, QCO_NO_OPTIONS); } - [Uninteresting stuff omitted...] For a modular QAPI schema (see section Include directives), code for
Makefile +3 -1
@@ -117,6 +117,7 @@ GENERATED_QAPI_FILES += qapi/qapi-builtin-visit.h qapi/qapi-builtin-visit.c GENERATED_QAPI_FILES += qapi/qapi-visit.h qapi/qapi-visit.c GENERATED_QAPI_FILES += $(QAPI_MODULES:%=qapi/qapi-visit-%.h) GENERATED_QAPI_FILES += $(QAPI_MODULES:%=qapi/qapi-visit-%.c) +GENERATED_QAPI_FILES += qapi/qapi-init-commands.h qapi/qapi-init-commands.c GENERATED_QAPI_FILES += qapi/qapi-commands.h qapi/qapi-commands.c GENERATED_QAPI_FILES += $(QAPI_MODULES:%=qapi/qapi-commands-%.h) GENERATED_QAPI_FILES += $(QAPI_MODULES:%=qapi/qapi-commands-%.c)
@@ -610,6 +611,7 @@ $(SRC_PATH)/scripts/qapi-gen.py qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \ qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \ qga/qapi-generated/qga-qapi-commands.h qga/qapi-generated/qga-qapi-commands.c \ +qga/qapi-generated/qga-qapi-init-commands.h qga/qapi-generated/qga-qapi-init-commands.c \ qga/qapi-generated/qga-qapi-doc.texi: \ qga/qapi-generated/qapi-gen-timestamp ; qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py)
@@ -628,7 +630,7 @@ qapi-gen-timestamp: $(qapi-modules) $(qapi-py) "GEN","$(@:%-timestamp=%)") @>$@ -QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qapi-commands.h) +QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qapi-commands.h qga-qapi-init-commands.h) $(qga-obj-y): $(QGALIB_GEN) qemu-ga$(EXESUF): $(qga-obj-y) $(COMMON_LDADDS)
monitor/misc.c +6 -1
@@ -66,8 +66,13 @@#include "qemu/option.h" #include "qemu/thread.h" #include "block/qapi.h" -#include "qapi/qapi-commands.h" +#include "qapi/qapi-commands-char.h" +#include "qapi/qapi-commands-migration.h" +#include "qapi/qapi-commands-misc.h" +#include "qapi/qapi-commands-qom.h" +#include "qapi/qapi-commands-trace.h" #include "qapi/qapi-emit-events.h" +#include "qapi/qapi-init-commands.h" #include "qapi/error.h" #include "qapi/qmp-event.h" #include "qapi/qapi-introspect.h"
qga/main.c +1 -1
@@ -24,7 +24,7 @@#include "qapi/qmp/qjson.h" #include "qapi/qmp/qstring.h" #include "guest-agent-core.h" -#include "qga-qapi-commands.h" +#include "qga-qapi-init-commands.h" #include "qapi/qmp/qerror.h" #include "qapi/error.h" #include "channel.h"
tests/test-qmp-cmds.c +1 -0
@@ -7,6 +7,7 @@#include "tests/test-qapi-types.h" #include "tests/test-qapi-visit.h" #include "test-qapi-commands.h" +#include "test-qapi-init-commands.h" static QmpCommandList qmp_commands;
.gitignore +1 -0
@@ -37,6 +37,7 @@ /qapi/qapi-emit-events.[ch] /qapi/qapi-events-*.[ch] /qapi/qapi-events.[ch] +/qapi/qapi-init-commands.[ch] /qapi/qapi-introspect.[ch] /qapi/qapi-types-*.[ch] /qapi/qapi-types.[ch]
qapi/Makefile.objs +1 -0
@@ -30,3 +30,4 @@ obj-y += $(QAPI_TARGET_MODULES:%=qapi-events-%.o) obj-y += qapi-events.o obj-y += $(QAPI_TARGET_MODULES:%=qapi-commands-%.o) obj-y += qapi-commands.o +obj-y += qapi-init-commands.o
qga/Makefile.objs +1 -0
@@ -5,5 +5,6 @@ qga-obj-$(CONFIG_WIN32) += commands-win32.o channel-win32.o service-win32.o qga-obj-$(CONFIG_WIN32) += vss-win32.o qga-obj-y += qapi-generated/qga-qapi-types.o qapi-generated/qga-qapi-visit.o qga-obj-y += qapi-generated/qga-qapi-commands.o +qga-obj-y += qapi-generated/qga-qapi-init-commands.o qga-vss-dll-obj-$(CONFIG_QGA_VSS) += vss-win32/
scripts/qapi/commands.py +11 -4
@@ -263,18 +263,25 @@ class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor): commands=commands, visit=visit)) self._genh.add(mcgen(''' #include "%(types)s.h" -#include "qapi/qmp/dispatch.h" ''', types=types)) def visit_end(self): - (genc, genh) = self._module[self._main_module] - genh.add(mcgen(''' + self._add_system_module('init', ' * QAPI Commands initialization') + self._genh.add(mcgen(''' +#include "qapi/qmp/dispatch.h" + void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds); ''', c_prefix=c_name(self._prefix, protect=False))) - genc.add(gen_registry(self._regy.get_content(), self._prefix)) + self._genc.preamble_add(mcgen(''' +#include "qemu/osdep.h" +#include "%(prefix)sqapi-commands.h" +#include "%(prefix)sqapi-init-commands.h" +''', + prefix=self._prefix)) + self._genc.add(gen_registry(self._regy.get_content(), self._prefix)) def visit_command(self, name, info, ifcond, arg_type, ret_type, gen, success_response, boxed, allow_oob, allow_preconfig,
tests/.gitignore +1 -0
@@ -12,6 +12,7 @@ test-* !test-*.c !docker/test-* test-qapi-commands.[ch] +test-qapi-init-commands.[ch] include/test-qapi-commands-sub-module.[ch] test-qapi-commands-sub-sub-module.[ch] test-qapi-emit-events.[ch]
tests/Makefile.include +4 -1
@@ -501,6 +501,7 @@ generated-files-y += tests/test-qapi-visit.h generated-files-y += tests/include/test-qapi-visit-sub-module.h generated-files-y += tests/test-qapi-visit-sub-sub-module.h generated-files-y += tests/test-qapi-commands.h +generated-files-y += tests/test-qapi-init-commands.h generated-files-y += tests/include/test-qapi-commands-sub-module.h generated-files-y += tests/test-qapi-commands-sub-sub-module.h generated-files-y += tests/test-qapi-emit-events.h
@@ -613,6 +614,8 @@ tests/test-qapi-commands-sub-sub-module.h \ tests/test-qapi-commands-sub-sub-module.c \ tests/test-qapi-emit-events.c tests/test-qapi-emit-events.h \ tests/test-qapi-events.c tests/test-qapi-events.h \ +tests/test-qapi-init-commands.c \ +tests/test-qapi-init-commands.h \ tests/include/test-qapi-events-sub-module.c \ tests/include/test-qapi-events-sub-module.h \ tests/test-qapi-events-sub-sub-module.c \
@@ -643,7 +646,7 @@ tests/test-qmp-event$(EXESUF): tests/test-qmp-event.o $(test-qapi-obj-y) tests/t tests/test-qobject-output-visitor$(EXESUF): tests/test-qobject-output-visitor.o $(test-qapi-obj-y) tests/test-clone-visitor$(EXESUF): tests/test-clone-visitor.o $(test-qapi-obj-y) tests/test-qobject-input-visitor$(EXESUF): tests/test-qobject-input-visitor.o $(test-qapi-obj-y) -tests/test-qmp-cmds$(EXESUF): tests/test-qmp-cmds.o tests/test-qapi-commands.o $(test-qapi-obj-y) +tests/test-qmp-cmds$(EXESUF): tests/test-qmp-cmds.o tests/test-qapi-commands.o tests/test-qapi-init-commands.o $(test-qapi-obj-y) tests/test-visitor-serialization$(EXESUF): tests/test-visitor-serialization.o $(test-qapi-obj-y) tests/test-opts-visitor$(EXESUF): tests/test-opts-visitor.o $(test-qapi-obj-y)
Signature · 1 line
2.21.0
Re: [PATCH 3/6] qapi: Generate command registration stuff into separate files
Quoted 10 lines · Markus Armbruster
Having to include
qapi-commands.hjust forqmp_init_marshal()is suboptimal. Generate it into separate files. This letsmonitor/misc.c,qga/main.c, and the generatedqapi-commands-FOO.hinclude less.Signed-off-by Markus Armbruster <[email protected]>---
+++
b/docs/devel/qapi-code-gen.txt
docs/devel/qapi-code-gen.txt $(prefix)qapi-commands.c: Command marshal/dispatch functions for each · line 1498
Quoted 6 lines
docs/devel/qapi-code-gen.txt +3
-0@@ -1493,6 +1493,10 @@ $(prefix)qapi-commands.c: Command marshal/dispatch functions for each $(prefix)qapi-commands.h: Function prototypes for the QMP commands specified in the schema +$(prefix)qapi-init-commands.h - Command initialization prototype + +$(prefix)qapi-init-commands.h - Command initialization code
Looks like you meant s/h/c/
docs/devel/qapi-code-gen.txt Example: · line 1583
Quoted 3 lines
docs/devel/qapi-code-gen.txt +3
-0+ #endif /* EXAMPLE_QAPI_INIT_COMMANDS_H */ + $ cat qapi-generated/example-qapi-init-commands. +[Uninteresting stuff omitted...]
missing a 'c'
Makefile qapi-gen-timestamp: $(qapi-modules) $(qapi-py) · line 633
Quoted 5 lines
Makefile +2
-1+++ b/Makefile -QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qapi-commands.h) +QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qapi-commands.h qga-qapi-init-commands.h)
Worth using \ for line-wrapping?
With those addressed,
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Markus Armbruster
Tested-by Stefan BergerRe: [PATCH 3/6] qapi: Generate command registration stuff into separate files
Quoted 9 lines · Eric Blake
Quoted 7 lines · Markus Armbruster
Having to include
qapi-commands.hjust forqmp_init_marshal()is suboptimal. Generate it into separate files. This letsmonitor/misc.c,qga/main.c, and the generatedqapi-commands-FOO.hinclude less.Signed-off-by Markus Armbruster <[email protected]>---
Quoted 9 lines
docs/devel/qapi-code-gen.txt $(prefix)qapi-commands.c: Command marshal/dispatch functions for each · line 1497
Quoted 7 lines
+++
b/docs/devel/qapi-code-gen.txtdocs/devel/qapi-code-gen.txt +2
-0@@ -1493,6 +1493,10 @@ $(prefix)qapi-commands.c: Command marshal/dispatch functions for each $(prefix)qapi-commands.h: Function prototypes for the QMP commands specified in the schema +$(prefix)qapi-init-commands.h - Command initialization prototype + +$(prefix)qapi-init-commands.h - Command initialization codeLooks like you meant s/h/c/
I did. Thanks!
Quoted 5 lines
docs/devel/qapi-code-gen.txt Example: · line 1583
Quoted 3 lines
docs/devel/qapi-code-gen.txt +3
-0+ #endif /* EXAMPLE_QAPI_INIT_COMMANDS_H */ + $ cat qapi-generated/example-qapi-init-commands. +[Uninteresting stuff omitted...]missing a 'c'
Yes.
Quoted 7 lines
Quoted 5 lines
(quoted diff) +1
-0+++ b/Makefile -QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h
qga-qapi-visit.hqga-qapi-commands.h)
+QGALIB_GEN=$(addprefix qga/qapi-generated/,qga-qapi-types.hqga-qapi-visit.hqga-qapi-commands.hqga-qapi-init-commands.h)Worth using \ for line-wrapping?
Hmm, it's used only in the following line, could just as well inline there and line-wrap that:
$(qga-obj-y): $(QGALIB_GEN)
Hmm^2, why do we even have that line? The compiler-generated .d should take care of dependencies. Dig, dig, ...
commit 016c77ad62a8ad607dd4349d8cb8ad1365bab831
Author: Michael Roth <[email protected]>
Date: Tue Jul 26 11:39:24 2011 -0500
Makefile: add missing deps on $(GENERATED_HEADERS)
This fixes a build issue with make -j6+ due to qapi-generated files being built before $(GENERATED_HEADERS) have been created.
Yes, but the same issue exists for all the other generated headers, and we solve it differently: put them into $(generated-files-y), have Makefile depend on them.
I'll clean this up on top. No need to beautify lines now that will go away then.
Quoted 2 lines
With those addressed,
Reviewed-by Eric Blake <[email protected]>
Thanks!
PATCH 4/6 qapi: Proper intermediate representation for modules
In reply to Markus ArmbrusterModules are represented only by their names so far. Introduce class QAPISchemaModule. So far, it merely wraps the name. The next patch will put it to more interesting use.
Once again, arrays spice up the patch a bit. For any other type, @info points to the definition, which lets us map from @info to module. For arrays, there is no definition, and @info points to the first use instead. We have to use the element type's module instead, which is only available after .check().
--- scripts/qapi/schema.py | 63 ++++++++++++++++++++++++++++-------------- 1 file changed, 43 insertions(+), 20 deletions(-)
scripts/qapi/schema.py +43 -20
@@ -50,9 +50,6 @@ class QAPISchemaEntity(object): def check(self, schema): assert not self._checked - if self.info: - self._module = os.path.relpath(self.info.fname, - os.path.dirname(schema.fname)) seen = {} for f in self.features: f.check_clash(self.info, seen)
@@ -68,6 +65,13 @@ class QAPISchemaEntity(object): if self.doc: self.doc.check() + def _set_module(self, schema, info): + assert self._checked + self._module = schema.module_by_fname(info and info.fname) + + def set_module(self, schema): + self._set_module(schema, self.info) + @property def ifcond(self): assert self._checked
@@ -75,7 +79,7 @@ class QAPISchemaEntity(object): @property def module(self): - assert self._checked + assert self._module or not self.info return self._module def is_implicit(self):
@@ -135,15 +139,19 @@ class QAPISchemaVisitor(object): pass +class QAPISchemaModule(object): + def __init__(self, name): + self.name = name + + class QAPISchemaInclude(QAPISchemaEntity): - - def __init__(self, fname, info): + def __init__(self, sub_module, info): QAPISchemaEntity.__init__(self, None, info, None) - self.fname = fname + self._sub_module = sub_module def visit(self, visitor): QAPISchemaEntity.visit(self, visitor) - visitor.visit_include(self.fname, self.info) + visitor.visit_include(self._sub_module.name, self.info) class QAPISchemaType(QAPISchemaEntity):
@@ -276,16 +284,14 @@ class QAPISchemaArrayType(QAPISchemaType): self.info and self.info.defn_meta) assert not isinstance(self.element_type, QAPISchemaArrayType) + def set_module(self, schema): + self._set_module(schema, self.element_type.info) + @property def ifcond(self): assert self._checked return self.element_type.ifcond - @property - def module(self): - assert self._checked - return self.element_type.module - def is_implicit(self): return True
@@ -783,6 +789,10 @@ class QAPISchema(object): self.docs = parser.docs self._entity_list = [] self._entity_dict = {} + self._module_dict = {} + self._schema_dir = os.path.dirname(fname) + self._make_module(None) # built-ins + self._make_module(fname) self._predefining = True self._def_predefineds() self._predefining = False
@@ -826,14 +836,26 @@ class QAPISchema(object): info, "%s uses unknown type '%s'" % (what, name)) return typ + def _module_name(self, fname): + if fname is None: + return None + return os.path.relpath(fname, self._schema_dir) + + def _make_module(self, fname): + name = self._module_name(fname) + if not name in self._module_dict: + self._module_dict[name] = QAPISchemaModule(name) + return self._module_dict[name] + + def module_by_fname(self, fname): + name = self._module_name(fname) + assert name in self._module_dict + return self._module_dict[name] + def _def_include(self, expr, info, doc): include = expr['include'] assert doc is None - main_info = info - while main_info.parent: - main_info = main_info.parent - fname = os.path.relpath(include, os.path.dirname(main_info.fname)) - self._def_entity(QAPISchemaInclude(fname, info)) + self._def_entity(QAPISchemaInclude(self._make_module(include), info)) def _def_builtin_type(self, name, json_type, c_type): self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
@@ -1065,15 +1087,16 @@ class QAPISchema(object): ent.check(self) ent.connect_doc() ent.check_doc() + for ent in self._entity_list: + ent.set_module(self) def visit(self, visitor): visitor.visit_begin(self) module = None - visitor.visit_module(module) for entity in self._entity_list: if visitor.visit_needed(entity): if entity.module != module: module = entity.module - visitor.visit_module(module) + visitor.visit_module(module.name) entity.visit(visitor) visitor.visit_end()
Signature · 1 line
2.21.0
Re: [PATCH 4/6] qapi: Proper intermediate representation for modules
Quoted 10 lines · Markus Armbruster
Modules are represented only by their names so far. Introduce class QAPISchemaModule. So far, it merely wraps the name. The next patch will put it to more interesting use.
Once again, arrays spice up the patch a bit. For any other type, @info points to the definition, which lets us map from @info to module. For arrays, there is no definition, and @info points to the first use instead. We have to use the element type's module instead, which is only available after .check().
Quoted 5 lines
Signed-off-by Markus Armbruster <[email protected]>--- scripts/qapi/schema.py | 63 ++++++++++++++++++++++++++++-------------- 1 file changed, 43 insertions(+), 20 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH 5/6 qapi: Fix code generation for empty modules
In reply to Markus ArmbrusterWhen a sub-module doesn't contain any definitions, we don't generate code for it, but we do generate the #include.
We generate code only for modules that get visited. QAPISchema.visit() visits only modules that have definitions. It can visit modules multiple times.
Clean this up as follows. Collect entities in their QAPISchemaModule. Have QAPISchema.visit() call QAPISchemaModule.visit() for each module. Have QAPISchemaModule.visit() call .visit_module() for itself, and QAPISchemaEntity.visit() for each of its entities. This way, we visit each module exactly once.
--- scripts/qapi/schema.py | 24 +++++++++++++----------- tests/qapi-schema/empty.out | 1 + tests/qapi-schema/include-repetition.out | 6 ++---- tests/qapi-schema/qapi-schema-test.out | 24 ++++++++++-------------- 4 files changed, 26 insertions(+), 29 deletions(-)
scripts/qapi/schema.py +13 -11
@@ -68,6 +68,7 @@ class QAPISchemaEntity(object): def _set_module(self, schema, info): assert self._checked self._module = schema.module_by_fname(info and info.fname) + self._module.add_entity(self) def set_module(self, schema): self._set_module(schema, self.info)
@@ -77,11 +78,6 @@ class QAPISchemaEntity(object): assert self._checked return self._ifcond - @property - def module(self): - assert self._module or not self.info - return self._module - def is_implicit(self): return not self.info
@@ -142,6 +138,16 @@ class QAPISchemaVisitor(object): class QAPISchemaModule(object): def __init__(self, name): self.name = name + self._entity_list = [] + + def add_entity(self, ent): + self._entity_list.append(ent) + + def visit(self, visitor): + visitor.visit_module(self.name) + for entity in self._entity_list: + if visitor.visit_needed(entity): + entity.visit(visitor) class QAPISchemaInclude(QAPISchemaEntity):
@@ -1093,10 +1099,6 @@ class QAPISchema(object): def visit(self, visitor): visitor.visit_begin(self) module = None - for entity in self._entity_list: - if visitor.visit_needed(entity): - if entity.module != module: - module = entity.module - visitor.visit_module(module.name) - entity.visit(visitor) + for mod in self._module_dict.values(): + mod.visit(visitor) visitor.visit_end()
tests/qapi-schema/empty.out +1 -0
@@ -9,3 +9,4 @@ enum QType member qdict member qlist member qbool +module empty.json
tests/qapi-schema/include-repetition.out +2 -4
@@ -11,15 +11,13 @@ enum QType member qbool module include-repetition.json include comments.json +include include-repetition-sub.json +include comments.json module comments.json enum Status member good member bad member ugly -module include-repetition.json -include include-repetition-sub.json module include-repetition-sub.json include comments.json include comments.json -module include-repetition.json -include comments.json
tests/qapi-schema/qapi-schema-test.out +10 -14
@@ -153,9 +153,6 @@ object q_obj_sizeList-wrapper member data: sizeList optional=False object q_obj_anyList-wrapper member data: anyList optional=False -module sub-sub-module.json -array StatusList Status -module qapi-schema-test.json object q_obj_StatusList-wrapper member data: StatusList optional=False enum UserDefListUnionKind
@@ -193,17 +190,6 @@ object UserDefListUnion case any: q_obj_anyList-wrapper case user: q_obj_StatusList-wrapper include include/sub-module.json -module include/sub-module.json -include sub-sub-module.json -module sub-sub-module.json -enum Status - member good - member bad - member ugly -module include/sub-module.json -object SecondArrayRef - member s: StatusList optional=False -module qapi-schema-test.json command user_def_cmd None -> None gen=True success_response=True boxed=False oob=False preconfig=False object q_obj_user_def_cmd1-arg
@@ -435,3 +421,13 @@ command test-command-cond-features3 None -> None gen=True success_response=True boxed=False oob=False preconfig=False feature feature1 if ['defined(TEST_IF_COND_1)', 'defined(TEST_IF_COND_2)'] +module include/sub-module.json +include sub-sub-module.json +object SecondArrayRef + member s: StatusList optional=False +module sub-sub-module.json +array StatusList Status +enum Status + member good + member bad + member ugly
Signature · 1 line
2.21.0
Re: [PATCH 5/6] qapi: Fix code generation for empty modules
Quoted 7 lines · Markus Armbruster
When a sub-module doesn't contain any definitions, we don't generate code for it, but we do generate the #include.
We generate code only for modules that get visited.
QAPISchema.visit()visits only modules that have definitions. It can visit modules multiple times.
Quoted 8 lines
Clean this up as follows. Collect entities in their QAPISchemaModule. Have
QAPISchema.visit()callQAPISchemaModule.visit()for each module. HaveQAPISchemaModule.visit()call .visit_module() for itself, andQAPISchemaEntity.visit()for each of its entities. This way, we visit each module exactly once.Signed-off-by Markus Armbruster <[email protected]>---
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH 6/6 qapi: Simplify QAPISchemaModularCVisitor
In reply to Markus ArmbrusterSince the previous commit, QAPISchemaVisitor.visit_module() is called just once. Simplify QAPISchemaModularCVisitor accordingly.
--- scripts/qapi/commands.py | 2 +- scripts/qapi/events.py | 2 +- scripts/qapi/gen.py | 28 ++++++++++++++-------------- scripts/qapi/types.py | 5 +++-- scripts/qapi/visit.py | 8 ++++---- 5 files changed, 23 insertions(+), 22 deletions(-)
scripts/qapi/commands.py +1 -1
@@ -239,7 +239,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor): def __init__(self, prefix): QAPISchemaModularCVisitor.__init__( self, prefix, 'qapi-commands', - ' * Schema-defined QAPI/QMP commands', __doc__) + ' * Schema-defined QAPI/QMP commands', None, __doc__) self._regy = QAPIGenCCode(None) self._visited_ret_types = {}
scripts/qapi/events.py +1 -1
@@ -140,7 +140,7 @@ class QAPISchemaGenEventVisitor(QAPISchemaModularCVisitor): def __init__(self, prefix): QAPISchemaModularCVisitor.__init__( self, prefix, 'qapi-events', - ' * Schema-defined QAPI/QMP events', __doc__) + ' * Schema-defined QAPI/QMP events', None, __doc__) self._event_enum_name = c_name(prefix + 'QAPIEvent', protect=False) self._event_enum_members = [] self._event_emit_name = c_name(prefix + 'qapi_event_emit')
scripts/qapi/gen.py +14 -14
@@ -201,10 +201,11 @@ class QAPISchemaMonolithicCVisitor(QAPISchemaVisitor): class QAPISchemaModularCVisitor(QAPISchemaVisitor): - def __init__(self, prefix, what, blurb, pydoc): + def __init__(self, prefix, what, user_blurb, builtin_blurb, pydoc): self._prefix = prefix self._what = what - self._blurb = blurb + self._user_blurb = user_blurb + self._builtin_blurb = builtin_blurb self._pydoc = pydoc self._genc = None self._genh = None
@@ -245,7 +246,7 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor): genc = QAPIGenC(basename + '.c', blurb, self._pydoc) genh = QAPIGenH(basename + '.h', blurb, self._pydoc) self._module[name] = (genc, genh) - self._set_module(name) + self._genc, self._genh = self._module[name] def _add_user_module(self, name, blurb): assert self._is_user_module(name)
@@ -256,9 +257,6 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor): def _add_system_module(self, name, blurb): self._add_module(name and './' + name, blurb) - def _set_module(self, name): - self._genc, self._genh = self._module[name] - def write(self, output_dir, opt_builtins=False): for name in self._module: if self._is_builtin_module(name) and not opt_builtins:
@@ -271,15 +269,17 @@ class QAPISchemaModularCVisitor(QAPISchemaVisitor): pass def visit_module(self, name): - if name in self._module: - self._set_module(name) - elif self._is_builtin_module(name): - # The built-in module has not been created. No code may - # be generated. - self._genc = None - self._genh = None + if name is None: + if self._builtin_blurb: + self._add_system_module(None, self._builtin_blurb) + self._begin_system_module(name) + else: + # The built-in module has not been created. No code may + # be generated. + self._genc = None + self._genh = None else: - self._add_user_module(name, self._blurb) + self._add_user_module(name, self._user_blurb) self._begin_user_module(name) def visit_include(self, name, info):
scripts/qapi/types.py +3 -2
@@ -243,8 +243,9 @@ class QAPISchemaGenTypeVisitor(QAPISchemaModularCVisitor): def __init__(self, prefix): QAPISchemaModularCVisitor.__init__( self, prefix, 'qapi-types', ' * Schema-defined QAPI types', - __doc__) - self._add_system_module(None, ' * Built-in QAPI types') + ' * Built-in QAPI types', __doc__) + + def _begin_system_module(self, name): self._genc.preamble_add(mcgen(''' #include "qemu/osdep.h" #include "qapi/dealloc-visitor.h"
scripts/qapi/visit.py +4 -4
@@ -285,8 +285,9 @@ class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor): def __init__(self, prefix): QAPISchemaModularCVisitor.__init__( self, prefix, 'qapi-visit', ' * Schema-defined QAPI visitors', - __doc__) - self._add_system_module(None, ' * Built-in QAPI visitors') + ' * Built-in QAPI visitors', __doc__) + + def _begin_system_module(self, name): self._genc.preamble_add(mcgen(''' #include "qemu/osdep.h" #include "qapi/error.h"
@@ -296,8 +297,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaModularCVisitor): #include "qapi/visitor.h" #include "qapi/qapi-builtin-types.h" -''', - prefix=prefix)) +''')) def _begin_user_module(self, name): types = self._module_basename('qapi-types', name)
Signature · 1 line
2.21.0
Re: [PATCH 6/6] qapi: Simplify QAPISchemaModularCVisitor
Quoted 5 lines · Markus Armbruster
Since the previous commit,
QAPISchemaVisitor.visit_module()is called just once. Simplify QAPISchemaModularCVisitor accordingly.Signed-off-by Markus Armbruster <[email protected]>---
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Fat-fingered Kevin's e-mail address...
Quoted 32 lines · Markus Armbruster
Kevin recently posted a minimally invasive fix for empty QAPI modules[*]. This is my attempt at a fix that also addresses the design weakness that led to the bug.
Markus Armbruster (6):
qapi: Tweak "command returns a nice type" check for clarity tests/Makefile.include: Fix missing test-qapi-emit-events.[ch]
qapi: Generate command registration stuff into separate files
qapi: Proper intermediate representation for modules
qapi: Fix code generation for empty modules
qapi: Simplify QAPISchemaModularCVisitordocs/devel/qapi-code-gen.txt | 19 ++++- Makefile | 4 +- monitor/misc.c | 7 +- qga/main.c | 2 +- tests/test-qmp-cmds.c | 1 + .gitignore | 1 + qapi/Makefile.objs | 1 + qga/Makefile.objs | 1 + scripts/qapi/commands.py | 17 +++-- scripts/qapi/events.py | 2 +- scripts/qapi/gen.py | 28 ++++---- scripts/qapi/schema.py | 92 +++++++++++++++--------- scripts/qapi/types.py | 5 +- scripts/qapi/visit.py | 8 +-- tests/.gitignore | 1 + tests/Makefile.include | 9 ++- tests/qapi-schema/empty.out | 1 + tests/qapi-schema/include-repetition.out | 6 +- tests/qapi-schema/qapi-schema-test.out | 24 +++---- 19 files changed, 144 insertions(+), 85 deletions(-)
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===
TEST iotest-qcow2: 268
Failures: 192
Failed 1 of 108 iotests
make: *** [check-tests/check-block.sh] Error 1 Traceback (most recent call last):
File "./tests/docker/docker.py", line 662, in <module>
sys.exit(main())
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=7206e3a304c44c7187ff2b4aa7fff8d6', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-wk8w7tot/src/docker-src.2019-11-20-18.46.04.18266:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=7206e3a304c44c7187ff2b4aa7fff8d6
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-wk8w7tot/src'
make: *** [docker-run-test-quick@centos7] Error 2
real 12m12.040s
user 0m8.724s
The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
Quoted 3 lines · Markus Armbruster
Kevin recently posted a minimally invasive fix for empty QAPI modules[*]. This is my attempt at a fix that also addresses the design weakness that led to the bug.
[*] Subject: [RFC PATCH 15/18] qapi: Support empty modules
Message-Id: <[email protected]>
[PATCH 7/6] Makefile: Make Makefile depend on generated qga files, too
In reply to Markus ArmbrusterGenerated .h need to be generated before compiling any .c using them. To know which .h a .c uses, we need to compile it.
Since commit 4115852bb0 "build: do not sprinkle around GENERATED_HEADERS dependencies", we break this circular dependency the simple & stupid way: the generated headers are a prerequisite of Makefile, which causes Make to generate them first, then start over.
Except for qga we still use the older method of making all its .o summarily depend on all its generated .h (commit 016c77ad62 "Makefile:
add missing deps on $(GENERATED_HEADERS)").
Add qga's generated files to generated-files-y to get rid of this exception.
--- Makefile | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)
Makefile +11 -10
@@ -130,6 +130,15 @@ GENERATED_QAPI_FILES += qapi/qapi-doc.texi generated-files-y += $(GENERATED_QAPI_FILES) +GENERATED_QGA_FILES := qga-qapi-types.c qga-qapi-types.h +GENERATED_QGA_FILES += qga-qapi-visit.c qga-qapi-visit.h +GENERATED_QGA_FILES += qga-qapi-commands.h qga-qapi-commands.c +GENERATED_QGA_FILES += qga-qapi-init-commands.h qga-qapi-init-commands.c +GENERATED_QGA_FILES += qga-qapi-doc.texi +GENERATED_QGA_FILES := $(addprefix qga/qapi-generated/, $(GENERATED_QGA_FILES)) + +generated-files-y += $(GENERATED_QGA_FILES) + generated-files-y += trace/generated-tcg-tracers.h generated-files-y += trace/generated-helpers-wrappers.h
@@ -608,12 +617,7 @@ $(SRC_PATH)/scripts/qapi/types.py \ $(SRC_PATH)/scripts/qapi/visit.py \ $(SRC_PATH)/scripts/qapi-gen.py -qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \ -qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \ -qga/qapi-generated/qga-qapi-commands.h qga/qapi-generated/qga-qapi-commands.c \ -qga/qapi-generated/qga-qapi-init-commands.h qga/qapi-generated/qga-qapi-init-commands.c \ -qga/qapi-generated/qga-qapi-doc.texi: \ -qga/qapi-generated/qapi-gen-timestamp ; +$(GENERATED_QGA_FILES): qga/qapi-generated/qapi-gen-timestamp ; qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(qapi-py) $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o qga/qapi-generated -p "qga-" $<, \
@@ -630,9 +634,6 @@ qapi-gen-timestamp: $(qapi-modules) $(qapi-py) "GEN","$(@:%-timestamp=%)") @>$@ -QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qapi-commands.h qga-qapi-init-commands.h) -$(qga-obj-y): $(QGALIB_GEN) - qemu-ga$(EXESUF): $(qga-obj-y) $(COMMON_LDADDS) $(call LINK, $^)
@@ -722,7 +723,7 @@ clean: recurse-clean rm -f trace/generated-tracers-dtrace.h* rm -f $(foreach f,$(generated-files-y),$(f) $(f)-timestamp) rm -f qapi-gen-timestamp - rm -rf qga/qapi-generated + rm -f qga/qapi-generated/qapi-gen-timestamp rm -f config-all-devices.mak VERSION ?= $(shell cat VERSION)
Signature · 1 line
2.21.0
Eric Blake
Reviewed-byRe: [PATCH 7/6] Makefile: Make Makefile depend on generated qga files, too
Quoted 7 lines · Markus Armbruster
Generated .h need to be generated before compiling any .c using them. To know which .h a .c uses, we need to compile it.
Since commit
4115852bb0"build: do not sprinkle aroundGENERATED_HEADERSdependencies", we break this circular dependency the simple & stupid way: the generated headers are a prerequisite ofMakefile, which causes Make to generate them first, then start over.
which is a pain when using 'make --dry-run' to see what would get built (a dependency of Makefile _is_ rebuilt if Makefile itself has to be updated), but not the fault of this patch.
Quoted 5 lines
Except for qga we still use the older method of making all its .o summarily depend on all its generated .h (commit
016c77ad62"Makefile:
add missing deps on $(GENERATED_HEADERS)").
Quoted 7 lines
Add qga's generated files to generated-files-y to get rid of this exception.
Signed-off-by Markus Armbruster <[email protected]>--- Makefile | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)
Makefile GENERATED_QAPI_FILES += qapi/qapi-doc.texi · line 138
Quoted 11 lines
+++ b/Makefile
Makefile +6
-0@@ -130,6 +130,15 @@ GENERATED_QAPI_FILES += qapi/qapi-doc.texi generated-files-y += $(GENERATED_QAPI_FILES) +GENERATED_QGA_FILES := qga-qapi-types.c qga-qapi-types.h +GENERATED_QGA_FILES += qga-qapi-visit.c qga-qapi-visit.h +GENERATED_QGA_FILES += qga-qapi-commands.h qga-qapi-commands.c +GENERATED_QGA_FILES += qga-qapi-init-commands.h qga-qapi-init-commands.c +GENERATED_QGA_FILES += qga-qapi-doc.texi +GENERATED_QGA_FILES := $(addprefix qga/qapi-generated/, $(GENERATED_QGA_FILES))
Would it be worth using two separate variable names (maybe GENERATED_QGA_BASEFILES for the first list) rather than exploiting the arcane knowledge that consecutive use of := causes GNU make to rewrite an existing variable with new contents?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 7/6] Makefile: Make Makefile depend on generated qga files, too
Quoted 4 lines · Eric Blake
Quoted 3 lines · Markus Armbruster
Generated .h need to be generated before compiling any .c using them. To know which .h a .c uses, we need to compile it.
Quoted 8 lines
Quoted 4 lines
Since commit
4115852bb0"build: do not sprinkle aroundGENERATED_HEADERSdependencies", we break this circular dependency the simple & stupid way: the generated headers are a prerequisite ofMakefile, which causes Make to generate them first, then start over.which is a pain when using 'make --dry-run' to see what would get built (a dependency of
Makefile_is_ rebuilt ifMakefileitself has to be updated), but not the fault of this patch.
Yes.
Quoted 7 lines
Quoted 7 lines
Except for qga we still use the older method of making all its .o summarily depend on all its generated .h (commit
016c77ad62"Makefile:
add missing deps on $(GENERATED_HEADERS)").Add qga's generated files to generated-files-y to get rid of this exception.
Quoted 6 lines
Quoted 4 lines
Signed-off-by Markus Armbruster <[email protected]>--- Makefile | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)Reviewed-by Eric Blake <[email protected]>
Thanks!
Quoted 6 lines
Quoted 6 lines
+++ b/Makefile
Makefile +2
-0@@ -130,6 +130,15 @@ GENERATED_QAPI_FILES += qapi/qapi-doc.texi generated-files-y += $(GENERATED_QAPI_FILES) +GENERATED_QGA_FILES := qga-qapi-types.c qga-qapi-types.h +GENERATED_QGA_FILES += qga-qapi-visit.c qga-qapi-visit.h +GENERATED_QGA_FILES += qga-qapi-commands.h qga-qapi-commands.c
Quoted 8 lines
Makefile GENERATED_QAPI_FILES += qapi/qapi-doc.texi · line 138
Quoted 3 lines
Makefile +3
-0+GENERATED_QGA_FILES += qga-qapi-init-commands.h qga-qapi-init-commands.c +GENERATED_QGA_FILES += qga-qapi-doc.texi +GENERATED_QGA_FILES := $(addprefix qga/qapi-generated/, $(GENERATED_QGA_FILES))Would it be worth using two separate variable names (maybe
GENERATED_QGA_BASEFILESfor the first list) rather than exploiting the arcane knowledge that consecutive use of := causes GNU make to rewrite an existing variable with new contents?
Our rules.mak relies on this already. It's full of magic, which admittedly diminishes its suitability to serve as a good example.
Your worry might be rooted in old "=" burns. "=" makes the variable recursively expanded, and
GENERATED_QGA_FILES = $(addprefix qga/qapi-generated/, $(GENERATED_QGA_FILES))
would be an infinite loop. ":=" makes it simply expanded; there's not even a loop, let alone an infinite one. The GNU Make manual explains this clearly at
https://www.gnu.org/software/make/manual/html_node/Flavors.html
Aside: there's a reason one of the two flavors is called "simple". It could additionally be called "not as slow". One of my pet makefile
peeves: unthinking use of recursively expanded variables, complicating semantics and slowing down builds.
Back to this patch. I had started to write the thing in longhand, but got tired of repeating qga/qapi-generated/, so I factored that out. Would longhand be easier to understand?
Re: [PATCH 7/6] Makefile: Make Makefile depend on generated qga files, too
Quoted 15 lines · Markus Armbruster
Quoted 6 lines
Quoted 6 lines
+++ b/Makefile
Makefile +2
-0@@ -130,6 +130,15 @@ GENERATED_QAPI_FILES += qapi/qapi-doc.texi generated-files-y += $(GENERATED_QAPI_FILES) +GENERATED_QGA_FILES := qga-qapi-types.c qga-qapi-types.h +GENERATED_QGA_FILES += qga-qapi-visit.c qga-qapi-visit.h +GENERATED_QGA_FILES += qga-qapi-commands.h qga-qapi-commands.cQuoted 8 lines
Makefile GENERATED_QAPI_FILES += qapi/qapi-doc.texi · line 138
Quoted 3 lines
Makefile +3
-0+GENERATED_QGA_FILES += qga-qapi-init-commands.h qga-qapi-init-commands.c +GENERATED_QGA_FILES += qga-qapi-doc.texi +GENERATED_QGA_FILES := $(addprefix qga/qapi-generated/, $(GENERATED_QGA_FILES))Would it be worth using two separate variable names (maybe
GENERATED_QGA_BASEFILESfor the first list) rather than exploiting the arcane knowledge that consecutive use of := causes GNU make to rewrite an existing variable with new contents?
Quoted 7 lines
Our rules.mak relies on this already. It's full of magic, which admittedly diminishes its suitability to serve as a good example.
Your worry might be rooted in old "=" burns. "=" makes the variable recursively expanded, and
GENERATED_QGA_FILES = $(addprefix qga/qapi-generated/, $(GENERATED_QGA_FILES))
Indeed, but I have to refer to the manual to remind myself of whether = or := is what I want in a given situation.
Quoted 6 lines
would be an infinite loop. ":=" makes it simply expanded; there's not even a loop, let alone an infinite one. The GNU Make manual explains this clearly at
https://www.gnu.org/software/make/manual/html_node/Flavors.html
Quoted 8 lines
Aside: there's a reason one of the two flavors is called "simple". It could additionally be called "not as slow". One of my pet makefile
peeves: unthinking use of recursively expanded variables, complicating semantics and slowing down builds.Back to this patch. I had started to write the thing in longhand, but got tired of repeating qga/qapi-generated/, so I factored that out. Would longhand be easier to understand?
It's more verbose. My suggestion was more:
GENERATED_QGA_BASENAMES := qga-qapi-types.c qga-qapi-types.h GENERATED_QGA_BASENAMES += qga-qapi-visit.c qga-qapi-visit.h
...GENERATED_QGA_FILES := $(addprefix qga/qapi-generated/, $(GENERATED_QGA_BASENAMES))
to avoid the reassignment-to-self issue altogether, while still remaining concise compared to longhand.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 7/6] Makefile: Make Makefile depend on generated qga files, too
Quoted 20 lines · Eric Blake
Quoted 18 lines · Markus Armbruster
Quoted 6 lines
Quoted 6 lines
+++ b/Makefile
Makefile +2
-0@@ -130,6 +130,15 @@ GENERATED_QAPI_FILES += qapi/qapi-doc.texi generated-files-y += $(GENERATED_QAPI_FILES) +GENERATED_QGA_FILES := qga-qapi-types.c qga-qapi-types.h +GENERATED_QGA_FILES += qga-qapi-visit.c qga-qapi-visit.h +GENERATED_QGA_FILES += qga-qapi-commands.h qga-qapi-commands.cQuoted 8 lines
Makefile GENERATED_QAPI_FILES += qapi/qapi-doc.texi · line 138
Quoted 3 lines
Makefile +3
-0+GENERATED_QGA_FILES += qga-qapi-init-commands.h qga-qapi-init-commands.c +GENERATED_QGA_FILES += qga-qapi-doc.texi +GENERATED_QGA_FILES := $(addprefix qga/qapi-generated/, $(GENERATED_QGA_FILES))Would it be worth using two separate variable names (maybe
GENERATED_QGA_BASEFILESfor the first list) rather than exploiting the arcane knowledge that consecutive use of := causes GNU make to rewrite an existing variable with new contents?Our rules.mak relies on this already. It's full of magic, which admittedly diminishes its suitability to serve as a good example.
Quoted 7 lines
Quoted 4 lines
Your worry might be rooted in old "=" burns. "=" makes the variable recursively expanded, and
GENERATED_QGA_FILES = $(addprefix qga/qapi-generated/, $(GENERATED_QGA_FILES))Indeed, but I have to refer to the manual to remind myself of whether = or := is what I want in a given situation.
Trust me, you're either sure you want "=", or you want ":=".
On a green field, I recommend a hard rule "no = without a comment explaining why".
Quoted 16 lines
Quoted 5 lines
would be an infinite loop. ":=" makes it simply expanded; there's not even a loop, let alone an infinite one. The GNU Make manual explains this clearly at
https://www.gnu.org/software/make/manual/html_node/Flavors.htmlQuoted 8 lines
Aside: there's a reason one of the two flavors is called "simple". It could additionally be called "not as slow". One of my pet makefile
peeves: unthinking use of recursively expanded variables, complicating semantics and slowing down builds.Back to this patch. I had started to write the thing in longhand, but got tired of repeating qga/qapi-generated/, so I factored that out. Would longhand be easier to understand?
It's more verbose. My suggestion was more:
Quoted 8 lines
GENERATED_QGA_BASENAMES:=qga-qapi-types.cqga-qapi-types.hGENERATED_QGA_BASENAMES+=qga-qapi-visit.cqga-qapi-visit.h
...GENERATED_QGA_FILES:= $(addprefix qga/qapi-generated/, $(GENERATED_QGA_BASENAMES))to avoid the reassignment-to-self issue altogether, while still remaining concise compared to longhand.
Either way, we use multiple assignments to build GENERATED_QGA_FILES. The only difference is that the version using two variables would also work with recursive expansion, due to the magic of +=.
Quoted 3 lines · Markus Armbruster
Kevin recently posted a minimally invasive fix for empty QAPI modules[*]. This is my attempt at a fix that also addresses the design weakness that led to the bug.
Queued for 5.0.