alternate layout (post-introspection cleanups, subset C)
v8 cover alternate layout (post-introspection cleanups, subset C)
Pending prerequisite: Markus' qapi-next branchgit://repo.or.cz/qemu/armbru.git qapi-next
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg06417.html
Also available as a tag at this location:
git fetch git://repo.or.cz/qemu/ericb.git qapi-cleanupv8c
as well as part of my branch with the rest of the v5 series, at:
http://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/qapi
v8 notes:
Minor changes when rebasing to latest, improved commit messages in a few places, plus the addition of a few new patches. Markus started reviewing v7, but hadn't gotten very far.
001/17:[----] [--] 'qapi: Use generated TestStruct machinery in tests' 002/17:[----] [--] 'qapi: Strengthen test of TestStructList' 003/17:[----] [--] 'qapi: Provide nicer array names in introspection' 004/17:[----] [--] 'qapi-introspect: Guarantee particular sorting' 005/17:[down] 'qapi: Track simple union tag in object.local_members' 006/17:[down] 'qapi-types: Consolidate gen_struct() and gen_union()' 007/17:[0007] [FC] 'qapi: Rework collision assertions' 008/17:[down] 'qapi: Remove outdated tests related to QMP/branch collisions' 009/17:[down] 'qapi: Add positive tests to qapi-schema-test' 010/17:[----] [-C] 'qapi: Simplify visiting of alternate types' 011/17:[----] [--] 'qapi: Fix alternates that accept 'number' but not 'int'' 012/17:[----] [--] 'qapi: Remove dead visitor code' 013/17:[----] [--] 'qapi: Plug leaks in test-qmp-*' 014/17:[----] [-C] 'qapi: Simplify error testing in test-qmp-*' 015/17:[----] [--] 'qapi: Test failure in middle of array parse' 016/17:[----] [--] 'qapi: More tests of input arrays' 017/17:[----] [--] 'qapi: Simplify visits of optional fields'
v7 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg04112.html
Patches 1-3 of the previous round have moved to subset B; patch 7 is gone, and a couple of new patches are present in this round. The latest version of subset B made it much easier to reason about collision detection (namely, tag values can't collide with QMP values thanks to a named rather than anonymous union in the C type), and I like how things turned out. I suspect the hardest part of the review will be patches 5/14 and 7/14, although none of this has really had much review in any earlier versions.
v6 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg01980.html
Add some patches and rebase onto work on subset B. Rearrange some patches from v5 (this set includes 17-20, 23, 25-27). Backport diff gets a bit confused by one patch title changing.
Not much direct review comments, although some of the changes here are updated based on comments made on other patches in the v5 series.
Subset D (and more?) will come later.
In v5:
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg05410.html
I _did_ rearrange patches to try and group related features:
1-2: Groundwork cleanups
3-5: Add more test cases
6-16: Front-end cleanups
17-18: Introspection output cleanups
19-20: 'alternate' type cleanups
21-29: qapi visitor cleanups
30-45: qapi-ify netdev_add
46: add qapi shorthand for flat unions
Lots of fixes based on additional testing, and rebased to track other changes that happened in the meantime. The series is huge; I can split off smaller portions as requested.
In v4:
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg02580.html
add some more clean up patches
rebase to Markus' recent work
pull in part of Zoltán's work to make netdev_add a flat union, further enhancing it to be introspectible
I might be able to rearrange some of these patches, or separate it into smaller independent series, if requested; but I'm posting now to get review started.
In v3:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg02059.html
redo cleanup of dealloc of partial struct
add patches to make all visit_type_*() avoid leaks on failure add patches to allow boxed command arguments and events
In v2:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00900.html
rebase to Markus' v3 series
rework how comments are emitted for fields inherited from base additional patches added for deleting colliding 'void *data' documentation updates to match code changes
v1 was here:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05266.html
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05325.html
Eric Blake (17):
qapi: Use generated TestStruct machinery in tests
qapi: Strengthen test of TestStructList
qapi: Provide nicer array names in introspection
qapi-introspect: Guarantee particular sorting
qapi: Track simple union tag in object.local_members
qapi-types: Consolidate gen_struct() and gen_union()
qapi: Rework collision assertions
qapi: Remove outdated tests related to QMP/branch collisions
qapi: Add positive tests to qapi-schema-test
qapi: Simplify visiting of alternate types
qapi: Fix alternates that accept 'number' but not 'int'
qapi: Remove dead visitor code
qapi: Plug leaks in test-qmp-*
qapi: Simplify error testing in test-qmp-*
qapi: Test failure in middle of array parse
qapi: More tests of input arrays
qapi: Simplify visits of optional fields
docs/qapi-code-gen.txt | 31 ++- include/qapi/visitor-impl.h | 27 ++- include/qapi/visitor.h | 22 +- qapi/introspect.json | 22 +- qapi/opts-visitor.c | 2 +- qapi/qapi-visit-core.c | 141 +++++-------- qapi/qmp-input-visitor.c | 11 +- qapi/string-input-visitor.c | 3 +- scripts/qapi-introspect.py | 17 +- scripts/qapi-types.py | 70 ++----- scripts/qapi-visit.py | 27 ++- scripts/qapi.py | 116 ++++++----- tests/Makefile | 3 - tests/qapi-schema/alternate-empty.out | 1 - tests/qapi-schema/flat-union-clash-branch.err | 0 tests/qapi-schema/flat-union-clash-branch.exit | 1 - tests/qapi-schema/flat-union-clash-branch.json | 18 -- tests/qapi-schema/flat-union-clash-branch.out | 14 -- tests/qapi-schema/flat-union-clash-type.err | 1 - tests/qapi-schema/flat-union-clash-type.exit | 1 - tests/qapi-schema/flat-union-clash-type.json | 14 -- tests/qapi-schema/flat-union-clash-type.out | 0 tests/qapi-schema/qapi-schema-test.json | 26 ++- tests/qapi-schema/qapi-schema-test.out | 46 ++++- tests/qapi-schema/union-clash-data.out | 1 + tests/qapi-schema/union-clash-type.err | 1 - tests/qapi-schema/union-clash-type.exit | 1 - tests/qapi-schema/union-clash-type.json | 9 - tests/qapi-schema/union-clash-type.out | 0 tests/qapi-schema/union-empty.out | 1 + tests/test-qmp-input-strict.c | 108 +++------- tests/test-qmp-input-visitor.c | 268 +++++++++++-------------- tests/test-qmp-output-visitor.c | 153 ++++---------- tests/test-visitor-serialization.c | 76 ++----- 34 files changed, 503 insertions(+), 729 deletions(-) delete mode 100644 tests/qapi-schema/flat-union-clash-branch.err delete mode 100644 tests/qapi-schema/flat-union-clash-branch.exit delete mode 100644 tests/qapi-schema/flat-union-clash-branch.json delete mode 100644 tests/qapi-schema/flat-union-clash-branch.out delete mode 100644 tests/qapi-schema/flat-union-clash-type.err delete mode 100644 tests/qapi-schema/flat-union-clash-type.exit delete mode 100644 tests/qapi-schema/flat-union-clash-type.json delete mode 100644 tests/qapi-schema/flat-union-clash-type.out delete mode 100644 tests/qapi-schema/union-clash-type.err delete mode 100644 tests/qapi-schema/union-clash-type.exit delete mode 100644 tests/qapi-schema/union-clash-type.json delete mode 100644 tests/qapi-schema/union-clash-type.out
Signature · 1 line
2.4.3
PATCH v8 1/17 qapi: Use generated TestStruct machinery in tests
Commit d88f5fd and friends first introduced the various test-qmp-* tests in 2011, with duplicated hand-rolled TestStruct machinery, to make sure the qapi visitor interface was tested. Later, commit 4f193e3 in 2013 added a .json file for further testing use by the files, but without consolidating any of the existing hand-rolled visitors. And with four copies, subtle differences have crept in, between the tests themselves (mainly whitespace differences, but also a question of whether to use NULL or "TestStruct" when calling visit_start_struct()) and from what the generator produces (the hand-rolled versions did not cater to partially-allocated objects, because they did not have a deallocation usage).
Of course, just because the visitor interface is tested does not mean it is a sane interface; and future patches will be changing some of the visitor contracts. Rather than having to duplicate the cleanup work in each copy of the TestStruct visitor, and keep each hand-rolled copy in sync with what the generator supplies, we might as well just test what the generator should give us in the first place.
---
v8: improve commit message
v7: rebase on top of subset B v9; defer unrelated trailing whitespace cleanups to later in series
v6: new patch
--- tests/qapi-schema/qapi-schema-test.json | 6 +++- tests/qapi-schema/qapi-schema-test.out | 5 +++ tests/test-qmp-input-strict.c | 35 -------------------- tests/test-qmp-input-visitor.c | 34 ------------------- tests/test-qmp-output-visitor.c | 58 --------------------------------- tests/test-visitor-serialization.c | 34 ------------------- 6 files changed, 10 insertions(+), 162 deletions(-)
tests/qapi-schema/qapi-schema-test.json +5 -1
@@ -3,6 +3,9 @@ # This file is a stress test of supported qapi constructs that must # parse and compile correctly. +{ 'struct': 'TestStruct', + 'data': { 'integer': 'int', 'boolean': 'bool', 'string': 'str' } } + # for testing enums { 'struct': 'NestedEnumsOne', 'data': { 'enum1': 'EnumOne', # Intentional forward reference
@@ -46,7 +49,8 @@ # dummy struct to force generation of array types not otherwise mentioned { 'struct': 'ForceArrays', - 'data': { 'unused1':['UserDefOne'], 'unused2':['UserDefTwo'] } } + 'data': { 'unused1':['UserDefOne'], 'unused2':['UserDefTwo'], + 'unused3':['TestStruct'] } } # for testing unions # Among other things, test that a name collision between branches does
tests/qapi-schema/qapi-schema-test.out +5 -0
@@ -92,6 +92,7 @@ object EventStructOne object ForceArrays member unused1: UserDefOneList optional=False member unused2: UserDefTwoList optional=False + member unused3: TestStructList optional=False enum MyEnum [] object NestedEnumsOne member enum1: EnumOne optional=False
@@ -100,6 +101,10 @@ object NestedEnumsOne member enum4: EnumOne optional=True enum QEnumTwo ['value1', 'value2'] prefix QENUM_TWO +object TestStruct + member integer: int optional=False + member boolean: bool optional=False + member string: str optional=False object UserDefA member boolean: bool optional=False member a_b: int optional=True
tests/test-qmp-input-strict.c +0 -35
@@ -89,41 +89,6 @@ static Visitor *validate_test_init_raw(TestInputVisitorData *data,return v; } -typedef struct TestStruct -{ -int64_t integer; -bool boolean; -char *string; -}TestStruct ; - -static void visit_type_TestStruct (Visitor *v,TestStruct **obj, -const char *name,Error **errp) -{ -Error *err =NULL ; - -visit_start_struct (v, (void **)obj,"TestStruct" , name,sizeof (TestStruct), - &err); -if (err) { - goto out; - } - -visit_type_int (v, &(*obj)->integer,"integer" , &err); -if (err) { - goto out_end; - } -visit_type_bool (v, &(*obj)->boolean,"boolean" , &err); -if (err) { - goto out_end; - } -visit_type_str (v, &(*obj)->string,"string" , &err); - -out_end: -error_propagate (errp, err); - err =NULL ; -visit_end_struct (v, &err); -out: -error_propagate (errp, err); -}static void test_validate_struct (TestInputVisitorData *data,const void *unused)
tests/test-qmp-input-visitor.c +0 -34
@@ -185,40 +185,6 @@ static void test_visitor_in_enum(TestInputVisitorData *data, data->qiv =NULL ; } -typedef struct TestStruct -{ -int64_t integer; -bool boolean; -char *string; -}TestStruct ; - -static void visit_type_TestStruct (Visitor *v,TestStruct **obj, -const char *name,Error **errp) -{ -Error *err =NULL ; - -visit_start_struct (v, (void **)obj,"TestStruct" , name,sizeof (TestStruct), - &err); -if (err) { - goto out; - } -visit_type_int (v, &(*obj)->integer,"integer" , &err); -if (err) { - goto out_end; - } -visit_type_bool (v, &(*obj)->boolean,"boolean" , &err); -if (err) { - goto out_end; - } -visit_type_str (v, &(*obj)->string,"string" , &err); - -out_end: -error_propagate (errp, err); - err =NULL ; -visit_end_struct (v, &err); -out: -error_propagate (errp, err); -}static void test_visitor_in_struct (TestInputVisitorData *data,const void *unused)
tests/test-qmp-output-visitor.c +0 -58
70 lines not on this page · raw message
tests/test-visitor-serialization.c +0 -34
@@ -186,40 +186,6 @@ static void visit_primitive_list(Visitor *v, void **native, Error **errp) } } -typedef struct TestStruct -{ -int64_t integer; -bool boolean; -char *string; -}TestStruct ; - -static void visit_type_TestStruct (Visitor *v,TestStruct **obj, -const char *name,Error **errp) -{ -Error *err =NULL ; - -visit_start_struct (v, (void **)obj,NULL , name,sizeof (TestStruct), &err); -if (err) { - goto out; - } - -visit_type_int (v, &(*obj)->integer,"integer" , &err); -if (err) { - goto out_end; - } -visit_type_bool (v, &(*obj)->boolean,"boolean" , &err); -if (err) { - goto out_end; - } -visit_type_str (v, &(*obj)->string,"string" , &err); - -out_end: -error_propagate (errp, err); - err =NULL ; -visit_end_struct (v, &err); -out: -error_propagate (errp, err); -}static TestStruct *struct_create (void ) {
Signature · 1 line
2.4.3
PATCH v8 2/17 qapi: Strengthen test of TestStructList
In reply to Eric BlakeMake each list element different, to ensure that order is preserved, and use the generated free function instead of hand-rolling our own to ensure (under valgrind) that the list is properly cleaned.
---
v8: no change
v7: new patch
--- tests/test-qmp-output-visitor.c | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-)
tests/test-qmp-output-visitor.c +7 -11
@@ -283,7 +283,7 @@ static void test_visitor_out_struct_errors(TestOutputVisitorData *data,static void test_visitor_out_list (TestOutputVisitorData *data,const void *unused) { -char *value_str = (char *)"list value" ; +const char *value_str ="list value" ;TestStructList *p, *head =NULL ;const int max_items =10 ;bool value_bool = true;
@@ -294,12 +294,13 @@ static void test_visitor_out_list(TestOutputVisitorData *data,QList *qlist;int i; +/* Build the list in reverse order... */ for (i =0 ; i < max_items; i++) { p =g_malloc0 (sizeof (*p)); p->value =g_malloc0 (sizeof (*p->value)); - p->value->integer = value_int; + p->value->integer = value_int + (max_items - i -1 ); p->value->boolean = value_bool; - p->value->string = value_str; + p->value->string =g_strdup (value_str); p->next = head; head = p;
@@ -315,6 +316,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data, qlist =qobject_to_qlist (obj);g_assert (!qlist_empty (qlist)); +/* ...and ensure that the visitor sees it in order */ i =0 ; QLIST_FOREACH_ENTRY(qlist, entry) {QDict *qdict;
@@ -322,7 +324,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data,g_assert (qobject_type (entry->value) ==QTYPE_QDICT ); qdict =qobject_to_qdict (entry->value); g_assert_cmpint(qdict_size(qdict), ==, 3); - g_assert_cmpint(qdict_get_int(qdict, "integer"), ==, value_int); + g_assert_cmpint(qdict_get_int(qdict, "integer"), ==, value_int + i); g_assert_cmpint(qdict_get_bool(qdict, "boolean"), ==, value_bool); g_assert_cmpstr(qdict_get_str(qdict, "string"), ==, value_str); i++;
@@ -330,13 +332,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data, g_assert_cmpint(i, ==, max_items);QDECREF (qlist); - -for (p = head; p;) { -TestStructList *tmp = p->next; -g_free (p->value); -g_free (p); - p = tmp; - } +qapi_free_TestStructList (head); } static void test_visitor_out_list_qapi_free(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
PATCH v8 3/17 qapi: Provide nicer array names in introspection
In reply to Eric BlakeFor the sake of humans reading introspection output, it is nice to have the name of implicit array types be recognizable as arrays of the underlying type. However, while this patch allows humans to skip from a command with return type "[123]" straight to the definition of type "123" without having to first inspect type "[123]", document that this shortcut should not be taken by client apps.
This makes the resulting introspection string slightly larger by default (just over 200 bytes), but it's in the noise (less than 0.3% of the overall 70k size of 'query-qmp-capabilities').
---
v8: commit message stats on the size increase
v7: no change
v6: no change
--- docs/qapi-code-gen.txt | 7 +++++-- scripts/qapi-introspect.py | 8 +++++--- 2 files changed, 10 insertions(+), 5 deletions(-)
docs/qapi-code-gen.txt +5 -2
@@ -662,11 +662,14 @@ Example: the SchemaInfo for BlockRef from section Alternate types The SchemaInfo for an array type has meta-type "array", and variant member "element-type", which names the array's element type. Array -types are implicitly defined. +types are implicitly defined. For convenience, the array's name may +resemble the element type; however, clients should examine member +"element-type" instead of making assumptions based on parsing member +"name". Example: the SchemaInfo for ['str'] - { "name": "strList", "meta-type": "array", + { "name": "[str]", "meta-type": "array", "element-type": "str" } The SchemaInfo for an enumeration type has meta-type "enum" and
scripts/qapi-introspect.py +5 -3
@@ -107,10 +107,12 @@ const char %(c_name)s[] = %(c_string)s; # characters. if isinstance(typ, QAPISchemaBuiltinType): return typ.name + if isinstance(typ, QAPISchemaArrayType): + return '[' + self._use_type(typ.element_type) + ']' return self._name(typ.name) def _gen_json(self, name, mtype, obj): - if mtype != 'command' and mtype != 'event' and mtype != 'builtin': + if mtype not in ('command', 'event', 'builtin', 'array'): name = self._name(name) obj['name'] = name obj['meta-type'] = mtype
@@ -136,8 +138,8 @@ const char %(c_name)s[] = %(c_string)s; self._gen_json(name, 'enum', {'values': values}) def visit_array_type(self, name, info, element_type): - self._gen_json(name, 'array', - {'element-type': self._use_type(element_type)}) + element = self._use_type(element_type) + self._gen_json('[' + element + ']', 'array', {'element-type': element}) def visit_object_type_flat(self, name, info, members, variants): obj = {'members': [self._gen_member(m) for m in members]}
Signature · 1 line
2.4.3
PATCH v8 4/17 qapi-introspect: Guarantee particular sorting
In reply to Eric BlakeSorting the values of an enum makes it easier to look up whether a particular value is present, via binary rather than linear search (probably most visible with QKeyCode, which has grown over several releases). Additionally, QMP clients need not know which C value is associated with an enum name, so sorting is an effective way to hide that non-ABI aspect of qapi.
While we are at it, it is also easy to sort the members and variants of objects, to allow for a similar binary search (although less compelling, as any struct with that many members should arguably be broken into hierarchical substructs), and equally valid since JSON objects have no specific order in which keys must appear. There is no trivial or obvious way to sort the types of an alternate, so that is left unchanged.
However, the overall SchemaInfo array remains unsorted. It might make sense to sort with 'meta-type' as a primary key and 'name' as a secondary key, but it is not obvious that this will provide benefits to end-user clients (we allow mutually recursive types, so there is no posible topological sorting where a single pass over the array could resolve all types, and while binary searches could be made possible by sorting, it would be even more efficient for clients to read the array into a hashtable for O(1) rather than O(log n) random-access lookups, at which point pre-sorting is wasted effort).
Document these conventions, so that clients will know what can and cannot be relied on.
---
TODO: should the documentation mention that the sorting is done in the C locale? Is there anything required to ensure that python sorts sanely (ie. that the choice of locale while building doesn't cause inadvertent sorting differences such as turning on case-insensitivity)?
v8: no change
v7: tweak commit wording
v6: no change from v5
--- docs/qapi-code-gen.txt | 21 +++++++++++++++++---- qapi/introspect.json | 22 +++++++++++++++++----- scripts/qapi-introspect.py | 9 ++++++--- 3 files changed, 40 insertions(+), 12 deletions(-)
docs/qapi-code-gen.txt +17 -4
@@ -516,6 +516,13 @@ query-qmp-schema. QGA currently doesn't support introspection. query-qmp-schema returns a JSON array of SchemaInfo objects. These objects together describe the wire ABI, as defined in the QAPI schema. +There is no specified order to the SchemaInfo objects returned; a +client must search for a particular name and meta-type throughout the +entire array to learn more about that name. For now, the name for +each SchemaInfo is unique thanks to qapi naming conventions; however +this may be changed in the future (such as allowing a command and an +event with the same name), so it is important that the client check +for the desired meta-type. However, the SchemaInfo can't reflect all the rules and restrictions that apply to QMP. It's interface introspection (figuring out what's
@@ -596,7 +603,8 @@ any. Each element is a JSON object with members "name" (the member's name), "type" (the name of its type), and optionally "default". The member is optional if "default" is present. Currently, "default" can only have value null. Other values are reserved for future -extensions. +extensions. The "members" array is sorted by "name", so that clients +can use a binary search to see if a particular member is supported. Example: the SchemaInfo for MyType from section Struct types
@@ -610,7 +618,9 @@ Example: the SchemaInfo for MyType from section Struct types "variants" is a JSON array describing the object's variant members. Each element is a JSON object with members "case" (the value of type tag this element applies to) and "type" (the name of an object type -that provides the variant members for this type tag value). +that provides the variant members for this type tag value). The +"variants" array is sorted by "case", so it appears in the same +order as the enum type matching "tag". Example: the SchemaInfo for flat union BlockdevOptions from section Union types
@@ -651,7 +661,8 @@ Union types The SchemaInfo for an alternate type has meta-type "alternate", and variant member "members". "members" is a JSON array. Each element is a JSON object with member "type", which names a type. Values of the -alternate type conform to exactly one of its member types. +alternate type conform to exactly one of its member types. There is +no guarantee on the order in which "members" will be listed. Example: the SchemaInfo for BlockRef from section Alternate types
@@ -673,7 +684,9 @@ Example: the SchemaInfo for ['str'] "element-type": "str" } The SchemaInfo for an enumeration type has meta-type "enum" and -variant member "values". +variant member "values". The values are listed in sorted order, +so clients can use a binary search to see if a particular value +is present. Example: the SchemaInfo for MyEnum from section Enumeration types
qapi/introspect.json +17 -5
@@ -25,6 +25,11 @@ # Returns: array of @SchemaInfo, where each element describes an # entity in the ABI: command, event, type, ... # +# The order of the various SchemaInfo is unspecified. For now, the +# name of each SchemaInfo is unique regardless of meta-type, but to be +# safe, clients should check that a given name has the expected +# meta-type. +# # Note: the QAPI schema is also used to help define *internal* # interfaces, by defining QAPI types. These are not part of the QMP # wire ABI, and therefore not returned by this command.
@@ -78,7 +83,8 @@ # Commands and events have the name defined in the QAPI schema. # Unlike command and event names, type names are not part of # the wire ABI. Consequently, type names are meaningless -# strings here. +# strings here. Although all names are currently unique +# regardless of @meta-type, clients should not rely on this. # # All references to other SchemaInfo are by name. #
@@ -130,7 +136,9 @@ # # Additional SchemaInfo members for meta-type 'enum'. # -# @values: the enumeration type's values. +# @values: the enumeration type's values. The values are sorted, so +# clients can use a binary search to see if a particular value +# is present. # # Values of this type are JSON string on the wire. #
@@ -158,14 +166,18 @@ # # Additional SchemaInfo members for meta-type 'object'. # -# @members: the object type's (non-variant) members. +# @members: the object type's (non-variant) members. The members are +# sorted by name, so clients can use a binary search to see +# if a given member is present. # # @tag: #optional the name of the member serving as type tag. # An element of @members with this name must exist. # # @variants: #optional variant members, i.e. additional members that # depend on the type tag's value. Present exactly when -# @tag is present. +# @tag is present. The variants are sorted by case, which +# means they appear in the same order as the values of the +# enum type of the @tag. # # Values of this type are JSON object on the wire. #
@@ -219,7 +231,7 @@ # # Additional SchemaInfo members for meta-type 'alternate'. # -# @members: the alternate type's members. +# @members: the alternate type's members, in no particular order. # The members' wire encoding is distinct, see # docs/qapi-code-gen.txt section Alternate types. #
scripts/qapi-introspect.py +6 -3
@@ -10,6 +10,7 @@ # See the COPYING file in the top-level directory. from qapi import * +from operator import attrgetter # Caveman's json.dumps() replacement (we're stuck at Python 2.4)
@@ -126,7 +127,8 @@ const char %(c_name)s[] = %(c_string)s; def _gen_variants(self, tag_name, variants): return {'tag': tag_name, - 'variants': [self._gen_variant(v) for v in variants]} + 'variants': [self._gen_variant(v) for v in + sorted(variants, key=attrgetter('name'))]} def _gen_variant(self, variant): return {'case': variant.name, 'type': self._use_type(variant.type)}
@@ -135,14 +137,15 @@ const char %(c_name)s[] = %(c_string)s; self._gen_json(name, 'builtin', {'json-type': json_type}) def visit_enum_type(self, name, info, values, prefix): - self._gen_json(name, 'enum', {'values': values}) + self._gen_json(name, 'enum', {'values': sorted(values)}) def visit_array_type(self, name, info, element_type): element = self._use_type(element_type) self._gen_json('[' + element + ']', 'array', {'element-type': element}) def visit_object_type_flat(self, name, info, members, variants): - obj = {'members': [self._gen_member(m) for m in members]} + obj = {'members': [self._gen_member(m) for m in + sorted(members, key=attrgetter('name'))]} if variants: obj.update(self._gen_variants(variants.tag_member.name, variants.variants))
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v8 04/17] qapi-introspect: Guarantee particular sorting
For now, only high-level review.
The main cost of sorting is interface complexity: we need to specify which things are sorted, and what the sorting order is (see your TODO below). Once we've done so, we can't go back.
There's also implementation complexity, but your patch shows it's low enough to be ignored.
Cost needs be justified by benefits. Let's have a look at the benefits.
Quoted 4 lines · Eric Blake
Sorting the values of an enum makes it easier to look up whether a particular value is present, via binary rather than linear search (probably most visible with QKeyCode, which has grown over several releases).
Making the interface guarantee "sorted" saves a client that wants it sorted (say for binary search) the trouble of sorting itself. We sort at QEMU compile time instead of client run time.
My qmp-introspect.c has 48 SchemaInfoEnum, ranging from one to 125 members. Histogram:
#enums with this #members
6 1
11 2
8 3
9 4
2 5
1 6
3 7
1 8
2 9
1 15
1 19
1 27
1 43
1 125Mean is less than eight. Median is *three*.
Sorting these member arrays in the client won't have a noticeable performance impact. Heck, I suspect even using linear search wouldn't! Therefore, we can't really claim client performance as a benefit.
We might be able to claim reduced client complexity as a benefit, but I'm rather skeptical. When your search space has a dozen members, you better stick to simple search techniques. Linear search in array or list and binary search in array look adequate, hash table already approaches overkill, and anything fancier definitely is. Of these, only binary search profits from a "sorted" guarantee. But when you got data in an array already, all it takes to sort it is a call to qsort() and a comparison function. Ten straightforward lines of code, tops. Less in a language that isn't quite as primitive as C. Not much of a benefit, I'm afraid.
Quoted 3 lines
Additionally, QMP clients need not know which
C value is associated with an enum name, so sorting is an effective way to hide that non-ABI aspect of qapi.
Sorting indeed hides the implementation detail of how enumerations are encoded in the server. However, I can't see what would tempt clients into relying on it. I can see for type names, and that's why we hide them (commit 1a9a507).
One more potential benefit: when the schema changes in a way that affects only order in introspection, sorting can hide the changes from clients. Can't think of a practical use of it, though.
Quoted 6 lines
While we are at it, it is also easy to sort the members and variants of objects, to allow for a similar binary search (although less compelling, as any struct with that many members should arguably be broken into hierarchical substructs), and equally valid since JSON objects have no specific order in which keys must appear.
My qmp-introspect.c has 48 SchemaInfoObject, ranging from zero to 27 members. Histogram:
#objs with this #members
3 0
102 2
58 3
28 4
14 5
17 6
15 7
3 8
2 9
7 10
2 11
3 12
1 13
2 14
1 15
1 16
1 27Mean is 9.5, median is 9.
The variants are also enums, and therefore can't be any worse than enums.
Again, client performance is hardly a benefit, and client complexity isn't particularly convincing, either.
Quoted 2 lines
There is no trivial or obvious way to sort the types of
an alternate, so that is left unchanged.
In the schema, an alternate's member has a name, a type and nothing else, so that's what we could use to sort. The name isn't visible in introspection, and the type is masked. Sorting by either hides schema change from the client, but isn't of much use to the client otherwise.
If you want to let the client use binary search without having to sort itself, you obviously have to sort by masked type.
My qmp-introspect.c has *two* alternate types, each with two members.
Quoted 10 lines
However, the overall SchemaInfo array remains unsorted. It might make sense to sort with 'meta-type' as a primary key and 'name' as a secondary key, but it is not obvious that this will provide benefits to end-user clients (we allow mutually recursive types, so there is no posible topological sorting where a single pass over the array could resolve all types, and while binary searches could be made possible by sorting, it would be even more efficient for clients to read the array into a hashtable for O(1) rather than O(log n) random-access lookups, at which point pre-sorting is wasted effort).
My qmp-introspect.c has:
2 alternate
55 array
5 builtin
126 command
48 enum
35 event
260 object
-------------
531 totalSince they all share the same entity name space, I'd use a single hash table to map from name to SchemaInfo, just like qapi.py's QAPISchema does.
Since we visit stuff in a defined order, qmp-introspect.c's order is stable. The order just isn't particularly useful for clients, let alone specified.
Quoted 8 lines
Document these conventions, so that clients will know what can and cannot be relied on.
Signed-off-by Eric Blake <[email protected]>---
TODO: should the documentation mention that the sorting is done in the C locale?
I'd assume C locale because we're sorting plain ASCII strings. But spelling it out can't hurt.
Quoted 4 lines
Is there anything required to ensure that python
sorts sanely (ie. that the choice of locale while building doesn't cause inadvertent sorting differences such as turning on case-insensitivity)?
Good question.
Quoted 14 lines
v8: no change
v7: tweak commit wording
v6: no change from v5--- docs/qapi-code-gen.txt | 21 +++++++++++++++++---- qapi/introspect.json | 22 +++++++++++++++++----- scripts/qapi-introspect.py | 9 ++++++--- 3 files changed, 40 insertions(+), 12 deletions(-)docs/qapi-code-gen.txt +0
-0
docs/qapi-code-gen.txt query-qmp-schema. QGA currently doesn't support introspection. · line 525
Quoted 10 lines
docs/qapi-code-gen.txt +7
-0@@ -516,6 +516,13 @@ query-qmp-schema. QGA currently doesn't support introspection. query-qmp-schema returns a JSON array of SchemaInfo objects. These objects together describe the wire ABI, as defined in the QAPI schema. +There is no specified order to the SchemaInfo objects returned; a +client must search for a particular name and meta-type throughout the +entire array to learn more about that name. For now, the name for +each SchemaInfo is unique thanks to qapi naming conventions; however +this may be changed in the future (such as allowing a command and an +event with the same name), so it is important that the client check +for the desired meta-type.
I feel separate name spaces aren't necessary and would be confusing, and I'd be prepared to cast "single entity name space" in stone now.
docs/qapi-code-gen.txt any. Each element is a JSON object with members "name" (the member's · line 610
Quoted 13 lines
docs/qapi-code-gen.txt +2
-1However, the SchemaInfo can't reflect all the rules and restrictions that apply to QMP. It's interface introspection (figuring out what's@@ -596,7 +603,8 @@ any. Each element is a JSON object with members "name" (the member's name), "type" (the name of its type), and optionally "default". The member is optional if "default" is present. Currently, "default" can only have value null. Other values are reserved for future -extensions. +extensions. The "members" array is sorted by "name", so that clients +can use a binary search to see if a particular member is supported. Example: the SchemaInfo for MyType from section Struct types
[...]
This all boils down to whether the increase in interface specification complexity is justified by the benefits. The cost is small, but I'm having a hard time seeing the benefits, to be honest.
Am I missing something?
Re: [Qemu-devel] [PATCH v8 04/17] qapi-introspect: Guarantee particular sorting
Quoted 15 lines · Markus Armbruster
For now, only high-level review.
The main cost of sorting is interface complexity: we need to specify which things are sorted, and what the sorting order is (see your TODO below). Once we've done so, we can't go back.
There's also implementation complexity, but your patch shows it's low enough to be ignored.
Cost needs be justified by benefits. Let's have a look at the benefits.
My
qmp-introspect.chas 48 SchemaInfoEnum, ranging from one to 125 members. Histogram:
Quoted 7 lines
#enums with this #members
1 27 1 43 1 125Mean is less than eight. Median is *three*.
Binary searches have more overhead on small lists; you don't get the benefits of the better algorithm until you overcome the initial hit of more expensive lookups (bsearch() has to use a callback function, which is intrinsically slower than doing direct comparison; the speedups are only present if the search set is large enough that you are able to skip enough comparisons to overcome the higher cost of the extra indirection). So leaving enums unsorted is unlikely to slow clients down (clients will have to do a linear search, but not the end of the world since we have few large lists, and even among those, the size is unlikely to grow much further).
Quoted 4 lines
Sorting these member arrays in the client won't have a noticeable performance impact. Heck, I suspect even using linear search wouldn't! Therefore, we can't really claim client performance as a benefit.
So I'm inclined to agree with your conclusion.
Quoted 8 lines
Quoted 3 lines
Additionally, QMP clients need not know which
C value is associated with an enum name, so sorting is an effective way to hide that non-ABI aspect of qapi.Sorting indeed hides the implementation detail of how enumerations are encoded in the server. However, I can't see what would tempt clients into relying on it. I can see for type names, and that's why we hide them (commit 1a9a507).
You have a point - even if a client figures out what integer value an enum name maps to, it doesn't really help, because the client never sends the integer over the wire.
Quoted 4 lines
One more potential benefit: when the schema changes in a way that affects only order in introspection, sorting can hide the changes from clients. Can't think of a practical use of it, though.
Because we send names, not numbers, over the wire, we can insert a new enum member in slot 0. Clients blindly doing strcmp(enum[0], "magic") will get thrown off if "magic" moves to a different position - but this is true whether or not we sort. So the only working clients are those that check every enum position, not just enum[0]. So I agree - broken clients are broken whether or not we sort, and working clients don't miss anything whether or not we sort.
Quoted 8 lines
My
qmp-introspect.chas 48 SchemaInfoObject, ranging from zero to 27 members. Histogram:#objs with this #members
1 15 1 16 1 27
Even less than the largest enum (and large structs are already unwieldy, compared to making trees of substructs, so we're less likely to make them too much bigger).
Quoted 16 lines
Since we visit stuff in a defined order, qmp-introspect.c's order is stable. The order just isn't particularly useful for clients, let alone specified.
Quoted 8 lines
Document these conventions, so that clients will know what can and cannot be relied on.
Signed-off-by Eric Blake <[email protected]>---
TODO: should the documentation mention that the sorting is done in the C locale?I'd assume C locale because we're sorting plain ASCII strings. But spelling it out can't hurt.
Quoted 6 lines
Quoted 4 lines
Is there anything required to ensure that python
sorts sanely (ie. that the choice of locale while building doesn't cause inadvertent sorting differences such as turning on case-insensitivity)?Good question.
And my biggest worry.
Quoted 15 lines
docs/qapi-code-gen.txt query-qmp-schema. QGA currently doesn't support introspection. · line 525
Quoted 12 lines
+++
b/docs/qapi-code-gen.txtdocs/qapi-code-gen.txt +7
-0@@ -516,6 +516,13 @@ query-qmp-schema. QGA currently doesn't support introspection. query-qmp-schema returns a JSON array of SchemaInfo objects. These objects together describe the wire ABI, as defined in the QAPI schema. +There is no specified order to the SchemaInfo objects returned; a +client must search for a particular name and meta-type throughout the +entire array to learn more about that name. For now, the name for +each SchemaInfo is unique thanks to qapi naming conventions; however +this may be changed in the future (such as allowing a command and an +event with the same name), so it is important that the client check +for the desired meta-type.I feel separate name spaces aren't necessary and would be confusing, and I'd be prepared to cast "single entity name space" in stone now.
Sounds fair to me.
Quoted 5 lines
This all boils down to whether the increase in interface specification complexity is justified by the benefits. The cost is small, but I'm having a hard time seeing the benefits, to be honest.
Am I missing something?
No, I think you gave a fair review. I put this patch out there for discussion and to see how hard it would be; but I'm not strongly tied to it (if I have to pick only one patch out of 3/17 and 4/17, I'd rather see 3/17 go in). So now that we've looked at the benefits (or rather, that they aren't very strong), I'm okay with the idea of making my v9 spin of this series use an alternative patch that _just_ documents the "single entity name space" and the fact that clients cannot rely on any ordering (we don't sort - we may be deterministic, but we make no guarantees on what that order is, and we are free to change the order in the future).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v8 5/17 qapi: Track simple union tag in object.local_members
In reply to Eric BlakeWe were previously creating all unions with an empty list for local_members. However, it will make it easier to unify struct and union generation if we include the generated tag member in local_members. That way, we can have a common code pattern:
visit the base (if any), visit the local members (if any), visit the variants (if any). The local_members of a flat union remains empty (because the discriminator is already visited as part of the base).
The various front end entities then map as follows:
struct: optional base, optional local_members, no variants simple union: no base, one-element local_members, variants with tag_member
from local_members
flat union: base, no local_members, variants with tag_member from base
alternate: no base, no local_members, variants
With the new local members, we require a bit of finesse to avoid assertions in the clients. No change to generated code.
---
v8: new patch
--- scripts/qapi-types.py | 4 +++- scripts/qapi-visit.py | 4 +++- scripts/qapi.py | 14 ++++++++++---- tests/qapi-schema/qapi-schema-test.out | 2 ++ tests/qapi-schema/union-clash-data.out | 1 + tests/qapi-schema/union-empty.out | 1 + 6 files changed, 20 insertions(+), 6 deletions(-)
scripts/qapi-types.py +3 -1
@@ -269,7 +269,9 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_object_type(self, name, info, base, members, variants): self._fwdecl += gen_fwd_object_or_array(name) if variants: - assert not members # not implemented + if members: + assert len(members) == 1 + assert members[0] == variants.tag_member self.decl += gen_union(name, base, variants) else: self.decl += gen_struct(name, base, members)
scripts/qapi-visit.py +3 -1
@@ -360,7 +360,9 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor): def visit_object_type(self, name, info, base, members, variants): self.decl += gen_visit_decl(name) if variants: - assert not members # not implemented + if members: + assert len(members) == 1 + assert members[0] == variants.tag_member self.defn += gen_visit_union(name, base, variants) else: self.defn += gen_visit_struct(name, base, members)
scripts/qapi.py +10 -4
@@ -957,6 +957,9 @@ class QAPISchemaArrayType(QAPISchemaType): class QAPISchemaObjectType(QAPISchemaType): def __init__(self, name, info, base, local_members, variants): + # struct has local_members, optional base, and no variants + # flat union has base, variants, and no local_members + # simple union has local_members, variants, and no base QAPISchemaType.__init__(self, name, info) assert base is None or isinstance(base, str) for m in local_members:
@@ -1048,9 +1051,11 @@ class QAPISchemaObjectTypeVariants(object): self.variants = variants def check(self, schema, members, seen): - if self.tag_name: + if self.tag_name: # flat union self.tag_member = seen[self.tag_name] - else: + elif seen: # simple union + assert self.tag_member in seen.itervalues() + else: # alternate self.tag_member.check(schema, members, seen) assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants:
@@ -1270,13 +1275,14 @@ class QAPISchema(object): if tag_name: variants = [self._make_variant(key, value) for (key, value) in data.iteritems()] + members = [] else: variants = [self._make_simple_variant(key, value, info) for (key, value) in data.iteritems()] tag_member = self._make_implicit_tag(name, info, variants) + members = [tag_member] self._def_entity( - QAPISchemaObjectType(name, info, base, - self._make_members(OrderedDict(), info), + QAPISchemaObjectType(name, info, base, members, QAPISchemaObjectTypeVariants(tag_name, tag_member, variants)))
tests/qapi-schema/qapi-schema-test.out +2 -0
@@ -132,6 +132,7 @@ object UserDefFlatUnion2 case value2: UserDefB case value3: UserDefA object UserDefNativeListUnion + member type: UserDefNativeListUnionKind optional=False case integer: :obj-intList-wrapper case s8: :obj-int8List-wrapper case s16: :obj-int16List-wrapper
@@ -187,6 +188,7 @@ object __org.qemu_x-Struct object __org.qemu_x-Struct2 member array: __org.qemu_x-Union1List optional=False object __org.qemu_x-Union1 + member type: __org.qemu_x-Union1Kind optional=False case __org.qemu_x-branch: :obj-str-wrapper enum __org.qemu_x-Union1Kind ['__org.qemu_x-branch'] object __org.qemu_x-Union2
tests/qapi-schema/union-clash-data.out +1 -0
@@ -2,5 +2,6 @@ object :empty object :obj-int-wrapper member data: int optional=False object TestUnion + member type: TestUnionKind optional=False case data: :obj-int-wrapper enum TestUnionKind ['data']
tests/qapi-schema/union-empty.out +1 -0
@@ -1,3 +1,4 @@ object :empty object Union + member type: UnionKind optional=False enum UnionKind []
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v8 05/17] qapi: Track simple union tag in object.local_members
Quoted 8 lines · Eric Blake
We were previously creating all unions with an empty list for
local_members. However, it will make it easier to unify struct and union generation if we include the generated tag member inlocal_members. That way, we can have a common code pattern:
visit the base (if any), visit the local members (if any), visit the variants (if any). Thelocal_membersof a flat union remains empty (because the discriminator is already visited as part of the base).
Keeping the implicit tag implicit by not including it in local_members was a conscious design decision, but if including it makes unifying struct and union into objects easier, go right ahead.
Quoted 28 lines
The various front end entities then map as follows:
struct: optional base, optionallocal_members, no variants simple union: no base, one-elementlocal_members, variants withtag_member
fromlocal_members
flat union: base, nolocal_members, variants withtag_memberfrom base
alternate: no base, nolocal_members, variantsWith the new local members, we require a bit of finesse to avoid assertions in the clients. No change to generated code.
Signed-off-by Eric Blake <[email protected]>---
v8: new patch--- scripts/qapi-types.py | 4 +++- scripts/qapi-visit.py | 4 +++- scripts/qapi.py | 14 ++++++++++---- tests/qapi-schema/qapi-schema-test.out | 2 ++ tests/qapi-schema/union-clash-data.out | 1 + tests/qapi-schema/union-empty.out | 1 + 6 files changed, 20 insertions(+), 6 deletions(-)scripts/qapi-types.py +0
-0
scripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 277
Quoted 10 lines
scripts/qapi-types.py +3
-1@@ -269,7 +269,9 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_object_type(self, name, info, base, members, variants): self._fwdecl += gen_fwd_object_or_array(name) if variants: - assert not members # not implemented + if members: + assert len(members) == 1 + assert members[0] == variants.tag_member self.decl += gen_union(name, base, variants) else: self.decl += gen_struct(name, base, members)
The assertion checks that not passing members to gen_union() won't lose any. Before the patch, we assert there are none. After the patch, we assert there's either none or variants.tag_member. Before ans after, gen_union() takes care of variants.tag_member. Okay.
Let's keep the comment, though. Perhaps like this:
# Members other than variants.tag_member not implemented
assert len(members) == 1
assert members[0] == variants.tag_memberI hope this the whole conditional will eventually be replaced by
self.decl += gen_object(name, base, members, variants)
Quoted 5 lines
scripts/qapi-visit.py +0
-0
scripts/qapi-visit.py class QAPISchemaGenVisitVisitor(QAPISchemaVisitor): · line 368
Quoted 10 lines
scripts/qapi-visit.py +3
-1@@ -360,7 +360,9 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor): def visit_object_type(self, name, info, base, members, variants): self.decl += gen_visit_decl(name) if variants: - assert not members # not implemented + if members: + assert len(members) == 1 + assert members[0] == variants.tag_member self.defn += gen_visit_union(name, base, variants) else: self.defn += gen_visit_struct(name, base, members)
Likewise, let's keep the comment.
Quoted 18 lines
scripts/qapi.py +3
-0@@ -957,6 +957,9 @@ class QAPISchemaArrayType(QAPISchemaType): class QAPISchemaObjectType(QAPISchemaType): def __init__(self, name, info, base, local_members, variants): + # struct has local_members, optional base, and no variants + # flat union has base, variants, and no local_members + # simple union has local_members, variants, and no base QAPISchemaType.__init__(self, name, info) assert base is None or isinstance(base, str) for m in local_members:@@ -1048,9 +1051,11 @@ class QAPISchemaObjectTypeVariants(object): self.variants = variants def check(self, schema, members, seen):
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1061
Quoted 10 lines
scripts/qapi.py +4
-2- if self.tag_name: + if self.tag_name: # flat union self.tag_member = seen[self.tag_name] - else: + elif seen: # simple union + assert self.tag_member in seen.itervalues() + else: # alternate self.tag_member.check(schema, members, seen) assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants:
The test for simple union is hackish.
I guess you want to bypass self.tag_member.check() when for simple unions, because it's now checked when QAPISchemaObjectType.check() calls QAPISchemaObjectTypeMember.check() for each of self.local_members.
Could we instead make QAPISchemaAlternateType.check() call QAPISchemaObjectTypeMember.check() for its implicit tag, and drop the else here?
Quoted 7 lines
scripts/qapi.py +1
-0@@ -1270,13 +1275,14 @@ class QAPISchema(object): if tag_name: variants = [self._make_variant(key, value) for (key, value) in data.iteritems()] + members = [] else: variants = [self._make_simple_variant(key, value, info)
scripts/qapi.py class QAPISchema(object): · line 1288
Quoted 10 lines
scripts/qapi.py +2
-2for (key, value) in data.iteritems()] tag_member = self._make_implicit_tag(name, info, variants) + members = [tag_member] self._def_entity( - QAPISchemaObjectType(name, info, base, - self._make_members(OrderedDict(), info), + QAPISchemaObjectType(name, info, base, members, QAPISchemaObjectTypeVariants(tag_name, tag_member, variants)))
Two steps squashed together:
1. Simplify _make_members(OrderedDict(), info) to [].
2. Put a simple union's implicit tag_member into the members array passed to QAPISchemaObjectType().
Okay.
[...]
Re: [Qemu-devel] [PATCH v8 05/17] qapi: Track simple union tag in object.local_members
Quoted 46 lines · Markus Armbruster
Quoted 8 lines · Eric Blake
We were previously creating all unions with an empty list for
local_members. However, it will make it easier to unify struct and union generation if we include the generated tag member inlocal_members. That way, we can have a common code pattern:
visit the base (if any), visit the local members (if any), visit the variants (if any). Thelocal_membersof a flat union remains empty (because the discriminator is already visited as part of the base).Keeping the implicit tag implicit by not including it in
local_memberswas a conscious design decision, but if including it makes unifying struct and union into objects easier, go right ahead.Quoted 15 lines
The various front end entities then map as follows:
struct: optional base, optionallocal_members, no variants simple union: no base, one-elementlocal_members, variants withtag_member
fromlocal_members
flat union: base, nolocal_members, variants withtag_memberfrom base
alternate: no base, nolocal_members, variantsWith the new local members, we require a bit of finesse to avoid assertions in the clients. No change to generated code.
Signed-off-by Eric Blake <[email protected]>+++
b/scripts/qapi-types.pyscripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 277
Quoted 10 lines
scripts/qapi-types.py +3
-1@@ -269,7 +269,9 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_object_type(self, name, info, base, members, variants): self._fwdecl += gen_fwd_object_or_array(name) if variants: - assert not members # not implemented + if members: + assert len(members) == 1 + assert members[0] == variants.tag_member self.decl += gen_union(name, base, variants) else: self.decl += gen_struct(name, base, members)The assertion checks that not passing members to
gen_union()won't lose any. Before the patch, we assert there are none. After the patch, we assert there's either none or variants.tag_member. Before ans after,gen_union()takes care of variants.tag_member. Okay.
Quoted 5 lines
Let's keep the comment, though. Perhaps like this:
# Members other than variants.tag_member not implemented assert len(members) == 1 assert members[0] == variants.tag_member
Indeed, that sounds nicer.
Quoted 4 lines
I hope this the whole conditional will eventually be replaced by
self.decl += gen_object(name, base, members, variants)
Yes. It goes away in 6/17 for qapi-types (as you already saw), and I have a later patch queued that likewise makes it go away for qapi-visit (but not in subset C or D, since there's still a bit more to clean up there first).
Quoted 14 lines
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1061
Quoted 11 lines
def
check(self, schema, members, seen):scripts/qapi.py +4
-2- if self.tag_name: + if self.tag_name: # flat union self.tag_member = seen[self.tag_name] - else: + elif seen: # simple union + assert self.tag_member in seen.itervalues() + else: # alternate self.tag_member.check(schema, members, seen) assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants:The test for simple union is hackish.
Quoted 7 lines
I guess you want to bypass
self.tag_member.check()when for simple unions, because it's now checked whenQAPISchemaObjectType.check()callsQAPISchemaObjectTypeMember.check()for each of self.local_members.Could we instead make
QAPISchemaAlternateType.check()callQAPISchemaObjectTypeMember.check()for its implicit tag, and drop the else here?
I debated about that when writing my patch. Yes, I can make that change (it seems a little bit unclean to be calling self.variants.tag_member.check() in QAPISchemaAlternateType, but not too much worse; and adding a comment might help).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v8 6/17 qapi-types: Consolidate gen_struct() and gen_union()
In reply to Eric BlakeThese two methods are now close enough that we can finally merge them, relying on the fact that simple unions now provide a reasonable local_members. Change gen_struct() to gen_object() that handles all forms of QAPISchemaObjectType, and rename and shrink gen_union() to gen_variants() to handle the portion of gen_object() needed when variants are present.
gen_struct_fields() now has a single caller, so it no longer needs an optional parameter; however, I did not choose to inline it into the caller.
No difference to generated code.
---
v8: new patch
--- scripts/qapi-types.py | 36 +++++++++++------------------------- 1 file changed, 11 insertions(+), 25 deletions(-)
scripts/qapi-types.py +11 -25
@@ -51,7 +51,7 @@ def gen_struct_field(member): return ret -def gen_struct_fields(local_members, base=None): +def gen_struct_fields(local_members, base): ret = '' if base:
@@ -70,7 +70,7 @@ def gen_struct_fields(local_members, base=None): return ret -def gen_struct(name, base, members): +def gen_object(name, base, members, variants): ret = mcgen(''' struct %(c_name)s {
@@ -79,11 +79,14 @@ struct %(c_name)s { ret += gen_struct_fields(members, base) + if variants: + ret += gen_variants(variants) + # Make sure that all structs have at least one field; this avoids # potential issues with attempting to malloc space for zero-length # structs in C, and also incompatibility with C++ (where an empty # struct is size 1). - if not (base and base.members) and not members: + if not (base and base.members) and not members and not variants: ret += mcgen(''' char qapi_dummy_field_for_empty_struct; ''')
@@ -140,17 +143,7 @@ const int %(c_name)s_qtypes[QTYPE_MAX] = { return ret -def gen_union(name, base, variants): - ret = mcgen(''' - -struct %(c_name)s { -''', - c_name=c_name(name)) - if base: - ret += gen_struct_fields([], base) - else: - ret += gen_struct_field(variants.tag_member) - +def gen_variants(variants): # FIXME: What purpose does data serve, besides preventing a union that # has a branch named 'data'? We use it in qapi-visit.py to decide # whether to bypass the switch statement if visiting the discriminator
@@ -159,11 +152,11 @@ struct %(c_name)s { # should not be any data leaks even without a data pointer. Or, if # 'data' is merely added to guarantee we don't have an empty union, # shouldn't we enforce that at .json parse time? - ret += mcgen(''' + ret = mcgen(''' union { /* union tag is @%(c_name)s */ void *data; ''', - c_name=c_name(variants.tag_member.name)) + c_name=c_name(variants.tag_member.name)) for var in variants.variants: # Ugly special case for simple union TODO get rid of it
@@ -176,7 +169,6 @@ struct %(c_name)s { ret += mcgen(''' } u; -}; ''') return ret
@@ -268,13 +260,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_object_type(self, name, info, base, members, variants): self._fwdecl += gen_fwd_object_or_array(name) - if variants: - if members: - assert len(members) == 1 - assert members[0] == variants.tag_member - self.decl += gen_union(name, base, variants) - else: - self.decl += gen_struct(name, base, members) + self.decl += gen_object(name, base, members, variants) if base: self.decl += gen_upcast(name, base) self._gen_type_cleanup(name)
@@ -282,7 +268,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_alternate_type(self, name, info, variants): self._fwdecl += gen_fwd_object_or_array(name) self._fwdefn += gen_alternate_qtypes(name, variants) - self.decl += gen_union(name, None, variants) + self.decl += gen_object(name, None, [variants.tag_member], variants) self.decl += gen_alternate_qtypes_decl(name) self._gen_type_cleanup(name)
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v8 06/17] qapi-types: Consolidate gen_struct() and gen_union()
Quoted 116 lines · Eric Blake
These two methods are now close enough that we can finally merge them, relying on the fact that simple unions now provide a reasonable
local_members. Changegen_struct()togen_object()that handles all forms of QAPISchemaObjectType, and rename and shrinkgen_union()togen_variants()to handle the portion ofgen_object()needed when variants are present.
gen_struct_fields()now has a single caller, so it no longer needs an optional parameter; however, I did not choose to inline it into the caller.No difference to generated code.
Signed-off-by Eric Blake <[email protected]>---
v8: new patch--- scripts/qapi-types.py | 36 +++++++++++------------------------- 1 file changed, 11 insertions(+), 25 deletions(-)scripts/qapi-types.py +10
-24@@ -51,7 +51,7 @@ def gen_struct_field(member): return ret -def gen_struct_fields(local_members, base=None): +def gen_struct_fields(local_members, base): ret = '' if base:@@ -70,7 +70,7 @@ def gen_struct_fields(local_members, base=None): return ret -def gen_struct(name, base, members): +def gen_object(name, base, members, variants): ret = mcgen(''' struct %(c_name)s {@@ -79,11 +79,14 @@ struct %(c_name)s { ret += gen_struct_fields(members, base) + if variants: + ret += gen_variants(variants) + # Make sure that all structs have at least one field; this avoids # potential issues with attempting to malloc space for zero-length # structs in C, and also incompatibility with C++ (where an empty # struct is size 1). - if not (base and base.members) and not members: + if not (base and base.members) and not members and not variants: ret += mcgen(''' char qapi_dummy_field_for_empty_struct; ''')@@ -140,17 +143,7 @@ const int %(c_name)s_qtypes[QTYPE_MAX] = { return ret -def gen_union(name, base, variants): - ret = mcgen(''' - -struct %(c_name)s { -''', - c_name=c_name(name)) - if base: - ret += gen_struct_fields([], base) - else: - ret += gen_struct_field(variants.tag_member) - +def gen_variants(variants): # FIXME: What purpose does data serve, besides preventing a union that # has a branch named 'data'? We use it in qapi-visit.py to decide # whether to bypass the switch statement if visiting the discriminator@@ -159,11 +152,11 @@ struct %(c_name)s { # should not be any data leaks even without a data pointer. Or, if # 'data' is merely added to guarantee we don't have an empty union, # shouldn't we enforce that at .json parse time? - ret += mcgen(''' + ret = mcgen(''' union { /* union tag is @%(c_name)s */ void *data; ''', - c_name=c_name(variants.tag_member.name)) + c_name=c_name(variants.tag_member.name)) for var in variants.variants: # Ugly special case for simple union TODO get rid of it@@ -176,7 +169,6 @@ struct %(c_name)s { ret += mcgen(''' } u; -}; ''') return ret@@ -268,13 +260,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_object_type(self, name, info, base, members, variants): self._fwdecl += gen_fwd_object_or_array(name) - if variants: - if members: - assert len(members) == 1 - assert members[0] == variants.tag_member - self.decl += gen_union(name, base, variants) - else: - self.decl += gen_struct(name, base, members) + self.decl += gen_object(name, base, members, variants) if base: self.decl += gen_upcast(name, base) self._gen_type_cleanup(name)
scripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 273
Quoted 7 lines
scripts/qapi-types.py +1
-1@@ -282,7 +268,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_alternate_type(self, name, info, variants): self._fwdecl += gen_fwd_object_or_array(name) self._fwdefn += gen_alternate_qtypes(name, variants) - self.decl += gen_union(name, None, variants) + self.decl += gen_object(name, None, [variants.tag_member], variants) self.decl += gen_alternate_qtypes_decl(name) self._gen_type_cleanup(name)
Turned out nicely.
We could morph gen_struct_field() back into its original shape in a separate patch:
def gen_struct_fields(members):
ret = ''
for memb in members:
ret += gen_struct_field(memb.name, memb.type, memb.optional)
return retwith calling code
ret += mcgen('''
/* Members inherited from %(c_name)s: */
''',
c_name=base.c_name())
ret += gen_struct_fields(base.members)
ret += mcgen('''
/* Own members: */
''')
ret += gen_struct_fields(local_members)Matter of taste.
Re: [Qemu-devel] [PATCH v8 06/17] qapi-types: Consolidate gen_struct() and gen_union()
Quoted 33 lines · Markus Armbruster
Quoted 31 lines · Eric Blake
These two methods are now close enough that we can finally merge them, relying on the fact that simple unions now provide a reasonable
local_members. Changegen_struct()togen_object()that handles all forms of QAPISchemaObjectType, and rename and shrinkgen_union()togen_variants()to handle the portion ofgen_object()needed when variants are present.
gen_struct_fields()now has a single caller, so it no longer needs an optional parameter; however, I did not choose to inline it into the caller.No difference to generated code.
Signed-off-by Eric Blake <[email protected]>def
visit_object_type(self, name, info, base, members, variants):
self._fwdecl +=gen_fwd_object_or_array(name)scripts/qapi-types.py +1
-7- if variants: - if members: - assert len(members) == 1 - assert members[0] == variants.tag_member - self.decl += gen_union(name, base, variants) - else: - self.decl += gen_struct(name, base, members) + self.decl += gen_object(name, base, members, variants) if base: self.decl += gen_upcast(name, base) self._gen_type_cleanup(name)@@ -282,7 +268,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_alternate_type(self, name, info, variants):
Quoted 8 lines
scripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 273
Quoted 6 lines
scripts/qapi-types.py +1
-1self._fwdecl += gen_fwd_object_or_array(name) self._fwdefn += gen_alternate_qtypes(name, variants) - self.decl += gen_union(name, None, variants) + self.decl += gen_object(name, None, [variants.tag_member], variants) self.decl += gen_alternate_qtypes_decl(name) self._gen_type_cleanup(name)Turned out nicely.
Yes, I was pretty pleased with it. It gives more credence to our choice of introspection representation.
Quoted 23 lines
We could morph
gen_struct_field()back into its original shape in a separate patch:def gen_struct_fields(members): ret = '' for memb in members: ret += gen_struct_field(memb.name, memb.type, memb.optional) return retwith calling code
ret += mcgen(''' /* Members inherited from %(c_name)s: */ ''', c_name=base.c_name()) ret += gen_struct_fields(base.members) ret += mcgen(''' /* Own members: */ ''') ret += gen_struct_fields(local_members)
Quoted 1 line
Matter of taste.
Reasonable idea; I'll add it in for the v9 spin.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v8 7/17 qapi: Rework collision assertions
In reply to Eric BlakeNow that we have separate namespaces for QMP vs. tag values, we can simplify how the QAPISchema*.check() methods check for collisions. Each QAPISchemaObjectTypeMember check() call is given a single set of names it must not collide with; this set is either the QMP names (when this member is used by an ObjectType) or the case names (when this member is used by an ObjectTypeVariants). We no longer need an all_members parameter, as it can be computed by seen.values(). When used by a union, QAPISchemaObjectTypeVariant must also perform a QMP collision check for each member of its corresponding type.
The new ObjectType.check_qmp() is an idempotent subset of check(), and can be called multiple times over different seen sets (useful, since the members of one type can be applied into more than one other location via inheritance or flat union variants).
The code needs a temporary hack of passing a 'union' flag through Variants.check(), since we do not inline the branches of an alternate type into a parent QMP object. A later patch will rework how alternates are laid out, by adding a new subclass, and that will allow us to drop the extra parameter.
There are no changes to generated code.
Future patches will enhance testsuite coverage, improve error message quality on actual collisions, and move collision checks out of ad hoc parse code into the check() methods.
---
v8: rebase to earlier patches, defer positive test additions to later in series
v7: new patch, although it is a much cleaner implementation of what was attempted by subset B v8 15/18
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg03042.html
--- scripts/qapi.py | 55 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 22 deletions(-)
scripts/qapi.py +33 -22
@@ -972,28 +972,28 @@ class QAPISchemaObjectType(QAPISchemaType): self.variants = variants self.members = None + # Finish construction, and validate that all members are usable def check(self, schema): assert self.members is not False # not running in cycles if self.members: return self.members = False # mark as being checked + seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType) - assert not self.base.variants # not implemented - self.base.check(schema) - members = list(self.base.members) - else: - members = [] - seen = {} - for m in members: - assert c_name(m.name) not in seen - seen[m.name] = m + self.base.check_qmp(schema, seen) for m in self.local_members: - m.check(schema, members, seen) + m.check(schema, seen) if self.variants: - self.variants.check(schema, members, seen) - self.members = members + self.variants.check(schema, seen) + self.members = seen.values() + + # Check that this type does not introduce QMP collisions into seen + def check_qmp(self, schema, seen): + self.check(schema) + assert not self.variants # not implemented + for m in self.members: + m.check(schema, seen) def is_implicit(self): # See QAPISchema._make_implicit_object_type()
@@ -1027,11 +1027,13 @@ class QAPISchemaObjectTypeMember(object): self.type = None self.optional = optional - def check(self, schema, all_members, seen): + def check(self, schema, seen): + # seen is a map of names we must not collide with (either QMP + # names, when called by ObjectType, or case names, when called + # by Variant). This method is safe to call over multiple 'seen'. assert self.name not in seen self.type = schema.lookup_type(self._type_name) assert self.type - all_members.append(self) seen[self.name] = self
@@ -1050,26 +1052,35 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member = tag_member self.variants = variants - def check(self, schema, members, seen): + # TODO drop union once alternates can be distinguished by tag_member + def check(self, schema, seen, union=True): if self.tag_name: # flat union self.tag_member = seen[self.tag_name] + assert self.tag_member elif seen: # simple union assert self.tag_member in seen.itervalues() else: # alternate - self.tag_member.check(schema, members, seen) + self.tag_member.check(schema, seen) assert isinstance(self.tag_member.type, QAPISchemaEnumType) + cases = OrderedDict() for v in self.variants: - vseen = dict(seen) - v.check(schema, self.tag_member.type, vseen) + # Reset seen array for each variant, since QMP names from one + # branch do not affect another branch, nor add to all_members + v.check(schema, self.tag_member.type, dict(seen), cases, union) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def __init__(self, name, typ): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) - def check(self, schema, tag_type, seen): - QAPISchemaObjectTypeMember.check(self, schema, [], seen) + # TODO drop union once alternates can be distinguished by tag_type + def check(self, schema, tag_type, seen, cases, union): + # cases is case names we must not collide with + QAPISchemaObjectTypeMember.check(self, schema, cases) assert self.name in tag_type.values + if union: + # seen is QMP names our members must not collide with + self.type.check_qmp(schema, seen) # This function exists to support ugly simple union special cases # TODO get rid of them, and drop the function
@@ -1090,7 +1101,7 @@ class QAPISchemaAlternateType(QAPISchemaType): self.variants = variants def check(self, schema): - self.variants.check(schema, [], {}) + self.variants.check(schema, {}, False) def json_type(self): return 'value'
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v8 07/17] qapi: Rework collision assertions
Quoted 1 line · Eric Blake
Now that we have separate namespaces for QMP vs. tag values,
What's the "QMP namespace"?
Quoted 2 lines
we can simplify how the QAPISchema*.check() methods check for collisions.
I *guess* the point of this patch is dropping checks that are obsolete now tag values no longer collide with non-variant member names in generated C, with simplifications on top. Correct?
Quoted 8 lines
Each QAPISchemaObjectTypeMember
check()call is
given a single set of names it must not collide with; this set is either the QMP names (when this member is used by an ObjectType) or the case names (when this member is used by an ObjectTypeVariants). We no longer need anall_membersparameter, as it can be computed byseen.values(). When used by a union, QAPISchemaObjectTypeVariant must also perform a QMP collision check for each member of its corresponding type.
I'm afraid this explanation is next to impossible to understand unless you know how the checking works. I should know, because I wrote it, but actually don't, at least not by heart. So let me relearn how checking works before your patch.
We're talking about a single invocation of QAPISchemaObjectType.check(). Its job is to compute self.members and self.base, and do sanity checking, which includes checking for collisions. It has two variables of interest:
* members is where we accumulate the list of members that'll become self.members. It's initialized to the inherited members (empty if no base, obviously).
* seen is a dictionary mapping names to members. This is merely for collision checking. It's initialized to contain the inherited members (whose names must be distinct, by induction, because we make sure the base type's check() completes first).
For each local member m of self, we call m.check(schema, members, seen). This is actually QAPISchemaObjectTypeMember.check(), and it uses members and seen as follows:
* Assert m.name doesn't collide with another member's name, i.e. not in seen.
* Append m to members.
* Insert m.name: m into seen.
Straightforward so far. Coming up next: variants.
Each variant's members are represented as a single member with the tag value as name. Its type is an object type that has the variant's members.
For each variant v:
* Copy seen to vseen. It holds the non-variant members.
* Call QAPISchemaObjectTypeMember.check(schema, [], vseen), which boils down to assert v.name doesn't collide with a non-variant member's name. This check is obsolete.
* Assert v.name is a member of tag_type.
Not checked: variant's name doesn't collide with another variant's name. That's because we copy seen inside the loop instead of before the loop.
Not checked: variant's members don't collide with non-variant members. I think this check got lost when we simplified QAPISchemaObjectTypeVariants to hold a single member.
Note that the real checking happens in check_union(), and the checks here are just sanity checks. As long as that's the case, holes here are harmless. However, we need them plugged them when we move the real checking from check_union() to the .check().
Looks like variant checking should be thrown out and redone.
I still don't get your description. Guess I have to read the patch after all ;)
Quoted 5 lines
The new
ObjectType.check_qmp()is an idempotent subset ofcheck(), and can be called multiple times over different seen sets (useful, since the members of one type can be applied into more than one other location via inheritance or flat union variants).
I think I get this argument. Except I don't get why the method's named check_qmp().
Quoted 3 lines
The code needs a temporary hack of passing a 'union' flag through
Variants.check(), since we do not inline the branches of an alternate type into a parent QMP object.
What a "QMP object"? Sounds like a JSON object on the QMP wire, but that makes no sense :)
Quoted 27 lines
A later patch
will rework how alternates are laid out, by adding a new subclass, and that will allow us to drop the extra parameter.There are no changes to generated code.
Future patches will enhance testsuite coverage, improve error message quality on actual collisions, and move collision checks out of ad hoc parse code into the
check()methods.Signed-off-by Eric Blake <[email protected]>---
v8: rebase to earlier patches, defer positive test additions to later in series
v7: new patch, although it is a much cleaner implementation of what was attempted by subset B v8 15/18
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg03042.html--- scripts/qapi.py | 55 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 22 deletions(-)scripts/qapi.py +0
-0
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 981
Quoted 10 lines
scripts/qapi.py +2
-0@@ -972,28 +972,28 @@ class QAPISchemaObjectType(QAPISchemaType): self.variants = variants self.members = None + # Finish construction, and validate that all members are usable def check(self, schema): assert self.members is not False # not running in cycles if self.members: return self.members = False # mark as being checked + seen = OrderedDict()
Why do you need to switch from {} to OrderedDict()?
Quoted 18 lines
scripts/qapi.py +2
-12if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType) - assert not self.base.variants # not implemented - self.base.check(schema) - members = list(self.base.members) - else: - members = [] - seen = {} - for m in members: - assert c_name(m.name) not in seen - seen[m.name] = m + self.base.check_qmp(schema, seen) for m in self.local_members: - m.check(schema, members, seen) + m.check(schema, seen) if self.variants: - self.variants.check(schema, members, seen)
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 996
Quoted 10 lines
scripts/qapi.py +9
-1- self.members = members + self.variants.check(schema, seen) + self.members = seen.values() + + # Check that this type does not introduce QMP collisions into seen + def check_qmp(self, schema, seen): + self.check(schema) + assert not self.variants # not implemented + for m in self.members: + m.check(schema, seen)
You said "ObjectType.check_qmp() is an idempotent subset of check()", but it's obviously a superset: it calls check(), then does some more work.
I think I'm now too confused to make further progress on this patch, probably because I've thought myself into a corner.
Before I give up, a remark on design. The QAPISchemaFOO classes all implement the same protocol:
* __init__() type-checks arguments and initializes instance variables, generally either to an argument or to None. This is basically AST construction.
* check() computes the remaining instance variables. This is semantic analysis, except it's stubbed out.
QAPISchema.__init__() calls its own check(), which calls all the others, directly (for entities), or indirect (for members and such). Each check() runs *once*.
Except for QAPISchemaType.check(). Why? Unlike other entities, the types need to be checked in a certain order: contained types (base and variant) before the containing type. For simplicity, we simply call their check(), and detect cycles. This is basically a topological sort and the real checking squashed together. The real checking still runs once.
It follows that you must not call check() except:
- An entity is responsible for calling check() for the non-entities it owns (e.g. an object type's check() calls its members' check()).
- A type may call check() for a type it contains (e.g. its base type). That's safe, because no type may contain itself. This kind of call drives the top-sort.
* Obviously, the instance variables computed by check() have valid values only after check(). Likewise, certain methods should be called only after check(), e.g. visit().
Of course, if we find a more suitable protocol, we're free to adopt it.
scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1033
Quoted 12 lines
scripts/qapi.py +4
-1def is_implicit(self): # See QAPISchema._make_implicit_object_type()@@ -1027,11 +1027,13 @@ class QAPISchemaObjectTypeMember(object): self.type = None self.optional = optional - def check(self, schema, all_members, seen): + def check(self, schema, seen): + # seen is a map of names we must not collide with (either QMP + # names, when called by ObjectType, or case names, when called + # by Variant). This method is safe to call over multiple 'seen'.
What are "QMP names"? Member names, perhaps?
Quoted 58 lines
scripts/qapi.py +16
-8assert self.name not in seen self.type = schema.lookup_type(self._type_name) assert self.type - all_members.append(self) seen[self.name] = self@@ -1050,26 +1052,35 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member = tag_member self.variants = variants - def check(self, schema, members, seen): + # TODO drop union once alternates can be distinguished by tag_member + def check(self, schema, seen, union=True): if self.tag_name: # flat union self.tag_member = seen[self.tag_name] + assert self.tag_member elif seen: # simple union assert self.tag_member in seen.itervalues() else: # alternate - self.tag_member.check(schema, members, seen) + self.tag_member.check(schema, seen) assert isinstance(self.tag_member.type, QAPISchemaEnumType) + cases = OrderedDict() for v in self.variants: - vseen = dict(seen) - v.check(schema, self.tag_member.type, vseen) + # Reset seen array for each variant, since QMP names from one + # branch do not affect another branch, nor add to all_members + v.check(schema, self.tag_member.type, dict(seen), cases, union) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def __init__(self, name, typ): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) - def check(self, schema, tag_type, seen): - QAPISchemaObjectTypeMember.check(self, schema, [], seen) + # TODO drop union once alternates can be distinguished by tag_type + def check(self, schema, tag_type, seen, cases, union): + # cases is case names we must not collide with + QAPISchemaObjectTypeMember.check(self, schema, cases) assert self.name in tag_type.values + if union: + # seen is QMP names our members must not collide with + self.type.check_qmp(schema, seen) # This function exists to support ugly simple union special cases # TODO get rid of them, and drop the function@@ -1090,7 +1101,7 @@ class QAPISchemaAlternateType(QAPISchemaType): self.variants = variants def check(self, schema): - self.variants.check(schema, [], {}) + self.variants.check(schema, {}, False) def json_type(self): return 'value'
Re: [Qemu-devel] [PATCH v8 07/17] qapi: Rework collision assertions
Quoted 5 lines · Markus Armbruster
Quoted 1 line · Eric Blake
Now that we have separate namespaces for QMP vs. tag values,
What's the "QMP namespace"?
I guess I need to be more explicit :)
In the generated C struct for a qapi object, we now have two separate namespaces:
struct Foo {
Type1 name1; /* namespace for QMP (aka non-variant) members */
Type2 name2;
union {
Type1 name1; /* namespace for tag values */
Type2 name2;
} u;
};Quoted 8 lines
Quoted 3 lines
we can simplify how the QAPISchema*.check() methods check for collisions.
I *guess* the point of this patch is dropping checks that are obsolete now tag values no longer collide with non-variant member names in generated C, with simplifications on top. Correct?
Almost. This is not actually dropping any of the old ad hoc checks in the parser, but you are correct that it IS fixing the check() methods to quit asserting things that are no longer forbidden now that tag values no longer collide with non-variant member names.
Quoted 6 lines
Quoted 6 lines
Each QAPISchemaObjectTypeMember
check()call is
given a single set of names it must not collide with; this set is either the QMP names (when this member is used by an ObjectType) or the case names (when this member is used by an ObjectTypeVariants). We no longer need anall_membersparameter, as it can be computed byseen.values().
This is point [1] mentioned below.
Quoted 8 lines
Quoted 3 lines
When used
by a union, QAPISchemaObjectTypeVariant must also perform a QMP collision check for each member of its corresponding type.I'm afraid this explanation is next to impossible to understand unless you know how the checking works. I should know, because I wrote it, but actually don't, at least not by heart. So let me relearn how checking works before your patch.
I guess I get to pick and choose from your wording, to try and make my commit body more comprehensible.
Quoted 9 lines
We're talking about a single invocation of
QAPISchemaObjectType.check(). Its job is to compute self.members and self.base, and do sanity checking, which includes checking for collisions. It has two variables of interest:* members is where we accumulate the list of members that'll become self.members. It's initialized to the inherited members (empty if no base, obviously).
True pre-patch; eliminated by point [1] mentioned above, by observing that seen.values() is identical to members after processing local_members, and that we don't further modify seen when processing variants.
Quoted 31 lines
* seen is a dictionary mapping names to members. This is merely for collision checking. It's initialized to contain the inherited members (whose names must be distinct, by induction, because we make sure the base type's
check()completes first).For each local member m of self, we call
m.check(schema, members, seen). This is actuallyQAPISchemaObjectTypeMember.check(), and it uses members and seen as follows:* Assert m.name doesn't collide with another member's name, i.e. not in seen.
* Append m to members.
* Insert m.name: m into seen.
Straightforward so far. Coming up next: variants.
Each variant's members are represented as a single member with the tag value as name. Its type is an object type that has the variant's members.
For each variant v:
* Copy seen to vseen. It holds the non-variant members.
* Call
QAPISchemaObjectTypeMember.check(schema, [], vseen), which boils down to assert v.name doesn't collide with a non-variant member's name. This check is obsolete.
Quoted 8 lines
* Assert v.name is a member of
tag_type.Not checked: variant's name doesn't collide with another variant's name. That's because we copy seen inside the loop instead of before the loop.
Not checked: variant's members don't collide with non-variant members. I think this check got lost when we simplified QAPISchemaObjectTypeVariants to hold a single member.
Perhaps so; our testsuite wasn't as complete at that time. Or maybe it's because our ad hoc checks in the parsing portion were preventing us from realizing that we needed to repeat things in check() methods.
Quoted 5 lines
Note that the real checking happens in
check_union(), and the checks here are just sanity checks. As long as that's the case, holes here are harmless. However, we need them plugged them when we move the real checking fromcheck_union()to the .check().
Yep, and that's what this patch is trying to do.
Quoted 6 lines
Looks like variant checking should be thrown out and redone.
I still don't get your description. Guess I have to read the patch after all ;)
Quoted 8 lines
Quoted 5 lines
The new
ObjectType.check_qmp()is an idempotent subset ofcheck(), and can be called multiple times over different seen sets (useful, since the members of one type can be applied into more than one other location via inheritance or flat union variants).I think I get this argument. Except I don't get why the method's named
check_qmp().
Check that the QMP field names introduced by this type into an overall object do not cause any collisions with QMP field names already present from other sources. I'm open to suggestions for a better name.
Quoted 7 lines
Quoted 4 lines
The code needs a temporary hack of passing a 'union' flag through
Variants.check(), since we do not inline the branches of an alternate type into a parent QMP object.What a "QMP object"? Sounds like a JSON object on the QMP wire, but that makes no sense :)
Let's try again:
The code needs a temporary hack of passing a 'union' flag through Variants.check(), in order to decide whether to check the various fields of the branch object type against the earlier QMP names in seen. The check is conditional on being a union, because the QMP field names of each union branch share the same QMP namespace as the non-variant fields in the overall JSON object (even though they are in a different C namespace due to boxing behind the branch type), while an alternate does not inline any QMP fields (because it is not part of a larger JSON object). Note that we do not need to distinguish between simple and flat unions, because simple unions have a generated wrapper type whose lone QMP field ('data') will be checked against the lone 'type' non-variant field.
Quoted 13 lines
Quoted 13 lines
A later patch
will rework how alternates are laid out, by adding a new subclass, and that will allow us to drop the extra parameter.There are no changes to generated code.
Future patches will enhance testsuite coverage, improve error message quality on actual collisions, and move collision checks out of ad hoc parse code into the
check()methods.Signed-off-by Eric Blake <[email protected]>
Quoted 9 lines
Quoted 9 lines
---
v8: rebase to earlier patches, defer positive test additions to later in series
v7: new patch, although it is a much cleaner implementation of what was attempted by subset B v8 15/18
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg03042.html--- scripts/qapi.py | 55 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 22 deletions(-)
Okay, let's see what you think of the actual patch, or if you have any better suggestions for how I should have worded the commit body.
Quoted 9 lines
Quoted 9 lines
scripts/qapi.py +0
-0@@ -972,28 +972,28 @@ class QAPISchemaObjectType(QAPISchemaType): self.variants = variants self.members = None
Quoted 9 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 981
Quoted 7 lines
scripts/qapi.py +2
-0+ # Finish construction, and validate that all members are usable def check(self, schema): assert self.members is not False # not running in cycles if self.members: return self.members = False # mark as being checked + seen = OrderedDict()Why do you need to switch from {} to
OrderedDict()?
Because I'm dropping the all_members argument, and I need seen.values() to maintain the same order in which members are seen, for it to be a replacement for all_members.
Maybe I need to split this patch into a few more pieces, each tackling a smaller cleanup, rather than trying to do it all in one?
Quoted 6 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 984
Quoted 6 lines
scripts/qapi.py +0
-3if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType) - assert not self.base.variants # not implemented - self.base.check(schema)
[2] The old code calls base.check()...
Quoted 1 line
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 985
Quoted 1 line
scripts/qapi.py +0
-1- members = list(self.base.members)
x>> - else:
Quoted 5 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 991
Quoted 5 lines
scripts/qapi.py +0
-5- members = [] - seen = {} - for m in members: - assert c_name(m.name) not in seen - seen[m.name] = m
...then adds each of base's members into seen.
Quoted 1 line
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 984
Quoted 1 line
scripts/qapi.py +1
-0+ self.base.check_qmp(schema, seen)
The new code combines all of those actions under the helper base.check_qmp().
Quoted 8 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 989
Quoted 8 lines
scripts/qapi.py +3
-3for m in self.local_members: - m.check(schema, members, seen) + m.check(schema, seen) if self.variants: - self.variants.check(schema, members, seen) - self.members = members + self.variants.check(schema, seen) + self.members = seen.values()
Nothing else in child calls check_qmp().
Quoted 11 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 996
Quoted 7 lines
scripts/qapi.py +7
-0+ + # Check that this type does not introduce QMP collisions into seen + def check_qmp(self, schema, seen): + self.check(schema) + assert not self.variants # not implemented + for m in self.members: + m.check(schema, seen)You said "
ObjectType.check_qmp()is an idempotent subset ofcheck()", but it's obviously a superset: it callscheck(), then does some more work.
See [2] above. I need to reword that then, or split it into a separate patch for easier reasoning. base.check_qmp() is a subset of the work done by child.check(). You are correct that it is NOT a subset of the work done by base.check() (because it calls base.check() and then does additional work).
Quoted 11 lines
I think I'm now too confused to make further progress on this patch, probably because I've thought myself into a corner.
Before I give up, a remark on design. The QAPISchemaFOO classes all implement the same protocol:
*
__init__()type-checks arguments and initializes instance variables, generally either to an argument or to None. This is basically AST construction.
Quoted 6 lines
*
check()computes the remaining instance variables. This is semantic analysis, except it's stubbed out.
QAPISchema.__init__()calls its owncheck(), which calls all the others, directly (for entities), or indirect (for members and such). Eachcheck()runs *once*.
And I'm intentionally reworking things to run a _subset_ of check() multiple times - namely, calling member.check(seen) to see if member collides with any names already present in seen, where seen can be either the set of inherited non-variant fields (for both local members and for the fields of a variant type from a qapi union), or where seen can be the set of tag names (for the use of QAPISchemaObjectTypeVariant, which is a subclass of QAPISchemaObjectTypeMember).
Quoted 30 lines
Except for
QAPISchemaType.check(). Why? Unlike other entities, the types need to be checked in a certain order: contained types (base and variant) before the containing type. For simplicity, we simply call theircheck(), and detect cycles. This is basically a topological sort and the real checking squashed together. The real checking still runs once.It follows that you must not call
check()except:- An entity is responsible for calling
check()for the non-entities it owns (e.g. an object type'scheck()calls its members'check()).- A type may call
check()for a type it contains (e.g. its base type). That's safe, because no type may contain itself. This kind of call drives the top-sort.* Obviously, the instance variables computed by
check()have valid values only aftercheck(). Likewise, certain methods should be called only aftercheck(), e.g.visit().Of course, if we find a more suitable protocol, we're free to adopt it.
Quoted 7 lines
scripts/qapi.py +0
-0def is_implicit(self): # See QAPISchema._make_implicit_object_type()@@ -1027,11 +1027,13 @@ class QAPISchemaObjectTypeMember(object): self.type = None self.optional = optional
Quoted 7 lines
scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1033
Quoted 5 lines
scripts/qapi.py +4
-1- def check(self, schema, all_members, seen): + def check(self, schema, seen): + # seen is a map of names we must not collide with (either QMP + # names, when called by ObjectType, or case names, when called + # by Variant). This method is safe to call over multiple 'seen'.What are "QMP names"? Member names, perhaps?
Yes, the member names that result from JSON object keys at the same level of nesting, whether or not those names are attributed to the same C namespace.
I still like my patch, but I can see it caused some confusion. So I'll try to break it into smaller pieces for v9.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v8 07/17] qapi: Rework collision assertions
In reply to Markus ArmbrusterQuoted 4 lines · Markus Armbruster
Not checked: variant's members don't collide with non-variant members. I think this check got lost when we simplified QAPISchemaObjectTypeVariants to hold a single member.
Yep, I found the culprit: in your v2 proposal for QAPISchema, you had:
+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
+ def __init__(self, name, typ, flat):
+ QAPISchemaObjectTypeMember.__init__(self, name, typ, False) + assert isinstance(flat, bool) + self.flat = flat + def check(self, schema, tag_type, seen): + QAPISchemaObjectTypeMember.check(self, schema, [], seen) + assert self.name in tag_type.values + if self.flat:
+ self.type.check(schema)
+ assert isinstance(self.type, QAPISchemaObjectType)
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg00394.html
but the 'if self.flat' clause was lost in v3:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00450.html
I am in fact reinstating it here, but for v9, will do it in a separate patch rather than blended in with the rest of the changes.
[wow - we've been hammering at this since July?]
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v8 07/17] qapi: Rework collision assertions
Quoted 21 lines · Eric Blake
Quoted 4 lines · Markus Armbruster
Not checked: variant's members don't collide with non-variant members. I think this check got lost when we simplified QAPISchemaObjectTypeVariants to hold a single member.
Yep, I found the culprit: in your v2 proposal for QAPISchema, you had:
(quoted diff) +11
-0+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): + def __init__(self, name, typ, flat): + QAPISchemaObjectTypeMember.__init__(self, name, typ, False) + assert isinstance(flat, bool) + self.flat = flat + def check(self, schema, tag_type, seen): + QAPISchemaObjectTypeMember.check(self, schema, [], seen) + assert self.name in tag_type.values + if self.flat: + self.type.check(schema) + assert isinstance(self.type, QAPISchemaObjectType)
Quoted 5 lines
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg00394.html
but the 'if self.flat' clause was lost in v3:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00450.html
Quoting v3's change log:
- Lower simple variants to flat ones as described on qapi-code-gen.txt. Replace QAPISchemaObjectType's .flat by .simple_union_type(), for use by pre-existing code-generation warts.
Quoted 2 lines
I am in fact reinstating it here, but for v9, will do it in a separate patch rather than blended in with the rest of the changes.
Any "is this union flat or simple" check signals a flaw. It's either a pointless difference in generated code (these should all be marked TODO by now), or something's wrong with the desugaring of simple to flat unions.
In this case, it looks like a collision check was lost when I switched from "simple unions are its own separate type" to "simple unions are sugar for flat unions".
Reinstating it makes sense, except for the "if self.flat" part. If the check's correct for flat unions, it must be correct for desugared simple unions, or else we screwed up the desugaring.
For a genuine flat union case
'tag-value': 'FlatVariant'
self.type is the object type 'FlatVariant'. Its members are the case's variant members, and they can clash with the flat union's non-variant members.
For a desugared simple union case
'tag-value': 'SimpleVariant'
self.type is an object type with a single member 'data' of type 'SimpleVariant'. Its member 'data' is the case's only variant member, and it can clash with the simple union's non-variant members. In theory only, because the only non-variant member is the implicit tag, and it's called 'type'.
Therefore, the if self.flat is superfluous. Good, because otherwise our desugaring must be flawed.
Quoted 1 line
[wow - we've been hammering at this since July?]
First RFC was in April, but we started hammering in earnest only in summer.
Re: [Qemu-devel] [PATCH v8 07/17] qapi: Rework collision assertions
Quoted 36 lines · Markus Armbruster
Quoted 21 lines · Eric Blake
Quoted 4 lines · Markus Armbruster
Not checked: variant's members don't collide with non-variant members. I think this check got lost when we simplified QAPISchemaObjectTypeVariants to hold a single member.
Yep, I found the culprit: in your v2 proposal for QAPISchema, you had:
(quoted diff) +11
-0+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): + def __init__(self, name, typ, flat): + QAPISchemaObjectTypeMember.__init__(self, name, typ, False) + assert isinstance(flat, bool) + self.flat = flat + def check(self, schema, tag_type, seen): + QAPISchemaObjectTypeMember.check(self, schema, [], seen) + assert self.name in tag_type.values + if self.flat: + self.type.check(schema) + assert isinstance(self.type, QAPISchemaObjectType)Quoted 5 lines
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg00394.html
but the 'if self.flat' clause was lost in v3:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00450.html
Quoting v3's change log:
- Lower simple variants to flat ones as described on qapi-code-gen.txt. Replace QAPISchemaObjectType's .flat by .simple_union_type(), for use by pre-existing code-generation warts.
Quoted 7 lines
Quoted 2 lines
I am in fact reinstating it here, but for v9, will do it in a separate patch rather than blended in with the rest of the changes.
Any "is this union flat or simple" check signals a flaw. It's either a pointless difference in generated code (these should all be marked TODO by now), or something's wrong with the desugaring of simple to flat unions.
Losing 'if self.flat' was correct, but we still need 'if union'; and that's what I add in 2/4.
Quoted 3 lines
Therefore, the if self.flat is superfluous. Good, because otherwise our desugaring must be flawed.
And things correctly work on simple unions due to our wrapper type, so that 'if union' was sufficient.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
cover rework of 7/17
In reply to Markus ArmbrusterBased on the confusion I caused Markus, I've tried to split the v8 patch 7/17 into smaller pieces that are hopefully easier to review in isolation.
Eric Blake (4):
qapi: Drop all_members parameter from check()
qapi: Check for QMP collisions of flat union branches
qapi: Fix check for variant tag values collision
qapi: Consolidate collision detection code
scripts/qapi.py | 62 ++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 39 insertions(+), 23 deletions(-)
Signature · 1 line
2.4.3
PATCH 1/4 qapi: Drop all_members parameter from check()
In reply to Markus ArmbrusterThe implementation of QAPISchemaObjectTypeMember.check() always adds the member currently being checked to both the all_members and seen parameters. However, the three callers of this method pass in the following parameters:
QAPISchemaObjectType.check():
- all_members contains all non-variant members seen to date, for use in populating self.members
- seen contains all non-variant members seen to date, for use in checking for collisions
QAPISchemaObjectTypeVariant.check():
- all_members is a throwaway empty list
- seen is a throwaway dictionary created as a copy by QAPISchemaObjectVariants.check() (since the members of one variant cannot collide with those from another), for use in checking for collisions (technically, we no longer need to check for collisions between tag values and QMP key names, but that's a cleanup for another patch)
QAPISchemaAlternateType.check():
- all_members is a throwaway empty list
- seen is a throwaway empty dict
Therefore, in the one case where we care about all_members after seen has been populated, we know that it contains the same members as seen.values(); changing seen to be an OrderedDict() is sufficient to pick up this information with one less parameter being passed around.
---
v9: new patch, split off from v8 7/17
--- scripts/qapi.py | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-)
scripts/qapi.py +13 -16
@@ -972,28 +972,26 @@ class QAPISchemaObjectType(QAPISchemaType): self.variants = variants self.members = None + # Finish construction, and validate that all members are usable def check(self, schema): assert self.members is not False # not running in cycles if self.members: return self.members = False # mark as being checked + seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) assert isinstance(self.base, QAPISchemaObjectType) assert not self.base.variants # not implemented self.base.check(schema) - members = list(self.base.members) - else: - members = [] - seen = {} - for m in members: - assert c_name(m.name) not in seen - seen[m.name] = m + for m in self.base.members: + assert c_name(m.name) not in seen + seen[m.name] = m for m in self.local_members: - m.check(schema, members, seen) + m.check(schema, seen) if self.variants: - self.variants.check(schema, members, seen) - self.members = members + self.variants.check(schema, seen) + self.members = seen.values() def is_implicit(self): # See QAPISchema._make_implicit_object_type()
@@ -1027,11 +1025,10 @@ class QAPISchemaObjectTypeMember(object): self.type = None self.optional = optional - def check(self, schema, all_members, seen): + def check(self, schema, seen): assert self.name not in seen self.type = schema.lookup_type(self._type_name) assert self.type - all_members.append(self) seen[self.name] = self
@@ -1050,7 +1047,7 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member = tag_member self.variants = variants - def check(self, schema, members, seen): + def check(self, schema, seen): if self.tag_name: # flat union self.tag_member = seen[self.tag_name] assert self.tag_member
@@ -1067,7 +1064,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) def check(self, schema, tag_type, seen): - QAPISchemaObjectTypeMember.check(self, schema, [], seen) + QAPISchemaObjectTypeMember.check(self, schema, seen) assert self.name in tag_type.values # This function exists to support ugly simple union special cases
@@ -1090,8 +1087,8 @@ class QAPISchemaAlternateType(QAPISchemaType): def check(self, schema): seen = {} - self.variants.tag_member.check(schema, [], seen) - self.variants.check(schema, [], seen) + self.variants.tag_member.check(schema, seen) + self.variants.check(schema, seen) def json_type(self): return 'value'
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()
Quoted 3 lines · Eric Blake
The implementation of
QAPISchemaObjectTypeMember.check()always adds the member currently being checked to both theall_membersand seen parameters.
QAPISchemaObjectTypeMember.check() does four things:
1. Compute self.type
Precondition: all types are defined.
2. Accumulate members
all_members serves as accumulator.
We'll see that its only actual use is the owning object type's check(), which uses it to compute self.members.
3. Check for collisions
This works by accumulating names in seen. Precondition: seen contains the names seen so far.
Note that this part uses seen like a set. See 4.
4. Accumulate a map from names to members
seen serves as accumulator.
We'll see that its only actual user is the owning object type's variants.check(), which uses it to compute variants.tag_member from variants.tag_name.
Quoted 8 lines
However, the three callers of this method
pass in the following parameters:
QAPISchemaObjectType.check():
-all_memberscontains all non-variant members seen to date, for use in populating self.members
- seen contains all non-variant members seen to date, for use in checking for collisions
Yes, and:
- we're calling it for m in self.local_members
- before the loop, all_members and seen are initialized to the inherited non-variant members
- after the loop, they therefore contain all non-variant members
This caller uses all four things done by QAPISchemaObjectType.check():
1. Compute m.type
2. Accumulate non-variant members
3. Check for collisions among non-variant members
Before the loop, seen contains the inherited members, which don't collide (self.base.check() ensures that). The loop adds the local members one by one, checking for collisions.
4. Accumulate a map from names to non-variant members Similar argument to 3.
Quoted 1 line
QAPISchemaObjectTypeVariant.check():
Do you mean QAPISchemaObjectVariants.check()?
Quoted 11 lines
-
all_membersis a throwaway empty list
- seen is a throwaway dictionary created as a copy byQAPISchemaObjectVariants.check()(since the members of one variant cannot collide with those from another), for use in checking for collisions (technically, we no longer need to check for collisions between tag values and QMP key names, but that's a cleanup for another patch)
QAPISchemaAlternateType.check():
-all_membersis a throwaway empty list
- seen is a throwaway empty dict
I'm afraid you're omitting a few steps here, and I think you missed QAPISchemaObjectVariants.check()'s self.tag_member.check(). Let me go through the remaining callers of QAPISchemaObjectTypeMember.check() real slow.
* QAPISchemaObjectType.check() calls it via v.check(), which is a thin wrapper that throws away 2. and asserts something of no interest here. seen is a map from names to non-variant members. Therefore, QAPISchemaObjectTypeMember.check() gets used here as follows:
1. Compute v.type
2. Thrown away
3. Check for collision of tag value with non-variant members This is obsolete now.
4. Thrown away
* QAPISchemaAlternateType.check() calls it via v.check(), which is the same thin wrapper. seen is {} here. Therefore:
1. Compute v.type
2. Thrown away
3. No-op
4. Thrown away
* QAPISchemaObjectTypeVariants.check() calls self.tag_member.check(schema, members, seen)
where members and seen contain the non-variant members. However, it gets called only when the owning type is an alternate, and then members and seen are both empty. Therefore:
1. Compute v.type
2. Thrown away
3. No-op
4. Thrown away
Quoted 5 lines
Therefore, in the one case where we care about
all_membersafter seen has been populated, we know that it contains the same members asseen.values(); changing seen to be anOrderedDict()is sufficient to pick up this information with one less parameter being passed around.
I believe the first step should be dropping the obsolete check for collision of tag value with non-variant members. I believe this should do:
@@ -1059,8 +1059,7 @@ class QAPISchemaObjectTypeVariants(object):
self.tag_member.check(schema, members, seen)
assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants:
- vseen = dict(seen)
- v.check(schema, self.tag_member.type, vseen)
+ v.check(schema, self.tag_member.type, {})class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Then only one caller about 2-4., namely QAPISchemaObjectType.check(). Simplify radically: move 2-4. to the caller that cares, drop parameters all_members and seen.
Still to do then: non-variant member collision checking. Factor out
3. into a helper function, use it for non-variant members.
What do you think?
Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()
Quoted 11 lines · Markus Armbruster
Quoted 3 lines · Eric Blake
The implementation of
QAPISchemaObjectTypeMember.check()always adds the member currently being checked to both theall_membersand seen parameters.
QAPISchemaObjectTypeMember.check()does four things:1. Compute self.type
Precondition: all types are defined.
Correct, unchanged by this patch.
Quoted 7 lines
2. Accumulate members
all_membersserves as accumulator.We'll see that its only actual use is the owning object type's
check(), which uses it to compute self.members.
This patch changes it to use seen.values(), which (once you use an OrderedDict() instead of plain {}) is identical to all_members.
Quoted 7 lines
3. Check for collisions
This works by accumulating names in seen. Precondition: seen contains the names seen so far.
Note that this part uses seen like a set. See 4.
Unchanged by this patch; but see also 2/4 and 3/4.
Quoted 5 lines
4. Accumulate a map from names to members
seen serves as accumulator.
Unchanged by this patch.
Quoted 15 lines
We'll see that its only actual user is the owning object type's
variants.check(), which uses it to compute variants.tag_member from variants.tag_name.Quoted 8 lines
However, the three callers of this method
pass in the following parameters:
QAPISchemaObjectType.check():
-all_memberscontains all non-variant members seen to date, for use in populating self.members
- seen contains all non-variant members seen to date, for use in checking for collisionsYes, and:
Quoted 8 lines
- we're calling it for m in self.local_members
- before the loop,all_membersand seen are initialized to the inherited non-variant members
- after the loop, they therefore contain all non-variant membersThis caller uses all four things done by
QAPISchemaObjectType.check():1. Compute m.type
Unchanged by this patch.
Quoted 1 line
2. Accumulate non-variant members
Whether the accumulation is done via all_members (pre-patch) or by seen.values() (post-patch), this step is still done.
Quoted 4 lines
3. Check for collisions among non-variant members
Before the loop, seen contains the inherited members, which don't collide (self.base.check()ensures that). The loop adds the local members one by one, checking for collisions.
Unchanged by this patch.
Quoted 2 lines
4. Accumulate a map from names to non-variant members Similar argument to 3.
Unchanged by this patch.
Quoted 4 lines
Quoted 2 lines
QAPISchemaObjectTypeVariant.check():Do you mean
QAPISchemaObjectVariants.check()?
QAPISchemaObjectTypeVariants.check() calls QAPISchemaObjectTypeVariant.check() for each variant, but with a fresh copy of seen. We'll later need to expand this copy of seen (patch 2/4), but for this patch its use is unchanged - we are appending a single value (the tag value) which is wrong, but no one cares that we appended it because it was a copy. Patch 3/4 fixes to not append to it.
Quoted 9 lines
Quoted 9 lines
-
all_membersis a throwaway empty list
- seen is a throwaway dictionary created as a copy byQAPISchemaObjectVariants.check()(since the members of one variant cannot collide with those from another), for use in checking for collisions (technically, we no longer need to check for collisions between tag values and QMP key names, but that's a cleanup for another patch)
Quoted 6 lines
Quoted 3 lines
QAPISchemaAlternateType.check():
-all_membersis a throwaway empty list
- seen is a throwaway empty dictI'm afraid you're omitting a few steps here, and I think you missed QAPISchemaObjectVariants.check()'s
self.tag_member.check().
There is no self.tag_member.check(), any more; rather, my earlier patches have reworked things so that tag_member is checked by the owner (a flat union's base type, a simple union's local_members, or directly in QAPISchemaAlternateType prior to calling Variants.check()). I guess that's a pitfall of seeing this patch without my rework of 5/17 to address your comments there.
Quoted 11 lines
Quoted 5 lines
Therefore, in the one case where we care about
all_membersafter seen has been populated, we know that it contains the same members asseen.values(); changing seen to be anOrderedDict()is sufficient to pick up this information with one less parameter being passed around.I believe the first step should be dropping the obsolete check for collision of tag value with non-variant members. I believe this should do:
Review class QAPISchemaObjectTypeVariants(object): · line 1062
Quoted 6 lines
(quoted diff) +1
-2@@ -1059,8 +1059,7 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member.check(schema, members, seen) assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: - vseen = dict(seen) - v.check(schema, self.tag_member.type, vseen) + v.check(schema, self.tag_member.type, {})
Close, but not quite. It should do:
+ cases = {}
for v in self.variants:
vseen = dict(seen)
- v.check(schema, self.tag_member.type, vseen)
+ v.check(schema, self.tag_member.type, vseen, cases)
coupled with this in QAPISchemaObjectTypeVariant:
- def check(self, schema, tag_type, seen):
- QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+ def check(self, schema, tag_type, seen, cases):
+ QAPISchemaObjectTypeMember.check(self, schema, [], cases)
so that we are now checking collisions between tag values, rather than cases. But that's what I did in patch 3/4. And we still need seen passed to Variant.check(), because that's the checking added in 2/4.
Okay, you've convinced me - when I post v9, I'll reorder these four patches to put 3/4 first.
Quoted 7 lines
(quoted diff) +0
-0class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):Then only one caller about 2-4., namely
QAPISchemaObjectType.check(). Simplify radically: move 2-4. to the caller that cares, drop parametersall_membersand seen.
Nope - because seen (well, a copy of seen) is still important to patch 2/4.
Quoted 3 lines
Still to do then: non-variant member collision checking. Factor out
3. into a helper function, use it for non-variant members.
Factoring into a helper function is done in 4/4. I can try and rearrange that earlier, too.
Quoted 3 lines
What do you think?
Can you at least look at 2, 3, and 4 to see where I'm headed, and then I can rearrange things for the v9 spin? We're probably talking a bit past each other, with the same end goal, but a muddle in the middle of how to get there.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()
Quoted 115 lines · Eric Blake
Quoted 11 lines · Markus Armbruster
Quoted 3 lines · Eric Blake
The implementation of
QAPISchemaObjectTypeMember.check()always adds the member currently being checked to both theall_membersand seen parameters.
QAPISchemaObjectTypeMember.check()does four things:1. Compute self.type
Precondition: all types are defined.
Correct, unchanged by this patch.
Quoted 7 lines
2. Accumulate members
all_membersserves as accumulator.We'll see that its only actual use is the owning object type's
check(), which uses it to compute self.members.This patch changes it to use
seen.values(), which (once you use anOrderedDict()instead of plain {}) is identical toall_members.Quoted 7 lines
3. Check for collisions
This works by accumulating names in seen. Precondition: seen contains the names seen so far.
Note that this part uses seen like a set. See 4.
Unchanged by this patch; but see also 2/4 and 3/4.
Quoted 5 lines
4. Accumulate a map from names to members
seen serves as accumulator.
Unchanged by this patch.
Quoted 15 lines
We'll see that its only actual user is the owning object type's
variants.check(), which uses it to compute variants.tag_member from variants.tag_name.Quoted 8 lines
However, the three callers of this method
pass in the following parameters:
QAPISchemaObjectType.check():
-all_memberscontains all non-variant members seen to date, for use in populating self.members
- seen contains all non-variant members seen to date, for use in checking for collisionsYes, and:
Quoted 8 lines
- we're calling it for m in self.local_members
- before the loop,all_membersand seen are initialized to the inherited non-variant members
- after the loop, they therefore contain all non-variant membersThis caller uses all four things done by
QAPISchemaObjectType.check():1. Compute m.type
Unchanged by this patch.
Quoted 1 line
2. Accumulate non-variant members
Whether the accumulation is done via
all_members(pre-patch) or byseen.values()(post-patch), this step is still done.Quoted 4 lines
3. Check for collisions among non-variant members
Before the loop, seen contains the inherited members, which don't collide (self.base.check()ensures that). The loop adds the local members one by one, checking for collisions.Unchanged by this patch.
Quoted 2 lines
4. Accumulate a map from names to non-variant members Similar argument to 3.
Unchanged by this patch.
Quoted 4 lines
Quoted 2 lines
QAPISchemaObjectTypeVariant.check():Do you mean
QAPISchemaObjectVariants.check()?
QAPISchemaObjectTypeVariants.check()callsQAPISchemaObjectTypeVariant.check()for each variant, but with a fresh copy of seen. We'll later need to expand this copy of seen (patch 2/4), but for this patch its use is unchanged - we are appending a single value (the tag value) which is wrong, but no one cares that we appended it because it was a copy. Patch 3/4 fixes to not append to it.Quoted 9 lines
Quoted 9 lines
-
all_membersis a throwaway empty list
- seen is a throwaway dictionary created as a copy byQAPISchemaObjectVariants.check()(since the members of one variant cannot collide with those from another), for use in checking for collisions (technically, we no longer need to check for collisions between tag values and QMP key names, but that's a cleanup for another patch)Quoted 6 lines
Quoted 3 lines
QAPISchemaAlternateType.check():
-all_membersis a throwaway empty list
- seen is a throwaway empty dictI'm afraid you're omitting a few steps here, and I think you missed QAPISchemaObjectVariants.check()'s
self.tag_member.check().
Quoted 6 lines
There is no
self.tag_member.check(), any more; rather, my earlier patches have reworked things so thattag_memberis checked by the owner (a flat union's base type, a simple union'slocal_members, or directly in QAPISchemaAlternateType prior to callingVariants.check()). I guess that's a pitfall of seeing this patch without my rework of 5/17 to address your comments there.
I'm assuming this patch is based on
[PATCH v8 06/17] qapi-types: Consolidate gen_struct() and gen_union() which has
def check(self, schema, members, seen):
if self.tag_name: # flat union
self.tag_member = seen[self.tag_name]
elif seen: # simple union
assert self.tag_member in seen.itervalues()
else: # alternate
---> self.tag_member.check(schema, members, seen)
in QAPISchemaObjectTypeVariants.
Quoted 33 lines
Quoted 11 lines
Quoted 5 lines
Therefore, in the one case where we care about
all_membersafter seen has been populated, we know that it contains the same members asseen.values(); changing seen to be anOrderedDict()is sufficient to pick up this information with one less parameter being passed around.I believe the first step should be dropping the obsolete check for collision of tag value with non-variant members. I believe this should do:
Review class QAPISchemaObjectTypeVariants(object): · line 1062
Quoted 6 lines
(quoted diff) +1
-2@@ -1059,8 +1059,7 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member.check(schema, members, seen) assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: - vseen = dict(seen) - v.check(schema, self.tag_member.type, vseen) + v.check(schema, self.tag_member.type, {})Close, but not quite. It should do:
(quoted diff) +2
-1+ cases = {} for v in self.variants: vseen = dict(seen) - v.check(schema, self.tag_member.type, vseen) + v.check(schema, self.tag_member.type, vseen, cases)coupled with this in QAPISchemaObjectTypeVariant:
(quoted diff) +2
-2- def check(self, schema, tag_type, seen): - QAPISchemaObjectTypeMember.check(self, schema, [], seen) + def check(self, schema, tag_type, seen, cases): + QAPISchemaObjectTypeMember.check(self, schema, [], cases)
Quoted 3 lines
so that we are now checking collisions between tag values, rather than cases. But that's what I did in patch 3/4. And we still need seen passed to
Variant.check(), because that's the checking added in 2/4.
A sanity check "no collisions between tag values" doesn't hurt, but I'd simply rely on the tag type's .check(). The tag type is an enumeration type, its check() ensures the enumeration values are distinct. For variants, checking the tag value is a is a value of the tag type suffices to ensure they don't collide.
Quoted 21 lines
Okay, you've convinced me - when I post v9, I'll reorder these four patches to put 3/4 first.
Quoted 7 lines
(quoted diff) +0
-0class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):Then only one caller about 2-4., namely
QAPISchemaObjectType.check(). Simplify radically: move 2-4. to the caller that cares, drop parametersall_membersand seen.Nope - because seen (well, a copy of seen) is still important to patch 2/4.
Quoted 3 lines
Still to do then: non-variant member collision checking. Factor out
3. into a helper function, use it for non-variant members.Factoring into a helper function is done in 4/4. I can try and rearrange that earlier, too.
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
What do you think?
Can you at least look at 2, 3, and 4 to see where I'm headed, and then I can rearrange things for the v9 spin? We're probably talking a bit past each other, with the same end goal, but a muddle in the middle of how to get there.
Yes, we're almost certainly headed in the same direction. But I got thoroughly confused and lost in the details on the way, so I had to hack things up myself to clear my head. The result looks nice to me, so I'll send it out in a jiffie in the hope you'll find it useful.
Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()
Quoted 22 lines · Markus Armbruster
Quoted 9 lines
Quoted 2 lines
I'm afraid you're omitting a few steps here, and I think you missed QAPISchemaObjectVariants.check()'s
self.tag_member.check().There is no
self.tag_member.check(), any more; rather, my earlier patches have reworked things so thattag_memberis checked by the owner (a flat union's base type, a simple union'slocal_members, or directly in QAPISchemaAlternateType prior to callingVariants.check()). I guess that's a pitfall of seeing this patch without my rework of 5/17 to address your comments there.I'm assuming this patch is based on
[PATCH v8 06/17] qapi-types: Consolidategen_struct()andgen_union()which hasdef
check(self, schema, members, seen):
if self.tag_name: # flat union
self.tag_member = seen[self.tag_name]
elif seen: # simple union
assert self.tag_member inseen.itervalues()
else: # alternate
--->self.tag_member.check(schema, members, seen)
Quoted 1 line
in QAPISchemaObjectTypeVariants.
That was true in v8, but not in my pending v9 which did essentially what your 7/7 did (move the tag_member.check() into Alternate.check() back in patch 5).
At any rate, my first glance of your series shows that it is reasonable, so my task today is to spit out a v9 of my series, but using your seven patches in place of my four.
Quoted 5 lines
Yes, we're almost certainly headed in the same direction. But I got thoroughly confused and lost in the details on the way, so I had to hack things up myself to clear my head. The result looks nice to me, so I'll send it out in a jiffie in the hope you'll find it useful.
Yes, it should make v9 easier when it includes your patches, because then you know what to look for :)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()
Quoted 26 lines · Eric Blake
Quoted 22 lines · Markus Armbruster
Quoted 9 lines
Quoted 2 lines
I'm afraid you're omitting a few steps here, and I think you missed QAPISchemaObjectVariants.check()'s
self.tag_member.check().There is no
self.tag_member.check(), any more; rather, my earlier patches have reworked things so thattag_memberis checked by the owner (a flat union's base type, a simple union'slocal_members, or directly in QAPISchemaAlternateType prior to callingVariants.check()). I guess that's a pitfall of seeing this patch without my rework of 5/17 to address your comments there.I'm assuming this patch is based on
[PATCH v8 06/17] qapi-types: Consolidategen_struct()andgen_union()which hasdef
check(self, schema, members, seen):
if self.tag_name: # flat union
self.tag_member = seen[self.tag_name]
elif seen: # simple union
assert self.tag_member inseen.itervalues()
else: # alternate
--->self.tag_member.check(schema, members, seen)Quoted 1 line
in QAPISchemaObjectTypeVariants.
Quoted 7 lines
That was true in v8, but not in my pending v9 which did essentially what your 7/7 did (move the
tag_member.check()intoAlternate.check()back in patch 5).At any rate, my first glance of your series shows that it is reasonable, so my task today is to spit out a v9 of my series, but using your seven patches in place of my four.
Buyer beware: I'm not sure my seven do everything your four do.
Quoted 8 lines
Quoted 5 lines
Yes, we're almost certainly headed in the same direction. But I got thoroughly confused and lost in the details on the way, so I had to hack things up myself to clear my head. The result looks nice to me, so I'll send it out in a jiffie in the hope you'll find it useful.
Yes, it should make v9 easier when it includes your patches, because then you know what to look for :)
I hope I'll be less easily confused going forward...
Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()
Quoted 12 lines · Markus Armbruster
Quoted 12 lines
Quoted 12 lines
I'm assuming this patch is based on
[PATCH v8 06/17] qapi-types: Consolidategen_struct()andgen_union()which hasdef
check(self, schema, members, seen):
if self.tag_name: # flat union
self.tag_member = seen[self.tag_name]
elif seen: # simple union
assert self.tag_member inseen.itervalues()
else: # alternate
--->self.tag_member.check(schema, members, seen)
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
in QAPISchemaObjectTypeVariants.
That was true in v8, but not in my pending v9 which did essentially what your 7/7 did (move the
tag_member.check()intoAlternate.check()back in patch 5).
Caused me some churn in incorporating your patches, but not too bad.
Quoted 6 lines
Quoted 4 lines
At any rate, my first glance of your series shows that it is reasonable, so my task today is to spit out a v9 of my series, but using your seven patches in place of my four.
Buyer beware: I'm not sure my seven do everything your four do.
Yours do the same as my patch 1. You also made a compelling argument for eliminating my patch 3 (the enum values are already collision-proof, therefore if we check that a variant name is in the enum, we don't need to check for collisions) - except that in my 10/17, when I get rid of the generated enum for alternates, I have to reinstate that check somewhere; but adding it directly to QAPISchemaAlternateType.check() at that point feels better.
My patch 2 is still needed, but looks a bit nicer on top of some of your refactoring. And my patch 4 is still useful, as additional refactoring to share the code used by my patch 2 and your code for ObjectType.check().
We'll see how it all turns out in v9.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 1/7 qapi: Drop obsolete tag value collision assertions
In reply to Markus ArmbrusterUnion tag values can't clash with member names in generated C anymore since commit e4ba22b, but QAPISchemaObjectTypeVariants.check() still asserts they don't. Drop it.
--- scripts/qapi.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi.py +1 -2
@@ -1059,8 +1059,7 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member.check(schema, members, seen) assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: - vseen = dict(seen) - v.check(schema, self.tag_member.type, vseen) + v.check(schema, self.tag_member.type, {}) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Signature · 1 line
2.4.3
PATCH 2/7 qapi: Simplify QAPISchemaObjectTypeMember.check()
QAPISchemaObjectTypeMember.check() currently does four things:
1. Compute self.type
2. Accumulate members in all_members
Only one caller cares: QAPISchemaObjectType.check() uses it to compute self.members. The other callers pass a throw-away accumulator.
3. Accumulate a map from names to members in seen
Only one caller cares: QAPISchemaObjectType.check() uses it to compute its local variable seen, for self.variants.check(), which uses it to compute self.variants.tag_member from self.variants.tag_name. The other callers pass a throw-away accumulator.
4. Check for collisions
This piggyback on 3: before adding a new entry, we assert it's new.
Only one caller cares: QAPISchemaObjectType.check() uses it to assert non-variant members don't clash.
Simplify QAPISchemaObjectType.check(): move 2.-4. to QAPISchemaObjectType.check(), and drop parameters all_members and seen.
--- scripts/qapi.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
scripts/qapi.py +7 -7
@@ -990,7 +990,10 @@ class QAPISchemaObjectType(QAPISchemaType): assert c_name(m.name) not in seen seen[m.name] = m for m in self.local_members: - m.check(schema, members, seen) + m.check(schema) + assert m.name not in seen + seen[m.name] = m + members.append(m) if self.variants: self.variants.check(schema, members, seen) self.members = members
@@ -1027,12 +1030,9 @@ class QAPISchemaObjectTypeMember(object): self.type = None self.optional = optional - def check(self, schema, all_members, seen): - assert self.name not in seen + def check(self, schema): self.type = schema.lookup_type(self._type_name) assert self.type - all_members.append(self) - seen[self.name] = self class QAPISchemaObjectTypeVariants(object):
@@ -1056,7 +1056,7 @@ class QAPISchemaObjectTypeVariants(object): elif seen: # simple union assert self.tag_member in seen.itervalues() else: # alternate - self.tag_member.check(schema, members, seen) + self.tag_member.check(schema) assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: v.check(schema, self.tag_member.type, {})
@@ -1067,7 +1067,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) def check(self, schema, tag_type, seen): - QAPISchemaObjectTypeMember.check(self, schema, [], seen) + QAPISchemaObjectTypeMember.check(self, schema) assert self.name in tag_type.values # This function exists to support ugly simple union special cases
Signature · 1 line
2.4.3
PATCH 3/7 qapi: Clean up after previous commit
In reply to Markus ArmbrusterQAPISchemaObjectTypeVariants.check() parameter members and QAPISchemaObjectTypeVariant.check() parameter seen are no longer used, drop them.
--- scripts/qapi.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
scripts/qapi.py +5 -5
@@ -995,7 +995,7 @@ class QAPISchemaObjectType(QAPISchemaType): seen[m.name] = m members.append(m) if self.variants: - self.variants.check(schema, members, seen) + self.variants.check(schema, seen) self.members = members def is_implicit(self):
@@ -1050,7 +1050,7 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member = tag_member self.variants = variants - def check(self, schema, members, seen): + def check(self, schema, seen): if self.tag_name: # flat union self.tag_member = seen[self.tag_name] elif seen: # simple union
@@ -1059,14 +1059,14 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member.check(schema) assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: - v.check(schema, self.tag_member.type, {}) + v.check(schema, self.tag_member.type) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def __init__(self, name, typ): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) - def check(self, schema, tag_type, seen): + def check(self, schema, tag_type): QAPISchemaObjectTypeMember.check(self, schema) assert self.name in tag_type.values
@@ -1089,7 +1089,7 @@ class QAPISchemaAlternateType(QAPISchemaType): self.variants = variants def check(self, schema): - self.variants.check(schema, [], {}) + self.variants.check(schema, {}) def json_type(self): return 'value'
Signature · 1 line
2.4.3
PATCH 4/7 qapi: Fix up commit 7618b91's clash sanity checking change
In reply to Markus ArmbrusterThis hunk
@@ -964,6 +965,7 @@ class QAPISchemaObjectType(QAPISchemaType):
members = []
seen = {}
for m in members:
+ assert c_name(m.name) not in seen
seen[m.name] = m
for m in self.local_members:
m.check(schema, members, seen)
is plainly broken.
Asserting the members inherited from base don't clash is somewhat redundant, because self.base.check() just checked that. But it doesn't hurt.
The idea to use c_name(m.name) instead of m.name for collision checking is sound, because we need to catch clashes between the m.name and between the c_name(m.name), and when two m.name clash, then their c_name() also clash.
However, using c_name(m.name) instead of m.name in one of several places doesn't work. See the very next line.
Keep the assertion, but drop the c_name() for now. A future commit will bring it back.
--- scripts/qapi.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi.py +1 -1
@@ -987,7 +987,7 @@ class QAPISchemaObjectType(QAPISchemaType): members = [] seen = {} for m in members: - assert c_name(m.name) not in seen + assert m.name not in seen seen[m.name] = m for m in self.local_members: m.check(schema)
Signature · 1 line
2.4.3
PATCH 5/7 qapi: Eliminate QAPISchemaObjectType.check() variable members
In reply to Markus ArmbrusterWe can use seen.values() instead if we make it an OrderedDict.
--- scripts/qapi.py | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-)
scripts/qapi.py +5 -9
@@ -977,26 +977,22 @@ class QAPISchemaObjectType(QAPISchemaType): if self.members: return self.members = False # mark as being checked + seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) assert isinstance(self.base, QAPISchemaObjectType) assert not self.base.variants # not implemented self.base.check(schema) - members = list(self.base.members) - else: - members = [] - seen = {} - for m in members: - assert m.name not in seen - seen[m.name] = m + for m in self.base.members: + assert m.name not in seen + seen[m.name] = m for m in self.local_members: m.check(schema) assert m.name not in seen seen[m.name] = m - members.append(m) if self.variants: self.variants.check(schema, seen) - self.members = members + self.members = seen.values() def is_implicit(self): # See QAPISchema._make_implicit_object_type()
Signature · 1 line
2.4.3
PATCH 6/7 qapi: Factor out QAPISchemaObjectTypeMember.check_clash()
In reply to Markus ArmbrusterWhile there, stick in a TODO change key of seen from QAPI name to C name. Can't do it right away, because it would fail the assertion for tests/qapi-schema/args-has-clash.json.
--- scripts/qapi.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
scripts/qapi.py +7 -4
@@ -984,12 +984,10 @@ class QAPISchemaObjectType(QAPISchemaType): assert not self.base.variants # not implemented self.base.check(schema) for m in self.base.members: - assert m.name not in seen - seen[m.name] = m + m.check_clash(seen) for m in self.local_members: m.check(schema) - assert m.name not in seen - seen[m.name] = m + m.check_clash(seen) if self.variants: self.variants.check(schema, seen) self.members = seen.values()
@@ -1030,6 +1028,11 @@ class QAPISchemaObjectTypeMember(object): self.type = schema.lookup_type(self._type_name) assert self.type + def check_clash(self, seen): + # TODO change key of seen from QAPI name to C name + assert self.name not in seen + seen[self.name] = self + class QAPISchemaObjectTypeVariants(object): def __init__(self, tag_name, tag_member, variants):
Signature · 1 line
2.4.3
PATCH 7/7 qapi: QAPISchemaObjectTypeVariants.check()
In reply to Markus ArmbrusterAvoid the ugly flat union / simple union / alternate conditional by doing just the essential work here, namely setting self.tag_member. Move the rest to callers.
--- scripts/qapi.py | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
scripts/qapi.py +4 -6
@@ -988,9 +988,10 @@ class QAPISchemaObjectType(QAPISchemaType): for m in self.local_members: m.check(schema) m.check_clash(seen) + self.members = seen.values() if self.variants: self.variants.check(schema, seen) - self.members = seen.values() + assert self.variants.tag_member in self.members def is_implicit(self): # See QAPISchema._make_implicit_object_type()
@@ -1050,12 +1051,8 @@ class QAPISchemaObjectTypeVariants(object): self.variants = variants def check(self, schema, seen): - if self.tag_name: # flat union + if not self.tag_member: # flat union self.tag_member = seen[self.tag_name] - elif seen: # simple union - assert self.tag_member in seen.itervalues() - else: # alternate - self.tag_member.check(schema) assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: v.check(schema, self.tag_member.type)
@@ -1088,6 +1085,7 @@ class QAPISchemaAlternateType(QAPISchemaType): self.variants = variants def check(self, schema): + self.variants.tag_member.check(schema) self.variants.check(schema, {}) def json_type(self):
Signature · 1 line
2.4.3
PATCH 2/4 qapi: Check for QMP collisions of flat union branches
In reply to Markus ArmbrusterRight now, our ad hoc parser ensures that we cannot have a flat union that introduces any QMP member names that would conflict with the non-variant QMP members already present from the union's base type (see flat-union-clash-member.json). However, we want QAPISchema*.check() to make the same check, so we can later reduce some of the ad hoc checks.
Basically, all branches of a flat union must be qapi structs with no variants, at which point those members appear in the same JSON object as all non-variant members. We already have a map 'seen' of all non-variant members passed in to QAPISchemaObjectTypeVariants.check(), which we clone for each particular variant (since the members of one variant do not clash with another); all that is additionally needed is to actually check the each member of the variant type do not add any collisions.
For simple unions, the same code happens to work (however, our synthesized wrapper type has a single member 'data' which will never collide with the one non-variant member 'type').
But for alternates, we do NOT want to check the type members for adding collisions (an alternate has no parent JSON object that is merging in member names, the way a flat union does), so we have to pass in an extra flag to distinguish whether we are working with a union or an alternate. The flag is temporary; a later patch will rework how alternates are laid out by creating a new subclass of QAPISchemaObjectTypeMember, and detecting the use of this subclass for variants.tag_member will serve the same purpose.
Note that an early proposal [1] for what eventually became commit ac88219a had QAPISchemaObjectTypeVariant.check() ensure that the variant type was complete, although it was later removed [2]; the checks added here happen to match what that earlier attempt meant to do.
[1] https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg00394.html
[2] https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00450.html
---
v9: new patch, split off from v8 7/17
--- scripts/qapi.py | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-)
scripts/qapi.py +19 -6
@@ -1047,7 +1047,8 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member = tag_member self.variants = variants - def check(self, schema, seen): + # TODO drop 'union' param once tag_member is sufficient to spot alternates + def check(self, schema, seen, union=True): if self.tag_name: # flat union self.tag_member = seen[self.tag_name] assert self.tag_member
@@ -1055,17 +1056,29 @@ class QAPISchemaObjectTypeVariants(object): assert self.tag_member in seen.itervalues() assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: - vseen = dict(seen) - v.check(schema, self.tag_member.type, vseen) + # Reset seen array for each variant, since QMP names from one + # branch do not affect another branch + v.check(schema, self.tag_member.type, dict(seen), union) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def __init__(self, name, typ): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) - def check(self, schema, tag_type, seen): - QAPISchemaObjectTypeMember.check(self, schema, seen) + # TODO drop 'union' param once tag_type is sufficient to spot alternates + def check(self, schema, tag_type, seen, union): + QAPISchemaObjectTypeMember.check(self, schema, dict(seen)) assert self.name in tag_type.values + if union: + # If this variant is used within a union, then each member + # field must avoid collisions with the non-variant members + # already present in the union. + assert isinstance(self.type, QAPISchemaObjectType) + assert not self.type.variants # not implemented + self.type.check(schema) + for m in self.type.members: + assert c_name(m.name) not in seen + seen[m.name] = m # This function exists to support ugly simple union special cases # TODO get rid of them, and drop the function
@@ -1088,7 +1101,7 @@ class QAPISchemaAlternateType(QAPISchemaType): def check(self, schema): seen = {} self.variants.tag_member.check(schema, seen) - self.variants.check(schema, seen) + self.variants.check(schema, seen, False) def json_type(self): return 'value'
Signature · 1 line
2.4.3
PATCH 3/4 qapi: Fix check for variant tag values collision
In reply to Markus ArmbrusterNow that commit e4ba22b3 has separated the C representation of qapi unions so that tag values no longer collide with non-variant members, we must adjust QAPISchemaObjectTypeVariant.check() to match. The fix is conceptually simple - track a separate dictionary of tag names we have seen so far, different from the dictionary of non-variant names. And while the non-variant seen array gets reset for each new variant (because the JSON object does not have collisions between separate branches), the map of tag names is not reset.
---
v9: new patch, split off from v8 7/17
--- scripts/qapi.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
scripts/qapi.py +4 -3
@@ -1055,10 +1055,11 @@ class QAPISchemaObjectTypeVariants(object): else: # simple union or alternate assert self.tag_member in seen.itervalues() assert isinstance(self.tag_member.type, QAPISchemaEnumType) + cases = {} for v in self.variants: # Reset seen array for each variant, since QMP names from one # branch do not affect another branch - v.check(schema, self.tag_member.type, dict(seen), union) + v.check(schema, self.tag_member.type, dict(seen), cases, union) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
@@ -1066,8 +1067,8 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) # TODO drop 'union' param once tag_type is sufficient to spot alternates - def check(self, schema, tag_type, seen, union): - QAPISchemaObjectTypeMember.check(self, schema, dict(seen)) + def check(self, schema, tag_type, seen, cases, union): + QAPISchemaObjectTypeMember.check(self, schema, cases) assert self.name in tag_type.values if union: # If this variant is used within a union, then each member
Signature · 1 line
2.4.3
PATCH 4/4 qapi: Consolidate collision detection code
In reply to Markus ArmbrusterRather than having three separate places populate the seen map, it is easier to just factor out a subset of Member.check() that does this as a new method check_collision(), and have the remaining places in ObjectType.check() and Variant.check() call into it. This likewise means a new helper method ObjectType.check_collision(). Later patches can then change the handling of the seen array in just one place, when moving away from ad hoc parser tests.
Note that there was some discrepancy in the existing code on whether name or c_name(name) could not previously be in the seen map; a future patch will clean this up to consistently populate the map via c_name().
---
v9: new patch, split off from v8 7/17; name change from ObjectType.check_qmp() to check_collision(), and new method Member.check_collision(). I'm open to naming suggestions.
--- scripts/qapi.py | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-)
scripts/qapi.py +18 -13
@@ -981,18 +981,21 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType) - assert not self.base.variants # not implemented - self.base.check(schema) - for m in self.base.members: - assert c_name(m.name) not in seen - seen[m.name] = m + self.base.check_collision(schema, seen) for m in self.local_members: m.check(schema, seen) if self.variants: self.variants.check(schema, seen) self.members = seen.values() + # Check that the members of this type do not cause duplicate JSON fields, + # and update seen to track the members seen so far + def check_collision(self, schema, seen): + assert not self.variants # not implemented + self.check(schema) + for m in self.members: + m.check_collision(seen) + def is_implicit(self): # See QAPISchema._make_implicit_object_type() return self.name[0] == ':'
@@ -1026,9 +1029,16 @@ class QAPISchemaObjectTypeMember(object): self.optional = optional def check(self, schema, seen): - assert self.name not in seen self.type = schema.lookup_type(self._type_name) + self.check_collision(seen) + + # Check that this member does not collide with anything in seen (the + # set of non-variant members when called from QAPISchemaObjectType, + # or the set of tag values when called from QAPISchemaObjectTypeVariant), + # and update seen accordingly. + def check_collision(self, seen): assert self.type + assert self.name not in seen seen[self.name] = self
@@ -1074,12 +1084,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): # If this variant is used within a union, then each member # field must avoid collisions with the non-variant members # already present in the union. - assert isinstance(self.type, QAPISchemaObjectType) - assert not self.type.variants # not implemented - self.type.check(schema) - for m in self.type.members: - assert c_name(m.name) not in seen - seen[m.name] = m + self.type.check_collision(schema, seen) # This function exists to support ugly simple union special cases # TODO get rid of them, and drop the function
Signature · 1 line
2.4.3
PATCH v8 8/17 qapi: Remove outdated tests related to QMP/branch collisions
In reply to Eric BlakeNow that branches are in a separate C namespace, we can remove the restrictions in the parser that claim a branch name would collide with QMP, and delete the negative tests that are no longer problematic. A separate patch can then add positive tests to qapi-schema-test to test that any corner cases will compile correctly.
This reverts the scripts/qapi.py portion of commit 7b2a5c2, now that the assertions that it plugged are no longer possible.
---
v8: retitle and update commit message, finish reversion of 7b2a5c2
v7: new patch (also temporarily appeared in subset B v10)
--- scripts/qapi.py | 11 ++--------- tests/Makefile | 3 --- tests/qapi-schema/flat-union-clash-branch.err | 0 tests/qapi-schema/flat-union-clash-branch.exit | 1 - tests/qapi-schema/flat-union-clash-branch.json | 18 ------------------ tests/qapi-schema/flat-union-clash-branch.out | 14 -------------- tests/qapi-schema/flat-union-clash-type.err | 1 - tests/qapi-schema/flat-union-clash-type.exit | 1 - tests/qapi-schema/flat-union-clash-type.json | 14 -------------- tests/qapi-schema/flat-union-clash-type.out | 0 tests/qapi-schema/union-clash-type.err | 1 - tests/qapi-schema/union-clash-type.exit | 1 - tests/qapi-schema/union-clash-type.json | 9 --------- tests/qapi-schema/union-clash-type.out | 0 14 files changed, 2 insertions(+), 72 deletions(-) delete mode 100644 tests/qapi-schema/flat-union-clash-branch.err delete mode 100644 tests/qapi-schema/flat-union-clash-branch.exit delete mode 100644 tests/qapi-schema/flat-union-clash-branch.json delete mode 100644 tests/qapi-schema/flat-union-clash-branch.out delete mode 100644 tests/qapi-schema/flat-union-clash-type.err delete mode 100644 tests/qapi-schema/flat-union-clash-type.exit delete mode 100644 tests/qapi-schema/flat-union-clash-type.json delete mode 100644 tests/qapi-schema/flat-union-clash-type.out delete mode 100644 tests/qapi-schema/union-clash-type.err delete mode 100644 tests/qapi-schema/union-clash-type.exit delete mode 100644 tests/qapi-schema/union-clash-type.json delete mode 100644 tests/qapi-schema/union-clash-type.out
scripts/qapi.py +2 -9
@@ -548,8 +548,7 @@ def check_union(expr, expr_info): base = expr.get('base') discriminator = expr.get('discriminator') members = expr['data'] - values = {'MAX': '(automatic)', 'KIND': '(automatic)', - 'TYPE': '(automatic)'} + values = {'MAX': '(automatic)'} # Two types of unions, determined by discriminator.
@@ -607,19 +606,13 @@ def check_union(expr, expr_info): " of branch '%s'" % key) # If the discriminator names an enum type, then all members - # of 'data' must also be members of the enum type, which in turn - # must not collide with the discriminator name. + # of 'data' must also be members of the enum type. if enum_define: if key not in enum_define['enum_values']: raise QAPIExprError(expr_info, "Discriminator value '%s' is not found in " "enum '%s'" % (key, enum_define["enum_name"])) - if discriminator in enum_define['enum_values']: - raise QAPIExprError(expr_info, - "Discriminator name '%s' collides with " - "enum value in '%s'" % - (discriminator, enum_define["enum_name"])) # Otherwise, check for conflicts in the generated enum else:
tests/Makefile +0 -3
@@ -277,9 +277,7 @@ qapi-schema += flat-union-bad-base.json qapi-schema += flat-union-bad-discriminator.json qapi-schema += flat-union-base-any.json qapi-schema += flat-union-base-union.json -qapi-schema += flat-union-clash-branch.json qapi-schema += flat-union-clash-member.json -qapi-schema += flat-union-clash-type.json qapi-schema += flat-union-empty.json qapi-schema += flat-union-inline.json qapi-schema += flat-union-int-branch.json
@@ -341,7 +339,6 @@ qapi-schema += union-bad-branch.json qapi-schema += union-base-no-discriminator.json qapi-schema += union-clash-branches.json qapi-schema += union-clash-data.json -qapi-schema += union-clash-type.json qapi-schema += union-empty.json qapi-schema += union-invalid-base.json qapi-schema += union-max.json
tests/qapi-schema/flat-union-clash-branch.err deleted +0 -0
tests/qapi-schema/flat-union-clash-branch.exit deleted +0 -1
@@ -1 +0,0 @@ -0
tests/qapi-schema/flat-union-clash-branch.json deleted +0 -18
@@ -1,18 +0,0 @@ -# Flat union branch name collision -# FIXME: this parses, but then fails to compile due to a duplicate 'c_d' -# (one from the base member, the other from the branch name). We should -# either reject the collision at parse time, or munge the generated branch -# name to allow this to compile. -{ 'enum': 'TestEnum', - 'data': [ 'base', 'c-d' ] } -{ 'struct': 'Base', - 'data': { 'enum1': 'TestEnum', '*c_d': 'str' } } -{ 'struct': 'Branch1', - 'data': { 'string': 'str' } } -{ 'struct': 'Branch2', - 'data': { 'value': 'int' } } -{ 'union': 'TestUnion', - 'base': 'Base', - 'discriminator': 'enum1', - 'data': { 'base': 'Branch1', - 'c-d': 'Branch2' } }
tests/qapi-schema/flat-union-clash-branch.out deleted +0 -14
@@ -1,14 +0,0 @@ -object :empty -object Base - member enum1: TestEnum optional=False - member c_d: str optional=True -object Branch1 - member string: str optional=False -object Branch2 - member value: int optional=False -enum TestEnum ['base', 'c-d'] -object TestUnion - base Base - tag enum1 - case base: Branch1 - case c-d: Branch2
tests/qapi-schema/flat-union-clash-type.err deleted +0 -1
@@ -1 +0,0 @@ -tests/qapi-schema/flat-union-clash-type.json:11: Discriminator name 'type' collides with enum value in 'TestEnum'
tests/qapi-schema/flat-union-clash-type.exit deleted +0 -1
@@ -1 +0,0 @@ -1
tests/qapi-schema/flat-union-clash-type.json deleted +0 -14
@@ -1,14 +0,0 @@ -# Flat union branch 'type' -# Reject this, because we would have a clash in generated C, between the -# outer tag 'type' and the branch name 'type' within the union. -# TODO: We could munge the generated C branch name to let it compile. -{ 'enum': 'TestEnum', - 'data': [ 'type' ] } -{ 'struct': 'Base', - 'data': { 'type': 'TestEnum' } } -{ 'struct': 'Branch1', - 'data': { 'string': 'str' } } -{ 'union': 'TestUnion', - 'base': 'Base', - 'discriminator': 'type', - 'data': { 'type': 'Branch1' } }
tests/qapi-schema/flat-union-clash-type.out deleted +0 -0
tests/qapi-schema/union-clash-type.err deleted +0 -1
@@ -1 +0,0 @@ -tests/qapi-schema/union-clash-type.json:8: Union 'TestUnion' member 'kind' clashes with '(automatic)'
tests/qapi-schema/union-clash-type.exit deleted +0 -1
@@ -1 +0,0 @@ -1
tests/qapi-schema/union-clash-type.json deleted +0 -9
@@ -1,9 +0,0 @@ -# Union branch 'type' -# Reject this, because we would have a clash in generated C, between the -# simple union's implicit tag member 'kind' and the branch name 'kind' -# within the union. -# TODO: Even when the generated C is switched to use 'type' rather than -# 'kind', to match the QMP spelling, the collision should still be detected. -# Or, we could munge the branch name to allow compilation. -{ 'union': 'TestUnion', - 'data': { 'kind': 'int', 'type': 'str' } }
tests/qapi-schema/union-clash-type.out deleted +0 -0
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v8 08/17] qapi: Remove outdated tests related to QMP/branch collisions
Quoted 11 lines · Eric Blake
Now that branches are in a separate C namespace, we can remove the restrictions in the parser that claim a branch name would collide with QMP, and delete the negative tests that are no longer problematic. A separate patch can then add positive tests to qapi-schema-test to test that any corner cases will compile correctly.
This reverts the
scripts/qapi.pyportion of commit 7b2a5c2, now that the assertions that it plugged are no longer possible.Signed-off-by Eric Blake <[email protected]>
Yay, an easy patch!
PATCH v8 9/17 qapi: Add positive tests to qapi-schema-test
In reply to Eric BlakeAdd positive tests to qapi-schema-test for things that were made possible by recent patches but which caused compile errors due to collisions prior to that point.
This includes:
Use of a member 'base' in a struct with a base class Use of a member name ending in 'Kind' or 'List' Use of a type name starting with 'has_'
Use of a type named 'u'
Use of a union branch name of 'u'
Use of a union branch name starting with 'has_' Use of a union branch name of 'type'
---
v8: new, but collects portions of subset B v10 patches 2, 3, and 16 and subset C v7 patch 6 that were deferred to later.
It might be worth dropping or simplifying this patch, depending on how many corner cases we actually want to test.
--- tests/qapi-schema/qapi-schema-test.json | 20 ++++++++++++++++++++ tests/qapi-schema/qapi-schema-test.out | 31 +++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+)
tests/qapi-schema/qapi-schema-test.json +20 -0
@@ -47,6 +47,10 @@ 'data': { 'string0': 'str', 'dict1': 'UserDefTwoDict' } } +# ensure that we don't have an artificial collision on 'base' +{ 'struct': 'UserDefThree', + 'base': 'UserDefOne', 'data': { 'base': 'str' } } + # dummy struct to force generation of array types not otherwise mentioned { 'struct': 'ForceArrays', 'data': { 'unused1':['UserDefOne'], 'unused2':['UserDefTwo'],
@@ -113,6 +117,22 @@ 'sizes': ['size'], 'any': ['any'] } } +# Even though 'u' and 'has_*' are forbidden as struct member names, they +# should still be valid as a type or union branch name. And although +# '*Kind' and '*List' are forbidden as type names, they should not be +# forbidden as a member or branch name. Flat union branches do not +# collide with base members. +{ 'enum': 'EnumName', 'data': [ 'value1', 'has_a', 'u', 'type' ] } +{ 'struct': 'has_a', 'data': { 'MyKind': 'int', 'MyList': ['int'], + 'value1': 'EnumName' } } +{ 'union': 'u', 'data': { 'u': 'uint8', 'myKind': 'has_a', + 'myList': 'has_a', 'has_a': 'has_a' } } +{ 'union': 'UnionName', 'base': 'has_a', 'discriminator': 'value1', + 'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero', + 'u': 'UserDefZero', 'type': 'UserDefZero' } } +{ 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool', + 'myKind': 'has_a' } } + # testing commands { 'command': 'user_def_cmd', 'data': {} } { 'command': 'user_def_cmd1', 'data': {'ud1a': 'UserDefOne'} }
tests/qapi-schema/qapi-schema-test.out +31 -0
@@ -22,6 +22,8 @@ object :obj-guest-get-time-arg member b: int optional=True object :obj-guest-sync-arg member arg: any optional=False +object :obj-has_a-wrapper + member data: has_a optional=False object :obj-int16List-wrapper member data: int16List optional=False object :obj-int32List-wrapper
@@ -46,6 +48,8 @@ object :obj-uint32List-wrapper member data: uint32List optional=False object :obj-uint64List-wrapper member data: uint64List optional=False +object :obj-uint8-wrapper + member data: uint8 optional=False object :obj-uint8List-wrapper member data: uint8List optional=False object :obj-user_def_cmd1-arg
@@ -57,6 +61,11 @@ alternate AltIntNum case i: int case n: number enum AltIntNumKind ['i', 'n'] +alternate AltName + case type: int + case u: bool + case myKind: has_a +enum AltNameKind ['type', 'u', 'myKind'] alternate AltNumInt case n: number case i: int
@@ -84,6 +93,7 @@ event EVENT_D :obj-EVENT_D-arg object Empty1 object Empty2 base Empty1 +enum EnumName ['value1', 'has_a', 'u', 'type'] enum EnumOne ['value1', 'value2', 'value3'] object EventStructOne member struct1: UserDefOne optional=False
@@ -105,6 +115,13 @@ object TestStruct member integer: int optional=False member boolean: bool optional=False member string: str optional=False +object UnionName + base has_a + tag value1 + case value1: UserDefZero + case has_a: UserDefZero + case u: UserDefZero + case type: UserDefZero object UserDefA member boolean: bool optional=False member a_b: int optional=True
@@ -158,6 +175,9 @@ object UserDefOptions member u16: uint16List optional=True member i64x: int optional=True member u64x: uint64 optional=True +object UserDefThree + base UserDefOne + member base: str optional=False object UserDefTwo member string0: str optional=False member dict1: UserDefTwoDict optional=False
@@ -201,6 +221,17 @@ command guest-get-time :obj-guest-get-time-arg -> int gen=True success_response=True command guest-sync :obj-guest-sync-arg -> any gen=True success_response=True +object has_a + member MyKind: int optional=False + member MyList: intList optional=False + member value1: EnumName optional=False +object u + member type: uKind optional=False + case u: :obj-uint8-wrapper + case myKind: :obj-has_a-wrapper + case myList: :obj-has_a-wrapper + case has_a: :obj-has_a-wrapper +enum uKind ['u', 'myKind', 'myList', 'has_a'] command user_def_cmd None -> None gen=True success_response=True command user_def_cmd1 :obj-user_def_cmd1-arg -> None
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v8 09/17] qapi: Add positive tests to qapi-schema-test
Quoted 13 lines · Eric Blake
Add positive tests to qapi-schema-test for things that were made possible by recent patches but which caused compile errors due to collisions prior to that point.
This includes:
Use of a member 'base' in a struct with a base class Use of a member name ending in 'Kind' or 'List' Use of a type name starting with 'has_'
Use of a type named 'u'
Use of a union branch name of 'u'
Use of a union branch name starting with 'has_' Use of a union branch name of 'type'
Quoted 8 lines
Signed-off-by Eric Blake <[email protected]>---
v8: new, but collects portions of subset B v10 patches 2, 3, and 16 and subset C v7 patch 6 that were deferred to later.It might be worth dropping or simplifying this patch, depending on how many corner cases we actually want to test.
Let's go through your list:
* Use of a member 'base' in a struct with a base class
'base' is no longer special, and testing it is as useful as testing any other non-special member name. I'd drop it. Or am I missing something?
* Use of a member name ending in 'Kind' or 'List'
These aren't special as member names, but they are reserved type names. The test ensures we don't accidentally reserve them as member names. Low probability * low damage = very low risk. But since you wrote the test already, we can just as well keep it.
* Use of a type name starting with 'has_'
* Use of a type named 'u'
* Use of a union branch name of 'u'
* Use of a union branch name starting with 'has_'
Similarly: these are only reserved as member names, very low risk, but why not keep the test.
* Use of a union branch name of 'type'
As far as I know, 'type' isn't reserved anywhere, it's just the name of a simple union's implicit tag member. Testing a tag value doesn't clash with 'type' is as useful as testing any other non-variant member name. I'd drop it. Or am I missing something?
Anything else being tested here?
Re: [Qemu-devel] [PATCH v8 09/17] qapi: Add positive tests to qapi-schema-test
Quoted 24 lines · Markus Armbruster
Quoted 13 lines · Eric Blake
Add positive tests to qapi-schema-test for things that were made possible by recent patches but which caused compile errors due to collisions prior to that point.
This includes:
Use of a member 'base' in a struct with a base class Use of a member name ending in 'Kind' or 'List' Use of a type name starting with 'has_'
Use of a type named 'u'
Use of a union branch name of 'u'
Use of a union branch name starting with 'has_' Use of a union branch name of 'type'Quoted 8 lines
Signed-off-by Eric Blake <[email protected]>---
v8: new, but collects portions of subset B v10 patches 2, 3, and 16 and subset C v7 patch 6 that were deferred to later.It might be worth dropping or simplifying this patch, depending on how many corner cases we actually want to test.
Quoted 7 lines
Let's go through your list:
* Use of a member 'base' in a struct with a base class
'base' is no longer special, and testing it is as useful as testing any other non-special member name. I'd drop it. Or am I missing something?
Broken prior to commit ddf2190, and no reservation exists. I don't mind dropping it, especially since it quite distinct from the other grouping of collision testing.
Quoted 10 lines
* Use of a member name ending in 'Kind' or 'List'
These aren't special as member names, but they are reserved type names. The test ensures we don't accidentally reserve them as member names. Low probability * low damage = very low risk. But since you wrote the test already, we can just as well keep it.
* Use of a type name starting with '
has_'
* Use of a type named 'u'
Never have been broken, but this adds insurance that we don't break it (and that our reservations added earlier aren't too greedy).
Quoted 1 line
* Use of a union branch name of 'u'
Broken prior to commit e4ba22b3; also proves that the reservation in 5359baf is not too greedy.
Quoted 1 line
* Use of a union branch name starting with '
has_'
Broken prior to commit e4ba22b3; also proves that the reservation in 9fb081e is not too greedy.
Quoted 5 lines
Similarly: these are only reserved as member names, very low risk, but why not keep the test.
* Use of a union branch name of 'type'
Broken prior to commit e4ba22b3. Doesn't have a reservation, so...
Quoted 5 lines
As far as I know, 'type' isn't reserved anywhere, it's just the name of a simple union's implicit tag member. Testing a tag value doesn't clash with 'type' is as useful as testing any other non-variant member name. I'd drop it. Or am I missing something?
...I could indeed drop this part of the test, on the same grounds as dropping the 'base' non-collision tests.
Quoted 3 lines
Anything else being tested here?
I think that covers it. There's also one more positive test added in 10/17, when I rework the layout of alternate types (at that point, it is possible to have an alternate branch named 'max' because it no longer collides with a generated enum _MAX value).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v8 10/17 qapi: Simplify visiting of alternate types
In reply to Eric BlakePreviously, working with alternates required two enums, and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a member of FooKind_lookup[], then use FooKind_lookup[] like we do for other union types.
This has a subtle bug: since the values of FooKind_lookup start at zero, all entries of Foo_qtypes that were not explicitly initialized map to the same branch of the union as the first member of the alternate, rather than triggering a failure in visit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).
However, it IS observable in one case: the behavior of an alternate that contains a 'number' member but no 'int' member differs according to whether the 'number' was first in the qapi definition, and when the input being parsed is an integer; this is because the 'number' parser accepts QTYPE_QINT in addition to the expected QTYPE_QFLOAT. A later patch will worry about fixing alternates to parse all inputs that a non-alternate 'number' would accept, for now it is still marked FIXME.
This patch fixes the validation bug by deleting the indirection, and modifying get_next_type() to directly return a qtype code. There is no longer a need to generate an implicit FooKind array associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names); that also means we no longer have a collision with an alternate branch named 'max'. Next, the generated visitor is fixed to properly detect unexpected qtypes in the switch statement. This is done via the use of a new QAPISchemaAlternateTypeTag subclass and the use of a new member.c_type() method when producing qapi-types. The new subtype also allows us to clean up a TODO left in the previous commit.
Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. If that gets too confusing, we could reintroduce FooKind, but initialize it differently than most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A = QTYPE_QDICT,
FOO_KIND_B = QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to use foo->a or foo->b when inspecting foo->type. But without a current client, I didn't see the point of doing it now.
There is a user-visible side effect to this change, but I consider it to be an improvement. Previously, the invalid QMP command:
{"execute":"blockdev-add", "arguments":{"options":
{"driver":"raw", "id":"a", "file":true}}}
failed with:
{"error": {"class": "GenericError",
"desc": "Invalid parameter type for 'file', expected: QDict"}}
Now it fails with:
{"error": {"class": "GenericError",
"desc": "Invalid parameter type for 'file', expected: BlockdevRef"}}---
v8: no change
v7: rebase onto earlier changes, rework how subtype makes things work
v6: rebase onto tag_member subclass, testsuite, gen_err_check(), and info improvements
--- docs/qapi-code-gen.txt | 3 --- include/qapi/visitor-impl.h | 3 ++- include/qapi/visitor.h | 8 +++++- qapi/qapi-visit-core.c | 4 +-- qapi/qmp-input-visitor.c | 4 +-- scripts/qapi-types.py | 36 +-------------------------- scripts/qapi-visit.py | 12 +++++---- scripts/qapi.py | 43 ++++++++++++++++++++++----------- tests/qapi-schema/alternate-empty.out | 1 - tests/qapi-schema/qapi-schema-test.json | 2 +- tests/qapi-schema/qapi-schema-test.out | 10 +------- tests/test-qmp-input-visitor.c | 33 +++++++++++++------------ tests/test-qmp-output-visitor.c | 21 ++++++++++++---- 13 files changed, 86 insertions(+), 94 deletions(-)
docs/qapi-code-gen.txt +0 -3
@@ -383,9 +383,6 @@ where each branch of the union names a QAPI type. For example: 'data': { 'definition': 'BlockdevOptions', 'reference': 'str' } } -Just like for a simple union, an implicit C enum 'NameKind' is created -to enumerate the branches for the alternate 'Name'. - Unlike a union, the discriminator string is never passed on the wire for the Client JSON Protocol. Instead, the value's JSON type serves as an implicit discriminator, which in turn means that an alternate
include/qapi/visitor-impl.h +2 -1
@@ -32,7 +32,8 @@ struct Visitorvoid (*type_enum)(Visitor *v,int *obj,const char *const strings[],const char *kind,const char *name,Error **errp); -void (*get_next_type)(Visitor *v,int *kind,const int *qobjects, +/* May be NULL; most useful for input visitors. */ +void (*get_next_type)(Visitor *v,qtype_code *type,const char *name,Error **errp);void (*type_int)(Visitor *v,int64_t *obj,const char *name,Error **errp);
include/qapi/visitor.h +7 -1
@@ -41,7 +41,13 @@ GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp);void visit_end_list (Visitor *v,Error **errp);void visit_optional (Visitor *v,bool *present,const char *name,Error **errp); -void visit_get_next_type (Visitor *v,int *obj,const int *qtypes, + +/** +* Determine the qtype of the item @name in the current object visit. +* For input visitors, set *@type to the correct qtype of a qapi +* alternate type; for other visitors, leave *@type unchanged. +*/ +void visit_get_next_type (Visitor *v,qtype_code *type,const char *name,Error **errp);void visit_type_enum (Visitor *v,int *obj,const char *const strings[],const char *kind,const char *name,Error **errp);
qapi/qapi-visit-core.c +2 -2
@@ -81,11 +81,11 @@ void visit_optional(Visitor *v, bool *present, const char *name, } } -void visit_get_next_type (Visitor *v,int *obj,const int *qtypes, +void visit_get_next_type (Visitor *v,qtype_code *type,const char *name,Error **errp) {if (v->get_next_type) { - v->get_next_type (v, obj, qtypes, name, errp); + v->get_next_type (v, type, name, errp); } }
qapi/qmp-input-visitor.c +2 -2
@@ -208,7 +208,7 @@ static void qmp_input_end_list(Visitor *v, Error **errp)qmp_input_pop (qiv, errp); } -static void qmp_input_get_next_type (Visitor *v,int *kind,const int *qobjects, +static void qmp_input_get_next_type (Visitor *v,qtype_code *type,const char *name,Error **errp) {QmpInputVisitor *qiv =to_qiv (v);
@@ -218,7 +218,7 @@ static void qmp_input_get_next_type(Visitor *v, int *kind, const int *qobjects,error_setg (errp,QERR_MISSING_PARAMETER , name ? name :"null" );return ; } - *kind = qobjects[qobject_type (qobj)]; + *type =qobject_type (qobj); } static void qmp_input_type_int(Visitor *v, int64_t *obj, const char *name,
scripts/qapi-types.py +1 -35
@@ -47,7 +47,7 @@ def gen_struct_field(member): ret += mcgen(''' %(c_type)s %(c_name)s; ''', - c_type=member.type.c_type(), c_name=c_name(member.name)) + c_type=member.c_type(), c_name=c_name(member.name)) return ret
@@ -111,38 +111,6 @@ static inline %(base)s *qapi_%(c_name)s_base(const %(c_name)s *obj) c_name=c_name(name), base=base.c_name()) -def gen_alternate_qtypes_decl(name): - return mcgen(''' - -extern const int %(c_name)s_qtypes[]; -''', - c_name=c_name(name)) - - -def gen_alternate_qtypes(name, variants): - ret = mcgen(''' - -const int %(c_name)s_qtypes[QTYPE_MAX] = { -''', - c_name=c_name(name)) - - for var in variants.variants: - qtype = var.type.alternate_qtype() - assert qtype - - ret += mcgen(''' - [%(qtype)s] = %(enum_const)s, -''', - qtype=qtype, - enum_const=c_enum_const(variants.tag_member.type.name, - var.name)) - - ret += mcgen(''' -}; -''') - return ret - - def gen_variants(variants): # FIXME: What purpose does data serve, besides preventing a union that # has a branch named 'data'? We use it in qapi-visit.py to decide
@@ -267,9 +235,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_alternate_type(self, name, info, variants): self._fwdecl += gen_fwd_object_or_array(name) - self._fwdefn += gen_alternate_qtypes(name, variants) self.decl += gen_object(name, None, [variants.tag_member], variants) - self.decl += gen_alternate_qtypes_decl(name) self._gen_type_cleanup(name) # If you link code generated from multiple schemata, you want only one
scripts/qapi-visit.py +7 -5
@@ -189,7 +189,7 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error if (err) { goto out; } - visit_get_next_type(v, (int*) &(*obj)->type, %(c_name)s_qtypes, name, &err); + visit_get_next_type(v, &(*obj)->type, name, &err); if (err) { goto out_obj; }
@@ -203,14 +203,14 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, name, &err); break; ''', - case=c_enum_const(variants.tag_member.type.name, - var.name), + case=var.type.alternate_qtype(), c_type=var.type.c_name(), c_name=c_name(var.name)) ret += mcgen(''' default: - abort(); + error_setg(&err, QERR_INVALID_PARAMETER_TYPE, name ? name : "null", + "%(name)s"); } out_obj: error_propagate(errp, err);
@@ -219,7 +219,8 @@ out_obj: out: error_propagate(errp, err); } -''') +''', + name=name) return ret
@@ -425,6 +426,7 @@ fdef.write(mcgen(''' fdecl.write(mcgen(''' #include "qapi/visitor.h" +#include "qapi/qmp/qerror.h" #include "%(prefix)sqapi-types.h" ''',
scripts/qapi.py +29 -14
110 lines not on this page · raw message
tests/qapi-schema/alternate-empty.out +0 -1
@@ -1,4 +1,3 @@ object :empty alternate Alt case i: int -enum AltKind ['i']
tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -131,7 +131,7 @@ 'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero', 'u': 'UserDefZero', 'type': 'UserDefZero' } } { 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool', - 'myKind': 'has_a' } } + 'myKind': 'has_a', 'max': 'str' } } # testing commands { 'command': 'user_def_cmd', 'data': {} }
tests/qapi-schema/qapi-schema-test.out +1 -9
@@ -60,32 +60,26 @@ object :obj-user_def_cmd2-arg alternate AltIntNum case i: int case n: number -enum AltIntNumKind ['i', 'n'] alternate AltName case type: int case u: bool case myKind: has_a -enum AltNameKind ['type', 'u', 'myKind'] + case max: str alternate AltNumInt case n: number case i: int -enum AltNumIntKind ['n', 'i'] alternate AltNumStr case n: number case s: str -enum AltNumStrKind ['n', 's'] alternate AltStrBool case s: str case b: bool -enum AltStrBoolKind ['s', 'b'] alternate AltStrInt case s: str case i: int -enum AltStrIntKind ['s', 'i'] alternate AltStrNum case s: str case n: number -enum AltStrNumKind ['s', 'n'] event EVENT_A None event EVENT_B None event EVENT_C :obj-EVENT_C-arg
@@ -129,7 +123,6 @@ alternate UserDefAlternate case uda: UserDefA case s: str case i: int -enum UserDefAlternateKind ['uda', 's', 'i'] object UserDefB member intb: int optional=False member a-b: bool optional=True
@@ -198,7 +191,6 @@ event __ORG.QEMU_X-EVENT __org.qemu_x-Struct alternate __org.qemu_x-Alt case __org.qemu_x-branch: str case b: __org.qemu_x-Base -enum __org.qemu_x-AltKind ['__org.qemu_x-branch', 'b'] object __org.qemu_x-Base member __org.qemu_x-member1: __org.qemu_x-Enum optional=False enum __org.qemu_x-Enum ['__org.qemu_x-value']
tests/test-qmp-input-visitor.c +18 -15
@@ -343,14 +343,14 @@ static void test_visitor_in_alternate(TestInputVisitorData *data, v =visitor_input_test_init (data,"42" );visit_type_UserDefAlternate (v, &tmp,NULL , &error_abort); - g_assert_cmpint(tmp->type, ==, USER_DEF_ALTERNATE_KIND_I); + g_assert_cmpint(tmp->type, ==, QTYPE_QINT); g_assert_cmpint(tmp->u.i, ==, 42);qapi_free_UserDefAlternate (tmp);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"'string'" );visit_type_UserDefAlternate (v, &tmp,NULL , &error_abort); - g_assert_cmpint(tmp->type, ==, USER_DEF_ALTERNATE_KIND_S); + g_assert_cmpint(tmp->type, ==, QTYPE_QSTRING); g_assert_cmpstr(tmp->u.s, ==, "string");qapi_free_UserDefAlternate (tmp);visitor_input_teardown (data,NULL );
@@ -386,11 +386,10 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,qapi_free_AltStrBool (asb);visitor_input_teardown (data,NULL ); -/* FIXME: Order of alternate should not affect semantics; asn should -* parse the same as ans */ +/* FIXME: integer should parse as number */ v =visitor_input_test_init (data,"42" );visit_type_AltStrNum (v, &asn,NULL , &err); -/* FIXME g_assert_cmpint(asn->type, == ALT_STR_NUM_KIND_N); */ +/* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */ /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */ g_assert (err);error_free (err);
@@ -398,30 +397,34 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,qapi_free_AltStrNum (asn);visitor_input_teardown (data,NULL ); +/* FIXME: integer should parse as number */ v =visitor_input_test_init (data,"42" ); -visit_type_AltNumStr (v, &ans,NULL , &error_abort); - g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N); - g_assert_cmpfloat(ans->u.n, ==, 42); +visit_type_AltNumStr (v, &ans,NULL , &err); +/* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */ +/* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */ +g_assert (err); +error_free (err); + err =NULL ;qapi_free_AltNumStr (ans);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"42" );visit_type_AltStrInt (v, &asi,NULL , &error_abort); - g_assert_cmpint(asi->type, ==, ALT_STR_INT_KIND_I); + g_assert_cmpint(asi->type, ==, QTYPE_QINT); g_assert_cmpint(asi->u.i, ==, 42);qapi_free_AltStrInt (asi);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"42" );visit_type_AltIntNum (v, &ain,NULL , &error_abort); - g_assert_cmpint(ain->type, ==, ALT_INT_NUM_KIND_I); + g_assert_cmpint(ain->type, ==, QTYPE_QINT); g_assert_cmpint(ain->u.i, ==, 42);qapi_free_AltIntNum (ain);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"42" );visit_type_AltNumInt (v, &ani,NULL , &error_abort); - g_assert_cmpint(ani->type, ==, ALT_NUM_INT_KIND_I); + g_assert_cmpint(ani->type, ==, QTYPE_QINT); g_assert_cmpint(ani->u.i, ==, 42);qapi_free_AltNumInt (ani);visitor_input_teardown (data,NULL );
@@ -438,14 +441,14 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data, v =visitor_input_test_init (data,"42.5" );visit_type_AltStrNum (v, &asn,NULL , &error_abort); - g_assert_cmpint(asn->type, ==, ALT_STR_NUM_KIND_N); + g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); g_assert_cmpfloat(asn->u.n, ==, 42.5);qapi_free_AltStrNum (asn);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"42.5" );visit_type_AltNumStr (v, &ans,NULL , &error_abort); - g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N); + g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); g_assert_cmpfloat(ans->u.n, ==, 42.5);qapi_free_AltNumStr (ans);visitor_input_teardown (data,NULL );
@@ -460,14 +463,14 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data, v =visitor_input_test_init (data,"42.5" );visit_type_AltIntNum (v, &ain,NULL , &error_abort); - g_assert_cmpint(ain->type, ==, ALT_INT_NUM_KIND_N); + g_assert_cmpint(ain->type, ==, QTYPE_QFLOAT); g_assert_cmpfloat(ain->u.n, ==, 42.5);qapi_free_AltIntNum (ain);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"42.5" );visit_type_AltNumInt (v, &ani,NULL , &error_abort); - g_assert_cmpint(ani->type, ==, ALT_NUM_INT_KIND_N); + g_assert_cmpint(ani->type, ==, QTYPE_QFLOAT); g_assert_cmpfloat(ani->u.n, ==, 42.5);qapi_free_AltNumInt (ani);visitor_input_teardown (data,NULL );
tests/test-qmp-output-visitor.c +16 -5
@@ -449,20 +449,31 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data, const void *unused) {QObject *arg; -Error *err =NULL ; +UserDefAlternate *tmp; -UserDefAlternate *tmp =g_malloc0 (sizeof (UserDefAlternate)); - tmp->type =USER_DEF_ALTERNATE_KIND_I ; + tmp =g_new0 (UserDefAlternate,1 ); + tmp->type =QTYPE_QINT ; tmp->u.i =42 ; -visit_type_UserDefAlternate (data->ov, &tmp,NULL , &err); -g_assert (err ==NULL ); +visit_type_UserDefAlternate (data->ov, &tmp,NULL , &error_abort); arg =qmp_output_get_qobject (data->qov);g_assert (qobject_type (arg) ==QTYPE_QINT ); g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42);qapi_free_UserDefAlternate (tmp); + + tmp =g_malloc0 (sizeof (UserDefAlternate)); + tmp->type =QTYPE_QSTRING ; + tmp->u.s =g_strdup ("hello" ); + +visit_type_UserDefAlternate (data->ov, &tmp,NULL , &error_abort); + arg =qmp_output_get_qobject (data->qov); + +g_assert (qobject_type (arg) ==QTYPE_QSTRING ); + g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello"); + +qapi_free_UserDefAlternate (tmp); } static void test_visitor_out_empty(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types
Quoted 3 lines · Eric Blake
Previously, working with alternates required two enums, and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a member of FooKind_lookup[], then use
member of FooKind, actually.
Quoted 1 line
FooKind_lookup[] like we do for other union types.
You probably mean FooKind here as well.
Quoted 10 lines
This has a subtle bug: since the values of
FooKind_lookupstart at zero, all entries ofFoo_qtypesthat were not explicitly initialized map to the same branch of the union as the first member of the alternate, rather than triggering a failure invisit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).
Yes, I remember us discussing this bug.
While reading code to double-check your description, I stumbled over this beauty in generated qapi-visit.c:
visit_get_next_type(v, (int*) &(*obj)->type, BlockdevRef_qtypes, name, &err);
This casts enum BlockdevRefKind * to int *, which assumes the compiler represents the enum BlockdevRefKind as int or unsigned. It is free to use any integer type, though. Common mistake of programmers with insufficiently developed wariness of C's subtleties.
visit_get_next_type() passes the fishy int * on to v->get_next_type(). Only implementation is qmp_input_get_next_type(), which uses it so:
*kind = qobjects[qobject_type(qobj)];
Latent death trap.
Does your patch clean this up?
Quoted 11 lines
However, it IS observable in one case: the behavior of an alternate that contains a 'number' member but no 'int' member differs according to whether the 'number' was first in the qapi definition, and when the input being parsed is an integer; this is because the 'number' parser accepts
QTYPE_QINTin addition to the expectedQTYPE_QFLOAT. A later patch will worry about fixing alternates to parse all inputs that a non-alternate 'number' would accept, for now it is still marked FIXME.This patch fixes the validation bug by deleting the indirection, and modifying
get_next_type()to directly return a qtype code.
get_next_type() doesn't return anything. Do you mean "store a qtype code"?
Quoted 1 line
There is no longer a need to generate an implicit FooKind array
FooKind is an enum, not an array.
Quoted 28 lines
associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names); that also means we no longer have a collision with an alternate branch named 'max'. Next, the generated visitor is fixed to properly detect unexpected qtypes in the switch statement. This is done via the use of a new QAPISchemaAlternateTypeTag subclass and the use of a new
member.c_type()method when producing qapi-types. The new subtype also allows us to clean up a TODO left in the previous commit.Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. If that gets too confusing, we could reintroduce FooKind, but initialize it differently than most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A=QTYPE_QDICT,
FOO_KIND_B=QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to usefoo->aorfoo->bwhen inspectingfoo->type. But without a current client, I didn't see the point of doing it now.There is a user-visible side effect to this change, but I consider it to be an improvement. Previously, the invalid QMP command:
{"execute":"blockdev-add", "arguments":{"options":
Quoted 7 lines
{"driver":"raw", "id":"a", "file":true}}}
failed with:
{"error": {"class": "GenericError",
"desc": "Invalid parameter type for 'file', expected: QDict"}}
Now it fails with:
{"error": {"class": "GenericError",
"desc": "Invalid parameter type for 'file', expected: BlockdevRef"}}I wonder how that happens. Perhaps it's obvious in the patch.
QMP introspection isn't affected, because we carefully minimized the information to expose there.
Quoted 58 lines
Signed-off-by Eric Blake <[email protected]>---
v8: no change
v7: rebase onto earlier changes, rework how subtype makes things work
v6: rebase ontotag_membersubclass, testsuite,gen_err_check(), and info improvements--- docs/qapi-code-gen.txt | 3 --- include/qapi/visitor-impl.h | 3 ++- include/qapi/visitor.h | 8 +++++- qapi/qapi-visit-core.c | 4 +-- qapi/qmp-input-visitor.c | 4 +-- scripts/qapi-types.py | 36 +-------------------------- scripts/qapi-visit.py | 12 +++++---- scripts/qapi.py | 43 ++++++++++++++++++++++----------- tests/qapi-schema/alternate-empty.out | 1 - tests/qapi-schema/qapi-schema-test.json | 2 +- tests/qapi-schema/qapi-schema-test.out | 10 +------- tests/test-qmp-input-visitor.c | 33 +++++++++++++------------ tests/test-qmp-output-visitor.c | 21 ++++++++++++---- 13 files changed, 86 insertions(+), 94 deletions(-)docs/qapi-code-gen.txt +0
-3@@ -383,9 +383,6 @@ where each branch of the union names a QAPI type. For example: 'data': { 'definition': 'BlockdevOptions', 'reference': 'str' } } -Just like for a simple union, an implicit C enum 'NameKind' is created -to enumerate the branches for the alternate 'Name'. - Unlike a union, the discriminator string is never passed on the wire for the Client JSON Protocol. Instead, the value's JSON type serves as an implicit discriminator, which in turn means that an alternateinclude/qapi/visitor-impl.h +2
-1@@ -32,7 +32,8 @@ struct Visitorvoid (*type_enum)(Visitor *v,int *obj,const char *const strings[],const char *kind,const char *name,Error **errp); -void (*get_next_type)(Visitor *v,int *kind,const int *qobjects, +/* May be NULL; most useful for input visitors. */ +void (*get_next_type)(Visitor *v,qtype_code *type,const char *name,Error **errp);void (*type_int)(Visitor *v,int64_t *obj,const char *name,Error **errp);include/qapi/visitor.h +0
-0@@ -41,7 +41,13 @@ GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp);void visit_end_list (Visitor *v,Error **errp); void visit_optional(Visitor *v, bool *present, const char *name,
include/qapi/visitor.h GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp); · line 51
Quoted 10 lines
include/qapi/visitor.h +7
-1Error **errp); -void visit_get_next_type (Visitor *v,int *obj,const int *qtypes, + +/** +* Determine the qtype of the item @name in the current object visit. +* For input visitors, set *@type to the correct qtype of a qapi +* alternate type; for other visitors, leave *@type unchanged. +*/ +void visit_get_next_type (Visitor *v,qtype_code *type,const char *name,Error **errp);
Naive question: what makes a visitor an input visitor?
Quoted 47 lines
include/qapi/visitor.h +0
-0void visit_type_enum (Visitor *v,int *obj,const char *const strings[],const char *kind,const char *name,Error **errp);qapi/qapi-visit-core.c +2
-2@@ -81,11 +81,11 @@ void visit_optional(Visitor *v, bool *present, const char *name, } } -void visit_get_next_type (Visitor *v,int *obj,const int *qtypes, +void visit_get_next_type (Visitor *v,qtype_code *type,const char *name,Error **errp) {if (v->get_next_type) { - v->get_next_type (v, obj, qtypes, name, errp); + v->get_next_type (v, type, name, errp); } }qapi/qmp-input-visitor.c +2
-2@@ -208,7 +208,7 @@ static void qmp_input_end_list(Visitor *v, Error **errp)qmp_input_pop (qiv, errp); } -static void qmp_input_get_next_type (Visitor *v,int *kind,const int *qobjects, +static void qmp_input_get_next_type (Visitor *v,qtype_code *type,const char *name,Error **errp) {QmpInputVisitor *qiv =to_qiv (v);@@ -218,7 +218,7 @@ static void qmp_input_get_next_type(Visitor *v, int *kind, const int *qobjects,error_setg (errp,QERR_MISSING_PARAMETER , name ? name :"null" );return ; } - *kind = qobjects[qobject_type (qobj)]; + *type =qobject_type (qobj); } static void qmp_input_type_int(Visitor *v, int64_t *obj, const char *name,scripts/qapi-types.py +0
-0
scripts/qapi-types.py def gen_struct_field(member): · line 50
Quoted 5 lines
scripts/qapi-types.py +1
-1@@ -47,7 +47,7 @@ def gen_struct_field(member): ret += mcgen(''' %(c_type)s %(c_name)s; ''', - c_type=member.type.c_type(), c_name=c_name(member.name)) + c_type=member.c_type(), c_name=c_name(member.name))
This uses the new member.c_type() defined further down in the patch.
No change if member is an instance of QAPISchemaObjectTypeMember.
Quoted 57 lines
scripts/qapi-types.py +0
-34return ret@@ -111,38 +111,6 @@ static inline %(base)s *qapi_%(c_name)s_base(const %(c_name)s *obj) c_name=c_name(name), base=base.c_name()) -def gen_alternate_qtypes_decl(name): - return mcgen(''' - -extern const int %(c_name)s_qtypes[]; -''', - c_name=c_name(name)) - - -def gen_alternate_qtypes(name, variants): - ret = mcgen(''' - -const int %(c_name)s_qtypes[QTYPE_MAX] = { -''', - c_name=c_name(name)) - - for var in variants.variants: - qtype = var.type.alternate_qtype() - assert qtype - - ret += mcgen(''' - [%(qtype)s] = %(enum_const)s, -''', - qtype=qtype, - enum_const=c_enum_const(variants.tag_member.type.name, - var.name)) - - ret += mcgen(''' -}; -''') - return ret - - def gen_variants(variants): # FIXME: What purpose does data serve, besides preventing a union that # has a branch named 'data'? We use it in qapi-visit.py to decide@@ -267,9 +235,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_alternate_type(self, name, info, variants): self._fwdecl += gen_fwd_object_or_array(name) - self._fwdefn += gen_alternate_qtypes(name, variants) self.decl += gen_object(name, None, [variants.tag_member], variants) - self.decl += gen_alternate_qtypes_decl(name) self._gen_type_cleanup(name) # If you link code generated from multiple schemata, you want only onescripts/qapi-visit.py +0
-0
scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 195
Quoted 8 lines
scripts/qapi-visit.py +1
-1@@ -189,7 +189,7 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error if (err) { goto out; } - visit_get_next_type(v, (int*) &(*obj)->type, %(c_name)s_qtypes, name, &err); + visit_get_next_type(v, &(*obj)->type, name, &err); if (err) { goto out_obj; }
Yes, your patch disarms the latent death trap: no more pointer casting.
Quoted 5 lines
scripts/qapi-visit.py +0
-1@@ -203,14 +203,14 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, name, &err); break; ''', - case=c_enum_const(variants.tag_member.type.name,
scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 213
Quoted 10 lines
scripts/qapi-visit.py +3
-2- var.name), + case=var.type.alternate_qtype(), c_type=var.type.c_name(), c_name=c_name(var.name)) ret += mcgen(''' default: - abort(); + error_setg(&err, QERR_INVALID_PARAMETER_TYPE, name ? name : "null", + "%(name)s");
Okay, this is where the new error message comes from.
Before, default is unreachable, because (*obj)->type got erroneously set the enum's first member when none of the alternate's variants matches the qtype.
After, (*obj)->type *is* the qtype, and we do reach default when no variant matches.
How can name be null?
I really need to finish the QERR_ killing job.
Quoted 31 lines
scripts/qapi-visit.py +3
-1} out_obj: error_propagate(errp, err);@@ -219,7 +219,8 @@ out_obj: out: error_propagate(errp, err); } -''') +''', + name=name) return ret@@ -425,6 +426,7 @@ fdef.write(mcgen(''' fdecl.write(mcgen(''' #include "qapi/visitor.h" +#include "qapi/qmp/qerror.h" #include "%(prefix)sqapi-types.h" ''',scripts/qapi.py +1
-1@@ -627,15 +627,15 @@ def check_union(expr, expr_info): def check_alternate(expr, expr_info): name = expr['alternate'] members = expr['data'] - values = {'MAX': '(automatic)'} + values = {}
scripts/qapi.py def check_union(expr, expr_info): · line 638
Quoted 10 lines
scripts/qapi.py +2
-2types_seen = {} # Check every branch for (key, value) in members.items(): check_name(expr_info, "Member of alternate '%s'" % name, key) - # Check for conflicts in the generated enum - c_key = camel_to_upper(key) + # Check for conflicts in the branch names + c_key = c_name(key)
Why c_name()?
Quoted 9 lines
scripts/qapi.py +2
-0if c_key in values: raise QAPIExprError(expr_info, "Alternate '%s' member '%s' clashes with '%s'"@@ -1029,13 +1029,17 @@ class QAPISchemaObjectTypeMember(object): assert self.type seen[self.name] = self + def c_type(self): + return self.type.c_type()
scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1042
Quoted 10 lines
scripts/qapi.py +3
-1+ class QAPISchemaObjectTypeVariants(object): def __init__(self, tag_name, tag_member, variants): # Flat unions pass tag_name but not tag_member. # Simple unions and alternates pass tag_member but not tag_name. # After check(), tag_member is always set, and tag_name remains - # a reliable witness of being used by a flat union. + # a reliable witness of being used by a flat union, and + # tag_member.type being None is a reliable witness of an alternate.
A member without a type? Ugh! I wouldn't dare breaking invariants like that.
Of course, an alternate's tag member still has a type: qtype_code. It's just not declared in the schema. Should it be a built-in type then?
Quoted 16 lines
scripts/qapi.py +1
-2assert bool(tag_member) != bool(tag_name) assert (isinstance(tag_name, str) or isinstance(tag_member, QAPISchemaObjectTypeMember))@@ -1045,8 +1049,7 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member = tag_member self.variants = variants - # TODO drop union once alternates can be distinguished by tag_member - def check(self, schema, seen, union=True): + def check(self, schema, seen): if self.tag_name: # flat union self.tag_member = seen[self.tag_name] assert self.tag_member@@ -1054,25 +1057,25 @@ class QAPISchemaObjectTypeVariants(object): assert self.tag_member in seen.itervalues() else: # alternate
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066
Quoted 10 lines
scripts/qapi.py +3
-2self.tag_member.check(schema, seen) - assert isinstance(self.tag_member.type, QAPISchemaEnumType) + if not isinstance(self.tag_member, QAPISchemaAlternateTypeTag): + assert isinstance(self.tag_member.type, QAPISchemaEnumType) cases = OrderedDict() for v in self.variants: # Reset seen array for each variant, since QMP names from one # branch do not affect another branch, nor add to all_members - v.check(schema, self.tag_member.type, dict(seen), cases, union) + v.check(schema, self.tag_member.type, dict(seen), cases)
I expect some rebase churn around here, so I'm not reviewing closely.
Quoted 8 lines
scripts/qapi.py +0
-2class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def __init__(self, name, typ): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) - # TODO drop union once alternates can be distinguished by tag_type - def check(self, schema, tag_type, seen, cases, union):
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1080
Quoted 10 lines
scripts/qapi.py +3
-2+ def check(self, schema, tag_type, seen, cases): # cases is case names we must not collide with QAPISchemaObjectTypeMember.check(self, schema, cases) - assert self.name in tag_type.values - if union: + if tag_type: # seen is QMP names our members must not collide with + assert self.name in tag_type.values self.type.check_qmp(schema, seen)
My patches move the member name collision checking to QAPISchemaObjectType.check().
I suspect alternate branch name collision checking should similarly move to QAPISchemaAlternateType.check().
Union branch name collision checking is superfluous, because they already get checked by enum member name collision checking.
Quoted 16 lines
scripts/qapi.py +3
-1# This function exists to support ugly simple union special cases@@ -1094,7 +1097,7 @@ class QAPISchemaAlternateType(QAPISchemaType): self.variants = variants def check(self, schema): - self.variants.check(schema, {}, False) + self.variants.check(schema, {}) def json_type(self): return 'value'@@ -1103,6 +1106,18 @@ class QAPISchemaAlternateType(QAPISchemaType): visitor.visit_alternate_type(self.name, self.info, self.variants) +class QAPISchemaAlternateTypeTag(QAPISchemaObjectTypeMember): + def __init__(self):
scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1120
Quoted 10 lines
scripts/qapi.py +10
-0+ QAPISchemaObjectTypeMember.__init__(self, 'type', '', False) + + def check(self, schema, seen): + assert len(seen) == 0 + seen[self.name] = self + + def c_type(self): + return 'qtype_code' + +
This is a hack to work around the lack of a qtype_code type. I suspect creating such a type would be simpler in the end. Safer, too, because it would avoid having members without a type, which scares me.
Quoted 26 lines
scripts/qapi.py +1
-1class QAPISchemaCommand(QAPISchemaEntity): def __init__(self, name, info, arg_type, ret_type, gen, success_response): QAPISchemaEntity.__init__(self, name, info)@@ -1296,7 +1311,7 @@ class QAPISchema(object): data = expr['data'] variants = [self._make_variant(key, value) for (key, value) in data.iteritems()] - tag_member = self._make_implicit_tag(name, info, variants) + tag_member = QAPISchemaAlternateTypeTag() self._def_entity( QAPISchemaAlternateType(name, info, QAPISchemaObjectTypeVariants(None,tests/qapi-schema/alternate-empty.out +0
-1@@ -1,4 +1,3 @@ object :empty alternate Alt case i: int -enum AltKind ['i']tests/qapi-schema/qapi-schema-test.json +0
-0
tests/qapi-schema/qapi-schema-test.json line 134
Quoted 5 lines
tests/qapi-schema/qapi-schema-test.json +1
-1@@ -131,7 +131,7 @@ 'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero', 'u': 'UserDefZero', 'type': 'UserDefZero' } } { 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool', - 'myKind': 'has_a' } } + 'myKind': 'has_a', 'max': 'str' } }
Here, you add the positive test that alternate name 'max' works.
One, not mentioned in the commit message.
Two, the commit message says we may reintroduce FooKind if working with qtype_code turns out to be too confusing. If we ever do that, alternate name 'max' breaks, doesn't it? Shouldn't we keep it reserved then, just in case?
Quoted 97 lines
tests/qapi-schema/qapi-schema-test.json +0
-0# testing commands { 'command': 'user_def_cmd', 'data': {} }tests/qapi-schema/qapi-schema-test.out +1
-9@@ -60,32 +60,26 @@ object :obj-user_def_cmd2-arg alternate AltIntNum case i: int case n: number -enum AltIntNumKind ['i', 'n'] alternate AltName case type: int case u: bool case myKind: has_a -enum AltNameKind ['type', 'u', 'myKind'] + case max: str alternate AltNumInt case n: number case i: int -enum AltNumIntKind ['n', 'i'] alternate AltNumStr case n: number case s: str -enum AltNumStrKind ['n', 's'] alternate AltStrBool case s: str case b: bool -enum AltStrBoolKind ['s', 'b'] alternate AltStrInt case s: str case i: int -enum AltStrIntKind ['s', 'i'] alternate AltStrNum case s: str case n: number -enum AltStrNumKind ['s', 'n'] event EVENT_A None event EVENT_B None event EVENT_C :obj-EVENT_C-arg@@ -129,7 +123,6 @@ alternate UserDefAlternate case uda: UserDefA case s: str case i: int -enum UserDefAlternateKind ['uda', 's', 'i'] object UserDefB member intb: int optional=False member a-b: bool optional=True@@ -198,7 +191,6 @@ event __ORG.QEMU_X-EVENT __org.qemu_x-Struct alternate __org.qemu_x-Alt case __org.qemu_x-branch: str case b: __org.qemu_x-Base -enum __org.qemu_x-AltKind ['__org.qemu_x-branch', 'b'] object __org.qemu_x-Base member __org.qemu_x-member1: __org.qemu_x-Enum optional=False enum __org.qemu_x-Enum ['__org.qemu_x-value']tests/test-qmp-input-visitor.c +5
-5@@ -343,14 +343,14 @@ static void test_visitor_in_alternate(TestInputVisitorData *data, v =visitor_input_test_init (data,"42" );visit_type_UserDefAlternate (v, &tmp,NULL , &error_abort); - g_assert_cmpint(tmp->type, ==, USER_DEF_ALTERNATE_KIND_I); + g_assert_cmpint(tmp->type, ==, QTYPE_QINT); g_assert_cmpint(tmp->u.i, ==, 42);qapi_free_UserDefAlternate (tmp);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"'string'" );visit_type_UserDefAlternate (v, &tmp,NULL , &error_abort); - g_assert_cmpint(tmp->type, ==, USER_DEF_ALTERNATE_KIND_S); + g_assert_cmpint(tmp->type, ==, QTYPE_QSTRING); g_assert_cmpstr(tmp->u.s, ==, "string");qapi_free_UserDefAlternate (tmp);visitor_input_teardown (data,NULL );@@ -386,11 +386,10 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,qapi_free_AltStrBool (asb);visitor_input_teardown (data,NULL ); -/* FIXME: Order of alternate should not affect semantics; asn should -* parse the same as ans */ +/* FIXME: integer should parse as number */ v =visitor_input_test_init (data,"42" );visit_type_AltStrNum (v, &asn,NULL , &err); -/* FIXME g_assert_cmpint(asn->type, == ALT_STR_NUM_KIND_N); */ +/* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */ /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */ g_assert (err);error_free (err);@@ -398,30 +397,34 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,qapi_free_AltStrNum (asn);visitor_input_teardown (data,NULL ); +/* FIXME: integer should parse as number */
tests/test-qmp-input-visitor.c static void test_visitor_in_alternate_number(TestInputVisitorData *data, · line 407
Quoted 10 lines
tests/test-qmp-input-visitor.c +6
-3v =visitor_input_test_init (data,"42" ); -visit_type_AltNumStr (v, &ans,NULL , &error_abort); - g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N); - g_assert_cmpfloat(ans->u.n, ==, 42); +visit_type_AltNumStr (v, &ans,NULL , &err); +/* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */ +/* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */ +g_assert (err); +error_free (err); + err =NULL ;
What's happening here? Whatever it is, the commit message didn't prepare me for it...
Quoted 66 lines
tests/test-qmp-input-visitor.c +7
-7qapi_free_AltNumStr (ans);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"42" );visit_type_AltStrInt (v, &asi,NULL , &error_abort); - g_assert_cmpint(asi->type, ==, ALT_STR_INT_KIND_I); + g_assert_cmpint(asi->type, ==, QTYPE_QINT); g_assert_cmpint(asi->u.i, ==, 42);qapi_free_AltStrInt (asi);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"42" );visit_type_AltIntNum (v, &ain,NULL , &error_abort); - g_assert_cmpint(ain->type, ==, ALT_INT_NUM_KIND_I); + g_assert_cmpint(ain->type, ==, QTYPE_QINT); g_assert_cmpint(ain->u.i, ==, 42);qapi_free_AltIntNum (ain);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"42" );visit_type_AltNumInt (v, &ani,NULL , &error_abort); - g_assert_cmpint(ani->type, ==, ALT_NUM_INT_KIND_I); + g_assert_cmpint(ani->type, ==, QTYPE_QINT); g_assert_cmpint(ani->u.i, ==, 42);qapi_free_AltNumInt (ani);visitor_input_teardown (data,NULL );@@ -438,14 +441,14 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data, v =visitor_input_test_init (data,"42.5" );visit_type_AltStrNum (v, &asn,NULL , &error_abort); - g_assert_cmpint(asn->type, ==, ALT_STR_NUM_KIND_N); + g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); g_assert_cmpfloat(asn->u.n, ==, 42.5);qapi_free_AltStrNum (asn);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"42.5" );visit_type_AltNumStr (v, &ans,NULL , &error_abort); - g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N); + g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); g_assert_cmpfloat(ans->u.n, ==, 42.5);qapi_free_AltNumStr (ans);visitor_input_teardown (data,NULL );@@ -460,14 +463,14 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data, v =visitor_input_test_init (data,"42.5" );visit_type_AltIntNum (v, &ain,NULL , &error_abort); - g_assert_cmpint(ain->type, ==, ALT_INT_NUM_KIND_N); + g_assert_cmpint(ain->type, ==, QTYPE_QFLOAT); g_assert_cmpfloat(ain->u.n, ==, 42.5);qapi_free_AltIntNum (ain);visitor_input_teardown (data,NULL ); v =visitor_input_test_init (data,"42.5" );visit_type_AltNumInt (v, &ani,NULL , &error_abort); - g_assert_cmpint(ani->type, ==, ALT_NUM_INT_KIND_N); + g_assert_cmpint(ani->type, ==, QTYPE_QFLOAT); g_assert_cmpfloat(ani->u.n, ==, 42.5);qapi_free_AltNumInt (ani);visitor_input_teardown (data,NULL );tests/test-qmp-output-visitor.c +0
-0@@ -449,20 +449,31 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data, const void *unused)
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 456
Quoted 10 lines
tests/test-qmp-output-visitor.c +3
-3{QObject *arg; -Error *err =NULL ; +UserDefAlternate *tmp; -UserDefAlternate *tmp =g_malloc0 (sizeof (UserDefAlternate)); - tmp->type =USER_DEF_ALTERNATE_KIND_I ; + tmp =g_new0 (UserDefAlternate,1 ); + tmp->type =QTYPE_QINT ; tmp->u.i =42 ;
Coding style touched up. Okay.
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 458
Quoted 4 lines
tests/test-qmp-output-visitor.c +1
-2-visit_type_UserDefAlternate (data->ov, &tmp,NULL , &err); -g_assert (err ==NULL ); +visit_type_UserDefAlternate (data->ov, &tmp,NULL , &error_abort);
We need to make up our mind whether to use g_assert(err == NULL) or &error_abort in tests. Wholesale conversion could be in order. I like &error_abort, because it's more concise.
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 466
Quoted 8 lines
tests/test-qmp-output-visitor.c +2
-0arg =qmp_output_get_qobject (data->qov);g_assert (qobject_type (arg) ==QTYPE_QINT ); g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42);qapi_free_UserDefAlternate (tmp); + + tmp =g_malloc0 (sizeof (UserDefAlternate));
g_new0(UserDefAlternate, 1), please.
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 476
Quoted 10 lines
tests/test-qmp-output-visitor.c +10
-0+ tmp->type =QTYPE_QSTRING ; + tmp->u.s =g_strdup ("hello" ); + +visit_type_UserDefAlternate (data->ov, &tmp,NULL , &error_abort); + arg =qmp_output_get_qobject (data->qov); + +g_assert (qobject_type (arg) ==QTYPE_QSTRING ); + g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello"); + +qapi_free_UserDefAlternate (tmp);
New test, not mentioned in commit message. Separate patch, perhaps, along with the nearby coding style touch ups?
Quoted 3 lines
tests/test-qmp-output-visitor.c +0
-0} static void test_visitor_out_empty(TestOutputVisitorData *data,
Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types
Quoted 7 lines · Markus Armbruster
Quoted 3 lines · Eric Blake
Previously, working with alternates required two enums, and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a member of FooKind_lookup[], then use
member of FooKind, actually.
Or entry in the FooKind_lookup[] array.
Quoted 4 lines
Quoted 2 lines
FooKind_lookup[] like we do for other union types.
You probably mean FooKind here as well.
I'll play with the wording.
Quoted 24 lines
Quoted 11 lines
This has a subtle bug: since the values of
FooKind_lookupstart at zero, all entries ofFoo_qtypesthat were not explicitly initialized map to the same branch of the union as the first member of the alternate, rather than triggering a failure invisit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).Yes, I remember us discussing this bug.
While reading code to double-check your description, I stumbled over this beauty in generated
qapi-visit.c:visit_get_next_type(v, (int*) &(*obj)->type, BlockdevRef_qtypes, name, &err);This casts enum BlockdevRefKind * to int *, which assumes the compiler represents the enum BlockdevRefKind as int or unsigned. It is free to use any integer type, though. Common mistake of programmers with insufficiently developed wariness of C's subtleties.
Quoted 8 lines
visit_get_next_type()passes the fishy int * on tov->get_next_type(). Only implementation isqmp_input_get_next_type(), which uses it so:*kind = qobjects[qobject_type(qobj)];Latent death trap.
Does your patch clean this up?
Yes, and I need to also document that this is an additional bug fix.
Quoted 9 lines
Quoted 9 lines
However, it IS observable in one case: the behavior of an alternate that contains a 'number' member but no 'int' member differs according to whether the 'number' was first in the qapi definition, and when the input being parsed is an integer; this is because the 'number' parser accepts
QTYPE_QINTin addition to the expectedQTYPE_QFLOAT. A later patch will worry about fixing alternates to parse all inputs that a non-alternate 'number' would accept, for now it is still marked FIXME.
See [1] below.
Quoted 6 lines
Quoted 3 lines
This patch fixes the validation bug by deleting the indirection, and modifying
get_next_type()to directly return a qtype code.
get_next_type()doesn't return anything. Do you mean "store a qtype code"?
Yes.
Quoted 4 lines
Quoted 2 lines
There is no longer a need to generate an implicit FooKind array
FooKind is an enum, not an array.
...to generate an implicit FooKind enum, nor FooKind_lookup[] array.
Quoted 12 lines
Quoted 12 lines
associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names); that also means we no longer have a collision with an alternate branch named 'max'. Next, the generated visitor is fixed to properly detect unexpected qtypes in the switch statement. This is done via the use of a new QAPISchemaAlternateTypeTag subclass and the use of a new
member.c_type()method when producing qapi-types. The new subtype also allows us to clean up a TODO left in the previous commit.
Quoted 12 lines
Quoted 4 lines
Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. If that gets too confusing, we could reintroduce FooKind, but initialize it differently than
Quoted 8 lines
most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A=QTYPE_QDICT,
FOO_KIND_B=QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to usefoo->aorfoo->bwhen inspectingfoo->type. But without a current client, I didn't see the point of doing it now.
You have a point below that we either need to reserve MAX and require no case-insensitive clashes, or that we will never want to add it. I'm leaning towards never going back, because the new way feels so much nicer.
Quoted 13 lines
Quoted 5 lines
There is a user-visible side effect to this change, but I consider it to be an improvement. Previously, the invalid QMP command:
{"execute":"blockdev-add", "arguments":{"options":Quoted 7 lines
{"driver":"raw", "id":"a", "file":true}}} failed with: {"error": {"class": "GenericError", "desc": "Invalid parameter type for 'file', expected: QDict"}} Now it fails with: {"error": {"class": "GenericError", "desc": "Invalid parameter type for 'file', expected: BlockdevRef"}}
Quoted 1 line
I wonder how that happens. Perhaps it's obvious in the patch.
I think you found it below.
Quoted 3 lines
QMP introspection isn't affected, because we carefully minimized the information to expose there.
Ooh, nice tidbit to add.
Quoted 9 lines
include/qapi/visitor.h GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp); · line 51
Quoted 7 lines
include/qapi/visitor.h +6
-0+/** +* Determine the qtype of the item @name in the current object visit. +* For input visitors, set *@type to the correct qtype of a qapi +* alternate type; for other visitors, leave *@type unchanged. +*/ +void visit_get_next_type (Visitor *v,qtype_code *type,const char *name,Error **errp);Naive question: what makes a visitor an input visitor?
I've got a later patch in my queue that adds a lot more documentation:
http://repo.or.cz/qemu/ericb.git/commitdiff/f7674a87e72
+/* This file describes the client view for visiting a map between
+ * generated QAPI C structs and another representation (command line
+ * options, strings, or QObjects). An input visitor converts from
+ * some other form into QAPI representation; an output visitor
+ * converts from QAPI back into another form. In the descriptions
+ * below, an object or dictionary refers to a JSON '{}', and an array
+ * or list refers to a JSON '[]'. These functions seldom need to be
+ * called directly, but are instead used by code generated by
+ * scripts/qapi-visit.py. For the visitor callback contracts, see
+ * visitor-impl.h. */
Quoted 4 lines
Quoted 4 lines
+++
b/scripts/qapi-visit.pyscripts/qapi-visit.py +0
-0@@ -189,7 +189,7 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error if (err) { goto out;
Quoted 8 lines
scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 195
Quoted 6 lines
scripts/qapi-visit.py +1
-1} - visit_get_next_type(v, (int*) &(*obj)->type, %(c_name)s_qtypes, name, &err); + visit_get_next_type(v, &(*obj)->type, name, &err); if (err) { goto out_obj; }Yes, your patch disarms the latent death trap: no more pointer casting.
Indeed, I noticed the cleanup as well (I'm quite familiar with the unsafe nature of casting enum* because you cannot guarantee its size), but failed to call out the trap in my commit message.
Quoted 19 lines
Quoted 6 lines
scripts/qapi-visit.py +0
-1@@ -203,14 +203,14 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, name, &err); break; ''', - case=c_enum_const(variants.tag_member.type.name,scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 213
Quoted 10 lines
scripts/qapi-visit.py +3
-2- var.name), + case=var.type.alternate_qtype(), c_type=var.type.c_name(), c_name=c_name(var.name)) ret += mcgen(''' default: - abort(); + error_setg(&err, QERR_INVALID_PARAMETER_TYPE, name ? name : "null", + "%(name)s");Okay, this is where the new error message comes from.
Quoted 8 lines
Before, default is unreachable, because (*obj)->type got erroneously set the enum's first member when none of the alternate's variants matches the qtype.
After, (*obj)->type *is* the qtype, and we do reach default when no variant matches.
How can name be null?
When you have the qapi representation ['MyAlternate'], you will have qapi_visit_type_MyAlternateList() which passes NULL for the name of each list member (because names are only present for objects, not lists).
Quoted 2 lines
I really need to finish the
QERR_killing job.
Agreed. But shouldn't stall this patch, though.
Quoted 10 lines
scripts/qapi.py def check_union(expr, expr_info): · line 638
Quoted 8 lines
scripts/qapi.py +2
-2# Check every branch for (key, value) in members.items(): check_name(expr_info, "Member of alternate '%s'" % name, key) - # Check for conflicts in the generated enum - c_key = camel_to_upper(key) + # Check for conflicts in the branch names + c_key = c_name(key)Why
c_name()?
So that 'a-b' and 'a_b' are properly flagged as conflicting (they map to the same c_name).
Quoted 9 lines
scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1042
Quoted 8 lines
class QAPISchemaObjectTypeVariants(object):
def__init__(self, tag_name, tag_member, variants):
# Flat unions passtag_namebut nottag_member.
# Simple unions and alternates passtag_memberbut nottag_name. # Aftercheck(),tag_memberis always set, andtag_nameremainsscripts/qapi.py +2
-1- # a reliable witness of being used by a flat union. + # a reliable witness of being used by a flat union, and + # tag_member.type being None is a reliable witness of an alternate.
Quoted 5 lines
A member without a type? Ugh! I wouldn't dare breaking invariants like that.
Of course, an alternate's tag member still has a type:
qtype_code. It's just not declared in the schema. Should it be a built-in type then?
It's not a builtin that can ever be referenced in the .json files. But I could probably come up with something, if it would make you feel better.
Quoted 8 lines
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066
Quoted 5 lines
for v in self.variants:
# Reset seen array for each variant, since QMP names from one # branch do not affect another branch, nor add toall_membersscripts/qapi.py +1
-1- v.check(schema, self.tag_member.type, dict(seen), cases, union) + v.check(schema, self.tag_member.type, dict(seen), cases)I expect some rebase churn around here, so I'm not reviewing closely.
Indeed. All the more reason for me to post a v9 spin (and maybe defer the question of a non-None type for tag_member until after that post).
Quoted 6 lines
My patches move the member name collision checking to
QAPISchemaObjectType.check().I suspect alternate branch name collision checking should similarly move to
QAPISchemaAlternateType.check().
Yep, already that way in my pending v9 series after incorporating your patches.
Quoted 8 lines
Quoted 8 lines
scripts/qapi.py +8
-0+class QAPISchemaAlternateTypeTag(QAPISchemaObjectTypeMember): + def __init__(self): + QAPISchemaObjectTypeMember.__init__(self, 'type', '', False) + + def check(self, schema, seen): + assert len(seen) == 0 + seen[self.name] = self +
Quoted 8 lines
scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1120
Quoted 4 lines
scripts/qapi.py +4
-0+ def c_type(self): + return 'qtype_code' + +This is a hack to work around the lack of a
qtype_codetype. I suspect creating such a type would be simpler in the end. Safer, too, because it would avoid having members without a type, which scares me.
I can play with dropping c_type() here in favor of adding a qtype_code special class, but I may still need to keep this QAPISchemaAlternateTypeTag subclass.
Quoted 11 lines
tests/qapi-schema/qapi-schema-test.json line 134
Quoted 7 lines
+++
b/tests/qapi-schema/qapi-schema-test.jsontests/qapi-schema/qapi-schema-test.json +1
-1@@ -131,7 +131,7 @@ 'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero', 'u': 'UserDefZero', 'type': 'UserDefZero' } } { 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool', - 'myKind': 'has_a' } } + 'myKind': 'has_a', 'max': 'str' } }Here, you add the positive test that alternate name 'max' works.
One, not mentioned in the commit message.
D'oh.
Quoted 6 lines
Two, the commit message says we may reintroduce FooKind if working with
qtype_codeturns out to be too confusing. If we ever do that, alternate name 'max' breaks, doesn't it? Shouldn't we keep it reserved then, just in case?
See my comment above; at this point, I doubt we'll ever want to go back, so maybe I just need to be more definitive in stating that.
Quoted 20 lines
Quoted 20 lines
tests/test-qmp-input-visitor.c +3
-3+++ b/tests/test-qmp-input-visitor.c@@ -386,11 +386,10 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,qapi_free_AltStrBool (asb);visitor_input_teardown (data,NULL ); -/* FIXME: Order of alternate should not affect semantics; asn should -* parse the same as ans */ +/* FIXME: integer should parse as number */ v =visitor_input_test_init (data,"42" );visit_type_AltStrNum (v, &asn,NULL , &err); -/* FIXME g_assert_cmpint(asn->type, == ALT_STR_NUM_KIND_N); */ +/* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */ /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */ g_assert (err);error_free (err);@@ -398,30 +397,34 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,qapi_free_AltStrNum (asn);visitor_input_teardown (data,NULL );
Quoted 14 lines
tests/test-qmp-input-visitor.c static void test_visitor_in_alternate_number(TestInputVisitorData *data, · line 407
Quoted 11 lines
tests/test-qmp-input-visitor.c +7
-3+/* FIXME: integer should parse as number */ v =visitor_input_test_init (data,"42" ); -visit_type_AltNumStr (v, &ans,NULL , &error_abort); - g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N); - g_assert_cmpfloat(ans->u.n, ==, 42); +visit_type_AltNumStr (v, &ans,NULL , &err); +/* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */ +/* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */ +g_assert (err); +error_free (err); + err =NULL ;What's happening here? Whatever it is, the commit message didn't prepare me for it...
See [1] above. 'asn' is now parsing the same as 'ans' (we are no longer sensitive to whether 'number' was the first member of the alternate), but it isn't until patch 11/17 that we fix things that 'ans' and 'asn' both properly parse '1' as a number instead of rejecting it as an integer.
Quoted 17 lines
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 456
Quoted 13 lines
tests/test-qmp-output-visitor.c +4
-3+++ b/tests/test-qmp-output-visitor.c@@ -449,20 +449,31 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data, const void *unused) {QObject *arg; -Error *err =NULL ; +UserDefAlternate *tmp; -UserDefAlternate *tmp =g_malloc0 (sizeof (UserDefAlternate)); - tmp->type =USER_DEF_ALTERNATE_KIND_I ; + tmp =g_new0 (UserDefAlternate,1 ); + tmp->type =QTYPE_QINT ; tmp->u.i =42 ;Coding style touched up. Okay.
Quoted 1 line
Quoted 7 lines
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 458
Quoted 3 lines
tests/test-qmp-output-visitor.c +1
-2-visit_type_UserDefAlternate (data->ov, &tmp,NULL , &err); -g_assert (err ==NULL ); +visit_type_UserDefAlternate (data->ov, &tmp,NULL , &error_abort);We need to make up our mind whether to use
g_assert(err == NULL)or&error_abortin tests. Wholesale conversion could be in order. I like&error_abort, because it's more concise.
Wholesale conversion dead-ahead, in 14/17 of this subset.
Quoted 12 lines
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 466
Quoted 9 lines
arg = qmp_output_get_qobject(data->qov); g_assert(qobject_type(arg) == QTYPE_QINT); g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42); qapi_free_UserDefAlternate(tmp);tests/test-qmp-output-visitor.c +2
-0+ + tmp =g_malloc0 (sizeof (UserDefAlternate));
g_new0(UserDefAlternate, 1), please.
Quoted 13 lines
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 476
Quoted 10 lines
tests/test-qmp-output-visitor.c +10
-0+ tmp->type =QTYPE_QSTRING ; + tmp->u.s =g_strdup ("hello" ); + +visit_type_UserDefAlternate (data->ov, &tmp,NULL , &error_abort); + arg =qmp_output_get_qobject (data->qov); + +g_assert (qobject_type (arg) ==QTYPE_QSTRING ); + g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello"); + +qapi_free_UserDefAlternate (tmp);New test, not mentioned in commit message. Separate patch, perhaps, along with the nearby coding style touch ups?
Yes, I will split this portion of the test changes out to a separate commit.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types
Quoted 10 lines · Eric Blake
Quoted 7 lines · Markus Armbruster
Quoted 3 lines · Eric Blake
Previously, working with alternates required two enums, and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a member of FooKind_lookup[], then use
member of FooKind, actually.
Or entry in the FooKind_lookup[] array.
The generated
const int BlockdevRef_qtypes[QTYPE_MAX] = {
[QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION, [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE,
};
maps from qtype_code to BlockdevRefKind, except it uses int instead of BlockdevRefKind, so it can be passed to visit_get_next_type().
Quoted 61 lines
Quoted 3 lines
Quoted 1 line
FooKind_lookup[] like we do for other union types.
You probably mean FooKind here as well.
I'll play with the wording.
Quoted 25 lines
Quoted 11 lines
This has a subtle bug: since the values of
FooKind_lookupstart at zero, all entries ofFoo_qtypesthat were not explicitly initialized map to the same branch of the union as the first member of the alternate, rather than triggering a failure invisit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).Yes, I remember us discussing this bug.
While reading code to double-check your description, I stumbled over this beauty in generated
qapi-visit.c:visit_get_next_type(v, (int*) &(*obj)->type, BlockdevRef_qtypes,name, &err);
This casts enum BlockdevRefKind * to int *, which assumes the compiler represents the enum BlockdevRefKind as int or unsigned. It is free to use any integer type, though. Common mistake of programmers with insufficiently developed wariness of C's subtleties.
Quoted 8 lines
visit_get_next_type()passes the fishy int * on tov->get_next_type(). Only implementation isqmp_input_get_next_type(), which uses it so:*kind = qobjects[qobject_type(qobj)];Latent death trap.
Does your patch clean this up?
Yes, and I need to also document that this is an additional bug fix.
Quoted 9 lines
Quoted 9 lines
However, it IS observable in one case: the behavior of an alternate that contains a 'number' member but no 'int' member differs according to whether the 'number' was first in the qapi definition, and when the input being parsed is an integer; this is because the 'number' parser accepts
QTYPE_QINTin addition to the expectedQTYPE_QFLOAT. A later patch will worry about fixing alternates to parse all inputs that a non-alternate 'number' would accept, for now it is still marked FIXME.See [1] below.
Quoted 6 lines
Quoted 3 lines
This patch fixes the validation bug by deleting the indirection, and modifying
get_next_type()to directly return a qtype code.
get_next_type()doesn't return anything. Do you mean "store a qtype code"?
Quoted 8 lines
Yes.
Quoted 4 lines
Quoted 2 lines
There is no longer a need to generate an implicit FooKind array
FooKind is an enum, not an array.
...to generate an implicit FooKind enum, nor FooKind_lookup[] array.
Yes.
Quoted 24 lines
Quoted 11 lines
Quoted 11 lines
associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names); that also means we no longer have a collision with an alternate branch named 'max'. Next, the generated visitor is fixed to properly detect unexpected qtypes in the switch statement. This is done via the use of a new QAPISchemaAlternateTypeTag subclass and the use of a new
member.c_type()method when producing qapi-types. The new subtype also allows us to clean up a TODO left in the previous commit.Quoted 12 lines
Quoted 4 lines
Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. If that gets too confusing, we could reintroduce FooKind, but initialize it differently than
Quoted 8 lines
most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A=QTYPE_QDICT,
FOO_KIND_B=QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to usefoo->aorfoo->bwhen inspectingfoo->type. But without a current client, I didn't see the point of doing it now.
Quoted 3 lines
You have a point below that we either need to reserve MAX and require no case-insensitive clashes, or that we will never want to add it. I'm leaning towards never going back, because the new way feels so much nicer.
Perhaps un-reserving MAX and making the collision checking more lenient should be a separate follow-up patch we could easily revert. Probably useful only if this patch doesn't make 2.5.
Quoted 37 lines
Quoted 13 lines
Quoted 5 lines
There is a user-visible side effect to this change, but I consider it to be an improvement. Previously, the invalid QMP command:
{"execute":"blockdev-add", "arguments":{"options":Quoted 7 lines
{"driver":"raw", "id":"a", "file":true}}} failed with: {"error": {"class": "GenericError", "desc": "Invalid parameter type for 'file', expected: QDict"}} Now it fails with: {"error": {"class": "GenericError", "desc": "Invalid parameter type for 'file', expected: BlockdevRef"}}Quoted 1 line
I wonder how that happens. Perhaps it's obvious in the patch.
I think you found it below.
Quoted 3 lines
QMP introspection isn't affected, because we carefully minimized the information to expose there.
Ooh, nice tidbit to add.
Quoted 9 lines
include/qapi/visitor.h GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp); · line 51
Quoted 7 lines
include/qapi/visitor.h +6
-0+/** +* Determine the qtype of the item @name in the current object visit. +* For input visitors, set *@type to the correct qtype of a qapi +* alternate type; for other visitors, leave *@type unchanged. +*/ +void visit_get_next_type (Visitor *v,qtype_code *type,const char *name,Error **errp);Naive question: what makes a visitor an input visitor?
I've got a later patch in my queue that adds a lot more documentation:
http://repo.or.cz/qemu/ericb.git/commitdiff/f7674a87e72
Quoted 10 lines
+/* This file describes the client view for visiting a map between
+ * generated QAPI C structs and another representation (command line
+ * options, strings, or QObjects). An input visitor converts from
+ * some other form into QAPI representation; an output visitor
+ * converts from QAPI back into another form. In the descriptions
+ * below, an object or dictionary refers to a JSON '{}', and an array
+ * or list refers to a JSON '[]'. These functions seldom need to be
+ * called directly, but are instead used by code generated by
+ *scripts/qapi-visit.py. For the visitor callback contracts, see
+ *visitor-impl.h. */
That's a useful step towards a visitors contract.
Quoted 40 lines
Quoted 4 lines
Quoted 4 lines
+++
b/scripts/qapi-visit.pyscripts/qapi-visit.py +0
-0@@ -189,7 +189,7 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error if (err) { goto out;Quoted 8 lines
scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 195
Quoted 6 lines
scripts/qapi-visit.py +1
-1} - visit_get_next_type(v, (int*) &(*obj)->type, %(c_name)s_qtypes, name, &err); + visit_get_next_type(v, &(*obj)->type, name, &err); if (err) { goto out_obj; }Yes, your patch disarms the latent death trap: no more pointer casting.
Indeed, I noticed the cleanup as well (I'm quite familiar with the unsafe nature of casting enum* because you cannot guarantee its size), but failed to call out the trap in my commit message.
Quoted 23 lines
Quoted 6 lines
scripts/qapi-visit.py +0
-1@@ -203,14 +203,14 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, name, &err); break; ''', - case=c_enum_const(variants.tag_member.type.name,scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 213
Quoted 10 lines
scripts/qapi-visit.py +3
-2- var.name), + case=var.type.alternate_qtype(), c_type=var.type.c_name(), c_name=c_name(var.name)) ret += mcgen(''' default: - abort(); + error_setg(&err, QERR_INVALID_PARAMETER_TYPE, name ? name : "null", + "%(name)s");Okay, this is where the new error message comes from.
Before, default is unreachable, because (*obj)->type got erroneously set the enum's first member when none of the alternate's variants matches the qtype.
Quoted 8 lines
Quoted 4 lines
After, (*obj)->type *is* the qtype, and we do reach default when no variant matches.
How can name be null?
When you have the qapi representation ['MyAlternate'], you will have
qapi_visit_type_MyAlternateList()which passes NULL for the name of each list member (because names are only present for objects, not lists).
Examining the occurences of visit_type_BlockdevRef* in generated qapi-visit.c shows that parameter name is commonly the member name. It appears to be used only for error messages.
Works fine when the object containing the member is obvious. With multiple objects, the recipient of the error message has to guess the object containing the member. If the member name isn't unique, the error message is ambiguous. Crap, but not crap we can afford to improve right now.
Array elements are anonymous, and I guess that's what led to NULL. Stupid choice. Make it "array element" or something. Code becomes slightly simpler, error message becomes slightly less cruel. This could be simple enough to improve now.
Quoted 3 lines
Quoted 1 line
I really need to finish the
QERR_killing job.Agreed. But shouldn't stall this patch, though.
Certainly not.
Quoted 73 lines
Quoted 10 lines
scripts/qapi.py def check_union(expr, expr_info): · line 638
Quoted 8 lines
scripts/qapi.py +2
-2# Check every branch for (key, value) in members.items(): check_name(expr_info, "Member of alternate '%s'" % name, key) - # Check for conflicts in the generated enum - c_key = camel_to_upper(key) + # Check for conflicts in the branch names + c_key = c_name(key)Why
c_name()?So that 'a-b' and '
a_b' are properly flagged as conflicting (they map to the samec_name).Quoted 9 lines
scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1042
Quoted 8 lines
class QAPISchemaObjectTypeVariants(object):
def__init__(self, tag_name, tag_member, variants):
# Flat unions passtag_namebut nottag_member.
# Simple unions and alternates passtag_memberbut nottag_name. # Aftercheck(),tag_memberis always set, andtag_nameremainsscripts/qapi.py +2
-1- # a reliable witness of being used by a flat union. + # a reliable witness of being used by a flat union, and + # tag_member.type being None is a reliable witness of an alternate.Quoted 5 lines
A member without a type? Ugh! I wouldn't dare breaking invariants like that.
Of course, an alternate's tag member still has a type:
qtype_code. It's just not declared in the schema. Should it be a built-in type then?It's not a builtin that can ever be referenced in the .json files. But I could probably come up with something, if it would make you feel better.
Quoted 8 lines
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066
Quoted 5 lines
for v in self.variants:
# Reset seen array for each variant, since QMP names from one # branch do not affect another branch, nor add toall_membersscripts/qapi.py +1
-1- v.check(schema, self.tag_member.type, dict(seen), cases, union) + v.check(schema, self.tag_member.type, dict(seen), cases)I expect some rebase churn around here, so I'm not reviewing closely.
Indeed. All the more reason for me to post a v9 spin (and maybe defer the question of a non-None type for
tag_memberuntil after that post).Quoted 6 lines
My patches move the member name collision checking to
QAPISchemaObjectType.check().I suspect alternate branch name collision checking should similarly move to
QAPISchemaAlternateType.check().Yep, already that way in my pending v9 series after incorporating your patches.
Quoted 8 lines
Quoted 8 lines
scripts/qapi.py +8
-0+class QAPISchemaAlternateTypeTag(QAPISchemaObjectTypeMember): + def __init__(self): + QAPISchemaObjectTypeMember.__init__(self, 'type', '', False) + + def check(self, schema, seen): + assert len(seen) == 0 + seen[self.name] = self +Quoted 8 lines
scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1120
Quoted 4 lines
scripts/qapi.py +4
-0+ def c_type(self): + return 'qtype_code' + +This is a hack to work around the lack of a
qtype_codetype. I suspect creating such a type would be simpler in the end. Safer, too, because it would avoid having members without a type, which scares me.
Quoted 3 lines
I can play with dropping
c_type()here in favor of adding aqtype_codespecial class, but I may still need to keep this QAPISchemaAlternateTypeTag subclass.
Let's give it a try and see.
Quoted 59 lines
Quoted 11 lines
tests/qapi-schema/qapi-schema-test.json line 134
Quoted 7 lines
+++
b/tests/qapi-schema/qapi-schema-test.jsontests/qapi-schema/qapi-schema-test.json +1
-1@@ -131,7 +131,7 @@ 'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero', 'u': 'UserDefZero', 'type': 'UserDefZero' } } { 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool', - 'myKind': 'has_a' } } + 'myKind': 'has_a', 'max': 'str' } }Here, you add the positive test that alternate name 'max' works.
One, not mentioned in the commit message.
D'oh.
Quoted 6 lines
Two, the commit message says we may reintroduce FooKind if working with
qtype_codeturns out to be too confusing. If we ever do that, alternate name 'max' breaks, doesn't it? Shouldn't we keep it reserved then, just in case?See my comment above; at this point, I doubt we'll ever want to go back, so maybe I just need to be more definitive in stating that.
Quoted 20 lines
Quoted 20 lines
tests/test-qmp-input-visitor.c +3
-3+++ b/tests/test-qmp-input-visitor.c@@ -386,11 +386,10 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,qapi_free_AltStrBool (asb);visitor_input_teardown (data,NULL ); -/* FIXME: Order of alternate should not affect semantics; asn should -* parse the same as ans */ +/* FIXME: integer should parse as number */ v =visitor_input_test_init (data,"42" );visit_type_AltStrNum (v, &asn,NULL , &err); -/* FIXME g_assert_cmpint(asn->type, == ALT_STR_NUM_KIND_N); */ +/* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */ /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */ g_assert (err);error_free (err);@@ -398,30 +397,34 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,qapi_free_AltStrNum (asn);visitor_input_teardown (data,NULL );Quoted 14 lines
tests/test-qmp-input-visitor.c static void test_visitor_in_alternate_number(TestInputVisitorData *data, · line 407
Quoted 11 lines
tests/test-qmp-input-visitor.c +7
-3+/* FIXME: integer should parse as number */ v =visitor_input_test_init (data,"42" ); -visit_type_AltNumStr (v, &ans,NULL , &error_abort); - g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N); - g_assert_cmpfloat(ans->u.n, ==, 42); +visit_type_AltNumStr (v, &ans,NULL , &err); +/* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */ +/* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */ +g_assert (err); +error_free (err); + err =NULL ;What's happening here? Whatever it is, the commit message didn't prepare me for it...
Quoted 4 lines
See [1] above. 'asn' is now parsing the same as 'ans' (we are no longer sensitive to whether 'number' was the first member of the alternate), but it isn't until patch 11/17 that we fix things that 'ans' and 'asn' both properly parse '1' as a number instead of rejecting it as an integer.
Recommend to stick a brief note into the commit message that this temporarily breaks test so-and-so, marked FIXME.
Quoted 54 lines
Quoted 17 lines
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 456
Quoted 13 lines
tests/test-qmp-output-visitor.c +4
-3+++ b/tests/test-qmp-output-visitor.c@@ -449,20 +449,31 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data, const void *unused) {QObject *arg; -Error *err =NULL ; +UserDefAlternate *tmp; -UserDefAlternate *tmp =g_malloc0 (sizeof (UserDefAlternate)); - tmp->type =USER_DEF_ALTERNATE_KIND_I ; + tmp =g_new0 (UserDefAlternate,1 ); + tmp->type =QTYPE_QINT ; tmp->u.i =42 ;Coding style touched up. Okay.
Quoted 1 line
Quoted 7 lines
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 458
Quoted 3 lines
tests/test-qmp-output-visitor.c +1
-2-visit_type_UserDefAlternate (data->ov, &tmp,NULL , &err); -g_assert (err ==NULL ); +visit_type_UserDefAlternate (data->ov, &tmp,NULL , &error_abort);We need to make up our mind whether to use
g_assert(err == NULL)or&error_abortin tests. Wholesale conversion could be in order. I like&error_abort, because it's more concise.Wholesale conversion dead-ahead, in 14/17 of this subset.
Quoted 12 lines
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 466
Quoted 9 lines
arg = qmp_output_get_qobject(data->qov); g_assert(qobject_type(arg) == QTYPE_QINT); g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42); qapi_free_UserDefAlternate(tmp);tests/test-qmp-output-visitor.c +2
-0+ + tmp =g_malloc0 (sizeof (UserDefAlternate));
g_new0(UserDefAlternate, 1), please.Quoted 13 lines
tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 476
Quoted 10 lines
tests/test-qmp-output-visitor.c +10
-0+ tmp->type =QTYPE_QSTRING ; + tmp->u.s =g_strdup ("hello" ); + +visit_type_UserDefAlternate (data->ov, &tmp,NULL , &error_abort); + arg =qmp_output_get_qobject (data->qov); + +g_assert (qobject_type (arg) ==QTYPE_QSTRING ); + g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello"); + +qapi_free_UserDefAlternate (tmp);New test, not mentioned in commit message. Separate patch, perhaps, along with the nearby coding style touch ups?
Yes, I will split this portion of the test changes out to a separate commit.
Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types
In reply to Eric BlakeQuoted 110 lines · Eric Blake
Quoted 7 lines · Markus Armbruster
Quoted 3 lines · Eric Blake
Previously, working with alternates required two enums, and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a member of FooKind_lookup[], then use
member of FooKind, actually.
Or entry in the FooKind_lookup[] array.
Quoted 4 lines
Quoted 2 lines
FooKind_lookup[] like we do for other union types.
You probably mean FooKind here as well.
I'll play with the wording.
Quoted 25 lines
Quoted 11 lines
This has a subtle bug: since the values of
FooKind_lookupstart at zero, all entries ofFoo_qtypesthat were not explicitly initialized map to the same branch of the union as the first member of the alternate, rather than triggering a failure invisit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).Yes, I remember us discussing this bug.
While reading code to double-check your description, I stumbled over this beauty in generated
qapi-visit.c:visit_get_next_type(v, (int*) &(*obj)->type, BlockdevRef_qtypes,name, &err);
This casts enum BlockdevRefKind * to int *, which assumes the compiler represents the enum BlockdevRefKind as int or unsigned. It is free to use any integer type, though. Common mistake of programmers with insufficiently developed wariness of C's subtleties.
Quoted 8 lines
visit_get_next_type()passes the fishy int * on tov->get_next_type(). Only implementation isqmp_input_get_next_type(), which uses it so:*kind = qobjects[qobject_type(qobj)];Latent death trap.
Does your patch clean this up?
Yes, and I need to also document that this is an additional bug fix.
Quoted 9 lines
Quoted 9 lines
However, it IS observable in one case: the behavior of an alternate that contains a 'number' member but no 'int' member differs according to whether the 'number' was first in the qapi definition, and when the input being parsed is an integer; this is because the 'number' parser accepts
QTYPE_QINTin addition to the expectedQTYPE_QFLOAT. A later patch will worry about fixing alternates to parse all inputs that a non-alternate 'number' would accept, for now it is still marked FIXME.See [1] below.
Quoted 6 lines
Quoted 3 lines
This patch fixes the validation bug by deleting the indirection, and modifying
get_next_type()to directly return a qtype code.
get_next_type()doesn't return anything. Do you mean "store a qtype code"?Yes.
Quoted 4 lines
Quoted 2 lines
There is no longer a need to generate an implicit FooKind array
FooKind is an enum, not an array.
...to generate an implicit FooKind enum, nor FooKind_lookup[] array.
Quoted 12 lines
Quoted 12 lines
associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names); that also means we no longer have a collision with an alternate branch named 'max'. Next, the generated visitor is fixed to properly detect unexpected qtypes in the switch statement. This is done via the use of a new QAPISchemaAlternateTypeTag subclass and the use of a new
member.c_type()method when producing qapi-types. The new subtype also allows us to clean up a TODO left in the previous commit.Quoted 12 lines
Quoted 4 lines
Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. If that gets too confusing, we could reintroduce FooKind, but initialize it differently than
Quoted 8 lines
most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A=QTYPE_QDICT,
FOO_KIND_B=QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to usefoo->aorfoo->bwhen inspectingfoo->type. But without a current client, I didn't see the point of doing it now.You have a point below that we either need to reserve MAX and require no case-insensitive clashes, or that we will never want to add it. I'm leaning towards never going back, because the new way feels so much nicer.
[...]
Quoted 15 lines
Quoted 11 lines
tests/qapi-schema/qapi-schema-test.json line 134
Quoted 7 lines
+++
b/tests/qapi-schema/qapi-schema-test.jsontests/qapi-schema/qapi-schema-test.json +1
-1@@ -131,7 +131,7 @@ 'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero', 'u': 'UserDefZero', 'type': 'UserDefZero' } } { 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool', - 'myKind': 'has_a' } } + 'myKind': 'has_a', 'max': 'str' } }Here, you add the positive test that alternate name 'max' works.
One, not mentioned in the commit message.
D'oh.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Two, the commit message says we may reintroduce FooKind if working with
qtype_codeturns out to be too confusing. If we ever do that, alternate name 'max' breaks, doesn't it? Shouldn't we keep it reserved then, just in case?See my comment above; at this point, I doubt we'll ever want to go back, so maybe I just need to be more definitive in stating that.
Fair enough.
However, we should also consider QAPI language regularity and simplicity. To make that argument, I need to back up a bit.
There are three kinds of name collisions: QAPI schema, QMP wire, generated C.
QAPI schema collisions are the obvious ones:
* JSON object member names must be distinct (enforced by our JSON parser).
* Enumeration values must be distinct.
* Each flat union's variant's members must be distinct from the non-variant members.
A translation of the schema into a protocol or code can add additional restrictions.
I believe the translation to QMP wire doesn't add any. We don't mangle names, we don't add object members or enumeration values.
The translation to C does add some:
* Mangled names can collide even when unmangled names don't.
* Names can collide with names used by the implementation.
It can also remove collision possibilities (e.g. variant and non-variant members end up in separate name spaces), but that's not important here.
One possible approach would be to let qapi.py worry about QAPI schema / QMP wire collisions, and the C compiler about generated C collisions. We rejected that approach, because navigating from the C compiler's error messages to the broken spot in the QAPI schema is a bother.
Instead, qapi.py knows enough about the generators to predict collisions in generated C. This is feasible only because the generators follow simple and regular rules:
* Names are mangled by c_name(), prefixes or suffixes may be tacked on.
* Exception: enumeration values are mangled by camel_to_upper(). Although the way c_enum_const() is defined, you have to look closely to see it.
* Any pair of names that clashes before mangling also clashes after. Permits omitting collision checks for unmangled names when the mangled names are checked.
* Tag values are mangled *both* ways, because they occur both as C union members and as C enum members.
* We reserve a bunch of names for generator use: mangled names can't start with 'q_', mangled member names can't start with 'has_', mangled enumeration values can't be 'MAX', and so forth.
Remarks:
* A simple union's member names are also tag values, and are therefore mangled both ways, and both reserved member and enum value names apply.
* The moment you use an enumeration as type of a flat union tag, the other mangling and reserved names kicks in.
Conclusions:
* Having two different name manglers is a headache we could do without, especially since the second one camel_to_upper() is pretty magic.
We have it only to get
typedef enum BlockDeviceIoStatus {
BLOCK_DEVICE_IO_STATUS_OK = 0,
BLOCK_DEVICE_IO_STATUS_FAILED = 1,
BLOCK_DEVICE_IO_STATUS_NOSPACE = 2,
BLOCK_DEVICE_IO_STATUS_MAX = 3,
} BlockDeviceIoStatus;
instead of
typedef enum BlockDeviceIoStatus {
BlockDeviceIoStatus_ok = 0,
BlockDeviceIoStatus_failed = 1,
BlockDeviceIoStatus_nospace = 2,
BlockDeviceIoStatus_MAX = 3,
} BlockDeviceIoStatus;
Bah! CODING_STYLE doesn't even ask for shouting enumeration constants. Can't see why we do.
* Keeping the complexity of the rules under control is good both for qapi.py and for the QAPI schema language.
To that end, I think we should consider reserving the same set of names both for members and tag values. It gets rid of complications like enumerations you can't use as flat union tags.
Additionally, the question whether to keep the door open for generating an enum for the alternate cases becomes moot.
What do you think?
[...]
Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types
Quoted 23 lines · Markus Armbruster
Conclusions:
* Having two different name manglers is a headache we could do without, especially since the second one
camel_to_upper()is pretty magic.We have it only to get
typedef enum BlockDeviceIoStatus {
BLOCK_DEVICE_IO_STATUS_OK= 0,
BLOCK_DEVICE_IO_STATUS_FAILED= 1,
BLOCK_DEVICE_IO_STATUS_NOSPACE= 2,
BLOCK_DEVICE_IO_STATUS_MAX= 3,
} BlockDeviceIoStatus;instead of
typedef enum BlockDeviceIoStatus {
BlockDeviceIoStatus_ok= 0,
BlockDeviceIoStatus_failed= 1,
BlockDeviceIoStatus_nospace= 2,
BlockDeviceIoStatus_MAX= 3,
} BlockDeviceIoStatus;
Quoted 2 lines
Bah!
CODING_STYLEdoesn't even ask for shouting enumeration constants. Can't see why we do.
Interesting idea. In fact, if we went one step further:
BlockDeviceIoStatus_ok = 0,
...
BlockDeviceIoStatusMAX = 3.
(that is, typename + '_' + value for user values, and typename + 'MAX' for the sentinel), then the max value _cannot_ collide with any of the other values.
Quoted 4 lines
* Keeping the complexity of the rules under control is good both for
qapi.pyand for the QAPI schema language.
Quoted 8 lines
To that end, I think we should consider reserving the same set of names both for members and tag values. It gets rid of complications like enumerations you can't use as flat union tags.
Additionally, the question whether to keep the door open for generating an enum for the alternate cases becomes moot.
What do you think?
I like the idea. Don't know if it's too late for 2.5, though.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types
Quoted 30 lines · Eric Blake
Quoted 23 lines · Markus Armbruster
Conclusions:
* Having two different name manglers is a headache we could do without, especially since the second one
camel_to_upper()is pretty magic.We have it only to get
typedef enum BlockDeviceIoStatus {
BLOCK_DEVICE_IO_STATUS_OK= 0,
BLOCK_DEVICE_IO_STATUS_FAILED= 1,
BLOCK_DEVICE_IO_STATUS_NOSPACE= 2,
BLOCK_DEVICE_IO_STATUS_MAX= 3,
} BlockDeviceIoStatus;instead of
typedef enum BlockDeviceIoStatus {
BlockDeviceIoStatus_ok= 0,
BlockDeviceIoStatus_failed= 1,
BlockDeviceIoStatus_nospace= 2,
BlockDeviceIoStatus_MAX= 3,
} BlockDeviceIoStatus;Quoted 2 lines
Bah!
CODING_STYLEdoesn't even ask for shouting enumeration constants. Can't see why we do.Interesting idea. In fact, if we went one step further:
Quoted 7 lines
BlockDeviceIoStatus_ok= 0,
...
BlockDeviceIoStatusMAX = 3.(that is, typename + '_' + value for user values, and typename + 'MAX' for the sentinel), then the max value _cannot_ collide with any of the other values.
Another arbitrary rule killed.
Quoted 7 lines
Quoted 7 lines
* Keeping the complexity of the rules under control is good both for
qapi.pyand for the QAPI schema language.To that end, I think we should consider reserving the same set of names both for members and tag values. It gets rid of complications like enumerations you can't use as flat union tags.
Quoted 6 lines
Quoted 4 lines
Additionally, the question whether to keep the door open for generating an enum for the alternate cases becomes moot.
What do you think?
I like the idea. Don't know if it's too late for 2.5, though.
The only risk with renaming obviously unique identifiers is producing merge conflicts. Annoying, but easy enough to resolve.
How much churn exactly? I count 1423 occurences of generated enumeration constants in 143 out of 4442 source files. Top-scorer is of course QKeyCode, with 471 occurences in 9 files, almost all in tables. I've seen worse. I wouldn't ask for such a pull during hard freeze, of course. During soft freeze, merge conflict risk might be lower than usual.
PATCH v8 11/17 qapi: Fix alternates that accept 'number' but not 'int'
In reply to Eric BlakeThe QMP input visitor allows integral values to be assigned by promotion to a QTYPE_QFLOAT. However, when parsing an alternate, we did not take this into account, such that an alternate that accepts 'number' but not 'int' would reject integral values.
With this patch, we now have the following desirable table:
alternate has case selected for
'int' 'number' QTYPE_QINT QTYPE_QFLOAT
no no error error
no yes 'number' 'number'
yes no 'int' error
yes yes 'int' 'number'While it is unlikely that we will ever use 'number' in an alternate other than in the testsuite, it never hurts to be more precise in what we allow.
---
v8: no change
v7: rebase to named .u union
v6: rebase onto earlier testsuite and gen_err_check() improvements
--- include/qapi/visitor-impl.h | 2 +- include/qapi/visitor.h | 3 ++- qapi/qapi-visit-core.c | 4 ++-- qapi/qmp-input-visitor.c | 4 ++++ scripts/qapi-visit.py | 9 +++++++-- tests/test-qmp-input-visitor.c | 20 ++++++-------------- 6 files changed, 22 insertions(+), 20 deletions(-)
include/qapi/visitor-impl.h +1 -1
@@ -33,7 +33,7 @@ struct Visitorvoid (*type_enum)(Visitor *v,int *obj,const char *const strings[],const char *kind,const char *name,Error **errp);/* May be NULL; most useful for input visitors. */ -void (*get_next_type)(Visitor *v,qtype_code *type, +void (*get_next_type)(Visitor *v,qtype_code *type,bool promote_int,const char *name,Error **errp);void (*type_int)(Visitor *v,int64_t *obj,const char *name,Error **errp);
include/qapi/visitor.h +2 -1
@@ -46,8 +46,9 @@ void visit_optional(Visitor *v, bool *present, const char *name,* Determine the qtype of the item @name in the current object visit. * For input visitors, set *@type to the correct qtype of a qapi * alternate type; for other visitors, leave *@type unchanged. +* If @promote_int, treat integers as QTYPE_FLOAT. */ -void visit_get_next_type (Visitor *v,qtype_code *type, +void visit_get_next_type (Visitor *v,qtype_code *type,bool promote_int,const char *name,Error **errp);void visit_type_enum (Visitor *v,int *obj,const char *const strings[],const char *kind,const char *name,Error **errp);
qapi/qapi-visit-core.c +2 -2
@@ -81,11 +81,11 @@ void visit_optional(Visitor *v, bool *present, const char *name, } } -void visit_get_next_type (Visitor *v,qtype_code *type, +void visit_get_next_type (Visitor *v,qtype_code *type,bool promote_int,const char *name,Error **errp) {if (v->get_next_type) { - v->get_next_type (v, type, name, errp); + v->get_next_type (v, type, promote_int, name, errp); } }
qapi/qmp-input-visitor.c +4 -0
@@ -209,6 +209,7 @@ static void qmp_input_end_list(Visitor *v, Error **errp) }static void qmp_input_get_next_type (Visitor *v,qtype_code *type, +bool promote_int,const char *name,Error **errp) {QmpInputVisitor *qiv =to_qiv (v);
@@ -219,6 +220,9 @@ static void qmp_input_get_next_type(Visitor *v, qtype_code *type,return ; } *type =qobject_type (qobj); +if (promote_int && *type ==QTYPE_QINT ) { + *type =QTYPE_QFLOAT ; + } } static void qmp_input_type_int(Visitor *v, int64_t *obj, const char *name,
scripts/qapi-visit.py +7 -2
@@ -179,6 +179,11 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s *obj, const char *name, Error def gen_visit_alternate(name, variants): + promote_int = 'true' + for var in variants.variants: + if var.type.alternate_qtype() == 'QTYPE_QINT': + promote_int = 'false' + ret = mcgen(''' void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error **errp)
@@ -189,13 +194,13 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error if (err) { goto out; } - visit_get_next_type(v, &(*obj)->type, name, &err); + visit_get_next_type(v, &(*obj)->type, %(promote_int)s, name, &err); if (err) { goto out_obj; } switch ((*obj)->type) { ''', - c_name=c_name(name)) + c_name=c_name(name), promote_int=promote_int) for var in variants.variants: ret += mcgen('''
tests/test-qmp-input-visitor.c +6 -14
@@ -386,25 +386,17 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,qapi_free_AltStrBool (asb);visitor_input_teardown (data,NULL ); -/* FIXME: integer should parse as number */ v =visitor_input_test_init (data,"42" ); -visit_type_AltStrNum (v, &asn,NULL , &err); -/* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */ -/* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */ -g_assert (err); -error_free (err); - err =NULL ; +visit_type_AltStrNum (v, &asn,NULL , &error_abort); + g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); + g_assert_cmpfloat(asn->u.n, ==, 42);qapi_free_AltStrNum (asn);visitor_input_teardown (data,NULL ); -/* FIXME: integer should parse as number */ v =visitor_input_test_init (data,"42" ); -visit_type_AltNumStr (v, &ans,NULL , &err); -/* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */ -/* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */ -g_assert (err); -error_free (err); - err =NULL ; +visit_type_AltNumStr (v, &ans,NULL , &error_abort); + g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); + g_assert_cmpfloat(ans->u.n, ==, 42);qapi_free_AltNumStr (ans);visitor_input_teardown (data,NULL );
Signature · 1 line
2.4.3
PATCH v8 12/17 qapi: Remove dead visitor code
In reply to Eric BlakeCommit cbc95538 removed unused start_handle() and end_handle(), but forgot got remove their declarations.
Commit 4e27e819 introduced optional visitor callbacks for all sorts of int types, but except for type_uint64 and type_size, none of them have ever been supplied (the generic implementation based on using type_int then bounds-checking works just fine). In the interest of simplicity, it's easier to make the visitor callback interface not have to worry about the other sizes.
---
v8: no change
v7: no change
v6: no change
--- include/qapi/visitor-impl.h | 19 +++---- include/qapi/visitor.h | 3 - qapi/qapi-visit-core.c | 131 +++++++++++++++++--------------------------- 3 files changed, 58 insertions(+), 95 deletions(-)
include/qapi/visitor-impl.h +8 -11
@@ -1,7 +1,7 @@/* * Core Definitions for QAPI Visitor implementations * -* Copyright (C) 2012 Red Hat, Inc. +* Copyright (C) 2012, 2015 Red Hat, Inc. * * Author: Paolo Bonizni <[email protected]> *
@@ -48,18 +48,15 @@ struct Visitorvoid (*optional)(Visitor *v,bool *present,const char *name,Error **errp); -void (*type_uint8)(Visitor *v,uint8_t *obj,const char *name,Error **errp); -void (*type_uint16)(Visitor *v,uint16_t *obj,const char *name,Error **errp); -void (*type_uint32)(Visitor *v,uint32_t *obj,const char *name,Error **errp); -void (*type_uint64)(Visitor *v,uint64_t *obj,const char *name,Error **errp); -void (*type_int8)(Visitor *v,int8_t *obj,const char *name,Error **errp); -void (*type_int16)(Visitor *v,int16_t *obj,const char *name,Error **errp); -void (*type_int32)(Visitor *v,int32_t *obj,const char *name,Error **errp); -void (*type_int64)(Visitor *v,int64_t *obj,const char *name,Error **errp); -/* visit_type_size() falls back to (*type_uint64)() if type_size is unset */ -void (*type_size)(Visitor *v,uint64_t *obj,const char *name,Error **errp);bool (*start_union)(Visitor *v,bool data_present,Error **errp);void (*end_union)(Visitor *v,bool data_present,Error **errp); + +/* Only required to visit uint64 differently than (*type_int)(). */ +void (*type_uint64)(Visitor *v,uint64_t *obj,const char *name, +Error **errp); +/* Only required to visit sizes differently than (*type_uint64)(). */ +void (*type_size)(Visitor *v,uint64_t *obj,const char *name, +Error **errp); }; void input_type_enum(Visitor *v, int *obj, const char * const strings[],
include/qapi/visitor.h +0 -3
@@ -27,9 +27,6 @@ typedef struct GenericListstruct GenericList *next; } GenericList; -void visit_start_handle (Visitor *v,void **obj,const char *kind, -const char *name,Error **errp); -void visit_end_handle (Visitor *v,Error **errp);void visit_start_struct (Visitor *v,void **obj,const char *kind,const char *name,size_t size,Error **errp);void visit_end_struct (Visitor *v,Error **errp);
qapi/qapi-visit-core.c +50 -81
184 lines not on this page · raw message
Signature · 1 line
2.4.3
PATCH v8 13/17 qapi: Plug leaks in test-qmp-*
In reply to Eric BlakeMake valgrind happy with the current state of the tests, so that it is easier to see if future patches introduce new memory problems without being drowned in noise. Many of the leaks were due to calling a second init without tearing down the data from an earlier visit. But since teardown is already idempotent, and we already register teardown as part of input_visitor_test_add(), it is nicer to just make init() safe to call multiple times than it is to have to make all tests call teardown.
Another common leak was forgetting to clean up an error object, after testing that an error was raised.
Another leak was in test_visitor_in_struct_nested(), failing to clean the base member of UserDefTwo. Cleaning that up left check_and_free_str() as dead code (since using the qapi_free_* takes care of recursion, and we don't want double frees).
---
v8: no change
v7: no change
v6: make init repeatable rather than adding teardown everywhere, fix additional leak with UserDefTwo base, plug additional files
--- tests/test-qmp-input-strict.c | 10 ++++++++++ tests/test-qmp-input-visitor.c | 41 +++++++---------------------------------- tests/test-qmp-output-visitor.c | 4 +++- 3 files changed, 20 insertions(+), 35 deletions(-)
tests/test-qmp-input-strict.c +10 -0
@@ -77,6 +77,8 @@ static Visitor *validate_test_init_raw(TestInputVisitorData *data, {Visitor *v; +validate_teardown (data,NULL ); + data->obj =qobject_from_json (json_string);g_assert (data->obj !=NULL );
@@ -193,6 +195,8 @@ static void test_validate_fail_struct(TestInputVisitorData *data,visit_type_TestStruct (v, &p,NULL , &err);g_assert (err); +error_free (err); +/* FIXME: visitor should not allocate p when returning error */ if (p) {g_free (p->string); }
@@ -210,6 +214,7 @@ static void test_validate_fail_struct_nested(TestInputVisitorData *data,visit_type_UserDefTwo (v, &udp,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefTwo (udp); }
@@ -224,6 +229,7 @@ static void test_validate_fail_list(TestInputVisitorData *data,visit_type_UserDefOneList (v, &head,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefOneList (head); }
@@ -239,6 +245,7 @@ static void test_validate_fail_union_native_list(TestInputVisitorData *data,visit_type_UserDefNativeListUnion (v, &tmp,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefNativeListUnion (tmp); }
@@ -253,6 +260,7 @@ static void test_validate_fail_union_flat(TestInputVisitorData *data,visit_type_UserDefFlatUnion (v, &tmp,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefFlatUnion (tmp); }
@@ -268,6 +276,7 @@ static void test_validate_fail_union_flat_no_discrim(TestInputVisitorData *data,visit_type_UserDefFlatUnion2 (v, &tmp,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefFlatUnion2 (tmp); }
@@ -282,6 +291,7 @@ static void test_validate_fail_alternate(TestInputVisitorData *data,visit_type_UserDefAlternate (v, &tmp,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefAlternate (tmp); }
tests/test-qmp-input-visitor.c +7 -34
159 lines not on this page · raw message
tests/test-qmp-output-visitor.c +3 -1
@@ -391,6 +391,7 @@ static void test_visitor_out_any(TestOutputVisitorData *data, qobj =QOBJECT (qdict);visit_type_any (data->ov, &qobj,NULL , &err);g_assert (!err); +qobject_decref (qobj); obj =qmp_output_get_qobject (data->qov);g_assert (obj !=NULL ); qdict =qobject_to_qdict (obj);
@@ -411,7 +412,6 @@ static void test_visitor_out_any(TestOutputVisitorData *data,g_assert (qstring); g_assert_cmpstr(qstring_get_str(qstring), ==, "foo");qobject_decref (obj); -qobject_decref (qobj); } static void test_visitor_out_union_flat(TestOutputVisitorData *data,
@@ -462,6 +462,7 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data, g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42);qapi_free_UserDefAlternate (tmp); +qobject_decref (arg); tmp =g_malloc0 (sizeof (UserDefAlternate)); tmp->type =QTYPE_QSTRING ;
@@ -474,6 +475,7 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data, g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello");qapi_free_UserDefAlternate (tmp); +qobject_decref (arg); } static void test_visitor_out_empty(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
PATCH v8 14/17 qapi: Simplify error testing in test-qmp-*
In reply to Eric BlakeBy using &error_abort, we can avoid a local err variable in situations where we expect success.
By moving err into data, we can let test teardown take care of cleaning up any collected error (and allowing for fewer lines of code between repeated tests where init runs teardown on our behalf).
---
v8: no change
v7: pick up whitespace changes dropped from earlier commit
v6: new patch
--- tests/test-qmp-input-strict.c | 77 +++++++++------------------- tests/test-qmp-input-visitor.c | 101 ++++++++++++------------------------- tests/test-qmp-output-visitor.c | 52 +++++-------------- tests/test-visitor-serialization.c | 42 +++++++-------- 4 files changed, 86 insertions(+), 186 deletions(-)
tests/test-qmp-input-strict.c +25 -52
@@ -26,6 +26,7 @@typedef struct TestInputVisitorData {QObject *obj;QmpInputVisitor *qiv; +Error *err; }TestInputVisitorData ; static void validate_teardown(TestInputVisitorData *data,
@@ -34,6 +35,9 @@ static void validate_teardown(TestInputVisitorData *data,qobject_decref (data->obj); data->obj =NULL ; +error_free (data->err); + data->err =NULL ; +if (data->qiv) {qmp_input_visitor_cleanup (data->qiv); data->qiv =NULL ;
@@ -96,13 +100,11 @@ static void test_validate_struct(TestInputVisitorData *data, const void *unused) {TestStruct *p =NULL ; -Error *err =NULL ;Visitor *v; v =validate_test_init (data,"{ 'integer': -42, 'boolean': true, 'string': 'foo' }" ); -visit_type_TestStruct (v, &p,NULL , &err); -g_assert (!err); +visit_type_TestStruct (v, &p,NULL , &error_abort);g_free (p->string);g_free (p); }
@@ -111,7 +113,6 @@ static void test_validate_struct_nested(TestInputVisitorData *data, const void *unused) {UserDefTwo *udp =NULL ; -Error *err =NULL ;Visitor *v; v = validate_test_init(data, "{ 'string0': 'string0', "
@@ -119,8 +120,7 @@ static void test_validate_struct_nested(TestInputVisitorData *data,"'dict2': { 'userdef': { 'integer': 42, " "'string': 'string' }, 'string': 'string2'}}}"); -visit_type_UserDefTwo (v, &udp,NULL , &err); -g_assert (!err); +visit_type_UserDefTwo (v, &udp,NULL , &error_abort);qapi_free_UserDefTwo (udp); }
@@ -128,13 +128,11 @@ static void test_validate_list(TestInputVisitorData *data, const void *unused) {UserDefOneList *head =NULL ; -Error *err =NULL ;Visitor *v; v =validate_test_init (data,"[ { 'string': 'string0', 'integer': 42 }, { 'string': 'string1', 'integer': 43 }, { 'string': 'string2', 'integer': 44 } ]" ); -visit_type_UserDefOneList (v, &head,NULL , &err); -g_assert (!err); +visit_type_UserDefOneList (v, &head,NULL , &error_abort);qapi_free_UserDefOneList (head); }
@@ -143,12 +141,10 @@ static void test_validate_union_native_list(TestInputVisitorData *data, {UserDefNativeListUnion *tmp =NULL ;Visitor *v; -Error *err =NULL ; v =validate_test_init (data,"{ 'type': 'integer', 'data' : [ 1, 2 ] }" ); -visit_type_UserDefNativeListUnion (v, &tmp,NULL , &err); -g_assert (!err); +visit_type_UserDefNativeListUnion (v, &tmp,NULL , &error_abort);qapi_free_UserDefNativeListUnion (tmp); }
@@ -157,7 +153,6 @@ static void test_validate_union_flat(TestInputVisitorData *data, {UserDefFlatUnion *tmp =NULL ;Visitor *v; -Error *err =NULL ; v =validate_test_init (data, "{ 'enum1': 'value1', "
@@ -165,8 +160,7 @@ static void test_validate_union_flat(TestInputVisitorData *data,"'string': 'str', " "'boolean': true }"); -visit_type_UserDefFlatUnion (v, &tmp,NULL , &err); -g_assert (!err); +visit_type_UserDefFlatUnion (v, &tmp,NULL , &error_abort);qapi_free_UserDefFlatUnion (tmp); }
@@ -175,12 +169,10 @@ static void test_validate_alternate(TestInputVisitorData *data, {UserDefAlternate *tmp =NULL ;Visitor *v; -Error *err =NULL ; v =validate_test_init (data,"42" ); -visit_type_UserDefAlternate (v, &tmp,NULL , &err); -g_assert (!err); +visit_type_UserDefAlternate (v, &tmp,NULL , &error_abort);qapi_free_UserDefAlternate (tmp); }
@@ -188,14 +180,12 @@ static void test_validate_fail_struct(TestInputVisitorData *data, const void *unused) {TestStruct *p =NULL ; -Error *err =NULL ;Visitor *v; v =validate_test_init (data,"{ 'integer': -42, 'boolean': true, 'string': 'foo', 'extra': 42 }" ); -visit_type_TestStruct (v, &p,NULL , &err); -g_assert (err); -error_free (err); +visit_type_TestStruct (v, &p,NULL , &data->err); +g_assert (data->err);/* FIXME: visitor should not allocate p when returning error */ if (p) {g_free (p->string);
@@ -207,14 +197,12 @@ static void test_validate_fail_struct_nested(TestInputVisitorData *data, const void *unused) {UserDefTwo *udp =NULL ; -Error *err =NULL ;Visitor *v; v =validate_test_init (data,"{ 'string0': 'string0', 'dict1': { 'string1': 'string1', 'dict2': { 'userdef1': { 'integer': 42, 'string': 'string', 'extra': [42, 23, {'foo':'bar'}] }, 'string2': 'string2'}}}" ); -visit_type_UserDefTwo (v, &udp,NULL , &err); -g_assert (err); -error_free (err); +visit_type_UserDefTwo (v, &udp,NULL , &data->err); +g_assert (data->err);qapi_free_UserDefTwo (udp); }
@@ -222,14 +210,12 @@ static void test_validate_fail_list(TestInputVisitorData *data, const void *unused) {UserDefOneList *head =NULL ; -Error *err =NULL ;Visitor *v; v =validate_test_init (data,"[ { 'string': 'string0', 'integer': 42 }, { 'string': 'string1', 'integer': 43 }, { 'string': 'string2', 'integer': 44, 'extra': 'ggg' } ]" ); -visit_type_UserDefOneList (v, &head,NULL , &err); -g_assert (err); -error_free (err); +visit_type_UserDefOneList (v, &head,NULL , &data->err); +g_assert (data->err);qapi_free_UserDefOneList (head); }
@@ -237,15 +223,13 @@ static void test_validate_fail_union_native_list(TestInputVisitorData *data, const void *unused) {UserDefNativeListUnion *tmp =NULL ; -Error *err =NULL ;Visitor *v; v =validate_test_init (data,"{ 'type': 'integer', 'data' : [ 'string' ] }" ); -visit_type_UserDefNativeListUnion (v, &tmp,NULL , &err); -g_assert (err); -error_free (err); +visit_type_UserDefNativeListUnion (v, &tmp,NULL , &data->err); +g_assert (data->err);qapi_free_UserDefNativeListUnion (tmp); }
@@ -253,14 +237,12 @@ static void test_validate_fail_union_flat(TestInputVisitorData *data, const void *unused) {UserDefFlatUnion *tmp =NULL ; -Error *err =NULL ;Visitor *v; v =validate_test_init (data,"{ 'string': 'c', 'integer': 41, 'boolean': true }" ); -visit_type_UserDefFlatUnion (v, &tmp,NULL , &err); -g_assert (err); -error_free (err); +visit_type_UserDefFlatUnion (v, &tmp,NULL , &data->err); +g_assert (data->err);qapi_free_UserDefFlatUnion (tmp); }
@@ -268,15 +250,13 @@ static void test_validate_fail_union_flat_no_discrim(TestInputVisitorData *data, const void *unused) {UserDefFlatUnion2 *tmp =NULL ; -Error *err =NULL ;Visitor *v;/* test situation where discriminator field ('enum1' here) is missing */ v =validate_test_init (data,"{ 'integer': 42, 'string': 'c', 'string1': 'd', 'string2': 'e' }" ); -visit_type_UserDefFlatUnion2 (v, &tmp,NULL , &err); -g_assert (err); -error_free (err); +visit_type_UserDefFlatUnion2 (v, &tmp,NULL , &data->err); +g_assert (data->err);qapi_free_UserDefFlatUnion2 (tmp); }
@@ -285,13 +265,11 @@ static void test_validate_fail_alternate(TestInputVisitorData *data, {UserDefAlternate *tmp =NULL ;Visitor *v; -Error *err =NULL ; v =validate_test_init (data,"3.14" ); -visit_type_UserDefAlternate (v, &tmp,NULL , &err); -g_assert (err); -error_free (err); +visit_type_UserDefAlternate (v, &tmp,NULL , &data->err); +g_assert (data->err);qapi_free_UserDefAlternate (tmp); }
@@ -299,16 +277,11 @@ static void do_test_validate_qmp_introspect(TestInputVisitorData *data, const char *schema_json) {SchemaInfoList *schema =NULL ; -Error *err =NULL ;Visitor *v; v =validate_test_init_raw (data, schema_json); -visit_type_SchemaInfoList (v, &schema,NULL , &err); -if (err) { -fprintf (stderr,"%s" ,error_get_pretty (err)); - } -g_assert (!err); +visit_type_SchemaInfoList (v, &schema,NULL , &error_abort);g_assert (schema);qapi_free_SchemaInfoList (schema);
tests/test-qmp-input-visitor.c +31 -70
338 lines not on this page · raw message
tests/test-qmp-output-visitor.c +13 -39
178 lines not on this page · raw message
tests/test-visitor-serialization.c +17 -25
103 lines not on this page · raw message
Signature · 1 line
2.4.3
PATCH v8 15/17 qapi: Test failure in middle of array parse
In reply to Eric BlakeOur generated list visitors have the same problem as has been mentioned elsewhere (see commit 2f52e20): they allocate data even on failure. An upcoming patch will correct things to provide saner guarantees, but first we need to expose the behavior in the testsuite to ensure we aren't introducing any memory usage bugs.
---
v8: no change
v7: no change
v6: rebase onto earlier gen_err_check() and testsuite improvements
--- scripts/qapi-visit.py | 4 ++++ tests/test-qmp-input-visitor.c | 10 ++++++++++ 2 files changed, 14 insertions(+)
scripts/qapi-visit.py +4 -0
@@ -138,6 +138,10 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error def gen_visit_list(name, element_type): + # FIXME: if *obj is NULL on entry, and the first visit_next_list() + # assigns to *obj, while a later one fails, we should clean up *obj + # rather than leaving it non-NULL. As currently written, the caller must + # call qapi_free_FOOList() to avoid a memory leak of the partial FOOList. return mcgen(''' void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error **errp)
tests/test-qmp-input-visitor.c +10 -0
@@ -698,6 +698,7 @@ static void test_visitor_in_errors(TestInputVisitorData *data, {TestStruct *p =NULL ;Visitor *v; +strList *q =NULL ; v =visitor_input_test_init (data,"{ 'integer': false, 'boolean': 'foo', 'string': -42 }" );
@@ -709,6 +710,15 @@ static void test_visitor_in_errors(TestInputVisitorData *data,g_free (p->string);g_free (p); + + v =visitor_input_test_init (data,"[ '1', '2', false, '3' ]" ); +/* FIXME - a failed parse should not leave a partially-allocated +* array for us to clean up; this could cause callers to leak +* memory. */ +visit_type_strList (v, &q,NULL , &data->err); +assert (q); +assert (data->err); +qapi_free_strList (q); }int main (int argc,char **argv)
Signature · 1 line
2.4.3
PATCH v8 16/17 qapi: More tests of input arrays
In reply to Eric BlakeOur testsuite had no coverage of empty arrays, nor of what happens when the input does not match the expected type. Useful to have, especially if we start changing the visitor contracts.
---
v8: no change
v7: no change
v6: new patch
--- tests/test-qmp-input-visitor.c | 51 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+)
tests/test-qmp-input-visitor.c +51 -0
@@ -236,6 +236,12 @@ static void test_visitor_in_list(TestInputVisitorData *data, }qapi_free_UserDefOneList (head); + head =NULL ; + +/* An empty list is valid */ + v =visitor_input_test_init (data,"[]" ); +visit_type_UserDefOneList (v, &head,NULL , &error_abort); +g_assert (!head); } static void test_visitor_in_any(TestInputVisitorData *data,
@@ -721,6 +727,49 @@ static void test_visitor_in_errors(TestInputVisitorData *data,qapi_free_strList (q); } +static void test_visitor_in_wrong_type (TestInputVisitorData *data, +const void *unused) +{ +TestStruct *p =NULL ; +Visitor *v; +strList *q =NULL ; +int64_t i; + +/* Make sure arrays and structs cannot be confused */ + + v =visitor_input_test_init (data,"[]" ); +visit_type_TestStruct (v, &p,NULL , &data->err); +g_assert (data->err); +g_assert (!p); + + v =visitor_input_test_init (data,"{}" ); +visit_type_strList (v, &q,NULL , &data->err); +assert (data->err); +assert (!q); + +/* Make sure primitives and struct cannot be confused */ + + v =visitor_input_test_init (data,"1" ); +visit_type_TestStruct (v, &p,NULL , &data->err); +g_assert (data->err); +g_assert (!p); + + v =visitor_input_test_init (data,"{}" ); +visit_type_int (v, &i,NULL , &data->err); +assert (data->err); + +/* Make sure primitives and arrays cannot be confused */ + + v =visitor_input_test_init (data,"1" ); +visit_type_strList (v, &q,NULL , &data->err); +assert (data->err); +assert (!q); + + v =visitor_input_test_init (data,"[]" ); +visit_type_int (v, &i,NULL , &data->err); +assert (data->err); +} +int main (int argc,char **argv) {TestInputVisitorData in_visitor_data;
@@ -753,6 +802,8 @@ int main(int argc, char **argv) &in_visitor_data, test_visitor_in_alternate);input_visitor_test_add ("/visitor/input/errors" , &in_visitor_data, test_visitor_in_errors); +input_visitor_test_add ("/visitor/input/wrong-type" , + &in_visitor_data, test_visitor_in_wrong_type);input_visitor_test_add ("/visitor/input/alternate-number" , &in_visitor_data, test_visitor_in_alternate_number); input_visitor_test_add("/visitor/input/native_list/int",
Signature · 1 line
2.4.3
PATCH v8 17/17 qapi: Simplify visits of optional fields
In reply to Eric BlakeNone of the visitor callbacks would set an error when testing if an optional field was present; make this part of the interface contract by eliminating the errp argument. Then, for less code, reflect the determined boolean value back to the caller instead of making the caller read the boolean after the fact.
The resulting generated code has a nice diff:
|- visit_optional(v, &has_fdset_id, "fdset-id", &err);
|- if (err) {
|- goto out;
|- }
|- if (has_fdset_id) {
|+ if (visit_optional(v, &has_fdset_id, "fdset-id")) {
| visit_type_int(v, &fdset_id, "fdset-id", &err);
| if (err) {
| goto out;
| }
| }---
v8: no change
v7: rebase to no member.c_name()
v6: rebase onto earlier testsuite and gen_err_check() improvements
--- include/qapi/visitor-impl.h | 5 ++--- include/qapi/visitor.h | 10 ++++++++-- qapi/opts-visitor.c | 2 +- qapi/qapi-visit-core.c | 6 +++--- qapi/qmp-input-visitor.c | 3 +-- qapi/string-input-visitor.c | 3 +-- scripts/qapi.py | 7 +------ 7 files changed, 17 insertions(+), 19 deletions(-)
include/qapi/visitor-impl.h +2 -3
@@ -44,9 +44,8 @@ struct Visitorvoid (*type_any)(Visitor *v,QObject **obj,const char *name,Error **errp); -/* May be NULL */ -void (*optional)(Visitor *v,bool *present,const char *name, -Error **errp); +/* May be NULL; most useful for input visitors. */ +void (*optional)(Visitor *v,bool *present,const char *name);bool (*start_union)(Visitor *v,bool data_present,Error **errp);void (*end_union)(Visitor *v,bool data_present,Error **errp);
include/qapi/visitor.h +8 -2
@@ -36,8 +36,14 @@ void visit_end_implicit_struct(Visitor *v, Error **errp);void visit_start_list (Visitor *v,const char *name,Error **errp);GenericList *visit_next_list (Visitor *v,GenericList **list,Error **errp);void visit_end_list (Visitor *v,Error **errp); -void visit_optional (Visitor *v,bool *present,const char *name, -Error **errp); + +/** +* Check if an optional member @name of an object needs visiting. +* For input visitors, set *@present according to whether the +* corresponding visit_type_*() needs calling; for other visitors, +* leave *@present unchanged. Return *@present for convenience. +*/ +bool visit_optional (Visitor *v,bool *present,const char *name);/** * Determine the qtype of the item @name in the current object visit.
qapi/opts-visitor.c +1 -1
@@ -488,7 +488,7 @@ opts_type_size(Visitor *v, uint64_t *obj, const char *name, Error **errp)static void -opts_optional (Visitor *v,bool *present,const char *name,Error **errp) +opts_optional (Visitor *v,bool *present,const char *name) {OptsVisitor *ov =DO_UPCAST (OptsVisitor, visitor, v);
qapi/qapi-visit-core.c +3 -3
@@ -73,12 +73,12 @@ void visit_end_union(Visitor *v, bool data_present, Error **errp) } } -void visit_optional (Visitor *v,bool *present,const char *name, -Error **errp) +bool visit_optional (Visitor *v,bool *present,const char *name) {if (v->optional) { - v->optional (v, present, name, errp); + v->optional (v, present, name); } +return *present; } void visit_get_next_type(Visitor *v, qtype_code *type, bool promote_int,
qapi/qmp-input-visitor.c +1 -2
@@ -300,8 +300,7 @@ static void qmp_input_type_any(Visitor *v, QObject **obj, const char *name, *obj = qobj; } -static void qmp_input_optional (Visitor *v,bool *present,const char *name, -Error **errp) +static void qmp_input_optional (Visitor *v,bool *present,const char *name) {QmpInputVisitor *qiv =to_qiv (v);QObject *qobj =qmp_input_get_object (qiv, name, true);
qapi/string-input-visitor.c +1 -2
@@ -299,8 +299,7 @@ static void parse_type_number(Visitor *v, double *obj, const char *name, *obj = val; } -static void parse_optional (Visitor *v,bool *present,const char *name, -Error **errp) +static void parse_optional (Visitor *v,bool *present,const char *name) {StringInputVisitor *siv =DO_UPCAST (StringInputVisitor, visitor, v);
scripts/qapi.py +1 -6
@@ -1619,15 +1619,10 @@ def gen_visit_fields(members, prefix='', need_cast=False, skiperr=False): for memb in members: if memb.optional: ret += mcgen(''' - visit_optional(v, &%(prefix)shas_%(c_name)s, "%(name)s", %(errp)s); + if (visit_optional(v, &%(prefix)shas_%(c_name)s, "%(name)s")) { ''', prefix=prefix, c_name=c_name(memb.name), name=memb.name, errp=errparg) - ret += gen_err_check(skiperr=skiperr) - ret += mcgen(''' - if (%(prefix)shas_%(c_name)s) { -''', - prefix=prefix, c_name=c_name(memb.name)) push_indent() # Ugly: sometimes we need to cast away const
Signature · 1 line
2.4.3