Skip to contentSkip to conversation

alternate layout (post-introspection cleanups, subset C)

PATCH Superseded Eric Blake 2 participants 56 messages Oct 28, 2015 – Nov 5, 2015

v8 of 14 Superseded Latest is v14 →

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

v8 cover alternate layout (post-introspection cleanups, subset C)

Pending prerequisite: Markus' qapi-next branch
git://repo.or.cz/qemu/armbru.git qapi-next
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg06417.html

Also available as a tag at this location:
git fetch git://repo.or.cz/qemu/ericb.git qapi-cleanupv8c

as well as part of my branch with the rest of the v5 series, at:
http://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/qapi

v8 notes:
Minor changes when rebasing to latest, improved commit messages in a few places, plus the addition of a few new patches. Markus started reviewing v7, but hadn't gotten very far.

001/17:[----] [--] 'qapi: Use generated TestStruct machinery in tests' 002/17:[----] [--] 'qapi: Strengthen test of TestStructList' 003/17:[----] [--] 'qapi: Provide nicer array names in introspection' 004/17:[----] [--] 'qapi-introspect: Guarantee particular sorting' 005/17:[down] 'qapi: Track simple union tag in object.local_members' 006/17:[down] 'qapi-types: Consolidate gen_struct() and gen_union()' 007/17:[0007] [FC] 'qapi: Rework collision assertions' 008/17:[down] 'qapi: Remove outdated tests related to QMP/branch collisions' 009/17:[down] 'qapi: Add positive tests to qapi-schema-test' 010/17:[----] [-C] 'qapi: Simplify visiting of alternate types' 011/17:[----] [--] 'qapi: Fix alternates that accept 'number' but not 'int'' 012/17:[----] [--] 'qapi: Remove dead visitor code' 013/17:[----] [--] 'qapi: Plug leaks in test-qmp-*' 014/17:[----] [-C] 'qapi: Simplify error testing in test-qmp-*' 015/17:[----] [--] 'qapi: Test failure in middle of array parse' 016/17:[----] [--] 'qapi: More tests of input arrays' 017/17:[----] [--] 'qapi: Simplify visits of optional fields'

v7 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg04112.html
Patches 1-3 of the previous round have moved to subset B; patch 7 is gone, and a couple of new patches are present in this round. The latest version of subset B made it much easier to reason about collision detection (namely, tag values can't collide with QMP values thanks to a named rather than anonymous union in the C type), and I like how things turned out. I suspect the hardest part of the review will be patches 5/14 and 7/14, although none of this has really had much review in any earlier versions.

v6 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg01980.html
Add some patches and rebase onto work on subset B. Rearrange some patches from v5 (this set includes 17-20, 23, 25-27). Backport diff gets a bit confused by one patch title changing.

Not much direct review comments, although some of the changes here are updated based on comments made on other patches in the v5 series.

Subset D (and more?) will come later.

In v5:
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg05410.html
I _did_ rearrange patches to try and group related features:

1-2: Groundwork cleanups
3-5: Add more test cases
6-16: Front-end cleanups
17-18: Introspection output cleanups
19-20: 'alternate' type cleanups
21-29: qapi visitor cleanups
30-45: qapi-ify netdev_add
46: add qapi shorthand for flat unions

Lots of fixes based on additional testing, and rebased to track other changes that happened in the meantime. The series is huge; I can split off smaller portions as requested.

In v4:
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg02580.html
add some more clean up patches
rebase to Markus' recent work
pull in part of Zoltán's work to make netdev_add a flat union, further enhancing it to be introspectible

I might be able to rearrange some of these patches, or separate it into smaller independent series, if requested; but I'm posting now to get review started.

In v3:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg02059.html
redo cleanup of dealloc of partial struct
add patches to make all visit_type_*() avoid leaks on failure add patches to allow boxed command arguments and events

In v2:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00900.html
rebase to Markus' v3 series
rework how comments are emitted for fields inherited from base additional patches added for deleting colliding 'void *data' documentation updates to match code changes

v1 was here:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05266.html
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05325.html

Eric Blake (17):
qapi: Use generated TestStruct machinery in tests
qapi: Strengthen test of TestStructList
qapi: Provide nicer array names in introspection
qapi-introspect: Guarantee particular sorting
qapi: Track simple union tag in object.local_members
qapi-types: Consolidate gen_struct() and gen_union()
qapi: Rework collision assertions
qapi: Remove outdated tests related to QMP/branch collisions
qapi: Add positive tests to qapi-schema-test
qapi: Simplify visiting of alternate types
qapi: Fix alternates that accept 'number' but not 'int'
qapi: Remove dead visitor code
qapi: Plug leaks in test-qmp-*
qapi: Simplify error testing in test-qmp-*
qapi: Test failure in middle of array parse
qapi: More tests of input arrays
qapi: Simplify visits of optional fields

 docs/qapi-code-gen.txt                         |  31 ++-
 include/qapi/visitor-impl.h                    |  27 ++-
 include/qapi/visitor.h                         |  22 +-
 qapi/introspect.json                           |  22 +-
 qapi/opts-visitor.c                            |   2 +-
 qapi/qapi-visit-core.c                         | 141 +++++--------
 qapi/qmp-input-visitor.c                       |  11 +-
 qapi/string-input-visitor.c                    |   3 +-
 scripts/qapi-introspect.py                     |  17 +-
 scripts/qapi-types.py                          |  70 ++-----
 scripts/qapi-visit.py                          |  27 ++-
 scripts/qapi.py                                | 116 ++++++-----
 tests/Makefile                                 |   3 -
 tests/qapi-schema/alternate-empty.out          |   1 -
 tests/qapi-schema/flat-union-clash-branch.err  |   0
 tests/qapi-schema/flat-union-clash-branch.exit |   1 -
 tests/qapi-schema/flat-union-clash-branch.json |  18 --
 tests/qapi-schema/flat-union-clash-branch.out  |  14 --
 tests/qapi-schema/flat-union-clash-type.err    |   1 -
 tests/qapi-schema/flat-union-clash-type.exit   |   1 -
 tests/qapi-schema/flat-union-clash-type.json   |  14 --
 tests/qapi-schema/flat-union-clash-type.out    |   0
 tests/qapi-schema/qapi-schema-test.json        |  26 ++-
 tests/qapi-schema/qapi-schema-test.out         |  46 ++++-
 tests/qapi-schema/union-clash-data.out         |   1 +
 tests/qapi-schema/union-clash-type.err         |   1 -
 tests/qapi-schema/union-clash-type.exit        |   1 -
 tests/qapi-schema/union-clash-type.json        |   9 -
 tests/qapi-schema/union-clash-type.out         |   0
 tests/qapi-schema/union-empty.out              |   1 +
 tests/test-qmp-input-strict.c                  | 108 +++-------
 tests/test-qmp-input-visitor.c                 | 268 +++++++++++--------------
 tests/test-qmp-output-visitor.c                | 153 ++++----------
 tests/test-visitor-serialization.c             |  76 ++-----
 34 files changed, 503 insertions(+), 729 deletions(-)
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.exit
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.json
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.out
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.exit
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.json
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.out
 delete mode 100644 tests/qapi-schema/union-clash-type.err
 delete mode 100644 tests/qapi-schema/union-clash-type.exit
 delete mode 100644 tests/qapi-schema/union-clash-type.json
 delete mode 100644 tests/qapi-schema/union-clash-type.out
Signature · 1 line
2.4.3
author
#1 lore raw

PATCH v8 1/17 qapi: Use generated TestStruct machinery in tests

Commit d88f5fd and friends first introduced the various test-qmp-* tests in 2011, with duplicated hand-rolled TestStruct machinery, to make sure the qapi visitor interface was tested. Later, commit 4f193e3 in 2013 added a .json file for further testing use by the files, but without consolidating any of the existing hand-rolled visitors. And with four copies, subtle differences have crept in, between the tests themselves (mainly whitespace differences, but also a question of whether to use NULL or "TestStruct" when calling visit_start_struct()) and from what the generator produces (the hand-rolled versions did not cater to partially-allocated objects, because they did not have a deallocation usage).

Of course, just because the visitor interface is tested does not mean it is a sane interface; and future patches will be changing some of the visitor contracts. Rather than having to duplicate the cleanup work in each copy of the TestStruct visitor, and keep each hand-rolled copy in sync with what the generator supplies, we might as well just test what the generator should give us in the first place.

---
v8: improve commit message
v7: rebase on top of subset B v9; defer unrelated trailing whitespace cleanups to later in series
v6: new patch

---
 tests/qapi-schema/qapi-schema-test.json |  6 +++-
 tests/qapi-schema/qapi-schema-test.out  |  5 +++
 tests/test-qmp-input-strict.c           | 35 --------------------
 tests/test-qmp-input-visitor.c          | 34 -------------------
 tests/test-qmp-output-visitor.c         | 58 ---------------------------------
 tests/test-visitor-serialization.c      | 34 -------------------
 6 files changed, 10 insertions(+), 162 deletions(-)
tests/qapi-schema/qapi-schema-test.json +5 -1
index 48e104b..44638da 100644
@@ -3,6 +3,9 @@
 # This file is a stress test of supported qapi constructs that must
 # parse and compile correctly.
 
+{ 'struct': 'TestStruct',
+  'data': { 'integer': 'int', 'boolean': 'bool', 'string': 'str' } }
+
 # for testing enums
 { 'struct': 'NestedEnumsOne',
   'data': { 'enum1': 'EnumOne',   # Intentional forward reference
@@ -46,7 +49,8 @@
 
 # dummy struct to force generation of array types not otherwise mentioned
 { 'struct': 'ForceArrays',
-  'data': { 'unused1':['UserDefOne'], 'unused2':['UserDefTwo'] } }
+  'data': { 'unused1':['UserDefOne'], 'unused2':['UserDefTwo'],
+            'unused3':['TestStruct'] } }
 
 # for testing unions
 # Among other things, test that a name collision between branches does
tests/qapi-schema/qapi-schema-test.out +5 -0
index a7e9aab..e20a823 100644
@@ -92,6 +92,7 @@ object EventStructOne
 object ForceArrays
     member unused1: UserDefOneList optional=False
     member unused2: UserDefTwoList optional=False
+    member unused3: TestStructList optional=False
 enum MyEnum []
 object NestedEnumsOne
     member enum1: EnumOne optional=False
@@ -100,6 +101,10 @@ object NestedEnumsOne
     member enum4: EnumOne optional=True
 enum QEnumTwo ['value1', 'value2']
     prefix QENUM_TWO
+object TestStruct
+    member integer: int optional=False
+    member boolean: bool optional=False
+    member string: str optional=False
 object UserDefA
     member boolean: bool optional=False
     member a_b: int optional=True
tests/test-qmp-input-strict.c +0 -35
index 53a7693..b44184f 100644
@@ -89,41 +89,6 @@ static Visitor *validate_test_init_raw(TestInputVisitorData *data,
     return v;
 }
 
-typedef struct TestStruct
-{
-    int64_t integer;
-    bool boolean;
-    char *string;
-} TestStruct;
-
-static void visit_type_TestStruct(Visitor *v, TestStruct **obj,
-                                  const char *name, Error **errp)
-{
-    Error *err = NULL;
-
-    visit_start_struct(v, (void **)obj, "TestStruct", name, sizeof(TestStruct),
-                       &err);
-    if (err) {
-        goto out;
-    }
-
-    visit_type_int(v, &(*obj)->integer, "integer", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_bool(v, &(*obj)->boolean, "boolean", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_str(v, &(*obj)->string, "string", &err);
-
-out_end:
-    error_propagate(errp, err);
-    err = NULL;
-    visit_end_struct(v, &err);
-out:
-    error_propagate(errp, err);
-}
 
 static void test_validate_struct(TestInputVisitorData *data,
                                   const void *unused)
tests/test-qmp-input-visitor.c +0 -34
index de65982..3f6bc4d 100644
@@ -185,40 +185,6 @@ static void test_visitor_in_enum(TestInputVisitorData *data,
     data->qiv = NULL;
 }
 
-typedef struct TestStruct
-{
-    int64_t integer;
-    bool boolean;
-    char *string;
-} TestStruct;
-
-static void visit_type_TestStruct(Visitor *v, TestStruct **obj,
-                                  const char *name, Error **errp)
-{
-    Error *err = NULL;
-
-    visit_start_struct(v, (void **)obj, "TestStruct", name, sizeof(TestStruct),
-                       &err);
-    if (err) {
-        goto out;
-    }
-    visit_type_int(v, &(*obj)->integer, "integer", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_bool(v, &(*obj)->boolean, "boolean", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_str(v, &(*obj)->string, "string", &err);
-
-out_end:
-    error_propagate(errp, err);
-    err = NULL;
-    visit_end_struct(v, &err);
-out:
-    error_propagate(errp, err);
-}
 
 static void test_visitor_in_struct(TestInputVisitorData *data,
                                    const void *unused)
tests/test-qmp-output-visitor.c +0 -58

70 lines not on this page · raw message

tests/test-visitor-serialization.c +0 -34
index 634563b..c024e5e 100644
@@ -186,40 +186,6 @@ static void visit_primitive_list(Visitor *v, void **native, Error **errp)
     }
 }
 
-typedef struct TestStruct
-{
-    int64_t integer;
-    bool boolean;
-    char *string;
-} TestStruct;
-
-static void visit_type_TestStruct(Visitor *v, TestStruct **obj,
-                                  const char *name, Error **errp)
-{
-    Error *err = NULL;
-
-    visit_start_struct(v, (void **)obj, NULL, name, sizeof(TestStruct), &err);
-    if (err) {
-        goto out;
-    }
-
-    visit_type_int(v, &(*obj)->integer, "integer", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_bool(v, &(*obj)->boolean, "boolean", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_str(v, &(*obj)->string, "string", &err);
-
-out_end:
-    error_propagate(errp, err);
-    err = NULL;
-    visit_end_struct(v, &err);
-out:
-    error_propagate(errp, err);
-}
 
 static TestStruct *struct_create(void)
 {
Signature · 1 line
2.4.3
author
#2 lore raw

PATCH v8 2/17 qapi: Strengthen test of TestStructList

In reply to Eric Blake

Make each list element different, to ensure that order is preserved, and use the generated free function instead of hand-rolling our own to ensure (under valgrind) that the list is properly cleaned.

---
v8: no change
v7: new patch

---
 tests/test-qmp-output-visitor.c | 18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)
tests/test-qmp-output-visitor.c +7 -11
index baf58dc..9364843 100644
@@ -283,7 +283,7 @@ static void test_visitor_out_struct_errors(TestOutputVisitorData *data,
 static void test_visitor_out_list(TestOutputVisitorData *data,
                                   const void *unused)
 {
-    char *value_str = (char *) "list value";
+    const char *value_str = "list value";
     TestStructList *p, *head = NULL;
     const int max_items = 10;
     bool value_bool = true;
@@ -294,12 +294,13 @@ static void test_visitor_out_list(TestOutputVisitorData *data,
     QList *qlist;
     int i;
 
+    /* Build the list in reverse order... */
     for (i = 0; i < max_items; i++) {
         p = g_malloc0(sizeof(*p));
         p->value = g_malloc0(sizeof(*p->value));
-        p->value->integer = value_int;
+        p->value->integer = value_int + (max_items - i - 1);
         p->value->boolean = value_bool;
-        p->value->string = value_str;
+        p->value->string = g_strdup(value_str);
 
         p->next = head;
         head = p;
@@ -315,6 +316,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data,
     qlist = qobject_to_qlist(obj);
     g_assert(!qlist_empty(qlist));
 
+    /* ...and ensure that the visitor sees it in order */
     i = 0;
     QLIST_FOREACH_ENTRY(qlist, entry) {
         QDict *qdict;
@@ -322,7 +324,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data,
         g_assert(qobject_type(entry->value) == QTYPE_QDICT);
         qdict = qobject_to_qdict(entry->value);
         g_assert_cmpint(qdict_size(qdict), ==, 3);
-        g_assert_cmpint(qdict_get_int(qdict, "integer"), ==, value_int);
+        g_assert_cmpint(qdict_get_int(qdict, "integer"), ==, value_int + i);
         g_assert_cmpint(qdict_get_bool(qdict, "boolean"), ==, value_bool);
         g_assert_cmpstr(qdict_get_str(qdict, "string"), ==, value_str);
         i++;
@@ -330,13 +332,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data,
     g_assert_cmpint(i, ==, max_items);
 
     QDECREF(qlist);
-
-    for (p = head; p;) {
-        TestStructList *tmp = p->next;
-        g_free(p->value);
-        g_free(p);
-        p = tmp;
-    }
+    qapi_free_TestStructList(head);
 }
 
 static void test_visitor_out_list_qapi_free(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
author
#3 lore raw

PATCH v8 3/17 qapi: Provide nicer array names in introspection

In reply to Eric Blake

For the sake of humans reading introspection output, it is nice to have the name of implicit array types be recognizable as arrays of the underlying type. However, while this patch allows humans to skip from a command with return type "[123]" straight to the definition of type "123" without having to first inspect type "[123]", document that this shortcut should not be taken by client apps.

This makes the resulting introspection string slightly larger by default (just over 200 bytes), but it's in the noise (less than 0.3% of the overall 70k size of 'query-qmp-capabilities').

---
v8: commit message stats on the size increase
v7: no change
v6: no change

---
 docs/qapi-code-gen.txt     | 7 +++++--
 scripts/qapi-introspect.py | 8 +++++---
 2 files changed, 10 insertions(+), 5 deletions(-)
docs/qapi-code-gen.txt +5 -2
index 4d8c2fc..ba29bc6 100644
@@ -662,11 +662,14 @@ Example: the SchemaInfo for BlockRef from section Alternate types
 
 The SchemaInfo for an array type has meta-type "array", and variant
 member "element-type", which names the array's element type.  Array
-types are implicitly defined.
+types are implicitly defined.  For convenience, the array's name may
+resemble the element type; however, clients should examine member
+"element-type" instead of making assumptions based on parsing member
+"name".
 
 Example: the SchemaInfo for ['str']
 
-    { "name": "strList", "meta-type": "array",
+    { "name": "[str]", "meta-type": "array",
       "element-type": "str" }
 
 The SchemaInfo for an enumeration type has meta-type "enum" and
scripts/qapi-introspect.py +5 -3
index c0dad66..64f2cd0 100644
@@ -107,10 +107,12 @@ const char %(c_name)s[] = %(c_string)s;
         # characters.
         if isinstance(typ, QAPISchemaBuiltinType):
             return typ.name
+        if isinstance(typ, QAPISchemaArrayType):
+            return '[' + self._use_type(typ.element_type) + ']'
         return self._name(typ.name)
 
     def _gen_json(self, name, mtype, obj):
-        if mtype != 'command' and mtype != 'event' and mtype != 'builtin':
+        if mtype not in ('command', 'event', 'builtin', 'array'):
             name = self._name(name)
         obj['name'] = name
         obj['meta-type'] = mtype
@@ -136,8 +138,8 @@ const char %(c_name)s[] = %(c_string)s;
         self._gen_json(name, 'enum', {'values': values})
 
     def visit_array_type(self, name, info, element_type):
-        self._gen_json(name, 'array',
-                       {'element-type': self._use_type(element_type)})
+        element = self._use_type(element_type)
+        self._gen_json('[' + element + ']', 'array', {'element-type': element})
 
     def visit_object_type_flat(self, name, info, members, variants):
         obj = {'members': [self._gen_member(m) for m in members]}
Signature · 1 line
2.4.3
author
#4 lore raw

PATCH v8 4/17 qapi-introspect: Guarantee particular sorting

In reply to Eric Blake

Sorting the values of an enum makes it easier to look up whether a particular value is present, via binary rather than linear search (probably most visible with QKeyCode, which has grown over several releases). Additionally, QMP clients need not know which C value is associated with an enum name, so sorting is an effective way to hide that non-ABI aspect of qapi.

While we are at it, it is also easy to sort the members and variants of objects, to allow for a similar binary search (although less compelling, as any struct with that many members should arguably be broken into hierarchical substructs), and equally valid since JSON objects have no specific order in which keys must appear. There is no trivial or obvious way to sort the types of an alternate, so that is left unchanged.

However, the overall SchemaInfo array remains unsorted. It might make sense to sort with 'meta-type' as a primary key and 'name' as a secondary key, but it is not obvious that this will provide benefits to end-user clients (we allow mutually recursive types, so there is no posible topological sorting where a single pass over the array could resolve all types, and while binary searches could be made possible by sorting, it would be even more efficient for clients to read the array into a hashtable for O(1) rather than O(log n) random-access lookups, at which point pre-sorting is wasted effort).

Document these conventions, so that clients will know what can and cannot be relied on.

---
TODO: should the documentation mention that the sorting is done in the C locale? Is there anything required to ensure that python sorts sanely (ie. that the choice of locale while building doesn't cause inadvertent sorting differences such as turning on case-insensitivity)?

v8: no change
v7: tweak commit wording
v6: no change from v5

---
 docs/qapi-code-gen.txt     | 21 +++++++++++++++++----
 qapi/introspect.json       | 22 +++++++++++++++++-----
 scripts/qapi-introspect.py |  9 ++++++---
 3 files changed, 40 insertions(+), 12 deletions(-)
docs/qapi-code-gen.txt +17 -4
index ba29bc6..163f547 100644
@@ -516,6 +516,13 @@ query-qmp-schema.  QGA currently doesn't support introspection.
 
 query-qmp-schema returns a JSON array of SchemaInfo objects.  These
 objects together describe the wire ABI, as defined in the QAPI schema.
+There is no specified order to the SchemaInfo objects returned; a
+client must search for a particular name and meta-type throughout the
+entire array to learn more about that name.  For now, the name for
+each SchemaInfo is unique thanks to qapi naming conventions; however
+this may be changed in the future (such as allowing a command and an
+event with the same name), so it is important that the client check
+for the desired meta-type.
 
 However, the SchemaInfo can't reflect all the rules and restrictions
 that apply to QMP.  It's interface introspection (figuring out what's
@@ -596,7 +603,8 @@ any.  Each element is a JSON object with members "name" (the member's
 name), "type" (the name of its type), and optionally "default".  The
 member is optional if "default" is present.  Currently, "default" can
 only have value null.  Other values are reserved for future
-extensions.
+extensions.  The "members" array is sorted by "name", so that clients
+can use a binary search to see if a particular member is supported.
 
 Example: the SchemaInfo for MyType from section Struct types
 
@@ -610,7 +618,9 @@ Example: the SchemaInfo for MyType from section Struct types
 "variants" is a JSON array describing the object's variant members.
 Each element is a JSON object with members "case" (the value of type
 tag this element applies to) and "type" (the name of an object type
-that provides the variant members for this type tag value).
+that provides the variant members for this type tag value).  The
+"variants" array is sorted by "case", so it appears in the same
+order as the enum type matching "tag".
 
 Example: the SchemaInfo for flat union BlockdevOptions from section
 Union types
@@ -651,7 +661,8 @@ Union types
 The SchemaInfo for an alternate type has meta-type "alternate", and
 variant member "members".  "members" is a JSON array.  Each element is
 a JSON object with member "type", which names a type.  Values of the
-alternate type conform to exactly one of its member types.
+alternate type conform to exactly one of its member types.  There is
+no guarantee on the order in which "members" will be listed.
 
 Example: the SchemaInfo for BlockRef from section Alternate types
 
@@ -673,7 +684,9 @@ Example: the SchemaInfo for ['str']
       "element-type": "str" }
 
 The SchemaInfo for an enumeration type has meta-type "enum" and
-variant member "values".
+variant member "values".  The values are listed in sorted order,
+so clients can use a binary search to see if a particular value
+is present.
 
 Example: the SchemaInfo for MyEnum from section Enumeration types
 
qapi/introspect.json +17 -5
index cc50dc6..71632af 100644
@@ -25,6 +25,11 @@
 # Returns: array of @SchemaInfo, where each element describes an
 # entity in the ABI: command, event, type, ...
 #
+# The order of the various SchemaInfo is unspecified.  For now, the
+# name of each SchemaInfo is unique regardless of meta-type, but to be
+# safe, clients should check that a given name has the expected
+# meta-type.
+#
 # Note: the QAPI schema is also used to help define *internal*
 # interfaces, by defining QAPI types.  These are not part of the QMP
 # wire ABI, and therefore not returned by this command.
@@ -78,7 +83,8 @@
 #        Commands and events have the name defined in the QAPI schema.
 #        Unlike command and event names, type names are not part of
 #        the wire ABI.  Consequently, type names are meaningless
-#        strings here.
+#        strings here.  Although all names are currently unique
+#        regardless of @meta-type, clients should not rely on this.
 #
 # All references to other SchemaInfo are by name.
 #
@@ -130,7 +136,9 @@
 #
 # Additional SchemaInfo members for meta-type 'enum'.
 #
-# @values: the enumeration type's values.
+# @values: the enumeration type's values.  The values are sorted, so
+#          clients can use a binary search to see if a particular value
+#          is present.
 #
 # Values of this type are JSON string on the wire.
 #
@@ -158,14 +166,18 @@
 #
 # Additional SchemaInfo members for meta-type 'object'.
 #
-# @members: the object type's (non-variant) members.
+# @members: the object type's (non-variant) members.  The members are
+#           sorted by name, so clients can use a binary search to see
+#           if a given member is present.
 #
 # @tag: #optional the name of the member serving as type tag.
 #       An element of @members with this name must exist.
 #
 # @variants: #optional variant members, i.e. additional members that
 #            depend on the type tag's value.  Present exactly when
-#            @tag is present.
+#            @tag is present.  The variants are sorted by case, which
+#            means they appear in the same order as the values of the
+#            enum type of the @tag.
 #
 # Values of this type are JSON object on the wire.
 #
@@ -219,7 +231,7 @@
 #
 # Additional SchemaInfo members for meta-type 'alternate'.
 #
-# @members: the alternate type's members.
+# @members: the alternate type's members, in no particular order.
 #           The members' wire encoding is distinct, see
 #           docs/qapi-code-gen.txt section Alternate types.
 #
scripts/qapi-introspect.py +6 -3
index 64f2cd0..6a5a843 100644
@@ -10,6 +10,7 @@
 # See the COPYING file in the top-level directory.
 
 from qapi import *
+from operator import attrgetter
 
 
 # Caveman's json.dumps() replacement (we're stuck at Python 2.4)
@@ -126,7 +127,8 @@ const char %(c_name)s[] = %(c_string)s;
 
     def _gen_variants(self, tag_name, variants):
         return {'tag': tag_name,
-                'variants': [self._gen_variant(v) for v in variants]}
+                'variants': [self._gen_variant(v) for v in
+                             sorted(variants, key=attrgetter('name'))]}
 
     def _gen_variant(self, variant):
         return {'case': variant.name, 'type': self._use_type(variant.type)}
@@ -135,14 +137,15 @@ const char %(c_name)s[] = %(c_string)s;
         self._gen_json(name, 'builtin', {'json-type': json_type})
 
     def visit_enum_type(self, name, info, values, prefix):
-        self._gen_json(name, 'enum', {'values': values})
+        self._gen_json(name, 'enum', {'values': sorted(values)})
 
     def visit_array_type(self, name, info, element_type):
         element = self._use_type(element_type)
         self._gen_json('[' + element + ']', 'array', {'element-type': element})
 
     def visit_object_type_flat(self, name, info, members, variants):
-        obj = {'members': [self._gen_member(m) for m in members]}
+        obj = {'members': [self._gen_member(m) for m in
+                           sorted(members, key=attrgetter('name'))]}
         if variants:
             obj.update(self._gen_variants(variants.tag_member.name,
                                           variants.variants))
Signature · 1 line
2.4.3
#5 lore raw

Re: [Qemu-devel] [PATCH v8 04/17] qapi-introspect: Guarantee particular sorting

For now, only high-level review.

The main cost of sorting is interface complexity: we need to specify which things are sorted, and what the sorting order is (see your TODO below). Once we've done so, we can't go back.

There's also implementation complexity, but your patch shows it's low enough to be ignored.

Cost needs be justified by benefits. Let's have a look at the benefits.

Quoted 4 lines · Eric Blake

Sorting the values of an enum makes it easier to look up whether a particular value is present, via binary rather than linear search (probably most visible with QKeyCode, which has grown over several releases).

Making the interface guarantee "sorted" saves a client that wants it sorted (say for binary search) the trouble of sorting itself. We sort at QEMU compile time instead of client run time.

My qmp-introspect.c has 48 SchemaInfoEnum, ranging from one to 125 members. Histogram:

#enums with this #members

      6 1
     11 2
      8 3
      9 4
      2 5
      1 6
      3 7
      1 8
      2 9
      1 15
      1 19
      1 27
      1 43
      1 125

Mean is less than eight. Median is *three*.

Sorting these member arrays in the client won't have a noticeable performance impact. Heck, I suspect even using linear search wouldn't! Therefore, we can't really claim client performance as a benefit.

We might be able to claim reduced client complexity as a benefit, but I'm rather skeptical. When your search space has a dozen members, you better stick to simple search techniques. Linear search in array or list and binary search in array look adequate, hash table already approaches overkill, and anything fancier definitely is. Of these, only binary search profits from a "sorted" guarantee. But when you got data in an array already, all it takes to sort it is a call to qsort() and a comparison function. Ten straightforward lines of code, tops. Less in a language that isn't quite as primitive as C. Not much of a benefit, I'm afraid.

Quoted 3 lines

Additionally, QMP clients need not know which
C value is associated with an enum name, so sorting is an effective way to hide that non-ABI aspect of qapi.

Sorting indeed hides the implementation detail of how enumerations are encoded in the server. However, I can't see what would tempt clients into relying on it. I can see for type names, and that's why we hide them (commit 1a9a507).

One more potential benefit: when the schema changes in a way that affects only order in introspection, sorting can hide the changes from clients. Can't think of a practical use of it, though.

Quoted 6 lines

While we are at it, it is also easy to sort the members and variants of objects, to allow for a similar binary search (although less compelling, as any struct with that many members should arguably be broken into hierarchical substructs), and equally valid since JSON objects have no specific order in which keys must appear.

My qmp-introspect.c has 48 SchemaInfoObject, ranging from zero to 27 members. Histogram:

#objs with this #members

      3 0
    102 2
     58 3
     28 4
     14 5
     17 6
     15 7
      3 8
      2 9
      7 10
      2 11
      3 12
      1 13
      2 14
      1 15
      1 16
      1 27

Mean is 9.5, median is 9.

The variants are also enums, and therefore can't be any worse than enums.

Again, client performance is hardly a benefit, and client complexity isn't particularly convincing, either.

Quoted 2 lines

There is no trivial or obvious way to sort the types of
an alternate, so that is left unchanged.

In the schema, an alternate's member has a name, a type and nothing else, so that's what we could use to sort. The name isn't visible in introspection, and the type is masked. Sorting by either hides schema change from the client, but isn't of much use to the client otherwise.

If you want to let the client use binary search without having to sort itself, you obviously have to sort by masked type.

My qmp-introspect.c has *two* alternate types, each with two members.

Quoted 10 lines

However, the overall SchemaInfo array remains unsorted. It might make sense to sort with 'meta-type' as a primary key and 'name' as a secondary key, but it is not obvious that this will provide benefits to end-user clients (we allow mutually recursive types, so there is no posible topological sorting where a single pass over the array could resolve all types, and while binary searches could be made possible by sorting, it would be even more efficient for clients to read the array into a hashtable for O(1) rather than O(log n) random-access lookups, at which point pre-sorting is wasted effort).

My qmp-introspect.c has:

      2 alternate
     55 array
      5 builtin
    126 command
     48 enum
     35 event
    260 object
    -------------
    531 total

Since they all share the same entity name space, I'd use a single hash table to map from name to SchemaInfo, just like qapi.py's QAPISchema does.

Since we visit stuff in a defined order, qmp-introspect.c's order is stable. The order just isn't particularly useful for clients, let alone specified.

Quoted 8 lines

Document these conventions, so that clients will know what can and cannot be relied on.

---
TODO: should the documentation mention that the sorting is done in the C locale?

I'd assume C locale because we're sorting plain ASCII strings. But spelling it out can't hurt.

Quoted 4 lines

Is there anything required to ensure that python
sorts sanely (ie. that the choice of locale while building doesn't cause inadvertent sorting differences such as turning on case-insensitivity)?

Good question.

Quoted 14 lines

v8: no change
v7: tweak commit wording
v6: no change from v5

---
 docs/qapi-code-gen.txt     | 21 +++++++++++++++++----
 qapi/introspect.json       | 22 +++++++++++++++++-----
 scripts/qapi-introspect.py |  9 ++++++---
 3 files changed, 40 insertions(+), 12 deletions(-)
docs/qapi-code-gen.txt +0 -0
index ba29bc6..163f547 100644

docs/qapi-code-gen.txt query-qmp-schema. QGA currently doesn't support introspection. · line 525

Quoted 10 lines
docs/qapi-code-gen.txt +7 -0
@@ -516,6 +516,13 @@ query-qmp-schema.  QGA currently doesn't support introspection.
 
 query-qmp-schema returns a JSON array of SchemaInfo objects.  These
 objects together describe the wire ABI, as defined in the QAPI schema.
+There is no specified order to the SchemaInfo objects returned; a
+client must search for a particular name and meta-type throughout the
+entire array to learn more about that name.  For now, the name for
+each SchemaInfo is unique thanks to qapi naming conventions; however
+this may be changed in the future (such as allowing a command and an
+event with the same name), so it is important that the client check
+for the desired meta-type.

I feel separate name spaces aren't necessary and would be confusing, and I'd be prepared to cast "single entity name space" in stone now.

docs/qapi-code-gen.txt any. Each element is a JSON object with members "name" (the member's · line 610

Quoted 13 lines
docs/qapi-code-gen.txt +2 -1
 However, the SchemaInfo can't reflect all the rules and restrictions
 that apply to QMP.  It's interface introspection (figuring out what's
@@ -596,7 +603,8 @@ any.  Each element is a JSON object with members "name" (the member's
 name), "type" (the name of its type), and optionally "default".  The
 member is optional if "default" is present.  Currently, "default" can
 only have value null.  Other values are reserved for future
-extensions.
+extensions.  The "members" array is sorted by "name", so that clients
+can use a binary search to see if a particular member is supported.
 
 Example: the SchemaInfo for MyType from section Struct types
 

[...]

This all boils down to whether the increase in interface specification complexity is justified by the benefits. The cost is small, but I'm having a hard time seeing the benefits, to be honest.

Am I missing something?

author
#6 lore raw

Re: [Qemu-devel] [PATCH v8 04/17] qapi-introspect: Guarantee particular sorting

Quoted 15 lines · Markus Armbruster

For now, only high-level review.

The main cost of sorting is interface complexity: we need to specify which things are sorted, and what the sorting order is (see your TODO below). Once we've done so, we can't go back.

There's also implementation complexity, but your patch shows it's low enough to be ignored.

Cost needs be justified by benefits. Let's have a look at the benefits.

My qmp-introspect.c has 48 SchemaInfoEnum, ranging from one to 125 members. Histogram:

Quoted 7 lines

#enums with this #members

      1 27
      1 43
      1 125

Mean is less than eight. Median is *three*.

Binary searches have more overhead on small lists; you don't get the benefits of the better algorithm until you overcome the initial hit of more expensive lookups (bsearch() has to use a callback function, which is intrinsically slower than doing direct comparison; the speedups are only present if the search set is large enough that you are able to skip enough comparisons to overcome the higher cost of the extra indirection). So leaving enums unsorted is unlikely to slow clients down (clients will have to do a linear search, but not the end of the world since we have few large lists, and even among those, the size is unlikely to grow much further).

Quoted 4 lines

Sorting these member arrays in the client won't have a noticeable performance impact. Heck, I suspect even using linear search wouldn't! Therefore, we can't really claim client performance as a benefit.

So I'm inclined to agree with your conclusion.

Quoted 8 lines
Quoted 3 lines

Additionally, QMP clients need not know which
C value is associated with an enum name, so sorting is an effective way to hide that non-ABI aspect of qapi.

Sorting indeed hides the implementation detail of how enumerations are encoded in the server. However, I can't see what would tempt clients into relying on it. I can see for type names, and that's why we hide them (commit 1a9a507).

You have a point - even if a client figures out what integer value an enum name maps to, it doesn't really help, because the client never sends the integer over the wire.

Quoted 4 lines

One more potential benefit: when the schema changes in a way that affects only order in introspection, sorting can hide the changes from clients. Can't think of a practical use of it, though.

Because we send names, not numbers, over the wire, we can insert a new enum member in slot 0. Clients blindly doing strcmp(enum[0], "magic") will get thrown off if "magic" moves to a different position - but this is true whether or not we sort. So the only working clients are those that check every enum position, not just enum[0]. So I agree - broken clients are broken whether or not we sort, and working clients don't miss anything whether or not we sort.

Quoted 8 lines

My qmp-introspect.c has 48 SchemaInfoObject, ranging from zero to 27 members. Histogram:

#objs with this #members

      1 15
      1 16
      1 27

Even less than the largest enum (and large structs are already unwieldy, compared to making trees of substructs, so we're less likely to make them too much bigger).

Quoted 16 lines

Since we visit stuff in a defined order, qmp-introspect.c's order is stable. The order just isn't particularly useful for clients, let alone specified.

Quoted 8 lines

Document these conventions, so that clients will know what can and cannot be relied on.

---
TODO: should the documentation mention that the sorting is done in the C locale?

I'd assume C locale because we're sorting plain ASCII strings. But spelling it out can't hurt.

Quoted 6 lines
Quoted 4 lines

Is there anything required to ensure that python
sorts sanely (ie. that the choice of locale while building doesn't cause inadvertent sorting differences such as turning on case-insensitivity)?

Good question.

And my biggest worry.

Quoted 15 lines

docs/qapi-code-gen.txt query-qmp-schema. QGA currently doesn't support introspection. · line 525

Quoted 12 lines

+++ b/docs/qapi-code-gen.txt

docs/qapi-code-gen.txt +7 -0
@@ -516,6 +516,13 @@ query-qmp-schema.  QGA currently doesn't support introspection.
 
 query-qmp-schema returns a JSON array of SchemaInfo objects.  These
 objects together describe the wire ABI, as defined in the QAPI schema.
+There is no specified order to the SchemaInfo objects returned; a
+client must search for a particular name and meta-type throughout the
+entire array to learn more about that name.  For now, the name for
+each SchemaInfo is unique thanks to qapi naming conventions; however
+this may be changed in the future (such as allowing a command and an
+event with the same name), so it is important that the client check
+for the desired meta-type.

I feel separate name spaces aren't necessary and would be confusing, and I'd be prepared to cast "single entity name space" in stone now.

Sounds fair to me.

Quoted 5 lines

This all boils down to whether the increase in interface specification complexity is justified by the benefits. The cost is small, but I'm having a hard time seeing the benefits, to be honest.

Am I missing something?

No, I think you gave a fair review. I put this patch out there for discussion and to see how hard it would be; but I'm not strongly tied to it (if I have to pick only one patch out of 3/17 and 4/17, I'd rather see 3/17 go in). So now that we've looked at the benefits (or rather, that they aren't very strong), I'm okay with the idea of making my v9 spin of this series use an alternative patch that _just_ documents the "single entity name space" and the fact that clients cannot rely on any ordering (we don't sort - we may be deterministic, but we make no guarantees on what that order is, and we are free to change the order in the future).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#7 lore raw

PATCH v8 5/17 qapi: Track simple union tag in object.local_members

In reply to Eric Blake

We were previously creating all unions with an empty list for local_members. However, it will make it easier to unify struct and union generation if we include the generated tag member in local_members. That way, we can have a common code pattern:
visit the base (if any), visit the local members (if any), visit the variants (if any). The local_members of a flat union remains empty (because the discriminator is already visited as part of the base).

The various front end entities then map as follows:
struct: optional base, optional local_members, no variants simple union: no base, one-element local_members, variants with tag_member
from local_members
flat union: base, no local_members, variants with tag_member from base
alternate: no base, no local_members, variants

With the new local members, we require a bit of finesse to avoid assertions in the clients. No change to generated code.

---
v8: new patch

---
 scripts/qapi-types.py                  |  4 +++-
 scripts/qapi-visit.py                  |  4 +++-
 scripts/qapi.py                        | 14 ++++++++++----
 tests/qapi-schema/qapi-schema-test.out |  2 ++
 tests/qapi-schema/union-clash-data.out |  1 +
 tests/qapi-schema/union-empty.out      |  1 +
 6 files changed, 20 insertions(+), 6 deletions(-)
scripts/qapi-types.py +3 -1
index b37900f..a6bf95d 100644
@@ -269,7 +269,9 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def visit_object_type(self, name, info, base, members, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
         if variants:
-            assert not members      # not implemented
+            if members:
+                assert len(members) == 1
+                assert members[0] == variants.tag_member
             self.decl += gen_union(name, base, variants)
         else:
             self.decl += gen_struct(name, base, members)
scripts/qapi-visit.py +3 -1
index f40c3c7..318b8e6 100644
@@ -360,7 +360,9 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
     def visit_object_type(self, name, info, base, members, variants):
         self.decl += gen_visit_decl(name)
         if variants:
-            assert not members      # not implemented
+            if members:
+                assert len(members) == 1
+                assert members[0] == variants.tag_member
             self.defn += gen_visit_union(name, base, variants)
         else:
             self.defn += gen_visit_struct(name, base, members)
scripts/qapi.py +10 -4
index 7c50cc4..84ac151 100644
@@ -957,6 +957,9 @@ class QAPISchemaArrayType(QAPISchemaType):
 
 class QAPISchemaObjectType(QAPISchemaType):
     def __init__(self, name, info, base, local_members, variants):
+        # struct has local_members, optional base, and no variants
+        # flat union has base, variants, and no local_members
+        # simple union has local_members, variants, and no base
         QAPISchemaType.__init__(self, name, info)
         assert base is None or isinstance(base, str)
         for m in local_members:
@@ -1048,9 +1051,11 @@ class QAPISchemaObjectTypeVariants(object):
         self.variants = variants
 
     def check(self, schema, members, seen):
-        if self.tag_name:
+        if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
-        else:
+        elif seen:           # simple union
+            assert self.tag_member in seen.itervalues()
+        else:                # alternate
             self.tag_member.check(schema, members, seen)
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
@@ -1270,13 +1275,14 @@ class QAPISchema(object):
         if tag_name:
             variants = [self._make_variant(key, value)
                         for (key, value) in data.iteritems()]
+            members = []
         else:
             variants = [self._make_simple_variant(key, value, info)
                         for (key, value) in data.iteritems()]
             tag_member = self._make_implicit_tag(name, info, variants)
+            members = [tag_member]
         self._def_entity(
-            QAPISchemaObjectType(name, info, base,
-                                 self._make_members(OrderedDict(), info),
+            QAPISchemaObjectType(name, info, base, members,
                                  QAPISchemaObjectTypeVariants(tag_name,
                                                               tag_member,
                                                               variants)))
tests/qapi-schema/qapi-schema-test.out +2 -0
index e20a823..786024e 100644
@@ -132,6 +132,7 @@ object UserDefFlatUnion2
     case value2: UserDefB
     case value3: UserDefA
 object UserDefNativeListUnion
+    member type: UserDefNativeListUnionKind optional=False
     case integer: :obj-intList-wrapper
     case s8: :obj-int8List-wrapper
     case s16: :obj-int16List-wrapper
@@ -187,6 +188,7 @@ object __org.qemu_x-Struct
 object __org.qemu_x-Struct2
     member array: __org.qemu_x-Union1List optional=False
 object __org.qemu_x-Union1
+    member type: __org.qemu_x-Union1Kind optional=False
     case __org.qemu_x-branch: :obj-str-wrapper
 enum __org.qemu_x-Union1Kind ['__org.qemu_x-branch']
 object __org.qemu_x-Union2
tests/qapi-schema/union-clash-data.out +1 -0
index 6277239..cea8551 100644
@@ -2,5 +2,6 @@ object :empty
 object :obj-int-wrapper
     member data: int optional=False
 object TestUnion
+    member type: TestUnionKind optional=False
     case data: :obj-int-wrapper
 enum TestUnionKind ['data']
tests/qapi-schema/union-empty.out +1 -0
index 8b5a7bf..9c89fd1 100644
@@ -1,3 +1,4 @@
 object :empty
 object Union
+    member type: UnionKind optional=False
 enum UnionKind []
Signature · 1 line
2.4.3
#8 lore raw

Re: [Qemu-devel] [PATCH v8 05/17] qapi: Track simple union tag in object.local_members

Quoted 8 lines · Eric Blake

We were previously creating all unions with an empty list for local_members. However, it will make it easier to unify struct and union generation if we include the generated tag member in local_members. That way, we can have a common code pattern:
visit the base (if any), visit the local members (if any), visit the variants (if any). The local_members of a flat union remains empty (because the discriminator is already visited as part of the base).

Keeping the implicit tag implicit by not including it in local_members was a conscious design decision, but if including it makes unifying struct and union into objects easier, go right ahead.

Quoted 28 lines

The various front end entities then map as follows:
struct: optional base, optional local_members, no variants simple union: no base, one-element local_members, variants with tag_member
from local_members
flat union: base, no local_members, variants with tag_member from base
alternate: no base, no local_members, variants

With the new local members, we require a bit of finesse to avoid assertions in the clients. No change to generated code.

---
v8: new patch

---
 scripts/qapi-types.py                  |  4 +++-
 scripts/qapi-visit.py                  |  4 +++-
 scripts/qapi.py                        | 14 ++++++++++----
 tests/qapi-schema/qapi-schema-test.out |  2 ++
 tests/qapi-schema/union-clash-data.out |  1 +
 tests/qapi-schema/union-empty.out      |  1 +
 6 files changed, 20 insertions(+), 6 deletions(-)
scripts/qapi-types.py +0 -0
index b37900f..a6bf95d 100644

scripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 277

Quoted 10 lines
scripts/qapi-types.py +3 -1
@@ -269,7 +269,9 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def visit_object_type(self, name, info, base, members, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
         if variants:
-            assert not members      # not implemented
+            if members:
+                assert len(members) == 1
+                assert members[0] == variants.tag_member
             self.decl += gen_union(name, base, variants)
         else:
             self.decl += gen_struct(name, base, members)

The assertion checks that not passing members to gen_union() won't lose any. Before the patch, we assert there are none. After the patch, we assert there's either none or variants.tag_member. Before ans after, gen_union() takes care of variants.tag_member. Okay.

Let's keep the comment, though. Perhaps like this:

                # Members other than variants.tag_member not implemented
                assert len(members) == 1
                assert members[0] == variants.tag_member

I hope this the whole conditional will eventually be replaced by

           self.decl += gen_object(name, base, members, variants)
Quoted 5 lines
scripts/qapi-visit.py +0 -0
index f40c3c7..318b8e6 100644

scripts/qapi-visit.py class QAPISchemaGenVisitVisitor(QAPISchemaVisitor): · line 368

Quoted 10 lines
scripts/qapi-visit.py +3 -1
@@ -360,7 +360,9 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
     def visit_object_type(self, name, info, base, members, variants):
         self.decl += gen_visit_decl(name)
         if variants:
-            assert not members      # not implemented
+            if members:
+                assert len(members) == 1
+                assert members[0] == variants.tag_member
             self.defn += gen_visit_union(name, base, variants)
         else:
             self.defn += gen_visit_struct(name, base, members)

Likewise, let's keep the comment.

Quoted 18 lines
scripts/qapi.py +3 -0
index 7c50cc4..84ac151 100644
@@ -957,6 +957,9 @@ class QAPISchemaArrayType(QAPISchemaType):
 
 class QAPISchemaObjectType(QAPISchemaType):
     def __init__(self, name, info, base, local_members, variants):
+        # struct has local_members, optional base, and no variants
+        # flat union has base, variants, and no local_members
+        # simple union has local_members, variants, and no base
         QAPISchemaType.__init__(self, name, info)
         assert base is None or isinstance(base, str)
         for m in local_members:
@@ -1048,9 +1051,11 @@ class QAPISchemaObjectTypeVariants(object):
         self.variants = variants
 
     def check(self, schema, members, seen):

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1061

Quoted 10 lines
scripts/qapi.py +4 -2
-        if self.tag_name:
+        if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
-        else:
+        elif seen:           # simple union
+            assert self.tag_member in seen.itervalues()
+        else:                # alternate
             self.tag_member.check(schema, members, seen)
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:

The test for simple union is hackish.

I guess you want to bypass self.tag_member.check() when for simple unions, because it's now checked when QAPISchemaObjectType.check() calls QAPISchemaObjectTypeMember.check() for each of self.local_members.

Could we instead make QAPISchemaAlternateType.check() call QAPISchemaObjectTypeMember.check() for its implicit tag, and drop the else here?

Quoted 7 lines
scripts/qapi.py +1 -0
@@ -1270,13 +1275,14 @@ class QAPISchema(object):
         if tag_name:
             variants = [self._make_variant(key, value)
                         for (key, value) in data.iteritems()]
+            members = []
         else:
             variants = [self._make_simple_variant(key, value, info)

scripts/qapi.py class QAPISchema(object): · line 1288

Quoted 10 lines
scripts/qapi.py +2 -2
                         for (key, value) in data.iteritems()]
             tag_member = self._make_implicit_tag(name, info, variants)
+            members = [tag_member]
         self._def_entity(
-            QAPISchemaObjectType(name, info, base,
-                                 self._make_members(OrderedDict(), info),
+            QAPISchemaObjectType(name, info, base, members,
                                  QAPISchemaObjectTypeVariants(tag_name,
                                                               tag_member,
                                                               variants)))

Two steps squashed together:

1. Simplify _make_members(OrderedDict(), info) to [].

2. Put a simple union's implicit tag_member into the members array passed to QAPISchemaObjectType().

Okay.

[...]

author
#9 lore raw

Re: [Qemu-devel] [PATCH v8 05/17] qapi: Track simple union tag in object.local_members

Quoted 46 lines · Markus Armbruster
Quoted 8 lines · Eric Blake

We were previously creating all unions with an empty list for local_members. However, it will make it easier to unify struct and union generation if we include the generated tag member in local_members. That way, we can have a common code pattern:
visit the base (if any), visit the local members (if any), visit the variants (if any). The local_members of a flat union remains empty (because the discriminator is already visited as part of the base).

Keeping the implicit tag implicit by not including it in local_members was a conscious design decision, but if including it makes unifying struct and union into objects easier, go right ahead.

Quoted 15 lines

The various front end entities then map as follows:
struct: optional base, optional local_members, no variants simple union: no base, one-element local_members, variants with tag_member
from local_members
flat union: base, no local_members, variants with tag_member from base
alternate: no base, no local_members, variants

With the new local members, we require a bit of finesse to avoid assertions in the clients. No change to generated code.

+++ b/scripts/qapi-types.py

scripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 277

Quoted 10 lines
scripts/qapi-types.py +3 -1
@@ -269,7 +269,9 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def visit_object_type(self, name, info, base, members, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
         if variants:
-            assert not members      # not implemented
+            if members:
+                assert len(members) == 1
+                assert members[0] == variants.tag_member
             self.decl += gen_union(name, base, variants)
         else:
             self.decl += gen_struct(name, base, members)

The assertion checks that not passing members to gen_union() won't lose any. Before the patch, we assert there are none. After the patch, we assert there's either none or variants.tag_member. Before ans after, gen_union() takes care of variants.tag_member. Okay.

Quoted 5 lines

Let's keep the comment, though. Perhaps like this:

                # Members other than variants.tag_member not implemented
                assert len(members) == 1
                assert members[0] == variants.tag_member

Indeed, that sounds nicer.

Quoted 4 lines

I hope this the whole conditional will eventually be replaced by

           self.decl += gen_object(name, base, members, variants)

Yes. It goes away in 6/17 for qapi-types (as you already saw), and I have a later patch queued that likewise makes it go away for qapi-visit (but not in subset C or D, since there's still a bit more to clean up there first).

Quoted 14 lines

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1061

Quoted 11 lines

def check(self, schema, members, seen):

scripts/qapi.py +4 -2
-        if self.tag_name:
+        if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
-        else:
+        elif seen:           # simple union
+            assert self.tag_member in seen.itervalues()
+        else:                # alternate
             self.tag_member.check(schema, members, seen)
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:

The test for simple union is hackish.

Quoted 7 lines

I guess you want to bypass self.tag_member.check() when for simple unions, because it's now checked when QAPISchemaObjectType.check() calls QAPISchemaObjectTypeMember.check() for each of self.local_members.

Could we instead make QAPISchemaAlternateType.check() call QAPISchemaObjectTypeMember.check() for its implicit tag, and drop the else here?

I debated about that when writing my patch. Yes, I can make that change (it seems a little bit unclean to be calling self.variants.tag_member.check() in QAPISchemaAlternateType, but not too much worse; and adding a comment might help).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#10 lore raw

PATCH v8 6/17 qapi-types: Consolidate gen_struct() and gen_union()

In reply to Eric Blake

These two methods are now close enough that we can finally merge them, relying on the fact that simple unions now provide a reasonable local_members. Change gen_struct() to gen_object() that handles all forms of QAPISchemaObjectType, and rename and shrink gen_union() to gen_variants() to handle the portion of gen_object() needed when variants are present.

gen_struct_fields() now has a single caller, so it no longer needs an optional parameter; however, I did not choose to inline it into the caller.

No difference to generated code.

---
v8: new patch

---
 scripts/qapi-types.py | 36 +++++++++++-------------------------
 1 file changed, 11 insertions(+), 25 deletions(-)
scripts/qapi-types.py +11 -25
index a6bf95d..403768b 100644
@@ -51,7 +51,7 @@ def gen_struct_field(member):
     return ret
 
 
-def gen_struct_fields(local_members, base=None):
+def gen_struct_fields(local_members, base):
     ret = ''
 
     if base:
@@ -70,7 +70,7 @@ def gen_struct_fields(local_members, base=None):
     return ret
 
 
-def gen_struct(name, base, members):
+def gen_object(name, base, members, variants):
     ret = mcgen('''
 
 struct %(c_name)s {
@@ -79,11 +79,14 @@ struct %(c_name)s {
 
     ret += gen_struct_fields(members, base)
 
+    if variants:
+        ret += gen_variants(variants)
+
     # Make sure that all structs have at least one field; this avoids
     # potential issues with attempting to malloc space for zero-length
     # structs in C, and also incompatibility with C++ (where an empty
     # struct is size 1).
-    if not (base and base.members) and not members:
+    if not (base and base.members) and not members and not variants:
         ret += mcgen('''
     char qapi_dummy_field_for_empty_struct;
 ''')
@@ -140,17 +143,7 @@ const int %(c_name)s_qtypes[QTYPE_MAX] = {
     return ret
 
 
-def gen_union(name, base, variants):
-    ret = mcgen('''
-
-struct %(c_name)s {
-''',
-                c_name=c_name(name))
-    if base:
-        ret += gen_struct_fields([], base)
-    else:
-        ret += gen_struct_field(variants.tag_member)
-
+def gen_variants(variants):
     # FIXME: What purpose does data serve, besides preventing a union that
     # has a branch named 'data'? We use it in qapi-visit.py to decide
     # whether to bypass the switch statement if visiting the discriminator
@@ -159,11 +152,11 @@ struct %(c_name)s {
     # should not be any data leaks even without a data pointer.  Or, if
     # 'data' is merely added to guarantee we don't have an empty union,
     # shouldn't we enforce that at .json parse time?
-    ret += mcgen('''
+    ret = mcgen('''
     union { /* union tag is @%(c_name)s */
         void *data;
 ''',
-                 c_name=c_name(variants.tag_member.name))
+                c_name=c_name(variants.tag_member.name))
 
     for var in variants.variants:
         # Ugly special case for simple union TODO get rid of it
@@ -176,7 +169,6 @@ struct %(c_name)s {
 
     ret += mcgen('''
     } u;
-};
 ''')
 
     return ret
@@ -268,13 +260,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
     def visit_object_type(self, name, info, base, members, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
-        if variants:
-            if members:
-                assert len(members) == 1
-                assert members[0] == variants.tag_member
-            self.decl += gen_union(name, base, variants)
-        else:
-            self.decl += gen_struct(name, base, members)
+        self.decl += gen_object(name, base, members, variants)
         if base:
             self.decl += gen_upcast(name, base)
         self._gen_type_cleanup(name)
@@ -282,7 +268,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def visit_alternate_type(self, name, info, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
         self._fwdefn += gen_alternate_qtypes(name, variants)
-        self.decl += gen_union(name, None, variants)
+        self.decl += gen_object(name, None, [variants.tag_member], variants)
         self.decl += gen_alternate_qtypes_decl(name)
         self._gen_type_cleanup(name)
 
Signature · 1 line
2.4.3
#11 lore raw

Re: [Qemu-devel] [PATCH v8 06/17] qapi-types: Consolidate gen_struct() and gen_union()

Quoted 116 lines · Eric Blake

These two methods are now close enough that we can finally merge them, relying on the fact that simple unions now provide a reasonable local_members. Change gen_struct() to gen_object() that handles all forms of QAPISchemaObjectType, and rename and shrink gen_union() to gen_variants() to handle the portion of gen_object() needed when variants are present.

gen_struct_fields() now has a single caller, so it no longer needs an optional parameter; however, I did not choose to inline it into the caller.

No difference to generated code.

---
v8: new patch

---
 scripts/qapi-types.py | 36 +++++++++++-------------------------
 1 file changed, 11 insertions(+), 25 deletions(-)
scripts/qapi-types.py +10 -24
index a6bf95d..403768b 100644
@@ -51,7 +51,7 @@ def gen_struct_field(member):
     return ret
 
 
-def gen_struct_fields(local_members, base=None):
+def gen_struct_fields(local_members, base):
     ret = ''
 
     if base:
@@ -70,7 +70,7 @@ def gen_struct_fields(local_members, base=None):
     return ret
 
 
-def gen_struct(name, base, members):
+def gen_object(name, base, members, variants):
     ret = mcgen('''
 
 struct %(c_name)s {
@@ -79,11 +79,14 @@ struct %(c_name)s {
 
     ret += gen_struct_fields(members, base)
 
+    if variants:
+        ret += gen_variants(variants)
+
     # Make sure that all structs have at least one field; this avoids
     # potential issues with attempting to malloc space for zero-length
     # structs in C, and also incompatibility with C++ (where an empty
     # struct is size 1).
-    if not (base and base.members) and not members:
+    if not (base and base.members) and not members and not variants:
         ret += mcgen('''
     char qapi_dummy_field_for_empty_struct;
 ''')
@@ -140,17 +143,7 @@ const int %(c_name)s_qtypes[QTYPE_MAX] = {
     return ret
 
 
-def gen_union(name, base, variants):
-    ret = mcgen('''
-
-struct %(c_name)s {
-''',
-                c_name=c_name(name))
-    if base:
-        ret += gen_struct_fields([], base)
-    else:
-        ret += gen_struct_field(variants.tag_member)
-
+def gen_variants(variants):
     # FIXME: What purpose does data serve, besides preventing a union that
     # has a branch named 'data'? We use it in qapi-visit.py to decide
     # whether to bypass the switch statement if visiting the discriminator
@@ -159,11 +152,11 @@ struct %(c_name)s {
     # should not be any data leaks even without a data pointer.  Or, if
     # 'data' is merely added to guarantee we don't have an empty union,
     # shouldn't we enforce that at .json parse time?
-    ret += mcgen('''
+    ret = mcgen('''
     union { /* union tag is @%(c_name)s */
         void *data;
 ''',
-                 c_name=c_name(variants.tag_member.name))
+                c_name=c_name(variants.tag_member.name))
 
     for var in variants.variants:
         # Ugly special case for simple union TODO get rid of it
@@ -176,7 +169,6 @@ struct %(c_name)s {
 
     ret += mcgen('''
     } u;
-};
 ''')
 
     return ret
@@ -268,13 +260,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
     def visit_object_type(self, name, info, base, members, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
-        if variants:
-            if members:
-                assert len(members) == 1
-                assert members[0] == variants.tag_member
-            self.decl += gen_union(name, base, variants)
-        else:
-            self.decl += gen_struct(name, base, members)
+        self.decl += gen_object(name, base, members, variants)
         if base:
             self.decl += gen_upcast(name, base)
         self._gen_type_cleanup(name)

scripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 273

Quoted 7 lines
scripts/qapi-types.py +1 -1
@@ -282,7 +268,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def visit_alternate_type(self, name, info, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
         self._fwdefn += gen_alternate_qtypes(name, variants)
-        self.decl += gen_union(name, None, variants)
+        self.decl += gen_object(name, None, [variants.tag_member], variants)
         self.decl += gen_alternate_qtypes_decl(name)
         self._gen_type_cleanup(name)

Turned out nicely.

We could morph gen_struct_field() back into its original shape in a separate patch:

    def gen_struct_fields(members):
        ret = ''

        for memb in members:
            ret += gen_struct_field(memb.name, memb.type, memb.optional)
        return ret

with calling code

    ret += mcgen('''
    /* Members inherited from %(c_name)s: */
''',
                 c_name=base.c_name())
    ret += gen_struct_fields(base.members)
    ret += mcgen('''
    /* Own members: */
''')
    ret += gen_struct_fields(local_members)

Matter of taste.

author
#12 lore raw

Re: [Qemu-devel] [PATCH v8 06/17] qapi-types: Consolidate gen_struct() and gen_union()

Quoted 33 lines · Markus Armbruster
Quoted 31 lines · Eric Blake

These two methods are now close enough that we can finally merge them, relying on the fact that simple unions now provide a reasonable local_members. Change gen_struct() to gen_object() that handles all forms of QAPISchemaObjectType, and rename and shrink gen_union() to gen_variants() to handle the portion of gen_object() needed when variants are present.

gen_struct_fields() now has a single caller, so it no longer needs an optional parameter; however, I did not choose to inline it into the caller.

No difference to generated code.

def visit_object_type(self, name, info, base, members, variants):
self._fwdecl += gen_fwd_object_or_array(name)

scripts/qapi-types.py +1 -7
-        if variants:
-            if members:
-                assert len(members) == 1
-                assert members[0] == variants.tag_member
-            self.decl += gen_union(name, base, variants)
-        else:
-            self.decl += gen_struct(name, base, members)
+        self.decl += gen_object(name, base, members, variants)
         if base:
             self.decl += gen_upcast(name, base)
         self._gen_type_cleanup(name)
@@ -282,7 +268,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def visit_alternate_type(self, name, info, variants):
Quoted 8 lines

scripts/qapi-types.py class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): · line 273

Quoted 6 lines
scripts/qapi-types.py +1 -1
         self._fwdecl += gen_fwd_object_or_array(name)
         self._fwdefn += gen_alternate_qtypes(name, variants)
-        self.decl += gen_union(name, None, variants)
+        self.decl += gen_object(name, None, [variants.tag_member], variants)
         self.decl += gen_alternate_qtypes_decl(name)
         self._gen_type_cleanup(name)

Turned out nicely.

Yes, I was pretty pleased with it. It gives more credence to our choice of introspection representation.

Quoted 23 lines

We could morph gen_struct_field() back into its original shape in a separate patch:

    def gen_struct_fields(members):
        ret = ''

        for memb in members:
            ret += gen_struct_field(memb.name, memb.type, memb.optional)
        return ret

with calling code

    ret += mcgen('''
    /* Members inherited from %(c_name)s: */
''',
                 c_name=base.c_name())
    ret += gen_struct_fields(base.members)
    ret += mcgen('''
    /* Own members: */
''')
    ret += gen_struct_fields(local_members)
Quoted 1 line

Matter of taste.

Reasonable idea; I'll add it in for the v9 spin.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#13 lore raw

PATCH v8 7/17 qapi: Rework collision assertions

In reply to Eric Blake

Now that we have separate namespaces for QMP vs. tag values, we can simplify how the QAPISchema*.check() methods check for collisions. Each QAPISchemaObjectTypeMember check() call is given a single set of names it must not collide with; this set is either the QMP names (when this member is used by an ObjectType) or the case names (when this member is used by an ObjectTypeVariants). We no longer need an all_members parameter, as it can be computed by seen.values(). When used by a union, QAPISchemaObjectTypeVariant must also perform a QMP collision check for each member of its corresponding type.

The new ObjectType.check_qmp() is an idempotent subset of check(), and can be called multiple times over different seen sets (useful, since the members of one type can be applied into more than one other location via inheritance or flat union variants).

The code needs a temporary hack of passing a 'union' flag through Variants.check(), since we do not inline the branches of an alternate type into a parent QMP object. A later patch will rework how alternates are laid out, by adding a new subclass, and that will allow us to drop the extra parameter.

There are no changes to generated code.

Future patches will enhance testsuite coverage, improve error message quality on actual collisions, and move collision checks out of ad hoc parse code into the check() methods.

---
v8: rebase to earlier patches, defer positive test additions to later in series
v7: new patch, although it is a much cleaner implementation of what was attempted by subset B v8 15/18
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg03042.html

---
 scripts/qapi.py | 55 +++++++++++++++++++++++++++++++++----------------------
 1 file changed, 33 insertions(+), 22 deletions(-)
scripts/qapi.py +33 -22
index 84ac151..c571709 100644
@@ -972,28 +972,28 @@ class QAPISchemaObjectType(QAPISchemaType):
         self.variants = variants
         self.members = None
 
+    # Finish construction, and validate that all members are usable
     def check(self, schema):
         assert self.members is not False        # not running in cycles
         if self.members:
             return
         self.members = False                    # mark as being checked
+        seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)
-            members = list(self.base.members)
-        else:
-            members = []
-        seen = {}
-        for m in members:
-            assert c_name(m.name) not in seen
-            seen[m.name] = m
+            self.base.check_qmp(schema, seen)
         for m in self.local_members:
-            m.check(schema, members, seen)
+            m.check(schema, seen)
         if self.variants:
-            self.variants.check(schema, members, seen)
-        self.members = members
+            self.variants.check(schema, seen)
+        self.members = seen.values()
+
+    # Check that this type does not introduce QMP collisions into seen
+    def check_qmp(self, schema, seen):
+        self.check(schema)
+        assert not self.variants       # not implemented
+        for m in self.members:
+            m.check(schema, seen)
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
@@ -1027,11 +1027,13 @@ class QAPISchemaObjectTypeMember(object):
         self.type = None
         self.optional = optional
 
-    def check(self, schema, all_members, seen):
+    def check(self, schema, seen):
+        # seen is a map of names we must not collide with (either QMP
+        # names, when called by ObjectType, or case names, when called
+        # by Variant). This method is safe to call over multiple 'seen'.
         assert self.name not in seen
         self.type = schema.lookup_type(self._type_name)
         assert self.type
-        all_members.append(self)
         seen[self.name] = self
 
 
@@ -1050,26 +1052,35 @@ class QAPISchemaObjectTypeVariants(object):
         self.tag_member = tag_member
         self.variants = variants
 
-    def check(self, schema, members, seen):
+    # TODO drop union once alternates can be distinguished by tag_member
+    def check(self, schema, seen, union=True):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
+            assert self.tag_member
         elif seen:           # simple union
             assert self.tag_member in seen.itervalues()
         else:                # alternate
-            self.tag_member.check(schema, members, seen)
+            self.tag_member.check(schema, seen)
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        cases = OrderedDict()
         for v in self.variants:
-            vseen = dict(seen)
-            v.check(schema, self.tag_member.type, vseen)
+            # Reset seen array for each variant, since QMP names from one
+            # branch do not affect another branch, nor add to all_members
+            v.check(schema, self.tag_member.type, dict(seen), cases, union)
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
-    def check(self, schema, tag_type, seen):
-        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+    # TODO drop union once alternates can be distinguished by tag_type
+    def check(self, schema, tag_type, seen, cases, union):
+        # cases is case names we must not collide with
+        QAPISchemaObjectTypeMember.check(self, schema, cases)
         assert self.name in tag_type.values
+        if union:
+            # seen is QMP names our members must not collide with
+            self.type.check_qmp(schema, seen)
 
     # This function exists to support ugly simple union special cases
     # TODO get rid of them, and drop the function
@@ -1090,7 +1101,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
 
     def check(self, schema):
-        self.variants.check(schema, [], {})
+        self.variants.check(schema, {}, False)
 
     def json_type(self):
         return 'value'
Signature · 1 line
2.4.3
#14 lore raw

Re: [Qemu-devel] [PATCH v8 07/17] qapi: Rework collision assertions

Quoted 1 line · Eric Blake

Now that we have separate namespaces for QMP vs. tag values,

What's the "QMP namespace"?

Quoted 2 lines

we can simplify how the QAPISchema*.check() methods check for collisions.

I *guess* the point of this patch is dropping checks that are obsolete now tag values no longer collide with non-variant member names in generated C, with simplifications on top. Correct?

Quoted 8 lines

Each QAPISchemaObjectTypeMember check() call is
given a single set of names it must not collide with; this set is either the QMP names (when this member is used by an ObjectType) or the case names (when this member is used by an ObjectTypeVariants). We no longer need an all_members parameter, as it can be computed by seen.values(). When used by a union, QAPISchemaObjectTypeVariant must also perform a QMP collision check for each member of its corresponding type.

I'm afraid this explanation is next to impossible to understand unless you know how the checking works. I should know, because I wrote it, but actually don't, at least not by heart. So let me relearn how checking works before your patch.

We're talking about a single invocation of QAPISchemaObjectType.check(). Its job is to compute self.members and self.base, and do sanity checking, which includes checking for collisions. It has two variables of interest:

* members is where we accumulate the list of members that'll become self.members. It's initialized to the inherited members (empty if no base, obviously).

* seen is a dictionary mapping names to members. This is merely for collision checking. It's initialized to contain the inherited members (whose names must be distinct, by induction, because we make sure the base type's check() completes first).

For each local member m of self, we call m.check(schema, members, seen). This is actually QAPISchemaObjectTypeMember.check(), and it uses members and seen as follows:

* Assert m.name doesn't collide with another member's name, i.e. not in seen.

* Append m to members.

* Insert m.name: m into seen.

Straightforward so far. Coming up next: variants.

Each variant's members are represented as a single member with the tag value as name. Its type is an object type that has the variant's members.

For each variant v:

* Copy seen to vseen. It holds the non-variant members.

* Call QAPISchemaObjectTypeMember.check(schema, [], vseen), which boils down to assert v.name doesn't collide with a non-variant member's name. This check is obsolete.

* Assert v.name is a member of tag_type.

Not checked: variant's name doesn't collide with another variant's name. That's because we copy seen inside the loop instead of before the loop.

Not checked: variant's members don't collide with non-variant members. I think this check got lost when we simplified QAPISchemaObjectTypeVariants to hold a single member.

Note that the real checking happens in check_union(), and the checks here are just sanity checks. As long as that's the case, holes here are harmless. However, we need them plugged them when we move the real checking from check_union() to the .check().

Looks like variant checking should be thrown out and redone.

I still don't get your description. Guess I have to read the patch after all ;)

Quoted 5 lines

The new ObjectType.check_qmp() is an idempotent subset of check(), and can be called multiple times over different seen sets (useful, since the members of one type can be applied into more than one other location via inheritance or flat union variants).

I think I get this argument. Except I don't get why the method's named check_qmp().

Quoted 3 lines

The code needs a temporary hack of passing a 'union' flag through Variants.check(), since we do not inline the branches of an alternate type into a parent QMP object.

What a "QMP object"? Sounds like a JSON object on the QMP wire, but that makes no sense :)

Quoted 27 lines

A later patch
will rework how alternates are laid out, by adding a new subclass, and that will allow us to drop the extra parameter.

There are no changes to generated code.

Future patches will enhance testsuite coverage, improve error message quality on actual collisions, and move collision checks out of ad hoc parse code into the check() methods.

---
v8: rebase to earlier patches, defer positive test additions to later in series
v7: new patch, although it is a much cleaner implementation of what was attempted by subset B v8 15/18
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg03042.html

---
 scripts/qapi.py | 55 +++++++++++++++++++++++++++++++++----------------------
 1 file changed, 33 insertions(+), 22 deletions(-)
scripts/qapi.py +0 -0
index 84ac151..c571709 100644

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 981

Quoted 10 lines
scripts/qapi.py +2 -0
@@ -972,28 +972,28 @@ class QAPISchemaObjectType(QAPISchemaType):
         self.variants = variants
         self.members = None
 
+    # Finish construction, and validate that all members are usable
     def check(self, schema):
         assert self.members is not False        # not running in cycles
         if self.members:
             return
         self.members = False                    # mark as being checked
+        seen = OrderedDict()

Why do you need to switch from {} to OrderedDict()?

Quoted 18 lines
scripts/qapi.py +2 -12
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)
-            members = list(self.base.members)
-        else:
-            members = []
-        seen = {}
-        for m in members:
-            assert c_name(m.name) not in seen
-            seen[m.name] = m
+            self.base.check_qmp(schema, seen)
         for m in self.local_members:
-            m.check(schema, members, seen)
+            m.check(schema, seen)
         if self.variants:
-            self.variants.check(schema, members, seen)

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 996

Quoted 10 lines
scripts/qapi.py +9 -1
-        self.members = members
+            self.variants.check(schema, seen)
+        self.members = seen.values()
+
+    # Check that this type does not introduce QMP collisions into seen
+    def check_qmp(self, schema, seen):
+        self.check(schema)
+        assert not self.variants       # not implemented
+        for m in self.members:
+            m.check(schema, seen)

You said "ObjectType.check_qmp() is an idempotent subset of check()", but it's obviously a superset: it calls check(), then does some more work.

I think I'm now too confused to make further progress on this patch, probably because I've thought myself into a corner.

Before I give up, a remark on design. The QAPISchemaFOO classes all implement the same protocol:

* __init__() type-checks arguments and initializes instance variables, generally either to an argument or to None. This is basically AST construction.

* check() computes the remaining instance variables. This is semantic analysis, except it's stubbed out.

QAPISchema.__init__() calls its own check(), which calls all the others, directly (for entities), or indirect (for members and such). Each check() runs *once*.

Except for QAPISchemaType.check(). Why? Unlike other entities, the types need to be checked in a certain order: contained types (base and variant) before the containing type. For simplicity, we simply call their check(), and detect cycles. This is basically a topological sort and the real checking squashed together. The real checking still runs once.

It follows that you must not call check() except:

- An entity is responsible for calling check() for the non-entities it owns (e.g. an object type's check() calls its members' check()).

- A type may call check() for a type it contains (e.g. its base type). That's safe, because no type may contain itself. This kind of call drives the top-sort.

* Obviously, the instance variables computed by check() have valid values only after check(). Likewise, certain methods should be called only after check(), e.g. visit().

Of course, if we find a more suitable protocol, we're free to adopt it.

scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1033

Quoted 12 lines
scripts/qapi.py +4 -1
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
@@ -1027,11 +1027,13 @@ class QAPISchemaObjectTypeMember(object):
         self.type = None
         self.optional = optional
 
-    def check(self, schema, all_members, seen):
+    def check(self, schema, seen):
+        # seen is a map of names we must not collide with (either QMP
+        # names, when called by ObjectType, or case names, when called
+        # by Variant). This method is safe to call over multiple 'seen'.

What are "QMP names"? Member names, perhaps?

Quoted 58 lines
scripts/qapi.py +16 -8
         assert self.name not in seen
         self.type = schema.lookup_type(self._type_name)
         assert self.type
-        all_members.append(self)
         seen[self.name] = self
 
 
@@ -1050,26 +1052,35 @@ class QAPISchemaObjectTypeVariants(object):
         self.tag_member = tag_member
         self.variants = variants
 
-    def check(self, schema, members, seen):
+    # TODO drop union once alternates can be distinguished by tag_member
+    def check(self, schema, seen, union=True):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
+            assert self.tag_member
         elif seen:           # simple union
             assert self.tag_member in seen.itervalues()
         else:                # alternate
-            self.tag_member.check(schema, members, seen)
+            self.tag_member.check(schema, seen)
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        cases = OrderedDict()
         for v in self.variants:
-            vseen = dict(seen)
-            v.check(schema, self.tag_member.type, vseen)
+            # Reset seen array for each variant, since QMP names from one
+            # branch do not affect another branch, nor add to all_members
+            v.check(schema, self.tag_member.type, dict(seen), cases, union)
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
-    def check(self, schema, tag_type, seen):
-        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+    # TODO drop union once alternates can be distinguished by tag_type
+    def check(self, schema, tag_type, seen, cases, union):
+        # cases is case names we must not collide with
+        QAPISchemaObjectTypeMember.check(self, schema, cases)
         assert self.name in tag_type.values
+        if union:
+            # seen is QMP names our members must not collide with
+            self.type.check_qmp(schema, seen)
 
     # This function exists to support ugly simple union special cases
     # TODO get rid of them, and drop the function
@@ -1090,7 +1101,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
 
     def check(self, schema):
-        self.variants.check(schema, [], {})
+        self.variants.check(schema, {}, False)
 
     def json_type(self):
         return 'value'
author
#15 lore raw

Re: [Qemu-devel] [PATCH v8 07/17] qapi: Rework collision assertions

Quoted 5 lines · Markus Armbruster
Quoted 1 line · Eric Blake

Now that we have separate namespaces for QMP vs. tag values,

What's the "QMP namespace"?

I guess I need to be more explicit :)

In the generated C struct for a qapi object, we now have two separate namespaces:

struct Foo {
    Type1 name1; /* namespace for QMP (aka non-variant) members */
    Type2 name2;
    union {
        Type1 name1; /* namespace for tag values */
        Type2 name2;
    } u;
};
Quoted 8 lines
Quoted 3 lines

we can simplify how the QAPISchema*.check() methods check for collisions.

I *guess* the point of this patch is dropping checks that are obsolete now tag values no longer collide with non-variant member names in generated C, with simplifications on top. Correct?

Almost. This is not actually dropping any of the old ad hoc checks in the parser, but you are correct that it IS fixing the check() methods to quit asserting things that are no longer forbidden now that tag values no longer collide with non-variant member names.

Quoted 6 lines
Quoted 6 lines

Each QAPISchemaObjectTypeMember check() call is
given a single set of names it must not collide with; this set is either the QMP names (when this member is used by an ObjectType) or the case names (when this member is used by an ObjectTypeVariants). We no longer need an all_members parameter, as it can be computed by seen.values().

This is point [1] mentioned below.

Quoted 8 lines
Quoted 3 lines

When used
by a union, QAPISchemaObjectTypeVariant must also perform a QMP collision check for each member of its corresponding type.

I'm afraid this explanation is next to impossible to understand unless you know how the checking works. I should know, because I wrote it, but actually don't, at least not by heart. So let me relearn how checking works before your patch.

I guess I get to pick and choose from your wording, to try and make my commit body more comprehensible.

Quoted 9 lines

We're talking about a single invocation of QAPISchemaObjectType.check(). Its job is to compute self.members and self.base, and do sanity checking, which includes checking for collisions. It has two variables of interest:

* members is where we accumulate the list of members that'll become self.members. It's initialized to the inherited members (empty if no base, obviously).

True pre-patch; eliminated by point [1] mentioned above, by observing that seen.values() is identical to members after processing local_members, and that we don't further modify seen when processing variants.

Quoted 31 lines

* seen is a dictionary mapping names to members. This is merely for collision checking. It's initialized to contain the inherited members (whose names must be distinct, by induction, because we make sure the base type's check() completes first).

For each local member m of self, we call m.check(schema, members, seen). This is actually QAPISchemaObjectTypeMember.check(), and it uses members and seen as follows:

* Assert m.name doesn't collide with another member's name, i.e. not in seen.

* Append m to members.

* Insert m.name: m into seen.

Straightforward so far. Coming up next: variants.

Each variant's members are represented as a single member with the tag value as name. Its type is an object type that has the variant's members.

For each variant v:

* Copy seen to vseen. It holds the non-variant members.

* Call QAPISchemaObjectTypeMember.check(schema, [], vseen), which boils down to assert v.name doesn't collide with a non-variant member's name. This check is obsolete.

Quoted 8 lines

* Assert v.name is a member of tag_type.

Not checked: variant's name doesn't collide with another variant's name. That's because we copy seen inside the loop instead of before the loop.

Not checked: variant's members don't collide with non-variant members. I think this check got lost when we simplified QAPISchemaObjectTypeVariants to hold a single member.

Perhaps so; our testsuite wasn't as complete at that time. Or maybe it's because our ad hoc checks in the parsing portion were preventing us from realizing that we needed to repeat things in check() methods.

Quoted 5 lines

Note that the real checking happens in check_union(), and the checks here are just sanity checks. As long as that's the case, holes here are harmless. However, we need them plugged them when we move the real checking from check_union() to the .check().

Yep, and that's what this patch is trying to do.

Quoted 6 lines

Looks like variant checking should be thrown out and redone.

I still don't get your description. Guess I have to read the patch after all ;)

Quoted 8 lines
Quoted 5 lines

The new ObjectType.check_qmp() is an idempotent subset of check(), and can be called multiple times over different seen sets (useful, since the members of one type can be applied into more than one other location via inheritance or flat union variants).

I think I get this argument. Except I don't get why the method's named check_qmp().

Check that the QMP field names introduced by this type into an overall object do not cause any collisions with QMP field names already present from other sources. I'm open to suggestions for a better name.

Quoted 7 lines
Quoted 4 lines

The code needs a temporary hack of passing a 'union' flag through Variants.check(), since we do not inline the branches of an alternate type into a parent QMP object.

What a "QMP object"? Sounds like a JSON object on the QMP wire, but that makes no sense :)

Let's try again:

The code needs a temporary hack of passing a 'union' flag through Variants.check(), in order to decide whether to check the various fields of the branch object type against the earlier QMP names in seen. The check is conditional on being a union, because the QMP field names of each union branch share the same QMP namespace as the non-variant fields in the overall JSON object (even though they are in a different C namespace due to boxing behind the branch type), while an alternate does not inline any QMP fields (because it is not part of a larger JSON object). Note that we do not need to distinguish between simple and flat unions, because simple unions have a generated wrapper type whose lone QMP field ('data') will be checked against the lone 'type' non-variant field.

Quoted 13 lines
Quoted 13 lines

A later patch
will rework how alternates are laid out, by adding a new subclass, and that will allow us to drop the extra parameter.

There are no changes to generated code.

Future patches will enhance testsuite coverage, improve error message quality on actual collisions, and move collision checks out of ad hoc parse code into the check() methods.

Quoted 9 lines
Quoted 9 lines

---
v8: rebase to earlier patches, defer positive test additions to later in series
v7: new patch, although it is a much cleaner implementation of what was attempted by subset B v8 15/18
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg03042.html

---
 scripts/qapi.py | 55 +++++++++++++++++++++++++++++++++----------------------
 1 file changed, 33 insertions(+), 22 deletions(-)

Okay, let's see what you think of the actual patch, or if you have any better suggestions for how I should have worded the commit body.

Quoted 9 lines
Quoted 9 lines
scripts/qapi.py +0 -0
index 84ac151..c571709 100644
@@ -972,28 +972,28 @@ class QAPISchemaObjectType(QAPISchemaType):
         self.variants = variants
         self.members = None
 
Quoted 9 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 981

Quoted 7 lines
scripts/qapi.py +2 -0
+    # Finish construction, and validate that all members are usable
     def check(self, schema):
         assert self.members is not False        # not running in cycles
         if self.members:
             return
         self.members = False                    # mark as being checked
+        seen = OrderedDict()

Why do you need to switch from {} to OrderedDict()?

Because I'm dropping the all_members argument, and I need seen.values() to maintain the same order in which members are seen, for it to be a replacement for all_members.

Maybe I need to split this patch into a few more pieces, each tackling a smaller cleanup, rather than trying to do it all in one?

Quoted 6 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 984

Quoted 6 lines
scripts/qapi.py +0 -3
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)

[2] The old code calls base.check()...

Quoted 1 line

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 985

Quoted 1 line
scripts/qapi.py +0 -1
-            members = list(self.base.members)

x>> - else:

Quoted 5 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 991

Quoted 5 lines
scripts/qapi.py +0 -5
-            members = []
-        seen = {}
-        for m in members:
-            assert c_name(m.name) not in seen
-            seen[m.name] = m

...then adds each of base's members into seen.

Quoted 1 line

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 984

Quoted 1 line
scripts/qapi.py +1 -0
+            self.base.check_qmp(schema, seen)

The new code combines all of those actions under the helper base.check_qmp().

Quoted 8 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 989

Quoted 8 lines
scripts/qapi.py +3 -3
         for m in self.local_members:
-            m.check(schema, members, seen)
+            m.check(schema, seen)
         if self.variants:
-            self.variants.check(schema, members, seen)
-        self.members = members
+            self.variants.check(schema, seen)
+        self.members = seen.values()

Nothing else in child calls check_qmp().

Quoted 11 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 996

Quoted 7 lines
scripts/qapi.py +7 -0
+
+    # Check that this type does not introduce QMP collisions into seen
+    def check_qmp(self, schema, seen):
+        self.check(schema)
+        assert not self.variants       # not implemented
+        for m in self.members:
+            m.check(schema, seen)

You said "ObjectType.check_qmp() is an idempotent subset of check()", but it's obviously a superset: it calls check(), then does some more work.

See [2] above. I need to reword that then, or split it into a separate patch for easier reasoning. base.check_qmp() is a subset of the work done by child.check(). You are correct that it is NOT a subset of the work done by base.check() (because it calls base.check() and then does additional work).

Quoted 11 lines

I think I'm now too confused to make further progress on this patch, probably because I've thought myself into a corner.

Before I give up, a remark on design. The QAPISchemaFOO classes all implement the same protocol:

* __init__() type-checks arguments and initializes instance variables, generally either to an argument or to None. This is basically AST construction.

Quoted 6 lines

* check() computes the remaining instance variables. This is semantic analysis, except it's stubbed out.

QAPISchema.__init__() calls its own check(), which calls all the others, directly (for entities), or indirect (for members and such). Each check() runs *once*.

And I'm intentionally reworking things to run a _subset_ of check() multiple times - namely, calling member.check(seen) to see if member collides with any names already present in seen, where seen can be either the set of inherited non-variant fields (for both local members and for the fields of a variant type from a qapi union), or where seen can be the set of tag names (for the use of QAPISchemaObjectTypeVariant, which is a subclass of QAPISchemaObjectTypeMember).

Quoted 30 lines

Except for QAPISchemaType.check(). Why? Unlike other entities, the types need to be checked in a certain order: contained types (base and variant) before the containing type. For simplicity, we simply call their check(), and detect cycles. This is basically a topological sort and the real checking squashed together. The real checking still runs once.

It follows that you must not call check() except:

- An entity is responsible for calling check() for the non-entities it owns (e.g. an object type's check() calls its members' check()).

- A type may call check() for a type it contains (e.g. its base type). That's safe, because no type may contain itself. This kind of call drives the top-sort.

* Obviously, the instance variables computed by check() have valid values only after check(). Likewise, certain methods should be called only after check(), e.g. visit().

Of course, if we find a more suitable protocol, we're free to adopt it.

Quoted 7 lines
scripts/qapi.py +0 -0
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
@@ -1027,11 +1027,13 @@ class QAPISchemaObjectTypeMember(object):
         self.type = None
         self.optional = optional
 
Quoted 7 lines

scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1033

Quoted 5 lines
scripts/qapi.py +4 -1
-    def check(self, schema, all_members, seen):
+    def check(self, schema, seen):
+        # seen is a map of names we must not collide with (either QMP
+        # names, when called by ObjectType, or case names, when called
+        # by Variant). This method is safe to call over multiple 'seen'.

What are "QMP names"? Member names, perhaps?

Yes, the member names that result from JSON object keys at the same level of nesting, whether or not those names are attributed to the same C namespace.

I still like my patch, but I can see it caused some confusion. So I'll try to break it into smaller pieces for v9.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#16 lore raw

Re: [Qemu-devel] [PATCH v8 07/17] qapi: Rework collision assertions

In reply to Markus Armbruster
Quoted 4 lines · Markus Armbruster

Not checked: variant's members don't collide with non-variant members. I think this check got lost when we simplified QAPISchemaObjectTypeVariants to hold a single member.

Yep, I found the culprit: in your v2 proposal for QAPISchema, you had:

+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
+ def __init__(self, name, typ, flat):

+        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
+        assert isinstance(flat, bool)
+        self.flat = flat
+    def check(self, schema, tag_type, seen):
+        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        assert self.name in tag_type.values
+        if self.flat:

+ self.type.check(schema)
+ assert isinstance(self.type, QAPISchemaObjectType)

https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg00394.html

but the 'if self.flat' clause was lost in v3:

https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00450.html

I am in fact reinstating it here, but for v9, will do it in a separate patch rather than blended in with the rest of the changes.

[wow - we've been hammering at this since July?]

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#17 lore raw

Re: [Qemu-devel] [PATCH v8 07/17] qapi: Rework collision assertions

Quoted 21 lines · Eric Blake
Quoted 4 lines · Markus Armbruster

Not checked: variant's members don't collide with non-variant members. I think this check got lost when we simplified QAPISchemaObjectTypeVariants to hold a single member.

Yep, I found the culprit: in your v2 proposal for QAPISchema, you had:

(quoted diff) +11 -0
+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
+    def __init__(self, name, typ, flat):
+        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
+        assert isinstance(flat, bool)
+        self.flat = flat
+    def check(self, schema, tag_type, seen):
+        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        assert self.name in tag_type.values
+        if self.flat:
+            self.type.check(schema)
+            assert isinstance(self.type, QAPISchemaObjectType)
Quoted 5 lines

https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg00394.html

but the 'if self.flat' clause was lost in v3:

https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00450.html

Quoting v3's change log:

- Lower simple variants to flat ones as described on qapi-code-gen.txt. Replace QAPISchemaObjectType's .flat by .simple_union_type(), for use by pre-existing code-generation warts.

Quoted 2 lines

I am in fact reinstating it here, but for v9, will do it in a separate patch rather than blended in with the rest of the changes.

Any "is this union flat or simple" check signals a flaw. It's either a pointless difference in generated code (these should all be marked TODO by now), or something's wrong with the desugaring of simple to flat unions.

In this case, it looks like a collision check was lost when I switched from "simple unions are its own separate type" to "simple unions are sugar for flat unions".

Reinstating it makes sense, except for the "if self.flat" part. If the check's correct for flat unions, it must be correct for desugared simple unions, or else we screwed up the desugaring.

For a genuine flat union case

'tag-value': 'FlatVariant'

self.type is the object type 'FlatVariant'. Its members are the case's variant members, and they can clash with the flat union's non-variant members.

For a desugared simple union case

'tag-value': 'SimpleVariant'

self.type is an object type with a single member 'data' of type 'SimpleVariant'. Its member 'data' is the case's only variant member, and it can clash with the simple union's non-variant members. In theory only, because the only non-variant member is the implicit tag, and it's called 'type'.

Therefore, the if self.flat is superfluous. Good, because otherwise our desugaring must be flawed.

Quoted 1 line

[wow - we've been hammering at this since July?]

First RFC was in April, but we started hammering in earnest only in summer.

author
#18 lore raw

Re: [Qemu-devel] [PATCH v8 07/17] qapi: Rework collision assertions

Quoted 36 lines · Markus Armbruster
Quoted 21 lines · Eric Blake
Quoted 4 lines · Markus Armbruster

Not checked: variant's members don't collide with non-variant members. I think this check got lost when we simplified QAPISchemaObjectTypeVariants to hold a single member.

Yep, I found the culprit: in your v2 proposal for QAPISchema, you had:

(quoted diff) +11 -0
+class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
+    def __init__(self, name, typ, flat):
+        QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
+        assert isinstance(flat, bool)
+        self.flat = flat
+    def check(self, schema, tag_type, seen):
+        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        assert self.name in tag_type.values
+        if self.flat:
+            self.type.check(schema)
+            assert isinstance(self.type, QAPISchemaObjectType)
Quoted 5 lines

https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg00394.html

but the 'if self.flat' clause was lost in v3:

https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00450.html

Quoting v3's change log:

- Lower simple variants to flat ones as described on qapi-code-gen.txt. Replace QAPISchemaObjectType's .flat by .simple_union_type(), for use by pre-existing code-generation warts.

Quoted 7 lines
Quoted 2 lines

I am in fact reinstating it here, but for v9, will do it in a separate patch rather than blended in with the rest of the changes.

Any "is this union flat or simple" check signals a flaw. It's either a pointless difference in generated code (these should all be marked TODO by now), or something's wrong with the desugaring of simple to flat unions.

Losing 'if self.flat' was correct, but we still need 'if union'; and that's what I add in 2/4.

Quoted 3 lines

Therefore, the if self.flat is superfluous. Good, because otherwise our desugaring must be flawed.

And things correctly work on simple unions due to our wrapper type, so that 'if union' was sufficient.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#19 lore raw

cover rework of 7/17

In reply to Markus Armbruster

Based on the confusion I caused Markus, I've tried to split the v8 patch 7/17 into smaller pieces that are hopefully easier to review in isolation.

Eric Blake (4):
qapi: Drop all_members parameter from check()
qapi: Check for QMP collisions of flat union branches
qapi: Fix check for variant tag values collision
qapi: Consolidate collision detection code

 scripts/qapi.py | 62 ++++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 39 insertions(+), 23 deletions(-)
Signature · 1 line
2.4.3
author
#20 lore raw

PATCH 1/4 qapi: Drop all_members parameter from check()

In reply to Markus Armbruster

The implementation of QAPISchemaObjectTypeMember.check() always adds the member currently being checked to both the all_members and seen parameters. However, the three callers of this method pass in the following parameters:

QAPISchemaObjectType.check():
- all_members contains all non-variant members seen to date, for use in populating self.members
- seen contains all non-variant members seen to date, for use in checking for collisions

QAPISchemaObjectTypeVariant.check():
- all_members is a throwaway empty list
- seen is a throwaway dictionary created as a copy by QAPISchemaObjectVariants.check() (since the members of one variant cannot collide with those from another), for use in checking for collisions (technically, we no longer need to check for collisions between tag values and QMP key names, but that's a cleanup for another patch)

QAPISchemaAlternateType.check():
- all_members is a throwaway empty list
- seen is a throwaway empty dict

Therefore, in the one case where we care about all_members after seen has been populated, we know that it contains the same members as seen.values(); changing seen to be an OrderedDict() is sufficient to pick up this information with one less parameter being passed around.

---
v9: new patch, split off from v8 7/17

---
 scripts/qapi.py | 29 +++++++++++++----------------
 1 file changed, 13 insertions(+), 16 deletions(-)
scripts/qapi.py +13 -16
index 6b6ba5f..fff4adb 100644
@@ -972,28 +972,26 @@ class QAPISchemaObjectType(QAPISchemaType):
         self.variants = variants
         self.members = None
 
+    # Finish construction, and validate that all members are usable
     def check(self, schema):
         assert self.members is not False        # not running in cycles
         if self.members:
             return
         self.members = False                    # mark as being checked
+        seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
             assert isinstance(self.base, QAPISchemaObjectType)
             assert not self.base.variants       # not implemented
             self.base.check(schema)
-            members = list(self.base.members)
-        else:
-            members = []
-        seen = {}
-        for m in members:
-            assert c_name(m.name) not in seen
-            seen[m.name] = m
+            for m in self.base.members:
+                assert c_name(m.name) not in seen
+                seen[m.name] = m
         for m in self.local_members:
-            m.check(schema, members, seen)
+            m.check(schema, seen)
         if self.variants:
-            self.variants.check(schema, members, seen)
-        self.members = members
+            self.variants.check(schema, seen)
+        self.members = seen.values()
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
@@ -1027,11 +1025,10 @@ class QAPISchemaObjectTypeMember(object):
         self.type = None
         self.optional = optional
 
-    def check(self, schema, all_members, seen):
+    def check(self, schema, seen):
         assert self.name not in seen
         self.type = schema.lookup_type(self._type_name)
         assert self.type
-        all_members.append(self)
         seen[self.name] = self
 
 
@@ -1050,7 +1047,7 @@ class QAPISchemaObjectTypeVariants(object):
         self.tag_member = tag_member
         self.variants = variants
 
-    def check(self, schema, members, seen):
+    def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
             assert self.tag_member
@@ -1067,7 +1064,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
     def check(self, schema, tag_type, seen):
-        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        QAPISchemaObjectTypeMember.check(self, schema, seen)
         assert self.name in tag_type.values
 
     # This function exists to support ugly simple union special cases
@@ -1090,8 +1087,8 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
     def check(self, schema):
         seen = {}
-        self.variants.tag_member.check(schema, [], seen)
-        self.variants.check(schema, [], seen)
+        self.variants.tag_member.check(schema, seen)
+        self.variants.check(schema, seen)
 
     def json_type(self):
         return 'value'
Signature · 1 line
2.4.3
#21 lore raw

Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()

Quoted 3 lines · Eric Blake

The implementation of QAPISchemaObjectTypeMember.check() always adds the member currently being checked to both the all_members and seen parameters.

QAPISchemaObjectTypeMember.check() does four things:

1. Compute self.type

Precondition: all types are defined.

2. Accumulate members

all_members serves as accumulator.

We'll see that its only actual use is the owning object type's check(), which uses it to compute self.members.

3. Check for collisions

This works by accumulating names in seen. Precondition: seen contains the names seen so far.

Note that this part uses seen like a set. See 4.

4. Accumulate a map from names to members

seen serves as accumulator.

We'll see that its only actual user is the owning object type's variants.check(), which uses it to compute variants.tag_member from variants.tag_name.

Quoted 8 lines

However, the three callers of this method
pass in the following parameters:

QAPISchemaObjectType.check():
- all_members contains all non-variant members seen to date, for use in populating self.members
- seen contains all non-variant members seen to date, for use in checking for collisions

Yes, and:

- we're calling it for m in self.local_members
- before the loop, all_members and seen are initialized to the inherited non-variant members
- after the loop, they therefore contain all non-variant members

This caller uses all four things done by QAPISchemaObjectType.check():

1. Compute m.type
2. Accumulate non-variant members
3. Check for collisions among non-variant members
Before the loop, seen contains the inherited members, which don't collide (self.base.check() ensures that). The loop adds the local members one by one, checking for collisions.
4. Accumulate a map from names to non-variant members Similar argument to 3.

Quoted 1 line

QAPISchemaObjectTypeVariant.check():

Do you mean QAPISchemaObjectVariants.check()?

Quoted 11 lines

- all_members is a throwaway empty list
- seen is a throwaway dictionary created as a copy by QAPISchemaObjectVariants.check() (since the members of one variant cannot collide with those from another), for use in checking for collisions (technically, we no longer need to check for collisions between tag values and QMP key names, but that's a cleanup for another patch)

QAPISchemaAlternateType.check():
- all_members is a throwaway empty list
- seen is a throwaway empty dict

I'm afraid you're omitting a few steps here, and I think you missed QAPISchemaObjectVariants.check()'s self.tag_member.check(). Let me go through the remaining callers of QAPISchemaObjectTypeMember.check() real slow.

* QAPISchemaObjectType.check() calls it via v.check(), which is a thin wrapper that throws away 2. and asserts something of no interest here. seen is a map from names to non-variant members. Therefore, QAPISchemaObjectTypeMember.check() gets used here as follows:

1. Compute v.type
2. Thrown away
3. Check for collision of tag value with non-variant members This is obsolete now.
4. Thrown away

* QAPISchemaAlternateType.check() calls it via v.check(), which is the same thin wrapper. seen is {} here. Therefore:

1. Compute v.type
2. Thrown away
3. No-op
4. Thrown away

* QAPISchemaObjectTypeVariants.check() calls self.tag_member.check(schema, members, seen)
where members and seen contain the non-variant members. However, it gets called only when the owning type is an alternate, and then members and seen are both empty. Therefore:

1. Compute v.type
2. Thrown away
3. No-op
4. Thrown away

Quoted 5 lines

Therefore, in the one case where we care about all_members after seen has been populated, we know that it contains the same members as seen.values(); changing seen to be an OrderedDict() is sufficient to pick up this information with one less parameter being passed around.

I believe the first step should be dropping the obsolete check for collision of tag value with non-variant members. I believe this should do:

@@ -1059,8 +1059,7 @@ class QAPISchemaObjectTypeVariants(object):
self.tag_member.check(schema, members, seen)
assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants:

-            vseen = dict(seen)
-            v.check(schema, self.tag_member.type, vseen)
+            v.check(schema, self.tag_member.type, {})

class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):

Then only one caller about 2-4., namely QAPISchemaObjectType.check(). Simplify radically: move 2-4. to the caller that cares, drop parameters all_members and seen.

Still to do then: non-variant member collision checking. Factor out
3. into a helper function, use it for non-variant members.

What do you think?

author
#22 lore raw

Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()

Quoted 11 lines · Markus Armbruster
Quoted 3 lines · Eric Blake

The implementation of QAPISchemaObjectTypeMember.check() always adds the member currently being checked to both the all_members and seen parameters.

QAPISchemaObjectTypeMember.check() does four things:

1. Compute self.type

Precondition: all types are defined.

Correct, unchanged by this patch.

Quoted 7 lines

2. Accumulate members

all_members serves as accumulator.

We'll see that its only actual use is the owning object type's check(), which uses it to compute self.members.

This patch changes it to use seen.values(), which (once you use an OrderedDict() instead of plain {}) is identical to all_members.

Quoted 7 lines

3. Check for collisions

This works by accumulating names in seen. Precondition: seen contains the names seen so far.

Note that this part uses seen like a set. See 4.

Unchanged by this patch; but see also 2/4 and 3/4.

Quoted 5 lines

4. Accumulate a map from names to members

seen serves as accumulator.

Unchanged by this patch.

Quoted 15 lines

We'll see that its only actual user is the owning object type's variants.check(), which uses it to compute variants.tag_member from variants.tag_name.

Quoted 8 lines

However, the three callers of this method
pass in the following parameters:

QAPISchemaObjectType.check():
- all_members contains all non-variant members seen to date, for use in populating self.members
- seen contains all non-variant members seen to date, for use in checking for collisions

Yes, and:

Quoted 8 lines

- we're calling it for m in self.local_members
- before the loop, all_members and seen are initialized to the inherited non-variant members
- after the loop, they therefore contain all non-variant members

This caller uses all four things done by QAPISchemaObjectType.check():

1. Compute m.type

Unchanged by this patch.

Quoted 1 line

2. Accumulate non-variant members

Whether the accumulation is done via all_members (pre-patch) or by seen.values() (post-patch), this step is still done.

Quoted 4 lines

3. Check for collisions among non-variant members
Before the loop, seen contains the inherited members, which don't collide (self.base.check() ensures that). The loop adds the local members one by one, checking for collisions.

Unchanged by this patch.

Quoted 2 lines

4. Accumulate a map from names to non-variant members Similar argument to 3.

Unchanged by this patch.

Quoted 4 lines
Quoted 2 lines

QAPISchemaObjectTypeVariant.check():

Do you mean QAPISchemaObjectVariants.check()?

QAPISchemaObjectTypeVariants.check() calls QAPISchemaObjectTypeVariant.check() for each variant, but with a fresh copy of seen. We'll later need to expand this copy of seen (patch 2/4), but for this patch its use is unchanged - we are appending a single value (the tag value) which is wrong, but no one cares that we appended it because it was a copy. Patch 3/4 fixes to not append to it.

Quoted 9 lines
Quoted 9 lines

- all_members is a throwaway empty list
- seen is a throwaway dictionary created as a copy by QAPISchemaObjectVariants.check() (since the members of one variant cannot collide with those from another), for use in checking for collisions (technically, we no longer need to check for collisions between tag values and QMP key names, but that's a cleanup for another patch)

Quoted 6 lines
Quoted 3 lines

QAPISchemaAlternateType.check():
- all_members is a throwaway empty list
- seen is a throwaway empty dict

I'm afraid you're omitting a few steps here, and I think you missed QAPISchemaObjectVariants.check()'s self.tag_member.check().

There is no self.tag_member.check(), any more; rather, my earlier patches have reworked things so that tag_member is checked by the owner (a flat union's base type, a simple union's local_members, or directly in QAPISchemaAlternateType prior to calling Variants.check()). I guess that's a pitfall of seeing this patch without my rework of 5/17 to address your comments there.

Quoted 11 lines
Quoted 5 lines

Therefore, in the one case where we care about all_members after seen has been populated, we know that it contains the same members as seen.values(); changing seen to be an OrderedDict() is sufficient to pick up this information with one less parameter being passed around.

I believe the first step should be dropping the obsolete check for collision of tag value with non-variant members. I believe this should do:

Review class QAPISchemaObjectTypeVariants(object): · line 1062

Quoted 6 lines
(quoted diff) +1 -2
@@ -1059,8 +1059,7 @@ class QAPISchemaObjectTypeVariants(object):
             self.tag_member.check(schema, members, seen)
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
-            vseen = dict(seen)
-            v.check(schema, self.tag_member.type, vseen)
+            v.check(schema, self.tag_member.type, {})

Close, but not quite. It should do:

+ cases = {}
for v in self.variants:
vseen = dict(seen)
- v.check(schema, self.tag_member.type, vseen)
+ v.check(schema, self.tag_member.type, vseen, cases)

coupled with this in QAPISchemaObjectTypeVariant:

- def check(self, schema, tag_type, seen):
- QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+ def check(self, schema, tag_type, seen, cases):
+ QAPISchemaObjectTypeMember.check(self, schema, [], cases)

so that we are now checking collisions between tag values, rather than cases. But that's what I did in patch 3/4. And we still need seen passed to Variant.check(), because that's the checking added in 2/4.

Okay, you've convinced me - when I post v9, I'll reorder these four patches to put 3/4 first.

Quoted 7 lines
(quoted diff) +0 -0
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
 

Then only one caller about 2-4., namely QAPISchemaObjectType.check(). Simplify radically: move 2-4. to the caller that cares, drop parameters all_members and seen.

Nope - because seen (well, a copy of seen) is still important to patch 2/4.

Quoted 3 lines

Still to do then: non-variant member collision checking. Factor out
3. into a helper function, use it for non-variant members.

Factoring into a helper function is done in 4/4. I can try and rearrange that earlier, too.

Quoted 3 lines

What do you think?

Can you at least look at 2, 3, and 4 to see where I'm headed, and then I can rearrange things for the v9 spin? We're probably talking a bit past each other, with the same end goal, but a muddle in the middle of how to get there.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#23 lore raw

Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()

Quoted 115 lines · Eric Blake
Quoted 11 lines · Markus Armbruster
Quoted 3 lines · Eric Blake

The implementation of QAPISchemaObjectTypeMember.check() always adds the member currently being checked to both the all_members and seen parameters.

QAPISchemaObjectTypeMember.check() does four things:

1. Compute self.type

Precondition: all types are defined.

Correct, unchanged by this patch.

Quoted 7 lines

2. Accumulate members

all_members serves as accumulator.

We'll see that its only actual use is the owning object type's check(), which uses it to compute self.members.

This patch changes it to use seen.values(), which (once you use an OrderedDict() instead of plain {}) is identical to all_members.

Quoted 7 lines

3. Check for collisions

This works by accumulating names in seen. Precondition: seen contains the names seen so far.

Note that this part uses seen like a set. See 4.

Unchanged by this patch; but see also 2/4 and 3/4.

Quoted 5 lines

4. Accumulate a map from names to members

seen serves as accumulator.

Unchanged by this patch.

Quoted 15 lines

We'll see that its only actual user is the owning object type's variants.check(), which uses it to compute variants.tag_member from variants.tag_name.

Quoted 8 lines

However, the three callers of this method
pass in the following parameters:

QAPISchemaObjectType.check():
- all_members contains all non-variant members seen to date, for use in populating self.members
- seen contains all non-variant members seen to date, for use in checking for collisions

Yes, and:

Quoted 8 lines

- we're calling it for m in self.local_members
- before the loop, all_members and seen are initialized to the inherited non-variant members
- after the loop, they therefore contain all non-variant members

This caller uses all four things done by QAPISchemaObjectType.check():

1. Compute m.type

Unchanged by this patch.

Quoted 1 line

2. Accumulate non-variant members

Whether the accumulation is done via all_members (pre-patch) or by seen.values() (post-patch), this step is still done.

Quoted 4 lines

3. Check for collisions among non-variant members
Before the loop, seen contains the inherited members, which don't collide (self.base.check() ensures that). The loop adds the local members one by one, checking for collisions.

Unchanged by this patch.

Quoted 2 lines

4. Accumulate a map from names to non-variant members Similar argument to 3.

Unchanged by this patch.

Quoted 4 lines
Quoted 2 lines

QAPISchemaObjectTypeVariant.check():

Do you mean QAPISchemaObjectVariants.check()?

QAPISchemaObjectTypeVariants.check() calls QAPISchemaObjectTypeVariant.check() for each variant, but with a fresh copy of seen. We'll later need to expand this copy of seen (patch 2/4), but for this patch its use is unchanged - we are appending a single value (the tag value) which is wrong, but no one cares that we appended it because it was a copy. Patch 3/4 fixes to not append to it.

Quoted 9 lines
Quoted 9 lines

- all_members is a throwaway empty list
- seen is a throwaway dictionary created as a copy by QAPISchemaObjectVariants.check() (since the members of one variant cannot collide with those from another), for use in checking for collisions (technically, we no longer need to check for collisions between tag values and QMP key names, but that's a cleanup for another patch)

Quoted 6 lines
Quoted 3 lines

QAPISchemaAlternateType.check():
- all_members is a throwaway empty list
- seen is a throwaway empty dict

I'm afraid you're omitting a few steps here, and I think you missed QAPISchemaObjectVariants.check()'s self.tag_member.check().

Quoted 6 lines

There is no self.tag_member.check(), any more; rather, my earlier patches have reworked things so that tag_member is checked by the owner (a flat union's base type, a simple union's local_members, or directly in QAPISchemaAlternateType prior to calling Variants.check()). I guess that's a pitfall of seeing this patch without my rework of 5/17 to address your comments there.

I'm assuming this patch is based on
[PATCH v8 06/17] qapi-types: Consolidate gen_struct() and gen_union() which has

def check(self, schema, members, seen):
if self.tag_name: # flat union
self.tag_member = seen[self.tag_name]
elif seen: # simple union
assert self.tag_member in seen.itervalues()
else: # alternate
---> self.tag_member.check(schema, members, seen)

in QAPISchemaObjectTypeVariants.

Quoted 33 lines
Quoted 11 lines
Quoted 5 lines

Therefore, in the one case where we care about all_members after seen has been populated, we know that it contains the same members as seen.values(); changing seen to be an OrderedDict() is sufficient to pick up this information with one less parameter being passed around.

I believe the first step should be dropping the obsolete check for collision of tag value with non-variant members. I believe this should do:

Review class QAPISchemaObjectTypeVariants(object): · line 1062

Quoted 6 lines
(quoted diff) +1 -2
@@ -1059,8 +1059,7 @@ class QAPISchemaObjectTypeVariants(object):
             self.tag_member.check(schema, members, seen)
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
-            vseen = dict(seen)
-            v.check(schema, self.tag_member.type, vseen)
+            v.check(schema, self.tag_member.type, {})

Close, but not quite. It should do:

(quoted diff) +2 -1
+          cases = {}
           for v in self.variants:
               vseen = dict(seen)
-              v.check(schema, self.tag_member.type, vseen)
+              v.check(schema, self.tag_member.type, vseen, cases)

coupled with this in QAPISchemaObjectTypeVariant:

(quoted diff) +2 -2
-    def check(self, schema, tag_type, seen):
-        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+    def check(self, schema, tag_type, seen, cases):
+        QAPISchemaObjectTypeMember.check(self, schema, [], cases)
Quoted 3 lines

so that we are now checking collisions between tag values, rather than cases. But that's what I did in patch 3/4. And we still need seen passed to Variant.check(), because that's the checking added in 2/4.

A sanity check "no collisions between tag values" doesn't hurt, but I'd simply rely on the tag type's .check(). The tag type is an enumeration type, its check() ensures the enumeration values are distinct. For variants, checking the tag value is a is a value of the tag type suffices to ensure they don't collide.

Quoted 21 lines

Okay, you've convinced me - when I post v9, I'll reorder these four patches to put 3/4 first.

Quoted 7 lines
(quoted diff) +0 -0
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
 

Then only one caller about 2-4., namely QAPISchemaObjectType.check(). Simplify radically: move 2-4. to the caller that cares, drop parameters all_members and seen.

Nope - because seen (well, a copy of seen) is still important to patch 2/4.

Quoted 3 lines

Still to do then: non-variant member collision checking. Factor out
3. into a helper function, use it for non-variant members.

Factoring into a helper function is done in 4/4. I can try and rearrange that earlier, too.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

What do you think?

Can you at least look at 2, 3, and 4 to see where I'm headed, and then I can rearrange things for the v9 spin? We're probably talking a bit past each other, with the same end goal, but a muddle in the middle of how to get there.

Yes, we're almost certainly headed in the same direction. But I got thoroughly confused and lost in the details on the way, so I had to hack things up myself to clear my head. The result looks nice to me, so I'll send it out in a jiffie in the hope you'll find it useful.

author
#24 lore raw

Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()

Quoted 22 lines · Markus Armbruster
Quoted 9 lines
Quoted 2 lines

I'm afraid you're omitting a few steps here, and I think you missed QAPISchemaObjectVariants.check()'s self.tag_member.check().

There is no self.tag_member.check(), any more; rather, my earlier patches have reworked things so that tag_member is checked by the owner (a flat union's base type, a simple union's local_members, or directly in QAPISchemaAlternateType prior to calling Variants.check()). I guess that's a pitfall of seeing this patch without my rework of 5/17 to address your comments there.

I'm assuming this patch is based on
[PATCH v8 06/17] qapi-types: Consolidate gen_struct() and gen_union() which has

def check(self, schema, members, seen):
if self.tag_name: # flat union
self.tag_member = seen[self.tag_name]
elif seen: # simple union
assert self.tag_member in seen.itervalues()
else: # alternate
---> self.tag_member.check(schema, members, seen)

Quoted 1 line

in QAPISchemaObjectTypeVariants.

That was true in v8, but not in my pending v9 which did essentially what your 7/7 did (move the tag_member.check() into Alternate.check() back in patch 5).

At any rate, my first glance of your series shows that it is reasonable, so my task today is to spit out a v9 of my series, but using your seven patches in place of my four.

Quoted 5 lines

Yes, we're almost certainly headed in the same direction. But I got thoroughly confused and lost in the details on the way, so I had to hack things up myself to clear my head. The result looks nice to me, so I'll send it out in a jiffie in the hope you'll find it useful.

Yes, it should make v9 easier when it includes your patches, because then you know what to look for :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#25 lore raw

Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()

Quoted 26 lines · Eric Blake
Quoted 22 lines · Markus Armbruster
Quoted 9 lines
Quoted 2 lines

I'm afraid you're omitting a few steps here, and I think you missed QAPISchemaObjectVariants.check()'s self.tag_member.check().

There is no self.tag_member.check(), any more; rather, my earlier patches have reworked things so that tag_member is checked by the owner (a flat union's base type, a simple union's local_members, or directly in QAPISchemaAlternateType prior to calling Variants.check()). I guess that's a pitfall of seeing this patch without my rework of 5/17 to address your comments there.

I'm assuming this patch is based on
[PATCH v8 06/17] qapi-types: Consolidate gen_struct() and gen_union() which has

def check(self, schema, members, seen):
if self.tag_name: # flat union
self.tag_member = seen[self.tag_name]
elif seen: # simple union
assert self.tag_member in seen.itervalues()
else: # alternate
---> self.tag_member.check(schema, members, seen)

Quoted 1 line

in QAPISchemaObjectTypeVariants.

Quoted 7 lines

That was true in v8, but not in my pending v9 which did essentially what your 7/7 did (move the tag_member.check() into Alternate.check() back in patch 5).

At any rate, my first glance of your series shows that it is reasonable, so my task today is to spit out a v9 of my series, but using your seven patches in place of my four.

Buyer beware: I'm not sure my seven do everything your four do.

Quoted 8 lines
Quoted 5 lines

Yes, we're almost certainly headed in the same direction. But I got thoroughly confused and lost in the details on the way, so I had to hack things up myself to clear my head. The result looks nice to me, so I'll send it out in a jiffie in the hope you'll find it useful.

Yes, it should make v9 easier when it includes your patches, because then you know what to look for :)

I hope I'll be less easily confused going forward...

author
#26 lore raw

Re: [Qemu-devel] [PATCH v8.5 1/4] qapi: Drop all_members parameter from check()

Quoted 12 lines · Markus Armbruster
Quoted 12 lines
Quoted 12 lines

I'm assuming this patch is based on
[PATCH v8 06/17] qapi-types: Consolidate gen_struct() and gen_union() which has

def check(self, schema, members, seen):
if self.tag_name: # flat union
self.tag_member = seen[self.tag_name]
elif seen: # simple union
assert self.tag_member in seen.itervalues()
else: # alternate
---> self.tag_member.check(schema, members, seen)

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

in QAPISchemaObjectTypeVariants.

That was true in v8, but not in my pending v9 which did essentially what your 7/7 did (move the tag_member.check() into Alternate.check() back in patch 5).

Caused me some churn in incorporating your patches, but not too bad.

Quoted 6 lines
Quoted 4 lines

At any rate, my first glance of your series shows that it is reasonable, so my task today is to spit out a v9 of my series, but using your seven patches in place of my four.

Buyer beware: I'm not sure my seven do everything your four do.

Yours do the same as my patch 1. You also made a compelling argument for eliminating my patch 3 (the enum values are already collision-proof, therefore if we check that a variant name is in the enum, we don't need to check for collisions) - except that in my 10/17, when I get rid of the generated enum for alternates, I have to reinstate that check somewhere; but adding it directly to QAPISchemaAlternateType.check() at that point feels better.

My patch 2 is still needed, but looks a bit nicer on top of some of your refactoring. And my patch 4 is still useful, as additional refactoring to share the code used by my patch 2 and your code for ObjectType.check().

We'll see how it all turns out in v9.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#27 lore raw

PATCH 1/7 qapi: Drop obsolete tag value collision assertions

In reply to Markus Armbruster

Union tag values can't clash with member names in generated C anymore since commit e4ba22b, but QAPISchemaObjectTypeVariants.check() still asserts they don't. Drop it.

---
 scripts/qapi.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi.py +1 -2
index 84ac151..a303929 100644
@@ -1059,8 +1059,7 @@ class QAPISchemaObjectTypeVariants(object):
             self.tag_member.check(schema, members, seen)
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
-            vseen = dict(seen)
-            v.check(schema, self.tag_member.type, vseen)
+            v.check(schema, self.tag_member.type, {})
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Signature · 1 line
2.4.3
#28 lore raw

PATCH 2/7 qapi: Simplify QAPISchemaObjectTypeMember.check()

QAPISchemaObjectTypeMember.check() currently does four things:

1. Compute self.type

2. Accumulate members in all_members

Only one caller cares: QAPISchemaObjectType.check() uses it to compute self.members. The other callers pass a throw-away accumulator.

3. Accumulate a map from names to members in seen

Only one caller cares: QAPISchemaObjectType.check() uses it to compute its local variable seen, for self.variants.check(), which uses it to compute self.variants.tag_member from self.variants.tag_name. The other callers pass a throw-away accumulator.

4. Check for collisions

This piggyback on 3: before adding a new entry, we assert it's new.

Only one caller cares: QAPISchemaObjectType.check() uses it to assert non-variant members don't clash.

Simplify QAPISchemaObjectType.check(): move 2.-4. to QAPISchemaObjectType.check(), and drop parameters all_members and seen.

---
 scripts/qapi.py | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
scripts/qapi.py +7 -7
index a303929..20fdfbf 100644
@@ -990,7 +990,10 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert c_name(m.name) not in seen
             seen[m.name] = m
         for m in self.local_members:
-            m.check(schema, members, seen)
+            m.check(schema)
+            assert m.name not in seen
+            seen[m.name] = m
+            members.append(m)
         if self.variants:
             self.variants.check(schema, members, seen)
         self.members = members
@@ -1027,12 +1030,9 @@ class QAPISchemaObjectTypeMember(object):
         self.type = None
         self.optional = optional
 
-    def check(self, schema, all_members, seen):
-        assert self.name not in seen
+    def check(self, schema):
         self.type = schema.lookup_type(self._type_name)
         assert self.type
-        all_members.append(self)
-        seen[self.name] = self
 
 
 class QAPISchemaObjectTypeVariants(object):
@@ -1056,7 +1056,7 @@ class QAPISchemaObjectTypeVariants(object):
         elif seen:           # simple union
             assert self.tag_member in seen.itervalues()
         else:                # alternate
-            self.tag_member.check(schema, members, seen)
+            self.tag_member.check(schema)
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
             v.check(schema, self.tag_member.type, {})
@@ -1067,7 +1067,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
     def check(self, schema, tag_type, seen):
-        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        QAPISchemaObjectTypeMember.check(self, schema)
         assert self.name in tag_type.values
 
     # This function exists to support ugly simple union special cases
Signature · 1 line
2.4.3
#29 lore raw

PATCH 3/7 qapi: Clean up after previous commit

In reply to Markus Armbruster

QAPISchemaObjectTypeVariants.check() parameter members and QAPISchemaObjectTypeVariant.check() parameter seen are no longer used, drop them.

---
 scripts/qapi.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
scripts/qapi.py +5 -5
index 20fdfbf..7f8af41 100644
@@ -995,7 +995,7 @@ class QAPISchemaObjectType(QAPISchemaType):
             seen[m.name] = m
             members.append(m)
         if self.variants:
-            self.variants.check(schema, members, seen)
+            self.variants.check(schema, seen)
         self.members = members
 
     def is_implicit(self):
@@ -1050,7 +1050,7 @@ class QAPISchemaObjectTypeVariants(object):
         self.tag_member = tag_member
         self.variants = variants
 
-    def check(self, schema, members, seen):
+    def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
         elif seen:           # simple union
@@ -1059,14 +1059,14 @@ class QAPISchemaObjectTypeVariants(object):
             self.tag_member.check(schema)
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
-            v.check(schema, self.tag_member.type, {})
+            v.check(schema, self.tag_member.type)
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
-    def check(self, schema, tag_type, seen):
+    def check(self, schema, tag_type):
         QAPISchemaObjectTypeMember.check(self, schema)
         assert self.name in tag_type.values
 
@@ -1089,7 +1089,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
 
     def check(self, schema):
-        self.variants.check(schema, [], {})
+        self.variants.check(schema, {})
 
     def json_type(self):
         return 'value'
Signature · 1 line
2.4.3
#30 lore raw

PATCH 4/7 qapi: Fix up commit 7618b91's clash sanity checking change

In reply to Markus Armbruster

This hunk

@@ -964,6 +965,7 @@ class QAPISchemaObjectType(QAPISchemaType):
members = []
seen = {}
for m in members:
+ assert c_name(m.name) not in seen
seen[m.name] = m
for m in self.local_members:
m.check(schema, members, seen)

is plainly broken.

Asserting the members inherited from base don't clash is somewhat redundant, because self.base.check() just checked that. But it doesn't hurt.

The idea to use c_name(m.name) instead of m.name for collision checking is sound, because we need to catch clashes between the m.name and between the c_name(m.name), and when two m.name clash, then their c_name() also clash.

However, using c_name(m.name) instead of m.name in one of several places doesn't work. See the very next line.

Keep the assertion, but drop the c_name() for now. A future commit will bring it back.

---
 scripts/qapi.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi.py +1 -1
index 7f8af41..a6fdd76 100644
@@ -987,7 +987,7 @@ class QAPISchemaObjectType(QAPISchemaType):
             members = []
         seen = {}
         for m in members:
-            assert c_name(m.name) not in seen
+            assert m.name not in seen
             seen[m.name] = m
         for m in self.local_members:
             m.check(schema)
Signature · 1 line
2.4.3
#31 lore raw

PATCH 5/7 qapi: Eliminate QAPISchemaObjectType.check() variable members

In reply to Markus Armbruster

We can use seen.values() instead if we make it an OrderedDict.

---
 scripts/qapi.py | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)
scripts/qapi.py +5 -9
index a6fdd76..a46fded 100644
@@ -977,26 +977,22 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self.members:
             return
         self.members = False                    # mark as being checked
+        seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
             assert isinstance(self.base, QAPISchemaObjectType)
             assert not self.base.variants       # not implemented
             self.base.check(schema)
-            members = list(self.base.members)
-        else:
-            members = []
-        seen = {}
-        for m in members:
-            assert m.name not in seen
-            seen[m.name] = m
+            for m in self.base.members:
+                assert m.name not in seen
+                seen[m.name] = m
         for m in self.local_members:
             m.check(schema)
             assert m.name not in seen
             seen[m.name] = m
-            members.append(m)
         if self.variants:
             self.variants.check(schema, seen)
-        self.members = members
+        self.members = seen.values()
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
Signature · 1 line
2.4.3
#32 lore raw

PATCH 6/7 qapi: Factor out QAPISchemaObjectTypeMember.check_clash()

In reply to Markus Armbruster

While there, stick in a TODO change key of seen from QAPI name to C name. Can't do it right away, because it would fail the assertion for tests/qapi-schema/args-has-clash.json.

---
 scripts/qapi.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
scripts/qapi.py +7 -4
index a46fded..f0cbb7e 100644
@@ -984,12 +984,10 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert not self.base.variants       # not implemented
             self.base.check(schema)
             for m in self.base.members:
-                assert m.name not in seen
-                seen[m.name] = m
+                m.check_clash(seen)
         for m in self.local_members:
             m.check(schema)
-            assert m.name not in seen
-            seen[m.name] = m
+            m.check_clash(seen)
         if self.variants:
             self.variants.check(schema, seen)
         self.members = seen.values()
@@ -1030,6 +1028,11 @@ class QAPISchemaObjectTypeMember(object):
         self.type = schema.lookup_type(self._type_name)
         assert self.type
 
+    def check_clash(self, seen):
+        # TODO change key of seen from QAPI name to C name
+        assert self.name not in seen
+        seen[self.name] = self
+
 
 class QAPISchemaObjectTypeVariants(object):
     def __init__(self, tag_name, tag_member, variants):
Signature · 1 line
2.4.3
#33 lore raw

PATCH 7/7 qapi: QAPISchemaObjectTypeVariants.check()

In reply to Markus Armbruster

Avoid the ugly flat union / simple union / alternate conditional by doing just the essential work here, namely setting self.tag_member. Move the rest to callers.

---
 scripts/qapi.py | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
scripts/qapi.py +4 -6
index f0cbb7e..86341e6 100644
@@ -988,9 +988,10 @@ class QAPISchemaObjectType(QAPISchemaType):
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)
+        self.members = seen.values()
         if self.variants:
             self.variants.check(schema, seen)
-        self.members = seen.values()
+            assert self.variants.tag_member in self.members
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
@@ -1050,12 +1051,8 @@ class QAPISchemaObjectTypeVariants(object):
         self.variants = variants
 
     def check(self, schema, seen):
-        if self.tag_name:    # flat union
+        if not self.tag_member: # flat union
             self.tag_member = seen[self.tag_name]
-        elif seen:           # simple union
-            assert self.tag_member in seen.itervalues()
-        else:                # alternate
-            self.tag_member.check(schema)
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
             v.check(schema, self.tag_member.type)
@@ -1088,6 +1085,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
 
     def check(self, schema):
+        self.variants.tag_member.check(schema)
         self.variants.check(schema, {})
 
     def json_type(self):
Signature · 1 line
2.4.3
author
#34 lore raw

PATCH 2/4 qapi: Check for QMP collisions of flat union branches

In reply to Markus Armbruster

Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any QMP member names that would conflict with the non-variant QMP members already present from the union's base type (see flat-union-clash-member.json). However, we want QAPISchema*.check() to make the same check, so we can later reduce some of the ad hoc checks.

Basically, all branches of a flat union must be qapi structs with no variants, at which point those members appear in the same JSON object as all non-variant members. We already have a map 'seen' of all non-variant members passed in to QAPISchemaObjectTypeVariants.check(), which we clone for each particular variant (since the members of one variant do not clash with another); all that is additionally needed is to actually check the each member of the variant type do not add any collisions.

For simple unions, the same code happens to work (however, our synthesized wrapper type has a single member 'data' which will never collide with the one non-variant member 'type').

But for alternates, we do NOT want to check the type members for adding collisions (an alternate has no parent JSON object that is merging in member names, the way a flat union does), so we have to pass in an extra flag to distinguish whether we are working with a union or an alternate. The flag is temporary; a later patch will rework how alternates are laid out by creating a new subclass of QAPISchemaObjectTypeMember, and detecting the use of this subclass for variants.tag_member will serve the same purpose.

Note that an early proposal [1] for what eventually became commit ac88219a had QAPISchemaObjectTypeVariant.check() ensure that the variant type was complete, although it was later removed [2]; the checks added here happen to match what that earlier attempt meant to do.

[1] https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg00394.html
[2] https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00450.html

---
v9: new patch, split off from v8 7/17

---
 scripts/qapi.py | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)
scripts/qapi.py +19 -6
index fff4adb..3cf051f 100644
@@ -1047,7 +1047,8 @@ class QAPISchemaObjectTypeVariants(object):
         self.tag_member = tag_member
         self.variants = variants
 
-    def check(self, schema, seen):
+    # TODO drop 'union' param once tag_member is sufficient to spot alternates
+    def check(self, schema, seen, union=True):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
             assert self.tag_member
@@ -1055,17 +1056,29 @@ class QAPISchemaObjectTypeVariants(object):
             assert self.tag_member in seen.itervalues()
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
-            vseen = dict(seen)
-            v.check(schema, self.tag_member.type, vseen)
+            # Reset seen array for each variant, since QMP names from one
+            # branch do not affect another branch
+            v.check(schema, self.tag_member.type, dict(seen), union)
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
-    def check(self, schema, tag_type, seen):
-        QAPISchemaObjectTypeMember.check(self, schema, seen)
+    # TODO drop 'union' param once tag_type is sufficient to spot alternates
+    def check(self, schema, tag_type, seen, union):
+        QAPISchemaObjectTypeMember.check(self, schema, dict(seen))
         assert self.name in tag_type.values
+        if union:
+            # If this variant is used within a union, then each member
+            # field must avoid collisions with the non-variant members
+            # already present in the union.
+            assert isinstance(self.type, QAPISchemaObjectType)
+            assert not self.type.variants       # not implemented
+            self.type.check(schema)
+            for m in self.type.members:
+                assert c_name(m.name) not in seen
+                seen[m.name] = m
 
     # This function exists to support ugly simple union special cases
     # TODO get rid of them, and drop the function
@@ -1088,7 +1101,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
     def check(self, schema):
         seen = {}
         self.variants.tag_member.check(schema, seen)
-        self.variants.check(schema, seen)
+        self.variants.check(schema, seen, False)
 
     def json_type(self):
         return 'value'
Signature · 1 line
2.4.3
author
#35 lore raw

PATCH 3/4 qapi: Fix check for variant tag values collision

In reply to Markus Armbruster

Now that commit e4ba22b3 has separated the C representation of qapi unions so that tag values no longer collide with non-variant members, we must adjust QAPISchemaObjectTypeVariant.check() to match. The fix is conceptually simple - track a separate dictionary of tag names we have seen so far, different from the dictionary of non-variant names. And while the non-variant seen array gets reset for each new variant (because the JSON object does not have collisions between separate branches), the map of tag names is not reset.

---
v9: new patch, split off from v8 7/17

---
 scripts/qapi.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
scripts/qapi.py +4 -3
index 3cf051f..10bf16f 100644
@@ -1055,10 +1055,11 @@ class QAPISchemaObjectTypeVariants(object):
         else:                # simple union or alternate
             assert self.tag_member in seen.itervalues()
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        cases = {}
         for v in self.variants:
             # Reset seen array for each variant, since QMP names from one
             # branch do not affect another branch
-            v.check(schema, self.tag_member.type, dict(seen), union)
+            v.check(schema, self.tag_member.type, dict(seen), cases, union)
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
@@ -1066,8 +1067,8 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
     # TODO drop 'union' param once tag_type is sufficient to spot alternates
-    def check(self, schema, tag_type, seen, union):
-        QAPISchemaObjectTypeMember.check(self, schema, dict(seen))
+    def check(self, schema, tag_type, seen, cases, union):
+        QAPISchemaObjectTypeMember.check(self, schema, cases)
         assert self.name in tag_type.values
         if union:
             # If this variant is used within a union, then each member
Signature · 1 line
2.4.3
author
#36 lore raw

PATCH 4/4 qapi: Consolidate collision detection code

In reply to Markus Armbruster

Rather than having three separate places populate the seen map, it is easier to just factor out a subset of Member.check() that does this as a new method check_collision(), and have the remaining places in ObjectType.check() and Variant.check() call into it. This likewise means a new helper method ObjectType.check_collision(). Later patches can then change the handling of the seen array in just one place, when moving away from ad hoc parser tests.

Note that there was some discrepancy in the existing code on whether name or c_name(name) could not previously be in the seen map; a future patch will clean this up to consistently populate the map via c_name().

---
v9: new patch, split off from v8 7/17; name change from ObjectType.check_qmp() to check_collision(), and new method Member.check_collision(). I'm open to naming suggestions.

---
 scripts/qapi.py | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)
scripts/qapi.py +18 -13
index 10bf16f..b519d30 100644
@@ -981,18 +981,21 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)
-            for m in self.base.members:
-                assert c_name(m.name) not in seen
-                seen[m.name] = m
+            self.base.check_collision(schema, seen)
         for m in self.local_members:
             m.check(schema, seen)
         if self.variants:
             self.variants.check(schema, seen)
         self.members = seen.values()
 
+    # Check that the members of this type do not cause duplicate JSON fields,
+    # and update seen to track the members seen so far
+    def check_collision(self, schema, seen):
+        assert not self.variants       # not implemented
+        self.check(schema)
+        for m in self.members:
+            m.check_collision(seen)
+
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
         return self.name[0] == ':'
@@ -1026,9 +1029,16 @@ class QAPISchemaObjectTypeMember(object):
         self.optional = optional
 
     def check(self, schema, seen):
-        assert self.name not in seen
         self.type = schema.lookup_type(self._type_name)
+        self.check_collision(seen)
+
+    # Check that this member does not collide with anything in seen (the
+    # set of non-variant members when called from QAPISchemaObjectType,
+    # or the set of tag values when called from QAPISchemaObjectTypeVariant),
+    # and update seen accordingly.
+    def check_collision(self, seen):
         assert self.type
+        assert self.name not in seen
         seen[self.name] = self
 
 
@@ -1074,12 +1084,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
             # If this variant is used within a union, then each member
             # field must avoid collisions with the non-variant members
             # already present in the union.
-            assert isinstance(self.type, QAPISchemaObjectType)
-            assert not self.type.variants       # not implemented
-            self.type.check(schema)
-            for m in self.type.members:
-                assert c_name(m.name) not in seen
-                seen[m.name] = m
+            self.type.check_collision(schema, seen)
 
     # This function exists to support ugly simple union special cases
     # TODO get rid of them, and drop the function
Signature · 1 line
2.4.3
author
#37 lore raw

PATCH v8 8/17 qapi: Remove outdated tests related to QMP/branch collisions

In reply to Eric Blake

Now that branches are in a separate C namespace, we can remove the restrictions in the parser that claim a branch name would collide with QMP, and delete the negative tests that are no longer problematic. A separate patch can then add positive tests to qapi-schema-test to test that any corner cases will compile correctly.

This reverts the scripts/qapi.py portion of commit 7b2a5c2, now that the assertions that it plugged are no longer possible.

---
v8: retitle and update commit message, finish reversion of 7b2a5c2
v7: new patch (also temporarily appeared in subset B v10)

---
 scripts/qapi.py                                | 11 ++---------
 tests/Makefile                                 |  3 ---
 tests/qapi-schema/flat-union-clash-branch.err  |  0
 tests/qapi-schema/flat-union-clash-branch.exit |  1 -
 tests/qapi-schema/flat-union-clash-branch.json | 18 ------------------
 tests/qapi-schema/flat-union-clash-branch.out  | 14 --------------
 tests/qapi-schema/flat-union-clash-type.err    |  1 -
 tests/qapi-schema/flat-union-clash-type.exit   |  1 -
 tests/qapi-schema/flat-union-clash-type.json   | 14 --------------
 tests/qapi-schema/flat-union-clash-type.out    |  0
 tests/qapi-schema/union-clash-type.err         |  1 -
 tests/qapi-schema/union-clash-type.exit        |  1 -
 tests/qapi-schema/union-clash-type.json        |  9 ---------
 tests/qapi-schema/union-clash-type.out         |  0
 14 files changed, 2 insertions(+), 72 deletions(-)
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.exit
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.json
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.out
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.exit
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.json
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.out
 delete mode 100644 tests/qapi-schema/union-clash-type.err
 delete mode 100644 tests/qapi-schema/union-clash-type.exit
 delete mode 100644 tests/qapi-schema/union-clash-type.json
 delete mode 100644 tests/qapi-schema/union-clash-type.out
scripts/qapi.py +2 -9
index c571709..a0639e4 100644
@@ -548,8 +548,7 @@ def check_union(expr, expr_info):
     base = expr.get('base')
     discriminator = expr.get('discriminator')
     members = expr['data']
-    values = {'MAX': '(automatic)', 'KIND': '(automatic)',
-              'TYPE': '(automatic)'}
+    values = {'MAX': '(automatic)'}
 
     # Two types of unions, determined by discriminator.
 
@@ -607,19 +606,13 @@ def check_union(expr, expr_info):
                                " of branch '%s'" % key)
 
         # If the discriminator names an enum type, then all members
-        # of 'data' must also be members of the enum type, which in turn
-        # must not collide with the discriminator name.
+        # of 'data' must also be members of the enum type.
         if enum_define:
             if key not in enum_define['enum_values']:
                 raise QAPIExprError(expr_info,
                                     "Discriminator value '%s' is not found in "
                                     "enum '%s'" %
                                     (key, enum_define["enum_name"]))
-            if discriminator in enum_define['enum_values']:
-                raise QAPIExprError(expr_info,
-                                    "Discriminator name '%s' collides with "
-                                    "enum value in '%s'" %
-                                    (discriminator, enum_define["enum_name"]))
 
         # Otherwise, check for conflicts in the generated enum
         else:
tests/Makefile +0 -3
index f448b5a..30879ba 100644
@@ -277,9 +277,7 @@ qapi-schema += flat-union-bad-base.json
 qapi-schema += flat-union-bad-discriminator.json
 qapi-schema += flat-union-base-any.json
 qapi-schema += flat-union-base-union.json
-qapi-schema += flat-union-clash-branch.json
 qapi-schema += flat-union-clash-member.json
-qapi-schema += flat-union-clash-type.json
 qapi-schema += flat-union-empty.json
 qapi-schema += flat-union-inline.json
 qapi-schema += flat-union-int-branch.json
@@ -341,7 +339,6 @@ qapi-schema += union-bad-branch.json
 qapi-schema += union-base-no-discriminator.json
 qapi-schema += union-clash-branches.json
 qapi-schema += union-clash-data.json
-qapi-schema += union-clash-type.json
 qapi-schema += union-empty.json
 qapi-schema += union-invalid-base.json
 qapi-schema += union-max.json
tests/qapi-schema/flat-union-clash-branch.err deleted +0 -0
deleted file mode 100644
index e69de29..0000000
tests/qapi-schema/flat-union-clash-branch.exit deleted +0 -1
deleted file mode 100644
index 573541a..0000000
@@ -1 +0,0 @@
-0
tests/qapi-schema/flat-union-clash-branch.json deleted +0 -18
deleted file mode 100644
index e593336..0000000
@@ -1,18 +0,0 @@
-# Flat union branch name collision
-# FIXME: this parses, but then fails to compile due to a duplicate 'c_d'
-# (one from the base member, the other from the branch name).  We should
-# either reject the collision at parse time, or munge the generated branch
-# name to allow this to compile.
-{ 'enum': 'TestEnum',
-  'data': [ 'base', 'c-d' ] }
-{ 'struct': 'Base',
-  'data': { 'enum1': 'TestEnum', '*c_d': 'str' } }
-{ 'struct': 'Branch1',
-  'data': { 'string': 'str' } }
-{ 'struct': 'Branch2',
-  'data': { 'value': 'int' } }
-{ 'union': 'TestUnion',
-  'base': 'Base',
-  'discriminator': 'enum1',
-  'data': { 'base': 'Branch1',
-            'c-d': 'Branch2' } }
tests/qapi-schema/flat-union-clash-branch.out deleted +0 -14
deleted file mode 100644
index 8e0da73..0000000
@@ -1,14 +0,0 @@
-object :empty
-object Base
-    member enum1: TestEnum optional=False
-    member c_d: str optional=True
-object Branch1
-    member string: str optional=False
-object Branch2
-    member value: int optional=False
-enum TestEnum ['base', 'c-d']
-object TestUnion
-    base Base
-    tag enum1
-    case base: Branch1
-    case c-d: Branch2
tests/qapi-schema/flat-union-clash-type.err deleted +0 -1
deleted file mode 100644
index b44dd40..0000000
@@ -1 +0,0 @@
-tests/qapi-schema/flat-union-clash-type.json:11: Discriminator name 'type' collides with enum value in 'TestEnum'
tests/qapi-schema/flat-union-clash-type.exit deleted +0 -1
deleted file mode 100644
index d00491f..0000000
@@ -1 +0,0 @@
-1
tests/qapi-schema/flat-union-clash-type.json deleted +0 -14
deleted file mode 100644
index 8f710f0..0000000
@@ -1,14 +0,0 @@
-# Flat union branch 'type'
-# Reject this, because we would have a clash in generated C, between the
-# outer tag 'type' and the branch name 'type' within the union.
-# TODO: We could munge the generated C branch name to let it compile.
-{ 'enum': 'TestEnum',
-  'data': [ 'type' ] }
-{ 'struct': 'Base',
-  'data': { 'type': 'TestEnum' } }
-{ 'struct': 'Branch1',
-  'data': { 'string': 'str' } }
-{ 'union': 'TestUnion',
-  'base': 'Base',
-  'discriminator': 'type',
-  'data': { 'type': 'Branch1' } }
tests/qapi-schema/flat-union-clash-type.out deleted +0 -0
deleted file mode 100644
index e69de29..0000000
tests/qapi-schema/union-clash-type.err deleted +0 -1
deleted file mode 100644
index a5dead1..0000000
@@ -1 +0,0 @@
-tests/qapi-schema/union-clash-type.json:8: Union 'TestUnion' member 'kind' clashes with '(automatic)'
tests/qapi-schema/union-clash-type.exit deleted +0 -1
deleted file mode 100644
index d00491f..0000000
@@ -1 +0,0 @@
-1
tests/qapi-schema/union-clash-type.json deleted +0 -9
deleted file mode 100644
index cfc256b..0000000
@@ -1,9 +0,0 @@
-# Union branch 'type'
-# Reject this, because we would have a clash in generated C, between the
-# simple union's implicit tag member 'kind' and the branch name 'kind'
-# within the union.
-# TODO: Even when the generated C is switched to use 'type' rather than
-# 'kind', to match the QMP spelling, the collision should still be detected.
-# Or, we could munge the branch name to allow compilation.
-{ 'union': 'TestUnion',
-  'data': { 'kind': 'int', 'type': 'str' } }
tests/qapi-schema/union-clash-type.out deleted +0 -0
deleted file mode 100644
index e69de29..0000000
Signature · 1 line
2.4.3
#38 lore raw

Re: [Qemu-devel] [PATCH v8 08/17] qapi: Remove outdated tests related to QMP/branch collisions

Quoted 11 lines · Eric Blake

Now that branches are in a separate C namespace, we can remove the restrictions in the parser that claim a branch name would collide with QMP, and delete the negative tests that are no longer problematic. A separate patch can then add positive tests to qapi-schema-test to test that any corner cases will compile correctly.

This reverts the scripts/qapi.py portion of commit 7b2a5c2, now that the assertions that it plugged are no longer possible.

Yay, an easy patch!

author
#39 lore raw

PATCH v8 9/17 qapi: Add positive tests to qapi-schema-test

In reply to Eric Blake

Add positive tests to qapi-schema-test for things that were made possible by recent patches but which caused compile errors due to collisions prior to that point.

This includes:
Use of a member 'base' in a struct with a base class Use of a member name ending in 'Kind' or 'List' Use of a type name starting with 'has_'
Use of a type named 'u'
Use of a union branch name of 'u'
Use of a union branch name starting with 'has_' Use of a union branch name of 'type'

---
v8: new, but collects portions of subset B v10 patches 2, 3, and 16 and subset C v7 patch 6 that were deferred to later.

It might be worth dropping or simplifying this patch, depending on how many corner cases we actually want to test.

---
 tests/qapi-schema/qapi-schema-test.json | 20 ++++++++++++++++++++
 tests/qapi-schema/qapi-schema-test.out  | 31 +++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+)
tests/qapi-schema/qapi-schema-test.json +20 -0
index 44638da..4354604 100644
@@ -47,6 +47,10 @@
   'data': { 'string0': 'str',
             'dict1': 'UserDefTwoDict' } }
 
+# ensure that we don't have an artificial collision on 'base'
+{ 'struct': 'UserDefThree',
+  'base': 'UserDefOne', 'data': { 'base': 'str' } }
+
 # dummy struct to force generation of array types not otherwise mentioned
 { 'struct': 'ForceArrays',
   'data': { 'unused1':['UserDefOne'], 'unused2':['UserDefTwo'],
@@ -113,6 +117,22 @@
             'sizes': ['size'],
             'any': ['any'] } }
 
+# Even though 'u' and 'has_*' are forbidden as struct member names, they
+# should still be valid as a type or union branch name. And although
+# '*Kind' and '*List' are forbidden as type names, they should not be
+# forbidden as a member or branch name.  Flat union branches do not
+# collide with base members.
+{ 'enum': 'EnumName', 'data': [ 'value1', 'has_a', 'u', 'type' ] }
+{ 'struct': 'has_a', 'data': { 'MyKind': 'int', 'MyList': ['int'],
+                               'value1': 'EnumName' } }
+{ 'union': 'u', 'data': { 'u': 'uint8', 'myKind': 'has_a',
+                          'myList': 'has_a', 'has_a': 'has_a' } }
+{ 'union': 'UnionName', 'base': 'has_a', 'discriminator': 'value1',
+  'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero',
+            'u': 'UserDefZero', 'type': 'UserDefZero' } }
+{ 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool',
+                                    'myKind': 'has_a' } }
+
 # testing commands
 { 'command': 'user_def_cmd', 'data': {} }
 { 'command': 'user_def_cmd1', 'data': {'ud1a': 'UserDefOne'} }
tests/qapi-schema/qapi-schema-test.out +31 -0
index 786024e..29c46da 100644
@@ -22,6 +22,8 @@ object :obj-guest-get-time-arg
     member b: int optional=True
 object :obj-guest-sync-arg
     member arg: any optional=False
+object :obj-has_a-wrapper
+    member data: has_a optional=False
 object :obj-int16List-wrapper
     member data: int16List optional=False
 object :obj-int32List-wrapper
@@ -46,6 +48,8 @@ object :obj-uint32List-wrapper
     member data: uint32List optional=False
 object :obj-uint64List-wrapper
     member data: uint64List optional=False
+object :obj-uint8-wrapper
+    member data: uint8 optional=False
 object :obj-uint8List-wrapper
     member data: uint8List optional=False
 object :obj-user_def_cmd1-arg
@@ -57,6 +61,11 @@ alternate AltIntNum
     case i: int
     case n: number
 enum AltIntNumKind ['i', 'n']
+alternate AltName
+    case type: int
+    case u: bool
+    case myKind: has_a
+enum AltNameKind ['type', 'u', 'myKind']
 alternate AltNumInt
     case n: number
     case i: int
@@ -84,6 +93,7 @@ event EVENT_D :obj-EVENT_D-arg
 object Empty1
 object Empty2
     base Empty1
+enum EnumName ['value1', 'has_a', 'u', 'type']
 enum EnumOne ['value1', 'value2', 'value3']
 object EventStructOne
     member struct1: UserDefOne optional=False
@@ -105,6 +115,13 @@ object TestStruct
     member integer: int optional=False
     member boolean: bool optional=False
     member string: str optional=False
+object UnionName
+    base has_a
+    tag value1
+    case value1: UserDefZero
+    case has_a: UserDefZero
+    case u: UserDefZero
+    case type: UserDefZero
 object UserDefA
     member boolean: bool optional=False
     member a_b: int optional=True
@@ -158,6 +175,9 @@ object UserDefOptions
     member u16: uint16List optional=True
     member i64x: int optional=True
     member u64x: uint64 optional=True
+object UserDefThree
+    base UserDefOne
+    member base: str optional=False
 object UserDefTwo
     member string0: str optional=False
     member dict1: UserDefTwoDict optional=False
@@ -201,6 +221,17 @@ command guest-get-time :obj-guest-get-time-arg -> int
    gen=True success_response=True
 command guest-sync :obj-guest-sync-arg -> any
    gen=True success_response=True
+object has_a
+    member MyKind: int optional=False
+    member MyList: intList optional=False
+    member value1: EnumName optional=False
+object u
+    member type: uKind optional=False
+    case u: :obj-uint8-wrapper
+    case myKind: :obj-has_a-wrapper
+    case myList: :obj-has_a-wrapper
+    case has_a: :obj-has_a-wrapper
+enum uKind ['u', 'myKind', 'myList', 'has_a']
 command user_def_cmd None -> None
    gen=True success_response=True
 command user_def_cmd1 :obj-user_def_cmd1-arg -> None
Signature · 1 line
2.4.3
#40 lore raw

Re: [Qemu-devel] [PATCH v8 09/17] qapi: Add positive tests to qapi-schema-test

Quoted 13 lines · Eric Blake

Add positive tests to qapi-schema-test for things that were made possible by recent patches but which caused compile errors due to collisions prior to that point.

This includes:
Use of a member 'base' in a struct with a base class Use of a member name ending in 'Kind' or 'List' Use of a type name starting with 'has_'
Use of a type named 'u'
Use of a union branch name of 'u'
Use of a union branch name starting with 'has_' Use of a union branch name of 'type'

Quoted 8 lines

---
v8: new, but collects portions of subset B v10 patches 2, 3, and 16 and subset C v7 patch 6 that were deferred to later.

It might be worth dropping or simplifying this patch, depending on how many corner cases we actually want to test.

Let's go through your list:

* Use of a member 'base' in a struct with a base class

'base' is no longer special, and testing it is as useful as testing any other non-special member name. I'd drop it. Or am I missing something?

* Use of a member name ending in 'Kind' or 'List'

These aren't special as member names, but they are reserved type names. The test ensures we don't accidentally reserve them as member names. Low probability * low damage = very low risk. But since you wrote the test already, we can just as well keep it.

* Use of a type name starting with 'has_'
* Use of a type named 'u'
* Use of a union branch name of 'u'
* Use of a union branch name starting with 'has_'

Similarly: these are only reserved as member names, very low risk, but why not keep the test.

* Use of a union branch name of 'type'

As far as I know, 'type' isn't reserved anywhere, it's just the name of a simple union's implicit tag member. Testing a tag value doesn't clash with 'type' is as useful as testing any other non-variant member name. I'd drop it. Or am I missing something?

Anything else being tested here?

author
#41 lore raw

Re: [Qemu-devel] [PATCH v8 09/17] qapi: Add positive tests to qapi-schema-test

Quoted 24 lines · Markus Armbruster
Quoted 13 lines · Eric Blake

Add positive tests to qapi-schema-test for things that were made possible by recent patches but which caused compile errors due to collisions prior to that point.

This includes:
Use of a member 'base' in a struct with a base class Use of a member name ending in 'Kind' or 'List' Use of a type name starting with 'has_'
Use of a type named 'u'
Use of a union branch name of 'u'
Use of a union branch name starting with 'has_' Use of a union branch name of 'type'

Quoted 8 lines

---
v8: new, but collects portions of subset B v10 patches 2, 3, and 16 and subset C v7 patch 6 that were deferred to later.

It might be worth dropping or simplifying this patch, depending on how many corner cases we actually want to test.

Quoted 7 lines

Let's go through your list:

* Use of a member 'base' in a struct with a base class

'base' is no longer special, and testing it is as useful as testing any other non-special member name. I'd drop it. Or am I missing something?

Broken prior to commit ddf2190, and no reservation exists. I don't mind dropping it, especially since it quite distinct from the other grouping of collision testing.

Quoted 10 lines

* Use of a member name ending in 'Kind' or 'List'

These aren't special as member names, but they are reserved type names. The test ensures we don't accidentally reserve them as member names. Low probability * low damage = very low risk. But since you wrote the test already, we can just as well keep it.

* Use of a type name starting with 'has_'
* Use of a type named 'u'

Never have been broken, but this adds insurance that we don't break it (and that our reservations added earlier aren't too greedy).

Quoted 1 line

* Use of a union branch name of 'u'

Broken prior to commit e4ba22b3; also proves that the reservation in 5359baf is not too greedy.

Quoted 1 line

* Use of a union branch name starting with 'has_'

Broken prior to commit e4ba22b3; also proves that the reservation in 9fb081e is not too greedy.

Quoted 5 lines

Similarly: these are only reserved as member names, very low risk, but why not keep the test.

* Use of a union branch name of 'type'

Broken prior to commit e4ba22b3. Doesn't have a reservation, so...

Quoted 5 lines

As far as I know, 'type' isn't reserved anywhere, it's just the name of a simple union's implicit tag member. Testing a tag value doesn't clash with 'type' is as useful as testing any other non-variant member name. I'd drop it. Or am I missing something?

...I could indeed drop this part of the test, on the same grounds as dropping the 'base' non-collision tests.

Quoted 3 lines

Anything else being tested here?

I think that covers it. There's also one more positive test added in 10/17, when I rework the layout of alternate types (at that point, it is possible to have an alternate branch named 'max' because it no longer collides with a generated enum _MAX value).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#42 lore raw

PATCH v8 10/17 qapi: Simplify visiting of alternate types

In reply to Eric Blake

Previously, working with alternates required two enums, and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a member of FooKind_lookup[], then use FooKind_lookup[] like we do for other union types.

This has a subtle bug: since the values of FooKind_lookup start at zero, all entries of Foo_qtypes that were not explicitly initialized map to the same branch of the union as the first member of the alternate, rather than triggering a failure in visit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).

However, it IS observable in one case: the behavior of an alternate that contains a 'number' member but no 'int' member differs according to whether the 'number' was first in the qapi definition, and when the input being parsed is an integer; this is because the 'number' parser accepts QTYPE_QINT in addition to the expected QTYPE_QFLOAT. A later patch will worry about fixing alternates to parse all inputs that a non-alternate 'number' would accept, for now it is still marked FIXME.

This patch fixes the validation bug by deleting the indirection, and modifying get_next_type() to directly return a qtype code. There is no longer a need to generate an implicit FooKind array associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names); that also means we no longer have a collision with an alternate branch named 'max'. Next, the generated visitor is fixed to properly detect unexpected qtypes in the switch statement. This is done via the use of a new QAPISchemaAlternateTypeTag subclass and the use of a new member.c_type() method when producing qapi-types. The new subtype also allows us to clean up a TODO left in the previous commit.

Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. If that gets too confusing, we could reintroduce FooKind, but initialize it differently than most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A = QTYPE_QDICT,
FOO_KIND_B = QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to use foo->a or foo->b when inspecting foo->type. But without a current client, I didn't see the point of doing it now.

There is a user-visible side effect to this change, but I consider it to be an improvement. Previously, the invalid QMP command:
{"execute":"blockdev-add", "arguments":{"options":

    {"driver":"raw", "id":"a", "file":true}}}
failed with:
  {"error": {"class": "GenericError",
    "desc": "Invalid parameter type for 'file', expected: QDict"}}
Now it fails with:
  {"error": {"class": "GenericError",
    "desc": "Invalid parameter type for 'file', expected: BlockdevRef"}}

---
v8: no change
v7: rebase onto earlier changes, rework how subtype makes things work
v6: rebase onto tag_member subclass, testsuite, gen_err_check(), and info improvements

---
 docs/qapi-code-gen.txt                  |  3 ---
 include/qapi/visitor-impl.h             |  3 ++-
 include/qapi/visitor.h                  |  8 +++++-
 qapi/qapi-visit-core.c                  |  4 +--
 qapi/qmp-input-visitor.c                |  4 +--
 scripts/qapi-types.py                   | 36 +--------------------------
 scripts/qapi-visit.py                   | 12 +++++----
 scripts/qapi.py                         | 43 ++++++++++++++++++++++-----------
 tests/qapi-schema/alternate-empty.out   |  1 -
 tests/qapi-schema/qapi-schema-test.json |  2 +-
 tests/qapi-schema/qapi-schema-test.out  | 10 +-------
 tests/test-qmp-input-visitor.c          | 33 +++++++++++++------------
 tests/test-qmp-output-visitor.c         | 21 ++++++++++++----
 13 files changed, 86 insertions(+), 94 deletions(-)
docs/qapi-code-gen.txt +0 -3
index 163f547..0c1bd36 100644
@@ -383,9 +383,6 @@ where each branch of the union names a QAPI type.  For example:
    'data': { 'definition': 'BlockdevOptions',
              'reference': 'str' } }
 
-Just like for a simple union, an implicit C enum 'NameKind' is created
-to enumerate the branches for the alternate 'Name'.
-
 Unlike a union, the discriminator string is never passed on the wire
 for the Client JSON Protocol.  Instead, the value's JSON type serves
 as an implicit discriminator, which in turn means that an alternate
include/qapi/visitor-impl.h +2 -1
index 8c0ba57..6d95b36 100644
@@ -32,7 +32,8 @@ struct Visitor
 
     void (*type_enum)(Visitor *v, int *obj, const char * const strings[],
                       const char *kind, const char *name, Error **errp);
-    void (*get_next_type)(Visitor *v, int *kind, const int *qobjects,
+    /* May be NULL; most useful for input visitors. */
+    void (*get_next_type)(Visitor *v, qtype_code *type,
                           const char *name, Error **errp);
 
     void (*type_int)(Visitor *v, int64_t *obj, const char *name, Error **errp);
include/qapi/visitor.h +7 -1
index cfc19a6..b765993 100644
@@ -41,7 +41,13 @@ GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp);
 void visit_end_list(Visitor *v, Error **errp);
 void visit_optional(Visitor *v, bool *present, const char *name,
                     Error **errp);
-void visit_get_next_type(Visitor *v, int *obj, const int *qtypes,
+
+/**
+ * Determine the qtype of the item @name in the current object visit.
+ * For input visitors, set *@type to the correct qtype of a qapi
+ * alternate type; for other visitors, leave *@type unchanged.
+ */
+void visit_get_next_type(Visitor *v, qtype_code *type,
                          const char *name, Error **errp);
 void visit_type_enum(Visitor *v, int *obj, const char * const strings[],
                      const char *kind, const char *name, Error **errp);
qapi/qapi-visit-core.c +2 -2
index 59ed506..3f24daa 100644
@@ -81,11 +81,11 @@ void visit_optional(Visitor *v, bool *present, const char *name,
     }
 }
 
-void visit_get_next_type(Visitor *v, int *obj, const int *qtypes,
+void visit_get_next_type(Visitor *v, qtype_code *type,
                          const char *name, Error **errp)
 {
     if (v->get_next_type) {
-        v->get_next_type(v, obj, qtypes, name, errp);
+        v->get_next_type(v, type, name, errp);
     }
 }
 
qapi/qmp-input-visitor.c +2 -2
index 5dd9ed5..803ffad 100644
@@ -208,7 +208,7 @@ static void qmp_input_end_list(Visitor *v, Error **errp)
     qmp_input_pop(qiv, errp);
 }
 
-static void qmp_input_get_next_type(Visitor *v, int *kind, const int *qobjects,
+static void qmp_input_get_next_type(Visitor *v, qtype_code *type,
                                     const char *name, Error **errp)
 {
     QmpInputVisitor *qiv = to_qiv(v);
@@ -218,7 +218,7 @@ static void qmp_input_get_next_type(Visitor *v, int *kind, const int *qobjects,
         error_setg(errp, QERR_MISSING_PARAMETER, name ? name : "null");
         return;
     }
-    *kind = qobjects[qobject_type(qobj)];
+    *type = qobject_type(qobj);
 }
 
 static void qmp_input_type_int(Visitor *v, int64_t *obj, const char *name,
scripts/qapi-types.py +1 -35
index 403768b..3fd07fd 100644
@@ -47,7 +47,7 @@ def gen_struct_field(member):
     ret += mcgen('''
     %(c_type)s %(c_name)s;
 ''',
-                 c_type=member.type.c_type(), c_name=c_name(member.name))
+                 c_type=member.c_type(), c_name=c_name(member.name))
     return ret
 
 
@@ -111,38 +111,6 @@ static inline %(base)s *qapi_%(c_name)s_base(const %(c_name)s *obj)
                  c_name=c_name(name), base=base.c_name())
 
 
-def gen_alternate_qtypes_decl(name):
-    return mcgen('''
-
-extern const int %(c_name)s_qtypes[];
-''',
-                 c_name=c_name(name))
-
-
-def gen_alternate_qtypes(name, variants):
-    ret = mcgen('''
-
-const int %(c_name)s_qtypes[QTYPE_MAX] = {
-''',
-                c_name=c_name(name))
-
-    for var in variants.variants:
-        qtype = var.type.alternate_qtype()
-        assert qtype
-
-        ret += mcgen('''
-    [%(qtype)s] = %(enum_const)s,
-''',
-                     qtype=qtype,
-                     enum_const=c_enum_const(variants.tag_member.type.name,
-                                             var.name))
-
-    ret += mcgen('''
-};
-''')
-    return ret
-
-
 def gen_variants(variants):
     # FIXME: What purpose does data serve, besides preventing a union that
     # has a branch named 'data'? We use it in qapi-visit.py to decide
@@ -267,9 +235,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
     def visit_alternate_type(self, name, info, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
-        self._fwdefn += gen_alternate_qtypes(name, variants)
         self.decl += gen_object(name, None, [variants.tag_member], variants)
-        self.decl += gen_alternate_qtypes_decl(name)
         self._gen_type_cleanup(name)
 
 # If you link code generated from multiple schemata, you want only one
scripts/qapi-visit.py +7 -5
index 318b8e6..2091a0f 100644
@@ -189,7 +189,7 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
     if (err) {
         goto out;
     }
-    visit_get_next_type(v, (int*) &(*obj)->type, %(c_name)s_qtypes, name, &err);
+    visit_get_next_type(v, &(*obj)->type, name, &err);
     if (err) {
         goto out_obj;
     }
@@ -203,14 +203,14 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
         visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, name, &err);
         break;
 ''',
-                     case=c_enum_const(variants.tag_member.type.name,
-                                       var.name),
+                     case=var.type.alternate_qtype(),
                      c_type=var.type.c_name(),
                      c_name=c_name(var.name))
 
     ret += mcgen('''
     default:
-        abort();
+        error_setg(&err, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "%(name)s");
     }
 out_obj:
     error_propagate(errp, err);
@@ -219,7 +219,8 @@ out_obj:
 out:
     error_propagate(errp, err);
 }
-''')
+''',
+                 name=name)
 
     return ret
 
@@ -425,6 +426,7 @@ fdef.write(mcgen('''
 
 fdecl.write(mcgen('''
 #include "qapi/visitor.h"
+#include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
scripts/qapi.py +29 -14

110 lines not on this page · raw message

tests/qapi-schema/alternate-empty.out +0 -1
index 0f153b6..9b010d8 100644
@@ -1,4 +1,3 @@
 object :empty
 alternate Alt
     case i: int
-enum AltKind ['i']
tests/qapi-schema/qapi-schema-test.json +1 -1
index 4354604..4513d94 100644
@@ -131,7 +131,7 @@
   'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero',
             'u': 'UserDefZero', 'type': 'UserDefZero' } }
 { 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool',
-                                    'myKind': 'has_a' } }
+                                    'myKind': 'has_a', 'max': 'str' } }
 
 # testing commands
 { 'command': 'user_def_cmd', 'data': {} }
tests/qapi-schema/qapi-schema-test.out +1 -9
index 29c46da..eda71b9 100644
@@ -60,32 +60,26 @@ object :obj-user_def_cmd2-arg
 alternate AltIntNum
     case i: int
     case n: number
-enum AltIntNumKind ['i', 'n']
 alternate AltName
     case type: int
     case u: bool
     case myKind: has_a
-enum AltNameKind ['type', 'u', 'myKind']
+    case max: str
 alternate AltNumInt
     case n: number
     case i: int
-enum AltNumIntKind ['n', 'i']
 alternate AltNumStr
     case n: number
     case s: str
-enum AltNumStrKind ['n', 's']
 alternate AltStrBool
     case s: str
     case b: bool
-enum AltStrBoolKind ['s', 'b']
 alternate AltStrInt
     case s: str
     case i: int
-enum AltStrIntKind ['s', 'i']
 alternate AltStrNum
     case s: str
     case n: number
-enum AltStrNumKind ['s', 'n']
 event EVENT_A None
 event EVENT_B None
 event EVENT_C :obj-EVENT_C-arg
@@ -129,7 +123,6 @@ alternate UserDefAlternate
     case uda: UserDefA
     case s: str
     case i: int
-enum UserDefAlternateKind ['uda', 's', 'i']
 object UserDefB
     member intb: int optional=False
     member a-b: bool optional=True
@@ -198,7 +191,6 @@ event __ORG.QEMU_X-EVENT __org.qemu_x-Struct
 alternate __org.qemu_x-Alt
     case __org.qemu_x-branch: str
     case b: __org.qemu_x-Base
-enum __org.qemu_x-AltKind ['__org.qemu_x-branch', 'b']
 object __org.qemu_x-Base
     member __org.qemu_x-member1: __org.qemu_x-Enum optional=False
 enum __org.qemu_x-Enum ['__org.qemu_x-value']
tests/test-qmp-input-visitor.c +18 -15
index 3f6bc4d..552fc74 100644
@@ -343,14 +343,14 @@ static void test_visitor_in_alternate(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42");
     visit_type_UserDefAlternate(v, &tmp, NULL, &error_abort);
-    g_assert_cmpint(tmp->type, ==, USER_DEF_ALTERNATE_KIND_I);
+    g_assert_cmpint(tmp->type, ==, QTYPE_QINT);
     g_assert_cmpint(tmp->u.i, ==, 42);
     qapi_free_UserDefAlternate(tmp);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "'string'");
     visit_type_UserDefAlternate(v, &tmp, NULL, &error_abort);
-    g_assert_cmpint(tmp->type, ==, USER_DEF_ALTERNATE_KIND_S);
+    g_assert_cmpint(tmp->type, ==, QTYPE_QSTRING);
     g_assert_cmpstr(tmp->u.s, ==, "string");
     qapi_free_UserDefAlternate(tmp);
     visitor_input_teardown(data, NULL);
@@ -386,11 +386,10 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     qapi_free_AltStrBool(asb);
     visitor_input_teardown(data, NULL);
 
-    /* FIXME: Order of alternate should not affect semantics; asn should
-     * parse the same as ans */
+    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
     visit_type_AltStrNum(v, &asn, NULL, &err);
-    /* FIXME g_assert_cmpint(asn->type, == ALT_STR_NUM_KIND_N); */
+    /* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */
     /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */
     g_assert(err);
     error_free(err);
@@ -398,30 +397,34 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     qapi_free_AltStrNum(asn);
     visitor_input_teardown(data, NULL);
 
+    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
-    visit_type_AltNumStr(v, &ans, NULL, &error_abort);
-    g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N);
-    g_assert_cmpfloat(ans->u.n, ==, 42);
+    visit_type_AltNumStr(v, &ans, NULL, &err);
+    /* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */
+    /* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */
+    g_assert(err);
+    error_free(err);
+    err = NULL;
     qapi_free_AltNumStr(ans);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "42");
     visit_type_AltStrInt(v, &asi, NULL, &error_abort);
-    g_assert_cmpint(asi->type, ==, ALT_STR_INT_KIND_I);
+    g_assert_cmpint(asi->type, ==, QTYPE_QINT);
     g_assert_cmpint(asi->u.i, ==, 42);
     qapi_free_AltStrInt(asi);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "42");
     visit_type_AltIntNum(v, &ain, NULL, &error_abort);
-    g_assert_cmpint(ain->type, ==, ALT_INT_NUM_KIND_I);
+    g_assert_cmpint(ain->type, ==, QTYPE_QINT);
     g_assert_cmpint(ain->u.i, ==, 42);
     qapi_free_AltIntNum(ain);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "42");
     visit_type_AltNumInt(v, &ani, NULL, &error_abort);
-    g_assert_cmpint(ani->type, ==, ALT_NUM_INT_KIND_I);
+    g_assert_cmpint(ani->type, ==, QTYPE_QINT);
     g_assert_cmpint(ani->u.i, ==, 42);
     qapi_free_AltNumInt(ani);
     visitor_input_teardown(data, NULL);
@@ -438,14 +441,14 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltStrNum(v, &asn, NULL, &error_abort);
-    g_assert_cmpint(asn->type, ==, ALT_STR_NUM_KIND_N);
+    g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(asn->u.n, ==, 42.5);
     qapi_free_AltStrNum(asn);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltNumStr(v, &ans, NULL, &error_abort);
-    g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N);
+    g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(ans->u.n, ==, 42.5);
     qapi_free_AltNumStr(ans);
     visitor_input_teardown(data, NULL);
@@ -460,14 +463,14 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltIntNum(v, &ain, NULL, &error_abort);
-    g_assert_cmpint(ain->type, ==, ALT_INT_NUM_KIND_N);
+    g_assert_cmpint(ain->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(ain->u.n, ==, 42.5);
     qapi_free_AltIntNum(ain);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltNumInt(v, &ani, NULL, &error_abort);
-    g_assert_cmpint(ani->type, ==, ALT_NUM_INT_KIND_N);
+    g_assert_cmpint(ani->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(ani->u.n, ==, 42.5);
     qapi_free_AltNumInt(ani);
     visitor_input_teardown(data, NULL);
tests/test-qmp-output-visitor.c +16 -5
index 9364843..695db32 100644
@@ -449,20 +449,31 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,
                                        const void *unused)
 {
     QObject *arg;
-    Error *err = NULL;
+    UserDefAlternate *tmp;
 
-    UserDefAlternate *tmp = g_malloc0(sizeof(UserDefAlternate));
-    tmp->type = USER_DEF_ALTERNATE_KIND_I;
+    tmp = g_new0(UserDefAlternate, 1);
+    tmp->type = QTYPE_QINT;
     tmp->u.i = 42;
 
-    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &err);
-    g_assert(err == NULL);
+    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &error_abort);
     arg = qmp_output_get_qobject(data->qov);
 
     g_assert(qobject_type(arg) == QTYPE_QINT);
     g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42);
 
     qapi_free_UserDefAlternate(tmp);
+
+    tmp = g_malloc0(sizeof(UserDefAlternate));
+    tmp->type = QTYPE_QSTRING;
+    tmp->u.s = g_strdup("hello");
+
+    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &error_abort);
+    arg = qmp_output_get_qobject(data->qov);
+
+    g_assert(qobject_type(arg) == QTYPE_QSTRING);
+    g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello");
+
+    qapi_free_UserDefAlternate(tmp);
 }
 
 static void test_visitor_out_empty(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
#43 lore raw

Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types

Quoted 3 lines · Eric Blake

Previously, working with alternates required two enums, and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a member of FooKind_lookup[], then use

member of FooKind, actually.

Quoted 1 line

FooKind_lookup[] like we do for other union types.

You probably mean FooKind here as well.

Quoted 10 lines

This has a subtle bug: since the values of FooKind_lookup start at zero, all entries of Foo_qtypes that were not explicitly initialized map to the same branch of the union as the first member of the alternate, rather than triggering a failure in visit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).

Yes, I remember us discussing this bug.

While reading code to double-check your description, I stumbled over this beauty in generated qapi-visit.c:

    visit_get_next_type(v, (int*) &(*obj)->type, BlockdevRef_qtypes, name, &err);

This casts enum BlockdevRefKind * to int *, which assumes the compiler represents the enum BlockdevRefKind as int or unsigned. It is free to use any integer type, though. Common mistake of programmers with insufficiently developed wariness of C's subtleties.

visit_get_next_type() passes the fishy int * on to v->get_next_type(). Only implementation is qmp_input_get_next_type(), which uses it so:

    *kind = qobjects[qobject_type(qobj)];

Latent death trap.

Does your patch clean this up?

Quoted 11 lines

However, it IS observable in one case: the behavior of an alternate that contains a 'number' member but no 'int' member differs according to whether the 'number' was first in the qapi definition, and when the input being parsed is an integer; this is because the 'number' parser accepts QTYPE_QINT in addition to the expected QTYPE_QFLOAT. A later patch will worry about fixing alternates to parse all inputs that a non-alternate 'number' would accept, for now it is still marked FIXME.

This patch fixes the validation bug by deleting the indirection, and modifying get_next_type() to directly return a qtype code.

get_next_type() doesn't return anything. Do you mean "store a qtype code"?

Quoted 1 line

There is no longer a need to generate an implicit FooKind array

FooKind is an enum, not an array.

Quoted 28 lines

associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names); that also means we no longer have a collision with an alternate branch named 'max'. Next, the generated visitor is fixed to properly detect unexpected qtypes in the switch statement. This is done via the use of a new QAPISchemaAlternateTypeTag subclass and the use of a new member.c_type() method when producing qapi-types. The new subtype also allows us to clean up a TODO left in the previous commit.

Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. If that gets too confusing, we could reintroduce FooKind, but initialize it differently than most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A = QTYPE_QDICT,
FOO_KIND_B = QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to use foo->a or foo->b when inspecting foo->type. But without a current client, I didn't see the point of doing it now.

There is a user-visible side effect to this change, but I consider it to be an improvement. Previously, the invalid QMP command:
{"execute":"blockdev-add", "arguments":{"options":

Quoted 7 lines
    {"driver":"raw", "id":"a", "file":true}}}
failed with:
  {"error": {"class": "GenericError",
    "desc": "Invalid parameter type for 'file', expected: QDict"}}
Now it fails with:
  {"error": {"class": "GenericError",
    "desc": "Invalid parameter type for 'file', expected: BlockdevRef"}}

I wonder how that happens. Perhaps it's obvious in the patch.

QMP introspection isn't affected, because we carefully minimized the information to expose there.

Quoted 58 lines

---
v8: no change
v7: rebase onto earlier changes, rework how subtype makes things work
v6: rebase onto tag_member subclass, testsuite, gen_err_check(), and info improvements

---
 docs/qapi-code-gen.txt                  |  3 ---
 include/qapi/visitor-impl.h             |  3 ++-
 include/qapi/visitor.h                  |  8 +++++-
 qapi/qapi-visit-core.c                  |  4 +--
 qapi/qmp-input-visitor.c                |  4 +--
 scripts/qapi-types.py                   | 36 +--------------------------
 scripts/qapi-visit.py                   | 12 +++++----
 scripts/qapi.py                         | 43 ++++++++++++++++++++++-----------
 tests/qapi-schema/alternate-empty.out   |  1 -
 tests/qapi-schema/qapi-schema-test.json |  2 +-
 tests/qapi-schema/qapi-schema-test.out  | 10 +-------
 tests/test-qmp-input-visitor.c          | 33 +++++++++++++------------
 tests/test-qmp-output-visitor.c         | 21 ++++++++++++----
 13 files changed, 86 insertions(+), 94 deletions(-)
docs/qapi-code-gen.txt +0 -3
index 163f547..0c1bd36 100644
@@ -383,9 +383,6 @@ where each branch of the union names a QAPI type.  For example:
    'data': { 'definition': 'BlockdevOptions',
              'reference': 'str' } }
 
-Just like for a simple union, an implicit C enum 'NameKind' is created
-to enumerate the branches for the alternate 'Name'.
-
 Unlike a union, the discriminator string is never passed on the wire
 for the Client JSON Protocol.  Instead, the value's JSON type serves
 as an implicit discriminator, which in turn means that an alternate
include/qapi/visitor-impl.h +2 -1
index 8c0ba57..6d95b36 100644
@@ -32,7 +32,8 @@ struct Visitor
 
     void (*type_enum)(Visitor *v, int *obj, const char * const strings[],
                       const char *kind, const char *name, Error **errp);
-    void (*get_next_type)(Visitor *v, int *kind, const int *qobjects,
+    /* May be NULL; most useful for input visitors. */
+    void (*get_next_type)(Visitor *v, qtype_code *type,
                           const char *name, Error **errp);
 
     void (*type_int)(Visitor *v, int64_t *obj, const char *name, Error **errp);
include/qapi/visitor.h +0 -0
index cfc19a6..b765993 100644
@@ -41,7 +41,13 @@ GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp);
 void visit_end_list(Visitor *v, Error **errp);
 void visit_optional(Visitor *v, bool *present, const char *name,

include/qapi/visitor.h GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp); · line 51

Quoted 10 lines
include/qapi/visitor.h +7 -1
                     Error **errp);
-void visit_get_next_type(Visitor *v, int *obj, const int *qtypes,
+
+/**
+ * Determine the qtype of the item @name in the current object visit.
+ * For input visitors, set *@type to the correct qtype of a qapi
+ * alternate type; for other visitors, leave *@type unchanged.
+ */
+void visit_get_next_type(Visitor *v, qtype_code *type,
                          const char *name, Error **errp);

Naive question: what makes a visitor an input visitor?

Quoted 47 lines
include/qapi/visitor.h +0 -0
 void visit_type_enum(Visitor *v, int *obj, const char * const strings[],
                      const char *kind, const char *name, Error **errp);
qapi/qapi-visit-core.c +2 -2
index 59ed506..3f24daa 100644
@@ -81,11 +81,11 @@ void visit_optional(Visitor *v, bool *present, const char *name,
     }
 }
 
-void visit_get_next_type(Visitor *v, int *obj, const int *qtypes,
+void visit_get_next_type(Visitor *v, qtype_code *type,
                          const char *name, Error **errp)
 {
     if (v->get_next_type) {
-        v->get_next_type(v, obj, qtypes, name, errp);
+        v->get_next_type(v, type, name, errp);
     }
 }
 
qapi/qmp-input-visitor.c +2 -2
index 5dd9ed5..803ffad 100644
@@ -208,7 +208,7 @@ static void qmp_input_end_list(Visitor *v, Error **errp)
     qmp_input_pop(qiv, errp);
 }
 
-static void qmp_input_get_next_type(Visitor *v, int *kind, const int *qobjects,
+static void qmp_input_get_next_type(Visitor *v, qtype_code *type,
                                     const char *name, Error **errp)
 {
     QmpInputVisitor *qiv = to_qiv(v);
@@ -218,7 +218,7 @@ static void qmp_input_get_next_type(Visitor *v, int *kind, const int *qobjects,
         error_setg(errp, QERR_MISSING_PARAMETER, name ? name : "null");
         return;
     }
-    *kind = qobjects[qobject_type(qobj)];
+    *type = qobject_type(qobj);
 }
 
 static void qmp_input_type_int(Visitor *v, int64_t *obj, const char *name,
scripts/qapi-types.py +0 -0
index 403768b..3fd07fd 100644

scripts/qapi-types.py def gen_struct_field(member): · line 50

Quoted 5 lines
scripts/qapi-types.py +1 -1
@@ -47,7 +47,7 @@ def gen_struct_field(member):
     ret += mcgen('''
     %(c_type)s %(c_name)s;
 ''',
-                 c_type=member.type.c_type(), c_name=c_name(member.name))
+                 c_type=member.c_type(), c_name=c_name(member.name))

This uses the new member.c_type() defined further down in the patch.

No change if member is an instance of QAPISchemaObjectTypeMember.

Quoted 57 lines
scripts/qapi-types.py +0 -34
     return ret
 
 
@@ -111,38 +111,6 @@ static inline %(base)s *qapi_%(c_name)s_base(const %(c_name)s *obj)
                  c_name=c_name(name), base=base.c_name())
 
 
-def gen_alternate_qtypes_decl(name):
-    return mcgen('''
-
-extern const int %(c_name)s_qtypes[];
-''',
-                 c_name=c_name(name))
-
-
-def gen_alternate_qtypes(name, variants):
-    ret = mcgen('''
-
-const int %(c_name)s_qtypes[QTYPE_MAX] = {
-''',
-                c_name=c_name(name))
-
-    for var in variants.variants:
-        qtype = var.type.alternate_qtype()
-        assert qtype
-
-        ret += mcgen('''
-    [%(qtype)s] = %(enum_const)s,
-''',
-                     qtype=qtype,
-                     enum_const=c_enum_const(variants.tag_member.type.name,
-                                             var.name))
-
-    ret += mcgen('''
-};
-''')
-    return ret
-
-
 def gen_variants(variants):
     # FIXME: What purpose does data serve, besides preventing a union that
     # has a branch named 'data'? We use it in qapi-visit.py to decide
@@ -267,9 +235,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
     def visit_alternate_type(self, name, info, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
-        self._fwdefn += gen_alternate_qtypes(name, variants)
         self.decl += gen_object(name, None, [variants.tag_member], variants)
-        self.decl += gen_alternate_qtypes_decl(name)
         self._gen_type_cleanup(name)
 
 # If you link code generated from multiple schemata, you want only one
scripts/qapi-visit.py +0 -0
index 318b8e6..2091a0f 100644

scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 195

Quoted 8 lines
scripts/qapi-visit.py +1 -1
@@ -189,7 +189,7 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
     if (err) {
         goto out;
     }
-    visit_get_next_type(v, (int*) &(*obj)->type, %(c_name)s_qtypes, name, &err);
+    visit_get_next_type(v, &(*obj)->type, name, &err);
     if (err) {
         goto out_obj;
     }

Yes, your patch disarms the latent death trap: no more pointer casting.

Quoted 5 lines
scripts/qapi-visit.py +0 -1
@@ -203,14 +203,14 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
         visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, name, &err);
         break;
 ''',
-                     case=c_enum_const(variants.tag_member.type.name,

scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 213

Quoted 10 lines
scripts/qapi-visit.py +3 -2
-                                       var.name),
+                     case=var.type.alternate_qtype(),
                      c_type=var.type.c_name(),
                      c_name=c_name(var.name))
 
     ret += mcgen('''
     default:
-        abort();
+        error_setg(&err, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "%(name)s");

Okay, this is where the new error message comes from.

Before, default is unreachable, because (*obj)->type got erroneously set the enum's first member when none of the alternate's variants matches the qtype.

After, (*obj)->type *is* the qtype, and we do reach default when no variant matches.

How can name be null?

I really need to finish the QERR_ killing job.

Quoted 31 lines
scripts/qapi-visit.py +3 -1
     }
 out_obj:
     error_propagate(errp, err);
@@ -219,7 +219,8 @@ out_obj:
 out:
     error_propagate(errp, err);
 }
-''')
+''',
+                 name=name)
 
     return ret
 
@@ -425,6 +426,7 @@ fdef.write(mcgen('''
 
 fdecl.write(mcgen('''
 #include "qapi/visitor.h"
+#include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
scripts/qapi.py +1 -1
index a0639e4..bd74470 100644
@@ -627,15 +627,15 @@ def check_union(expr, expr_info):
 def check_alternate(expr, expr_info):
     name = expr['alternate']
     members = expr['data']
-    values = {'MAX': '(automatic)'}
+    values = {}

scripts/qapi.py def check_union(expr, expr_info): · line 638

Quoted 10 lines
scripts/qapi.py +2 -2
     types_seen = {}
 
     # Check every branch
     for (key, value) in members.items():
         check_name(expr_info, "Member of alternate '%s'" % name, key)
 
-        # Check for conflicts in the generated enum
-        c_key = camel_to_upper(key)
+        # Check for conflicts in the branch names
+        c_key = c_name(key)

Why c_name()?

Quoted 9 lines
scripts/qapi.py +2 -0
         if c_key in values:
             raise QAPIExprError(expr_info,
                                 "Alternate '%s' member '%s' clashes with '%s'"
@@ -1029,13 +1029,17 @@ class QAPISchemaObjectTypeMember(object):
         assert self.type
         seen[self.name] = self
 
+    def c_type(self):
+        return self.type.c_type()

scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1042

Quoted 10 lines
scripts/qapi.py +3 -1
+
 
 class QAPISchemaObjectTypeVariants(object):
     def __init__(self, tag_name, tag_member, variants):
         # Flat unions pass tag_name but not tag_member.
         # Simple unions and alternates pass tag_member but not tag_name.
         # After check(), tag_member is always set, and tag_name remains
-        # a reliable witness of being used by a flat union.
+        # a reliable witness of being used by a flat union, and
+        # tag_member.type being None is a reliable witness of an alternate.

A member without a type? Ugh! I wouldn't dare breaking invariants like that.

Of course, an alternate's tag member still has a type: qtype_code. It's just not declared in the schema. Should it be a built-in type then?

Quoted 16 lines
scripts/qapi.py +1 -2
         assert bool(tag_member) != bool(tag_name)
         assert (isinstance(tag_name, str) or
                 isinstance(tag_member, QAPISchemaObjectTypeMember))
@@ -1045,8 +1049,7 @@ class QAPISchemaObjectTypeVariants(object):
         self.tag_member = tag_member
         self.variants = variants
 
-    # TODO drop union once alternates can be distinguished by tag_member
-    def check(self, schema, seen, union=True):
+    def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
             assert self.tag_member
@@ -1054,25 +1057,25 @@ class QAPISchemaObjectTypeVariants(object):
             assert self.tag_member in seen.itervalues()
         else:                # alternate

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066

Quoted 10 lines
scripts/qapi.py +3 -2
             self.tag_member.check(schema, seen)
-        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        if not isinstance(self.tag_member, QAPISchemaAlternateTypeTag):
+            assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         cases = OrderedDict()
         for v in self.variants:
             # Reset seen array for each variant, since QMP names from one
             # branch do not affect another branch, nor add to all_members
-            v.check(schema, self.tag_member.type, dict(seen), cases, union)
+            v.check(schema, self.tag_member.type, dict(seen), cases)

I expect some rebase churn around here, so I'm not reviewing closely.

Quoted 8 lines
scripts/qapi.py +0 -2
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
-    # TODO drop union once alternates can be distinguished by tag_type
-    def check(self, schema, tag_type, seen, cases, union):

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1080

Quoted 10 lines
scripts/qapi.py +3 -2
+    def check(self, schema, tag_type, seen, cases):
         # cases is case names we must not collide with
         QAPISchemaObjectTypeMember.check(self, schema, cases)
-        assert self.name in tag_type.values
-        if union:
+        if tag_type:
             # seen is QMP names our members must not collide with
+            assert self.name in tag_type.values
             self.type.check_qmp(schema, seen)
 

My patches move the member name collision checking to QAPISchemaObjectType.check().

I suspect alternate branch name collision checking should similarly move to QAPISchemaAlternateType.check().

Union branch name collision checking is superfluous, because they already get checked by enum member name collision checking.

Quoted 16 lines
scripts/qapi.py +3 -1
     # This function exists to support ugly simple union special cases
@@ -1094,7 +1097,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
 
     def check(self, schema):
-        self.variants.check(schema, {}, False)
+        self.variants.check(schema, {})
 
     def json_type(self):
         return 'value'
@@ -1103,6 +1106,18 @@ class QAPISchemaAlternateType(QAPISchemaType):
         visitor.visit_alternate_type(self.name, self.info, self.variants)
 
 
+class QAPISchemaAlternateTypeTag(QAPISchemaObjectTypeMember):
+    def __init__(self):

scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1120

Quoted 10 lines
scripts/qapi.py +10 -0
+        QAPISchemaObjectTypeMember.__init__(self, 'type', '', False)
+
+    def check(self, schema, seen):
+        assert len(seen) == 0
+        seen[self.name] = self
+
+    def c_type(self):
+        return 'qtype_code'
+
+

This is a hack to work around the lack of a qtype_code type. I suspect creating such a type would be simpler in the end. Safer, too, because it would avoid having members without a type, which scares me.

Quoted 26 lines
scripts/qapi.py +1 -1
 class QAPISchemaCommand(QAPISchemaEntity):
     def __init__(self, name, info, arg_type, ret_type, gen, success_response):
         QAPISchemaEntity.__init__(self, name, info)
@@ -1296,7 +1311,7 @@ class QAPISchema(object):
         data = expr['data']
         variants = [self._make_variant(key, value)
                     for (key, value) in data.iteritems()]
-        tag_member = self._make_implicit_tag(name, info, variants)
+        tag_member = QAPISchemaAlternateTypeTag()
         self._def_entity(
             QAPISchemaAlternateType(name, info,
                                     QAPISchemaObjectTypeVariants(None,
tests/qapi-schema/alternate-empty.out +0 -1
index 0f153b6..9b010d8 100644
@@ -1,4 +1,3 @@
 object :empty
 alternate Alt
     case i: int
-enum AltKind ['i']
tests/qapi-schema/qapi-schema-test.json +0 -0
index 4354604..4513d94 100644

tests/qapi-schema/qapi-schema-test.json line 134

Quoted 5 lines
tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -131,7 +131,7 @@
   'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero',
             'u': 'UserDefZero', 'type': 'UserDefZero' } }
 { 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool',
-                                    'myKind': 'has_a' } }
+                                    'myKind': 'has_a', 'max': 'str' } }

Here, you add the positive test that alternate name 'max' works.

One, not mentioned in the commit message.

Two, the commit message says we may reintroduce FooKind if working with qtype_code turns out to be too confusing. If we ever do that, alternate name 'max' breaks, doesn't it? Shouldn't we keep it reserved then, just in case?

Quoted 97 lines
tests/qapi-schema/qapi-schema-test.json +0 -0
 # testing commands
 { 'command': 'user_def_cmd', 'data': {} }
tests/qapi-schema/qapi-schema-test.out +1 -9
index 29c46da..eda71b9 100644
@@ -60,32 +60,26 @@ object :obj-user_def_cmd2-arg
 alternate AltIntNum
     case i: int
     case n: number
-enum AltIntNumKind ['i', 'n']
 alternate AltName
     case type: int
     case u: bool
     case myKind: has_a
-enum AltNameKind ['type', 'u', 'myKind']
+    case max: str
 alternate AltNumInt
     case n: number
     case i: int
-enum AltNumIntKind ['n', 'i']
 alternate AltNumStr
     case n: number
     case s: str
-enum AltNumStrKind ['n', 's']
 alternate AltStrBool
     case s: str
     case b: bool
-enum AltStrBoolKind ['s', 'b']
 alternate AltStrInt
     case s: str
     case i: int
-enum AltStrIntKind ['s', 'i']
 alternate AltStrNum
     case s: str
     case n: number
-enum AltStrNumKind ['s', 'n']
 event EVENT_A None
 event EVENT_B None
 event EVENT_C :obj-EVENT_C-arg
@@ -129,7 +123,6 @@ alternate UserDefAlternate
     case uda: UserDefA
     case s: str
     case i: int
-enum UserDefAlternateKind ['uda', 's', 'i']
 object UserDefB
     member intb: int optional=False
     member a-b: bool optional=True
@@ -198,7 +191,6 @@ event __ORG.QEMU_X-EVENT __org.qemu_x-Struct
 alternate __org.qemu_x-Alt
     case __org.qemu_x-branch: str
     case b: __org.qemu_x-Base
-enum __org.qemu_x-AltKind ['__org.qemu_x-branch', 'b']
 object __org.qemu_x-Base
     member __org.qemu_x-member1: __org.qemu_x-Enum optional=False
 enum __org.qemu_x-Enum ['__org.qemu_x-value']
tests/test-qmp-input-visitor.c +5 -5
index 3f6bc4d..552fc74 100644
@@ -343,14 +343,14 @@ static void test_visitor_in_alternate(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42");
     visit_type_UserDefAlternate(v, &tmp, NULL, &error_abort);
-    g_assert_cmpint(tmp->type, ==, USER_DEF_ALTERNATE_KIND_I);
+    g_assert_cmpint(tmp->type, ==, QTYPE_QINT);
     g_assert_cmpint(tmp->u.i, ==, 42);
     qapi_free_UserDefAlternate(tmp);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "'string'");
     visit_type_UserDefAlternate(v, &tmp, NULL, &error_abort);
-    g_assert_cmpint(tmp->type, ==, USER_DEF_ALTERNATE_KIND_S);
+    g_assert_cmpint(tmp->type, ==, QTYPE_QSTRING);
     g_assert_cmpstr(tmp->u.s, ==, "string");
     qapi_free_UserDefAlternate(tmp);
     visitor_input_teardown(data, NULL);
@@ -386,11 +386,10 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     qapi_free_AltStrBool(asb);
     visitor_input_teardown(data, NULL);
 
-    /* FIXME: Order of alternate should not affect semantics; asn should
-     * parse the same as ans */
+    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
     visit_type_AltStrNum(v, &asn, NULL, &err);
-    /* FIXME g_assert_cmpint(asn->type, == ALT_STR_NUM_KIND_N); */
+    /* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */
     /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */
     g_assert(err);
     error_free(err);
@@ -398,30 +397,34 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     qapi_free_AltStrNum(asn);
     visitor_input_teardown(data, NULL);
 
+    /* FIXME: integer should parse as number */

tests/test-qmp-input-visitor.c static void test_visitor_in_alternate_number(TestInputVisitorData *data, · line 407

Quoted 10 lines
tests/test-qmp-input-visitor.c +6 -3
     v = visitor_input_test_init(data, "42");
-    visit_type_AltNumStr(v, &ans, NULL, &error_abort);
-    g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N);
-    g_assert_cmpfloat(ans->u.n, ==, 42);
+    visit_type_AltNumStr(v, &ans, NULL, &err);
+    /* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */
+    /* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */
+    g_assert(err);
+    error_free(err);
+    err = NULL;

What's happening here? Whatever it is, the commit message didn't prepare me for it...

Quoted 66 lines
tests/test-qmp-input-visitor.c +7 -7
     qapi_free_AltNumStr(ans);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "42");
     visit_type_AltStrInt(v, &asi, NULL, &error_abort);
-    g_assert_cmpint(asi->type, ==, ALT_STR_INT_KIND_I);
+    g_assert_cmpint(asi->type, ==, QTYPE_QINT);
     g_assert_cmpint(asi->u.i, ==, 42);
     qapi_free_AltStrInt(asi);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "42");
     visit_type_AltIntNum(v, &ain, NULL, &error_abort);
-    g_assert_cmpint(ain->type, ==, ALT_INT_NUM_KIND_I);
+    g_assert_cmpint(ain->type, ==, QTYPE_QINT);
     g_assert_cmpint(ain->u.i, ==, 42);
     qapi_free_AltIntNum(ain);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "42");
     visit_type_AltNumInt(v, &ani, NULL, &error_abort);
-    g_assert_cmpint(ani->type, ==, ALT_NUM_INT_KIND_I);
+    g_assert_cmpint(ani->type, ==, QTYPE_QINT);
     g_assert_cmpint(ani->u.i, ==, 42);
     qapi_free_AltNumInt(ani);
     visitor_input_teardown(data, NULL);
@@ -438,14 +441,14 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltStrNum(v, &asn, NULL, &error_abort);
-    g_assert_cmpint(asn->type, ==, ALT_STR_NUM_KIND_N);
+    g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(asn->u.n, ==, 42.5);
     qapi_free_AltStrNum(asn);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltNumStr(v, &ans, NULL, &error_abort);
-    g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N);
+    g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(ans->u.n, ==, 42.5);
     qapi_free_AltNumStr(ans);
     visitor_input_teardown(data, NULL);
@@ -460,14 +463,14 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltIntNum(v, &ain, NULL, &error_abort);
-    g_assert_cmpint(ain->type, ==, ALT_INT_NUM_KIND_N);
+    g_assert_cmpint(ain->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(ain->u.n, ==, 42.5);
     qapi_free_AltIntNum(ain);
     visitor_input_teardown(data, NULL);
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltNumInt(v, &ani, NULL, &error_abort);
-    g_assert_cmpint(ani->type, ==, ALT_NUM_INT_KIND_N);
+    g_assert_cmpint(ani->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(ani->u.n, ==, 42.5);
     qapi_free_AltNumInt(ani);
     visitor_input_teardown(data, NULL);
tests/test-qmp-output-visitor.c +0 -0
index 9364843..695db32 100644
@@ -449,20 +449,31 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,
                                        const void *unused)

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 456

Quoted 10 lines
tests/test-qmp-output-visitor.c +3 -3
 {
     QObject *arg;
-    Error *err = NULL;
+    UserDefAlternate *tmp;
 
-    UserDefAlternate *tmp = g_malloc0(sizeof(UserDefAlternate));
-    tmp->type = USER_DEF_ALTERNATE_KIND_I;
+    tmp = g_new0(UserDefAlternate, 1);
+    tmp->type = QTYPE_QINT;
     tmp->u.i = 42;

Coding style touched up. Okay.

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 458

Quoted 4 lines
tests/test-qmp-output-visitor.c +1 -2
-    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &err);
-    g_assert(err == NULL);
+    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &error_abort);

We need to make up our mind whether to use g_assert(err == NULL) or &error_abort in tests. Wholesale conversion could be in order. I like &error_abort, because it's more concise.

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 466

Quoted 8 lines
tests/test-qmp-output-visitor.c +2 -0
     arg = qmp_output_get_qobject(data->qov);
 
     g_assert(qobject_type(arg) == QTYPE_QINT);
     g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42);
 
     qapi_free_UserDefAlternate(tmp);
+
+    tmp = g_malloc0(sizeof(UserDefAlternate));

g_new0(UserDefAlternate, 1), please.

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 476

Quoted 10 lines
tests/test-qmp-output-visitor.c +10 -0
+    tmp->type = QTYPE_QSTRING;
+    tmp->u.s = g_strdup("hello");
+
+    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &error_abort);
+    arg = qmp_output_get_qobject(data->qov);
+
+    g_assert(qobject_type(arg) == QTYPE_QSTRING);
+    g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello");
+
+    qapi_free_UserDefAlternate(tmp);

New test, not mentioned in commit message. Separate patch, perhaps, along with the nearby coding style touch ups?

Quoted 3 lines
tests/test-qmp-output-visitor.c +0 -0
 }
 
 static void test_visitor_out_empty(TestOutputVisitorData *data,
author
#44 lore raw

Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types

Quoted 7 lines · Markus Armbruster
Quoted 3 lines · Eric Blake

Previously, working with alternates required two enums, and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a member of FooKind_lookup[], then use

member of FooKind, actually.

Or entry in the FooKind_lookup[] array.

Quoted 4 lines
Quoted 2 lines

FooKind_lookup[] like we do for other union types.

You probably mean FooKind here as well.

I'll play with the wording.

Quoted 24 lines
Quoted 11 lines

This has a subtle bug: since the values of FooKind_lookup start at zero, all entries of Foo_qtypes that were not explicitly initialized map to the same branch of the union as the first member of the alternate, rather than triggering a failure in visit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).

Yes, I remember us discussing this bug.

While reading code to double-check your description, I stumbled over this beauty in generated qapi-visit.c:

    visit_get_next_type(v, (int*) &(*obj)->type, BlockdevRef_qtypes, name, &err);

This casts enum BlockdevRefKind * to int *, which assumes the compiler represents the enum BlockdevRefKind as int or unsigned. It is free to use any integer type, though. Common mistake of programmers with insufficiently developed wariness of C's subtleties.

Quoted 8 lines

visit_get_next_type() passes the fishy int * on to v->get_next_type(). Only implementation is qmp_input_get_next_type(), which uses it so:

    *kind = qobjects[qobject_type(qobj)];

Latent death trap.

Does your patch clean this up?

Yes, and I need to also document that this is an additional bug fix.

Quoted 9 lines
Quoted 9 lines

However, it IS observable in one case: the behavior of an alternate that contains a 'number' member but no 'int' member differs according to whether the 'number' was first in the qapi definition, and when the input being parsed is an integer; this is because the 'number' parser accepts QTYPE_QINT in addition to the expected QTYPE_QFLOAT. A later patch will worry about fixing alternates to parse all inputs that a non-alternate 'number' would accept, for now it is still marked FIXME.

See [1] below.

Quoted 6 lines
Quoted 3 lines

This patch fixes the validation bug by deleting the indirection, and modifying get_next_type() to directly return a qtype code.

get_next_type() doesn't return anything. Do you mean "store a qtype code"?

Yes.

Quoted 4 lines
Quoted 2 lines

There is no longer a need to generate an implicit FooKind array

FooKind is an enum, not an array.

...to generate an implicit FooKind enum, nor FooKind_lookup[] array.

Quoted 12 lines
Quoted 12 lines

associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names); that also means we no longer have a collision with an alternate branch named 'max'. Next, the generated visitor is fixed to properly detect unexpected qtypes in the switch statement. This is done via the use of a new QAPISchemaAlternateTypeTag subclass and the use of a new member.c_type() method when producing qapi-types. The new subtype also allows us to clean up a TODO left in the previous commit.

Quoted 12 lines
Quoted 4 lines

Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. If that gets too confusing, we could reintroduce FooKind, but initialize it differently than

Quoted 8 lines

most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A = QTYPE_QDICT,
FOO_KIND_B = QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to use foo->a or foo->b when inspecting foo->type. But without a current client, I didn't see the point of doing it now.

You have a point below that we either need to reserve MAX and require no case-insensitive clashes, or that we will never want to add it. I'm leaning towards never going back, because the new way feels so much nicer.

Quoted 13 lines
Quoted 5 lines

There is a user-visible side effect to this change, but I consider it to be an improvement. Previously, the invalid QMP command:
{"execute":"blockdev-add", "arguments":{"options":

Quoted 7 lines
    {"driver":"raw", "id":"a", "file":true}}}
failed with:
  {"error": {"class": "GenericError",
    "desc": "Invalid parameter type for 'file', expected: QDict"}}
Now it fails with:
  {"error": {"class": "GenericError",
    "desc": "Invalid parameter type for 'file', expected: BlockdevRef"}}
Quoted 1 line

I wonder how that happens. Perhaps it's obvious in the patch.

I think you found it below.

Quoted 3 lines

QMP introspection isn't affected, because we carefully minimized the information to expose there.

Ooh, nice tidbit to add.

Quoted 9 lines

include/qapi/visitor.h GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp); · line 51

Quoted 7 lines
include/qapi/visitor.h +6 -0
+/**
+ * Determine the qtype of the item @name in the current object visit.
+ * For input visitors, set *@type to the correct qtype of a qapi
+ * alternate type; for other visitors, leave *@type unchanged.
+ */
+void visit_get_next_type(Visitor *v, qtype_code *type,
                          const char *name, Error **errp);

Naive question: what makes a visitor an input visitor?

I've got a later patch in my queue that adds a lot more documentation:
http://repo.or.cz/qemu/ericb.git/commitdiff/f7674a87e72

+/* This file describes the client view for visiting a map between
+ * generated QAPI C structs and another representation (command line
+ * options, strings, or QObjects). An input visitor converts from
+ * some other form into QAPI representation; an output visitor
+ * converts from QAPI back into another form. In the descriptions
+ * below, an object or dictionary refers to a JSON '{}', and an array
+ * or list refers to a JSON '[]'. These functions seldom need to be
+ * called directly, but are instead used by code generated by
+ * scripts/qapi-visit.py. For the visitor callback contracts, see
+ * visitor-impl.h. */

Quoted 4 lines
Quoted 4 lines

+++ b/scripts/qapi-visit.py

scripts/qapi-visit.py +0 -0
@@ -189,7 +189,7 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
     if (err) {
         goto out;
Quoted 8 lines

scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 195

Quoted 6 lines
scripts/qapi-visit.py +1 -1
     }
-    visit_get_next_type(v, (int*) &(*obj)->type, %(c_name)s_qtypes, name, &err);
+    visit_get_next_type(v, &(*obj)->type, name, &err);
     if (err) {
         goto out_obj;
     }

Yes, your patch disarms the latent death trap: no more pointer casting.

Indeed, I noticed the cleanup as well (I'm quite familiar with the unsafe nature of casting enum* because you cannot guarantee its size), but failed to call out the trap in my commit message.

Quoted 19 lines
Quoted 6 lines
scripts/qapi-visit.py +0 -1
@@ -203,14 +203,14 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
         visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, name, &err);
         break;
 ''',
-                     case=c_enum_const(variants.tag_member.type.name,

scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 213

Quoted 10 lines
scripts/qapi-visit.py +3 -2
-                                       var.name),
+                     case=var.type.alternate_qtype(),
                      c_type=var.type.c_name(),
                      c_name=c_name(var.name))
 
     ret += mcgen('''
     default:
-        abort();
+        error_setg(&err, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "%(name)s");

Okay, this is where the new error message comes from.

Quoted 8 lines

Before, default is unreachable, because (*obj)->type got erroneously set the enum's first member when none of the alternate's variants matches the qtype.

After, (*obj)->type *is* the qtype, and we do reach default when no variant matches.

How can name be null?

When you have the qapi representation ['MyAlternate'], you will have qapi_visit_type_MyAlternateList() which passes NULL for the name of each list member (because names are only present for objects, not lists).

Quoted 2 lines

I really need to finish the QERR_ killing job.

Agreed. But shouldn't stall this patch, though.

Quoted 10 lines

scripts/qapi.py def check_union(expr, expr_info): · line 638

Quoted 8 lines
scripts/qapi.py +2 -2
     # Check every branch
     for (key, value) in members.items():
         check_name(expr_info, "Member of alternate '%s'" % name, key)
 
-        # Check for conflicts in the generated enum
-        c_key = camel_to_upper(key)
+        # Check for conflicts in the branch names
+        c_key = c_name(key)

Why c_name()?

So that 'a-b' and 'a_b' are properly flagged as conflicting (they map to the same c_name).

Quoted 9 lines

scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1042

Quoted 8 lines

class QAPISchemaObjectTypeVariants(object):
def __init__(self, tag_name, tag_member, variants):
# Flat unions pass tag_name but not tag_member.
# Simple unions and alternates pass tag_member but not tag_name. # After check(), tag_member is always set, and tag_name remains

scripts/qapi.py +2 -1
-        # a reliable witness of being used by a flat union.
+        # a reliable witness of being used by a flat union, and
+        # tag_member.type being None is a reliable witness of an alternate.
Quoted 5 lines

A member without a type? Ugh! I wouldn't dare breaking invariants like that.

Of course, an alternate's tag member still has a type: qtype_code. It's just not declared in the schema. Should it be a built-in type then?

It's not a builtin that can ever be referenced in the .json files. But I could probably come up with something, if it would make you feel better.

Quoted 8 lines

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066

Quoted 5 lines

for v in self.variants:
# Reset seen array for each variant, since QMP names from one # branch do not affect another branch, nor add to all_members

scripts/qapi.py +1 -1
-            v.check(schema, self.tag_member.type, dict(seen), cases, union)
+            v.check(schema, self.tag_member.type, dict(seen), cases)

I expect some rebase churn around here, so I'm not reviewing closely.

Indeed. All the more reason for me to post a v9 spin (and maybe defer the question of a non-None type for tag_member until after that post).

Quoted 6 lines

My patches move the member name collision checking to QAPISchemaObjectType.check().

I suspect alternate branch name collision checking should similarly move to QAPISchemaAlternateType.check().

Yep, already that way in my pending v9 series after incorporating your patches.

Quoted 8 lines
Quoted 8 lines
scripts/qapi.py +8 -0
+class QAPISchemaAlternateTypeTag(QAPISchemaObjectTypeMember):
+    def __init__(self):
+        QAPISchemaObjectTypeMember.__init__(self, 'type', '', False)
+
+    def check(self, schema, seen):
+        assert len(seen) == 0
+        seen[self.name] = self
+
Quoted 8 lines

scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1120

Quoted 4 lines
scripts/qapi.py +4 -0
+    def c_type(self):
+        return 'qtype_code'
+
+

This is a hack to work around the lack of a qtype_code type. I suspect creating such a type would be simpler in the end. Safer, too, because it would avoid having members without a type, which scares me.

I can play with dropping c_type() here in favor of adding a qtype_code special class, but I may still need to keep this QAPISchemaAlternateTypeTag subclass.

Quoted 11 lines

tests/qapi-schema/qapi-schema-test.json line 134

Quoted 7 lines

+++ b/tests/qapi-schema/qapi-schema-test.json

tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -131,7 +131,7 @@
   'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero',
             'u': 'UserDefZero', 'type': 'UserDefZero' } }
 { 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool',
-                                    'myKind': 'has_a' } }
+                                    'myKind': 'has_a', 'max': 'str' } }

Here, you add the positive test that alternate name 'max' works.

One, not mentioned in the commit message.

D'oh.

Quoted 6 lines

Two, the commit message says we may reintroduce FooKind if working with qtype_code turns out to be too confusing. If we ever do that, alternate name 'max' breaks, doesn't it? Shouldn't we keep it reserved then, just in case?

See my comment above; at this point, I doubt we'll ever want to go back, so maybe I just need to be more definitive in stating that.

Quoted 20 lines
Quoted 20 lines
tests/test-qmp-input-visitor.c +3 -3
+++ b/tests/test-qmp-input-visitor.c
 
@@ -386,11 +386,10 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     qapi_free_AltStrBool(asb);
     visitor_input_teardown(data, NULL);
 
-    /* FIXME: Order of alternate should not affect semantics; asn should
-     * parse the same as ans */
+    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
     visit_type_AltStrNum(v, &asn, NULL, &err);
-    /* FIXME g_assert_cmpint(asn->type, == ALT_STR_NUM_KIND_N); */
+    /* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */
     /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */
     g_assert(err);
     error_free(err);
@@ -398,30 +397,34 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     qapi_free_AltStrNum(asn);
     visitor_input_teardown(data, NULL);
 
Quoted 14 lines

tests/test-qmp-input-visitor.c static void test_visitor_in_alternate_number(TestInputVisitorData *data, · line 407

Quoted 11 lines
tests/test-qmp-input-visitor.c +7 -3
+    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
-    visit_type_AltNumStr(v, &ans, NULL, &error_abort);
-    g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N);
-    g_assert_cmpfloat(ans->u.n, ==, 42);
+    visit_type_AltNumStr(v, &ans, NULL, &err);
+    /* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */
+    /* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */
+    g_assert(err);
+    error_free(err);
+    err = NULL;

What's happening here? Whatever it is, the commit message didn't prepare me for it...

See [1] above. 'asn' is now parsing the same as 'ans' (we are no longer sensitive to whether 'number' was the first member of the alternate), but it isn't until patch 11/17 that we fix things that 'ans' and 'asn' both properly parse '1' as a number instead of rejecting it as an integer.

Quoted 17 lines

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 456

Quoted 13 lines
tests/test-qmp-output-visitor.c +4 -3
+++ b/tests/test-qmp-output-visitor.c
@@ -449,20 +449,31 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,
                                        const void *unused)
 {
     QObject *arg;
-    Error *err = NULL;
+    UserDefAlternate *tmp;
 
-    UserDefAlternate *tmp = g_malloc0(sizeof(UserDefAlternate));
-    tmp->type = USER_DEF_ALTERNATE_KIND_I;
+    tmp = g_new0(UserDefAlternate, 1);
+    tmp->type = QTYPE_QINT;
     tmp->u.i = 42;

Coding style touched up. Okay.

Quoted 1 line
Quoted 7 lines

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 458

Quoted 3 lines
tests/test-qmp-output-visitor.c +1 -2
-    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &err);
-    g_assert(err == NULL);
+    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &error_abort);

We need to make up our mind whether to use g_assert(err == NULL) or &error_abort in tests. Wholesale conversion could be in order. I like &error_abort, because it's more concise.

Wholesale conversion dead-ahead, in 14/17 of this subset.

Quoted 12 lines

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 466

Quoted 9 lines
     arg = qmp_output_get_qobject(data->qov);

     g_assert(qobject_type(arg) == QTYPE_QINT);
     g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42);

     qapi_free_UserDefAlternate(tmp);
tests/test-qmp-output-visitor.c +2 -0
+
+    tmp = g_malloc0(sizeof(UserDefAlternate));

g_new0(UserDefAlternate, 1), please.

Quoted 13 lines

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 476

Quoted 10 lines
tests/test-qmp-output-visitor.c +10 -0
+    tmp->type = QTYPE_QSTRING;
+    tmp->u.s = g_strdup("hello");
+
+    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &error_abort);
+    arg = qmp_output_get_qobject(data->qov);
+
+    g_assert(qobject_type(arg) == QTYPE_QSTRING);
+    g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello");
+
+    qapi_free_UserDefAlternate(tmp);

New test, not mentioned in commit message. Separate patch, perhaps, along with the nearby coding style touch ups?

Yes, I will split this portion of the test changes out to a separate commit.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#45 lore raw

Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types

Quoted 10 lines · Eric Blake
Quoted 7 lines · Markus Armbruster
Quoted 3 lines · Eric Blake

Previously, working with alternates required two enums, and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a member of FooKind_lookup[], then use

member of FooKind, actually.

Or entry in the FooKind_lookup[] array.

The generated

const int BlockdevRef_qtypes[QTYPE_MAX] = {
[QTYPE_QDICT] = BLOCKDEV_REF_KIND_DEFINITION, [QTYPE_QSTRING] = BLOCKDEV_REF_KIND_REFERENCE,
};

maps from qtype_code to BlockdevRefKind, except it uses int instead of BlockdevRefKind, so it can be passed to visit_get_next_type().

Quoted 61 lines
Quoted 3 lines
Quoted 1 line

FooKind_lookup[] like we do for other union types.

You probably mean FooKind here as well.

I'll play with the wording.

Quoted 25 lines
Quoted 11 lines

This has a subtle bug: since the values of FooKind_lookup start at zero, all entries of Foo_qtypes that were not explicitly initialized map to the same branch of the union as the first member of the alternate, rather than triggering a failure in visit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).

Yes, I remember us discussing this bug.

While reading code to double-check your description, I stumbled over this beauty in generated qapi-visit.c:

    visit_get_next_type(v, (int*) &(*obj)->type, BlockdevRef_qtypes,

name, &err);

This casts enum BlockdevRefKind * to int *, which assumes the compiler represents the enum BlockdevRefKind as int or unsigned. It is free to use any integer type, though. Common mistake of programmers with insufficiently developed wariness of C's subtleties.

Quoted 8 lines

visit_get_next_type() passes the fishy int * on to v->get_next_type(). Only implementation is qmp_input_get_next_type(), which uses it so:

    *kind = qobjects[qobject_type(qobj)];

Latent death trap.

Does your patch clean this up?

Yes, and I need to also document that this is an additional bug fix.

Quoted 9 lines
Quoted 9 lines

However, it IS observable in one case: the behavior of an alternate that contains a 'number' member but no 'int' member differs according to whether the 'number' was first in the qapi definition, and when the input being parsed is an integer; this is because the 'number' parser accepts QTYPE_QINT in addition to the expected QTYPE_QFLOAT. A later patch will worry about fixing alternates to parse all inputs that a non-alternate 'number' would accept, for now it is still marked FIXME.

See [1] below.

Quoted 6 lines
Quoted 3 lines

This patch fixes the validation bug by deleting the indirection, and modifying get_next_type() to directly return a qtype code.

get_next_type() doesn't return anything. Do you mean "store a qtype code"?

Quoted 8 lines

Yes.

Quoted 4 lines
Quoted 2 lines

There is no longer a need to generate an implicit FooKind array

FooKind is an enum, not an array.

...to generate an implicit FooKind enum, nor FooKind_lookup[] array.

Yes.

Quoted 24 lines
Quoted 11 lines
Quoted 11 lines

associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names); that also means we no longer have a collision with an alternate branch named 'max'. Next, the generated visitor is fixed to properly detect unexpected qtypes in the switch statement. This is done via the use of a new QAPISchemaAlternateTypeTag subclass and the use of a new member.c_type() method when producing qapi-types. The new subtype also allows us to clean up a TODO left in the previous commit.

Quoted 12 lines
Quoted 4 lines

Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. If that gets too confusing, we could reintroduce FooKind, but initialize it differently than

Quoted 8 lines

most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A = QTYPE_QDICT,
FOO_KIND_B = QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to use foo->a or foo->b when inspecting foo->type. But without a current client, I didn't see the point of doing it now.

Quoted 3 lines

You have a point below that we either need to reserve MAX and require no case-insensitive clashes, or that we will never want to add it. I'm leaning towards never going back, because the new way feels so much nicer.

Perhaps un-reserving MAX and making the collision checking more lenient should be a separate follow-up patch we could easily revert. Probably useful only if this patch doesn't make 2.5.

Quoted 37 lines
Quoted 13 lines
Quoted 5 lines

There is a user-visible side effect to this change, but I consider it to be an improvement. Previously, the invalid QMP command:
{"execute":"blockdev-add", "arguments":{"options":

Quoted 7 lines
    {"driver":"raw", "id":"a", "file":true}}}
failed with:
  {"error": {"class": "GenericError",
    "desc": "Invalid parameter type for 'file', expected: QDict"}}
Now it fails with:
  {"error": {"class": "GenericError",
    "desc": "Invalid parameter type for 'file', expected: BlockdevRef"}}
Quoted 1 line

I wonder how that happens. Perhaps it's obvious in the patch.

I think you found it below.

Quoted 3 lines

QMP introspection isn't affected, because we carefully minimized the information to expose there.

Ooh, nice tidbit to add.

Quoted 9 lines

include/qapi/visitor.h GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp); · line 51

Quoted 7 lines
include/qapi/visitor.h +6 -0
+/**
+ * Determine the qtype of the item @name in the current object visit.
+ * For input visitors, set *@type to the correct qtype of a qapi
+ * alternate type; for other visitors, leave *@type unchanged.
+ */
+void visit_get_next_type(Visitor *v, qtype_code *type,
                          const char *name, Error **errp);

Naive question: what makes a visitor an input visitor?

I've got a later patch in my queue that adds a lot more documentation:
http://repo.or.cz/qemu/ericb.git/commitdiff/f7674a87e72

Quoted 10 lines

+/* This file describes the client view for visiting a map between
+ * generated QAPI C structs and another representation (command line
+ * options, strings, or QObjects). An input visitor converts from
+ * some other form into QAPI representation; an output visitor
+ * converts from QAPI back into another form. In the descriptions
+ * below, an object or dictionary refers to a JSON '{}', and an array
+ * or list refers to a JSON '[]'. These functions seldom need to be
+ * called directly, but are instead used by code generated by
+ * scripts/qapi-visit.py. For the visitor callback contracts, see
+ * visitor-impl.h. */

That's a useful step towards a visitors contract.

Quoted 40 lines
Quoted 4 lines
Quoted 4 lines

+++ b/scripts/qapi-visit.py

scripts/qapi-visit.py +0 -0
@@ -189,7 +189,7 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
     if (err) {
         goto out;
Quoted 8 lines

scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 195

Quoted 6 lines
scripts/qapi-visit.py +1 -1
     }
-    visit_get_next_type(v, (int*) &(*obj)->type, %(c_name)s_qtypes, name, &err);
+    visit_get_next_type(v, &(*obj)->type, name, &err);
     if (err) {
         goto out_obj;
     }

Yes, your patch disarms the latent death trap: no more pointer casting.

Indeed, I noticed the cleanup as well (I'm quite familiar with the unsafe nature of casting enum* because you cannot guarantee its size), but failed to call out the trap in my commit message.

Quoted 23 lines
Quoted 6 lines
scripts/qapi-visit.py +0 -1
@@ -203,14 +203,14 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
         visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, name, &err);
         break;
 ''',
-                     case=c_enum_const(variants.tag_member.type.name,

scripts/qapi-visit.py void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error · line 213

Quoted 10 lines
scripts/qapi-visit.py +3 -2
-                                       var.name),
+                     case=var.type.alternate_qtype(),
                      c_type=var.type.c_name(),
                      c_name=c_name(var.name))
 
     ret += mcgen('''
     default:
-        abort();
+        error_setg(&err, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "%(name)s");

Okay, this is where the new error message comes from.

Before, default is unreachable, because (*obj)->type got erroneously set the enum's first member when none of the alternate's variants matches the qtype.

Quoted 8 lines
Quoted 4 lines

After, (*obj)->type *is* the qtype, and we do reach default when no variant matches.

How can name be null?

When you have the qapi representation ['MyAlternate'], you will have qapi_visit_type_MyAlternateList() which passes NULL for the name of each list member (because names are only present for objects, not lists).

Examining the occurences of visit_type_BlockdevRef* in generated qapi-visit.c shows that parameter name is commonly the member name. It appears to be used only for error messages.

Works fine when the object containing the member is obvious. With multiple objects, the recipient of the error message has to guess the object containing the member. If the member name isn't unique, the error message is ambiguous. Crap, but not crap we can afford to improve right now.

Array elements are anonymous, and I guess that's what led to NULL. Stupid choice. Make it "array element" or something. Code becomes slightly simpler, error message becomes slightly less cruel. This could be simple enough to improve now.

Quoted 3 lines
Quoted 1 line

I really need to finish the QERR_ killing job.

Agreed. But shouldn't stall this patch, though.

Certainly not.

Quoted 73 lines
Quoted 10 lines

scripts/qapi.py def check_union(expr, expr_info): · line 638

Quoted 8 lines
scripts/qapi.py +2 -2
     # Check every branch
     for (key, value) in members.items():
         check_name(expr_info, "Member of alternate '%s'" % name, key)
 
-        # Check for conflicts in the generated enum
-        c_key = camel_to_upper(key)
+        # Check for conflicts in the branch names
+        c_key = c_name(key)

Why c_name()?

So that 'a-b' and 'a_b' are properly flagged as conflicting (they map to the same c_name).

Quoted 9 lines

scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1042

Quoted 8 lines

class QAPISchemaObjectTypeVariants(object):
def __init__(self, tag_name, tag_member, variants):
# Flat unions pass tag_name but not tag_member.
# Simple unions and alternates pass tag_member but not tag_name. # After check(), tag_member is always set, and tag_name remains

scripts/qapi.py +2 -1
-        # a reliable witness of being used by a flat union.
+        # a reliable witness of being used by a flat union, and
+        # tag_member.type being None is a reliable witness of an alternate.
Quoted 5 lines

A member without a type? Ugh! I wouldn't dare breaking invariants like that.

Of course, an alternate's tag member still has a type: qtype_code. It's just not declared in the schema. Should it be a built-in type then?

It's not a builtin that can ever be referenced in the .json files. But I could probably come up with something, if it would make you feel better.

Quoted 8 lines

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066

Quoted 5 lines

for v in self.variants:
# Reset seen array for each variant, since QMP names from one # branch do not affect another branch, nor add to all_members

scripts/qapi.py +1 -1
-            v.check(schema, self.tag_member.type, dict(seen), cases, union)
+            v.check(schema, self.tag_member.type, dict(seen), cases)

I expect some rebase churn around here, so I'm not reviewing closely.

Indeed. All the more reason for me to post a v9 spin (and maybe defer the question of a non-None type for tag_member until after that post).

Quoted 6 lines

My patches move the member name collision checking to QAPISchemaObjectType.check().

I suspect alternate branch name collision checking should similarly move to QAPISchemaAlternateType.check().

Yep, already that way in my pending v9 series after incorporating your patches.

Quoted 8 lines
Quoted 8 lines
scripts/qapi.py +8 -0
+class QAPISchemaAlternateTypeTag(QAPISchemaObjectTypeMember):
+    def __init__(self):
+        QAPISchemaObjectTypeMember.__init__(self, 'type', '', False)
+
+    def check(self, schema, seen):
+        assert len(seen) == 0
+        seen[self.name] = self
+
Quoted 8 lines

scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1120

Quoted 4 lines
scripts/qapi.py +4 -0
+    def c_type(self):
+        return 'qtype_code'
+
+

This is a hack to work around the lack of a qtype_code type. I suspect creating such a type would be simpler in the end. Safer, too, because it would avoid having members without a type, which scares me.

Quoted 3 lines

I can play with dropping c_type() here in favor of adding a qtype_code special class, but I may still need to keep this QAPISchemaAlternateTypeTag subclass.

Let's give it a try and see.

Quoted 59 lines
Quoted 11 lines

tests/qapi-schema/qapi-schema-test.json line 134

Quoted 7 lines

+++ b/tests/qapi-schema/qapi-schema-test.json

tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -131,7 +131,7 @@
   'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero',
             'u': 'UserDefZero', 'type': 'UserDefZero' } }
 { 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool',
-                                    'myKind': 'has_a' } }
+                                    'myKind': 'has_a', 'max': 'str' } }

Here, you add the positive test that alternate name 'max' works.

One, not mentioned in the commit message.

D'oh.

Quoted 6 lines

Two, the commit message says we may reintroduce FooKind if working with qtype_code turns out to be too confusing. If we ever do that, alternate name 'max' breaks, doesn't it? Shouldn't we keep it reserved then, just in case?

See my comment above; at this point, I doubt we'll ever want to go back, so maybe I just need to be more definitive in stating that.

Quoted 20 lines
Quoted 20 lines
tests/test-qmp-input-visitor.c +3 -3
+++ b/tests/test-qmp-input-visitor.c
 
@@ -386,11 +386,10 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     qapi_free_AltStrBool(asb);
     visitor_input_teardown(data, NULL);
 
-    /* FIXME: Order of alternate should not affect semantics; asn should
-     * parse the same as ans */
+    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
     visit_type_AltStrNum(v, &asn, NULL, &err);
-    /* FIXME g_assert_cmpint(asn->type, == ALT_STR_NUM_KIND_N); */
+    /* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */
     /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */
     g_assert(err);
     error_free(err);
@@ -398,30 +397,34 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     qapi_free_AltStrNum(asn);
     visitor_input_teardown(data, NULL);
 
Quoted 14 lines

tests/test-qmp-input-visitor.c static void test_visitor_in_alternate_number(TestInputVisitorData *data, · line 407

Quoted 11 lines
tests/test-qmp-input-visitor.c +7 -3
+    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
-    visit_type_AltNumStr(v, &ans, NULL, &error_abort);
-    g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N);
-    g_assert_cmpfloat(ans->u.n, ==, 42);
+    visit_type_AltNumStr(v, &ans, NULL, &err);
+    /* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */
+    /* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */
+    g_assert(err);
+    error_free(err);
+    err = NULL;

What's happening here? Whatever it is, the commit message didn't prepare me for it...

Quoted 4 lines

See [1] above. 'asn' is now parsing the same as 'ans' (we are no longer sensitive to whether 'number' was the first member of the alternate), but it isn't until patch 11/17 that we fix things that 'ans' and 'asn' both properly parse '1' as a number instead of rejecting it as an integer.

Recommend to stick a brief note into the commit message that this temporarily breaks test so-and-so, marked FIXME.

Quoted 54 lines
Quoted 17 lines

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 456

Quoted 13 lines
tests/test-qmp-output-visitor.c +4 -3
+++ b/tests/test-qmp-output-visitor.c
@@ -449,20 +449,31 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,
                                        const void *unused)
 {
     QObject *arg;
-    Error *err = NULL;
+    UserDefAlternate *tmp;
 
-    UserDefAlternate *tmp = g_malloc0(sizeof(UserDefAlternate));
-    tmp->type = USER_DEF_ALTERNATE_KIND_I;
+    tmp = g_new0(UserDefAlternate, 1);
+    tmp->type = QTYPE_QINT;
     tmp->u.i = 42;

Coding style touched up. Okay.

Quoted 1 line
Quoted 7 lines

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 458

Quoted 3 lines
tests/test-qmp-output-visitor.c +1 -2
-    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &err);
-    g_assert(err == NULL);
+    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &error_abort);

We need to make up our mind whether to use g_assert(err == NULL) or &error_abort in tests. Wholesale conversion could be in order. I like &error_abort, because it's more concise.

Wholesale conversion dead-ahead, in 14/17 of this subset.

Quoted 12 lines

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 466

Quoted 9 lines
     arg = qmp_output_get_qobject(data->qov);

     g_assert(qobject_type(arg) == QTYPE_QINT);
     g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42);

     qapi_free_UserDefAlternate(tmp);
tests/test-qmp-output-visitor.c +2 -0
+
+    tmp = g_malloc0(sizeof(UserDefAlternate));

g_new0(UserDefAlternate, 1), please.

Quoted 13 lines

tests/test-qmp-output-visitor.c static void test_visitor_out_alternate(TestOutputVisitorData *data, · line 476

Quoted 10 lines
tests/test-qmp-output-visitor.c +10 -0
+    tmp->type = QTYPE_QSTRING;
+    tmp->u.s = g_strdup("hello");
+
+    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &error_abort);
+    arg = qmp_output_get_qobject(data->qov);
+
+    g_assert(qobject_type(arg) == QTYPE_QSTRING);
+    g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello");
+
+    qapi_free_UserDefAlternate(tmp);

New test, not mentioned in commit message. Separate patch, perhaps, along with the nearby coding style touch ups?

Yes, I will split this portion of the test changes out to a separate commit.

#46 lore raw

Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types

In reply to Eric Blake
Quoted 110 lines · Eric Blake
Quoted 7 lines · Markus Armbruster
Quoted 3 lines · Eric Blake

Previously, working with alternates required two enums, and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a member of FooKind_lookup[], then use

member of FooKind, actually.

Or entry in the FooKind_lookup[] array.

Quoted 4 lines
Quoted 2 lines

FooKind_lookup[] like we do for other union types.

You probably mean FooKind here as well.

I'll play with the wording.

Quoted 25 lines
Quoted 11 lines

This has a subtle bug: since the values of FooKind_lookup start at zero, all entries of Foo_qtypes that were not explicitly initialized map to the same branch of the union as the first member of the alternate, rather than triggering a failure in visit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).

Yes, I remember us discussing this bug.

While reading code to double-check your description, I stumbled over this beauty in generated qapi-visit.c:

    visit_get_next_type(v, (int*) &(*obj)->type, BlockdevRef_qtypes,

name, &err);

This casts enum BlockdevRefKind * to int *, which assumes the compiler represents the enum BlockdevRefKind as int or unsigned. It is free to use any integer type, though. Common mistake of programmers with insufficiently developed wariness of C's subtleties.

Quoted 8 lines

visit_get_next_type() passes the fishy int * on to v->get_next_type(). Only implementation is qmp_input_get_next_type(), which uses it so:

    *kind = qobjects[qobject_type(qobj)];

Latent death trap.

Does your patch clean this up?

Yes, and I need to also document that this is an additional bug fix.

Quoted 9 lines
Quoted 9 lines

However, it IS observable in one case: the behavior of an alternate that contains a 'number' member but no 'int' member differs according to whether the 'number' was first in the qapi definition, and when the input being parsed is an integer; this is because the 'number' parser accepts QTYPE_QINT in addition to the expected QTYPE_QFLOAT. A later patch will worry about fixing alternates to parse all inputs that a non-alternate 'number' would accept, for now it is still marked FIXME.

See [1] below.

Quoted 6 lines
Quoted 3 lines

This patch fixes the validation bug by deleting the indirection, and modifying get_next_type() to directly return a qtype code.

get_next_type() doesn't return anything. Do you mean "store a qtype code"?

Yes.

Quoted 4 lines
Quoted 2 lines

There is no longer a need to generate an implicit FooKind array

FooKind is an enum, not an array.

...to generate an implicit FooKind enum, nor FooKind_lookup[] array.

Quoted 12 lines
Quoted 12 lines

associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names); that also means we no longer have a collision with an alternate branch named 'max'. Next, the generated visitor is fixed to properly detect unexpected qtypes in the switch statement. This is done via the use of a new QAPISchemaAlternateTypeTag subclass and the use of a new member.c_type() method when producing qapi-types. The new subtype also allows us to clean up a TODO left in the previous commit.

Quoted 12 lines
Quoted 4 lines

Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. If that gets too confusing, we could reintroduce FooKind, but initialize it differently than

Quoted 8 lines

most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A = QTYPE_QDICT,
FOO_KIND_B = QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to use foo->a or foo->b when inspecting foo->type. But without a current client, I didn't see the point of doing it now.

You have a point below that we either need to reserve MAX and require no case-insensitive clashes, or that we will never want to add it. I'm leaning towards never going back, because the new way feels so much nicer.

[...]

Quoted 15 lines
Quoted 11 lines

tests/qapi-schema/qapi-schema-test.json line 134

Quoted 7 lines

+++ b/tests/qapi-schema/qapi-schema-test.json

tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -131,7 +131,7 @@
   'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero',
             'u': 'UserDefZero', 'type': 'UserDefZero' } }
 { 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool',
-                                    'myKind': 'has_a' } }
+                                    'myKind': 'has_a', 'max': 'str' } }

Here, you add the positive test that alternate name 'max' works.

One, not mentioned in the commit message.

D'oh.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines

Two, the commit message says we may reintroduce FooKind if working with qtype_code turns out to be too confusing. If we ever do that, alternate name 'max' breaks, doesn't it? Shouldn't we keep it reserved then, just in case?

See my comment above; at this point, I doubt we'll ever want to go back, so maybe I just need to be more definitive in stating that.

Fair enough.

However, we should also consider QAPI language regularity and simplicity. To make that argument, I need to back up a bit.

There are three kinds of name collisions: QAPI schema, QMP wire, generated C.

QAPI schema collisions are the obvious ones:

* JSON object member names must be distinct (enforced by our JSON parser).

* Enumeration values must be distinct.

* Each flat union's variant's members must be distinct from the non-variant members.

A translation of the schema into a protocol or code can add additional restrictions.

I believe the translation to QMP wire doesn't add any. We don't mangle names, we don't add object members or enumeration values.

The translation to C does add some:

* Mangled names can collide even when unmangled names don't.

* Names can collide with names used by the implementation.

It can also remove collision possibilities (e.g. variant and non-variant members end up in separate name spaces), but that's not important here.

One possible approach would be to let qapi.py worry about QAPI schema / QMP wire collisions, and the C compiler about generated C collisions. We rejected that approach, because navigating from the C compiler's error messages to the broken spot in the QAPI schema is a bother.

Instead, qapi.py knows enough about the generators to predict collisions in generated C. This is feasible only because the generators follow simple and regular rules:

* Names are mangled by c_name(), prefixes or suffixes may be tacked on.

* Exception: enumeration values are mangled by camel_to_upper(). Although the way c_enum_const() is defined, you have to look closely to see it.

* Any pair of names that clashes before mangling also clashes after. Permits omitting collision checks for unmangled names when the mangled names are checked.

* Tag values are mangled *both* ways, because they occur both as C union members and as C enum members.

* We reserve a bunch of names for generator use: mangled names can't start with 'q_', mangled member names can't start with 'has_', mangled enumeration values can't be 'MAX', and so forth.

Remarks:

* A simple union's member names are also tag values, and are therefore mangled both ways, and both reserved member and enum value names apply.

* The moment you use an enumeration as type of a flat union tag, the other mangling and reserved names kicks in.

Conclusions:

* Having two different name manglers is a headache we could do without, especially since the second one camel_to_upper() is pretty magic.

We have it only to get

typedef enum BlockDeviceIoStatus {
BLOCK_DEVICE_IO_STATUS_OK = 0,
BLOCK_DEVICE_IO_STATUS_FAILED = 1,
BLOCK_DEVICE_IO_STATUS_NOSPACE = 2,
BLOCK_DEVICE_IO_STATUS_MAX = 3,
} BlockDeviceIoStatus;

instead of

typedef enum BlockDeviceIoStatus {
BlockDeviceIoStatus_ok = 0,
BlockDeviceIoStatus_failed = 1,
BlockDeviceIoStatus_nospace = 2,
BlockDeviceIoStatus_MAX = 3,
} BlockDeviceIoStatus;

Bah! CODING_STYLE doesn't even ask for shouting enumeration constants. Can't see why we do.

* Keeping the complexity of the rules under control is good both for qapi.py and for the QAPI schema language.

To that end, I think we should consider reserving the same set of names both for members and tag values. It gets rid of complications like enumerations you can't use as flat union tags.

Additionally, the question whether to keep the door open for generating an enum for the alternate cases becomes moot.

What do you think?

[...]

author
#47 lore raw

Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types

Quoted 23 lines · Markus Armbruster

Conclusions:

* Having two different name manglers is a headache we could do without, especially since the second one camel_to_upper() is pretty magic.

We have it only to get

typedef enum BlockDeviceIoStatus {
BLOCK_DEVICE_IO_STATUS_OK = 0,
BLOCK_DEVICE_IO_STATUS_FAILED = 1,
BLOCK_DEVICE_IO_STATUS_NOSPACE = 2,
BLOCK_DEVICE_IO_STATUS_MAX = 3,
} BlockDeviceIoStatus;

instead of

typedef enum BlockDeviceIoStatus {
BlockDeviceIoStatus_ok = 0,
BlockDeviceIoStatus_failed = 1,
BlockDeviceIoStatus_nospace = 2,
BlockDeviceIoStatus_MAX = 3,
} BlockDeviceIoStatus;

Quoted 2 lines

Bah! CODING_STYLE doesn't even ask for shouting enumeration constants. Can't see why we do.

Interesting idea. In fact, if we went one step further:

BlockDeviceIoStatus_ok = 0,
...
BlockDeviceIoStatusMAX = 3.

(that is, typename + '_' + value for user values, and typename + 'MAX' for the sentinel), then the max value _cannot_ collide with any of the other values.

Quoted 4 lines

* Keeping the complexity of the rules under control is good both for qapi.py and for the QAPI schema language.

Quoted 8 lines

To that end, I think we should consider reserving the same set of names both for members and tag values. It gets rid of complications like enumerations you can't use as flat union tags.

Additionally, the question whether to keep the door open for generating an enum for the alternate cases becomes moot.

What do you think?

I like the idea. Don't know if it's too late for 2.5, though.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#48 lore raw

Re: [Qemu-devel] [PATCH v8 10/17] qapi: Simplify visiting of alternate types

Quoted 30 lines · Eric Blake
Quoted 23 lines · Markus Armbruster

Conclusions:

* Having two different name manglers is a headache we could do without, especially since the second one camel_to_upper() is pretty magic.

We have it only to get

typedef enum BlockDeviceIoStatus {
BLOCK_DEVICE_IO_STATUS_OK = 0,
BLOCK_DEVICE_IO_STATUS_FAILED = 1,
BLOCK_DEVICE_IO_STATUS_NOSPACE = 2,
BLOCK_DEVICE_IO_STATUS_MAX = 3,
} BlockDeviceIoStatus;

instead of

typedef enum BlockDeviceIoStatus {
BlockDeviceIoStatus_ok = 0,
BlockDeviceIoStatus_failed = 1,
BlockDeviceIoStatus_nospace = 2,
BlockDeviceIoStatus_MAX = 3,
} BlockDeviceIoStatus;

Quoted 2 lines

Bah! CODING_STYLE doesn't even ask for shouting enumeration constants. Can't see why we do.

Interesting idea. In fact, if we went one step further:

Quoted 7 lines

BlockDeviceIoStatus_ok = 0,
...
BlockDeviceIoStatusMAX = 3.

(that is, typename + '_' + value for user values, and typename + 'MAX' for the sentinel), then the max value _cannot_ collide with any of the other values.

Another arbitrary rule killed.

Quoted 7 lines
Quoted 7 lines

* Keeping the complexity of the rules under control is good both for qapi.py and for the QAPI schema language.

To that end, I think we should consider reserving the same set of names both for members and tag values. It gets rid of complications like enumerations you can't use as flat union tags.

Quoted 6 lines
Quoted 4 lines

Additionally, the question whether to keep the door open for generating an enum for the alternate cases becomes moot.

What do you think?

I like the idea. Don't know if it's too late for 2.5, though.

The only risk with renaming obviously unique identifiers is producing merge conflicts. Annoying, but easy enough to resolve.

How much churn exactly? I count 1423 occurences of generated enumeration constants in 143 out of 4442 source files. Top-scorer is of course QKeyCode, with 471 occurences in 9 files, almost all in tables. I've seen worse. I wouldn't ask for such a pull during hard freeze, of course. During soft freeze, merge conflict risk might be lower than usual.

author
#49 lore raw

PATCH v8 11/17 qapi: Fix alternates that accept 'number' but not 'int'

In reply to Eric Blake

The QMP input visitor allows integral values to be assigned by promotion to a QTYPE_QFLOAT. However, when parsing an alternate, we did not take this into account, such that an alternate that accepts 'number' but not 'int' would reject integral values.

With this patch, we now have the following desirable table:

    alternate has      case selected for
    'int'  'number'    QTYPE_QINT  QTYPE_QFLOAT
      no        no     error       error
      no       yes     'number'    'number'
     yes        no     'int'       error
     yes       yes     'int'       'number'

While it is unlikely that we will ever use 'number' in an alternate other than in the testsuite, it never hurts to be more precise in what we allow.

---
v8: no change
v7: rebase to named .u union
v6: rebase onto earlier testsuite and gen_err_check() improvements

---
 include/qapi/visitor-impl.h    |  2 +-
 include/qapi/visitor.h         |  3 ++-
 qapi/qapi-visit-core.c         |  4 ++--
 qapi/qmp-input-visitor.c       |  4 ++++
 scripts/qapi-visit.py          |  9 +++++++--
 tests/test-qmp-input-visitor.c | 20 ++++++--------------
 6 files changed, 22 insertions(+), 20 deletions(-)
include/qapi/visitor-impl.h +1 -1
index 6d95b36..1d09b7b 100644
@@ -33,7 +33,7 @@ struct Visitor
     void (*type_enum)(Visitor *v, int *obj, const char * const strings[],
                       const char *kind, const char *name, Error **errp);
     /* May be NULL; most useful for input visitors. */
-    void (*get_next_type)(Visitor *v, qtype_code *type,
+    void (*get_next_type)(Visitor *v, qtype_code *type, bool promote_int,
                           const char *name, Error **errp);
 
     void (*type_int)(Visitor *v, int64_t *obj, const char *name, Error **errp);
include/qapi/visitor.h +2 -1
index b765993..baea594 100644
@@ -46,8 +46,9 @@ void visit_optional(Visitor *v, bool *present, const char *name,
  * Determine the qtype of the item @name in the current object visit.
  * For input visitors, set *@type to the correct qtype of a qapi
  * alternate type; for other visitors, leave *@type unchanged.
+ * If @promote_int, treat integers as QTYPE_FLOAT.
  */
-void visit_get_next_type(Visitor *v, qtype_code *type,
+void visit_get_next_type(Visitor *v, qtype_code *type, bool promote_int,
                          const char *name, Error **errp);
 void visit_type_enum(Visitor *v, int *obj, const char * const strings[],
                      const char *kind, const char *name, Error **errp);
qapi/qapi-visit-core.c +2 -2
index 3f24daa..884fe94 100644
@@ -81,11 +81,11 @@ void visit_optional(Visitor *v, bool *present, const char *name,
     }
 }
 
-void visit_get_next_type(Visitor *v, qtype_code *type,
+void visit_get_next_type(Visitor *v, qtype_code *type, bool promote_int,
                          const char *name, Error **errp)
 {
     if (v->get_next_type) {
-        v->get_next_type(v, type, name, errp);
+        v->get_next_type(v, type, promote_int, name, errp);
     }
 }
 
qapi/qmp-input-visitor.c +4 -0
index 803ffad..5310db5 100644
@@ -209,6 +209,7 @@ static void qmp_input_end_list(Visitor *v, Error **errp)
 }
 
 static void qmp_input_get_next_type(Visitor *v, qtype_code *type,
+                                    bool promote_int,
                                     const char *name, Error **errp)
 {
     QmpInputVisitor *qiv = to_qiv(v);
@@ -219,6 +220,9 @@ static void qmp_input_get_next_type(Visitor *v, qtype_code *type,
         return;
     }
     *type = qobject_type(qobj);
+    if (promote_int && *type == QTYPE_QINT) {
+        *type = QTYPE_QFLOAT;
+    }
 }
 
 static void qmp_input_type_int(Visitor *v, int64_t *obj, const char *name,
scripts/qapi-visit.py +7 -2
index 2091a0f..948016d 100644
@@ -179,6 +179,11 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s *obj, const char *name, Error
 
 
 def gen_visit_alternate(name, variants):
+    promote_int = 'true'
+    for var in variants.variants:
+        if var.type.alternate_qtype() == 'QTYPE_QINT':
+            promote_int = 'false'
+
     ret = mcgen('''
 
 void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error **errp)
@@ -189,13 +194,13 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
     if (err) {
         goto out;
     }
-    visit_get_next_type(v, &(*obj)->type, name, &err);
+    visit_get_next_type(v, &(*obj)->type, %(promote_int)s, name, &err);
     if (err) {
         goto out_obj;
     }
     switch ((*obj)->type) {
 ''',
-                c_name=c_name(name))
+                c_name=c_name(name), promote_int=promote_int)
 
     for var in variants.variants:
         ret += mcgen('''
tests/test-qmp-input-visitor.c +6 -14
index 552fc74..5a5014b 100644
@@ -386,25 +386,17 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     qapi_free_AltStrBool(asb);
     visitor_input_teardown(data, NULL);
 
-    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
-    visit_type_AltStrNum(v, &asn, NULL, &err);
-    /* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */
-    /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */
-    g_assert(err);
-    error_free(err);
-    err = NULL;
+    visit_type_AltStrNum(v, &asn, NULL, &error_abort);
+    g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT);
+    g_assert_cmpfloat(asn->u.n, ==, 42);
     qapi_free_AltStrNum(asn);
     visitor_input_teardown(data, NULL);
 
-    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
-    visit_type_AltNumStr(v, &ans, NULL, &err);
-    /* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */
-    /* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */
-    g_assert(err);
-    error_free(err);
-    err = NULL;
+    visit_type_AltNumStr(v, &ans, NULL, &error_abort);
+    g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT);
+    g_assert_cmpfloat(ans->u.n, ==, 42);
     qapi_free_AltNumStr(ans);
     visitor_input_teardown(data, NULL);
 
Signature · 1 line
2.4.3
author
#50 lore raw

PATCH v8 12/17 qapi: Remove dead visitor code

In reply to Eric Blake

Commit cbc95538 removed unused start_handle() and end_handle(), but forgot got remove their declarations.

Commit 4e27e819 introduced optional visitor callbacks for all sorts of int types, but except for type_uint64 and type_size, none of them have ever been supplied (the generic implementation based on using type_int then bounds-checking works just fine). In the interest of simplicity, it's easier to make the visitor callback interface not have to worry about the other sizes.

---
v8: no change
v7: no change
v6: no change

---
 include/qapi/visitor-impl.h |  19 +++----
 include/qapi/visitor.h      |   3 -
 qapi/qapi-visit-core.c      | 131 +++++++++++++++++---------------------------
 3 files changed, 58 insertions(+), 95 deletions(-)
include/qapi/visitor-impl.h +8 -11
index 1d09b7b..370935a 100644
@@ -1,7 +1,7 @@
 /*
  * Core Definitions for QAPI Visitor implementations
  *
- * Copyright (C) 2012 Red Hat, Inc.
+ * Copyright (C) 2012, 2015 Red Hat, Inc.
  *
  * Author: Paolo Bonizni <[email protected]>
  *
@@ -48,18 +48,15 @@ struct Visitor
     void (*optional)(Visitor *v, bool *present, const char *name,
                      Error **errp);
 
-    void (*type_uint8)(Visitor *v, uint8_t *obj, const char *name, Error **errp);
-    void (*type_uint16)(Visitor *v, uint16_t *obj, const char *name, Error **errp);
-    void (*type_uint32)(Visitor *v, uint32_t *obj, const char *name, Error **errp);
-    void (*type_uint64)(Visitor *v, uint64_t *obj, const char *name, Error **errp);
-    void (*type_int8)(Visitor *v, int8_t *obj, const char *name, Error **errp);
-    void (*type_int16)(Visitor *v, int16_t *obj, const char *name, Error **errp);
-    void (*type_int32)(Visitor *v, int32_t *obj, const char *name, Error **errp);
-    void (*type_int64)(Visitor *v, int64_t *obj, const char *name, Error **errp);
-    /* visit_type_size() falls back to (*type_uint64)() if type_size is unset */
-    void (*type_size)(Visitor *v, uint64_t *obj, const char *name, Error **errp);
     bool (*start_union)(Visitor *v, bool data_present, Error **errp);
     void (*end_union)(Visitor *v, bool data_present, Error **errp);
+
+    /* Only required to visit uint64 differently than (*type_int)().  */
+    void (*type_uint64)(Visitor *v, uint64_t *obj, const char *name,
+                        Error **errp);
+    /* Only required to visit sizes differently than (*type_uint64)().  */
+    void (*type_size)(Visitor *v, uint64_t *obj, const char *name,
+                      Error **errp);
 };
 
 void input_type_enum(Visitor *v, int *obj, const char * const strings[],
include/qapi/visitor.h +0 -3
index baea594..67ddd83 100644
@@ -27,9 +27,6 @@ typedef struct GenericList
     struct GenericList *next;
 } GenericList;
 
-void visit_start_handle(Visitor *v, void **obj, const char *kind,
-                        const char *name, Error **errp);
-void visit_end_handle(Visitor *v, Error **errp);
 void visit_start_struct(Visitor *v, void **obj, const char *kind,
                         const char *name, size_t size, Error **errp);
 void visit_end_struct(Visitor *v, Error **errp);
qapi/qapi-visit-core.c +50 -81

184 lines not on this page · raw message

Signature · 1 line
2.4.3
author
#51 lore raw

PATCH v8 13/17 qapi: Plug leaks in test-qmp-*

In reply to Eric Blake

Make valgrind happy with the current state of the tests, so that it is easier to see if future patches introduce new memory problems without being drowned in noise. Many of the leaks were due to calling a second init without tearing down the data from an earlier visit. But since teardown is already idempotent, and we already register teardown as part of input_visitor_test_add(), it is nicer to just make init() safe to call multiple times than it is to have to make all tests call teardown.

Another common leak was forgetting to clean up an error object, after testing that an error was raised.

Another leak was in test_visitor_in_struct_nested(), failing to clean the base member of UserDefTwo. Cleaning that up left check_and_free_str() as dead code (since using the qapi_free_* takes care of recursion, and we don't want double frees).

---
v8: no change
v7: no change
v6: make init repeatable rather than adding teardown everywhere, fix additional leak with UserDefTwo base, plug additional files

---
 tests/test-qmp-input-strict.c   | 10 ++++++++++
 tests/test-qmp-input-visitor.c  | 41 +++++++----------------------------------
 tests/test-qmp-output-visitor.c |  4 +++-
 3 files changed, 20 insertions(+), 35 deletions(-)
tests/test-qmp-input-strict.c +10 -0
index b44184f..910e2f9 100644
@@ -77,6 +77,8 @@ static Visitor *validate_test_init_raw(TestInputVisitorData *data,
 {
     Visitor *v;
 
+    validate_teardown(data, NULL);
+
     data->obj = qobject_from_json(json_string);
     g_assert(data->obj != NULL);
 
@@ -193,6 +195,8 @@ static void test_validate_fail_struct(TestInputVisitorData *data,
 
     visit_type_TestStruct(v, &p, NULL, &err);
     g_assert(err);
+    error_free(err);
+    /* FIXME: visitor should not allocate p when returning error */
     if (p) {
         g_free(p->string);
     }
@@ -210,6 +214,7 @@ static void test_validate_fail_struct_nested(TestInputVisitorData *data,
 
     visit_type_UserDefTwo(v, &udp, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefTwo(udp);
 }
 
@@ -224,6 +229,7 @@ static void test_validate_fail_list(TestInputVisitorData *data,
 
     visit_type_UserDefOneList(v, &head, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefOneList(head);
 }
 
@@ -239,6 +245,7 @@ static void test_validate_fail_union_native_list(TestInputVisitorData *data,
 
     visit_type_UserDefNativeListUnion(v, &tmp, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefNativeListUnion(tmp);
 }
 
@@ -253,6 +260,7 @@ static void test_validate_fail_union_flat(TestInputVisitorData *data,
 
     visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefFlatUnion(tmp);
 }
 
@@ -268,6 +276,7 @@ static void test_validate_fail_union_flat_no_discrim(TestInputVisitorData *data,
 
     visit_type_UserDefFlatUnion2(v, &tmp, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefFlatUnion2(tmp);
 }
 
@@ -282,6 +291,7 @@ static void test_validate_fail_alternate(TestInputVisitorData *data,
 
     visit_type_UserDefAlternate(v, &tmp, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefAlternate(tmp);
 }
 
tests/test-qmp-input-visitor.c +7 -34

159 lines not on this page · raw message

tests/test-qmp-output-visitor.c +3 -1
index 695db32..42458d7 100644
@@ -391,6 +391,7 @@ static void test_visitor_out_any(TestOutputVisitorData *data,
     qobj = QOBJECT(qdict);
     visit_type_any(data->ov, &qobj, NULL, &err);
     g_assert(!err);
+    qobject_decref(qobj);
     obj = qmp_output_get_qobject(data->qov);
     g_assert(obj != NULL);
     qdict = qobject_to_qdict(obj);
@@ -411,7 +412,6 @@ static void test_visitor_out_any(TestOutputVisitorData *data,
     g_assert(qstring);
     g_assert_cmpstr(qstring_get_str(qstring), ==, "foo");
     qobject_decref(obj);
-    qobject_decref(qobj);
 }
 
 static void test_visitor_out_union_flat(TestOutputVisitorData *data,
@@ -462,6 +462,7 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,
     g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42);
 
     qapi_free_UserDefAlternate(tmp);
+    qobject_decref(arg);
 
     tmp = g_malloc0(sizeof(UserDefAlternate));
     tmp->type = QTYPE_QSTRING;
@@ -474,6 +475,7 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,
     g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello");
 
     qapi_free_UserDefAlternate(tmp);
+    qobject_decref(arg);
 }
 
 static void test_visitor_out_empty(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
author
#52 lore raw

PATCH v8 14/17 qapi: Simplify error testing in test-qmp-*

In reply to Eric Blake

By using &error_abort, we can avoid a local err variable in situations where we expect success.

By moving err into data, we can let test teardown take care of cleaning up any collected error (and allowing for fewer lines of code between repeated tests where init runs teardown on our behalf).

---
v8: no change
v7: pick up whitespace changes dropped from earlier commit
v6: new patch

---
 tests/test-qmp-input-strict.c      |  77 +++++++++-------------------
 tests/test-qmp-input-visitor.c     | 101 ++++++++++++-------------------------
 tests/test-qmp-output-visitor.c    |  52 +++++--------------
 tests/test-visitor-serialization.c |  42 +++++++--------
 4 files changed, 86 insertions(+), 186 deletions(-)
tests/test-qmp-input-strict.c +25 -52
index 910e2f9..f8da75c 100644
@@ -26,6 +26,7 @@
 typedef struct TestInputVisitorData {
     QObject *obj;
     QmpInputVisitor *qiv;
+    Error *err;
 } TestInputVisitorData;
 
 static void validate_teardown(TestInputVisitorData *data,
@@ -34,6 +35,9 @@ static void validate_teardown(TestInputVisitorData *data,
     qobject_decref(data->obj);
     data->obj = NULL;
 
+    error_free(data->err);
+    data->err = NULL;
+
     if (data->qiv) {
         qmp_input_visitor_cleanup(data->qiv);
         data->qiv = NULL;
@@ -96,13 +100,11 @@ static void test_validate_struct(TestInputVisitorData *data,
                                   const void *unused)
 {
     TestStruct *p = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init(data, "{ 'integer': -42, 'boolean': true, 'string': 'foo' }");
 
-    visit_type_TestStruct(v, &p, NULL, &err);
-    g_assert(!err);
+    visit_type_TestStruct(v, &p, NULL, &error_abort);
     g_free(p->string);
     g_free(p);
 }
@@ -111,7 +113,6 @@ static void test_validate_struct_nested(TestInputVisitorData *data,
                                          const void *unused)
 {
     UserDefTwo *udp = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init(data, "{ 'string0': 'string0', "
@@ -119,8 +120,7 @@ static void test_validate_struct_nested(TestInputVisitorData *data,
                            "'dict2': { 'userdef': { 'integer': 42, "
                            "'string': 'string' }, 'string': 'string2'}}}");
 
-    visit_type_UserDefTwo(v, &udp, NULL, &err);
-    g_assert(!err);
+    visit_type_UserDefTwo(v, &udp, NULL, &error_abort);
     qapi_free_UserDefTwo(udp);
 }
 
@@ -128,13 +128,11 @@ static void test_validate_list(TestInputVisitorData *data,
                                 const void *unused)
 {
     UserDefOneList *head = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init(data, "[ { 'string': 'string0', 'integer': 42 }, { 'string': 'string1', 'integer': 43 }, { 'string': 'string2', 'integer': 44 } ]");
 
-    visit_type_UserDefOneList(v, &head, NULL, &err);
-    g_assert(!err);
+    visit_type_UserDefOneList(v, &head, NULL, &error_abort);
     qapi_free_UserDefOneList(head);
 }
 
@@ -143,12 +141,10 @@ static void test_validate_union_native_list(TestInputVisitorData *data,
 {
     UserDefNativeListUnion *tmp = NULL;
     Visitor *v;
-    Error *err = NULL;
 
     v = validate_test_init(data, "{ 'type': 'integer', 'data' : [ 1, 2 ] }");
 
-    visit_type_UserDefNativeListUnion(v, &tmp, NULL, &err);
-    g_assert(!err);
+    visit_type_UserDefNativeListUnion(v, &tmp, NULL, &error_abort);
     qapi_free_UserDefNativeListUnion(tmp);
 }
 
@@ -157,7 +153,6 @@ static void test_validate_union_flat(TestInputVisitorData *data,
 {
     UserDefFlatUnion *tmp = NULL;
     Visitor *v;
-    Error *err = NULL;
 
     v = validate_test_init(data,
                            "{ 'enum1': 'value1', "
@@ -165,8 +160,7 @@ static void test_validate_union_flat(TestInputVisitorData *data,
                            "'string': 'str', "
                            "'boolean': true }");
 
-    visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
-    g_assert(!err);
+    visit_type_UserDefFlatUnion(v, &tmp, NULL, &error_abort);
     qapi_free_UserDefFlatUnion(tmp);
 }
 
@@ -175,12 +169,10 @@ static void test_validate_alternate(TestInputVisitorData *data,
 {
     UserDefAlternate *tmp = NULL;
     Visitor *v;
-    Error *err = NULL;
 
     v = validate_test_init(data, "42");
 
-    visit_type_UserDefAlternate(v, &tmp, NULL, &err);
-    g_assert(!err);
+    visit_type_UserDefAlternate(v, &tmp, NULL, &error_abort);
     qapi_free_UserDefAlternate(tmp);
 }
 
@@ -188,14 +180,12 @@ static void test_validate_fail_struct(TestInputVisitorData *data,
                                        const void *unused)
 {
     TestStruct *p = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init(data, "{ 'integer': -42, 'boolean': true, 'string': 'foo', 'extra': 42 }");
 
-    visit_type_TestStruct(v, &p, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    visit_type_TestStruct(v, &p, NULL, &data->err);
+    g_assert(data->err);
     /* FIXME: visitor should not allocate p when returning error */
     if (p) {
         g_free(p->string);
@@ -207,14 +197,12 @@ static void test_validate_fail_struct_nested(TestInputVisitorData *data,
                                               const void *unused)
 {
     UserDefTwo *udp = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init(data, "{ 'string0': 'string0', 'dict1': { 'string1': 'string1', 'dict2': { 'userdef1': { 'integer': 42, 'string': 'string', 'extra': [42, 23, {'foo':'bar'}] }, 'string2': 'string2'}}}");
 
-    visit_type_UserDefTwo(v, &udp, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    visit_type_UserDefTwo(v, &udp, NULL, &data->err);
+    g_assert(data->err);
     qapi_free_UserDefTwo(udp);
 }
 
@@ -222,14 +210,12 @@ static void test_validate_fail_list(TestInputVisitorData *data,
                                      const void *unused)
 {
     UserDefOneList *head = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init(data, "[ { 'string': 'string0', 'integer': 42 }, { 'string': 'string1', 'integer': 43 }, { 'string': 'string2', 'integer': 44, 'extra': 'ggg' } ]");
 
-    visit_type_UserDefOneList(v, &head, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    visit_type_UserDefOneList(v, &head, NULL, &data->err);
+    g_assert(data->err);
     qapi_free_UserDefOneList(head);
 }
 
@@ -237,15 +223,13 @@ static void test_validate_fail_union_native_list(TestInputVisitorData *data,
                                                  const void *unused)
 {
     UserDefNativeListUnion *tmp = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init(data,
                            "{ 'type': 'integer', 'data' : [ 'string' ] }");
 
-    visit_type_UserDefNativeListUnion(v, &tmp, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    visit_type_UserDefNativeListUnion(v, &tmp, NULL, &data->err);
+    g_assert(data->err);
     qapi_free_UserDefNativeListUnion(tmp);
 }
 
@@ -253,14 +237,12 @@ static void test_validate_fail_union_flat(TestInputVisitorData *data,
                                           const void *unused)
 {
     UserDefFlatUnion *tmp = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init(data, "{ 'string': 'c', 'integer': 41, 'boolean': true }");
 
-    visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    visit_type_UserDefFlatUnion(v, &tmp, NULL, &data->err);
+    g_assert(data->err);
     qapi_free_UserDefFlatUnion(tmp);
 }
 
@@ -268,15 +250,13 @@ static void test_validate_fail_union_flat_no_discrim(TestInputVisitorData *data,
                                                      const void *unused)
 {
     UserDefFlatUnion2 *tmp = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     /* test situation where discriminator field ('enum1' here) is missing */
     v = validate_test_init(data, "{ 'integer': 42, 'string': 'c', 'string1': 'd', 'string2': 'e' }");
 
-    visit_type_UserDefFlatUnion2(v, &tmp, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    visit_type_UserDefFlatUnion2(v, &tmp, NULL, &data->err);
+    g_assert(data->err);
     qapi_free_UserDefFlatUnion2(tmp);
 }
 
@@ -285,13 +265,11 @@ static void test_validate_fail_alternate(TestInputVisitorData *data,
 {
     UserDefAlternate *tmp = NULL;
     Visitor *v;
-    Error *err = NULL;
 
     v = validate_test_init(data, "3.14");
 
-    visit_type_UserDefAlternate(v, &tmp, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    visit_type_UserDefAlternate(v, &tmp, NULL, &data->err);
+    g_assert(data->err);
     qapi_free_UserDefAlternate(tmp);
 }
 
@@ -299,16 +277,11 @@ static void do_test_validate_qmp_introspect(TestInputVisitorData *data,
                                             const char *schema_json)
 {
     SchemaInfoList *schema = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init_raw(data, schema_json);
 
-    visit_type_SchemaInfoList(v, &schema, NULL, &err);
-    if (err) {
-        fprintf(stderr, "%s", error_get_pretty(err));
-    }
-    g_assert(!err);
+    visit_type_SchemaInfoList(v, &schema, NULL, &error_abort);
     g_assert(schema);
 
     qapi_free_SchemaInfoList(schema);
tests/test-qmp-input-visitor.c +31 -70

338 lines not on this page · raw message

tests/test-qmp-output-visitor.c +13 -39

178 lines not on this page · raw message

tests/test-visitor-serialization.c +17 -25

103 lines not on this page · raw message

Signature · 1 line
2.4.3
author
#53 lore raw

PATCH v8 15/17 qapi: Test failure in middle of array parse

In reply to Eric Blake

Our generated list visitors have the same problem as has been mentioned elsewhere (see commit 2f52e20): they allocate data even on failure. An upcoming patch will correct things to provide saner guarantees, but first we need to expose the behavior in the testsuite to ensure we aren't introducing any memory usage bugs.

---
v8: no change
v7: no change
v6: rebase onto earlier gen_err_check() and testsuite improvements

---
 scripts/qapi-visit.py          |  4 ++++
 tests/test-qmp-input-visitor.c | 10 ++++++++++
 2 files changed, 14 insertions(+)
scripts/qapi-visit.py +4 -0
index 948016d..4152f40 100644
@@ -138,6 +138,10 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
 
 
 def gen_visit_list(name, element_type):
+    # FIXME: if *obj is NULL on entry, and the first visit_next_list()
+    # assigns to *obj, while a later one fails, we should clean up *obj
+    # rather than leaving it non-NULL. As currently written, the caller must
+    # call qapi_free_FOOList() to avoid a memory leak of the partial FOOList.
     return mcgen('''
 
 void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error **errp)
tests/test-qmp-input-visitor.c +10 -0
index 4553210..df0bceb 100644
@@ -698,6 +698,7 @@ static void test_visitor_in_errors(TestInputVisitorData *data,
 {
     TestStruct *p = NULL;
     Visitor *v;
+    strList *q = NULL;
 
     v = visitor_input_test_init(data, "{ 'integer': false, 'boolean': 'foo', 'string': -42 }");
 
@@ -709,6 +710,15 @@ static void test_visitor_in_errors(TestInputVisitorData *data,
 
     g_free(p->string);
     g_free(p);
+
+    v = visitor_input_test_init(data, "[ '1', '2', false, '3' ]");
+    /* FIXME - a failed parse should not leave a partially-allocated
+     * array for us to clean up; this could cause callers to leak
+     * memory. */
+    visit_type_strList(v, &q, NULL, &data->err);
+    assert(q);
+    assert(data->err);
+    qapi_free_strList(q);
 }
 
 int main(int argc, char **argv)
Signature · 1 line
2.4.3
author
#54 lore raw

PATCH v8 16/17 qapi: More tests of input arrays

In reply to Eric Blake

Our testsuite had no coverage of empty arrays, nor of what happens when the input does not match the expected type. Useful to have, especially if we start changing the visitor contracts.

---
v8: no change
v7: no change
v6: new patch

---
 tests/test-qmp-input-visitor.c | 51 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)
tests/test-qmp-input-visitor.c +51 -0
index df0bceb..b93f698 100644
@@ -236,6 +236,12 @@ static void test_visitor_in_list(TestInputVisitorData *data,
     }
 
     qapi_free_UserDefOneList(head);
+    head = NULL;
+
+    /* An empty list is valid */
+    v = visitor_input_test_init(data, "[]");
+    visit_type_UserDefOneList(v, &head, NULL, &error_abort);
+    g_assert(!head);
 }
 
 static void test_visitor_in_any(TestInputVisitorData *data,
@@ -721,6 +727,49 @@ static void test_visitor_in_errors(TestInputVisitorData *data,
     qapi_free_strList(q);
 }
 
+static void test_visitor_in_wrong_type(TestInputVisitorData *data,
+                                       const void *unused)
+{
+    TestStruct *p = NULL;
+    Visitor *v;
+    strList *q = NULL;
+    int64_t i;
+
+    /* Make sure arrays and structs cannot be confused */
+
+    v = visitor_input_test_init(data, "[]");
+    visit_type_TestStruct(v, &p, NULL, &data->err);
+    g_assert(data->err);
+    g_assert(!p);
+
+    v = visitor_input_test_init(data, "{}");
+    visit_type_strList(v, &q, NULL, &data->err);
+    assert(data->err);
+    assert(!q);
+
+    /* Make sure primitives and struct cannot be confused */
+
+    v = visitor_input_test_init(data, "1");
+    visit_type_TestStruct(v, &p, NULL, &data->err);
+    g_assert(data->err);
+    g_assert(!p);
+
+    v = visitor_input_test_init(data, "{}");
+    visit_type_int(v, &i, NULL, &data->err);
+    assert(data->err);
+
+    /* Make sure primitives and arrays cannot be confused */
+
+    v = visitor_input_test_init(data, "1");
+    visit_type_strList(v, &q, NULL, &data->err);
+    assert(data->err);
+    assert(!q);
+
+    v = visitor_input_test_init(data, "[]");
+    visit_type_int(v, &i, NULL, &data->err);
+    assert(data->err);
+}
+
 int main(int argc, char **argv)
 {
     TestInputVisitorData in_visitor_data;
@@ -753,6 +802,8 @@ int main(int argc, char **argv)
                            &in_visitor_data, test_visitor_in_alternate);
     input_visitor_test_add("/visitor/input/errors",
                            &in_visitor_data, test_visitor_in_errors);
+    input_visitor_test_add("/visitor/input/wrong-type",
+                           &in_visitor_data, test_visitor_in_wrong_type);
     input_visitor_test_add("/visitor/input/alternate-number",
                            &in_visitor_data, test_visitor_in_alternate_number);
     input_visitor_test_add("/visitor/input/native_list/int",
Signature · 1 line
2.4.3
author
#55 lore raw

PATCH v8 17/17 qapi: Simplify visits of optional fields

In reply to Eric Blake

None of the visitor callbacks would set an error when testing if an optional field was present; make this part of the interface contract by eliminating the errp argument. Then, for less code, reflect the determined boolean value back to the caller instead of making the caller read the boolean after the fact.

The resulting generated code has a nice diff:

|-    visit_optional(v, &has_fdset_id, "fdset-id", &err);
|-    if (err) {
|-        goto out;
|-    }
|-    if (has_fdset_id) {
|+    if (visit_optional(v, &has_fdset_id, "fdset-id")) {
|         visit_type_int(v, &fdset_id, "fdset-id", &err);
|         if (err) {
|             goto out;
|         }
|     }

---
v8: no change
v7: rebase to no member.c_name()
v6: rebase onto earlier testsuite and gen_err_check() improvements

---
 include/qapi/visitor-impl.h |  5 ++---
 include/qapi/visitor.h      | 10 ++++++++--
 qapi/opts-visitor.c         |  2 +-
 qapi/qapi-visit-core.c      |  6 +++---
 qapi/qmp-input-visitor.c    |  3 +--
 qapi/string-input-visitor.c |  3 +--
 scripts/qapi.py             |  7 +------
 7 files changed, 17 insertions(+), 19 deletions(-)
include/qapi/visitor-impl.h +2 -3
index 370935a..fd2e905 100644
@@ -44,9 +44,8 @@ struct Visitor
     void (*type_any)(Visitor *v, QObject **obj, const char *name,
                      Error **errp);
 
-    /* May be NULL */
-    void (*optional)(Visitor *v, bool *present, const char *name,
-                     Error **errp);
+    /* May be NULL; most useful for input visitors. */
+    void (*optional)(Visitor *v, bool *present, const char *name);
 
     bool (*start_union)(Visitor *v, bool data_present, Error **errp);
     void (*end_union)(Visitor *v, bool data_present, Error **errp);
include/qapi/visitor.h +8 -2
index 67ddd83..e52ad39 100644
@@ -36,8 +36,14 @@ void visit_end_implicit_struct(Visitor *v, Error **errp);
 void visit_start_list(Visitor *v, const char *name, Error **errp);
 GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp);
 void visit_end_list(Visitor *v, Error **errp);
-void visit_optional(Visitor *v, bool *present, const char *name,
-                    Error **errp);
+
+/**
+ * Check if an optional member @name of an object needs visiting.
+ * For input visitors, set *@present according to whether the
+ * corresponding visit_type_*() needs calling; for other visitors,
+ * leave *@present unchanged.  Return *@present for convenience.
+ */
+bool visit_optional(Visitor *v, bool *present, const char *name);
 
 /**
  * Determine the qtype of the item @name in the current object visit.
qapi/opts-visitor.c +1 -1
index cd10392..ef5fb8b 100644
@@ -488,7 +488,7 @@ opts_type_size(Visitor *v, uint64_t *obj, const char *name, Error **errp)
 
 
 static void
-opts_optional(Visitor *v, bool *present, const char *name, Error **errp)
+opts_optional(Visitor *v, bool *present, const char *name)
 {
     OptsVisitor *ov = DO_UPCAST(OptsVisitor, visitor, v);
 
qapi/qapi-visit-core.c +3 -3
index cbf7780..2594147 100644
@@ -73,12 +73,12 @@ void visit_end_union(Visitor *v, bool data_present, Error **errp)
     }
 }
 
-void visit_optional(Visitor *v, bool *present, const char *name,
-                    Error **errp)
+bool visit_optional(Visitor *v, bool *present, const char *name)
 {
     if (v->optional) {
-        v->optional(v, present, name, errp);
+        v->optional(v, present, name);
     }
+    return *present;
 }
 
 void visit_get_next_type(Visitor *v, qtype_code *type, bool promote_int,
qapi/qmp-input-visitor.c +1 -2
index 5310db5..f714dfc 100644
@@ -300,8 +300,7 @@ static void qmp_input_type_any(Visitor *v, QObject **obj, const char *name,
     *obj = qobj;
 }
 
-static void qmp_input_optional(Visitor *v, bool *present, const char *name,
-                               Error **errp)
+static void qmp_input_optional(Visitor *v, bool *present, const char *name)
 {
     QmpInputVisitor *qiv = to_qiv(v);
     QObject *qobj = qmp_input_get_object(qiv, name, true);
qapi/string-input-visitor.c +1 -2
index bbd6a54..dee780a 100644
@@ -299,8 +299,7 @@ static void parse_type_number(Visitor *v, double *obj, const char *name,
     *obj = val;
 }
 
-static void parse_optional(Visitor *v, bool *present, const char *name,
-                           Error **errp)
+static void parse_optional(Visitor *v, bool *present, const char *name)
 {
     StringInputVisitor *siv = DO_UPCAST(StringInputVisitor, visitor, v);
 
scripts/qapi.py +1 -6
index bd74470..0eb5d9a 100644
@@ -1619,15 +1619,10 @@ def gen_visit_fields(members, prefix='', need_cast=False, skiperr=False):
     for memb in members:
         if memb.optional:
             ret += mcgen('''
-    visit_optional(v, &%(prefix)shas_%(c_name)s, "%(name)s", %(errp)s);
+    if (visit_optional(v, &%(prefix)shas_%(c_name)s, "%(name)s")) {
 ''',
                          prefix=prefix, c_name=c_name(memb.name),
                          name=memb.name, errp=errparg)
-            ret += gen_err_check(skiperr=skiperr)
-            ret += mcgen('''
-    if (%(prefix)shas_%(c_name)s) {
-''',
-                         prefix=prefix, c_name=c_name(memb.name))
             push_indent()
 
         # Ugly: sometimes we need to cast away const
Signature · 1 line
2.4.3