Skip to contentSkip to conversation

qapi member collision (post-introspection cleanups, subset C')

PATCH Superseded Eric Blake 2 participants 85 messages Nov 6, 2015 – Nov 11, 2015

v10 of 14 Superseded Latest is v14 →

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

v10 cover qapi member collision (post-introspection cleanups, subset C')

No pending prerequisites; based on qemu.git master

Also available as a tag at this location:
git fetch git://repo.or.cz/qemu/ericb.git qapi-cleanupv9c

and will soon be part of my branch with the rest of the v5 series, at:
http://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/qapi

v10 notes:
Split several patches, redo the middle patches from Markus to be back in the order they were first posted, some fallout change to my patches due to the nicer pattern of minimizing conditionals inside .check(), by instead calling .check_clash() as needed. Change data->err magic in tests to instead use a new helper error_free_or_abort(). Add a patch that would prevent qapi case-insensitive clashes.

I am redoing my subset boundaries slightly: patches 23-27 of v9 (updating the alternate layout) will be delayed to subset D, and 2 other patches previously posted in subset D are now here (turning qapi clash checking into actual error messages), so the subject line of this cover letter is slightly different.

Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.

Backport diff in relation to v9:

001/30:[----] [--] 'qapi: Use generated TestStruct machinery in tests' 002/30:[----] [--] 'qapi: Strengthen test of TestStructList' 003/30:[down] 'qobject: Protect against use-after-free in qobject_decref()' 004/30:[down] 'qapi: Share test_init code in test-qmp-input*' 005/30:[0001] [FC] 'qapi: Plug leaks in test-qmp-*'
006/30:[down] 'qapi: Simplify non-error testing in test-qmp-*' 007/30:[down] 'qapi: Simplify error cleanup in test-qmp-*' 008/30:[----] [--] 'qapi: More tests of alternate output' 009/30:[0010] [FC] 'qapi: Test failure in middle of array parse' 010/30:[0025] [FC] 'qapi: More tests of input arrays'
011/30:[----] [--] 'qapi: Provide nicer array names in introspection' 012/30:[0014] [FC] 'qapi-introspect: Document lack of sorting' 013/30:[----] [--] 'qapi: Track simple union tag in object.local_members' 014/30:[----] [--] 'qapi-types: Consolidate gen_struct() and gen_union()' 015/30:[----] [--] 'qapi-types: Simplify gen_struct_field[s]' 016/30:[0005] [FC] 'qapi: Drop obsolete tag value collision assertions' 017/30:[0002] [FC] 'qapi: Simplify QAPISchemaObjectTypeMember.check()' 018/30:[down] 'qapi: Clean up after previous commit'
019/30:[----] [-C] 'qapi: Fix up commit 7618b91's clash sanity checking change' 020/30:[0002] [FC] 'qapi: Eliminate QAPISchemaObjectType.check() variable members' 021/30:[----] [--] 'qapi: Factor out QAPISchemaObjectTypeMember.check_clash()' 022/30:[----] [-C] 'qapi: Simplify QAPISchemaObjectTypeVariants.check()' 023/30:[0023] [FC] 'qapi: Check for qapi collisions of flat union branches' 024/30:[0013] [FC] 'qapi: Factor out QAPISchemaObjectType.check_clash()' 025/30:[down] 'qapi: Hoist tag collision check to Variants.check()' 026/30:[----] [--] 'qapi: Remove outdated tests related to QMP/branch collisions' 027/30:[down] 'qapi: Track owner of each object member'
028/30:[down] 'qapi: Detect collisions in C member names' 029/30:[down] 'cpu: Convert CpuInfo into flat union'
030/30:[down] 'qapi: Forbid case-insensitive clashes'

v9 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg00652.html
More patches added, and several reorganized. Lots of new patches from Markus, although not in the order originally proposed.

The first 8 patches are fairly straightforward, and could probably be taken as-is. Patch 9 is a rewrite of v8 4/17, but in the opposite direction (document that no sorting is done, rather than attempting to sort), so it may need further fine-tuning. Patches 12-21 represents a fusion of Markus' and my attempts to rewrite v5 7/17 into a more-reviewable set of patches, and caused further churn later in the series.

Patch 23 still uses tag_member.type == None; I ran out of time to work on Markus' idea of providing an instance of QAPISchemaBuiltinType to fill the role for 'qtype_code' without being exposed through .json files and without breaking the invariant of a valid member.type after check(), and wanted to get the rest of the series started under revew. So I may need a followup patch or even a v10 of the later half of this series after exploring that idea more.

v8 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg06674.html
Minor changes when rebasing to latest, improved commit messages in a few places, plus the addition of a few new patches. Markus started reviewing v7, but hadn't gotten very far.

v7 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg04112.html
Patches 1-3 of the previous round have moved to subset B; patch 7 is gone, and a couple of new patches are present in this round. The latest version of subset B made it much easier to reason about collision detection (namely, tag values can't collide with QMP values thanks to a named rather than anonymous union in the C type), and I like how things turned out. I suspect the hardest part of the review will be patches 5/14 and 7/14, although none of this has really had much review in any earlier versions.

v6 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg01980.html
Add some patches and rebase onto work on subset B. Rearrange some patches from v5 (this set includes 17-20, 23, 25-27). Backport diff gets a bit confused by one patch title changing.

Not much direct review comments, although some of the changes here are updated based on comments made on other patches in the v5 series.

Subset D (and more?) will come later.

In v5:
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg05410.html
I _did_ rearrange patches to try and group related features:

1-2: Groundwork cleanups
3-5: Add more test cases
6-16: Front-end cleanups
17-18: Introspection output cleanups
19-20: 'alternate' type cleanups
21-29: qapi visitor cleanups
30-45: qapi-ify netdev_add
46: add qapi shorthand for flat unions

Lots of fixes based on additional testing, and rebased to track other changes that happened in the meantime. The series is huge; I can split off smaller portions as requested.

In v4:
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg02580.html
add some more clean up patches
rebase to Markus' recent work
pull in part of Zoltán's work to make netdev_add a flat union, further enhancing it to be introspectible

I might be able to rearrange some of these patches, or separate it into smaller independent series, if requested; but I'm posting now to get review started.

In v3:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg02059.html
redo cleanup of dealloc of partial struct
add patches to make all visit_type_*() avoid leaks on failure add patches to allow boxed command arguments and events

In v2:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00900.html
rebase to Markus' v3 series
rework how comments are emitted for fields inherited from base additional patches added for deleting colliding 'void *data' documentation updates to match code changes

v1 was here:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05266.html
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05325.html

Eric Blake (23):
qapi: Use generated TestStruct machinery in tests
qapi: Strengthen test of TestStructList
qobject: Protect against use-after-free in qobject_decref()
qapi: Share test_init code in test-qmp-input*
qapi: Plug leaks in test-qmp-*
qapi: Simplify non-error testing in test-qmp-*
qapi: Simplify error cleanup in test-qmp-*
qapi: More tests of alternate output
qapi: Test failure in middle of array parse
qapi: More tests of input arrays
qapi: Provide nicer array names in introspection
qapi-introspect: Document lack of sorting
qapi: Track simple union tag in object.local_members
qapi-types: Consolidate gen_struct() and gen_union()
qapi-types: Simplify gen_struct_field[s]
qapi: Check for qapi collisions of flat union branches
qapi: Factor out QAPISchemaObjectType.check_clash()
qapi: Hoist tag collision check to Variants.check()
qapi: Remove outdated tests related to QMP/branch collisions
qapi: Track owner of each object member
qapi: Detect collisions in C member names
cpu: Convert CpuInfo into flat union
qapi: Forbid case-insensitive clashes

Markus Armbruster (7):
qapi: Drop obsolete tag value collision assertions
qapi: Simplify QAPISchemaObjectTypeMember.check()
qapi: Clean up after previous commit
qapi: Fix up commit 7618b91's clash sanity checking change
qapi: Eliminate QAPISchemaObjectType.check() variable members
qapi: Factor out QAPISchemaObjectTypeMember.check_clash()
qapi: Simplify QAPISchemaObjectTypeVariants.check()

 cpus.c                                             |  31 ++-
 docs/qapi-code-gen.txt                             |  31 ++-
 hmp.c                                              |  30 ++-
 include/qapi/qmp/qobject.h                         |   1 +
 qapi-schema.json                                   | 120 +++++++--
 qapi/introspect.json                               |  17 +-
 scripts/qapi-introspect.py                         |   8 +-
 scripts/qapi-types.py                              |  68 ++----
 scripts/qapi-visit.py                              |   9 +-
 scripts/qapi.py                                    | 135 ++++++----
 tests/Makefile                                     |   4 +-
 tests/qapi-schema/args-case-clash.err              |   1 +
 ...{union-clash-type.exit => args-case-clash.exit} |   0
 tests/qapi-schema/args-case-clash.json             |   5 +
 .../{union-clash-type.out => args-case-clash.out}  |   0
 tests/qapi-schema/args-name-clash.err              |   1 +
 tests/qapi-schema/args-name-clash.exit             |   2 +-
 tests/qapi-schema/args-name-clash.json             |   6 +-
 tests/qapi-schema/args-name-clash.out              |   6 -
 tests/qapi-schema/flat-union-clash-branch.err      |   0
 tests/qapi-schema/flat-union-clash-branch.exit     |   1 -
 tests/qapi-schema/flat-union-clash-branch.json     |  18 --
 tests/qapi-schema/flat-union-clash-branch.out      |  14 --
 tests/qapi-schema/flat-union-clash-type.err        |   1 -
 tests/qapi-schema/flat-union-clash-type.exit       |   1 -
 tests/qapi-schema/flat-union-clash-type.json       |  14 --
 tests/qapi-schema/flat-union-clash-type.out        |   0
 tests/qapi-schema/qapi-schema-test.json            |   6 +-
 tests/qapi-schema/qapi-schema-test.out             |  15 +-
 tests/qapi-schema/union-clash-data.out             |   1 +
 tests/qapi-schema/union-clash-type.err             |   1 -
 tests/qapi-schema/union-clash-type.json            |   9 -
 tests/qapi-schema/union-empty.out                  |   1 +
 tests/test-qmp-commands.c                          |   8 +-
 tests/test-qmp-common.h                            |  22 ++
 tests/test-qmp-event.c                             |   1 +
 tests/test-qmp-input-strict.c                      | 131 +++-------
 tests/test-qmp-input-visitor.c                     | 272 +++++++++------------
 tests/test-qmp-output-visitor.c                    | 151 +++---------
 tests/test-visitor-serialization.c                 |  76 ++----
 40 files changed, 570 insertions(+), 648 deletions(-)
 create mode 100644 tests/qapi-schema/args-case-clash.err
 rename tests/qapi-schema/{union-clash-type.exit => args-case-clash.exit} (100%)
 create mode 100644 tests/qapi-schema/args-case-clash.json
 rename tests/qapi-schema/{union-clash-type.out => args-case-clash.out} (100%)
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.exit
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.json
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.out
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.exit
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.json
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.out
 delete mode 100644 tests/qapi-schema/union-clash-type.err
 delete mode 100644 tests/qapi-schema/union-clash-type.json
 create mode 100644 tests/test-qmp-common.h
Signature · 1 line
2.4.3
author
#1 lore raw

PATCH v10 1/30 qapi: Use generated TestStruct machinery in tests

Commit d88f5fd and friends first introduced the various test-qmp-* tests in 2011, with duplicated hand-rolled TestStruct machinery, to make sure the qapi visitor interface was tested. Later, commit 4f193e3 in 2013 added a .json file for further testing use by the files, but without consolidating any of the existing hand-rolled visitors. And with four copies, subtle differences have crept in, between the tests themselves (mainly whitespace differences, but also a question of whether to use NULL or "TestStruct" when calling visit_start_struct()) and from what the generator produces (the hand-rolled versions did not cater to partially-allocated objects, because they did not have a deallocation usage).

Of course, just because the visitor interface is tested does not mean it is a sane interface; and future patches will be changing some of the visitor contracts. Rather than having to duplicate the cleanup work in each copy of the TestStruct visitor, and keep each hand-rolled copy in sync with what the generator supplies, we might as well just test what the generator should give us in the first place.

---
v10: no change
v9: no change
v8: improve commit message
v7: rebase on top of subset B v9; defer unrelated trailing whitespace cleanups to later in series
v6: new patch

---
 tests/qapi-schema/qapi-schema-test.json |  6 +++-
 tests/qapi-schema/qapi-schema-test.out  |  5 +++
 tests/test-qmp-input-strict.c           | 35 --------------------
 tests/test-qmp-input-visitor.c          | 34 -------------------
 tests/test-qmp-output-visitor.c         | 58 ---------------------------------
 tests/test-visitor-serialization.c      | 34 -------------------
 6 files changed, 10 insertions(+), 162 deletions(-)
tests/qapi-schema/qapi-schema-test.json +5 -1
index 48e104b..44638da 100644
@@ -3,6 +3,9 @@
 # This file is a stress test of supported qapi constructs that must
 # parse and compile correctly.
 
+{ 'struct': 'TestStruct',
+  'data': { 'integer': 'int', 'boolean': 'bool', 'string': 'str' } }
+
 # for testing enums
 { 'struct': 'NestedEnumsOne',
   'data': { 'enum1': 'EnumOne',   # Intentional forward reference
@@ -46,7 +49,8 @@
 
 # dummy struct to force generation of array types not otherwise mentioned
 { 'struct': 'ForceArrays',
-  'data': { 'unused1':['UserDefOne'], 'unused2':['UserDefTwo'] } }
+  'data': { 'unused1':['UserDefOne'], 'unused2':['UserDefTwo'],
+            'unused3':['TestStruct'] } }
 
 # for testing unions
 # Among other things, test that a name collision between branches does
tests/qapi-schema/qapi-schema-test.out +5 -0
index a7e9aab..e20a823 100644
@@ -92,6 +92,7 @@ object EventStructOne
 object ForceArrays
     member unused1: UserDefOneList optional=False
     member unused2: UserDefTwoList optional=False
+    member unused3: TestStructList optional=False
 enum MyEnum []
 object NestedEnumsOne
     member enum1: EnumOne optional=False
@@ -100,6 +101,10 @@ object NestedEnumsOne
     member enum4: EnumOne optional=True
 enum QEnumTwo ['value1', 'value2']
     prefix QENUM_TWO
+object TestStruct
+    member integer: int optional=False
+    member boolean: bool optional=False
+    member string: str optional=False
 object UserDefA
     member boolean: bool optional=False
     member a_b: int optional=True
tests/test-qmp-input-strict.c +0 -35
index 53a7693..b44184f 100644
@@ -89,41 +89,6 @@ static Visitor *validate_test_init_raw(TestInputVisitorData *data,
     return v;
 }
 
-typedef struct TestStruct
-{
-    int64_t integer;
-    bool boolean;
-    char *string;
-} TestStruct;
-
-static void visit_type_TestStruct(Visitor *v, TestStruct **obj,
-                                  const char *name, Error **errp)
-{
-    Error *err = NULL;
-
-    visit_start_struct(v, (void **)obj, "TestStruct", name, sizeof(TestStruct),
-                       &err);
-    if (err) {
-        goto out;
-    }
-
-    visit_type_int(v, &(*obj)->integer, "integer", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_bool(v, &(*obj)->boolean, "boolean", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_str(v, &(*obj)->string, "string", &err);
-
-out_end:
-    error_propagate(errp, err);
-    err = NULL;
-    visit_end_struct(v, &err);
-out:
-    error_propagate(errp, err);
-}
 
 static void test_validate_struct(TestInputVisitorData *data,
                                   const void *unused)
tests/test-qmp-input-visitor.c +0 -34
index de65982..3f6bc4d 100644
@@ -185,40 +185,6 @@ static void test_visitor_in_enum(TestInputVisitorData *data,
     data->qiv = NULL;
 }
 
-typedef struct TestStruct
-{
-    int64_t integer;
-    bool boolean;
-    char *string;
-} TestStruct;
-
-static void visit_type_TestStruct(Visitor *v, TestStruct **obj,
-                                  const char *name, Error **errp)
-{
-    Error *err = NULL;
-
-    visit_start_struct(v, (void **)obj, "TestStruct", name, sizeof(TestStruct),
-                       &err);
-    if (err) {
-        goto out;
-    }
-    visit_type_int(v, &(*obj)->integer, "integer", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_bool(v, &(*obj)->boolean, "boolean", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_str(v, &(*obj)->string, "string", &err);
-
-out_end:
-    error_propagate(errp, err);
-    err = NULL;
-    visit_end_struct(v, &err);
-out:
-    error_propagate(errp, err);
-}
 
 static void test_visitor_in_struct(TestInputVisitorData *data,
                                    const void *unused)
tests/test-qmp-output-visitor.c +0 -58

70 lines not on this page · raw message

tests/test-visitor-serialization.c +0 -34
index 634563b..c024e5e 100644
@@ -186,40 +186,6 @@ static void visit_primitive_list(Visitor *v, void **native, Error **errp)
     }
 }
 
-typedef struct TestStruct
-{
-    int64_t integer;
-    bool boolean;
-    char *string;
-} TestStruct;
-
-static void visit_type_TestStruct(Visitor *v, TestStruct **obj,
-                                  const char *name, Error **errp)
-{
-    Error *err = NULL;
-
-    visit_start_struct(v, (void **)obj, NULL, name, sizeof(TestStruct), &err);
-    if (err) {
-        goto out;
-    }
-
-    visit_type_int(v, &(*obj)->integer, "integer", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_bool(v, &(*obj)->boolean, "boolean", &err);
-    if (err) {
-        goto out_end;
-    }
-    visit_type_str(v, &(*obj)->string, "string", &err);
-
-out_end:
-    error_propagate(errp, err);
-    err = NULL;
-    visit_end_struct(v, &err);
-out:
-    error_propagate(errp, err);
-}
 
 static TestStruct *struct_create(void)
 {
Signature · 1 line
2.4.3
author
#2 lore raw

PATCH v10 2/30 qapi: Strengthen test of TestStructList

In reply to Eric Blake

Make each list element different, to ensure that order is preserved, and use the generated free function instead of hand-rolling our own to ensure (under valgrind) that the list is properly cleaned.

---
v10: no change
v9: no change
v8: no change
v7: new patch

---
 tests/test-qmp-output-visitor.c | 18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)
tests/test-qmp-output-visitor.c +7 -11
index baf58dc..9364843 100644
@@ -283,7 +283,7 @@ static void test_visitor_out_struct_errors(TestOutputVisitorData *data,
 static void test_visitor_out_list(TestOutputVisitorData *data,
                                   const void *unused)
 {
-    char *value_str = (char *) "list value";
+    const char *value_str = "list value";
     TestStructList *p, *head = NULL;
     const int max_items = 10;
     bool value_bool = true;
@@ -294,12 +294,13 @@ static void test_visitor_out_list(TestOutputVisitorData *data,
     QList *qlist;
     int i;
 
+    /* Build the list in reverse order... */
     for (i = 0; i < max_items; i++) {
         p = g_malloc0(sizeof(*p));
         p->value = g_malloc0(sizeof(*p->value));
-        p->value->integer = value_int;
+        p->value->integer = value_int + (max_items - i - 1);
         p->value->boolean = value_bool;
-        p->value->string = value_str;
+        p->value->string = g_strdup(value_str);
 
         p->next = head;
         head = p;
@@ -315,6 +316,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data,
     qlist = qobject_to_qlist(obj);
     g_assert(!qlist_empty(qlist));
 
+    /* ...and ensure that the visitor sees it in order */
     i = 0;
     QLIST_FOREACH_ENTRY(qlist, entry) {
         QDict *qdict;
@@ -322,7 +324,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data,
         g_assert(qobject_type(entry->value) == QTYPE_QDICT);
         qdict = qobject_to_qdict(entry->value);
         g_assert_cmpint(qdict_size(qdict), ==, 3);
-        g_assert_cmpint(qdict_get_int(qdict, "integer"), ==, value_int);
+        g_assert_cmpint(qdict_get_int(qdict, "integer"), ==, value_int + i);
         g_assert_cmpint(qdict_get_bool(qdict, "boolean"), ==, value_bool);
         g_assert_cmpstr(qdict_get_str(qdict, "string"), ==, value_str);
         i++;
@@ -330,13 +332,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data,
     g_assert_cmpint(i, ==, max_items);
 
     QDECREF(qlist);
-
-    for (p = head; p;) {
-        TestStructList *tmp = p->next;
-        g_free(p->value);
-        g_free(p);
-        p = tmp;
-    }
+    qapi_free_TestStructList(head);
 }
 
 static void test_visitor_out_list_qapi_free(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
author
#3 lore raw

PATCH v10 3/30 qobject: Protect against use-after-free in qobject_decref()

In reply to Eric Blake

Adding an assertion to qobject_decref() will ensure that a programming error causing use-after-free will result in immediate failure (provided no other thread has started using the memory) instead of silently attempting to wrap refcnt around and leaving the problem to potentially bite later at a harder point to diagnose.

---
v10: new patch

---
 include/qapi/qmp/qobject.h | 1 +
 1 file changed, 1 insertion(+)
include/qapi/qmp/qobject.h +1 -0
index c856f55..4b96ed5 100644
@@ -90,6 +90,7 @@ static inline void qobject_incref(QObject *obj)
  */
 static inline void qobject_decref(QObject *obj)
 {
+    assert(!obj || obj->refcnt);
     if (obj && --obj->refcnt == 0) {
         assert(obj->type != NULL);
         assert(obj->type->destroy != NULL);
Signature · 1 line
2.4.3
author
#4 lore raw

PATCH v10 4/30 qapi: Share test_init code in test-qmp-input*

In reply to Eric Blake

Rather than duplicate the body of two functions just to decide between qobject_from_jsonv() and qobject_from_json(), exploit the fact that qobject_from_jsonv() intentionally takes 'va_list *' instead of the more common 'va_list', and that qobject_from_json() just calls qobject_from_jsonv(,NULL). For each file, our two existing init functions then become thin wrappers around a new internal function, and future updates to initialization don't have to be duplicated.

---
v10: new patch

---
 tests/test-qmp-input-strict.c  | 48 ++++++++++++++++++++---------------------
 tests/test-qmp-input-visitor.c | 49 ++++++++++++++++++++----------------------
 2 files changed, 46 insertions(+), 51 deletions(-)
tests/test-qmp-input-strict.c +23 -25
index b44184f..77151de 100644
@@ -40,9 +40,27 @@ static void validate_teardown(TestInputVisitorData *data,
     }
 }
 
-/* This is provided instead of a test setup function so that the JSON
-   string used by the tests are kept in the test functions (and not
-   int main()) */
+/* The various test_init functions are provided instead of a test setup
+   function so that the JSON string used by the tests are kept in the test
+   functions (and not int main()). */
+static Visitor *validate_test_init_internal(TestInputVisitorData *data,
+                                            const char *json_string,
+                                            va_list *ap)
+{
+    Visitor *v;
+
+    data->obj = qobject_from_jsonv(json_string, ap);
+    g_assert(data->obj);
+
+    data->qiv = qmp_input_visitor_new_strict(data->obj);
+    g_assert(data->qiv);
+
+    v = qmp_input_get_visitor(data->qiv);
+    g_assert(v);
+
+    return v;
+}
+
 static GCC_FMT_ATTR(2, 3)
 Visitor *validate_test_init(TestInputVisitorData *data,
                              const char *json_string, ...)
@@ -51,17 +69,8 @@ Visitor *validate_test_init(TestInputVisitorData *data,
     va_list ap;
 
     va_start(ap, json_string);
-    data->obj = qobject_from_jsonv(json_string, &ap);
+    v = validate_test_init_internal(data, json_string, &ap);
     va_end(ap);
-
-    g_assert(data->obj != NULL);
-
-    data->qiv = qmp_input_visitor_new_strict(data->obj);
-    g_assert(data->qiv != NULL);
-
-    v = qmp_input_get_visitor(data->qiv);
-    g_assert(v != NULL);
-
     return v;
 }
 
@@ -75,18 +84,7 @@ Visitor *validate_test_init(TestInputVisitorData *data,
 static Visitor *validate_test_init_raw(TestInputVisitorData *data,
                                        const char *json_string)
 {
-    Visitor *v;
-
-    data->obj = qobject_from_json(json_string);
-    g_assert(data->obj != NULL);
-
-    data->qiv = qmp_input_visitor_new_strict(data->obj);
-    g_assert(data->qiv != NULL);
-
-    v = qmp_input_get_visitor(data->qiv);
-    g_assert(v != NULL);
-
-    return v;
+    return validate_test_init_internal(data, json_string, NULL);
 }
 
 
tests/test-qmp-input-visitor.c +23 -26

68 lines not on this page · raw message

Signature · 1 line
2.4.3
#5 lore raw

Re: [Qemu-devel] [PATCH v10 04/30] qapi: Share test_init code in test-qmp-input*

Quoted 5 lines · Eric Blake

Rather than duplicate the body of two functions just to decide between qobject_from_jsonv() and qobject_from_json(), exploit the fact that qobject_from_jsonv() intentionally takes 'va_list *' instead of the more common 'va_list', and that qobject_from_json() just calls qobject_from_jsonv(,NULL).

Maximizes the common code. Relies on qobject_from_jsonv() treating a null ap argument specially. Okay, but I would've left the initialization of data->obj alone, to avoid the less than obvious null argument.

Two trivial remarks inline.

Quoted 19 lines

For each file, our two existing init functions then become thin wrappers around a new internal function, and future updates to initialization don't have to be duplicated.

---
v10: new patch

---
 tests/test-qmp-input-strict.c  | 48 ++++++++++++++++++++---------------------
 tests/test-qmp-input-visitor.c | 49 ++++++++++++++++++++----------------------
 2 files changed, 46 insertions(+), 51 deletions(-)
tests/test-qmp-input-strict.c +0 -0
index b44184f..77151de 100644

tests/test-qmp-input-strict.c static void validate_teardown(TestInputVisitorData *data, · line 45

Quoted 9 lines
tests/test-qmp-input-strict.c +3 -3
@@ -40,9 +40,27 @@ static void validate_teardown(TestInputVisitorData *data,
     }
 }
 
-/* This is provided instead of a test setup function so that the JSON
-   string used by the tests are kept in the test functions (and not
-   int main()) */
+/* The various test_init functions are provided instead of a test setup
+   function so that the JSON string used by the tests are kept in the test
+   functions (and not int main()). */

Since you're touching it: "not in main()".

Quoted 65 lines
tests/test-qmp-input-strict.c +20 -22
+static Visitor *validate_test_init_internal(TestInputVisitorData *data,
+                                            const char *json_string,
+                                            va_list *ap)
+{
+    Visitor *v;
+
+    data->obj = qobject_from_jsonv(json_string, ap);
+    g_assert(data->obj);
+
+    data->qiv = qmp_input_visitor_new_strict(data->obj);
+    g_assert(data->qiv);
+
+    v = qmp_input_get_visitor(data->qiv);
+    g_assert(v);
+
+    return v;
+}
+
 static GCC_FMT_ATTR(2, 3)
 Visitor *validate_test_init(TestInputVisitorData *data,
                              const char *json_string, ...)
@@ -51,17 +69,8 @@ Visitor *validate_test_init(TestInputVisitorData *data,
     va_list ap;
 
     va_start(ap, json_string);
-    data->obj = qobject_from_jsonv(json_string, &ap);
+    v = validate_test_init_internal(data, json_string, &ap);
     va_end(ap);
-
-    g_assert(data->obj != NULL);
-
-    data->qiv = qmp_input_visitor_new_strict(data->obj);
-    g_assert(data->qiv != NULL);
-
-    v = qmp_input_get_visitor(data->qiv);
-    g_assert(v != NULL);
-
     return v;
 }
 
@@ -75,18 +84,7 @@ Visitor *validate_test_init(TestInputVisitorData *data,
 static Visitor *validate_test_init_raw(TestInputVisitorData *data,
                                        const char *json_string)
 {
-    Visitor *v;
-
-    data->obj = qobject_from_json(json_string);
-    g_assert(data->obj != NULL);
-
-    data->qiv = qmp_input_visitor_new_strict(data->obj);
-    g_assert(data->qiv != NULL);
-
-    v = qmp_input_get_visitor(data->qiv);
-    g_assert(v != NULL);
-
-    return v;
+    return validate_test_init_internal(data, json_string, NULL);
 }
 
 
tests/test-qmp-input-visitor.c +0 -0
index 3f6bc4d..933243d 100644

tests/test-qmp-input-visitor.c static void visitor_input_teardown(TestInputVisitorData *data, · line 41

Quoted 9 lines
tests/test-qmp-input-visitor.c +3 -3
@@ -36,9 +36,27 @@ static void visitor_input_teardown(TestInputVisitorData *data,
     }
 }
 
-/* This is provided instead of a test setup function so that the JSON
-   string used by the tests are kept in the test functions (and not
-   int main()) */
+/* The various test_init functions are provided instead of a test setup
+   function so that the JSON string used by the tests are kept in the test
+   functions (and not int main()). */

Likewise.

Quoted 61 lines
tests/test-qmp-input-visitor.c +20 -23
+static Visitor *visitor_input_test_init_internal(TestInputVisitorData *data,
+                                                 const char *json_string,
+                                                 va_list *ap)
+{
+    Visitor *v;
+
+    data->obj = qobject_from_jsonv(json_string, ap);
+    g_assert(data->obj);
+
+    data->qiv = qmp_input_visitor_new(data->obj);
+    g_assert(data->qiv);
+
+    v = qmp_input_get_visitor(data->qiv);
+    g_assert(v);
+
+    return v;
+}
+
 static GCC_FMT_ATTR(2, 3)
 Visitor *visitor_input_test_init(TestInputVisitorData *data,
                                  const char *json_string, ...)
@@ -47,17 +65,8 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,
     va_list ap;
 
     va_start(ap, json_string);
-    data->obj = qobject_from_jsonv(json_string, &ap);
+    v = visitor_input_test_init_internal(data, json_string, &ap);
     va_end(ap);
-
-    g_assert(data->obj != NULL);
-
-    data->qiv = qmp_input_visitor_new(data->obj);
-    g_assert(data->qiv != NULL);
-
-    v = qmp_input_get_visitor(data->qiv);
-    g_assert(v != NULL);
-
     return v;
 }
 
@@ -71,19 +80,7 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,
 static Visitor *visitor_input_test_init_raw(TestInputVisitorData *data,
                                             const char *json_string)
 {
-    Visitor *v;
-
-    data->obj = qobject_from_json(json_string);
-
-    g_assert(data->obj != NULL);
-
-    data->qiv = qmp_input_visitor_new(data->obj);
-    g_assert(data->qiv != NULL);
-
-    v = qmp_input_get_visitor(data->qiv);
-    g_assert(v != NULL);
-
-    return v;
+    return visitor_input_test_init_internal(data, json_string, NULL);
 }
 
 static void test_visitor_in_int(TestInputVisitorData *data,
author
#6 lore raw

PATCH v10 5/30 qapi: Plug leaks in test-qmp-*

In reply to Eric Blake

Make valgrind happy with the current state of the tests, so that it is easier to see if future patches introduce new memory problems without being drowned in noise. Many of the leaks were due to calling a second init without tearing down the data from an earlier visit. But since teardown is already idempotent, and we already register teardown as part of input_visitor_test_add(), it is nicer to just make init() safe to call multiple times than it is to have to make all tests call teardown.

Another common leak was forgetting to clean up an error object, after testing that an error was raised.

Another leak was in test_visitor_in_struct_nested(), failing to clean the base member of UserDefTwo. Cleaning that up left check_and_free_str() as dead code (since using the qapi_free_* takes care of recursion, and we don't want double frees).

A final leak was in test_visitor_out_any(), which was reassigning the qobj local variable to a subset of the overall structure needing freeing; it did not result in a use-after-free, but was not cleaning up all the qdict.

test-qmp-event and test-qmp-commands were already clean.

---
v10: improve commit message, split out refactor of test init so that both init and init_raw benefit from change
v9: move earlier in series (was 13/17)
v8: no change
v7: no change
v6: make init repeatable rather than adding teardown everywhere, fix additional leak with UserDefTwo base, plug additional files

---
 tests/test-qmp-input-strict.c   |  9 +++++++++
 tests/test-qmp-input-visitor.c  | 41 +++++++----------------------------------
 tests/test-qmp-output-visitor.c |  3 ++-
 3 files changed, 18 insertions(+), 35 deletions(-)
tests/test-qmp-input-strict.c +9 -0
index 77151de..95cd8e1 100644
@@ -49,6 +49,8 @@ static Visitor *validate_test_init_internal(TestInputVisitorData *data,
 {
     Visitor *v;
 
+    validate_teardown(data, NULL);
+
     data->obj = qobject_from_jsonv(json_string, ap);
     g_assert(data->obj);
 
@@ -191,6 +193,7 @@ static void test_validate_fail_struct(TestInputVisitorData *data,
 
     visit_type_TestStruct(v, &p, NULL, &err);
     g_assert(err);
+    error_free(err);
     if (p) {
         g_free(p->string);
     }
@@ -208,6 +211,7 @@ static void test_validate_fail_struct_nested(TestInputVisitorData *data,
 
     visit_type_UserDefTwo(v, &udp, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefTwo(udp);
 }
 
@@ -222,6 +226,7 @@ static void test_validate_fail_list(TestInputVisitorData *data,
 
     visit_type_UserDefOneList(v, &head, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefOneList(head);
 }
 
@@ -237,6 +242,7 @@ static void test_validate_fail_union_native_list(TestInputVisitorData *data,
 
     visit_type_UserDefNativeListUnion(v, &tmp, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefNativeListUnion(tmp);
 }
 
@@ -251,6 +257,7 @@ static void test_validate_fail_union_flat(TestInputVisitorData *data,
 
     visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefFlatUnion(tmp);
 }
 
@@ -266,6 +273,7 @@ static void test_validate_fail_union_flat_no_discrim(TestInputVisitorData *data,
 
     visit_type_UserDefFlatUnion2(v, &tmp, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefFlatUnion2(tmp);
 }
 
@@ -280,6 +288,7 @@ static void test_validate_fail_alternate(TestInputVisitorData *data,
 
     visit_type_UserDefAlternate(v, &tmp, NULL, &err);
     g_assert(err);
+    error_free(err);
     qapi_free_UserDefAlternate(tmp);
 }
 
tests/test-qmp-input-visitor.c +7 -34

159 lines not on this page · raw message

tests/test-qmp-output-visitor.c +2 -1
index 9364843..8606111 100644
@@ -391,6 +391,7 @@ static void test_visitor_out_any(TestOutputVisitorData *data,
     qobj = QOBJECT(qdict);
     visit_type_any(data->ov, &qobj, NULL, &err);
     g_assert(!err);
+    qobject_decref(qobj);
     obj = qmp_output_get_qobject(data->qov);
     g_assert(obj != NULL);
     qdict = qobject_to_qdict(obj);
@@ -411,7 +412,6 @@ static void test_visitor_out_any(TestOutputVisitorData *data,
     g_assert(qstring);
     g_assert_cmpstr(qstring_get_str(qstring), ==, "foo");
     qobject_decref(obj);
-    qobject_decref(qobj);
 }
 
 static void test_visitor_out_union_flat(TestOutputVisitorData *data,
@@ -463,6 +463,7 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,
     g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42);
 
     qapi_free_UserDefAlternate(tmp);
+    qobject_decref(arg);
 }
 
 static void test_visitor_out_empty(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
#7 lore raw

Re: [Qemu-devel] [PATCH v10 05/30] qapi: Plug leaks in test-qmp-*

Quoted 54 lines · Eric Blake

Make valgrind happy with the current state of the tests, so that it is easier to see if future patches introduce new memory problems without being drowned in noise. Many of the leaks were due to calling a second init without tearing down the data from an earlier visit. But since teardown is already idempotent, and we already register teardown as part of input_visitor_test_add(), it is nicer to just make init() safe to call multiple times than it is to have to make all tests call teardown.

Another common leak was forgetting to clean up an error object, after testing that an error was raised.

Another leak was in test_visitor_in_struct_nested(), failing to clean the base member of UserDefTwo. Cleaning that up left check_and_free_str() as dead code (since using the qapi_free_* takes care of recursion, and we don't want double frees).

A final leak was in test_visitor_out_any(), which was reassigning the qobj local variable to a subset of the overall structure needing freeing; it did not result in a use-after-free, but was not cleaning up all the qdict.

test-qmp-event and test-qmp-commands were already clean.

---
v10: improve commit message, split out refactor of test init so that both init and init_raw benefit from change
v9: move earlier in series (was 13/17)
v8: no change
v7: no change
v6: make init repeatable rather than adding teardown everywhere, fix additional leak with UserDefTwo base, plug additional files

---
 tests/test-qmp-input-strict.c   |  9 +++++++++
 tests/test-qmp-input-visitor.c  | 41 +++++++----------------------------------
 tests/test-qmp-output-visitor.c |  3 ++-
 3 files changed, 18 insertions(+), 35 deletions(-)
tests/test-qmp-input-strict.c +2 -0
index 77151de..95cd8e1 100644
@@ -49,6 +49,8 @@ static Visitor *validate_test_init_internal(TestInputVisitorData *data,
 {
     Visitor *v;
 
+    validate_teardown(data, NULL);
+
     data->obj = qobject_from_jsonv(json_string, ap);
     g_assert(data->obj);
 

tests/test-qmp-input-strict.c static void test_validate_fail_struct(TestInputVisitorData *data, · line 196

Quoted 4 lines
tests/test-qmp-input-strict.c +1 -0
@@ -191,6 +193,7 @@ static void test_validate_fail_struct(TestInputVisitorData *data,
 
     visit_type_TestStruct(v, &p, NULL, &err);
     g_assert(err);
+    error_free(err);

v9 had /* FIXME: visitor should not allocate p when returning error */ here. Did you drop it intentionally?

If you put it back, please mention it in the commit message.

Quoted 3 lines
tests/test-qmp-input-strict.c +0 -0
     if (p) {
         g_free(p->string);
     }

[...]

author
#8 lore raw

Re: [Qemu-devel] [PATCH v10 05/30] qapi: Plug leaks in test-qmp-*

Quoted 12 lines · Markus Armbruster
Quoted 9 lines · Eric Blake

Make valgrind happy with the current state of the tests, so that it is easier to see if future patches introduce new memory problems without being drowned in noise. Many of the leaks were due to calling a second init without tearing down the data from an earlier visit. But since teardown is already idempotent, and we already register teardown as part of input_visitor_test_add(), it is nicer to just make init() safe to call multiple times than it is to have to make all tests call teardown.

Quoted 9 lines

tests/test-qmp-input-strict.c static void test_validate_fail_struct(TestInputVisitorData *data, · line 196

Quoted 5 lines
tests/test-qmp-input-strict.c +1 -0
@@ -191,6 +193,7 @@ static void test_validate_fail_struct(TestInputVisitorData *data,
 
     visit_type_TestStruct(v, &p, NULL, &err);
     g_assert(err);
+    error_free(err);

v9 had /* FIXME: visitor should not allocate p when returning error */ here. Did you drop it intentionally?

Yes, I dropped it intentionally.

Quoted 1 line

If you put it back, please mention it in the commit message.

Rather, I decided that it was sufficient to just fix the bug later (patch already queued, but not posted), and minimize the churn (no need to mark things up just to pull down the markings later):
http://repo.or.cz/qemu/ericb.git/commitdiff/ac584b53a

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#9 lore raw

PATCH v10 6/30 qapi: Simplify non-error testing in test-qmp-*

In reply to Eric Blake

By using &error_abort, we can avoid a local err variable in situations where we expect success. It also has the nice effect that if the test breaks, the error message from error_abort tends to be nicer than that of g_assert().

---
v10: split into two pieces
v9: move earlier in series (was 14/17), reword commit message
v8: no change
v7: pick up whitespace changes dropped from earlier commit
v6: new patch

---
 tests/test-qmp-input-strict.c      | 31 +++++---------------
 tests/test-qmp-input-visitor.c     | 59 ++++++++++----------------------------
 tests/test-qmp-output-visitor.c    | 56 +++++++++---------------------------
 tests/test-visitor-serialization.c | 42 +++++++++++----------------
 4 files changed, 53 insertions(+), 135 deletions(-)
tests/test-qmp-input-strict.c +7 -24
index 95cd8e1..6226397 100644
@@ -94,13 +94,11 @@ static void test_validate_struct(TestInputVisitorData *data,
                                   const void *unused)
 {
     TestStruct *p = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init(data, "{ 'integer': -42, 'boolean': true, 'string': 'foo' }");
 
-    visit_type_TestStruct(v, &p, NULL, &err);
-    g_assert(!err);
+    visit_type_TestStruct(v, &p, NULL, &error_abort);
     g_free(p->string);
     g_free(p);
 }
@@ -109,7 +107,6 @@ static void test_validate_struct_nested(TestInputVisitorData *data,
                                          const void *unused)
 {
     UserDefTwo *udp = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init(data, "{ 'string0': 'string0', "
@@ -117,8 +114,7 @@ static void test_validate_struct_nested(TestInputVisitorData *data,
                            "'dict2': { 'userdef': { 'integer': 42, "
                            "'string': 'string' }, 'string': 'string2'}}}");
 
-    visit_type_UserDefTwo(v, &udp, NULL, &err);
-    g_assert(!err);
+    visit_type_UserDefTwo(v, &udp, NULL, &error_abort);
     qapi_free_UserDefTwo(udp);
 }
 
@@ -126,13 +122,11 @@ static void test_validate_list(TestInputVisitorData *data,
                                 const void *unused)
 {
     UserDefOneList *head = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init(data, "[ { 'string': 'string0', 'integer': 42 }, { 'string': 'string1', 'integer': 43 }, { 'string': 'string2', 'integer': 44 } ]");
 
-    visit_type_UserDefOneList(v, &head, NULL, &err);
-    g_assert(!err);
+    visit_type_UserDefOneList(v, &head, NULL, &error_abort);
     qapi_free_UserDefOneList(head);
 }
 
@@ -141,12 +135,10 @@ static void test_validate_union_native_list(TestInputVisitorData *data,
 {
     UserDefNativeListUnion *tmp = NULL;
     Visitor *v;
-    Error *err = NULL;
 
     v = validate_test_init(data, "{ 'type': 'integer', 'data' : [ 1, 2 ] }");
 
-    visit_type_UserDefNativeListUnion(v, &tmp, NULL, &err);
-    g_assert(!err);
+    visit_type_UserDefNativeListUnion(v, &tmp, NULL, &error_abort);
     qapi_free_UserDefNativeListUnion(tmp);
 }
 
@@ -155,7 +147,6 @@ static void test_validate_union_flat(TestInputVisitorData *data,
 {
     UserDefFlatUnion *tmp = NULL;
     Visitor *v;
-    Error *err = NULL;
 
     v = validate_test_init(data,
                            "{ 'enum1': 'value1', "
@@ -163,8 +154,7 @@ static void test_validate_union_flat(TestInputVisitorData *data,
                            "'string': 'str', "
                            "'boolean': true }");
 
-    visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
-    g_assert(!err);
+    visit_type_UserDefFlatUnion(v, &tmp, NULL, &error_abort);
     qapi_free_UserDefFlatUnion(tmp);
 }
 
@@ -173,12 +163,10 @@ static void test_validate_alternate(TestInputVisitorData *data,
 {
     UserDefAlternate *tmp = NULL;
     Visitor *v;
-    Error *err = NULL;
 
     v = validate_test_init(data, "42");
 
-    visit_type_UserDefAlternate(v, &tmp, NULL, &err);
-    g_assert(!err);
+    visit_type_UserDefAlternate(v, &tmp, NULL, &error_abort);
     qapi_free_UserDefAlternate(tmp);
 }
 
@@ -296,16 +284,11 @@ static void do_test_validate_qmp_introspect(TestInputVisitorData *data,
                                             const char *schema_json)
 {
     SchemaInfoList *schema = NULL;
-    Error *err = NULL;
     Visitor *v;
 
     v = validate_test_init_raw(data, schema_json);
 
-    visit_type_SchemaInfoList(v, &schema, NULL, &err);
-    if (err) {
-        fprintf(stderr, "%s", error_get_pretty(err));
-    }
-    g_assert(!err);
+    visit_type_SchemaInfoList(v, &schema, NULL, &error_abort);
     g_assert(schema);
 
     qapi_free_SchemaInfoList(schema);
tests/test-qmp-input-visitor.c +15 -44

223 lines not on this page · raw message

tests/test-qmp-output-visitor.c +14 -42

193 lines not on this page · raw message

tests/test-visitor-serialization.c +17 -25

103 lines not on this page · raw message

Signature · 1 line
2.4.3
#10 lore raw

Re: [Qemu-devel] [PATCH v10 06/30] qapi: Simplify non-error testing in test-qmp-*

Quoted 6 lines · Eric Blake

By using &error_abort, we can avoid a local err variable in situations where we expect success. It also has the nice effect that if the test breaks, the error message from error_abort tends to be nicer than that of g_assert().

[Boring mechanical changes snipped...]

Quoted 9 lines
tests/test-visitor-serialization.c +0 -1
index c024e5e..9f67f9e 100644
@@ -302,14 +302,13 @@ static void test_primitives(gconstpointer opaque)
     const SerializeOps *ops = args->ops;
     PrimitiveType *pt = args->test_data;
     PrimitiveType *pt_copy = g_malloc0(sizeof(*pt_copy));
-    Error *err = NULL;

tests/test-visitor-serialization.c static void test_primitives(gconstpointer opaque) · line 312

Quoted 10 lines
tests/test-visitor-serialization.c +3 -3
     void *serialize_data;
 
     pt_copy->type = pt->type;
-    ops->serialize(pt, &serialize_data, visit_primitive_type, &err);
-    ops->deserialize((void **)&pt_copy, serialize_data, visit_primitive_type, &err);
+    ops->serialize(pt, &serialize_data, visit_primitive_type, &error_abort);
+    ops->deserialize((void **)&pt_copy, serialize_data, visit_primitive_type,
+                     &error_abort);
 
-    g_assert(err == NULL);

This looks like a (very minor) bug fix / cleanup: you're not supposed to pass the same &err to multiple functions without checking and clearing it in between, because the second failure trips assert(*errp == NULL) in error_setv(). Harmless here, but it's nice to get rid of a bad example.

Several more below.

Quoted 86 lines
tests/test-visitor-serialization.c +11 -18
     g_assert(pt_copy != NULL);
     if (pt->type == PTYPE_STRING) {
         g_assert_cmpstr(pt->value.string, ==, pt_copy->value.string);
@@ -345,7 +344,6 @@ static void test_primitive_lists(gconstpointer opaque)
     PrimitiveList pl = { .value = { NULL } };
     PrimitiveList pl_copy = { .value = { NULL } };
     PrimitiveList *pl_copy_ptr = &pl_copy;
-    Error *err = NULL;
     void *serialize_data;
     void *cur_head = NULL;
     int i;
@@ -492,10 +490,11 @@ static void test_primitive_lists(gconstpointer opaque)
         }
     }
 
-    ops->serialize((void **)&pl, &serialize_data, visit_primitive_list, &err);
-    ops->deserialize((void **)&pl_copy_ptr, serialize_data, visit_primitive_list, &err);
+    ops->serialize((void **)&pl, &serialize_data, visit_primitive_list,
+                   &error_abort);
+    ops->deserialize((void **)&pl_copy_ptr, serialize_data,
+                     visit_primitive_list, &error_abort);
 
-    g_assert(err == NULL);
     i = 0;
 
     /* compare our deserialized list of primitives to the original */
@@ -652,10 +651,8 @@ static void test_primitive_lists(gconstpointer opaque)
     g_assert_cmpint(i, ==, 33);
 
     ops->cleanup(serialize_data);
-    dealloc_helper(&pl, visit_primitive_list, &err);
-    g_assert(!err);
-    dealloc_helper(&pl_copy, visit_primitive_list, &err);
-    g_assert(!err);
+    dealloc_helper(&pl, visit_primitive_list, &error_abort);
+    dealloc_helper(&pl_copy, visit_primitive_list, &error_abort);
     g_free(args);
 }
 
@@ -665,13 +662,12 @@ static void test_struct(gconstpointer opaque)
     const SerializeOps *ops = args->ops;
     TestStruct *ts = struct_create();
     TestStruct *ts_copy = NULL;
-    Error *err = NULL;
     void *serialize_data;
 
-    ops->serialize(ts, &serialize_data, visit_struct, &err);
-    ops->deserialize((void **)&ts_copy, serialize_data, visit_struct, &err); 
+    ops->serialize(ts, &serialize_data, visit_struct, &error_abort);
+    ops->deserialize((void **)&ts_copy, serialize_data, visit_struct,
+                     &error_abort);
 
-    g_assert(err == NULL);
     struct_compare(ts, ts_copy);
 
     struct_cleanup(ts);
@@ -687,14 +683,12 @@ static void test_nested_struct(gconstpointer opaque)
     const SerializeOps *ops = args->ops;
     UserDefTwo *udnp = nested_struct_create();
     UserDefTwo *udnp_copy = NULL;
-    Error *err = NULL;
     void *serialize_data;
 
-    ops->serialize(udnp, &serialize_data, visit_nested_struct, &err);
+    ops->serialize(udnp, &serialize_data, visit_nested_struct, &error_abort);
     ops->deserialize((void **)&udnp_copy, serialize_data, visit_nested_struct,
-                     &err);
+                     &error_abort);
 
-    g_assert(err == NULL);
     nested_struct_compare(udnp, udnp_copy);
 
     nested_struct_cleanup(udnp);
@@ -709,7 +703,6 @@ static void test_nested_struct_list(gconstpointer opaque)
     TestArgs *args = (TestArgs *) opaque;
     const SerializeOps *ops = args->ops;
     UserDefTwoList *listp = NULL, *tmp, *tmp_copy, *listp_copy = NULL;
-    Error *err = NULL;
     void *serialize_data;
     int i = 0;
 
@@ -720,11 +713,10 @@ static void test_nested_struct_list(gconstpointer opaque)
         listp = tmp;
     }
 
-    ops->serialize(listp, &serialize_data, visit_nested_struct_list, &err);

tests/test-visitor-serialization.c static void test_nested_struct_list(gconstpointer opaque) · line 722

Quoted 10 lines
tests/test-visitor-serialization.c +3 -3
+    ops->serialize(listp, &serialize_data, visit_nested_struct_list,
+                   &error_abort);
     ops->deserialize((void **)&listp_copy, serialize_data,
-                     visit_nested_struct_list, &err); 
-
-    g_assert(err == NULL);
+                     visit_nested_struct_list, &error_abort);
 
     tmp = listp;
     tmp_copy = listp_copy;

Suggest to note the cleanup in the commit message.

author
#11 lore raw

Re: [Qemu-devel] [PATCH v10 06/30] qapi: Simplify non-error testing in test-qmp-*

Quoted 17 lines · Markus Armbruster
Quoted 6 lines · Eric Blake

By using &error_abort, we can avoid a local err variable in situations where we expect success. It also has the nice effect that if the test breaks, the error message from error_abort tends to be nicer than that of g_assert().

[Boring mechanical changes snipped...]

Quoted 7 lines
     pt_copy->type = pt->type;
tests/test-visitor-serialization.c +3 -2
-    ops->serialize(pt, &serialize_data, visit_primitive_type, &err);
-    ops->deserialize((void **)&pt_copy, serialize_data, visit_primitive_type, &err);
+    ops->serialize(pt, &serialize_data, visit_primitive_type, &error_abort);
+    ops->deserialize((void **)&pt_copy, serialize_data, visit_primitive_type,
+                     &error_abort);
Quoted 6 lines
Quoted 1 line

- g_assert(err == NULL);

This looks like a (very minor) bug fix / cleanup: you're not supposed to pass the same &err to multiple functions without checking and clearing it in between, because the second failure trips assert(*errp == NULL) in error_setv(). Harmless here, but it's nice to get rid of a bad example.

Harmless here because we are asserting that err is still NULL after the chain (which means it was NULL at all points during the chain). But I agree that it is nice to get rid of poor practice, and that adding a paragraph to the commit message to point it out would be a nice idea.

Quoted 2 lines

Suggest to note the cleanup in the commit message.

We may be close enough to take the series without needing a v11; if that's the case, and you are the one squashing in the change, how about this text:

This patch has an additional bonus of fixing several call sites that were passing &err to two different functions without checking it in between. In general that is unsafe practice; because if the first function sets an error, the second function could abort() if it tries to set a different error. We got away with it because we were asserting that err was NULL through the entire chain, but switching to &error_abort avoids the questionable practice up front.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#12 lore raw

Re: [Qemu-devel] [PATCH v10 06/30] qapi: Simplify non-error testing in test-qmp-*

Quoted 39 lines · Eric Blake
Quoted 19 lines · Markus Armbruster
Quoted 6 lines · Eric Blake

By using &error_abort, we can avoid a local err variable in situations where we expect success. It also has the nice effect that if the test breaks, the error message from error_abort tends to be nicer than that of g_assert().

[Boring mechanical changes snipped...]

Quoted 9 lines
     pt_copy->type = pt->type;

- ops->serialize(pt, &serialize_data, visit_primitive_type, &err);
- ops->deserialize((void **)&pt_copy, serialize_data, visit_primitive_type, &err);
+ ops->serialize(pt, &serialize_data, visit_primitive_type, &error_abort);
+ ops->deserialize((void **)&pt_copy, serialize_data, visit_primitive_type,
+ &error_abort);

Quoted 6 lines
Quoted 1 line

- g_assert(err == NULL);

This looks like a (very minor) bug fix / cleanup: you're not supposed to pass the same &err to multiple functions without checking and clearing it in between, because the second failure trips assert(*errp == NULL) in error_setv(). Harmless here, but it's nice to get rid of a bad example.

Harmless here because we are asserting that err is still NULL after the chain (which means it was NULL at all points during the chain). But I agree that it is nice to get rid of poor practice, and that adding a paragraph to the commit message to point it out would be a nice idea.

Quoted 2 lines

Suggest to note the cleanup in the commit message.

We may be close enough to take the series without needing a v11; if that's the case, and you are the one squashing in the change, how about this text:

Quoted 7 lines

This patch has an additional bonus of fixing several call sites that were passing &err to two different functions without checking it in between. In general that is unsafe practice; because if the first function sets an error, the second function could abort() if it tries to set a different error. We got away with it because we were asserting that err was NULL through the entire chain, but switching to &error_abort avoids the questionable practice up front.

Works for me.

author
#13 lore raw

PATCH v10 7/30 qapi: Simplify error cleanup in test-qmp-*

In reply to Eric Blake

By moving err into data, we can let test teardown take care of cleaning up any collected error; it also gives us fewer lines of code between repeated tests where init runs teardown on our behalf.

Rather than duplicate code between .c files, I added a new test-qmp-common.h. I debated about putting error_free_or_abort() in error.h, but it seems like something that is only useful for tests.

---
v10: split in two pieces
v9: move earlier in series (was 14/17), reword commit message
v8: no change
v7: pick up whitespace changes dropped from earlier commit
v6: new patch

---
 tests/test-qmp-commands.c      |  8 ++++----
 tests/test-qmp-common.h        | 22 ++++++++++++++++++++++
 tests/test-qmp-event.c         |  1 +
 tests/test-qmp-input-strict.c  | 22 ++++++++--------------
 tests/test-qmp-input-visitor.c | 27 ++++++++-------------------
 5 files changed, 43 insertions(+), 37 deletions(-)
 create mode 100644 tests/test-qmp-common.h
tests/test-qmp-commands.c +4 -4
index f23d8ea..9f65fc2 100644
@@ -1,12 +1,13 @@
 #include <glib.h>
 #include "qemu-common.h"
+#include "test-qmp-common.h"
 #include "qapi/qmp/types.h"
 #include "test-qmp-commands.h"
 #include "qapi/qmp/dispatch.h"
 #include "qemu/module.h"
 #include "qapi/qmp-input-visitor.h"
-#include "tests/test-qapi-types.h"
-#include "tests/test-qapi-visit.h"
+#include "test-qapi-types.h"
+#include "test-qapi-visit.h"
 
 void qmp_user_def_cmd(Error **errp)
 {
@@ -225,8 +226,7 @@ static void test_dealloc_partial(void)
     assert(ud2->dict1 == NULL);
 
     /* confirm & release construction error */
-    assert(err != NULL);
-    error_free(err);
+    error_free_or_abort(&err);
 
     /* tear down partial object */
     qapi_free_UserDefTwo(ud2);
tests/test-qmp-common.h new file +22 -0
new file mode 100644
index 0000000..043f49c
@@ -0,0 +1,22 @@
+/*
+ * Code common to qmp/qapi unit-tests.
+ *
+ * Copyright (C) 2015 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#ifndef TEST_QMP_COMMON_H__
+#define TEST_QMP_COMMON_H__
+
+/* Expect an error, abort() if there is none. */
+static inline void error_free_or_abort(Error **errp)
+{
+    g_assert(*errp);
+    error_free(*errp);
+    *errp = NULL;
+}
+
+#endif
tests/test-qmp-event.c +1 -0
index 035c65c..c0ac3ea 100644
@@ -22,6 +22,7 @@
 #include "qapi/qmp/qint.h"
 #include "qapi/qmp/qobject.h"
 #include "qapi/qmp-event.h"
+#include "test-qmp-common.h"
 
 typedef struct TestEventData {
     QDict *expect;
tests/test-qmp-input-strict.c +8 -14
index 6226397..7e15b09 100644
@@ -15,6 +15,7 @@
 #include <stdarg.h>
 
 #include "qemu-common.h"
+#include "test-qmp-common.h"
 #include "qapi/qmp-input-visitor.h"
 #include "test-qapi-types.h"
 #include "test-qapi-visit.h"
@@ -180,8 +181,7 @@ static void test_validate_fail_struct(TestInputVisitorData *data,
     v = validate_test_init(data, "{ 'integer': -42, 'boolean': true, 'string': 'foo', 'extra': 42 }");
 
     visit_type_TestStruct(v, &p, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    error_free_or_abort(&err);
     if (p) {
         g_free(p->string);
     }
@@ -198,8 +198,7 @@ static void test_validate_fail_struct_nested(TestInputVisitorData *data,
     v = validate_test_init(data, "{ 'string0': 'string0', 'dict1': { 'string1': 'string1', 'dict2': { 'userdef1': { 'integer': 42, 'string': 'string', 'extra': [42, 23, {'foo':'bar'}] }, 'string2': 'string2'}}}");
 
     visit_type_UserDefTwo(v, &udp, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    error_free_or_abort(&err);
     qapi_free_UserDefTwo(udp);
 }
 
@@ -213,8 +212,7 @@ static void test_validate_fail_list(TestInputVisitorData *data,
     v = validate_test_init(data, "[ { 'string': 'string0', 'integer': 42 }, { 'string': 'string1', 'integer': 43 }, { 'string': 'string2', 'integer': 44, 'extra': 'ggg' } ]");
 
     visit_type_UserDefOneList(v, &head, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    error_free_or_abort(&err);
     qapi_free_UserDefOneList(head);
 }
 
@@ -229,8 +227,7 @@ static void test_validate_fail_union_native_list(TestInputVisitorData *data,
                            "{ 'type': 'integer', 'data' : [ 'string' ] }");
 
     visit_type_UserDefNativeListUnion(v, &tmp, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    error_free_or_abort(&err);
     qapi_free_UserDefNativeListUnion(tmp);
 }
 
@@ -244,8 +241,7 @@ static void test_validate_fail_union_flat(TestInputVisitorData *data,
     v = validate_test_init(data, "{ 'string': 'c', 'integer': 41, 'boolean': true }");
 
     visit_type_UserDefFlatUnion(v, &tmp, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    error_free_or_abort(&err);
     qapi_free_UserDefFlatUnion(tmp);
 }
 
@@ -260,8 +256,7 @@ static void test_validate_fail_union_flat_no_discrim(TestInputVisitorData *data,
     v = validate_test_init(data, "{ 'integer': 42, 'string': 'c', 'string1': 'd', 'string2': 'e' }");
 
     visit_type_UserDefFlatUnion2(v, &tmp, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    error_free_or_abort(&err);
     qapi_free_UserDefFlatUnion2(tmp);
 }
 
@@ -275,8 +270,7 @@ static void test_validate_fail_alternate(TestInputVisitorData *data,
     v = validate_test_init(data, "3.14");
 
     visit_type_UserDefAlternate(v, &tmp, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    error_free_or_abort(&err);
     qapi_free_UserDefAlternate(tmp);
 }
 
tests/test-qmp-input-visitor.c +8 -19
index 46566bc..2b59b6f 100644
@@ -14,6 +14,7 @@
 #include <stdarg.h>
 
 #include "qemu-common.h"
+#include "test-qmp-common.h"
 #include "qapi/qmp-input-visitor.h"
 #include "test-qapi-types.h"
 #include "test-qapi-visit.h"
@@ -111,8 +112,7 @@ static void test_visitor_in_int_overflow(TestInputVisitorData *data,
     v = visitor_input_test_init(data, "%f", DBL_MAX);
 
     visit_type_int(v, &res, NULL, &err);
-    g_assert(err);
-    error_free(err);
+    error_free_or_abort(&err);
 }
 
 static void test_visitor_in_bool(TestInputVisitorData *data,
@@ -319,9 +319,7 @@ static void test_visitor_in_alternate(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "false");
     visit_type_UserDefAlternate(v, &tmp, NULL, &err);
-    g_assert(err);
-    error_free(err);
-    err = NULL;
+    error_free_or_abort(&err);
     qapi_free_UserDefAlternate(tmp);
 }
 
@@ -341,9 +339,7 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42");
     visit_type_AltStrBool(v, &asb, NULL, &err);
-    g_assert(err);
-    error_free(err);
-    err = NULL;
+    error_free_or_abort(&err);
     qapi_free_AltStrBool(asb);
 
     /* FIXME: Order of alternate should not affect semantics; asn should
@@ -352,9 +348,7 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     visit_type_AltStrNum(v, &asn, NULL, &err);
     /* FIXME g_assert_cmpint(asn->type, == ALT_STR_NUM_KIND_N); */
     /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */
-    g_assert(err);
-    error_free(err);
-    err = NULL;
+    error_free_or_abort(&err);
     qapi_free_AltStrNum(asn);
 
     v = visitor_input_test_init(data, "42");
@@ -385,9 +379,7 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltStrBool(v, &asb, NULL, &err);
-    g_assert(err);
-    error_free(err);
-    err = NULL;
+    error_free_or_abort(&err);
     qapi_free_AltStrBool(asb);
 
     v = visitor_input_test_init(data, "42.5");
@@ -404,9 +396,7 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltStrInt(v, &asi, NULL, &err);
-    g_assert(err);
-    error_free(err);
-    err = NULL;
+    error_free_or_abort(&err);
     qapi_free_AltStrInt(asi);
 
     v = visitor_input_test_init(data, "42.5");
@@ -713,12 +703,11 @@ static void test_visitor_in_errors(TestInputVisitorData *data,
     v = visitor_input_test_init(data, "{ 'integer': false, 'boolean': 'foo', 'string': -42 }");
 
     visit_type_TestStruct(v, &p, NULL, &err);
-    g_assert(err);
+    error_free_or_abort(&err);
     /* FIXME - a failed parse should not leave a partially-allocated p
      * for us to clean up; this could cause callers to leak memory. */
     g_assert(p->string == NULL);
 
-    error_free(err);
     g_free(p->string);
     g_free(p);
 }
Signature · 1 line
2.4.3
#14 lore raw

Re: [Qemu-devel] [PATCH v10 07/30] qapi: Simplify error cleanup in test-qmp-*

Quoted 4 lines · Eric Blake

By moving err into data, we can let test teardown take care of cleaning up any collected error; it also gives us fewer lines of code between repeated tests where init runs teardown on our behalf.

I think this paragraph is no longer valid: you aren't moving err anywhere in this version.

Quoted 4 lines

Rather than duplicate code between .c files, I added a new test-qmp-common.h. I debated about putting error_free_or_abort() in error.h, but it seems like something that is only useful for tests.

Maybe, maybe not. I'd accept it into error.h.

Quoted 1 line

Patch looks okay.

author
#15 lore raw

Re: [Qemu-devel] [PATCH v10 07/30] qapi: Simplify error cleanup in test-qmp-*

Quoted 9 lines · Markus Armbruster
Quoted 4 lines · Eric Blake

By moving err into data, we can let test teardown take care of cleaning up any collected error; it also gives us fewer lines of code between repeated tests where init runs teardown on our behalf.

I think this paragraph is no longer valid: you aren't moving err anywhere in this version.

D'oh. I scrubbed the code, but not the commit message. At least it was a faithful split of the v9 version of the commit, before I reworked the mechanism :)

How about:

We have several tests that perform multiple sub-actions that are expected to fail. Asserting that an error occurred, then clearing it up to prepare for the next action, turned into enough boilerplate that it was sometimes forgotten, risking memory leak or invalidating the efforts of the second action (passing a non-NULL err into a function is generally a bad idea). Encapsulate the boilerplate into a single helper function error_free_or_abort(), and consistently use it.

Quoted 7 lines
Quoted 5 lines

Rather than duplicate code between .c files, I added a new test-qmp-common.h. I debated about putting error_free_or_abort() in error.h, but it seems like something that is only useful for tests.

Maybe, maybe not. I'd accept it into error.h.

Do you want me to post a fixup patch that relocates it into error.h?

Quoted 5 lines
Quoted 2 lines

Patch looks okay.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#16 lore raw

Re: [Qemu-devel] [PATCH v10 07/30] qapi: Simplify error cleanup in test-qmp-*

Quoted 15 lines · Eric Blake
Quoted 9 lines · Markus Armbruster
Quoted 4 lines · Eric Blake

By moving err into data, we can let test teardown take care of cleaning up any collected error; it also gives us fewer lines of code between repeated tests where init runs teardown on our behalf.

I think this paragraph is no longer valid: you aren't moving err anywhere in this version.

D'oh. I scrubbed the code, but not the commit message. At least it was a faithful split of the v9 version of the commit, before I reworked the mechanism :)

Quoted 6 lines

How about:

We have several tests that perform multiple sub-actions that are expected to fail. Asserting that an error occurred, then clearing it up to prepare for the next action, turned into enough boilerplate that it was sometimes forgotten,

If you got suitable commit SHAs handy, lets insert some here.

Quoted 4 lines

risking memory leak or invalidating the efforts
of the second action (passing a non-NULL err into a function is generally a bad idea). Encapsulate the boilerplate into a single helper function error_free_or_abort(), and consistently use it.

Works for me.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Rather than duplicate code between .c files, I added a new test-qmp-common.h. I debated about putting error_free_or_abort() in error.h, but it seems like something that is only useful for tests.

Maybe, maybe not. I'd accept it into error.h.

Do you want me to post a fixup patch that relocates it into error.h?

I guess the result would be slightly simpler, so go ahead.

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

Patch looks okay.

author
#17 lore raw

Re: [Qemu-devel] [PATCH v10 07/30] qapi: Simplify error cleanup in test-qmp-*

Quoted 17 lines · Markus Armbruster
Quoted 15 lines · Eric Blake
Quoted 9 lines · Markus Armbruster
Quoted 4 lines · Eric Blake

By moving err into data, we can let test teardown take care of cleaning up any collected error; it also gives us fewer lines of code between repeated tests where init runs teardown on our behalf.

I think this paragraph is no longer valid: you aren't moving err anywhere in this version.

D'oh. I scrubbed the code, but not the commit message. At least it was a faithful split of the v9 version of the commit, before I reworked the mechanism :)

Quoted 8 lines
Quoted 6 lines

How about:

We have several tests that perform multiple sub-actions that are expected to fail. Asserting that an error occurred, then clearing it up to prepare for the next action, turned into enough boilerplate that it was sometimes forgotten,

If you got suitable commit SHAs handy, lets insert some here.

A number of them were introduced in d88f5fd (look at test-qmp-input-visitor.c:test_validate_fail_struct_nested(), for example), and only barely cleaned up in 5/30 of this series. I'm not finding other commits off-hand, though.

Quoted 13 lines
Quoted 5 lines

risking memory leak or invalidating the efforts
of the second action (passing a non-NULL err into a function is generally a bad idea). Encapsulate the boilerplate into a single helper function error_free_or_abort(), and consistently use it.

Works for me.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Rather than duplicate code between .c files, I added a new test-qmp-common.h. I debated about putting error_free_or_abort() in error.h, but it seems like something that is only useful for tests.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Maybe, maybe not. I'd accept it into error.h.

Do you want me to post a fixup patch that relocates it into error.h?

I guess the result would be slightly simpler, so go ahead.

Coming up soon.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#18 lore raw

PATCH 1/1 fixup! qapi: Simplify error cleanup in test-qmp-*

In reply to Eric Blake

[As part of the fixup, replace the old commit message with this]

We have several tests that perform multiple sub-actions that are expected to fail. Asserting that an error occurred, then clearing it up to prepare for the next action, turned into enough boilerplate that it was sometimes forgotten (for example, a number of tests added to test-qmp-input-visitor.c in d88f5fd leaked err). Worse, if an error is not reset to NULL, we risk invalidating later use of that error (passing a non-NULL err into a function is generally a bad idea). Encapsulate the boilerplate into a single helper function error_free_or_abort(), and consistently use it.

The new function is added into error.c for use everywhere, although it is anticipated that testsuites will be the main client.

---
Drop test-qmp-common.h, and move to error.h instead

---
 include/qapi/error.h           |  9 +++++++++
 tests/test-qmp-commands.c      |  5 ++---
 tests/test-qmp-common.h        | 22 ----------------------
 tests/test-qmp-event.c         |  1 -
 tests/test-qmp-input-strict.c  |  1 -
 tests/test-qmp-input-visitor.c |  1 -
 util/error.c                   |  7 +++++++
 7 files changed, 18 insertions(+), 28 deletions(-)
 delete mode 100644 tests/test-qmp-common.h
include/qapi/error.h +9 -0
index c69dddb..faad466 100644
@@ -30,6 +30,10 @@
  * Handle an error without reporting it (just for completeness):
  *     error_free(err);
  *
+ * Assert than an expected error occurred, but clean it up without
+ * reporting it (primarily useful in testsuites):
+ *     error_free_or_abort(&err);
+ *
  * Pass an existing error to the caller:
  *     error_propagate(errp, err);
  * where Error **errp is a parameter, by convention the last one.
@@ -190,6 +194,11 @@ Error *error_copy(const Error *err);
 void error_free(Error *err);
 
 /*
+ * Convenience function to assert that *@errp is set, then silently free it.
+ */
+void error_free_or_abort(Error **errp);
+
+/*
  * Convenience function to error_report() and free @err.
  */
 void error_report_err(Error *);
tests/test-qmp-commands.c +2 -3
index 9f65fc2..888fb5f 100644
@@ -1,13 +1,12 @@
 #include <glib.h>
 #include "qemu-common.h"
-#include "test-qmp-common.h"
 #include "qapi/qmp/types.h"
 #include "test-qmp-commands.h"
 #include "qapi/qmp/dispatch.h"
 #include "qemu/module.h"
 #include "qapi/qmp-input-visitor.h"
-#include "test-qapi-types.h"
-#include "test-qapi-visit.h"
+#include "tests/test-qapi-types.h"
+#include "tests/test-qapi-visit.h"
 
 void qmp_user_def_cmd(Error **errp)
 {
tests/test-qmp-common.h deleted +0 -22
deleted file mode 100644
index 043f49c..0000000
@@ -1,22 +0,0 @@
-/*
- * Code common to qmp/qapi unit-tests.
- *
- * Copyright (C) 2015 Red Hat, Inc.
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef TEST_QMP_COMMON_H__
-#define TEST_QMP_COMMON_H__
-
-/* Expect an error, abort() if there is none. */
-static inline void error_free_or_abort(Error **errp)
-{
-    g_assert(*errp);
-    error_free(*errp);
-    *errp = NULL;
-}
-
-#endif
tests/test-qmp-event.c +0 -1
index c0ac3ea..035c65c 100644
@@ -22,7 +22,6 @@
 #include "qapi/qmp/qint.h"
 #include "qapi/qmp/qobject.h"
 #include "qapi/qmp-event.h"
-#include "test-qmp-common.h"
 
 typedef struct TestEventData {
     QDict *expect;
tests/test-qmp-input-strict.c +0 -1
index 7e15b09..201221c 100644
@@ -15,7 +15,6 @@
 #include <stdarg.h>
 
 #include "qemu-common.h"
-#include "test-qmp-common.h"
 #include "qapi/qmp-input-visitor.h"
 #include "test-qapi-types.h"
 #include "test-qapi-visit.h"
tests/test-qmp-input-visitor.c +0 -1
index 2b59b6f..ff4e7c9 100644
@@ -14,7 +14,6 @@
 #include <stdarg.h>
 
 #include "qemu-common.h"
-#include "test-qmp-common.h"
 #include "qapi/qmp-input-visitor.h"
 #include "test-qapi-types.h"
 #include "test-qapi-visit.h"
util/error.c +7 -0
index 8b86490..80c89a2 100644
@@ -220,6 +220,13 @@ void error_free(Error *err)
     }
 }
 
+void error_free_or_abort(Error **errp)
+{
+    assert(errp && *errp);
+    error_free(*errp);
+    *errp = NULL;
+}
+
 void error_propagate(Error **dst_errp, Error *local_err)
 {
     if (!local_err) {
Signature · 1 line
2.4.3
author
#19 lore raw

PATCH v10 8/30 qapi: More tests of alternate output

In reply to Eric Blake

The testsuite was only covering that we could output the 'int' branch of an alternate (no additional allocation/cleanup required). Add a test of the 'str' branch, to make sure that things still work even when a branch involves allocation.

Update to modern style of g_new0() over g_malloc0() while touching it.

---
v10: improve commit message
v9: new patch, split off of 10/17

---
 tests/test-qmp-output-visitor.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
tests/test-qmp-output-visitor.c +15 -1
index 0164984..0d0c859 100644
@@ -425,8 +425,9 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,
                                        const void *unused)
 {
     QObject *arg;
+    UserDefAlternate *tmp;
 
-    UserDefAlternate *tmp = g_malloc0(sizeof(UserDefAlternate));
+    tmp = g_new0(UserDefAlternate, 1);
     tmp->type = USER_DEF_ALTERNATE_KIND_I;
     tmp->u.i = 42;
 
@@ -438,6 +439,19 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,
 
     qapi_free_UserDefAlternate(tmp);
     qobject_decref(arg);
+
+    tmp = g_new0(UserDefAlternate, 1);
+    tmp->type = USER_DEF_ALTERNATE_KIND_S;
+    tmp->u.s = g_strdup("hello");
+
+    visit_type_UserDefAlternate(data->ov, &tmp, NULL, &error_abort);
+    arg = qmp_output_get_qobject(data->qov);
+
+    g_assert(qobject_type(arg) == QTYPE_QSTRING);
+    g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello");
+
+    qapi_free_UserDefAlternate(tmp);
+    qobject_decref(arg);
 }
 
 static void test_visitor_out_empty(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
author
#20 lore raw

PATCH v10 9/30 qapi: Test failure in middle of array parse

In reply to Eric Blake

Our generated list visitors have the same problem as has been mentioned elsewhere (see commit 2f52e20): they allocate data even on failure. An upcoming patch will correct things to provide saner guarantees, but first we need to expose the behavior in the testsuite to ensure we aren't introducing any memory usage bugs.

There are more test cases throughout the test-qmp-input-* tests that already deal with partial allocation; a later commit will clean up all visit_type_FOO(), without marking all of the tests with FIXME at this time.

---
v10: rebase to earlier changes, more commit message text
v9: move earlier in series (was 15/17)
v8: no change
v7: no change
v6: rebase onto earlier gen_err_check() and testsuite improvements

---
 scripts/qapi-visit.py          |  4 ++++
 tests/test-qmp-input-visitor.c | 10 +++++++++-
 2 files changed, 13 insertions(+), 1 deletion(-)
scripts/qapi-visit.py +4 -0
index f40c3c7..3ef5c16 100644
@@ -138,6 +138,10 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
 
 
 def gen_visit_list(name, element_type):
+    # FIXME: if *obj is NULL on entry, and the first visit_next_list()
+    # assigns to *obj, while a later one fails, we should clean up *obj
+    # rather than leaving it non-NULL. As currently written, the caller must
+    # call qapi_free_FOOList() to avoid a memory leak of the partial FOOList.
     return mcgen('''
 
 void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error **errp)
tests/test-qmp-input-visitor.c +9 -1
index 2b59b6f..b9b0c31 100644
@@ -699,8 +699,10 @@ static void test_visitor_in_errors(TestInputVisitorData *data,
     TestStruct *p = NULL;
     Error *err = NULL;
     Visitor *v;
+    strList *q = NULL;
 
-    v = visitor_input_test_init(data, "{ 'integer': false, 'boolean': 'foo', 'string': -42 }");
+    v = visitor_input_test_init(data, "{ 'integer': false, 'boolean': 'foo', "
+                                "'string': -42 }");
 
     visit_type_TestStruct(v, &p, NULL, &err);
     error_free_or_abort(&err);
@@ -710,6 +712,12 @@ static void test_visitor_in_errors(TestInputVisitorData *data,
 
     g_free(p->string);
     g_free(p);
+
+    v = visitor_input_test_init(data, "[ '1', '2', false, '3' ]");
+    visit_type_strList(v, &q, NULL, &err);
+    error_free_or_abort(&err);
+    assert(q);
+    qapi_free_strList(q);
 }
 
 int main(int argc, char **argv)
Signature · 1 line
2.4.3
author
#21 lore raw

PATCH v10 10/30 qapi: More tests of input arrays

In reply to Eric Blake

Our testsuite had no coverage of empty arrays, nor of what happens when the input does not match the expected type. Useful to have, especially if we start changing the visitor contracts.

I did not think it worth duplicating these additions to test-qmp-input-strict; since all strict mode does is add the ability to reject JSON input that has more keys than what the visitor expects, yet the additions in this patch error out earlier than that point regardless of whether strict mode was requested.

---
v10: rebase to earlier changes, more in commit message
v9: move earlier in series (was 16/17)
v8: no change
v7: no change
v6: new patch

---
 tests/test-qmp-input-visitor.c | 52 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
tests/test-qmp-input-visitor.c +52 -0
index b9b0c31..6a94683 100644
@@ -231,6 +231,12 @@ static void test_visitor_in_list(TestInputVisitorData *data,
     }
 
     qapi_free_UserDefOneList(head);
+    head = NULL;
+
+    /* An empty list is valid */
+    v = visitor_input_test_init(data, "[]");
+    visit_type_UserDefOneList(v, &head, NULL, &error_abort);
+    g_assert(!head);
 }
 
 static void test_visitor_in_any(TestInputVisitorData *data,
@@ -720,6 +726,50 @@ static void test_visitor_in_errors(TestInputVisitorData *data,
     qapi_free_strList(q);
 }
 
+static void test_visitor_in_wrong_type(TestInputVisitorData *data,
+                                       const void *unused)
+{
+    TestStruct *p = NULL;
+    Visitor *v;
+    strList *q = NULL;
+    int64_t i;
+    Error *err = NULL;
+
+    /* Make sure arrays and structs cannot be confused */
+
+    v = visitor_input_test_init(data, "[]");
+    visit_type_TestStruct(v, &p, NULL, &err);
+    error_free_or_abort(&err);
+    g_assert(!p);
+
+    v = visitor_input_test_init(data, "{}");
+    visit_type_strList(v, &q, NULL, &err);
+    error_free_or_abort(&err);
+    assert(!q);
+
+    /* Make sure primitives and struct cannot be confused */
+
+    v = visitor_input_test_init(data, "1");
+    visit_type_TestStruct(v, &p, NULL, &err);
+    error_free_or_abort(&err);
+    g_assert(!p);
+
+    v = visitor_input_test_init(data, "{}");
+    visit_type_int(v, &i, NULL, &err);
+    error_free_or_abort(&err);
+
+    /* Make sure primitives and arrays cannot be confused */
+
+    v = visitor_input_test_init(data, "1");
+    visit_type_strList(v, &q, NULL, &err);
+    error_free_or_abort(&err);
+    assert(!q);
+
+    v = visitor_input_test_init(data, "[]");
+    visit_type_int(v, &i, NULL, &err);
+    error_free_or_abort(&err);
+}
+
 int main(int argc, char **argv)
 {
     TestInputVisitorData in_visitor_data;
@@ -752,6 +802,8 @@ int main(int argc, char **argv)
                            &in_visitor_data, test_visitor_in_alternate);
     input_visitor_test_add("/visitor/input/errors",
                            &in_visitor_data, test_visitor_in_errors);
+    input_visitor_test_add("/visitor/input/wrong-type",
+                           &in_visitor_data, test_visitor_in_wrong_type);
     input_visitor_test_add("/visitor/input/alternate-number",
                            &in_visitor_data, test_visitor_in_alternate_number);
     input_visitor_test_add("/visitor/input/native_list/int",
Signature · 1 line
2.4.3
author
#22 lore raw

PATCH v10 11/30 qapi: Provide nicer array names in introspection

In reply to Eric Blake

For the sake of humans reading introspection output, it is nice to have the name of implicit array types be recognizable as arrays of the underlying type. However, while this patch allows humans to skip from a command with return type "[123]" straight to the definition of type "123" without having to first inspect type "[123]", document that this shortcut should not be taken by client apps.

This makes the resulting introspection string slightly larger by default (just over 200 bytes), but it's in the noise (less than 0.3% of the overall 70k size of 'query-qmp-capabilities').

---
v10: no change
v9: no change
v8: commit message stats on the size increase
v7: no change
v6: no change

---
 docs/qapi-code-gen.txt     | 7 +++++--
 scripts/qapi-introspect.py | 8 +++++---
 2 files changed, 10 insertions(+), 5 deletions(-)
docs/qapi-code-gen.txt +5 -2
index 4d8c2fc..ba29bc6 100644
@@ -662,11 +662,14 @@ Example: the SchemaInfo for BlockRef from section Alternate types
 
 The SchemaInfo for an array type has meta-type "array", and variant
 member "element-type", which names the array's element type.  Array
-types are implicitly defined.
+types are implicitly defined.  For convenience, the array's name may
+resemble the element type; however, clients should examine member
+"element-type" instead of making assumptions based on parsing member
+"name".
 
 Example: the SchemaInfo for ['str']
 
-    { "name": "strList", "meta-type": "array",
+    { "name": "[str]", "meta-type": "array",
       "element-type": "str" }
 
 The SchemaInfo for an enumeration type has meta-type "enum" and
scripts/qapi-introspect.py +5 -3
index c0dad66..64f2cd0 100644
@@ -107,10 +107,12 @@ const char %(c_name)s[] = %(c_string)s;
         # characters.
         if isinstance(typ, QAPISchemaBuiltinType):
             return typ.name
+        if isinstance(typ, QAPISchemaArrayType):
+            return '[' + self._use_type(typ.element_type) + ']'
         return self._name(typ.name)
 
     def _gen_json(self, name, mtype, obj):
-        if mtype != 'command' and mtype != 'event' and mtype != 'builtin':
+        if mtype not in ('command', 'event', 'builtin', 'array'):
             name = self._name(name)
         obj['name'] = name
         obj['meta-type'] = mtype
@@ -136,8 +138,8 @@ const char %(c_name)s[] = %(c_string)s;
         self._gen_json(name, 'enum', {'values': values})
 
     def visit_array_type(self, name, info, element_type):
-        self._gen_json(name, 'array',
-                       {'element-type': self._use_type(element_type)})
+        element = self._use_type(element_type)
+        self._gen_json('[' + element + ']', 'array', {'element-type': element})
 
     def visit_object_type_flat(self, name, info, members, variants):
         obj = {'members': [self._gen_member(m) for m in members]}
Signature · 1 line
2.4.3
author
#23 lore raw

PATCH v10 12/30 qapi-introspect: Document lack of sorting

In reply to Eric Blake

qapi-code-gen.txt already claims that types, commands, and events share a common namespace; set this in stone by further documenting that our introspection output will never have collisions with the same name tied to more than one meta-type.

Our largest QMP enum currently has 125 values, our largest object type has 27 members, and the mean for each is less than
10. These sizes are small enough that the per-element overhead of O(log n) binary searching probably outweighs the speed possible with direct O(n) linear searching (a better algorithm with more overhead will only beat a leaner naive algorithm only as you scale to larger input sizes).

Arguably, the overall SchemaInfo array could be sorted by name; there, we currently have 531 entities, large enough for a binary search to be faster than linear. However, remember that we have mutually-recursive types, which means there is no topological ordering that will allow clients to learn all information about that type in a single linear pass; thus clients will want to do random access over the data, and they will probably read the introspection output into a hashtable for O(1) lookup rather than O(log n) binary searching, at which point, pre-sorting our introspection output doesn't help the client.

It doesn't help that sorting can be subjective if you introduce locales into the mix (I'm not experienced enough with Python to know for sure, but at least it looks like it defaults to sorting in the C locale even when run under a different locale). And while our current introspection output is deterministic (because we visit entities in a sorted order), we may want to change that order in the future (such as using OrderedDict to stick to .json declaration order).

For these reasons, we simply document that clients should not rely on any particular order of items in introspection output. And since it is now a documented part of the contract, we have the freedom to later rearrange output if needed, without worrying about breaking well-written clients that were already aware that they had to do linear searches.

---
v10: wording touchups based on review
v9: retitle; and swing in the opposite direction: give the user no guarantee about order
v8: no change
v7: tweak commit wording
v6: no change from v5

---
 docs/qapi-code-gen.txt | 19 +++++++++++++++----
 qapi/introspect.json   | 17 ++++++++++++-----
 2 files changed, 27 insertions(+), 9 deletions(-)
docs/qapi-code-gen.txt +15 -4
index ba29bc6..f9fa6f3 100644
@@ -516,6 +516,10 @@ query-qmp-schema.  QGA currently doesn't support introspection.
 
 query-qmp-schema returns a JSON array of SchemaInfo objects.  These
 objects together describe the wire ABI, as defined in the QAPI schema.
+There is no specified order to the SchemaInfo objects returned; a
+client must search for a particular name throughout the entire array
+to learn more about that name, but is at least guaranteed that there
+will be no collisions between type, command, and event names.
 
 However, the SchemaInfo can't reflect all the rules and restrictions
 that apply to QMP.  It's interface introspection (figuring out what's
@@ -596,7 +600,9 @@ any.  Each element is a JSON object with members "name" (the member's
 name), "type" (the name of its type), and optionally "default".  The
 member is optional if "default" is present.  Currently, "default" can
 only have value null.  Other values are reserved for future
-extensions.
+extensions.  The "members" array is in no particular order; clients
+must search the entire object when learning whether a particular
+member is supported.
 
 Example: the SchemaInfo for MyType from section Struct types
 
@@ -610,7 +616,9 @@ Example: the SchemaInfo for MyType from section Struct types
 "variants" is a JSON array describing the object's variant members.
 Each element is a JSON object with members "case" (the value of type
 tag this element applies to) and "type" (the name of an object type
-that provides the variant members for this type tag value).
+that provides the variant members for this type tag value).  The
+"variants" array is in no particular order, and is not guaranteed to
+list cases in the same order as the corresponding "tag" enum type.
 
 Example: the SchemaInfo for flat union BlockdevOptions from section
 Union types
@@ -651,7 +659,8 @@ Union types
 The SchemaInfo for an alternate type has meta-type "alternate", and
 variant member "members".  "members" is a JSON array.  Each element is
 a JSON object with member "type", which names a type.  Values of the
-alternate type conform to exactly one of its member types.
+alternate type conform to exactly one of its member types.  There is
+no guarantee on the order in which "members" will be listed.
 
 Example: the SchemaInfo for BlockRef from section Alternate types
 
@@ -673,7 +682,9 @@ Example: the SchemaInfo for ['str']
       "element-type": "str" }
 
 The SchemaInfo for an enumeration type has meta-type "enum" and
-variant member "values".
+variant member "values".  The values are listed in no particular
+order; clients must search the entire enum when learning whether a
+particular value is supported.
 
 Example: the SchemaInfo for MyEnum from section Enumeration types
 
qapi/introspect.json +12 -5
index cc50dc6..e7c4c3e 100644
@@ -25,6 +25,10 @@
 # Returns: array of @SchemaInfo, where each element describes an
 # entity in the ABI: command, event, type, ...
 #
+# The order of the various SchemaInfo is unspecified; however, all
+# names are guaranteed to be unique (no name will be duplicated with
+# different meta-types).
+#
 # Note: the QAPI schema is also used to help define *internal*
 # interfaces, by defining QAPI types.  These are not part of the QMP
 # wire ABI, and therefore not returned by this command.
@@ -78,7 +82,8 @@
 #        Commands and events have the name defined in the QAPI schema.
 #        Unlike command and event names, type names are not part of
 #        the wire ABI.  Consequently, type names are meaningless
-#        strings here.
+#        strings here, although they are still guaranteed unique
+#        regardless of @meta-type.
 #
 # All references to other SchemaInfo are by name.
 #
@@ -130,7 +135,7 @@
 #
 # Additional SchemaInfo members for meta-type 'enum'.
 #
-# @values: the enumeration type's values.
+# @values: the enumeration type's values, in no particular order.
 #
 # Values of this type are JSON string on the wire.
 #
@@ -158,14 +163,16 @@
 #
 # Additional SchemaInfo members for meta-type 'object'.
 #
-# @members: the object type's (non-variant) members.
+# @members: the object type's (non-variant) members, in no particular order.
 #
 # @tag: #optional the name of the member serving as type tag.
 #       An element of @members with this name must exist.
 #
 # @variants: #optional variant members, i.e. additional members that
 #            depend on the type tag's value.  Present exactly when
-#            @tag is present.
+#            @tag is present.  The variants are in no particular order,
+#            and may even differ from the order of the values of the
+#            enum type of the @tag.
 #
 # Values of this type are JSON object on the wire.
 #
@@ -219,7 +226,7 @@
 #
 # Additional SchemaInfo members for meta-type 'alternate'.
 #
-# @members: the alternate type's members.
+# @members: the alternate type's members, in no particular order.
 #           The members' wire encoding is distinct, see
 #           docs/qapi-code-gen.txt section Alternate types.
 #
Signature · 1 line
2.4.3
#24 lore raw

Re: [Qemu-devel] [PATCH v10 12/30] qapi-introspect: Document lack of sorting

Quoted 33 lines · Eric Blake

qapi-code-gen.txt already claims that types, commands, and events share a common namespace; set this in stone by further documenting that our introspection output will never have collisions with the same name tied to more than one meta-type.

Our largest QMP enum currently has 125 values, our largest object type has 27 members, and the mean for each is less than
10. These sizes are small enough that the per-element overhead of O(log n) binary searching probably outweighs the speed possible with direct O(n) linear searching (a better algorithm with more overhead will only beat a leaner naive algorithm only as you scale to larger input sizes).

Arguably, the overall SchemaInfo array could be sorted by name; there, we currently have 531 entities, large enough for a binary search to be faster than linear. However, remember that we have mutually-recursive types, which means there is no topological ordering that will allow clients to learn all information about that type in a single linear pass; thus clients will want to do random access over the data, and they will probably read the introspection output into a hashtable for O(1) lookup rather than O(log n) binary searching, at which point, pre-sorting our introspection output doesn't help the client.

It doesn't help that sorting can be subjective if you introduce locales into the mix (I'm not experienced enough with Python to know for sure, but at least it looks like it defaults to sorting in the C locale even when run under a different locale). And while our current introspection output is deterministic (because we visit entities in a sorted order), we may want to change that order in the future (such as using OrderedDict to stick to .json declaration order).

Quoted 6 lines

For these reasons, we simply document that clients should not rely on any particular order of items in introspection output. And since it is now a documented part of the contract, we have the freedom to later rearrange output if needed, without worrying about breaking well-written clients that were already aware that they had to do linear searches.

Well, any kind of search that doesn't rely on sorting. Suggest to drop the "that were already..." clause. Happy to do that on commit.

author
#25 lore raw

Re: [Qemu-devel] [PATCH v10 12/30] qapi-introspect: Document lack of sorting

Quoted 15 lines · Markus Armbruster
Quoted 6 lines · Eric Blake

qapi-code-gen.txt already claims that types, commands, and events share a common namespace; set this in stone by further documenting that our introspection output will never have collisions with the same name tied to more than one meta-type.

Quoted 6 lines

For these reasons, we simply document that clients should not rely on any particular order of items in introspection output. And since it is now a documented part of the contract, we have the freedom to later rearrange output if needed, without worrying about breaking well-written clients that were already aware that they had to do linear searches.

Quoted 2 lines

Well, any kind of search that doesn't rely on sorting. Suggest to drop the "that were already..." clause. Happy to do that on commit.

I don't see that phrase dropped in your v1 pull request. Worth fixing?

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#26 lore raw

Re: [Qemu-devel] [PATCH v10 12/30] qapi-introspect: Document lack of sorting

Quoted 16 lines · Eric Blake
Quoted 15 lines · Markus Armbruster
Quoted 12 lines · Eric Blake

qapi-code-gen.txt already claims that types, commands, and events share a common namespace; set this in stone by further documenting that our introspection output will never have collisions with the same name tied to more than one meta-type.

For these reasons, we simply document that clients should not rely on any particular order of items in introspection output. And since it is now a documented part of the contract, we have the freedom to later rearrange output if needed, without worrying about breaking well-written clients that were already aware that they had to do linear searches.

Quoted 4 lines
Quoted 2 lines

Well, any kind of search that doesn't rely on sorting. Suggest to drop the "that were already..." clause. Happy to do that on commit.

I don't see that phrase dropped in your v1 pull request. Worth fixing?

Fixed in PULL v2, thanks!

author
#27 lore raw

PATCH v10 13/30 qapi: Track simple union tag in object.local_members

In reply to Eric Blake

We were previously creating all unions with an empty list for local_members. However, it will make it easier to unify struct and union generation if we include the generated tag member in local_members. That way, we can have a common code pattern:
visit the base (if any), visit the local members (if any), visit the variants (if any). The local_members of a flat union remains empty (because the discriminator is already visited as part of the base). Then, by visiting tag_member.check() during AlternateType.check(), we no longer need to call it during Variants.check().

The various front end entities now exist as follows:
struct: optional base, optional local_members, no variants simple union: no base, one-element local_members, variants with tag_member
from local_members
flat union: base, no local_members, variants with tag_member from base
alternate: no base, no local_members, variants

With the new local members, we require a bit of finesse to avoid assertions in the clients.

No change to generated code.

---
v10: no change
v9: improve commit message, add comments, tweak how alternates check tag_member
v8: new patch

---
 scripts/qapi-types.py                  |  5 ++++-
 scripts/qapi-visit.py                  |  5 ++++-
 scripts/qapi.py                        | 16 +++++++++++-----
 tests/qapi-schema/qapi-schema-test.out |  2 ++
 tests/qapi-schema/union-clash-data.out |  1 +
 tests/qapi-schema/union-empty.out      |  1 +
 6 files changed, 23 insertions(+), 7 deletions(-)
scripts/qapi-types.py +4 -1
index b37900f..946afab 100644
@@ -269,7 +269,10 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def visit_object_type(self, name, info, base, members, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
         if variants:
-            assert not members      # not implemented
+            if members:
+                # Members other than variants.tag_member not implemented
+                assert len(members) == 1
+                assert members[0] == variants.tag_member
             self.decl += gen_union(name, base, variants)
         else:
             self.decl += gen_struct(name, base, members)
scripts/qapi-visit.py +4 -1
index 3ef5c16..94cd113 100644
@@ -364,7 +364,10 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
     def visit_object_type(self, name, info, base, members, variants):
         self.decl += gen_visit_decl(name)
         if variants:
-            assert not members      # not implemented
+            if members:
+                # Members other than variants.tag_member not implemented
+                assert len(members) == 1
+                assert members[0] == variants.tag_member
             self.defn += gen_visit_union(name, base, variants)
         else:
             self.defn += gen_visit_struct(name, base, members)
scripts/qapi.py +11 -5
index 7c50cc4..a814e20 100644
@@ -957,6 +957,9 @@ class QAPISchemaArrayType(QAPISchemaType):
 
 class QAPISchemaObjectType(QAPISchemaType):
     def __init__(self, name, info, base, local_members, variants):
+        # struct has local_members, optional base, and no variants
+        # flat union has base, variants, and no local_members
+        # simple union has local_members, variants, and no base
         QAPISchemaType.__init__(self, name, info)
         assert base is None or isinstance(base, str)
         for m in local_members:
@@ -1048,10 +1051,11 @@ class QAPISchemaObjectTypeVariants(object):
         self.variants = variants
 
     def check(self, schema, members, seen):
-        if self.tag_name:
+        # seen is non-empty for unions, empty for alternates
+        if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
-        else:
-            self.tag_member.check(schema, members, seen)
+        if seen:
+            assert self.tag_member in seen.itervalues()
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
             vseen = dict(seen)
@@ -1085,6 +1089,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
 
     def check(self, schema):
+        self.variants.tag_member.check(schema, [], {})
         self.variants.check(schema, [], {})
 
     def json_type(self):
@@ -1270,13 +1275,14 @@ class QAPISchema(object):
         if tag_name:
             variants = [self._make_variant(key, value)
                         for (key, value) in data.iteritems()]
+            members = []
         else:
             variants = [self._make_simple_variant(key, value, info)
                         for (key, value) in data.iteritems()]
             tag_member = self._make_implicit_tag(name, info, variants)
+            members = [tag_member]
         self._def_entity(
-            QAPISchemaObjectType(name, info, base,
-                                 self._make_members(OrderedDict(), info),
+            QAPISchemaObjectType(name, info, base, members,
                                  QAPISchemaObjectTypeVariants(tag_name,
                                                               tag_member,
                                                               variants)))
tests/qapi-schema/qapi-schema-test.out +2 -0
index e20a823..786024e 100644
@@ -132,6 +132,7 @@ object UserDefFlatUnion2
     case value2: UserDefB
     case value3: UserDefA
 object UserDefNativeListUnion
+    member type: UserDefNativeListUnionKind optional=False
     case integer: :obj-intList-wrapper
     case s8: :obj-int8List-wrapper
     case s16: :obj-int16List-wrapper
@@ -187,6 +188,7 @@ object __org.qemu_x-Struct
 object __org.qemu_x-Struct2
     member array: __org.qemu_x-Union1List optional=False
 object __org.qemu_x-Union1
+    member type: __org.qemu_x-Union1Kind optional=False
     case __org.qemu_x-branch: :obj-str-wrapper
 enum __org.qemu_x-Union1Kind ['__org.qemu_x-branch']
 object __org.qemu_x-Union2
tests/qapi-schema/union-clash-data.out +1 -0
index 6277239..cea8551 100644
@@ -2,5 +2,6 @@ object :empty
 object :obj-int-wrapper
     member data: int optional=False
 object TestUnion
+    member type: TestUnionKind optional=False
     case data: :obj-int-wrapper
 enum TestUnionKind ['data']
tests/qapi-schema/union-empty.out +1 -0
index 8b5a7bf..9c89fd1 100644
@@ -1,3 +1,4 @@
 object :empty
 object Union
+    member type: UnionKind optional=False
 enum UnionKind []
Signature · 1 line
2.4.3
author
#28 lore raw

PATCH v10 14/30 qapi-types: Consolidate gen_struct() and gen_union()

In reply to Eric Blake

These two methods are now close enough that we can finally merge them, relying on the fact that simple unions now provide a reasonable local_members. Change gen_struct() to gen_object() that handles all forms of QAPISchemaObjectType, and rename and shrink gen_union() to gen_variants() to handle the portion of gen_object() needed when variants are present.

gen_struct_fields() now has a single caller, so it no longer needs an optional parameter; however, I did not choose to inline it into the caller.

No difference to generated code.

---
v10: no change
v9: rebase to earlier changes
v8: new patch

---
 scripts/qapi-types.py | 37 +++++++++++--------------------------
 1 file changed, 11 insertions(+), 26 deletions(-)
scripts/qapi-types.py +11 -26
index 946afab..403768b 100644
@@ -51,7 +51,7 @@ def gen_struct_field(member):
     return ret
 
 
-def gen_struct_fields(local_members, base=None):
+def gen_struct_fields(local_members, base):
     ret = ''
 
     if base:
@@ -70,7 +70,7 @@ def gen_struct_fields(local_members, base=None):
     return ret
 
 
-def gen_struct(name, base, members):
+def gen_object(name, base, members, variants):
     ret = mcgen('''
 
 struct %(c_name)s {
@@ -79,11 +79,14 @@ struct %(c_name)s {
 
     ret += gen_struct_fields(members, base)
 
+    if variants:
+        ret += gen_variants(variants)
+
     # Make sure that all structs have at least one field; this avoids
     # potential issues with attempting to malloc space for zero-length
     # structs in C, and also incompatibility with C++ (where an empty
     # struct is size 1).
-    if not (base and base.members) and not members:
+    if not (base and base.members) and not members and not variants:
         ret += mcgen('''
     char qapi_dummy_field_for_empty_struct;
 ''')
@@ -140,17 +143,7 @@ const int %(c_name)s_qtypes[QTYPE_MAX] = {
     return ret
 
 
-def gen_union(name, base, variants):
-    ret = mcgen('''
-
-struct %(c_name)s {
-''',
-                c_name=c_name(name))
-    if base:
-        ret += gen_struct_fields([], base)
-    else:
-        ret += gen_struct_field(variants.tag_member)
-
+def gen_variants(variants):
     # FIXME: What purpose does data serve, besides preventing a union that
     # has a branch named 'data'? We use it in qapi-visit.py to decide
     # whether to bypass the switch statement if visiting the discriminator
@@ -159,11 +152,11 @@ struct %(c_name)s {
     # should not be any data leaks even without a data pointer.  Or, if
     # 'data' is merely added to guarantee we don't have an empty union,
     # shouldn't we enforce that at .json parse time?
-    ret += mcgen('''
+    ret = mcgen('''
     union { /* union tag is @%(c_name)s */
         void *data;
 ''',
-                 c_name=c_name(variants.tag_member.name))
+                c_name=c_name(variants.tag_member.name))
 
     for var in variants.variants:
         # Ugly special case for simple union TODO get rid of it
@@ -176,7 +169,6 @@ struct %(c_name)s {
 
     ret += mcgen('''
     } u;
-};
 ''')
 
     return ret
@@ -268,14 +260,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
     def visit_object_type(self, name, info, base, members, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
-        if variants:
-            if members:
-                # Members other than variants.tag_member not implemented
-                assert len(members) == 1
-                assert members[0] == variants.tag_member
-            self.decl += gen_union(name, base, variants)
-        else:
-            self.decl += gen_struct(name, base, members)
+        self.decl += gen_object(name, base, members, variants)
         if base:
             self.decl += gen_upcast(name, base)
         self._gen_type_cleanup(name)
@@ -283,7 +268,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def visit_alternate_type(self, name, info, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
         self._fwdefn += gen_alternate_qtypes(name, variants)
-        self.decl += gen_union(name, None, variants)
+        self.decl += gen_object(name, None, [variants.tag_member], variants)
         self.decl += gen_alternate_qtypes_decl(name)
         self._gen_type_cleanup(name)
 
Signature · 1 line
2.4.3
author
#29 lore raw

PATCH v10 15/30 qapi-types: Simplify gen_struct_field[s]

In reply to Eric Blake

Simplify gen_struct_fields() back to a single iteration over a list of fields (like it was prior to commit f87ab7f9), by moving the generated comments to gen_object(). Then, inline gen_struct_field() into its only caller.

---
v10: no change
v9: new patch

---
 scripts/qapi-types.py | 40 +++++++++++++++-------------------------
 1 file changed, 15 insertions(+), 25 deletions(-)
scripts/qapi-types.py +15 -25
index 403768b..2f2f7df 100644
@@ -36,48 +36,38 @@ struct %(c_name)s {
                  c_name=c_name(name), c_type=element_type.c_type())
 
 
-def gen_struct_field(member):
+def gen_struct_fields(members):
     ret = ''
-
-    if member.optional:
-        ret += mcgen('''
+    for memb in members:
+        if memb.optional:
+            ret += mcgen('''
     bool has_%(c_name)s;
 ''',
-                     c_name=c_name(member.name))
-    ret += mcgen('''
+                         c_name=c_name(memb.name))
+        ret += mcgen('''
     %(c_type)s %(c_name)s;
 ''',
-                 c_type=member.type.c_type(), c_name=c_name(member.name))
+                     c_type=memb.type.c_type(), c_name=c_name(memb.name))
     return ret
 
 
-def gen_struct_fields(local_members, base):
-    ret = ''
+def gen_object(name, base, members, variants):
+    ret = mcgen('''
+
+struct %(c_name)s {
+''',
+                c_name=c_name(name))
 
     if base:
         ret += mcgen('''
     /* Members inherited from %(c_name)s: */
 ''',
                      c_name=base.c_name())
-        for memb in base.members:
-            ret += gen_struct_field(memb)
+        ret += gen_struct_fields(base.members)
         ret += mcgen('''
     /* Own members: */
 ''')
-
-    for memb in local_members:
-        ret += gen_struct_field(memb)
-    return ret
-
-
-def gen_object(name, base, members, variants):
-    ret = mcgen('''
-
-struct %(c_name)s {
-''',
-                c_name=c_name(name))
-
-    ret += gen_struct_fields(members, base)
+    ret += gen_struct_fields(members)
 
     if variants:
         ret += gen_variants(variants)
Signature · 1 line
2.4.3
author
#30 lore raw

PATCH v10 16/30 qapi: Drop obsolete tag value collision assertions

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

Union tag values can't clash with member names in generated C anymore since commit e4ba22b, but QAPISchemaObjectTypeVariants.check() still asserts they don't. Drop it.

Message-Id: <[email protected]>

---
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi.py +1 -2
index a814e20..1ac870d 100644
@@ -1058,8 +1058,7 @@ class QAPISchemaObjectTypeVariants(object):
             assert self.tag_member in seen.itervalues()
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
-            vseen = dict(seen)
-            v.check(schema, self.tag_member.type, vseen)
+            v.check(schema, self.tag_member.type, {})
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Signature · 1 line
2.4.3
author
#31 lore raw

PATCH v10 17/30 qapi: Simplify QAPISchemaObjectTypeMember.check()

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

QAPISchemaObjectTypeMember.check() currently does four things:

1. Compute self.type

2. Accumulate members in all_members

Only one caller cares: QAPISchemaObjectType.check() uses it to compute self.members. The other callers pass a throw-away accumulator.

3. Accumulate a map from names to members in seen

Only one caller cares: QAPISchemaObjectType.check() uses it to compute its local variable seen, for self.variants.check(), which uses it to compute self.variants.tag_member from self.variants.tag_name. The other callers pass a throw-away accumulator.

4. Check for collisions

This piggyback on 3: before adding a new entry, we assert it's new.

Only one caller cares: QAPISchemaObjectType.check() uses it to assert non-variant members don't clash.

Simplify QAPISchemaObjectType.check(): move 2.-4. to QAPISchemaObjectType.check(), and drop parameters all_members and seen.

Message-Id: <[email protected]> [rebase to earlier changes that moved tag_member.check() of alternate types]

---
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
scripts/qapi.py +7 -7
index 1ac870d..a4350b2 100644
@@ -990,7 +990,10 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert c_name(m.name) not in seen
             seen[m.name] = m
         for m in self.local_members:
-            m.check(schema, members, seen)
+            m.check(schema)
+            assert m.name not in seen
+            seen[m.name] = m
+            members.append(m)
         if self.variants:
             self.variants.check(schema, members, seen)
         self.members = members
@@ -1027,12 +1030,9 @@ class QAPISchemaObjectTypeMember(object):
         self.type = None
         self.optional = optional
 
-    def check(self, schema, all_members, seen):
-        assert self.name not in seen
+    def check(self, schema):
         self.type = schema.lookup_type(self._type_name)
         assert self.type
-        all_members.append(self)
-        seen[self.name] = self
 
 
 class QAPISchemaObjectTypeVariants(object):
@@ -1066,7 +1066,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
     def check(self, schema, tag_type, seen):
-        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        QAPISchemaObjectTypeMember.check(self, schema)
         assert self.name in tag_type.values
 
     # This function exists to support ugly simple union special cases
@@ -1088,7 +1088,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
 
     def check(self, schema):
-        self.variants.tag_member.check(schema, [], {})
+        self.variants.tag_member.check(schema)
         self.variants.check(schema, [], {})
 
     def json_type(self):
Signature · 1 line
2.4.3
#32 lore raw

Re: [Qemu-devel] [PATCH v10 17/30] qapi: Simplify QAPISchemaObjectTypeMember.check()

Quoted 20 lines · Eric Blake

From: Markus Armbruster <[email protected]>

QAPISchemaObjectTypeMember.check() currently does four things:

1. Compute self.type

2. Accumulate members in all_members

Only one caller cares: QAPISchemaObjectType.check() uses it to compute self.members. The other callers pass a throw-away accumulator.

3. Accumulate a map from names to members in seen

Only one caller cares: QAPISchemaObjectType.check() uses it to compute its local variable seen, for self.variants.check(), which uses it to compute self.variants.tag_member from self.variants.tag_name. The other callers pass a throw-away accumulator.

Quoted 3 lines

4. Check for collisions

This piggyback on 3: before adding a new entry, we assert it's new.

piggybacks (typo is mine)

Quoted 8 lines

Only one caller cares: QAPISchemaObjectType.check() uses it to assert non-variant members don't clash.

Simplify QAPISchemaObjectType.check(): move 2.-4. to QAPISchemaObjectType.check(), and drop parameters all_members and seen.

Quoted 5 lines

Message-Id: <[email protected]> [rebase to earlier changes that moved tag_member.check() of alternate types]

Patch is as good as ever ;)

author
#33 lore raw

Re: [Qemu-devel] [PATCH v10 17/30] qapi: Simplify QAPISchemaObjectTypeMember.check()

Quoted 4 lines · Markus Armbruster
Quoted 2 lines · Eric Blake

From: Markus Armbruster <[email protected]>

Quoted 8 lines
Quoted 6 lines

QAPISchemaObjectTypeMember.check() currently does four things:

4. Check for collisions

This piggyback on 3: before adding a new entry, we assert it's new.

piggybacks (typo is mine)

What's more, I noticed it in v9, but in redoing v10 to go closer to your original series, I reverted my touchup. Oh well :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#34 lore raw

PATCH v10 18/30 qapi: Clean up after previous commit

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

QAPISchemaObjectTypeVariants.check() parameter members and QAPISchemaObjectTypeVariant.check() parameter seen are no longer used, drop them.

Message-Id: <[email protected]> [rebase to earlier changes that moved tag_member.check() of alternate types]

---
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
scripts/qapi.py +5 -6
index a4350b2..0bf8235 100644
@@ -995,7 +995,7 @@ class QAPISchemaObjectType(QAPISchemaType):
             seen[m.name] = m
             members.append(m)
         if self.variants:
-            self.variants.check(schema, members, seen)
+            self.variants.check(schema, seen)
         self.members = members
 
     def is_implicit(self):
@@ -1050,22 +1050,21 @@ class QAPISchemaObjectTypeVariants(object):
         self.tag_member = tag_member
         self.variants = variants
 
-    def check(self, schema, members, seen):
-        # seen is non-empty for unions, empty for alternates
+    def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
         if seen:
             assert self.tag_member in seen.itervalues()
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
-            v.check(schema, self.tag_member.type, {})
+            v.check(schema, self.tag_member.type)
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
-    def check(self, schema, tag_type, seen):
+    def check(self, schema, tag_type):
         QAPISchemaObjectTypeMember.check(self, schema)
         assert self.name in tag_type.values
 
@@ -1089,7 +1088,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
     def check(self, schema):
         self.variants.tag_member.check(schema)
-        self.variants.check(schema, [], {})
+        self.variants.check(schema, {})
 
     def json_type(self):
         return 'value'
Signature · 1 line
2.4.3
author
#35 lore raw

PATCH v10 19/30 qapi: Fix up commit 7618b91's clash sanity checking change

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

This hunk

@@ -964,6 +965,7 @@ class QAPISchemaObjectType(QAPISchemaType):
members = []
seen = {}
for m in members:
+ assert c_name(m.name) not in seen
seen[m.name] = m
for m in self.local_members:
m.check(schema, members, seen)

is plainly broken.

Asserting the members inherited from base don't clash is somewhat redundant, because self.base.check() just checked that. But it doesn't hurt.

The idea to use c_name(m.name) instead of m.name for collision checking is sound, because we need to catch clashes between the m.name and between the c_name(m.name), and when two m.name clash, then their c_name() also clash.

However, using c_name(m.name) instead of m.name in one of several places doesn't work. See the very next line.

Keep the assertion, but drop the c_name() for now. A future commit will bring it back.

Message-Id: <[email protected]>

[change TABs in commit message to space]

---
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi.py +1 -1
index 0bf8235..86d2adc 100644
@@ -987,7 +987,7 @@ class QAPISchemaObjectType(QAPISchemaType):
             members = []
         seen = {}
         for m in members:
-            assert c_name(m.name) not in seen
+            assert m.name not in seen
             seen[m.name] = m
         for m in self.local_members:
             m.check(schema)
Signature · 1 line
2.4.3
author
#36 lore raw

PATCH v10 20/30 qapi: Eliminate QAPISchemaObjectType.check() variable members

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

We can use seen.values() instead if we make it an OrderedDict.

Message-Id: <[email protected]>

---
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)
scripts/qapi.py +5 -9
index 86d2adc..44d08c1 100644
@@ -977,26 +977,22 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self.members:
             return
         self.members = False                    # mark as being checked
+        seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
             assert isinstance(self.base, QAPISchemaObjectType)
             assert not self.base.variants       # not implemented
             self.base.check(schema)
-            members = list(self.base.members)
-        else:
-            members = []
-        seen = {}
-        for m in members:
-            assert m.name not in seen
-            seen[m.name] = m
+            for m in self.base.members:
+                assert m.name not in seen
+                seen[m.name] = m
         for m in self.local_members:
             m.check(schema)
             assert m.name not in seen
             seen[m.name] = m
-            members.append(m)
         if self.variants:
             self.variants.check(schema, seen)
-        self.members = members
+        self.members = seen.values()
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
Signature · 1 line
2.4.3
author
#37 lore raw

PATCH v10 21/30 qapi: Factor out QAPISchemaObjectTypeMember.check_clash()

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

While there, stick in a TODO change key of seen from QAPI name to C name. Can't do it right away, because it would fail the assertion for tests/qapi-schema/args-has-clash.json.

Message-Id: <[email protected]>

---
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
scripts/qapi.py +7 -4
index 44d08c1..2a73b2b 100644
@@ -984,12 +984,10 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert not self.base.variants       # not implemented
             self.base.check(schema)
             for m in self.base.members:
-                assert m.name not in seen
-                seen[m.name] = m
+                m.check_clash(seen)
         for m in self.local_members:
             m.check(schema)
-            assert m.name not in seen
-            seen[m.name] = m
+            m.check_clash(seen)
         if self.variants:
             self.variants.check(schema, seen)
         self.members = seen.values()
@@ -1030,6 +1028,11 @@ class QAPISchemaObjectTypeMember(object):
         self.type = schema.lookup_type(self._type_name)
         assert self.type
 
+    def check_clash(self, seen):
+        # TODO change key of seen from QAPI name to C name
+        assert self.name not in seen
+        seen[self.name] = self
+
 
 class QAPISchemaObjectTypeVariants(object):
     def __init__(self, tag_name, tag_member, variants):
Signature · 1 line
2.4.3
author
#38 lore raw

PATCH v10 22/30 qapi: Simplify QAPISchemaObjectTypeVariants.check()

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

Reduce the ugly flat union / simple union conditional by doing just the essential work here, namely setting self.tag_member. Move the rest to callers.

Message-Id: <[email protected]> [rebase to earlier changes that moved tag_member.check() of alternate types, and tweak commit title and wording]

---
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
scripts/qapi.py +2 -3
index 2a73b2b..e057408 100644
@@ -988,9 +988,10 @@ class QAPISchemaObjectType(QAPISchemaType):
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)
+        self.members = seen.values()
         if self.variants:
             self.variants.check(schema, seen)
-        self.members = seen.values()
+            assert self.variants.tag_member in self.members
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
@@ -1052,8 +1053,6 @@ class QAPISchemaObjectTypeVariants(object):
     def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
-        if seen:
-            assert self.tag_member in seen.itervalues()
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
             v.check(schema, self.tag_member.type)
Signature · 1 line
2.4.3
#39 lore raw

Re: [Qemu-devel] [PATCH v10 22/30] qapi: Simplify QAPISchemaObjectTypeVariants.check()

Quoted 36 lines · Eric Blake

From: Markus Armbruster <[email protected]>

Reduce the ugly flat union / simple union conditional by doing just the essential work here, namely setting self.tag_member. Move the rest to callers.

Message-Id: <[email protected]> [rebase to earlier changes that moved tag_member.check() of alternate types, and tweak commit title and wording]

---
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
scripts/qapi.py +2 -1
index 2a73b2b..e057408 100644
@@ -988,9 +988,10 @@ class QAPISchemaObjectType(QAPISchemaType):
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)
+        self.members = seen.values()
         if self.variants:
             self.variants.check(schema, seen)
-        self.members = seen.values()
+            assert self.variants.tag_member in self.members
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1055

Quoted 3 lines
scripts/qapi.py +0 -0
@@ -1052,8 +1053,6 @@ class QAPISchemaObjectTypeVariants(object):
     def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]

My patch has:

- if self.tag_name: # flat union
+ if not self.tag_member: # flat union
self.tag_member = seen[self.tag_name]

Any particular reason for dropping it?

I like my change, because I feel it makes the assignment's purpose more
obvious: ensure .tag_member is set.

Quoted 5 lines
scripts/qapi.py +0 -2
-        if seen:
-            assert self.tag_member in seen.itervalues()
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
             v.check(schema, self.tag_member.type)
author
#40 lore raw

Re: [Qemu-devel] [PATCH v10 22/30] qapi: Simplify QAPISchemaObjectTypeVariants.check()

Quoted 16 lines · Markus Armbruster

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1055

Quoted 11 lines · Eric Blake

From: Markus Armbruster <[email protected]>

Reduce the ugly flat union / simple union conditional by doing just the essential work here, namely setting self.tag_member. Move the rest to callers.

scripts/qapi.py +0 -0
@@ -1052,8 +1053,6 @@ class QAPISchemaObjectTypeVariants(object):
     def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]

My patch has:

Quoted 5 lines

- if self.tag_name: # flat union
+ if not self.tag_member: # flat union
self.tag_member = seen[self.tag_name]

Any particular reason for dropping it?

Not really; I couldn't find anything in later patches that cared either way.

Quoted 3 lines

I like my change, because I feel it makes the assignment's purpose more
obvious: ensure .tag_member is set.

Sure, we'll go with your approach.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#41 lore raw

PATCH v10 23/30 qapi: Check for qapi collisions of flat union branches

In reply to Eric Blake

Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want QAPISchemaObjectType.check() to make the same check, so we can later reduce some of the ad hoc checks.

We already ensure that all branches of a flat union are qapi structs with no variants, at which point those members appear in the same JSON object as all non-variant members. And we already have a map 'seen' of all non-variant members. All we need is a new QAPISchemaObjectTypeVariants.check_clash(), which clones the seen map then checks for clashes with each member of the variant's qapi type.

Note that the clone of seen inside Variants.check_clash() resembles the one we just removed from Variants.check(); the difference here is that we are now checking for clashes among the qapi members of the variant type, rather than for a single clash with the variant tag name itself.

In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to variant.type.check() in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (and type.check() is already set up to allow multiple calls due to base types).

For simple unions, the same code happens to work by design, because of our synthesized wrapper classes (however, the wrapper has a single member 'data' which will never collide with the one non-variant member 'type', so it doesn't really matter).

There is no impact to alternates, which intentionally do not need to call variants.check_clash() (there, at most one of the variant's branches will be an ObjectType, and even if one exists, we are not inlining the qapi members of that object into a parent object, the way we do for unions).

No change to generated code.

---
v10: create new Variants.check_clash() rather than piggybacking on .check()
v9: new patch, split off from v8 7/17

---
 scripts/qapi.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)
scripts/qapi.py +12 -0
index e057408..4c56935 100644
@@ -992,6 +992,7 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self.variants:
             self.variants.check(schema, seen)
             assert self.variants.tag_member in self.members
+            self.variants.check_clash(schema, seen)
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
@@ -1057,6 +1058,17 @@ class QAPISchemaObjectTypeVariants(object):
         for v in self.variants:
             v.check(schema, self.tag_member.type)
 
+    def check_clash(self, schema, seen):
+        for v in self.variants:
+            # Reset seen map for each variant, since qapi names from one
+            # branch do not affect another branch
+            vseen = dict(seen)
+            assert isinstance(v.type, QAPISchemaObjectType)
+            assert not v.type.variants       # not implemented
+            v.type.check(schema)
+            for m in v.type.members:
+                m.check_clash(vseen)
+
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def __init__(self, name, typ):
Signature · 1 line
2.4.3
#42 lore raw

Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches

Quoted 21 lines · Eric Blake

Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want QAPISchemaObjectType.check() to make the same check, so we can later reduce some of the ad hoc checks.

We already ensure that all branches of a flat union are qapi structs with no variants, at which point those members appear in the same JSON object as all non-variant members. And we already have a map 'seen' of all non-variant members. All we need is a new QAPISchemaObjectTypeVariants.check_clash(), which clones the seen map then checks for clashes with each member of the variant's qapi type.

Note that the clone of seen inside Variants.check_clash() resembles the one we just removed from Variants.check(); the difference here is that we are now checking for clashes among the qapi members of the variant type, rather than for a single clash with the variant tag name itself.

Quoted 6 lines

In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to variant.type.check() in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (and type.check() is already set up to allow multiple calls due to base types).

Yes, a type cannot contain itself, neither as base nor as variant.

We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can see, we don't have tests covering the latter. Do we catch it?

Quoted 12 lines

For simple unions, the same code happens to work by design, because of our synthesized wrapper classes (however, the wrapper has a single member 'data' which will never collide with the one non-variant member 'type', so it doesn't really matter).

There is no impact to alternates, which intentionally do not need to call variants.check_clash() (there, at most one of the variant's branches will be an ObjectType, and even if one exists, we are not inlining the qapi members of that object into a parent object, the way we do for unions).

Quoted 3 lines

No change to generated code.

Patch looks good.

#43 lore raw

Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches

Quoted 30 lines · Markus Armbruster
Quoted 21 lines · Eric Blake

Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want QAPISchemaObjectType.check() to make the same check, so we can later reduce some of the ad hoc checks.

We already ensure that all branches of a flat union are qapi structs with no variants, at which point those members appear in the same JSON object as all non-variant members. And we already have a map 'seen' of all non-variant members. All we need is a new QAPISchemaObjectTypeVariants.check_clash(), which clones the seen map then checks for clashes with each member of the variant's qapi type.

Note that the clone of seen inside Variants.check_clash() resembles the one we just removed from Variants.check(); the difference here is that we are now checking for clashes among the qapi members of the variant type, rather than for a single clash with the variant tag name itself.

Quoted 6 lines

In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to variant.type.check() in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (and type.check() is already set up to allow multiple calls due to base types).

Quoted 17 lines

Yes, a type cannot contain itself, neither as base nor as variant.

We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can see, we don't have tests covering the latter. Do we catch it?

Quoted 11 lines

For simple unions, the same code happens to work by design, because of our synthesized wrapper classes (however, the wrapper has a single member 'data' which will never collide with the one non-variant member 'type', so it doesn't really matter).

There is no impact to alternates, which intentionally do not need to call variants.check_clash() (there, at most one of the variant's branches will be an ObjectType, and even if one exists, we are not inlining the qapi members of that object into a parent object, the way we do for unions).

Yes. QAPISchemaObjectTypeVariants.check_clash() checks for each variant's members clashing with other members in the same name space. For alternates, there are no such other members.

That said, should we add a comment to QAPISchemaAlternateType.check()? Perhaps:

# Not calling self.variant.check_clash(), because there's # nothing to clash with.

Quoted 5 lines
Quoted 3 lines

No change to generated code.

Patch looks good.

author
#44 lore raw

Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches

Quoted 22 lines · Markus Armbruster
Quoted 10 lines · Markus Armbruster
Quoted 7 lines · Eric Blake

Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want QAPISchemaObjectType.check() to make the same check, so we can later reduce some of the ad hoc checks.

Quoted 5 lines
Quoted 5 lines

There is no impact to alternates, which intentionally do not need to call variants.check_clash() (there, at most one of the variant's branches will be an ObjectType, and even if one exists, we are not inlining the qapi members of that object into a parent object, the way we do for unions).

Yes. QAPISchemaObjectTypeVariants.check_clash() checks for each variant's members clashing with other members in the same name space. For alternates, there are no such other members.

Quoted 5 lines

That said, should we add a comment to QAPISchemaAlternateType.check()? Perhaps:

# Not calling self.variant.check_clash(), because there's # nothing to clash with.

Sure; if there's another reason for a respin, I can add it here; otherwise, it's already slated to be added in my respin of the patches for redoing how alternates are laid out.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#45 lore raw

Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches

In reply to Markus Armbruster
Quoted 17 lines · Markus Armbruster
Quoted 8 lines · Eric Blake

Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want QAPISchemaObjectType.check() to make the same check, so we can later reduce some of the ad hoc checks.

Quoted 6 lines

In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to variant.type.check() in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (and type.check() is already set up to allow multiple calls due to base types).

Quoted 5 lines

Yes, a type cannot contain itself, neither as base nor as variant.

We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can see, we don't have tests covering the latter. Do we catch it?

Yes, at least by virtue of the ad hoc tests: attempting to reuse a base type of the flat union as a variant member will cause the qapi members of the base type to appear more than once in the JSON object (that is, the checks that reject flat-union-clash-member.json would also reject this scenario). To test:

(quoted diff) +0 -0

w/tests/qapi-schema/qapi-schema-test.json
index 44638da..16b2ffb 100644

tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -67,7 +67,7 @@
   'discriminator': 'enum1',
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
-            'value3' : 'UserDefB' } }
+            'value3' : 'UserDefUnionBase' } }
 
 { 'struct': 'UserDefUnionBase',
   'base': 'UserDefZero',

GEN tests/test-qapi-types.h
/home/eblake/qemu/tests/qapi-schema/qapi-schema-test.json:65: Member name 'string' of branch 'value3' clashes with base 'UserDefUnionBase' /home/eblake/qemu/tests/Makefile:415: recipe for target 'tests/test-qapi-types.h' failed

But you have me curious if this collision is still caught when the ad hoc tests are gone. If so, great; if not, I'll add a test here. (I'll know later when I get through rebasing to all of your comments.)

Quoted 5 lines
Quoted 3 lines

No change to generated code.

Patch looks good.

Yay; it's nice to see results after all our mental gymnastics over how collision testing should work.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#46 lore raw

Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches

Quoted 18 lines · Eric Blake
Quoted 17 lines · Markus Armbruster
Quoted 14 lines · Eric Blake

Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want QAPISchemaObjectType.check() to make the same check, so we can later reduce some of the ad hoc checks.

In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to variant.type.check() in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (and type.check() is already set up to allow multiple calls due to base types).

Quoted 4 lines
Quoted 4 lines

Yes, a type cannot contain itself, neither as base nor as variant.

We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can

Actually, these are just local, unpublished tests. They both make check_member_clash() recurse infinitely.

    # Direct inheritance loop
    # FIXME triggers infinite recursion
    { 'struct': 'Loopy', 'base': 'Loopy',
      'data': {} }

    # we reject a loop in base classes
    { 'struct': 'Base1', 'base': 'Base2', 'data': {} }
    { 'struct': 'Base2', 'base': 'Base1', 'data': {} }

The latter is actually yours, proposed as base-cycle.json in
Subject: qapi: Detect collisions in C member names
Message-Id: <[email protected]>

If I disable the recursive call, the cycle detection in QAPISchemaObjectType.check() is reached, and works.

Completing the move of clash detection to check() methods should improve things from "accidental infinite recursion" to "intentional assertion failure", because it should get rid of check_member_clash() and should not break the cycle detection.

Then we can turn the assertion into a proper error message, and add the tests.

Quoted 41 lines
Quoted 1 line

see, we don't have tests covering the latter. Do we catch it?

Yes, at least by virtue of the ad hoc tests: attempting to reuse a base type of the flat union as a variant member will cause the qapi members of the base type to appear more than once in the JSON object (that is, the checks that reject flat-union-clash-member.json would also reject this scenario). To test:

(quoted diff) +0 -0

w/tests/qapi-schema/qapi-schema-test.json
index 44638da..16b2ffb 100644

tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -67,7 +67,7 @@
   'discriminator': 'enum1',
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
-            'value3' : 'UserDefB' } }
+            'value3' : 'UserDefUnionBase' } }
 
 { 'struct': 'UserDefUnionBase',
   'base': 'UserDefZero',

GEN tests/test-qapi-types.h
/home/eblake/qemu/tests/qapi-schema/qapi-schema-test.json:65: Member name 'string' of branch 'value3' clashes with base 'UserDefUnionBase' /home/eblake/qemu/tests/Makefile:415: recipe for target 'tests/test-qapi-types.h' failed

But you have me curious if this collision is still caught when the ad hoc tests are gone. If so, great; if not, I'll add a test here. (I'll know later when I get through rebasing to all of your comments.)

Quoted 5 lines
Quoted 3 lines

No change to generated code.

Patch looks good.

Yay; it's nice to see results after all our mental gymnastics over how collision testing should work.

author
#47 lore raw

Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches

Quoted 25 lines · Markus Armbruster
Quoted 18 lines · Eric Blake
Quoted 17 lines · Markus Armbruster
Quoted 14 lines · Eric Blake

Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want QAPISchemaObjectType.check() to make the same check, so we can later reduce some of the ad hoc checks.

In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to variant.type.check() in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (and type.check() is already set up to allow multiple calls due to base types).

Quoted 4 lines
Quoted 4 lines

Yes, a type cannot contain itself, neither as base nor as variant.

We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can

Quoted 7 lines

Actually, these are just local, unpublished tests. They both make check_member_clash() recurse infinitely.

# Direct inheritance loop
# FIXME triggers infinite recursion
{ 'struct': 'Loopy', 'base': 'Loopy',
'data': {} }

Okay, I should add that into my pending patch that cleans up base loops,

Quoted 8 lines
    # we reject a loop in base classes
    { 'struct': 'Base1', 'base': 'Base2', 'data': {} }
    { 'struct': 'Base2', 'base': 'Base1', 'data': {} }

The latter is actually yours, proposed as base-cycle.json in
Subject: qapi: Detect collisions in C member names
Message-Id: <[email protected]>

and yes, that one is still in my queue for subset D:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg07001.html

and I may indeed at a test for reusing the base type of a flat union as one of the branches of the same union, depending on whether it uncovers anything different.

Quoted 11 lines

If I disable the recursive call, the cycle detection in QAPISchemaObjectType.check() is reached, and works.

Completing the move of clash detection to check() methods should improve things from "accidental infinite recursion" to "intentional assertion failure", because it should get rid of check_member_clash() and should not break the cycle detection.

Then we can turn the assertion into a proper error message, and add the tests.

Yep, that's what is pending in my queue, just further out than subset C.
Doesn't matter if it misses 2.5 (the bug is real, but is only triggered
by bad .json code, and we aren't going to add any bad .json code between now and 2.5).

Quoted 3 lines
Quoted 3 lines

But you have me curious if this collision is still caught when the ad hoc tests are gone. If so, great; if not, I'll add a test here. (I'll know later when I get through rebasing to all of your comments.)

Still true - I'm still plowing through earlier patches before deciding if my 'qapi: Detect base class loops' also needs to detect flat union loops.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#48 lore raw

Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches

In reply to Eric Blake
Quoted 16 lines · Eric Blake
Quoted 3 lines

We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can see, we don't have tests covering the latter. Do we catch it?

Yes, at least by virtue of the ad hoc tests: attempting to reuse a base type of the flat union as a variant member will cause the qapi members of the base type to appear more than once in the JSON object (that is, the checks that reject flat-union-clash-member.json would also reject this scenario). To test:

(quoted diff) +0 -0

w/tests/qapi-schema/qapi-schema-test.json
index 44638da..16b2ffb 100644

tests/qapi-schema/qapi-schema-test.json line 70

Quoted 5 lines
tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -67,7 +67,7 @@
   'discriminator': 'enum1',
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
-            'value3' : 'UserDefB' } }
+            'value3' : 'UserDefUnionBase' } }

Another test I just tried is creating a flat union with:

empty -> base -> union

then use empty as one of the union branches. In that case, there is no conflict (although base is included twice, neither inclusion adds members to the JSON object; and the inclusion is not circular so things compile just fine). So probably not worth adding a test for it.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#49 lore raw

Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches

Quoted 26 lines · Eric Blake
Quoted 16 lines · Eric Blake
Quoted 3 lines

We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can see, we don't have tests covering the latter. Do we catch it?

Yes, at least by virtue of the ad hoc tests: attempting to reuse a base type of the flat union as a variant member will cause the qapi members of the base type to appear more than once in the JSON object (that is, the checks that reject flat-union-clash-member.json would also reject this scenario). To test:

(quoted diff) +0 -0

w/tests/qapi-schema/qapi-schema-test.json
index 44638da..16b2ffb 100644

tests/qapi-schema/qapi-schema-test.json line 70

Quoted 5 lines
tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -67,7 +67,7 @@
   'discriminator': 'enum1',
   'data': { 'value1' : 'UserDefA',
             'value2' : 'UserDefB',
-            'value3' : 'UserDefB' } }
+            'value3' : 'UserDefUnionBase' } }

Another test I just tried is creating a flat union with:

Quoted 6 lines

empty -> base -> union

then use empty as one of the union branches. In that case, there is no conflict (although base is included twice, neither inclusion adds members to the JSON object; and the inclusion is not circular so things compile just fine). So probably not worth adding a test for it.

Member name clashes occur because:

1. Two separately defined members happen to have a clashing name.

Whether the members are local or inherited, variant or non-variant doesn't matter.

2. The same type gets spliced in twice, and conflict.

By "spliced", I mean the members get included. A bit like unboxed, but also "unwrapped".

Base types and variant types get spliced in. Variant types don't conflict with each other. Empty type can't conflict (outlawing it anyway would be pointless). Therefore, the only conflicting splice is a non-empty type occuring both as (base type of the) base type and as (base type of a) variant type.

We detect this kind of clash exactly like 1. That's fine, because the resulting error messages are plainly good enough.

3. Type contains itself.

The type gets spliced into itself, or has itself as unboxed member.

We don't unbox object members so far. That leaves the spliced ones:
base types and variants. Thus, a type contains itself if it's its own base or one of its own variants, or a direct or indirect base of either.

We detect this kind of clash separately, when we guard against infinite check() recursion. Reporting it right there will be easiest.

We want to cover all this with tests. I guess we're lacking only 3.

author
#50 lore raw

PATCH v10 24/30 qapi: Factor out QAPISchemaObjectType.check_clash()

In reply to Eric Blake

Consolidate two common sequences of clash detection into a new QAPISchemaObjectType.check_clash() helper method.

No change to generated code.

---
v10: rebase on new Variants.check_clash()
v9: new patch, split off from v8 7/17

---
 scripts/qapi.py | 19 ++++++++-----------
 1 file changed, 8 insertions(+), 11 deletions(-)
scripts/qapi.py +8 -11
index 4c56935..6d8c4c7 100644
@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)
-            for m in self.base.members:
-                m.check_clash(seen)
+            self.base.check_clash(schema, seen)
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)
@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert self.variants.tag_member in self.members
             self.variants.check_clash(schema, seen)
 
+    def check_clash(self, schema, seen):
+        self.check(schema)
+        assert not self.variants       # not implemented
+        for m in self.members:
+            m.check_clash(seen)
+
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
         return self.name[0] == ':'
@@ -1062,12 +1064,7 @@ class QAPISchemaObjectTypeVariants(object):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
-            vseen = dict(seen)
-            assert isinstance(v.type, QAPISchemaObjectType)
-            assert not v.type.variants       # not implemented
-            v.type.check(schema)
-            for m in v.type.members:
-                m.check_clash(vseen)
+            v.type.check_clash(schema, dict(seen))
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Signature · 1 line
2.4.3
#51 lore raw

Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()

Quoted 19 lines · Eric Blake

Consolidate two common sequences of clash detection into a new QAPISchemaObjectType.check_clash() helper method.

No change to generated code.

---
v10: rebase on new Variants.check_clash()
v9: new patch, split off from v8 7/17

---
 scripts/qapi.py | 19 ++++++++-----------
 1 file changed, 8 insertions(+), 11 deletions(-)
scripts/qapi.py +0 -0
index 4c56935..6d8c4c7 100644

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 983

Quoted 4 lines
scripts/qapi.py +0 -1
@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)

This assertion is lost.

Quoted 22 lines
scripts/qapi.py +7 -4
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)
-            for m in self.base.members:
-                m.check_clash(seen)
+            self.base.check_clash(schema, seen)
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)
@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert self.variants.tag_member in self.members
             self.variants.check_clash(schema, seen)
 
+    def check_clash(self, schema, seen):
+        self.check(schema)
+        assert not self.variants       # not implemented
+        for m in self.members:
+            m.check_clash(seen)
+
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
         return self.name[0] == ':'

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066

Quoted 5 lines
scripts/qapi.py +0 -2
@@ -1062,12 +1064,7 @@ class QAPISchemaObjectTypeVariants(object):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
-            vseen = dict(seen)
-            assert isinstance(v.type, QAPISchemaObjectType)

This assertion is lost.

Quoted 8 lines
scripts/qapi.py +1 -4
-            assert not v.type.variants       # not implemented
-            v.type.check(schema)
-            for m in v.type.members:
-                m.check_clash(vseen)
+            v.type.check_clash(schema, dict(seen))
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
author
#52 lore raw

Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()

Quoted 11 lines · Markus Armbruster
Quoted 8 lines · Eric Blake

Consolidate two common sequences of clash detection into a new QAPISchemaObjectType.check_clash() helper method.

No change to generated code.

---

Quoted 13 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 983

Quoted 5 lines
scripts/qapi.py +0 -1
@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)

This assertion is lost.

Quoted 5 lines
scripts/qapi.py +1 -4
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)
-            for m in self.base.members:
-                m.check_clash(seen)
+            self.base.check_clash(schema, seen)

Directly lost, but indirectly still present. The new code is calling QAPISchemaObjectType.check_clash(), which won't exist unless self.base is a QAPISchemaObjectType. Folding the assert into the refactored function makes no sense (the condition isinstance(self, QAPISchemaObjectType) would always be true), and leaving the assert prior to calling self.base.check_clash() adds no real protection against programming bugs.

Quoted 14 lines

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066

Quoted 6 lines
scripts/qapi.py +0 -2
@@ -1062,12 +1064,7 @@ class QAPISchemaObjectTypeVariants(object):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
-            vseen = dict(seen)
-            assert isinstance(v.type, QAPISchemaObjectType)

This assertion is lost.

Quoted 5 lines
scripts/qapi.py +1 -4
-            assert not v.type.variants       # not implemented
-            v.type.check(schema)
-            for m in v.type.members:
-                m.check_clash(vseen)
+            v.type.check_clash(schema, dict(seen))

Same explanation.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#53 lore raw

Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()

Quoted 26 lines · Eric Blake
Quoted 11 lines · Markus Armbruster
Quoted 8 lines · Eric Blake

Consolidate two common sequences of clash detection into a new QAPISchemaObjectType.check_clash() helper method.

No change to generated code.

---

Quoted 13 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 983

Quoted 5 lines
scripts/qapi.py +0 -1
@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)

This assertion is lost.

Quoted 5 lines
scripts/qapi.py +1 -4
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)
-            for m in self.base.members:
-                m.check_clash(seen)
+            self.base.check_clash(schema, seen)
Quoted 3 lines

Directly lost, but indirectly still present. The new code is calling QAPISchemaObjectType.check_clash(), which won't exist unless self.base is a QAPISchemaObjectType.

or a QAPISchemaObjectTypeMember, or a QAPISchemaObjectVariants, or whatever else acquires the method in the future.

Quoted 3 lines

Folding the assert into the refactored
function makes no sense (the condition isinstance(self, QAPISchemaObjectType) would always be true),

Correct.

Quoted 3 lines

and leaving the assert
prior to calling self.base.check_clash() adds no real protection against programming bugs.

Maybe, but the isinstance(self.base, QAPISchemaObjectType) will come right back anyway when we move the "'base' for FOO cannot use BAR type" check from the old semantic analysis into the check methods. Until then, it makes sense at least as a place holder.

Quoted 16 lines
Quoted 14 lines

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066

Quoted 6 lines
scripts/qapi.py +0 -2
@@ -1062,12 +1064,7 @@ class QAPISchemaObjectTypeVariants(object):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
-            vseen = dict(seen)
-            assert isinstance(v.type, QAPISchemaObjectType)

This assertion is lost.

Quoted 5 lines
scripts/qapi.py +1 -4
-            assert not v.type.variants       # not implemented
-            v.type.check(schema)
-            for m in v.type.members:
-                m.check_clash(vseen)
+            v.type.check_clash(schema, dict(seen))

Same explanation.

author
#54 lore raw

Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()

Quoted 31 lines · Markus Armbruster
Quoted 20 lines · Eric Blake
Quoted 9 lines · Markus Armbruster
Quoted 6 lines · Eric Blake

Consolidate two common sequences of clash detection into a new QAPISchemaObjectType.check_clash() helper method.

No change to generated code.

Quoted 8 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 983

Quoted 5 lines
scripts/qapi.py +0 -1
@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)

This assertion is lost.

Quoted 3 lines

Directly lost, but indirectly still present. The new code is calling QAPISchemaObjectType.check_clash(), which won't exist unless self.base is a QAPISchemaObjectType.

or a QAPISchemaObjectTypeMember, or a QAPISchemaObjectVariants, or whatever else acquires the method in the future.

Quoted 4 lines

Maybe, but the isinstance(self.base, QAPISchemaObjectType) will come right back anyway when we move the "'base' for FOO cannot use BAR type" check from the old semantic analysis into the check methods. Until then, it makes sense at least as a place holder.

Good point, I'll add it back in.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#55 lore raw

Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()

In reply to Eric Blake
Quoted 32 lines · Eric Blake

Consolidate two common sequences of clash detection into a new QAPISchemaObjectType.check_clash() helper method.

No change to generated code.

---
v10: rebase on new Variants.check_clash()
v9: new patch, split off from v8 7/17

---
 scripts/qapi.py | 19 ++++++++-----------
 1 file changed, 8 insertions(+), 11 deletions(-)
scripts/qapi.py +1 -5
index 4c56935..6d8c4c7 100644
@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)
-            for m in self.base.members:
-                m.check_clash(seen)
+            self.base.check_clash(schema, seen)
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 994

Quoted 5 lines
scripts/qapi.py +2 -0
@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert self.variants.tag_member in self.members
             self.variants.check_clash(schema, seen)
 
+    def check_clash(self, schema, seen):
+        self.check(schema)

Do we want to hide this .check() inside .check_clash()?

QAPISchemaObjectTypeMember.check() doesn't. I think the two better behave the same.

Quoted 21 lines
scripts/qapi.py +5 -6
+        assert not self.variants       # not implemented
+        for m in self.members:
+            m.check_clash(seen)
+
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
         return self.name[0] == ':'
@@ -1062,12 +1064,7 @@ class QAPISchemaObjectTypeVariants(object):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
-            vseen = dict(seen)
-            assert isinstance(v.type, QAPISchemaObjectType)
-            assert not v.type.variants       # not implemented
-            v.type.check(schema)
-            for m in v.type.members:
-                m.check_clash(vseen)
+            v.type.check_clash(schema, dict(seen))
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
author
#56 lore raw

Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()

Quoted 27 lines · Markus Armbruster
Quoted 25 lines · Eric Blake

Consolidate two common sequences of clash detection into a new QAPISchemaObjectType.check_clash() helper method.

No change to generated code.

scripts/qapi.py +1 -5
@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)
-            for m in self.base.members:
-                m.check_clash(seen)
+            self.base.check_clash(schema, seen)
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)
@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert self.variants.tag_member in self.members
             self.variants.check_clash(schema, seen)
 
Quoted 11 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 994

Quoted 2 lines
scripts/qapi.py +2 -0
+    def check_clash(self, schema, seen):
+        self.check(schema)

Do we want to hide this .check() inside .check_clash()?

QAPISchemaObjectTypeMember.check() doesn't. I think the two better behave the same.

Quoted 3 lines
scripts/qapi.py +3 -0
+        assert not self.variants       # not implemented
+        for m in self.members:
+            m.check_clash(seen)

The self.check(schema) call is necessary to get self.members populated.
We cannot iterate over self.members if the type has not had check()
called; this is true for both callers of type.check_clash() (ObjectType.check(), and Variants.check_clash()).

You are correct that neither Member.check() nor Member.check_clash() call a form of type.check() - but that's because at that level, there is no need to populate a type.members list.

On the other hand, we've been arguing that check() should populate everything after construction prior to anything else being run; and not running Variant.type.check() during Variants.check() of flat unions feels like we may have a hole (a flat union will have to inline its types to the overall JSON object, and inlining types requires access to type.members - but as written, we aren't populating them until Variants.check_clash()). I can play with hoisting the type.check() out of type.check_clash() and instead keep base.check() in type.check(), and add variant.type.check() in Variants.check() (but only for unions, not for alternates), if you are interested.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#57 lore raw

Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()

Quoted 36 lines · Eric Blake
Quoted 35 lines · Markus Armbruster
Quoted 22 lines · Eric Blake

Consolidate two common sequences of clash detection into a new QAPISchemaObjectType.check_clash() helper method.

No change to generated code.

(quoted diff) +1 -5
@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)
-            for m in self.base.members:
-                m.check_clash(seen)
+            self.base.check_clash(schema, seen)
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)

Review class QAPISchemaObjectType(QAPISchemaType): · line 994

Quoted 5 lines
(quoted diff) +2 -0
@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert self.variants.tag_member in self.members
             self.variants.check_clash(schema, seen)
 
+    def check_clash(self, schema, seen):
+        self.check(schema)

Do we want to hide this .check() inside .check_clash()?

QAPISchemaObjectTypeMember.check() doesn't. I think the two better behave the same.

Quoted 8 lines
Quoted 3 lines

Review class QAPISchemaObjectType(QAPISchemaType): · line 997

Quoted 3 lines
(quoted diff) +3 -0
+        assert not self.variants       # not implemented
+        for m in self.members:
+            m.check_clash(seen)

The self.check(schema) call is necessary to get self.members populated.
We cannot iterate over self.members if the type has not had check()
called; this is true for both callers of type.check_clash() (ObjectType.check(), and Variants.check_clash()).

Yes.

We have a common protocol for QAPISchemaFOO objects, namely that certain instance variables and methods are only valid after .check().

Quoted 4 lines

You are correct that neither Member.check() nor Member.check_clash() call a form of type.check() - but that's because at that level, there is no need to populate a type.members list.

Quoted 10 lines

On the other hand, we've been arguing that check() should populate everything after construction prior to anything else being run; and not running Variant.type.check() during Variants.check() of flat unions feels like we may have a hole (a flat union will have to inline its types to the overall JSON object, and inlining types requires access to type.members - but as written, we aren't populating them until Variants.check_clash()). I can play with hoisting the type.check() out of type.check_clash() and instead keep base.check() in type.check(), and add variant.type.check() in Variants.check() (but only for unions, not for alternates), if you are interested.

My "qapi: Factor out QAPISchemaObjectTypeMember.check_clash()" added QAPISchemaObjectTypeMember.check_clash() without changing the common protocol. The new QAPISchemaObjectTypeMember.check_clash() is merely a helper for QAPISchemaObjectType.check().

Your
Gcc: nnml:mail.redhat.xlst.qemu-devel
From: Markus Armbruster <[email protected]>
--text follows this line--

Quoted 36 lines · Eric Blake
Quoted 35 lines · Markus Armbruster
Quoted 22 lines · Eric Blake

Consolidate two common sequences of clash detection into a new QAPISchemaObjectType.check_clash() helper method.

No change to generated code.

(quoted diff) +1 -5
@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            assert isinstance(self.base, QAPISchemaObjectType)
-            assert not self.base.variants       # not implemented
-            self.base.check(schema)
-            for m in self.base.members:
-                m.check_clash(seen)
+            self.base.check_clash(schema, seen)
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)

Review class QAPISchemaObjectType(QAPISchemaType): · line 994

Quoted 5 lines
(quoted diff) +2 -0
@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert self.variants.tag_member in self.members
             self.variants.check_clash(schema, seen)
 
+    def check_clash(self, schema, seen):
+        self.check(schema)

Do we want to hide this .check() inside .check_clash()?

QAPISchemaObjectTypeMember.check() doesn't. I think the two better behave the same.

Quoted 8 lines
Quoted 3 lines

Review class QAPISchemaObjectType(QAPISchemaType): · line 997

Quoted 3 lines
(quoted diff) +3 -0
+        assert not self.variants       # not implemented
+        for m in self.members:
+            m.check_clash(seen)

The self.check(schema) call is necessary to get self.members populated.
We cannot iterate over self.members if the type has not had check()
called; this is true for both callers of type.check_clash() (ObjectType.check(), and Variants.check_clash()).

Yes.

We have a common protocol for QAPISchemaFOO objects, namely that certain instance variables and methods are only valid after .check().

Quoted 4 lines

You are correct that neither Member.check() nor Member.check_clash() call a form of type.check() - but that's because at that level, there is no need to populate a type.members list.

Quoted 10 lines

On the other hand, we've been arguing that check() should populate everything after construction prior to anything else being run; and not running Variant.type.check() during Variants.check() of flat unions feels like we may have a hole (a flat union will have to inline its types to the overall JSON object, and inlining types requires access to type.members - but as written, we aren't populating them until Variants.check_clash()). I can play with hoisting the type.check() out of type.check_clash() and instead keep base.check() in type.check(), and add variant.type.check() in Variants.check() (but only for unions, not for alternates), if you are interested.

My "qapi: Factor out QAPISchemaObjectTypeMember.check_clash()" adds QAPISchemaObjectTypeMember.check_clash() without changing the common protocol. The new QAPISchemaObjectTypeMember.check_clash() is merely a helper for QAPISchemaObjectType.check().

The two .check_clash() you add (one in this patch, one in the previous one) are different: both contain calls of QAPISchemaObjectType.check().

I feel the .check() calls are too important to be buried deep like that. I'd stick to prior practice and put the .check() calls right into .check(). Obviously, the .check_clash() methods may only called after .check() then, but that's nothing new.

Fixup for your previous patch:

scripts/qapi.py +1 -1
index 4c56935..357127d 100644
@@ -1065,7 +1065,6 @@ class QAPISchemaObjectTypeVariants(object):
             vseen = dict(seen)
             assert isinstance(v.type, QAPISchemaObjectType)
             assert not v.type.variants       # not implemented
-            v.type.check(schema)
             for m in v.type.members:
                 m.check_clash(vseen)
 
@@ -1077,6 +1076,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def check(self, schema, tag_type):
         QAPISchemaObjectTypeMember.check(self, schema)
         assert self.name in tag_type.values
+        self.type.check(schema)
 
     # This function exists to support ugly simple union special cases
     # TODO get rid of them, and drop the function

This patch with conflicts resolved and the same idea worked in then becomes:

scripts/qapi.py +7 -7
index 357127d..f443ec5 100644
@@ -981,10 +981,8 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
             assert isinstance(self.base, QAPISchemaObjectType)
-            assert not self.base.variants       # not implemented
             self.base.check(schema)
-            for m in self.base.members:
-                m.check_clash(seen)
+            self.base.check_clash(schema, seen)
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)
@@ -994,6 +992,11 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert self.variants.tag_member in self.members
             self.variants.check_clash(schema, seen)
 
+    def check_clash(self, schema, seen):
+        assert not self.variants       # not implemented
+        for m in self.members:
+            m.check_clash(seen)
+
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
         return self.name[0] == ':'
@@ -1062,11 +1065,8 @@ class QAPISchemaObjectTypeVariants(object):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
-            vseen = dict(seen)
             assert isinstance(v.type, QAPISchemaObjectType)
-            assert not v.type.variants       # not implemented
-            for m in v.type.members:
-                m.check_clash(vseen)
+            v.type.check_clash(schema, dict(seen))
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):

Your two check_clash() are then simple helpers, just like mine.

author
#58 lore raw

Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()

Quoted 39 lines · Markus Armbruster
Quoted 10 lines

On the other hand, we've been arguing that check() should populate everything after construction prior to anything else being run; and not running Variant.type.check() during Variants.check() of flat unions feels like we may have a hole (a flat union will have to inline its types to the overall JSON object, and inlining types requires access to type.members - but as written, we aren't populating them until Variants.check_clash()). I can play with hoisting the type.check() out of type.check_clash() and instead keep base.check() in type.check(), and add variant.type.check() in Variants.check() (but only for unions, not for alternates), if you are interested.

My "qapi: Factor out QAPISchemaObjectTypeMember.check_clash()" adds QAPISchemaObjectTypeMember.check_clash() without changing the common protocol. The new QAPISchemaObjectTypeMember.check_clash() is merely a helper for QAPISchemaObjectType.check().

The two .check_clash() you add (one in this patch, one in the previous one) are different: both contain calls of QAPISchemaObjectType.check().

I feel the .check() calls are too important to be buried deep like that. I'd stick to prior practice and put the .check() calls right into .check(). Obviously, the .check_clash() methods may only called after .check() then, but that's nothing new.

Fixup for your previous patch:

scripts/qapi.py +0 -1
index 4c56935..357127d 100644
@@ -1065,7 +1065,6 @@ class QAPISchemaObjectTypeVariants(object):
             vseen = dict(seen)
             assert isinstance(v.type, QAPISchemaObjectType)
             assert not v.type.variants       # not implemented
-            v.type.check(schema)
             for m in v.type.members:
                 m.check_clash(vseen)
 

scripts/qapi.py class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): · line 1080

Quoted 5 lines
scripts/qapi.py +1 -0
@@ -1077,6 +1076,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def check(self, schema, tag_type):
         QAPISchemaObjectTypeMember.check(self, schema)
         assert self.name in tag_type.values
+        self.type.check(schema)
 

Won't quite work. You are right that we must call self.type.check(schema) for variants used by a union; but calling it for ALL variants used by an alternate is wrong, because self.type for at least one branch of an alternate will not be an instance of QAPISchemaObjectType. However, I'm currently testing whether it is safe to check to just blindly check an object branch of an alternate, if present (and that should not lead to cycles, since alternates have no base class and since we don't allow one alternate type as a variant of another alternate), in which case the fixup for 23/30 is more like:

scripts/qapi.py +0 -1
index a005c87..25fa642 100644
@@ -1065,7 +1065,6 @@ class QAPISchemaObjectTypeVariants(object):
             vseen = dict(seen)
             assert isinstance(v.type, QAPISchemaObjectType)
             assert not v.type.variants       # not implemented
-            v.type.check(schema)
             for m in v.type.members:
                 m.check_clash(vseen)
 
@@ -1077,6 +1076,8 @@ class

QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
def check(self, schema, tag_type):
QAPISchemaObjectTypeMember.check(self, schema) assert self.name in tag_type.values
+ if isinstance(self.type, QAPISchemaObjectType):
+ self.type.check(schema)

# This function exists to support ugly simple union special cases # TODO get rid of them, and drop the function
@@ -1098,6 +1099,8 @@ class QAPISchemaAlternateType(QAPISchemaType):

def check(self, schema):
self.variants.tag_member.check(schema)
+ # Not calling self.variants.check_clash(), because there's
+ # nothing to clash with
self.variants.check(schema, {})

def json_type(self):

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#59 lore raw

Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()

Quoted 80 lines · Eric Blake
Quoted 39 lines · Markus Armbruster
Quoted 10 lines

On the other hand, we've been arguing that check() should populate everything after construction prior to anything else being run; and not running Variant.type.check() during Variants.check() of flat unions feels like we may have a hole (a flat union will have to inline its types to the overall JSON object, and inlining types requires access to type.members - but as written, we aren't populating them until Variants.check_clash()). I can play with hoisting the type.check() out of type.check_clash() and instead keep base.check() in type.check(), and add variant.type.check() in Variants.check() (but only for unions, not for alternates), if you are interested.

My "qapi: Factor out QAPISchemaObjectTypeMember.check_clash()" adds QAPISchemaObjectTypeMember.check_clash() without changing the common protocol. The new QAPISchemaObjectTypeMember.check_clash() is merely a helper for QAPISchemaObjectType.check().

The two .check_clash() you add (one in this patch, one in the previous one) are different: both contain calls of QAPISchemaObjectType.check().

I feel the .check() calls are too important to be buried deep like that. I'd stick to prior practice and put the .check() calls right into .check(). Obviously, the .check_clash() methods may only called after .check() then, but that's nothing new.

Fixup for your previous patch:

scripts/qapi.py +0 -1
index 4c56935..357127d 100644
@@ -1065,7 +1065,6 @@ class QAPISchemaObjectTypeVariants(object):
             vseen = dict(seen)
             assert isinstance(v.type, QAPISchemaObjectType)
             assert not v.type.variants       # not implemented
-            v.type.check(schema)
             for m in v.type.members:
                 m.check_clash(vseen)
 

scripts/qapi.py class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): · line 1080

Quoted 5 lines
scripts/qapi.py +1 -0
@@ -1077,6 +1076,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def check(self, schema, tag_type):
         QAPISchemaObjectTypeMember.check(self, schema)
         assert self.name in tag_type.values
+        self.type.check(schema)
 

Won't quite work. You are right that we must call self.type.check(schema) for variants used by a union; but calling it for ALL variants used by an alternate is wrong, because self.type for at least one branch of an alternate will not be an instance of QAPISchemaObjectType. However, I'm currently testing whether it is safe to check to just blindly check an object branch of an alternate, if present (and that should not lead to cycles, since alternates have no base class and since we don't allow one alternate type as a variant of another alternate), in which case the fixup for 23/30 is more like:

scripts/qapi.py +0 -1
index a005c87..25fa642 100644
@@ -1065,7 +1065,6 @@ class QAPISchemaObjectTypeVariants(object):
             vseen = dict(seen)
             assert isinstance(v.type, QAPISchemaObjectType)
             assert not v.type.variants       # not implemented
-            v.type.check(schema)
             for m in v.type.members:
                 m.check_clash(vseen)
 
@@ -1077,6 +1076,8 @@ class

QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
def check(self, schema, tag_type):
QAPISchemaObjectTypeMember.check(self, schema) assert self.name in tag_type.values

scripts/qapi.py +2 -0
+        if isinstance(self.type, QAPISchemaObjectType):
+            self.type.check(schema)
 
     # This function exists to support ugly simple union special cases
     # TODO get rid of them, and drop the function

scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1106

Quoted 8 lines
scripts/qapi.py +2 -0
@@ -1098,6 +1099,8 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
     def check(self, schema):
         self.variants.tag_member.check(schema)
+        # Not calling self.variants.check_clash(), because there's
+        # nothing to clash with
         self.variants.check(schema, {})
 
     def json_type(self):

Makes sense to me.

author
#60 lore raw

PATCH v10 25/30 qapi: Hoist tag collision check to Variants.check()

In reply to Eric Blake

Checking that a given QAPISchemaObjectTypeVariant.name is a member of the corresponding QAPISchemaEnumType of the owning QAPISchemaObjectTypeVariants.tag_member ensures that there are no collisions in the generated C union for those tag values (since the enum itself should have no collisions).

However, this check was the only thing that Variant.check() was doing beyond the work of the superclass ObjectTypeMember.check(), and resulted in a difference of the .check() signatures just to pass the enum type down.

Simplify things by instead doing the tag name check as part of Variants.check(), at which point we can rely on inheritance instead of overriding Variant.check().

---
v10: new patch

---
 scripts/qapi.py | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
scripts/qapi.py +4 -6
index 6d8c4c7..798df51 100644
@@ -1056,9 +1056,11 @@ class QAPISchemaObjectTypeVariants(object):
     def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
-        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        tag_type = self.tag_member.type
+        assert isinstance(tag_type, QAPISchemaEnumType)
         for v in self.variants:
-            v.check(schema, self.tag_member.type)
+            v.check(schema)
+            assert v.name in tag_type.values
 
     def check_clash(self, schema, seen):
         for v in self.variants:
@@ -1071,10 +1073,6 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
-    def check(self, schema, tag_type):
-        QAPISchemaObjectTypeMember.check(self, schema)
-        assert self.name in tag_type.values
-
     # This function exists to support ugly simple union special cases
     # TODO get rid of them, and drop the function
     def simple_union_type(self):
Signature · 1 line
2.4.3
#61 lore raw

Re: [Qemu-devel] [PATCH v10 25/30] qapi: Hoist tag collision check to Variants.check()

Quoted 28 lines · Eric Blake

Checking that a given QAPISchemaObjectTypeVariant.name is a member of the corresponding QAPISchemaEnumType of the owning QAPISchemaObjectTypeVariants.tag_member ensures that there are no collisions in the generated C union for those tag values (since the enum itself should have no collisions).

However, this check was the only thing that Variant.check() was doing beyond the work of the superclass ObjectTypeMember.check(), and resulted in a difference of the .check() signatures just to pass the enum type down.

Simplify things by instead doing the tag name check as part of Variants.check(), at which point we can rely on inheritance instead of overriding Variant.check().

---
v10: new patch

---
 scripts/qapi.py | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
scripts/qapi.py +0 -0
index 6d8c4c7..798df51 100644

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1063

Quoted 10 lines
scripts/qapi.py +4 -2
@@ -1056,9 +1056,11 @@ class QAPISchemaObjectTypeVariants(object):
     def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
-        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        tag_type = self.tag_member.type
+        assert isinstance(tag_type, QAPISchemaEnumType)
         for v in self.variants:
-            v.check(schema, self.tag_member.type)
+            v.check(schema)
+            assert v.name in tag_type.values

Two changes squashed together:

* Move the assertion from QAPISchemaObjectTypeVariant.check(),

* Capture self.tag_member.type in tag_type

The second part makes the patch slightly less obvious. Matter of taste.

Quoted 14 lines
scripts/qapi.py +0 -4
     def check_clash(self, schema, seen):
         for v in self.variants:
@@ -1071,10 +1073,6 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
-    def check(self, schema, tag_type):
-        QAPISchemaObjectTypeMember.check(self, schema)
-        assert self.name in tag_type.values
-
     # This function exists to support ugly simple union special cases
     # TODO get rid of them, and drop the function
     def simple_union_type(self):
author
#62 lore raw

Re: [Qemu-devel] [PATCH v10 25/30] qapi: Hoist tag collision check to Variants.check()

Quoted 29 lines · Markus Armbruster
Quoted 19 lines · Eric Blake

Checking that a given QAPISchemaObjectTypeVariant.name is a member of the corresponding QAPISchemaEnumType of the owning QAPISchemaObjectTypeVariants.tag_member ensures that there are no collisions in the generated C union for those tag values (since the enum itself should have no collisions).

However, this check was the only thing that Variant.check() was doing beyond the work of the superclass ObjectTypeMember.check(), and resulted in a difference of the .check() signatures just to pass the enum type down.

Simplify things by instead doing the tag name check as part of Variants.check(), at which point we can rely on inheritance instead of overriding Variant.check().

---

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1063

Quoted 7 lines
scripts/qapi.py +4 -2
-        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        tag_type = self.tag_member.type
+        assert isinstance(tag_type, QAPISchemaEnumType)
         for v in self.variants:
-            v.check(schema, self.tag_member.type)
+            v.check(schema)
+            assert v.name in tag_type.values
Quoted 7 lines

Two changes squashed together:

* Move the assertion from QAPISchemaObjectTypeVariant.check(),

* Capture self.tag_member.type in tag_type

The second part makes the patch slightly less obvious. Matter of taste.

I'm dropping the tag_type temporary member. Once I played more with making qtype_code a qapi builtin type, it became much more elegant to revisit this code (the use of tag_type here was for when my later patches were trying to key off of tag_type==None as a witness of alternates).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#63 lore raw

PATCH v10 26/30 qapi: Remove outdated tests related to QMP/branch collisions

In reply to Eric Blake

Now that branches are in a separate C namespace, we can remove the restrictions in the parser that claim a branch name would collide with QMP, and delete the negative tests that are no longer problematic. A separate patch can then add positive tests to qapi-schema-test to test that any corner cases will compile correctly.

This reverts the scripts/qapi.py portion of commit 7b2a5c2, now that the assertions that it plugged are no longer possible.

---
v10: no change
v9: no change
v8: retitle and update commit message, finish reversion of 7b2a5c2
v7: new patch (also temporarily appeared in subset B v10)

---
 scripts/qapi.py                                | 11 ++---------
 tests/Makefile                                 |  3 ---
 tests/qapi-schema/flat-union-clash-branch.err  |  0
 tests/qapi-schema/flat-union-clash-branch.exit |  1 -
 tests/qapi-schema/flat-union-clash-branch.json | 18 ------------------
 tests/qapi-schema/flat-union-clash-branch.out  | 14 --------------
 tests/qapi-schema/flat-union-clash-type.err    |  1 -
 tests/qapi-schema/flat-union-clash-type.exit   |  1 -
 tests/qapi-schema/flat-union-clash-type.json   | 14 --------------
 tests/qapi-schema/flat-union-clash-type.out    |  0
 tests/qapi-schema/union-clash-type.err         |  1 -
 tests/qapi-schema/union-clash-type.exit        |  1 -
 tests/qapi-schema/union-clash-type.json        |  9 ---------
 tests/qapi-schema/union-clash-type.out         |  0
 14 files changed, 2 insertions(+), 72 deletions(-)
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.exit
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.json
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.out
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.exit
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.json
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.out
 delete mode 100644 tests/qapi-schema/union-clash-type.err
 delete mode 100644 tests/qapi-schema/union-clash-type.exit
 delete mode 100644 tests/qapi-schema/union-clash-type.json
 delete mode 100644 tests/qapi-schema/union-clash-type.out
scripts/qapi.py +2 -9
index 798df51..b914785 100644
@@ -548,8 +548,7 @@ def check_union(expr, expr_info):
     base = expr.get('base')
     discriminator = expr.get('discriminator')
     members = expr['data']
-    values = {'MAX': '(automatic)', 'KIND': '(automatic)',
-              'TYPE': '(automatic)'}
+    values = {'MAX': '(automatic)'}
 
     # Two types of unions, determined by discriminator.
 
@@ -607,19 +606,13 @@ def check_union(expr, expr_info):
                                " of branch '%s'" % key)
 
         # If the discriminator names an enum type, then all members
-        # of 'data' must also be members of the enum type, which in turn
-        # must not collide with the discriminator name.
+        # of 'data' must also be members of the enum type.
         if enum_define:
             if key not in enum_define['enum_values']:
                 raise QAPIExprError(expr_info,
                                     "Discriminator value '%s' is not found in "
                                     "enum '%s'" %
                                     (key, enum_define["enum_name"]))
-            if discriminator in enum_define['enum_values']:
-                raise QAPIExprError(expr_info,
-                                    "Discriminator name '%s' collides with "
-                                    "enum value in '%s'" %
-                                    (discriminator, enum_define["enum_name"]))
 
         # Otherwise, check for conflicts in the generated enum
         else:
tests/Makefile +0 -3
index 92969e8..c84c6cb 100644
@@ -277,9 +277,7 @@ qapi-schema += flat-union-bad-base.json
 qapi-schema += flat-union-bad-discriminator.json
 qapi-schema += flat-union-base-any.json
 qapi-schema += flat-union-base-union.json
-qapi-schema += flat-union-clash-branch.json
 qapi-schema += flat-union-clash-member.json
-qapi-schema += flat-union-clash-type.json
 qapi-schema += flat-union-empty.json
 qapi-schema += flat-union-inline.json
 qapi-schema += flat-union-int-branch.json
@@ -341,7 +339,6 @@ qapi-schema += union-bad-branch.json
 qapi-schema += union-base-no-discriminator.json
 qapi-schema += union-clash-branches.json
 qapi-schema += union-clash-data.json
-qapi-schema += union-clash-type.json
 qapi-schema += union-empty.json
 qapi-schema += union-invalid-base.json
 qapi-schema += union-max.json
tests/qapi-schema/flat-union-clash-branch.err deleted +0 -0
deleted file mode 100644
index e69de29..0000000
tests/qapi-schema/flat-union-clash-branch.exit deleted +0 -1
deleted file mode 100644
index 573541a..0000000
@@ -1 +0,0 @@
-0
tests/qapi-schema/flat-union-clash-branch.json deleted +0 -18
deleted file mode 100644
index e593336..0000000
@@ -1,18 +0,0 @@
-# Flat union branch name collision
-# FIXME: this parses, but then fails to compile due to a duplicate 'c_d'
-# (one from the base member, the other from the branch name).  We should
-# either reject the collision at parse time, or munge the generated branch
-# name to allow this to compile.
-{ 'enum': 'TestEnum',
-  'data': [ 'base', 'c-d' ] }
-{ 'struct': 'Base',
-  'data': { 'enum1': 'TestEnum', '*c_d': 'str' } }
-{ 'struct': 'Branch1',
-  'data': { 'string': 'str' } }
-{ 'struct': 'Branch2',
-  'data': { 'value': 'int' } }
-{ 'union': 'TestUnion',
-  'base': 'Base',
-  'discriminator': 'enum1',
-  'data': { 'base': 'Branch1',
-            'c-d': 'Branch2' } }
tests/qapi-schema/flat-union-clash-branch.out deleted +0 -14
deleted file mode 100644
index 8e0da73..0000000
@@ -1,14 +0,0 @@
-object :empty
-object Base
-    member enum1: TestEnum optional=False
-    member c_d: str optional=True
-object Branch1
-    member string: str optional=False
-object Branch2
-    member value: int optional=False
-enum TestEnum ['base', 'c-d']
-object TestUnion
-    base Base
-    tag enum1
-    case base: Branch1
-    case c-d: Branch2
tests/qapi-schema/flat-union-clash-type.err deleted +0 -1
deleted file mode 100644
index b44dd40..0000000
@@ -1 +0,0 @@
-tests/qapi-schema/flat-union-clash-type.json:11: Discriminator name 'type' collides with enum value in 'TestEnum'
tests/qapi-schema/flat-union-clash-type.exit deleted +0 -1
deleted file mode 100644
index d00491f..0000000
@@ -1 +0,0 @@
-1
tests/qapi-schema/flat-union-clash-type.json deleted +0 -14
deleted file mode 100644
index 8f710f0..0000000
@@ -1,14 +0,0 @@
-# Flat union branch 'type'
-# Reject this, because we would have a clash in generated C, between the
-# outer tag 'type' and the branch name 'type' within the union.
-# TODO: We could munge the generated C branch name to let it compile.
-{ 'enum': 'TestEnum',
-  'data': [ 'type' ] }
-{ 'struct': 'Base',
-  'data': { 'type': 'TestEnum' } }
-{ 'struct': 'Branch1',
-  'data': { 'string': 'str' } }
-{ 'union': 'TestUnion',
-  'base': 'Base',
-  'discriminator': 'type',
-  'data': { 'type': 'Branch1' } }
tests/qapi-schema/flat-union-clash-type.out deleted +0 -0
deleted file mode 100644
index e69de29..0000000
tests/qapi-schema/union-clash-type.err deleted +0 -1
deleted file mode 100644
index a5dead1..0000000
@@ -1 +0,0 @@
-tests/qapi-schema/union-clash-type.json:8: Union 'TestUnion' member 'kind' clashes with '(automatic)'
tests/qapi-schema/union-clash-type.exit deleted +0 -1
deleted file mode 100644
index d00491f..0000000
@@ -1 +0,0 @@
-1
tests/qapi-schema/union-clash-type.json deleted +0 -9
deleted file mode 100644
index cfc256b..0000000
@@ -1,9 +0,0 @@
-# Union branch 'type'
-# Reject this, because we would have a clash in generated C, between the
-# simple union's implicit tag member 'kind' and the branch name 'kind'
-# within the union.
-# TODO: Even when the generated C is switched to use 'type' rather than
-# 'kind', to match the QMP spelling, the collision should still be detected.
-# Or, we could munge the branch name to allow compilation.
-{ 'union': 'TestUnion',
-  'data': { 'kind': 'int', 'type': 'str' } }
tests/qapi-schema/union-clash-type.out deleted +0 -0
deleted file mode 100644
index e69de29..0000000
Signature · 1 line
2.4.3
author
#64 lore raw

PATCH v10 27/30 qapi: Track owner of each object member

In reply to Eric Blake

Future commits will migrate semantic checking away from parsing and over to the various QAPISchema*.check() methods. But to report an error message about an incorrect semantic use of a member of an object type, it helps to know which type, command, or event owns the member. In particular, when a member is inherited from a base type, it is desirable to associate the member name with the base type (and not the type calling member.check()).

Rather than packing additional information into the seen array passed to each member.check() (as in seen[m.name] = {'member':m, 'owner':type}), it is easier to have each member track the name of the owner type in the first place (keeping things simpler with the existing seen[m.name] = m). The new member.owner field is set via a new set_owner() method, called when registering the members and variants arrays with an object or variant type. Track only a name, and not the actual type object, to avoid creating a circular python reference chain.

Note that Variants.set_owner() method does not set the owner for the tag_member field; this field is set earlier either as part of an object's non-variant members, or explicitly by alternates.

The source information is intended for human consumption in error messages, and a new describe() method is added to access the resulting information. For example, given the qapi:
{ 'command': 'foo', 'data': { 'string': 'str' } }
an implementation of visit_command() that calls
arg_type.members[0].describe()
will see "'string' (argument of foo)".

To make the human-readable name of implicit types work without duplicating efforts, the describe() method has to reverse the name of implicit types, via the helper _pretty_owner(), plus a tweak to report event data separately from command arguments.

No change to generated code.

---
v10 (now in subset C): rebase to latest, set alternate tag_member owner from alternate
v9 (now in subset D): rebase to earlier changes, hoist 'role' to top of class, split out _pretty_helper(), manage owner when tag_member appears as part of local_members for unions
v8: don't munge implicit type names [except for event data], and instead make describe() create nicer messages. Add set_owner(), and use field 'role' instead of method _describe()
v7: total rewrite: rework implicit object names, assign owner when initializing owner type rather than when creating member python object
v6: rebase on new lazy array creation and simple union 'type' motion; tweak commit message

---
 scripts/qapi.py                        | 44 +++++++++++++++++++++++++++++++---
 tests/qapi-schema/qapi-schema-test.out |  8 +++----
 2 files changed, 45 insertions(+), 7 deletions(-)
scripts/qapi.py +41 -3
index b914785..b3af973 100644
@@ -957,8 +957,10 @@ class QAPISchemaObjectType(QAPISchemaType):
         assert base is None or isinstance(base, str)
         for m in local_members:
             assert isinstance(m, QAPISchemaObjectTypeMember)
-        assert (variants is None or
-                isinstance(variants, QAPISchemaObjectTypeVariants))
+            m.set_owner(name)
+        if variants is not None:
+            assert isinstance(variants, QAPISchemaObjectTypeVariants)
+            variants.set_owner(name)
         self._base_name = base
         self.base = None
         self.local_members = local_members
@@ -1012,6 +1014,8 @@ class QAPISchemaObjectType(QAPISchemaType):
 
 
 class QAPISchemaObjectTypeMember(object):
+    role = 'member'
+
     def __init__(self, name, typ, optional):
         assert isinstance(name, str)
         assert isinstance(typ, str)
@@ -1020,8 +1024,14 @@ class QAPISchemaObjectTypeMember(object):
         self._type_name = typ
         self.type = None
         self.optional = optional
+        self.owner = None
+
+    def set_owner(self, name):
+        assert not self.owner
+        self.owner = name
 
     def check(self, schema):
+        assert self.owner
         self.type = schema.lookup_type(self._type_name)
         assert self.type
 
@@ -1030,6 +1040,25 @@ class QAPISchemaObjectTypeMember(object):
         assert self.name not in seen
         seen[self.name] = self
 
+    def _pretty_owner(self):
+        # See QAPISchema._make_implicit_object_type() - reverse the
+        # mapping there to create a nice human-readable description
+        owner = self.owner
+        if owner.startswith(':obj-'):
+            owner = owner[5:]
+            if owner.endswith('-arg'):
+                return '(argument of %s)' % owner[:-4]
+            elif owner.endswith('-data'):
+                return '(data of %s)' % owner[:-5]
+            else:
+                assert owner.endswith('-wrapper')
+                return '(branch of %s)' % owner[:-8]
+        else:
+            return '(%s of %s)' % (self.role, owner)
+
+    def describe(self):
+        return "'%s' %s" % (self.name, self._pretty_owner())
+
 
 class QAPISchemaObjectTypeVariants(object):
     def __init__(self, tag_name, tag_member, variants):
@@ -1046,6 +1075,10 @@ class QAPISchemaObjectTypeVariants(object):
         self.tag_member = tag_member
         self.variants = variants
 
+    def set_owner(self, name):
+        for v in self.variants:
+            v.set_owner(name)
+
     def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
@@ -1063,6 +1096,8 @@ class QAPISchemaObjectTypeVariants(object):
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
+    role = 'branch'
+
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
@@ -1082,9 +1117,11 @@ class QAPISchemaAlternateType(QAPISchemaType):
         QAPISchemaType.__init__(self, name, info)
         assert isinstance(variants, QAPISchemaObjectTypeVariants)
         assert not variants.tag_name
+        variants.set_owner(name)
         self.variants = variants
 
     def check(self, schema):
+        self.variants.tag_member.set_owner(self.name)
         self.variants.tag_member.check(schema)
         self.variants.check(schema, {})
 
@@ -1212,6 +1249,7 @@ class QAPISchema(object):
     def _make_implicit_object_type(self, name, info, role, members):
         if not members:
             return None
+        # See also QAPISchemaObjectTypeMember.describe()
         name = ':obj-%s-%s' % (name, role)
         if not self.lookup_entity(name, QAPISchemaObjectType):
             self._def_entity(QAPISchemaObjectType(name, info, None,
@@ -1315,7 +1353,7 @@ class QAPISchema(object):
         data = expr.get('data')
         if isinstance(data, OrderedDict):
             data = self._make_implicit_object_type(
-                name, info, 'arg', self._make_members(data, info))
+                name, info, 'data', self._make_members(data, info))
         self._def_entity(QAPISchemaEvent(name, info, data))
 
     def _def_exprs(self):
tests/qapi-schema/qapi-schema-test.out +4 -4
index 786024e..f78ef04 100644
@@ -1,9 +1,9 @@
 object :empty
-object :obj-EVENT_C-arg
+object :obj-EVENT_C-data
     member a: int optional=True
     member b: UserDefOne optional=True
     member c: str optional=False
-object :obj-EVENT_D-arg
+object :obj-EVENT_D-data
     member a: EventStructOne optional=False
     member b: str optional=False
     member c: str optional=True
@@ -79,8 +79,8 @@ alternate AltStrNum
 enum AltStrNumKind ['s', 'n']
 event EVENT_A None
 event EVENT_B None
-event EVENT_C :obj-EVENT_C-arg
-event EVENT_D :obj-EVENT_D-arg
+event EVENT_C :obj-EVENT_C-data
+event EVENT_D :obj-EVENT_D-data
 object Empty1
 object Empty2
     base Empty1
Signature · 1 line
2.4.3
#65 lore raw

Re: [Qemu-devel] [PATCH v10 27/30] qapi: Track owner of each object member

Quoted 102 lines · Eric Blake

Future commits will migrate semantic checking away from parsing and over to the various QAPISchema*.check() methods. But to report an error message about an incorrect semantic use of a member of an object type, it helps to know which type, command, or event owns the member. In particular, when a member is inherited from a base type, it is desirable to associate the member name with the base type (and not the type calling member.check()).

Rather than packing additional information into the seen array passed to each member.check() (as in seen[m.name] = {'member':m, 'owner':type}), it is easier to have each member track the name of the owner type in the first place (keeping things simpler with the existing seen[m.name] = m). The new member.owner field is set via a new set_owner() method, called when registering the members and variants arrays with an object or variant type. Track only a name, and not the actual type object, to avoid creating a circular python reference chain.

Note that Variants.set_owner() method does not set the owner for the tag_member field; this field is set earlier either as part of an object's non-variant members, or explicitly by alternates.

The source information is intended for human consumption in error messages, and a new describe() method is added to access the resulting information. For example, given the qapi:
{ 'command': 'foo', 'data': { 'string': 'str' } }
an implementation of visit_command() that calls
arg_type.members[0].describe()
will see "'string' (argument of foo)".

To make the human-readable name of implicit types work without duplicating efforts, the describe() method has to reverse the name of implicit types, via the helper _pretty_owner(), plus a tweak to report event data separately from command arguments.

No change to generated code.

---
v10 (now in subset C): rebase to latest, set alternate tag_member owner from alternate
v9 (now in subset D): rebase to earlier changes, hoist 'role' to top of class, split out _pretty_helper(), manage owner when tag_member appears as part of local_members for unions
v8: don't munge implicit type names [except for event data], and instead make describe() create nicer messages. Add set_owner(), and use field 'role' instead of method _describe()
v7: total rewrite: rework implicit object names, assign owner when initializing owner type rather than when creating member python object
v6: rebase on new lazy array creation and simple union 'type' motion; tweak commit message

---
 scripts/qapi.py                        | 44 +++++++++++++++++++++++++++++++---
 tests/qapi-schema/qapi-schema-test.out |  8 +++----
 2 files changed, 45 insertions(+), 7 deletions(-)
scripts/qapi.py +12 -2
index b914785..b3af973 100644
@@ -957,8 +957,10 @@ class QAPISchemaObjectType(QAPISchemaType):
         assert base is None or isinstance(base, str)
         for m in local_members:
             assert isinstance(m, QAPISchemaObjectTypeMember)
-        assert (variants is None or
-                isinstance(variants, QAPISchemaObjectTypeVariants))
+            m.set_owner(name)
+        if variants is not None:
+            assert isinstance(variants, QAPISchemaObjectTypeVariants)
+            variants.set_owner(name)
         self._base_name = base
         self.base = None
         self.local_members = local_members
@@ -1012,6 +1014,8 @@ class QAPISchemaObjectType(QAPISchemaType):
 
 
 class QAPISchemaObjectTypeMember(object):
+    role = 'member'
+
     def __init__(self, name, typ, optional):
         assert isinstance(name, str)
         assert isinstance(typ, str)
@@ -1020,8 +1024,14 @@ class QAPISchemaObjectTypeMember(object):
         self._type_name = typ
         self.type = None
         self.optional = optional
+        self.owner = None
+
+    def set_owner(self, name):
+        assert not self.owner
+        self.owner = name
 
     def check(self, schema):
+        assert self.owner
         self.type = schema.lookup_type(self._type_name)
         assert self.type
 

scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1045

Quoted 6 lines
scripts/qapi.py +3 -0
@@ -1030,6 +1040,25 @@ class QAPISchemaObjectTypeMember(object):
         assert self.name not in seen
         seen[self.name] = self
 
+    def _pretty_owner(self):
+        # See QAPISchema._make_implicit_object_type() - reverse the
+        # mapping there to create a nice human-readable description

This comment confused me briefly. It applies to the following conditionals if-part, but not its else-part. Move it into the if-part?

scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1055

Quoted 10 lines
scripts/qapi.py +10 -0
+        owner = self.owner
+        if owner.startswith(':obj-'):
+            owner = owner[5:]
+            if owner.endswith('-arg'):
+                return '(argument of %s)' % owner[:-4]
+            elif owner.endswith('-data'):
+                return '(data of %s)' % owner[:-5]
+            else:
+                assert owner.endswith('-wrapper')
+                return '(branch of %s)' % owner[:-8]

See last remark.

Quoted 31 lines
scripts/qapi.py +12 -0
+        else:
+            return '(%s of %s)' % (self.role, owner)
+
+    def describe(self):
+        return "'%s' %s" % (self.name, self._pretty_owner())
+
 
 class QAPISchemaObjectTypeVariants(object):
     def __init__(self, tag_name, tag_member, variants):
@@ -1046,6 +1075,10 @@ class QAPISchemaObjectTypeVariants(object):
         self.tag_member = tag_member
         self.variants = variants
 
+    def set_owner(self, name):
+        for v in self.variants:
+            v.set_owner(name)
+
     def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
@@ -1063,6 +1096,8 @@ class QAPISchemaObjectTypeVariants(object):
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
+    role = 'branch'
+
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
@@ -1082,9 +1117,11 @@ class QAPISchemaAlternateType(QAPISchemaType):
         QAPISchemaType.__init__(self, name, info)

scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1127

Quoted 10 lines
scripts/qapi.py +2 -0
         assert isinstance(variants, QAPISchemaObjectTypeVariants)
         assert not variants.tag_name
+        variants.set_owner(name)
         self.variants = variants
 
     def check(self, schema):
+        self.variants.tag_member.set_owner(self.name)
         self.variants.tag_member.check(schema)
         self.variants.check(schema, {})
 

Odd: all other .set_owner() calls are in .__init__() or .set_owner(). Can we move this one to __init__() for consistency?

I think we can: __init__() requires its variants argument to have a tag_member (it even asserts not variants.tag_name).

scripts/qapi.py class QAPISchema(object): · line 1252

Quoted 5 lines
scripts/qapi.py +1 -0
@@ -1212,6 +1249,7 @@ class QAPISchema(object):
     def _make_implicit_object_type(self, name, info, role, members):
         if not members:
             return None
+        # See also QAPISchemaObjectTypeMember.describe()

Should this point to ._pretty_owner() instead?

scripts/qapi.py class QAPISchema(object): · line 1356

Quoted 9 lines
scripts/qapi.py +1 -1
         name = ':obj-%s-%s' % (name, role)
         if not self.lookup_entity(name, QAPISchemaObjectType):
             self._def_entity(QAPISchemaObjectType(name, info, None,
@@ -1315,7 +1353,7 @@ class QAPISchema(object):
         data = expr.get('data')
         if isinstance(data, OrderedDict):
             data = self._make_implicit_object_type(
-                name, info, 'arg', self._make_members(data, info))
+                name, info, 'data', self._make_members(data, info))

This is necessary only to make .pretty_owner() say 'data of EVENT_NAME' instead of 'argument of EVENT_NAME'. Do we really need different wording for commands and events?

I'd make it say 'parameter of' for both commands and events. I habitually use "parameter" for formal parameters, and "argument" for actual arguments. Guess I've worked with the C standard too much...

For what it's worth, the QAPI schema language uses 'data' for both (and many other things, too), and the introspection schema uses 'arg-type' for both.

Quoted 30 lines
scripts/qapi.py +0 -0
         self._def_entity(QAPISchemaEvent(name, info, data))
 
     def _def_exprs(self):
tests/qapi-schema/qapi-schema-test.out +4 -4
index 786024e..f78ef04 100644
@@ -1,9 +1,9 @@
 object :empty
-object :obj-EVENT_C-arg
+object :obj-EVENT_C-data
     member a: int optional=True
     member b: UserDefOne optional=True
     member c: str optional=False
-object :obj-EVENT_D-arg
+object :obj-EVENT_D-data
     member a: EventStructOne optional=False
     member b: str optional=False
     member c: str optional=True
@@ -79,8 +79,8 @@ alternate AltStrNum
 enum AltStrNumKind ['s', 'n']
 event EVENT_A None
 event EVENT_B None
-event EVENT_C :obj-EVENT_C-arg
-event EVENT_D :obj-EVENT_D-arg
+event EVENT_C :obj-EVENT_C-data
+event EVENT_D :obj-EVENT_D-data
 object Empty1
 object Empty2
     base Empty1
author
#66 lore raw

Re: [Qemu-devel] [PATCH v10 27/30] qapi: Track owner of each object member

Quoted 13 lines · Markus Armbruster
Quoted 11 lines · Eric Blake

Future commits will migrate semantic checking away from parsing and over to the various QAPISchema*.check() methods. But to report an error message about an incorrect semantic use of a member of an object type, it helps to know which type, command, or event owns the member. In particular, when a member is inherited from a base type, it is desirable to associate the member name with the base type (and not the type calling member.check()).

Quoted 6 lines

scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1045

Quoted 3 lines
scripts/qapi.py +3 -0
+    def _pretty_owner(self):
+        # See QAPISchema._make_implicit_object_type() - reverse the
+        # mapping there to create a nice human-readable description

This comment confused me briefly. It applies to the following conditionals if-part, but not its else-part. Move it into the if-part?

Done.

Quoted 7 lines
Quoted 7 lines
scripts/qapi.py +1 -0
@@ -1082,9 +1117,11 @@ class QAPISchemaAlternateType(QAPISchemaType):
         QAPISchemaType.__init__(self, name, info)
         assert isinstance(variants, QAPISchemaObjectTypeVariants)
         assert not variants.tag_name
+        variants.set_owner(name)
         self.variants = variants
 
Quoted 8 lines

scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1127

Quoted 5 lines
scripts/qapi.py +1 -0
     def check(self, schema):
+        self.variants.tag_member.set_owner(self.name)
         self.variants.tag_member.check(schema)
         self.variants.check(schema, {})
 

Odd: all other .set_owner() calls are in .__init__() or .set_owner(). Can we move this one to __init__() for consistency?

Yes, done.

Quoted 11 lines

I think we can: __init__() requires its variants argument to have a tag_member (it even asserts not variants.tag_name).

scripts/qapi.py class QAPISchema(object): · line 1252

Quoted 5 lines
scripts/qapi.py +1 -0
@@ -1212,6 +1249,7 @@ class QAPISchema(object):
     def _make_implicit_object_type(self, name, info, role, members):
         if not members:
             return None
+        # See also QAPISchemaObjectTypeMember.describe()

Should this point to ._pretty_owner() instead?

Good call.

Quoted 15 lines

scripts/qapi.py class QAPISchema(object): · line 1356

Quoted 10 lines
scripts/qapi.py +1 -1
         name = ':obj-%s-%s' % (name, role)
         if not self.lookup_entity(name, QAPISchemaObjectType):
             self._def_entity(QAPISchemaObjectType(name, info, None,
@@ -1315,7 +1353,7 @@ class QAPISchema(object):
         data = expr.get('data')
         if isinstance(data, OrderedDict):
             data = self._make_implicit_object_type(
-                name, info, 'arg', self._make_members(data, info))
+                name, info, 'data', self._make_members(data, info))

This is necessary only to make .pretty_owner() say 'data of EVENT_NAME' instead of 'argument of EVENT_NAME'. Do we really need different wording for commands and events?

Quoted 3 lines

I'd make it say 'parameter of' for both commands and events. I habitually use "parameter" for formal parameters, and "argument" for actual arguments. Guess I've worked with the C standard too much...

'parameter' sounds better for both, at which point the messages no longer differ,...

Quoted 7 lines

For what it's worth, the QAPI schema language uses 'data' for both (and many other things, too), and the introspection schema uses 'arg-type' for both.

Quoted 2 lines
scripts/qapi.py +0 -0
         self._def_entity(QAPISchemaEvent(name, info, data))
 
Quoted 14 lines
Quoted 6 lines
scripts/qapi.py +0 -0
     def _def_exprs(self):
tests/qapi-schema/qapi-schema-test.out +0 -0
index 786024e..f78ef04 100644

tests/qapi-schema/qapi-schema-test.out line 6

Quoted 8 lines
tests/qapi-schema/qapi-schema-test.out +2 -2
@@ -1,9 +1,9 @@
 object :empty
-object :obj-EVENT_C-arg
+object :obj-EVENT_C-data
     member a: int optional=True
     member b: UserDefOne optional=True
     member c: str optional=False
-object :obj-EVENT_D-arg
+object :obj-EVENT_D-data

...so I no longer need to rename the implicit struct for events to allow the differentiation. v11 is thus a smaller patch :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#67 lore raw

PATCH v10 28/30 qapi: Detect collisions in C member names

In reply to Eric Blake

Detect attempts to declare two object members that would result in the same C member name, by keying the 'seen' dictionary off of the C name rather than the qapi name. It also requires passing info through the check_clash() methods.

This addresses a TODO and fixes the previously-broken args-name-clash test. The resulting error message demonstrates the utility of the .describe() method added previously. No change to generated code.

---
v10 (now in subset C): rebase to latest; update commit message v9 (now in subset D): rebase to earlier changes, now only one test affected
v8: rebase to earlier changes
v7: split out error reporting prep and member.c_name() addition
v6: rebase to earlier testsuite and info improvements

---
 scripts/qapi.py                        | 31 +++++++++++++++++++------------
 tests/qapi-schema/args-name-clash.err  |  1 +
 tests/qapi-schema/args-name-clash.exit |  2 +-
 tests/qapi-schema/args-name-clash.json |  6 +++---
 tests/qapi-schema/args-name-clash.out  |  6 ------
 5 files changed, 24 insertions(+), 22 deletions(-)
scripts/qapi.py +19 -12
index b3af973..3a359cb 100644
@@ -975,21 +975,24 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            self.base.check_clash(schema, seen)
+            self.base.check_clash(schema, self.info, seen)
         for m in self.local_members:
             m.check(schema)
-            m.check_clash(seen)
+            m.check_clash(self.info, seen)
         self.members = seen.values()
         if self.variants:
             self.variants.check(schema, seen)
             assert self.variants.tag_member in self.members
-            self.variants.check_clash(schema, seen)
+            self.variants.check_clash(schema, self.info, seen)
 
-    def check_clash(self, schema, seen):
+    # Check that the members of this type do not cause duplicate JSON fields,
+    # and update seen to track the members seen so far. Report any errors
+    # on behalf of info, which is not necessarily self.info
+    def check_clash(self, schema, info, seen):
         self.check(schema)
         assert not self.variants       # not implemented
         for m in self.members:
-            m.check_clash(seen)
+            m.check_clash(info, seen)
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
@@ -1035,10 +1038,13 @@ class QAPISchemaObjectTypeMember(object):
         self.type = schema.lookup_type(self._type_name)
         assert self.type
 
-    def check_clash(self, seen):
-        # TODO change key of seen from QAPI name to C name
-        assert self.name not in seen
-        seen[self.name] = self
+    def check_clash(self, info, seen):
+        name = c_name(self.name)
+        if name in seen:
+            raise QAPIExprError(info,
+                                "%s collides with %s"
+                                % (self.describe(), seen[name].describe()))
+        seen[name] = self
 
     def _pretty_owner(self):
         # See QAPISchema._make_implicit_object_type() - reverse the
@@ -1081,18 +1087,19 @@ class QAPISchemaObjectTypeVariants(object):
 
     def check(self, schema, seen):
         if self.tag_name:    # flat union
-            self.tag_member = seen[self.tag_name]
+            self.tag_member = seen[c_name(self.tag_name)]
+            assert self.tag_name == self.tag_member.name
         tag_type = self.tag_member.type
         assert isinstance(tag_type, QAPISchemaEnumType)
         for v in self.variants:
             v.check(schema)
             assert v.name in tag_type.values
 
-    def check_clash(self, schema, seen):
+    def check_clash(self, schema, info, seen):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
-            v.type.check_clash(schema, dict(seen))
+            v.type.check_clash(schema, info, dict(seen))
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
tests/qapi-schema/args-name-clash.err +1 -0
index e69de29..2735217 100644
@@ -0,0 +1 @@
+tests/qapi-schema/args-name-clash.json:5: 'a_b' (argument of oops) collides with 'a-b' (argument of oops)
tests/qapi-schema/args-name-clash.exit +1 -1
index 573541a..d00491f 100644
@@ -1 +1 @@
-0
+1
tests/qapi-schema/args-name-clash.json +3 -3
index 9e8f889..3fe4ea5 100644
@@ -1,5 +1,5 @@
 # C member name collision
-# FIXME - This parses, but fails to compile, because the C struct is given
-# two 'a_b' members.  Either reject this at parse time, or munge the C names
-# to avoid the collision.
+# Reject members that clash when mapped to C names (we would have two 'a_b'
+# members). It would also be possible to munge the C names to avoid the
+# collision, but unlikely to be worth the effort.
 { 'command': 'oops', 'data': { 'a-b': 'str', 'a_b': 'str' } }
tests/qapi-schema/args-name-clash.out +0 -6
index 9b2f6e4..e69de29 100644
@@ -1,6 +0,0 @@
-object :empty
-object :obj-oops-arg
-    member a-b: str optional=False
-    member a_b: str optional=False
-command oops :obj-oops-arg -> None
-   gen=True success_response=True
Signature · 1 line
2.4.3
#68 lore raw

Re: [Qemu-devel] [PATCH v10 28/30] qapi: Detect collisions in C member names

Quoted 32 lines · Eric Blake

Detect attempts to declare two object members that would result in the same C member name, by keying the 'seen' dictionary off of the C name rather than the qapi name. It also requires passing info through the check_clash() methods.

This addresses a TODO and fixes the previously-broken args-name-clash test. The resulting error message demonstrates the utility of the .describe() method added previously. No change to generated code.

---
v10 (now in subset C): rebase to latest; update commit message v9 (now in subset D): rebase to earlier changes, now only one test affected
v8: rebase to earlier changes
v7: split out error reporting prep and member.c_name() addition
v6: rebase to earlier testsuite and info improvements

---
 scripts/qapi.py                        | 31 +++++++++++++++++++------------
 tests/qapi-schema/args-name-clash.err  |  1 +
 tests/qapi-schema/args-name-clash.exit |  2 +-
 tests/qapi-schema/args-name-clash.json |  6 +++---
 tests/qapi-schema/args-name-clash.out  |  6 ------
 5 files changed, 24 insertions(+), 22 deletions(-)
scripts/qapi.py +0 -0
index b3af973..3a359cb 100644

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 978

Quoted 5 lines
scripts/qapi.py +1 -1
@@ -975,21 +975,24 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            self.base.check_clash(schema, seen)
+            self.base.check_clash(schema, self.info, seen)

Note that if this one reports an error for self.info, something went wrong. We first run self.base.check(), which we survive only when self.base doesn't have clashing members. Would be easier to see if self.base.check() wasn't hiding in self.base.check_clash().

Quoted 5 lines
scripts/qapi.py +1 -1
         for m in self.local_members:
             m.check(schema)
-            m.check_clash(seen)
+            m.check_clash(self.info, seen)
         self.members = seen.values()

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 990

Quoted 10 lines
scripts/qapi.py +4 -2
         if self.variants:
             self.variants.check(schema, seen)
             assert self.variants.tag_member in self.members
-            self.variants.check_clash(schema, seen)
+            self.variants.check_clash(schema, self.info, seen)
 
-    def check_clash(self, schema, seen):
+    # Check that the members of this type do not cause duplicate JSON fields,
+    # and update seen to track the members seen so far. Report any errors
+    # on behalf of info, which is not necessarily self.info

Do we actually need info != self.info? If yes, test case? If no, should the comment be adjusted?

Quoted 10 lines
scripts/qapi.py +2 -1
+    def check_clash(self, schema, info, seen):
         self.check(schema)
         assert not self.variants       # not implemented
         for m in self.members:
-            m.check_clash(seen)
+            m.check_clash(info, seen)
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()

scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1042

Quoted 9 lines
scripts/qapi.py +2 -4
@@ -1035,10 +1038,13 @@ class QAPISchemaObjectTypeMember(object):
         self.type = schema.lookup_type(self._type_name)
         assert self.type
 
-    def check_clash(self, seen):
-        # TODO change key of seen from QAPI name to C name
-        assert self.name not in seen
-        seen[self.name] = self
+    def check_clash(self, info, seen):
+        name = c_name(self.name)

I'd call it cname. Matter of taste, of course.

Quoted 18 lines
scripts/qapi.py +7 -1
+        if name in seen:
+            raise QAPIExprError(info,
+                                "%s collides with %s"
+                                % (self.describe(), seen[name].describe()))
+        seen[name] = self
 
     def _pretty_owner(self):
         # See QAPISchema._make_implicit_object_type() - reverse the
@@ -1081,18 +1087,19 @@ class QAPISchemaObjectTypeVariants(object):
 
     def check(self, schema, seen):
         if self.tag_name:    # flat union
-            self.tag_member = seen[self.tag_name]
+            self.tag_member = seen[c_name(self.tag_name)]
+            assert self.tag_name == self.tag_member.name
         tag_type = self.tag_member.type
         assert isinstance(tag_type, QAPISchemaEnumType)
         for v in self.variants:

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1102

Quoted 10 lines
scripts/qapi.py +2 -2
             v.check(schema)
             assert v.name in tag_type.values
 
-    def check_clash(self, schema, seen):
+    def check_clash(self, schema, info, seen):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
-            v.type.check_clash(schema, dict(seen))
+            v.type.check_clash(schema, info, dict(seen))

Since we can't inherit variant members, info is always the owning object type's info.

Quoted 21 lines
scripts/qapi.py +0 -0
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
tests/qapi-schema/args-name-clash.err +1 -0
index e69de29..2735217 100644
@@ -0,0 +1 @@
+tests/qapi-schema/args-name-clash.json:5: 'a_b' (argument of oops) collides with 'a-b' (argument of oops)
tests/qapi-schema/args-name-clash.exit +1 -1
index 573541a..d00491f 100644
@@ -1 +1 @@
-0
+1
tests/qapi-schema/args-name-clash.json +0 -0
index 9e8f889..3fe4ea5 100644

tests/qapi-schema/args-name-clash.json line 4

Quoted 7 lines
tests/qapi-schema/args-name-clash.json +3 -3
@@ -1,5 +1,5 @@
 # C member name collision
-# FIXME - This parses, but fails to compile, because the C struct is given
-# two 'a_b' members.  Either reject this at parse time, or munge the C names
-# to avoid the collision.
+# Reject members that clash when mapped to C names (we would have two 'a_b'
+# members). It would also be possible to munge the C names to avoid the
+# collision, but unlikely to be worth the effort.

I'd drop the second sentence.

Quoted 12 lines
tests/qapi-schema/args-name-clash.json +0 -0
 { 'command': 'oops', 'data': { 'a-b': 'str', 'a_b': 'str' } }
tests/qapi-schema/args-name-clash.out +0 -6
index 9b2f6e4..e69de29 100644
@@ -1,6 +0,0 @@
-object :empty
-object :obj-oops-arg
-    member a-b: str optional=False
-    member a_b: str optional=False
-command oops :obj-oops-arg -> None
-   gen=True success_response=True
author
#69 lore raw

Re: [Qemu-devel] [PATCH v10 28/30] qapi: Detect collisions in C member names

Quoted 16 lines · Markus Armbruster
Quoted 13 lines · Eric Blake

Detect attempts to declare two object members that would result in the same C member name, by keying the 'seen' dictionary off of the C name rather than the qapi name. It also requires passing info through the check_clash() methods.

This addresses a TODO and fixes the previously-broken args-name-clash test. The resulting error message demonstrates the utility of the .describe() method added previously. No change to generated code.

---

Quoted 12 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 978

Quoted 7 lines

+++ b/scripts/qapi.py

scripts/qapi.py +1 -1
@@ -975,21 +975,24 @@ class QAPISchemaObjectType(QAPISchemaType):
         seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
-            self.base.check_clash(schema, seen)
+            self.base.check_clash(schema, self.info, seen)

Note that if this one reports an error for self.info, something went wrong. We first run self.base.check(), which we survive only when self.base doesn't have clashing members. Would be easier to see if self.base.check() wasn't hiding in self.base.check_clash().

Hmm. I could pass None as a way of asserting that it won't fail. But I think that's a bit more confusing, so I'll probably just leave it alone.

Quoted 7 lines

scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 990

Quoted 4 lines
scripts/qapi.py +3 -1
-    def check_clash(self, schema, seen):
+    # Check that the members of this type do not cause duplicate JSON fields,
+    # and update seen to track the members seen so far. Report any errors
+    # on behalf of info, which is not necessarily self.info

Do we actually need info != self.info? If yes, test case? If no, should the comment be adjusted?

The comment is correct. For a flat union, if we have a variant associated with type 'Foo', and type 'Foo' has no clashes itself, but one of its members duplicates a member already present from the base of union, then we are calling Foo.check_clash(schema, Union.info, seen), so that the error message points to the location of Union (rather than Foo) as the place in the .json file introducing the clash.

flat-union-clash-member.json tests this; it's just that we first have to remove ad hoc testing before we can trigger this case. It's on my queue, just not in this particular posting of subset C:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg07000.html

Quoted 4 lines
Quoted 4 lines
scripts/qapi.py +0 -0
@@ -1035,10 +1038,13 @@ class QAPISchemaObjectTypeMember(object):
         self.type = schema.lookup_type(self._type_name)
         assert self.type
 
Quoted 8 lines

scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1042

Quoted 6 lines
scripts/qapi.py +2 -4
-    def check_clash(self, seen):
-        # TODO change key of seen from QAPI name to C name
-        assert self.name not in seen
-        seen[self.name] = self
+    def check_clash(self, info, seen):
+        name = c_name(self.name)

I'd call it cname. Matter of taste, of course.

Sure, that works. I first tried calling it c_name, but then python thought I was trying to redefine the global c_name().

Quoted 11 lines

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1102

Quoted 8 lines
scripts/qapi.py +2 -2
-    def check_clash(self, schema, seen):
+    def check_clash(self, schema, info, seen):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
-            v.type.check_clash(schema, dict(seen))
+            v.type.check_clash(schema, info, dict(seen))

Since we can't inherit variant members, info is always the owning object type's info.

Exactly. We want to output the passed-in info, and NOT v.type.info, so that errors are reported on behalf of the union that owns the variant, rather than at the location of the variant's type which independently passed .check() and therefore has no internal collisions.

And someday we might be able to inherit from objects with variants, but that's a lot further down the pipeline (if at all).

Quoted 11 lines

tests/qapi-schema/args-name-clash.json line 4

Quoted 9 lines

+++ b/tests/qapi-schema/args-name-clash.json

tests/qapi-schema/args-name-clash.json +3 -3
@@ -1,5 +1,5 @@
 # C member name collision
-# FIXME - This parses, but fails to compile, because the C struct is given
-# two 'a_b' members.  Either reject this at parse time, or munge the C names
-# to avoid the collision.
+# Reject members that clash when mapped to C names (we would have two 'a_b'
+# members). It would also be possible to munge the C names to avoid the
+# collision, but unlikely to be worth the effort.

I'd drop the second sentence.

Done.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#70 lore raw

PATCH v10 29/30 cpu: Convert CpuInfo into flat union

In reply to Eric Blake

When qapi type CpuInfo was originally created for 0.14, we had no notion of a flat union, and instead just listed a bunch of optional fields with documentation about the mutually-exclusive choice of which instruction pointer field(s) would be provided for a given architecture. But now that we have flat unions and introspection, it is better to segregate off which fields will be provided according to the actual architecture. With this in place, we no longer need the fields to be optional, because the choice of the discriminator serves that role.

This has an additional benefit: the old all-in-one struct was the only place in the code base that had a case-sensitive naming of members 'pc' vs. 'PC'. Separating these spellings into different branches of the flat union will allow us to add restrictions against future case-insensitive collisions, and make it possible for a future qemu to decide whether to enable case-insensitive QMP.

---
v10: new patch

---
 cpus.c           |  31 ++++++++------
 hmp.c            |  30 +++++++++-----
 qapi-schema.json | 120 ++++++++++++++++++++++++++++++++++++++++++++++---------
 3 files changed, 139 insertions(+), 42 deletions(-)
cpus.c +19 -12
index c6a5d0e..1c8078d 100644
@@ -1526,22 +1526,29 @@ CpuInfoList *qmp_query_cpus(Error **errp)
         info->value->qom_path = object_get_canonical_path(OBJECT(cpu));
         info->value->thread_id = cpu->thread_id;
 #if defined(TARGET_I386)
-        info->value->has_pc = true;
-        info->value->pc = env->eip + env->segs[R_CS].base;
+        info->value->arch = CPU_INFO_ARCH_X86;
+        info->value->u.x86 = g_new0(CpuInfoX86, 1);
+        info->value->u.x86->pc = env->eip + env->segs[R_CS].base;
 #elif defined(TARGET_PPC)
-        info->value->has_nip = true;
-        info->value->nip = env->nip;
+        info->value->arch = CPU_INFO_ARCH_PPC;
+        info->value->u.ppc = g_new0(CpuInfoPpc, 1);
+        info->value->u.ppc->nip = env->nip;
 #elif defined(TARGET_SPARC)
-        info->value->has_pc = true;
-        info->value->pc = env->pc;
-        info->value->has_npc = true;
-        info->value->npc = env->npc;
+        info->value->arch = CPU_INFO_ARCH_SPARC;
+        info->value->u.sparc = g_new0(CpuInfoSPARC, 1);
+        info->value->u.sparc->pc = env->pc;
+        info->value->u.sparc->npc = env->npc;
 #elif defined(TARGET_MIPS)
-        info->value->has_PC = true;
-        info->value->PC = env->active_tc.PC;
+        info->value->arch = CPU_INFO_ARCH_MIPS;
+        info->value->u.mips = g_new0(CpuInfoMips, 1);
+        info->value->u.mips->PC = env->active_tc.PC;
 #elif defined(TARGET_TRICORE)
-        info->value->has_PC = true;
-        info->value->PC = env->PC;
+        info->value->arch = CPU_INFO_ARCH_TRICORE;
+        info->value->u.tricore = g_new0(CpuInfoTricore, 1);
+        info->value->u.tricore->PC = env->PC;
+#else
+        info->value->arch = CPU_INFO_ARCH_OTHER;
+        info->value->u.other = g_new0(CpuInfoOther, 1);
 #endif
 
         /* XXX: waiting for the qapi to support GSList */
hmp.c +19 -11
index a15d00c..1307016 100644
@@ -310,17 +310,25 @@ void hmp_info_cpus(Monitor *mon, const QDict *qdict)
 
         monitor_printf(mon, "%c CPU #%" PRId64 ":", active, cpu->value->CPU);
 
-        if (cpu->value->has_pc) {
-            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->pc);
-        }
-        if (cpu->value->has_nip) {
-            monitor_printf(mon, " nip=0x%016" PRIx64, cpu->value->nip);
-        }
-        if (cpu->value->has_npc) {
-            monitor_printf(mon, " npc=0x%016" PRIx64, cpu->value->npc);
-        }
-        if (cpu->value->has_PC) {
-            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->PC);
+        switch (cpu->value->arch) {
+        case CPU_INFO_ARCH_X86:
+            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->u.x86->pc);
+            break;
+        case CPU_INFO_ARCH_PPC:
+            monitor_printf(mon, " nip=0x%016" PRIx64, cpu->value->u.ppc->nip);
+            break;
+        case CPU_INFO_ARCH_SPARC:
+            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->u.sparc->pc);
+            monitor_printf(mon, " npc=0x%016" PRIx64, cpu->value->u.sparc->npc);
+            break;
+        case CPU_INFO_ARCH_MIPS:
+            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->u.mips->PC);
+            break;
+        case CPU_INFO_ARCH_TRICORE:
+            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->u.tricore->PC);
+            break;
+        default:
+            break;
         }
 
         if (cpu->value->halted) {
qapi-schema.json +101 -19

144 lines not on this page · raw message

Signature · 1 line
2.4.3
#71 lore raw

Re: [Qemu-devel] [PATCH v10 29/30] cpu: Convert CpuInfo into flat union

Quoted 254 lines · Eric Blake

When qapi type CpuInfo was originally created for 0.14, we had no notion of a flat union, and instead just listed a bunch of optional fields with documentation about the mutually-exclusive choice of which instruction pointer field(s) would be provided for a given architecture. But now that we have flat unions and introspection, it is better to segregate off which fields will be provided according to the actual architecture. With this in place, we no longer need the fields to be optional, because the choice of the discriminator serves that role.

This has an additional benefit: the old all-in-one struct was the only place in the code base that had a case-sensitive naming of members 'pc' vs. 'PC'. Separating these spellings into different branches of the flat union will allow us to add restrictions against future case-insensitive collisions, and make it possible for a future qemu to decide whether to enable case-insensitive QMP.

---
v10: new patch

---
 cpus.c           |  31 ++++++++------
 hmp.c            |  30 +++++++++-----
 qapi-schema.json | 120 ++++++++++++++++++++++++++++++++++++++++++++++---------
 3 files changed, 139 insertions(+), 42 deletions(-)
cpus.c +19 -12
index c6a5d0e..1c8078d 100644
@@ -1526,22 +1526,29 @@ CpuInfoList *qmp_query_cpus(Error **errp)
         info->value->qom_path = object_get_canonical_path(OBJECT(cpu));
         info->value->thread_id = cpu->thread_id;
 #if defined(TARGET_I386)
-        info->value->has_pc = true;
-        info->value->pc = env->eip + env->segs[R_CS].base;
+        info->value->arch = CPU_INFO_ARCH_X86;
+        info->value->u.x86 = g_new0(CpuInfoX86, 1);
+        info->value->u.x86->pc = env->eip + env->segs[R_CS].base;
 #elif defined(TARGET_PPC)
-        info->value->has_nip = true;
-        info->value->nip = env->nip;
+        info->value->arch = CPU_INFO_ARCH_PPC;
+        info->value->u.ppc = g_new0(CpuInfoPpc, 1);
+        info->value->u.ppc->nip = env->nip;
 #elif defined(TARGET_SPARC)
-        info->value->has_pc = true;
-        info->value->pc = env->pc;
-        info->value->has_npc = true;
-        info->value->npc = env->npc;
+        info->value->arch = CPU_INFO_ARCH_SPARC;
+        info->value->u.sparc = g_new0(CpuInfoSPARC, 1);
+        info->value->u.sparc->pc = env->pc;
+        info->value->u.sparc->npc = env->npc;
 #elif defined(TARGET_MIPS)
-        info->value->has_PC = true;
-        info->value->PC = env->active_tc.PC;
+        info->value->arch = CPU_INFO_ARCH_MIPS;
+        info->value->u.mips = g_new0(CpuInfoMips, 1);
+        info->value->u.mips->PC = env->active_tc.PC;
 #elif defined(TARGET_TRICORE)
-        info->value->has_PC = true;
-        info->value->PC = env->PC;
+        info->value->arch = CPU_INFO_ARCH_TRICORE;
+        info->value->u.tricore = g_new0(CpuInfoTricore, 1);
+        info->value->u.tricore->PC = env->PC;
+#else
+        info->value->arch = CPU_INFO_ARCH_OTHER;
+        info->value->u.other = g_new0(CpuInfoOther, 1);
 #endif
 
         /* XXX: waiting for the qapi to support GSList */
hmp.c +19 -11
index a15d00c..1307016 100644
@@ -310,17 +310,25 @@ void hmp_info_cpus(Monitor *mon, const QDict *qdict)
 
         monitor_printf(mon, "%c CPU #%" PRId64 ":", active, cpu->value->CPU);
 
-        if (cpu->value->has_pc) {
-            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->pc);
-        }
-        if (cpu->value->has_nip) {
-            monitor_printf(mon, " nip=0x%016" PRIx64, cpu->value->nip);
-        }
-        if (cpu->value->has_npc) {
-            monitor_printf(mon, " npc=0x%016" PRIx64, cpu->value->npc);
-        }
-        if (cpu->value->has_PC) {
-            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->PC);
+        switch (cpu->value->arch) {
+        case CPU_INFO_ARCH_X86:
+            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->u.x86->pc);
+            break;
+        case CPU_INFO_ARCH_PPC:
+            monitor_printf(mon, " nip=0x%016" PRIx64, cpu->value->u.ppc->nip);
+            break;
+        case CPU_INFO_ARCH_SPARC:
+            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->u.sparc->pc);
+            monitor_printf(mon, " npc=0x%016" PRIx64, cpu->value->u.sparc->npc);
+            break;
+        case CPU_INFO_ARCH_MIPS:
+            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->u.mips->PC);
+            break;
+        case CPU_INFO_ARCH_TRICORE:
+            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->u.tricore->PC);
+            break;
+        default:
+            break;
         }
 
         if (cpu->value->halted) {
qapi-schema.json +94 -19
index 702b7b5..dd497ea 100644
@@ -728,43 +728,125 @@
 { 'command': 'query-mice', 'returns': ['MouseInfo'] }
 
 ##
-# @CpuInfo:
+# @CpuInfoArch:
 #
-# Information about a virtual CPU
+# An enumeration of cpu types that enable additional information during
+# @query-cpus.
+#
+# Since: 2.5
+##
+{ 'enum': 'CpuInfoArch',
+  'data': ['x86', 'sparc', 'ppc', 'mips', 'tricore', 'other' ] }
+
+##
+# @CpuInfoBase:
+#
+# Common information about a virtual CPU
 #
 # @CPU: the index of the virtual CPU
 #
-# @current: this only exists for backwards compatible and should be ignored
+# @current: this only exists for backwards compatibility and should be ignored
 #
 # @halted: true if the virtual CPU is in the halt state.  Halt usually refers
 #          to a processor specific low power mode.
 #
 # @qom_path: path to the CPU object in the QOM tree (since 2.4)
 #
-# @pc: #optional If the target is i386 or x86_64, this is the 64-bit instruction
-#                pointer.
-#                If the target is Sparc, this is the PC component of the
-#                instruction pointer.
-#
-# @nip: #optional If the target is PPC, the instruction pointer
-#
-# @npc: #optional If the target is Sparc, the NPC component of the instruction
-#                 pointer
-#
-# @PC: #optional If the target is MIPS, the instruction pointer
-#
 # @thread_id: ID of the underlying host thread
 #
+# @arch: architecture of the cpu, which determines which additional fields
+#        will be listed (since 2.5)
+#
 # Since: 0.14.0
 #
 # Notes: @halted is a transient state that changes frequently.  By the time the
 #        data is sent to the client, the guest may no longer be halted.
 ##
-{ 'struct': 'CpuInfo',
+{ 'struct': 'CpuInfoBase',
   'data': {'CPU': 'int', 'current': 'bool', 'halted': 'bool',
-           'qom_path': 'str',
-           '*pc': 'int', '*nip': 'int', '*npc': 'int', '*PC': 'int',
-           'thread_id': 'int'} }
+           'qom_path': 'str', 'thread_id': 'int', 'arch': 'CpuInfoArch' } }
+
+##
+# @CpuInfo:
+#
+# Information about a virtual CPU
+#
+# Since: 0.14.0
+##
+{ 'union': 'CpuInfo', 'base': 'CpuInfoBase', 'discriminator': 'arch',
+  'data': { 'x86': 'CpuInfoX86',
+            'sparc': 'CpuInfoSparc',
+            'ppc': 'CpuInfoPpc',
+            'mips': 'CpuInfoMips',
+            'tricore': 'CpuInfoTricore',
+            'other': 'CpuInfoOther' } }
+
+##
+# @CpuInfoX86:
+#
+# Additional information about a virtual i386 or x86_64 CPU
+#
+# @pc: the 64-bit instruction pointer
+#
+# Since 2.5
+##
+{ 'struct': 'CpuInfoX86', 'data': { 'pc': 'int' } }
+
+##
+# @CpuInfoSparc:
+#
+# Additional information about a virtual Sparc CPU
+#
+# @pc: the PC component of the instruction pointer
+#
+# @npc: the NPC component of the instruction pointer
+#
+# Since 2.5
+##
+{ 'struct': 'CpuInfoSparc', 'data': { 'pc': 'int', 'npc': 'int' } }
+
+##
+# @CpuInfoPpc:
+#
+# Additional information about a virtual PPC CPU
+#
+# @nip: the instruction pointer
+#
+# Since 2.5
+##
+{ 'struct': 'CpuInfoPpc', 'data': { 'nip': 'int' } }
+
+##
+# @CpuInfoMips:
+#
+# Additional information about a virtual MIPS CPU
+#
+# @PC: the instruction pointer
+#
+# Since 2.5
+##
+{ 'struct': 'CpuInfoMips', 'data': { 'PC': 'int' } }
+
+##
+# @CpuInfoMips:
+#
+# Additional information about a virtual Tricore CPU
+#
+# @PC: the instruction pointer
+#
+# Since 2.5
+##
+{ 'struct': 'CpuInfoTricore', 'data': { 'PC': 'int' } }
+
+##
+# @CpuInfoOther:

qapi-schema.json line 852

Quoted 10 lines
qapi-schema.json +7 -0
+#
+# No additional information is available about the virtual CPU
+#
+# Since 2.5
+#
+##
+{ 'struct': 'CpuInfoOther', 'data': { } }
 
 ##
 # @query-cpus:

CpuInfo is only used as return type of query-cpus.

If I understand the change correctly, the value of query-cpus is not changed at all. Its introspection, however, is.

Do we need this to make 2.5?

Any other messy optionals that should really be (flat) unions?

author
#72 lore raw

Re: [Qemu-devel] [PATCH v10 29/30] cpu: Convert CpuInfo into flat union

Quoted 20 lines · Markus Armbruster
Quoted 17 lines · Eric Blake

When qapi type CpuInfo was originally created for 0.14, we had no notion of a flat union, and instead just listed a bunch of optional fields with documentation about the mutually-exclusive choice of which instruction pointer field(s) would be provided for a given architecture. But now that we have flat unions and introspection, it is better to segregate off which fields will be provided according to the actual architecture. With this in place, we no longer need the fields to be optional, because the choice of the discriminator serves that role.

This has an additional benefit: the old all-in-one struct was the only place in the code base that had a case-sensitive naming of members 'pc' vs. 'PC'. Separating these spellings into different branches of the flat union will allow us to add restrictions against future case-insensitive collisions, and make it possible for a future qemu to decide whether to enable case-insensitive QMP.

Quoted 7 lines
Quoted 2 lines

##
# @query-cpus:

CpuInfo is only used as return type of query-cpus.

If I understand the change correctly, the value of query-cpus is not changed at all. Its introspection, however, is.

Almost. The output of query-cpus gains a new 'arch':'FOO' string per cpu, saying which branch of the flat union is in use. But as it is output-only, adding a new output field shouldn't break any existing clients (because QMP already documents that clients should ignore new fields); and since the struct is output-only, it can't break any input fields.

I probably can spell that out better in the commit message, though.

Oh, and I need to update qmp-commands.hx, as part of my v11 spin.

Quoted 2 lines

Do we need this to make 2.5?

It's true that the introspection will change (instead of seeing flat optional members, you now have to chase down variants). But I don't think it is pressing enough to rush into 2.5; the change is backwards-compatible no matter when we do it, and introspection clients are going to have to get used to the idea of chasing down different paths to find all members of a struct (that is, if we don't change this until 2.6, I'm sure it won't be the last case of introspection changing while keeping QMP wire format back-compatible as formerly non-variant fields become variant).

Quoted 3 lines

Any other messy optionals that should really be (flat) unions?

Possibly.
/me goes and audits all 0.14 interfaces...

add_client is an input command, but it would be nicer as a flat union (we can't do that until commands can take unions; but that happens as part of my pending queue for netdev_add), so that's not going to make 2.5.

client_migrate_info looks odd, since it requires 'protocol':'str' to be the value 'spice'. Probably several functions that take or produce a finite set of strings that should be using enums, but conversion from 'str' to 'enum' is backwards-compatible according to QMP wire format. Doing it sooner rather than later makes introspection nicer.

But I didn't spot any other obvious commands from that far back with mutually-exclusive optional parameters, and certainly nothing else with case clashes.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#73 lore raw

Re: [Qemu-devel] [PATCH v10 29/30] cpu: Convert CpuInfo into flat union

Quoted 29 lines · Eric Blake
Quoted 20 lines · Markus Armbruster
Quoted 17 lines · Eric Blake

When qapi type CpuInfo was originally created for 0.14, we had no notion of a flat union, and instead just listed a bunch of optional fields with documentation about the mutually-exclusive choice of which instruction pointer field(s) would be provided for a given architecture. But now that we have flat unions and introspection, it is better to segregate off which fields will be provided according to the actual architecture. With this in place, we no longer need the fields to be optional, because the choice of the discriminator serves that role.

This has an additional benefit: the old all-in-one struct was the only place in the code base that had a case-sensitive naming of members 'pc' vs. 'PC'. Separating these spellings into different branches of the flat union will allow us to add restrictions against future case-insensitive collisions, and make it possible for a future qemu to decide whether to enable case-insensitive QMP.

Quoted 7 lines
Quoted 2 lines

##
# @query-cpus:

CpuInfo is only used as return type of query-cpus.

If I understand the change correctly, the value of query-cpus is not changed at all. Its introspection, however, is.

Quoted 6 lines

Almost. The output of query-cpus gains a new 'arch':'FOO' string per cpu, saying which branch of the flat union is in use. But as it is output-only, adding a new output field shouldn't break any existing clients (because QMP already documents that clients should ignore new fields); and since the struct is output-only, it can't break any input fields.

Understood.

Quoted 1 line

I probably can spell that out better in the commit message, though.

Yes, please

Quoted 1 line

Oh, and I need to update qmp-commands.hx, as part of my v11 spin.

Yes, please.

Quoted 11 lines
Quoted 1 line

Do we need this to make 2.5?

It's true that the introspection will change (instead of seeing flat optional members, you now have to chase down variants). But I don't think it is pressing enough to rush into 2.5; the change is backwards-compatible no matter when we do it, and introspection clients are going to have to get used to the idea of chasing down different paths to find all members of a struct (that is, if we don't change this until 2.6, I'm sure it won't be the last case of introspection changing while keeping QMP wire format back-compatible as formerly non-variant fields become variant).

We can mess with the schema as long as the QMP wire format stays compatible.

With introspection, schema changes get exposed in QMP that weren't exposed before. Begs the question whether the ABI promise extends to the introspection value.

I think we don't want to extend it, at least for now. In other words, we refuse to put additional constraints on schema changes to keep the introspection value stable. Makes introspection a bit harder to use. Not ideal, but better than committing to constraints we don't even fully understand, yet.

I think we should spell this out introspection documentation.

Our only example so far is converting between optional members and flat unions. Can we think of others? Hmm, you found one below.

Quoted 9 lines
Quoted 2 lines

Any other messy optionals that should really be (flat) unions?

Possibly.
/me goes and audits all 0.14 interfaces...

add_client is an input command, but it would be nicer as a flat union (we can't do that until commands can take unions; but that happens as part of my pending queue for netdev_add), so that's not going to make 2.5.

Since the command parameters are just an object type, this is merely another case of converting between optional members and flat unions.

Quoted 5 lines

client_migrate_info looks odd, since it requires 'protocol':'str' to be the value 'spice'. Probably several functions that take or produce a finite set of strings that should be using enums, but conversion from 'str' to 'enum' is backwards-compatible according to QMP wire format. Doing it sooner rather than later makes introspection nicer.

This is another example: converting between string and enum. Going from string to enum only adds information, and should be easy enough for clients. Going the other way removes information, and probably should not be done.

Quoted 3 lines

But I didn't spot any other obvious commands from that far back with mutually-exclusive optional parameters, and certainly nothing else with case clashes.

Okay. I guess finding all the examples that matter may take a few development cycles. Perhaps we want to commit to some additional compatibility promises for introspection then.

author
#74 lore raw

Re: [Qemu-devel] [PATCH v10 29/30] cpu: Convert CpuInfo into flat union

Quoted 19 lines · Markus Armbruster
Quoted 11 lines
Quoted 1 line

Do we need this to make 2.5?

It's true that the introspection will change (instead of seeing flat optional members, you now have to chase down variants). But I don't think it is pressing enough to rush into 2.5; the change is backwards-compatible no matter when we do it, and introspection clients are going to have to get used to the idea of chasing down different paths to find all members of a struct (that is, if we don't change this until 2.6, I'm sure it won't be the last case of introspection changing while keeping QMP wire format back-compatible as formerly non-variant fields become variant).

We can mess with the schema as long as the QMP wire format stays compatible.

With introspection, schema changes get exposed in QMP that weren't exposed before. Begs the question whether the ABI promise extends to the introspection value.

Quoted 7 lines

I think we don't want to extend it, at least for now. In other words, we refuse to put additional constraints on schema changes to keep the introspection value stable. Makes introspection a bit harder to use. Not ideal, but better than committing to constraints we don't even fully understand, yet.

I think we should spell this out introspection documentation.

Sounds like I need to prep a doc patch for 2.5 then.

Quoted 3 lines

Our only example so far is converting between optional members and flat unions. Can we think of others? Hmm, you found one below.

Converting from optional non-variant to variant, converting from string to enum, converting from single type to alternate. Might be other conversions, and the set of conversions for input types may differ from those of output conversions.

Quoted 3 lines

Okay. I guess finding all the examples that matter may take a few development cycles. Perhaps we want to commit to some additional compatibility promises for introspection then.

Maybe, but until clients are no longer worried about older clients, it may be several years before taking advantage of whatever we later document (as it is, upstream libvirt is just now bumping minimum qemu support up to 0.12 [thanks to RHEL 6] and discarding older qemu from RHEL 5 - quite a few years behind upstream qemu 2.5, although it is finding a lot of code in libvirt that is no longer necessary).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#75 lore raw

Re: [Qemu-devel] [PATCH v10 29/30] cpu: Convert CpuInfo into flat union

Quoted 27 lines · Eric Blake
Quoted 25 lines · Markus Armbruster
Quoted 11 lines
Quoted 1 line

Do we need this to make 2.5?

It's true that the introspection will change (instead of seeing flat optional members, you now have to chase down variants). But I don't think it is pressing enough to rush into 2.5; the change is backwards-compatible no matter when we do it, and introspection clients are going to have to get used to the idea of chasing down different paths to find all members of a struct (that is, if we don't change this until 2.6, I'm sure it won't be the last case of introspection changing while keeping QMP wire format back-compatible as formerly non-variant fields become variant).

We can mess with the schema as long as the QMP wire format stays compatible.

With introspection, schema changes get exposed in QMP that weren't exposed before. Begs the question whether the ABI promise extends to the introspection value.

I think we don't want to extend it, at least for now. In other words, we refuse to put additional constraints on schema changes to keep the introspection value stable. Makes introspection a bit harder to use. Not ideal, but better than committing to constraints we don't even fully understand, yet.

Quoted 3 lines
Quoted 1 line

I think we should spell this out introspection documentation.

Sounds like I need to prep a doc patch for 2.5 then.

Yes, please!

Quoted 8 lines
Quoted 2 lines

Our only example so far is converting between optional members and flat unions. Can we think of others? Hmm, you found one below.

Converting from optional non-variant to variant, converting from string to enum, converting from single type to alternate. Might be other conversions, and the set of conversions for input types may differ from those of output conversions.

Quoted 7 lines
Quoted 3 lines

Okay. I guess finding all the examples that matter may take a few development cycles. Perhaps we want to commit to some additional compatibility promises for introspection then.

Maybe, but until clients are no longer worried about older clients, it may be several years before taking advantage of whatever we later document

That's life.

I fully expect the consumers of QMP introspection to get a few things wrong initially, requiring their users to upgrade to fixed versions. That's life, too.

Quoted 4 lines

(as it is, upstream libvirt is just now bumping minimum qemu
support up to 0.12 [thanks to RHEL 6] and discarding older qemu from RHEL 5 - quite a few years behind upstream qemu 2.5, although it is finding a lot of code in libvirt that is no longer necessary).

In my opinion, attempting to support old versions of QEMU is a complete waste of resources unless you actually test them to stop the bit-rot.

RHEL-6 0.12 is by now quite different from upstream 0.12. Unlike upstream 0.12, it's actually still relevant and useful today.

I encourage you to identify relevant upstream and downstream versions, keep them tested, and drop support for the rest.

author
#76 lore raw

PATCH v10 30/30 qapi: Forbid case-insensitive clashes

In reply to Eric Blake

We have toyed on list with the idea of a future extension to QMP of teaching it to be case-insensitive (the user could request command 'Quit' instead of 'quit', or could spell a struct field as 'CPU' instead of 'cpu'). But for that to be a practical extension, we cannot break backwards compatibility with any existing struct that was already relying on case sensitivity. Fortunately, after the previous patch cleaned up CpuInfo, there are no such existing qapi structs.

Another benefit of enforcing a restriction against case-insensitive is that we no longer have to worry about the case of enum values that could be distinguished by case if mapped by c_name(), but which cannot be distinguished when mapped to C as ALL_CAPS by camel_to_uppper(). Having the generator look for case collisions up front will prevent developers from worrying whether different munging rules for member names compared to enum values as a discriminator will cause any problems in qapi unions.

Of course, if we never implement a case-insensitive QMP extension, or find a legitimate reason to need qapi members that differ only by case, we could always relax this restriction. But it is easier to relax later than it is to wish we had the restriction in place earlier.

---
v10: new patch

---
 docs/qapi-code-gen.txt                 | 5 +++++
 scripts/qapi.py                        | 4 ++--
 tests/Makefile                         | 1 +
 tests/qapi-schema/args-case-clash.err  | 1 +
 tests/qapi-schema/args-case-clash.exit | 1 +
 tests/qapi-schema/args-case-clash.json | 5 +++++
 tests/qapi-schema/args-case-clash.out  | 0
 7 files changed, 15 insertions(+), 2 deletions(-)
 create mode 100644 tests/qapi-schema/args-case-clash.err
 create mode 100644 tests/qapi-schema/args-case-clash.exit
 create mode 100644 tests/qapi-schema/args-case-clash.json
 create mode 100644 tests/qapi-schema/args-case-clash.out
docs/qapi-code-gen.txt +5 -0
index f9fa6f3..13cec10 100644
@@ -116,6 +116,11 @@ names should be ALL_CAPS with words separated by underscore.  Field
 names cannot start with 'has-' or 'has_', as this is reserved for
 tracking optional fields.
 
+For now, Client JSON Protocol is case-sensitive, but future extensions
+may allow for case-insensitive recognition of command and event names,
+or of member field names.  As such, the generator rejects attempts to
+create entities that only differ by case.
+
 Any name (command, event, type, field, or enum value) beginning with
 "x-" is marked experimental, and may be withdrawn or changed
 incompatibly in a future release.  Downstream vendors may add
scripts/qapi.py +2 -2
index 3a359cb..7e7ad6e 100644
@@ -1039,7 +1039,7 @@ class QAPISchemaObjectTypeMember(object):
         assert self.type
 
     def check_clash(self, info, seen):
-        name = c_name(self.name)
+        name = c_name(self.name).upper()
         if name in seen:
             raise QAPIExprError(info,
                                 "%s collides with %s"
@@ -1087,7 +1087,7 @@ class QAPISchemaObjectTypeVariants(object):
 
     def check(self, schema, seen):
         if self.tag_name:    # flat union
-            self.tag_member = seen[c_name(self.tag_name)]
+            self.tag_member = seen[c_name(self.tag_name).upper()]
             assert self.tag_name == self.tag_member.name
         tag_type = self.tag_member.type
         assert isinstance(tag_type, QAPISchemaEnumType)
tests/Makefile +1 -0
index c84c6cb..d1c6817 100644
@@ -239,6 +239,7 @@ qapi-schema += args-alternate.json
 qapi-schema += args-any.json
 qapi-schema += args-array-empty.json
 qapi-schema += args-array-unknown.json
+qapi-schema += args-case-clash.json
 qapi-schema += args-int.json
 qapi-schema += args-invalid.json
 qapi-schema += args-member-array-bad.json
tests/qapi-schema/args-case-clash.err new file +1 -0
new file mode 100644
index 0000000..5495ab8
@@ -0,0 +1 @@
+tests/qapi-schema/args-case-clash.json:5: 'A' (argument of oops) collides with 'a' (argument of oops)
tests/qapi-schema/args-case-clash.exit new file +1 -0
new file mode 100644
index 0000000..d00491f
@@ -0,0 +1 @@
+1
tests/qapi-schema/args-case-clash.json new file +5 -0
new file mode 100644
index 0000000..55ae488
@@ -0,0 +1,5 @@
+# C member name collision
+# Reject members that clash case-insensitively (our mapping to C names
+# preserves case, but allowing these members now would prevent a future
+# relaxing of QMP to be case-insensitive).
+{ 'command': 'oops', 'data': { 'a': 'str', 'A': 'int' } }
tests/qapi-schema/args-case-clash.out new file +0 -0
new file mode 100644
index 0000000..e69de29
Signature · 1 line
2.4.3
#77 lore raw

Re: [Qemu-devel] [PATCH v10 30/30] qapi: Forbid case-insensitive clashes

Quoted 9 lines · Eric Blake

We have toyed on list with the idea of a future extension to QMP of teaching it to be case-insensitive (the user could request command 'Quit' instead of 'quit', or could spell a struct field as 'CPU' instead of 'cpu'). But for that to be a practical extension, we cannot break backwards compatibility with any existing struct that was already relying on case sensitivity. Fortunately, after the previous patch cleaned up CpuInfo, there are no such existing qapi structs.

Quoted 5 lines

Another benefit of enforcing a restriction against case-insensitive is that we no longer have to worry about the case of enum values that could be distinguished by case if mapped by c_name(), but which cannot be distinguished when mapped to C as ALL_CAPS by camel_to_uppper(). Having the

camel_to_upper()

Quoted 5 lines

generator look for case collisions up front will prevent developers from worrying whether different munging rules for member names compared to enum values as a discriminator will cause any problems in qapi unions.

Quoted 7 lines

Of course, if we never implement a case-insensitive QMP extension, or find a legitimate reason to need qapi members that differ only by case, we could always relax this restriction. But it is easier to relax later than it is to wish we had the restriction in place earlier.

I think "make QMP case-insensitive" is a very weak justification for anything.

Your second reason is much stronger: forbidding case-insensitive clashes lets the generators shout without fear of introducing clashes in generated code.

However, since camel_to_upper() does more than just shout, forbidding case-insensitive clashes protects against clashes in generated code only for names where camel_to_upper() just shouts. It does for lower case names, i.e. almost all of them.

The commit would make more sense if we first defanged c_enum_const():
make it mangle the const_name part like c_name(const_name).upper(). This is independent of the argument I'm having with Dan on how the prefix should be mangled.

One more good argument for forbidding case-insensitive clashes: an interface that sports names differing only in case is a bad interface.

Quoted 21 lines

---
v10: new patch

---
 docs/qapi-code-gen.txt                 | 5 +++++
 scripts/qapi.py                        | 4 ++--
 tests/Makefile                         | 1 +
 tests/qapi-schema/args-case-clash.err  | 1 +
 tests/qapi-schema/args-case-clash.exit | 1 +
 tests/qapi-schema/args-case-clash.json | 5 +++++
 tests/qapi-schema/args-case-clash.out  | 0
 7 files changed, 15 insertions(+), 2 deletions(-)
 create mode 100644 tests/qapi-schema/args-case-clash.err
 create mode 100644 tests/qapi-schema/args-case-clash.exit
 create mode 100644 tests/qapi-schema/args-case-clash.json
 create mode 100644 tests/qapi-schema/args-case-clash.out
docs/qapi-code-gen.txt +0 -0
index f9fa6f3..13cec10 100644

docs/qapi-code-gen.txt names should be ALL_CAPS with words separated by underscore. Field · line 123

Quoted 8 lines
docs/qapi-code-gen.txt +5 -0
@@ -116,6 +116,11 @@ names should be ALL_CAPS with words separated by underscore.  Field
 names cannot start with 'has-' or 'has_', as this is reserved for
 tracking optional fields.
 
+For now, Client JSON Protocol is case-sensitive, but future extensions
+may allow for case-insensitive recognition of command and event names,
+or of member field names.  As such, the generator rejects attempts to
+create entities that only differ by case.
+

This suggests we're actually planning to make QMP case-insensitive. Let's avoid that. Perhaps:

Since we don't want interfaces with different names that differ only in case, the generator rejects such names.

Quoted 17 lines
docs/qapi-code-gen.txt +0 -0
 Any name (command, event, type, field, or enum value) beginning with
 "x-" is marked experimental, and may be withdrawn or changed
 incompatibly in a future release.  Downstream vendors may add
scripts/qapi.py +1 -1
index 3a359cb..7e7ad6e 100644
@@ -1039,7 +1039,7 @@ class QAPISchemaObjectTypeMember(object):
         assert self.type
 
     def check_clash(self, info, seen):
-        name = c_name(self.name)
+        name = c_name(self.name).upper()
         if name in seen:
             raise QAPIExprError(info,
                                 "%s collides with %s"

scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1093

Quoted 8 lines
scripts/qapi.py +1 -1
@@ -1087,7 +1087,7 @@ class QAPISchemaObjectTypeVariants(object):
 
     def check(self, schema, seen):
         if self.tag_name:    # flat union
-            self.tag_member = seen[c_name(self.tag_name)]
+            self.tag_member = seen[c_name(self.tag_name).upper()]
             assert self.tag_name == self.tag_member.name
         tag_type = self.tag_member.type
         assert isinstance(tag_type, QAPISchemaEnumType)

If we ever acquire more lookups in seen, we'll need a lookup function, so we don't duplicate c_name(NAME).upper(). But this will do for now.

Quoted 32 lines
tests/Makefile +1 -0
index c84c6cb..d1c6817 100644
@@ -239,6 +239,7 @@ qapi-schema += args-alternate.json
 qapi-schema += args-any.json
 qapi-schema += args-array-empty.json
 qapi-schema += args-array-unknown.json
+qapi-schema += args-case-clash.json
 qapi-schema += args-int.json
 qapi-schema += args-invalid.json
 qapi-schema += args-member-array-bad.json
tests/qapi-schema/args-case-clash.err new file +1 -0
new file mode 100644
index 0000000..5495ab8
@@ -0,0 +1 @@
+tests/qapi-schema/args-case-clash.json:5: 'A' (argument of oops) collides with 'a' (argument of oops)
tests/qapi-schema/args-case-clash.exit new file +1 -0
new file mode 100644
index 0000000..d00491f
@@ -0,0 +1 @@
+1
tests/qapi-schema/args-case-clash.json new file +0 -0
new file mode 100644
index 0000000..55ae488

tests/qapi-schema/args-case-clash.json line 4

Quoted 4 lines
tests/qapi-schema/args-case-clash.json new file +4 -0
@@ -0,0 +1,5 @@
+# C member name collision
+# Reject members that clash case-insensitively (our mapping to C names
+# preserves case, but allowing these members now would prevent a future
+# relaxing of QMP to be case-insensitive).

Again, let's not suggest we're planning to make QMP case-insensitive.

Quoted 4 lines
tests/qapi-schema/args-case-clash.json new file +1 -0
+{ 'command': 'oops', 'data': { 'a': 'str', 'A': 'int' } }
tests/qapi-schema/args-case-clash.out new file +0 -0
new file mode 100644
index 0000000..e69de29
#78 lore raw
In reply to Eric Blake
Quoted 23 lines · Eric Blake

No pending prerequisites; based on qemu.git master

Also available as a tag at this location:
git fetch git://repo.or.cz/qemu/ericb.git qapi-cleanupv9c

and will soon be part of my branch with the rest of the v5 series, at:
http://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/qapi

v10 notes:
Split several patches, redo the middle patches from Markus to be back in the order they were first posted, some fallout change to my patches due to the nicer pattern of minimizing conditionals inside .check(), by instead calling .check_clash() as needed. Change data->err magic in tests to instead use a new helper error_free_or_abort(). Add a patch that would prevent qapi case-insensitive clashes.

I am redoing my subset boundaries slightly: patches 23-27 of v9 (updating the alternate layout) will be delayed to subset D, and 2 other patches previously posted in subset D are now here (turning qapi clash checking into actual error messages), so the subject line of this cover letter is slightly different.

Quoted 3 lines

Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.

I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.

I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.

author
#79 lore raw
Quoted 6 lines · Markus Armbruster
Quoted 4 lines · Eric Blake

No pending prerequisites; based on qemu.git master

Also available as a tag at this location:
git fetch git://repo.or.cz/qemu/ericb.git qapi-cleanupv9c

Make that the qapi-cleanupv10c label.

Quoted 23 lines
Quoted 19 lines

and will soon be part of my branch with the rest of the v5 series, at:
http://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/qapi

v10 notes:
Split several patches, redo the middle patches from Markus to be back in the order they were first posted, some fallout change to my patches due to the nicer pattern of minimizing conditionals inside .check(), by instead calling .check_clash() as needed. Change data->err magic in tests to instead use a new helper error_free_or_abort(). Add a patch that would prevent qapi case-insensitive clashes.

I am redoing my subset boundaries slightly: patches 23-27 of v9 (updating the alternate layout) will be delayed to subset D, and 2 other patches previously posted in subset D are now here (turning qapi clash checking into actual error messages), so the subject line of this cover letter is slightly different.

Quoted 3 lines

Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.

Quoted 5 lines

I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.

I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.

Yes, 01-12 seems like a good first set, if you want to make those touchups (I've supplied some potential text improvements in reply to some of your comments); I'm happy, as always, to take a peek over your staging repo to double check what you are prepping for the pull request.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#80 lore raw
Quoted 2 lines · Eric Blake
Quoted 1 line · Markus Armbruster

Eric Blake <[email protected]> writes:

[...]

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.

I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.

Quoted 7 lines
Quoted 2 lines

I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.

Yes, 01-12 seems like a good first set, if you want to make those touchups (I've supplied some potential text improvements in reply to some of your comments); I'm happy, as always, to take a peek over your staging repo to double check what you are prepping for the pull request.

Please have a look at my qapi-next branch.

author
#81 lore raw
Quoted 15 lines · Markus Armbruster
Quoted 2 lines · Eric Blake
Quoted 1 line · Markus Armbruster

Eric Blake <[email protected]> writes:

[...]

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.

I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.

I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.

Quoted 6 lines
Quoted 4 lines

Yes, 01-12 seems like a good first set, if you want to make those touchups (I've supplied some potential text improvements in reply to some of your comments); I'm happy, as always, to take a peek over your staging repo to double check what you are prepping for the pull request.

Please have a look at my qapi-next branch.

Close, but commit 5f72bb85 on that branch is missing the one-line summary:

qapi: Simplify error cleanup in test-qmp-*

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#82 lore raw
Quoted 21 lines · Eric Blake
Quoted 20 lines · Markus Armbruster
Quoted 2 lines · Eric Blake
Quoted 1 line · Markus Armbruster

Eric Blake <[email protected]> writes:

[...]

Quoted 14 lines
Quoted 9 lines
Quoted 3 lines

Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.

I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.

I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.

Yes, 01-12 seems like a good first set, if you want to make those touchups (I've supplied some potential text improvements in reply to some of your comments); I'm happy, as always, to take a peek over your staging repo to double check what you are prepping for the pull request.

Quoted 5 lines
Quoted 1 line

Please have a look at my qapi-next branch.

Close, but commit 5f72bb85 on that branch is missing the one-line summary:

qapi: Simplify error cleanup in test-qmp-*

Fixed & sent pull request, thanks!

#83 lore raw
Quoted 23 lines · Markus Armbruster
Quoted 21 lines · Eric Blake
Quoted 20 lines · Markus Armbruster
Quoted 2 lines · Eric Blake
Quoted 1 line · Markus Armbruster

Eric Blake <[email protected]> writes:

[...]

Quoted 14 lines
Quoted 9 lines
Quoted 3 lines

Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.

I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.

I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.

Yes, 01-12 seems like a good first set, if you want to make those touchups (I've supplied some potential text improvements in reply to some of your comments); I'm happy, as always, to take a peek over your staging repo to double check what you are prepping for the pull request.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Please have a look at my qapi-next branch.

Close, but commit 5f72bb85 on that branch is missing the one-line summary:

qapi: Simplify error cleanup in test-qmp-*

Fixed & sent pull request, thanks!

Pulled.

I think I'll close the QAPI floodgates for 2.5 now. Bug fixes are of course exempted, and if we find something that impacts ABI, I'm willing to consider patches. Work on our backlog can continue uninterrupted; I'm happy to collect patches that are ready, and will take care of getting them into master once 2.6 opens.

author
#84 lore raw
Quoted 5 lines · Markus Armbruster

I think I'll close the QAPI floodgates for 2.5 now. Bug fixes are of course exempted, and if we find something that impacts ABI, I'm willing to consider patches. Work on our backlog can continue uninterrupted; I'm happy to collect patches that are ready, and will take care of getting them into master once 2.6 opens.

I think I found something that ought to go into 2.5: we have nothing in current query-qmp-introspection that shows the set of valid 'ErrorClass' enum values that a client can expect on error (this is because no existing commands have a reference to the type). While we are unlikely to be adding new ErrorClass values any time soon, this feels like something that should be exposed over the wire.

I'm trying to play with ideas on how best to expose it.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org