qapi member collision (post-introspection cleanups, subset C')
v10 cover qapi member collision (post-introspection cleanups, subset C')
No pending prerequisites; based on qemu.git master
Also available as a tag at this location:
git fetch git://repo.or.cz/qemu/ericb.git qapi-cleanupv9c
and will soon be part of my branch with the rest of the v5 series, at:
http://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/qapi
v10 notes:
Split several patches, redo the middle patches from Markus to be back in the order they were first posted, some fallout change to my patches due to the nicer pattern of minimizing conditionals inside .check(), by instead calling .check_clash() as needed. Change data->err magic in tests to instead use a new helper error_free_or_abort(). Add a patch that would prevent qapi case-insensitive clashes.
I am redoing my subset boundaries slightly: patches 23-27 of v9 (updating the alternate layout) will be delayed to subset D, and 2 other patches previously posted in subset D are now here (turning qapi clash checking into actual error messages), so the subject line of this cover letter is slightly different.
Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.
Backport diff in relation to v9:
001/30:[----] [--] 'qapi: Use generated TestStruct machinery in tests' 002/30:[----] [--] 'qapi: Strengthen test of TestStructList' 003/30:[down] 'qobject: Protect against use-after-free in qobject_decref()' 004/30:[down] 'qapi: Share test_init code in test-qmp-input*' 005/30:[0001] [FC] 'qapi: Plug leaks in test-qmp-*'
006/30:[down] 'qapi: Simplify non-error testing in test-qmp-*' 007/30:[down] 'qapi: Simplify error cleanup in test-qmp-*' 008/30:[----] [--] 'qapi: More tests of alternate output' 009/30:[0010] [FC] 'qapi: Test failure in middle of array parse' 010/30:[0025] [FC] 'qapi: More tests of input arrays'
011/30:[----] [--] 'qapi: Provide nicer array names in introspection' 012/30:[0014] [FC] 'qapi-introspect: Document lack of sorting' 013/30:[----] [--] 'qapi: Track simple union tag in object.local_members' 014/30:[----] [--] 'qapi-types: Consolidate gen_struct() and gen_union()' 015/30:[----] [--] 'qapi-types: Simplify gen_struct_field[s]' 016/30:[0005] [FC] 'qapi: Drop obsolete tag value collision assertions' 017/30:[0002] [FC] 'qapi: Simplify QAPISchemaObjectTypeMember.check()' 018/30:[down] 'qapi: Clean up after previous commit'
019/30:[----] [-C] 'qapi: Fix up commit 7618b91's clash sanity checking change' 020/30:[0002] [FC] 'qapi: Eliminate QAPISchemaObjectType.check() variable members' 021/30:[----] [--] 'qapi: Factor out QAPISchemaObjectTypeMember.check_clash()' 022/30:[----] [-C] 'qapi: Simplify QAPISchemaObjectTypeVariants.check()' 023/30:[0023] [FC] 'qapi: Check for qapi collisions of flat union branches' 024/30:[0013] [FC] 'qapi: Factor out QAPISchemaObjectType.check_clash()' 025/30:[down] 'qapi: Hoist tag collision check to Variants.check()' 026/30:[----] [--] 'qapi: Remove outdated tests related to QMP/branch collisions' 027/30:[down] 'qapi: Track owner of each object member'
028/30:[down] 'qapi: Detect collisions in C member names' 029/30:[down] 'cpu: Convert CpuInfo into flat union'
030/30:[down] 'qapi: Forbid case-insensitive clashes'
v9 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg00652.html
More patches added, and several reorganized. Lots of new patches from Markus, although not in the order originally proposed.
The first 8 patches are fairly straightforward, and could probably be taken as-is. Patch 9 is a rewrite of v8 4/17, but in the opposite direction (document that no sorting is done, rather than attempting to sort), so it may need further fine-tuning. Patches 12-21 represents a fusion of Markus' and my attempts to rewrite v5 7/17 into a more-reviewable set of patches, and caused further churn later in the series.
Patch 23 still uses tag_member.type == None; I ran out of time to work on Markus' idea of providing an instance of QAPISchemaBuiltinType to fill the role for 'qtype_code' without being exposed through .json files and without breaking the invariant of a valid member.type after check(), and wanted to get the rest of the series started under revew. So I may need a followup patch or even a v10 of the later half of this series after exploring that idea more.
v8 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg06674.html
Minor changes when rebasing to latest, improved commit messages in a few places, plus the addition of a few new patches. Markus started reviewing v7, but hadn't gotten very far.
v7 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg04112.html
Patches 1-3 of the previous round have moved to subset B; patch 7 is gone, and a couple of new patches are present in this round. The latest version of subset B made it much easier to reason about collision detection (namely, tag values can't collide with QMP values thanks to a named rather than anonymous union in the C type), and I like how things turned out. I suspect the hardest part of the review will be patches 5/14 and 7/14, although none of this has really had much review in any earlier versions.
v6 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg01980.html
Add some patches and rebase onto work on subset B. Rearrange some patches from v5 (this set includes 17-20, 23, 25-27). Backport diff gets a bit confused by one patch title changing.
Not much direct review comments, although some of the changes here are updated based on comments made on other patches in the v5 series.
Subset D (and more?) will come later.
In v5:
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg05410.html
I _did_ rearrange patches to try and group related features:
1-2: Groundwork cleanups
3-5: Add more test cases
6-16: Front-end cleanups
17-18: Introspection output cleanups
19-20: 'alternate' type cleanups
21-29: qapi visitor cleanups
30-45: qapi-ify netdev_add
46: add qapi shorthand for flat unions
Lots of fixes based on additional testing, and rebased to track other changes that happened in the meantime. The series is huge; I can split off smaller portions as requested.
In v4:
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg02580.html
add some more clean up patches
rebase to Markus' recent work
pull in part of Zoltán's work to make netdev_add a flat union, further enhancing it to be introspectible
I might be able to rearrange some of these patches, or separate it into smaller independent series, if requested; but I'm posting now to get review started.
In v3:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg02059.html
redo cleanup of dealloc of partial struct
add patches to make all visit_type_*() avoid leaks on failure add patches to allow boxed command arguments and events
In v2:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00900.html
rebase to Markus' v3 series
rework how comments are emitted for fields inherited from base additional patches added for deleting colliding 'void *data' documentation updates to match code changes
v1 was here:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05266.html
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05325.html
Eric Blake (23):
qapi: Use generated TestStruct machinery in tests
qapi: Strengthen test of TestStructList
qobject: Protect against use-after-free in qobject_decref()
qapi: Share test_init code in test-qmp-input*
qapi: Plug leaks in test-qmp-*
qapi: Simplify non-error testing in test-qmp-*
qapi: Simplify error cleanup in test-qmp-*
qapi: More tests of alternate output
qapi: Test failure in middle of array parse
qapi: More tests of input arrays
qapi: Provide nicer array names in introspection
qapi-introspect: Document lack of sorting
qapi: Track simple union tag in object.local_members
qapi-types: Consolidate gen_struct() and gen_union()
qapi-types: Simplify gen_struct_field[s]
qapi: Check for qapi collisions of flat union branches
qapi: Factor out QAPISchemaObjectType.check_clash()
qapi: Hoist tag collision check to Variants.check()
qapi: Remove outdated tests related to QMP/branch collisions
qapi: Track owner of each object member
qapi: Detect collisions in C member names
cpu: Convert CpuInfo into flat union
qapi: Forbid case-insensitive clashes
Markus Armbruster (7):
qapi: Drop obsolete tag value collision assertions
qapi: Simplify QAPISchemaObjectTypeMember.check()
qapi: Clean up after previous commit
qapi: Fix up commit 7618b91's clash sanity checking change
qapi: Eliminate QAPISchemaObjectType.check() variable members
qapi: Factor out QAPISchemaObjectTypeMember.check_clash()
qapi: Simplify QAPISchemaObjectTypeVariants.check()
cpus.c | 31 ++-
docs/qapi-code-gen.txt | 31 ++-
hmp.c | 30 ++-
include/qapi/qmp/qobject.h | 1 +
qapi-schema.json | 120 +++++++--
qapi/introspect.json | 17 +-
scripts/qapi-introspect.py | 8 +-
scripts/qapi-types.py | 68 ++----
scripts/qapi-visit.py | 9 +-
scripts/qapi.py | 135 ++++++----
tests/Makefile | 4 +-
tests/qapi-schema/args-case-clash.err | 1 +
...{union-clash-type.exit => args-case-clash.exit} | 0
tests/qapi-schema/args-case-clash.json | 5 +
.../{union-clash-type.out => args-case-clash.out} | 0
tests/qapi-schema/args-name-clash.err | 1 +
tests/qapi-schema/args-name-clash.exit | 2 +-
tests/qapi-schema/args-name-clash.json | 6 +-
tests/qapi-schema/args-name-clash.out | 6 -
tests/qapi-schema/flat-union-clash-branch.err | 0
tests/qapi-schema/flat-union-clash-branch.exit | 1 -
tests/qapi-schema/flat-union-clash-branch.json | 18 --
tests/qapi-schema/flat-union-clash-branch.out | 14 --
tests/qapi-schema/flat-union-clash-type.err | 1 -
tests/qapi-schema/flat-union-clash-type.exit | 1 -
tests/qapi-schema/flat-union-clash-type.json | 14 --
tests/qapi-schema/flat-union-clash-type.out | 0
tests/qapi-schema/qapi-schema-test.json | 6 +-
tests/qapi-schema/qapi-schema-test.out | 15 +-
tests/qapi-schema/union-clash-data.out | 1 +
tests/qapi-schema/union-clash-type.err | 1 -
tests/qapi-schema/union-clash-type.json | 9 -
tests/qapi-schema/union-empty.out | 1 +
tests/test-qmp-commands.c | 8 +-
tests/test-qmp-common.h | 22 ++
tests/test-qmp-event.c | 1 +
tests/test-qmp-input-strict.c | 131 +++-------
tests/test-qmp-input-visitor.c | 272 +++++++++------------
tests/test-qmp-output-visitor.c | 151 +++---------
tests/test-visitor-serialization.c | 76 ++----
40 files changed, 570 insertions(+), 648 deletions(-)
create mode 100644 tests/qapi-schema/args-case-clash.err
rename tests/qapi-schema/{union-clash-type.exit => args-case-clash.exit} (100%)
create mode 100644 tests/qapi-schema/args-case-clash.json
rename tests/qapi-schema/{union-clash-type.out => args-case-clash.out} (100%)
delete mode 100644 tests/qapi-schema/flat-union-clash-branch.err
delete mode 100644 tests/qapi-schema/flat-union-clash-branch.exit
delete mode 100644 tests/qapi-schema/flat-union-clash-branch.json
delete mode 100644 tests/qapi-schema/flat-union-clash-branch.out
delete mode 100644 tests/qapi-schema/flat-union-clash-type.err
delete mode 100644 tests/qapi-schema/flat-union-clash-type.exit
delete mode 100644 tests/qapi-schema/flat-union-clash-type.json
delete mode 100644 tests/qapi-schema/flat-union-clash-type.out
delete mode 100644 tests/qapi-schema/union-clash-type.err
delete mode 100644 tests/qapi-schema/union-clash-type.json
create mode 100644 tests/test-qmp-common.hSignature · 1 line
2.4.3
PATCH v10 1/30 qapi: Use generated TestStruct machinery in tests
Commit d88f5fd and friends first introduced the various test-qmp-* tests in 2011, with duplicated hand-rolled TestStruct machinery, to make sure the qapi visitor interface was tested. Later, commit 4f193e3 in 2013 added a .json file for further testing use by the files, but without consolidating any of the existing hand-rolled visitors. And with four copies, subtle differences have crept in, between the tests themselves (mainly whitespace differences, but also a question of whether to use NULL or "TestStruct" when calling visit_start_struct()) and from what the generator produces (the hand-rolled versions did not cater to partially-allocated objects, because they did not have a deallocation usage).
Of course, just because the visitor interface is tested does not mean it is a sane interface; and future patches will be changing some of the visitor contracts. Rather than having to duplicate the cleanup work in each copy of the TestStruct visitor, and keep each hand-rolled copy in sync with what the generator supplies, we might as well just test what the generator should give us in the first place.
---
v10: no change
v9: no change
v8: improve commit message
v7: rebase on top of subset B v9; defer unrelated trailing whitespace cleanups to later in series
v6: new patch
--- tests/qapi-schema/qapi-schema-test.json | 6 +++- tests/qapi-schema/qapi-schema-test.out | 5 +++ tests/test-qmp-input-strict.c | 35 -------------------- tests/test-qmp-input-visitor.c | 34 ------------------- tests/test-qmp-output-visitor.c | 58 --------------------------------- tests/test-visitor-serialization.c | 34 ------------------- 6 files changed, 10 insertions(+), 162 deletions(-)
tests/qapi-schema/qapi-schema-test.json +5 -1
@@ -3,6 +3,9 @@ # This file is a stress test of supported qapi constructs that must # parse and compile correctly. +{ 'struct': 'TestStruct', + 'data': { 'integer': 'int', 'boolean': 'bool', 'string': 'str' } } + # for testing enums { 'struct': 'NestedEnumsOne', 'data': { 'enum1': 'EnumOne', # Intentional forward reference
@@ -46,7 +49,8 @@ # dummy struct to force generation of array types not otherwise mentioned { 'struct': 'ForceArrays', - 'data': { 'unused1':['UserDefOne'], 'unused2':['UserDefTwo'] } } + 'data': { 'unused1':['UserDefOne'], 'unused2':['UserDefTwo'], + 'unused3':['TestStruct'] } } # for testing unions # Among other things, test that a name collision between branches does
tests/qapi-schema/qapi-schema-test.out +5 -0
@@ -92,6 +92,7 @@ object EventStructOne object ForceArrays member unused1: UserDefOneList optional=False member unused2: UserDefTwoList optional=False + member unused3: TestStructList optional=False enum MyEnum [] object NestedEnumsOne member enum1: EnumOne optional=False
@@ -100,6 +101,10 @@ object NestedEnumsOne member enum4: EnumOne optional=True enum QEnumTwo ['value1', 'value2'] prefix QENUM_TWO +object TestStruct + member integer: int optional=False + member boolean: bool optional=False + member string: str optional=False object UserDefA member boolean: bool optional=False member a_b: int optional=True
tests/test-qmp-input-strict.c +0 -35
@@ -89,41 +89,6 @@ static Visitor *validate_test_init_raw(TestInputVisitorData *data,return v; } -typedef struct TestStruct -{ -int64_t integer; -bool boolean; -char *string; -}TestStruct ; - -static void visit_type_TestStruct (Visitor *v,TestStruct **obj, -const char *name,Error **errp) -{ -Error *err =NULL ; - -visit_start_struct (v, (void **)obj,"TestStruct" , name,sizeof (TestStruct), - &err); -if (err) { - goto out; - } - -visit_type_int (v, &(*obj)->integer,"integer" , &err); -if (err) { - goto out_end; - } -visit_type_bool (v, &(*obj)->boolean,"boolean" , &err); -if (err) { - goto out_end; - } -visit_type_str (v, &(*obj)->string,"string" , &err); - -out_end: -error_propagate (errp, err); - err =NULL ; -visit_end_struct (v, &err); -out: -error_propagate (errp, err); -}static void test_validate_struct (TestInputVisitorData *data,const void *unused)
tests/test-qmp-input-visitor.c +0 -34
@@ -185,40 +185,6 @@ static void test_visitor_in_enum(TestInputVisitorData *data, data->qiv =NULL ; } -typedef struct TestStruct -{ -int64_t integer; -bool boolean; -char *string; -}TestStruct ; - -static void visit_type_TestStruct (Visitor *v,TestStruct **obj, -const char *name,Error **errp) -{ -Error *err =NULL ; - -visit_start_struct (v, (void **)obj,"TestStruct" , name,sizeof (TestStruct), - &err); -if (err) { - goto out; - } -visit_type_int (v, &(*obj)->integer,"integer" , &err); -if (err) { - goto out_end; - } -visit_type_bool (v, &(*obj)->boolean,"boolean" , &err); -if (err) { - goto out_end; - } -visit_type_str (v, &(*obj)->string,"string" , &err); - -out_end: -error_propagate (errp, err); - err =NULL ; -visit_end_struct (v, &err); -out: -error_propagate (errp, err); -}static void test_visitor_in_struct (TestInputVisitorData *data,const void *unused)
tests/test-qmp-output-visitor.c +0 -58
70 lines not on this page · raw message
tests/test-visitor-serialization.c +0 -34
@@ -186,40 +186,6 @@ static void visit_primitive_list(Visitor *v, void **native, Error **errp) } } -typedef struct TestStruct -{ -int64_t integer; -bool boolean; -char *string; -}TestStruct ; - -static void visit_type_TestStruct (Visitor *v,TestStruct **obj, -const char *name,Error **errp) -{ -Error *err =NULL ; - -visit_start_struct (v, (void **)obj,NULL , name,sizeof (TestStruct), &err); -if (err) { - goto out; - } - -visit_type_int (v, &(*obj)->integer,"integer" , &err); -if (err) { - goto out_end; - } -visit_type_bool (v, &(*obj)->boolean,"boolean" , &err); -if (err) { - goto out_end; - } -visit_type_str (v, &(*obj)->string,"string" , &err); - -out_end: -error_propagate (errp, err); - err =NULL ; -visit_end_struct (v, &err); -out: -error_propagate (errp, err); -}static TestStruct *struct_create (void ) {
Signature · 1 line
2.4.3
PATCH v10 2/30 qapi: Strengthen test of TestStructList
In reply to Eric BlakeMake each list element different, to ensure that order is preserved, and use the generated free function instead of hand-rolling our own to ensure (under valgrind) that the list is properly cleaned.
---
v10: no change
v9: no change
v8: no change
v7: new patch
--- tests/test-qmp-output-visitor.c | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-)
tests/test-qmp-output-visitor.c +7 -11
@@ -283,7 +283,7 @@ static void test_visitor_out_struct_errors(TestOutputVisitorData *data,static void test_visitor_out_list (TestOutputVisitorData *data,const void *unused) { -char *value_str = (char *)"list value" ; +const char *value_str ="list value" ;TestStructList *p, *head =NULL ;const int max_items =10 ;bool value_bool = true;
@@ -294,12 +294,13 @@ static void test_visitor_out_list(TestOutputVisitorData *data,QList *qlist;int i; +/* Build the list in reverse order... */ for (i =0 ; i < max_items; i++) { p =g_malloc0 (sizeof (*p)); p->value =g_malloc0 (sizeof (*p->value)); - p->value->integer = value_int; + p->value->integer = value_int + (max_items - i -1 ); p->value->boolean = value_bool; - p->value->string = value_str; + p->value->string =g_strdup (value_str); p->next = head; head = p;
@@ -315,6 +316,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data, qlist =qobject_to_qlist (obj);g_assert (!qlist_empty (qlist)); +/* ...and ensure that the visitor sees it in order */ i =0 ; QLIST_FOREACH_ENTRY(qlist, entry) {QDict *qdict;
@@ -322,7 +324,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data,g_assert (qobject_type (entry->value) ==QTYPE_QDICT ); qdict =qobject_to_qdict (entry->value); g_assert_cmpint(qdict_size(qdict), ==, 3); - g_assert_cmpint(qdict_get_int(qdict, "integer"), ==, value_int); + g_assert_cmpint(qdict_get_int(qdict, "integer"), ==, value_int + i); g_assert_cmpint(qdict_get_bool(qdict, "boolean"), ==, value_bool); g_assert_cmpstr(qdict_get_str(qdict, "string"), ==, value_str); i++;
@@ -330,13 +332,7 @@ static void test_visitor_out_list(TestOutputVisitorData *data, g_assert_cmpint(i, ==, max_items);QDECREF (qlist); - -for (p = head; p;) { -TestStructList *tmp = p->next; -g_free (p->value); -g_free (p); - p = tmp; - } +qapi_free_TestStructList (head); } static void test_visitor_out_list_qapi_free(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
PATCH v10 3/30 qobject: Protect against use-after-free in qobject_decref()
In reply to Eric BlakeAdding an assertion to qobject_decref() will ensure that a programming error causing use-after-free will result in immediate failure (provided no other thread has started using the memory) instead of silently attempting to wrap refcnt around and leaving the problem to potentially bite later at a harder point to diagnose.
---
v10: new patch
--- include/qapi/qmp/qobject.h | 1 + 1 file changed, 1 insertion(+)
include/qapi/qmp/qobject.h +1 -0
@@ -90,6 +90,7 @@ static inline void qobject_incref(QObject *obj)*/ static inline void qobject_decref (QObject *obj) { +assert (!obj || obj->refcnt);if (obj && --obj->refcnt ==0 ) {assert (obj->type !=NULL );assert (obj->type->destroy !=NULL );
Signature · 1 line
2.4.3
PATCH v10 4/30 qapi: Share test_init code in test-qmp-input*
In reply to Eric BlakeRather than duplicate the body of two functions just to decide between qobject_from_jsonv() and qobject_from_json(), exploit the fact that qobject_from_jsonv() intentionally takes 'va_list *' instead of the more common 'va_list', and that qobject_from_json() just calls qobject_from_jsonv(,NULL). For each file, our two existing init functions then become thin wrappers around a new internal function, and future updates to initialization don't have to be duplicated.
---
v10: new patch
--- tests/test-qmp-input-strict.c | 48 ++++++++++++++++++++--------------------- tests/test-qmp-input-visitor.c | 49 ++++++++++++++++++++---------------------- 2 files changed, 46 insertions(+), 51 deletions(-)
tests/test-qmp-input-strict.c +23 -25
@@ -40,9 +40,27 @@ static void validate_teardown(TestInputVisitorData *data, } } -/* This is provided instead of a test setup function so that the JSON -string used by the tests are kept in the test functions (and not -int main()) */ +/* The various test_init functions are provided instead of a test setup +function so that the JSON string used by the tests are kept in the test +functions (and not int main()). */ +static Visitor *validate_test_init_internal (TestInputVisitorData *data, +const char *json_string, +va_list *ap) +{ +Visitor *v; + + data->obj =qobject_from_jsonv (json_string, ap); +g_assert (data->obj); + + data->qiv =qmp_input_visitor_new_strict (data->obj); +g_assert (data->qiv); + + v =qmp_input_get_visitor (data->qiv); +g_assert (v); + +return v; +} + static GCC_FMT_ATTR(2, 3) Visitor *validate_test_init(TestInputVisitorData *data, const char *json_string, ...)
@@ -51,17 +69,8 @@ Visitor *validate_test_init(TestInputVisitorData *data,va_list ap;va_start (ap, json_string); - data->obj =qobject_from_jsonv (json_string, &ap); + v =validate_test_init_internal (data, json_string, &ap);va_end (ap); - -g_assert (data->obj !=NULL ); - - data->qiv =qmp_input_visitor_new_strict (data->obj); -g_assert (data->qiv !=NULL ); - - v =qmp_input_get_visitor (data->qiv); -g_assert (v !=NULL ); -return v; }
@@ -75,18 +84,7 @@ Visitor *validate_test_init(TestInputVisitorData *data,static Visitor *validate_test_init_raw (TestInputVisitorData *data,const char *json_string) { -Visitor *v; - - data->obj =qobject_from_json (json_string); -g_assert (data->obj !=NULL ); - - data->qiv =qmp_input_visitor_new_strict (data->obj); -g_assert (data->qiv !=NULL ); - - v =qmp_input_get_visitor (data->qiv); -g_assert (v !=NULL ); - -return v; +return validate_test_init_internal (data, json_string,NULL ); }
tests/test-qmp-input-visitor.c +23 -26
68 lines not on this page · raw message
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 04/30] qapi: Share test_init code in test-qmp-input*
Quoted 5 lines · Eric Blake
Rather than duplicate the body of two functions just to decide between
qobject_from_jsonv()andqobject_from_json(), exploit the fact thatqobject_from_jsonv()intentionally takes 'va_list*' instead of the more common 'va_list', and thatqobject_from_json()just callsqobject_from_jsonv(,NULL).
Maximizes the common code. Relies on qobject_from_jsonv() treating a null ap argument specially. Okay, but I would've left the initialization of data->obj alone, to avoid the less than obvious null argument.
Two trivial remarks inline.
Quoted 19 lines
For each file, our two existing init functions then become thin wrappers around a new internal function, and future updates to initialization don't have to be duplicated.
Suggested-by Markus Armbruster <[email protected]>Signed-off-by Eric Blake <[email protected]>---
v10: new patch--- tests/test-qmp-input-strict.c | 48 ++++++++++++++++++++--------------------- tests/test-qmp-input-visitor.c | 49 ++++++++++++++++++++---------------------- 2 files changed, 46 insertions(+), 51 deletions(-)tests/test-qmp-input-strict.c +0
-0
tests/test-qmp-input-strict.c static void validate_teardown(TestInputVisitorData *data, · line 45
Quoted 9 lines
tests/test-qmp-input-strict.c +3
-3@@ -40,9 +40,27 @@ static void validate_teardown(TestInputVisitorData *data, } } -/* This is provided instead of a test setup function so that the JSON -string used by the tests are kept in the test functions (and not -int main()) */ +/* The various test_init functions are provided instead of a test setup +function so that the JSON string used by the tests are kept in the test +functions (and not int main()). */
Since you're touching it: "not in main()".
Quoted 65 lines
tests/test-qmp-input-strict.c +20
-22+static Visitor *validate_test_init_internal (TestInputVisitorData *data, +const char *json_string, +va_list *ap) +{ +Visitor *v; + + data->obj =qobject_from_jsonv (json_string, ap); +g_assert (data->obj); + + data->qiv =qmp_input_visitor_new_strict (data->obj); +g_assert (data->qiv); + + v =qmp_input_get_visitor (data->qiv); +g_assert (v); + +return v; +} + static GCC_FMT_ATTR(2, 3) Visitor *validate_test_init(TestInputVisitorData *data, const char *json_string, ...)@@ -51,17 +69,8 @@ Visitor *validate_test_init(TestInputVisitorData *data,va_list ap;va_start (ap, json_string); - data->obj =qobject_from_jsonv (json_string, &ap); + v =validate_test_init_internal (data, json_string, &ap);va_end (ap); - -g_assert (data->obj !=NULL ); - - data->qiv =qmp_input_visitor_new_strict (data->obj); -g_assert (data->qiv !=NULL ); - - v =qmp_input_get_visitor (data->qiv); -g_assert (v !=NULL ); -return v; }@@ -75,18 +84,7 @@ Visitor *validate_test_init(TestInputVisitorData *data,static Visitor *validate_test_init_raw (TestInputVisitorData *data,const char *json_string) { -Visitor *v; - - data->obj =qobject_from_json (json_string); -g_assert (data->obj !=NULL ); - - data->qiv =qmp_input_visitor_new_strict (data->obj); -g_assert (data->qiv !=NULL ); - - v =qmp_input_get_visitor (data->qiv); -g_assert (v !=NULL ); - -return v; +return validate_test_init_internal (data, json_string,NULL ); }tests/test-qmp-input-visitor.c +0
-0
tests/test-qmp-input-visitor.c static void visitor_input_teardown(TestInputVisitorData *data, · line 41
Quoted 9 lines
tests/test-qmp-input-visitor.c +3
-3@@ -36,9 +36,27 @@ static void visitor_input_teardown(TestInputVisitorData *data, } } -/* This is provided instead of a test setup function so that the JSON -string used by the tests are kept in the test functions (and not -int main()) */ +/* The various test_init functions are provided instead of a test setup +function so that the JSON string used by the tests are kept in the test +functions (and not int main()). */
Likewise.
Quoted 61 lines
tests/test-qmp-input-visitor.c +20
-23+static Visitor *visitor_input_test_init_internal (TestInputVisitorData *data, +const char *json_string, +va_list *ap) +{ +Visitor *v; + + data->obj =qobject_from_jsonv (json_string, ap); +g_assert (data->obj); + + data->qiv =qmp_input_visitor_new (data->obj); +g_assert (data->qiv); + + v =qmp_input_get_visitor (data->qiv); +g_assert (v); + +return v; +} + static GCC_FMT_ATTR(2, 3) Visitor *visitor_input_test_init(TestInputVisitorData *data, const char *json_string, ...)@@ -47,17 +65,8 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,va_list ap;va_start (ap, json_string); - data->obj =qobject_from_jsonv (json_string, &ap); + v =visitor_input_test_init_internal (data, json_string, &ap);va_end (ap); - -g_assert (data->obj !=NULL ); - - data->qiv =qmp_input_visitor_new (data->obj); -g_assert (data->qiv !=NULL ); - - v =qmp_input_get_visitor (data->qiv); -g_assert (v !=NULL ); -return v; }@@ -71,19 +80,7 @@ Visitor *visitor_input_test_init(TestInputVisitorData *data,static Visitor *visitor_input_test_init_raw (TestInputVisitorData *data,const char *json_string) { -Visitor *v; - - data->obj =qobject_from_json (json_string); - -g_assert (data->obj !=NULL ); - - data->qiv =qmp_input_visitor_new (data->obj); -g_assert (data->qiv !=NULL ); - - v =qmp_input_get_visitor (data->qiv); -g_assert (v !=NULL ); - -return v; +return visitor_input_test_init_internal (data, json_string,NULL ); } static void test_visitor_in_int(TestInputVisitorData *data,
PATCH v10 5/30 qapi: Plug leaks in test-qmp-*
In reply to Eric BlakeMake valgrind happy with the current state of the tests, so that it is easier to see if future patches introduce new memory problems without being drowned in noise. Many of the leaks were due to calling a second init without tearing down the data from an earlier visit. But since teardown is already idempotent, and we already register teardown as part of input_visitor_test_add(), it is nicer to just make init() safe to call multiple times than it is to have to make all tests call teardown.
Another common leak was forgetting to clean up an error object, after testing that an error was raised.
Another leak was in test_visitor_in_struct_nested(), failing to clean the base member of UserDefTwo. Cleaning that up left check_and_free_str() as dead code (since using the qapi_free_* takes care of recursion, and we don't want double frees).
A final leak was in test_visitor_out_any(), which was reassigning the qobj local variable to a subset of the overall structure needing freeing; it did not result in a use-after-free, but was not cleaning up all the qdict.
test-qmp-event and test-qmp-commands were already clean.
---
v10: improve commit message, split out refactor of test init so that both init and init_raw benefit from change
v9: move earlier in series (was 13/17)
v8: no change
v7: no change
v6: make init repeatable rather than adding teardown everywhere, fix additional leak with UserDefTwo base, plug additional files
--- tests/test-qmp-input-strict.c | 9 +++++++++ tests/test-qmp-input-visitor.c | 41 +++++++---------------------------------- tests/test-qmp-output-visitor.c | 3 ++- 3 files changed, 18 insertions(+), 35 deletions(-)
tests/test-qmp-input-strict.c +9 -0
@@ -49,6 +49,8 @@ static Visitor *validate_test_init_internal(TestInputVisitorData *data, {Visitor *v; +validate_teardown (data,NULL ); + data->obj =qobject_from_jsonv (json_string, ap);g_assert (data->obj);
@@ -191,6 +193,7 @@ static void test_validate_fail_struct(TestInputVisitorData *data,visit_type_TestStruct (v, &p,NULL , &err);g_assert (err); +error_free (err);if (p) {g_free (p->string); }
@@ -208,6 +211,7 @@ static void test_validate_fail_struct_nested(TestInputVisitorData *data,visit_type_UserDefTwo (v, &udp,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefTwo (udp); }
@@ -222,6 +226,7 @@ static void test_validate_fail_list(TestInputVisitorData *data,visit_type_UserDefOneList (v, &head,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefOneList (head); }
@@ -237,6 +242,7 @@ static void test_validate_fail_union_native_list(TestInputVisitorData *data,visit_type_UserDefNativeListUnion (v, &tmp,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefNativeListUnion (tmp); }
@@ -251,6 +257,7 @@ static void test_validate_fail_union_flat(TestInputVisitorData *data,visit_type_UserDefFlatUnion (v, &tmp,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefFlatUnion (tmp); }
@@ -266,6 +273,7 @@ static void test_validate_fail_union_flat_no_discrim(TestInputVisitorData *data,visit_type_UserDefFlatUnion2 (v, &tmp,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefFlatUnion2 (tmp); }
@@ -280,6 +288,7 @@ static void test_validate_fail_alternate(TestInputVisitorData *data,visit_type_UserDefAlternate (v, &tmp,NULL , &err);g_assert (err); +error_free (err);qapi_free_UserDefAlternate (tmp); }
tests/test-qmp-input-visitor.c +7 -34
159 lines not on this page · raw message
tests/test-qmp-output-visitor.c +2 -1
@@ -391,6 +391,7 @@ static void test_visitor_out_any(TestOutputVisitorData *data, qobj =QOBJECT (qdict);visit_type_any (data->ov, &qobj,NULL , &err);g_assert (!err); +qobject_decref (qobj); obj =qmp_output_get_qobject (data->qov);g_assert (obj !=NULL ); qdict =qobject_to_qdict (obj);
@@ -411,7 +412,6 @@ static void test_visitor_out_any(TestOutputVisitorData *data,g_assert (qstring); g_assert_cmpstr(qstring_get_str(qstring), ==, "foo");qobject_decref (obj); -qobject_decref (qobj); } static void test_visitor_out_union_flat(TestOutputVisitorData *data,
@@ -463,6 +463,7 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data, g_assert_cmpint(qint_get_int(qobject_to_qint(arg)), ==, 42);qapi_free_UserDefAlternate (tmp); +qobject_decref (arg); } static void test_visitor_out_empty(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 05/30] qapi: Plug leaks in test-qmp-*
Quoted 54 lines · Eric Blake
Make valgrind happy with the current state of the tests, so that it is easier to see if future patches introduce new memory problems without being drowned in noise. Many of the leaks were due to calling a second init without tearing down the data from an earlier visit. But since teardown is already idempotent, and we already register teardown as part of
input_visitor_test_add(), it is nicer to just makeinit()safe to call multiple times than it is to have to make all tests call teardown.Another common leak was forgetting to clean up an error object, after testing that an error was raised.
Another leak was in
test_visitor_in_struct_nested(), failing to clean the base member of UserDefTwo. Cleaning that up leftcheck_and_free_str()as dead code (since using the qapi_free_* takes care of recursion, and we don't want double frees).A final leak was in
test_visitor_out_any(), which was reassigning the qobj local variable to a subset of the overall structure needing freeing; it did not result in a use-after-free, but was not cleaning up all the qdict.test-qmp-event and test-qmp-commands were already clean.
Signed-off-by Eric Blake <[email protected]>---
v10: improve commit message, split out refactor of test init so that both init andinit_rawbenefit from change
v9: move earlier in series (was 13/17)
v8: no change
v7: no change
v6: make init repeatable rather than adding teardown everywhere, fix additional leak with UserDefTwo base, plug additional files--- tests/test-qmp-input-strict.c | 9 +++++++++ tests/test-qmp-input-visitor.c | 41 +++++++---------------------------------- tests/test-qmp-output-visitor.c | 3 ++- 3 files changed, 18 insertions(+), 35 deletions(-)tests/test-qmp-input-strict.c +2
-0@@ -49,6 +49,8 @@ static Visitor *validate_test_init_internal(TestInputVisitorData *data, {Visitor *v; +validate_teardown (data,NULL ); + data->obj =qobject_from_jsonv (json_string, ap);g_assert (data->obj);
tests/test-qmp-input-strict.c static void test_validate_fail_struct(TestInputVisitorData *data, · line 196
Quoted 4 lines
tests/test-qmp-input-strict.c +1
-0@@ -191,6 +193,7 @@ static void test_validate_fail_struct(TestInputVisitorData *data,visit_type_TestStruct (v, &p,NULL , &err);g_assert (err); +error_free (err);
v9 had /* FIXME: visitor should not allocate p when returning error */ here. Did you drop it intentionally?
If you put it back, please mention it in the commit message.
Quoted 3 lines
tests/test-qmp-input-strict.c +0
-0if (p) {g_free (p->string); }
[...]
Re: [Qemu-devel] [PATCH v10 05/30] qapi: Plug leaks in test-qmp-*
Quoted 12 lines · Markus Armbruster
Quoted 9 lines · Eric Blake
Make valgrind happy with the current state of the tests, so that it is easier to see if future patches introduce new memory problems without being drowned in noise. Many of the leaks were due to calling a second init without tearing down the data from an earlier visit. But since teardown is already idempotent, and we already register teardown as part of
input_visitor_test_add(), it is nicer to just makeinit()safe to call multiple times than it is to have to make all tests call teardown.
Quoted 9 lines
tests/test-qmp-input-strict.c static void test_validate_fail_struct(TestInputVisitorData *data, · line 196
Quoted 5 lines
tests/test-qmp-input-strict.c +1
-0@@ -191,6 +193,7 @@ static void test_validate_fail_struct(TestInputVisitorData *data,visit_type_TestStruct (v, &p,NULL , &err);g_assert (err); +error_free (err);v9 had /* FIXME: visitor should not allocate p when returning error */ here. Did you drop it intentionally?
Yes, I dropped it intentionally.
Quoted 1 line
If you put it back, please mention it in the commit message.
Rather, I decided that it was sufficient to just fix the bug later (patch already queued, but not posted), and minimize the churn (no need to mark things up just to pull down the markings later):
http://repo.or.cz/qemu/ericb.git/commitdiff/ac584b53a
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v10 6/30 qapi: Simplify non-error testing in test-qmp-*
In reply to Eric BlakeBy using &error_abort, we can avoid a local err variable in situations where we expect success. It also has the nice effect that if the test breaks, the error message from error_abort tends to be nicer than that of g_assert().
---
v10: split into two pieces
v9: move earlier in series (was 14/17), reword commit message
v8: no change
v7: pick up whitespace changes dropped from earlier commit
v6: new patch
--- tests/test-qmp-input-strict.c | 31 +++++--------------- tests/test-qmp-input-visitor.c | 59 ++++++++++---------------------------- tests/test-qmp-output-visitor.c | 56 +++++++++--------------------------- tests/test-visitor-serialization.c | 42 +++++++++++---------------- 4 files changed, 53 insertions(+), 135 deletions(-)
tests/test-qmp-input-strict.c +7 -24
@@ -94,13 +94,11 @@ static void test_validate_struct(TestInputVisitorData *data, const void *unused) {TestStruct *p =NULL ; -Error *err =NULL ;Visitor *v; v =validate_test_init (data,"{ 'integer': -42, 'boolean': true, 'string': 'foo' }" ); -visit_type_TestStruct (v, &p,NULL , &err); -g_assert (!err); +visit_type_TestStruct (v, &p,NULL , &error_abort);g_free (p->string);g_free (p); }
@@ -109,7 +107,6 @@ static void test_validate_struct_nested(TestInputVisitorData *data, const void *unused) {UserDefTwo *udp =NULL ; -Error *err =NULL ;Visitor *v; v = validate_test_init(data, "{ 'string0': 'string0', "
@@ -117,8 +114,7 @@ static void test_validate_struct_nested(TestInputVisitorData *data,"'dict2': { 'userdef': { 'integer': 42, " "'string': 'string' }, 'string': 'string2'}}}"); -visit_type_UserDefTwo (v, &udp,NULL , &err); -g_assert (!err); +visit_type_UserDefTwo (v, &udp,NULL , &error_abort);qapi_free_UserDefTwo (udp); }
@@ -126,13 +122,11 @@ static void test_validate_list(TestInputVisitorData *data, const void *unused) {UserDefOneList *head =NULL ; -Error *err =NULL ;Visitor *v; v =validate_test_init (data,"[ { 'string': 'string0', 'integer': 42 }, { 'string': 'string1', 'integer': 43 }, { 'string': 'string2', 'integer': 44 } ]" ); -visit_type_UserDefOneList (v, &head,NULL , &err); -g_assert (!err); +visit_type_UserDefOneList (v, &head,NULL , &error_abort);qapi_free_UserDefOneList (head); }
@@ -141,12 +135,10 @@ static void test_validate_union_native_list(TestInputVisitorData *data, {UserDefNativeListUnion *tmp =NULL ;Visitor *v; -Error *err =NULL ; v =validate_test_init (data,"{ 'type': 'integer', 'data' : [ 1, 2 ] }" ); -visit_type_UserDefNativeListUnion (v, &tmp,NULL , &err); -g_assert (!err); +visit_type_UserDefNativeListUnion (v, &tmp,NULL , &error_abort);qapi_free_UserDefNativeListUnion (tmp); }
@@ -155,7 +147,6 @@ static void test_validate_union_flat(TestInputVisitorData *data, {UserDefFlatUnion *tmp =NULL ;Visitor *v; -Error *err =NULL ; v =validate_test_init (data, "{ 'enum1': 'value1', "
@@ -163,8 +154,7 @@ static void test_validate_union_flat(TestInputVisitorData *data,"'string': 'str', " "'boolean': true }"); -visit_type_UserDefFlatUnion (v, &tmp,NULL , &err); -g_assert (!err); +visit_type_UserDefFlatUnion (v, &tmp,NULL , &error_abort);qapi_free_UserDefFlatUnion (tmp); }
@@ -173,12 +163,10 @@ static void test_validate_alternate(TestInputVisitorData *data, {UserDefAlternate *tmp =NULL ;Visitor *v; -Error *err =NULL ; v =validate_test_init (data,"42" ); -visit_type_UserDefAlternate (v, &tmp,NULL , &err); -g_assert (!err); +visit_type_UserDefAlternate (v, &tmp,NULL , &error_abort);qapi_free_UserDefAlternate (tmp); }
@@ -296,16 +284,11 @@ static void do_test_validate_qmp_introspect(TestInputVisitorData *data, const char *schema_json) {SchemaInfoList *schema =NULL ; -Error *err =NULL ;Visitor *v; v =validate_test_init_raw (data, schema_json); -visit_type_SchemaInfoList (v, &schema,NULL , &err); -if (err) { -fprintf (stderr,"%s" ,error_get_pretty (err)); - } -g_assert (!err); +visit_type_SchemaInfoList (v, &schema,NULL , &error_abort);g_assert (schema);qapi_free_SchemaInfoList (schema);
tests/test-qmp-input-visitor.c +15 -44
223 lines not on this page · raw message
tests/test-qmp-output-visitor.c +14 -42
193 lines not on this page · raw message
tests/test-visitor-serialization.c +17 -25
103 lines not on this page · raw message
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 06/30] qapi: Simplify non-error testing in test-qmp-*
Quoted 6 lines · Eric Blake
By using
&error_abort, we can avoid a local err variable in situations where we expect success. It also has the nice effect that if the test breaks, the error message fromerror_aborttends to be nicer than that ofg_assert().Signed-off-by Eric Blake <[email protected]>
[Boring mechanical changes snipped...]
Quoted 9 lines
tests/test-visitor-serialization.c +0
-1@@ -302,14 +302,13 @@ static void test_primitives(gconstpointer opaque)const SerializeOps *ops = args->ops;PrimitiveType *pt = args->test_data;PrimitiveType *pt_copy =g_malloc0 (sizeof (*pt_copy)); -Error *err =NULL ;
tests/test-visitor-serialization.c static void test_primitives(gconstpointer opaque) · line 312
Quoted 10 lines
tests/test-visitor-serialization.c +3
-3void *serialize_data; pt_copy->type = pt->type; - ops->serialize (pt, &serialize_data, visit_primitive_type, &err); - ops->deserialize ((void **)&pt_copy, serialize_data, visit_primitive_type, &err); + ops->serialize (pt, &serialize_data, visit_primitive_type, &error_abort); + ops->deserialize ((void **)&pt_copy, serialize_data, visit_primitive_type, + &error_abort); -g_assert (err ==NULL );
This looks like a (very minor) bug fix / cleanup: you're not supposed to pass the same &err to multiple functions without checking and clearing it in between, because the second failure trips assert(*errp == NULL) in error_setv(). Harmless here, but it's nice to get rid of a bad example.
Several more below.
Quoted 86 lines
tests/test-visitor-serialization.c +11
-18g_assert (pt_copy !=NULL );if (pt->type ==PTYPE_STRING ) { g_assert_cmpstr(pt->value.string, ==, pt_copy->value.string);@@ -345,7 +344,6 @@ static void test_primitive_lists(gconstpointer opaque)PrimitiveList pl = { .value = {NULL } };PrimitiveList pl_copy = { .value = {NULL } };PrimitiveList *pl_copy_ptr = &pl_copy; -Error *err =NULL ;void *serialize_data;void *cur_head =NULL ;int i;@@ -492,10 +490,11 @@ static void test_primitive_lists(gconstpointer opaque) } } - ops->serialize ((void **)&pl, &serialize_data, visit_primitive_list, &err); - ops->deserialize ((void **)&pl_copy_ptr, serialize_data, visit_primitive_list, &err); + ops->serialize ((void **)&pl, &serialize_data, visit_primitive_list, + &error_abort); + ops->deserialize ((void **)&pl_copy_ptr, serialize_data, + visit_primitive_list, &error_abort); -g_assert (err ==NULL ); i =0 ;/* compare our deserialized list of primitives to the original */ @@ -652,10 +651,8 @@ static void test_primitive_lists(gconstpointer opaque) g_assert_cmpint(i, ==, 33); ops->cleanup (serialize_data); -dealloc_helper (&pl, visit_primitive_list, &err); -g_assert (!err); -dealloc_helper (&pl_copy, visit_primitive_list, &err); -g_assert (!err); +dealloc_helper (&pl, visit_primitive_list, &error_abort); +dealloc_helper (&pl_copy, visit_primitive_list, &error_abort);g_free (args); }@@ -665,13 +662,12 @@ static void test_struct(gconstpointer opaque)const SerializeOps *ops = args->ops;TestStruct *ts =struct_create ();TestStruct *ts_copy =NULL ; -Error *err =NULL ;void *serialize_data; - ops->serialize (ts, &serialize_data, visit_struct, &err); - ops->deserialize ((void **)&ts_copy, serialize_data, visit_struct, &err); + ops->serialize (ts, &serialize_data, visit_struct, &error_abort); + ops->deserialize ((void **)&ts_copy, serialize_data, visit_struct, + &error_abort); -g_assert (err ==NULL );struct_compare (ts, ts_copy);struct_cleanup (ts);@@ -687,14 +683,12 @@ static void test_nested_struct(gconstpointer opaque)const SerializeOps *ops = args->ops;UserDefTwo *udnp =nested_struct_create ();UserDefTwo *udnp_copy =NULL ; -Error *err =NULL ;void *serialize_data; - ops->serialize (udnp, &serialize_data, visit_nested_struct, &err); + ops->serialize (udnp, &serialize_data, visit_nested_struct, &error_abort); ops->deserialize ((void **)&udnp_copy, serialize_data, visit_nested_struct, - &err); + &error_abort); -g_assert (err ==NULL );nested_struct_compare (udnp, udnp_copy);nested_struct_cleanup (udnp);@@ -709,7 +703,6 @@ static void test_nested_struct_list(gconstpointer opaque)TestArgs *args = (TestArgs *) opaque;const SerializeOps *ops = args->ops;UserDefTwoList *listp =NULL , *tmp, *tmp_copy, *listp_copy =NULL ; -Error *err =NULL ;void *serialize_data;int i =0 ;@@ -720,11 +713,10 @@ static void test_nested_struct_list(gconstpointer opaque) listp = tmp; } - ops->serialize (listp, &serialize_data, visit_nested_struct_list, &err);
tests/test-visitor-serialization.c static void test_nested_struct_list(gconstpointer opaque) · line 722
Quoted 10 lines
tests/test-visitor-serialization.c +3
-3+ ops->serialize (listp, &serialize_data, visit_nested_struct_list, + &error_abort); ops->deserialize ((void **)&listp_copy, serialize_data, - visit_nested_struct_list, &err); - -g_assert (err ==NULL ); + visit_nested_struct_list, &error_abort); tmp = listp; tmp_copy = listp_copy;
Suggest to note the cleanup in the commit message.
Re: [Qemu-devel] [PATCH v10 06/30] qapi: Simplify non-error testing in test-qmp-*
Quoted 17 lines · Markus Armbruster
Quoted 6 lines · Eric Blake
By using
&error_abort, we can avoid a local err variable in situations where we expect success. It also has the nice effect that if the test breaks, the error message fromerror_aborttends to be nicer than that ofg_assert().Signed-off-by Eric Blake <[email protected]>[Boring mechanical changes snipped...]
Quoted 7 lines
pt_copy->type = pt->type;tests/test-visitor-serialization.c +3
-2- ops->serialize (pt, &serialize_data, visit_primitive_type, &err); - ops->deserialize ((void **)&pt_copy, serialize_data, visit_primitive_type, &err); + ops->serialize (pt, &serialize_data, visit_primitive_type, &error_abort); + ops->deserialize ((void **)&pt_copy, serialize_data, visit_primitive_type, + &error_abort);
Quoted 6 lines
Quoted 1 line
-
g_assert(err == NULL);This looks like a (very minor) bug fix / cleanup: you're not supposed to pass the same &err to multiple functions without checking and clearing it in between, because the second failure trips
assert(*errp == NULL)inerror_setv(). Harmless here, but it's nice to get rid of a bad example.
Harmless here because we are asserting that err is still NULL after the chain (which means it was NULL at all points during the chain). But I agree that it is nice to get rid of poor practice, and that adding a paragraph to the commit message to point it out would be a nice idea.
Quoted 2 lines
Suggest to note the cleanup in the commit message.
We may be close enough to take the series without needing a v11; if that's the case, and you are the one squashing in the change, how about this text:
This patch has an additional bonus of fixing several call sites that were passing &err to two different functions without checking it in between. In general that is unsafe practice; because if the first function sets an error, the second function could abort() if it tries to set a different error. We got away with it because we were asserting that err was NULL through the entire chain, but switching to &error_abort avoids the questionable practice up front.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 06/30] qapi: Simplify non-error testing in test-qmp-*
Quoted 39 lines · Eric Blake
Quoted 19 lines · Markus Armbruster
Quoted 6 lines · Eric Blake
By using
&error_abort, we can avoid a local err variable in situations where we expect success. It also has the nice effect that if the test breaks, the error message fromerror_aborttends to be nicer than that ofg_assert().Signed-off-by Eric Blake <[email protected]>[Boring mechanical changes snipped...]
Quoted 9 lines
pt_copy->type = pt->type;-
ops->serialize(pt, &serialize_data, visit_primitive_type, &err);
-ops->deserialize((void **)&pt_copy, serialize_data, visit_primitive_type, &err);
+ops->serialize(pt, &serialize_data, visit_primitive_type, &error_abort);
+ ops->deserialize((void **)&pt_copy,serialize_data,visit_primitive_type,
+&error_abort);Quoted 6 lines
Quoted 1 line
-
g_assert(err == NULL);This looks like a (very minor) bug fix / cleanup: you're not supposed to pass the same &err to multiple functions without checking and clearing it in between, because the second failure trips
assert(*errp == NULL)inerror_setv(). Harmless here, but it's nice to get rid of a bad example.Harmless here because we are asserting that err is still NULL after the chain (which means it was NULL at all points during the chain). But I agree that it is nice to get rid of poor practice, and that adding a paragraph to the commit message to point it out would be a nice idea.
Quoted 2 lines
Suggest to note the cleanup in the commit message.
We may be close enough to take the series without needing a v11; if that's the case, and you are the one squashing in the change, how about this text:
Quoted 7 lines
This patch has an additional bonus of fixing several call sites that were passing &err to two different functions without checking it in between. In general that is unsafe practice; because if the first function sets an error, the second function could
abort()if it tries to set a different error. We got away with it because we were asserting that err was NULL through the entire chain, but switching to&error_abortavoids the questionable practice up front.
Works for me.
PATCH v10 7/30 qapi: Simplify error cleanup in test-qmp-*
In reply to Eric BlakeBy moving err into data, we can let test teardown take care of cleaning up any collected error; it also gives us fewer lines of code between repeated tests where init runs teardown on our behalf.
Rather than duplicate code between .c files, I added a new test-qmp-common.h. I debated about putting error_free_or_abort() in error.h, but it seems like something that is only useful for tests.
---
v10: split in two pieces
v9: move earlier in series (was 14/17), reword commit message
v8: no change
v7: pick up whitespace changes dropped from earlier commit
v6: new patch
--- tests/test-qmp-commands.c | 8 ++++---- tests/test-qmp-common.h | 22 ++++++++++++++++++++++ tests/test-qmp-event.c | 1 + tests/test-qmp-input-strict.c | 22 ++++++++-------------- tests/test-qmp-input-visitor.c | 27 ++++++++------------------- 5 files changed, 43 insertions(+), 37 deletions(-) create mode 100644 tests/test-qmp-common.h
tests/test-qmp-commands.c +4 -4
@@ -1,12 +1,13 @@#include <glib.h> #include "qemu-common.h" +#include "test-qmp-common.h" #include "qapi/qmp/types.h" #include "test-qmp-commands.h" #include "qapi/qmp/dispatch.h" #include "qemu/module.h" #include "qapi/qmp-input-visitor.h" -#include "tests/test-qapi-types.h" -#include "tests/test-qapi-visit.h" +#include "test-qapi-types.h" +#include "test-qapi-visit.h" void qmp_user_def_cmd (Error **errp) {
@@ -225,8 +226,7 @@ static void test_dealloc_partial(void)assert (ud2->dict1 ==NULL );/* confirm & release construction error */ -assert (err !=NULL ); -error_free (err); +error_free_or_abort (&err);/* tear down partial object */ qapi_free_UserDefTwo (ud2);
tests/test-qmp-common.h new file +22 -0
@@ -0,0 +1,22 @@ +/* +* Code common to qmp/qapi unit-tests. +* +* Copyright (C) 2015 Red Hat, Inc. +* +* This work is licensed under the terms of the GNU LGPL, version 2.1 or later. +* See the COPYING.LIB file in the top-level directory. +* +*/ + +#ifndef TEST_QMP_COMMON_H__ +#define TEST_QMP_COMMON_H__ + +/* Expect an error, abort() if there is none. */ +static inline void error_free_or_abort (Error **errp) +{ +g_assert (*errp); +error_free (*errp); + *errp =NULL ; +} + +#endif
tests/test-qmp-event.c +1 -0
@@ -22,6 +22,7 @@#include "qapi/qmp/qint.h" #include "qapi/qmp/qobject.h" #include "qapi/qmp-event.h" +#include "test-qmp-common.h" typedef struct TestEventData { QDict *expect;
tests/test-qmp-input-strict.c +8 -14
@@ -15,6 +15,7 @@#include <stdarg.h> #include "qemu-common.h" +#include "test-qmp-common.h" #include "qapi/qmp-input-visitor.h" #include "test-qapi-types.h" #include "test-qapi-visit.h"
@@ -180,8 +181,7 @@ static void test_validate_fail_struct(TestInputVisitorData *data, v =validate_test_init (data,"{ 'integer': -42, 'boolean': true, 'string': 'foo', 'extra': 42 }" );visit_type_TestStruct (v, &p,NULL , &err); -g_assert (err); -error_free (err); +error_free_or_abort (&err);if (p) {g_free (p->string); }
@@ -198,8 +198,7 @@ static void test_validate_fail_struct_nested(TestInputVisitorData *data, v =validate_test_init (data,"{ 'string0': 'string0', 'dict1': { 'string1': 'string1', 'dict2': { 'userdef1': { 'integer': 42, 'string': 'string', 'extra': [42, 23, {'foo':'bar'}] }, 'string2': 'string2'}}}" );visit_type_UserDefTwo (v, &udp,NULL , &err); -g_assert (err); -error_free (err); +error_free_or_abort (&err);qapi_free_UserDefTwo (udp); }
@@ -213,8 +212,7 @@ static void test_validate_fail_list(TestInputVisitorData *data, v =validate_test_init (data,"[ { 'string': 'string0', 'integer': 42 }, { 'string': 'string1', 'integer': 43 }, { 'string': 'string2', 'integer': 44, 'extra': 'ggg' } ]" );visit_type_UserDefOneList (v, &head,NULL , &err); -g_assert (err); -error_free (err); +error_free_or_abort (&err);qapi_free_UserDefOneList (head); }
@@ -229,8 +227,7 @@ static void test_validate_fail_union_native_list(TestInputVisitorData *data, "{ 'type': 'integer', 'data' : [ 'string' ] }");visit_type_UserDefNativeListUnion (v, &tmp,NULL , &err); -g_assert (err); -error_free (err); +error_free_or_abort (&err);qapi_free_UserDefNativeListUnion (tmp); }
@@ -244,8 +241,7 @@ static void test_validate_fail_union_flat(TestInputVisitorData *data, v =validate_test_init (data,"{ 'string': 'c', 'integer': 41, 'boolean': true }" );visit_type_UserDefFlatUnion (v, &tmp,NULL , &err); -g_assert (err); -error_free (err); +error_free_or_abort (&err);qapi_free_UserDefFlatUnion (tmp); }
@@ -260,8 +256,7 @@ static void test_validate_fail_union_flat_no_discrim(TestInputVisitorData *data, v =validate_test_init (data,"{ 'integer': 42, 'string': 'c', 'string1': 'd', 'string2': 'e' }" );visit_type_UserDefFlatUnion2 (v, &tmp,NULL , &err); -g_assert (err); -error_free (err); +error_free_or_abort (&err);qapi_free_UserDefFlatUnion2 (tmp); }
@@ -275,8 +270,7 @@ static void test_validate_fail_alternate(TestInputVisitorData *data, v =validate_test_init (data,"3.14" );visit_type_UserDefAlternate (v, &tmp,NULL , &err); -g_assert (err); -error_free (err); +error_free_or_abort (&err);qapi_free_UserDefAlternate (tmp); }
tests/test-qmp-input-visitor.c +8 -19
@@ -14,6 +14,7 @@#include <stdarg.h> #include "qemu-common.h" +#include "test-qmp-common.h" #include "qapi/qmp-input-visitor.h" #include "test-qapi-types.h" #include "test-qapi-visit.h"
@@ -111,8 +112,7 @@ static void test_visitor_in_int_overflow(TestInputVisitorData *data, v =visitor_input_test_init (data,"%f" ,DBL_MAX );visit_type_int (v, &res,NULL , &err); -g_assert (err); -error_free (err); +error_free_or_abort (&err); } static void test_visitor_in_bool(TestInputVisitorData *data,
@@ -319,9 +319,7 @@ static void test_visitor_in_alternate(TestInputVisitorData *data, v =visitor_input_test_init (data,"false" );visit_type_UserDefAlternate (v, &tmp,NULL , &err); -g_assert (err); -error_free (err); - err =NULL ; +error_free_or_abort (&err);qapi_free_UserDefAlternate (tmp); }
@@ -341,9 +339,7 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data, v =visitor_input_test_init (data,"42" );visit_type_AltStrBool (v, &asb,NULL , &err); -g_assert (err); -error_free (err); - err =NULL ; +error_free_or_abort (&err);qapi_free_AltStrBool (asb);/* FIXME: Order of alternate should not affect semantics; asn should
@@ -352,9 +348,7 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,visit_type_AltStrNum (v, &asn,NULL , &err);/* FIXME g_assert_cmpint(asn->type, == ALT_STR_NUM_KIND_N); */ /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */ -g_assert (err); -error_free (err); - err =NULL ; +error_free_or_abort (&err);qapi_free_AltStrNum (asn); v =visitor_input_test_init (data,"42" );
@@ -385,9 +379,7 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data, v =visitor_input_test_init (data,"42.5" );visit_type_AltStrBool (v, &asb,NULL , &err); -g_assert (err); -error_free (err); - err =NULL ; +error_free_or_abort (&err);qapi_free_AltStrBool (asb); v =visitor_input_test_init (data,"42.5" );
@@ -404,9 +396,7 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data, v =visitor_input_test_init (data,"42.5" );visit_type_AltStrInt (v, &asi,NULL , &err); -g_assert (err); -error_free (err); - err =NULL ; +error_free_or_abort (&err);qapi_free_AltStrInt (asi); v =visitor_input_test_init (data,"42.5" );
@@ -713,12 +703,11 @@ static void test_visitor_in_errors(TestInputVisitorData *data, v =visitor_input_test_init (data,"{ 'integer': false, 'boolean': 'foo', 'string': -42 }" );visit_type_TestStruct (v, &p,NULL , &err); -g_assert (err); +error_free_or_abort (&err);/* FIXME - a failed parse should not leave a partially-allocated p * for us to clean up; this could cause callers to leak memory. */ g_assert (p->string ==NULL ); -error_free (err);g_free (p->string);g_free (p); }
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 07/30] qapi: Simplify error cleanup in test-qmp-*
Quoted 4 lines · Eric Blake
By moving err into data, we can let test teardown take care of cleaning up any collected error; it also gives us fewer lines of code between repeated tests where init runs teardown on our behalf.
I think this paragraph is no longer valid: you aren't moving err anywhere in this version.
Quoted 4 lines
Rather than duplicate code between .c files, I added a new
test-qmp-common.h. I debated about puttingerror_free_or_abort()inerror.h, but it seems like something that is only useful for tests.
Maybe, maybe not. I'd accept it into error.h.
Quoted 1 line
Signed-off-by Eric Blake <[email protected]>
Patch looks okay.
Re: [Qemu-devel] [PATCH v10 07/30] qapi: Simplify error cleanup in test-qmp-*
Quoted 9 lines · Markus Armbruster
Quoted 4 lines · Eric Blake
By moving err into data, we can let test teardown take care of cleaning up any collected error; it also gives us fewer lines of code between repeated tests where init runs teardown on our behalf.
I think this paragraph is no longer valid: you aren't moving err anywhere in this version.
D'oh. I scrubbed the code, but not the commit message. At least it was a faithful split of the v9 version of the commit, before I reworked the mechanism :)
How about:
We have several tests that perform multiple sub-actions that are expected to fail. Asserting that an error occurred, then clearing it up to prepare for the next action, turned into enough boilerplate that it was sometimes forgotten, risking memory leak or invalidating the efforts of the second action (passing a non-NULL err into a function is generally a bad idea). Encapsulate the boilerplate into a single helper function error_free_or_abort(), and consistently use it.
Quoted 7 lines
Quoted 5 lines
Rather than duplicate code between .c files, I added a new
test-qmp-common.h. I debated about puttingerror_free_or_abort()inerror.h, but it seems like something that is only useful for tests.Maybe, maybe not. I'd accept it into
error.h.
Do you want me to post a fixup patch that relocates it into error.h?
Quoted 5 lines
Quoted 2 lines
Signed-off-by Eric Blake <[email protected]>Patch looks okay.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 07/30] qapi: Simplify error cleanup in test-qmp-*
Quoted 15 lines · Eric Blake
Quoted 9 lines · Markus Armbruster
Quoted 4 lines · Eric Blake
By moving err into data, we can let test teardown take care of cleaning up any collected error; it also gives us fewer lines of code between repeated tests where init runs teardown on our behalf.
I think this paragraph is no longer valid: you aren't moving err anywhere in this version.
D'oh. I scrubbed the code, but not the commit message. At least it was a faithful split of the v9 version of the commit, before I reworked the mechanism :)
Quoted 6 lines
How about:
We have several tests that perform multiple sub-actions that are expected to fail. Asserting that an error occurred, then clearing it up to prepare for the next action, turned into enough boilerplate that it was sometimes forgotten,
If you got suitable commit SHAs handy, lets insert some here.
Quoted 4 lines
risking memory leak or invalidating the efforts
of the second action (passing a non-NULL err into a function is generally a bad idea). Encapsulate the boilerplate into a single helper functionerror_free_or_abort(), and consistently use it.
Works for me.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Rather than duplicate code between .c files, I added a new
test-qmp-common.h. I debated about puttingerror_free_or_abort()inerror.h, but it seems like something that is only useful for tests.Maybe, maybe not. I'd accept it into
error.h.Do you want me to post a fixup patch that relocates it into
error.h?
I guess the result would be slightly simpler, so go ahead.
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Eric Blake <[email protected]>Patch looks okay.
Re: [Qemu-devel] [PATCH v10 07/30] qapi: Simplify error cleanup in test-qmp-*
Quoted 17 lines · Markus Armbruster
Quoted 15 lines · Eric Blake
Quoted 9 lines · Markus Armbruster
Quoted 4 lines · Eric Blake
By moving err into data, we can let test teardown take care of cleaning up any collected error; it also gives us fewer lines of code between repeated tests where init runs teardown on our behalf.
I think this paragraph is no longer valid: you aren't moving err anywhere in this version.
D'oh. I scrubbed the code, but not the commit message. At least it was a faithful split of the v9 version of the commit, before I reworked the mechanism :)
Quoted 8 lines
Quoted 6 lines
How about:
We have several tests that perform multiple sub-actions that are expected to fail. Asserting that an error occurred, then clearing it up to prepare for the next action, turned into enough boilerplate that it was sometimes forgotten,
If you got suitable commit SHAs handy, lets insert some here.
A number of them were introduced in d88f5fd (look at test-qmp-input-visitor.c:test_validate_fail_struct_nested(), for example), and only barely cleaned up in 5/30 of this series. I'm not finding other commits off-hand, though.
Quoted 13 lines
Quoted 5 lines
risking memory leak or invalidating the efforts
of the second action (passing a non-NULL err into a function is generally a bad idea). Encapsulate the boilerplate into a single helper functionerror_free_or_abort(), and consistently use it.Works for me.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Rather than duplicate code between .c files, I added a new
test-qmp-common.h. I debated about puttingerror_free_or_abort()inerror.h, but it seems like something that is only useful for tests.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Maybe, maybe not. I'd accept it into
error.h.Do you want me to post a fixup patch that relocates it into
error.h?I guess the result would be slightly simpler, so go ahead.
Coming up soon.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 1/1 fixup! qapi: Simplify error cleanup in test-qmp-*
In reply to Eric Blake[As part of the fixup, replace the old commit message with this]
We have several tests that perform multiple sub-actions that are expected to fail. Asserting that an error occurred, then clearing it up to prepare for the next action, turned into enough boilerplate that it was sometimes forgotten (for example, a number of tests added to test-qmp-input-visitor.c in d88f5fd leaked err). Worse, if an error is not reset to NULL, we risk invalidating later use of that error (passing a non-NULL err into a function is generally a bad idea). Encapsulate the boilerplate into a single helper function error_free_or_abort(), and consistently use it.
The new function is added into error.c for use everywhere, although it is anticipated that testsuites will be the main client.
---
Drop test-qmp-common.h, and move to error.h instead
--- include/qapi/error.h | 9 +++++++++ tests/test-qmp-commands.c | 5 ++--- tests/test-qmp-common.h | 22 ---------------------- tests/test-qmp-event.c | 1 - tests/test-qmp-input-strict.c | 1 - tests/test-qmp-input-visitor.c | 1 - util/error.c | 7 +++++++ 7 files changed, 18 insertions(+), 28 deletions(-) delete mode 100644 tests/test-qmp-common.h
include/qapi/error.h +9 -0
@@ -30,6 +30,10 @@* Handle an error without reporting it (just for completeness): * error_free(err); * +* Assert than an expected error occurred, but clean it up without +* reporting it (primarily useful in testsuites): +* error_free_or_abort(&err); +* * Pass an existing error to the caller: * error_propagate(errp, err); * where Error **errp is a parameter, by convention the last one.
@@ -190,6 +194,11 @@ Error *error_copy(const Error *err);void error_free (Error *err);/* +* Convenience function to assert that *@errp is set, then silently free it. +*/ +void error_free_or_abort (Error **errp); + +/* * Convenience function to error_report() and free @err. */ void error_report_err (Error *);
tests/test-qmp-commands.c +2 -3
@@ -1,13 +1,12 @@#include <glib.h> #include "qemu-common.h" -#include "test-qmp-common.h" #include "qapi/qmp/types.h" #include "test-qmp-commands.h" #include "qapi/qmp/dispatch.h" #include "qemu/module.h" #include "qapi/qmp-input-visitor.h" -#include "test-qapi-types.h" -#include "test-qapi-visit.h" +#include "tests/test-qapi-types.h" +#include "tests/test-qapi-visit.h" void qmp_user_def_cmd (Error **errp) {
tests/test-qmp-common.h deleted +0 -22
@@ -1,22 +0,0 @@ -/* -* Code common to qmp/qapi unit-tests. -* -* Copyright (C) 2015 Red Hat, Inc. -* -* This work is licensed under the terms of the GNU LGPL, version 2.1 or later. -* See the COPYING.LIB file in the top-level directory. -* -*/ - -#ifndef TEST_QMP_COMMON_H__ -#define TEST_QMP_COMMON_H__ - -/* Expect an error, abort() if there is none. */ -static inline void error_free_or_abort (Error **errp) -{ -g_assert (*errp); -error_free (*errp); - *errp =NULL ; -} - -#endif
tests/test-qmp-event.c +0 -1
@@ -22,7 +22,6 @@#include "qapi/qmp/qint.h" #include "qapi/qmp/qobject.h" #include "qapi/qmp-event.h" -#include "test-qmp-common.h" typedef struct TestEventData { QDict *expect;
tests/test-qmp-input-strict.c +0 -1
@@ -15,7 +15,6 @@#include <stdarg.h> #include "qemu-common.h" -#include "test-qmp-common.h" #include "qapi/qmp-input-visitor.h" #include "test-qapi-types.h" #include "test-qapi-visit.h"
tests/test-qmp-input-visitor.c +0 -1
@@ -14,7 +14,6 @@#include <stdarg.h> #include "qemu-common.h" -#include "test-qmp-common.h" #include "qapi/qmp-input-visitor.h" #include "test-qapi-types.h" #include "test-qapi-visit.h"
util/error.c +7 -0
@@ -220,6 +220,13 @@ void error_free(Error *err) } } +void error_free_or_abort (Error **errp) +{ +assert (errp && *errp); +error_free (*errp); + *errp =NULL ; +} +void error_propagate (Error **dst_errp,Error *local_err) {if (!local_err) {
Signature · 1 line
2.4.3
PATCH v10 8/30 qapi: More tests of alternate output
In reply to Eric BlakeThe testsuite was only covering that we could output the 'int' branch of an alternate (no additional allocation/cleanup required). Add a test of the 'str' branch, to make sure that things still work even when a branch involves allocation.
Update to modern style of g_new0() over g_malloc0() while touching it.
---
v10: improve commit message
v9: new patch, split off of 10/17
--- tests/test-qmp-output-visitor.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
tests/test-qmp-output-visitor.c +15 -1
@@ -425,8 +425,9 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data, const void *unused) {QObject *arg; +UserDefAlternate *tmp; -UserDefAlternate *tmp =g_malloc0 (sizeof (UserDefAlternate)); + tmp =g_new0 (UserDefAlternate,1 ); tmp->type =USER_DEF_ALTERNATE_KIND_I ; tmp->u.i =42 ;
@@ -438,6 +439,19 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,qapi_free_UserDefAlternate (tmp);qobject_decref (arg); + + tmp =g_new0 (UserDefAlternate,1 ); + tmp->type =USER_DEF_ALTERNATE_KIND_S ; + tmp->u.s =g_strdup ("hello" ); + +visit_type_UserDefAlternate (data->ov, &tmp,NULL , &error_abort); + arg =qmp_output_get_qobject (data->qov); + +g_assert (qobject_type (arg) ==QTYPE_QSTRING ); + g_assert_cmpstr(qstring_get_str(qobject_to_qstring(arg)), ==, "hello"); + +qapi_free_UserDefAlternate (tmp); +qobject_decref (arg); } static void test_visitor_out_empty(TestOutputVisitorData *data,
Signature · 1 line
2.4.3
PATCH v10 9/30 qapi: Test failure in middle of array parse
In reply to Eric BlakeOur generated list visitors have the same problem as has been mentioned elsewhere (see commit 2f52e20): they allocate data even on failure. An upcoming patch will correct things to provide saner guarantees, but first we need to expose the behavior in the testsuite to ensure we aren't introducing any memory usage bugs.
There are more test cases throughout the test-qmp-input-* tests that already deal with partial allocation; a later commit will clean up all visit_type_FOO(), without marking all of the tests with FIXME at this time.
---
v10: rebase to earlier changes, more commit message text
v9: move earlier in series (was 15/17)
v8: no change
v7: no change
v6: rebase onto earlier gen_err_check() and testsuite improvements
--- scripts/qapi-visit.py | 4 ++++ tests/test-qmp-input-visitor.c | 10 +++++++++- 2 files changed, 13 insertions(+), 1 deletion(-)
scripts/qapi-visit.py +4 -0
@@ -138,6 +138,10 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error def gen_visit_list(name, element_type): + # FIXME: if *obj is NULL on entry, and the first visit_next_list() + # assigns to *obj, while a later one fails, we should clean up *obj + # rather than leaving it non-NULL. As currently written, the caller must + # call qapi_free_FOOList() to avoid a memory leak of the partial FOOList. return mcgen(''' void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error **errp)
tests/test-qmp-input-visitor.c +9 -1
@@ -699,8 +699,10 @@ static void test_visitor_in_errors(TestInputVisitorData *data,TestStruct *p =NULL ;Error *err =NULL ;Visitor *v; +strList *q =NULL ; - v =visitor_input_test_init (data,"{ 'integer': false, 'boolean': 'foo', 'string': -42 }" ); + v =visitor_input_test_init (data,"{ 'integer': false, 'boolean': 'foo', " +"'string': -42 }" );visit_type_TestStruct (v, &p,NULL , &err);error_free_or_abort (&err);
@@ -710,6 +712,12 @@ static void test_visitor_in_errors(TestInputVisitorData *data,g_free (p->string);g_free (p); + + v =visitor_input_test_init (data,"[ '1', '2', false, '3' ]" ); +visit_type_strList (v, &q,NULL , &err); +error_free_or_abort (&err); +assert (q); +qapi_free_strList (q); }int main (int argc,char **argv)
Signature · 1 line
2.4.3
PATCH v10 10/30 qapi: More tests of input arrays
In reply to Eric BlakeOur testsuite had no coverage of empty arrays, nor of what happens when the input does not match the expected type. Useful to have, especially if we start changing the visitor contracts.
I did not think it worth duplicating these additions to test-qmp-input-strict; since all strict mode does is add the ability to reject JSON input that has more keys than what the visitor expects, yet the additions in this patch error out earlier than that point regardless of whether strict mode was requested.
---
v10: rebase to earlier changes, more in commit message
v9: move earlier in series (was 16/17)
v8: no change
v7: no change
v6: new patch
--- tests/test-qmp-input-visitor.c | 52 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+)
tests/test-qmp-input-visitor.c +52 -0
@@ -231,6 +231,12 @@ static void test_visitor_in_list(TestInputVisitorData *data, }qapi_free_UserDefOneList (head); + head =NULL ; + +/* An empty list is valid */ + v =visitor_input_test_init (data,"[]" ); +visit_type_UserDefOneList (v, &head,NULL , &error_abort); +g_assert (!head); } static void test_visitor_in_any(TestInputVisitorData *data,
@@ -720,6 +726,50 @@ static void test_visitor_in_errors(TestInputVisitorData *data,qapi_free_strList (q); } +static void test_visitor_in_wrong_type (TestInputVisitorData *data, +const void *unused) +{ +TestStruct *p =NULL ; +Visitor *v; +strList *q =NULL ; +int64_t i; +Error *err =NULL ; + +/* Make sure arrays and structs cannot be confused */ + + v =visitor_input_test_init (data,"[]" ); +visit_type_TestStruct (v, &p,NULL , &err); +error_free_or_abort (&err); +g_assert (!p); + + v =visitor_input_test_init (data,"{}" ); +visit_type_strList (v, &q,NULL , &err); +error_free_or_abort (&err); +assert (!q); + +/* Make sure primitives and struct cannot be confused */ + + v =visitor_input_test_init (data,"1" ); +visit_type_TestStruct (v, &p,NULL , &err); +error_free_or_abort (&err); +g_assert (!p); + + v =visitor_input_test_init (data,"{}" ); +visit_type_int (v, &i,NULL , &err); +error_free_or_abort (&err); + +/* Make sure primitives and arrays cannot be confused */ + + v =visitor_input_test_init (data,"1" ); +visit_type_strList (v, &q,NULL , &err); +error_free_or_abort (&err); +assert (!q); + + v =visitor_input_test_init (data,"[]" ); +visit_type_int (v, &i,NULL , &err); +error_free_or_abort (&err); +} +int main (int argc,char **argv) {TestInputVisitorData in_visitor_data;
@@ -752,6 +802,8 @@ int main(int argc, char **argv) &in_visitor_data, test_visitor_in_alternate);input_visitor_test_add ("/visitor/input/errors" , &in_visitor_data, test_visitor_in_errors); +input_visitor_test_add ("/visitor/input/wrong-type" , + &in_visitor_data, test_visitor_in_wrong_type);input_visitor_test_add ("/visitor/input/alternate-number" , &in_visitor_data, test_visitor_in_alternate_number); input_visitor_test_add("/visitor/input/native_list/int",
Signature · 1 line
2.4.3
PATCH v10 11/30 qapi: Provide nicer array names in introspection
In reply to Eric BlakeFor the sake of humans reading introspection output, it is nice to have the name of implicit array types be recognizable as arrays of the underlying type. However, while this patch allows humans to skip from a command with return type "[123]" straight to the definition of type "123" without having to first inspect type "[123]", document that this shortcut should not be taken by client apps.
This makes the resulting introspection string slightly larger by default (just over 200 bytes), but it's in the noise (less than 0.3% of the overall 70k size of 'query-qmp-capabilities').
---
v10: no change
v9: no change
v8: commit message stats on the size increase
v7: no change
v6: no change
--- docs/qapi-code-gen.txt | 7 +++++-- scripts/qapi-introspect.py | 8 +++++--- 2 files changed, 10 insertions(+), 5 deletions(-)
docs/qapi-code-gen.txt +5 -2
@@ -662,11 +662,14 @@ Example: the SchemaInfo for BlockRef from section Alternate types The SchemaInfo for an array type has meta-type "array", and variant member "element-type", which names the array's element type. Array -types are implicitly defined. +types are implicitly defined. For convenience, the array's name may +resemble the element type; however, clients should examine member +"element-type" instead of making assumptions based on parsing member +"name". Example: the SchemaInfo for ['str'] - { "name": "strList", "meta-type": "array", + { "name": "[str]", "meta-type": "array", "element-type": "str" } The SchemaInfo for an enumeration type has meta-type "enum" and
scripts/qapi-introspect.py +5 -3
@@ -107,10 +107,12 @@ const char %(c_name)s[] = %(c_string)s; # characters. if isinstance(typ, QAPISchemaBuiltinType): return typ.name + if isinstance(typ, QAPISchemaArrayType): + return '[' + self._use_type(typ.element_type) + ']' return self._name(typ.name) def _gen_json(self, name, mtype, obj): - if mtype != 'command' and mtype != 'event' and mtype != 'builtin': + if mtype not in ('command', 'event', 'builtin', 'array'): name = self._name(name) obj['name'] = name obj['meta-type'] = mtype
@@ -136,8 +138,8 @@ const char %(c_name)s[] = %(c_string)s; self._gen_json(name, 'enum', {'values': values}) def visit_array_type(self, name, info, element_type): - self._gen_json(name, 'array', - {'element-type': self._use_type(element_type)}) + element = self._use_type(element_type) + self._gen_json('[' + element + ']', 'array', {'element-type': element}) def visit_object_type_flat(self, name, info, members, variants): obj = {'members': [self._gen_member(m) for m in members]}
Signature · 1 line
2.4.3
PATCH v10 12/30 qapi-introspect: Document lack of sorting
In reply to Eric Blakeqapi-code-gen.txt already claims that types, commands, and events share a common namespace; set this in stone by further documenting that our introspection output will never have collisions with the same name tied to more than one meta-type.
Our largest QMP enum currently has 125 values, our largest object type has 27 members, and the mean for each is less than
10. These sizes are small enough that the per-element overhead of O(log n) binary searching probably outweighs the speed possible with direct O(n) linear searching (a better algorithm with more overhead will only beat a leaner naive algorithm only as you scale to larger input sizes).
Arguably, the overall SchemaInfo array could be sorted by name; there, we currently have 531 entities, large enough for a binary search to be faster than linear. However, remember that we have mutually-recursive types, which means there is no topological ordering that will allow clients to learn all information about that type in a single linear pass; thus clients will want to do random access over the data, and they will probably read the introspection output into a hashtable for O(1) lookup rather than O(log n) binary searching, at which point, pre-sorting our introspection output doesn't help the client.
It doesn't help that sorting can be subjective if you introduce locales into the mix (I'm not experienced enough with Python to know for sure, but at least it looks like it defaults to sorting in the C locale even when run under a different locale). And while our current introspection output is deterministic (because we visit entities in a sorted order), we may want to change that order in the future (such as using OrderedDict to stick to .json declaration order).
For these reasons, we simply document that clients should not rely on any particular order of items in introspection output. And since it is now a documented part of the contract, we have the freedom to later rearrange output if needed, without worrying about breaking well-written clients that were already aware that they had to do linear searches.
---
v10: wording touchups based on review
v9: retitle; and swing in the opposite direction: give the user no guarantee about order
v8: no change
v7: tweak commit wording
v6: no change from v5
--- docs/qapi-code-gen.txt | 19 +++++++++++++++---- qapi/introspect.json | 17 ++++++++++++----- 2 files changed, 27 insertions(+), 9 deletions(-)
docs/qapi-code-gen.txt +15 -4
@@ -516,6 +516,10 @@ query-qmp-schema. QGA currently doesn't support introspection. query-qmp-schema returns a JSON array of SchemaInfo objects. These objects together describe the wire ABI, as defined in the QAPI schema. +There is no specified order to the SchemaInfo objects returned; a +client must search for a particular name throughout the entire array +to learn more about that name, but is at least guaranteed that there +will be no collisions between type, command, and event names. However, the SchemaInfo can't reflect all the rules and restrictions that apply to QMP. It's interface introspection (figuring out what's
@@ -596,7 +600,9 @@ any. Each element is a JSON object with members "name" (the member's name), "type" (the name of its type), and optionally "default". The member is optional if "default" is present. Currently, "default" can only have value null. Other values are reserved for future -extensions. +extensions. The "members" array is in no particular order; clients +must search the entire object when learning whether a particular +member is supported. Example: the SchemaInfo for MyType from section Struct types
@@ -610,7 +616,9 @@ Example: the SchemaInfo for MyType from section Struct types "variants" is a JSON array describing the object's variant members. Each element is a JSON object with members "case" (the value of type tag this element applies to) and "type" (the name of an object type -that provides the variant members for this type tag value). +that provides the variant members for this type tag value). The +"variants" array is in no particular order, and is not guaranteed to +list cases in the same order as the corresponding "tag" enum type. Example: the SchemaInfo for flat union BlockdevOptions from section Union types
@@ -651,7 +659,8 @@ Union types The SchemaInfo for an alternate type has meta-type "alternate", and variant member "members". "members" is a JSON array. Each element is a JSON object with member "type", which names a type. Values of the -alternate type conform to exactly one of its member types. +alternate type conform to exactly one of its member types. There is +no guarantee on the order in which "members" will be listed. Example: the SchemaInfo for BlockRef from section Alternate types
@@ -673,7 +682,9 @@ Example: the SchemaInfo for ['str'] "element-type": "str" } The SchemaInfo for an enumeration type has meta-type "enum" and -variant member "values". +variant member "values". The values are listed in no particular +order; clients must search the entire enum when learning whether a +particular value is supported. Example: the SchemaInfo for MyEnum from section Enumeration types
qapi/introspect.json +12 -5
@@ -25,6 +25,10 @@ # Returns: array of @SchemaInfo, where each element describes an # entity in the ABI: command, event, type, ... # +# The order of the various SchemaInfo is unspecified; however, all +# names are guaranteed to be unique (no name will be duplicated with +# different meta-types). +# # Note: the QAPI schema is also used to help define *internal* # interfaces, by defining QAPI types. These are not part of the QMP # wire ABI, and therefore not returned by this command.
@@ -78,7 +82,8 @@ # Commands and events have the name defined in the QAPI schema. # Unlike command and event names, type names are not part of # the wire ABI. Consequently, type names are meaningless -# strings here. +# strings here, although they are still guaranteed unique +# regardless of @meta-type. # # All references to other SchemaInfo are by name. #
@@ -130,7 +135,7 @@ # # Additional SchemaInfo members for meta-type 'enum'. # -# @values: the enumeration type's values. +# @values: the enumeration type's values, in no particular order. # # Values of this type are JSON string on the wire. #
@@ -158,14 +163,16 @@ # # Additional SchemaInfo members for meta-type 'object'. # -# @members: the object type's (non-variant) members. +# @members: the object type's (non-variant) members, in no particular order. # # @tag: #optional the name of the member serving as type tag. # An element of @members with this name must exist. # # @variants: #optional variant members, i.e. additional members that # depend on the type tag's value. Present exactly when -# @tag is present. +# @tag is present. The variants are in no particular order, +# and may even differ from the order of the values of the +# enum type of the @tag. # # Values of this type are JSON object on the wire. #
@@ -219,7 +226,7 @@ # # Additional SchemaInfo members for meta-type 'alternate'. # -# @members: the alternate type's members. +# @members: the alternate type's members, in no particular order. # The members' wire encoding is distinct, see # docs/qapi-code-gen.txt section Alternate types. #
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 12/30] qapi-introspect: Document lack of sorting
Quoted 33 lines · Eric Blake
qapi-code-gen.txt already claims that types, commands, and events share a common namespace; set this in stone by further documenting that our introspection output will never have collisions with the same name tied to more than one meta-type.
Our largest QMP enum currently has 125 values, our largest object type has 27 members, and the mean for each is less than
10. These sizes are small enough that the per-element overhead of O(log n) binary searching probably outweighs the speed possible with direct O(n) linear searching (a better algorithm with more overhead will only beat a leaner naive algorithm only as you scale to larger input sizes).Arguably, the overall SchemaInfo array could be sorted by name; there, we currently have 531 entities, large enough for a binary search to be faster than linear. However, remember that we have mutually-recursive types, which means there is no topological ordering that will allow clients to learn all information about that type in a single linear pass; thus clients will want to do random access over the data, and they will probably read the introspection output into a hashtable for O(1) lookup rather than O(log n) binary searching, at which point, pre-sorting our introspection output doesn't help the client.
It doesn't help that sorting can be subjective if you introduce locales into the mix (I'm not experienced enough with Python to know for sure, but at least it looks like it defaults to sorting in the C locale even when run under a different locale). And while our current introspection output is deterministic (because we visit entities in a sorted order), we may want to change that order in the future (such as using OrderedDict to stick to .json declaration order).
Quoted 6 lines
For these reasons, we simply document that clients should not rely on any particular order of items in introspection output. And since it is now a documented part of the contract, we have the freedom to later rearrange output if needed, without worrying about breaking well-written clients that were already aware that they had to do linear searches.
Well, any kind of search that doesn't rely on sorting. Suggest to drop the "that were already..." clause. Happy to do that on commit.
Re: [Qemu-devel] [PATCH v10 12/30] qapi-introspect: Document lack of sorting
Quoted 15 lines · Markus Armbruster
Quoted 6 lines · Eric Blake
qapi-code-gen.txt already claims that types, commands, and events share a common namespace; set this in stone by further documenting that our introspection output will never have collisions with the same name tied to more than one meta-type.
Quoted 6 lines
For these reasons, we simply document that clients should not rely on any particular order of items in introspection output. And since it is now a documented part of the contract, we have the freedom to later rearrange output if needed, without worrying about breaking well-written clients that were already aware that they had to do linear searches.
Quoted 2 lines
Well, any kind of search that doesn't rely on sorting. Suggest to drop the "that were already..." clause. Happy to do that on commit.
I don't see that phrase dropped in your v1 pull request. Worth fixing?
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 12/30] qapi-introspect: Document lack of sorting
Quoted 16 lines · Eric Blake
Quoted 15 lines · Markus Armbruster
Quoted 12 lines · Eric Blake
qapi-code-gen.txt already claims that types, commands, and events share a common namespace; set this in stone by further documenting that our introspection output will never have collisions with the same name tied to more than one meta-type.
For these reasons, we simply document that clients should not rely on any particular order of items in introspection output. And since it is now a documented part of the contract, we have the freedom to later rearrange output if needed, without worrying about breaking well-written clients that were already aware that they had to do linear searches.
Quoted 4 lines
Quoted 2 lines
Well, any kind of search that doesn't rely on sorting. Suggest to drop the "that were already..." clause. Happy to do that on commit.
I don't see that phrase dropped in your v1 pull request. Worth fixing?
Fixed in PULL v2, thanks!
PATCH v10 13/30 qapi: Track simple union tag in object.local_members
In reply to Eric BlakeWe were previously creating all unions with an empty list for local_members. However, it will make it easier to unify struct and union generation if we include the generated tag member in local_members. That way, we can have a common code pattern:
visit the base (if any), visit the local members (if any), visit the variants (if any). The local_members of a flat union remains empty (because the discriminator is already visited as part of the base). Then, by visiting tag_member.check() during AlternateType.check(), we no longer need to call it during Variants.check().
The various front end entities now exist as follows:
struct: optional base, optional local_members, no variants simple union: no base, one-element local_members, variants with tag_member
from local_members
flat union: base, no local_members, variants with tag_member from base
alternate: no base, no local_members, variants
With the new local members, we require a bit of finesse to avoid assertions in the clients.
No change to generated code.
---
v10: no change
v9: improve commit message, add comments, tweak how alternates check tag_member
v8: new patch
--- scripts/qapi-types.py | 5 ++++- scripts/qapi-visit.py | 5 ++++- scripts/qapi.py | 16 +++++++++++----- tests/qapi-schema/qapi-schema-test.out | 2 ++ tests/qapi-schema/union-clash-data.out | 1 + tests/qapi-schema/union-empty.out | 1 + 6 files changed, 23 insertions(+), 7 deletions(-)
scripts/qapi-types.py +4 -1
@@ -269,7 +269,10 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_object_type(self, name, info, base, members, variants): self._fwdecl += gen_fwd_object_or_array(name) if variants: - assert not members # not implemented + if members: + # Members other than variants.tag_member not implemented + assert len(members) == 1 + assert members[0] == variants.tag_member self.decl += gen_union(name, base, variants) else: self.decl += gen_struct(name, base, members)
scripts/qapi-visit.py +4 -1
@@ -364,7 +364,10 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor): def visit_object_type(self, name, info, base, members, variants): self.decl += gen_visit_decl(name) if variants: - assert not members # not implemented + if members: + # Members other than variants.tag_member not implemented + assert len(members) == 1 + assert members[0] == variants.tag_member self.defn += gen_visit_union(name, base, variants) else: self.defn += gen_visit_struct(name, base, members)
scripts/qapi.py +11 -5
@@ -957,6 +957,9 @@ class QAPISchemaArrayType(QAPISchemaType): class QAPISchemaObjectType(QAPISchemaType): def __init__(self, name, info, base, local_members, variants): + # struct has local_members, optional base, and no variants + # flat union has base, variants, and no local_members + # simple union has local_members, variants, and no base QAPISchemaType.__init__(self, name, info) assert base is None or isinstance(base, str) for m in local_members:
@@ -1048,10 +1051,11 @@ class QAPISchemaObjectTypeVariants(object): self.variants = variants def check(self, schema, members, seen): - if self.tag_name: + # seen is non-empty for unions, empty for alternates + if self.tag_name: # flat union self.tag_member = seen[self.tag_name] - else: - self.tag_member.check(schema, members, seen) + if seen: + assert self.tag_member in seen.itervalues() assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: vseen = dict(seen)
@@ -1085,6 +1089,7 @@ class QAPISchemaAlternateType(QAPISchemaType): self.variants = variants def check(self, schema): + self.variants.tag_member.check(schema, [], {}) self.variants.check(schema, [], {}) def json_type(self):
@@ -1270,13 +1275,14 @@ class QAPISchema(object): if tag_name: variants = [self._make_variant(key, value) for (key, value) in data.iteritems()] + members = [] else: variants = [self._make_simple_variant(key, value, info) for (key, value) in data.iteritems()] tag_member = self._make_implicit_tag(name, info, variants) + members = [tag_member] self._def_entity( - QAPISchemaObjectType(name, info, base, - self._make_members(OrderedDict(), info), + QAPISchemaObjectType(name, info, base, members, QAPISchemaObjectTypeVariants(tag_name, tag_member, variants)))
tests/qapi-schema/qapi-schema-test.out +2 -0
@@ -132,6 +132,7 @@ object UserDefFlatUnion2 case value2: UserDefB case value3: UserDefA object UserDefNativeListUnion + member type: UserDefNativeListUnionKind optional=False case integer: :obj-intList-wrapper case s8: :obj-int8List-wrapper case s16: :obj-int16List-wrapper
@@ -187,6 +188,7 @@ object __org.qemu_x-Struct object __org.qemu_x-Struct2 member array: __org.qemu_x-Union1List optional=False object __org.qemu_x-Union1 + member type: __org.qemu_x-Union1Kind optional=False case __org.qemu_x-branch: :obj-str-wrapper enum __org.qemu_x-Union1Kind ['__org.qemu_x-branch'] object __org.qemu_x-Union2
tests/qapi-schema/union-clash-data.out +1 -0
@@ -2,5 +2,6 @@ object :empty object :obj-int-wrapper member data: int optional=False object TestUnion + member type: TestUnionKind optional=False case data: :obj-int-wrapper enum TestUnionKind ['data']
tests/qapi-schema/union-empty.out +1 -0
@@ -1,3 +1,4 @@ object :empty object Union + member type: UnionKind optional=False enum UnionKind []
Signature · 1 line
2.4.3
PATCH v10 14/30 qapi-types: Consolidate gen_struct() and gen_union()
In reply to Eric BlakeThese two methods are now close enough that we can finally merge them, relying on the fact that simple unions now provide a reasonable local_members. Change gen_struct() to gen_object() that handles all forms of QAPISchemaObjectType, and rename and shrink gen_union() to gen_variants() to handle the portion of gen_object() needed when variants are present.
gen_struct_fields() now has a single caller, so it no longer needs an optional parameter; however, I did not choose to inline it into the caller.
No difference to generated code.
---
v10: no change
v9: rebase to earlier changes
v8: new patch
--- scripts/qapi-types.py | 37 +++++++++++-------------------------- 1 file changed, 11 insertions(+), 26 deletions(-)
scripts/qapi-types.py +11 -26
@@ -51,7 +51,7 @@ def gen_struct_field(member): return ret -def gen_struct_fields(local_members, base=None): +def gen_struct_fields(local_members, base): ret = '' if base:
@@ -70,7 +70,7 @@ def gen_struct_fields(local_members, base=None): return ret -def gen_struct(name, base, members): +def gen_object(name, base, members, variants): ret = mcgen(''' struct %(c_name)s {
@@ -79,11 +79,14 @@ struct %(c_name)s { ret += gen_struct_fields(members, base) + if variants: + ret += gen_variants(variants) + # Make sure that all structs have at least one field; this avoids # potential issues with attempting to malloc space for zero-length # structs in C, and also incompatibility with C++ (where an empty # struct is size 1). - if not (base and base.members) and not members: + if not (base and base.members) and not members and not variants: ret += mcgen(''' char qapi_dummy_field_for_empty_struct; ''')
@@ -140,17 +143,7 @@ const int %(c_name)s_qtypes[QTYPE_MAX] = { return ret -def gen_union(name, base, variants): - ret = mcgen(''' - -struct %(c_name)s { -''', - c_name=c_name(name)) - if base: - ret += gen_struct_fields([], base) - else: - ret += gen_struct_field(variants.tag_member) - +def gen_variants(variants): # FIXME: What purpose does data serve, besides preventing a union that # has a branch named 'data'? We use it in qapi-visit.py to decide # whether to bypass the switch statement if visiting the discriminator
@@ -159,11 +152,11 @@ struct %(c_name)s { # should not be any data leaks even without a data pointer. Or, if # 'data' is merely added to guarantee we don't have an empty union, # shouldn't we enforce that at .json parse time? - ret += mcgen(''' + ret = mcgen(''' union { /* union tag is @%(c_name)s */ void *data; ''', - c_name=c_name(variants.tag_member.name)) + c_name=c_name(variants.tag_member.name)) for var in variants.variants: # Ugly special case for simple union TODO get rid of it
@@ -176,7 +169,6 @@ struct %(c_name)s { ret += mcgen(''' } u; -}; ''') return ret
@@ -268,14 +260,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_object_type(self, name, info, base, members, variants): self._fwdecl += gen_fwd_object_or_array(name) - if variants: - if members: - # Members other than variants.tag_member not implemented - assert len(members) == 1 - assert members[0] == variants.tag_member - self.decl += gen_union(name, base, variants) - else: - self.decl += gen_struct(name, base, members) + self.decl += gen_object(name, base, members, variants) if base: self.decl += gen_upcast(name, base) self._gen_type_cleanup(name)
@@ -283,7 +268,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor): def visit_alternate_type(self, name, info, variants): self._fwdecl += gen_fwd_object_or_array(name) self._fwdefn += gen_alternate_qtypes(name, variants) - self.decl += gen_union(name, None, variants) + self.decl += gen_object(name, None, [variants.tag_member], variants) self.decl += gen_alternate_qtypes_decl(name) self._gen_type_cleanup(name)
Signature · 1 line
2.4.3
PATCH v10 15/30 qapi-types: Simplify gen_struct_field[s]
In reply to Eric BlakeSimplify gen_struct_fields() back to a single iteration over a list of fields (like it was prior to commit f87ab7f9), by moving the generated comments to gen_object(). Then, inline gen_struct_field() into its only caller.
---
v10: no change
v9: new patch
--- scripts/qapi-types.py | 40 +++++++++++++++------------------------- 1 file changed, 15 insertions(+), 25 deletions(-)
scripts/qapi-types.py +15 -25
@@ -36,48 +36,38 @@ struct %(c_name)s { c_name=c_name(name), c_type=element_type.c_type()) -def gen_struct_field(member): +def gen_struct_fields(members): ret = '' - - if member.optional: - ret += mcgen(''' + for memb in members: + if memb.optional: + ret += mcgen(''' bool has_%(c_name)s; ''', - c_name=c_name(member.name)) - ret += mcgen(''' + c_name=c_name(memb.name)) + ret += mcgen(''' %(c_type)s %(c_name)s; ''', - c_type=member.type.c_type(), c_name=c_name(member.name)) + c_type=memb.type.c_type(), c_name=c_name(memb.name)) return ret -def gen_struct_fields(local_members, base): - ret = '' +def gen_object(name, base, members, variants): + ret = mcgen(''' + +struct %(c_name)s { +''', + c_name=c_name(name)) if base: ret += mcgen(''' /* Members inherited from %(c_name)s: */ ''', c_name=base.c_name()) - for memb in base.members: - ret += gen_struct_field(memb) + ret += gen_struct_fields(base.members) ret += mcgen(''' /* Own members: */ ''') - - for memb in local_members: - ret += gen_struct_field(memb) - return ret - - -def gen_object(name, base, members, variants): - ret = mcgen(''' - -struct %(c_name)s { -''', - c_name=c_name(name)) - - ret += gen_struct_fields(members, base) + ret += gen_struct_fields(members) if variants: ret += gen_variants(variants)
Signature · 1 line
2.4.3
PATCH v10 16/30 qapi: Drop obsolete tag value collision assertions
In reply to Eric BlakeFrom: Markus Armbruster <[email protected]>
Union tag values can't clash with member names in generated C anymore since commit e4ba22b, but QAPISchemaObjectTypeVariants.check() still asserts they don't. Drop it.
Message-Id: <[email protected]>
---
v10: redo closer to Markus' original proposal
v9: new patch
--- scripts/qapi.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi.py +1 -2
@@ -1058,8 +1058,7 @@ class QAPISchemaObjectTypeVariants(object): assert self.tag_member in seen.itervalues() assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: - vseen = dict(seen) - v.check(schema, self.tag_member.type, vseen) + v.check(schema, self.tag_member.type, {}) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Signature · 1 line
2.4.3
PATCH v10 17/30 qapi: Simplify QAPISchemaObjectTypeMember.check()
In reply to Eric BlakeFrom: Markus Armbruster <[email protected]>
QAPISchemaObjectTypeMember.check() currently does four things:
1. Compute self.type
2. Accumulate members in all_members
Only one caller cares: QAPISchemaObjectType.check() uses it to compute self.members. The other callers pass a throw-away accumulator.
3. Accumulate a map from names to members in seen
Only one caller cares: QAPISchemaObjectType.check() uses it to compute its local variable seen, for self.variants.check(), which uses it to compute self.variants.tag_member from self.variants.tag_name. The other callers pass a throw-away accumulator.
4. Check for collisions
This piggyback on 3: before adding a new entry, we assert it's new.
Only one caller cares: QAPISchemaObjectType.check() uses it to assert non-variant members don't clash.
Simplify QAPISchemaObjectType.check(): move 2.-4. to QAPISchemaObjectType.check(), and drop parameters all_members and seen.
Message-Id: <[email protected]> [rebase to earlier changes that moved tag_member.check() of alternate types]
---
v10: redo closer to Markus' original proposal
v9: new patch
--- scripts/qapi.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
scripts/qapi.py +7 -7
@@ -990,7 +990,10 @@ class QAPISchemaObjectType(QAPISchemaType): assert c_name(m.name) not in seen seen[m.name] = m for m in self.local_members: - m.check(schema, members, seen) + m.check(schema) + assert m.name not in seen + seen[m.name] = m + members.append(m) if self.variants: self.variants.check(schema, members, seen) self.members = members
@@ -1027,12 +1030,9 @@ class QAPISchemaObjectTypeMember(object): self.type = None self.optional = optional - def check(self, schema, all_members, seen): - assert self.name not in seen + def check(self, schema): self.type = schema.lookup_type(self._type_name) assert self.type - all_members.append(self) - seen[self.name] = self class QAPISchemaObjectTypeVariants(object):
@@ -1066,7 +1066,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) def check(self, schema, tag_type, seen): - QAPISchemaObjectTypeMember.check(self, schema, [], seen) + QAPISchemaObjectTypeMember.check(self, schema) assert self.name in tag_type.values # This function exists to support ugly simple union special cases
@@ -1088,7 +1088,7 @@ class QAPISchemaAlternateType(QAPISchemaType): self.variants = variants def check(self, schema): - self.variants.tag_member.check(schema, [], {}) + self.variants.tag_member.check(schema) self.variants.check(schema, [], {}) def json_type(self):
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 17/30] qapi: Simplify QAPISchemaObjectTypeMember.check()
Quoted 20 lines · Eric Blake
From: Markus Armbruster <[email protected]>
QAPISchemaObjectTypeMember.check()currently does four things:1. Compute self.type
2. Accumulate members in
all_membersOnly one caller cares:
QAPISchemaObjectType.check()uses it to compute self.members. The other callers pass a throw-away accumulator.3. Accumulate a map from names to members in seen
Only one caller cares:
QAPISchemaObjectType.check()uses it to compute its local variable seen, forself.variants.check(), which uses it to compute self.variants.tag_member from self.variants.tag_name. The other callers pass a throw-away accumulator.
Quoted 3 lines
4. Check for collisions
This piggyback on 3: before adding a new entry, we assert it's new.
piggybacks (typo is mine)
Quoted 8 lines
Only one caller cares:
QAPISchemaObjectType.check()uses it to assert non-variant members don't clash.Simplify
QAPISchemaObjectType.check(): move 2.-4. toQAPISchemaObjectType.check(), and drop parametersall_membersand seen.
Quoted 5 lines
Signed-off-by Markus Armbruster <[email protected]>Message-Id: <[email protected]> [rebase to earlier changes that moved
tag_member.check()of alternate types]Signed-off-by Eric Blake <[email protected]>
Patch is as good as ever ;)
Re: [Qemu-devel] [PATCH v10 17/30] qapi: Simplify QAPISchemaObjectTypeMember.check()
Quoted 4 lines · Markus Armbruster
Quoted 2 lines · Eric Blake
From: Markus Armbruster <[email protected]>
Quoted 8 lines
Quoted 6 lines
QAPISchemaObjectTypeMember.check()currently does four things:4. Check for collisions
This piggyback on 3: before adding a new entry, we assert it's new.
piggybacks (typo is mine)
What's more, I noticed it in v9, but in redoing v10 to go closer to your original series, I reverted my touchup. Oh well :)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v10 18/30 qapi: Clean up after previous commit
In reply to Eric BlakeFrom: Markus Armbruster <[email protected]>
QAPISchemaObjectTypeVariants.check() parameter members and QAPISchemaObjectTypeVariant.check() parameter seen are no longer used, drop them.
Message-Id: <[email protected]> [rebase to earlier changes that moved tag_member.check() of alternate types]
---
v10: redo closer to Markus' original proposal
v9: new patch
--- scripts/qapi.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
scripts/qapi.py +5 -6
@@ -995,7 +995,7 @@ class QAPISchemaObjectType(QAPISchemaType): seen[m.name] = m members.append(m) if self.variants: - self.variants.check(schema, members, seen) + self.variants.check(schema, seen) self.members = members def is_implicit(self):
@@ -1050,22 +1050,21 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member = tag_member self.variants = variants - def check(self, schema, members, seen): - # seen is non-empty for unions, empty for alternates + def check(self, schema, seen): if self.tag_name: # flat union self.tag_member = seen[self.tag_name] if seen: assert self.tag_member in seen.itervalues() assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: - v.check(schema, self.tag_member.type, {}) + v.check(schema, self.tag_member.type) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def __init__(self, name, typ): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) - def check(self, schema, tag_type, seen): + def check(self, schema, tag_type): QAPISchemaObjectTypeMember.check(self, schema) assert self.name in tag_type.values
@@ -1089,7 +1088,7 @@ class QAPISchemaAlternateType(QAPISchemaType): def check(self, schema): self.variants.tag_member.check(schema) - self.variants.check(schema, [], {}) + self.variants.check(schema, {}) def json_type(self): return 'value'
Signature · 1 line
2.4.3
PATCH v10 19/30 qapi: Fix up commit 7618b91's clash sanity checking change
In reply to Eric BlakeFrom: Markus Armbruster <[email protected]>
This hunk
@@ -964,6 +965,7 @@ class QAPISchemaObjectType(QAPISchemaType):
members = []
seen = {}
for m in members:
+ assert c_name(m.name) not in seen
seen[m.name] = m
for m in self.local_members:
m.check(schema, members, seen)
is plainly broken.
Asserting the members inherited from base don't clash is somewhat redundant, because self.base.check() just checked that. But it doesn't hurt.
The idea to use c_name(m.name) instead of m.name for collision checking is sound, because we need to catch clashes between the m.name and between the c_name(m.name), and when two m.name clash, then their c_name() also clash.
However, using c_name(m.name) instead of m.name in one of several places doesn't work. See the very next line.
Keep the assertion, but drop the c_name() for now. A future commit will bring it back.
Message-Id: <[email protected]>
---
v10: redo closer to Markus' original proposal
v9: new patch
--- scripts/qapi.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi.py +1 -1
@@ -987,7 +987,7 @@ class QAPISchemaObjectType(QAPISchemaType): members = [] seen = {} for m in members: - assert c_name(m.name) not in seen + assert m.name not in seen seen[m.name] = m for m in self.local_members: m.check(schema)
Signature · 1 line
2.4.3
PATCH v10 20/30 qapi: Eliminate QAPISchemaObjectType.check() variable members
In reply to Eric BlakeFrom: Markus Armbruster <[email protected]>
We can use seen.values() instead if we make it an OrderedDict.
Message-Id: <[email protected]>
---
v10: redo closer to Markus' original proposal
v9: new patch
--- scripts/qapi.py | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-)
scripts/qapi.py +5 -9
@@ -977,26 +977,22 @@ class QAPISchemaObjectType(QAPISchemaType): if self.members: return self.members = False # mark as being checked + seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) assert isinstance(self.base, QAPISchemaObjectType) assert not self.base.variants # not implemented self.base.check(schema) - members = list(self.base.members) - else: - members = [] - seen = {} - for m in members: - assert m.name not in seen - seen[m.name] = m + for m in self.base.members: + assert m.name not in seen + seen[m.name] = m for m in self.local_members: m.check(schema) assert m.name not in seen seen[m.name] = m - members.append(m) if self.variants: self.variants.check(schema, seen) - self.members = members + self.members = seen.values() def is_implicit(self): # See QAPISchema._make_implicit_object_type()
Signature · 1 line
2.4.3
PATCH v10 21/30 qapi: Factor out QAPISchemaObjectTypeMember.check_clash()
In reply to Eric BlakeFrom: Markus Armbruster <[email protected]>
While there, stick in a TODO change key of seen from QAPI name to C name. Can't do it right away, because it would fail the assertion for tests/qapi-schema/args-has-clash.json.
Message-Id: <[email protected]>
---
v10: redo closer to Markus' original proposal
v9: new patch
--- scripts/qapi.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
scripts/qapi.py +7 -4
@@ -984,12 +984,10 @@ class QAPISchemaObjectType(QAPISchemaType): assert not self.base.variants # not implemented self.base.check(schema) for m in self.base.members: - assert m.name not in seen - seen[m.name] = m + m.check_clash(seen) for m in self.local_members: m.check(schema) - assert m.name not in seen - seen[m.name] = m + m.check_clash(seen) if self.variants: self.variants.check(schema, seen) self.members = seen.values()
@@ -1030,6 +1028,11 @@ class QAPISchemaObjectTypeMember(object): self.type = schema.lookup_type(self._type_name) assert self.type + def check_clash(self, seen): + # TODO change key of seen from QAPI name to C name + assert self.name not in seen + seen[self.name] = self + class QAPISchemaObjectTypeVariants(object): def __init__(self, tag_name, tag_member, variants):
Signature · 1 line
2.4.3
PATCH v10 22/30 qapi: Simplify QAPISchemaObjectTypeVariants.check()
In reply to Eric BlakeFrom: Markus Armbruster <[email protected]>
Reduce the ugly flat union / simple union conditional by doing just the essential work here, namely setting self.tag_member. Move the rest to callers.
Message-Id: <[email protected]> [rebase to earlier changes that moved tag_member.check() of alternate types, and tweak commit title and wording]
---
v10: redo closer to Markus' original proposal
v9: new patch
--- scripts/qapi.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
scripts/qapi.py +2 -3
@@ -988,9 +988,10 @@ class QAPISchemaObjectType(QAPISchemaType): for m in self.local_members: m.check(schema) m.check_clash(seen) + self.members = seen.values() if self.variants: self.variants.check(schema, seen) - self.members = seen.values() + assert self.variants.tag_member in self.members def is_implicit(self): # See QAPISchema._make_implicit_object_type()
@@ -1052,8 +1053,6 @@ class QAPISchemaObjectTypeVariants(object): def check(self, schema, seen): if self.tag_name: # flat union self.tag_member = seen[self.tag_name] - if seen: - assert self.tag_member in seen.itervalues() assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: v.check(schema, self.tag_member.type)
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 22/30] qapi: Simplify QAPISchemaObjectTypeVariants.check()
Quoted 36 lines · Eric Blake
From: Markus Armbruster <[email protected]>
Reduce the ugly flat union / simple union conditional by doing just the essential work here, namely setting self.tag_member. Move the rest to callers.
Signed-off-by Markus Armbruster <[email protected]>Message-Id: <[email protected]> [rebase to earlier changes that moved
tag_member.check()of alternate types, and tweak commit title and wording]Signed-off-by Eric Blake <[email protected]>---
v10: redo closer to Markus' original proposal
v9: new patch--- scripts/qapi.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)scripts/qapi.py +2
-1@@ -988,9 +988,10 @@ class QAPISchemaObjectType(QAPISchemaType): for m in self.local_members: m.check(schema) m.check_clash(seen) + self.members = seen.values() if self.variants: self.variants.check(schema, seen) - self.members = seen.values() + assert self.variants.tag_member in self.members def is_implicit(self): # See QAPISchema._make_implicit_object_type()
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1055
Quoted 3 lines
scripts/qapi.py +0
-0@@ -1052,8 +1053,6 @@ class QAPISchemaObjectTypeVariants(object): def check(self, schema, seen): if self.tag_name: # flat union self.tag_member = seen[self.tag_name]
My patch has:
- if self.tag_name: # flat union
+ if not self.tag_member: # flat union
self.tag_member = seen[self.tag_name]
Any particular reason for dropping it?
I like my change, because I feel it makes the assignment's purpose more
obvious: ensure .tag_member is set.
Quoted 5 lines
scripts/qapi.py +0
-2- if seen: - assert self.tag_member in seen.itervalues() assert isinstance(self.tag_member.type, QAPISchemaEnumType) for v in self.variants: v.check(schema, self.tag_member.type)
Re: [Qemu-devel] [PATCH v10 22/30] qapi: Simplify QAPISchemaObjectTypeVariants.check()
Quoted 16 lines · Markus Armbruster
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1055
Quoted 11 lines · Eric Blake
From: Markus Armbruster <[email protected]>
Reduce the ugly flat union / simple union conditional by doing just the essential work here, namely setting self.tag_member. Move the rest to callers.
scripts/qapi.py +0
-0@@ -1052,8 +1053,6 @@ class QAPISchemaObjectTypeVariants(object): def check(self, schema, seen): if self.tag_name: # flat union self.tag_member = seen[self.tag_name]My patch has:
Quoted 5 lines
- if self.tag_name: # flat union
+ if not self.tag_member: # flat union
self.tag_member = seen[self.tag_name]Any particular reason for dropping it?
Not really; I couldn't find anything in later patches that cared either way.
Quoted 3 lines
I like my change, because I feel it makes the assignment's purpose more
obvious: ensure .tag_member is set.
Sure, we'll go with your approach.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v10 23/30 qapi: Check for qapi collisions of flat union branches
In reply to Eric BlakeRight now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want QAPISchemaObjectType.check() to make the same check, so we can later reduce some of the ad hoc checks.
We already ensure that all branches of a flat union are qapi structs with no variants, at which point those members appear in the same JSON object as all non-variant members. And we already have a map 'seen' of all non-variant members. All we need is a new QAPISchemaObjectTypeVariants.check_clash(), which clones the seen map then checks for clashes with each member of the variant's qapi type.
Note that the clone of seen inside Variants.check_clash() resembles the one we just removed from Variants.check(); the difference here is that we are now checking for clashes among the qapi members of the variant type, rather than for a single clash with the variant tag name itself.
In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to variant.type.check() in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (and type.check() is already set up to allow multiple calls due to base types).
For simple unions, the same code happens to work by design, because of our synthesized wrapper classes (however, the wrapper has a single member 'data' which will never collide with the one non-variant member 'type', so it doesn't really matter).
There is no impact to alternates, which intentionally do not need to call variants.check_clash() (there, at most one of the variant's branches will be an ObjectType, and even if one exists, we are not inlining the qapi members of that object into a parent object, the way we do for unions).
No change to generated code.
---
v10: create new Variants.check_clash() rather than piggybacking on .check()
v9: new patch, split off from v8 7/17
--- scripts/qapi.py | 12 ++++++++++++ 1 file changed, 12 insertions(+)
scripts/qapi.py +12 -0
@@ -992,6 +992,7 @@ class QAPISchemaObjectType(QAPISchemaType): if self.variants: self.variants.check(schema, seen) assert self.variants.tag_member in self.members + self.variants.check_clash(schema, seen) def is_implicit(self): # See QAPISchema._make_implicit_object_type()
@@ -1057,6 +1058,17 @@ class QAPISchemaObjectTypeVariants(object): for v in self.variants: v.check(schema, self.tag_member.type) + def check_clash(self, schema, seen): + for v in self.variants: + # Reset seen map for each variant, since qapi names from one + # branch do not affect another branch + vseen = dict(seen) + assert isinstance(v.type, QAPISchemaObjectType) + assert not v.type.variants # not implemented + v.type.check(schema) + for m in v.type.members: + m.check_clash(vseen) + class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def __init__(self, name, typ):
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches
Quoted 21 lines · Eric Blake
Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want
QAPISchemaObjectType.check()to make the same check, so we can later reduce some of the ad hoc checks.We already ensure that all branches of a flat union are qapi structs with no variants, at which point those members appear in the same JSON object as all non-variant members. And we already have a map 'seen' of all non-variant members. All we need is a new
QAPISchemaObjectTypeVariants.check_clash(), which clones the seen map then checks for clashes with each member of the variant's qapi type.Note that the clone of seen inside
Variants.check_clash()resembles the one we just removed fromVariants.check(); the difference here is that we are now checking for clashes among the qapi members of the variant type, rather than for a single clash with the variant tag name itself.
Quoted 6 lines
In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to
variant.type.check()in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (andtype.check()is already set up to allow multiple calls due to base types).
Yes, a type cannot contain itself, neither as base nor as variant.
We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can see, we don't have tests covering the latter. Do we catch it?
Quoted 12 lines
For simple unions, the same code happens to work by design, because of our synthesized wrapper classes (however, the wrapper has a single member 'data' which will never collide with the one non-variant member 'type', so it doesn't really matter).
There is no impact to alternates, which intentionally do not need to call
variants.check_clash()(there, at most one of the variant's branches will be an ObjectType, and even if one exists, we are not inlining the qapi members of that object into a parent object, the way we do for unions).
Quoted 3 lines
No change to generated code.
Signed-off-by Eric Blake <[email protected]>
Patch looks good.
Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches
Quoted 30 lines · Markus Armbruster
Quoted 21 lines · Eric Blake
Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want
QAPISchemaObjectType.check()to make the same check, so we can later reduce some of the ad hoc checks.We already ensure that all branches of a flat union are qapi structs with no variants, at which point those members appear in the same JSON object as all non-variant members. And we already have a map 'seen' of all non-variant members. All we need is a new
QAPISchemaObjectTypeVariants.check_clash(), which clones the seen map then checks for clashes with each member of the variant's qapi type.Note that the clone of seen inside
Variants.check_clash()resembles the one we just removed fromVariants.check(); the difference here is that we are now checking for clashes among the qapi members of the variant type, rather than for a single clash with the variant tag name itself.Quoted 6 lines
In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to
variant.type.check()in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (andtype.check()is already set up to allow multiple calls due to base types).
Quoted 17 lines
Yes, a type cannot contain itself, neither as base nor as variant.
We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can see, we don't have tests covering the latter. Do we catch it?
Quoted 11 lines
For simple unions, the same code happens to work by design, because of our synthesized wrapper classes (however, the wrapper has a single member 'data' which will never collide with the one non-variant member 'type', so it doesn't really matter).
There is no impact to alternates, which intentionally do not need to call
variants.check_clash()(there, at most one of the variant's branches will be an ObjectType, and even if one exists, we are not inlining the qapi members of that object into a parent object, the way we do for unions).
Yes. QAPISchemaObjectTypeVariants.check_clash() checks for each variant's members clashing with other members in the same name space. For alternates, there are no such other members.
That said, should we add a comment to QAPISchemaAlternateType.check()? Perhaps:
# Not calling self.variant.check_clash(), because there's # nothing to clash with.
Quoted 5 lines
Quoted 3 lines
No change to generated code.
Signed-off-by Eric Blake <[email protected]>Patch looks good.
Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches
Quoted 22 lines · Markus Armbruster
Quoted 10 lines · Markus Armbruster
Quoted 7 lines · Eric Blake
Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want
QAPISchemaObjectType.check()to make the same check, so we can later reduce some of the ad hoc checks.Quoted 5 lines
Quoted 5 lines
There is no impact to alternates, which intentionally do not need to call
variants.check_clash()(there, at most one of the variant's branches will be an ObjectType, and even if one exists, we are not inlining the qapi members of that object into a parent object, the way we do for unions).Yes.
QAPISchemaObjectTypeVariants.check_clash()checks for each variant's members clashing with other members in the same name space. For alternates, there are no such other members.
Quoted 5 lines
That said, should we add a comment to
QAPISchemaAlternateType.check()? Perhaps:# Not calling
self.variant.check_clash(), because there's # nothing to clash with.
Sure; if there's another reason for a respin, I can add it here; otherwise, it's already slated to be added in my respin of the patches for redoing how alternates are laid out.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches
In reply to Markus ArmbrusterQuoted 17 lines · Markus Armbruster
Quoted 8 lines · Eric Blake
Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want
QAPISchemaObjectType.check()to make the same check, so we can later reduce some of the ad hoc checks.Quoted 6 lines
In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to
variant.type.check()in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (andtype.check()is already set up to allow multiple calls due to base types).
Quoted 5 lines
Yes, a type cannot contain itself, neither as base nor as variant.
We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can see, we don't have tests covering the latter. Do we catch it?
Yes, at least by virtue of the ad hoc tests: attempting to reuse a base type of the flat union as a variant member will cause the qapi members of the base type to appear more than once in the JSON object (that is, the checks that reject flat-union-clash-member.json would also reject this scenario). To test:
(quoted diff) +0 -0
w/tests/qapi-schema/qapi-schema-test.json
index 44638da..16b2ffb 100644
tests/qapi-schema/qapi-schema-test.json +1 -1
@@ -67,7 +67,7 @@ 'discriminator': 'enum1', 'data': { 'value1' : 'UserDefA', 'value2' : 'UserDefB', - 'value3' : 'UserDefB' } } + 'value3' : 'UserDefUnionBase' } } { 'struct': 'UserDefUnionBase', 'base': 'UserDefZero',
GEN tests/test-qapi-types.h/home/eblake/qemu/tests/qapi-schema/qapi-schema-test.json:65: Member name 'string' of branch 'value3' clashes with base 'UserDefUnionBase' /home/eblake/qemu/tests/Makefile:415: recipe for target 'tests/test-qapi-types.h' failed
But you have me curious if this collision is still caught when the ad hoc tests are gone. If so, great; if not, I'll add a test here. (I'll know later when I get through rebasing to all of your comments.)
Quoted 5 lines
Quoted 3 lines
No change to generated code.
Signed-off-by Eric Blake <[email protected]>Patch looks good.
Yay; it's nice to see results after all our mental gymnastics over how collision testing should work.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches
Quoted 18 lines · Eric Blake
Quoted 17 lines · Markus Armbruster
Quoted 14 lines · Eric Blake
Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want
QAPISchemaObjectType.check()to make the same check, so we can later reduce some of the ad hoc checks.In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to
variant.type.check()in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (andtype.check()is already set up to allow multiple calls due to base types).
Quoted 4 lines
Quoted 4 lines
Yes, a type cannot contain itself, neither as base nor as variant.
We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can
Actually, these are just local, unpublished tests. They both make check_member_clash() recurse infinitely.
# Direct inheritance loop
# FIXME triggers infinite recursion
{ 'struct': 'Loopy', 'base': 'Loopy',
'data': {} }
# we reject a loop in base classes
{ 'struct': 'Base1', 'base': 'Base2', 'data': {} }
{ 'struct': 'Base2', 'base': 'Base1', 'data': {} }The latter is actually yours, proposed as base-cycle.json in
Subject: qapi: Detect collisions in C member names
Message-Id: <[email protected]>
If I disable the recursive call, the cycle detection in QAPISchemaObjectType.check() is reached, and works.
Completing the move of clash detection to check() methods should improve things from "accidental infinite recursion" to "intentional assertion failure", because it should get rid of check_member_clash() and should not break the cycle detection.
Then we can turn the assertion into a proper error message, and add the tests.
Quoted 41 lines
Quoted 1 line
see, we don't have tests covering the latter. Do we catch it?
Yes, at least by virtue of the ad hoc tests: attempting to reuse a base type of the flat union as a variant member will cause the qapi members of the base type to appear more than once in the JSON object (that is, the checks that reject flat-union-clash-member.json would also reject this scenario). To test:
(quoted diff) +0
-0
w/tests/qapi-schema/qapi-schema-test.json
index 44638da..16b2ffb 100644tests/qapi-schema/qapi-schema-test.json +1
-1@@ -67,7 +67,7 @@ 'discriminator': 'enum1', 'data': { 'value1' : 'UserDefA', 'value2' : 'UserDefB', - 'value3' : 'UserDefB' } } + 'value3' : 'UserDefUnionBase' } } { 'struct': 'UserDefUnionBase', 'base': 'UserDefZero',GEN
tests/test-qapi-types.h/home/eblake/qemu/tests/qapi-schema/qapi-schema-test.json:65: Member name 'string' of branch 'value3' clashes with base 'UserDefUnionBase'/home/eblake/qemu/tests/Makefile:415: recipe for target 'tests/test-qapi-types.h' failedBut you have me curious if this collision is still caught when the ad hoc tests are gone. If so, great; if not, I'll add a test here. (I'll know later when I get through rebasing to all of your comments.)
Quoted 5 lines
Quoted 3 lines
No change to generated code.
Signed-off-by Eric Blake <[email protected]>Patch looks good.
Yay; it's nice to see results after all our mental gymnastics over how collision testing should work.
Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches
Quoted 25 lines · Markus Armbruster
Quoted 18 lines · Eric Blake
Quoted 17 lines · Markus Armbruster
Quoted 14 lines · Eric Blake
Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any qapi member names that would conflict with the non-variant qapi members already present from the union's base type (see flat-union-clash-member.json). We want
QAPISchemaObjectType.check()to make the same check, so we can later reduce some of the ad hoc checks.In general, a type used as a branch of a flat union cannot also be the base type of the flat union, so even though we are adding a call to
variant.type.check()in order to populate variant.type.members, this is merely a case of gaining topological sorting of how types are visited (andtype.check()is already set up to allow multiple calls due to base types).Quoted 4 lines
Quoted 4 lines
Yes, a type cannot contain itself, neither as base nor as variant.
We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can
Quoted 7 lines
Actually, these are just local, unpublished tests. They both make
check_member_clash()recurse infinitely.# Direct inheritance loop
# FIXME triggers infinite recursion
{ 'struct': 'Loopy', 'base': 'Loopy',
'data': {} }
Okay, I should add that into my pending patch that cleans up base loops,
Quoted 8 lines
# we reject a loop in base classes { 'struct': 'Base1', 'base': 'Base2', 'data': {} } { 'struct': 'Base2', 'base': 'Base1', 'data': {} }The latter is actually yours, proposed as base-cycle.json in
Subject: qapi: Detect collisions in C member names
Message-Id: <[email protected]>
and yes, that one is still in my queue for subset D:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg07001.html
and I may indeed at a test for reusing the base type of a flat union as one of the branches of the same union, depending on whether it uncovers anything different.
Quoted 11 lines
If I disable the recursive call, the cycle detection in
QAPISchemaObjectType.check()is reached, and works.Completing the move of clash detection to
check()methods should improve things from "accidental infinite recursion" to "intentional assertion failure", because it should get rid ofcheck_member_clash()and should not break the cycle detection.Then we can turn the assertion into a proper error message, and add the tests.
Yep, that's what is pending in my queue, just further out than subset C.
Doesn't matter if it misses 2.5 (the bug is real, but is only triggered
by bad .json code, and we aren't going to add any bad .json code between now and 2.5).
Quoted 3 lines
Quoted 3 lines
But you have me curious if this collision is still caught when the ad hoc tests are gone. If so, great; if not, I'll add a test here. (I'll know later when I get through rebasing to all of your comments.)
Still true - I'm still plowing through earlier patches before deciding if my 'qapi: Detect base class loops' also needs to detect flat union loops.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches
In reply to Eric BlakeQuoted 16 lines · Eric Blake
Quoted 3 lines
We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can see, we don't have tests covering the latter. Do we catch it?
Yes, at least by virtue of the ad hoc tests: attempting to reuse a base type of the flat union as a variant member will cause the qapi members of the base type to appear more than once in the JSON object (that is, the checks that reject flat-union-clash-member.json would also reject this scenario). To test:
(quoted diff) +0
-0
w/tests/qapi-schema/qapi-schema-test.json
index 44638da..16b2ffb 100644
tests/qapi-schema/qapi-schema-test.json line 70
Quoted 5 lines
tests/qapi-schema/qapi-schema-test.json +1
-1@@ -67,7 +67,7 @@ 'discriminator': 'enum1', 'data': { 'value1' : 'UserDefA', 'value2' : 'UserDefB', - 'value3' : 'UserDefB' } } + 'value3' : 'UserDefUnionBase' } }
Another test I just tried is creating a flat union with:
empty -> base -> union
then use empty as one of the union branches. In that case, there is no conflict (although base is included twice, neither inclusion adds members to the JSON object; and the inclusion is not circular so things compile just fine). So probably not worth adding a test for it.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 23/30] qapi: Check for qapi collisions of flat union branches
Quoted 26 lines · Eric Blake
Quoted 16 lines · Eric Blake
Quoted 3 lines
We have tests covering attempts to do the former (struct-cycle-direct.json, struct-cycle-indirect.json). As far as I can see, we don't have tests covering the latter. Do we catch it?
Yes, at least by virtue of the ad hoc tests: attempting to reuse a base type of the flat union as a variant member will cause the qapi members of the base type to appear more than once in the JSON object (that is, the checks that reject flat-union-clash-member.json would also reject this scenario). To test:
(quoted diff) +0
-0
w/tests/qapi-schema/qapi-schema-test.json
index 44638da..16b2ffb 100644tests/qapi-schema/qapi-schema-test.json line 70
Quoted 5 lines
tests/qapi-schema/qapi-schema-test.json +1
-1@@ -67,7 +67,7 @@ 'discriminator': 'enum1', 'data': { 'value1' : 'UserDefA', 'value2' : 'UserDefB', - 'value3' : 'UserDefB' } } + 'value3' : 'UserDefUnionBase' } }Another test I just tried is creating a flat union with:
Quoted 6 lines
empty -> base -> union
then use empty as one of the union branches. In that case, there is no conflict (although base is included twice, neither inclusion adds members to the JSON object; and the inclusion is not circular so things compile just fine). So probably not worth adding a test for it.
Member name clashes occur because:
1. Two separately defined members happen to have a clashing name.
Whether the members are local or inherited, variant or non-variant doesn't matter.
2. The same type gets spliced in twice, and conflict.
By "spliced", I mean the members get included. A bit like unboxed, but also "unwrapped".
Base types and variant types get spliced in. Variant types don't conflict with each other. Empty type can't conflict (outlawing it anyway would be pointless). Therefore, the only conflicting splice is a non-empty type occuring both as (base type of the) base type and as (base type of a) variant type.
We detect this kind of clash exactly like 1. That's fine, because the resulting error messages are plainly good enough.
3. Type contains itself.
The type gets spliced into itself, or has itself as unboxed member.
We don't unbox object members so far. That leaves the spliced ones:
base types and variants. Thus, a type contains itself if it's its own base or one of its own variants, or a direct or indirect base of either.
We detect this kind of clash separately, when we guard against infinite check() recursion. Reporting it right there will be easiest.
We want to cover all this with tests. I guess we're lacking only 3.
PATCH v10 24/30 qapi: Factor out QAPISchemaObjectType.check_clash()
In reply to Eric BlakeConsolidate two common sequences of clash detection into a new QAPISchemaObjectType.check_clash() helper method.
No change to generated code.
---
v10: rebase on new Variants.check_clash()
v9: new patch, split off from v8 7/17
--- scripts/qapi.py | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-)
scripts/qapi.py +8 -11
@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType) - assert not self.base.variants # not implemented - self.base.check(schema) - for m in self.base.members: - m.check_clash(seen) + self.base.check_clash(schema, seen) for m in self.local_members: m.check(schema) m.check_clash(seen)
@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType): assert self.variants.tag_member in self.members self.variants.check_clash(schema, seen) + def check_clash(self, schema, seen): + self.check(schema) + assert not self.variants # not implemented + for m in self.members: + m.check_clash(seen) + def is_implicit(self): # See QAPISchema._make_implicit_object_type() return self.name[0] == ':'
@@ -1062,12 +1064,7 @@ class QAPISchemaObjectTypeVariants(object): for v in self.variants: # Reset seen map for each variant, since qapi names from one # branch do not affect another branch - vseen = dict(seen) - assert isinstance(v.type, QAPISchemaObjectType) - assert not v.type.variants # not implemented - v.type.check(schema) - for m in v.type.members: - m.check_clash(vseen) + v.type.check_clash(schema, dict(seen)) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()
Quoted 19 lines · Eric Blake
Consolidate two common sequences of clash detection into a new
QAPISchemaObjectType.check_clash()helper method.No change to generated code.
Signed-off-by Eric Blake <[email protected]>---
v10: rebase on newVariants.check_clash()
v9: new patch, split off from v8 7/17--- scripts/qapi.py | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-)scripts/qapi.py +0
-0
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 983
Quoted 4 lines
scripts/qapi.py +0
-1@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType)
This assertion is lost.
Quoted 22 lines
scripts/qapi.py +7
-4- assert not self.base.variants # not implemented - self.base.check(schema) - for m in self.base.members: - m.check_clash(seen) + self.base.check_clash(schema, seen) for m in self.local_members: m.check(schema) m.check_clash(seen)@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType): assert self.variants.tag_member in self.members self.variants.check_clash(schema, seen) + def check_clash(self, schema, seen): + self.check(schema) + assert not self.variants # not implemented + for m in self.members: + m.check_clash(seen) + def is_implicit(self): # See QAPISchema._make_implicit_object_type() return self.name[0] == ':'
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066
Quoted 5 lines
scripts/qapi.py +0
-2@@ -1062,12 +1064,7 @@ class QAPISchemaObjectTypeVariants(object): for v in self.variants: # Reset seen map for each variant, since qapi names from one # branch do not affect another branch - vseen = dict(seen) - assert isinstance(v.type, QAPISchemaObjectType)
This assertion is lost.
Quoted 8 lines
scripts/qapi.py +1
-4- assert not v.type.variants # not implemented - v.type.check(schema) - for m in v.type.members: - m.check_clash(vseen) + v.type.check_clash(schema, dict(seen)) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()
Quoted 11 lines · Markus Armbruster
Quoted 8 lines · Eric Blake
Consolidate two common sequences of clash detection into a new
QAPISchemaObjectType.check_clash()helper method.No change to generated code.
Signed-off-by Eric Blake <[email protected]>---
Quoted 13 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 983
Quoted 5 lines
scripts/qapi.py +0
-1@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType)This assertion is lost.
Quoted 5 lines
scripts/qapi.py +1
-4- assert not self.base.variants # not implemented - self.base.check(schema) - for m in self.base.members: - m.check_clash(seen) + self.base.check_clash(schema, seen)
Directly lost, but indirectly still present. The new code is calling QAPISchemaObjectType.check_clash(), which won't exist unless self.base is a QAPISchemaObjectType. Folding the assert into the refactored function makes no sense (the condition isinstance(self, QAPISchemaObjectType) would always be true), and leaving the assert prior to calling self.base.check_clash() adds no real protection against programming bugs.
Quoted 14 lines
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066
Quoted 6 lines
scripts/qapi.py +0
-2@@ -1062,12 +1064,7 @@ class QAPISchemaObjectTypeVariants(object): for v in self.variants: # Reset seen map for each variant, since qapi names from one # branch do not affect another branch - vseen = dict(seen) - assert isinstance(v.type, QAPISchemaObjectType)This assertion is lost.
Quoted 5 lines
scripts/qapi.py +1
-4- assert not v.type.variants # not implemented - v.type.check(schema) - for m in v.type.members: - m.check_clash(vseen) + v.type.check_clash(schema, dict(seen))
Same explanation.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()
Quoted 26 lines · Eric Blake
Quoted 11 lines · Markus Armbruster
Quoted 8 lines · Eric Blake
Consolidate two common sequences of clash detection into a new
QAPISchemaObjectType.check_clash()helper method.No change to generated code.
Signed-off-by Eric Blake <[email protected]>---
Quoted 13 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 983
Quoted 5 lines
scripts/qapi.py +0
-1@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType)This assertion is lost.
Quoted 5 lines
scripts/qapi.py +1
-4- assert not self.base.variants # not implemented - self.base.check(schema) - for m in self.base.members: - m.check_clash(seen) + self.base.check_clash(schema, seen)
Quoted 3 lines
Directly lost, but indirectly still present. The new code is calling
QAPISchemaObjectType.check_clash(), which won't exist unless self.base is a QAPISchemaObjectType.
or a QAPISchemaObjectTypeMember, or a QAPISchemaObjectVariants, or whatever else acquires the method in the future.
Quoted 3 lines
Folding the assert into the refactored
function makes no sense (the conditionisinstance(self, QAPISchemaObjectType)would always be true),
Correct.
Quoted 3 lines
and leaving the assert
prior to callingself.base.check_clash()adds no real protection against programming bugs.
Maybe, but the isinstance(self.base, QAPISchemaObjectType) will come right back anyway when we move the "'base' for FOO cannot use BAR type" check from the old semantic analysis into the check methods. Until then, it makes sense at least as a place holder.
Quoted 16 lines
Quoted 14 lines
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1066
Quoted 6 lines
scripts/qapi.py +0
-2@@ -1062,12 +1064,7 @@ class QAPISchemaObjectTypeVariants(object): for v in self.variants: # Reset seen map for each variant, since qapi names from one # branch do not affect another branch - vseen = dict(seen) - assert isinstance(v.type, QAPISchemaObjectType)This assertion is lost.
Quoted 5 lines
scripts/qapi.py +1
-4- assert not v.type.variants # not implemented - v.type.check(schema) - for m in v.type.members: - m.check_clash(vseen) + v.type.check_clash(schema, dict(seen))Same explanation.
Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()
Quoted 31 lines · Markus Armbruster
Quoted 20 lines · Eric Blake
Quoted 9 lines · Markus Armbruster
Quoted 6 lines · Eric Blake
Consolidate two common sequences of clash detection into a new
QAPISchemaObjectType.check_clash()helper method.No change to generated code.
Signed-off-by Eric Blake <[email protected]>Quoted 8 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 983
Quoted 5 lines
scripts/qapi.py +0
-1@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType)This assertion is lost.
Quoted 3 lines
Directly lost, but indirectly still present. The new code is calling
QAPISchemaObjectType.check_clash(), which won't exist unless self.base is a QAPISchemaObjectType.or a QAPISchemaObjectTypeMember, or a QAPISchemaObjectVariants, or whatever else acquires the method in the future.
Quoted 4 lines
Maybe, but the
isinstance(self.base, QAPISchemaObjectType)will come right back anyway when we move the "'base' for FOO cannot use BAR type" check from the old semantic analysis into the check methods. Until then, it makes sense at least as a place holder.
Good point, I'll add it back in.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()
In reply to Eric BlakeQuoted 32 lines · Eric Blake
Consolidate two common sequences of clash detection into a new
QAPISchemaObjectType.check_clash()helper method.No change to generated code.
Signed-off-by Eric Blake <[email protected]>---
v10: rebase on newVariants.check_clash()
v9: new patch, split off from v8 7/17--- scripts/qapi.py | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-)scripts/qapi.py +1
-5@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType) - assert not self.base.variants # not implemented - self.base.check(schema) - for m in self.base.members: - m.check_clash(seen) + self.base.check_clash(schema, seen) for m in self.local_members: m.check(schema) m.check_clash(seen)
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 994
Quoted 5 lines
scripts/qapi.py +2
-0@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType): assert self.variants.tag_member in self.members self.variants.check_clash(schema, seen) + def check_clash(self, schema, seen): + self.check(schema)
Do we want to hide this .check() inside .check_clash()?
QAPISchemaObjectTypeMember.check() doesn't. I think the two better behave the same.
Quoted 21 lines
scripts/qapi.py +5
-6+ assert not self.variants # not implemented + for m in self.members: + m.check_clash(seen) + def is_implicit(self): # See QAPISchema._make_implicit_object_type() return self.name[0] == ':'@@ -1062,12 +1064,7 @@ class QAPISchemaObjectTypeVariants(object): for v in self.variants: # Reset seen map for each variant, since qapi names from one # branch do not affect another branch - vseen = dict(seen) - assert isinstance(v.type, QAPISchemaObjectType) - assert not v.type.variants # not implemented - v.type.check(schema) - for m in v.type.members: - m.check_clash(vseen) + v.type.check_clash(schema, dict(seen)) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()
Quoted 27 lines · Markus Armbruster
Quoted 25 lines · Eric Blake
Consolidate two common sequences of clash detection into a new
QAPISchemaObjectType.check_clash()helper method.No change to generated code.
Signed-off-by Eric Blake <[email protected]>scripts/qapi.py +1
-5@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType) - assert not self.base.variants # not implemented - self.base.check(schema) - for m in self.base.members: - m.check_clash(seen) + self.base.check_clash(schema, seen) for m in self.local_members: m.check(schema) m.check_clash(seen)@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType): assert self.variants.tag_member in self.members self.variants.check_clash(schema, seen)
Quoted 11 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 994
Quoted 2 lines
scripts/qapi.py +2
-0+ def check_clash(self, schema, seen): + self.check(schema)Do we want to hide this .check() inside .check_clash()?
QAPISchemaObjectTypeMember.check()doesn't. I think the two better behave the same.Quoted 3 lines
scripts/qapi.py +3
-0+ assert not self.variants # not implemented + for m in self.members: + m.check_clash(seen)
The self.check(schema) call is necessary to get self.members populated.
We cannot iterate over self.members if the type has not had check()
called; this is true for both callers of type.check_clash() (ObjectType.check(), and Variants.check_clash()).
You are correct that neither Member.check() nor Member.check_clash() call a form of type.check() - but that's because at that level, there is no need to populate a type.members list.
On the other hand, we've been arguing that check() should populate everything after construction prior to anything else being run; and not running Variant.type.check() during Variants.check() of flat unions feels like we may have a hole (a flat union will have to inline its types to the overall JSON object, and inlining types requires access to type.members - but as written, we aren't populating them until Variants.check_clash()). I can play with hoisting the type.check() out of type.check_clash() and instead keep base.check() in type.check(), and add variant.type.check() in Variants.check() (but only for unions, not for alternates), if you are interested.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()
Quoted 36 lines · Eric Blake
Quoted 35 lines · Markus Armbruster
Quoted 22 lines · Eric Blake
Consolidate two common sequences of clash detection into a new
QAPISchemaObjectType.check_clash()helper method.No change to generated code.
Signed-off-by Eric Blake <[email protected]>(quoted diff) +1
-5@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType) - assert not self.base.variants # not implemented - self.base.check(schema) - for m in self.base.members: - m.check_clash(seen) + self.base.check_clash(schema, seen) for m in self.local_members: m.check(schema) m.check_clash(seen)Review class QAPISchemaObjectType(QAPISchemaType): · line 994
Quoted 5 lines
(quoted diff) +2
-0@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType): assert self.variants.tag_member in self.members self.variants.check_clash(schema, seen) + def check_clash(self, schema, seen): + self.check(schema)Do we want to hide this .check() inside .check_clash()?
QAPISchemaObjectTypeMember.check()doesn't. I think the two better behave the same.
Quoted 8 lines
Quoted 3 lines
Review class QAPISchemaObjectType(QAPISchemaType): · line 997
Quoted 3 lines
(quoted diff) +3
-0+ assert not self.variants # not implemented + for m in self.members: + m.check_clash(seen)The
self.check(schema)call is necessary to get self.members populated.
We cannot iterate over self.members if the type has not hadcheck()
called; this is true for both callers oftype.check_clash()(ObjectType.check(), andVariants.check_clash()).
Yes.
We have a common protocol for QAPISchemaFOO objects, namely that certain instance variables and methods are only valid after .check().
Quoted 4 lines
You are correct that neither
Member.check()norMember.check_clash()call a form oftype.check()- but that's because at that level, there is no need to populate a type.members list.
Quoted 10 lines
On the other hand, we've been arguing that
check()should populate everything after construction prior to anything else being run; and not runningVariant.type.check()duringVariants.check()of flat unions feels like we may have a hole (a flat union will have to inline its types to the overall JSON object, and inlining types requires access to type.members - but as written, we aren't populating them untilVariants.check_clash()). I can play with hoisting thetype.check()out oftype.check_clash()and instead keepbase.check()intype.check(), and addvariant.type.check()inVariants.check()(but only for unions, not for alternates), if you are interested.
My "qapi: Factor out QAPISchemaObjectTypeMember.check_clash()" added QAPISchemaObjectTypeMember.check_clash() without changing the common protocol. The new QAPISchemaObjectTypeMember.check_clash() is merely a helper for QAPISchemaObjectType.check().
Your
Gcc: nnml:mail.redhat.xlst.qemu-devel
From: Markus Armbruster <[email protected]>
--text follows this line--
Quoted 36 lines · Eric Blake
Quoted 35 lines · Markus Armbruster
Quoted 22 lines · Eric Blake
Consolidate two common sequences of clash detection into a new
QAPISchemaObjectType.check_clash()helper method.No change to generated code.
Signed-off-by Eric Blake <[email protected]>(quoted diff) +1
-5@@ -980,11 +980,7 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - assert isinstance(self.base, QAPISchemaObjectType) - assert not self.base.variants # not implemented - self.base.check(schema) - for m in self.base.members: - m.check_clash(seen) + self.base.check_clash(schema, seen) for m in self.local_members: m.check(schema) m.check_clash(seen)Review class QAPISchemaObjectType(QAPISchemaType): · line 994
Quoted 5 lines
(quoted diff) +2
-0@@ -994,6 +990,12 @@ class QAPISchemaObjectType(QAPISchemaType): assert self.variants.tag_member in self.members self.variants.check_clash(schema, seen) + def check_clash(self, schema, seen): + self.check(schema)Do we want to hide this .check() inside .check_clash()?
QAPISchemaObjectTypeMember.check()doesn't. I think the two better behave the same.
Quoted 8 lines
Quoted 3 lines
Review class QAPISchemaObjectType(QAPISchemaType): · line 997
Quoted 3 lines
(quoted diff) +3
-0+ assert not self.variants # not implemented + for m in self.members: + m.check_clash(seen)The
self.check(schema)call is necessary to get self.members populated.
We cannot iterate over self.members if the type has not hadcheck()
called; this is true for both callers oftype.check_clash()(ObjectType.check(), andVariants.check_clash()).
Yes.
We have a common protocol for QAPISchemaFOO objects, namely that certain instance variables and methods are only valid after .check().
Quoted 4 lines
You are correct that neither
Member.check()norMember.check_clash()call a form oftype.check()- but that's because at that level, there is no need to populate a type.members list.
Quoted 10 lines
On the other hand, we've been arguing that
check()should populate everything after construction prior to anything else being run; and not runningVariant.type.check()duringVariants.check()of flat unions feels like we may have a hole (a flat union will have to inline its types to the overall JSON object, and inlining types requires access to type.members - but as written, we aren't populating them untilVariants.check_clash()). I can play with hoisting thetype.check()out oftype.check_clash()and instead keepbase.check()intype.check(), and addvariant.type.check()inVariants.check()(but only for unions, not for alternates), if you are interested.
My "qapi: Factor out QAPISchemaObjectTypeMember.check_clash()" adds QAPISchemaObjectTypeMember.check_clash() without changing the common protocol. The new QAPISchemaObjectTypeMember.check_clash() is merely a helper for QAPISchemaObjectType.check().
The two .check_clash() you add (one in this patch, one in the previous one) are different: both contain calls of QAPISchemaObjectType.check().
I feel the .check() calls are too important to be buried deep like that. I'd stick to prior practice and put the .check() calls right into .check(). Obviously, the .check_clash() methods may only called after .check() then, but that's nothing new.
Fixup for your previous patch:
scripts/qapi.py +1 -1
@@ -1065,7 +1065,6 @@ class QAPISchemaObjectTypeVariants(object): vseen = dict(seen) assert isinstance(v.type, QAPISchemaObjectType) assert not v.type.variants # not implemented - v.type.check(schema) for m in v.type.members: m.check_clash(vseen)
@@ -1077,6 +1076,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def check(self, schema, tag_type): QAPISchemaObjectTypeMember.check(self, schema) assert self.name in tag_type.values + self.type.check(schema) # This function exists to support ugly simple union special cases # TODO get rid of them, and drop the function
This patch with conflicts resolved and the same idea worked in then becomes:
scripts/qapi.py +7 -7
@@ -981,10 +981,8 @@ class QAPISchemaObjectType(QAPISchemaType): if self._base_name: self.base = schema.lookup_type(self._base_name) assert isinstance(self.base, QAPISchemaObjectType) - assert not self.base.variants # not implemented self.base.check(schema) - for m in self.base.members: - m.check_clash(seen) + self.base.check_clash(schema, seen) for m in self.local_members: m.check(schema) m.check_clash(seen)
@@ -994,6 +992,11 @@ class QAPISchemaObjectType(QAPISchemaType): assert self.variants.tag_member in self.members self.variants.check_clash(schema, seen) + def check_clash(self, schema, seen): + assert not self.variants # not implemented + for m in self.members: + m.check_clash(seen) + def is_implicit(self): # See QAPISchema._make_implicit_object_type() return self.name[0] == ':'
@@ -1062,11 +1065,8 @@ class QAPISchemaObjectTypeVariants(object): for v in self.variants: # Reset seen map for each variant, since qapi names from one # branch do not affect another branch - vseen = dict(seen) assert isinstance(v.type, QAPISchemaObjectType) - assert not v.type.variants # not implemented - for m in v.type.members: - m.check_clash(vseen) + v.type.check_clash(schema, dict(seen)) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Your two check_clash() are then simple helpers, just like mine.
Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()
Quoted 39 lines · Markus Armbruster
Quoted 10 lines
On the other hand, we've been arguing that
check()should populate everything after construction prior to anything else being run; and not runningVariant.type.check()duringVariants.check()of flat unions feels like we may have a hole (a flat union will have to inline its types to the overall JSON object, and inlining types requires access to type.members - but as written, we aren't populating them untilVariants.check_clash()). I can play with hoisting thetype.check()out oftype.check_clash()and instead keepbase.check()intype.check(), and addvariant.type.check()inVariants.check()(but only for unions, not for alternates), if you are interested.My "qapi: Factor out
QAPISchemaObjectTypeMember.check_clash()" addsQAPISchemaObjectTypeMember.check_clash()without changing the common protocol. The newQAPISchemaObjectTypeMember.check_clash()is merely a helper forQAPISchemaObjectType.check().The two .check_clash() you add (one in this patch, one in the previous one) are different: both contain calls of
QAPISchemaObjectType.check().I feel the .check() calls are too important to be buried deep like that. I'd stick to prior practice and put the .check() calls right into .check(). Obviously, the .check_clash() methods may only called after .check() then, but that's nothing new.
Fixup for your previous patch:
scripts/qapi.py +0
-1@@ -1065,7 +1065,6 @@ class QAPISchemaObjectTypeVariants(object): vseen = dict(seen) assert isinstance(v.type, QAPISchemaObjectType) assert not v.type.variants # not implemented - v.type.check(schema) for m in v.type.members: m.check_clash(vseen)
scripts/qapi.py class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): · line 1080
Quoted 5 lines
scripts/qapi.py +1
-0@@ -1077,6 +1076,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def check(self, schema, tag_type): QAPISchemaObjectTypeMember.check(self, schema) assert self.name in tag_type.values + self.type.check(schema)
Won't quite work. You are right that we must call self.type.check(schema) for variants used by a union; but calling it for ALL variants used by an alternate is wrong, because self.type for at least one branch of an alternate will not be an instance of QAPISchemaObjectType. However, I'm currently testing whether it is safe to check to just blindly check an object branch of an alternate, if present (and that should not lead to cycles, since alternates have no base class and since we don't allow one alternate type as a variant of another alternate), in which case the fixup for 23/30 is more like:
scripts/qapi.py +0 -1
@@ -1065,7 +1065,6 @@ class QAPISchemaObjectTypeVariants(object): vseen = dict(seen) assert isinstance(v.type, QAPISchemaObjectType) assert not v.type.variants # not implemented - v.type.check(schema) for m in v.type.members: m.check_clash(vseen)
@@ -1077,6 +1076,8 @@ class
QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
def check(self, schema, tag_type):
QAPISchemaObjectTypeMember.check(self, schema) assert self.name in tag_type.values
+ if isinstance(self.type, QAPISchemaObjectType):
+ self.type.check(schema)
# This function exists to support ugly simple union special cases # TODO get rid of them, and drop the function
@@ -1098,6 +1099,8 @@ class QAPISchemaAlternateType(QAPISchemaType):
def check(self, schema):
self.variants.tag_member.check(schema)
+ # Not calling self.variants.check_clash(), because there's
+ # nothing to clash with
self.variants.check(schema, {})
def json_type(self):
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 24/30] qapi: Factor out QAPISchemaObjectType.check_clash()
Quoted 80 lines · Eric Blake
Quoted 39 lines · Markus Armbruster
Quoted 10 lines
On the other hand, we've been arguing that
check()should populate everything after construction prior to anything else being run; and not runningVariant.type.check()duringVariants.check()of flat unions feels like we may have a hole (a flat union will have to inline its types to the overall JSON object, and inlining types requires access to type.members - but as written, we aren't populating them untilVariants.check_clash()). I can play with hoisting thetype.check()out oftype.check_clash()and instead keepbase.check()intype.check(), and addvariant.type.check()inVariants.check()(but only for unions, not for alternates), if you are interested.My "qapi: Factor out
QAPISchemaObjectTypeMember.check_clash()" addsQAPISchemaObjectTypeMember.check_clash()without changing the common protocol. The newQAPISchemaObjectTypeMember.check_clash()is merely a helper forQAPISchemaObjectType.check().The two .check_clash() you add (one in this patch, one in the previous one) are different: both contain calls of
QAPISchemaObjectType.check().I feel the .check() calls are too important to be buried deep like that. I'd stick to prior practice and put the .check() calls right into .check(). Obviously, the .check_clash() methods may only called after .check() then, but that's nothing new.
Fixup for your previous patch:
scripts/qapi.py +0
-1@@ -1065,7 +1065,6 @@ class QAPISchemaObjectTypeVariants(object): vseen = dict(seen) assert isinstance(v.type, QAPISchemaObjectType) assert not v.type.variants # not implemented - v.type.check(schema) for m in v.type.members: m.check_clash(vseen)scripts/qapi.py class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): · line 1080
Quoted 5 lines
scripts/qapi.py +1
-0@@ -1077,6 +1076,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def check(self, schema, tag_type): QAPISchemaObjectTypeMember.check(self, schema) assert self.name in tag_type.values + self.type.check(schema)Won't quite work. You are right that we must call
self.type.check(schema)for variants used by a union; but calling it for ALL variants used by an alternate is wrong, because self.type for at least one branch of an alternate will not be an instance of QAPISchemaObjectType. However, I'm currently testing whether it is safe to check to just blindly check an object branch of an alternate, if present (and that should not lead to cycles, since alternates have no base class and since we don't allow one alternate type as a variant of another alternate), in which case the fixup for 23/30 is more like:scripts/qapi.py +0
-1@@ -1065,7 +1065,6 @@ class QAPISchemaObjectTypeVariants(object): vseen = dict(seen) assert isinstance(v.type, QAPISchemaObjectType) assert not v.type.variants # not implemented - v.type.check(schema) for m in v.type.members: m.check_clash(vseen)@@ -1077,6 +1076,8 @@ classQAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
defcheck(self, schema, tag_type):
QAPISchemaObjectTypeMember.check(self, schema)assert self.name in tag_type.valuesscripts/qapi.py +2
-0+ if isinstance(self.type, QAPISchemaObjectType): + self.type.check(schema) # This function exists to support ugly simple union special cases # TODO get rid of them, and drop the function
scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1106
Quoted 8 lines
scripts/qapi.py +2
-0@@ -1098,6 +1099,8 @@ class QAPISchemaAlternateType(QAPISchemaType): def check(self, schema): self.variants.tag_member.check(schema) + # Not calling self.variants.check_clash(), because there's + # nothing to clash with self.variants.check(schema, {}) def json_type(self):
Makes sense to me.
PATCH v10 25/30 qapi: Hoist tag collision check to Variants.check()
In reply to Eric BlakeChecking that a given QAPISchemaObjectTypeVariant.name is a member of the corresponding QAPISchemaEnumType of the owning QAPISchemaObjectTypeVariants.tag_member ensures that there are no collisions in the generated C union for those tag values (since the enum itself should have no collisions).
However, this check was the only thing that Variant.check() was doing beyond the work of the superclass ObjectTypeMember.check(), and resulted in a difference of the .check() signatures just to pass the enum type down.
Simplify things by instead doing the tag name check as part of Variants.check(), at which point we can rely on inheritance instead of overriding Variant.check().
---
v10: new patch
--- scripts/qapi.py | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
scripts/qapi.py +4 -6
@@ -1056,9 +1056,11 @@ class QAPISchemaObjectTypeVariants(object): def check(self, schema, seen): if self.tag_name: # flat union self.tag_member = seen[self.tag_name] - assert isinstance(self.tag_member.type, QAPISchemaEnumType) + tag_type = self.tag_member.type + assert isinstance(tag_type, QAPISchemaEnumType) for v in self.variants: - v.check(schema, self.tag_member.type) + v.check(schema) + assert v.name in tag_type.values def check_clash(self, schema, seen): for v in self.variants:
@@ -1071,10 +1073,6 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def __init__(self, name, typ): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) - def check(self, schema, tag_type): - QAPISchemaObjectTypeMember.check(self, schema) - assert self.name in tag_type.values - # This function exists to support ugly simple union special cases # TODO get rid of them, and drop the function def simple_union_type(self):
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 25/30] qapi: Hoist tag collision check to Variants.check()
Quoted 28 lines · Eric Blake
Checking that a given QAPISchemaObjectTypeVariant.name is a member of the corresponding QAPISchemaEnumType of the owning QAPISchemaObjectTypeVariants.tag_member ensures that there are no collisions in the generated C union for those tag values (since the enum itself should have no collisions).
However, this check was the only thing that
Variant.check()was doing beyond the work of the superclassObjectTypeMember.check(), and resulted in a difference of the .check() signatures just to pass the enum type down.Simplify things by instead doing the tag name check as part of
Variants.check(), at which point we can rely on inheritance instead of overridingVariant.check().Signed-off-by Eric Blake <[email protected]>---
v10: new patch--- scripts/qapi.py | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)scripts/qapi.py +0
-0
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1063
Quoted 10 lines
scripts/qapi.py +4
-2@@ -1056,9 +1056,11 @@ class QAPISchemaObjectTypeVariants(object): def check(self, schema, seen): if self.tag_name: # flat union self.tag_member = seen[self.tag_name] - assert isinstance(self.tag_member.type, QAPISchemaEnumType) + tag_type = self.tag_member.type + assert isinstance(tag_type, QAPISchemaEnumType) for v in self.variants: - v.check(schema, self.tag_member.type) + v.check(schema) + assert v.name in tag_type.values
Two changes squashed together:
* Move the assertion from QAPISchemaObjectTypeVariant.check(),
* Capture self.tag_member.type in tag_type
The second part makes the patch slightly less obvious. Matter of taste.
Quoted 14 lines
scripts/qapi.py +0
-4def check_clash(self, schema, seen): for v in self.variants:@@ -1071,10 +1073,6 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): def __init__(self, name, typ): QAPISchemaObjectTypeMember.__init__(self, name, typ, False) - def check(self, schema, tag_type): - QAPISchemaObjectTypeMember.check(self, schema) - assert self.name in tag_type.values - # This function exists to support ugly simple union special cases # TODO get rid of them, and drop the function def simple_union_type(self):
Re: [Qemu-devel] [PATCH v10 25/30] qapi: Hoist tag collision check to Variants.check()
Quoted 29 lines · Markus Armbruster
Quoted 19 lines · Eric Blake
Checking that a given QAPISchemaObjectTypeVariant.name is a member of the corresponding QAPISchemaEnumType of the owning QAPISchemaObjectTypeVariants.tag_member ensures that there are no collisions in the generated C union for those tag values (since the enum itself should have no collisions).
However, this check was the only thing that
Variant.check()was doing beyond the work of the superclassObjectTypeMember.check(), and resulted in a difference of the .check() signatures just to pass the enum type down.Simplify things by instead doing the tag name check as part of
Variants.check(), at which point we can rely on inheritance instead of overridingVariant.check().Signed-off-by Eric Blake <[email protected]>---
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1063
Quoted 7 lines
scripts/qapi.py +4
-2- assert isinstance(self.tag_member.type, QAPISchemaEnumType) + tag_type = self.tag_member.type + assert isinstance(tag_type, QAPISchemaEnumType) for v in self.variants: - v.check(schema, self.tag_member.type) + v.check(schema) + assert v.name in tag_type.values
Quoted 7 lines
Two changes squashed together:
* Move the assertion from
QAPISchemaObjectTypeVariant.check(),* Capture self.tag_member.type in
tag_typeThe second part makes the patch slightly less obvious. Matter of taste.
I'm dropping the tag_type temporary member. Once I played more with making qtype_code a qapi builtin type, it became much more elegant to revisit this code (the use of tag_type here was for when my later patches were trying to key off of tag_type==None as a witness of alternates).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v10 26/30 qapi: Remove outdated tests related to QMP/branch collisions
In reply to Eric BlakeNow that branches are in a separate C namespace, we can remove the restrictions in the parser that claim a branch name would collide with QMP, and delete the negative tests that are no longer problematic. A separate patch can then add positive tests to qapi-schema-test to test that any corner cases will compile correctly.
This reverts the scripts/qapi.py portion of commit 7b2a5c2, now that the assertions that it plugged are no longer possible.
---
v10: no change
v9: no change
v8: retitle and update commit message, finish reversion of 7b2a5c2
v7: new patch (also temporarily appeared in subset B v10)
--- scripts/qapi.py | 11 ++--------- tests/Makefile | 3 --- tests/qapi-schema/flat-union-clash-branch.err | 0 tests/qapi-schema/flat-union-clash-branch.exit | 1 - tests/qapi-schema/flat-union-clash-branch.json | 18 ------------------ tests/qapi-schema/flat-union-clash-branch.out | 14 -------------- tests/qapi-schema/flat-union-clash-type.err | 1 - tests/qapi-schema/flat-union-clash-type.exit | 1 - tests/qapi-schema/flat-union-clash-type.json | 14 -------------- tests/qapi-schema/flat-union-clash-type.out | 0 tests/qapi-schema/union-clash-type.err | 1 - tests/qapi-schema/union-clash-type.exit | 1 - tests/qapi-schema/union-clash-type.json | 9 --------- tests/qapi-schema/union-clash-type.out | 0 14 files changed, 2 insertions(+), 72 deletions(-) delete mode 100644 tests/qapi-schema/flat-union-clash-branch.err delete mode 100644 tests/qapi-schema/flat-union-clash-branch.exit delete mode 100644 tests/qapi-schema/flat-union-clash-branch.json delete mode 100644 tests/qapi-schema/flat-union-clash-branch.out delete mode 100644 tests/qapi-schema/flat-union-clash-type.err delete mode 100644 tests/qapi-schema/flat-union-clash-type.exit delete mode 100644 tests/qapi-schema/flat-union-clash-type.json delete mode 100644 tests/qapi-schema/flat-union-clash-type.out delete mode 100644 tests/qapi-schema/union-clash-type.err delete mode 100644 tests/qapi-schema/union-clash-type.exit delete mode 100644 tests/qapi-schema/union-clash-type.json delete mode 100644 tests/qapi-schema/union-clash-type.out
scripts/qapi.py +2 -9
@@ -548,8 +548,7 @@ def check_union(expr, expr_info): base = expr.get('base') discriminator = expr.get('discriminator') members = expr['data'] - values = {'MAX': '(automatic)', 'KIND': '(automatic)', - 'TYPE': '(automatic)'} + values = {'MAX': '(automatic)'} # Two types of unions, determined by discriminator.
@@ -607,19 +606,13 @@ def check_union(expr, expr_info): " of branch '%s'" % key) # If the discriminator names an enum type, then all members - # of 'data' must also be members of the enum type, which in turn - # must not collide with the discriminator name. + # of 'data' must also be members of the enum type. if enum_define: if key not in enum_define['enum_values']: raise QAPIExprError(expr_info, "Discriminator value '%s' is not found in " "enum '%s'" % (key, enum_define["enum_name"])) - if discriminator in enum_define['enum_values']: - raise QAPIExprError(expr_info, - "Discriminator name '%s' collides with " - "enum value in '%s'" % - (discriminator, enum_define["enum_name"])) # Otherwise, check for conflicts in the generated enum else:
tests/Makefile +0 -3
@@ -277,9 +277,7 @@ qapi-schema += flat-union-bad-base.json qapi-schema += flat-union-bad-discriminator.json qapi-schema += flat-union-base-any.json qapi-schema += flat-union-base-union.json -qapi-schema += flat-union-clash-branch.json qapi-schema += flat-union-clash-member.json -qapi-schema += flat-union-clash-type.json qapi-schema += flat-union-empty.json qapi-schema += flat-union-inline.json qapi-schema += flat-union-int-branch.json
@@ -341,7 +339,6 @@ qapi-schema += union-bad-branch.json qapi-schema += union-base-no-discriminator.json qapi-schema += union-clash-branches.json qapi-schema += union-clash-data.json -qapi-schema += union-clash-type.json qapi-schema += union-empty.json qapi-schema += union-invalid-base.json qapi-schema += union-max.json
tests/qapi-schema/flat-union-clash-branch.err deleted +0 -0
tests/qapi-schema/flat-union-clash-branch.exit deleted +0 -1
@@ -1 +0,0 @@ -0
tests/qapi-schema/flat-union-clash-branch.json deleted +0 -18
@@ -1,18 +0,0 @@ -# Flat union branch name collision -# FIXME: this parses, but then fails to compile due to a duplicate 'c_d' -# (one from the base member, the other from the branch name). We should -# either reject the collision at parse time, or munge the generated branch -# name to allow this to compile. -{ 'enum': 'TestEnum', - 'data': [ 'base', 'c-d' ] } -{ 'struct': 'Base', - 'data': { 'enum1': 'TestEnum', '*c_d': 'str' } } -{ 'struct': 'Branch1', - 'data': { 'string': 'str' } } -{ 'struct': 'Branch2', - 'data': { 'value': 'int' } } -{ 'union': 'TestUnion', - 'base': 'Base', - 'discriminator': 'enum1', - 'data': { 'base': 'Branch1', - 'c-d': 'Branch2' } }
tests/qapi-schema/flat-union-clash-branch.out deleted +0 -14
@@ -1,14 +0,0 @@ -object :empty -object Base - member enum1: TestEnum optional=False - member c_d: str optional=True -object Branch1 - member string: str optional=False -object Branch2 - member value: int optional=False -enum TestEnum ['base', 'c-d'] -object TestUnion - base Base - tag enum1 - case base: Branch1 - case c-d: Branch2
tests/qapi-schema/flat-union-clash-type.err deleted +0 -1
@@ -1 +0,0 @@ -tests/qapi-schema/flat-union-clash-type.json:11: Discriminator name 'type' collides with enum value in 'TestEnum'
tests/qapi-schema/flat-union-clash-type.exit deleted +0 -1
@@ -1 +0,0 @@ -1
tests/qapi-schema/flat-union-clash-type.json deleted +0 -14
@@ -1,14 +0,0 @@ -# Flat union branch 'type' -# Reject this, because we would have a clash in generated C, between the -# outer tag 'type' and the branch name 'type' within the union. -# TODO: We could munge the generated C branch name to let it compile. -{ 'enum': 'TestEnum', - 'data': [ 'type' ] } -{ 'struct': 'Base', - 'data': { 'type': 'TestEnum' } } -{ 'struct': 'Branch1', - 'data': { 'string': 'str' } } -{ 'union': 'TestUnion', - 'base': 'Base', - 'discriminator': 'type', - 'data': { 'type': 'Branch1' } }
tests/qapi-schema/flat-union-clash-type.out deleted +0 -0
tests/qapi-schema/union-clash-type.err deleted +0 -1
@@ -1 +0,0 @@ -tests/qapi-schema/union-clash-type.json:8: Union 'TestUnion' member 'kind' clashes with '(automatic)'
tests/qapi-schema/union-clash-type.exit deleted +0 -1
@@ -1 +0,0 @@ -1
tests/qapi-schema/union-clash-type.json deleted +0 -9
@@ -1,9 +0,0 @@ -# Union branch 'type' -# Reject this, because we would have a clash in generated C, between the -# simple union's implicit tag member 'kind' and the branch name 'kind' -# within the union. -# TODO: Even when the generated C is switched to use 'type' rather than -# 'kind', to match the QMP spelling, the collision should still be detected. -# Or, we could munge the branch name to allow compilation. -{ 'union': 'TestUnion', - 'data': { 'kind': 'int', 'type': 'str' } }
tests/qapi-schema/union-clash-type.out deleted +0 -0
Signature · 1 line
2.4.3
PATCH v10 27/30 qapi: Track owner of each object member
In reply to Eric BlakeFuture commits will migrate semantic checking away from parsing and over to the various QAPISchema*.check() methods. But to report an error message about an incorrect semantic use of a member of an object type, it helps to know which type, command, or event owns the member. In particular, when a member is inherited from a base type, it is desirable to associate the member name with the base type (and not the type calling member.check()).
Rather than packing additional information into the seen array passed to each member.check() (as in seen[m.name] = {'member':m, 'owner':type}), it is easier to have each member track the name of the owner type in the first place (keeping things simpler with the existing seen[m.name] = m). The new member.owner field is set via a new set_owner() method, called when registering the members and variants arrays with an object or variant type. Track only a name, and not the actual type object, to avoid creating a circular python reference chain.
Note that Variants.set_owner() method does not set the owner for the tag_member field; this field is set earlier either as part of an object's non-variant members, or explicitly by alternates.
The source information is intended for human consumption in error messages, and a new describe() method is added to access the resulting information. For example, given the qapi:
{ 'command': 'foo', 'data': { 'string': 'str' } }
an implementation of visit_command() that calls
arg_type.members[0].describe()
will see "'string' (argument of foo)".
To make the human-readable name of implicit types work without duplicating efforts, the describe() method has to reverse the name of implicit types, via the helper _pretty_owner(), plus a tweak to report event data separately from command arguments.
No change to generated code.
---
v10 (now in subset C): rebase to latest, set alternate tag_member owner from alternate
v9 (now in subset D): rebase to earlier changes, hoist 'role' to top of class, split out _pretty_helper(), manage owner when tag_member appears as part of local_members for unions
v8: don't munge implicit type names [except for event data], and instead make describe() create nicer messages. Add set_owner(), and use field 'role' instead of method _describe()
v7: total rewrite: rework implicit object names, assign owner when initializing owner type rather than when creating member python object
v6: rebase on new lazy array creation and simple union 'type' motion; tweak commit message
--- scripts/qapi.py | 44 +++++++++++++++++++++++++++++++--- tests/qapi-schema/qapi-schema-test.out | 8 +++---- 2 files changed, 45 insertions(+), 7 deletions(-)
scripts/qapi.py +41 -3
@@ -957,8 +957,10 @@ class QAPISchemaObjectType(QAPISchemaType): assert base is None or isinstance(base, str) for m in local_members: assert isinstance(m, QAPISchemaObjectTypeMember) - assert (variants is None or - isinstance(variants, QAPISchemaObjectTypeVariants)) + m.set_owner(name) + if variants is not None: + assert isinstance(variants, QAPISchemaObjectTypeVariants) + variants.set_owner(name) self._base_name = base self.base = None self.local_members = local_members
@@ -1012,6 +1014,8 @@ class QAPISchemaObjectType(QAPISchemaType): class QAPISchemaObjectTypeMember(object): + role = 'member' + def __init__(self, name, typ, optional): assert isinstance(name, str) assert isinstance(typ, str)
@@ -1020,8 +1024,14 @@ class QAPISchemaObjectTypeMember(object): self._type_name = typ self.type = None self.optional = optional + self.owner = None + + def set_owner(self, name): + assert not self.owner + self.owner = name def check(self, schema): + assert self.owner self.type = schema.lookup_type(self._type_name) assert self.type
@@ -1030,6 +1040,25 @@ class QAPISchemaObjectTypeMember(object): assert self.name not in seen seen[self.name] = self + def _pretty_owner(self): + # See QAPISchema._make_implicit_object_type() - reverse the + # mapping there to create a nice human-readable description + owner = self.owner + if owner.startswith(':obj-'): + owner = owner[5:] + if owner.endswith('-arg'): + return '(argument of %s)' % owner[:-4] + elif owner.endswith('-data'): + return '(data of %s)' % owner[:-5] + else: + assert owner.endswith('-wrapper') + return '(branch of %s)' % owner[:-8] + else: + return '(%s of %s)' % (self.role, owner) + + def describe(self): + return "'%s' %s" % (self.name, self._pretty_owner()) + class QAPISchemaObjectTypeVariants(object): def __init__(self, tag_name, tag_member, variants):
@@ -1046,6 +1075,10 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member = tag_member self.variants = variants + def set_owner(self, name): + for v in self.variants: + v.set_owner(name) + def check(self, schema, seen): if self.tag_name: # flat union self.tag_member = seen[self.tag_name]
@@ -1063,6 +1096,8 @@ class QAPISchemaObjectTypeVariants(object): class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): + role = 'branch' + def __init__(self, name, typ): QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
@@ -1082,9 +1117,11 @@ class QAPISchemaAlternateType(QAPISchemaType): QAPISchemaType.__init__(self, name, info) assert isinstance(variants, QAPISchemaObjectTypeVariants) assert not variants.tag_name + variants.set_owner(name) self.variants = variants def check(self, schema): + self.variants.tag_member.set_owner(self.name) self.variants.tag_member.check(schema) self.variants.check(schema, {})
@@ -1212,6 +1249,7 @@ class QAPISchema(object): def _make_implicit_object_type(self, name, info, role, members): if not members: return None + # See also QAPISchemaObjectTypeMember.describe() name = ':obj-%s-%s' % (name, role) if not self.lookup_entity(name, QAPISchemaObjectType): self._def_entity(QAPISchemaObjectType(name, info, None,
@@ -1315,7 +1353,7 @@ class QAPISchema(object): data = expr.get('data') if isinstance(data, OrderedDict): data = self._make_implicit_object_type( - name, info, 'arg', self._make_members(data, info)) + name, info, 'data', self._make_members(data, info)) self._def_entity(QAPISchemaEvent(name, info, data)) def _def_exprs(self):
tests/qapi-schema/qapi-schema-test.out +4 -4
@@ -1,9 +1,9 @@ object :empty -object :obj-EVENT_C-arg +object :obj-EVENT_C-data member a: int optional=True member b: UserDefOne optional=True member c: str optional=False -object :obj-EVENT_D-arg +object :obj-EVENT_D-data member a: EventStructOne optional=False member b: str optional=False member c: str optional=True
@@ -79,8 +79,8 @@ alternate AltStrNum enum AltStrNumKind ['s', 'n'] event EVENT_A None event EVENT_B None -event EVENT_C :obj-EVENT_C-arg -event EVENT_D :obj-EVENT_D-arg +event EVENT_C :obj-EVENT_C-data +event EVENT_D :obj-EVENT_D-data object Empty1 object Empty2 base Empty1
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 27/30] qapi: Track owner of each object member
Quoted 102 lines · Eric Blake
Future commits will migrate semantic checking away from parsing and over to the various QAPISchema*.check() methods. But to report an error message about an incorrect semantic use of a member of an object type, it helps to know which type, command, or event owns the member. In particular, when a member is inherited from a base type, it is desirable to associate the member name with the base type (and not the type calling
member.check()).Rather than packing additional information into the seen array passed to each
member.check()(as in seen[m.name] = {'member':m, 'owner':type}), it is easier to have each member track the name of the owner type in the first place (keeping things simpler with the existing seen[m.name] = m). The new member.owner field is set via a newset_owner()method, called when registering the members and variants arrays with an object or variant type. Track only a name, and not the actual type object, to avoid creating a circular python reference chain.Note that
Variants.set_owner()method does not set the owner for thetag_memberfield; this field is set earlier either as part of an object's non-variant members, or explicitly by alternates.The source information is intended for human consumption in error messages, and a new
describe()method is added to access the resulting information. For example, given the qapi:
{ 'command': 'foo', 'data': { 'string': 'str' } }
an implementation ofvisit_command()that calls
arg_type.members[0].describe()
will see "'string' (argument of foo)".To make the human-readable name of implicit types work without duplicating efforts, the
describe()method has to reverse the name of implicit types, via the helper_pretty_owner(), plus a tweak to report event data separately from command arguments.No change to generated code.
Signed-off-by Eric Blake <[email protected]>---
v10 (now in subset C): rebase to latest, set alternatetag_memberowner from alternate
v9 (now in subset D): rebase to earlier changes, hoist 'role' to top of class, split out_pretty_helper(), manage owner whentag_memberappears as part oflocal_membersfor unions
v8: don't munge implicit type names [except for event data], and instead makedescribe()create nicer messages. Addset_owner(), and use field 'role' instead of method_describe()
v7: total rewrite: rework implicit object names, assign owner when initializing owner type rather than when creating member python object
v6: rebase on new lazy array creation and simple union 'type' motion; tweak commit message--- scripts/qapi.py | 44 +++++++++++++++++++++++++++++++--- tests/qapi-schema/qapi-schema-test.out | 8 +++---- 2 files changed, 45 insertions(+), 7 deletions(-)scripts/qapi.py +12
-2@@ -957,8 +957,10 @@ class QAPISchemaObjectType(QAPISchemaType): assert base is None or isinstance(base, str) for m in local_members: assert isinstance(m, QAPISchemaObjectTypeMember) - assert (variants is None or - isinstance(variants, QAPISchemaObjectTypeVariants)) + m.set_owner(name) + if variants is not None: + assert isinstance(variants, QAPISchemaObjectTypeVariants) + variants.set_owner(name) self._base_name = base self.base = None self.local_members = local_members@@ -1012,6 +1014,8 @@ class QAPISchemaObjectType(QAPISchemaType): class QAPISchemaObjectTypeMember(object): + role = 'member' + def __init__(self, name, typ, optional): assert isinstance(name, str) assert isinstance(typ, str)@@ -1020,8 +1024,14 @@ class QAPISchemaObjectTypeMember(object): self._type_name = typ self.type = None self.optional = optional + self.owner = None + + def set_owner(self, name): + assert not self.owner + self.owner = name def check(self, schema): + assert self.owner self.type = schema.lookup_type(self._type_name) assert self.type
scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1045
Quoted 6 lines
scripts/qapi.py +3
-0@@ -1030,6 +1040,25 @@ class QAPISchemaObjectTypeMember(object): assert self.name not in seen seen[self.name] = self + def _pretty_owner(self): + # See QAPISchema._make_implicit_object_type() - reverse the + # mapping there to create a nice human-readable description
This comment confused me briefly. It applies to the following conditionals if-part, but not its else-part. Move it into the if-part?
scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1055
Quoted 10 lines
scripts/qapi.py +10
-0+ owner = self.owner + if owner.startswith(':obj-'): + owner = owner[5:] + if owner.endswith('-arg'): + return '(argument of %s)' % owner[:-4] + elif owner.endswith('-data'): + return '(data of %s)' % owner[:-5] + else: + assert owner.endswith('-wrapper') + return '(branch of %s)' % owner[:-8]
See last remark.
Quoted 31 lines
scripts/qapi.py +12
-0+ else: + return '(%s of %s)' % (self.role, owner) + + def describe(self): + return "'%s' %s" % (self.name, self._pretty_owner()) + class QAPISchemaObjectTypeVariants(object): def __init__(self, tag_name, tag_member, variants):@@ -1046,6 +1075,10 @@ class QAPISchemaObjectTypeVariants(object): self.tag_member = tag_member self.variants = variants + def set_owner(self, name): + for v in self.variants: + v.set_owner(name) + def check(self, schema, seen): if self.tag_name: # flat union self.tag_member = seen[self.tag_name]@@ -1063,6 +1096,8 @@ class QAPISchemaObjectTypeVariants(object): class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember): + role = 'branch' + def __init__(self, name, typ): QAPISchemaObjectTypeMember.__init__(self, name, typ, False)@@ -1082,9 +1117,11 @@ class QAPISchemaAlternateType(QAPISchemaType): QAPISchemaType.__init__(self, name, info)
scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1127
Quoted 10 lines
scripts/qapi.py +2
-0assert isinstance(variants, QAPISchemaObjectTypeVariants) assert not variants.tag_name + variants.set_owner(name) self.variants = variants def check(self, schema): + self.variants.tag_member.set_owner(self.name) self.variants.tag_member.check(schema) self.variants.check(schema, {})
Odd: all other .set_owner() calls are in .__init__() or .set_owner(). Can we move this one to __init__() for consistency?
I think we can: __init__() requires its variants argument to have a tag_member (it even asserts not variants.tag_name).
scripts/qapi.py class QAPISchema(object): · line 1252
Quoted 5 lines
scripts/qapi.py +1
-0@@ -1212,6 +1249,7 @@ class QAPISchema(object): def _make_implicit_object_type(self, name, info, role, members): if not members: return None + # See also QAPISchemaObjectTypeMember.describe()
Should this point to ._pretty_owner() instead?
scripts/qapi.py class QAPISchema(object): · line 1356
Quoted 9 lines
scripts/qapi.py +1
-1name = ':obj-%s-%s' % (name, role) if not self.lookup_entity(name, QAPISchemaObjectType): self._def_entity(QAPISchemaObjectType(name, info, None,@@ -1315,7 +1353,7 @@ class QAPISchema(object): data = expr.get('data') if isinstance(data, OrderedDict): data = self._make_implicit_object_type( - name, info, 'arg', self._make_members(data, info)) + name, info, 'data', self._make_members(data, info))
This is necessary only to make .pretty_owner() say 'data of EVENT_NAME' instead of 'argument of EVENT_NAME'. Do we really need different wording for commands and events?
I'd make it say 'parameter of' for both commands and events. I habitually use "parameter" for formal parameters, and "argument" for actual arguments. Guess I've worked with the C standard too much...
For what it's worth, the QAPI schema language uses 'data' for both (and many other things, too), and the introspection schema uses 'arg-type' for both.
Quoted 30 lines
scripts/qapi.py +0
-0self._def_entity(QAPISchemaEvent(name, info, data)) def _def_exprs(self):tests/qapi-schema/qapi-schema-test.out +4
-4@@ -1,9 +1,9 @@ object :empty -object :obj-EVENT_C-arg +object :obj-EVENT_C-data member a: int optional=True member b: UserDefOne optional=True member c: str optional=False -object :obj-EVENT_D-arg +object :obj-EVENT_D-data member a: EventStructOne optional=False member b: str optional=False member c: str optional=True@@ -79,8 +79,8 @@ alternate AltStrNum enum AltStrNumKind ['s', 'n'] event EVENT_A None event EVENT_B None -event EVENT_C :obj-EVENT_C-arg -event EVENT_D :obj-EVENT_D-arg +event EVENT_C :obj-EVENT_C-data +event EVENT_D :obj-EVENT_D-data object Empty1 object Empty2 base Empty1
Re: [Qemu-devel] [PATCH v10 27/30] qapi: Track owner of each object member
Quoted 13 lines · Markus Armbruster
Quoted 11 lines · Eric Blake
Future commits will migrate semantic checking away from parsing and over to the various QAPISchema*.check() methods. But to report an error message about an incorrect semantic use of a member of an object type, it helps to know which type, command, or event owns the member. In particular, when a member is inherited from a base type, it is desirable to associate the member name with the base type (and not the type calling
member.check()).
Quoted 6 lines
scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1045
Quoted 3 lines
scripts/qapi.py +3
-0+ def _pretty_owner(self): + # See QAPISchema._make_implicit_object_type() - reverse the + # mapping there to create a nice human-readable descriptionThis comment confused me briefly. It applies to the following conditionals if-part, but not its else-part. Move it into the if-part?
Done.
Quoted 7 lines
Quoted 7 lines
scripts/qapi.py +1
-0@@ -1082,9 +1117,11 @@ class QAPISchemaAlternateType(QAPISchemaType): QAPISchemaType.__init__(self, name, info) assert isinstance(variants, QAPISchemaObjectTypeVariants) assert not variants.tag_name + variants.set_owner(name) self.variants = variants
Quoted 8 lines
scripts/qapi.py class QAPISchemaAlternateType(QAPISchemaType): · line 1127
Quoted 5 lines
scripts/qapi.py +1
-0def check(self, schema): + self.variants.tag_member.set_owner(self.name) self.variants.tag_member.check(schema) self.variants.check(schema, {})Odd: all other .set_owner() calls are in .__init__() or .set_owner(). Can we move this one to
__init__()for consistency?
Yes, done.
Quoted 11 lines
I think we can:
__init__()requires its variants argument to have atag_member(it even asserts not variants.tag_name).scripts/qapi.py class QAPISchema(object): · line 1252
Quoted 5 lines
scripts/qapi.py +1
-0@@ -1212,6 +1249,7 @@ class QAPISchema(object): def _make_implicit_object_type(self, name, info, role, members): if not members: return None + # See also QAPISchemaObjectTypeMember.describe()Should this point to ._pretty_owner() instead?
Good call.
Quoted 15 lines
scripts/qapi.py class QAPISchema(object): · line 1356
Quoted 10 lines
scripts/qapi.py +1
-1name = ':obj-%s-%s' % (name, role) if not self.lookup_entity(name, QAPISchemaObjectType): self._def_entity(QAPISchemaObjectType(name, info, None,@@ -1315,7 +1353,7 @@ class QAPISchema(object): data = expr.get('data') if isinstance(data, OrderedDict): data = self._make_implicit_object_type( - name, info, 'arg', self._make_members(data, info)) + name, info, 'data', self._make_members(data, info))This is necessary only to make .pretty_owner() say 'data of
EVENT_NAME' instead of 'argument ofEVENT_NAME'. Do we really need different wording for commands and events?
Quoted 3 lines
I'd make it say 'parameter of' for both commands and events. I habitually use "parameter" for formal parameters, and "argument" for actual arguments. Guess I've worked with the C standard too much...
'parameter' sounds better for both, at which point the messages no longer differ,...
Quoted 7 lines
For what it's worth, the QAPI schema language uses 'data' for both (and many other things, too), and the introspection schema uses 'arg-type' for both.
Quoted 2 lines
scripts/qapi.py +0
-0self._def_entity(QAPISchemaEvent(name, info, data))
Quoted 14 lines
Quoted 6 lines
scripts/qapi.py +0
-0def _def_exprs(self):tests/qapi-schema/qapi-schema-test.out +0
-0tests/qapi-schema/qapi-schema-test.out line 6
Quoted 8 lines
tests/qapi-schema/qapi-schema-test.out +2
-2@@ -1,9 +1,9 @@ object :empty -object :obj-EVENT_C-arg +object :obj-EVENT_C-data member a: int optional=True member b: UserDefOne optional=True member c: str optional=False -object :obj-EVENT_D-arg +object :obj-EVENT_D-data
...so I no longer need to rename the implicit struct for events to allow the differentiation. v11 is thus a smaller patch :)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v10 28/30 qapi: Detect collisions in C member names
In reply to Eric BlakeDetect attempts to declare two object members that would result in the same C member name, by keying the 'seen' dictionary off of the C name rather than the qapi name. It also requires passing info through the check_clash() methods.
This addresses a TODO and fixes the previously-broken args-name-clash test. The resulting error message demonstrates the utility of the .describe() method added previously. No change to generated code.
---
v10 (now in subset C): rebase to latest; update commit message v9 (now in subset D): rebase to earlier changes, now only one test affected
v8: rebase to earlier changes
v7: split out error reporting prep and member.c_name() addition
v6: rebase to earlier testsuite and info improvements
--- scripts/qapi.py | 31 +++++++++++++++++++------------ tests/qapi-schema/args-name-clash.err | 1 + tests/qapi-schema/args-name-clash.exit | 2 +- tests/qapi-schema/args-name-clash.json | 6 +++--- tests/qapi-schema/args-name-clash.out | 6 ------ 5 files changed, 24 insertions(+), 22 deletions(-)
scripts/qapi.py +19 -12
@@ -975,21 +975,24 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - self.base.check_clash(schema, seen) + self.base.check_clash(schema, self.info, seen) for m in self.local_members: m.check(schema) - m.check_clash(seen) + m.check_clash(self.info, seen) self.members = seen.values() if self.variants: self.variants.check(schema, seen) assert self.variants.tag_member in self.members - self.variants.check_clash(schema, seen) + self.variants.check_clash(schema, self.info, seen) - def check_clash(self, schema, seen): + # Check that the members of this type do not cause duplicate JSON fields, + # and update seen to track the members seen so far. Report any errors + # on behalf of info, which is not necessarily self.info + def check_clash(self, schema, info, seen): self.check(schema) assert not self.variants # not implemented for m in self.members: - m.check_clash(seen) + m.check_clash(info, seen) def is_implicit(self): # See QAPISchema._make_implicit_object_type()
@@ -1035,10 +1038,13 @@ class QAPISchemaObjectTypeMember(object): self.type = schema.lookup_type(self._type_name) assert self.type - def check_clash(self, seen): - # TODO change key of seen from QAPI name to C name - assert self.name not in seen - seen[self.name] = self + def check_clash(self, info, seen): + name = c_name(self.name) + if name in seen: + raise QAPIExprError(info, + "%s collides with %s" + % (self.describe(), seen[name].describe())) + seen[name] = self def _pretty_owner(self): # See QAPISchema._make_implicit_object_type() - reverse the
@@ -1081,18 +1087,19 @@ class QAPISchemaObjectTypeVariants(object): def check(self, schema, seen): if self.tag_name: # flat union - self.tag_member = seen[self.tag_name] + self.tag_member = seen[c_name(self.tag_name)] + assert self.tag_name == self.tag_member.name tag_type = self.tag_member.type assert isinstance(tag_type, QAPISchemaEnumType) for v in self.variants: v.check(schema) assert v.name in tag_type.values - def check_clash(self, schema, seen): + def check_clash(self, schema, info, seen): for v in self.variants: # Reset seen map for each variant, since qapi names from one # branch do not affect another branch - v.type.check_clash(schema, dict(seen)) + v.type.check_clash(schema, info, dict(seen)) class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
tests/qapi-schema/args-name-clash.err +1 -0
@@ -0,0 +1 @@ +tests/qapi-schema/args-name-clash.json:5: 'a_b' (argument of oops) collides with 'a-b' (argument of oops)
tests/qapi-schema/args-name-clash.exit +1 -1
@@ -1 +1 @@ -0 +1
tests/qapi-schema/args-name-clash.json +3 -3
@@ -1,5 +1,5 @@ # C member name collision -# FIXME - This parses, but fails to compile, because the C struct is given -# two 'a_b' members. Either reject this at parse time, or munge the C names -# to avoid the collision. +# Reject members that clash when mapped to C names (we would have two 'a_b' +# members). It would also be possible to munge the C names to avoid the +# collision, but unlikely to be worth the effort. { 'command': 'oops', 'data': { 'a-b': 'str', 'a_b': 'str' } }
tests/qapi-schema/args-name-clash.out +0 -6
@@ -1,6 +0,0 @@ -object :empty -object :obj-oops-arg - member a-b: str optional=False - member a_b: str optional=False -command oops :obj-oops-arg -> None - gen=True success_response=True
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 28/30] qapi: Detect collisions in C member names
Quoted 32 lines · Eric Blake
Detect attempts to declare two object members that would result in the same C member name, by keying the 'seen' dictionary off of the C name rather than the qapi name. It also requires passing info through the
check_clash()methods.This addresses a TODO and fixes the previously-broken args-name-clash test. The resulting error message demonstrates the utility of the .describe() method added previously. No change to generated code.
Signed-off-by Eric Blake <[email protected]>---
v10 (now in subset C): rebase to latest; update commit message v9 (now in subset D): rebase to earlier changes, now only one test affected
v8: rebase to earlier changes
v7: split out error reporting prep andmember.c_name()addition
v6: rebase to earlier testsuite and info improvements--- scripts/qapi.py | 31 +++++++++++++++++++------------ tests/qapi-schema/args-name-clash.err | 1 + tests/qapi-schema/args-name-clash.exit | 2 +- tests/qapi-schema/args-name-clash.json | 6 +++--- tests/qapi-schema/args-name-clash.out | 6 ------ 5 files changed, 24 insertions(+), 22 deletions(-)scripts/qapi.py +0
-0
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 978
Quoted 5 lines
scripts/qapi.py +1
-1@@ -975,21 +975,24 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - self.base.check_clash(schema, seen) + self.base.check_clash(schema, self.info, seen)
Note that if this one reports an error for self.info, something went wrong. We first run self.base.check(), which we survive only when self.base doesn't have clashing members. Would be easier to see if self.base.check() wasn't hiding in self.base.check_clash().
Quoted 5 lines
scripts/qapi.py +1
-1for m in self.local_members: m.check(schema) - m.check_clash(seen) + m.check_clash(self.info, seen) self.members = seen.values()
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 990
Quoted 10 lines
scripts/qapi.py +4
-2if self.variants: self.variants.check(schema, seen) assert self.variants.tag_member in self.members - self.variants.check_clash(schema, seen) + self.variants.check_clash(schema, self.info, seen) - def check_clash(self, schema, seen): + # Check that the members of this type do not cause duplicate JSON fields, + # and update seen to track the members seen so far. Report any errors + # on behalf of info, which is not necessarily self.info
Do we actually need info != self.info? If yes, test case? If no, should the comment be adjusted?
Quoted 10 lines
scripts/qapi.py +2
-1+ def check_clash(self, schema, info, seen): self.check(schema) assert not self.variants # not implemented for m in self.members: - m.check_clash(seen) + m.check_clash(info, seen) def is_implicit(self): # See QAPISchema._make_implicit_object_type()
scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1042
Quoted 9 lines
scripts/qapi.py +2
-4@@ -1035,10 +1038,13 @@ class QAPISchemaObjectTypeMember(object): self.type = schema.lookup_type(self._type_name) assert self.type - def check_clash(self, seen): - # TODO change key of seen from QAPI name to C name - assert self.name not in seen - seen[self.name] = self + def check_clash(self, info, seen): + name = c_name(self.name)
I'd call it cname. Matter of taste, of course.
Quoted 18 lines
scripts/qapi.py +7
-1+ if name in seen: + raise QAPIExprError(info, + "%s collides with %s" + % (self.describe(), seen[name].describe())) + seen[name] = self def _pretty_owner(self): # See QAPISchema._make_implicit_object_type() - reverse the@@ -1081,18 +1087,19 @@ class QAPISchemaObjectTypeVariants(object): def check(self, schema, seen): if self.tag_name: # flat union - self.tag_member = seen[self.tag_name] + self.tag_member = seen[c_name(self.tag_name)] + assert self.tag_name == self.tag_member.name tag_type = self.tag_member.type assert isinstance(tag_type, QAPISchemaEnumType) for v in self.variants:
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1102
Quoted 10 lines
scripts/qapi.py +2
-2v.check(schema) assert v.name in tag_type.values - def check_clash(self, schema, seen): + def check_clash(self, schema, info, seen): for v in self.variants: # Reset seen map for each variant, since qapi names from one # branch do not affect another branch - v.type.check_clash(schema, dict(seen)) + v.type.check_clash(schema, info, dict(seen))
Since we can't inherit variant members, info is always the owning object type's info.
Quoted 21 lines
scripts/qapi.py +0
-0class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):tests/qapi-schema/args-name-clash.err +1
-0@@ -0,0 +1 @@ +tests/qapi-schema/args-name-clash.json:5: 'a_b' (argument of oops) collides with 'a-b' (argument of oops)tests/qapi-schema/args-name-clash.exit +1
-1@@ -1 +1 @@ -0 +1tests/qapi-schema/args-name-clash.json +0
-0
tests/qapi-schema/args-name-clash.json line 4
Quoted 7 lines
tests/qapi-schema/args-name-clash.json +3
-3@@ -1,5 +1,5 @@ # C member name collision -# FIXME - This parses, but fails to compile, because the C struct is given -# two 'a_b' members. Either reject this at parse time, or munge the C names -# to avoid the collision. +# Reject members that clash when mapped to C names (we would have two 'a_b' +# members). It would also be possible to munge the C names to avoid the +# collision, but unlikely to be worth the effort.
I'd drop the second sentence.
Quoted 12 lines
tests/qapi-schema/args-name-clash.json +0
-0{ 'command': 'oops', 'data': { 'a-b': 'str', 'a_b': 'str' } }tests/qapi-schema/args-name-clash.out +0
-6@@ -1,6 +0,0 @@ -object :empty -object :obj-oops-arg - member a-b: str optional=False - member a_b: str optional=False -command oops :obj-oops-arg -> None - gen=True success_response=True
Re: [Qemu-devel] [PATCH v10 28/30] qapi: Detect collisions in C member names
Quoted 16 lines · Markus Armbruster
Quoted 13 lines · Eric Blake
Detect attempts to declare two object members that would result in the same C member name, by keying the 'seen' dictionary off of the C name rather than the qapi name. It also requires passing info through the
check_clash()methods.This addresses a TODO and fixes the previously-broken args-name-clash test. The resulting error message demonstrates the utility of the .describe() method added previously. No change to generated code.
Signed-off-by Eric Blake <[email protected]>---
Quoted 12 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 978
Quoted 7 lines
+++
b/scripts/qapi.pyscripts/qapi.py +1
-1@@ -975,21 +975,24 @@ class QAPISchemaObjectType(QAPISchemaType): seen = OrderedDict() if self._base_name: self.base = schema.lookup_type(self._base_name) - self.base.check_clash(schema, seen) + self.base.check_clash(schema, self.info, seen)Note that if this one reports an error for self.info, something went wrong. We first run
self.base.check(), which we survive only when self.base doesn't have clashing members. Would be easier to see ifself.base.check()wasn't hiding inself.base.check_clash().
Hmm. I could pass None as a way of asserting that it won't fail. But I think that's a bit more confusing, so I'll probably just leave it alone.
Quoted 7 lines
scripts/qapi.py class QAPISchemaObjectType(QAPISchemaType): · line 990
Quoted 4 lines
scripts/qapi.py +3
-1- def check_clash(self, schema, seen): + # Check that the members of this type do not cause duplicate JSON fields, + # and update seen to track the members seen so far. Report any errors + # on behalf of info, which is not necessarily self.infoDo we actually need info != self.info? If yes, test case? If no, should the comment be adjusted?
The comment is correct. For a flat union, if we have a variant associated with type 'Foo', and type 'Foo' has no clashes itself, but one of its members duplicates a member already present from the base of union, then we are calling Foo.check_clash(schema, Union.info, seen), so that the error message points to the location of Union (rather than Foo) as the place in the .json file introducing the clash.
flat-union-clash-member.json tests this; it's just that we first have to remove ad hoc testing before we can trigger this case. It's on my queue, just not in this particular posting of subset C:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg07000.html
Quoted 4 lines
Quoted 4 lines
scripts/qapi.py +0
-0@@ -1035,10 +1038,13 @@ class QAPISchemaObjectTypeMember(object): self.type = schema.lookup_type(self._type_name) assert self.type
Quoted 8 lines
scripts/qapi.py class QAPISchemaObjectTypeMember(object): · line 1042
Quoted 6 lines
scripts/qapi.py +2
-4- def check_clash(self, seen): - # TODO change key of seen from QAPI name to C name - assert self.name not in seen - seen[self.name] = self + def check_clash(self, info, seen): + name = c_name(self.name)I'd call it cname. Matter of taste, of course.
Sure, that works. I first tried calling it c_name, but then python thought I was trying to redefine the global c_name().
Quoted 11 lines
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1102
Quoted 8 lines
scripts/qapi.py +2
-2- def check_clash(self, schema, seen): + def check_clash(self, schema, info, seen): for v in self.variants: # Reset seen map for each variant, since qapi names from one # branch do not affect another branch - v.type.check_clash(schema, dict(seen)) + v.type.check_clash(schema, info, dict(seen))Since we can't inherit variant members, info is always the owning object type's info.
Exactly. We want to output the passed-in info, and NOT v.type.info, so that errors are reported on behalf of the union that owns the variant, rather than at the location of the variant's type which independently passed .check() and therefore has no internal collisions.
And someday we might be able to inherit from objects with variants, but that's a lot further down the pipeline (if at all).
Quoted 11 lines
tests/qapi-schema/args-name-clash.json line 4
Quoted 9 lines
+++
b/tests/qapi-schema/args-name-clash.jsontests/qapi-schema/args-name-clash.json +3
-3@@ -1,5 +1,5 @@ # C member name collision -# FIXME - This parses, but fails to compile, because the C struct is given -# two 'a_b' members. Either reject this at parse time, or munge the C names -# to avoid the collision. +# Reject members that clash when mapped to C names (we would have two 'a_b' +# members). It would also be possible to munge the C names to avoid the +# collision, but unlikely to be worth the effort.I'd drop the second sentence.
Done.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH v10 29/30 cpu: Convert CpuInfo into flat union
In reply to Eric BlakeWhen qapi type CpuInfo was originally created for 0.14, we had no notion of a flat union, and instead just listed a bunch of optional fields with documentation about the mutually-exclusive choice of which instruction pointer field(s) would be provided for a given architecture. But now that we have flat unions and introspection, it is better to segregate off which fields will be provided according to the actual architecture. With this in place, we no longer need the fields to be optional, because the choice of the discriminator serves that role.
This has an additional benefit: the old all-in-one struct was the only place in the code base that had a case-sensitive naming of members 'pc' vs. 'PC'. Separating these spellings into different branches of the flat union will allow us to add restrictions against future case-insensitive collisions, and make it possible for a future qemu to decide whether to enable case-insensitive QMP.
---
v10: new patch
--- cpus.c | 31 ++++++++------ hmp.c | 30 +++++++++----- qapi-schema.json | 120 ++++++++++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 139 insertions(+), 42 deletions(-)
cpus.c +19 -12
@@ -1526,22 +1526,29 @@ CpuInfoList *qmp_query_cpus(Error **errp) info->value->qom_path =object_get_canonical_path (OBJECT (cpu)); info->value->thread_id = cpu->thread_id;#if defined(TARGET_I386 ) - info->value->has_pc = true; - info->value->pc = env->eip + env->segs[R_CS ].base; + info->value->arch =CPU_INFO_ARCH_X86 ; + info->value->u.x86 =g_new0 (CpuInfoX86,1 ); + info->value->u.x86->pc = env->eip + env->segs[R_CS ].base;#elif defined(TARGET_PPC ) - info->value->has_nip = true; - info->value->nip = env->nip; + info->value->arch =CPU_INFO_ARCH_PPC ; + info->value->u.ppc =g_new0 (CpuInfoPpc,1 ); + info->value->u.ppc->nip = env->nip;#elif defined(TARGET_SPARC ) - info->value->has_pc = true; - info->value->pc = env->pc; - info->value->has_npc = true; - info->value->npc = env->npc; + info->value->arch =CPU_INFO_ARCH_SPARC ; + info->value->u.sparc =g_new0 (CpuInfoSPARC,1 ); + info->value->u.sparc->pc = env->pc; + info->value->u.sparc->npc = env->npc;#elif defined(TARGET_MIPS ) - info->value->has_PC = true; - info->value->PC = env->active_tc.PC; + info->value->arch =CPU_INFO_ARCH_MIPS ; + info->value->u.mips =g_new0 (CpuInfoMips,1 ); + info->value->u.mips->PC = env->active_tc.PC;#elif defined(TARGET_TRICORE ) - info->value->has_PC = true; - info->value->PC = env->PC; + info->value->arch =CPU_INFO_ARCH_TRICORE ; + info->value->u.tricore =g_new0 (CpuInfoTricore,1 ); + info->value->u.tricore->PC = env->PC; +#else + info->value->arch =CPU_INFO_ARCH_OTHER ; + info->value->u.other =g_new0 (CpuInfoOther,1 );#endif /* XXX: waiting for the qapi to support GSList */
hmp.c +19 -11
@@ -310,17 +310,25 @@ void hmp_info_cpus(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%c CPU #%" PRId64":" , active, cpu->value->CPU); -if (cpu->value->has_pc) { -monitor_printf (mon," pc=0x%016" PRIx64, cpu->value->pc); - } -if (cpu->value->has_nip) { -monitor_printf (mon," nip=0x%016" PRIx64, cpu->value->nip); - } -if (cpu->value->has_npc) { -monitor_printf (mon," npc=0x%016" PRIx64, cpu->value->npc); - } -if (cpu->value->has_PC) { -monitor_printf (mon," PC=0x%016" PRIx64, cpu->value->PC); +switch (cpu->value->arch) { +case CPU_INFO_ARCH_X86 : +monitor_printf (mon," pc=0x%016" PRIx64, cpu->value->u.x86->pc); +break ; +case CPU_INFO_ARCH_PPC : +monitor_printf (mon," nip=0x%016" PRIx64, cpu->value->u.ppc->nip); +break ; +case CPU_INFO_ARCH_SPARC : +monitor_printf (mon," pc=0x%016" PRIx64, cpu->value->u.sparc->pc); +monitor_printf (mon," npc=0x%016" PRIx64, cpu->value->u.sparc->npc); +break ; +case CPU_INFO_ARCH_MIPS : +monitor_printf (mon," PC=0x%016" PRIx64, cpu->value->u.mips->PC); +break ; +case CPU_INFO_ARCH_TRICORE : +monitor_printf (mon," PC=0x%016" PRIx64, cpu->value->u.tricore->PC); +break ; +default : +break ; }if (cpu->value->halted) {
qapi-schema.json +101 -19
144 lines not on this page · raw message
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 29/30] cpu: Convert CpuInfo into flat union
Quoted 254 lines · Eric Blake
When qapi type CpuInfo was originally created for 0.14, we had no notion of a flat union, and instead just listed a bunch of optional fields with documentation about the mutually-exclusive choice of which instruction pointer field(s) would be provided for a given architecture. But now that we have flat unions and introspection, it is better to segregate off which fields will be provided according to the actual architecture. With this in place, we no longer need the fields to be optional, because the choice of the discriminator serves that role.
This has an additional benefit: the old all-in-one struct was the only place in the code base that had a case-sensitive naming of members 'pc' vs. 'PC'. Separating these spellings into different branches of the flat union will allow us to add restrictions against future case-insensitive collisions, and make it possible for a future qemu to decide whether to enable case-insensitive QMP.
Signed-off-by Eric Blake <[email protected]>---
v10: new patch--- cpus.c | 31 ++++++++------ hmp.c | 30 +++++++++----- qapi-schema.json | 120 ++++++++++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 139 insertions(+), 42 deletions(-)cpus.c +19
-12@@ -1526,22 +1526,29 @@ CpuInfoList *qmp_query_cpus(Error **errp) info->value->qom_path =object_get_canonical_path (OBJECT (cpu)); info->value->thread_id = cpu->thread_id;#if defined(TARGET_I386 ) - info->value->has_pc = true; - info->value->pc = env->eip + env->segs[R_CS ].base; + info->value->arch =CPU_INFO_ARCH_X86 ; + info->value->u.x86 =g_new0 (CpuInfoX86,1 ); + info->value->u.x86->pc = env->eip + env->segs[R_CS ].base;#elif defined(TARGET_PPC ) - info->value->has_nip = true; - info->value->nip = env->nip; + info->value->arch =CPU_INFO_ARCH_PPC ; + info->value->u.ppc =g_new0 (CpuInfoPpc,1 ); + info->value->u.ppc->nip = env->nip;#elif defined(TARGET_SPARC ) - info->value->has_pc = true; - info->value->pc = env->pc; - info->value->has_npc = true; - info->value->npc = env->npc; + info->value->arch =CPU_INFO_ARCH_SPARC ; + info->value->u.sparc =g_new0 (CpuInfoSPARC,1 ); + info->value->u.sparc->pc = env->pc; + info->value->u.sparc->npc = env->npc;#elif defined(TARGET_MIPS ) - info->value->has_PC = true; - info->value->PC = env->active_tc.PC; + info->value->arch =CPU_INFO_ARCH_MIPS ; + info->value->u.mips =g_new0 (CpuInfoMips,1 ); + info->value->u.mips->PC = env->active_tc.PC;#elif defined(TARGET_TRICORE ) - info->value->has_PC = true; - info->value->PC = env->PC; + info->value->arch =CPU_INFO_ARCH_TRICORE ; + info->value->u.tricore =g_new0 (CpuInfoTricore,1 ); + info->value->u.tricore->PC = env->PC; +#else + info->value->arch =CPU_INFO_ARCH_OTHER ; + info->value->u.other =g_new0 (CpuInfoOther,1 );#endif /* XXX: waiting for the qapi to support GSList */ hmp.c +19
-11@@ -310,17 +310,25 @@ void hmp_info_cpus(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%c CPU #%" PRId64":" , active, cpu->value->CPU); -if (cpu->value->has_pc) { -monitor_printf (mon," pc=0x%016" PRIx64, cpu->value->pc); - } -if (cpu->value->has_nip) { -monitor_printf (mon," nip=0x%016" PRIx64, cpu->value->nip); - } -if (cpu->value->has_npc) { -monitor_printf (mon," npc=0x%016" PRIx64, cpu->value->npc); - } -if (cpu->value->has_PC) { -monitor_printf (mon," PC=0x%016" PRIx64, cpu->value->PC); +switch (cpu->value->arch) { +case CPU_INFO_ARCH_X86 : +monitor_printf (mon," pc=0x%016" PRIx64, cpu->value->u.x86->pc); +break ; +case CPU_INFO_ARCH_PPC : +monitor_printf (mon," nip=0x%016" PRIx64, cpu->value->u.ppc->nip); +break ; +case CPU_INFO_ARCH_SPARC : +monitor_printf (mon," pc=0x%016" PRIx64, cpu->value->u.sparc->pc); +monitor_printf (mon," npc=0x%016" PRIx64, cpu->value->u.sparc->npc); +break ; +case CPU_INFO_ARCH_MIPS : +monitor_printf (mon," PC=0x%016" PRIx64, cpu->value->u.mips->PC); +break ; +case CPU_INFO_ARCH_TRICORE : +monitor_printf (mon," PC=0x%016" PRIx64, cpu->value->u.tricore->PC); +break ; +default : +break ; }if (cpu->value->halted) {qapi-schema.json +94
-19@@ -728,43 +728,125 @@ { 'command': 'query-mice', 'returns': ['MouseInfo'] } ## -# @CpuInfo: +# @CpuInfoArch: # -# Information about a virtual CPU +# An enumeration of cpu types that enable additional information during +# @query-cpus. +# +# Since: 2.5 +## +{ 'enum': 'CpuInfoArch', + 'data': ['x86', 'sparc', 'ppc', 'mips', 'tricore', 'other' ] } + +## +# @CpuInfoBase: +# +# Common information about a virtual CPU # # @CPU: the index of the virtual CPU # -# @current: this only exists for backwards compatible and should be ignored +# @current: this only exists for backwards compatibility and should be ignored # # @halted: true if the virtual CPU is in the halt state. Halt usually refers # to a processor specific low power mode. # # @qom_path: path to the CPU object in the QOM tree (since 2.4) # -# @pc: #optional If the target is i386 or x86_64, this is the 64-bit instruction -# pointer. -# If the target is Sparc, this is the PC component of the -# instruction pointer. -# -# @nip: #optional If the target is PPC, the instruction pointer -# -# @npc: #optional If the target is Sparc, the NPC component of the instruction -# pointer -# -# @PC: #optional If the target is MIPS, the instruction pointer -# # @thread_id: ID of the underlying host thread # +# @arch: architecture of the cpu, which determines which additional fields +# will be listed (since 2.5) +# # Since: 0.14.0 # # Notes: @halted is a transient state that changes frequently. By the time the # data is sent to the client, the guest may no longer be halted. ## -{ 'struct': 'CpuInfo', +{ 'struct': 'CpuInfoBase', 'data': {'CPU': 'int', 'current': 'bool', 'halted': 'bool', - 'qom_path': 'str', - '*pc': 'int', '*nip': 'int', '*npc': 'int', '*PC': 'int', - 'thread_id': 'int'} } + 'qom_path': 'str', 'thread_id': 'int', 'arch': 'CpuInfoArch' } } + +## +# @CpuInfo: +# +# Information about a virtual CPU +# +# Since: 0.14.0 +## +{ 'union': 'CpuInfo', 'base': 'CpuInfoBase', 'discriminator': 'arch', + 'data': { 'x86': 'CpuInfoX86', + 'sparc': 'CpuInfoSparc', + 'ppc': 'CpuInfoPpc', + 'mips': 'CpuInfoMips', + 'tricore': 'CpuInfoTricore', + 'other': 'CpuInfoOther' } } + +## +# @CpuInfoX86: +# +# Additional information about a virtual i386 or x86_64 CPU +# +# @pc: the 64-bit instruction pointer +# +# Since 2.5 +## +{ 'struct': 'CpuInfoX86', 'data': { 'pc': 'int' } } + +## +# @CpuInfoSparc: +# +# Additional information about a virtual Sparc CPU +# +# @pc: the PC component of the instruction pointer +# +# @npc: the NPC component of the instruction pointer +# +# Since 2.5 +## +{ 'struct': 'CpuInfoSparc', 'data': { 'pc': 'int', 'npc': 'int' } } + +## +# @CpuInfoPpc: +# +# Additional information about a virtual PPC CPU +# +# @nip: the instruction pointer +# +# Since 2.5 +## +{ 'struct': 'CpuInfoPpc', 'data': { 'nip': 'int' } } + +## +# @CpuInfoMips: +# +# Additional information about a virtual MIPS CPU +# +# @PC: the instruction pointer +# +# Since 2.5 +## +{ 'struct': 'CpuInfoMips', 'data': { 'PC': 'int' } } + +## +# @CpuInfoMips: +# +# Additional information about a virtual Tricore CPU +# +# @PC: the instruction pointer +# +# Since 2.5 +## +{ 'struct': 'CpuInfoTricore', 'data': { 'PC': 'int' } } + +## +# @CpuInfoOther:
qapi-schema.json line 852
Quoted 10 lines
qapi-schema.json +7
-0+# +# No additional information is available about the virtual CPU +# +# Since 2.5 +# +## +{ 'struct': 'CpuInfoOther', 'data': { } } ## # @query-cpus:
CpuInfo is only used as return type of query-cpus.
If I understand the change correctly, the value of query-cpus is not changed at all. Its introspection, however, is.
Do we need this to make 2.5?
Any other messy optionals that should really be (flat) unions?
Re: [Qemu-devel] [PATCH v10 29/30] cpu: Convert CpuInfo into flat union
Quoted 20 lines · Markus Armbruster
Quoted 17 lines · Eric Blake
When qapi type CpuInfo was originally created for 0.14, we had no notion of a flat union, and instead just listed a bunch of optional fields with documentation about the mutually-exclusive choice of which instruction pointer field(s) would be provided for a given architecture. But now that we have flat unions and introspection, it is better to segregate off which fields will be provided according to the actual architecture. With this in place, we no longer need the fields to be optional, because the choice of the discriminator serves that role.
This has an additional benefit: the old all-in-one struct was the only place in the code base that had a case-sensitive naming of members 'pc' vs. 'PC'. Separating these spellings into different branches of the flat union will allow us to add restrictions against future case-insensitive collisions, and make it possible for a future qemu to decide whether to enable case-insensitive QMP.
Quoted 7 lines
Quoted 2 lines
##
# @query-cpus:CpuInfo is only used as return type of query-cpus.
If I understand the change correctly, the value of query-cpus is not changed at all. Its introspection, however, is.
Almost. The output of query-cpus gains a new 'arch':'FOO' string per cpu, saying which branch of the flat union is in use. But as it is output-only, adding a new output field shouldn't break any existing clients (because QMP already documents that clients should ignore new fields); and since the struct is output-only, it can't break any input fields.
I probably can spell that out better in the commit message, though.
Oh, and I need to update qmp-commands.hx, as part of my v11 spin.
Quoted 2 lines
Do we need this to make 2.5?
It's true that the introspection will change (instead of seeing flat optional members, you now have to chase down variants). But I don't think it is pressing enough to rush into 2.5; the change is backwards-compatible no matter when we do it, and introspection clients are going to have to get used to the idea of chasing down different paths to find all members of a struct (that is, if we don't change this until 2.6, I'm sure it won't be the last case of introspection changing while keeping QMP wire format back-compatible as formerly non-variant fields become variant).
Quoted 3 lines
Any other messy optionals that should really be (flat) unions?
Possibly.
/me goes and audits all 0.14 interfaces...
add_client is an input command, but it would be nicer as a flat union (we can't do that until commands can take unions; but that happens as part of my pending queue for netdev_add), so that's not going to make 2.5.
client_migrate_info looks odd, since it requires 'protocol':'str' to be the value 'spice'. Probably several functions that take or produce a finite set of strings that should be using enums, but conversion from 'str' to 'enum' is backwards-compatible according to QMP wire format. Doing it sooner rather than later makes introspection nicer.
But I didn't spot any other obvious commands from that far back with mutually-exclusive optional parameters, and certainly nothing else with case clashes.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 29/30] cpu: Convert CpuInfo into flat union
Quoted 29 lines · Eric Blake
Quoted 20 lines · Markus Armbruster
Quoted 17 lines · Eric Blake
When qapi type CpuInfo was originally created for 0.14, we had no notion of a flat union, and instead just listed a bunch of optional fields with documentation about the mutually-exclusive choice of which instruction pointer field(s) would be provided for a given architecture. But now that we have flat unions and introspection, it is better to segregate off which fields will be provided according to the actual architecture. With this in place, we no longer need the fields to be optional, because the choice of the discriminator serves that role.
This has an additional benefit: the old all-in-one struct was the only place in the code base that had a case-sensitive naming of members 'pc' vs. 'PC'. Separating these spellings into different branches of the flat union will allow us to add restrictions against future case-insensitive collisions, and make it possible for a future qemu to decide whether to enable case-insensitive QMP.
Quoted 7 lines
Quoted 2 lines
##
# @query-cpus:CpuInfo is only used as return type of query-cpus.
If I understand the change correctly, the value of query-cpus is not changed at all. Its introspection, however, is.
Quoted 6 lines
Almost. The output of query-cpus gains a new 'arch':'FOO' string per cpu, saying which branch of the flat union is in use. But as it is output-only, adding a new output field shouldn't break any existing clients (because QMP already documents that clients should ignore new fields); and since the struct is output-only, it can't break any input fields.
Understood.
Quoted 1 line
I probably can spell that out better in the commit message, though.
Yes, please
Quoted 1 line
Oh, and I need to update qmp-commands.hx, as part of my v11 spin.
Yes, please.
Quoted 11 lines
Quoted 1 line
Do we need this to make 2.5?
It's true that the introspection will change (instead of seeing flat optional members, you now have to chase down variants). But I don't think it is pressing enough to rush into 2.5; the change is backwards-compatible no matter when we do it, and introspection clients are going to have to get used to the idea of chasing down different paths to find all members of a struct (that is, if we don't change this until 2.6, I'm sure it won't be the last case of introspection changing while keeping QMP wire format back-compatible as formerly non-variant fields become variant).
We can mess with the schema as long as the QMP wire format stays compatible.
With introspection, schema changes get exposed in QMP that weren't exposed before. Begs the question whether the ABI promise extends to the introspection value.
I think we don't want to extend it, at least for now. In other words, we refuse to put additional constraints on schema changes to keep the introspection value stable. Makes introspection a bit harder to use. Not ideal, but better than committing to constraints we don't even fully understand, yet.
I think we should spell this out introspection documentation.
Our only example so far is converting between optional members and flat unions. Can we think of others? Hmm, you found one below.
Quoted 9 lines
Quoted 2 lines
Any other messy optionals that should really be (flat) unions?
Possibly.
/me goes and audits all 0.14 interfaces...
add_clientis an input command, but it would be nicer as a flat union (we can't do that until commands can take unions; but that happens as part of my pending queue fornetdev_add), so that's not going to make 2.5.
Since the command parameters are just an object type, this is merely another case of converting between optional members and flat unions.
Quoted 5 lines
client_migrate_infolooks odd, since it requires 'protocol':'str' to be the value 'spice'. Probably several functions that take or produce a finite set of strings that should be using enums, but conversion from 'str' to 'enum' is backwards-compatible according to QMP wire format. Doing it sooner rather than later makes introspection nicer.
This is another example: converting between string and enum. Going from string to enum only adds information, and should be easy enough for clients. Going the other way removes information, and probably should not be done.
Quoted 3 lines
But I didn't spot any other obvious commands from that far back with mutually-exclusive optional parameters, and certainly nothing else with case clashes.
Okay. I guess finding all the examples that matter may take a few development cycles. Perhaps we want to commit to some additional compatibility promises for introspection then.
Re: [Qemu-devel] [PATCH v10 29/30] cpu: Convert CpuInfo into flat union
Quoted 19 lines · Markus Armbruster
Quoted 11 lines
Quoted 1 line
Do we need this to make 2.5?
It's true that the introspection will change (instead of seeing flat optional members, you now have to chase down variants). But I don't think it is pressing enough to rush into 2.5; the change is backwards-compatible no matter when we do it, and introspection clients are going to have to get used to the idea of chasing down different paths to find all members of a struct (that is, if we don't change this until 2.6, I'm sure it won't be the last case of introspection changing while keeping QMP wire format back-compatible as formerly non-variant fields become variant).
We can mess with the schema as long as the QMP wire format stays compatible.
With introspection, schema changes get exposed in QMP that weren't exposed before. Begs the question whether the ABI promise extends to the introspection value.
Quoted 7 lines
I think we don't want to extend it, at least for now. In other words, we refuse to put additional constraints on schema changes to keep the introspection value stable. Makes introspection a bit harder to use. Not ideal, but better than committing to constraints we don't even fully understand, yet.
I think we should spell this out introspection documentation.
Sounds like I need to prep a doc patch for 2.5 then.
Quoted 3 lines
Our only example so far is converting between optional members and flat unions. Can we think of others? Hmm, you found one below.
Converting from optional non-variant to variant, converting from string to enum, converting from single type to alternate. Might be other conversions, and the set of conversions for input types may differ from those of output conversions.
Quoted 3 lines
Okay. I guess finding all the examples that matter may take a few development cycles. Perhaps we want to commit to some additional compatibility promises for introspection then.
Maybe, but until clients are no longer worried about older clients, it may be several years before taking advantage of whatever we later document (as it is, upstream libvirt is just now bumping minimum qemu support up to 0.12 [thanks to RHEL 6] and discarding older qemu from RHEL 5 - quite a few years behind upstream qemu 2.5, although it is finding a lot of code in libvirt that is no longer necessary).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH v10 29/30] cpu: Convert CpuInfo into flat union
Quoted 27 lines · Eric Blake
Quoted 25 lines · Markus Armbruster
Quoted 11 lines
Quoted 1 line
Do we need this to make 2.5?
It's true that the introspection will change (instead of seeing flat optional members, you now have to chase down variants). But I don't think it is pressing enough to rush into 2.5; the change is backwards-compatible no matter when we do it, and introspection clients are going to have to get used to the idea of chasing down different paths to find all members of a struct (that is, if we don't change this until 2.6, I'm sure it won't be the last case of introspection changing while keeping QMP wire format back-compatible as formerly non-variant fields become variant).
We can mess with the schema as long as the QMP wire format stays compatible.
With introspection, schema changes get exposed in QMP that weren't exposed before. Begs the question whether the ABI promise extends to the introspection value.
I think we don't want to extend it, at least for now. In other words, we refuse to put additional constraints on schema changes to keep the introspection value stable. Makes introspection a bit harder to use. Not ideal, but better than committing to constraints we don't even fully understand, yet.
Quoted 3 lines
Quoted 1 line
I think we should spell this out introspection documentation.
Sounds like I need to prep a doc patch for 2.5 then.
Yes, please!
Quoted 8 lines
Quoted 2 lines
Our only example so far is converting between optional members and flat unions. Can we think of others? Hmm, you found one below.
Converting from optional non-variant to variant, converting from string to enum, converting from single type to alternate. Might be other conversions, and the set of conversions for input types may differ from those of output conversions.
Quoted 7 lines
Quoted 3 lines
Okay. I guess finding all the examples that matter may take a few development cycles. Perhaps we want to commit to some additional compatibility promises for introspection then.
Maybe, but until clients are no longer worried about older clients, it may be several years before taking advantage of whatever we later document
That's life.
I fully expect the consumers of QMP introspection to get a few things wrong initially, requiring their users to upgrade to fixed versions. That's life, too.
Quoted 4 lines
(as it is, upstream libvirt is just now bumping minimum qemu
support up to 0.12 [thanks to RHEL 6] and discarding older qemu from RHEL 5 - quite a few years behind upstream qemu 2.5, although it is finding a lot of code in libvirt that is no longer necessary).
In my opinion, attempting to support old versions of QEMU is a complete waste of resources unless you actually test them to stop the bit-rot.
RHEL-6 0.12 is by now quite different from upstream 0.12. Unlike upstream 0.12, it's actually still relevant and useful today.
I encourage you to identify relevant upstream and downstream versions, keep them tested, and drop support for the rest.
PATCH v10 30/30 qapi: Forbid case-insensitive clashes
In reply to Eric BlakeWe have toyed on list with the idea of a future extension to QMP of teaching it to be case-insensitive (the user could request command 'Quit' instead of 'quit', or could spell a struct field as 'CPU' instead of 'cpu'). But for that to be a practical extension, we cannot break backwards compatibility with any existing struct that was already relying on case sensitivity. Fortunately, after the previous patch cleaned up CpuInfo, there are no such existing qapi structs.
Another benefit of enforcing a restriction against case-insensitive is that we no longer have to worry about the case of enum values that could be distinguished by case if mapped by c_name(), but which cannot be distinguished when mapped to C as ALL_CAPS by camel_to_uppper(). Having the generator look for case collisions up front will prevent developers from worrying whether different munging rules for member names compared to enum values as a discriminator will cause any problems in qapi unions.
Of course, if we never implement a case-insensitive QMP extension, or find a legitimate reason to need qapi members that differ only by case, we could always relax this restriction. But it is easier to relax later than it is to wish we had the restriction in place earlier.
---
v10: new patch
--- docs/qapi-code-gen.txt | 5 +++++ scripts/qapi.py | 4 ++-- tests/Makefile | 1 + tests/qapi-schema/args-case-clash.err | 1 + tests/qapi-schema/args-case-clash.exit | 1 + tests/qapi-schema/args-case-clash.json | 5 +++++ tests/qapi-schema/args-case-clash.out | 0 7 files changed, 15 insertions(+), 2 deletions(-) create mode 100644 tests/qapi-schema/args-case-clash.err create mode 100644 tests/qapi-schema/args-case-clash.exit create mode 100644 tests/qapi-schema/args-case-clash.json create mode 100644 tests/qapi-schema/args-case-clash.out
docs/qapi-code-gen.txt +5 -0
@@ -116,6 +116,11 @@ names should be ALL_CAPS with words separated by underscore. Field names cannot start with 'has-' or 'has_', as this is reserved for tracking optional fields. +For now, Client JSON Protocol is case-sensitive, but future extensions +may allow for case-insensitive recognition of command and event names, +or of member field names. As such, the generator rejects attempts to +create entities that only differ by case. + Any name (command, event, type, field, or enum value) beginning with "x-" is marked experimental, and may be withdrawn or changed incompatibly in a future release. Downstream vendors may add
scripts/qapi.py +2 -2
@@ -1039,7 +1039,7 @@ class QAPISchemaObjectTypeMember(object): assert self.type def check_clash(self, info, seen): - name = c_name(self.name) + name = c_name(self.name).upper() if name in seen: raise QAPIExprError(info, "%s collides with %s"
@@ -1087,7 +1087,7 @@ class QAPISchemaObjectTypeVariants(object): def check(self, schema, seen): if self.tag_name: # flat union - self.tag_member = seen[c_name(self.tag_name)] + self.tag_member = seen[c_name(self.tag_name).upper()] assert self.tag_name == self.tag_member.name tag_type = self.tag_member.type assert isinstance(tag_type, QAPISchemaEnumType)
tests/Makefile +1 -0
@@ -239,6 +239,7 @@ qapi-schema += args-alternate.json qapi-schema += args-any.json qapi-schema += args-array-empty.json qapi-schema += args-array-unknown.json +qapi-schema += args-case-clash.json qapi-schema += args-int.json qapi-schema += args-invalid.json qapi-schema += args-member-array-bad.json
tests/qapi-schema/args-case-clash.err new file +1 -0
@@ -0,0 +1 @@ +tests/qapi-schema/args-case-clash.json:5: 'A' (argument of oops) collides with 'a' (argument of oops)
tests/qapi-schema/args-case-clash.exit new file +1 -0
@@ -0,0 +1 @@ +1
tests/qapi-schema/args-case-clash.json new file +5 -0
@@ -0,0 +1,5 @@ +# C member name collision +# Reject members that clash case-insensitively (our mapping to C names +# preserves case, but allowing these members now would prevent a future +# relaxing of QMP to be case-insensitive). +{ 'command': 'oops', 'data': { 'a': 'str', 'A': 'int' } }
tests/qapi-schema/args-case-clash.out new file +0 -0
Signature · 1 line
2.4.3
Re: [Qemu-devel] [PATCH v10 30/30] qapi: Forbid case-insensitive clashes
Quoted 9 lines · Eric Blake
We have toyed on list with the idea of a future extension to QMP of teaching it to be case-insensitive (the user could request command 'Quit' instead of 'quit', or could spell a struct field as 'CPU' instead of 'cpu'). But for that to be a practical extension, we cannot break backwards compatibility with any existing struct that was already relying on case sensitivity. Fortunately, after the previous patch cleaned up CpuInfo, there are no such existing qapi structs.
Quoted 5 lines
Another benefit of enforcing a restriction against case-insensitive is that we no longer have to worry about the case of enum values that could be distinguished by case if mapped by
c_name(), but which cannot be distinguished when mapped to C asALL_CAPSbycamel_to_uppper(). Having the
camel_to_upper()
Quoted 5 lines
generator look for case collisions up front will prevent developers from worrying whether different munging rules for member names compared to enum values as a discriminator will cause any problems in qapi unions.
Quoted 7 lines
Of course, if we never implement a case-insensitive QMP extension, or find a legitimate reason to need qapi members that differ only by case, we could always relax this restriction. But it is easier to relax later than it is to wish we had the restriction in place earlier.
Signed-off-by Eric Blake <[email protected]>
I think "make QMP case-insensitive" is a very weak justification for anything.
Your second reason is much stronger: forbidding case-insensitive clashes lets the generators shout without fear of introducing clashes in generated code.
However, since camel_to_upper() does more than just shout, forbidding case-insensitive clashes protects against clashes in generated code only for names where camel_to_upper() just shouts. It does for lower case names, i.e. almost all of them.
The commit would make more sense if we first defanged c_enum_const():
make it mangle the const_name part like c_name(const_name).upper(). This is independent of the argument I'm having with Dan on how the prefix should be mangled.
One more good argument for forbidding case-insensitive clashes: an interface that sports names differing only in case is a bad interface.
Quoted 21 lines
---
v10: new patch--- docs/qapi-code-gen.txt | 5 +++++ scripts/qapi.py | 4 ++-- tests/Makefile | 1 + tests/qapi-schema/args-case-clash.err | 1 + tests/qapi-schema/args-case-clash.exit | 1 + tests/qapi-schema/args-case-clash.json | 5 +++++ tests/qapi-schema/args-case-clash.out | 0 7 files changed, 15 insertions(+), 2 deletions(-) create mode 100644 tests/qapi-schema/args-case-clash.err create mode 100644 tests/qapi-schema/args-case-clash.exit create mode 100644 tests/qapi-schema/args-case-clash.json create mode 100644 tests/qapi-schema/args-case-clash.outdocs/qapi-code-gen.txt +0
-0
docs/qapi-code-gen.txt names should be ALL_CAPS with words separated by underscore. Field · line 123
Quoted 8 lines
docs/qapi-code-gen.txt +5
-0@@ -116,6 +116,11 @@ names should be ALL_CAPS with words separated by underscore. Field names cannot start with 'has-' or 'has_', as this is reserved for tracking optional fields. +For now, Client JSON Protocol is case-sensitive, but future extensions +may allow for case-insensitive recognition of command and event names, +or of member field names. As such, the generator rejects attempts to +create entities that only differ by case. +
This suggests we're actually planning to make QMP case-insensitive. Let's avoid that. Perhaps:
Since we don't want interfaces with different names that differ only in case, the generator rejects such names.
Quoted 17 lines
docs/qapi-code-gen.txt +0
-0Any name (command, event, type, field, or enum value) beginning with "x-" is marked experimental, and may be withdrawn or changed incompatibly in a future release. Downstream vendors may addscripts/qapi.py +1
-1@@ -1039,7 +1039,7 @@ class QAPISchemaObjectTypeMember(object): assert self.type def check_clash(self, info, seen): - name = c_name(self.name) + name = c_name(self.name).upper() if name in seen: raise QAPIExprError(info, "%s collides with %s"
scripts/qapi.py class QAPISchemaObjectTypeVariants(object): · line 1093
Quoted 8 lines
scripts/qapi.py +1
-1@@ -1087,7 +1087,7 @@ class QAPISchemaObjectTypeVariants(object): def check(self, schema, seen): if self.tag_name: # flat union - self.tag_member = seen[c_name(self.tag_name)] + self.tag_member = seen[c_name(self.tag_name).upper()] assert self.tag_name == self.tag_member.name tag_type = self.tag_member.type assert isinstance(tag_type, QAPISchemaEnumType)
If we ever acquire more lookups in seen, we'll need a lookup function, so we don't duplicate c_name(NAME).upper(). But this will do for now.
Quoted 32 lines
tests/Makefile +1
-0@@ -239,6 +239,7 @@ qapi-schema += args-alternate.json qapi-schema += args-any.json qapi-schema += args-array-empty.json qapi-schema += args-array-unknown.json +qapi-schema += args-case-clash.json qapi-schema += args-int.json qapi-schema += args-invalid.json qapi-schema += args-member-array-bad.jsontests/qapi-schema/args-case-clash.err new file +1
-0@@ -0,0 +1 @@ +tests/qapi-schema/args-case-clash.json:5: 'A' (argument of oops) collides with 'a' (argument of oops)tests/qapi-schema/args-case-clash.exit new file +1
-0@@ -0,0 +1 @@ +1tests/qapi-schema/args-case-clash.json new file +0
-0
tests/qapi-schema/args-case-clash.json line 4
Quoted 4 lines
tests/qapi-schema/args-case-clash.json new file +4
-0@@ -0,0 +1,5 @@ +# C member name collision +# Reject members that clash case-insensitively (our mapping to C names +# preserves case, but allowing these members now would prevent a future +# relaxing of QMP to be case-insensitive).
Again, let's not suggest we're planning to make QMP case-insensitive.
Quoted 4 lines
tests/qapi-schema/args-case-clash.json new file +1
-0+{ 'command': 'oops', 'data': { 'a': 'str', 'A': 'int' } }tests/qapi-schema/args-case-clash.out new file +0
-0
Quoted 23 lines · Eric Blake
No pending prerequisites; based on qemu.git master
Also available as a tag at this location:
git fetchgit://repo.or.cz/qemu/ericb.gitqapi-cleanupv9cand will soon be part of my branch with the rest of the v5 series, at:
http://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/qapiv10 notes:
Split several patches, redo the middle patches from Markus to be back in the order they were first posted, some fallout change to my patches due to the nicer pattern of minimizing conditionals inside .check(), by instead calling .check_clash() as needed. Changedata->errmagic in tests to instead use a new helpererror_free_or_abort(). Add a patch that would prevent qapi case-insensitive clashes.I am redoing my subset boundaries slightly: patches 23-27 of v9 (updating the alternate layout) will be delayed to subset D, and 2 other patches previously posted in subset D are now here (turning qapi clash checking into actual error messages), so the subject line of this cover letter is slightly different.
Quoted 3 lines
Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.
I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.
I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.
Quoted 6 lines · Markus Armbruster
Quoted 4 lines · Eric Blake
No pending prerequisites; based on qemu.git master
Also available as a tag at this location:
git fetchgit://repo.or.cz/qemu/ericb.gitqapi-cleanupv9c
Make that the qapi-cleanupv10c label.
Quoted 23 lines
Quoted 19 lines
and will soon be part of my branch with the rest of the v5 series, at:
http://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/qapiv10 notes:
Split several patches, redo the middle patches from Markus to be back in the order they were first posted, some fallout change to my patches due to the nicer pattern of minimizing conditionals inside .check(), by instead calling .check_clash() as needed. Changedata->errmagic in tests to instead use a new helpererror_free_or_abort(). Add a patch that would prevent qapi case-insensitive clashes.I am redoing my subset boundaries slightly: patches 23-27 of v9 (updating the alternate layout) will be delayed to subset D, and 2 other patches previously posted in subset D are now here (turning qapi clash checking into actual error messages), so the subject line of this cover letter is slightly different.
Quoted 3 lines
Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.
Quoted 5 lines
I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.
I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.
Yes, 01-12 seems like a good first set, if you want to make those touchups (I've supplied some potential text improvements in reply to some of your comments); I'm happy, as always, to take a peek over your staging repo to double check what you are prepping for the pull request.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Quoted 2 lines · Eric Blake
Quoted 1 line · Markus Armbruster
Eric Blake <[email protected]> writes:
[...]
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.
I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.
Quoted 7 lines
Quoted 2 lines
I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.
Yes, 01-12 seems like a good first set, if you want to make those touchups (I've supplied some potential text improvements in reply to some of your comments); I'm happy, as always, to take a peek over your staging repo to double check what you are prepping for the pull request.
Please have a look at my qapi-next branch.
Quoted 15 lines · Markus Armbruster
Quoted 2 lines · Eric Blake
Quoted 1 line · Markus Armbruster
Eric Blake <[email protected]> writes:
[...]
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.
I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.
I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.
Quoted 6 lines
Quoted 4 lines
Yes, 01-12 seems like a good first set, if you want to make those touchups (I've supplied some potential text improvements in reply to some of your comments); I'm happy, as always, to take a peek over your staging repo to double check what you are prepping for the pull request.
Please have a look at my qapi-next branch.
Close, but commit 5f72bb85 on that branch is missing the one-line summary:
qapi: Simplify error cleanup in test-qmp-*
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Quoted 21 lines · Eric Blake
Quoted 20 lines · Markus Armbruster
Quoted 2 lines · Eric Blake
Quoted 1 line · Markus Armbruster
Eric Blake <[email protected]> writes:
[...]
Quoted 14 lines
Quoted 9 lines
Quoted 3 lines
Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.
I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.
I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.
Yes, 01-12 seems like a good first set, if you want to make those touchups (I've supplied some potential text improvements in reply to some of your comments); I'm happy, as always, to take a peek over your staging repo to double check what you are prepping for the pull request.
Quoted 5 lines
Quoted 1 line
Please have a look at my qapi-next branch.
Close, but commit
5f72bb85on that branch is missing the one-line summary:qapi: Simplify error cleanup in test-qmp-*
Fixed & sent pull request, thanks!
Quoted 23 lines · Markus Armbruster
Quoted 21 lines · Eric Blake
Quoted 20 lines · Markus Armbruster
Quoted 2 lines · Eric Blake
Quoted 1 line · Markus Armbruster
Eric Blake <[email protected]> writes:
[...]
Quoted 14 lines
Quoted 9 lines
Quoted 3 lines
Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.
I guess you mean PATCH 01-12. I had a few questions, but the most likely outcome seems to be minor touchups I could apply in my tree.
I'm okay with trying to get more patches in, but let's get these out of the way meanwhile.
Yes, 01-12 seems like a good first set, if you want to make those touchups (I've supplied some potential text improvements in reply to some of your comments); I'm happy, as always, to take a peek over your staging repo to double check what you are prepping for the pull request.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Please have a look at my qapi-next branch.
Close, but commit
5f72bb85on that branch is missing the one-line summary:qapi: Simplify error cleanup in test-qmp-*
Fixed & sent pull request, thanks!
Pulled.
I think I'll close the QAPI floodgates for 2.5 now. Bug fixes are of course exempted, and if we find something that impacts ABI, I'm willing to consider patches. Work on our backlog can continue uninterrupted; I'm happy to collect patches that are ready, and will take care of getting them into master once 2.6 opens.
Quoted 5 lines · Markus Armbruster
I think I'll close the QAPI floodgates for 2.5 now. Bug fixes are of course exempted, and if we find something that impacts ABI, I'm willing to consider patches. Work on our backlog can continue uninterrupted; I'm happy to collect patches that are ready, and will take care of getting them into master once 2.6 opens.
I think I found something that ought to go into 2.5: we have nothing in current query-qmp-introspection that shows the set of valid 'ErrorClass' enum values that a client can expect on error (this is because no existing commands have a reference to the type). While we are unlikely to be adding new ErrorClass values any time soon, this feels like something that should be exposed over the wire.
I'm trying to play with ideas on how best to expose it.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org