Skip to contentSkip to conversation

qapi member collision, alternate layout (post-introspection cleanups, subset D)

PATCH Superseded Eric Blake 6 participants 102 messages Nov 18, 2015 – Feb 2, 2016

Subsystems (2)

v12 of 14 Superseded Latest is v14 →

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

v12 cover qapi member collision, alternate layout (post-introspection cleanups, subset D)

No pending prerequisites; based on qemu.git master

Also available as a tag at this location:
git fetch git://repo.or.cz/qemu/ericb.git qapi-cleanupv12d

and will soon be part of my branch with the rest of the v5 series, at:
http://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/qapi

v12 notes:
Delay 26-28 to a later subset, and instead add lots of new prereq patches that tackle some cleanups that make case-insensitive collision detection easier. Series is now slated for 2.6.

001/36:[----] [--] 'qapi: Track simple union tag in object.local_members' 002/36:[----] [--] 'qapi-types: Consolidate gen_struct() and gen_union()' 003/36:[----] [--] 'qapi-types: Simplify gen_struct_field[s]' 004/36:[----] [--] 'qapi: Drop obsolete tag value collision assertions' 005/36:[----] [--] 'qapi: Simplify QAPISchemaObjectTypeMember.check()' 006/36:[----] [--] 'qapi: Clean up after previous commit' 007/36:[----] [--] 'qapi: Fix up commit 7618b91's clash sanity checking change' 008/36:[----] [--] 'qapi: Eliminate QAPISchemaObjectType.check() variable members' 009/36:[----] [--] 'qapi: Factor out QAPISchemaObjectTypeMember.check_clash()' 010/36:[----] [--] 'qapi: Simplify QAPISchemaObjectTypeVariants.check()' 011/36:[down] 'qapi: Check for qapi collisions involving variant members' 012/36:[----] [--] 'qapi: Factor out QAPISchemaObjectType.check_clash()' 013/36:[----] [--] 'qapi: Hoist tag collision check to Variants.check()' 014/36:[----] [--] 'qapi: Remove outdated tests related to QMP/branch collisions' 015/36:[----] [--] 'qapi: Track owner of each object member' 016/36:[----] [--] 'qapi: Detect collisions in C member names' 017/36:[down] 'qapi: Fix c_name() munging'
018/36:[down] 'qapi: Remove dead visitor code'
019/36:[down] 'blkdebug: Merge hand-rolled and qapi BlkdebugEvent enum' 020/36:[down] 'blkdebug: Avoid '.' in enum values'
021/36:[down] 'qapi: Tighten the regex on valid names'
022/36:[down] 'qapi: Don't let implicit enum MAX member collide' 023/36:[down] 'qapi: Remove dead tests for max collision' 024/36:[0018] [FC] 'cpu: Convert CpuInfo into flat union' 025/36:[down] 'qapi: Add alias for ErrorClass'
026/36:[0098] [FC] 'qapi: Change munging of CamelCase enum values' 027/36:[0049] [FC] 'qapi: Forbid case-insensitive clashes' 028/36:[down] 'qapi: Simplify QObject'
029/36:[down] 'qobject: Rename qtype_code to QType'
030/36:[down] 'qapi: Convert QType into qapi builtin enum type' 031/36:[0020] [FC] 'qapi: Simplify visiting of alternate types' 032/36:[down] 'qapi: Inline _make_implicit_tag()'
033/36:[0015] [FC] 'qapi: Fix alternates that accept 'number' but not 'int'' 034/36:[----] [--] 'qapi: Add positive tests to qapi-schema-test' 035/36:[0010] [FC] 'qapi: Simplify visits of optional fields' 036/36:[down] 'qapi: Shorter visits of optional fields'

v11 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg02563.html
First half of v10 subset C has been applied, so consolidate the last half of that along with all of subset D (which was at v9) into one group. Address list reviews, in particular, add a new patch 21 that makes alternate layouts a lot nicer by making qtype_code a builtin qapi type; and new patches 18-19 that try to reduce confusion on the use of camel_to_upper() in c_enum_const().

Probably too late to get these into 2.5, in which case 17/28 will need tweaks to call out 2.6.

v10 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg01249.html
Split several patches, redo the middle patches from Markus to be back in the order they were first posted, some fallout change to my patches due to the nicer pattern of minimizing conditionals inside .check(), by instead calling .check_clash() as needed. Change data->err magic in tests to instead use a new helper error_free_or_abort(). Add a patch that would prevent qapi case-insensitive clashes.

I am redoing my subset boundaries slightly: patches 23-27 of v9 (updating the alternate layout) will be delayed to subset D, and 2 other patches previously posted in subset D are now here (turning qapi clash checking into actual error messages), so the subject line of this cover letter is slightly different.

Hopefully, we are converging on something that will be ready for a pull request, especially for the earlier patches of this subset.

v9 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg00652.html
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg06999.html
More patches added, and several reorganized. Lots of new patches from Markus, although not in the order originally proposed.

The first 8 patches are fairly straightforward, and could probably be taken as-is. Patch 9 is a rewrite of v8 4/17, but in the opposite direction (document that no sorting is done, rather than attempting to sort), so it may need further fine-tuning. Patches 12-21 represents a fusion of Markus' and my attempts to rewrite v5 7/17 into a more-reviewable set of patches, and caused further churn later in the series.

Patch 23 still uses tag_member.type == None; I ran out of time to work on Markus' idea of providing an instance of QAPISchemaBuiltinType to fill the role for 'qtype_code' without being exposed through .json files and without breaking the invariant of a valid member.type after check(), and wanted to get the rest of the series started under revew. So I may need a followup patch or even a v10 of the later half of this series after exploring that idea more.

v8 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg06674.html
Minor changes when rebasing to latest, improved commit messages in a few places, plus the addition of a few new patches. Markus started reviewing v7, but hadn't gotten very far.

v7 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg04112.html
Patches 1-3 of the previous round have moved to subset B; patch 7 is gone, and a couple of new patches are present in this round. The latest version of subset B made it much easier to reason about collision detection (namely, tag values can't collide with QMP values thanks to a named rather than anonymous union in the C type), and I like how things turned out. I suspect the hardest part of the review will be patches 5/14 and 7/14, although none of this has really had much review in any earlier versions.

v6 notes:
https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg01980.html
Add some patches and rebase onto work on subset B. Rearrange some patches from v5 (this set includes 17-20, 23, 25-27). Backport diff gets a bit confused by one patch title changing.

Not much direct review comments, although some of the changes here are updated based on comments made on other patches in the v5 series.

Subset D (and more?) will come later.

In v5:
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg05410.html
I _did_ rearrange patches to try and group related features:

1-2: Groundwork cleanups
3-5: Add more test cases
6-16: Front-end cleanups
17-18: Introspection output cleanups
19-20: 'alternate' type cleanups
21-29: qapi visitor cleanups
30-45: qapi-ify netdev_add
46: add qapi shorthand for flat unions

Lots of fixes based on additional testing, and rebased to track other changes that happened in the meantime. The series is huge; I can split off smaller portions as requested.

In v4:
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg02580.html
add some more clean up patches
rebase to Markus' recent work
pull in part of Zoltán's work to make netdev_add a flat union, further enhancing it to be introspectible

I might be able to rearrange some of these patches, or separate it into smaller independent series, if requested; but I'm posting now to get review started.

In v3:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg02059.html
redo cleanup of dealloc of partial struct
add patches to make all visit_type_*() avoid leaks on failure add patches to allow boxed command arguments and events

In v2:
https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg00900.html
rebase to Markus' v3 series
rework how comments are emitted for fields inherited from base additional patches added for deleting colliding 'void *data' documentation updates to match code changes

v1 was here:
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05266.html
https://lists.gnu.org/archive/html/qemu-devel/2015-07/msg05325.html

Eric Blake (29):
qapi: Track simple union tag in object.local_members
qapi-types: Consolidate gen_struct() and gen_union()
qapi-types: Simplify gen_struct_field[s]
qapi: Check for qapi collisions involving variant members
qapi: Factor out QAPISchemaObjectType.check_clash()
qapi: Hoist tag collision check to Variants.check()
qapi: Remove outdated tests related to QMP/branch collisions
qapi: Track owner of each object member
qapi: Detect collisions in C member names
qapi: Fix c_name() munging
qapi: Remove dead visitor code
blkdebug: Merge hand-rolled and qapi BlkdebugEvent enum
blkdebug: Avoid '.' in enum values
qapi: Tighten the regex on valid names
qapi: Don't let implicit enum MAX member collide
qapi: Remove dead tests for max collision
cpu: Convert CpuInfo into flat union
qapi: Add alias for ErrorClass
qapi: Change munging of CamelCase enum values
qapi: Forbid case-insensitive clashes
qapi: Simplify QObject
qobject: Rename qtype_code to QType
qapi: Convert QType into qapi builtin enum type
qapi: Simplify visiting of alternate types
qapi: Inline _make_implicit_tag()
qapi: Fix alternates that accept 'number' but not 'int'
qapi: Add positive tests to qapi-schema-test
qapi: Simplify visits of optional fields
qapi: Shorter visits of optional fields

Markus Armbruster (7):
qapi: Drop obsolete tag value collision assertions
qapi: Simplify QAPISchemaObjectTypeMember.check()
qapi: Clean up after previous commit
qapi: Fix up commit 7618b91's clash sanity checking change
qapi: Eliminate QAPISchemaObjectType.check() variable members
qapi: Factor out QAPISchemaObjectTypeMember.check_clash()
qapi: Simplify QAPISchemaObjectTypeVariants.check()

 block.c                                            |   2 +-
 block/blkdebug.c                                   |  79 +------
 block/parallels.c                                  |   4 +-
 block/qapi.c                                       |   4 +-
 block/qcow2.c                                      |   2 +-
 block/quorum.c                                     |   2 +-
 block/raw-posix.c                                  |   2 +-
 blockdev.c                                         |   2 +-
 cpus.c                                             |  31 +--
 docs/blkdebug.txt                                  |   7 +-
 docs/qapi-code-gen.txt                             |  34 +--
 hmp.c                                              |  44 ++--
 hw/char/escc.c                                     |   2 +-
 hw/i386/pc_piix.c                                  |   2 +-
 hw/i386/pc_q35.c                                   |   2 +-
 hw/input/hid.c                                     |   6 +-
 hw/input/ps2.c                                     |   6 +-
 hw/input/virtio-input-hid.c                        |  12 +-
 include/block/block.h                              |  62 +-----
 include/block/block_int.h                          |   2 +-
 include/hw/qdev-core.h                             |   2 +-
 include/migration/migration.h                      |   4 +-
 include/qapi/error.h                               |  14 ++
 include/qapi/qmp/qobject.h                         |  53 ++---
 include/qapi/visitor-impl.h                        |   8 +-
 include/qapi/visitor.h                             |  22 +-
 migration/migration.c                              |   4 +-
 monitor.c                                          |  18 +-
 net/net.c                                          |   4 +-
 qapi-schema.json                                   | 120 +++++++++--
 qapi/block-core.json                               |  20 +-
 qapi/common.json                                   |   5 +-
 qapi/opts-visitor.c                                |   2 +-
 qapi/qapi-visit-core.c                             |  10 +-
 qapi/qmp-dispatch.c                                |   2 +-
 qapi/qmp-input-visitor.c                           |  10 +-
 qapi/string-input-visitor.c                        |   3 +-
 qemu-nbd.c                                         |   2 +-
 qmp-commands.hx                                    |   4 +
 qobject/qbool.c                                    |   7 +-
 qobject/qdict.c                                    |  10 +-
 qobject/qfloat.c                                   |   7 +-
 qobject/qint.c                                     |   7 +-
 qobject/qlist.c                                    |   7 +-
 qobject/qnull.c                                    |  13 +-
 qobject/qstring.c                                  |   7 +-
 replay/replay-input.c                              |   8 +-
 scripts/qapi-types.py                              | 116 +++-------
 scripts/qapi-visit.py                              |  35 +++-
 scripts/qapi.py                                    | 233 ++++++++++++++-------
 tests/Makefile                                     |   9 +-
 tests/qapi-schema/alternate-empty.out              |   3 +-
 tests/qapi-schema/args-case-clash.err              |   1 +
 .../{union-max.exit => args-case-clash.exit}       |   0
 tests/qapi-schema/args-case-clash.json             |   4 +
 .../{union-max.out => args-case-clash.out}         |   0
 tests/qapi-schema/args-name-clash.err              |   1 +
 tests/qapi-schema/args-name-clash.exit             |   2 +-
 tests/qapi-schema/args-name-clash.json             |   5 +-
 tests/qapi-schema/args-name-clash.out              |   6 -
 tests/qapi-schema/command-type-case-clash.err      |   1 +
 ...lash-type.exit => command-type-case-clash.exit} |   0
 tests/qapi-schema/command-type-case-clash.json     |   3 +
 ...-clash-type.out => command-type-case-clash.out} |   0
 tests/qapi-schema/comments.out                     |   2 +
 tests/qapi-schema/empty.out                        |   2 +
 tests/qapi-schema/enum-max-member.err              |   1 -
 tests/qapi-schema/enum-max-member.exit             |   1 -
 tests/qapi-schema/enum-max-member.json             |   3 -
 tests/qapi-schema/enum-max-member.out              |   0
 tests/qapi-schema/event-case.out                   |   2 +
 tests/qapi-schema/event-max.err                    |   1 -
 tests/qapi-schema/event-max.exit                   |   1 -
 tests/qapi-schema/event-max.json                   |   2 -
 tests/qapi-schema/event-max.out                    |   0
 tests/qapi-schema/flat-union-clash-branch.err      |   0
 tests/qapi-schema/flat-union-clash-branch.exit     |   1 -
 tests/qapi-schema/flat-union-clash-branch.json     |  18 --
 tests/qapi-schema/flat-union-clash-branch.out      |  14 --
 tests/qapi-schema/flat-union-clash-type.err        |   1 -
 tests/qapi-schema/flat-union-clash-type.json       |  14 --
 tests/qapi-schema/flat-union-empty.out             |   2 +
 tests/qapi-schema/ident-with-escape.out            |   2 +
 tests/qapi-schema/include-relpath.out              |   2 +
 tests/qapi-schema/include-repetition.out           |   2 +
 tests/qapi-schema/include-simple.out               |   2 +
 tests/qapi-schema/indented-expr.out                |   2 +
 tests/qapi-schema/qapi-schema-test.json            |  21 +-
 tests/qapi-schema/qapi-schema-test.out             |  39 +++-
 tests/qapi-schema/reserved-member-underscore.err   |   1 +
 ...h-type.exit => reserved-member-underscore.exit} |   0
 tests/qapi-schema/reserved-member-underscore.json  |   4 +
 ...ash-type.out => reserved-member-underscore.out} |   0
 tests/qapi-schema/union-clash-data.out             |   3 +
 tests/qapi-schema/union-clash-type.err             |   1 -
 tests/qapi-schema/union-clash-type.json            |   9 -
 tests/qapi-schema/union-empty.out                  |   3 +
 tests/qapi-schema/union-max.err                    |   1 -
 tests/qapi-schema/union-max.json                   |   3 -
 tests/qemu-iotests/026                             |  18 +-
 tests/qemu-iotests/026.out                         |  80 +++----
 tests/qemu-iotests/026.out.nocache                 |  80 +++----
 tests/qemu-iotests/077                             |  24 +--
 tests/test-qmp-commands.c                          |   4 +
 tests/test-qmp-input-visitor.c                     |  29 ++-
 tests/test-qmp-output-visitor.c                    |  10 +-
 tests/test-string-output-visitor.c                 |   4 +-
 tpm.c                                              |  10 +-
 ui/cocoa.m                                         |   6 +-
 ui/console.c                                       |   2 +-
 ui/gtk.c                                           |   4 +-
 ui/input-keymap.c                                  |   4 +-
 ui/input-legacy.c                                  |  10 +-
 ui/input.c                                         |   6 +-
 ui/sdl.c                                           |   6 +-
 ui/sdl2.c                                          |   6 +-
 ui/spice-input.c                                   |   6 +-
 ui/vnc.c                                           |   6 +-
 vl.c                                               |  18 +-
 119 files changed, 806 insertions(+), 786 deletions(-)
 create mode 100644 tests/qapi-schema/args-case-clash.err
 rename tests/qapi-schema/{union-max.exit => args-case-clash.exit} (100%)
 create mode 100644 tests/qapi-schema/args-case-clash.json
 rename tests/qapi-schema/{union-max.out => args-case-clash.out} (100%)
 create mode 100644 tests/qapi-schema/command-type-case-clash.err
 rename tests/qapi-schema/{union-clash-type.exit => command-type-case-clash.exit} (100%)
 create mode 100644 tests/qapi-schema/command-type-case-clash.json
 rename tests/qapi-schema/{union-clash-type.out => command-type-case-clash.out} (100%)
 delete mode 100644 tests/qapi-schema/enum-max-member.err
 delete mode 100644 tests/qapi-schema/enum-max-member.exit
 delete mode 100644 tests/qapi-schema/enum-max-member.json
 delete mode 100644 tests/qapi-schema/enum-max-member.out
 delete mode 100644 tests/qapi-schema/event-max.err
 delete mode 100644 tests/qapi-schema/event-max.exit
 delete mode 100644 tests/qapi-schema/event-max.json
 delete mode 100644 tests/qapi-schema/event-max.out
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.exit
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.json
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.out
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.json
 create mode 100644 tests/qapi-schema/reserved-member-underscore.err
 rename tests/qapi-schema/{flat-union-clash-type.exit => reserved-member-underscore.exit} (100%)
 create mode 100644 tests/qapi-schema/reserved-member-underscore.json
 rename tests/qapi-schema/{flat-union-clash-type.out => reserved-member-underscore.out} (100%)
 delete mode 100644 tests/qapi-schema/union-clash-type.err
 delete mode 100644 tests/qapi-schema/union-clash-type.json
 delete mode 100644 tests/qapi-schema/union-max.err
 delete mode 100644 tests/qapi-schema/union-max.json
Signature · 1 line
2.4.3
author
#1 lore raw

PATCH v12 1/36 qapi: Track simple union tag in object.local_members

We were previously creating all unions with an empty list for local_members. However, it will make it easier to unify struct and union generation if we include the generated tag member in local_members. That way, we can have a common code pattern:
visit the base (if any), visit the local members (if any), visit the variants (if any). The local_members of a flat union remains empty (because the discriminator is already visited as part of the base). Then, by visiting tag_member.check() during AlternateType.check(), we no longer need to call it during Variants.check().

The various front end entities now exist as follows:
struct: optional base, optional local_members, no variants simple union: no base, one-element local_members, variants with tag_member
from local_members
flat union: base, no local_members, variants with tag_member from base
alternate: no base, no local_members, variants

With the new local members, we require a bit of finesse to avoid assertions in the clients.

No change to generated code.

---
v12: no change
v11: drop comment churn
v10: no change
v9: improve commit message, add comments, tweak how alternates check tag_member
v8: new patch

---
 scripts/qapi-types.py                  |  5 ++++-
 scripts/qapi-visit.py                  |  5 ++++-
 scripts/qapi.py                        | 15 ++++++++++-----
 tests/qapi-schema/qapi-schema-test.out |  2 ++
 tests/qapi-schema/union-clash-data.out |  1 +
 tests/qapi-schema/union-empty.out      |  1 +
 6 files changed, 22 insertions(+), 7 deletions(-)
scripts/qapi-types.py +4 -1
index b37900f..946afab 100644
@@ -269,7 +269,10 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def visit_object_type(self, name, info, base, members, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
         if variants:
-            assert not members      # not implemented
+            if members:
+                # Members other than variants.tag_member not implemented
+                assert len(members) == 1
+                assert members[0] == variants.tag_member
             self.decl += gen_union(name, base, variants)
         else:
             self.decl += gen_struct(name, base, members)
scripts/qapi-visit.py +4 -1
index 3ef5c16..94cd113 100644
@@ -364,7 +364,10 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
     def visit_object_type(self, name, info, base, members, variants):
         self.decl += gen_visit_decl(name)
         if variants:
-            assert not members      # not implemented
+            if members:
+                # Members other than variants.tag_member not implemented
+                assert len(members) == 1
+                assert members[0] == variants.tag_member
             self.defn += gen_visit_union(name, base, variants)
         else:
             self.defn += gen_visit_struct(name, base, members)
scripts/qapi.py +10 -5
index 7c50cc4..687d9dc 100644
@@ -957,6 +957,9 @@ class QAPISchemaArrayType(QAPISchemaType):
 
 class QAPISchemaObjectType(QAPISchemaType):
     def __init__(self, name, info, base, local_members, variants):
+        # struct has local_members, optional base, and no variants
+        # flat union has base, variants, and no local_members
+        # simple union has local_members, variants, and no base
         QAPISchemaType.__init__(self, name, info)
         assert base is None or isinstance(base, str)
         for m in local_members:
@@ -1048,10 +1051,10 @@ class QAPISchemaObjectTypeVariants(object):
         self.variants = variants
 
     def check(self, schema, members, seen):
-        if self.tag_name:
+        if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
-        else:
-            self.tag_member.check(schema, members, seen)
+        if seen:
+            assert self.tag_member in seen.itervalues()
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
             vseen = dict(seen)
@@ -1085,6 +1088,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
 
     def check(self, schema):
+        self.variants.tag_member.check(schema, [], {})
         self.variants.check(schema, [], {})
 
     def json_type(self):
@@ -1270,13 +1274,14 @@ class QAPISchema(object):
         if tag_name:
             variants = [self._make_variant(key, value)
                         for (key, value) in data.iteritems()]
+            members = []
         else:
             variants = [self._make_simple_variant(key, value, info)
                         for (key, value) in data.iteritems()]
             tag_member = self._make_implicit_tag(name, info, variants)
+            members = [tag_member]
         self._def_entity(
-            QAPISchemaObjectType(name, info, base,
-                                 self._make_members(OrderedDict(), info),
+            QAPISchemaObjectType(name, info, base, members,
                                  QAPISchemaObjectTypeVariants(tag_name,
                                                               tag_member,
                                                               variants)))
tests/qapi-schema/qapi-schema-test.out +2 -0
index e20a823..786024e 100644
@@ -132,6 +132,7 @@ object UserDefFlatUnion2
     case value2: UserDefB
     case value3: UserDefA
 object UserDefNativeListUnion
+    member type: UserDefNativeListUnionKind optional=False
     case integer: :obj-intList-wrapper
     case s8: :obj-int8List-wrapper
     case s16: :obj-int16List-wrapper
@@ -187,6 +188,7 @@ object __org.qemu_x-Struct
 object __org.qemu_x-Struct2
     member array: __org.qemu_x-Union1List optional=False
 object __org.qemu_x-Union1
+    member type: __org.qemu_x-Union1Kind optional=False
     case __org.qemu_x-branch: :obj-str-wrapper
 enum __org.qemu_x-Union1Kind ['__org.qemu_x-branch']
 object __org.qemu_x-Union2
tests/qapi-schema/union-clash-data.out +1 -0
index 6277239..cea8551 100644
@@ -2,5 +2,6 @@ object :empty
 object :obj-int-wrapper
     member data: int optional=False
 object TestUnion
+    member type: TestUnionKind optional=False
     case data: :obj-int-wrapper
 enum TestUnionKind ['data']
tests/qapi-schema/union-empty.out +1 -0
index 8b5a7bf..9c89fd1 100644
@@ -1,3 +1,4 @@
 object :empty
 object Union
+    member type: UnionKind optional=False
 enum UnionKind []
Signature · 1 line
2.4.3
author
#2 lore raw

PATCH v12 2/36 qapi-types: Consolidate gen_struct() and gen_union()

In reply to Eric Blake

These two methods are now close enough that we can finally merge them, relying on the fact that simple unions now provide a reasonable local_members. Change gen_struct() to gen_object() that handles all forms of QAPISchemaObjectType, and rename and shrink gen_union() to gen_variants() to handle the portion of gen_object() needed when variants are present.

gen_struct_fields() now has a single caller, so it no longer needs an optional parameter; however, I did not choose to inline it into the caller.

No difference to generated code.

---
v12: no change
v11: no change
v10: no change
v9: rebase to earlier changes
v8: new patch

---
 scripts/qapi-types.py | 37 +++++++++++--------------------------
 1 file changed, 11 insertions(+), 26 deletions(-)
scripts/qapi-types.py +11 -26
index 946afab..403768b 100644
@@ -51,7 +51,7 @@ def gen_struct_field(member):
     return ret
 
 
-def gen_struct_fields(local_members, base=None):
+def gen_struct_fields(local_members, base):
     ret = ''
 
     if base:
@@ -70,7 +70,7 @@ def gen_struct_fields(local_members, base=None):
     return ret
 
 
-def gen_struct(name, base, members):
+def gen_object(name, base, members, variants):
     ret = mcgen('''
 
 struct %(c_name)s {
@@ -79,11 +79,14 @@ struct %(c_name)s {
 
     ret += gen_struct_fields(members, base)
 
+    if variants:
+        ret += gen_variants(variants)
+
     # Make sure that all structs have at least one field; this avoids
     # potential issues with attempting to malloc space for zero-length
     # structs in C, and also incompatibility with C++ (where an empty
     # struct is size 1).
-    if not (base and base.members) and not members:
+    if not (base and base.members) and not members and not variants:
         ret += mcgen('''
     char qapi_dummy_field_for_empty_struct;
 ''')
@@ -140,17 +143,7 @@ const int %(c_name)s_qtypes[QTYPE_MAX] = {
     return ret
 
 
-def gen_union(name, base, variants):
-    ret = mcgen('''
-
-struct %(c_name)s {
-''',
-                c_name=c_name(name))
-    if base:
-        ret += gen_struct_fields([], base)
-    else:
-        ret += gen_struct_field(variants.tag_member)
-
+def gen_variants(variants):
     # FIXME: What purpose does data serve, besides preventing a union that
     # has a branch named 'data'? We use it in qapi-visit.py to decide
     # whether to bypass the switch statement if visiting the discriminator
@@ -159,11 +152,11 @@ struct %(c_name)s {
     # should not be any data leaks even without a data pointer.  Or, if
     # 'data' is merely added to guarantee we don't have an empty union,
     # shouldn't we enforce that at .json parse time?
-    ret += mcgen('''
+    ret = mcgen('''
     union { /* union tag is @%(c_name)s */
         void *data;
 ''',
-                 c_name=c_name(variants.tag_member.name))
+                c_name=c_name(variants.tag_member.name))
 
     for var in variants.variants:
         # Ugly special case for simple union TODO get rid of it
@@ -176,7 +169,6 @@ struct %(c_name)s {
 
     ret += mcgen('''
     } u;
-};
 ''')
 
     return ret
@@ -268,14 +260,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
     def visit_object_type(self, name, info, base, members, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
-        if variants:
-            if members:
-                # Members other than variants.tag_member not implemented
-                assert len(members) == 1
-                assert members[0] == variants.tag_member
-            self.decl += gen_union(name, base, variants)
-        else:
-            self.decl += gen_struct(name, base, members)
+        self.decl += gen_object(name, base, members, variants)
         if base:
             self.decl += gen_upcast(name, base)
         self._gen_type_cleanup(name)
@@ -283,7 +268,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
     def visit_alternate_type(self, name, info, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
         self._fwdefn += gen_alternate_qtypes(name, variants)
-        self.decl += gen_union(name, None, variants)
+        self.decl += gen_object(name, None, [variants.tag_member], variants)
         self.decl += gen_alternate_qtypes_decl(name)
         self._gen_type_cleanup(name)
 
Signature · 1 line
2.4.3
author
#3 lore raw

PATCH v12 3/36 qapi-types: Simplify gen_struct_field[s]

In reply to Eric Blake

Simplify gen_struct_fields() back to a single iteration over a list of fields (like it was prior to commit f87ab7f9), by moving the generated comments to gen_object(). Then, inline gen_struct_field() into its only caller.

---
v12: no change
v11: no change
v10: no change
v9: new patch

---
 scripts/qapi-types.py | 40 +++++++++++++++-------------------------
 1 file changed, 15 insertions(+), 25 deletions(-)
scripts/qapi-types.py +15 -25
index 403768b..2f2f7df 100644
@@ -36,48 +36,38 @@ struct %(c_name)s {
                  c_name=c_name(name), c_type=element_type.c_type())
 
 
-def gen_struct_field(member):
+def gen_struct_fields(members):
     ret = ''
-
-    if member.optional:
-        ret += mcgen('''
+    for memb in members:
+        if memb.optional:
+            ret += mcgen('''
     bool has_%(c_name)s;
 ''',
-                     c_name=c_name(member.name))
-    ret += mcgen('''
+                         c_name=c_name(memb.name))
+        ret += mcgen('''
     %(c_type)s %(c_name)s;
 ''',
-                 c_type=member.type.c_type(), c_name=c_name(member.name))
+                     c_type=memb.type.c_type(), c_name=c_name(memb.name))
     return ret
 
 
-def gen_struct_fields(local_members, base):
-    ret = ''
+def gen_object(name, base, members, variants):
+    ret = mcgen('''
+
+struct %(c_name)s {
+''',
+                c_name=c_name(name))
 
     if base:
         ret += mcgen('''
     /* Members inherited from %(c_name)s: */
 ''',
                      c_name=base.c_name())
-        for memb in base.members:
-            ret += gen_struct_field(memb)
+        ret += gen_struct_fields(base.members)
         ret += mcgen('''
     /* Own members: */
 ''')
-
-    for memb in local_members:
-        ret += gen_struct_field(memb)
-    return ret
-
-
-def gen_object(name, base, members, variants):
-    ret = mcgen('''
-
-struct %(c_name)s {
-''',
-                c_name=c_name(name))
-
-    ret += gen_struct_fields(members, base)
+    ret += gen_struct_fields(members)
 
     if variants:
         ret += gen_variants(variants)
Signature · 1 line
2.4.3
author
#4 lore raw

PATCH v12 4/36 qapi: Drop obsolete tag value collision assertions

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

Union tag values can't clash with member names in generated C anymore since commit e4ba22b, but QAPISchemaObjectTypeVariants.check() still asserts they don't. Drop it.

Message-Id: <[email protected]>

---
v12: no change
v11: no change
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/qapi.py +1 -2
index 687d9dc..29377d6 100644
@@ -1057,8 +1057,7 @@ class QAPISchemaObjectTypeVariants(object):
             assert self.tag_member in seen.itervalues()
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
-            vseen = dict(seen)
-            v.check(schema, self.tag_member.type, vseen)
+            v.check(schema, self.tag_member.type, {})
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Signature · 1 line
2.4.3
author
#5 lore raw

PATCH v12 5/36 qapi: Simplify QAPISchemaObjectTypeMember.check()

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

QAPISchemaObjectTypeMember.check() currently does four things:

1. Compute self.type

2. Accumulate members in all_members

Only one caller cares: QAPISchemaObjectType.check() uses it to compute self.members. The other callers pass a throw-away accumulator.

3. Accumulate a map from names to members in seen

Only one caller cares: QAPISchemaObjectType.check() uses it to compute its local variable seen, for self.variants.check(), which uses it to compute self.variants.tag_member from self.variants.tag_name. The other callers pass a throw-away accumulator.

4. Check for collisions

This piggybacks on 3: before adding a new entry, we assert it's new.

Only one caller cares: QAPISchemaObjectType.check() uses it to assert non-variant members don't clash.

Simplify QAPISchemaObjectType.check(): move 2.-4. to QAPISchemaObjectType.check(), and drop parameters all_members and seen.

Message-Id: <[email protected]> [rebase to earlier changes that moved tag_member.check() of alternate types, commit message typo fix]

---
v12: no change
v11: typo fix
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
scripts/qapi.py +7 -7
index 29377d6..63d39e4 100644
@@ -990,7 +990,10 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert c_name(m.name) not in seen
             seen[m.name] = m
         for m in self.local_members:
-            m.check(schema, members, seen)
+            m.check(schema)
+            assert m.name not in seen
+            seen[m.name] = m
+            members.append(m)
         if self.variants:
             self.variants.check(schema, members, seen)
         self.members = members
@@ -1027,12 +1030,9 @@ class QAPISchemaObjectTypeMember(object):
         self.type = None
         self.optional = optional
 
-    def check(self, schema, all_members, seen):
-        assert self.name not in seen
+    def check(self, schema):
         self.type = schema.lookup_type(self._type_name)
         assert self.type
-        all_members.append(self)
-        seen[self.name] = self
 
 
 class QAPISchemaObjectTypeVariants(object):
@@ -1065,7 +1065,7 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
     def check(self, schema, tag_type, seen):
-        QAPISchemaObjectTypeMember.check(self, schema, [], seen)
+        QAPISchemaObjectTypeMember.check(self, schema)
         assert self.name in tag_type.values
 
     # This function exists to support ugly simple union special cases
@@ -1087,7 +1087,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
         self.variants = variants
 
     def check(self, schema):
-        self.variants.tag_member.check(schema, [], {})
+        self.variants.tag_member.check(schema)
         self.variants.check(schema, [], {})
 
     def json_type(self):
Signature · 1 line
2.4.3
author
#6 lore raw

PATCH v12 6/36 qapi: Clean up after previous commit

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

QAPISchemaObjectTypeVariants.check() parameter members and QAPISchemaObjectTypeVariant.check() parameter seen are no longer used, drop them.

Message-Id: <[email protected]> [rebase to earlier changes that moved tag_member.check() of alternate types]

---
v12: fix git authorship
v11: rebase for less comment churn
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
scripts/qapi.py +5 -5
index 63d39e4..0bf8235 100644
@@ -995,7 +995,7 @@ class QAPISchemaObjectType(QAPISchemaType):
             seen[m.name] = m
             members.append(m)
         if self.variants:
-            self.variants.check(schema, members, seen)
+            self.variants.check(schema, seen)
         self.members = members
 
     def is_implicit(self):
@@ -1050,21 +1050,21 @@ class QAPISchemaObjectTypeVariants(object):
         self.tag_member = tag_member
         self.variants = variants
 
-    def check(self, schema, members, seen):
+    def check(self, schema, seen):
         if self.tag_name:    # flat union
             self.tag_member = seen[self.tag_name]
         if seen:
             assert self.tag_member in seen.itervalues()
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
-            v.check(schema, self.tag_member.type, {})
+            v.check(schema, self.tag_member.type)
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
-    def check(self, schema, tag_type, seen):
+    def check(self, schema, tag_type):
         QAPISchemaObjectTypeMember.check(self, schema)
         assert self.name in tag_type.values
 
@@ -1088,7 +1088,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
     def check(self, schema):
         self.variants.tag_member.check(schema)
-        self.variants.check(schema, [], {})
+        self.variants.check(schema, {})
 
     def json_type(self):
         return 'value'
Signature · 1 line
2.4.3
author
#7 lore raw

PATCH v12 7/36 qapi: Fix up commit 7618b91's clash sanity checking change

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

This hunk

@@ -964,6 +965,7 @@ class QAPISchemaObjectType(QAPISchemaType):
members = []
seen = {}
for m in members:
+ assert c_name(m.name) not in seen
seen[m.name] = m
for m in self.local_members:
m.check(schema, members, seen)

is plainly broken.

Asserting the members inherited from base don't clash is somewhat redundant, because self.base.check() just checked that. But it doesn't hurt.

The idea to use c_name(m.name) instead of m.name for collision checking is sound, because we need to catch clashes between the m.name and between the c_name(m.name), and when two m.name clash, then their c_name() also clash.

However, using c_name(m.name) instead of m.name in one of several places doesn't work. See the very next line.

Keep the assertion, but drop the c_name() for now. A future commit will bring it back.

Message-Id: <[email protected]>

[change TABs in commit message to space]

---
v12: no change
v11: no change
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/qapi.py +1 -1
index 0bf8235..86d2adc 100644
@@ -987,7 +987,7 @@ class QAPISchemaObjectType(QAPISchemaType):
             members = []
         seen = {}
         for m in members:
-            assert c_name(m.name) not in seen
+            assert m.name not in seen
             seen[m.name] = m
         for m in self.local_members:
             m.check(schema)
Signature · 1 line
2.4.3
author
#8 lore raw

PATCH v12 8/36 qapi: Eliminate QAPISchemaObjectType.check() variable members

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

We can use seen.values() instead if we make it an OrderedDict.

Message-Id: <[email protected]>

---
v12: no change
v11: no change
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)
scripts/qapi.py +5 -9
index 86d2adc..44d08c1 100644
@@ -977,26 +977,22 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self.members:
             return
         self.members = False                    # mark as being checked
+        seen = OrderedDict()
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
             assert isinstance(self.base, QAPISchemaObjectType)
             assert not self.base.variants       # not implemented
             self.base.check(schema)
-            members = list(self.base.members)
-        else:
-            members = []
-        seen = {}
-        for m in members:
-            assert m.name not in seen
-            seen[m.name] = m
+            for m in self.base.members:
+                assert m.name not in seen
+                seen[m.name] = m
         for m in self.local_members:
             m.check(schema)
             assert m.name not in seen
             seen[m.name] = m
-            members.append(m)
         if self.variants:
             self.variants.check(schema, seen)
-        self.members = members
+        self.members = seen.values()
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
Signature · 1 line
2.4.3
author
#9 lore raw

PATCH v12 9/36 qapi: Factor out QAPISchemaObjectTypeMember.check_clash()

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

While there, stick in a TODO change key of seen from QAPI name to C name. Can't do it right away, because it would fail the assertion for tests/qapi-schema/args-has-clash.json.

Message-Id: <[email protected]>

---
v12: no change
v11: no change
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
scripts/qapi.py +7 -4
index 44d08c1..2a73b2b 100644
@@ -984,12 +984,10 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert not self.base.variants       # not implemented
             self.base.check(schema)
             for m in self.base.members:
-                assert m.name not in seen
-                seen[m.name] = m
+                m.check_clash(seen)
         for m in self.local_members:
             m.check(schema)
-            assert m.name not in seen
-            seen[m.name] = m
+            m.check_clash(seen)
         if self.variants:
             self.variants.check(schema, seen)
         self.members = seen.values()
@@ -1030,6 +1028,11 @@ class QAPISchemaObjectTypeMember(object):
         self.type = schema.lookup_type(self._type_name)
         assert self.type
 
+    def check_clash(self, seen):
+        # TODO change key of seen from QAPI name to C name
+        assert self.name not in seen
+        seen[self.name] = self
+
 
 class QAPISchemaObjectTypeVariants(object):
     def __init__(self, tag_name, tag_member, variants):
Signature · 1 line
2.4.3
author
#10 lore raw

PATCH v12 10/36 qapi: Simplify QAPISchemaObjectTypeVariants.check()

In reply to Eric Blake

From: Markus Armbruster <[email protected]>

Reduce the ugly flat union / simple union conditional by doing just the essential work here, namely setting self.tag_member. Move the rest to callers.

Message-Id: <[email protected]> [rebase to earlier changes that moved tag_member.check() of alternate types, and tweak commit title and wording]

---
v12: no change
v11: don't drop change of 'if not self.tag_member'
v10: redo closer to Markus' original proposal
v9: new patch

---
 scripts/qapi.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
scripts/qapi.py +3 -4
index 2a73b2b..c6cb17b 100644
@@ -988,9 +988,10 @@ class QAPISchemaObjectType(QAPISchemaType):
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)
+        self.members = seen.values()
         if self.variants:
             self.variants.check(schema, seen)
-        self.members = seen.values()
+            assert self.variants.tag_member in self.members
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
@@ -1050,10 +1051,8 @@ class QAPISchemaObjectTypeVariants(object):
         self.variants = variants
 
     def check(self, schema, seen):
-        if self.tag_name:    # flat union
+        if not self.tag_member:    # flat union
             self.tag_member = seen[self.tag_name]
-        if seen:
-            assert self.tag_member in seen.itervalues()
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
             v.check(schema, self.tag_member.type)
Signature · 1 line
2.4.3
author
#11 lore raw

PATCH v12 11/36 qapi: Check for qapi collisions involving variant members

In reply to Eric Blake

Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any members that would clash with non-variant members inherited from the union's base type (see flat-union-clash-member.json). We want ObjectType.check() to make the same check, so we can later reduce some of the ad hoc checks.

We already have a map 'seen' of all non-variant members. We still need to check for collisions between each variant type's members and the non-variant ones.

To know the variant type's members, we need to call variant.type.check(). This also detects when a type contains itself in a variant, exactly like the existing base.check() detects when a type contains itself as a base. (Except that we currently forbid anything but a struct as the type of a variant, so we can't actually trigger this type of loop yet.)

Slight complication: an alternate's variant can have arbitrary type, but only an object type's check() may be called outside QAPISchema.check(). We could either skip the call for variants of alternates, or skip it for non-object types. For now, do the latter, because it's easier.

Then we call each variant member's check_clash() with the appropriate 'seen' map. Since members of different variants can't clash, we have to clone a fresh seen for each variant. Wrap this in a new helper method Variants.check_clash().

Note that cloning 'seen' inside Variants.check_clash() resembles the one we just removed from Variants.check() in 'qapi: Drop obsolete tag value collision assertions'; the difference here is that we are now checking for clashes among the qapi members of the variant type, rather than for a single clash with the variant tag name itself.

Note that, by construction, collisions can't actually happen for simple unions: each variant's type is a wrapper with a single member 'data', which will never collide with the only non-variant member 'type'.

For alternates, there's nothing for a variant object type's members to clash with, and therefore no need to call the new variants.check_clash().

No change to generated code.

---
v12: retitle; use Markus' improved commit wording
v11: keep type.check() in check(), add comment to alternate
v10: create new Variants.check_clash() rather than piggybacking on .check()
v9: new patch, split off from v8 7/17

---
 scripts/qapi.py | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
scripts/qapi.py +15 -0
index c6cb17b..b2d071f 100644
@@ -992,6 +992,7 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self.variants:
             self.variants.check(schema, seen)
             assert self.variants.tag_member in self.members
+            self.variants.check_clash(schema, seen)
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
@@ -1056,6 +1057,18 @@ class QAPISchemaObjectTypeVariants(object):
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
             v.check(schema, self.tag_member.type)
+            if isinstance(v.type, QAPISchemaObjectType):
+                v.type.check(schema)
+
+    def check_clash(self, schema, seen):
+        for v in self.variants:
+            # Reset seen map for each variant, since qapi names from one
+            # branch do not affect another branch
+            vseen = dict(seen)
+            assert isinstance(v.type, QAPISchemaObjectType)
+            assert not v.type.variants       # not implemented
+            for m in v.type.members:
+                m.check_clash(vseen)
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
@@ -1086,6 +1099,8 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
     def check(self, schema):
         self.variants.tag_member.check(schema)
+        # Not calling self.variants.check_clash(), because there's nothing
+        # to clash with
         self.variants.check(schema, {})
 
     def json_type(self):
Signature · 1 line
2.4.3
#12 lore raw

Re: [Qemu-devel] [PATCH v12 11/36] qapi: Check for qapi collisions involving variant members

Quoted 4 lines · Eric Blake

Right now, our ad hoc parser ensures that we cannot have a flat union that introduces any members that would clash with non-variant members inherited from the union's base type (see flat-union-clash-member.json). We want ObjectType.check() to

Recommend not to abbreviate QAPISchemaObjectType.check().

Quoted 20 lines

make the same check, so we can later reduce some of the ad hoc checks.

We already have a map 'seen' of all non-variant members. We still need to check for collisions between each variant type's members and the non-variant ones.

To know the variant type's members, we need to call variant.type.check(). This also detects when a type contains itself in a variant, exactly like the existing base.check() detects when a type contains itself as a base. (Except that we currently forbid anything but a struct as the type of a variant, so we can't actually trigger this type of loop yet.)

Slight complication: an alternate's variant can have arbitrary type, but only an object type's check() may be called outside QAPISchema.check(). We could either skip the call for variants of alternates, or skip it for non-object types. For now, do the latter, because it's easier.

Quoted 4 lines

Then we call each variant member's check_clash() with the appropriate 'seen' map. Since members of different variants can't clash, we have to clone a fresh seen for each variant. Wrap this in a new helper method Variants.check_clash().

Likewise: QAPISchemaObjectTypeVariants.check_clash().

Quoted 2 lines

Note that cloning 'seen' inside Variants.check_clash() resembles the one we just removed from Variants.check() in 'qapi: Drop

Here, we can say .check_clash() and .check(), and leave the class to context.

Quoted 10 lines

obsolete tag value collision assertions'; the difference here is that we are now checking for clashes among the qapi members of the variant type, rather than for a single clash with the variant tag name itself.

Note that, by construction, collisions can't actually happen for simple unions: each variant's type is a wrapper with a single member 'data', which will never collide with the only non-variant member 'type'.

Quoted 7 lines

For alternates, there's nothing for a variant object type's members to clash with, and therefore no need to call the new variants.check_clash().

No change to generated code.

Only commit message polishing, happy to do that in my tree.

author
#13 lore raw

PATCH v12 12/36 qapi: Factor out QAPISchemaObjectType.check_clash()

In reply to Eric Blake

Consolidate two common sequences of clash detection into a new QAPISchemaObjectType.check_clash() helper method.

No change to generated code.

---
v12: no change
v11: don't lose isinstance check, and don't hide type.check() inside check_clash()
v10: rebase on new Variants.check_clash()
v9: new patch, split off from v8 7/17

---
 scripts/qapi.py | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
scripts/qapi.py +7 -7
index b2d071f..296b9bb 100644
@@ -981,10 +981,8 @@ class QAPISchemaObjectType(QAPISchemaType):
         if self._base_name:
             self.base = schema.lookup_type(self._base_name)
             assert isinstance(self.base, QAPISchemaObjectType)
-            assert not self.base.variants       # not implemented
             self.base.check(schema)
-            for m in self.base.members:
-                m.check_clash(seen)
+            self.base.check_clash(schema, seen)
         for m in self.local_members:
             m.check(schema)
             m.check_clash(seen)
@@ -994,6 +992,11 @@ class QAPISchemaObjectType(QAPISchemaType):
             assert self.variants.tag_member in self.members
             self.variants.check_clash(schema, seen)
 
+    def check_clash(self, schema, seen):
+        assert not self.variants       # not implemented
+        for m in self.members:
+            m.check_clash(seen)
+
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
         return self.name[0] == ':'
@@ -1064,11 +1067,8 @@ class QAPISchemaObjectTypeVariants(object):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
-            vseen = dict(seen)
             assert isinstance(v.type, QAPISchemaObjectType)
-            assert not v.type.variants       # not implemented
-            for m in v.type.members:
-                m.check_clash(vseen)
+            v.type.check_clash(schema, dict(seen))
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
Signature · 1 line
2.4.3
author
#14 lore raw

PATCH v12 13/36 qapi: Hoist tag collision check to Variants.check()

In reply to Eric Blake

Checking that a given QAPISchemaObjectTypeVariant.name is a member of the corresponding QAPISchemaEnumType of the owning QAPISchemaObjectTypeVariants.tag_member ensures that there are no collisions in the generated C union for those tag values (since the enum itself should have no collisions).

However, ever since its introduction in f51d8c3d, this was the only additional action of of Variant.check(), beyond calling the superclass Member.check(). This forces a difference in .check() signatures, just to pass the enum type down.

Simplify things by instead doing the tag name check as part of Variants.check(), at which point we can rely on inheritance instead of overriding Variant.check().

---
v12: improve commit message
v11: don't use tag_type local variable, rebase to v.type.check()
v10: new patch

---
 scripts/qapi.py | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
scripts/qapi.py +2 -5
index 296b9bb..c6f3fce 100644
@@ -1059,7 +1059,8 @@ class QAPISchemaObjectTypeVariants(object):
             self.tag_member = seen[self.tag_name]
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
-            v.check(schema, self.tag_member.type)
+            v.check(schema)
+            assert v.name in self.tag_member.type.values
             if isinstance(v.type, QAPISchemaObjectType):
                 v.type.check(schema)
 
@@ -1075,10 +1076,6 @@ class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
-    def check(self, schema, tag_type):
-        QAPISchemaObjectTypeMember.check(self, schema)
-        assert self.name in tag_type.values
-
     # This function exists to support ugly simple union special cases
     # TODO get rid of them, and drop the function
     def simple_union_type(self):
Signature · 1 line
2.4.3
#15 lore raw

Re: [Qemu-devel] [PATCH v12 13/36] qapi: Hoist tag collision check to Variants.check()

Quoted 4 lines · Eric Blake

*** WARNING: THE ATTACHED DOCUMENT(S) CONTAIN MACROS ***
*** MACROS MAY CONTAIN MALICIOUS CODE ***
*** Open only if you can verify and trust the sender ***
*** Please contact [email protected] if you have questions or concerns **

Looks like infosec crapped over your commit message.

Quoted 11 lines

Checking that a given QAPISchemaObjectTypeVariant.name is a member of the corresponding QAPISchemaEnumType of the owning QAPISchemaObjectTypeVariants.tag_member ensures that there are no collisions in the generated C union for those tag values (since the enum itself should have no collisions).

However, ever since its introduction in f51d8c3d, this was the only additional action of of Variant.check(), beyond calling the superclass Member.check(). This forces a difference in .check() signatures, just to pass the enum type down.

Quoted 5 lines

Simplify things by instead doing the tag name check as part of Variants.check(), at which point we can rely on inheritance instead of overriding Variant.check().

Can bury the infosec turd on merge.

#16 lore raw

Re: [Qemu-devel] [PATCH v12 13/36] qapi: Hoist tag collision check to Variants.check()

Quoted 8 lines · Markus Armbruster
Quoted 4 lines · Eric Blake

*** WARNING: THE ATTACHED DOCUMENT(S) CONTAIN MACROS ***
*** MACROS MAY CONTAIN MALICIOUS CODE ***
*** Open only if you can verify and trust the sender ***
*** Please contact [email protected] if you have questions or concerns **

Looks like infosec crapped over your commit message.

I don't see that text in my copy of Eric's mail, nor in the archives

https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg04028.html

Wierd that it would get added to the mail you receive, but not to the mail I receive, given that we have the same incoming mail server. I guess the difference was you got CC'd directly but I got it via the mailing list, so something about that delivery path caused it not to trigger for me.

Regards,
Daniel

Signature · 4 lines
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
author
#17 lore raw

PATCH v12 14/36 qapi: Remove outdated tests related to QMP/branch collisions

In reply to Eric Blake

Now that branches are in a separate C namespace, we can remove the restrictions in the parser that claim a branch name would collide with QMP, and delete the negative tests that are no longer problematic. A separate patch can then add positive tests to qapi-schema-test to test that any corner cases will compile correctly.

This reverts the scripts/qapi.py portion of commit 7b2a5c2, now that the assertions that it plugged are no longer possible.

---
v12: no change
v11: no change
v10: no change
v9: no change
v8: retitle and update commit message, finish reversion of 7b2a5c2
v7: new patch (also temporarily appeared in subset B v10)

---
 scripts/qapi.py                                | 11 ++---------
 tests/Makefile                                 |  3 ---
 tests/qapi-schema/flat-union-clash-branch.err  |  0
 tests/qapi-schema/flat-union-clash-branch.exit |  1 -
 tests/qapi-schema/flat-union-clash-branch.json | 18 ------------------
 tests/qapi-schema/flat-union-clash-branch.out  | 14 --------------
 tests/qapi-schema/flat-union-clash-type.err    |  1 -
 tests/qapi-schema/flat-union-clash-type.exit   |  1 -
 tests/qapi-schema/flat-union-clash-type.json   | 14 --------------
 tests/qapi-schema/flat-union-clash-type.out    |  0
 tests/qapi-schema/union-clash-type.err         |  1 -
 tests/qapi-schema/union-clash-type.exit        |  1 -
 tests/qapi-schema/union-clash-type.json        |  9 ---------
 tests/qapi-schema/union-clash-type.out         |  0
 14 files changed, 2 insertions(+), 72 deletions(-)
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.exit
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.json
 delete mode 100644 tests/qapi-schema/flat-union-clash-branch.out
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.err
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.exit
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.json
 delete mode 100644 tests/qapi-schema/flat-union-clash-type.out
 delete mode 100644 tests/qapi-schema/union-clash-type.err
 delete mode 100644 tests/qapi-schema/union-clash-type.exit
 delete mode 100644 tests/qapi-schema/union-clash-type.json
 delete mode 100644 tests/qapi-schema/union-clash-type.out
scripts/qapi.py +2 -9
index c6f3fce..6fc14be 100644
@@ -548,8 +548,7 @@ def check_union(expr, expr_info):
     base = expr.get('base')
     discriminator = expr.get('discriminator')
     members = expr['data']
-    values = {'MAX': '(automatic)', 'KIND': '(automatic)',
-              'TYPE': '(automatic)'}
+    values = {'MAX': '(automatic)'}
 
     # Two types of unions, determined by discriminator.
 
@@ -607,19 +606,13 @@ def check_union(expr, expr_info):
                                " of branch '%s'" % key)
 
         # If the discriminator names an enum type, then all members
-        # of 'data' must also be members of the enum type, which in turn
-        # must not collide with the discriminator name.
+        # of 'data' must also be members of the enum type.
         if enum_define:
             if key not in enum_define['enum_values']:
                 raise QAPIExprError(expr_info,
                                     "Discriminator value '%s' is not found in "
                                     "enum '%s'" %
                                     (key, enum_define["enum_name"]))
-            if discriminator in enum_define['enum_values']:
-                raise QAPIExprError(expr_info,
-                                    "Discriminator name '%s' collides with "
-                                    "enum value in '%s'" %
-                                    (discriminator, enum_define["enum_name"]))
 
         # Otherwise, check for conflicts in the generated enum
         else:
tests/Makefile +0 -3
index 90c4141..1c2c8ee 100644
@@ -280,9 +280,7 @@ qapi-schema += flat-union-bad-base.json
 qapi-schema += flat-union-bad-discriminator.json
 qapi-schema += flat-union-base-any.json
 qapi-schema += flat-union-base-union.json
-qapi-schema += flat-union-clash-branch.json
 qapi-schema += flat-union-clash-member.json
-qapi-schema += flat-union-clash-type.json
 qapi-schema += flat-union-empty.json
 qapi-schema += flat-union-inline.json
 qapi-schema += flat-union-int-branch.json
@@ -344,7 +342,6 @@ qapi-schema += union-bad-branch.json
 qapi-schema += union-base-no-discriminator.json
 qapi-schema += union-clash-branches.json
 qapi-schema += union-clash-data.json
-qapi-schema += union-clash-type.json
 qapi-schema += union-empty.json
 qapi-schema += union-invalid-base.json
 qapi-schema += union-max.json
tests/qapi-schema/flat-union-clash-branch.err deleted +0 -0
deleted file mode 100644
index e69de29..0000000
tests/qapi-schema/flat-union-clash-branch.exit deleted +0 -1
deleted file mode 100644
index 573541a..0000000
@@ -1 +0,0 @@
-0
tests/qapi-schema/flat-union-clash-branch.json deleted +0 -18
deleted file mode 100644
index e593336..0000000
@@ -1,18 +0,0 @@
-# Flat union branch name collision
-# FIXME: this parses, but then fails to compile due to a duplicate 'c_d'
-# (one from the base member, the other from the branch name).  We should
-# either reject the collision at parse time, or munge the generated branch
-# name to allow this to compile.
-{ 'enum': 'TestEnum',
-  'data': [ 'base', 'c-d' ] }
-{ 'struct': 'Base',
-  'data': { 'enum1': 'TestEnum', '*c_d': 'str' } }
-{ 'struct': 'Branch1',
-  'data': { 'string': 'str' } }
-{ 'struct': 'Branch2',
-  'data': { 'value': 'int' } }
-{ 'union': 'TestUnion',
-  'base': 'Base',
-  'discriminator': 'enum1',
-  'data': { 'base': 'Branch1',
-            'c-d': 'Branch2' } }
tests/qapi-schema/flat-union-clash-branch.out deleted +0 -14
deleted file mode 100644
index 8e0da73..0000000
@@ -1,14 +0,0 @@
-object :empty
-object Base
-    member enum1: TestEnum optional=False
-    member c_d: str optional=True
-object Branch1
-    member string: str optional=False
-object Branch2
-    member value: int optional=False
-enum TestEnum ['base', 'c-d']
-object TestUnion
-    base Base
-    tag enum1
-    case base: Branch1
-    case c-d: Branch2
tests/qapi-schema/flat-union-clash-type.err deleted +0 -1
deleted file mode 100644
index b44dd40..0000000
@@ -1 +0,0 @@
-tests/qapi-schema/flat-union-clash-type.json:11: Discriminator name 'type' collides with enum value in 'TestEnum'
tests/qapi-schema/flat-union-clash-type.exit deleted +0 -1
deleted file mode 100644
index d00491f..0000000
@@ -1 +0,0 @@
-1
tests/qapi-schema/flat-union-clash-type.json deleted +0 -14
deleted file mode 100644
index 8f710f0..0000000
@@ -1,14 +0,0 @@
-# Flat union branch 'type'
-# Reject this, because we would have a clash in generated C, between the
-# outer tag 'type' and the branch name 'type' within the union.
-# TODO: We could munge the generated C branch name to let it compile.
-{ 'enum': 'TestEnum',
-  'data': [ 'type' ] }
-{ 'struct': 'Base',
-  'data': { 'type': 'TestEnum' } }
-{ 'struct': 'Branch1',
-  'data': { 'string': 'str' } }
-{ 'union': 'TestUnion',
-  'base': 'Base',
-  'discriminator': 'type',
-  'data': { 'type': 'Branch1' } }
tests/qapi-schema/flat-union-clash-type.out deleted +0 -0
deleted file mode 100644
index e69de29..0000000
tests/qapi-schema/union-clash-type.err deleted +0 -1
deleted file mode 100644
index a5dead1..0000000
@@ -1 +0,0 @@
-tests/qapi-schema/union-clash-type.json:8: Union 'TestUnion' member 'kind' clashes with '(automatic)'
tests/qapi-schema/union-clash-type.exit deleted +0 -1
deleted file mode 100644
index d00491f..0000000
@@ -1 +0,0 @@
-1
tests/qapi-schema/union-clash-type.json deleted +0 -9
deleted file mode 100644
index cfc256b..0000000
@@ -1,9 +0,0 @@
-# Union branch 'type'
-# Reject this, because we would have a clash in generated C, between the
-# simple union's implicit tag member 'kind' and the branch name 'kind'
-# within the union.
-# TODO: Even when the generated C is switched to use 'type' rather than
-# 'kind', to match the QMP spelling, the collision should still be detected.
-# Or, we could munge the branch name to allow compilation.
-{ 'union': 'TestUnion',
-  'data': { 'kind': 'int', 'type': 'str' } }
tests/qapi-schema/union-clash-type.out deleted +0 -0
deleted file mode 100644
index e69de29..0000000
Signature · 1 line
2.4.3
author
#18 lore raw

PATCH v12 15/36 qapi: Track owner of each object member

In reply to Eric Blake

Future commits will migrate semantic checking away from parsing and over to the various QAPISchema*.check() methods. But to report an error message about an incorrect semantic use of a member of an object type, it helps to know which type, command, or event owns the member. In particular, when a member is inherited from a base type, it is desirable to associate the member name with the base type (and not the type calling member.check()).

Rather than packing additional information into the seen array passed to each member.check() (as in seen[m.name] = {'member':m, 'owner':type}), it is easier to have each member track the name of the owner type in the first place (keeping things simpler with the existing seen[m.name] = m). The new member.owner field is set via a new set_owner() method, called when registering the members and variants arrays with an object or variant type. Track only a name, and not the actual type object, to avoid creating a circular python reference chain.

Note that Variants.set_owner() method does not set the owner for the tag_member field; this field is set earlier either as part of an object's non-variant members, or explicitly by alternates.

The source information is intended for human consumption in error messages, and a new describe() method is added to access the resulting information. For example, given the qapi:
{ 'command': 'foo', 'data': { 'string': 'str' } }
an implementation of visit_command() that calls
arg_type.members[0].describe()
will see "'string' (parameter of foo)".

To make the human-readable name of implicit types work without duplicating efforts, the describe() method has to reverse the name of implicit types, via the helper _pretty_owner().

No change to generated code.

---
v12: no change
v11: set alternate tag_member owner during init, tweak comments, keep events with '-args' instead of '-data', prefer '(parameter of foo)'
v10 (now in subset C): rebase to latest, set alternate tag_member owner from alternate
v9 (now in subset D): rebase to earlier changes, hoist 'role' to top of class, split out _pretty_helper(), manage owner when tag_member appears as part of local_members for unions
v8: don't munge implicit type names [except for event data], and instead make describe() create nicer messages. Add set_owner(), and use field 'role' instead of method _describe()
v7: total rewrite: rework implicit object names, assign owner when initializing owner type rather than when creating member python object
v6: rebase on new lazy array creation and simple union 'type' motion; tweak commit message

---
 scripts/qapi.py | 39 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 37 insertions(+), 2 deletions(-)
scripts/qapi.py +37 -2
index 6fc14be..79038a8 100644
@@ -957,8 +957,10 @@ class QAPISchemaObjectType(QAPISchemaType):
         assert base is None or isinstance(base, str)
         for m in local_members:
             assert isinstance(m, QAPISchemaObjectTypeMember)
-        assert (variants is None or
-                isinstance(variants, QAPISchemaObjectTypeVariants))
+            m.set_owner(name)
+        if variants is not None:
+            assert isinstance(variants, QAPISchemaObjectTypeVariants)
+            variants.set_owner(name)
         self._base_name = base
         self.base = None
         self.local_members = local_members
@@ -1013,6 +1015,8 @@ class QAPISchemaObjectType(QAPISchemaType):
 
 
 class QAPISchemaObjectTypeMember(object):
+    role = 'member'
+
     def __init__(self, name, typ, optional):
         assert isinstance(name, str)
         assert isinstance(typ, str)
@@ -1021,8 +1025,14 @@ class QAPISchemaObjectTypeMember(object):
         self._type_name = typ
         self.type = None
         self.optional = optional
+        self.owner = None
+
+    def set_owner(self, name):
+        assert not self.owner
+        self.owner = name
 
     def check(self, schema):
+        assert self.owner
         self.type = schema.lookup_type(self._type_name)
         assert self.type
 
@@ -1031,6 +1041,22 @@ class QAPISchemaObjectTypeMember(object):
         assert self.name not in seen
         seen[self.name] = self
 
+    def _pretty_owner(self):
+        owner = self.owner
+        if owner.startswith(':obj-'):
+            # See QAPISchema._make_implicit_object_type() - reverse the
+            # mapping there to create a nice human-readable description
+            owner = owner[5:]
+            if owner.endswith('-arg'):
+                return '(parameter of %s)' % owner[:-4]
+            else:
+                assert owner.endswith('-wrapper')
+                return '(branch of %s)' % owner[:-8]
+        return '(%s of %s)' % (self.role, owner)
+
+    def describe(self):
+        return "'%s' %s" % (self.name, self._pretty_owner())
+
 
 class QAPISchemaObjectTypeVariants(object):
     def __init__(self, tag_name, tag_member, variants):
@@ -1047,6 +1073,10 @@ class QAPISchemaObjectTypeVariants(object):
         self.tag_member = tag_member
         self.variants = variants
 
+    def set_owner(self, name):
+        for v in self.variants:
+            v.set_owner(name)
+
     def check(self, schema, seen):
         if not self.tag_member:    # flat union
             self.tag_member = seen[self.tag_name]
@@ -1066,6 +1096,8 @@ class QAPISchemaObjectTypeVariants(object):
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
+    role = 'branch'
+
     def __init__(self, name, typ):
         QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
 
@@ -1085,6 +1117,8 @@ class QAPISchemaAlternateType(QAPISchemaType):
         QAPISchemaType.__init__(self, name, info)
         assert isinstance(variants, QAPISchemaObjectTypeVariants)
         assert not variants.tag_name
+        variants.set_owner(name)
+        variants.tag_member.set_owner(self.name)
         self.variants = variants
 
     def check(self, schema):
@@ -1217,6 +1251,7 @@ class QAPISchema(object):
     def _make_implicit_object_type(self, name, info, role, members):
         if not members:
             return None
+        # See also QAPISchemaObjectTypeMember._pretty_owner()
         name = ':obj-%s-%s' % (name, role)
         if not self.lookup_entity(name, QAPISchemaObjectType):
             self._def_entity(QAPISchemaObjectType(name, info, None,
Signature · 1 line
2.4.3
author
#19 lore raw

PATCH v12 16/36 qapi: Detect collisions in C member names

In reply to Eric Blake

Detect attempts to declare two object members that would result in the same C member name, by keying the 'seen' dictionary off of the C name rather than the qapi name. It also requires passing info through the check_clash() methods.

This addresses a TODO and fixes the previously-broken args-name-clash test. The resulting error message demonstrates the utility of the .describe() method added previously. No change to generated code.

---
v12: no change
v11: rebase to earlier changes, use 'cname' local variable, shorter message in test file
v10 (now in subset C): rebase to latest; update commit message v9 (now in subset D): rebase to earlier changes, now only one test affected
v8: rebase to earlier changes
v7: split out error reporting prep and member.c_name() addition
v6: rebase to earlier testsuite and info improvements

---
 scripts/qapi.py                        | 31 +++++++++++++++++++------------
 tests/qapi-schema/args-name-clash.err  |  1 +
 tests/qapi-schema/args-name-clash.exit |  2 +-
 tests/qapi-schema/args-name-clash.json |  5 ++---
 tests/qapi-schema/args-name-clash.out  |  6 ------
 5 files changed, 23 insertions(+), 22 deletions(-)
scripts/qapi.py +19 -12
index 79038a8..b068141 100644
@@ -977,20 +977,23 @@ class QAPISchemaObjectType(QAPISchemaType):
             self.base = schema.lookup_type(self._base_name)
             assert isinstance(self.base, QAPISchemaObjectType)
             self.base.check(schema)
-            self.base.check_clash(schema, seen)
+            self.base.check_clash(schema, self.info, seen)
         for m in self.local_members:
             m.check(schema)
-            m.check_clash(seen)
+            m.check_clash(self.info, seen)
         self.members = seen.values()
         if self.variants:
             self.variants.check(schema, seen)
             assert self.variants.tag_member in self.members
-            self.variants.check_clash(schema, seen)
+            self.variants.check_clash(schema, self.info, seen)
 
-    def check_clash(self, schema, seen):
+    # Check that the members of this type do not cause duplicate JSON fields,
+    # and update seen to track the members seen so far. Report any errors
+    # on behalf of info, which is not necessarily self.info
+    def check_clash(self, schema, info, seen):
         assert not self.variants       # not implemented
         for m in self.members:
-            m.check_clash(seen)
+            m.check_clash(info, seen)
 
     def is_implicit(self):
         # See QAPISchema._make_implicit_object_type()
@@ -1036,10 +1039,13 @@ class QAPISchemaObjectTypeMember(object):
         self.type = schema.lookup_type(self._type_name)
         assert self.type
 
-    def check_clash(self, seen):
-        # TODO change key of seen from QAPI name to C name
-        assert self.name not in seen
-        seen[self.name] = self
+    def check_clash(self, info, seen):
+        cname = c_name(self.name)
+        if cname in seen:
+            raise QAPIExprError(info,
+                                "%s collides with %s"
+                                % (self.describe(), seen[cname].describe()))
+        seen[cname] = self
 
     def _pretty_owner(self):
         owner = self.owner
@@ -1079,7 +1085,8 @@ class QAPISchemaObjectTypeVariants(object):
 
     def check(self, schema, seen):
         if not self.tag_member:    # flat union
-            self.tag_member = seen[self.tag_name]
+            self.tag_member = seen[c_name(self.tag_name)]
+            assert self.tag_name == self.tag_member.name
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
             v.check(schema)
@@ -1087,12 +1094,12 @@ class QAPISchemaObjectTypeVariants(object):
             if isinstance(v.type, QAPISchemaObjectType):
                 v.type.check(schema)
 
-    def check_clash(self, schema, seen):
+    def check_clash(self, schema, info, seen):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
             assert isinstance(v.type, QAPISchemaObjectType)
-            v.type.check_clash(schema, dict(seen))
+            v.type.check_clash(schema, info, dict(seen))
 
 
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
tests/qapi-schema/args-name-clash.err +1 -0
index e69de29..d953e8d 100644
@@ -0,0 +1 @@
+tests/qapi-schema/args-name-clash.json:4: 'a_b' (parameter of oops) collides with 'a-b' (parameter of oops)
tests/qapi-schema/args-name-clash.exit +1 -1
index 573541a..d00491f 100644
@@ -1 +1 @@
-0
+1
tests/qapi-schema/args-name-clash.json +2 -3
index 9e8f889..61423cb 100644
@@ -1,5 +1,4 @@
 # C member name collision
-# FIXME - This parses, but fails to compile, because the C struct is given
-# two 'a_b' members.  Either reject this at parse time, or munge the C names
-# to avoid the collision.
+# Reject members that clash when mapped to C names (we would have two 'a_b'
+# members).
 { 'command': 'oops', 'data': { 'a-b': 'str', 'a_b': 'str' } }
tests/qapi-schema/args-name-clash.out +0 -6
index 9b2f6e4..e69de29 100644
@@ -1,6 +0,0 @@
-object :empty
-object :obj-oops-arg
-    member a-b: str optional=False
-    member a_b: str optional=False
-command oops :obj-oops-arg -> None
-   gen=True success_response=True
Signature · 1 line
2.4.3
author
#20 lore raw

PATCH v12 17/36 qapi: Fix c_name() munging

In reply to Eric Blake

The method c_name() is supposed to do two different actions: munge '-' into '_', and add a 'q_' prefix to ticklish names. But it did these steps out of order, making it possible to submit input that is not ticklish until after munging, where the output then lacked the desired prefix.

The failure is exposed easily if you have a compiler that recognizes C11 keywords, and try to name a member '_Thread-local', as it would result in trying to compile the declaration 'uint64_t _Thread_local;' which is not valid. However, this name violates our conventions (ultimately, want to enforce that no qapi names start with single underscore), so the test is slightly weaker by instead testing 'wchar-t'; the declaration 'uint64_t wchar_t;' is valid in C (where wchar_t is only a typedef) but fails with a C++ compiler (where it is a keyword).

Fix things by reversing the order of actions within c_name().

---
v12: new patch

---
 scripts/qapi.py                         | 3 ++-
 tests/qapi-schema/qapi-schema-test.json | 5 +++--
 tests/qapi-schema/qapi-schema-test.out  | 1 +
 tests/test-qmp-commands.c               | 4 ++++
 4 files changed, 10 insertions(+), 3 deletions(-)
scripts/qapi.py +2 -1
index b068141..4a30bc0 100644
@@ -1481,10 +1481,11 @@ def c_name(name, protect=True):
                      'not_eq', 'or', 'or_eq', 'xor', 'xor_eq'])
     # namespace pollution:
     polluted_words = set(['unix', 'errno'])
+    name = name.translate(c_name_trans)
     if protect and (name in c89_words | c99_words | c11_words | gcc_words
                     | cpp_words | polluted_words):
         return "q_" + name
-    return name.translate(c_name_trans)
+    return name
 
 eatspace = '\033EATSPACE.'
 pointer_suffix = ' *' + eatspace
tests/qapi-schema/qapi-schema-test.json +3 -2
index 44638da..4b89527 100644
@@ -152,12 +152,13 @@
 { 'event': 'EVENT_D',
   'data': { 'a' : 'EventStructOne', 'b' : 'str', '*c': 'str', '*enum3': 'EnumOne' } }
 
-# test that we correctly compile downstream extensions
+# test that we correctly compile downstream extensions, as well as munge
+# ticklish names
 { 'enum': '__org.qemu_x-Enum', 'data': [ '__org.qemu_x-value' ] }
 { 'struct': '__org.qemu_x-Base',
   'data': { '__org.qemu_x-member1': '__org.qemu_x-Enum' } }
 { 'struct': '__org.qemu_x-Struct', 'base': '__org.qemu_x-Base',
-  'data': { '__org.qemu_x-member2': 'str' } }
+  'data': { '__org.qemu_x-member2': 'str', '*wchar-t': 'int' } }
 { 'union': '__org.qemu_x-Union1', 'data': { '__org.qemu_x-branch': 'str' } }
 { 'struct': '__org.qemu_x-Struct2',
   'data': { 'array': ['__org.qemu_x-Union1'] } }
tests/qapi-schema/qapi-schema-test.out +1 -0
index 786024e..0724a9f 100644
@@ -185,6 +185,7 @@ enum __org.qemu_x-Enum ['__org.qemu_x-value']
 object __org.qemu_x-Struct
     base __org.qemu_x-Base
     member __org.qemu_x-member2: str optional=False
+    member wchar-t: int optional=True
 object __org.qemu_x-Struct2
     member array: __org.qemu_x-Union1List optional=False
 object __org.qemu_x-Union1
tests/test-qmp-commands.c +4 -0
index 888fb5f..9f35b80 100644
@@ -65,6 +65,10 @@ __org_qemu_x_Union1 *qmp___org_qemu_x_command(__org_qemu_x_EnumList *a,
     ret->type = ORG_QEMU_X_UNION1_KIND___ORG_QEMU_X_BRANCH;
     ret->u.__org_qemu_x_branch = strdup("blah1");
 
+    /* Also test that 'wchar-t' was munged to 'q_wchar_t' */
+    if (b && b->value && !b->value->has_q_wchar_t) {
+        b->value->q_wchar_t = 1;
+    }
     return ret;
 }
 
Signature · 1 line
2.4.3
#21 lore raw

Re: [Qemu-devel] [PATCH v12 17/36] qapi: Fix c_name() munging

Quoted 4 lines · Eric Blake

*** WARNING: THE ATTACHED DOCUMENT(S) CONTAIN MACROS ***
*** MACROS MAY CONTAIN MALICIOUS CODE ***
*** Open only if you can verify and trust the sender ***
*** Please contact [email protected] if you have questions or concerns **

Another one.

Quoted 6 lines

The method c_name() is supposed to do two different actions: munge '-' into '_', and add a 'q_' prefix to ticklish names. But it did these steps out of order, making it possible to submit input that is not ticklish until after munging, where the output then lacked the desired prefix.

Quoted 9 lines

The failure is exposed easily if you have a compiler that recognizes C11 keywords, and try to name a member '_Thread-local', as it would result in trying to compile the declaration 'uint64_t _Thread_local;' which is not valid. However, this name violates our conventions (ultimately, want to enforce that no qapi names start with single underscore), so the test is slightly weaker by instead testing 'wchar-t'; the declaration 'uint64_t wchar_t;' is valid in C (where wchar_t is only a typedef) but fails with a C++ compiler (where it is a keyword).

Do we support compiling it with a C++ compiler? To sidestep the question, I'd say "but would fail".

In my private opinion, the one sane way to compile C code with a C++ compiler is wrapping it in extern "C" { ... }.

Quoted 3 lines

Fix things by reversing the order of actions within c_name().

Again, just commit message polish, can do on merge.

author
#22 lore raw

Re: [Qemu-devel] [PATCH v12 17/36] qapi: Fix c_name() munging

Quoted 25 lines · Markus Armbruster
Quoted 4 lines · Eric Blake

*** WARNING: THE ATTACHED DOCUMENT(S) CONTAIN MACROS ***
*** MACROS MAY CONTAIN MALICIOUS CODE ***
*** Open only if you can verify and trust the sender ***
*** Please contact [email protected] if you have questions or concerns **

Another one.

Quoted 6 lines

The method c_name() is supposed to do two different actions: munge '-' into '_', and add a 'q_' prefix to ticklish names. But it did these steps out of order, making it possible to submit input that is not ticklish until after munging, where the output then lacked the desired prefix.

Quoted 9 lines

The failure is exposed easily if you have a compiler that recognizes C11 keywords, and try to name a member '_Thread-local', as it would result in trying to compile the declaration 'uint64_t _Thread_local;' which is not valid. However, this name violates our conventions (ultimately, want to enforce that no qapi names start with single underscore), so the test is slightly weaker by instead testing 'wchar-t'; the declaration 'uint64_t wchar_t;' is valid in C (where wchar_t is only a typedef) but fails with a C++ compiler (where it is a keyword).

Quoted 2 lines

Do we support compiling it with a C++ compiler? To sidestep the question, I'd say "but would fail".

I know we require a C++ compiler for qga on Windows, and qga uses qapi, so I think the problem is real; but as I do not have a working setup for compiling qga for windows, I can only speculate. Changing s/fails/but would fail/ is a nice hedge.

Quoted 3 lines

In my private opinion, the one sane way to compile C code with a C++ compiler is wrapping it in extern "C" { ... }.

True - but I don't think that changes the set of C++ keywords inside the extern block, does it? (The thought of context-sensitive keywords makes me cringe for how one would write a sane parser for that language).

Quoted 7 lines
Quoted 4 lines

Fix things by reversing the order of actions within c_name().

Again, just commit message polish, can do on merge.

Don't know why you got it on some messages and not others; I also got a round of those pollutions on other threads. It looks like the responsible party has cleaned up their false positives in the meantime, so hopefully we aren't hit by more of the annoyance.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#23 lore raw

Re: [Qemu-devel] [PATCH v12 17/36] qapi: Fix c_name() munging

Quoted 35 lines · Eric Blake
Quoted 25 lines · Markus Armbruster
Quoted 4 lines · Eric Blake

*** WARNING: THE ATTACHED DOCUMENT(S) CONTAIN MACROS ***
*** MACROS MAY CONTAIN MALICIOUS CODE ***
*** Open only if you can verify and trust the sender ***
*** Please contact [email protected] if you have questions or concerns **

Another one.

Quoted 6 lines

The method c_name() is supposed to do two different actions: munge '-' into '_', and add a 'q_' prefix to ticklish names. But it did these steps out of order, making it possible to submit input that is not ticklish until after munging, where the output then lacked the desired prefix.

Quoted 9 lines

The failure is exposed easily if you have a compiler that recognizes C11 keywords, and try to name a member '_Thread-local', as it would result in trying to compile the declaration 'uint64_t _Thread_local;' which is not valid. However, this name violates our conventions (ultimately, want to enforce that no qapi names start with single underscore), so the test is slightly weaker by instead testing 'wchar-t'; the declaration 'uint64_t wchar_t;' is valid in C (where wchar_t is only a typedef) but fails with a C++ compiler (where it is a keyword).

Quoted 2 lines

Do we support compiling it with a C++ compiler? To sidestep the question, I'd say "but would fail".

I know we require a C++ compiler for qga on Windows, and qga uses qapi, so I think the problem is real; but as I do not have a working setup for compiling qga for windows, I can only speculate. Changing s/fails/but would fail/ is a nice hedge.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

In my private opinion, the one sane way to compile C code with a C++ compiler is wrapping it in extern "C" { ... }.

True - but I don't think that changes the set of C++ keywords inside the extern block, does it? (The thought of context-sensitive keywords makes me cringe for how one would write a sane parser for that language).

The truly sane way to compile C with a C++ compiler is of course "not":
compile it with C and link it.

The exception is headers. Stick to declaring extern symbols and the types they need.

Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

Fix things by reversing the order of actions within c_name().

Again, just commit message polish, can do on merge.

Don't know why you got it on some messages and not others; I also got a round of those pollutions on other threads. It looks like the responsible party has cleaned up their false positives in the meantime, so hopefully we aren't hit by more of the annoyance.

I reported it, and was told it's been fixed.

author
#24 lore raw

PATCH v12 18/36 qapi: Remove dead visitor code

In reply to Eric Blake

Commit cbc95538 removed unused start_handle() and end_handle(), but forgot to remove their declarations.

---
v12 (no v10-11): Split off obvious piece from v9 26/27
v9: no change
v8: no change
v7: no change
v6: no change

---
 include/qapi/visitor.h | 3 ---
 1 file changed, 3 deletions(-)
include/qapi/visitor.h +0 -3
index cfc19a6..a2ad66c 100644
@@ -27,9 +27,6 @@ typedef struct GenericList
     struct GenericList *next;
 } GenericList;
 
-void visit_start_handle(Visitor *v, void **obj, const char *kind,
-                        const char *name, Error **errp);
-void visit_end_handle(Visitor *v, Error **errp);
 void visit_start_struct(Visitor *v, void **obj, const char *kind,
                         const char *name, size_t size, Error **errp);
 void visit_end_struct(Visitor *v, Error **errp);
Signature · 1 line
2.4.3
author
#25 lore raw

PATCH v12 19/36 blkdebug: Merge hand-rolled and qapi BlkdebugEvent enum

In reply to Eric Blake

No need to keep two separate enums, where editing one is likely to forget the other. Now that we can specify a qapi enum prefix, we don't even have to change the bulk of the uses.

get_event_by_name() could perhaps be replaced by qapi_enum_parse(), but I left that for another day.

---
v12: new patch

---
 block.c                   |  2 +-
 block/blkdebug.c          | 79 +++++++----------------------------------------
 docs/blkdebug.txt         |  7 +++--
 include/block/block.h     | 62 +------------------------------------
 include/block/block_int.h |  2 +-
 qapi/block-core.json      |  4 ++-
 6 files changed, 21 insertions(+), 135 deletions(-)
block.c +1 -1
index 3a7324b..9971976 100644
@@ -2851,7 +2851,7 @@ ImageInfoSpecific *bdrv_get_specific_info(BlockDriverState *bs)
     return NULL;
 }
 
-void bdrv_debug_event(BlockDriverState *bs, BlkDebugEvent event)
+void bdrv_debug_event(BlockDriverState *bs, BlkdebugEvent event)
 {
     if (!bs || !bs->drv || !bs->drv->bdrv_debug_event) {
         return;
block/blkdebug.c +11 -68

135 lines not on this page · raw message

docs/blkdebug.txt +4 -3
index b67a36d..43d8e8f 100644
@@ -1,6 +1,6 @@
 Block I/O error injection using blkdebug
 ----------------------------------------
-Copyright (C) 2014 Red Hat Inc
+Copyright (C) 2014-2015 Red Hat Inc
 
 This work is licensed under the terms of the GNU GPL, version 2 or later.  See
 the COPYING file in the top-level directory.
@@ -92,8 +92,9 @@ The core events are:
 
   flush_to_disk - flush the host block device's disk cache
 
-See block/blkdebug.c:event_names[] for the full list of events.  You may need
-to grep block driver source code to understand the meaning of specific events.
+See qapi/block-core.json:BlkdebugEvent for the full list of events.
+You may need to grep block driver source code to understand the
+meaning of specific events.
 
 State transitions
 -----------------
include/block/block.h +1 -61

74 lines not on this page · raw message

include/block/block_int.h +1 -1
index 4012e36..66e208d 100644
@@ -244,7 +244,7 @@ struct BlockDriver {
     int (*bdrv_amend_options)(BlockDriverState *bs, QemuOpts *opts,
                               BlockDriverAmendStatusCB *status_cb);
 
-    void (*bdrv_debug_event)(BlockDriverState *bs, BlkDebugEvent event);
+    void (*bdrv_debug_event)(BlockDriverState *bs, BlkdebugEvent event);
 
     /* TODO Better pass a option string/QDict/QemuOpts to add any rule? */
     int (*bdrv_debug_breakpoint)(BlockDriverState *bs, const char *event,
qapi/block-core.json +3 -1
index a07b13f..603eb60 100644
@@ -1776,8 +1776,10 @@
 # @BlkdebugEvent
 #
 # Trigger events supported by blkdebug.
+#
+# Since: 2.0
 ##
-{ 'enum': 'BlkdebugEvent',
+{ 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
   'data': [ 'l1_update', 'l1_grow.alloc_table', 'l1_grow.write_table',
             'l1_grow.activate_table', 'l2_load', 'l2_update',
             'l2_update_compressed', 'l2_alloc.cow_read', 'l2_alloc.write',
Signature · 1 line
2.4.3
#26 lore raw

Re: [Qemu-devel] [PATCH v12 19/36] blkdebug: Merge hand-rolled and qapi BlkdebugEvent enum

Quoted 304 lines · Eric Blake

No need to keep two separate enums, where editing one is likely to forget the other. Now that we can specify a qapi enum prefix, we don't even have to change the bulk of the uses.

get_event_by_name() could perhaps be replaced by qapi_enum_parse(), but I left that for another day.

---
v12: new patch

---
 block.c                   |  2 +-
 block/blkdebug.c          | 79 +++++++----------------------------------------
 docs/blkdebug.txt         |  7 +++--
 include/block/block.h     | 62 +------------------------------------
 include/block/block_int.h |  2 +-
 qapi/block-core.json      |  4 ++-
 6 files changed, 21 insertions(+), 135 deletions(-)
block.c +1 -1
index 3a7324b..9971976 100644
@@ -2851,7 +2851,7 @@ ImageInfoSpecific *bdrv_get_specific_info(BlockDriverState *bs)
     return NULL;
 }
 
-void bdrv_debug_event(BlockDriverState *bs, BlkDebugEvent event)
+void bdrv_debug_event(BlockDriverState *bs, BlkdebugEvent event)
 {
     if (!bs || !bs->drv || !bs->drv->bdrv_debug_event) {
         return;
block/blkdebug.c +11 -68
index 6860a2b..76805a6 100644
@@ -35,7 +35,7 @@ typedef struct BDRVBlkdebugState {
     int state;
     int new_state;
 
-    QLIST_HEAD(, BlkdebugRule) rules[BLKDBG_EVENT_MAX];
+    QLIST_HEAD(, BlkdebugRule) rules[BLKDBG_MAX];
     QSIMPLEQ_HEAD(, BlkdebugRule) active_rules;
     QLIST_HEAD(, BlkdebugSuspendedReq) suspended_reqs;
 } BDRVBlkdebugState;
@@ -63,7 +63,7 @@ enum {
 };
 
 typedef struct BlkdebugRule {
-    BlkDebugEvent event;
+    BlkdebugEvent event;
     int action;
     int state;
     union {
@@ -142,69 +142,12 @@ static QemuOptsList *config_groups[] = {
     NULL
 };
 
-static const char *event_names[BLKDBG_EVENT_MAX] = {
-    [BLKDBG_L1_UPDATE]                      = "l1_update",
-    [BLKDBG_L1_GROW_ALLOC_TABLE]            = "l1_grow.alloc_table",
-    [BLKDBG_L1_GROW_WRITE_TABLE]            = "l1_grow.write_table",
-    [BLKDBG_L1_GROW_ACTIVATE_TABLE]         = "l1_grow.activate_table",
-
-    [BLKDBG_L2_LOAD]                        = "l2_load",
-    [BLKDBG_L2_UPDATE]                      = "l2_update",
-    [BLKDBG_L2_UPDATE_COMPRESSED]           = "l2_update_compressed",
-    [BLKDBG_L2_ALLOC_COW_READ]              = "l2_alloc.cow_read",
-    [BLKDBG_L2_ALLOC_WRITE]                 = "l2_alloc.write",
-
-    [BLKDBG_READ_AIO]                       = "read_aio",
-    [BLKDBG_READ_BACKING_AIO]               = "read_backing_aio",
-    [BLKDBG_READ_COMPRESSED]                = "read_compressed",
-
-    [BLKDBG_WRITE_AIO]                      = "write_aio",
-    [BLKDBG_WRITE_COMPRESSED]               = "write_compressed",
-
-    [BLKDBG_VMSTATE_LOAD]                   = "vmstate_load",
-    [BLKDBG_VMSTATE_SAVE]                   = "vmstate_save",
-
-    [BLKDBG_COW_READ]                       = "cow_read",
-    [BLKDBG_COW_WRITE]                      = "cow_write",
-
-    [BLKDBG_REFTABLE_LOAD]                  = "reftable_load",
-    [BLKDBG_REFTABLE_GROW]                  = "reftable_grow",
-    [BLKDBG_REFTABLE_UPDATE]                = "reftable_update",
-
-    [BLKDBG_REFBLOCK_LOAD]                  = "refblock_load",
-    [BLKDBG_REFBLOCK_UPDATE]                = "refblock_update",
-    [BLKDBG_REFBLOCK_UPDATE_PART]           = "refblock_update_part",
-    [BLKDBG_REFBLOCK_ALLOC]                 = "refblock_alloc",
-    [BLKDBG_REFBLOCK_ALLOC_HOOKUP]          = "refblock_alloc.hookup",
-    [BLKDBG_REFBLOCK_ALLOC_WRITE]           = "refblock_alloc.write",
-    [BLKDBG_REFBLOCK_ALLOC_WRITE_BLOCKS]    = "refblock_alloc.write_blocks",
-    [BLKDBG_REFBLOCK_ALLOC_WRITE_TABLE]     = "refblock_alloc.write_table",
-    [BLKDBG_REFBLOCK_ALLOC_SWITCH_TABLE]    = "refblock_alloc.switch_table",
-
-    [BLKDBG_CLUSTER_ALLOC]                  = "cluster_alloc",
-    [BLKDBG_CLUSTER_ALLOC_BYTES]            = "cluster_alloc_bytes",
-    [BLKDBG_CLUSTER_FREE]                   = "cluster_free",
-
-    [BLKDBG_FLUSH_TO_OS]                    = "flush_to_os",
-    [BLKDBG_FLUSH_TO_DISK]                  = "flush_to_disk",
-
-    [BLKDBG_PWRITEV_RMW_HEAD]               = "pwritev_rmw.head",
-    [BLKDBG_PWRITEV_RMW_AFTER_HEAD]         = "pwritev_rmw.after_head",
-    [BLKDBG_PWRITEV_RMW_TAIL]               = "pwritev_rmw.tail",
-    [BLKDBG_PWRITEV_RMW_AFTER_TAIL]         = "pwritev_rmw.after_tail",
-    [BLKDBG_PWRITEV]                        = "pwritev",
-    [BLKDBG_PWRITEV_ZERO]                   = "pwritev_zero",
-    [BLKDBG_PWRITEV_DONE]                   = "pwritev_done",
-
-    [BLKDBG_EMPTY_IMAGE_PREPARE]            = "empty_image_prepare",
-};
-
-static int get_event_by_name(const char *name, BlkDebugEvent *event)
+static int get_event_by_name(const char *name, BlkdebugEvent *event)
 {
     int i;
 
-    for (i = 0; i < BLKDBG_EVENT_MAX; i++) {
-        if (!strcmp(event_names[i], name)) {
+    for (i = 0; i < BLKDBG_MAX; i++) {
+        if (!strcmp(BlkdebugEvent_lookup[i], name)) {
             *event = i;
             return 0;
         }
@@ -223,7 +166,7 @@ static int add_rule(void *opaque, QemuOpts *opts, Error **errp)
     struct add_rule_data *d = opaque;
     BDRVBlkdebugState *s = d->s;
     const char* event_name;
-    BlkDebugEvent event;
+    BlkdebugEvent event;
     struct BlkdebugRule *rule;
 
     /* Find the right event for the rule */
@@ -563,7 +506,7 @@ static void blkdebug_close(BlockDriverState *bs)
     BlkdebugRule *rule, *next;
     int i;
 
-    for (i = 0; i < BLKDBG_EVENT_MAX; i++) {
+    for (i = 0; i < BLKDBG_MAX; i++) {
         QLIST_FOREACH_SAFE(rule, &s->rules[i], next, next) {
             remove_rule(rule);
         }
@@ -622,13 +565,13 @@ static bool process_rule(BlockDriverState *bs, struct BlkdebugRule *rule,
     return injected;
 }
 
-static void blkdebug_debug_event(BlockDriverState *bs, BlkDebugEvent event)
+static void blkdebug_debug_event(BlockDriverState *bs, BlkdebugEvent event)
 {
     BDRVBlkdebugState *s = bs->opaque;
     struct BlkdebugRule *rule, *next;
     bool injected;
 
-    assert((int)event >= 0 && event < BLKDBG_EVENT_MAX);
+    assert((int)event >= 0 && event < BLKDBG_MAX);
 
     injected = false;
     s->new_state = s->state;
@@ -643,7 +586,7 @@ static int blkdebug_debug_breakpoint(BlockDriverState *bs, const char *event,
 {
     BDRVBlkdebugState *s = bs->opaque;
     struct BlkdebugRule *rule;
-    BlkDebugEvent blkdebug_event;
+    BlkdebugEvent blkdebug_event;
 
     if (get_event_by_name(event, &blkdebug_event) < 0) {
         return -ENOENT;
@@ -685,7 +628,7 @@ static int blkdebug_debug_remove_breakpoint(BlockDriverState *bs,
     BlkdebugRule *rule, *next;
     int i, ret = -ENOENT;
 
-    for (i = 0; i < BLKDBG_EVENT_MAX; i++) {
+    for (i = 0; i < BLKDBG_MAX; i++) {
         QLIST_FOREACH_SAFE(rule, &s->rules[i], next, next) {
             if (rule->action == ACTION_SUSPEND &&
                 !strcmp(rule->options.suspend.tag, tag)) {
docs/blkdebug.txt +4 -3
index b67a36d..43d8e8f 100644
@@ -1,6 +1,6 @@
 Block I/O error injection using blkdebug
 ----------------------------------------
-Copyright (C) 2014 Red Hat Inc
+Copyright (C) 2014-2015 Red Hat Inc
 
 This work is licensed under the terms of the GNU GPL, version 2 or later.  See
 the COPYING file in the top-level directory.
@@ -92,8 +92,9 @@ The core events are:
 
   flush_to_disk - flush the host block device's disk cache
 
-See block/blkdebug.c:event_names[] for the full list of events.  You may need
-to grep block driver source code to understand the meaning of specific events.
+See qapi/block-core.json:BlkdebugEvent for the full list of events.
+You may need to grep block driver source code to understand the
+meaning of specific events.
 
 State transitions
 -----------------
include/block/block.h +1 -61
index 73edb1a..2a72969 100644
@@ -520,66 +520,6 @@ void bdrv_op_block_all(BlockDriverState *bs, Error *reason);
 void bdrv_op_unblock_all(BlockDriverState *bs, Error *reason);
 bool bdrv_op_blocker_is_empty(BlockDriverState *bs);
 
-typedef enum {
-    BLKDBG_L1_UPDATE,
-
-    BLKDBG_L1_GROW_ALLOC_TABLE,
-    BLKDBG_L1_GROW_WRITE_TABLE,
-    BLKDBG_L1_GROW_ACTIVATE_TABLE,
-
-    BLKDBG_L2_LOAD,
-    BLKDBG_L2_UPDATE,
-    BLKDBG_L2_UPDATE_COMPRESSED,
-    BLKDBG_L2_ALLOC_COW_READ,
-    BLKDBG_L2_ALLOC_WRITE,
-
-    BLKDBG_READ_AIO,
-    BLKDBG_READ_BACKING_AIO,
-    BLKDBG_READ_COMPRESSED,
-
-    BLKDBG_WRITE_AIO,
-    BLKDBG_WRITE_COMPRESSED,
-
-    BLKDBG_VMSTATE_LOAD,
-    BLKDBG_VMSTATE_SAVE,
-
-    BLKDBG_COW_READ,
-    BLKDBG_COW_WRITE,
-
-    BLKDBG_REFTABLE_LOAD,
-    BLKDBG_REFTABLE_GROW,
-    BLKDBG_REFTABLE_UPDATE,
-
-    BLKDBG_REFBLOCK_LOAD,
-    BLKDBG_REFBLOCK_UPDATE,
-    BLKDBG_REFBLOCK_UPDATE_PART,
-    BLKDBG_REFBLOCK_ALLOC,
-    BLKDBG_REFBLOCK_ALLOC_HOOKUP,
-    BLKDBG_REFBLOCK_ALLOC_WRITE,
-    BLKDBG_REFBLOCK_ALLOC_WRITE_BLOCKS,
-    BLKDBG_REFBLOCK_ALLOC_WRITE_TABLE,
-    BLKDBG_REFBLOCK_ALLOC_SWITCH_TABLE,
-
-    BLKDBG_CLUSTER_ALLOC,
-    BLKDBG_CLUSTER_ALLOC_BYTES,
-    BLKDBG_CLUSTER_FREE,
-
-    BLKDBG_FLUSH_TO_OS,
-    BLKDBG_FLUSH_TO_DISK,
-
-    BLKDBG_PWRITEV_RMW_HEAD,
-    BLKDBG_PWRITEV_RMW_AFTER_HEAD,
-    BLKDBG_PWRITEV_RMW_TAIL,
-    BLKDBG_PWRITEV_RMW_AFTER_TAIL,
-    BLKDBG_PWRITEV,
-    BLKDBG_PWRITEV_ZERO,
-    BLKDBG_PWRITEV_DONE,
-
-    BLKDBG_EMPTY_IMAGE_PREPARE,
-
-    BLKDBG_EVENT_MAX,
-} BlkDebugEvent;
-
 #define BLKDBG_EVENT(child, evt) \
     do { \
         if (child) { \
@@ -587,7 +527,7 @@ typedef enum {
         } \
     } while (0)
 
-void bdrv_debug_event(BlockDriverState *bs, BlkDebugEvent event);
+void bdrv_debug_event(BlockDriverState *bs, BlkdebugEvent event);
 
 int bdrv_debug_breakpoint(BlockDriverState *bs, const char *event,
                            const char *tag);
include/block/block_int.h +1 -1
index 4012e36..66e208d 100644
@@ -244,7 +244,7 @@ struct BlockDriver {
     int (*bdrv_amend_options)(BlockDriverState *bs, QemuOpts *opts,
                               BlockDriverAmendStatusCB *status_cb);
 
-    void (*bdrv_debug_event)(BlockDriverState *bs, BlkDebugEvent event);
+    void (*bdrv_debug_event)(BlockDriverState *bs, BlkdebugEvent event);
 
     /* TODO Better pass a option string/QDict/QemuOpts to add any rule? */
     int (*bdrv_debug_breakpoint)(BlockDriverState *bs, const char *event,
qapi/block-core.json +0 -0
index a07b13f..603eb60 100644
@@ -1776,8 +1776,10 @@
 # @BlkdebugEvent

qapi/block-core.json line 1785

Quoted 10 lines
qapi/block-core.json +3 -1
 #
 # Trigger events supported by blkdebug.
+#
+# Since: 2.0
 ##
-{ 'enum': 'BlkdebugEvent',
+{ 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
   'data': [ 'l1_update', 'l1_grow.alloc_table', 'l1_grow.write_table',
             'l1_grow.activate_table', 'l2_load', 'l2_update',
             'l2_update_compressed', 'l2_alloc.cow_read', 'l2_alloc.write',

I'm no friend of the 'prefix' feature. You could avoid it here by renaming BlkdebugEvent to Blkdbg. No additional churn, because you already replace hand-written BlkDebugEvent by generated BlkdebugEvent.

Alternatively, you could reduce churn by renaming it to BlkDebugEvent.

Matter of taste. Perhaps Kevin has a preference.

#27 lore raw

Re: [Qemu-devel] [PATCH v12 19/36] blkdebug: Merge hand-rolled and qapi BlkdebugEvent enum

Am 18.11.2015 um 11:30 hat Markus Armbruster geschrieben:

Quoted 33 lines
Quoted 16 lines · Eric Blake

No need to keep two separate enums, where editing one is likely to forget the other. Now that we can specify a qapi enum prefix, we don't even have to change the bulk of the uses.

get_event_by_name() could perhaps be replaced by qapi_enum_parse(), but I left that for another day.

qapi/block-core.json +0 -0
index a07b13f..603eb60 100644
@@ -1776,8 +1776,10 @@
 # @BlkdebugEvent

qapi/block-core.json line 1785

Quoted 10 lines
qapi/block-core.json +3 -1
 #
 # Trigger events supported by blkdebug.
+#
+# Since: 2.0
 ##
-{ 'enum': 'BlkdebugEvent',
+{ 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
   'data': [ 'l1_update', 'l1_grow.alloc_table', 'l1_grow.write_table',
             'l1_grow.activate_table', 'l2_load', 'l2_update',
             'l2_update_compressed', 'l2_alloc.cow_read', 'l2_alloc.write',

I'm no friend of the 'prefix' feature. You could avoid it here by renaming BlkdebugEvent to Blkdbg. No additional churn, because you already replace hand-written BlkDebugEvent by generated BlkdebugEvent.

Quoted 3 lines

Alternatively, you could reduce churn by renaming it to BlkDebugEvent.

Matter of taste. Perhaps Kevin has a preference.

Wouldn't that mean that we end up with a C type called Blkdbg? In my opinion that's a bit unspecific, so if you ask me, I would paint my bikeshed in a different colour.

Kevin

author
#28 lore raw

Re: [Qemu-devel] [PATCH v12 19/36] blkdebug: Merge hand-rolled and qapi BlkdebugEvent enum

Quoted 19 lines · Kevin Wolf

Am 18.11.2015 um 11:30 hat Markus Armbruster geschrieben:

Quoted 18 lines

qapi/block-core.json line 1785

Quoted 11 lines · Eric Blake

No need to keep two separate enums, where editing one is likely to forget the other. Now that we can specify a qapi enum prefix, we don't even have to change the bulk of the uses.

##

qapi/block-core.json +1 -1
-{ 'enum': 'BlkdebugEvent',
+{ 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
   'data': [ 'l1_update', 'l1_grow.alloc_table', 'l1_grow.write_table',
             'l1_grow.activate_table', 'l2_load', 'l2_update',
             'l2_update_compressed', 'l2_alloc.cow_read', 'l2_alloc.write',

I'm no friend of the 'prefix' feature. You could avoid it here by renaming BlkdebugEvent to Blkdbg. No additional churn, because you already replace hand-written BlkDebugEvent by generated BlkdebugEvent.

Quoted 7 lines
Quoted 3 lines

Alternatively, you could reduce churn by renaming it to BlkDebugEvent.

Matter of taste. Perhaps Kevin has a preference.

Wouldn't that mean that we end up with a C type called Blkdbg? In my opinion that's a bit unspecific, so if you ask me, I would paint my bikeshed in a different colour.

Most of the existing qapi names are Blkdebug, it was only the internal enum being replaced that used BlkDebug. Also, qapi would munge BlkDebug into BLK_DEBUG, so I think we want to stick with the lowercase 'd' so that the munging (without 'prefix') would stick to BLKDEBUG.

I'm also fine if you want me to do a followup patch that renames all enums from BLKDBG_L1_UPDATE to BLKDEBUG_EVENT_L1_UPDATE, at which point we could drop the 'prefix' (I don't know how many lines it would make long enough to need different wrapping, but modulo wrapping it would all be a mechanically scripted change).

I don't have a favorite color in this painting match, so let me know yours :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#29 lore raw

Re: [Qemu-devel] [PATCH v12 19/36] blkdebug: Merge hand-rolled and qapi BlkdebugEvent enum

Am 18.11.2015 um 17:26 hat Eric Blake geschrieben:

Quoted 24 lines · Kevin Wolf
Quoted 23 lines

Am 18.11.2015 um 11:30 hat Markus Armbruster geschrieben:

Quoted 21 lines

qapi/block-core.json line 1785

Quoted 11 lines · Eric Blake

No need to keep two separate enums, where editing one is likely to forget the other. Now that we can specify a qapi enum prefix, we don't even have to change the bulk of the uses.

##

qapi/block-core.json +1 -1
-{ 'enum': 'BlkdebugEvent',
+{ 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
   'data': [ 'l1_update', 'l1_grow.alloc_table', 'l1_grow.write_table',
             'l1_grow.activate_table', 'l2_load', 'l2_update',
             'l2_update_compressed', 'l2_alloc.cow_read', 'l2_alloc.write',

I'm no friend of the 'prefix' feature. You could avoid it here by renaming BlkdebugEvent to Blkdbg. No additional churn, because you already replace hand-written BlkDebugEvent by generated BlkdebugEvent.

Alternatively, you could reduce churn by renaming it to BlkDebugEvent.

Matter of taste. Perhaps Kevin has a preference.

Quoted 8 lines
Quoted 3 lines

Wouldn't that mean that we end up with a C type called Blkdbg? In my opinion that's a bit unspecific, so if you ask me, I would paint my bikeshed in a different colour.

Most of the existing qapi names are Blkdebug, it was only the internal enum being replaced that used BlkDebug. Also, qapi would munge BlkDebug into BLK_DEBUG, so I think we want to stick with the lowercase 'd' so that the munging (without 'prefix') would stick to BLKDEBUG.

Oh, I don't really have an opinion whether BlockDebug, BlkDebug or Blkdebug is the best spelling. I think Blkdebug makes sense.

But I do have a strong opinion on whether the enum should be called BlkdebugEvent or just Blkdbg (without any mention of "Event"). The latter doesn't describe any more what the type is about.

Quoted 5 lines

I'm also fine if you want me to do a followup patch that renames all enums from BLKDBG_L1_UPDATE to BLKDEBUG_EVENT_L1_UPDATE, at which point we could drop the 'prefix' (I don't know how many lines it would make long enough to need different wrapping, but modulo wrapping it would all be a mechanically scripted change).

I don't mind the 'prefix' feature. I think it's nice to have short, but still unique enough constant names, but if Markus is bothered enough to send a patch, I can live with BLKDEBUG_EVENT_*.

Kevin

#30 lore raw

Re: [Qemu-devel] [PATCH v12 19/36] blkdebug: Merge hand-rolled and qapi BlkdebugEvent enum

Quoted 34 lines · Kevin Wolf

Am 18.11.2015 um 17:26 hat Eric Blake geschrieben:

Quoted 24 lines · Kevin Wolf
Quoted 23 lines

Am 18.11.2015 um 11:30 hat Markus Armbruster geschrieben:

Quoted 21 lines

qapi/block-core.json line 1785

Quoted 11 lines · Eric Blake

No need to keep two separate enums, where editing one is likely to forget the other. Now that we can specify a qapi enum prefix, we don't even have to change the bulk of the uses.

##

qapi/block-core.json +1 -1
-{ 'enum': 'BlkdebugEvent',
+{ 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
   'data': [ 'l1_update', 'l1_grow.alloc_table', 'l1_grow.write_table',
             'l1_grow.activate_table', 'l2_load', 'l2_update',
             'l2_update_compressed', 'l2_alloc.cow_read', 'l2_alloc.write',

I'm no friend of the 'prefix' feature. You could avoid it here by renaming BlkdebugEvent to Blkdbg. No additional churn, because you already replace hand-written BlkDebugEvent by generated BlkdebugEvent.

Alternatively, you could reduce churn by renaming it to BlkDebugEvent.

Matter of taste. Perhaps Kevin has a preference.

Quoted 8 lines
Quoted 3 lines

Wouldn't that mean that we end up with a C type called Blkdbg? In my opinion that's a bit unspecific, so if you ask me, I would paint my bikeshed in a different colour.

Most of the existing qapi names are Blkdebug, it was only the internal enum being replaced that used BlkDebug. Also, qapi would munge BlkDebug into BLK_DEBUG, so I think we want to stick with the lowercase 'd' so that the munging (without 'prefix') would stick to BLKDEBUG.

Quoted 6 lines

Oh, I don't really have an opinion whether BlockDebug, BlkDebug or Blkdebug is the best spelling. I think Blkdebug makes sense.

But I do have a strong opinion on whether the enum should be called BlkdebugEvent or just Blkdbg (without any mention of "Event"). The latter doesn't describe any more what the type is about.

Okay, we'll take the patch as is then.

Quoted 9 lines
Quoted 5 lines

I'm also fine if you want me to do a followup patch that renames all enums from BLKDBG_L1_UPDATE to BLKDEBUG_EVENT_L1_UPDATE, at which point we could drop the 'prefix' (I don't know how many lines it would make long enough to need different wrapping, but modulo wrapping it would all be a mechanically scripted change).

I don't mind the 'prefix' feature. I think it's nice to have short, but still unique enough constant names, but if Markus is bothered enough to send a patch, I can live with BLKDEBUG_EVENT_*.

Fair enough :) For now, let's move the queue forward.

author
#31 lore raw

PATCH v12 20/36 blkdebug: Avoid '.' in enum values

In reply to Eric Blake

Our qapi conventions document that '.' should only be used in the prefix of downstream names. BlkdebugEvent was a lone exception to this. Changing this is not backwards compatible to the 'blockdev-add' QMP command; however, that command is not yet fully stable. It can also be argued that the testsuite is the biggest user of blkdebug, and that any other user can be taught to deal with the change by paying attention to introspection results.

Done with:

$ for str in \
     l1_grow.{alloc,write,activate}_table \
     l2_alloc.{cow_read,write} \
     refblock_alloc.{hookup,write,write_blocks,write_table,switch_table} \
     pwritev_rmw.{head,after_head,tail,after_tail}; do
   str1=$(echo "$str" | sed 's/\./\\./')
   str2=$(echo "$str" | sed 's/\./_/')
   git grep -l "$str1" | xargs -r sed -i "s/$str1/$str2/g"
 done

followed by a manual touchup to test 77 to keep the test working.

---
v12: new commit. Arguably, test 77 should have failed immediately on an unrecognized event name instead of hanging forever, but that fix belongs in a separate patch, not part of this series.

---
 qapi/block-core.json               | 16 ++++----
 tests/qemu-iotests/026             | 18 ++++-----
 tests/qemu-iotests/026.out         | 80 +++++++++++++++++++-------------------
 tests/qemu-iotests/026.out.nocache | 80 +++++++++++++++++++-------------------
 tests/qemu-iotests/077             | 24 ++++++------
 5 files changed, 109 insertions(+), 109 deletions(-)
qapi/block-core.json +8 -8
index 603eb60..190b4e5 100644
@@ -1780,19 +1780,19 @@
 # Since: 2.0
 ##
 { 'enum': 'BlkdebugEvent', 'prefix': 'BLKDBG',
-  'data': [ 'l1_update', 'l1_grow.alloc_table', 'l1_grow.write_table',
-            'l1_grow.activate_table', 'l2_load', 'l2_update',
-            'l2_update_compressed', 'l2_alloc.cow_read', 'l2_alloc.write',
+  'data': [ 'l1_update', 'l1_grow_alloc_table', 'l1_grow_write_table',
+            'l1_grow_activate_table', 'l2_load', 'l2_update',
+            'l2_update_compressed', 'l2_alloc_cow_read', 'l2_alloc_write',
             'read_aio', 'read_backing_aio', 'read_compressed', 'write_aio',
             'write_compressed', 'vmstate_load', 'vmstate_save', 'cow_read',
             'cow_write', 'reftable_load', 'reftable_grow', 'reftable_update',
             'refblock_load', 'refblock_update', 'refblock_update_part',
-            'refblock_alloc', 'refblock_alloc.hookup', 'refblock_alloc.write',
-            'refblock_alloc.write_blocks', 'refblock_alloc.write_table',
-            'refblock_alloc.switch_table', 'cluster_alloc',
+            'refblock_alloc', 'refblock_alloc_hookup', 'refblock_alloc_write',
+            'refblock_alloc_write_blocks', 'refblock_alloc_write_table',
+            'refblock_alloc_switch_table', 'cluster_alloc',
             'cluster_alloc_bytes', 'cluster_free', 'flush_to_os',
-            'flush_to_disk', 'pwritev_rmw.head', 'pwritev_rmw.after_head',
-            'pwritev_rmw.tail', 'pwritev_rmw.after_tail', 'pwritev',
+            'flush_to_disk', 'pwritev_rmw_head', 'pwritev_rmw_after_head',
+            'pwritev_rmw_tail', 'pwritev_rmw_after_tail', 'pwritev',
             'pwritev_zero', 'pwritev_done', 'empty_image_prepare' ] }
 
 ##
tests/qemu-iotests/026 +9 -9
index 0fc3244..ba1047c 100755
@@ -66,7 +66,7 @@ for event in \
     \
     l2_load \
     l2_update \
-    l2_alloc.write \
+    l2_alloc_write \
     \
     write_aio \
     \
@@ -126,11 +126,11 @@ CLUSTER_SIZE=512
 
 
 for event in \
-    refblock_alloc.hookup \
-    refblock_alloc.write \
-    refblock_alloc.write_blocks \
-    refblock_alloc.write_table \
-    refblock_alloc.switch_table \
+    refblock_alloc_hookup \
+    refblock_alloc_write \
+    refblock_alloc_write_blocks \
+    refblock_alloc_write_table \
+    refblock_alloc_switch_table \
 
 do
 
@@ -170,9 +170,9 @@ CLUSTER_SIZE=1024
 
 
 for event in \
-    l1_grow.alloc_table \
-    l1_grow.write_table \
-    l1_grow.activate_table \
+    l1_grow_alloc_table \
+    l1_grow_write_table \
+    l1_grow_activate_table \
 
 do
 
tests/qemu-iotests/026.out +40 -40

280 lines not on this page · raw message

tests/qemu-iotests/026.out.nocache +40 -40

264 lines not on this page · raw message

tests/qemu-iotests/077 +12 -12
index 42a9085..8a7223f 100755
@@ -63,7 +63,7 @@ EOF
 off=0x1000
 for ev in "head" "after_head" "tail" "after_tail"; do
 cat  <<EOF
-break pwritev_rmw.$ev A
+break pwritev_rmw_$ev A
 aio_write -P 10 $((off + 0x200)) 0x200
 wait_break A
 aio_write -P 11 $((off + 0x400)) 0x200
@@ -76,7 +76,7 @@ done
 
 # Chained dependencies
 cat  <<EOF
-break pwritev_rmw.after_tail A
+break pwritev_rmw_after_tail A
 aio_write -P 10 0x5000 0x200
 wait_break A
 aio_write -P 11 0x5200 0x200
@@ -93,10 +93,10 @@ EOF
 
 # Overlapping multiple requests
 cat  <<EOF
-break pwritev_rmw.after_tail A
+break pwritev_rmw_after_tail A
 aio_write -P 10 0x6000 0x200
 wait_break A
-break pwritev_rmw.after_head B
+break pwritev_rmw_after_head B
 aio_write -P 10 0x7e00 0x200
 wait_break B
 aio_write -P 11 0x6800 0x1000
@@ -107,10 +107,10 @@ aio_flush
 EOF
 
 cat  <<EOF
-break pwritev_rmw.after_tail A
+break pwritev_rmw_after_tail A
 aio_write -P 10 0x8000 0x200
 wait_break A
-break pwritev_rmw.after_head B
+break pwritev_rmw_after_head B
 aio_write -P 10 0x9e00 0x200
 wait_break B
 aio_write -P 11 0x8800 0x1000
@@ -121,11 +121,11 @@ aio_flush
 EOF
 
 cat  <<EOF
-break pwritev_rmw.after_tail A
+break pwritev_rmw_after_tail A
 aio_write -P 10 0xa000 0x200
 wait_break A
 aio_write -P 11 0xa800 0x1000
-break pwritev_rmw.after_head B
+break pwritev_rmw_after_head B
 aio_write -P 10 0xbe00 0x200
 wait_break B
 resume A
@@ -135,11 +135,11 @@ aio_flush
 EOF
 
 cat  <<EOF
-break pwritev_rmw.after_tail A
+break pwritev_rmw_after_tail A
 aio_write -P 10 0xc000 0x200
 wait_break A
 aio_write -P 11 0xc800 0x1000
-break pwritev_rmw.after_head B
+break pwritev_rmw_after_head B
 aio_write -P 10 0xde00 0x200
 wait_break B
 resume B
@@ -150,7 +150,7 @@ EOF
 
 # Only RMW for the tail part
 cat  <<EOF
-break pwritev_rmw.after_tail A
+break pwritev_rmw_after_tail A
 aio_write -P 10 0xe000 0x1800
 wait_break A
 aio_write -P 11 0xf000 0xc00
@@ -163,7 +163,7 @@ cat  <<EOF
 break pwritev A
 aio_write -P 10 0x10000 0x800
 wait_break A
-break pwritev_rmw.after_tail B
+break pwritev_rmw_after_tail B
 aio_write -P 11 0x10000 0x400
 break pwritev_done C
 resume A
Signature · 1 line
2.4.3
#32 lore raw

Re: [Qemu-devel] [PATCH v12 20/36] blkdebug: Avoid '.' in enum values

git-am complains about the trailing whitespace in expected test output. Not this patch's fault. Would be nice to avoid, but not in this series.

author
#33 lore raw

PATCH v12 21/36 qapi: Tighten the regex on valid names

In reply to Eric Blake

We already documented that qapi names should match specific patterns (such as starting with a letter unless it was an enum value or a downstream extension). Tighten that from a suggestion into a hard requirement, which frees up names beginning with a single underscore for qapi internal usage. Also restrict things to avoid 'a__b' or 'a--b' (that is, dash and underscore must not occur consecutively).

Add a new test, reserved-member-underscore, to demonstrate the tighter checking.

---
v12: new patch

---
 docs/qapi-code-gen.txt                            | 19 ++++++++++---------
 scripts/qapi.py                                   | 12 +++++++-----
 tests/Makefile                                    |  1 +
 tests/qapi-schema/reserved-member-underscore.err  |  1 +
 tests/qapi-schema/reserved-member-underscore.exit |  1 +
 tests/qapi-schema/reserved-member-underscore.json |  4 ++++
 tests/qapi-schema/reserved-member-underscore.out  |  0
 7 files changed, 24 insertions(+), 14 deletions(-)
 create mode 100644 tests/qapi-schema/reserved-member-underscore.err
 create mode 100644 tests/qapi-schema/reserved-member-underscore.exit
 create mode 100644 tests/qapi-schema/reserved-member-underscore.json
 create mode 100644 tests/qapi-schema/reserved-member-underscore.out
docs/qapi-code-gen.txt +10 -9
index ceb9a78..ec225d1 100644
@@ -118,17 +118,18 @@ tracking optional fields.
 
 Any name (command, event, type, field, or enum value) beginning with
 "x-" is marked experimental, and may be withdrawn or changed
-incompatibly in a future release.  Downstream vendors may add
-extensions; such extensions should begin with a prefix matching
+incompatibly in a future release.  All names must begin with a letter,
+and contain only ASCII letters, digits, dash, and underscore, where
+dash and underscore cannot occur consecutively.  There are two
+exceptions: enum values may start with a digit, and any extensions
+added by downstream vendors should start with a prefix matching
 "__RFQDN_" (for the reverse-fully-qualified-domain-name of the
 vendor), even if the rest of the name uses dash (example:
-__com.redhat_drive-mirror).  Other than downstream extensions (with
-leading underscore and the use of dots), all names should begin with a
-letter, and contain only ASCII letters, digits, dash, and underscore.
-Names beginning with 'q_' are reserved for the generator: QMP names
-that resemble C keywords or other problematic strings will be munged
-in C to use this prefix.  For example, a field named "default" in
-qapi becomes "q_default" in the generated C code.
+__com.redhat_drive-mirror).  Names beginning with 'q_' are reserved
+for the generator: QMP names that resemble C keywords or other
+problematic strings will be munged in C to use this prefix.  For
+example, a field named "default" in qapi becomes "q_default" in the
+generated C code.
 
 In the rest of this document, usage lines are given for each
 expression type, with literal strings written in lower case and
scripts/qapi.py +7 -5
index 4a30bc0..e6d014b 100644
@@ -353,9 +353,11 @@ def discriminator_find_enum_define(expr):
     return find_enum(discriminator_type)
 
 
-# FIXME should enforce "other than downstream extensions [...], all
-# names should begin with a letter".
-valid_name = re.compile('^[a-zA-Z_][a-zA-Z0-9_.-]*$')
+# Names must be letters, numbers, -, and _.  They must start with letter,
+# except for downstream extensions which must start with __RFQDN_.
+# Dots are only valid in the downstream extension prefix.
+valid_name = re.compile('^(__[a-zA-Z][a-zA-Z0-9.]*_)?'
+                        '[a-zA-Z][a-zA-Z0-9]*([_-][a-zA-Z0-9]+)*$')
 
 
 def check_name(expr_info, source, name, allow_optional=False,
@@ -374,8 +376,8 @@ def check_name(expr_info, source, name, allow_optional=False,
                                 % (source, name))
     # Enum members can start with a digit, because the generated C
     # code always prefixes it with the enum name
-    if enum_member:
-        membername = '_' + membername
+    if enum_member and membername[0].isdigit():
+        membername = 'D' + membername
     # Reserve the entire 'q_' namespace for c_name()
     if not valid_name.match(membername) or \
        c_name(membername, False).startswith('q_'):
tests/Makefile +1 -0
index 1c2c8ee..b70d246 100644
@@ -320,6 +320,7 @@ qapi-schema += reserved-command-q.json
 qapi-schema += reserved-member-has.json
 qapi-schema += reserved-member-q.json
 qapi-schema += reserved-member-u.json
+qapi-schema += reserved-member-underscore.json
 qapi-schema += reserved-type-kind.json
 qapi-schema += reserved-type-list.json
 qapi-schema += returns-alternate.json
tests/qapi-schema/reserved-member-underscore.err new file +1 -0
new file mode 100644
index 0000000..65ff0da
@@ -0,0 +1 @@
+tests/qapi-schema/reserved-member-underscore.json:4: Member of 'data' for struct 'Oops' uses invalid name '_oops'
tests/qapi-schema/reserved-member-underscore.exit new file +1 -0
new file mode 100644
index 0000000..d00491f
@@ -0,0 +1 @@
+1
tests/qapi-schema/reserved-member-underscore.json new file +4 -0
new file mode 100644
index 0000000..4a3a017
@@ -0,0 +1,4 @@
+# C member name collision
+# We reject use of a single leading underscore in all names (names must
+# begin with a letter or a downstream extension double-underscore prefix).
+{ 'struct': 'Oops', 'data': { '_oops': 'str' } }
tests/qapi-schema/reserved-member-underscore.out new file +0 -0
new file mode 100644
index 0000000..e69de29
Signature · 1 line
2.4.3
#34 lore raw

Re: [Qemu-devel] [PATCH v12 21/36] qapi: Tighten the regex on valid names

Quoted 5 lines · Eric Blake

We already documented that qapi names should match specific patterns (such as starting with a letter unless it was an enum value or a downstream extension). Tighten that from a suggestion into a hard requirement, which frees up names beginning with a single underscore for qapi internal usage.

If we care enough about naming conventions to document them, enforcing them makes only sense.

Quoted 3 lines

Also restrict things
to avoid 'a__b' or 'a--b' (that is, dash and underscore must not occur consecutively).

I feel this is entering "foolish names" territory, where things like "IcAnFiNdNaMeSeVeNlEsSrEaDaBlEtHaNStudlyCaps" live. Catching fools automatically is generally futile, they're too creative :)

Let's drop this part.

Quoted 59 lines

Add a new test, reserved-member-underscore, to demonstrate the tighter checking.

---
v12: new patch

---
 docs/qapi-code-gen.txt                            | 19 ++++++++++---------
 scripts/qapi.py                                   | 12 +++++++-----
 tests/Makefile                                    |  1 +
 tests/qapi-schema/reserved-member-underscore.err  |  1 +
 tests/qapi-schema/reserved-member-underscore.exit |  1 +
 tests/qapi-schema/reserved-member-underscore.json |  4 ++++
 tests/qapi-schema/reserved-member-underscore.out  |  0
 7 files changed, 24 insertions(+), 14 deletions(-)
 create mode 100644 tests/qapi-schema/reserved-member-underscore.err
 create mode 100644 tests/qapi-schema/reserved-member-underscore.exit
 create mode 100644 tests/qapi-schema/reserved-member-underscore.json
 create mode 100644 tests/qapi-schema/reserved-member-underscore.out
docs/qapi-code-gen.txt +10 -9
index ceb9a78..ec225d1 100644
@@ -118,17 +118,18 @@ tracking optional fields.
 
 Any name (command, event, type, field, or enum value) beginning with
 "x-" is marked experimental, and may be withdrawn or changed
-incompatibly in a future release.  Downstream vendors may add
-extensions; such extensions should begin with a prefix matching
+incompatibly in a future release.  All names must begin with a letter,
+and contain only ASCII letters, digits, dash, and underscore, where
+dash and underscore cannot occur consecutively.  There are two
+exceptions: enum values may start with a digit, and any extensions
+added by downstream vendors should start with a prefix matching
 "__RFQDN_" (for the reverse-fully-qualified-domain-name of the
 vendor), even if the rest of the name uses dash (example:
-__com.redhat_drive-mirror).  Other than downstream extensions (with
-leading underscore and the use of dots), all names should begin with a
-letter, and contain only ASCII letters, digits, dash, and underscore.
-Names beginning with 'q_' are reserved for the generator: QMP names
-that resemble C keywords or other problematic strings will be munged
-in C to use this prefix.  For example, a field named "default" in
-qapi becomes "q_default" in the generated C code.
+__com.redhat_drive-mirror).  Names beginning with 'q_' are reserved
+for the generator: QMP names that resemble C keywords or other
+problematic strings will be munged in C to use this prefix.  For
+example, a field named "default" in qapi becomes "q_default" in the
+generated C code.
 
 In the rest of this document, usage lines are given for each
 expression type, with literal strings written in lower case and
scripts/qapi.py +0 -0
index 4a30bc0..e6d014b 100644
@@ -353,9 +353,11 @@ def discriminator_find_enum_define(expr):
     return find_enum(discriminator_type)

scripts/qapi.py def discriminator_find_enum_define(expr): · line 360

Quoted 10 lines
scripts/qapi.py +5 -3
 
 
-# FIXME should enforce "other than downstream extensions [...], all
-# names should begin with a letter".
-valid_name = re.compile('^[a-zA-Z_][a-zA-Z0-9_.-]*$')
+# Names must be letters, numbers, -, and _.  They must start with letter,
+# except for downstream extensions which must start with __RFQDN_.
+# Dots are only valid in the downstream extension prefix.
+valid_name = re.compile('^(__[a-zA-Z][a-zA-Z0-9.]*_)?'
+                        '[a-zA-Z][a-zA-Z0-9]*([_-][a-zA-Z0-9]+)*$')

This regexp consists of two parts: optional __RFQDN_ prefix and the name proper.

The latter stays simpler if we don't attempt to catch adjacent [-_].

The former isn't quite right. According to RFC 822 Appendix 1 - Domain Name Syntax Specification:

* We must accept '-' in addition to digits.

* We require RFQDN to start with a letter. This is still a loose match. The tight match is "labels separated by dots", where labels consist of letters, digits and '-', starting with a letter. I wouldn't bother checking first characters are letters at all.

Recommend

valid_name = re.compile('^(__[a-zA-Z0-9.-]+_)?'

                           '[a-zA-Z][a-zA-Z0-9_-]*$')

scripts/qapi.py def check_name(expr_info, source, name, allow_optional=False, · line 379

Quoted 10 lines
scripts/qapi.py +1 -2
 def check_name(expr_info, source, name, allow_optional=False,
@@ -374,8 +376,8 @@ def check_name(expr_info, source, name, allow_optional=False,
                                 % (source, name))
     # Enum members can start with a digit, because the generated C
     # code always prefixes it with the enum name
-    if enum_member:
-        membername = '_' + membername
+    if enum_member and membername[0].isdigit():

What's wrong with the old condition?

Quoted 43 lines
scripts/qapi.py +1 -0
+        membername = 'D' + membername
     # Reserve the entire 'q_' namespace for c_name()
     if not valid_name.match(membername) or \
        c_name(membername, False).startswith('q_'):
tests/Makefile +1 -0
index 1c2c8ee..b70d246 100644
@@ -320,6 +320,7 @@ qapi-schema += reserved-command-q.json
 qapi-schema += reserved-member-has.json
 qapi-schema += reserved-member-q.json
 qapi-schema += reserved-member-u.json
+qapi-schema += reserved-member-underscore.json
 qapi-schema += reserved-type-kind.json
 qapi-schema += reserved-type-list.json
 qapi-schema += returns-alternate.json
tests/qapi-schema/reserved-member-underscore.err new file +1 -0
new file mode 100644
index 0000000..65ff0da
@@ -0,0 +1 @@
+tests/qapi-schema/reserved-member-underscore.json:4: Member of 'data' for struct 'Oops' uses invalid name '_oops'
tests/qapi-schema/reserved-member-underscore.exit new file +1 -0
new file mode 100644
index 0000000..d00491f
@@ -0,0 +1 @@
+1
tests/qapi-schema/reserved-member-underscore.json new file +4 -0
new file mode 100644
index 0000000..4a3a017
@@ -0,0 +1,4 @@
+# C member name collision
+# We reject use of a single leading underscore in all names (names must
+# begin with a letter or a downstream extension double-underscore prefix).
+{ 'struct': 'Oops', 'data': { '_oops': 'str' } }
tests/qapi-schema/reserved-member-underscore.out new file +0 -0
new file mode 100644
index 0000000..e69de29
author
#35 lore raw

Re: [Qemu-devel] [PATCH v12 21/36] qapi: Tighten the regex on valid names

Quoted 11 lines · Markus Armbruster
Quoted 5 lines · Eric Blake

We already documented that qapi names should match specific patterns (such as starting with a letter unless it was an enum value or a downstream extension). Tighten that from a suggestion into a hard requirement, which frees up names beginning with a single underscore for qapi internal usage.

If we care enough about naming conventions to document them, enforcing them makes only sense.

Quoted 6 lines
Quoted 3 lines

Also restrict things
to avoid 'a__b' or 'a--b' (that is, dash and underscore must not occur consecutively).

I feel this is entering "foolish names" territory, where things like "IcAnFiNdNaMeSeVeNlEsSrEaDaBlEtHaNStudlyCaps" live. Catching fools

IhAdToOmUcHfUnReAdInGtHaT

[It's amazing that scholars can ever manage to correctly interpret old written languages, thanks to omitted word breaks (scriptio continua) or omitted vowels (such as in Hebrew)]

Quoted 3 lines

automatically is generally futile, they're too creative :)

Let's drop this part.

Sure, I can do that. I'll post a fixup patch, as it will affect docs too.

Quoted 6 lines
Quoted 6 lines

Add a new test, reserved-member-underscore, to demonstrate the tighter checking.

Quoted 7 lines

docs/qapi-code-gen.txt tracking optional fields. · line 123

Quoted 7 lines

---
v12: new patch
---

docs/qapi-code-gen.txt +3 -0
+incompatibly in a future release.  All names must begin with a letter,
+and contain only ASCII letters, digits, dash, and underscore, where
+dash and underscore cannot occur consecutively.  There are two

Namely, right here.

Quoted 21 lines

scripts/qapi.py def discriminator_find_enum_define(expr): · line 360

Quoted 5 lines
scripts/qapi.py +5 -0
+# Names must be letters, numbers, -, and _.  They must start with letter,
+# except for downstream extensions which must start with __RFQDN_.
+# Dots are only valid in the downstream extension prefix.
+valid_name = re.compile('^(__[a-zA-Z][a-zA-Z0-9.]*_)?'
+                        '[a-zA-Z][a-zA-Z0-9]*([_-][a-zA-Z0-9]+)*$')

This regexp consists of two parts: optional __RFQDN_ prefix and the name proper.

The latter stays simpler if we don't attempt to catch adjacent [-_].

The former isn't quite right. According to RFC 822 Appendix 1 - Domain Name Syntax Specification:

* We must accept '-' in addition to digits.

* We require RFQDN to start with a letter. This is still a loose match. The tight match is "labels separated by dots", where labels consist of letters, digits and '-', starting with a letter. I wouldn't bother checking first characters are letters at all.

Quoted 4 lines

Recommend

   valid_name = re.compile('^(__[a-zA-Z0-9.-]+_)?'
                           '[a-zA-Z][a-zA-Z0-9_-]*$')

Sure, that works for me. It's tighter than what we had before, but looser than what I proposed so that it allows more RFQDN. It potentially lets users confuse us by abusing 'foo__max' or similar, but I'm less worried about that abuse - it's okay to stop the blatantly obvious mistakes without worrying about the corner cases, if it makes the maintenance easier for the cases we do catch.

Quoted 15 lines

scripts/qapi.py def check_name(expr_info, source, name, allow_optional=False, · line 379

Quoted 11 lines

def check_name(expr_info, source, name, allow_optional=False,

scripts/qapi.py +1 -2
@@ -374,8 +376,8 @@ def check_name(expr_info, source, name, allow_optional=False,
                                 % (source, name))
     # Enum members can start with a digit, because the generated C
     # code always prefixes it with the enum name
-    if enum_member:
-        membername = '_' + membername
+    if enum_member and membername[0].isdigit():

What's wrong with the old condition?

Quoted 1 line
scripts/qapi.py +1 -0
+        membername = 'D' + membername

The old code prepended a lone '_', which doesn't work any more with the tighter valid_name regex, so we have to use some other prefix (I picked 'D').

Quoted 3 lines

scripts/qapi.py def check_name(expr_info, source, name, allow_optional=False, · line 383

Quoted 3 lines
scripts/qapi.py +0 -0
     # Reserve the entire 'q_' namespace for c_name()
     if not valid_name.match(membername) or \
        c_name(membername, False).startswith('q_'):

The other thing is that I realized that an enum value of 'q-int' was getting munged to '_q-int' which no longer gets flagged by the c_name() filter. But neither would 'Dq-int' get flagged. So limiting the munging to just enum values that start with a digit (where we know it doesn't start with q) means we don't weaken the second condition.

I guess I need to call that out in the commit message; all the more reason for me to send a fixup.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#36 lore raw

Re: [Qemu-devel] [PATCH v12 21/36] qapi: Tighten the regex on valid names

Quoted 16 lines · Eric Blake
Quoted 15 lines · Markus Armbruster
Quoted 5 lines · Eric Blake

We already documented that qapi names should match specific patterns (such as starting with a letter unless it was an enum value or a downstream extension). Tighten that from a suggestion into a hard requirement, which frees up names beginning with a single underscore for qapi internal usage.

If we care enough about naming conventions to document them, enforcing them makes only sense.

Quoted 3 lines

Also restrict things
to avoid 'a__b' or 'a--b' (that is, dash and underscore must not occur consecutively).

Quoted 8 lines
Quoted 2 lines

I feel this is entering "foolish names" territory, where things like "IcAnFiNdNaMeSeVeNlEsSrEaDaBlEtHaNStudlyCaps" live. Catching fools

IhAdToOmUcHfUnReAdInGtHaT

[It's amazing that scholars can ever manage to correctly interpret old written languages, thanks to omitted word breaks (scriptio continua) or omitted vowels (such as in Hebrew)]

Indeed.

Quoted 79 lines
Quoted 3 lines

automatically is generally futile, they're too creative :)

Let's drop this part.

Sure, I can do that. I'll post a fixup patch, as it will affect docs too.

Quoted 6 lines
Quoted 6 lines

Add a new test, reserved-member-underscore, to demonstrate the tighter checking.

Quoted 7 lines

docs/qapi-code-gen.txt tracking optional fields. · line 123

Quoted 7 lines

---
v12: new patch
---

docs/qapi-code-gen.txt +3 -0
+incompatibly in a future release.  All names must begin with a letter,
+and contain only ASCII letters, digits, dash, and underscore, where
+dash and underscore cannot occur consecutively.  There are two

Namely, right here.

Quoted 21 lines

scripts/qapi.py def discriminator_find_enum_define(expr): · line 360

Quoted 5 lines
scripts/qapi.py +5 -0
+# Names must be letters, numbers, -, and _.  They must start with letter,
+# except for downstream extensions which must start with __RFQDN_.
+# Dots are only valid in the downstream extension prefix.
+valid_name = re.compile('^(__[a-zA-Z][a-zA-Z0-9.]*_)?'
+                        '[a-zA-Z][a-zA-Z0-9]*([_-][a-zA-Z0-9]+)*$')

This regexp consists of two parts: optional __RFQDN_ prefix and the name proper.

The latter stays simpler if we don't attempt to catch adjacent [-_].

The former isn't quite right. According to RFC 822 Appendix 1 - Domain Name Syntax Specification:

* We must accept '-' in addition to digits.

* We require RFQDN to start with a letter. This is still a loose match. The tight match is "labels separated by dots", where labels consist of letters, digits and '-', starting with a letter. I wouldn't bother checking first characters are letters at all.

Quoted 4 lines

Recommend

   valid_name = re.compile('^(__[a-zA-Z0-9.-]+_)?'
                           '[a-zA-Z][a-zA-Z0-9_-]*$')

Sure, that works for me. It's tighter than what we had before, but looser than what I proposed so that it allows more RFQDN. It potentially lets users confuse us by abusing 'foo__max' or similar, but I'm less worried about that abuse - it's okay to stop the blatantly obvious mistakes without worrying about the corner cases, if it makes the maintenance easier for the cases we do catch.

Quoted 15 lines

scripts/qapi.py def check_name(expr_info, source, name, allow_optional=False, · line 379

Quoted 11 lines

def check_name(expr_info, source, name, allow_optional=False,

scripts/qapi.py +1 -2
@@ -374,8 +376,8 @@ def check_name(expr_info, source, name, allow_optional=False,
                                 % (source, name))
     # Enum members can start with a digit, because the generated C
     # code always prefixes it with the enum name
-    if enum_member:
-        membername = '_' + membername
+    if enum_member and membername[0].isdigit():

What's wrong with the old condition?

Quoted 1 line
scripts/qapi.py +1 -0
+        membername = 'D' + membername

The old code prepended a lone '_', which doesn't work any more with the tighter valid_name regex, so we have to use some other prefix (I picked 'D').

Quoted 3 lines

scripts/qapi.py def check_name(expr_info, source, name, allow_optional=False, · line 383

Quoted 3 lines
scripts/qapi.py +0 -0
     # Reserve the entire 'q_' namespace for c_name()
     if not valid_name.match(membername) or \
        c_name(membername, False).startswith('q_'):
Quoted 5 lines

The other thing is that I realized that an enum value of 'q-int' was getting munged to '_q-int' which no longer gets flagged by the c_name() filter. But neither would 'Dq-int' get flagged. So limiting the munging to just enum values that start with a digit (where we know it doesn't start with q) means we don't weaken the second condition.

Aha! Worth a comment, I think.

Quoted 2 lines

I guess I need to call that out in the commit message; all the more reason for me to send a fixup.

Yes, please!

author
#37 lore raw

PATCH 1/1 fixup! qapi: Tighten the regex on valid names

In reply to Eric Blake

[Replace the old commit message with this:]

qapi: Tighten the regex on valid names

We already documented that qapi names should match specific patterns (such as starting with a letter unless it was an enum value or a downstream extension). Tighten that from a suggestion into a hard requirement, which frees up names beginning with a single underscore for qapi internal usage.

The tighter regex doesn't forbid everything insane that a user could provide (for example, a user could name a type 'Foo-lookup' to collide with the generated 'Foo_lookup[]' for an enum 'Foo'), but does a good job at protecting the most obvious uses, and also happens to reserve single leading underscore for later use.

The handling of enum values starting with a digit is tricky:
commit 9fb081e introduced a subtle bug by using c_name() on a munged value, which would allow an enum to include the member 'q-int' in spite of our reservation. Furthermore, munging with a leading '_' would fail our tighter regex. So fix it by only munging for leading digits (which are never ticklish in c_name()) and by using a different prefix (I picked 'D', although any letter should do).

Add new tests, reserved-member-underscore and reserved-enum-q, to demonstrate the tighter checking.

---
 docs/qapi-code-gen.txt                 | 11 +++++------
 scripts/qapi.py                        |  4 ++--
 tests/Makefile                         |  1 +
 tests/qapi-schema/reserved-enum-q.err  |  1 +
 tests/qapi-schema/reserved-enum-q.exit |  1 +
 tests/qapi-schema/reserved-enum-q.json |  4 ++++
 tests/qapi-schema/reserved-enum-q.out  |  0
 7 files changed, 14 insertions(+), 8 deletions(-)
 create mode 100644 tests/qapi-schema/reserved-enum-q.err
 create mode 100644 tests/qapi-schema/reserved-enum-q.exit
 create mode 100644 tests/qapi-schema/reserved-enum-q.json
 create mode 100644 tests/qapi-schema/reserved-enum-q.out
docs/qapi-code-gen.txt +5 -6
index ec225d1..32cc68f 100644
@@ -119,12 +119,11 @@ tracking optional fields.
 Any name (command, event, type, field, or enum value) beginning with
 "x-" is marked experimental, and may be withdrawn or changed
 incompatibly in a future release.  All names must begin with a letter,
-and contain only ASCII letters, digits, dash, and underscore, where
-dash and underscore cannot occur consecutively.  There are two
-exceptions: enum values may start with a digit, and any extensions
-added by downstream vendors should start with a prefix matching
-"__RFQDN_" (for the reverse-fully-qualified-domain-name of the
-vendor), even if the rest of the name uses dash (example:
+and contain only ASCII letters, digits, dash, and underscore.  There
+are two exceptions: enum values may start with a digit, and any
+extensions added by downstream vendors should start with a prefix
+matching "__RFQDN_" (for the reverse-fully-qualified-domain-name of
+the vendor), even if the rest of the name uses dash (example:
 __com.redhat_drive-mirror).  Names beginning with 'q_' are reserved
 for the generator: QMP names that resemble C keywords or other
 problematic strings will be munged in C to use this prefix.  For
scripts/qapi.py +2 -2
index e6d014b..5b6d18d 100644
@@ -356,8 +356,8 @@ def discriminator_find_enum_define(expr):
 # Names must be letters, numbers, -, and _.  They must start with letter,
 # except for downstream extensions which must start with __RFQDN_.
 # Dots are only valid in the downstream extension prefix.
-valid_name = re.compile('^(__[a-zA-Z][a-zA-Z0-9.]*_)?'
-                        '[a-zA-Z][a-zA-Z0-9]*([_-][a-zA-Z0-9]+)*$')
+valid_name = re.compile('^(__[a-zA-Z0-9.-]+_)?'
+                        '[a-zA-Z][a-zA-Z0-9_-]*$')
 
 
 def check_name(expr_info, source, name, allow_optional=False,
tests/Makefile +1 -0
index b70d246..a3f63f5 100644
@@ -317,6 +317,7 @@ qapi-schema += redefined-command.json
 qapi-schema += redefined-event.json
 qapi-schema += redefined-type.json
 qapi-schema += reserved-command-q.json
+qapi-schema += reserved-enum-q.json
 qapi-schema += reserved-member-has.json
 qapi-schema += reserved-member-q.json
 qapi-schema += reserved-member-u.json
tests/qapi-schema/reserved-enum-q.err new file +1 -0
new file mode 100644
index 0000000..e1c3480
@@ -0,0 +1 @@
+tests/qapi-schema/reserved-enum-q.json:4: Member of enum 'Foo' uses invalid name 'q-Unix'
tests/qapi-schema/reserved-enum-q.exit new file +1 -0
new file mode 100644
index 0000000..d00491f
@@ -0,0 +1 @@
+1
tests/qapi-schema/reserved-enum-q.json new file +4 -0
new file mode 100644
index 0000000..3593a76
@@ -0,0 +1,4 @@
+# C entity name collision
+# We reject names like 'q-unix', because they can collide with the mangled
+# name for 'unix' in generated C.
+{ 'enum': 'Foo', 'data': [ 'unix', 'q-Unix' ] }
tests/qapi-schema/reserved-enum-q.out new file +0 -0
new file mode 100644
index 0000000..e69de29
Signature · 1 line
2.4.3
#38 lore raw

Re: [Qemu-devel] [PATCH] fixup! qapi: Tighten the regex on valid names

Quoted 1 line · Eric Blake

[Replace the old commit message with this:]

Fixup patch squashed, commit message replaced.

author
#39 lore raw

Re: [Qemu-devel] [PATCH] fixup! qapi: Tighten the regex on valid names

Quoted 5 lines · Markus Armbruster
Quoted 1 line · Eric Blake

[Replace the old commit message with this:]

Fixup patch squashed, commit message replaced.

The version of the patch currently on qapi-next (id 22c38fe) missed the addition of the new tests/qapi-schema/reserved-enum-q.* files but caught the Makefile change, which means it fails 'make check-qapi-schema'.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#40 lore raw

Re: [Qemu-devel] [PATCH] fixup! qapi: Tighten the regex on valid names

Quoted 10 lines · Eric Blake
Quoted 5 lines · Markus Armbruster
Quoted 1 line · Eric Blake

[Replace the old commit message with this:]

Fixup patch squashed, commit message replaced.

The version of the patch currently on qapi-next (id 22c38fe) missed the addition of the new tests/qapi-schema/reserved-enum-q.* files but caught the Makefile change, which means it fails 'make check-qapi-schema'.

I suck at squashing patches. Fixed & Thanks!

author
#41 lore raw

PATCH v12 22/36 qapi: Don't let implicit enum MAX member collide

In reply to Eric Blake

Now that we guarantee the user doesn't have any enum values beginning with a single underscore, we can use that for our own purposes. Renaming ENUM_MAX to ENUM__MAX makes it obvious that the sentinel is generated.

This patch was mostly generated by applying a temporary patch:

|diff --git a/scripts/qapi.py b/scripts/qapi.py
|index e6d014b..b862ec9 100644
|--- a/scripts/qapi.py
|+++ b/scripts/qapi.py
|@@ -1570,6 +1570,7 @@ const char *const %(c_name)s_lookup[] = {
|     max_index = c_enum_const(name, 'MAX', prefix)
|     ret += mcgen('''
|     [%(max_index)s] = NULL,
|+// %(max_index)s
| };
| ''',
|                max_index=max_index)

then running:

$ cat qapi-{types,event}.c tests/test-qapi-types.c |
    sed -n 's,^// \(.*\)MAX,s|\1MAX|\1_MAX|g,p' > list
$ git grep -l _MAX | xargs sed -i -f list

The only things not generated are the changes in scripts/qapi.py.

---
v12: new patch

---
 block/blkdebug.c                   | 10 +++++-----
 block/parallels.c                  |  4 ++--
 block/qcow2.c                      |  2 +-
 block/quorum.c                     |  2 +-
 block/raw-posix.c                  |  2 +-
 blockdev.c                         |  2 +-
 docs/qapi-code-gen.txt             |  4 ++--
 hmp.c                              | 14 +++++++-------
 hw/char/escc.c                     |  2 +-
 hw/i386/pc_piix.c                  |  2 +-
 hw/i386/pc_q35.c                   |  2 +-
 hw/input/hid.c                     |  2 +-
 hw/input/ps2.c                     |  2 +-
 hw/input/virtio-input-hid.c        |  8 ++++----
 include/migration/migration.h      |  4 ++--
 migration/migration.c              |  4 ++--
 monitor.c                          | 14 +++++++-------
 net/net.c                          |  4 ++--
 qemu-nbd.c                         |  2 +-
 replay/replay-input.c              |  8 ++++----
 scripts/qapi.py                    |  6 +++---
 tests/test-qmp-output-visitor.c    |  6 +++---
 tests/test-string-output-visitor.c |  4 ++--
 tpm.c                              | 10 +++++-----
 ui/cocoa.m                         |  2 +-
 ui/console.c                       |  2 +-
 ui/input-keymap.c                  |  4 ++--
 ui/input-legacy.c                  |  6 +++---
 ui/input.c                         |  6 +++---
 ui/sdl.c                           |  2 +-
 ui/sdl2.c                          |  2 +-
 ui/spice-input.c                   |  2 +-
 ui/vnc.c                           |  2 +-
 vl.c                               | 18 +++++++++---------
 34 files changed, 83 insertions(+), 83 deletions(-)
block/blkdebug.c +5 -5
index 76805a6..7cc30d5 100644
@@ -35,7 +35,7 @@ typedef struct BDRVBlkdebugState {
     int state;
     int new_state;
 
-    QLIST_HEAD(, BlkdebugRule) rules[BLKDBG_MAX];
+    QLIST_HEAD(, BlkdebugRule) rules[BLKDBG__MAX];
     QSIMPLEQ_HEAD(, BlkdebugRule) active_rules;
     QLIST_HEAD(, BlkdebugSuspendedReq) suspended_reqs;
 } BDRVBlkdebugState;
@@ -146,7 +146,7 @@ static int get_event_by_name(const char *name, BlkdebugEvent *event)
 {
     int i;
 
-    for (i = 0; i < BLKDBG_MAX; i++) {
+    for (i = 0; i < BLKDBG__MAX; i++) {
         if (!strcmp(BlkdebugEvent_lookup[i], name)) {
             *event = i;
             return 0;
@@ -506,7 +506,7 @@ static void blkdebug_close(BlockDriverState *bs)
     BlkdebugRule *rule, *next;
     int i;
 
-    for (i = 0; i < BLKDBG_MAX; i++) {
+    for (i = 0; i < BLKDBG__MAX; i++) {
         QLIST_FOREACH_SAFE(rule, &s->rules[i], next, next) {
             remove_rule(rule);
         }
@@ -571,7 +571,7 @@ static void blkdebug_debug_event(BlockDriverState *bs, BlkdebugEvent event)
     struct BlkdebugRule *rule, *next;
     bool injected;
 
-    assert((int)event >= 0 && event < BLKDBG_MAX);
+    assert((int)event >= 0 && event < BLKDBG__MAX);
 
     injected = false;
     s->new_state = s->state;
@@ -628,7 +628,7 @@ static int blkdebug_debug_remove_breakpoint(BlockDriverState *bs,
     BlkdebugRule *rule, *next;
     int i, ret = -ENOENT;
 
-    for (i = 0; i < BLKDBG_MAX; i++) {
+    for (i = 0; i < BLKDBG__MAX; i++) {
         QLIST_FOREACH_SAFE(rule, &s->rules[i], next, next) {
             if (rule->action == ACTION_SUSPEND &&
                 !strcmp(rule->options.suspend.tag, tag)) {
block/parallels.c +2 -2
index 4f79293..eb93321 100644
@@ -61,7 +61,7 @@ typedef struct ParallelsHeader {
 typedef enum ParallelsPreallocMode {
     PRL_PREALLOC_MODE_FALLOCATE = 0,
     PRL_PREALLOC_MODE_TRUNCATE = 1,
-    PRL_PREALLOC_MODE_MAX = 2,
+    PRL_PREALLOC_MODE__MAX = 2,
 } ParallelsPreallocMode;
 
 static const char *prealloc_mode_lookup[] = {
@@ -660,7 +660,7 @@ static int parallels_open(BlockDriverState *bs, QDict *options, int flags,
     s->prealloc_size = MAX(s->tracks, s->prealloc_size >> BDRV_SECTOR_BITS);
     buf = qemu_opt_get_del(opts, PARALLELS_OPT_PREALLOC_MODE);
     s->prealloc_mode = qapi_enum_parse(prealloc_mode_lookup, buf,
-            PRL_PREALLOC_MODE_MAX, PRL_PREALLOC_MODE_FALLOCATE, &local_err);
+            PRL_PREALLOC_MODE__MAX, PRL_PREALLOC_MODE_FALLOCATE, &local_err);
     g_free(buf);
     if (local_err != NULL) {
         goto fail_options;
block/qcow2.c +1 -1
index 88f56c8..5b59fa3 100644
@@ -2269,7 +2269,7 @@ static int qcow2_create(const char *filename, QemuOpts *opts, Error **errp)
                                          DEFAULT_CLUSTER_SIZE);
     buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
     prealloc = qapi_enum_parse(PreallocMode_lookup, buf,
-                               PREALLOC_MODE_MAX, PREALLOC_MODE_OFF,
+                               PREALLOC_MODE__MAX, PREALLOC_MODE_OFF,
                                &local_err);
     if (local_err) {
         error_propagate(errp, local_err);
block/quorum.c +1 -1
index b9ba028..d162459 100644
@@ -847,7 +847,7 @@ static int parse_read_pattern(const char *opt)
         return QUORUM_READ_PATTERN_QUORUM;
     }
 
-    for (i = 0; i < QUORUM_READ_PATTERN_MAX; i++) {
+    for (i = 0; i < QUORUM_READ_PATTERN__MAX; i++) {
         if (!strcmp(opt, QuorumReadPattern_lookup[i])) {
             return i;
         }
block/raw-posix.c +1 -1
index aec9ec6..c7f1327 100644
@@ -1636,7 +1636,7 @@ static int raw_create(const char *filename, QemuOpts *opts, Error **errp)
     nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false);
     buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
     prealloc = qapi_enum_parse(PreallocMode_lookup, buf,
-                               PREALLOC_MODE_MAX, PREALLOC_MODE_OFF,
+                               PREALLOC_MODE__MAX, PREALLOC_MODE_OFF,
                                &local_err);
     g_free(buf);
     if (local_err) {
blockdev.c +1 -1
index 917ae06..5649a55 100644
@@ -454,7 +454,7 @@ static void extract_common_blockdev_options(QemuOpts *opts, int *bdrv_flags,
         *detect_zeroes =
             qapi_enum_parse(BlockdevDetectZeroesOptions_lookup,
                             qemu_opt_get(opts, "detect-zeroes"),
-                            BLOCKDEV_DETECT_ZEROES_OPTIONS_MAX,
+                            BLOCKDEV_DETECT_ZEROES_OPTIONS__MAX,
                             BLOCKDEV_DETECT_ZEROES_OPTIONS_OFF,
                             &local_error);
         if (local_error) {
docs/qapi-code-gen.txt +2 -2
index ec225d1..b01a691 100644
@@ -1054,7 +1054,7 @@ Example:
 
     const char *const example_QAPIEvent_lookup[] = {
         [EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
-        [EXAMPLE_QAPI_EVENT_MAX] = NULL,
+        [EXAMPLE_QAPI_EVENT__MAX] = NULL,
     };
     $ cat qapi-generated/example-qapi-event.h
 [Uninteresting stuff omitted...]
@@ -1071,7 +1071,7 @@ Example:
 
     typedef enum example_QAPIEvent {
         EXAMPLE_QAPI_EVENT_MY_EVENT = 0,
-        EXAMPLE_QAPI_EVENT_MAX = 1,
+        EXAMPLE_QAPI_EVENT__MAX = 1,
     } example_QAPIEvent;
 
     extern const char *const example_QAPIEvent_lookup[];
hmp.c +7 -7
index 2140605..1b402c4 100644
@@ -855,7 +855,7 @@ void hmp_info_tpm(Monitor *mon, const QDict *qdict)
                            tpo->has_cancel_path ? ",cancel-path=" : "",
                            tpo->has_cancel_path ? tpo->cancel_path : "");
             break;
-        case TPM_TYPE_OPTIONS_KIND_MAX:
+        case TPM_TYPE_OPTIONS_KIND__MAX:
             break;
         }
         monitor_printf(mon, "\n");
@@ -1203,7 +1203,7 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
     MigrationCapabilityStatusList *caps = g_malloc0(sizeof(*caps));
     int i;
 
-    for (i = 0; i < MIGRATION_CAPABILITY_MAX; i++) {
+    for (i = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
         if (strcmp(cap, MigrationCapability_lookup[i]) == 0) {
             caps->value = g_malloc0(sizeof(*caps->value));
             caps->value->capability = i;
@@ -1214,7 +1214,7 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
         }
     }
 
-    if (i == MIGRATION_CAPABILITY_MAX) {
+    if (i == MIGRATION_CAPABILITY__MAX) {
         error_setg(&err, QERR_INVALID_PARAMETER, cap);
     }
 
@@ -1239,7 +1239,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
     bool has_x_cpu_throttle_increment = false;
     int i;
 
-    for (i = 0; i < MIGRATION_PARAMETER_MAX; i++) {
+    for (i = 0; i < MIGRATION_PARAMETER__MAX; i++) {
         if (strcmp(param, MigrationParameter_lookup[i]) == 0) {
             switch (i) {
             case MIGRATION_PARAMETER_COMPRESS_LEVEL:
@@ -1268,7 +1268,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
         }
     }
 
-    if (i == MIGRATION_PARAMETER_MAX) {
+    if (i == MIGRATION_PARAMETER__MAX) {
         error_setg(&err, QERR_INVALID_PARAMETER, param);
     }
 
@@ -1368,7 +1368,7 @@ void hmp_change(Monitor *mon, const QDict *qdict)
         if (read_only) {
             read_only_mode =
                 qapi_enum_parse(BlockdevChangeReadOnlyMode_lookup,
-                                read_only, BLOCKDEV_CHANGE_READ_ONLY_MODE_MAX,
+                                read_only, BLOCKDEV_CHANGE_READ_ONLY_MODE__MAX,
                                 BLOCKDEV_CHANGE_READ_ONLY_MODE_RETAIN, &err);
             if (err) {
                 hmp_handle_error(mon, &err);
@@ -1771,7 +1771,7 @@ void hmp_sendkey(Monitor *mon, const QDict *qdict)
             keylist->value->u.number = value;
         } else {
             int idx = index_from_key(keyname_buf);
-            if (idx == Q_KEY_CODE_MAX) {
+            if (idx == Q_KEY_CODE__MAX) {
                 goto err_out;
             }
             keylist->value->type = KEY_VALUE_KIND_QCODE;
hw/char/escc.c +1 -1
index c9840e1..b351214 100644
@@ -714,7 +714,7 @@ MemoryRegion *escc_init(hwaddr base, qemu_irq irqA, qemu_irq irqB,
     return &d->mmio;
 }
 
-static const uint8_t qcode_to_keycode[Q_KEY_CODE_MAX] = {
+static const uint8_t qcode_to_keycode[Q_KEY_CODE__MAX] = {
     [Q_KEY_CODE_SHIFT]         = 99,
     [Q_KEY_CODE_SHIFT_R]       = 110,
     [Q_KEY_CODE_ALT]           = 19,
hw/i386/pc_piix.c +1 -1
index 07d0baa..7354add 100644
@@ -225,7 +225,7 @@ static void pc_init1(MachineState *machine,
 
     pc_vga_init(isa_bus, pci_enabled ? pci_bus : NULL);
 
-    assert(pcms->vmport != ON_OFF_AUTO_MAX);
+    assert(pcms->vmport != ON_OFF_AUTO__MAX);
     if (pcms->vmport == ON_OFF_AUTO_AUTO) {
         pcms->vmport = xen_enabled() ? ON_OFF_AUTO_OFF : ON_OFF_AUTO_ON;
     }
hw/i386/pc_q35.c +1 -1
index 0fdae09..1061663 100644
@@ -233,7 +233,7 @@ static void pc_q35_init(MachineState *machine)
 
     pc_register_ferr_irq(gsi[13]);
 
-    assert(pcms->vmport != ON_OFF_AUTO_MAX);
+    assert(pcms->vmport != ON_OFF_AUTO__MAX);
     if (pcms->vmport == ON_OFF_AUTO_AUTO) {
         pcms->vmport = xen_enabled() ? ON_OFF_AUTO_OFF : ON_OFF_AUTO_ON;
     }
hw/input/hid.c +1 -1
index e39269f..12075c9 100644
@@ -108,7 +108,7 @@ void hid_set_next_idle(HIDState *hs)
 static void hid_pointer_event(DeviceState *dev, QemuConsole *src,
                               InputEvent *evt)
 {
-    static const int bmap[INPUT_BUTTON_MAX] = {
+    static const int bmap[INPUT_BUTTON__MAX] = {
         [INPUT_BUTTON_LEFT]   = 0x01,
         [INPUT_BUTTON_RIGHT]  = 0x02,
         [INPUT_BUTTON_MIDDLE] = 0x04,
hw/input/ps2.c +1 -1
index 3d6d496..9096d21 100644
@@ -382,7 +382,7 @@ static void ps2_mouse_send_packet(PS2MouseState *s)
 static void ps2_mouse_event(DeviceState *dev, QemuConsole *src,
                             InputEvent *evt)
 {
-    static const int bmap[INPUT_BUTTON_MAX] = {
+    static const int bmap[INPUT_BUTTON__MAX] = {
         [INPUT_BUTTON_LEFT]   = MOUSE_EVENT_LBUTTON,
         [INPUT_BUTTON_MIDDLE] = MOUSE_EVENT_MBUTTON,
         [INPUT_BUTTON_RIGHT]  = MOUSE_EVENT_RBUTTON,
hw/input/virtio-input-hid.c +4 -4
index bdd479c..5d00a03 100644
@@ -21,7 +21,7 @@
 
 /* ----------------------------------------------------------------- */
 
-static const unsigned int keymap_qcode[Q_KEY_CODE_MAX] = {
+static const unsigned int keymap_qcode[Q_KEY_CODE__MAX] = {
     [Q_KEY_CODE_ESC]                 = KEY_ESC,
     [Q_KEY_CODE_1]                   = KEY_1,
     [Q_KEY_CODE_2]                   = KEY_2,
@@ -138,7 +138,7 @@ static const unsigned int keymap_qcode[Q_KEY_CODE_MAX] = {
     [Q_KEY_CODE_MENU]                = KEY_MENU,
 };
 
-static const unsigned int keymap_button[INPUT_BUTTON_MAX] = {
+static const unsigned int keymap_button[INPUT_BUTTON__MAX] = {
     [INPUT_BUTTON_LEFT]              = BTN_LEFT,
     [INPUT_BUTTON_RIGHT]             = BTN_RIGHT,
     [INPUT_BUTTON_MIDDLE]            = BTN_MIDDLE,
@@ -146,12 +146,12 @@ static const unsigned int keymap_button[INPUT_BUTTON_MAX] = {
     [INPUT_BUTTON_WHEEL_DOWN]        = BTN_GEAR_DOWN,
 };
 
-static const unsigned int axismap_rel[INPUT_AXIS_MAX] = {
+static const unsigned int axismap_rel[INPUT_AXIS__MAX] = {
     [INPUT_AXIS_X]                   = REL_X,
     [INPUT_AXIS_Y]                   = REL_Y,
 };
 
-static const unsigned int axismap_abs[INPUT_AXIS_MAX] = {
+static const unsigned int axismap_abs[INPUT_AXIS__MAX] = {
     [INPUT_AXIS_X]                   = ABS_X,
     [INPUT_AXIS_Y]                   = ABS_Y,
 };
include/migration/migration.h +2 -2
index fd018b7..d9494b8 100644
@@ -133,7 +133,7 @@ struct MigrationState
     QemuThread thread;
     QEMUBH *cleanup_bh;
     QEMUFile *file;
-    int parameters[MIGRATION_PARAMETER_MAX];
+    int parameters[MIGRATION_PARAMETER__MAX];
 
     int state;
     MigrationParams params;
@@ -151,7 +151,7 @@ struct MigrationState
     int64_t expected_downtime;
     int64_t dirty_pages_rate;
     int64_t dirty_bytes_rate;
-    bool enabled_capabilities[MIGRATION_CAPABILITY_MAX];
+    bool enabled_capabilities[MIGRATION_CAPABILITY__MAX];
     int64_t xbzrle_cache_size;
     int64_t setup_time;
     int64_t dirty_sync_count;
migration/migration.c +2 -2
index 7e4e27b..7008890 100644
@@ -202,7 +202,7 @@ static int global_state_post_load(void *opaque, int version_id)
     s->received = true;
     trace_migrate_global_state_post_load(runstate);
 
-    r = qapi_enum_parse(RunState_lookup, runstate, RUN_STATE_MAX,
+    r = qapi_enum_parse(RunState_lookup, runstate, RUN_STATE__MAX,
                                 -1, &local_err);
 
     if (r == -1) {
@@ -479,7 +479,7 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)
     int i;
 
     caps = NULL; /* silence compiler warning */
-    for (i = 0; i < MIGRATION_CAPABILITY_MAX; i++) {
+    for (i = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
         if (head == NULL) {
             head = g_malloc0(sizeof(*caps));
             caps = head;
monitor.c +7 -7
index e4cf34e..3a0df08 100644
@@ -441,7 +441,7 @@ static void monitor_protocol_emitter(Monitor *mon, QObject *data,
 }
 
 
-static MonitorQAPIEventConf monitor_qapi_event_conf[QAPI_EVENT_MAX] = {
+static MonitorQAPIEventConf monitor_qapi_event_conf[QAPI_EVENT__MAX] = {
     /* Limit guest-triggerable events to 1 per second */
     [QAPI_EVENT_RTC_CHANGE]        = { 1000 * SCALE_MS },
     [QAPI_EVENT_WATCHDOG]          = { 1000 * SCALE_MS },
@@ -481,7 +481,7 @@ monitor_qapi_event_queue(QAPIEvent event, QDict *qdict, Error **errp)
     MonitorQAPIEventConf *evconf;
     MonitorQAPIEventState *evstate;
 
-    assert(event < QAPI_EVENT_MAX);
+    assert(event < QAPI_EVENT__MAX);
     evconf = &monitor_qapi_event_conf[event];
     trace_monitor_protocol_event_queue(event, qdict, evconf->rate);
 
@@ -946,7 +946,7 @@ EventInfoList *qmp_query_events(Error **errp)
     EventInfoList *info, *ev_list = NULL;
     QAPIEvent e;
 
-    for (e = 0 ; e < QAPI_EVENT_MAX ; e++) {
+    for (e = 0 ; e < QAPI_EVENT__MAX ; e++) {
         const char *event_name = QAPIEvent_lookup[e];
         assert(event_name != NULL);
         info = g_malloc0(sizeof(*info));
@@ -1386,7 +1386,7 @@ static void hmp_mouse_move(Monitor *mon, const QDict *qdict)
 
 static void hmp_mouse_button(Monitor *mon, const QDict *qdict)
 {
-    static uint32_t bmap[INPUT_BUTTON_MAX] = {
+    static uint32_t bmap[INPUT_BUTTON__MAX] = {
         [INPUT_BUTTON_LEFT]       = MOUSE_EVENT_LBUTTON,
         [INPUT_BUTTON_MIDDLE]     = MOUSE_EVENT_MBUTTON,
         [INPUT_BUTTON_RIGHT]      = MOUSE_EVENT_RBUTTON,
@@ -3217,7 +3217,7 @@ void sendkey_completion(ReadLineState *rs, int nb_args, const char *str)
     }
     len = strlen(str);
     readline_set_completion_index(rs, len);
-    for (i = 0; i < Q_KEY_CODE_MAX; i++) {
+    for (i = 0; i < Q_KEY_CODE__MAX; i++) {
         if (!strncmp(str, QKeyCode_lookup[i], len)) {
             readline_add_completion(rs, QKeyCode_lookup[i]);
         }
@@ -3316,7 +3316,7 @@ void migrate_set_capability_completion(ReadLineState *rs, int nb_args,
     readline_set_completion_index(rs, len);
     if (nb_args == 2) {
         int i;
-        for (i = 0; i < MIGRATION_CAPABILITY_MAX; i++) {
+        for (i = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
             const char *name = MigrationCapability_lookup[i];
             if (!strncmp(str, name, len)) {
                 readline_add_completion(rs, name);
@@ -3337,7 +3337,7 @@ void migrate_set_parameter_completion(ReadLineState *rs, int nb_args,
     readline_set_completion_index(rs, len);
     if (nb_args == 2) {
         int i;
-        for (i = 0; i < MIGRATION_PARAMETER_MAX; i++) {
+        for (i = 0; i < MIGRATION_PARAMETER__MAX; i++) {
             const char *name = MigrationParameter_lookup[i];
             if (!strncmp(str, name, len)) {
                 readline_add_completion(rs, name);
net/net.c +2 -2
index ade6051..87dd356 100644
@@ -943,7 +943,7 @@ static int net_init_nic(const NetClientOptions *opts, const char *name,
 }
 
 
-static int (* const net_client_init_fun[NET_CLIENT_OPTIONS_KIND_MAX])(
+static int (* const net_client_init_fun[NET_CLIENT_OPTIONS_KIND__MAX])(
     const NetClientOptions *opts,
     const char *name,
     NetClientState *peer, Error **errp) = {
@@ -1296,7 +1296,7 @@ void qmp_set_link(const char *name, bool up, Error **errp)
     int queues, i;
 
     queues = qemu_find_net_clients_except(name, ncs,
-                                          NET_CLIENT_OPTIONS_KIND_MAX,
+                                          NET_CLIENT_OPTIONS_KIND__MAX,
                                           MAX_QUEUE_NUM);
 
     if (queues == 0) {
qemu-nbd.c +1 -1
index 3afec76..65dc30c 100644
@@ -487,7 +487,7 @@ int main(int argc, char **argv)
             detect_zeroes =
                 qapi_enum_parse(BlockdevDetectZeroesOptions_lookup,
                                 optarg,
-                                BLOCKDEV_DETECT_ZEROES_OPTIONS_MAX,
+                                BLOCKDEV_DETECT_ZEROES_OPTIONS__MAX,
                                 BLOCKDEV_DETECT_ZEROES_OPTIONS_OFF,
                                 &local_err);
             if (local_err) {
replay/replay-input.c +4 -4
index 9879895..bc15f59 100644
@@ -61,7 +61,7 @@ void replay_save_input_event(InputEvent *evt)
             replay_put_dword(evt->u.key->key->u.qcode);
             replay_put_byte(evt->u.key->down);
             break;
-        case KEY_VALUE_KIND_MAX:
+        case KEY_VALUE_KIND__MAX:
             /* keep gcc happy */
             break;
         }
@@ -78,7 +78,7 @@ void replay_save_input_event(InputEvent *evt)
         replay_put_dword(evt->u.abs->axis);
         replay_put_qword(evt->u.abs->value);
         break;
-    case INPUT_EVENT_KIND_MAX:
+    case INPUT_EVENT_KIND__MAX:
         /* keep gcc happy */
         break;
     }
@@ -109,7 +109,7 @@ InputEvent *replay_read_input_event(void)
             evt.u.key->key->u.qcode = (QKeyCode)replay_get_dword();
             evt.u.key->down = replay_get_byte();
             break;
-        case KEY_VALUE_KIND_MAX:
+        case KEY_VALUE_KIND__MAX:
             /* keep gcc happy */
             break;
         }
@@ -129,7 +129,7 @@ InputEvent *replay_read_input_event(void)
         evt.u.abs->axis = (InputAxis)replay_get_dword();
         evt.u.abs->value = replay_get_qword();
         break;
-    case INPUT_EVENT_KIND_MAX:
+    case INPUT_EVENT_KIND__MAX:
         /* keep gcc happy */
         break;
     }
scripts/qapi.py +3 -3
index e6d014b..aa15adb 100644
@@ -918,7 +918,7 @@ class QAPISchemaEnumType(QAPISchemaType):
         return c_name(self.name)
 
     def c_null(self):
-        return c_enum_const(self.name, (self.values + ['MAX'])[0],
+        return c_enum_const(self.name, (self.values + ['_MAX'])[0],
                             self.prefix)
 
     def json_type(self):
@@ -1567,7 +1567,7 @@ const char *const %(c_name)s_lookup[] = {
 ''',
                      index=index, value=value)
 
-    max_index = c_enum_const(name, 'MAX', prefix)
+    max_index = c_enum_const(name, '_MAX', prefix)
     ret += mcgen('''
     [%(max_index)s] = NULL,
 };
@@ -1578,7 +1578,7 @@ const char *const %(c_name)s_lookup[] = {
 
 def gen_enum(name, values, prefix=None):
     # append automatically generated _MAX value
-    enum_values = values + ['MAX']
+    enum_values = values + ['_MAX']
 
     ret = mcgen('''
 
tests/test-qmp-output-visitor.c +3 -3
index 0d0c859..4196e66 100644
@@ -128,7 +128,7 @@ static void test_visitor_out_enum(TestOutputVisitorData *data,
     QObject *obj;
     EnumOne i;
 
-    for (i = 0; i < ENUM_ONE_MAX; i++) {
+    for (i = 0; i < ENUM_ONE__MAX; i++) {
         visit_type_EnumOne(data->ov, &i, "unused", &error_abort);
 
         obj = qmp_output_get_qobject(data->qov);
@@ -143,7 +143,7 @@ static void test_visitor_out_enum(TestOutputVisitorData *data,
 static void test_visitor_out_enum_errors(TestOutputVisitorData *data,
                                          const void *unused)
 {
-    EnumOne i, bad_values[] = { ENUM_ONE_MAX, -1 };
+    EnumOne i, bad_values[] = { ENUM_ONE__MAX, -1 };
     Error *err;
 
     for (i = 0; i < ARRAY_SIZE(bad_values) ; i++) {
@@ -247,7 +247,7 @@ static void test_visitor_out_struct_nested(TestOutputVisitorData *data,
 static void test_visitor_out_struct_errors(TestOutputVisitorData *data,
                                            const void *unused)
 {
-    EnumOne bad_values[] = { ENUM_ONE_MAX, -1 };
+    EnumOne bad_values[] = { ENUM_ONE__MAX, -1 };
     UserDefOne u = {0};
     UserDefOne *pu = &u;
     Error *err;
tests/test-string-output-visitor.c +2 -2
index fd5e67b..9585252 100644
@@ -194,7 +194,7 @@ static void test_visitor_out_enum(TestOutputVisitorData *data,
     char *str;
     EnumOne i;
 
-    for (i = 0; i < ENUM_ONE_MAX; i++) {
+    for (i = 0; i < ENUM_ONE__MAX; i++) {
         char *str_human;
 
         visit_type_EnumOne(data->ov, &i, "unused", &err);
@@ -217,7 +217,7 @@ static void test_visitor_out_enum(TestOutputVisitorData *data,
 static void test_visitor_out_enum_errors(TestOutputVisitorData *data,
                                          const void *unused)
 {
-    EnumOne i, bad_values[] = { ENUM_ONE_MAX, -1 };
+    EnumOne i, bad_values[] = { ENUM_ONE__MAX, -1 };
     Error *err;
 
     for (i = 0; i < ARRAY_SIZE(bad_values) ; i++) {
tpm.c +5 -5
index f2c59d1..0a3e3d5 100644
@@ -32,7 +32,7 @@ static TPMDriverOps const *be_drivers[TPM_MAX_DRIVERS] = {
 };
 
 static enum TpmModel tpm_models[TPM_MAX_MODELS] = {
-    TPM_MODEL_MAX,
+    TPM_MODEL__MAX,
 };
 
 int tpm_register_model(enum TpmModel model)
@@ -40,7 +40,7 @@ int tpm_register_model(enum TpmModel model)
     int i;
 
     for (i = 0; i < TPM_MAX_MODELS; i++) {
-        if (tpm_models[i] == TPM_MODEL_MAX) {
+        if (tpm_models[i] == TPM_MODEL__MAX) {
             tpm_models[i] = model;
             return 0;
         }
@@ -272,7 +272,7 @@ static TPMInfo *qmp_query_tpm_inst(TPMBackend *drv)
             tpo->has_cancel_path = true;
         }
         break;
-    case TPM_TYPE_MAX:
+    case TPM_TYPE__MAX:
         break;
     }
 
@@ -311,7 +311,7 @@ TpmTypeList *qmp_query_tpm_types(Error **errp)
     unsigned int i = 0;
     TpmTypeList *head = NULL, *prev = NULL, *cur_item;
 
-    for (i = 0; i < TPM_TYPE_MAX; i++) {
+    for (i = 0; i < TPM_TYPE__MAX; i++) {
         if (!tpm_driver_find_by_type(i)) {
             continue;
         }
@@ -335,7 +335,7 @@ TpmModelList *qmp_query_tpm_models(Error **errp)
     unsigned int i = 0;
     TpmModelList *head = NULL, *prev = NULL, *cur_item;
 
-    for (i = 0; i < TPM_MODEL_MAX; i++) {
+    for (i = 0; i < TPM_MODEL__MAX; i++) {
         if (!tpm_model_is_registered(i)) {
             continue;
         }
ui/cocoa.m +1 -1
index 1554331..5a0a2cf 100644
@@ -725,7 +725,7 @@ QemuCocoaView *cocoaView;
 
     if (mouse_event) {
         if (last_buttons != buttons) {
-            static uint32_t bmap[INPUT_BUTTON_MAX] = {
+            static uint32_t bmap[INPUT_BUTTON__MAX] = {
                 [INPUT_BUTTON_LEFT]       = MOUSE_EVENT_LBUTTON,
                 [INPUT_BUTTON_MIDDLE]     = MOUSE_EVENT_MBUTTON,
                 [INPUT_BUTTON_RIGHT]      = MOUSE_EVENT_RBUTTON,
ui/console.c +1 -1
index 745c354..27a2cdc 100644
@@ -1108,7 +1108,7 @@ void kbd_put_keysym_console(QemuConsole *s, int keysym)
     }
 }
 
-static const int qcode_to_keysym[Q_KEY_CODE_MAX] = {
+static const int qcode_to_keysym[Q_KEY_CODE__MAX] = {
     [Q_KEY_CODE_UP]     = QEMU_KEY_UP,
     [Q_KEY_CODE_DOWN]   = QEMU_KEY_DOWN,
     [Q_KEY_CODE_RIGHT]  = QEMU_KEY_RIGHT,
ui/input-keymap.c +2 -2
index d36be4b..63d71d2 100644
@@ -132,7 +132,7 @@ static const int qcode_to_number[] = {
     [Q_KEY_CODE_RO] = 0x73,
     [Q_KEY_CODE_KP_COMMA] = 0x7e,
 
-    [Q_KEY_CODE_MAX] = 0,
+    [Q_KEY_CODE__MAX] = 0,
 };
 
 static int number_to_qcode[0x100];
@@ -154,7 +154,7 @@ int qemu_input_key_number_to_qcode(uint8_t nr)
     if (first) {
         int qcode, number;
         first = false;
-        for (qcode = 0; qcode < Q_KEY_CODE_MAX; qcode++) {
+        for (qcode = 0; qcode < Q_KEY_CODE__MAX; qcode++) {
             number = qcode_to_number[qcode];
             assert(number < ARRAY_SIZE(number_to_qcode));
             number_to_qcode[number] = qcode;
ui/input-legacy.c +3 -3
index e0a39f0..3bc29bd 100644
@@ -38,7 +38,7 @@ struct QEMUPutMouseEntry {
     /* new input core */
     QemuInputHandler h;
     QemuInputHandlerState *s;
-    int axis[INPUT_AXIS_MAX];
+    int axis[INPUT_AXIS__MAX];
     int buttons;
 };
 
@@ -67,7 +67,7 @@ int index_from_key(const char *key)
         }
     }
 
-    /* Return Q_KEY_CODE_MAX if the key is invalid */
+    /* Return Q_KEY_CODE__MAX if the key is invalid */
     return i;
 }
 
@@ -143,7 +143,7 @@ QEMUPutKbdEntry *qemu_add_kbd_event_handler(QEMUPutKBDEvent *func, void *opaque)
 static void legacy_mouse_event(DeviceState *dev, QemuConsole *src,
                                InputEvent *evt)
 {
-    static const int bmap[INPUT_BUTTON_MAX] = {
+    static const int bmap[INPUT_BUTTON__MAX] = {
         [INPUT_BUTTON_LEFT]   = MOUSE_EVENT_LBUTTON,
         [INPUT_BUTTON_MIDDLE] = MOUSE_EVENT_MBUTTON,
         [INPUT_BUTTON_RIGHT]  = MOUSE_EVENT_RBUTTON,
ui/input.c +3 -3
index a0f9873..006667b 100644
@@ -211,7 +211,7 @@ static void qemu_input_event_trace(QemuConsole *src, InputEvent *evt)
             name = QKeyCode_lookup[evt->u.key->key->u.qcode];
             trace_input_event_key_qcode(idx, name, evt->u.key->down);
             break;
-        case KEY_VALUE_KIND_MAX:
+        case KEY_VALUE_KIND__MAX:
             /* keep gcc happy */
             break;
         }
@@ -228,7 +228,7 @@ static void qemu_input_event_trace(QemuConsole *src, InputEvent *evt)
         name = InputAxis_lookup[evt->u.abs->axis];
         trace_input_event_abs(idx, name, evt->u.abs->value);
         break;
-    case INPUT_EVENT_KIND_MAX:
+    case INPUT_EVENT_KIND__MAX:
         /* keep gcc happy */
         break;
     }
@@ -430,7 +430,7 @@ void qemu_input_update_buttons(QemuConsole *src, uint32_t *button_map,
     InputButton btn;
     uint32_t mask;
 
-    for (btn = 0; btn < INPUT_BUTTON_MAX; btn++) {
+    for (btn = 0; btn < INPUT_BUTTON__MAX; btn++) {
         mask = button_map[btn];
         if ((button_old & mask) == (button_new & mask)) {
             continue;
ui/sdl.c +1 -1
index 570cb99..f4aa0f2 100644
@@ -465,7 +465,7 @@ static void sdl_mouse_mode_change(Notifier *notify, void *data)
 
 static void sdl_send_mouse_event(int dx, int dy, int x, int y, int state)
 {
-    static uint32_t bmap[INPUT_BUTTON_MAX] = {
+    static uint32_t bmap[INPUT_BUTTON__MAX] = {
         [INPUT_BUTTON_LEFT]       = SDL_BUTTON(SDL_BUTTON_LEFT),
         [INPUT_BUTTON_MIDDLE]     = SDL_BUTTON(SDL_BUTTON_MIDDLE),
         [INPUT_BUTTON_RIGHT]      = SDL_BUTTON(SDL_BUTTON_RIGHT),
ui/sdl2.c +1 -1
index 5cb75aa..4be992a 100644
@@ -256,7 +256,7 @@ static void sdl_mouse_mode_change(Notifier *notify, void *data)
 static void sdl_send_mouse_event(struct sdl2_console *scon, int dx, int dy,
                                  int x, int y, int state)
 {
-    static uint32_t bmap[INPUT_BUTTON_MAX] = {
+    static uint32_t bmap[INPUT_BUTTON__MAX] = {
         [INPUT_BUTTON_LEFT]       = SDL_BUTTON(SDL_BUTTON_LEFT),
         [INPUT_BUTTON_MIDDLE]     = SDL_BUTTON(SDL_BUTTON_MIDDLE),
         [INPUT_BUTTON_RIGHT]      = SDL_BUTTON(SDL_BUTTON_RIGHT),
ui/spice-input.c +1 -1
index c342e0d..96f19aa 100644
@@ -107,7 +107,7 @@ typedef struct QemuSpicePointer {
 static void spice_update_buttons(QemuSpicePointer *pointer,
                                  int wheel, uint32_t button_mask)
 {
-    static uint32_t bmap[INPUT_BUTTON_MAX] = {
+    static uint32_t bmap[INPUT_BUTTON__MAX] = {
         [INPUT_BUTTON_LEFT]        = 0x01,
         [INPUT_BUTTON_MIDDLE]      = 0x04,
         [INPUT_BUTTON_RIGHT]       = 0x02,
ui/vnc.c +1 -1
index c9f2fed..c6cbb9d 100644
@@ -1671,7 +1671,7 @@ static void check_pointer_type_change(Notifier *notifier, void *data)
 
 static void pointer_event(VncState *vs, int button_mask, int x, int y)
 {
-    static uint32_t bmap[INPUT_BUTTON_MAX] = {
+    static uint32_t bmap[INPUT_BUTTON__MAX] = {
         [INPUT_BUTTON_LEFT]       = 0x01,
         [INPUT_BUTTON_MIDDLE]     = 0x02,
         [INPUT_BUTTON_RIGHT]      = 0x04,
vl.c +9 -9
index 7d993a5..3c38e02 100644
@@ -570,8 +570,8 @@ static int default_driver_check(void *opaque, QemuOpts *opts, Error **errp)
 
 static RunState current_run_state = RUN_STATE_PRELAUNCH;
 
-/* We use RUN_STATE_MAX but any invalid value will do */
-static RunState vmstop_requested = RUN_STATE_MAX;
+/* We use RUN_STATE__MAX but any invalid value will do */
+static RunState vmstop_requested = RUN_STATE__MAX;
 static QemuMutex vmstop_lock;
 
 typedef struct {
@@ -642,10 +642,10 @@ static const RunStateTransition runstate_transitions_def[] = {
     { RUN_STATE_GUEST_PANICKED, RUN_STATE_RUNNING },
     { RUN_STATE_GUEST_PANICKED, RUN_STATE_FINISH_MIGRATE },
 
-    { RUN_STATE_MAX, RUN_STATE_MAX },
+    { RUN_STATE__MAX, RUN_STATE__MAX },
 };
 
-static bool runstate_valid_transitions[RUN_STATE_MAX][RUN_STATE_MAX];
+static bool runstate_valid_transitions[RUN_STATE__MAX][RUN_STATE__MAX];
 
 bool runstate_check(RunState state)
 {
@@ -669,7 +669,7 @@ static void runstate_init(void)
     const RunStateTransition *p;
 
     memset(&runstate_valid_transitions, 0, sizeof(runstate_valid_transitions));
-    for (p = &runstate_transitions_def[0]; p->from != RUN_STATE_MAX; p++) {
+    for (p = &runstate_transitions_def[0]; p->from != RUN_STATE__MAX; p++) {
         runstate_valid_transitions[p->from][p->to] = true;
     }
 
@@ -679,7 +679,7 @@ static void runstate_init(void)
 /* This function will abort() on invalid state transitions */
 void runstate_set(RunState new_state)
 {
-    assert(new_state < RUN_STATE_MAX);
+    assert(new_state < RUN_STATE__MAX);
 
     if (!runstate_valid_transitions[current_run_state][new_state]) {
         error_report("invalid runstate transition: '%s' -> '%s'",
@@ -717,9 +717,9 @@ static bool qemu_vmstop_requested(RunState *r)
 {
     qemu_mutex_lock(&vmstop_lock);
     *r = vmstop_requested;
-    vmstop_requested = RUN_STATE_MAX;
+    vmstop_requested = RUN_STATE__MAX;
     qemu_mutex_unlock(&vmstop_lock);
-    return *r < RUN_STATE_MAX;
+    return *r < RUN_STATE__MAX;
 }
 
 void qemu_system_vmstop_request_prepare(void)
@@ -739,7 +739,7 @@ void vm_start(void)
     RunState requested;
 
     qemu_vmstop_requested(&requested);
-    if (runstate_is_running() && requested == RUN_STATE_MAX) {
+    if (runstate_is_running() && requested == RUN_STATE__MAX) {
         return;
     }
 
Signature · 1 line
2.4.3
Reviewed-by on v12 22/36
#42 lore raw

Re: [Qemu-devel] [PATCH v12 22/36] qapi: Don't let implicit enum MAX member collide

Quoted 22 lines · Eric Blake

Now that we guarantee the user doesn't have any enum values beginning with a single underscore, we can use that for our own purposes. Renaming ENUM_MAX to ENUM__MAX makes it obvious that the sentinel is generated.

This patch was mostly generated by applying a temporary patch:

|diff --git a/scripts/qapi.py b/scripts/qapi.py
|index e6d014b..b862ec9 100644
|--- a/scripts/qapi.py
|+++ b/scripts/qapi.py
|@@ -1570,6 +1570,7 @@ const char *const %(c_name)s_lookup[] = {
|     max_index = c_enum_const(name, 'MAX', prefix)
|     ret += mcgen('''
|     [%(max_index)s] = NULL,
|+// %(max_index)s
| };
| ''',
|                max_index=max_index)

then running:

Quoted 7 lines
$ cat qapi-{types,event}.c tests/test-qapi-types.c |
    sed -n 's,^// \(.*\)MAX,s|\1MAX|\1_MAX|g,p' > list
$ git grep -l _MAX | xargs sed -i -f list

The only things not generated are the changes in scripts/qapi.py.

For migration bits, I have zero objections about the changes. I trust you that you have done all the required changes (i.e. I haven't compiled it). Rest of the patch is as trivial as the commit log explains, so

Thanks, Juan.

#43 lore raw

Re: [Qemu-devel] [PATCH v12 22/36] qapi: Don't let implicit enum MAX member collide

In reply to Eric Blake
Quoted 4 lines · Eric Blake

Now that we guarantee the user doesn't have any enum values beginning with a single underscore, we can use that for our own purposes. Renaming ENUM_MAX to ENUM__MAX makes it obvious that the sentinel is generated.

The ENUM__MAX are mildly ugly. If we cared, we could get rid of most or all uses. Right now, I don't.

Hmm, perhaps these ENUM__MAX are your motivation for rejecting adjacent [-_] in the previous patch, to catch clashes like (foolishly named) type 'FOO--MAX' with enumeration type 'Foo'. I acknowledge the clash. However, there are many more clashes we don't attempt to catch, e.g. type 'Foo-lookup' with enumeration type 'Foo'. Perhaps we want to build a real fence later, but right now I don't want us to ram in more scattered fence laths.

Might want to say something like "Rejecting enum members named 'MAX' is now useless, and will be dropped in the next patch".

Quoted 17 lines

This patch was mostly generated by applying a temporary patch:

|diff --git a/scripts/qapi.py b/scripts/qapi.py
|index e6d014b..b862ec9 100644
|--- a/scripts/qapi.py
|+++ b/scripts/qapi.py
|@@ -1570,6 +1570,7 @@ const char *const %(c_name)s_lookup[] = {
|     max_index = c_enum_const(name, 'MAX', prefix)
|     ret += mcgen('''
|     [%(max_index)s] = NULL,
|+// %(max_index)s
| };
| ''',
|                max_index=max_index)

then running:

Quoted 7 lines
$ cat qapi-{types,event}.c tests/test-qapi-types.c |
    sed -n 's,^// \(.*\)MAX,s|\1MAX|\1_MAX|g,p' > list
$ git grep -l _MAX | xargs sed -i -f list

The only things not generated are the changes in scripts/qapi.py.

Patch looks good.

author
#44 lore raw

Re: [Qemu-devel] [PATCH v12 22/36] qapi: Don't let implicit enum MAX member collide

Quoted 7 lines · Markus Armbruster
Quoted 4 lines · Eric Blake

Now that we guarantee the user doesn't have any enum values beginning with a single underscore, we can use that for our own purposes. Renaming ENUM_MAX to ENUM__MAX makes it obvious that the sentinel is generated.

Quoted 6 lines

The ENUM__MAX are mildly ugly. If we cared, we could get rid of most or all uses. Right now, I don't.

Hmm, perhaps these ENUM__MAX are your motivation for rejecting adjacent [-_] in the previous patch, to catch clashes like (foolishly named) type 'FOO--MAX' with enumeration type 'Foo'. I acknowledge the clash.

Yes, and that was part of my reasoning on 21/36 - but you've convinced me to relax that one, so it no longer applies as an argument here.

Quoted 7 lines

However, there are many more clashes we don't attempt to catch, e.g. type 'Foo-lookup' with enumeration type 'Foo'. Perhaps we want to build a real fence later, but right now I don't want us to ram in more scattered fence laths.

Might want to say something like "Rejecting enum members named 'MAX' is now useless, and will be dropped in the next patch".

Yes, that helps the commit message. Can you handle it, or do I need to send a fixup?

Quoted 22 lines
Quoted 22 lines

This patch was mostly generated by applying a temporary patch:

|diff --git a/scripts/qapi.py b/scripts/qapi.py
|index e6d014b..b862ec9 100644
|--- a/scripts/qapi.py
|+++ b/scripts/qapi.py
|@@ -1570,6 +1570,7 @@ const char *const %(c_name)s_lookup[] = {
|     max_index = c_enum_const(name, 'MAX', prefix)
|     ret += mcgen('''
|     [%(max_index)s] = NULL,
|+// %(max_index)s
| };
| ''',
|                max_index=max_index)

then running:

$ cat qapi-{types,event}.c tests/test-qapi-types.c |
    sed -n 's,^// \(.*\)MAX,s|\1MAX|\1_MAX|g,p' > list
$ git grep -l _MAX | xargs sed -i -f list
Quoted 5 lines
Quoted 3 lines

The only things not generated are the changes in scripts/qapi.py.

Patch looks good.

The fun part was coming up with the automation, and then making sure it was reproducible. :)

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#45 lore raw

Re: [Qemu-devel] [PATCH v12 22/36] qapi: Don't let implicit enum MAX member collide

Quoted 23 lines · Eric Blake
Quoted 7 lines · Markus Armbruster
Quoted 4 lines · Eric Blake

Now that we guarantee the user doesn't have any enum values beginning with a single underscore, we can use that for our own purposes. Renaming ENUM_MAX to ENUM__MAX makes it obvious that the sentinel is generated.

Quoted 6 lines

The ENUM__MAX are mildly ugly. If we cared, we could get rid of most or all uses. Right now, I don't.

Hmm, perhaps these ENUM__MAX are your motivation for rejecting adjacent [-_] in the previous patch, to catch clashes like (foolishly named) type 'FOO--MAX' with enumeration type 'Foo'. I acknowledge the clash.

Yes, and that was part of my reasoning on 21/36 - but you've convinced me to relax that one, so it no longer applies as an argument here.

Quoted 5 lines

However, there are many more clashes we don't attempt to catch, e.g. type 'Foo-lookup' with enumeration type 'Foo'. Perhaps we want to build a real fence later, but right now I don't want us to ram in more scattered fence laths.

Quoted 5 lines
Quoted 2 lines

Might want to say something like "Rejecting enum members named 'MAX' is now useless, and will be dropped in the next patch".

Yes, that helps the commit message. Can you handle it, or do I need to send a fixup?

Can do!

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

This patch was mostly generated by applying a temporary patch:

|diff --git a/scripts/qapi.py b/scripts/qapi.py
|index e6d014b..b862ec9 100644
|--- a/scripts/qapi.py
|+++ b/scripts/qapi.py
|@@ -1570,6 +1570,7 @@ const char *const %(c_name)s_lookup[] = {
|     max_index = c_enum_const(name, 'MAX', prefix)
|     ret += mcgen('''
|     [%(max_index)s] = NULL,
|+// %(max_index)s
| };
| ''',
|                max_index=max_index)

then running:

$ cat qapi-{types,event}.c tests/test-qapi-types.c |
    sed -n 's,^// \(.*\)MAX,s|\1MAX|\1_MAX|g,p' > list
$ git grep -l _MAX | xargs sed -i -f list
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

The only things not generated are the changes in scripts/qapi.py.

Patch looks good.

The fun part was coming up with the automation, and then making sure it was reproducible. :)

May not always be faster than manual, but it sure is a lot more fun! More reliable, too, in my experience.

author
#46 lore raw

PATCH v12 23/36 qapi: Remove dead tests for max collision

In reply to Eric Blake

Now that we no longer collide with an implicit _MAX enum member, we no longer need to reject it in the ad hoc parser, and can remove several tests that are no longer needed.

---
v12: new patch

---
 scripts/qapi.py                        | 8 +++-----
 tests/Makefile                         | 3 ---
 tests/qapi-schema/enum-max-member.err  | 1 -
 tests/qapi-schema/enum-max-member.exit | 1 -
 tests/qapi-schema/enum-max-member.json | 3 ---
 tests/qapi-schema/enum-max-member.out  | 0
 tests/qapi-schema/event-max.err        | 1 -
 tests/qapi-schema/event-max.exit       | 1 -
 tests/qapi-schema/event-max.json       | 2 --
 tests/qapi-schema/event-max.out        | 0
 tests/qapi-schema/union-max.err        | 1 -
 tests/qapi-schema/union-max.exit       | 1 -
 tests/qapi-schema/union-max.json       | 3 ---
 tests/qapi-schema/union-max.out        | 0
 14 files changed, 3 insertions(+), 22 deletions(-)
 delete mode 100644 tests/qapi-schema/enum-max-member.err
 delete mode 100644 tests/qapi-schema/enum-max-member.exit
 delete mode 100644 tests/qapi-schema/enum-max-member.json
 delete mode 100644 tests/qapi-schema/enum-max-member.out
 delete mode 100644 tests/qapi-schema/event-max.err
 delete mode 100644 tests/qapi-schema/event-max.exit
 delete mode 100644 tests/qapi-schema/event-max.json
 delete mode 100644 tests/qapi-schema/event-max.out
 delete mode 100644 tests/qapi-schema/union-max.err
 delete mode 100644 tests/qapi-schema/union-max.exit
 delete mode 100644 tests/qapi-schema/union-max.json
 delete mode 100644 tests/qapi-schema/union-max.out
scripts/qapi.py +3 -5
index aa15adb..cd2982a 100644
@@ -537,8 +537,6 @@ def check_event(expr, expr_info):
     global events
     name = expr['event']
 
-    if name.upper() == 'MAX':
-        raise QAPIExprError(expr_info, "Event name 'MAX' cannot be created")
     events.append(name)
     check_type(expr_info, "'data' for event '%s'" % name,
                expr.get('data'), allow_dict=True, allow_optional=True,
@@ -550,7 +548,7 @@ def check_union(expr, expr_info):
     base = expr.get('base')
     discriminator = expr.get('discriminator')
     members = expr['data']
-    values = {'MAX': '(automatic)'}
+    values = {}
 
     # Two types of unions, determined by discriminator.
 
@@ -629,7 +627,7 @@ def check_union(expr, expr_info):
 def check_alternate(expr, expr_info):
     name = expr['alternate']
     members = expr['data']
-    values = {'MAX': '(automatic)'}
+    values = {}
     types_seen = {}
 
     # Check every branch
@@ -662,7 +660,7 @@ def check_enum(expr, expr_info):
     name = expr['enum']
     members = expr.get('data')
     prefix = expr.get('prefix')
-    values = {'MAX': '(automatic)'}
+    values = {}
 
     if not isinstance(members, list):
         raise QAPIExprError(expr_info,
tests/Makefile +0 -3
index b70d246..a8a3b12 100644
@@ -266,14 +266,12 @@ qapi-schema += enum-bad-prefix.json
 qapi-schema += enum-clash-member.json
 qapi-schema += enum-dict-member.json
 qapi-schema += enum-int-member.json
-qapi-schema += enum-max-member.json
 qapi-schema += enum-missing-data.json
 qapi-schema += enum-wrong-data.json
 qapi-schema += escape-outside-string.json
 qapi-schema += escape-too-big.json
 qapi-schema += escape-too-short.json
 qapi-schema += event-case.json
-qapi-schema += event-max.json
 qapi-schema += event-nest-struct.json
 qapi-schema += flat-union-array-branch.json
 qapi-schema += flat-union-bad-base.json
@@ -345,7 +343,6 @@ qapi-schema += union-clash-branches.json
 qapi-schema += union-clash-data.json
 qapi-schema += union-empty.json
 qapi-schema += union-invalid-base.json
-qapi-schema += union-max.json
 qapi-schema += union-optional-branch.json
 qapi-schema += union-unknown.json
 qapi-schema += unknown-escape.json
tests/qapi-schema/enum-max-member.err deleted +0 -1
deleted file mode 100644
index f77837f..0000000
@@ -1 +0,0 @@
-tests/qapi-schema/enum-max-member.json:3: Enum 'MyEnum' member 'max' clashes with '(automatic)'
tests/qapi-schema/enum-max-member.exit deleted +0 -1
deleted file mode 100644
index d00491f..0000000
@@ -1 +0,0 @@
-1
tests/qapi-schema/enum-max-member.json deleted +0 -3
deleted file mode 100644
index 4bcda0b..0000000
@@ -1,3 +0,0 @@
-# we reject user-supplied 'max' for clashing with implicit enum end
-# TODO: should we instead munge the implicit value to avoid the clash?
-{ 'enum': 'MyEnum', 'data': [ 'max' ] }
tests/qapi-schema/enum-max-member.out deleted +0 -0
deleted file mode 100644
index e69de29..0000000
tests/qapi-schema/event-max.err deleted +0 -1
deleted file mode 100644
index c856534..0000000
@@ -1 +0,0 @@
-tests/qapi-schema/event-max.json:2: Event name 'MAX' cannot be created
tests/qapi-schema/event-max.exit deleted +0 -1
deleted file mode 100644
index d00491f..0000000
@@ -1 +0,0 @@
-1
tests/qapi-schema/event-max.json deleted +0 -2
deleted file mode 100644
index f3d7de2..0000000
@@ -1,2 +0,0 @@
-# an event named 'MAX' would conflict with implicit C enum
-{ 'event': 'MAX' }
tests/qapi-schema/event-max.out deleted +0 -0
deleted file mode 100644
index e69de29..0000000
tests/qapi-schema/union-max.err deleted +0 -1
deleted file mode 100644
index 55ce439..0000000
@@ -1 +0,0 @@
-tests/qapi-schema/union-max.json:2: Union 'Union' member 'max' clashes with '(automatic)'
tests/qapi-schema/union-max.exit deleted +0 -1
deleted file mode 100644
index d00491f..0000000
@@ -1 +0,0 @@
-1
tests/qapi-schema/union-max.json deleted +0 -3
deleted file mode 100644
index d6ad986..0000000
@@ -1,3 +0,0 @@
-# we reject 'max' branch in a union, for collision with C enum
-{ 'union': 'Union',
-  'data': { 'max': 'int' } }
tests/qapi-schema/union-max.out deleted +0 -0
deleted file mode 100644
index e69de29..0000000
Signature · 1 line
2.4.3
#47 lore raw

Re: [Qemu-devel] [PATCH v12 23/36] qapi: Remove dead tests for max collision

Tweaking subject to "qapi: Remove obsolete tests for MAX collision".

author
#48 lore raw

PATCH v12 24/36 cpu: Convert CpuInfo into flat union

In reply to Eric Blake

The CpuInfo struct is used only by the 'query-cpus' output command, so we are free to modify it by adding fields (clients are already supposed to ignore unknown output fields), or by changing optional members to mandatory, while still keeping QMP wire compatibility with older versions of qemu.

When qapi type CpuInfo was originally created for 0.14, we had no notion of a flat union, and instead just listed a bunch of optional fields with documentation about the mutually-exclusive choice of which instruction pointer field(s) would be provided for a given architecture. But now that we have flat unions and introspection, it is better to segregate off which fields will be provided according to the actual architecture. With this in place, we no longer need the fields to be optional, because the choice of the new 'arch' discriminator serves that role.

This has an additional benefit: the old all-in-one struct was the only place in the code base that had a case-sensitive naming of members 'pc' vs. 'PC'. Separating these spellings into different branches of the flat union will allow us to add restrictions against future case-insensitive collisions, since that is generally a poor interface practice.

---
v12: s/2.5/2.6/, typo fix
v11: also fix qmp-commands.hx, improve commit message. If this misses 2.5 (likely), it will need updates to call out 2.6
v10: new patch

---
 cpus.c           |  31 ++++++++------
 hmp.c            |  30 +++++++++-----
 qapi-schema.json | 120 ++++++++++++++++++++++++++++++++++++++++++++++---------
 qmp-commands.hx  |   4 ++
 4 files changed, 143 insertions(+), 42 deletions(-)
cpus.c +19 -12
index 877bd70..77ba15b 100644
@@ -1556,22 +1556,29 @@ CpuInfoList *qmp_query_cpus(Error **errp)
         info->value->qom_path = object_get_canonical_path(OBJECT(cpu));
         info->value->thread_id = cpu->thread_id;
 #if defined(TARGET_I386)
-        info->value->has_pc = true;
-        info->value->pc = env->eip + env->segs[R_CS].base;
+        info->value->arch = CPU_INFO_ARCH_X86;
+        info->value->u.x86 = g_new0(CpuInfoX86, 1);
+        info->value->u.x86->pc = env->eip + env->segs[R_CS].base;
 #elif defined(TARGET_PPC)
-        info->value->has_nip = true;
-        info->value->nip = env->nip;
+        info->value->arch = CPU_INFO_ARCH_PPC;
+        info->value->u.ppc = g_new0(CpuInfoPpc, 1);
+        info->value->u.ppc->nip = env->nip;
 #elif defined(TARGET_SPARC)
-        info->value->has_pc = true;
-        info->value->pc = env->pc;
-        info->value->has_npc = true;
-        info->value->npc = env->npc;
+        info->value->arch = CPU_INFO_ARCH_SPARC;
+        info->value->u.sparc = g_new0(CpuInfoSPARC, 1);
+        info->value->u.sparc->pc = env->pc;
+        info->value->u.sparc->npc = env->npc;
 #elif defined(TARGET_MIPS)
-        info->value->has_PC = true;
-        info->value->PC = env->active_tc.PC;
+        info->value->arch = CPU_INFO_ARCH_MIPS;
+        info->value->u.mips = g_new0(CpuInfoMips, 1);
+        info->value->u.mips->PC = env->active_tc.PC;
 #elif defined(TARGET_TRICORE)
-        info->value->has_PC = true;
-        info->value->PC = env->PC;
+        info->value->arch = CPU_INFO_ARCH_TRICORE;
+        info->value->u.tricore = g_new0(CpuInfoTricore, 1);
+        info->value->u.tricore->PC = env->PC;
+#else
+        info->value->arch = CPU_INFO_ARCH_OTHER;
+        info->value->u.other = g_new0(CpuInfoOther, 1);
 #endif
 
         /* XXX: waiting for the qapi to support GSList */
hmp.c +19 -11
index 1b402c4..c2b2c16 100644
@@ -311,17 +311,25 @@ void hmp_info_cpus(Monitor *mon, const QDict *qdict)
 
         monitor_printf(mon, "%c CPU #%" PRId64 ":", active, cpu->value->CPU);
 
-        if (cpu->value->has_pc) {
-            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->pc);
-        }
-        if (cpu->value->has_nip) {
-            monitor_printf(mon, " nip=0x%016" PRIx64, cpu->value->nip);
-        }
-        if (cpu->value->has_npc) {
-            monitor_printf(mon, " npc=0x%016" PRIx64, cpu->value->npc);
-        }
-        if (cpu->value->has_PC) {
-            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->PC);
+        switch (cpu->value->arch) {
+        case CPU_INFO_ARCH_X86:
+            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->u.x86->pc);
+            break;
+        case CPU_INFO_ARCH_PPC:
+            monitor_printf(mon, " nip=0x%016" PRIx64, cpu->value->u.ppc->nip);
+            break;
+        case CPU_INFO_ARCH_SPARC:
+            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->u.sparc->pc);
+            monitor_printf(mon, " npc=0x%016" PRIx64, cpu->value->u.sparc->npc);
+            break;
+        case CPU_INFO_ARCH_MIPS:
+            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->u.mips->PC);
+            break;
+        case CPU_INFO_ARCH_TRICORE:
+            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->u.tricore->PC);
+            break;
+        default:
+            break;
         }
 
         if (cpu->value->halted) {
qapi-schema.json +101 -19

144 lines not on this page · raw message

qmp-commands.hx +4 -0
index 9d8b42f..1b5ecb3 100644
@@ -2765,6 +2765,8 @@ Return a json-array. Each CPU is represented by a json-object, which contains:
 - "current": true if this is the current CPU, false otherwise (json-bool)
 - "halted": true if the cpu is halted, false otherwise (json-bool)
 - "qom_path": path to the CPU object in the QOM tree (json-str)
+- "arch": architecture of the cpu, which determines what additional
+          keys will be present (json-str)
 - Current program counter. The key's name depends on the architecture:
      "pc": i386/x86_64 (json-int)
      "nip": PPC (json-int)
@@ -2782,6 +2784,7 @@ Example:
             "current":true,
             "halted":false,
             "qom_path":"/machine/unattached/device[0]",
+            "arch":"x86",
             "pc":3227107138,
             "thread_id":3134
          },
@@ -2790,6 +2793,7 @@ Example:
             "current":false,
             "halted":true,
             "qom_path":"/machine/unattached/device[2]",
+            "arch":"x86",
             "pc":7108165,
             "thread_id":3135
          }
Signature · 1 line
2.4.3
#49 lore raw

Re: [Qemu-devel] [PATCH v12 24/36] cpu: Convert CpuInfo into flat union

Quoted 53 lines · Eric Blake

The CpuInfo struct is used only by the 'query-cpus' output command, so we are free to modify it by adding fields (clients are already supposed to ignore unknown output fields), or by changing optional members to mandatory, while still keeping QMP wire compatibility with older versions of qemu.

When qapi type CpuInfo was originally created for 0.14, we had no notion of a flat union, and instead just listed a bunch of optional fields with documentation about the mutually-exclusive choice of which instruction pointer field(s) would be provided for a given architecture. But now that we have flat unions and introspection, it is better to segregate off which fields will be provided according to the actual architecture. With this in place, we no longer need the fields to be optional, because the choice of the new 'arch' discriminator serves that role.

This has an additional benefit: the old all-in-one struct was the only place in the code base that had a case-sensitive naming of members 'pc' vs. 'PC'. Separating these spellings into different branches of the flat union will allow us to add restrictions against future case-insensitive collisions, since that is generally a poor interface practice.

---
v12: s/2.5/2.6/, typo fix
v11: also fix qmp-commands.hx, improve commit message. If this misses 2.5 (likely), it will need updates to call out 2.6
v10: new patch

---
 cpus.c           |  31 ++++++++------
 hmp.c            |  30 +++++++++-----
 qapi-schema.json | 120 ++++++++++++++++++++++++++++++++++++++++++++++---------
 qmp-commands.hx  |   4 ++
 4 files changed, 143 insertions(+), 42 deletions(-)
cpus.c +3 -4
index 877bd70..77ba15b 100644
@@ -1556,22 +1556,29 @@ CpuInfoList *qmp_query_cpus(Error **errp)
         info->value->qom_path = object_get_canonical_path(OBJECT(cpu));
         info->value->thread_id = cpu->thread_id;
 #if defined(TARGET_I386)
-        info->value->has_pc = true;
-        info->value->pc = env->eip + env->segs[R_CS].base;
+        info->value->arch = CPU_INFO_ARCH_X86;
+        info->value->u.x86 = g_new0(CpuInfoX86, 1);
+        info->value->u.x86->pc = env->eip + env->segs[R_CS].base;
 #elif defined(TARGET_PPC)
-        info->value->has_nip = true;
-        info->value->nip = env->nip;

cpus.c CpuInfoList *qmp_query_cpus(Error **errp) · line 1568

Quoted 10 lines
cpus.c +5 -4
+        info->value->arch = CPU_INFO_ARCH_PPC;
+        info->value->u.ppc = g_new0(CpuInfoPpc, 1);
+        info->value->u.ppc->nip = env->nip;
 #elif defined(TARGET_SPARC)
-        info->value->has_pc = true;
-        info->value->pc = env->pc;
-        info->value->has_npc = true;
-        info->value->npc = env->npc;
+        info->value->arch = CPU_INFO_ARCH_SPARC;
+        info->value->u.sparc = g_new0(CpuInfoSPARC, 1);

CpuInfoSparc.

Quoted 144 lines
cpus.c +11 -4
+        info->value->u.sparc->pc = env->pc;
+        info->value->u.sparc->npc = env->npc;
 #elif defined(TARGET_MIPS)
-        info->value->has_PC = true;
-        info->value->PC = env->active_tc.PC;
+        info->value->arch = CPU_INFO_ARCH_MIPS;
+        info->value->u.mips = g_new0(CpuInfoMips, 1);
+        info->value->u.mips->PC = env->active_tc.PC;
 #elif defined(TARGET_TRICORE)
-        info->value->has_PC = true;
-        info->value->PC = env->PC;
+        info->value->arch = CPU_INFO_ARCH_TRICORE;
+        info->value->u.tricore = g_new0(CpuInfoTricore, 1);
+        info->value->u.tricore->PC = env->PC;
+#else
+        info->value->arch = CPU_INFO_ARCH_OTHER;
+        info->value->u.other = g_new0(CpuInfoOther, 1);
 #endif
 
         /* XXX: waiting for the qapi to support GSList */
hmp.c +19 -11
index 1b402c4..c2b2c16 100644
@@ -311,17 +311,25 @@ void hmp_info_cpus(Monitor *mon, const QDict *qdict)
 
         monitor_printf(mon, "%c CPU #%" PRId64 ":", active, cpu->value->CPU);
 
-        if (cpu->value->has_pc) {
-            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->pc);
-        }
-        if (cpu->value->has_nip) {
-            monitor_printf(mon, " nip=0x%016" PRIx64, cpu->value->nip);
-        }
-        if (cpu->value->has_npc) {
-            monitor_printf(mon, " npc=0x%016" PRIx64, cpu->value->npc);
-        }
-        if (cpu->value->has_PC) {
-            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->PC);
+        switch (cpu->value->arch) {
+        case CPU_INFO_ARCH_X86:
+            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->u.x86->pc);
+            break;
+        case CPU_INFO_ARCH_PPC:
+            monitor_printf(mon, " nip=0x%016" PRIx64, cpu->value->u.ppc->nip);
+            break;
+        case CPU_INFO_ARCH_SPARC:
+            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->u.sparc->pc);
+            monitor_printf(mon, " npc=0x%016" PRIx64, cpu->value->u.sparc->npc);
+            break;
+        case CPU_INFO_ARCH_MIPS:
+            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->u.mips->PC);
+            break;
+        case CPU_INFO_ARCH_TRICORE:
+            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->u.tricore->PC);
+            break;
+        default:
+            break;
         }
 
         if (cpu->value->halted) {
qapi-schema.json +38 -19
index 8b1a423..67e80a5 100644
@@ -744,43 +744,125 @@
 { 'command': 'query-mice', 'returns': ['MouseInfo'] }
 
 ##
-# @CpuInfo:
+# @CpuInfoArch:
 #
-# Information about a virtual CPU
+# An enumeration of cpu types that enable additional information during
+# @query-cpus.
+#
+# Since: 2.6
+##
+{ 'enum': 'CpuInfoArch',
+  'data': ['x86', 'sparc', 'ppc', 'mips', 'tricore', 'other' ] }
+
+##
+# @CpuInfoBase:
+#
+# Common information about a virtual CPU
 #
 # @CPU: the index of the virtual CPU
 #
-# @current: this only exists for backwards compatible and should be ignored
+# @current: this only exists for backwards compatibility and should be ignored
 #
 # @halted: true if the virtual CPU is in the halt state.  Halt usually refers
 #          to a processor specific low power mode.
 #
 # @qom_path: path to the CPU object in the QOM tree (since 2.4)
 #
-# @pc: #optional If the target is i386 or x86_64, this is the 64-bit instruction
-#                pointer.
-#                If the target is Sparc, this is the PC component of the
-#                instruction pointer.
-#
-# @nip: #optional If the target is PPC, the instruction pointer
-#
-# @npc: #optional If the target is Sparc, the NPC component of the instruction
-#                 pointer
-#
-# @PC: #optional If the target is MIPS, the instruction pointer
-#
 # @thread_id: ID of the underlying host thread
 #
+# @arch: architecture of the cpu, which determines which additional fields
+#        will be listed (since 2.6)
+#
 # Since: 0.14.0
 #
 # Notes: @halted is a transient state that changes frequently.  By the time the
 #        data is sent to the client, the guest may no longer be halted.
 ##
-{ 'struct': 'CpuInfo',
+{ 'struct': 'CpuInfoBase',
   'data': {'CPU': 'int', 'current': 'bool', 'halted': 'bool',
-           'qom_path': 'str',
-           '*pc': 'int', '*nip': 'int', '*npc': 'int', '*PC': 'int',
-           'thread_id': 'int'} }
+           'qom_path': 'str', 'thread_id': 'int', 'arch': 'CpuInfoArch' } }
+
+##
+# @CpuInfo:
+#
+# Information about a virtual CPU
+#
+# Since: 0.14.0
+##
+{ 'union': 'CpuInfo', 'base': 'CpuInfoBase', 'discriminator': 'arch',
+  'data': { 'x86': 'CpuInfoX86',
+            'sparc': 'CpuInfoSparc',
+            'ppc': 'CpuInfoPpc',
+            'mips': 'CpuInfoMips',
+            'tricore': 'CpuInfoTricore',
+            'other': 'CpuInfoOther' } }
+
+##
+# @CpuInfoX86:
+#

qapi-schema.json line 812

Quoted 10 lines
qapi-schema.json +10 -0
+# Additional information about a virtual i386 or x86_64 CPU
+#
+# @pc: the 64-bit instruction pointer
+#
+# Since 2.6
+##
+{ 'struct': 'CpuInfoX86', 'data': { 'pc': 'int' } }
+
+##
+# @CpuInfoSparc:

CpuInfoSPARC would be slightly prettier, actually.

qapi-schema.json line 825

Quoted 13 lines
qapi-schema.json +13 -0
+#
+# Additional information about a virtual Sparc CPU
+#
+# @pc: the PC component of the instruction pointer
+#
+# @npc: the NPC component of the instruction pointer
+#
+# Since 2.6
+##
+{ 'struct': 'CpuInfoSparc', 'data': { 'pc': 'int', 'npc': 'int' } }
+
+##
+# @CpuInfoPpc:

CpuInfoPPC would definitely be prettier.

qapi-schema.json line 836

Quoted 11 lines
qapi-schema.json +11 -0
+#
+# Additional information about a virtual PPC CPU
+#
+# @nip: the instruction pointer
+#
+# Since 2.6
+##
+{ 'struct': 'CpuInfoPpc', 'data': { 'nip': 'int' } }
+
+##
+# @CpuInfoMips:

CpuInfoMIPS?

Quoted 61 lines
qapi-schema.json +29 -0
+#
+# Additional information about a virtual MIPS CPU
+#
+# @PC: the instruction pointer
+#
+# Since 2.6
+##
+{ 'struct': 'CpuInfoMips', 'data': { 'PC': 'int' } }
+
+##
+# @CpuInfoTricore:
+#
+# Additional information about a virtual Tricore CPU
+#
+# @PC: the instruction pointer
+#
+# Since 2.6
+##
+{ 'struct': 'CpuInfoTricore', 'data': { 'PC': 'int' } }
+
+##
+# @CpuInfoOther:
+#
+# No additional information is available about the virtual CPU
+#
+# Since 2.6
+#
+##
+{ 'struct': 'CpuInfoOther', 'data': { } }
 
 ##
 # @query-cpus:
qmp-commands.hx +4 -0
index 9d8b42f..1b5ecb3 100644
@@ -2765,6 +2765,8 @@ Return a json-array. Each CPU is represented by a json-object, which contains:
 - "current": true if this is the current CPU, false otherwise (json-bool)
 - "halted": true if the cpu is halted, false otherwise (json-bool)
 - "qom_path": path to the CPU object in the QOM tree (json-str)
+- "arch": architecture of the cpu, which determines what additional
+          keys will be present (json-str)
 - Current program counter. The key's name depends on the architecture:
      "pc": i386/x86_64 (json-int)
      "nip": PPC (json-int)
@@ -2782,6 +2784,7 @@ Example:
             "current":true,
             "halted":false,
             "qom_path":"/machine/unattached/device[0]",
+            "arch":"x86",
             "pc":3227107138,
             "thread_id":3134
          },
@@ -2790,6 +2793,7 @@ Example:
             "current":false,
             "halted":true,
             "qom_path":"/machine/unattached/device[2]",
+            "arch":"x86",
             "pc":7108165,
             "thread_id":3135
          }
author
#50 lore raw

Re: [Qemu-devel] [PATCH v12 24/36] cpu: Convert CpuInfo into flat union

Quoted 26 lines · Markus Armbruster
Quoted 24 lines · Eric Blake

The CpuInfo struct is used only by the 'query-cpus' output command, so we are free to modify it by adding fields (clients are already supposed to ignore unknown output fields), or by changing optional members to mandatory, while still keeping QMP wire compatibility with older versions of qemu.

+++ b/cpus.c

cpus.c +6 -4
@@ -1556,22 +1556,29 @@ CpuInfoList *qmp_query_cpus(Error **errp)
         info->value->qom_path = object_get_canonical_path(OBJECT(cpu));
         info->value->thread_id = cpu->thread_id;
 #if defined(TARGET_I386)
-        info->value->has_pc = true;
-        info->value->pc = env->eip + env->segs[R_CS].base;
+        info->value->arch = CPU_INFO_ARCH_X86;
+        info->value->u.x86 = g_new0(CpuInfoX86, 1);
+        info->value->u.x86->pc = env->eip + env->segs[R_CS].base;
 #elif defined(TARGET_PPC)
-        info->value->has_nip = true;
-        info->value->nip = env->nip;
+        info->value->arch = CPU_INFO_ARCH_PPC;
+        info->value->u.ppc = g_new0(CpuInfoPpc, 1);
+        info->value->u.ppc->nip = env->nip;
 #elif defined(TARGET_SPARC)
Quoted 8 lines

cpus.c CpuInfoList *qmp_query_cpus(Error **errp) · line 1568

Quoted 6 lines
cpus.c +2 -4
-        info->value->has_pc = true;
-        info->value->pc = env->pc;
-        info->value->has_npc = true;
-        info->value->npc = env->npc;
+        info->value->arch = CPU_INFO_ARCH_SPARC;
+        info->value->u.sparc = g_new0(CpuInfoSPARC, 1);

CpuInfoSparc.

#ifdef compilation problems. Yes, we'll have to double-check that whatever case we use here matches our qapi.

Quoted 18 lines
Quoted 17 lines
qapi-schema.json +10 -2
+++ b/qapi-schema.json
@@ -744,43 +744,125 @@
 { 'command': 'query-mice', 'returns': ['MouseInfo'] }
 
 ##
-# @CpuInfo:
+# @CpuInfoArch:
 #
-# Information about a virtual CPU
+# An enumeration of cpu types that enable additional information during
+# @query-cpus.
+#
+# Since: 2.6
+##
+{ 'enum': 'CpuInfoArch',
+  'data': ['x86', 'sparc', 'ppc', 'mips', 'tricore', 'other' ] }
+
Quoted 14 lines
Quoted 4 lines
qapi-schema.json +4 -0
+##
+# @CpuInfo:
+#
+# Information about a virtual CPU

qapi-schema.json line 798

Quoted 10 lines
qapi-schema.json +10 -0
+#
+# Since: 0.14.0
+##
+{ 'union': 'CpuInfo', 'base': 'CpuInfoBase', 'discriminator': 'arch',
+  'data': { 'x86': 'CpuInfoX86',
+            'sparc': 'CpuInfoSparc',
+            'ppc': 'CpuInfoPpc',
+            'mips': 'CpuInfoMips',
+            'tricore': 'CpuInfoTricore',
+            'other': 'CpuInfoOther' } }

As to the qapi types, changing the case to CpuInfoSPARC, CpuInfoPPC, and CpuInfoMIPS all work for me. For MIPS, it definitely matches prevailing preference (here searching for any use of the arch name as part of a larger CamelCase name); for SPARC and PPC, we could sway the vote:

$ git grep '[a-z]Sparc' origin | wc
     39     262    2705
$ git grep '[a-z]SPARC' origin | wc
     38     316    3385
$ git grep '[a-z]Ppc' origin | wc
      1       4      48
$ git grep '[a-z]PPC' origin | wc
      1       4      56
$ git grep '[a-z]Mips' origin | wc
      0       0       0
$ git grep '[a-z]MIPS' origin | wc
     69     407    5475

I'm assuming you plan on doing the touchup?

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#51 lore raw

Re: [Qemu-devel] [PATCH v12 24/36] cpu: Convert CpuInfo into flat union

Quoted 88 lines · Eric Blake
Quoted 26 lines · Markus Armbruster
Quoted 24 lines · Eric Blake

The CpuInfo struct is used only by the 'query-cpus' output command, so we are free to modify it by adding fields (clients are already supposed to ignore unknown output fields), or by changing optional members to mandatory, while still keeping QMP wire compatibility with older versions of qemu.

+++ b/cpus.c

cpus.c +6 -4
@@ -1556,22 +1556,29 @@ CpuInfoList *qmp_query_cpus(Error **errp)
         info->value->qom_path = object_get_canonical_path(OBJECT(cpu));
         info->value->thread_id = cpu->thread_id;
 #if defined(TARGET_I386)
-        info->value->has_pc = true;
-        info->value->pc = env->eip + env->segs[R_CS].base;
+        info->value->arch = CPU_INFO_ARCH_X86;
+        info->value->u.x86 = g_new0(CpuInfoX86, 1);
+        info->value->u.x86->pc = env->eip + env->segs[R_CS].base;
 #elif defined(TARGET_PPC)
-        info->value->has_nip = true;
-        info->value->nip = env->nip;
+        info->value->arch = CPU_INFO_ARCH_PPC;
+        info->value->u.ppc = g_new0(CpuInfoPpc, 1);
+        info->value->u.ppc->nip = env->nip;
 #elif defined(TARGET_SPARC)
Quoted 8 lines

cpus.c CpuInfoList *qmp_query_cpus(Error **errp) · line 1568

Quoted 6 lines
cpus.c +2 -4
-        info->value->has_pc = true;
-        info->value->pc = env->pc;
-        info->value->has_npc = true;
-        info->value->npc = env->npc;
+        info->value->arch = CPU_INFO_ARCH_SPARC;
+        info->value->u.sparc = g_new0(CpuInfoSPARC, 1);

CpuInfoSparc.

#ifdef compilation problems. Yes, we'll have to double-check that whatever case we use here matches our qapi.

Quoted 18 lines
Quoted 17 lines
qapi-schema.json +10 -2
+++ b/qapi-schema.json
@@ -744,43 +744,125 @@
 { 'command': 'query-mice', 'returns': ['MouseInfo'] }
 
 ##
-# @CpuInfo:
+# @CpuInfoArch:
 #
-# Information about a virtual CPU
+# An enumeration of cpu types that enable additional information during
+# @query-cpus.
+#
+# Since: 2.6
+##
+{ 'enum': 'CpuInfoArch',
+  'data': ['x86', 'sparc', 'ppc', 'mips', 'tricore', 'other' ] }
+
Quoted 14 lines
Quoted 4 lines
qapi-schema.json +4 -0
+##
+# @CpuInfo:
+#
+# Information about a virtual CPU

qapi-schema.json line 798

Quoted 10 lines
qapi-schema.json +10 -0
+#
+# Since: 0.14.0
+##
+{ 'union': 'CpuInfo', 'base': 'CpuInfoBase', 'discriminator': 'arch',
+  'data': { 'x86': 'CpuInfoX86',
+            'sparc': 'CpuInfoSparc',
+            'ppc': 'CpuInfoPpc',
+            'mips': 'CpuInfoMips',
+            'tricore': 'CpuInfoTricore',
+            'other': 'CpuInfoOther' } }

As to the qapi types, changing the case to CpuInfoSPARC, CpuInfoPPC, and CpuInfoMIPS all work for me. For MIPS, it definitely matches prevailing preference (here searching for any use of the arch name as part of a larger CamelCase name); for SPARC and PPC, we could sway the vote:

$ git grep '[a-z]Sparc' origin | wc
     39     262    2705
$ git grep '[a-z]SPARC' origin | wc
     38     316    3385
$ git grep '[a-z]Ppc' origin | wc
      1       4      48
$ git grep '[a-z]PPC' origin | wc
      1       4      56
Quoted 4 lines
$ git grep '[a-z]Mips' origin | wc
      0       0       0
$ git grep '[a-z]MIPS' origin | wc
     69     407    5475

Wikipedia and USPTO spell it SPARC, MIPS and PowerPC. PPC is a common abbreviation of the latter. I find CpuInfoPpc even uglier than CpuInfoPPC.

Quoted 1 line

I'm assuming you plan on doing the touchup?

Yes, fixup is being compiled.

#52 lore raw

Re: [Qemu-devel] [PATCH v12 24/36] cpu: Convert CpuInfo into flat union

In reply to Eric Blake

Hi Eric,

Quoted 8 lines · Eric Blake
(quoted diff) +8 -0
+# An enumeration of cpu types that enable additional information during
+# @query-cpus.
+#
+# Since: 2.6
+##
+{ 'enum': 'CpuInfoArch',
+  'data': ['x86', 'sparc', 'ppc', 'mips', 'tricore', 'other' ] }
+

...

Review

Quoted 12 lines
(quoted diff) +12 -0
+##
+# @CpuInfo:
+#
+# Information about a virtual CPU
+#
+# Since: 0.14.0
+##
+{ 'union': 'CpuInfo', 'base': 'CpuInfoBase', 'discriminator': 'arch',
+  'data': { 'x86': 'CpuInfoX86',
+            'sparc': 'CpuInfoSparc',
+            'ppc': 'CpuInfoPpc',
+            'mips': 'CpuInfoMips',

This is causing compilation issues on MIPS, because gcc bizarrely defines mips=1:

$ mips-img-linux-gnu-gcc -dM -E - </dev/null | grep '\bmips\b'
#define mips 1

For example:
CC qga/commands.o
In file included from /work/mips/qemu/msa/include/qapi/qmp/qobject.h:37:0,

                from /work/mips/qemu/msa/include/qapi/qmp/dispatch.h:17,
                from ./qga/guest-agent-core.h:13,
                from qga/commands.c:14:
./qapi-types.h:2568:22: error: expected identifier or ‘(’ before
numeric constant
        CpuInfoMIPS *mips;
                     ^

/work/mips/qemu/msa/rules.mak:57: recipe for target 'qga/commands.o' failed
make: *** [qga/commands.o] Error 1

Any suggestions for a fix that is less ugly than this:

cpus.c +0 -0
index 882b61851a44..395ec70bdf22 100644
@@ -1580,9 +1580,9 @@ CpuInfoList *qmp_query_cpus(Error **errp)
        info->value->u.sparc->pc = env->pc;
        info->value->u.sparc->npc = env->npc;

#elif defined(TARGET_MIPS)

-        info->value->arch = CPU_INFO_ARCH_MIPS;
-        info->value->u.mips = g_new0(CpuInfoMIPS, 1);
-        info->value->u.mips->PC = env->active_tc.PC;
+        info->value->arch = CPU_INFO_ARCH_MIPSARCH;
+        info->value->u.mipsarch = g_new0(CpuInfoMIPS, 1);
+        info->value->u.mipsarch->PC = env->active_tc.PC;
#elif defined(TARGET_TRICORE)
        info->value->arch = CPU_INFO_ARCH_TRICORE;
        info->value->u.tricore = g_new0(CpuInfoTricore, 1);
hmp.c +2 -2
index 54f2620690b1..e4cba874352e 100644
@@ -321,8 +321,8 @@ void hmp_info_cpus(Monitor *mon, const QDict *qdict)
            monitor_printf(mon, " pc=0x%016" PRIx64, cpu->value->u.sparc->pc);
            monitor_printf(mon, " npc=0x%016" PRIx64, cpu->value->u.sparc->npc);
            break;
-        case CPU_INFO_ARCH_MIPS:
-            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->u.mips->PC);
+        case CPU_INFO_ARCH_MIPSARCH:
+            monitor_printf(mon, " PC=0x%016" PRIx64,
cpu->value->u.mipsarch->PC);
            break;
        case CPU_INFO_ARCH_TRICORE:
            monitor_printf(mon, " PC=0x%016" PRIx64, cpu->value->u.tricore->PC);
qapi-schema.json +0 -0
index 8d04897922ec..c72e0615e074 100644
@@ -750,7 +750,7 @@

# Since: 2.6
##
{ 'enum': 'CpuInfoArch',
- 'data': ['x86', 'sparc', 'ppc', 'mips', 'tricore', 'other' ] }
+ 'data': ['x86', 'sparc', 'ppc', 'mipsarch', 'tricore', 'other' ] }

##
# @CpuInfoBase:
@@ -791,7 +791,7 @@
'data': { 'x86': 'CpuInfoX86',

            'sparc': 'CpuInfoSPARC',
            'ppc': 'CpuInfoPPC',
-            'mips': 'CpuInfoMIPS',
+            'mipsarch': 'CpuInfoMIPS',
            'tricore': 'CpuInfoTricore',
            'other': 'CpuInfoOther' } }

Cheers
James

#53 lore raw

Re: [Qemu-devel] [PATCH v12 24/36] cpu: Convert CpuInfo into flat union

Quoted 30 lines · James Hogan

Hi Eric,

Quoted 8 lines · Eric Blake
(quoted diff) +8 -0
+# An enumeration of cpu types that enable additional information during
+# @query-cpus.
+#
+# Since: 2.6
+##
+{ 'enum': 'CpuInfoArch',
+  'data': ['x86', 'sparc', 'ppc', 'mips', 'tricore', 'other' ] }
+

...

Review

Quoted 12 lines
(quoted diff) +12 -0
+##
+# @CpuInfo:
+#
+# Information about a virtual CPU
+#
+# Since: 0.14.0
+##
+{ 'union': 'CpuInfo', 'base': 'CpuInfoBase', 'discriminator': 'arch',
+  'data': { 'x86': 'CpuInfoX86',
+            'sparc': 'CpuInfoSparc',
+            'ppc': 'CpuInfoPpc',
+            'mips': 'CpuInfoMips',

This is causing compilation issues on MIPS, because gcc bizarrely defines mips=1:

Quoted 2 lines
$ mips-img-linux-gnu-gcc -dM -E - </dev/null | grep '\bmips\b'
#define mips 1

Bizarre indeed.

See also
https://gcc.gnu.org/onlinedocs/gcc-5.3.0/cpp/System-specific-Predefined-Macros.html#System-specific-Predefined-Macros

Quoted 10 lines

For example:
CC qga/commands.o
In file included from /work/mips/qemu/msa/include/qapi/qmp/qobject.h:37:0,

                from /work/mips/qemu/msa/include/qapi/qmp/dispatch.h:17,
                from ./qga/guest-agent-core.h:13,
                from qga/commands.c:14:
./qapi-types.h:2568:22: error: expected identifier or ‘(’ before
numeric constant
        CpuInfoMIPS *mips;
                     ^
Quoted 5 lines

/work/mips/qemu/msa/rules.mak:57: recipe for target 'qga/commands.o' failed
make: *** [qga/commands.o] Error 1

Any suggestions for a fix that is less ugly than this:

#undef mips with a suitable comment?

[...]

author
#54 lore raw

Re: [Qemu-devel] [PATCH v12 24/36] cpu: Convert CpuInfo into flat union

In reply to James Hogan
Quoted 16 lines · James Hogan

Hi Eric,

Review

Quoted 8 lines · Eric Blake
(quoted diff) +8 -0
+# An enumeration of cpu types that enable additional information during
+# @query-cpus.
+#
+# Since: 2.6
+##
+{ 'enum': 'CpuInfoArch',
+  'data': ['x86', 'sparc', 'ppc', 'mips', 'tricore', 'other' ] }
+

This is causing compilation issues on MIPS, because gcc bizarrely defines mips=1:

Quoted 2 lines
$ mips-img-linux-gnu-gcc -dM -E - </dev/null | grep '\bmips\b'
#define mips 1

Not the first instance of namespace pollution.

Quoted 1 line

Any suggestions for a fix that is less ugly than this:

Yes, I'm posting the proper patch to poison QAPI 'mips' into C 'q_mips' shortly. That way, QMP remains backwards compatible (your approach did not preserve that).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#55 lore raw

PATCH v12 25/36 qapi: Add alias for ErrorClass

In reply to Eric Blake

The qapi enum ErrorClass is unusual that it uses 'CamelCase' names, contrary to our documented convention of preferring 'lower-case'. However, this enum is entrenched in the API; we cannot change what strings QMP outputs. Meanwhile, we want to simplify how c_enum_const() is used to generate enum constants, by moving away from the heuristics of camel_to_upper() to a more straightforward c_name(N).upper() - but doing so will rename all of the ErrorClass constants and cause churn to all client files, where the new names are aesthetically less pleasing (ERROR_CLASS_DEVICENOTFOUND looks like we can't make up our minds on whether to break between words).

So as always in computer science, solve the problem by some more
indirection: rename the qapi type to QapiErrorClass, and add a new enum ErrorClass in error.h whose members are aliases of the qapi type, but with the spelling expected elsewhere in the tree. Then, when c_enum_const() changes the munging, we only have to adjust the one alias spot.

Suggested by: Markus Armbruster <[email protected]>

---
v12: new patch

---
 include/qapi/error.h | 14 ++++++++++++++
 monitor.c            |  2 +-
 qapi/common.json     |  5 +++--
 qapi/qmp-dispatch.c  |  2 +-
 4 files changed, 19 insertions(+), 4 deletions(-)
include/qapi/error.h +14 -0
index b2362a5..299cbf4 100644
@@ -108,6 +108,20 @@
 typedef struct Error Error;
 
 /*
+ * Overall category of an error.
+ * Based on the qapi type QapiErrorClass, but reproduced here for nicer
+ * enum names.
+ */
+typedef enum ErrorClass {
+    ERROR_CLASS_GENERIC_ERROR = QAPI_ERROR_CLASS_GENERIC_ERROR,
+    ERROR_CLASS_COMMAND_NOT_FOUND = QAPI_ERROR_CLASS_COMMAND_NOT_FOUND,
+    ERROR_CLASS_DEVICE_ENCRYPTED = QAPI_ERROR_CLASS_DEVICE_ENCRYPTED,
+    ERROR_CLASS_DEVICE_NOT_ACTIVE = QAPI_ERROR_CLASS_DEVICE_NOT_ACTIVE,
+    ERROR_CLASS_DEVICE_NOT_FOUND = QAPI_ERROR_CLASS_DEVICE_NOT_FOUND,
+    ERROR_CLASS_KVM_MISSING_CAP = QAPI_ERROR_CLASS_KVM_MISSING_CAP,
+} ErrorClass;
+
+/*
  * Get @err's human-readable error message.
  */
 const char *error_get_pretty(Error *err);
monitor.c +1 -1
index 3a0df08..030e2b3 100644
@@ -403,7 +403,7 @@ static QDict *build_qmp_error_dict(Error *err)
     QObject *obj;
 
     obj = qobject_from_jsonf("{ 'error': { 'class': %s, 'desc': %s } }",
-                             ErrorClass_lookup[error_get_class(err)],
+                             QapiErrorClass_lookup[error_get_class(err)],
                              error_get_pretty(err));
 
     return qobject_to_qdict(obj);
qapi/common.json +3 -2
index bad56bf..6fb40e7 100644
@@ -3,7 +3,7 @@
 # QAPI common definitions
 
 ##
-# @ErrorClass
+# @QapiErrorClass
 #
 # QEMU error classes
 #
@@ -24,7 +24,8 @@
 #
 # Since: 1.2
 ##
-{ 'enum': 'ErrorClass',
+{ 'enum': 'QapiErrorClass',
+  # Keep this in sync with ErrorClass in error.h
   'data': [ 'GenericError', 'CommandNotFound', 'DeviceEncrypted',
             'DeviceNotActive', 'DeviceNotFound', 'KVMMissingCap' ] }
 
qapi/qmp-dispatch.c +1 -1
index 7bcc860..f36933d 100644
@@ -114,7 +114,7 @@ static QObject *do_qmp_dispatch(QObject *request, Error **errp)
 QObject *qmp_build_error_object(Error *err)
 {
     return qobject_from_jsonf("{ 'class': %s, 'desc': %s }",
-                              ErrorClass_lookup[error_get_class(err)],
+                              QapiErrorClass_lookup[error_get_class(err)],
                               error_get_pretty(err));
 }
 
Signature · 1 line
2.4.3
author
#56 lore raw

PATCH v12 26/36 qapi: Change munging of CamelCase enum values

In reply to Eric Blake

When munging enum values, the fact that we were passing the entire prefix + value through camel_to_upper() meant that enum values spelled with CamelCase could be turned into CAMEL_CASE. However, this provides a potential collision (both OneTwo and One-Two would munge into ONE_TWO) for enum types, when the same two names are valid side-by-side as qapi member names. By changing the generation of enum constants to always be prefix + '_' + c_name(value, False).upper(), and ensuring that there are no case collisions (in the next patches), we no longer have to worry about names that would be distinct as qapi members but collide as variant tag names, without having to think about what munging the heuristics in camel_to_upper() will actually perform on an enum value.

Making the change will affect enums that did not follow coding conventions, using 'CamelCase' rather than desired 'lower-case'.

Thankfully, there are only two culprits: InputButton and ErrorClass. We already tweaked ErrorClass to make it an alias of QapiErrorClass, where only the alias needs changingrather than the whole tree. So the bulk of this change is modifying INPUT_BUTTON_WHEEL_UP to the new INPUT_BUTTON_WHEELUP (and likewise for WHEELDOWN). That part of this commit may later need reverting if we rename the enum constants from 'WheelUp' to 'wheel-up' as part of moving x-input-send-event to a stable interface; but at least we have documentation bread crumbs in place to remind us (commit 513e7cd), and it matches the fact that SDL constants are also spelled SDL_BUTTON_WHEELUP.

Suggested by: Markus Armbruster <[email protected]>

---
v12: rebase to simpler ErrorClass changes, document decisions on InputButton
v11: new patch

---
 hw/input/hid.c              |  4 ++--
 hw/input/ps2.c              |  4 ++--
 hw/input/virtio-input-hid.c |  4 ++--
 include/qapi/error.h        | 12 ++++++------
 monitor.c                   |  2 +-
 scripts/qapi.py             |  2 +-
 ui/cocoa.m                  |  4 ++--
 ui/gtk.c                    |  4 ++--
 ui/input-legacy.c           |  4 ++--
 ui/sdl.c                    |  4 ++--
 ui/sdl2.c                   |  4 ++--
 ui/spice-input.c            |  4 ++--
 ui/vnc.c                    |  4 ++--
 13 files changed, 28 insertions(+), 28 deletions(-)
hw/input/hid.c +2 -2
index 12075c9..3221d29 100644
@@ -139,9 +139,9 @@ static void hid_pointer_event(DeviceState *dev, QemuConsole *src,
     case INPUT_EVENT_KIND_BTN:
         if (evt->u.btn->down) {
             e->buttons_state |= bmap[evt->u.btn->button];
-            if (evt->u.btn->button == INPUT_BUTTON_WHEEL_UP) {
+            if (evt->u.btn->button == INPUT_BUTTON_WHEELUP) {
                 e->dz--;
-            } else if (evt->u.btn->button == INPUT_BUTTON_WHEEL_DOWN) {
+            } else if (evt->u.btn->button == INPUT_BUTTON_WHEELDOWN) {
                 e->dz++;
             }
         } else {
hw/input/ps2.c +2 -2
index 9096d21..79754cd 100644
@@ -405,9 +405,9 @@ static void ps2_mouse_event(DeviceState *dev, QemuConsole *src,
     case INPUT_EVENT_KIND_BTN:
         if (evt->u.btn->down) {
             s->mouse_buttons |= bmap[evt->u.btn->button];
-            if (evt->u.btn->button == INPUT_BUTTON_WHEEL_UP) {
+            if (evt->u.btn->button == INPUT_BUTTON_WHEELUP) {
                 s->mouse_dz--;
-            } else if (evt->u.btn->button == INPUT_BUTTON_WHEEL_DOWN) {
+            } else if (evt->u.btn->button == INPUT_BUTTON_WHEELDOWN) {
                 s->mouse_dz++;
             }
         } else {
hw/input/virtio-input-hid.c +2 -2
index 5d00a03..a78d11c 100644
@@ -142,8 +142,8 @@ static const unsigned int keymap_button[INPUT_BUTTON__MAX] = {
     [INPUT_BUTTON_LEFT]              = BTN_LEFT,
     [INPUT_BUTTON_RIGHT]             = BTN_RIGHT,
     [INPUT_BUTTON_MIDDLE]            = BTN_MIDDLE,
-    [INPUT_BUTTON_WHEEL_UP]          = BTN_GEAR_UP,
-    [INPUT_BUTTON_WHEEL_DOWN]        = BTN_GEAR_DOWN,
+    [INPUT_BUTTON_WHEELUP]           = BTN_GEAR_UP,
+    [INPUT_BUTTON_WHEELDOWN]         = BTN_GEAR_DOWN,
 };
 
 static const unsigned int axismap_rel[INPUT_AXIS__MAX] = {
include/qapi/error.h +6 -6
index 299cbf4..1480f59 100644
@@ -113,12 +113,12 @@ typedef struct Error Error;
  * enum names.
  */
 typedef enum ErrorClass {
-    ERROR_CLASS_GENERIC_ERROR = QAPI_ERROR_CLASS_GENERIC_ERROR,
-    ERROR_CLASS_COMMAND_NOT_FOUND = QAPI_ERROR_CLASS_COMMAND_NOT_FOUND,
-    ERROR_CLASS_DEVICE_ENCRYPTED = QAPI_ERROR_CLASS_DEVICE_ENCRYPTED,
-    ERROR_CLASS_DEVICE_NOT_ACTIVE = QAPI_ERROR_CLASS_DEVICE_NOT_ACTIVE,
-    ERROR_CLASS_DEVICE_NOT_FOUND = QAPI_ERROR_CLASS_DEVICE_NOT_FOUND,
-    ERROR_CLASS_KVM_MISSING_CAP = QAPI_ERROR_CLASS_KVM_MISSING_CAP,
+    ERROR_CLASS_GENERIC_ERROR = QAPI_ERROR_CLASS_GENERICERROR,
+    ERROR_CLASS_COMMAND_NOT_FOUND = QAPI_ERROR_CLASS_COMMANDNOTFOUND,
+    ERROR_CLASS_DEVICE_ENCRYPTED = QAPI_ERROR_CLASS_DEVICEENCRYPTED,
+    ERROR_CLASS_DEVICE_NOT_ACTIVE = QAPI_ERROR_CLASS_DEVICENOTACTIVE,
+    ERROR_CLASS_DEVICE_NOT_FOUND = QAPI_ERROR_CLASS_DEVICENOTFOUND,
+    ERROR_CLASS_KVM_MISSING_CAP = QAPI_ERROR_CLASS_KVMMISSINGCAP,
 } ErrorClass;
 
 /*
monitor.c +1 -1
index 030e2b3..08df2b6 100644
@@ -1375,7 +1375,7 @@ static void hmp_mouse_move(Monitor *mon, const QDict *qdict)
     if (dz_str) {
         dz = strtol(dz_str, NULL, 0);
         if (dz != 0) {
-            button = (dz > 0) ? INPUT_BUTTON_WHEEL_UP : INPUT_BUTTON_WHEEL_DOWN;
+            button = (dz > 0) ? INPUT_BUTTON_WHEELUP : INPUT_BUTTON_WHEELDOWN;
             qemu_input_queue_btn(NULL, button, true);
             qemu_input_event_sync();
             qemu_input_queue_btn(NULL, button, false);
scripts/qapi.py +1 -1
index cd2982a..cde15f2 100644
@@ -1439,7 +1439,7 @@ def camel_to_upper(value):
 def c_enum_const(type_name, const_name, prefix=None):
     if prefix is not None:
         type_name = prefix
-    return camel_to_upper(type_name + '_' + const_name)
+    return camel_to_upper(type_name) + '_' + c_name(const_name, False).upper()
 
 c_name_trans = string.maketrans('.-', '__')
 
ui/cocoa.m +2 -2
index 5a0a2cf..96a41fe 100644
@@ -729,8 +729,8 @@ QemuCocoaView *cocoaView;
                 [INPUT_BUTTON_LEFT]       = MOUSE_EVENT_LBUTTON,
                 [INPUT_BUTTON_MIDDLE]     = MOUSE_EVENT_MBUTTON,
                 [INPUT_BUTTON_RIGHT]      = MOUSE_EVENT_RBUTTON,
-                [INPUT_BUTTON_WHEEL_UP]   = MOUSE_EVENT_WHEELUP,
-                [INPUT_BUTTON_WHEEL_DOWN] = MOUSE_EVENT_WHEELDN,
+                [INPUT_BUTTON_WHEELUP]    = MOUSE_EVENT_WHEELUP,
+                [INPUT_BUTTON_WHEELDOWN]  = MOUSE_EVENT_WHEELDN,
             };
             qemu_input_update_buttons(dcl->con, bmap, last_buttons, buttons);
             last_buttons = buttons;
ui/gtk.c +2 -2
index 47b37e1..40e78c5 100644
@@ -965,9 +965,9 @@ static gboolean gd_scroll_event(GtkWidget *widget, GdkEventScroll *scroll,
     InputButton btn;
 
     if (scroll->direction == GDK_SCROLL_UP) {
-        btn = INPUT_BUTTON_WHEEL_UP;
+        btn = INPUT_BUTTON_WHEELUP;
     } else if (scroll->direction == GDK_SCROLL_DOWN) {
-        btn = INPUT_BUTTON_WHEEL_DOWN;
+        btn = INPUT_BUTTON_WHEELDOWN;
     } else {
         return TRUE;
     }
ui/input-legacy.c +2 -2
index 3bc29bd..35dfc27 100644
@@ -157,7 +157,7 @@ static void legacy_mouse_event(DeviceState *dev, QemuConsole *src,
         } else {
             s->buttons &= ~bmap[evt->u.btn->button];
         }
-        if (evt->u.btn->down && evt->u.btn->button == INPUT_BUTTON_WHEEL_UP) {
+        if (evt->u.btn->down && evt->u.btn->button == INPUT_BUTTON_WHEELUP) {
             s->qemu_put_mouse_event(s->qemu_put_mouse_event_opaque,
                                     s->axis[INPUT_AXIS_X],
                                     s->axis[INPUT_AXIS_Y],
@@ -165,7 +165,7 @@ static void legacy_mouse_event(DeviceState *dev, QemuConsole *src,
                                     s->buttons);
         }
         if (evt->u.btn->down &&
-            evt->u.btn->button == INPUT_BUTTON_WHEEL_DOWN) {
+            evt->u.btn->button == INPUT_BUTTON_WHEELDOWN) {
             s->qemu_put_mouse_event(s->qemu_put_mouse_event_opaque,
                                     s->axis[INPUT_AXIS_X],
                                     s->axis[INPUT_AXIS_Y],
ui/sdl.c +2 -2
index f4aa0f2..c837436 100644
@@ -469,8 +469,8 @@ static void sdl_send_mouse_event(int dx, int dy, int x, int y, int state)
         [INPUT_BUTTON_LEFT]       = SDL_BUTTON(SDL_BUTTON_LEFT),
         [INPUT_BUTTON_MIDDLE]     = SDL_BUTTON(SDL_BUTTON_MIDDLE),
         [INPUT_BUTTON_RIGHT]      = SDL_BUTTON(SDL_BUTTON_RIGHT),
-        [INPUT_BUTTON_WHEEL_UP]   = SDL_BUTTON(SDL_BUTTON_WHEELUP),
-        [INPUT_BUTTON_WHEEL_DOWN] = SDL_BUTTON(SDL_BUTTON_WHEELDOWN),
+        [INPUT_BUTTON_WHEELUP]    = SDL_BUTTON(SDL_BUTTON_WHEELUP),
+        [INPUT_BUTTON_WHEELDOWN]  = SDL_BUTTON(SDL_BUTTON_WHEELDOWN),
     };
     static uint32_t prev_state;
 
ui/sdl2.c +2 -2
index 4be992a..cf38df2 100644
@@ -504,9 +504,9 @@ static void handle_mousewheel(SDL_Event *ev)
     InputButton btn;
 
     if (wev->y > 0) {
-        btn = INPUT_BUTTON_WHEEL_UP;
+        btn = INPUT_BUTTON_WHEELUP;
     } else if (wev->y < 0) {
-        btn = INPUT_BUTTON_WHEEL_DOWN;
+        btn = INPUT_BUTTON_WHEELDOWN;
     } else {
         return;
     }
ui/spice-input.c +2 -2
index 96f19aa..2b3b9ed 100644
@@ -111,8 +111,8 @@ static void spice_update_buttons(QemuSpicePointer *pointer,
         [INPUT_BUTTON_LEFT]        = 0x01,
         [INPUT_BUTTON_MIDDLE]      = 0x04,
         [INPUT_BUTTON_RIGHT]       = 0x02,
-        [INPUT_BUTTON_WHEEL_UP]    = 0x10,
-        [INPUT_BUTTON_WHEEL_DOWN]  = 0x20,
+        [INPUT_BUTTON_WHEELUP]     = 0x10,
+        [INPUT_BUTTON_WHEELDOWN]   = 0x20,
     };
 
     if (wheel < 0) {
ui/vnc.c +2 -2
index c6cbb9d..e74c84a 100644
@@ -1675,8 +1675,8 @@ static void pointer_event(VncState *vs, int button_mask, int x, int y)
         [INPUT_BUTTON_LEFT]       = 0x01,
         [INPUT_BUTTON_MIDDLE]     = 0x02,
         [INPUT_BUTTON_RIGHT]      = 0x04,
-        [INPUT_BUTTON_WHEEL_UP]   = 0x08,
-        [INPUT_BUTTON_WHEEL_DOWN] = 0x10,
+        [INPUT_BUTTON_WHEELUP]    = 0x08,
+        [INPUT_BUTTON_WHEELDOWN]  = 0x10,
     };
     QemuConsole *con = vs->vd->dcl.con;
     int width = pixman_image_get_width(vs->vd->server);
Signature · 1 line
2.4.3
#57 lore raw

Re: [Qemu-devel] [PATCH v12 26/36] qapi: Change munging of CamelCase enum values

Quoted 13 lines · Eric Blake

When munging enum values, the fact that we were passing the entire prefix + value through camel_to_upper() meant that enum values spelled with CamelCase could be turned into CAMEL_CASE. However, this provides a potential collision (both OneTwo and One-Two would munge into ONE_TWO) for enum types, when the same two names are valid side-by-side as qapi member names. By changing the generation of enum constants to always be prefix + '_' + c_name(value, False).upper(), and ensuring that there are no case collisions (in the next patches), we no longer have to worry about names that would be distinct as qapi members but collide as variant tag names, without having to think about what munging the heuristics in camel_to_upper() will actually perform on an enum value.

Quoted 6 lines

Making the change will affect enums that did not follow coding conventions, using 'CamelCase' rather than desired 'lower-case'.

Thankfully, there are only two culprits: InputButton and ErrorClass. We already tweaked ErrorClass to make it an alias of QapiErrorClass, where only the alias needs changingrather than the whole tree. So

s/changingrather/changing rather/

Quoted 9 lines

the bulk of this change is modifying INPUT_BUTTON_WHEEL_UP to the new INPUT_BUTTON_WHEELUP (and likewise for WHEELDOWN). That part of this commit may later need reverting if we rename the enum constants from 'WheelUp' to 'wheel-up' as part of moving x-input-send-event to a stable interface; but at least we have documentation bread crumbs in place to remind us (commit 513e7cd), and it matches the fact that SDL constants are also spelled SDL_BUTTON_WHEELUP.

Quoted 6 lines

Suggested by: Markus Armbruster <[email protected]>

---
v12: rebase to simpler ErrorClass changes, document decisions on InputButton

The ugliness is now confined to just the definition of enum ErrorClass. Good.

author
#58 lore raw

PATCH v12 27/36 qapi: Forbid case-insensitive clashes

In reply to Eric Blake

In general, designing user interfaces that rely on case distinction is poor practice. Another benefit of enforcing a restriction against case-insensitive clashes is that we no longer have to worry about the situation of enum values that could be distinguished by case if mapped by c_name(), but which cannot be distinguished when mapped to C as ALL_CAPS by c_enum_const() [via c_name(name, False).upper()]. Thus, having the generator look for case collisions up front will prevent developers from worrying whether different munging rules for member names compared to enum values as a discriminator will cause any problems in qapi unions.

We do not have to care about c_name(name, True) vs. c_name(name, False), as long as any pair of munged names being compared were munged using the same flag value to c_name(). This is because commit 9fb081e already reserved names munging to 'q_', and this patch extends the reservation to 'Q_'; and because recent patches fixed things to ensure the only thing done by the flag is adding the prefix 'q_', that the only use of '.' (which also munges to '_') is in downstream extension prefixes, and that enum munging does not add underscores to any CamelCase values.

However, we DO have to care about the fact that we have a command 'stop' and an event 'STOP' (currently the only case collision of any of our .json entities). To solve that, use some tricks in the lookup map for entities. With some careful choice of keys, we can bisect the map into two collision pools (name.upper() for events, name.lower() for all else). Since we already document that no two entities can have the exact same spelling, an entity can only occur under one of the two partitions of the map. We could go further and enforce that events are named with 'ALL_CAPS' and that nothing else is named in that manner; but that can be done as a followup if desired. We could also separate commands from type names, but then we no longer have a convenient upper vs. lower partitioning allowing us to share a single dictionary.

In order to keep things stable, the visit() method has to ensure that it visits entities sorted by their real name, and not by the new munged keys of the dictionary; Python's attrgetter is a lifesaver for this task.

There is also the possibility that we may want to add a future extension to QMP of teaching it to be case-insensitive (the user could request command 'Quit' instead of 'quit', or could spell a struct field as 'CPU' instead of 'cpu'). But for that to be a practical extension, we cannot break backwards compatibility with any existing struct that was already relying on case sensitivity. Fortunately, after a recent patch cleaned up CpuInfo, there are no such existing qapi structs. Of course, the idea of a future extension is not as strong of a reason to make the change.

At any rate, it is easier to be strict now, and relax things later if we find a reason to need case-sensitive QMP members, than it would be to wish we had the restriction in place.

Add new tests args-case-clash.json and command-type-case-clash.json to demonstrate that the collision detection works.

---
v12: add in entity case collisions (required sharing two maps), improve commit message
v11: rebase to latest, don't focus so hard on future case-insensitive extensions, adjust commit message
v10: new patch

---
 docs/qapi-code-gen.txt                         |  7 +++++
 scripts/qapi.py                                | 41 +++++++++++++++++++++-----
 tests/Makefile                                 |  2 ++
 tests/qapi-schema/args-case-clash.err          |  1 +
 tests/qapi-schema/args-case-clash.exit         |  1 +
 tests/qapi-schema/args-case-clash.json         |  4 +++
 tests/qapi-schema/args-case-clash.out          |  0
 tests/qapi-schema/command-type-case-clash.err  |  1 +
 tests/qapi-schema/command-type-case-clash.exit |  1 +
 tests/qapi-schema/command-type-case-clash.json |  3 ++
 tests/qapi-schema/command-type-case-clash.out  |  0
 11 files changed, 53 insertions(+), 8 deletions(-)
 create mode 100644 tests/qapi-schema/args-case-clash.err
 create mode 100644 tests/qapi-schema/args-case-clash.exit
 create mode 100644 tests/qapi-schema/args-case-clash.json
 create mode 100644 tests/qapi-schema/args-case-clash.out
 create mode 100644 tests/qapi-schema/command-type-case-clash.err
 create mode 100644 tests/qapi-schema/command-type-case-clash.exit
 create mode 100644 tests/qapi-schema/command-type-case-clash.json
 create mode 100644 tests/qapi-schema/command-type-case-clash.out
docs/qapi-code-gen.txt +7 -0
index b01a691..1f6cb16 100644
@@ -102,6 +102,13 @@ single-dimension array of that type; multi-dimension arrays are not
 directly supported (although an array of a complex struct that
 contains an array member is possible).
 
+Client JSON Protocol is case-sensitive.  However, the generator
+rejects attempts to create object members that differ only in case
+after normalization (thus 'a-b' and 'A_B' collide); and likewise
+rejects attempts to create commands or types that differ only in case,
+or events that differ only in case (it is possible to have a command
+and event map to the same case-insensitive name, though).
+
 Types, commands, and events share a common namespace.  Therefore,
 generally speaking, type definitions should always use CamelCase for
 user-defined type names, while built-in types are lowercase. Type
scripts/qapi.py +33 -8

86 lines not on this page · raw message

tests/Makefile +2 -0
index a8a3b12..762b0ca 100644
@@ -242,6 +242,7 @@ qapi-schema += args-alternate.json
 qapi-schema += args-any.json
 qapi-schema += args-array-empty.json
 qapi-schema += args-array-unknown.json
+qapi-schema += args-case-clash.json
 qapi-schema += args-int.json
 qapi-schema += args-invalid.json
 qapi-schema += args-member-array-bad.json
@@ -256,6 +257,7 @@ qapi-schema += bad-type-bool.json
 qapi-schema += bad-type-dict.json
 qapi-schema += bad-type-int.json
 qapi-schema += command-int.json
+qapi-schema += command-type-case-clash.json
 qapi-schema += comments.json
 qapi-schema += double-data.json
 qapi-schema += double-type.json
tests/qapi-schema/args-case-clash.err new file +1 -0
new file mode 100644
index 0000000..0fafe75
@@ -0,0 +1 @@
+tests/qapi-schema/args-case-clash.json:4: 'A' (parameter of oops) collides with 'a' (parameter of oops)
tests/qapi-schema/args-case-clash.exit new file +1 -0
new file mode 100644
index 0000000..d00491f
@@ -0,0 +1 @@
+1
tests/qapi-schema/args-case-clash.json new file +4 -0
new file mode 100644
index 0000000..e6f0625
@@ -0,0 +1,4 @@
+# C member name collision
+# Reject members that clash case-insensitively, even though our mapping to
+# C names preserves case.
+{ 'command': 'oops', 'data': { 'a': 'str', 'A': 'int' } }
tests/qapi-schema/args-case-clash.out new file +0 -0
new file mode 100644
index 0000000..e69de29
tests/qapi-schema/command-type-case-clash.err new file +1 -0
new file mode 100644
index 0000000..b1cc697
@@ -0,0 +1 @@
+tests/qapi-schema/command-type-case-clash.json:3: Entity 'foo' collides with 'Foo'
tests/qapi-schema/command-type-case-clash.exit new file +1 -0
new file mode 100644
index 0000000..d00491f
@@ -0,0 +1 @@
+1
tests/qapi-schema/command-type-case-clash.json new file +3 -0
new file mode 100644
index 0000000..1f6adee
@@ -0,0 +1,3 @@
+# we reject commands that would differ only case from a type
+{ 'struct': 'Foo', 'data': { 'i': 'int' } }
+{ 'command': 'foo', 'data': { 'character': 'str' } }
tests/qapi-schema/command-type-case-clash.out new file +0 -0
new file mode 100644
index 0000000..e69de29
Signature · 1 line
2.4.3
#59 lore raw

Re: [Qemu-devel] [PATCH v12 27/36] qapi: Forbid case-insensitive clashes

Quoted 123 lines · Eric Blake

In general, designing user interfaces that rely on case distinction is poor practice. Another benefit of enforcing a restriction against case-insensitive clashes is that we no longer have to worry about the situation of enum values that could be distinguished by case if mapped by c_name(), but which cannot be distinguished when mapped to C as ALL_CAPS by c_enum_const() [via c_name(name, False).upper()]. Thus, having the generator look for case collisions up front will prevent developers from worrying whether different munging rules for member names compared to enum values as a discriminator will cause any problems in qapi unions.

We do not have to care about c_name(name, True) vs. c_name(name, False), as long as any pair of munged names being compared were munged using the same flag value to c_name(). This is because commit 9fb081e already reserved names munging to 'q_', and this patch extends the reservation to 'Q_'; and because recent patches fixed things to ensure the only thing done by the flag is adding the prefix 'q_', that the only use of '.' (which also munges to '_') is in downstream extension prefixes, and that enum munging does not add underscores to any CamelCase values.

However, we DO have to care about the fact that we have a command 'stop' and an event 'STOP' (currently the only case collision of any of our .json entities). To solve that, use some tricks in the lookup map for entities. With some careful choice of keys, we can bisect the map into two collision pools (name.upper() for events, name.lower() for all else). Since we already document that no two entities can have the exact same spelling, an entity can only occur under one of the two partitions of the map. We could go further and enforce that events are named with 'ALL_CAPS' and that nothing else is named in that manner; but that can be done as a followup if desired. We could also separate commands from type names, but then we no longer have a convenient upper vs. lower partitioning allowing us to share a single dictionary.

In order to keep things stable, the visit() method has to ensure that it visits entities sorted by their real name, and not by the new munged keys of the dictionary; Python's attrgetter is a lifesaver for this task.

There is also the possibility that we may want to add a future extension to QMP of teaching it to be case-insensitive (the user could request command 'Quit' instead of 'quit', or could spell a struct field as 'CPU' instead of 'cpu'). But for that to be a practical extension, we cannot break backwards compatibility with any existing struct that was already relying on case sensitivity. Fortunately, after a recent patch cleaned up CpuInfo, there are no such existing qapi structs. Of course, the idea of a future extension is not as strong of a reason to make the change.

At any rate, it is easier to be strict now, and relax things later if we find a reason to need case-sensitive QMP members, than it would be to wish we had the restriction in place.

Add new tests args-case-clash.json and command-type-case-clash.json to demonstrate that the collision detection works.

---
v12: add in entity case collisions (required sharing two maps), improve commit message
v11: rebase to latest, don't focus so hard on future case-insensitive extensions, adjust commit message
v10: new patch

---
 docs/qapi-code-gen.txt                         |  7 +++++
 scripts/qapi.py                                | 41 +++++++++++++++++++++-----
 tests/Makefile                                 |  2 ++
 tests/qapi-schema/args-case-clash.err          |  1 +
 tests/qapi-schema/args-case-clash.exit         |  1 +
 tests/qapi-schema/args-case-clash.json         |  4 +++
 tests/qapi-schema/args-case-clash.out          |  0
 tests/qapi-schema/command-type-case-clash.err  |  1 +
 tests/qapi-schema/command-type-case-clash.exit |  1 +
 tests/qapi-schema/command-type-case-clash.json |  3 ++
 tests/qapi-schema/command-type-case-clash.out  |  0
 11 files changed, 53 insertions(+), 8 deletions(-)
 create mode 100644 tests/qapi-schema/args-case-clash.err
 create mode 100644 tests/qapi-schema/args-case-clash.exit
 create mode 100644 tests/qapi-schema/args-case-clash.json
 create mode 100644 tests/qapi-schema/args-case-clash.out
 create mode 100644 tests/qapi-schema/command-type-case-clash.err
 create mode 100644 tests/qapi-schema/command-type-case-clash.exit
 create mode 100644 tests/qapi-schema/command-type-case-clash.json
 create mode 100644 tests/qapi-schema/command-type-case-clash.out
docs/qapi-code-gen.txt +7 -0
index b01a691..1f6cb16 100644
@@ -102,6 +102,13 @@ single-dimension array of that type; multi-dimension arrays are not
 directly supported (although an array of a complex struct that
 contains an array member is possible).
 
+Client JSON Protocol is case-sensitive.  However, the generator
+rejects attempts to create object members that differ only in case
+after normalization (thus 'a-b' and 'A_B' collide); and likewise
+rejects attempts to create commands or types that differ only in case,
+or events that differ only in case (it is possible to have a command
+and event map to the same case-insensitive name, though).
+
 Types, commands, and events share a common namespace.  Therefore,
 generally speaking, type definitions should always use CamelCase for
 user-defined type names, while built-in types are lowercase. Type
scripts/qapi.py +1 -0
index cde15f2..e41dbaf 100644
@@ -13,6 +13,7 @@
 
 import re
 from ordereddict import OrderedDict
+from operator import attrgetter
 import errno
 import getopt
 import os
@@ -378,9 +379,9 @@ def check_name(expr_info, source, name, allow_optional=False,
     # code always prefixes it with the enum name

scripts/qapi.py def check_name(expr_info, source, name, allow_optional=False, · line 387

Quoted 10 lines
scripts/qapi.py +2 -2
     if enum_member and membername[0].isdigit():
         membername = 'D' + membername
-    # Reserve the entire 'q_' namespace for c_name()
+    # Reserve the entire 'q_'/'Q_' namespace for c_name()
     if not valid_name.match(membername) or \
-       c_name(membername, False).startswith('q_'):
+       c_name(membername, False).upper().startswith('Q_'):
         raise QAPIExprError(expr_info,
                             "%s uses invalid name '%s'" % (source, name))
 

We use .upper() to get a case-insensitive (in the C locale!) string compare. That's fine. .lower() would be just as fine. But see [*] below.

Quoted 53 lines
scripts/qapi.py +21 -5
@@ -1040,7 +1041,7 @@ class QAPISchemaObjectTypeMember(object):
         assert self.type
 
     def check_clash(self, info, seen):
-        cname = c_name(self.name)
+        cname = c_name(self.name).upper()
         if cname in seen:
             raise QAPIExprError(info,
                                 "%s collides with %s"
@@ -1085,7 +1086,7 @@ class QAPISchemaObjectTypeVariants(object):
 
     def check(self, schema, seen):
         if not self.tag_member:    # flat union
-            self.tag_member = seen[c_name(self.tag_name)]
+            self.tag_member = seen[c_name(self.tag_name).upper()]
             assert self.tag_name == self.tag_member.name
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
@@ -1189,6 +1190,8 @@ class QAPISchema(object):
     def __init__(self, fname):
         try:
             self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
+            # _entity_dict holds two namespaces: events are stored via
+            # name.upper(), commands and types are stored via name.lower().
             self._entity_dict = {}
             self._predefining = True
             self._def_predefineds()
@@ -1202,11 +1205,32 @@ class QAPISchema(object):
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
-        assert ent.name not in self._entity_dict
-        self._entity_dict[ent.name] = ent
+        # On insertion, we need to check for an exact match in either
+        # namespace, then for case collision in a single namespace
+        if self.lookup_entity(ent.name):
+            raise QAPIExprError(ent.info,
+                                "Entity '%s' already defined" % end.name)
+        cname = c_name(ent.name)
+        if isinstance(ent, QAPISchemaEvent):
+            cname = cname.upper()
+        else:
+            cname = cname.lower()
+        if cname in self._entity_dict:
+            raise QAPIExprError(ent.info,
+                                "Entity '%s' collides with '%s'"
+                                % (ent.name, self._entity_dict[cname].name))
+        self._entity_dict[cname] = ent
 
     def lookup_entity(self, name, typ=None):
-        ent = self._entity_dict.get(name)
+        # No thanks to 'stop'/'STOP', we have to check two namespaces during
+        # lookups, but only return a result on exact match.

scripts/qapi.py class QAPISchema(object): · line 1236

Quoted 10 lines
scripts/qapi.py +7 -0
+        ent1 = self._entity_dict.get(c_name(name).lower())   # commands, types
+        ent2 = self._entity_dict.get(c_name(name).upper())   # events
+        ent = None
+        if ent1 and ent1.name == name:
+            ent = ent1
+        elif ent2 and ent2.name == name:
+            ent = ent2
         if typ and not isinstance(ent, typ):
             return None
         return ent

Let's see how this works.

Before the patch, _entity_dict's key serves one purpose: mapping names to entities the obvious way: use entity name as dictionary key.

lookup_entity():
ent = self._entity_dict.get(name)
# not ent or ent.name == name

_def_entity():
assert ent.name not in self._entity_dict
self._entity_dict[ent.name] = ent

Your patch presses _entity_dict into clash detection service. I'm going to split the change into pieces I can understand.

First step changes the key from name to c_name(name) to ignore [-._] differences (the ticklish identifier protection has no effect). The dictionary lookup changes from "the entity with this name" to "the entity whose name clashes with this name". To preserve semantics, the two functions need to compensate.

lookup_entity():
ent = self._entity_dict.get(c_name(name))
if ent and ent.name != name
ent = None
# not ent or ent.name == name

_def_entity():
cname = c_name(ent.name)
assert cname not in self._entity_dict
self._entity_dict[cname] = ent

A bit less obvious, but not too bad.

We can then turn _def_entity()'s assertion into error reporting. It splits into two separate errors, "already defined" and "collides".

_def_entity():
if self.lookup_entity(ent.name):
raise QAPIExprError(ent.info,
"Entity '%s' already defined" % end.name)
cname = c_name(ent.name)
if cname in self._entity_dict:
raise QAPIExprError(ent.info,
"Entity '%s' collides with '%s'" % (ent.name, self._entity_dict[cname].name))
self._entity_dict[cname] = ent

To catch case-insensitive clashes, you change the key further to .lower(). No additional compensation necessary.

However, this makes command 'stop' clash with event 'STOP'. You cleverly sidestep that by partitioning _entity_dict's key space into upper case for events and lower case for everything else. Works, because every entity name contains at least one letter.

_def_entity()'s cname = c_name(ent.name).lower() becomes

cname = c_name(ent.name)
if isinstance(ent, QAPISchemaEvent):
cname = cname.upper()
else:
cname = cname.lower()

Digression [*]: I think using .lower() in check_name() would be slightly cleaner, because it matches the case mapping we use here.

lookup_entity() is even worse off. There, we don't know whether we're looking for an event or not, so we try both. If the entity is defined, exactly one will succeed, else none. Resulting code:

ent1 = self._entity_dict.get(c_name(name).lower()) # commands, types
ent2 = self._entity_dict.get(c_name(name).upper()) # events
ent = None
if ent1 and ent1.name == name:
ent = ent1
elif ent2 and ent2.name == name:
ent = ent2

All right, this should work.

Now let me try claw back some clarity.

First try: refine your approach.

Observe that all but one caller of lookup_entity() actually look up types. The one caller that doesn't is _def_entity(). Change it to:

cname = c_name(ent.name)
if isinstance(ent, QAPISchemaEvent):
cname = cname.upper()
else:
cname = cname.lower()
if self._lookup_entity(cname)
raise QAPIExprError(ent.info,
"Entity '%s' already defined" % end.name)
if cname in self._entity_dict:
raise QAPIExprError(ent.info,
"Entity '%s' collides with '%s'" % (ent.name, self._entity_dict[cname].name))

where _lookup_entity() is back to the much simpler version:

ent = self._entity_dict.get(cname)
if ent and ent.name != cname
ent = None

lookup_type() becomes:

def lookup_type(self, name, typ=QAPISchemaType):
return self._lookup_entity(name, typ)

and the third caller _make_implicit_object_type() goes from

if not self.lookup_entity(name, QAPISchemaObjectType):

to

if not self.lookup_type(name, QAPISchemaObjectType):

Another possible improvement is hiding the case-folding logic in methods. Have a QAPISchemaEntity.c_namecase() that returns self.c_name().lower(). Overwrite it in QAPISchemaEvent to return .upper() instead. Use it to make _def_entity() less ugly.

Probably not worthwhile unless more uses of .c_namecase() pop up.

Second approach: don't use _entity_dict for clash detection! Have a second dictionary.

lookup_entity() remains untouched. _def_entity() becomes something like:

if ent.name in self._entity_dict:
raise QAPIExprError(ent.info,
"Entity '%s' already defined" % end.name)
if ent.c_namecase() in self._clash_dict:
raise QAPIExprError(ent.info,
"Entity '%s' collides with '%s'" % (ent.name, self._entity_dict[cname].name))
self._entity_dict[ent.name] = ent self._clash_dict[cname] = ent

or the equivalent code without c_namecase().

scripts/qapi.py class QAPISchema(object): · line 1421

Quoted 10 lines
scripts/qapi.py +2 -1
@@ -1390,7 +1414,8 @@ class QAPISchema(object):
 
     def visit(self, visitor):
         visitor.visit_begin(self)
-        for (name, entity) in sorted(self._entity_dict.items()):
+        for entity in sorted(self._entity_dict.values(),
+                             key=attrgetter('name')):
             if visitor.visit_needed(entity):
                 entity.visit(visitor)
         visitor.visit_end()

Cool trick.

The order of items in dict.items() may vary:

Keys and values are listed in an arbitrary order which is non-random, varies across Python implementations, and depends on the dictionary’s history of insertions and deletions.

https://docs.python.org/2.7/library/stdtypes.html#dict.items

By sorting, we go from "arbitrary order that can vary" to "arbitrary fixed order". That's our only reason for sorting.

Your trick preserves the arbitrary order we use before the patch. Good idea, because it makes verifying the patch doesn't mess up output easier.

We could revert the trick in a followup patch, however.

Quoted 53 lines
tests/Makefile +2 -0
index a8a3b12..762b0ca 100644
@@ -242,6 +242,7 @@ qapi-schema += args-alternate.json
 qapi-schema += args-any.json
 qapi-schema += args-array-empty.json
 qapi-schema += args-array-unknown.json
+qapi-schema += args-case-clash.json
 qapi-schema += args-int.json
 qapi-schema += args-invalid.json
 qapi-schema += args-member-array-bad.json
@@ -256,6 +257,7 @@ qapi-schema += bad-type-bool.json
 qapi-schema += bad-type-dict.json
 qapi-schema += bad-type-int.json
 qapi-schema += command-int.json
+qapi-schema += command-type-case-clash.json
 qapi-schema += comments.json
 qapi-schema += double-data.json
 qapi-schema += double-type.json
tests/qapi-schema/args-case-clash.err new file +1 -0
new file mode 100644
index 0000000..0fafe75
@@ -0,0 +1 @@
+tests/qapi-schema/args-case-clash.json:4: 'A' (parameter of oops) collides with 'a' (parameter of oops)
tests/qapi-schema/args-case-clash.exit new file +1 -0
new file mode 100644
index 0000000..d00491f
@@ -0,0 +1 @@
+1
tests/qapi-schema/args-case-clash.json new file +4 -0
new file mode 100644
index 0000000..e6f0625
@@ -0,0 +1,4 @@
+# C member name collision
+# Reject members that clash case-insensitively, even though our mapping to
+# C names preserves case.
+{ 'command': 'oops', 'data': { 'a': 'str', 'A': 'int' } }
tests/qapi-schema/args-case-clash.out new file +0 -0
new file mode 100644
index 0000000..e69de29
tests/qapi-schema/command-type-case-clash.err new file +0 -0
new file mode 100644
index 0000000..b1cc697

tests/qapi-schema/command-type-case-clash.err line 1

Quoted 1 line
tests/qapi-schema/command-type-case-clash.err new file +1 -0
@@ -0,0 +1 @@
+tests/qapi-schema/command-type-case-clash.json:3: Entity 'foo' collides with 'Foo'

The message's location is foo's. An additional message "'Foo' defined here" would be nice. Just an idea, could be done later.

Quoted 19 lines
tests/qapi-schema/command-type-case-clash.exit new file +1 -0
new file mode 100644
index 0000000..d00491f
@@ -0,0 +1 @@
+1
tests/qapi-schema/command-type-case-clash.json new file +3 -0
new file mode 100644
index 0000000..1f6adee
@@ -0,0 +1,3 @@
+# we reject commands that would differ only case from a type
+{ 'struct': 'Foo', 'data': { 'i': 'int' } }
+{ 'command': 'foo', 'data': { 'character': 'str' } }
tests/qapi-schema/command-type-case-clash.out new file +0 -0
new file mode 100644
index 0000000..e69de29
author
#60 lore raw

Re: [Qemu-devel] [PATCH v12 27/36] qapi: Forbid case-insensitive clashes

Quoted 15 lines · Markus Armbruster
Quoted 12 lines · Eric Blake

In general, designing user interfaces that rely on case distinction is poor practice. Another benefit of enforcing a restriction against case-insensitive clashes is that we no longer have to worry about the situation of enum values that could be distinguished by case if mapped by c_name(), but which cannot be distinguished when mapped to C as ALL_CAPS by c_enum_const() [via c_name(name, False).upper()]. Thus, having the generator look for case collisions up front will prevent developers from worrying whether different munging rules for member names compared to enum values as a discriminator will cause any problems in qapi unions.

Quoted 11 lines

scripts/qapi.py def check_name(expr_info, source, name, allow_optional=False, · line 386

Quoted 11 lines
scripts/qapi.py +2 -2
@@ -378,9 +379,9 @@ def check_name(expr_info, source, name, allow_optional=False,
     # code always prefixes it with the enum name
     if enum_member and membername[0].isdigit():
         membername = 'D' + membername
-    # Reserve the entire 'q_' namespace for c_name()
+    # Reserve the entire 'q_'/'Q_' namespace for c_name()
     if not valid_name.match(membername) or \
-       c_name(membername, False).startswith('q_'):
+       c_name(membername, False).upper().startswith('Q_'):
         raise QAPIExprError(expr_info,
                             "%s uses invalid name '%s'" % (source, name))

I'll switch to lower() here, for consistency.

Quoted 7 lines

Now let me try claw back some clarity.

First try: refine your approach.

Observe that all but one caller of lookup_entity() actually look up types. The one caller that doesn't is _def_entity(). Change it to:

Quoted 6 lines

cname = c_name(ent.name)
if isinstance(ent, QAPISchemaEvent):
cname = cname.upper()
else:
cname = cname.lower()
if self._lookup_entity(cname)

I think lookup_entity would need two parameters: the munged name, and the actual name...

Quoted 11 lines

raise QAPIExprError(ent.info,
"Entity '%s' already defined" % end.name)
if cname in self._entity_dict:
raise QAPIExprError(ent.info,
"Entity '%s' collides with '%s'" % (ent.name, self._entity_dict[cname].name))

where _lookup_entity() is back to the much simpler version:

ent = self._entity_dict.get(cname)
if ent and ent.name != cname

...because ent.name does not have to match cname.

Quoted 6 lines

ent = None

lookup_type() becomes:

def lookup_type(self, name, typ=QAPISchemaType):
return self._lookup_entity(name, typ)

I don't know that lookup_type needs an optional parameter.

Quoted 8 lines

and the third caller _make_implicit_object_type() goes from

if not self.lookup_entity(name, QAPISchemaObjectType):

to

if not self.lookup_type(name, QAPISchemaObjectType):

And a plain lookup_type(name) is what we should have been using here all along.

Quoted 7 lines

Another possible improvement is hiding the case-folding logic in methods. Have a QAPISchemaEntity.c_namecase() that returns self.c_name().lower(). Overwrite it in QAPISchemaEvent to return .upper() instead. Use it to make _def_entity() less ugly.

Probably not worthwhile unless more uses of .c_namecase() pop up.

I kind of like it. I think I'll propose a 26.5 that implements that, then redo 27 on top of it (without reposting the entire series).

Quoted 3 lines

Second approach: don't use _entity_dict for clash detection! Have a second dictionary.

I debated about doing that the first time around. But it's probably a bit easier to follow, so I think I'll do it.

Quoted 12 lines

scripts/qapi.py class QAPISchema(object): · line 1421

Quoted 10 lines
scripts/qapi.py +2 -1
@@ -1390,7 +1414,8 @@ class QAPISchema(object):
 
     def visit(self, visitor):
         visitor.visit_begin(self)
-        for (name, entity) in sorted(self._entity_dict.items()):
+        for entity in sorted(self._entity_dict.values(),
+                             key=attrgetter('name')):
             if visitor.visit_needed(entity):
                 entity.visit(visitor)
         visitor.visit_end()

Cool trick.

and with two dicts, I wouldn't need this trick.

Quoted 6 lines

tests/qapi-schema/command-type-case-clash.err line 1

Quoted 3 lines

+++ b/tests/qapi-schema/command-type-case-clash.err

tests/qapi-schema/command-type-case-clash.err new file +1 -0
@@ -0,0 +1 @@
+tests/qapi-schema/command-type-case-clash.json:3: Entity 'foo' collides with 'Foo'

The message's location is foo's. An additional message "'Foo' defined here" would be nice. Just an idea, could be done later.

The way we currently generate a single line message is by raising a python exception. I guess we could rework things to build up lines at a time, then finally raise the complete multi-line message on a pre-formed string, rather than the current building the message as a parameter to the exception constructor. But any changes on this front will have to wait for later.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#61 lore raw

Re: [Qemu-devel] [PATCH v12 27/36] qapi: Forbid case-insensitive clashes

Quoted 22 lines · Eric Blake
Quoted 21 lines · Markus Armbruster
Quoted 12 lines · Eric Blake

In general, designing user interfaces that rely on case distinction is poor practice. Another benefit of enforcing a restriction against case-insensitive clashes is that we no longer have to worry about the situation of enum values that could be distinguished by case if mapped by c_name(), but which cannot be distinguished when mapped to C as ALL_CAPS by c_enum_const() [via c_name(name, False).upper()]. Thus, having the generator look for case collisions up front will prevent developers from worrying whether different munging rules for member names compared to enum values as a discriminator will cause any problems in qapi unions.

Another possible improvement is hiding the case-folding logic in methods. Have a QAPISchemaEntity.c_namecase() that returns self.c_name().lower(). Overwrite it in QAPISchemaEvent to return .upper() instead. Use it to make _def_entity() less ugly.

Quoted 4 lines
Quoted 1 line

Probably not worthwhile unless more uses of .c_namecase() pop up.

I kind of like it. I think I'll propose a 26.5 that implements that, then redo 27 on top of it (without reposting the entire series).

Actually, it's not worth splitting that into a separate patch (it was just 6 lines, and I couldn't find any other uses).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#62 lore raw

Re: [Qemu-devel] [PATCH v12 27/36] qapi: Forbid case-insensitive clashes

In reply to Eric Blake
Quoted 14 lines · Eric Blake

In general, designing user interfaces that rely on case distinction is poor practice. Another benefit of enforcing a restriction against case-insensitive clashes is that we no longer have to worry about the situation of enum values that could be distinguished by case if mapped by c_name(), but which cannot be distinguished when mapped to C as ALL_CAPS by c_enum_const() [via c_name(name, False).upper()]. Thus, having the generator look for case collisions up front will prevent developers from worrying whether different munging rules for member names compared to enum values as a discriminator will cause any problems in qapi unions.

scripts/qapi.py class QAPISchema(object): · line 1212

Quoted 10 lines
scripts/qapi.py +5 -2
@@ -1202,11 +1205,32 @@ class QAPISchema(object):
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
-        assert ent.name not in self._entity_dict
-        self._entity_dict[ent.name] = ent
+        # On insertion, we need to check for an exact match in either
+        # namespace, then for case collision in a single namespace
+        if self.lookup_entity(ent.name):
+            raise QAPIExprError(ent.info,
+                                "Entity '%s' already defined" % end.name)

Latent typo (s/end/ent/); we can't hit this line of code until we nuke ad hoc parser checks. So using an assert here is better in the meantime.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#63 lore raw

PATCH 1/1 fixup! qapi: Forbid case-insensitive clashes

In reply to Eric Blake

[Replace the old commit message with this:]

qapi: Forbid case-insensitive clashes

In general, designing user interfaces that rely on case distinction is poor practice. Another benefit of enforcing a restriction against case-insensitive clashes is that we no longer have to worry about the situation of enum values that could be distinguished by case if mapped by c_name(), but which cannot be distinguished when mapped to C as ALL_CAPS by c_enum_const() [via c_name(name, False).upper()]. Thus, having the generator look for case collisions up front will prevent developers from worrying whether different munging rules for member names compared to enum values as a discriminator will cause any problems in qapi unions.

We do not have to care about c_name(name, True) vs. c_name(name, False), as long as any pair of munged names being compared were munged using the same flag value to c_name(). This is because commit 9fb081e already reserved names munging to 'q_', and this patch extends the reservation to 'Q_'; and because recent patches fixed things to ensure the only thing done by the flag is adding the prefix 'q_', that the only use of '.' (which also munges to '_') is in downstream extension prefixes, and that enum munging does not add underscores to any CamelCase values.

However, we DO have to care about the fact that we have a command 'stop' and an event 'STOP' (currently the only case collision of any of our .json entities). To solve that, add a new QAPISchemaEntity.c_namecase() that returns a munged version of the name that can be used in checking for case clashes, and override it in QAPISchemaEvent to use a different case (thus, indexing by .c_namecase() will create two natural key partitions), then add a new _clash_dict to QAPISchema._def_entity() that checks for case collisions after ruling out identical spellings. This works because all entities have at least a leading letter in their name.

We could go further and enforce that events are named with 'ALL_CAPS' and that nothing else is named in that manner; but that can be done as a followup if desired. We could also separate commands from type names (the way we already separated events into a separate namespace), but then we'd have to make .c_namecase() pick something other than the convenient upper() vs. lower() in order to partition dictionary keys into three sets (perhaps a leading digit, since all entities start with a letter).

There is also the possibility that we may want to add a future extension to QMP of teaching it to be case-insensitive (the user could request command 'Quit' instead of 'quit', or could spell a struct field as 'CPU' instead of 'cpu'). But for that to be a practical extension, we cannot break backwards compatibility with any existing struct that was already relying on case sensitivity. Fortunately, after a recent patch cleaned up CpuInfo, there are no such existing qapi structs. Of course, the idea of a future extension is not as strong of a reason to make the change.

At any rate, it is easier to be strict now, and relax things later if we find a reason to need case-sensitive QMP members, than it would be to wish we had the restriction in place.

Add new tests args-case-clash.json and command-type-case-clash.json to demonstrate that the collision detection works.

---
Use a separate clash dictionary, fix a typo that we couldn't trigger, add .c_namecase(), no longer need to use attrgetter

---
 scripts/qapi.py | 48 ++++++++++++++++++------------------------------
 1 file changed, 18 insertions(+), 30 deletions(-)
scripts/qapi.py +18 -30
index 5207c12..114e07a 100644
@@ -13,7 +13,6 @@
 
 import re
 from ordereddict import OrderedDict
-from operator import attrgetter
 import errno
 import getopt
 import os
@@ -381,7 +380,7 @@ def check_name(expr_info, source, name, allow_optional=False,
         membername = 'D' + membername
     # Reserve the entire 'q_'/'Q_' namespace for c_name()
     if not valid_name.match(membername) or \
-       c_name(membername, False).upper().startswith('Q_'):
+       c_name(membername, False).lower().startswith('q_'):
         raise QAPIExprError(expr_info,
                             "%s uses invalid name '%s'" % (source, name))
 
@@ -801,6 +800,9 @@ class QAPISchemaEntity(object):
     def c_name(self):
         return c_name(self.name)
 
+    def c_namecase(self):
+        return c_name(self.name).lower()
+
     def check(self, schema):
         pass
 
@@ -1041,7 +1043,7 @@ class QAPISchemaObjectTypeMember(object):
         assert self.type
 
     def check_clash(self, info, seen):
-        cname = c_name(self.name).upper()
+        cname = c_name(self.name).lower()
         if cname in seen:
             raise QAPIExprError(info,
                                 "%s collides with %s"
@@ -1086,7 +1088,7 @@ class QAPISchemaObjectTypeVariants(object):
 
     def check(self, schema, seen):
         if not self.tag_member:    # flat union
-            self.tag_member = seen[c_name(self.tag_name).upper()]
+            self.tag_member = seen[c_name(self.tag_name).lower()]
             assert self.tag_name == self.tag_member.name
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
@@ -1185,14 +1187,16 @@ class QAPISchemaEvent(QAPISchemaEntity):
     def visit(self, visitor):
         visitor.visit_event(self.name, self.info, self.arg_type)
 
+    def c_namecase(self):
+        return c_name(self.name).upper()
+
 
 class QAPISchema(object):
     def __init__(self, fname):
         try:
             self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
-            # _entity_dict holds two namespaces: events are stored via
-            # name.upper(), commands and types are stored via name.lower().
             self._entity_dict = {}
+            self._clash_dict = {}
             self._predefining = True
             self._def_predefineds()
             self._predefining = False
@@ -1205,32 +1209,17 @@ class QAPISchema(object):
     def _def_entity(self, ent):
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
-        # On insertion, we need to check for an exact match in either
-        # namespace, then for case collision in a single namespace
-        if self.lookup_entity(ent.name):
-            raise QAPIExprError(ent.info,
-                                "Entity '%s' already defined" % end.name)
-        cname = c_name(ent.name)
-        if isinstance(ent, QAPISchemaEvent):
-            cname = cname.upper()
-        else:
-            cname = cname.lower()
-        if cname in self._entity_dict:
+        assert ent.name not in self._entity_dict
+        cname = ent.c_namecase()
+        if cname in self._clash_dict:
             raise QAPIExprError(ent.info,
                                 "Entity '%s' collides with '%s'"
-                                % (ent.name, self._entity_dict[cname].name))
-        self._entity_dict[cname] = ent
+                                % (ent.name, self._clash_dict[cname]))
+        self._entity_dict[ent.name] = ent
+        self._clash_dict[cname] = ent.name
 
     def lookup_entity(self, name, typ=None):
-        # No thanks to 'stop'/'STOP', we have to check two namespaces during
-        # lookups, but only return a result on exact match.
-        ent1 = self._entity_dict.get(c_name(name).lower())   # commands, types
-        ent2 = self._entity_dict.get(c_name(name).upper())   # events
-        ent = None
-        if ent1 and ent1.name == name:
-            ent = ent1
-        elif ent2 and ent2.name == name:
-            ent = ent2
+        ent = self._entity_dict.get(name)
         if typ and not isinstance(ent, typ):
             return None
         return ent
@@ -1414,8 +1403,7 @@ class QAPISchema(object):
 
     def visit(self, visitor):
         visitor.visit_begin(self)
-        for entity in sorted(self._entity_dict.values(),
-                             key=attrgetter('name')):
+        for (name, entity) in sorted(self._entity_dict.items()):
             if visitor.visit_needed(entity):
                 entity.visit(visitor)
         visitor.visit_end()
Signature · 1 line
2.4.3
#64 lore raw

Re: [Qemu-devel] [PATCH] fixup! qapi: Forbid case-insensitive clashes

Quoted 27 lines · Eric Blake

[Replace the old commit message with this:]

qapi: Forbid case-insensitive clashes

In general, designing user interfaces that rely on case distinction is poor practice. Another benefit of enforcing a restriction against case-insensitive clashes is that we no longer have to worry about the situation of enum values that could be distinguished by case if mapped by c_name(), but which cannot be distinguished when mapped to C as ALL_CAPS by c_enum_const() [via c_name(name, False).upper()]. Thus, having the generator look for case collisions up front will prevent developers from worrying whether different munging rules for member names compared to enum values as a discriminator will cause any problems in qapi unions.

We do not have to care about c_name(name, True) vs. c_name(name, False), as long as any pair of munged names being compared were munged using the same flag value to c_name(). This is because commit 9fb081e already reserved names munging to 'q_', and this patch extends the reservation to 'Q_'; and because recent patches fixed things to ensure the only thing done by the flag is adding the prefix 'q_', that the only use of '.' (which also munges to '_') is in downstream extension prefixes, and that enum munging does not add underscores to any CamelCase values.

Quoted 6 lines

However, we DO have to care about the fact that we have a command 'stop' and an event 'STOP' (currently the only case collision of any of our .json entities). To solve that, add a new QAPISchemaEntity.c_namecase() that returns a munged version of the name that can be used in checking for case clashes,

For [-._] clashes, too. Easiest fix it to just say "checking for clashes".

Quoted 4 lines

and override it in QAPISchemaEvent to use a different
case (thus, indexing by .c_namecase() will create two natural key partitions), then add a new _clash_dict to QAPISchema._def_entity() that checks for case collisions after

To QAPISchema, actually. In _def_entity(), we only add an entry.

Quoted 12 lines

ruling out identical spellings. This works because all entities have at least a leading letter in their name.

We could go further and enforce that events are named with 'ALL_CAPS' and that nothing else is named in that manner; but that can be done as a followup if desired. We could also separate commands from type names (the way we already separated events into a separate namespace), but then we'd have to make .c_namecase() pick something other than the convenient upper() vs. lower() in order to partition dictionary keys into three sets (perhaps a leading digit, since all entities start with a letter).

The upper / lower partition works for events / anything else, and is kind of cute, although two separate clash dictionaries might be clearer. I'd like to sketch that next, to help us decide.

Even if we decide to stick to the "partition the key space" idea now, I'm reluctant to push it farther. Prepending a digit would feel odd. Title case would be slightly better. But I think three separate clash dictionaries would be simpler and clearer then.

Depending on what we do, the paragraph above may need some more tweaking.

Quoted 18 lines

There is also the possibility that we may want to add a future extension to QMP of teaching it to be case-insensitive (the user could request command 'Quit' instead of 'quit', or could spell a struct field as 'CPU' instead of 'cpu'). But for that to be a practical extension, we cannot break backwards compatibility with any existing struct that was already relying on case sensitivity. Fortunately, after a recent patch cleaned up CpuInfo, there are no such existing qapi structs. Of course, the idea of a future extension is not as strong of a reason to make the change.

At any rate, it is easier to be strict now, and relax things later if we find a reason to need case-sensitive QMP members, than it would be to wish we had the restriction in place.

Add new tests args-case-clash.json and command-type-case-clash.json to demonstrate that the collision detection works.

Quoted 8 lines

---
Use a separate clash dictionary, fix a typo that we couldn't trigger, add .c_namecase(), no longer need to use attrgetter

---
 scripts/qapi.py | 48 ++++++++++++++++++------------------------------
 1 file changed, 18 insertions(+), 30 deletions(-)

I find the squashed patch easier to understand, so here it is:

| diff --git a/scripts/qapi.py b/scripts/qapi.py
| index 72925c3..114e07a 100644
| --- a/scripts/qapi.py
| +++ b/scripts/qapi.py
| @@ -378,9 +378,9 @@ def check_name(expr_info, source, name, allow_optional=False,
|      # code always prefixes it with the enum name
|      if enum_member and membername[0].isdigit():
|          membername = 'D' + membername
| -    # Reserve the entire 'q_' namespace for c_name()
| +    # Reserve the entire 'q_'/'Q_' namespace for c_name()
|      if not valid_name.match(membername) or \
| -       c_name(membername, False).startswith('q_'):
| +       c_name(membername, False).lower().startswith('q_'):
|          raise QAPIExprError(expr_info,
|                              "%s uses invalid name '%s'" % (source, name))
|  
| @@ -800,6 +800,9 @@ class QAPISchemaEntity(object):
|      def c_name(self):
|          return c_name(self.name)
|  
| +    def c_namecase(self):
| +        return c_name(self.name).lower()
| +
|      def check(self, schema):
|          pass
|  

The entities are: commands, events, the various types. This implementation of c_namecase() covers commands and types. It gets overwritten for events.

Commands should not use upper case. If we enforced that, the .lower() would be superfluous there. But it isn't wrong.

| @@ -1040,7 +1043,7 @@ class QAPISchemaObjectTypeMember(object):
|          assert self.type
|  
|      def check_clash(self, info, seen):
| -        cname = c_name(self.name)
| +        cname = c_name(self.name).lower()
|          if cname in seen:
|              raise QAPIExprError(info,
|                                  "%s collides with %s"

Not an entity, therefore QAPISchemaEntity.c_namecase() isn't inherited.

Members should not use upper case. If we enforced that, the .lower() would be superfluous.

| @@ -1085,7 +1088,7 @@ class QAPISchemaObjectTypeVariants(object):
|  
|      def check(self, schema, seen):
|          if not self.tag_member:    # flat union
| -            self.tag_member = seen[c_name(self.tag_name)]
| +            self.tag_member = seen[c_name(self.tag_name).lower()]
|              assert self.tag_name == self.tag_member.name
|          assert isinstance(self.tag_member.type, QAPISchemaEnumType)
|          for v in self.variants:

Key needs to match whatever we use in QAPISchemaObjectTypeMember.check_clash(). It does.

| @@ -1184,12 +1187,16 @@ class QAPISchemaEvent(QAPISchemaEntity):
|      def visit(self, visitor):
|          visitor.visit_event(self.name, self.info, self.arg_type)
|  
| +    def c_namecase(self):
| +        return c_name(self.name).upper()
| +

Event names should not use lower case. If we enforced that, the .upper() would be superfluous.

If we enforced our naming conventions, namely types in CamelCase, events in ALL_CAPS, everything else lower-case, the c_namecase() would become slightly simpler: it's same as c_name() except for types, where it additionally folds to lower case. The name would be misleading then, however. Anyway, we can consider that once we enforce naming conventions.

|  
|  class QAPISchema(object):
|      def __init__(self, fname):
|          try:
|              self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)
|              self._entity_dict = {}
| +            self._clash_dict = {}
|              self._predefining = True
|              self._def_predefineds()
|              self._predefining = False
| @@ -1203,7 +1210,13 @@ class QAPISchema(object):
|          # Only the predefined types are allowed to not have info
|          assert ent.info or self._predefining
|          assert ent.name not in self._entity_dict
| +        cname = ent.c_namecase()
| +        if cname in self._clash_dict:
| +            raise QAPIExprError(ent.info,
| +                                "Entity '%s' collides with '%s'"
| +                                % (ent.name, self._clash_dict[cname]))
|          self._entity_dict[ent.name] = ent
| +        self._clash_dict[cname] = ent.name
|  
|      def lookup_entity(self, name, typ=None):
|          ent = self._entity_dict.get(name)
author
#65 lore raw

Re: [Qemu-devel] [PATCH] fixup! qapi: Forbid case-insensitive clashes

Quoted 17 lines · Markus Armbruster
Quoted 11 lines · Eric Blake

[Replace the old commit message with this:]

qapi: Forbid case-insensitive clashes

However, we DO have to care about the fact that we have a command 'stop' and an event 'STOP' (currently the only case collision of any of our .json entities). To solve that, add a new QAPISchemaEntity.c_namecase() that returns a munged version of the name that can be used in checking for case clashes,

For [-._] clashes, too. Easiest fix it to just say "checking for clashes".

Quoted 6 lines
Quoted 4 lines

and override it in QAPISchemaEvent to use a different
case (thus, indexing by .c_namecase() will create two natural key partitions), then add a new _clash_dict to QAPISchema._def_entity() that checks for case collisions after

To QAPISchema, actually. In _def_entity(), we only add an entry.

Easy fixes.

Quoted 14 lines
Quoted 4 lines

ruling out identical spellings. This works because all entities have at least a leading letter in their name.

Quoted 9 lines

We could go further and enforce that events are named with 'ALL_CAPS' and that nothing else is named in that manner; but that can be done as a followup if desired. We could also separate commands from type names (the way we already separated events into a separate namespace), but then we'd have to make .c_namecase() pick something other than the convenient upper() vs. lower() in order to partition dictionary keys into three sets (perhaps a leading digit, since all entities start with a letter).

Quoted 3 lines

The upper / lower partition works for events / anything else, and is kind of cute, although two separate clash dictionaries might be clearer. I'd like to sketch that next, to help us decide.

Okay, sounds like I should post a v12.5 with the alternative of 2 separate clash dictionaries (3 dictionaries total), and as a complete patch rather than a fixup.

Quoted 11 lines
| @@ -800,6 +800,9 @@ class QAPISchemaEntity(object):
|      def c_name(self):
|          return c_name(self.name)
|  
| +    def c_namecase(self):
| +        return c_name(self.name).lower()
| +
|      def check(self, schema):
|          pass
|  
Quoted 6 lines

The entities are: commands, events, the various types. This implementation of c_namecase() covers commands and types. It gets overwritten for events.

Commands should not use upper case. If we enforced that, the .lower() would be superfluous there. But it isn't wrong.

I won't need c_namecase() with separate dictionaries. Once we have the two approaches to compare, we can decide which one looks nicer.

Quoted 11 lines
| @@ -1040,7 +1043,7 @@ class QAPISchemaObjectTypeMember(object):
|          assert self.type
|  
|      def check_clash(self, info, seen):
| -        cname = c_name(self.name)
| +        cname = c_name(self.name).lower()
|          if cname in seen:
|              raise QAPIExprError(info,
|                                  "%s collides with %s"
Quoted 4 lines

Not an entity, therefore QAPISchemaEntity.c_namecase() isn't inherited.

Members should not use upper case. If we enforced that, the .lower() would be superfluous.

Even if we enforce it, we have to whitelist exceptions for backward compatibility; those exceptions would have to use .lower(), or we can just skip collision detection on those whitelisted types (where the existence of the whitelist reminds us to maintain no case collisions by hand).

Quoted 10 lines
| @@ -1184,12 +1187,16 @@ class QAPISchemaEvent(QAPISchemaEntity):
|      def visit(self, visitor):
|          visitor.visit_event(self.name, self.info, self.arg_type)
|  
| +    def c_namecase(self):
| +        return c_name(self.name).upper()
| +

Event names should not use lower case. If we enforced that, the .upper() would be superfluous.

Looks like we could enforce this one without a whitelist.

Quoted 7 lines

If we enforced our naming conventions, namely types in CamelCase, events in ALL_CAPS, everything else lower-case, the c_namecase() would become slightly simpler: it's same as c_name() except for types, where it additionally folds to lower case. The name would be misleading then, however. Anyway, we can consider that once we enforce naming conventions.

And enforcing conventions certainly won't happen any sooner than the rest of the pending queue is flushed.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#66 lore raw

Re: [Qemu-devel] [PATCH v12 27/36] qapi: Forbid case-insensitive clashes

In reply to Eric Blake

Let's think through this on a higher level.

I figure the motivation for this patch is twofold:

1. C identifier clash detection

We generate C identifiers derived from QAPI names. These can clash with (1) each other, (2) C keywords and selected other well-known identifiers, and (3) the user's identifiers.

We avoid (2), and we make an effort to catch (1). because that's friendlier than letting the compiler catch it. (3) is the user's problem, and there's nothing we can do about it.

Clashes are due to (a) folding [-._] to _, (b) case folding, and (c) name space intrusion, e.g. a type Foo_lookup clashes with an enum type Foo's lookup table.

We make an effort to prevent (c) by reserving names. We make an effort to catch (a) and (b).

2. Dislike for interfaces that differ only in case

I agree defining such interfaces is a bad idea. However, forbidding bad ideas is problematic, because where do you stop? There's an infinite supply.

Forbidding makes sense when the bad idea is a recurring problem, or when it causes technical difficulties.

There's been one instance of this bad idea so far: command 'stop' and event 'STOP'. We can't fix it, because it's ABI. The patches proposed so far can't even prevent its recurrence; events and non-event entities that differ only in case remain valid. Any other combination becomes invalid.

A rule "cannot differ only in case" is okay, but with "except for events and non-events" tacked on, it becomes ugly.

The case for rejecting names that differ only in case appears weak.

Our naming conventions actually make clashes due to folding relatively unlikely. They are:

* All names consist of letters, digits, '-' and '_', starting with a letter.

* Event names are ALL_CAPS, i.e. they use neither lower case nor '-'.

Without lower case, '-' and '.', clash due to folding is impossible.

* Command names, member names and built-in type names are dashed-lower, i.e. they use neither upper case nor '_'.

Without upper case, '_', and '.', clash due to folding is impossible.

* Type names are CamelCase. They do not use '_' or '-'.

Can theoretically clash in amusing ways: ONegus, OneGus. Ain'tCamelCaseFun!

We can get rid of the clashes by not case folding type names. See "[PATCH RFC 3/5] qapi: Use common name mangling for enumeration constants". The patch additionally drops folding of enumeration member names, which isn't necessary. Controversial.

* Additionally, any name may be prefixed by __RFQDN_. RFQDN consists of letters, digits '-' and '.'.

Because unprefixed names start with a letter, and the prefix starts with '__', prefixed names cannot clash with unprefixed names.

If we additionally stipulated that event prefixes are upper case, and all others lower case, prefixes couldn't contribute to clashes at all.

Strict adherence to our naming conventions would eliminate clashes due to folding except for type names. A single extra dictionary mapping c_name(typ.name).lower() to typ would suffice.

What happens to the rest of your queue if we shelve this patch for now?

author
#67 lore raw

Re: [Qemu-devel] [PATCH v12 27/36] qapi: Forbid case-insensitive clashes

Quoted 9 lines · Markus Armbruster

Let's think through this on a higher level.

I figure the motivation for this patch is twofold:

1. C identifier clash detection

2. Dislike for interfaces that differ only in case

And the related dislike for interfaces that differ only in '-' or '_'.

Quoted 9 lines

Our naming conventions actually make clashes due to folding relatively unlikely. They are:

* All names consist of letters, digits, '-' and '_', starting with a letter.

* Event names are ALL_CAPS, i.e. they use neither lower case nor '-'.

Without lower case, '-' and '.', clash due to folding is impossible.

So if we enforce that convention, we don't have to worry about case clash, and I don't think we have any outliers to whitelist.

Quoted 5 lines

* Command names, member names and built-in type names are dashed-lower, i.e. they use neither upper case nor '_'.

Without upper case, '_', and '.', clash due to folding is impossible.

If we enforce that convention, we have to whitelist outliers (for example, 'netdev_add' as a command name, or a number of types whose members still use '_'). The existence of a whitelist will discourage future clashes, without any of the hassle of coding up clash checks.

Quoted 5 lines

* Type names are CamelCase. They do not use '_' or '-'.

Can theoretically clash in amusing ways: ONegus, OneGus. Ain'tCamelCaseFun!

Do we have any outliers?

Quoted 5 lines

We can get rid of the clashes by not case folding type names. See "[PATCH RFC 3/5] qapi: Use common name mangling for enumeration constants". The patch additionally drops folding of enumeration member names, which isn't necessary. Controversial.

There is even the possibility of mixed treatment (enumeration name portion as-is, member name portion shouted, as in 'MyTypeVALUE1'). Not sure if we'll get a clear answer to the controversy, but also not sure it is worth holding up other patches while discussing that.

Quoted 9 lines

* Additionally, any name may be prefixed by __RFQDN_. RFQDN consists of letters, digits '-' and '.'.

Because unprefixed names start with a letter, and the prefix starts with '__', prefixed names cannot clash with unprefixed names.

If we additionally stipulated that event prefixes are upper case, and all others lower case, prefixes couldn't contribute to clashes at all.

It's a bit weird that we'd have __org.qemu_command and __ORG.QEMU_EVENT for the same vendor. On the other hand, FQDN is already case-insensitive (qemu.org and QEMU.ORG resolve to the same address). So there won't be clashes between vendors (as no two vendors can share a FQDN that differs in case); beyond that, if a vendor wants to play weird case games, that's their (downstream) problem, not ours.

Quoted 4 lines

Strict adherence to our naming conventions would eliminate clashes due to folding except for type names. A single extra dictionary mapping c_name(typ.name).lower() to typ would suffice.

Certainly may be simpler than carrying three dictionaries for command/event/type.

Quoted 2 lines

What happens to the rest of your queue if we shelve this patch for now?

Not much; in fact, according to the patch version log:

---
v12: add in entity case collisions (required sharing two maps), improve commit message
v11: rebase to latest, don't focus so hard on future case-insensitive extensions, adjust commit message
v10: new patch
---

I only even added it due to conversations on v9, and intentionally kept it separate from 'qapi: Detect collisions in C member names' (we absolutely want to report exact-name collisions, whether or not we also decide to report case collisions). It should not be a problem to defer this patch (or a better version of it that enforces conventions, adds whitelist handling, then only worries about type name case collisions) to much later, if at all.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#68 lore raw

Re: [Qemu-devel] [PATCH v12 27/36] qapi: Forbid case-insensitive clashes

Quoted 5 lines · Eric Blake
Quoted 4 lines · Markus Armbruster

Let's think through this on a higher level.

I figure the motivation for this patch is twofold:

Quoted 7 lines
Quoted 5 lines

1. C identifier clash detection

2. Dislike for interfaces that differ only in case

And the related dislike for interfaces that differ only in '-' or '_'.

Yes.

Quoted 6 lines
Quoted 6 lines

Our naming conventions actually make clashes due to folding relatively unlikely. They are:

* All names consist of letters, digits, '-' and '_', starting with a letter.

Quoted 6 lines
Quoted 3 lines

* Event names are ALL_CAPS, i.e. they use neither lower case nor '-'.

Without lower case, '-' and '.', clash due to folding is impossible.

So if we enforce that convention, we don't have to worry about case clash, and I don't think we have any outliers to whitelist.

qmp-introspect.c shows none.

Quoted 8 lines
Quoted 4 lines

* Command names, member names and built-in type names are dashed-lower, i.e. they use neither upper case nor '_'.

Without upper case, '_', and '.', clash due to folding is impossible.

If we enforce that convention, we have to whitelist outliers (for example, 'netdev_add' as a command name, or a number of types whose members still use '_').

Yes.

Quoted 2 lines

The existence of a whitelist will discourage
future clashes, without any of the hassle of coding up clash checks.

I hope so.

Quoted 6 lines
Quoted 4 lines

* Type names are CamelCase. They do not use '_' or '-'.

Can theoretically clash in amusing ways: ONegus, OneGus. Ain'tCamelCaseFun!

Do we have any outliers?

Let's look. Stick

            print '%s %s' % (entity.__class__.__name__, name)

into QAPISchema.visit()'s loop.

Pipe output through "sed -n 's/^.*Type \([^:].*\)$/\1/p'" to get the type names. Grep for '[-_]', -v '[a-z]' and -v '[A-Z]' shows there are no names with [-_], no names without lower case letters, and the only names without upper case letters are the 14 built-in types. Both for qapi-schema.json and qga/qapi-schema.json.

Quoted 9 lines
Quoted 4 lines

We can get rid of the clashes by not case folding type names. See "[PATCH RFC 3/5] qapi: Use common name mangling for enumeration constants". The patch additionally drops folding of enumeration member names, which isn't necessary. Controversial.

There is even the possibility of mixed treatment (enumeration name portion as-is, member name portion shouted, as in 'MyTypeVALUE1'). Not sure if we'll get a clear answer to the controversy, but also not sure it is worth holding up other patches while discussing that.

I'm content to shout the member name. I'd stick a '_' between the prefix and the member name, though.

Quoted 11 lines
Quoted 8 lines

* Additionally, any name may be prefixed by __RFQDN_. RFQDN consists of letters, digits '-' and '.'.

Because unprefixed names start with a letter, and the prefix starts with '__', prefixed names cannot clash with unprefixed names.

If we additionally stipulated that event prefixes are upper case, and all others lower case, prefixes couldn't contribute to clashes at all.

It's a bit weird that we'd have __org.qemu_command and __ORG.QEMU_EVENT for the same vendor.

Yes.

Quoted 2 lines

On the other hand, FQDN is already
case-insensitive (qemu.org and QEMU.ORG resolve to the same address).

Yes, and that makes stupilating case now somewhat problematic.

Quoted 3 lines

So there won't be clashes between vendors (as no two vendors can share a FQDN that differs in case); beyond that, if a vendor wants to play weird case games, that's their (downstream) problem, not ours.

Yes, not our worry.

Quoted 20 lines
Quoted 3 lines

Strict adherence to our naming conventions would eliminate clashes due to folding except for type names. A single extra dictionary mapping c_name(typ.name).lower() to typ would suffice.

Certainly may be simpler than carrying three dictionaries for command/event/type.

Quoted 2 lines

What happens to the rest of your queue if we shelve this patch for now?

Not much; in fact, according to the patch version log:

---
v12: add in entity case collisions (required sharing two maps), improve commit message
v11: rebase to latest, don't focus so hard on future case-insensitive extensions, adjust commit message
v10: new patch
---

Quoted 7 lines

I only even added it due to conversations on v9, and intentionally kept it separate from 'qapi: Detect collisions in C member names' (we absolutely want to report exact-name collisions, whether or not we also decide to report case collisions). It should not be a problem to defer this patch (or a better version of it that enforces conventions, adds whitelist handling, then only worries about type name case collisions) to much later, if at all.

Okay, let's shelve the patch for now, keep the queue moving, and revisit the patch at a more opportune time.

I hope the detour didn't eat too much of your time. We learned a few things, and perhaps we can still reuse some of the work.

author
#69 lore raw

PATCH v12 28/36 qapi: Simplify QObject

In reply to Eric Blake

The QObject hierarchy is small enough, and unlikely to grow further (since we only use it to map to JSON and already cover all JSON types), that we can simplify things by not tracking a separate vtable, but just inline all two elements of the vtable QType directly into QObject. We can drop qnull_destroy_obj() in the process.

This also has the nice benefit of moving the typename 'QType' out of the way, so that the next patch can repurpose it for a nicer name for 'qtype_code'.

The various objects are now 8 bytes larger on 64-bit platforms, but hopefully this is in the noise. If size is absolutely critical due to cache line speed effects, we could do some bit packing to restore the previous size (qtype_code currently only requires three distinct bits, so we could assert an assumption that architecture ABIs will give function pointers at least an 8-byte alignment, and that the three low-order bits of destroy are always zero; or we could use a bitfield for refcnt, with a maximum of SIZE_MAX/8, and given that a QObject already occupies at least 8 bytes, that won't artificially limit us). Or we could expose the currently static destroy methods and instead create a table in qobject.h that maps QType to destructor. But let's save any size compression for a followup patch, if at all.

On the other hand, by avoiding a separate vtable, there is less indirection, so use of QObjects may be slightly more efficient.

However, as we don't have evidence pointing at either QObject size or access speed as being a hot spot, I didn't think it was worth benchmarking.

---
v12: new patch, split off from 21/28

---
 include/qapi/qmp/qobject.h | 38 +++++++++++++++++++++-----------------
 qobject/qbool.c            |  7 +------
 qobject/qdict.c            |  7 +------
 qobject/qfloat.c           |  7 +------
 qobject/qint.c             |  7 +------
 qobject/qlist.c            |  7 +------
 qobject/qnull.c            | 13 ++-----------
 qobject/qstring.c          |  7 +------
 8 files changed, 29 insertions(+), 64 deletions(-)
include/qapi/qmp/qobject.h +21 -17
index 4b96ed5..83ed70b 100644
@@ -47,21 +47,21 @@ typedef enum {
     QTYPE_MAX,
 } qtype_code;
 
-struct QObject;
+typedef struct QObject QObject;
 
-typedef struct QType {
-    qtype_code code;
-    void (*destroy)(struct QObject *);
-} QType;
-
-typedef struct QObject {
-    const QType *type;
+struct QObject {
+    qtype_code type;
     size_t refcnt;
-} QObject;
+    void (*destroy)(QObject *);
+};
 
 /* Get the 'base' part of an object */
 #define QOBJECT(obj) (&(obj)->base)
 
+/* High-level interface for qobject_init() */
+#define QOBJECT_INIT(obj, type, destroy) \
+    qobject_init(QOBJECT(obj), type, destroy)
+
 /* High-level interface for qobject_incref() */
 #define QINCREF(obj)      \
     qobject_incref(QOBJECT(obj))
@@ -71,9 +71,14 @@ typedef struct QObject {
     qobject_decref(obj ? QOBJECT(obj) : NULL)
 
 /* Initialize an object to default values */
-#define QOBJECT_INIT(obj, qtype_type)   \
-    obj->base.refcnt = 1;               \
-    obj->base.type   = qtype_type
+static inline void qobject_init(QObject *obj, qtype_code type,
+                                void (*destroy)(struct QObject *))
+{
+    assert(type);
+    obj->refcnt = 1;
+    obj->type = type;
+    obj->destroy = destroy;
+}
 
 /**
  * qobject_incref(): Increment QObject's reference count
@@ -92,9 +97,8 @@ static inline void qobject_decref(QObject *obj)
 {
     assert(!obj || obj->refcnt);
     if (obj && --obj->refcnt == 0) {
-        assert(obj->type != NULL);
-        assert(obj->type->destroy != NULL);
-        obj->type->destroy(obj);
+        assert(obj->destroy);
+        obj->destroy(obj);
     }
 }
 
@@ -103,8 +107,8 @@ static inline void qobject_decref(QObject *obj)
  */
 static inline qtype_code qobject_type(const QObject *obj)
 {
-    assert(obj->type != NULL);
-    return obj->type->code;
+    assert(obj->type);
+    return obj->type;
 }
 
 extern QObject qnull_;
qobject/qbool.c +1 -6
index bc6535f..9f46e67 100644
@@ -17,11 +17,6 @@
 
 static void qbool_destroy_obj(QObject *obj);
 
-static const QType qbool_type = {
-    .code = QTYPE_QBOOL,
-    .destroy = qbool_destroy_obj,
-};
-
 /**
  * qbool_from_bool(): Create a new QBool from a bool
  *
@@ -33,7 +28,7 @@ QBool *qbool_from_bool(bool value)
 
     qb = g_malloc(sizeof(*qb));
     qb->value = value;
-    QOBJECT_INIT(qb, &qbool_type);
+    QOBJECT_INIT(qb, QTYPE_QBOOL, qbool_destroy_obj);
 
     return qb;
 }
qobject/qdict.c +1 -6
index 2d67bf1..cf62269 100644
@@ -21,11 +21,6 @@
 
 static void qdict_destroy_obj(QObject *obj);
 
-static const QType qdict_type = {
-    .code = QTYPE_QDICT,
-    .destroy = qdict_destroy_obj,
-};
-
 /**
  * qdict_new(): Create a new QDict
  *
@@ -36,7 +31,7 @@ QDict *qdict_new(void)
     QDict *qdict;
 
     qdict = g_malloc0(sizeof(*qdict));
-    QOBJECT_INIT(qdict, &qdict_type);
+    QOBJECT_INIT(qdict, QTYPE_QDICT, qdict_destroy_obj);
 
     return qdict;
 }
qobject/qfloat.c +1 -6
index c865163..2472189 100644
@@ -17,11 +17,6 @@
 
 static void qfloat_destroy_obj(QObject *obj);
 
-static const QType qfloat_type = {
-    .code = QTYPE_QFLOAT,
-    .destroy = qfloat_destroy_obj,
-};
-
 /**
  * qfloat_from_int(): Create a new QFloat from a float
  *
@@ -33,7 +28,7 @@ QFloat *qfloat_from_double(double value)
 
     qf = g_malloc(sizeof(*qf));
     qf->value = value;
-    QOBJECT_INIT(qf, &qfloat_type);
+    QOBJECT_INIT(qf, QTYPE_QFLOAT, qfloat_destroy_obj);
 
     return qf;
 }
qobject/qint.c +1 -6
index 999688e..765307f 100644
@@ -16,11 +16,6 @@
 
 static void qint_destroy_obj(QObject *obj);
 
-static const QType qint_type = {
-    .code = QTYPE_QINT,
-    .destroy = qint_destroy_obj,
-};
-
 /**
  * qint_from_int(): Create a new QInt from an int64_t
  *
@@ -32,7 +27,7 @@ QInt *qint_from_int(int64_t value)
 
     qi = g_malloc(sizeof(*qi));
     qi->value = value;
-    QOBJECT_INIT(qi, &qint_type);
+    QOBJECT_INIT(qi, QTYPE_QINT, qint_destroy_obj);
 
     return qi;
 }
qobject/qlist.c +1 -6
index 298003a..1e2098a 100644
@@ -17,11 +17,6 @@
 
 static void qlist_destroy_obj(QObject *obj);
 
-static const QType qlist_type = {
-    .code = QTYPE_QLIST,
-    .destroy = qlist_destroy_obj,
-};
- 
 /**
  * qlist_new(): Create a new QList
  *
@@ -33,7 +28,7 @@ QList *qlist_new(void)
 
     qlist = g_malloc(sizeof(*qlist));
     QTAILQ_INIT(&qlist->head);
-    QOBJECT_INIT(qlist, &qlist_type);
+    QOBJECT_INIT(qlist, QTYPE_QLIST, qlist_destroy_obj);
 
     return qlist;
 }
qobject/qnull.c +2 -11
index 9873e26..22d59e9 100644
@@ -13,17 +13,8 @@
 #include "qemu-common.h"
 #include "qapi/qmp/qobject.h"
 
-static void qnull_destroy_obj(QObject *obj)
-{
-    assert(0);
-}
-
-static const QType qnull_type = {
-    .code = QTYPE_QNULL,
-    .destroy = qnull_destroy_obj,
-};
-
 QObject qnull_ = {
-    .type = &qnull_type,
+    .type = QTYPE_QNULL,
     .refcnt = 1,
+    .destroy = NULL,
 };
qobject/qstring.c +1 -6
index cb72dfb..ba59d00 100644
@@ -16,11 +16,6 @@
 
 static void qstring_destroy_obj(QObject *obj);
 
-static const QType qstring_type = {
-    .code = QTYPE_QSTRING,
-    .destroy = qstring_destroy_obj,
-};
-
 /**
  * qstring_new(): Create a new empty QString
  *
@@ -57,7 +52,7 @@ QString *qstring_from_substr(const char *str, int start, int end)
     memcpy(qstring->string, str + start, qstring->length);
     qstring->string[qstring->length] = 0;
 
-    QOBJECT_INIT(qstring, &qstring_type);
+    QOBJECT_INIT(qstring, QTYPE_QSTRING, qstring_destroy_obj);
 
     return qstring;
 }
Signature · 1 line
2.4.3
#70 lore raw

Re: [Qemu-devel] [PATCH v12 28/36] qapi: Simplify QObject

Quoted 11 lines · Eric Blake

The QObject hierarchy is small enough, and unlikely to grow further (since we only use it to map to JSON and already cover all JSON types), that we can simplify things by not tracking a separate vtable, but just inline all two elements of the vtable QType directly into QObject. We can drop qnull_destroy_obj() in the process.

This also has the nice benefit of moving the typename 'QType' out of the way, so that the next patch can repurpose it for a nicer name for 'qtype_code'.

Quoted 12 lines

The various objects are now 8 bytes larger on 64-bit platforms, but hopefully this is in the noise. If size is absolutely critical due to cache line speed effects, we could do some bit packing to restore the previous size (qtype_code currently only requires three distinct bits, so we could assert an assumption that architecture ABIs will give function pointers at least an 8-byte alignment, and that the three low-order bits of destroy are always zero; or we could use a bitfield for refcnt, with a maximum of SIZE_MAX/8, and given that a QObject already occupies at least 8 bytes, that won't artificially limit us). Or we could expose the currently static destroy methods and instead create a table in qobject.h that maps QType to destructor.

That's what I'd do.

A type registration function would keep the destructors static. But why bother.

Quoted 2 lines

But let's save any size compression for a
followup patch, if at all.

Okay.

Quoted 9 lines

On the other hand, by avoiding a separate vtable, there is less indirection, so use of QObjects may be slightly more efficient.

However, as we don't have evidence pointing at either QObject size or access speed as being a hot spot, I didn't think it was worth benchmarking.

Works for me.

author
#71 lore raw

PATCH 1/1 fixup? qapi: Simplify QObject

In reply to Eric Blake

[If we squash, replace the old commit message with this; you'll also have some merge conflicts in 29/36 and 30/36. Note that while this mail is net lines added, squashing would still result in a net reduction:

 16 files changed, 67 insertions(+), 83 deletions(-)

]

qapi: Simplify QObject

The QObject hierarchy is small enough, and unlikely to grow further (since we only use it to map to JSON and already cover all JSON types), that we can simplify things by not tracking a separate vtable, but just inline the refcnt element of the vtable QType directly into QObject, and track a separate array of destroy functions. We can drop qnull_destroy_obj() in the process.

The remaining QObject subclasses must export their destructor.

This also has the nice benefit of moving the typename 'QType' out of the way, so that the next patch can repurpose it for a nicer name for 'qtype_code'.

The various objects are still the same size (so no change in cache line pressure), but now have less indirection (although I didn't bother benchmarking to see if there is a noticeable speedup, as we don't have hard evidence that this was in a performance hotspot in the first place).

---

If we don't want to squash, I can instead rebase this to appear after 30/36 as a standalone patch, with a message something like:

qapi: Don't duplicate destructor pointer per QObject

The previous patch "qapi: Simplify QObject" made each QObject instance larger; restore the original size for less cache line pressure by moving the destroy function pointer out of each object and instead maintaining a global table of destroy functions. This requires exporting the various functions, and is made easier to read through a typedef.

---
 include/qapi/qmp/qbool.h   |  1 +
 include/qapi/qmp/qdict.h   |  1 +
 include/qapi/qmp/qfloat.h  |  1 +
 include/qapi/qmp/qint.h    |  1 +
 include/qapi/qmp/qlist.h   |  1 +
 include/qapi/qmp/qobject.h | 16 ++++++++--------
 include/qapi/qmp/qstring.h |  1 +
 qobject/Makefile.objs      |  2 +-
 qobject/qbool.c            |  6 ++----
 qobject/qdict.c            |  6 ++----
 qobject/qfloat.c           |  6 ++----
 qobject/qint.c             |  6 ++----
 qobject/qlist.c            |  6 ++----
 qobject/qnull.c            |  1 -
 qobject/qobject.c          | 26 ++++++++++++++++++++++++++
 qobject/qstring.c          |  6 ++----
 16 files changed, 53 insertions(+), 34 deletions(-)
 create mode 100644 qobject/qobject.c
include/qapi/qmp/qbool.h +1 -0
index d9256e4..836d078 100644
@@ -25,5 +25,6 @@ typedef struct QBool {
 QBool *qbool_from_bool(bool value);
 bool qbool_get_bool(const QBool *qb);
 QBool *qobject_to_qbool(const QObject *obj);
+void qbool_destroy_obj(QObject *obj);
 
 #endif /* QBOOL_H */
include/qapi/qmp/qdict.h +1 -0
index 787c658..6c2a0e5 100644
@@ -48,6 +48,7 @@ void qdict_iter(const QDict *qdict,
                 void *opaque);
 const QDictEntry *qdict_first(const QDict *qdict);
 const QDictEntry *qdict_next(const QDict *qdict, const QDictEntry *entry);
+void qdict_destroy_obj(QObject *obj);
 
 /* Helper to qdict_put_obj(), accepts any object */
 #define qdict_put(qdict, key, obj) \
include/qapi/qmp/qfloat.h +1 -0
index 46745e5..a8af2a8 100644
@@ -25,5 +25,6 @@ typedef struct QFloat {
 QFloat *qfloat_from_double(double value);
 double qfloat_get_double(const QFloat *qi);
 QFloat *qobject_to_qfloat(const QObject *obj);
+void qfloat_destroy_obj(QObject *obj);
 
 #endif /* QFLOAT_H */
include/qapi/qmp/qint.h +1 -0
index 339a9ab..049e528 100644
@@ -24,5 +24,6 @@ typedef struct QInt {
 QInt *qint_from_int(int64_t value);
 int64_t qint_get_int(const QInt *qi);
 QInt *qobject_to_qint(const QObject *obj);
+void qint_destroy_obj(QObject *obj);
 
 #endif /* QINT_H */
include/qapi/qmp/qlist.h +1 -0
index b1bf785..a84117e 100644
@@ -49,6 +49,7 @@ QObject *qlist_peek(QList *qlist);
 int qlist_empty(const QList *qlist);
 size_t qlist_size(const QList *qlist);
 QList *qobject_to_qlist(const QObject *obj);
+void qlist_destroy_obj(QObject *obj);
 
 static inline const QListEntry *qlist_first(const QList *qlist)
 {
include/qapi/qmp/qobject.h +8 -8
index 83ed70b..4e89a7f 100644
@@ -52,15 +52,17 @@ typedef struct QObject QObject;
 struct QObject {
     qtype_code type;
     size_t refcnt;
-    void (*destroy)(QObject *);
 };
 
+typedef void (*QDestroy)(QObject *);
+extern QDestroy qdestroy[QTYPE_MAX];
+
 /* Get the 'base' part of an object */
 #define QOBJECT(obj) (&(obj)->base)
 
 /* High-level interface for qobject_init() */
-#define QOBJECT_INIT(obj, type, destroy) \
-    qobject_init(QOBJECT(obj), type, destroy)
+#define QOBJECT_INIT(obj, type) \
+    qobject_init(QOBJECT(obj), type)
 
 /* High-level interface for qobject_incref() */
 #define QINCREF(obj)      \
@@ -71,13 +73,11 @@ struct QObject {
     qobject_decref(obj ? QOBJECT(obj) : NULL)
 
 /* Initialize an object to default values */
-static inline void qobject_init(QObject *obj, qtype_code type,
-                                void (*destroy)(struct QObject *))
+static inline void qobject_init(QObject *obj, qtype_code type)
 {
     assert(type);
     obj->refcnt = 1;
     obj->type = type;
-    obj->destroy = destroy;
 }
 
 /**
@@ -97,8 +97,8 @@ static inline void qobject_decref(QObject *obj)
 {
     assert(!obj || obj->refcnt);
     if (obj && --obj->refcnt == 0) {
-        assert(obj->destroy);
-        obj->destroy(obj);
+        assert(qdestroy[obj->type]);
+        qdestroy[obj->type](obj);
     }
 }
 
include/qapi/qmp/qstring.h +1 -0
index 34675a7..df7df55 100644
@@ -32,5 +32,6 @@ void qstring_append_int(QString *qstring, int64_t value);
 void qstring_append(QString *qstring, const char *str);
 void qstring_append_chr(QString *qstring, int c);
 QString *qobject_to_qstring(const QObject *obj);
+void qstring_destroy_obj(QObject *obj);
 
 #endif /* QSTRING_H */
qobject/Makefile.objs +1 -1
index 0031e8b..bed5508 100644
@@ -1,2 +1,2 @@
 util-obj-y = qnull.o qint.o qstring.o qdict.o qlist.o qfloat.o qbool.o
-util-obj-y += qjson.o json-lexer.o json-streamer.o json-parser.o
+util-obj-y += qjson.o qobject.o json-lexer.o json-streamer.o json-parser.o
qobject/qbool.c +2 -4
index 9f46e67..895fd4d 100644
@@ -15,8 +15,6 @@
 #include "qapi/qmp/qobject.h"
 #include "qemu-common.h"
 
-static void qbool_destroy_obj(QObject *obj);
-
 /**
  * qbool_from_bool(): Create a new QBool from a bool
  *
@@ -28,7 +26,7 @@ QBool *qbool_from_bool(bool value)
 
     qb = g_malloc(sizeof(*qb));
     qb->value = value;
-    QOBJECT_INIT(qb, QTYPE_QBOOL, qbool_destroy_obj);
+    QOBJECT_INIT(qb, QTYPE_QBOOL);
 
     return qb;
 }
@@ -56,7 +54,7 @@ QBool *qobject_to_qbool(const QObject *obj)
  * qbool_destroy_obj(): Free all memory allocated by a
  * QBool object
  */
-static void qbool_destroy_obj(QObject *obj)
+void qbool_destroy_obj(QObject *obj)
 {
     assert(obj != NULL);
     g_free(qobject_to_qbool(obj));
qobject/qdict.c +2 -4
index cf62269..dd3f1c2 100644
@@ -19,8 +19,6 @@
 #include "qemu/queue.h"
 #include "qemu-common.h"
 
-static void qdict_destroy_obj(QObject *obj);
-
 /**
  * qdict_new(): Create a new QDict
  *
@@ -31,7 +29,7 @@ QDict *qdict_new(void)
     QDict *qdict;
 
     qdict = g_malloc0(sizeof(*qdict));
-    QOBJECT_INIT(qdict, QTYPE_QDICT, qdict_destroy_obj);
+    QOBJECT_INIT(qdict, QTYPE_QDICT);
 
     return qdict;
 }
@@ -436,7 +434,7 @@ void qdict_del(QDict *qdict, const char *key)
 /**
  * qdict_destroy_obj(): Free all the memory allocated by a QDict
  */
-static void qdict_destroy_obj(QObject *obj)
+void qdict_destroy_obj(QObject *obj)
 {
     int i;
     QDict *qdict;
qobject/qfloat.c +2 -4
index 2472189..1dffb54 100644
@@ -15,8 +15,6 @@
 #include "qapi/qmp/qobject.h"
 #include "qemu-common.h"
 
-static void qfloat_destroy_obj(QObject *obj);
-
 /**
  * qfloat_from_int(): Create a new QFloat from a float
  *
@@ -28,7 +26,7 @@ QFloat *qfloat_from_double(double value)
 
     qf = g_malloc(sizeof(*qf));
     qf->value = value;
-    QOBJECT_INIT(qf, QTYPE_QFLOAT, qfloat_destroy_obj);
+    QOBJECT_INIT(qf, QTYPE_QFLOAT);
 
     return qf;
 }
@@ -56,7 +54,7 @@ QFloat *qobject_to_qfloat(const QObject *obj)
  * qfloat_destroy_obj(): Free all memory allocated by a
  * QFloat object
  */
-static void qfloat_destroy_obj(QObject *obj)
+void qfloat_destroy_obj(QObject *obj)
 {
     assert(obj != NULL);
     g_free(qobject_to_qfloat(obj));
qobject/qint.c +2 -4
index 765307f..112ee68 100644
@@ -14,8 +14,6 @@
 #include "qapi/qmp/qobject.h"
 #include "qemu-common.h"
 
-static void qint_destroy_obj(QObject *obj);
-
 /**
  * qint_from_int(): Create a new QInt from an int64_t
  *
@@ -27,7 +25,7 @@ QInt *qint_from_int(int64_t value)
 
     qi = g_malloc(sizeof(*qi));
     qi->value = value;
-    QOBJECT_INIT(qi, QTYPE_QINT, qint_destroy_obj);
+    QOBJECT_INIT(qi, QTYPE_QINT);
 
     return qi;
 }
@@ -55,7 +53,7 @@ QInt *qobject_to_qint(const QObject *obj)
  * qint_destroy_obj(): Free all memory allocated by a
  * QInt object
  */
-static void qint_destroy_obj(QObject *obj)
+void qint_destroy_obj(QObject *obj)
 {
     assert(obj != NULL);
     g_free(qobject_to_qint(obj));
qobject/qlist.c +2 -4
index 1e2098a..28be4f6 100644
@@ -15,8 +15,6 @@
 #include "qemu/queue.h"
 #include "qemu-common.h"
 
-static void qlist_destroy_obj(QObject *obj);
-
 /**
  * qlist_new(): Create a new QList
  *
@@ -28,7 +26,7 @@ QList *qlist_new(void)
 
     qlist = g_malloc(sizeof(*qlist));
     QTAILQ_INIT(&qlist->head);
-    QOBJECT_INIT(qlist, QTYPE_QLIST, qlist_destroy_obj);
+    QOBJECT_INIT(qlist, QTYPE_QLIST);
 
     return qlist;
 }
@@ -146,7 +144,7 @@ QList *qobject_to_qlist(const QObject *obj)
 /**
  * qlist_destroy_obj(): Free all the memory allocated by a QList
  */
-static void qlist_destroy_obj(QObject *obj)
+void qlist_destroy_obj(QObject *obj)
 {
     QList *qlist;
     QListEntry *entry, *next_entry;
qobject/qnull.c +0 -1
index 22d59e9..5f7ba4d 100644
@@ -16,5 +16,4 @@
 QObject qnull_ = {
     .type = QTYPE_QNULL,
     .refcnt = 1,
-    .destroy = NULL,
 };
qobject/qobject.c new file +26 -0
new file mode 100644
index 0000000..db86571
@@ -0,0 +1,26 @@
+/*
+ * QObject
+ *
+ * Copyright (C) 2015 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1
+ * or later.  See the COPYING.LIB file in the top-level directory.
+ */
+
+#include "qemu-common.h"
+#include "qapi/qmp/qbool.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qfloat.h"
+#include "qapi/qmp/qint.h"
+#include "qapi/qmp/qlist.h"
+#include "qapi/qmp/qstring.h"
+
+QDestroy qdestroy[QTYPE_MAX] = {
+    [QTYPE_QBOOL] = qbool_destroy_obj,
+    [QTYPE_QDICT] = qdict_destroy_obj,
+    [QTYPE_QFLOAT] = qfloat_destroy_obj,
+    [QTYPE_QINT] = qint_destroy_obj,
+    [QTYPE_QLIST] = qlist_destroy_obj,
+    [QTYPE_QSTRING] = qstring_destroy_obj,
+    /* [QTYPE_QNULL] = NULL, */
+};
qobject/qstring.c +2 -4
index ba59d00..a2f5903 100644
@@ -14,8 +14,6 @@
 #include "qapi/qmp/qstring.h"
 #include "qemu-common.h"
 
-static void qstring_destroy_obj(QObject *obj);
-
 /**
  * qstring_new(): Create a new empty QString
  *
@@ -52,7 +50,7 @@ QString *qstring_from_substr(const char *str, int start, int end)
     memcpy(qstring->string, str + start, qstring->length);
     qstring->string[qstring->length] = 0;
 
-    QOBJECT_INIT(qstring, QTYPE_QSTRING, qstring_destroy_obj);
+    QOBJECT_INIT(qstring, QTYPE_QSTRING);
 
     return qstring;
 }
@@ -133,7 +131,7 @@ const char *qstring_get_str(const QString *qstring)
  * qstring_destroy_obj(): Free all memory allocated by a QString
  * object
  */
-static void qstring_destroy_obj(QObject *obj)
+void qstring_destroy_obj(QObject *obj)
 {
     QString *qs;
 
Signature · 1 line
2.4.3
#72 lore raw

Re: [Qemu-devel] [PATCH] fixup? qapi: Simplify QObject

Quoted 162 lines · Eric Blake

[If we squash, replace the old commit message with this; you'll also have some merge conflicts in 29/36 and 30/36. Note that while this mail is net lines added, squashing would still result in a net reduction:

 16 files changed, 67 insertions(+), 83 deletions(-)

]

qapi: Simplify QObject

The QObject hierarchy is small enough, and unlikely to grow further (since we only use it to map to JSON and already cover all JSON types), that we can simplify things by not tracking a separate vtable, but just inline the refcnt element of the vtable QType directly into QObject, and track a separate array of destroy functions. We can drop qnull_destroy_obj() in the process.

The remaining QObject subclasses must export their destructor.

This also has the nice benefit of moving the typename 'QType' out of the way, so that the next patch can repurpose it for a nicer name for 'qtype_code'.

The various objects are still the same size (so no change in cache line pressure), but now have less indirection (although I didn't bother benchmarking to see if there is a noticeable speedup, as we don't have hard evidence that this was in a performance hotspot in the first place).

---

If we don't want to squash, I can instead rebase this to appear after 30/36 as a standalone patch, with a message something like:

qapi: Don't duplicate destructor pointer per QObject

The previous patch "qapi: Simplify QObject" made each QObject instance larger; restore the original size for less cache line pressure by moving the destroy function pointer out of each object and instead maintaining a global table of destroy functions. This requires exporting the various functions, and is made easier to read through a typedef.

---
 include/qapi/qmp/qbool.h   |  1 +
 include/qapi/qmp/qdict.h   |  1 +
 include/qapi/qmp/qfloat.h  |  1 +
 include/qapi/qmp/qint.h    |  1 +
 include/qapi/qmp/qlist.h   |  1 +
 include/qapi/qmp/qobject.h | 16 ++++++++--------
 include/qapi/qmp/qstring.h |  1 +
 qobject/Makefile.objs      |  2 +-
 qobject/qbool.c            |  6 ++----
 qobject/qdict.c            |  6 ++----
 qobject/qfloat.c           |  6 ++----
 qobject/qint.c             |  6 ++----
 qobject/qlist.c            |  6 ++----
 qobject/qnull.c            |  1 -
 qobject/qobject.c          | 26 ++++++++++++++++++++++++++
 qobject/qstring.c          |  6 ++----
 16 files changed, 53 insertions(+), 34 deletions(-)
 create mode 100644 qobject/qobject.c
include/qapi/qmp/qbool.h +1 -0
index d9256e4..836d078 100644
@@ -25,5 +25,6 @@ typedef struct QBool {
 QBool *qbool_from_bool(bool value);
 bool qbool_get_bool(const QBool *qb);
 QBool *qobject_to_qbool(const QObject *obj);
+void qbool_destroy_obj(QObject *obj);
 
 #endif /* QBOOL_H */
include/qapi/qmp/qdict.h +1 -0
index 787c658..6c2a0e5 100644
@@ -48,6 +48,7 @@ void qdict_iter(const QDict *qdict,
                 void *opaque);
 const QDictEntry *qdict_first(const QDict *qdict);
 const QDictEntry *qdict_next(const QDict *qdict, const QDictEntry *entry);
+void qdict_destroy_obj(QObject *obj);
 
 /* Helper to qdict_put_obj(), accepts any object */
 #define qdict_put(qdict, key, obj) \
include/qapi/qmp/qfloat.h +1 -0
index 46745e5..a8af2a8 100644
@@ -25,5 +25,6 @@ typedef struct QFloat {
 QFloat *qfloat_from_double(double value);
 double qfloat_get_double(const QFloat *qi);
 QFloat *qobject_to_qfloat(const QObject *obj);
+void qfloat_destroy_obj(QObject *obj);
 
 #endif /* QFLOAT_H */
include/qapi/qmp/qint.h +1 -0
index 339a9ab..049e528 100644
@@ -24,5 +24,6 @@ typedef struct QInt {
 QInt *qint_from_int(int64_t value);
 int64_t qint_get_int(const QInt *qi);
 QInt *qobject_to_qint(const QObject *obj);
+void qint_destroy_obj(QObject *obj);
 
 #endif /* QINT_H */
include/qapi/qmp/qlist.h +1 -0
index b1bf785..a84117e 100644
@@ -49,6 +49,7 @@ QObject *qlist_peek(QList *qlist);
 int qlist_empty(const QList *qlist);
 size_t qlist_size(const QList *qlist);
 QList *qobject_to_qlist(const QObject *obj);
+void qlist_destroy_obj(QObject *obj);
 
 static inline const QListEntry *qlist_first(const QList *qlist)
 {
include/qapi/qmp/qobject.h +6 -6
index 83ed70b..4e89a7f 100644
@@ -52,15 +52,17 @@ typedef struct QObject QObject;
 struct QObject {
     qtype_code type;
     size_t refcnt;
-    void (*destroy)(QObject *);
 };
 
+typedef void (*QDestroy)(QObject *);
+extern QDestroy qdestroy[QTYPE_MAX];
+
 /* Get the 'base' part of an object */
 #define QOBJECT(obj) (&(obj)->base)
 
 /* High-level interface for qobject_init() */
-#define QOBJECT_INIT(obj, type, destroy) \
-    qobject_init(QOBJECT(obj), type, destroy)
+#define QOBJECT_INIT(obj, type) \
+    qobject_init(QOBJECT(obj), type)
 
 /* High-level interface for qobject_incref() */
 #define QINCREF(obj)      \
@@ -71,13 +73,11 @@ struct QObject {
     qobject_decref(obj ? QOBJECT(obj) : NULL)
 
 /* Initialize an object to default values */
-static inline void qobject_init(QObject *obj, qtype_code type,
-                                void (*destroy)(struct QObject *))
+static inline void qobject_init(QObject *obj, qtype_code type)
 {
     assert(type);
     obj->refcnt = 1;
     obj->type = type;
-    obj->destroy = destroy;
 }
 
 /**

include/qapi/qmp/qobject.h static inline void qobject_decref(QObject *obj) · line 104

Quoted 10 lines
include/qapi/qmp/qobject.h +2 -2
@@ -97,8 +97,8 @@ static inline void qobject_decref(QObject *obj)
 {
     assert(!obj || obj->refcnt);
     if (obj && --obj->refcnt == 0) {
-        assert(obj->destroy);
-        obj->destroy(obj);
+        assert(qdestroy[obj->type]);
+        qdestroy[obj->type](obj);
     }
 }
 

Preexisting: the assertion is of little value, because all it does is convert one kind of crash into another.

Asserting obj->type is in range might be more useful, i.e.

  -        assert(obj->destroy);
  -        obj->destroy(obj);
  +        assert(QTYPE_QNULL < obj->type && obj->type < QTYPE_MAX);
  +        qdestroy[obj->type](obj);

We could outline the conditional part, and keep qdestroy[] static. Could save a bit of code (unless NDEBUG, but we shouldn't optimize for that).

Quoted 206 lines
include/qapi/qmp/qstring.h +1 -0
index 34675a7..df7df55 100644
@@ -32,5 +32,6 @@ void qstring_append_int(QString *qstring, int64_t value);
 void qstring_append(QString *qstring, const char *str);
 void qstring_append_chr(QString *qstring, int c);
 QString *qobject_to_qstring(const QObject *obj);
+void qstring_destroy_obj(QObject *obj);
 
 #endif /* QSTRING_H */
qobject/Makefile.objs +1 -1
index 0031e8b..bed5508 100644
@@ -1,2 +1,2 @@
 util-obj-y = qnull.o qint.o qstring.o qdict.o qlist.o qfloat.o qbool.o
-util-obj-y += qjson.o json-lexer.o json-streamer.o json-parser.o
+util-obj-y += qjson.o qobject.o json-lexer.o json-streamer.o json-parser.o
qobject/qbool.c +2 -4
index 9f46e67..895fd4d 100644
@@ -15,8 +15,6 @@
 #include "qapi/qmp/qobject.h"
 #include "qemu-common.h"
 
-static void qbool_destroy_obj(QObject *obj);
-
 /**
  * qbool_from_bool(): Create a new QBool from a bool
  *
@@ -28,7 +26,7 @@ QBool *qbool_from_bool(bool value)
 
     qb = g_malloc(sizeof(*qb));
     qb->value = value;
-    QOBJECT_INIT(qb, QTYPE_QBOOL, qbool_destroy_obj);
+    QOBJECT_INIT(qb, QTYPE_QBOOL);
 
     return qb;
 }
@@ -56,7 +54,7 @@ QBool *qobject_to_qbool(const QObject *obj)
  * qbool_destroy_obj(): Free all memory allocated by a
  * QBool object
  */
-static void qbool_destroy_obj(QObject *obj)
+void qbool_destroy_obj(QObject *obj)
 {
     assert(obj != NULL);
     g_free(qobject_to_qbool(obj));
qobject/qdict.c +2 -4
index cf62269..dd3f1c2 100644
@@ -19,8 +19,6 @@
 #include "qemu/queue.h"
 #include "qemu-common.h"
 
-static void qdict_destroy_obj(QObject *obj);
-
 /**
  * qdict_new(): Create a new QDict
  *
@@ -31,7 +29,7 @@ QDict *qdict_new(void)
     QDict *qdict;
 
     qdict = g_malloc0(sizeof(*qdict));
-    QOBJECT_INIT(qdict, QTYPE_QDICT, qdict_destroy_obj);
+    QOBJECT_INIT(qdict, QTYPE_QDICT);
 
     return qdict;
 }
@@ -436,7 +434,7 @@ void qdict_del(QDict *qdict, const char *key)
 /**
  * qdict_destroy_obj(): Free all the memory allocated by a QDict
  */
-static void qdict_destroy_obj(QObject *obj)
+void qdict_destroy_obj(QObject *obj)
 {
     int i;
     QDict *qdict;
qobject/qfloat.c +2 -4
index 2472189..1dffb54 100644
@@ -15,8 +15,6 @@
 #include "qapi/qmp/qobject.h"
 #include "qemu-common.h"
 
-static void qfloat_destroy_obj(QObject *obj);
-
 /**
  * qfloat_from_int(): Create a new QFloat from a float
  *
@@ -28,7 +26,7 @@ QFloat *qfloat_from_double(double value)
 
     qf = g_malloc(sizeof(*qf));
     qf->value = value;
-    QOBJECT_INIT(qf, QTYPE_QFLOAT, qfloat_destroy_obj);
+    QOBJECT_INIT(qf, QTYPE_QFLOAT);
 
     return qf;
 }
@@ -56,7 +54,7 @@ QFloat *qobject_to_qfloat(const QObject *obj)
  * qfloat_destroy_obj(): Free all memory allocated by a
  * QFloat object
  */
-static void qfloat_destroy_obj(QObject *obj)
+void qfloat_destroy_obj(QObject *obj)
 {
     assert(obj != NULL);
     g_free(qobject_to_qfloat(obj));
qobject/qint.c +2 -4
index 765307f..112ee68 100644
@@ -14,8 +14,6 @@
 #include "qapi/qmp/qobject.h"
 #include "qemu-common.h"
 
-static void qint_destroy_obj(QObject *obj);
-
 /**
  * qint_from_int(): Create a new QInt from an int64_t
  *
@@ -27,7 +25,7 @@ QInt *qint_from_int(int64_t value)
 
     qi = g_malloc(sizeof(*qi));
     qi->value = value;
-    QOBJECT_INIT(qi, QTYPE_QINT, qint_destroy_obj);
+    QOBJECT_INIT(qi, QTYPE_QINT);
 
     return qi;
 }
@@ -55,7 +53,7 @@ QInt *qobject_to_qint(const QObject *obj)
  * qint_destroy_obj(): Free all memory allocated by a
  * QInt object
  */
-static void qint_destroy_obj(QObject *obj)
+void qint_destroy_obj(QObject *obj)
 {
     assert(obj != NULL);
     g_free(qobject_to_qint(obj));
qobject/qlist.c +2 -4
index 1e2098a..28be4f6 100644
@@ -15,8 +15,6 @@
 #include "qemu/queue.h"
 #include "qemu-common.h"
 
-static void qlist_destroy_obj(QObject *obj);
-
 /**
  * qlist_new(): Create a new QList
  *
@@ -28,7 +26,7 @@ QList *qlist_new(void)
 
     qlist = g_malloc(sizeof(*qlist));
     QTAILQ_INIT(&qlist->head);
-    QOBJECT_INIT(qlist, QTYPE_QLIST, qlist_destroy_obj);
+    QOBJECT_INIT(qlist, QTYPE_QLIST);
 
     return qlist;
 }
@@ -146,7 +144,7 @@ QList *qobject_to_qlist(const QObject *obj)
 /**
  * qlist_destroy_obj(): Free all the memory allocated by a QList
  */
-static void qlist_destroy_obj(QObject *obj)
+void qlist_destroy_obj(QObject *obj)
 {
     QList *qlist;
     QListEntry *entry, *next_entry;
qobject/qnull.c +0 -1
index 22d59e9..5f7ba4d 100644
@@ -16,5 +16,4 @@
 QObject qnull_ = {
     .type = QTYPE_QNULL,
     .refcnt = 1,
-    .destroy = NULL,
 };
qobject/qobject.c new file +16 -0
new file mode 100644
index 0000000..db86571
@@ -0,0 +1,26 @@
+/*
+ * QObject
+ *
+ * Copyright (C) 2015 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1
+ * or later.  See the COPYING.LIB file in the top-level directory.
+ */
+
+#include "qemu-common.h"
+#include "qapi/qmp/qbool.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qfloat.h"
+#include "qapi/qmp/qint.h"
+#include "qapi/qmp/qlist.h"
+#include "qapi/qmp/qstring.h"

qobject/qobject.c line 26

Quoted 10 lines
qobject/qobject.c new file +10 -0
+
+QDestroy qdestroy[QTYPE_MAX] = {
+    [QTYPE_QBOOL] = qbool_destroy_obj,
+    [QTYPE_QDICT] = qdict_destroy_obj,
+    [QTYPE_QFLOAT] = qfloat_destroy_obj,
+    [QTYPE_QINT] = qint_destroy_obj,
+    [QTYPE_QLIST] = qlist_destroy_obj,
+    [QTYPE_QSTRING] = qstring_destroy_obj,
+    /* [QTYPE_QNULL] = NULL, */
+};

Suggest

   QDestroy qdestroy[QTYPE_MAX] = {
       [QTYPE_QNULL] = NULL,       /* no such object exists */
       [QTYPE_QNULL] = NULL,       /* qnull_ is indestructible */
       ...
   };
Quoted 23 lines
qobject/qstring.c +1 -3
index ba59d00..a2f5903 100644
@@ -14,8 +14,6 @@
 #include "qapi/qmp/qstring.h"
 #include "qemu-common.h"
 
-static void qstring_destroy_obj(QObject *obj);
-
 /**
  * qstring_new(): Create a new empty QString
  *
@@ -52,7 +50,7 @@ QString *qstring_from_substr(const char *str, int start, int end)
     memcpy(qstring->string, str + start, qstring->length);
     qstring->string[qstring->length] = 0;
 
-    QOBJECT_INIT(qstring, QTYPE_QSTRING, qstring_destroy_obj);
+    QOBJECT_INIT(qstring, QTYPE_QSTRING);
 
     return qstring;
 }

qobject/qstring.c const char *qstring_get_str(const QString *qstring) · line 136

Quoted 7 lines
qobject/qstring.c +1 -1
@@ -133,7 +131,7 @@ const char *qstring_get_str(const QString *qstring)
  * qstring_destroy_obj(): Free all memory allocated by a QString
  * object
  */
-static void qstring_destroy_obj(QObject *obj)
+void qstring_destroy_obj(QObject *obj)
 {
     QString *qs;

If you'd like to respin, I'd prefer a non-incremental [PATCH v12.5 28/36] replacing v12.

If not, I'm happy to squash in trivial changes like the one I suggested for qdestroy[]'s initializer.

#73 lore raw

Re: [Qemu-devel] [PATCH] fixup? qapi: Simplify QObject

Quoted 1 line · Markus Armbruster

Eric Blake <[email protected]> writes:

[...]

Quoted 35 lines
Quoted 22 lines
qobject/qobject.c new file +16 -0
new file mode 100644
index 0000000..db86571
@@ -0,0 +1,26 @@
+/*
+ * QObject
+ *
+ * Copyright (C) 2015 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1
+ * or later.  See the COPYING.LIB file in the top-level directory.
+ */
+
+#include "qemu-common.h"
+#include "qapi/qmp/qbool.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qfloat.h"
+#include "qapi/qmp/qint.h"
+#include "qapi/qmp/qlist.h"
+#include "qapi/qmp/qstring.h"

qobject/qobject.c line 26

Quoted 10 lines
qobject/qobject.c new file +10 -0
+
+QDestroy qdestroy[QTYPE_MAX] = {
+    [QTYPE_QBOOL] = qbool_destroy_obj,
+    [QTYPE_QDICT] = qdict_destroy_obj,
+    [QTYPE_QFLOAT] = qfloat_destroy_obj,
+    [QTYPE_QINT] = qint_destroy_obj,
+    [QTYPE_QLIST] = qlist_destroy_obj,
+    [QTYPE_QSTRING] = qstring_destroy_obj,
+    /* [QTYPE_QNULL] = NULL, */
+};

Suggest

Quoted 2 lines
   QDestroy qdestroy[QTYPE_MAX] = {
       [QTYPE_QNULL] = NULL,       /* no such object exists */

QTYPE_NONE, of course.

Quoted 3 lines

[QTYPE_QNULL] = NULL, /* qnull_ is indestructible */
...
};

[...]

author
#74 lore raw

Re: [Qemu-devel] [PATCH] fixup? qapi: Simplify QObject

In reply to Markus Armbruster
Quoted 8 lines · Markus Armbruster
Quoted 6 lines · Eric Blake

[If we squash, replace the old commit message with this; you'll also have some merge conflicts in 29/36 and 30/36. Note that while this mail is net lines added, squashing would still result in a net reduction:

 16 files changed, 67 insertions(+), 83 deletions(-)

]

I'm leaning towards squashing, so I'll post a v12.5 of 28-30.

Quoted 11 lines

include/qapi/qmp/qobject.h typedef struct QObject QObject; · line 59

Quoted 11 lines

+++ b/include/qapi/qmp/qobject.h

include/qapi/qmp/qobject.h +3 -1
@@ -52,15 +52,17 @@ typedef struct QObject QObject;
 struct QObject {
     qtype_code type;
     size_t refcnt;
-    void (*destroy)(QObject *);
 };
 
+typedef void (*QDestroy)(QObject *);
+extern QDestroy qdestroy[QTYPE_MAX];
+

So if I'm understanding you, instead of declaring the table,...

Quoted 17 lines

include/qapi/qmp/qobject.h static inline void qobject_decref(QObject *obj) · line 104

Quoted 11 lines
include/qapi/qmp/qobject.h +2 -2
@@ -97,8 +97,8 @@ static inline void qobject_decref(QObject *obj)
 {
     assert(!obj || obj->refcnt);
     if (obj && --obj->refcnt == 0) {
-        assert(obj->destroy);
-        obj->destroy(obj);
+        assert(qdestroy[obj->type]);
+        qdestroy[obj->type](obj);
     }
 }
 

Preexisting: the assertion is of little value, because all it does is convert one kind of crash into another.

Asserting obj->type is in range might be more useful, i.e.

Quoted 8 lines
  -        assert(obj->destroy);
  -        obj->destroy(obj);
  +        assert(QTYPE_QNULL < obj->type && obj->type < QTYPE_MAX);
  +        qdestroy[obj->type](obj);

We could outline the conditional part, and keep qdestroy[] static. Could save a bit of code (unless NDEBUG, but we shouldn't optimize for that).

...I could do:

/* High-level interface for qobject_decref() */

#define QDECREF(obj)              \
    do { \
        if (obj) { \
            assert(QOBJECT(obj)->refcnt)); \
            if (!--QOBJECT(obj)->refcnt) { \
                qobject_decref(QOBJECT(obj)); \
            } \
        } \
    } while (0)

and move qobject_decref() into qobject.c where the destroy table is then static to that file.

Correct?

Oh wait, it won't work quite like that. We have direct callers of qobject_decref() (QDECREF is only usable on subclasses, but if you have a QObject* you can't use QDECREF), so I can't change semantics by hoisting things into the macro.

Quoted 12 lines

qobject/qobject.c line 26

Quoted 9 lines
qobject/qobject.c new file +9 -0
+QDestroy qdestroy[QTYPE_MAX] = {
+    [QTYPE_QBOOL] = qbool_destroy_obj,
+    [QTYPE_QDICT] = qdict_destroy_obj,
+    [QTYPE_QFLOAT] = qfloat_destroy_obj,
+    [QTYPE_QINT] = qint_destroy_obj,
+    [QTYPE_QLIST] = qlist_destroy_obj,
+    [QTYPE_QSTRING] = qstring_destroy_obj,
+    /* [QTYPE_QNULL] = NULL, */
+};

Suggest

Quoted 3 lines
   QDestroy qdestroy[QTYPE_MAX] = {
       [QTYPE_QNULL] = NULL,       /* no such object exists */
       [QTYPE_QNULL] = NULL,       /* qnull_ is indestructible */

Makes sense (with the obvious fix for QTYPE_NONE).

Quoted 2 lines

If you'd like to respin, I'd prefer a non-incremental [PATCH v12.5 28/36] replacing v12.

Plus 29 and 30 for conflict resolutions.

Quoted 4 lines

If not, I'm happy to squash in trivial changes like the one I suggested for qdestroy[]'s initializer.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#75 lore raw

Re: [Qemu-devel] [PATCH] fixup? qapi: Simplify QObject

Quoted 11 lines · Eric Blake
Quoted 8 lines · Markus Armbruster
Quoted 6 lines · Eric Blake

[If we squash, replace the old commit message with this; you'll also have some merge conflicts in 29/36 and 30/36. Note that while this mail is net lines added, squashing would still result in a net reduction:

 16 files changed, 67 insertions(+), 83 deletions(-)

]

I'm leaning towards squashing, so I'll post a v12.5 of 28-30.

I'll push my current branch shortly. I'm happy with it up to PATCH 26. I'm still pondering PATCH 27.

Quoted 53 lines
Quoted 11 lines

include/qapi/qmp/qobject.h typedef struct QObject QObject; · line 59

Quoted 11 lines

+++ b/include/qapi/qmp/qobject.h

include/qapi/qmp/qobject.h +3 -1
@@ -52,15 +52,17 @@ typedef struct QObject QObject;
 struct QObject {
     qtype_code type;
     size_t refcnt;
-    void (*destroy)(QObject *);
 };
 
+typedef void (*QDestroy)(QObject *);
+extern QDestroy qdestroy[QTYPE_MAX];
+

So if I'm understanding you, instead of declaring the table,...

Quoted 17 lines

include/qapi/qmp/qobject.h static inline void qobject_decref(QObject *obj) · line 104

Quoted 11 lines
include/qapi/qmp/qobject.h +2 -2
@@ -97,8 +97,8 @@ static inline void qobject_decref(QObject *obj)
 {
     assert(!obj || obj->refcnt);
     if (obj && --obj->refcnt == 0) {
-        assert(obj->destroy);
-        obj->destroy(obj);
+        assert(qdestroy[obj->type]);
+        qdestroy[obj->type](obj);
     }
 }
 

Preexisting: the assertion is of little value, because all it does is convert one kind of crash into another.

Asserting obj->type is in range might be more useful, i.e.

Quoted 8 lines
  -        assert(obj->destroy);
  -        obj->destroy(obj);
  +        assert(QTYPE_QNULL < obj->type && obj->type < QTYPE_MAX);
  +        qdestroy[obj->type](obj);

We could outline the conditional part, and keep qdestroy[] static. Could save a bit of code (unless NDEBUG, but we shouldn't optimize for that).

...I could do:

/* High-level interface for qobject_decref() */

#define QDECREF(obj)              \
    do { \
        if (obj) { \
            assert(QOBJECT(obj)->refcnt)); \
            if (!--QOBJECT(obj)->refcnt) { \
                qobject_decref(QOBJECT(obj)); \
            } \
        } \
    } while (0)
Quoted 4 lines

and move qobject_decref() into qobject.c where the destroy table is then static to that file.

Correct?

Almost.

Quoted 4 lines

Oh wait, it won't work quite like that. We have direct callers of qobject_decref() (QDECREF is only usable on subclasses, but if you have a QObject* you can't use QDECREF), so I can't change semantics by hoisting things into the macro.

In qobject.h:

/**
* qobject_decref(): Decrement QObject's reference count, deallocate
* when it reaches zero
*/
static inline void qobject_decref(QObject *obj)

{
    assert(!obj || obj->refcnt);
    if (obj && --obj->refcnt == 0) {
        qobject_destroy(obj);
    }
}

In qobject.c:

void qobject_destroy(QObject *obj)

{
    assert(!obj->refcnt);
    assert(QTYPE_QNULL < obj->type && obj->type < QTYPE_MAX);
    qdestroy[obj->type](obj);
}
Quoted 27 lines
Quoted 12 lines

qobject/qobject.c line 26

Quoted 9 lines
qobject/qobject.c new file +9 -0
+QDestroy qdestroy[QTYPE_MAX] = {
+    [QTYPE_QBOOL] = qbool_destroy_obj,
+    [QTYPE_QDICT] = qdict_destroy_obj,
+    [QTYPE_QFLOAT] = qfloat_destroy_obj,
+    [QTYPE_QINT] = qint_destroy_obj,
+    [QTYPE_QLIST] = qlist_destroy_obj,
+    [QTYPE_QSTRING] = qstring_destroy_obj,
+    /* [QTYPE_QNULL] = NULL, */
+};

Suggest

Quoted 3 lines
   QDestroy qdestroy[QTYPE_MAX] = {
       [QTYPE_QNULL] = NULL,       /* no such object exists */
       [QTYPE_QNULL] = NULL,       /* qnull_ is indestructible */

Makes sense (with the obvious fix for QTYPE_NONE).

Quoted 2 lines

If you'd like to respin, I'd prefer a non-incremental [PATCH v12.5 28/36] replacing v12.

Plus 29 and 30 for conflict resolutions.

Quoted 3 lines

If not, I'm happy to squash in trivial changes like the one I suggested for qdestroy[]'s initializer.

author
#76 lore raw

PATCH v12 29/36 qobject: Rename qtype_code to QType

In reply to Eric Blake

The name QType is more in line with our conventions for qapi types, and matches the fact that each enum member has a prefix of QTYPE_.

---
v12: new patch split off of 21/28

---
 block/qapi.c               | 4 ++--
 include/hw/qdev-core.h     | 2 +-
 include/qapi/qmp/qobject.h | 8 ++++----
 qobject/qdict.c            | 3 +--
 scripts/qapi.py            | 2 +-
 5 files changed, 9 insertions(+), 10 deletions(-)
block/qapi.c +2 -2
index d20262d..4eb48fc 100644
@@ -590,7 +590,7 @@ static void dump_qlist(fprintf_function func_fprintf, void *f, int indentation,
     int i = 0;
 
     for (entry = qlist_first(list); entry; entry = qlist_next(entry), i++) {
-        qtype_code type = qobject_type(entry->value);
+        QType type = qobject_type(entry->value);
         bool composite = (type == QTYPE_QDICT || type == QTYPE_QLIST);
         const char *format = composite ? "%*s[%i]:\n" : "%*s[%i]: ";
 
@@ -608,7 +608,7 @@ static void dump_qdict(fprintf_function func_fprintf, void *f, int indentation,
     const QDictEntry *entry;
 
     for (entry = qdict_first(dict); entry; entry = qdict_next(dict, entry)) {
-        qtype_code type = qobject_type(entry->value);
+        QType type = qobject_type(entry->value);
         bool composite = (type == QTYPE_QDICT || type == QTYPE_QLIST);
         const char *format = composite ? "%*s%s:\n" : "%*s%s: ";
         char key[strlen(entry->key) + 1];
include/hw/qdev-core.h +1 -1
index e6dbde4..c66429a 100644
@@ -239,7 +239,7 @@ struct Property {
     PropertyInfo *info;
     int          offset;
     uint8_t      bitnr;
-    qtype_code   qtype;
+    QType        qtype;
     int64_t      defval;
     int          arrayoffset;
     PropertyInfo *arrayinfo;
include/qapi/qmp/qobject.h +4 -4
index 83ed70b..0558ff3 100644
@@ -45,12 +45,12 @@ typedef enum {
     QTYPE_QFLOAT,
     QTYPE_QBOOL,
     QTYPE_MAX,
-} qtype_code;
+} QType;
 
 typedef struct QObject QObject;
 
 struct QObject {
-    qtype_code type;
+    QType type;
     size_t refcnt;
     void (*destroy)(QObject *);
 };
@@ -71,7 +71,7 @@ struct QObject {
     qobject_decref(obj ? QOBJECT(obj) : NULL)
 
 /* Initialize an object to default values */
-static inline void qobject_init(QObject *obj, qtype_code type,
+static inline void qobject_init(QObject *obj, QType type,
                                 void (*destroy)(struct QObject *))
 {
     assert(type);
@@ -105,7 +105,7 @@ static inline void qobject_decref(QObject *obj)
 /**
  * qobject_type(): Return the QObject's type
  */
-static inline qtype_code qobject_type(const QObject *obj)
+static inline QType qobject_type(const QObject *obj)
 {
     assert(obj->type);
     return obj->type;
qobject/qdict.c +1 -2
index cf62269..d5e6df1 100644
@@ -179,8 +179,7 @@ size_t qdict_size(const QDict *qdict)
 /**
  * qdict_get_obj(): Get a QObject of a specific type
  */
-static QObject *qdict_get_obj(const QDict *qdict, const char *key,
-                              qtype_code type)
+static QObject *qdict_get_obj(const QDict *qdict, const char *key, QType type)
 {
     QObject *obj;
 
scripts/qapi.py +1 -1
index e41dbaf..7c0ef76 100644
@@ -34,7 +34,7 @@ builtin_types = {
     'uint32':   'QTYPE_QINT',
     'uint64':   'QTYPE_QINT',
     'size':     'QTYPE_QINT',
-    'any':      None,           # any qtype_code possible, actually
+    'any':      None,           # any QType possible, actually
 }
 
 # Whitelist of commands allowed to return a non-dictionary
Signature · 1 line
2.4.3
#77 lore raw

Re: [Qemu-devel] [PATCH v12 29/36] qobject: Rename qtype_code to QType

Quoted 5 lines · Eric Blake

The name QType is more in line with our conventions for qapi types, and matches the fact that each enum member has a prefix of QTYPE_.

At this point, the connection to QAPI is unobvious.

You can either point to CODING_STYLE, which ask for type names in CamelCase, or to the next patch.

Patch looks good.

author
#78 lore raw

Re: [Qemu-devel] [PATCH v12 29/36] qobject: Rename qtype_code to QType

Quoted 6 lines · Markus Armbruster
Quoted 4 lines · Eric Blake

The name QType is more in line with our conventions for qapi types, and matches the fact that each enum member has a prefix of QTYPE_.

Quoted 6 lines
Quoted 1 line

At this point, the connection to QAPI is unobvious.

You can either point to CODING_STYLE, which ask for type names in CamelCase, or to the next patch.

How about:

The name QType matches our CODING_STYLE conventions for type names in CamelCase. It also matches the fact that we are already naming all the enum members with a prefix of QTYPE, not QTYPE_CODE. And doing the rename will also make it easier for the next patch to use QAPI for the enum, which also wants CamelCase type names.

Feel free to touch up the commit message as you see fit.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#79 lore raw

Re: [Qemu-devel] [PATCH v12 29/36] qobject: Rename qtype_code to QType

Quoted 16 lines · Eric Blake
Quoted 6 lines · Markus Armbruster
Quoted 4 lines · Eric Blake

The name QType is more in line with our conventions for qapi types, and matches the fact that each enum member has a prefix of QTYPE_.

Quoted 6 lines
Quoted 1 line

At this point, the connection to QAPI is unobvious.

You can either point to CODING_STYLE, which ask for type names in CamelCase, or to the next patch.

How about:

Quoted 7 lines

The name QType matches our CODING_STYLE conventions for type names in CamelCase. It also matches the fact that we are already naming all the enum members with a prefix of QTYPE, not QTYPE_CODE. And doing the rename will also make it easier for the next patch to use QAPI for the enum, which also wants CamelCase type names.

Feel free to touch up the commit message as you see fit.

Sold!

author
#80 lore raw

PATCH v12 30/36 qapi: Convert QType into qapi builtin enum type

In reply to Eric Blake

What's more meta than using qapi to define qapi? :)

Convert QType into a full-fledged[*] builtin qapi enum type, so that a subsequent patch can then use it as the discriminator type of qapi alternate types. Fortunately, the judicious use of 'prefix' in the qapi definition avoids churn to the spelling of the enum constants.

To avoid circular definitions, we have to flip the order of inclusion between "qobject.h" vs. "qapi-types.h". Back in commit 28770e0, we had the latter include the former, so that we could use 'QObject *' for our implementation of 'any'. But that usage also works with only a forward declaration, whereas the definition of QObject requires QType to be a complete type.

[*] The type has to be builtin, rather than declared in qapi/common.json, because we want to use it for alternates even when common.json is not included. But since it is the first builtin enum type, we have to add special cases to qapi-types and qapi-visit to only emit definitions once, even when two qapi files are being compiled into the same binary (the way we already handled builtin list types like 'intList'). We may need to revisit how multiple qapi files share common types, but that's a project for another day.

---
v12: split out preparatory renames, retitle patch, use info rather than name comparison
v11: new patch

---
 docs/qapi-code-gen.txt                   |  1 +
 include/qapi/qmp/qobject.h               | 15 +--------------
 scripts/qapi-types.py                    | 16 ++++++++++++----
 scripts/qapi-visit.py                    | 11 +++++++++--
 scripts/qapi.py                          |  6 ++++++
 tests/qapi-schema/alternate-empty.out    |  2 ++
 tests/qapi-schema/comments.out           |  2 ++
 tests/qapi-schema/empty.out              |  2 ++
 tests/qapi-schema/event-case.out         |  2 ++
 tests/qapi-schema/flat-union-empty.out   |  2 ++
 tests/qapi-schema/ident-with-escape.out  |  2 ++
 tests/qapi-schema/include-relpath.out    |  2 ++
 tests/qapi-schema/include-repetition.out |  2 ++
 tests/qapi-schema/include-simple.out     |  2 ++
 tests/qapi-schema/indented-expr.out      |  2 ++
 tests/qapi-schema/qapi-schema-test.out   |  2 ++
 tests/qapi-schema/union-clash-data.out   |  2 ++
 tests/qapi-schema/union-empty.out        |  2 ++
 18 files changed, 55 insertions(+), 20 deletions(-)
docs/qapi-code-gen.txt +1 -0
index 1f6cb16..4edad89 100644
@@ -168,6 +168,7 @@ The following types are predefined, and map to C as follows:
                        accepts size suffixes
   bool      bool       JSON true or false
   any       QObject *  any JSON value
+  QType     QType      JSON string matching enum QType values
 
 
 === Includes ===
include/qapi/qmp/qobject.h +1 -14
index 0558ff3..16bf50d 100644
@@ -34,20 +34,7 @@
 
 #include <stddef.h>
 #include <assert.h>
-
-typedef enum {
-    QTYPE_NONE,    /* sentinel value, no QObject has this type code */
-    QTYPE_QNULL,
-    QTYPE_QINT,
-    QTYPE_QSTRING,
-    QTYPE_QDICT,
-    QTYPE_QLIST,
-    QTYPE_QFLOAT,
-    QTYPE_QBOOL,
-    QTYPE_MAX,
-} QType;
-
-typedef struct QObject QObject;
+#include "qapi-types.h"
 
 struct QObject {
     QType type;
scripts/qapi-types.py +12 -4
index 2f2f7df..99346ad 100644
@@ -112,7 +112,7 @@ extern const int %(c_name)s_qtypes[];
 def gen_alternate_qtypes(name, variants):
     ret = mcgen('''
 
-const int %(c_name)s_qtypes[QTYPE_MAX] = {
+const int %(c_name)s_qtypes[QTYPE__MAX] = {
 ''',
                 c_name=c_name(name))
 
@@ -233,8 +233,15 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
         self.defn += gen_type_cleanup(name)
 
     def visit_enum_type(self, name, info, values, prefix):
-        self._fwdecl += gen_enum(name, values, prefix)
-        self._fwdefn += gen_enum_lookup(name, values, prefix)
+        # Special case for our lone builtin enum type
+        # TODO use something cleaner than existence of info
+        if not info:
+            self._btin += gen_enum(name, values, prefix)
+            if do_builtins:
+                self.defn += gen_enum_lookup(name, values, prefix)
+        else:
+            self._fwdecl += gen_enum(name, values, prefix)
+            self._fwdefn += gen_enum_lookup(name, values, prefix)
 
     def visit_array_type(self, name, info, element_type):
         if isinstance(element_type, QAPISchemaBuiltinType):
@@ -319,7 +326,8 @@ fdef.write(mcgen('''
 fdecl.write(mcgen('''
 #include <stdbool.h>
 #include <stdint.h>
-#include "qapi/qmp/qobject.h"
+
+typedef struct QObject QObject;
 '''))
 
 schema = QAPISchema(input_file)
scripts/qapi-visit.py +9 -2
index 94cd113..7ceda18 100644
@@ -347,8 +347,15 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
                     isinstance(entity, QAPISchemaObjectType))
 
     def visit_enum_type(self, name, info, values, prefix):
-        self.decl += gen_visit_decl(name, scalar=True)
-        self.defn += gen_visit_enum(name)
+        # Special case for our lone builtin enum type
+        # TODO use something cleaner than existence of info
+        if not info:
+            self._btin += gen_visit_decl(name, scalar=True)
+            if do_builtins:
+                self.defn += gen_visit_enum(name)
+        else:
+            self.decl += gen_visit_decl(name, scalar=True)
+            self.defn += gen_visit_enum(name)
 
     def visit_array_type(self, name, info, element_type):
         decl = gen_visit_decl(name)
scripts/qapi.py +6 -0
index 7c0ef76..a90ea12 100644
@@ -35,6 +35,7 @@ builtin_types = {
     'uint64':   'QTYPE_QINT',
     'size':     'QTYPE_QINT',
     'any':      None,           # any QType possible, actually
+    'QType':    'QTYPE_QSTRING',
 }
 
 # Whitelist of commands allowed to return a non-dictionary
@@ -1267,6 +1268,11 @@ class QAPISchema(object):
         self.the_empty_object_type = QAPISchemaObjectType(':empty', None, None,
                                                           [], None)
         self._def_entity(self.the_empty_object_type)
+        self._def_entity(QAPISchemaEnumType('QType', None,
+                                            ['none', 'qnull', 'qint',
+                                             'qstring', 'qdict', 'qlist',
+                                             'qfloat', 'qbool'],
+                                            'QTYPE'))
 
     def _make_implicit_enum_type(self, name, info, values):
         name = name + 'Kind'   # Use namespace reserved by add_name()
tests/qapi-schema/alternate-empty.out +2 -0
index 0f153b6..02b9876 100644
@@ -2,3 +2,5 @@ object :empty
 alternate Alt
     case i: int
 enum AltKind ['i']
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
tests/qapi-schema/comments.out +2 -0
index 9e2c656..97be601 100644
@@ -1,2 +1,4 @@
 object :empty
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/empty.out +2 -0
index 272b161..6522940 100644
@@ -1 +1,3 @@
 object :empty
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
tests/qapi-schema/event-case.out +2 -0
index cdfd264..6350d64 100644
@@ -1,2 +1,4 @@
 object :empty
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
 event oops None
tests/qapi-schema/flat-union-empty.out +2 -0
index 0e0665a..eade2d5 100644
@@ -2,6 +2,8 @@ object :empty
 object Base
     member type: Empty optional=False
 enum Empty []
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
 object Union
     base Base
     tag type
tests/qapi-schema/ident-with-escape.out +2 -0
index f4542b1..453e0b2 100644
@@ -1,5 +1,7 @@
 object :empty
 object :obj-fooA-arg
     member bar1: str optional=False
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
 command fooA :obj-fooA-arg -> None
    gen=True success_response=True
tests/qapi-schema/include-relpath.out +2 -0
index 9e2c656..97be601 100644
@@ -1,2 +1,4 @@
 object :empty
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-repetition.out +2 -0
index 9e2c656..97be601 100644
@@ -1,2 +1,4 @@
 object :empty
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/include-simple.out +2 -0
index 9e2c656..97be601 100644
@@ -1,2 +1,4 @@
 object :empty
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
 enum Status ['good', 'bad', 'ugly']
tests/qapi-schema/indented-expr.out +2 -0
index 226d300..ce37ff5 100644
@@ -1,4 +1,6 @@
 object :empty
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
 command eins None -> None
    gen=True success_response=True
 command zwei None -> None
tests/qapi-schema/qapi-schema-test.out +2 -0
index 0724a9f..b87cfba 100644
@@ -101,6 +101,8 @@ object NestedEnumsOne
     member enum4: EnumOne optional=True
 enum QEnumTwo ['value1', 'value2']
     prefix QENUM_TWO
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
 object TestStruct
     member integer: int optional=False
     member boolean: bool optional=False
tests/qapi-schema/union-clash-data.out +2 -0
index cea8551..f5752f4 100644
@@ -1,6 +1,8 @@
 object :empty
 object :obj-int-wrapper
     member data: int optional=False
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
 object TestUnion
     member type: TestUnionKind optional=False
     case data: :obj-int-wrapper
tests/qapi-schema/union-empty.out +2 -0
index 9c89fd1..bdf17e5 100644
@@ -1,4 +1,6 @@
 object :empty
+enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
+    prefix QTYPE
 object Union
     member type: UnionKind optional=False
 enum UnionKind []
Signature · 1 line
2.4.3
#81 lore raw

Re: [Qemu-devel] [PATCH v12 30/36] qapi: Convert QType into qapi builtin enum type

Quoted 8 lines · Eric Blake

What's more meta than using qapi to define qapi? :)

Convert QType into a full-fledged[*] builtin qapi enum type, so that a subsequent patch can then use it as the discriminator type of qapi alternate types. Fortunately, the judicious use of 'prefix' in the qapi definition avoids churn to the spelling of the enum constants.

Quoted 6 lines

To avoid circular definitions, we have to flip the order of inclusion between "qobject.h" vs. "qapi-types.h". Back in commit 28770e0, we had the latter include the former, so that we could use 'QObject *' for our implementation of 'any'. But that usage also works with only a forward declaration, whereas the definition of QObject requires QType to be a complete type.

I figure putting the typedef in include/qemu/typedefs.h would be simpler. Didn't think of it when I reviewed v11.

Quoted 11 lines

[*] The type has to be builtin, rather than declared in qapi/common.json, because we want to use it for alternates even when common.json is not included. But since it is the first builtin enum type, we have to add special cases to qapi-types and qapi-visit to only emit definitions once, even when two qapi files are being compiled into the same binary (the way we already handled builtin list types like 'intList'). We may need to revisit how multiple qapi files share common types, but that's a project for another day.

Quoted 1 line

Patch looks sane.

#82 lore raw

Re: [Qemu-devel] [PATCH v12 30/36] qapi: Convert QType into qapi builtin enum type

In reply to Eric Blake

Suggest to spell it "QAPI built-in enum type" in the subject.

In general, I prefer "QAPI" when referring to the concept, because it's actually an acronym.

author
#83 lore raw

PATCH 1/1 fixup! qapi: Convert QType into qapi builtin enum type

In reply to Eric Blake

[keep the original commit body, but change the title:]

qapi: Convert QType into QAPI built-in enum type

---
 include/qemu/typedefs.h | 1 +
 scripts/qapi-types.py   | 3 +--
 2 files changed, 2 insertions(+), 2 deletions(-)
include/qemu/typedefs.h +1 -0
index 6b1093d..3c88a05 100644
@@ -80,6 +80,7 @@ typedef struct QEMUSGList QEMUSGList;
 typedef struct QEMUSizedBuffer QEMUSizedBuffer;
 typedef struct QEMUTimerListGroup QEMUTimerListGroup;
 typedef struct QEMUTimer QEMUTimer;
+typedef struct QObject QObject;
 typedef struct Range Range;
 typedef struct RAMBlock RAMBlock;
 typedef struct SerialState SerialState;
scripts/qapi-types.py +1 -2
index 99346ad..82635b0 100644
@@ -326,8 +326,7 @@ fdef.write(mcgen('''
 fdecl.write(mcgen('''
 #include <stdbool.h>
 #include <stdint.h>
-
-typedef struct QObject QObject;
+#include "qemu/typedefs.h"
 '''))
 
 schema = QAPISchema(input_file)
Signature · 1 line
2.4.3
author
#84 lore raw

PATCH v12 31/36 qapi: Simplify visiting of alternate types

In reply to Eric Blake

Previously, working with alternates required two lookup arrays and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a value of the generated FooKind enum, then look up that value in FooKind_lookup[] like we do for other union types.

This has a couple of subtle bugs. First, the generator was creating a call with a parameter '(int *) &(*obj)->type' where type is an enum type; this is unsafe if the compiler chooses to store the enum type in a different size than int, where assigning through the wrong size pointer can corrupt data or cause a SIGBUS. [We still have the casting bug for our enum visitors, but that's a topic for a different patch.]

Second, since the values of the FooKind enum start at zero, all entries of the Foo_qtypes[] array that were not explicitly initialized will map to the same branch of the union as the first member of the alternate, rather than triggering a desired failure in visit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which normally fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).

However, the second bug IS observable in one case: parsing an integer causes unusual behavior in an alternate that contains at least a 'number' member but no 'int' member, because the 'number' parser accepts QTYPE_QINT in addition to the expected QTYPE_QFLOAT (that is, since 'int' is not a member, the type QTYPE_QINT accidentally maps to FooKind 0; if this enum value is the 'number' branch the integer parses successfully, but if the 'number' branch is not first, some other branch tries to parse the integer and rejects it). A later patch will worry about fixing alternates to always parse all inputs that a non-alternate 'number' would accept, for now this is still marked FIXME in the updated test-qmp-input-visitor.c, to merely point out that new undesired behavior of 'ans' matches the existing undesired behavior of 'asn'.

This patch fixes the default-initialization bug by deleting the indirection, and modifying get_next_type() to directly assign a QTypeCode parameter. This in turn fixes the type-casting bug, as we are no longer casting a pointer to enum to a questionable size. There is no longer a need to generate an implicit FooKind enum associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names). Since the updated visit_get_next_type() does not know which qtypes are expected, the generated visitor is modified to generate an error statement if an unexpected type is encountered.

Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. I considered the possibility of keeping the internal enum FooKind, but initialized differently than most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A = QTYPE_QDICT,
FOO_KIND_B = QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to use foo->a or foo->b when inspecting foo->type; but it turned out to add too much complexity, especially without a client.

There is a user-visible side effect to this change, but I consider it to be an improvement. Previously, the invalid QMP command:
{"execute":"blockdev-add", "arguments":{"options":

    {"driver":"raw", "id":"a", "file":true}}}
failed with:
  {"error": {"class": "GenericError",
    "desc": "Invalid parameter type for 'file', expected: QDict"}}

(visit_get_next_type() succeeded, and the error comes from the visit_type_BlockdevOptions() expecting {}; there is no mention of the fact that a string would also work). Now it fails with:
{"error": {"class": "GenericError",

    "desc": "Invalid parameter type for 'file', expected: BlockdevRef"}}

(the error when the next type doesn't match any expected types for the overall alternate).

---
v12: rebase to earlier 'max' collision avoidance, some variable renames v11 (no v10): rebase to new QTypeCode, with fewer special cases; tweak commit message to match
v9: rebase to earlier changes, rework commit message to mention second bug fix; move positive test in qapi-schema-test to later patch
v8: no change
v7: rebase onto earlier changes, rework how subtype makes things work
v6: rebase onto tag_member subclass, testsuite, gen_err_check(), and info improvements

---
 docs/qapi-code-gen.txt                 |  3 ---
 include/qapi/visitor-impl.h            |  3 ++-
 include/qapi/visitor.h                 |  8 +++++++-
 qapi/qapi-visit-core.c                 |  4 ++--
 qapi/qmp-input-visitor.c               |  4 ++--
 scripts/qapi-types.py                  | 34 ----------------------------------
 scripts/qapi-visit.py                  | 14 +++++++++-----
 scripts/qapi.py                        | 18 +++++++++++++-----
 tests/qapi-schema/alternate-empty.out  |  1 -
 tests/qapi-schema/qapi-schema-test.out |  8 --------
 tests/test-qmp-input-visitor.c         | 31 ++++++++++++++++---------------
 tests/test-qmp-output-visitor.c        |  4 ++--
 12 files changed, 53 insertions(+), 79 deletions(-)
docs/qapi-code-gen.txt +0 -3
index 4edad89..286103b 100644
@@ -392,9 +392,6 @@ where each branch of the union names a QAPI type.  For example:
    'data': { 'definition': 'BlockdevOptions',
              'reference': 'str' } }
 
-Just like for a simple union, an implicit C enum 'NameKind' is created
-to enumerate the branches for the alternate 'Name'.
-
 Unlike a union, the discriminator string is never passed on the wire
 for the Client JSON Protocol.  Instead, the value's JSON type serves
 as an implicit discriminator, which in turn means that an alternate
include/qapi/visitor-impl.h +2 -1
index 8c0ba57..7cd1313 100644
@@ -32,7 +32,8 @@ struct Visitor
 
     void (*type_enum)(Visitor *v, int *obj, const char * const strings[],
                       const char *kind, const char *name, Error **errp);
-    void (*get_next_type)(Visitor *v, int *kind, const int *qobjects,
+    /* May be NULL; only needed for input visitors. */
+    void (*get_next_type)(Visitor *v, QType *type,
                           const char *name, Error **errp);
 
     void (*type_int)(Visitor *v, int64_t *obj, const char *name, Error **errp);
include/qapi/visitor.h +7 -1
index a2ad66c..6d25ad2 100644
@@ -38,7 +38,13 @@ GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp);
 void visit_end_list(Visitor *v, Error **errp);
 void visit_optional(Visitor *v, bool *present, const char *name,
                     Error **errp);
-void visit_get_next_type(Visitor *v, int *obj, const int *qtypes,
+
+/**
+ * Determine the qtype of the item @name in the current object visit.
+ * For input visitors, set *@type to the correct qtype of a qapi
+ * alternate type; for other visitors, leave *@type unchanged.
+ */
+void visit_get_next_type(Visitor *v, QType *type,
                          const char *name, Error **errp);
 void visit_type_enum(Visitor *v, int *obj, const char * const strings[],
                      const char *kind, const char *name, Error **errp);
qapi/qapi-visit-core.c +2 -2
index 59ed506..850ca03 100644
@@ -81,11 +81,11 @@ void visit_optional(Visitor *v, bool *present, const char *name,
     }
 }
 
-void visit_get_next_type(Visitor *v, int *obj, const int *qtypes,
+void visit_get_next_type(Visitor *v, QType *type,
                          const char *name, Error **errp)
 {
     if (v->get_next_type) {
-        v->get_next_type(v, obj, qtypes, name, errp);
+        v->get_next_type(v, type, name, errp);
     }
 }
 
qapi/qmp-input-visitor.c +2 -2
index eb6e110..d398de7 100644
@@ -208,7 +208,7 @@ static void qmp_input_end_list(Visitor *v, Error **errp)
     qmp_input_pop(qiv, errp);
 }
 
-static void qmp_input_get_next_type(Visitor *v, int *kind, const int *qobjects,
+static void qmp_input_get_next_type(Visitor *v, QType *type,
                                     const char *name, Error **errp)
 {
     QmpInputVisitor *qiv = to_qiv(v);
@@ -218,7 +218,7 @@ static void qmp_input_get_next_type(Visitor *v, int *kind, const int *qobjects,
         error_setg(errp, QERR_MISSING_PARAMETER, name ? name : "null");
         return;
     }
-    *kind = qobjects[qobject_type(qobj)];
+    *type = qobject_type(qobj);
 }
 
 static void qmp_input_type_int(Visitor *v, int64_t *obj, const char *name,
scripts/qapi-types.py +0 -34
index 99346ad..3c6b3d0 100644
@@ -101,38 +101,6 @@ static inline %(base)s *qapi_%(c_name)s_base(const %(c_name)s *obj)
                  c_name=c_name(name), base=base.c_name())
 
 
-def gen_alternate_qtypes_decl(name):
-    return mcgen('''
-
-extern const int %(c_name)s_qtypes[];
-''',
-                 c_name=c_name(name))
-
-
-def gen_alternate_qtypes(name, variants):
-    ret = mcgen('''
-
-const int %(c_name)s_qtypes[QTYPE__MAX] = {
-''',
-                c_name=c_name(name))
-
-    for var in variants.variants:
-        qtype = var.type.alternate_qtype()
-        assert qtype
-
-        ret += mcgen('''
-    [%(qtype)s] = %(enum_const)s,
-''',
-                     qtype=qtype,
-                     enum_const=c_enum_const(variants.tag_member.type.name,
-                                             var.name))
-
-    ret += mcgen('''
-};
-''')
-    return ret
-
-
 def gen_variants(variants):
     # FIXME: What purpose does data serve, besides preventing a union that
     # has a branch named 'data'? We use it in qapi-visit.py to decide
@@ -264,9 +232,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 
     def visit_alternate_type(self, name, info, variants):
         self._fwdecl += gen_fwd_object_or_array(name)
-        self._fwdefn += gen_alternate_qtypes(name, variants)
         self.decl += gen_object(name, None, [variants.tag_member], variants)
-        self.decl += gen_alternate_qtypes_decl(name)
         self._gen_type_cleanup(name)
 
 # If you link code generated from multiple schemata, you want only one
scripts/qapi-visit.py +9 -5
index 7ceda18..e8b53b3 100644
@@ -193,7 +193,7 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
     if (err) {
         goto out;
     }
-    visit_get_next_type(v, (int*) &(*obj)->type, %(c_name)s_qtypes, name, &err);
+    visit_get_next_type(v, &(*obj)->type, name, &err);
     if (err) {
         goto out_obj;
     }
@@ -201,20 +201,22 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
 ''',
                 c_name=c_name(name))
 
+    # FIXME: When 'number' but not 'int' is present in the alternate, we
+    # should allow QTYPE_INT to promote to QTYPE_FLOAT.
     for var in variants.variants:
         ret += mcgen('''
     case %(case)s:
         visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, name, &err);
         break;
 ''',
-                     case=c_enum_const(variants.tag_member.type.name,
-                                       var.name),
+                     case=var.type.alternate_qtype(),
                      c_type=var.type.c_name(),
                      c_name=c_name(var.name))
 
     ret += mcgen('''
     default:
-        abort();
+        error_setg(&err, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+                   "%(name)s");
     }
 out_obj:
     error_propagate(errp, err);
@@ -223,7 +225,8 @@ out_obj:
 out:
     error_propagate(errp, err);
 }
-''')
+''',
+                 name=name)
 
     return ret
 
@@ -437,6 +440,7 @@ fdef.write(mcgen('''
 
 fdecl.write(mcgen('''
 #include "qapi/visitor.h"
+#include "qapi/qmp/qerror.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
scripts/qapi.py +13 -5
index a90ea12..3756d41 100644
@@ -636,8 +636,8 @@ def check_alternate(expr, expr_info):
     for (key, value) in members.items():
         check_name(expr_info, "Member of alternate '%s'" % name, key)
 
-        # Check for conflicts in the generated enum
-        c_key = camel_to_upper(key)
+        # Check for conflicts in the branch names
+        c_key = c_name(key)
         if c_key in values:
             raise QAPIExprError(expr_info,
                                 "Alternate '%s' member '%s' clashes with '%s'"
@@ -1092,8 +1092,11 @@ class QAPISchemaObjectTypeVariants(object):
         assert isinstance(self.tag_member.type, QAPISchemaEnumType)
         for v in self.variants:
             v.check(schema)
-            assert v.name in self.tag_member.type.values
-            if isinstance(v.type, QAPISchemaObjectType):
+            # Union names must match enum values; alternate names are
+            # checked separately. Use 'seen' to tell the two apart.
+            if seen:
+                assert v.name in self.tag_member.type.values
+                assert isinstance(v.type, QAPISchemaObjectType)
                 v.type.check(schema)
 
     def check_clash(self, schema, info, seen):
@@ -1135,6 +1138,11 @@ class QAPISchemaAlternateType(QAPISchemaType):
         # Not calling self.variants.check_clash(), because there's nothing
         # to clash with
         self.variants.check(schema, {})
+        # Alternate branch names have no relation to the tag enum values;
+        # so we have to check for potential name collisions ourselves.
+        seen = {}
+        for v in self.variants.variants:
+            v.check_clash(self.info, seen)
 
     def json_type(self):
         return 'value'
@@ -1365,7 +1373,7 @@ class QAPISchema(object):
         data = expr['data']
         variants = [self._make_variant(key, value)
                     for (key, value) in data.iteritems()]
-        tag_member = self._make_implicit_tag(name, info, variants)
+        tag_member = QAPISchemaObjectTypeMember('type', 'QType', False)
         self._def_entity(
             QAPISchemaAlternateType(name, info,
                                     QAPISchemaObjectTypeVariants(None,
tests/qapi-schema/alternate-empty.out +0 -1
index 02b9876..f78f174 100644
@@ -1,6 +1,5 @@
 object :empty
 alternate Alt
     case i: int
-enum AltKind ['i']
 enum QType ['none', 'qnull', 'qint', 'qstring', 'qdict', 'qlist', 'qfloat', 'qbool']
     prefix QTYPE
tests/qapi-schema/qapi-schema-test.out +0 -8
index b87cfba..2c546b7 100644
@@ -56,27 +56,21 @@ object :obj-user_def_cmd2-arg
 alternate AltIntNum
     case i: int
     case n: number
-enum AltIntNumKind ['i', 'n']
 alternate AltNumInt
     case n: number
     case i: int
-enum AltNumIntKind ['n', 'i']
 alternate AltNumStr
     case n: number
     case s: str
-enum AltNumStrKind ['n', 's']
 alternate AltStrBool
     case s: str
     case b: bool
-enum AltStrBoolKind ['s', 'b']
 alternate AltStrInt
     case s: str
     case i: int
-enum AltStrIntKind ['s', 'i']
 alternate AltStrNum
     case s: str
     case n: number
-enum AltStrNumKind ['s', 'n']
 event EVENT_A None
 event EVENT_B None
 event EVENT_C :obj-EVENT_C-arg
@@ -114,7 +108,6 @@ alternate UserDefAlternate
     case uda: UserDefA
     case s: str
     case i: int
-enum UserDefAlternateKind ['uda', 's', 'i']
 object UserDefB
     member intb: int optional=False
     member a-b: bool optional=True
@@ -180,7 +173,6 @@ event __ORG.QEMU_X-EVENT __org.qemu_x-Struct
 alternate __org.qemu_x-Alt
     case __org.qemu_x-branch: str
     case b: __org.qemu_x-Base
-enum __org.qemu_x-AltKind ['__org.qemu_x-branch', 'b']
 object __org.qemu_x-Base
     member __org.qemu_x-member1: __org.qemu_x-Enum optional=False
 enum __org.qemu_x-Enum ['__org.qemu_x-value']
tests/test-qmp-input-visitor.c +16 -15
index d48ebdd..43b9e18 100644
@@ -312,13 +312,13 @@ static void test_visitor_in_alternate(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42");
     visit_type_UserDefAlternate(v, &tmp, NULL, &error_abort);
-    g_assert_cmpint(tmp->type, ==, USER_DEF_ALTERNATE_KIND_I);
+    g_assert_cmpint(tmp->type, ==, QTYPE_QINT);
     g_assert_cmpint(tmp->u.i, ==, 42);
     qapi_free_UserDefAlternate(tmp);
 
     v = visitor_input_test_init(data, "'string'");
     visit_type_UserDefAlternate(v, &tmp, NULL, &error_abort);
-    g_assert_cmpint(tmp->type, ==, USER_DEF_ALTERNATE_KIND_S);
+    g_assert_cmpint(tmp->type, ==, QTYPE_QSTRING);
     g_assert_cmpstr(tmp->u.s, ==, "string");
     qapi_free_UserDefAlternate(tmp);
 
@@ -347,36 +347,37 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     error_free_or_abort(&err);
     qapi_free_AltStrBool(asb);
 
-    /* FIXME: Order of alternate should not affect semantics; asn should
-     * parse the same as ans */
+    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
     visit_type_AltStrNum(v, &asn, NULL, &err);
-    /* FIXME g_assert_cmpint(asn->type, == ALT_STR_NUM_KIND_N); */
+    /* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */
     /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */
     error_free_or_abort(&err);
     qapi_free_AltStrNum(asn);
 
+    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
-    visit_type_AltNumStr(v, &ans, NULL, &error_abort);
-    g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N);
-    g_assert_cmpfloat(ans->u.n, ==, 42);
+    visit_type_AltNumStr(v, &ans, NULL, &err);
+    /* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */
+    /* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */
+    error_free_or_abort(&err);
     qapi_free_AltNumStr(ans);
 
     v = visitor_input_test_init(data, "42");
     visit_type_AltStrInt(v, &asi, NULL, &error_abort);
-    g_assert_cmpint(asi->type, ==, ALT_STR_INT_KIND_I);
+    g_assert_cmpint(asi->type, ==, QTYPE_QINT);
     g_assert_cmpint(asi->u.i, ==, 42);
     qapi_free_AltStrInt(asi);
 
     v = visitor_input_test_init(data, "42");
     visit_type_AltIntNum(v, &ain, NULL, &error_abort);
-    g_assert_cmpint(ain->type, ==, ALT_INT_NUM_KIND_I);
+    g_assert_cmpint(ain->type, ==, QTYPE_QINT);
     g_assert_cmpint(ain->u.i, ==, 42);
     qapi_free_AltIntNum(ain);
 
     v = visitor_input_test_init(data, "42");
     visit_type_AltNumInt(v, &ani, NULL, &error_abort);
-    g_assert_cmpint(ani->type, ==, ALT_NUM_INT_KIND_I);
+    g_assert_cmpint(ani->type, ==, QTYPE_QINT);
     g_assert_cmpint(ani->u.i, ==, 42);
     qapi_free_AltNumInt(ani);
 
@@ -389,13 +390,13 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltStrNum(v, &asn, NULL, &error_abort);
-    g_assert_cmpint(asn->type, ==, ALT_STR_NUM_KIND_N);
+    g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(asn->u.n, ==, 42.5);
     qapi_free_AltStrNum(asn);
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltNumStr(v, &ans, NULL, &error_abort);
-    g_assert_cmpint(ans->type, ==, ALT_NUM_STR_KIND_N);
+    g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(ans->u.n, ==, 42.5);
     qapi_free_AltNumStr(ans);
 
@@ -406,13 +407,13 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltIntNum(v, &ain, NULL, &error_abort);
-    g_assert_cmpint(ain->type, ==, ALT_INT_NUM_KIND_N);
+    g_assert_cmpint(ain->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(ain->u.n, ==, 42.5);
     qapi_free_AltIntNum(ain);
 
     v = visitor_input_test_init(data, "42.5");
     visit_type_AltNumInt(v, &ani, NULL, &error_abort);
-    g_assert_cmpint(ani->type, ==, ALT_NUM_INT_KIND_N);
+    g_assert_cmpint(ani->type, ==, QTYPE_QFLOAT);
     g_assert_cmpfloat(ani->u.n, ==, 42.5);
     qapi_free_AltNumInt(ani);
 }
tests/test-qmp-output-visitor.c +2 -2
index 4196e66..3078442 100644
@@ -428,7 +428,7 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,
     UserDefAlternate *tmp;
 
     tmp = g_new0(UserDefAlternate, 1);
-    tmp->type = USER_DEF_ALTERNATE_KIND_I;
+    tmp->type = QTYPE_QINT;
     tmp->u.i = 42;
 
     visit_type_UserDefAlternate(data->ov, &tmp, NULL, &error_abort);
@@ -441,7 +441,7 @@ static void test_visitor_out_alternate(TestOutputVisitorData *data,
     qobject_decref(arg);
 
     tmp = g_new0(UserDefAlternate, 1);
-    tmp->type = USER_DEF_ALTERNATE_KIND_S;
+    tmp->type = QTYPE_QSTRING;
     tmp->u.s = g_strdup("hello");
 
     visit_type_UserDefAlternate(data->ov, &tmp, NULL, &error_abort);
Signature · 1 line
2.4.3
#85 lore raw

Re: [Qemu-devel] [PATCH v12 31/36] qapi: Simplify visiting of alternate types

Quoted 6 lines · Eric Blake

Previously, working with alternates required two lookup arrays and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a value of the generated FooKind enum, then look up that value in FooKind_lookup[] like we do for other union types.

Quoted 7 lines

This has a couple of subtle bugs. First, the generator was creating a call with a parameter '(int *) &(*obj)->type' where type is an enum type; this is unsafe if the compiler chooses to store the enum type in a different size than int, where assigning through the wrong size pointer can corrupt data or cause a SIGBUS. [We still have the casting bug for our enum visitors, but that's a topic for a different patch.]

I'm not sure I get the last sentence.

Quoted 63 lines

Second, since the values of the FooKind enum start at zero, all entries of the Foo_qtypes[] array that were not explicitly initialized will map to the same branch of the union as the first member of the alternate, rather than triggering a desired failure in visit_get_next_type(). Fortunately, the bug seldom bites; the very next thing the input visitor does is try to parse the incoming JSON with the wrong parser, which normally fails; the output visitor is not used with a C struct in that state, and the dealloc visitor has nothing to clean up (so there is no leak).

However, the second bug IS observable in one case: parsing an integer causes unusual behavior in an alternate that contains at least a 'number' member but no 'int' member, because the 'number' parser accepts QTYPE_QINT in addition to the expected QTYPE_QFLOAT (that is, since 'int' is not a member, the type QTYPE_QINT accidentally maps to FooKind 0; if this enum value is the 'number' branch the integer parses successfully, but if the 'number' branch is not first, some other branch tries to parse the integer and rejects it). A later patch will worry about fixing alternates to always parse all inputs that a non-alternate 'number' would accept, for now this is still marked FIXME in the updated test-qmp-input-visitor.c, to merely point out that new undesired behavior of 'ans' matches the existing undesired behavior of 'asn'.

This patch fixes the default-initialization bug by deleting the indirection, and modifying get_next_type() to directly assign a QTypeCode parameter. This in turn fixes the type-casting bug, as we are no longer casting a pointer to enum to a questionable size. There is no longer a need to generate an implicit FooKind enum associated with the alternate type (since the QMP wire format never uses the stringized counterparts of the C union member names). Since the updated visit_get_next_type() does not know which qtypes are expected, the generated visitor is modified to generate an error statement if an unexpected type is encountered.

Callers now have to know the QTYPE_* mapping when looking at the discriminator; but so far, only the testsuite was even using the C struct of an alternate types. I considered the possibility of keeping the internal enum FooKind, but initialized differently than most generated arrays, as in:
typedef enum FooKind {
FOO_KIND_A = QTYPE_QDICT,
FOO_KIND_B = QTYPE_QINT,
} FooKind;
to create nicer aliases for knowing when to use foo->a or foo->b when inspecting foo->type; but it turned out to add too much complexity, especially without a client.

There is a user-visible side effect to this change, but I consider it to be an improvement. Previously, the invalid QMP command:
{"execute":"blockdev-add", "arguments":{"options":

    {"driver":"raw", "id":"a", "file":true}}}
failed with:
  {"error": {"class": "GenericError",
    "desc": "Invalid parameter type for 'file', expected: QDict"}}

(visit_get_next_type() succeeded, and the error comes from the visit_type_BlockdevOptions() expecting {}; there is no mention of the fact that a string would also work). Now it fails with:
{"error": {"class": "GenericError",

Quoted 5 lines
    "desc": "Invalid parameter type for 'file', expected: BlockdevRef"}}

(the error when the next type doesn't match any expected types for the overall alternate).

Patch looks good.

author
#86 lore raw

Re: [Qemu-devel] [PATCH v12 31/36] qapi: Simplify visiting of alternate types

Quoted 16 lines · Markus Armbruster
Quoted 6 lines · Eric Blake

Previously, working with alternates required two lookup arrays and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a value of the generated FooKind enum, then look up that value in FooKind_lookup[] like we do for other union types.

Quoted 7 lines

This has a couple of subtle bugs. First, the generator was creating a call with a parameter '(int *) &(*obj)->type' where type is an enum type; this is unsafe if the compiler chooses to store the enum type in a different size than int, where assigning through the wrong size pointer can corrupt data or cause a SIGBUS. [We still have the casting bug for our enum visitors, but that's a topic for a different patch.]

Quoted 1 line

I'm not sure I get the last sentence.

I was referring to our casts of enum types to int* inside visit_type_Enum():
https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg03408.html

If you have a better wording for it, or want to drop the parenthetical altogether, I'm fine.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#87 lore raw

Re: [Qemu-devel] [PATCH v12 31/36] qapi: Simplify visiting of alternate types

Quoted 17 lines · Eric Blake
Quoted 16 lines · Markus Armbruster
Quoted 13 lines · Eric Blake

Previously, working with alternates required two lookup arrays and some indirection: for type Foo, we created Foo_qtypes[] which maps each qtype to a value of the generated FooKind enum, then look up that value in FooKind_lookup[] like we do for other union types.

This has a couple of subtle bugs. First, the generator was creating a call with a parameter '(int *) &(*obj)->type' where type is an enum type; this is unsafe if the compiler chooses to store the enum type in a different size than int, where assigning through the wrong size pointer can corrupt data or cause a SIGBUS. [We still have the casting bug for our enum visitors, but that's a topic for a different patch.]

Quoted 7 lines
Quoted 1 line

I'm not sure I get the last sentence.

I was referring to our casts of enum types to int* inside visit_type_Enum():
https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg03408.html

If you have a better wording for it, or want to drop the parenthetical altogether, I'm fine.

What about this: replace the parenthetical with

Related bug, not fixed in this patch: qapi-visit.py's gen_visit_enum() generates a cast of its enum * argument to int *. Marked FIXME.

and squash in

scripts/qapi-visit.py +1 -0
index e8b53b3..4797d6e 100644
@@ -172,6 +172,7 @@ out:
 
 
 def gen_visit_enum(name):
+    # FIXME cast from enum *obj to int * invalidly assumes enum is int
     return mcgen('''
 
 void visit_type_%(c_name)s(Visitor *v, %(c_name)s *obj, const char *name, Error **errp)
author
#88 lore raw

Re: [Qemu-devel] [PATCH v12 31/36] qapi: Simplify visiting of alternate types

Quoted 29 lines · Markus Armbruster
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

This has a couple of subtle bugs. First, the generator was creating a call with a parameter '(int *) &(*obj)->type' where type is an enum type; this is unsafe if the compiler chooses to store the enum type in a different size than int, where assigning through the wrong size pointer can corrupt data or cause a SIGBUS. [We still have the casting bug for our enum visitors, but that's a topic for a different patch.]

Quoted 7 lines
Quoted 1 line

I'm not sure I get the last sentence.

I was referring to our casts of enum types to int* inside visit_type_Enum():
https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg03408.html

If you have a better wording for it, or want to drop the parenthetical altogether, I'm fine.

What about this: replace the parenthetical with

Related bug, not fixed in this patch: qapi-visit.py's gen_visit_enum() generates a cast of its enum * argument to int *. Marked FIXME.

and squash in

scripts/qapi-visit.py +0 -0
index e8b53b3..4797d6e 100644

scripts/qapi-visit.py out: · line 176

Quoted 5 lines
scripts/qapi-visit.py +1 -0
@@ -172,6 +172,7 @@ out:
 
 
 def gen_visit_enum(name):
+    # FIXME cast from enum *obj to int * invalidly assumes enum is int
     return mcgen('''

That works for me.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#89 lore raw

PATCH v12 32/36 qapi: Inline _make_implicit_tag()

In reply to Eric Blake

Now that alternates no longer use an implicit tag, we can inline _make_implicit_tag() into its one caller of _def_union_type().

No change to generated code.

---
v12: new patch

---
 scripts/qapi.py | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
scripts/qapi.py +3 -6
index 3756d41..362afa1 100644
@@ -1342,11 +1342,6 @@ class QAPISchema(object):
             typ, info, 'wrapper', [self._make_member('data', typ, info)])
         return QAPISchemaObjectTypeVariant(case, typ)
 
-    def _make_implicit_tag(self, type_name, info, variants):
-        typ = self._make_implicit_enum_type(type_name, info,
-                                            [v.name for v in variants])
-        return QAPISchemaObjectTypeMember('type', typ, False)
-
     def _def_union_type(self, expr, info):
         name = expr['union']
         data = expr['data']
@@ -1360,7 +1355,9 @@ class QAPISchema(object):
         else:
             variants = [self._make_simple_variant(key, value, info)
                         for (key, value) in data.iteritems()]
-            tag_member = self._make_implicit_tag(name, info, variants)
+            typ = self._make_implicit_enum_type(name, info,
+                                                [v.name for v in variants])
+            tag_member = QAPISchemaObjectTypeMember('type', typ, False)
             members = [tag_member]
         self._def_entity(
             QAPISchemaObjectType(name, info, base, members,
Signature · 1 line
2.4.3
author
#91 lore raw

PATCH v12 33/36 qapi: Fix alternates that accept 'number' but not 'int'

In reply to Eric Blake

The QMP input visitor allows integral values to be assigned by promotion to a QTYPE_QFLOAT. However, when parsing an alternate, we did not take this into account, such that an alternate that accepts 'number' and some other type, but not 'int', would reject integral values.

With this patch, we now have the following desirable table:

    alternate has      case selected for
    'int'  'number'    QTYPE_QINT  QTYPE_QFLOAT
      no        no     error       error
      no       yes     'number'    'number'
     yes        no     'int'       error
     yes       yes     'int'       'number'

While it is unlikely that we will ever use 'number' in an alternate other than in the testsuite, it never hurts to be more precise in what we allow.

---
v12: rebase to QType cleanups
v11 (no v10): slight commit message tweak, rebase to earlier changes
v9: rebase to earlier changes
v8: no change
v7: rebase to named .u union
v6: rebase onto earlier testsuite and gen_err_check() improvements

---
 include/qapi/visitor-impl.h    |  2 +-
 include/qapi/visitor.h         |  3 ++-
 qapi/qapi-visit-core.c         |  4 ++--
 qapi/qmp-input-visitor.c       |  5 ++++-
 scripts/qapi-visit.py          | 11 +++++++----
 tests/test-qmp-input-visitor.c | 16 ++++++----------
 6 files changed, 22 insertions(+), 19 deletions(-)
include/qapi/visitor-impl.h +1 -1
index 7cd1313..7419684 100644
@@ -33,7 +33,7 @@ struct Visitor
     void (*type_enum)(Visitor *v, int *obj, const char * const strings[],
                       const char *kind, const char *name, Error **errp);
     /* May be NULL; only needed for input visitors. */
-    void (*get_next_type)(Visitor *v, QType *type,
+    void (*get_next_type)(Visitor *v, QType *type, bool promote_int,
                           const char *name, Error **errp);
 
     void (*type_int)(Visitor *v, int64_t *obj, const char *name, Error **errp);
include/qapi/visitor.h +2 -1
index 6d25ad2..1414de1 100644
@@ -43,8 +43,9 @@ void visit_optional(Visitor *v, bool *present, const char *name,
  * Determine the qtype of the item @name in the current object visit.
  * For input visitors, set *@type to the correct qtype of a qapi
  * alternate type; for other visitors, leave *@type unchanged.
+ * If @promote_int, treat integers as QTYPE_FLOAT.
  */
-void visit_get_next_type(Visitor *v, QType *type,
+void visit_get_next_type(Visitor *v, QType *type, bool promote_int,
                          const char *name, Error **errp);
 void visit_type_enum(Visitor *v, int *obj, const char * const strings[],
                      const char *kind, const char *name, Error **errp);
qapi/qapi-visit-core.c +2 -2
index 850ca03..cee76bc 100644
@@ -81,11 +81,11 @@ void visit_optional(Visitor *v, bool *present, const char *name,
     }
 }
 
-void visit_get_next_type(Visitor *v, QType *type,
+void visit_get_next_type(Visitor *v, QType *type, bool promote_int,
                          const char *name, Error **errp)
 {
     if (v->get_next_type) {
-        v->get_next_type(v, type, name, errp);
+        v->get_next_type(v, type, promote_int, name, errp);
     }
 }
 
qapi/qmp-input-visitor.c +4 -1
index d398de7..26b7414 100644
@@ -208,7 +208,7 @@ static void qmp_input_end_list(Visitor *v, Error **errp)
     qmp_input_pop(qiv, errp);
 }
 
-static void qmp_input_get_next_type(Visitor *v, QType *type,
+static void qmp_input_get_next_type(Visitor *v, QType *type, bool promote_int,
                                     const char *name, Error **errp)
 {
     QmpInputVisitor *qiv = to_qiv(v);
@@ -219,6 +219,9 @@ static void qmp_input_get_next_type(Visitor *v, QType *type,
         return;
     }
     *type = qobject_type(qobj);
+    if (promote_int && *type == QTYPE_QINT) {
+        *type = QTYPE_QFLOAT;
+    }
 }
 
 static void qmp_input_type_int(Visitor *v, int64_t *obj, const char *name,
scripts/qapi-visit.py +7 -4
index e8b53b3..035ea59 100644
@@ -183,6 +183,11 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s *obj, const char *name, Error
 
 
 def gen_visit_alternate(name, variants):
+    promote_int = 'true'
+    for var in variants.variants:
+        if var.type.alternate_qtype() == 'QTYPE_QINT':
+            promote_int = 'false'
+
     ret = mcgen('''
 
 void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error **errp)
@@ -193,16 +198,14 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s **obj, const char *name, Error
     if (err) {
         goto out;
     }
-    visit_get_next_type(v, &(*obj)->type, name, &err);
+    visit_get_next_type(v, &(*obj)->type, %(promote_int)s, name, &err);
     if (err) {
         goto out_obj;
     }
     switch ((*obj)->type) {
 ''',
-                c_name=c_name(name))
+                c_name=c_name(name), promote_int=promote_int)
 
-    # FIXME: When 'number' but not 'int' is present in the alternate, we
-    # should allow QTYPE_INT to promote to QTYPE_FLOAT.
     for var in variants.variants:
         ret += mcgen('''
     case %(case)s:
tests/test-qmp-input-visitor.c +6 -10
index 43b9e18..b4a5bee 100644
@@ -347,20 +347,16 @@ static void test_visitor_in_alternate_number(TestInputVisitorData *data,
     error_free_or_abort(&err);
     qapi_free_AltStrBool(asb);
 
-    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
-    visit_type_AltStrNum(v, &asn, NULL, &err);
-    /* FIXME g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT); */
-    /* FIXME g_assert_cmpfloat(asn->u.n, ==, 42); */
-    error_free_or_abort(&err);
+    visit_type_AltStrNum(v, &asn, NULL, &error_abort);
+    g_assert_cmpint(asn->type, ==, QTYPE_QFLOAT);
+    g_assert_cmpfloat(asn->u.n, ==, 42);
     qapi_free_AltStrNum(asn);
 
-    /* FIXME: integer should parse as number */
     v = visitor_input_test_init(data, "42");
-    visit_type_AltNumStr(v, &ans, NULL, &err);
-    /* FIXME g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT); */
-    /* FIXME g_assert_cmpfloat(ans->u.n, ==, 42); */
-    error_free_or_abort(&err);
+    visit_type_AltNumStr(v, &ans, NULL, &error_abort);
+    g_assert_cmpint(ans->type, ==, QTYPE_QFLOAT);
+    g_assert_cmpfloat(ans->u.n, ==, 42);
     qapi_free_AltNumStr(ans);
 
     v = visitor_input_test_init(data, "42");
Signature · 1 line
2.4.3
author
#92 lore raw

PATCH v12 34/36 qapi: Add positive tests to qapi-schema-test

In reply to Eric Blake

Add positive tests to qapi-schema-test for things that were made possible by recent patches but which caused compile errors due to collisions prior to that point. The focus is mainly on collisions due to names we have reserved for qapi, even though it is unlikely that anyone will want to abuse these names in actual .json files.

The added tests includes:
Use of a member name ending in 'Kind' or 'List' [1, 3] Use of a type name starting with 'has_' [1, 4] Use of a type named 'u' [1, 5]
Use of a union branch name of 'u' [2, 5]
Use of a union branch name starting with 'has_' [2, 4]

[1] Never broken, but could break if reservations are too strict
[2] Broken prior to commit e4ba22b
[3] See reservations in commit 4dc2e69 and 255960d
[4] See reservations in commit 9fb081e
[5] See reservation in commit 5e59baf

Not worth testing here: we no longer have a collision with a member named 'base' (commit ddf2190), with a branch named 'type' (commit e4ba22b), or with an alternate member named 'max' (previous commits); these names were more accidental namespace pollutions than intentional reservations.

---
v12: no change
v11 (no v10): drop test of 'max'
v9: reorder in series (was 9/17); fewer tests of 'base' and 'type' non-collision; fold in alternate 'max' test; update commit message
v8: new, but collects portions of subset B v10 patches 2, 3, and 16 and subset C v7 patch 6 that were deferred to later.

It might be worth dropping or simplifying this patch, depending on how many corner cases we actually want to test.

---
 tests/qapi-schema/qapi-schema-test.json | 16 ++++++++++++++++
 tests/qapi-schema/qapi-schema-test.out  | 26 ++++++++++++++++++++++++++
 2 files changed, 42 insertions(+)
tests/qapi-schema/qapi-schema-test.json +16 -0
index 4b89527..793323d 100644
@@ -113,6 +113,22 @@
             'sizes': ['size'],
             'any': ['any'] } }
 
+# Even though 'u' and 'has_*' are forbidden as struct member names, they
+# should still be valid as a type or union branch name. And although
+# '*Kind' and '*List' are forbidden as type names, they should not be
+# forbidden as a member or branch name.  Flat union branches do not
+# collide with base members.
+{ 'enum': 'EnumName', 'data': [ 'value1', 'has_a', 'u' ] }
+{ 'struct': 'has_a', 'data': { 'MyKind': 'int', 'MyList': ['int'],
+                               'value1': 'EnumName' } }
+{ 'union': 'u', 'data': { 'u': 'uint8', 'myKind': 'has_a',
+                          'myList': 'has_a', 'has_a': 'has_a' } }
+{ 'union': 'UnionName', 'base': 'has_a', 'discriminator': 'value1',
+  'data': { 'value1': 'UserDefZero', 'has_a': 'UserDefZero',
+            'u': 'UserDefZero' } }
+{ 'alternate': 'AltName', 'data': { 'type': 'int', 'u': 'bool',
+                                    'myKind': 'has_a' } }
+
 # testing commands
 { 'command': 'user_def_cmd', 'data': {} }
 { 'command': 'user_def_cmd1', 'data': {'ud1a': 'UserDefOne'} }
tests/qapi-schema/qapi-schema-test.out +26 -0
index 2c546b7..03c465b 100644
@@ -22,6 +22,8 @@ object :obj-guest-get-time-arg
     member b: int optional=True
 object :obj-guest-sync-arg
     member arg: any optional=False
+object :obj-has_a-wrapper
+    member data: has_a optional=False
 object :obj-int16List-wrapper
     member data: int16List optional=False
 object :obj-int32List-wrapper
@@ -46,6 +48,8 @@ object :obj-uint32List-wrapper
     member data: uint32List optional=False
 object :obj-uint64List-wrapper
     member data: uint64List optional=False
+object :obj-uint8-wrapper
+    member data: uint8 optional=False
 object :obj-uint8List-wrapper
     member data: uint8List optional=False
 object :obj-user_def_cmd1-arg
@@ -56,6 +60,10 @@ object :obj-user_def_cmd2-arg
 alternate AltIntNum
     case i: int
     case n: number
+alternate AltName
+    case type: int
+    case u: bool
+    case myKind: has_a
 alternate AltNumInt
     case n: number
     case i: int
@@ -78,6 +86,7 @@ event EVENT_D :obj-EVENT_D-arg
 object Empty1
 object Empty2
     base Empty1
+enum EnumName ['value1', 'has_a', 'u']
 enum EnumOne ['value1', 'value2', 'value3']
 object EventStructOne
     member struct1: UserDefOne optional=False
@@ -101,6 +110,12 @@ object TestStruct
     member integer: int optional=False
     member boolean: bool optional=False
     member string: str optional=False
+object UnionName
+    base has_a
+    tag value1
+    case value1: UserDefZero
+    case has_a: UserDefZero
+    case u: UserDefZero
 object UserDefA
     member boolean: bool optional=False
     member a_b: int optional=True
@@ -196,6 +211,17 @@ command guest-get-time :obj-guest-get-time-arg -> int
    gen=True success_response=True
 command guest-sync :obj-guest-sync-arg -> any
    gen=True success_response=True
+object has_a
+    member MyKind: int optional=False
+    member MyList: intList optional=False
+    member value1: EnumName optional=False
+object u
+    member type: uKind optional=False
+    case u: :obj-uint8-wrapper
+    case myKind: :obj-has_a-wrapper
+    case myList: :obj-has_a-wrapper
+    case has_a: :obj-has_a-wrapper
+enum uKind ['u', 'myKind', 'myList', 'has_a']
 command user_def_cmd None -> None
    gen=True success_response=True
 command user_def_cmd1 :obj-user_def_cmd1-arg -> None
Signature · 1 line
2.4.3
author
#93 lore raw

Re: [Qemu-devel] [PATCH v12 34/36] qapi: Add positive tests to qapi-schema-test

Quoted 9 lines · Eric Blake

Add positive tests to qapi-schema-test for things that were made possible by recent patches but which caused compile errors due to collisions prior to that point. The focus is mainly on collisions due to names we have reserved for qapi, even though it is unlikely that anyone will want to abuse these names in actual .json files.

The added tests includes:
Use of a member name ending in 'Kind' or 'List' [1, 3]

Violates our coding convention of all lower case, so not worth testing.

Quoted 1 line

Use of a type name starting with 'has_' [1, 4]

Ditto.

Quoted 1 line

Use of a type named 'u' [1, 5]

Ditto.

Quoted 2 lines

Use of a union branch name of 'u' [2, 5]
Use of a union branch name starting with 'has_' [2, 4]

Leaving just these two feels pretty weak. I think I'll just drop this patch from my queue entirely (it was great at proving that recent layout improvements worked, but it's not the end of the world to lose it entirely, especially now that v13 is starting to enforce naming conventions except for a few whitelisted names, and I don't want to make the whitelist longer just for the testsuite).

Quoted 15 lines

[1] Never broken, but could break if reservations are too strict
[2] Broken prior to commit e4ba22b
[3] See reservations in commit 4dc2e69 and 255960d
[4] See reservations in commit 9fb081e
[5] See reservation in commit 5e59baf

Not worth testing here: we no longer have a collision with a member named 'base' (commit ddf2190), with a branch named 'type' (commit e4ba22b), or with an alternate member named 'max' (previous commits); these names were more accidental namespace pollutions than intentional reservations.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
#94 lore raw

Re: [Qemu-devel] [PATCH v12 34/36] qapi: Add positive tests to qapi-schema-test

Quoted 24 lines · Eric Blake
Quoted 9 lines · Eric Blake

Add positive tests to qapi-schema-test for things that were made possible by recent patches but which caused compile errors due to collisions prior to that point. The focus is mainly on collisions due to names we have reserved for qapi, even though it is unlikely that anyone will want to abuse these names in actual .json files.

The added tests includes:
Use of a member name ending in 'Kind' or 'List' [1, 3]

Violates our coding convention of all lower case, so not worth testing.

Quoted 1 line

Use of a type name starting with 'has_' [1, 4]

Ditto.

Quoted 1 line

Use of a type named 'u' [1, 5]

Ditto.

Quoted 2 lines

Use of a union branch name of 'u' [2, 5]
Use of a union branch name starting with 'has_' [2, 4]

Quoted 6 lines

Leaving just these two feels pretty weak. I think I'll just drop this patch from my queue entirely (it was great at proving that recent layout improvements worked, but it's not the end of the world to lose it entirely, especially now that v13 is starting to enforce naming conventions except for a few whitelisted names, and I don't want to make the whitelist longer just for the testsuite).

Makes sense.

[...]

author
#95 lore raw

PATCH v12 35/36 qapi: Simplify visits of optional fields

In reply to Eric Blake

None of the visitor callbacks would set an error when testing if an optional field was present; make this part of the interface contract by eliminating the errp argument.

The resulting generated code has a nice diff:

|-    visit_optional(v, &has_fdset_id, "fdset-id", &err);
|-    if (err) {
|-        goto out;
|-    }
|-    if (has_fdset_id) {
|+    visit_optional(v, &has_fdset_id, "fdset-id");
|+    if (has_fdset_id) {
|         visit_type_int(v, &fdset_id, "fdset-id", &err);
|         if (err) {
|             goto out;
|         }
|     }

---
v12: split error removal from 'if' compression
v11 (no v10): no change
v9: no change
v8: no change
v7: rebase to no member.c_name()
v6: rebase onto earlier testsuite and gen_err_check() improvements

---
 include/qapi/visitor-impl.h |  5 ++---
 include/qapi/visitor.h      | 10 ++++++++--
 qapi/opts-visitor.c         |  2 +-
 qapi/qapi-visit-core.c      |  5 ++---
 qapi/qmp-input-visitor.c    |  3 +--
 qapi/string-input-visitor.c |  3 +--
 scripts/qapi.py             |  8 ++------
 7 files changed, 17 insertions(+), 19 deletions(-)
include/qapi/visitor-impl.h +2 -3
index 7419684..44a21b7 100644
@@ -44,9 +44,8 @@ struct Visitor
     void (*type_any)(Visitor *v, QObject **obj, const char *name,
                      Error **errp);
 
-    /* May be NULL */
-    void (*optional)(Visitor *v, bool *present, const char *name,
-                     Error **errp);
+    /* May be NULL; most useful for input visitors. */
+    void (*optional)(Visitor *v, bool *present, const char *name);
 
     void (*type_uint8)(Visitor *v, uint8_t *obj, const char *name, Error **errp);
     void (*type_uint16)(Visitor *v, uint16_t *obj, const char *name, Error **errp);
include/qapi/visitor.h +8 -2
index 1414de1..9be60d4 100644
@@ -36,8 +36,14 @@ void visit_end_implicit_struct(Visitor *v, Error **errp);
 void visit_start_list(Visitor *v, const char *name, Error **errp);
 GenericList *visit_next_list(Visitor *v, GenericList **list, Error **errp);
 void visit_end_list(Visitor *v, Error **errp);
-void visit_optional(Visitor *v, bool *present, const char *name,
-                    Error **errp);
+
+/**
+ * Check if an optional member @name of an object needs visiting.
+ * For input visitors, set *@present according to whether the
+ * corresponding visit_type_*() needs calling; for other visitors,
+ * leave *@present unchanged.
+ */
+void visit_optional(Visitor *v, bool *present, const char *name);
 
 /**
  * Determine the qtype of the item @name in the current object visit.
qapi/opts-visitor.c +1 -1
index cd10392..ef5fb8b 100644
@@ -488,7 +488,7 @@ opts_type_size(Visitor *v, uint64_t *obj, const char *name, Error **errp)
 
 
 static void
-opts_optional(Visitor *v, bool *present, const char *name, Error **errp)
+opts_optional(Visitor *v, bool *present, const char *name)
 {
     OptsVisitor *ov = DO_UPCAST(OptsVisitor, visitor, v);
 
qapi/qapi-visit-core.c +2 -3
index cee76bc..e07d6f9 100644
@@ -73,11 +73,10 @@ void visit_end_union(Visitor *v, bool data_present, Error **errp)
     }
 }
 
-void visit_optional(Visitor *v, bool *present, const char *name,
-                    Error **errp)
+void visit_optional(Visitor *v, bool *present, const char *name)
 {
     if (v->optional) {
-        v->optional(v, present, name, errp);
+        v->optional(v, present, name);
     }
 }
 
qapi/qmp-input-visitor.c +1 -2
index 26b7414..932b5f3 100644
@@ -303,8 +303,7 @@ static void qmp_input_type_any(Visitor *v, QObject **obj, const char *name,
     *obj = qobj;
 }
 
-static void qmp_input_optional(Visitor *v, bool *present, const char *name,
-                               Error **errp)
+static void qmp_input_optional(Visitor *v, bool *present, const char *name)
 {
     QmpInputVisitor *qiv = to_qiv(v);
     QObject *qobj = qmp_input_get_object(qiv, name, true);
qapi/string-input-visitor.c +1 -2
index bbd6a54..dee780a 100644
@@ -299,8 +299,7 @@ static void parse_type_number(Visitor *v, double *obj, const char *name,
     *obj = val;
 }
 
-static void parse_optional(Visitor *v, bool *present, const char *name,
-                           Error **errp)
+static void parse_optional(Visitor *v, bool *present, const char *name)
 {
     StringInputVisitor *siv = DO_UPCAST(StringInputVisitor, visitor, v);
 
scripts/qapi.py +2 -6
index 362afa1..78b6400 100644
@@ -1680,15 +1680,11 @@ def gen_visit_fields(members, prefix='', need_cast=False, skiperr=False):
     for memb in members:
         if memb.optional:
             ret += mcgen('''
-    visit_optional(v, &%(prefix)shas_%(c_name)s, "%(name)s", %(errp)s);
+    visit_optional(v, &%(prefix)shas_%(c_name)s, "%(name)s");
+    if (%(prefix)shas_%(c_name)s) {
 ''',
                          prefix=prefix, c_name=c_name(memb.name),
                          name=memb.name, errp=errparg)
-            ret += gen_err_check(skiperr=skiperr)
-            ret += mcgen('''
-    if (%(prefix)shas_%(c_name)s) {
-''',
-                         prefix=prefix, c_name=c_name(memb.name))
             push_indent()
 
         # Ugly: sometimes we need to cast away const
Signature · 1 line
2.4.3
#96 lore raw

Re: [Qemu-devel] [PATCH v12 35/36] qapi: Simplify visits of optional fields

Quoted 6 lines · Eric Blake

None of the visitor callbacks would set an error when testing if an optional field was present; make this part of the interface contract by eliminating the errp argument.

The resulting generated code has a nice diff:

Quoted 12 lines
|-    visit_optional(v, &has_fdset_id, "fdset-id", &err);
|-    if (err) {
|-        goto out;
|-    }
|-    if (has_fdset_id) {
|+    visit_optional(v, &has_fdset_id, "fdset-id");
|+    if (has_fdset_id) {
|         visit_type_int(v, &fdset_id, "fdset-id", &err);
|         if (err) {
|             goto out;
|         }
|     }

I think this should be

  |-    visit_optional(v, &has_fdset_id, "fdset-id", &err);
  |-    if (err) {
  |-        goto out;
  |-    }
  |+    visit_optional(v, &has_fdset_id, "fdset-id");
  |     if (has_fdset_id) {
  |         visit_type_int(v, &fdset_id, "fdset-id", &err);
  |         if (err) {
  |             goto out;
  |         }
  |     }
Quoted 2 lines

Patch looks good.

author
#97 lore raw

Re: [Qemu-devel] [PATCH v12 35/36] qapi: Simplify visits of optional fields

Quoted 21 lines · Markus Armbruster
Quoted 6 lines · Eric Blake

None of the visitor callbacks would set an error when testing if an optional field was present; make this part of the interface contract by eliminating the errp argument.

The resulting generated code has a nice diff:

Quoted 12 lines
|-    visit_optional(v, &has_fdset_id, "fdset-id", &err);
|-    if (err) {
|-        goto out;
|-    }
|-    if (has_fdset_id) {
|+    visit_optional(v, &has_fdset_id, "fdset-id");
|+    if (has_fdset_id) {
|         visit_type_int(v, &fdset_id, "fdset-id", &err);
|         if (err) {
|             goto out;
|         }
|     }
Quoted 8 lines

I think this should be

  |-    visit_optional(v, &has_fdset_id, "fdset-id", &err);
  |-    if (err) {
  |-        goto out;
  |-    }
  |+    visit_optional(v, &has_fdset_id, "fdset-id");
  |     if (has_fdset_id) {

Oh, yeah. Serves me right for hand-editing a diff in my commit message.
(Patch applies either diff just fine, it's just that mine was longer
than necessary).

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org
author
#98 lore raw

PATCH v12 36/36 qapi: Shorter visits of optional fields

In reply to Eric Blake

For less code, reflect the determined boolean value of an optional visit back to the caller instead of making the caller read the boolean after the fact.

The resulting generated code has the following diff:

|-    visit_optional(v, &has_fdset_id, "fdset-id");
|-    if (has_fdset_id) {
|+    if (visit_optional(v, &has_fdset_id, "fdset-id")) {
|         visit_type_int(v, &fdset_id, "fdset-id", &err);
|         if (err) {
|             goto out;
|         }
|     }

---
v12: split error removal from 'if' compression
v11 (no v10): no change
v9: no change
v8: no change
v7: rebase to no member.c_name()
v6: rebase onto earlier testsuite and gen_err_check() improvements

---
 include/qapi/visitor.h | 4 ++--
 qapi/qapi-visit-core.c | 3 ++-
 scripts/qapi.py        | 3 +--
 3 files changed, 5 insertions(+), 5 deletions(-)
include/qapi/visitor.h +2 -2
index 9be60d4..a14a16d 100644
@@ -41,9 +41,9 @@ void visit_end_list(Visitor *v, Error **errp);
  * Check if an optional member @name of an object needs visiting.
  * For input visitors, set *@present according to whether the
  * corresponding visit_type_*() needs calling; for other visitors,
- * leave *@present unchanged.
+ * leave *@present unchanged.  Return *@present for convenience.
  */
-void visit_optional(Visitor *v, bool *present, const char *name);
+bool visit_optional(Visitor *v, bool *present, const char *name);
 
 /**
  * Determine the qtype of the item @name in the current object visit.
qapi/qapi-visit-core.c +2 -1
index e07d6f9..6d63e40 100644
@@ -73,11 +73,12 @@ void visit_end_union(Visitor *v, bool data_present, Error **errp)
     }
 }
 
-void visit_optional(Visitor *v, bool *present, const char *name)
+bool visit_optional(Visitor *v, bool *present, const char *name)
 {
     if (v->optional) {
         v->optional(v, present, name);
     }
+    return *present;
 }
 
 void visit_get_next_type(Visitor *v, QType *type, bool promote_int,
scripts/qapi.py +1 -2
index 78b6400..ae70a2d 100644
@@ -1680,8 +1680,7 @@ def gen_visit_fields(members, prefix='', need_cast=False, skiperr=False):
     for memb in members:
         if memb.optional:
             ret += mcgen('''
-    visit_optional(v, &%(prefix)shas_%(c_name)s, "%(name)s");
-    if (%(prefix)shas_%(c_name)s) {
+    if (visit_optional(v, &%(prefix)shas_%(c_name)s, "%(name)s")) {
 ''',
                          prefix=prefix, c_name=c_name(memb.name),
                          name=memb.name, errp=errparg)
Signature · 1 line
2.4.3
#99 lore raw

Re: [Qemu-devel] [PATCH v12 36/36] qapi: Shorter visits of optional fields

Quoted 15 lines · Eric Blake

For less code, reflect the determined boolean value of an optional visit back to the caller instead of making the caller read the boolean after the fact.

The resulting generated code has the following diff:

|-    visit_optional(v, &has_fdset_id, "fdset-id");
|-    if (has_fdset_id) {
|+    if (visit_optional(v, &has_fdset_id, "fdset-id")) {
|         visit_type_int(v, &fdset_id, "fdset-id", &err);
|         if (err) {
|             goto out;
|         }
|     }
Quoted 1 line

Feels like a wash to me, but I'm willing to take it anyway, in recognition of the massive amount of work you've been doing in this area :)

#100 lore raw
In reply to Eric Blake

Feels close. The only issues in need of discussion are in PATCH 27.

I expect the fixup to PATCH 21 to be straightforward. I'd do PATCH 28 a bit different, but it works as is. PATCH 30 could perhaps be simplified slightly. The rest are just commit message polish.

author
#101 lore raw
Quoted 5 lines · Markus Armbruster

Feels close. The only issues in need of discussion are in PATCH 27.

I expect the fixup to PATCH 21 to be straightforward. I'd do PATCH 28 a bit different, but it works as is. PATCH 30 could perhaps be simplified slightly. The rest are just commit message polish.

Fixups proposed to 21, 27, 28, and 30; at this point, I'll be happy to review whatever you post on your staging tree to make sure you caught all the commit message polish you suggested, and we can probably avoid the need for a (lucky?) v13.

Signature · 2 lines
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org