Skip to contentSkip to conversation

block: Try to create well typed json:{} filenames

PATCH Superseded Max Reitz 4 participants 42 messages May 9, 2018 – Jun 6, 2018 Block layer

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

cover block: Try to create well typed json:{} filenames

The “Try” in the subject of this series means more or less the same as the “Try” in the subject line of
http://lists.nongnu.org/archive/html/qemu-block/2018-05/msg00061.html .

The difference is that it’s not so important here, because the worst outcome is that in very narrow edge cases[1] (e.g. you trying to use password-secret for rbd) we fail to generate a properly typed json:{} filename -- but that is no regression over what we currently have. In the vast majority of cases[1], this series will result in proper typing.

Or, well, I should add that I’m not quite sure about all the edge cases[1], solely because the current bdrv_refresh_filename() code is a mess, especially when it comes to gathering the blockdev options. But I do know that after my series to address that (“block: Fix some filename generation issues”), it only comes down to a very small number of edge cases (the only hard edge case is rbd’s =keyvalue-pairs, but that is kaput anyway).[1]

But since this series does not have any hard requirement on that other one, I decided to send it independently.

[1] You might have noticed a number of [1]s above. This is because the above text is as it was in a version of this series with only five patches in it. Then I noticed a not-so-edge case where converting to QAPI broke down: qcow(2) encryption. Since qcow and qcow2 now support both the old AES and luks encryption, and every encryption type may offer its own runtime options, you now have a flat union there (under “encrypt”) which is discriminated based on encrypt.format. Now the qcow2 driver can of course detect the format from the image header, so you don’t really have to specify it. In fact, when your options don’t go through QAPI, you actually don’t have to. So this is exactly the not-so-edge case I was talking about.

QAPI cannot parse encrypt before it knows encrypt.format. So it’s much too late when the qcow2 driver detects encrypt.format in its .bdrv_open() plementation.

I’m more than happy to take alternative suggestions, but this is an issue we really cannot ignore (unlike =keyvalue-pairs), so the two solutions I came up with are:

(1) Say auto-detection was a bad idea and make encrypt.format mandatory, even without QAPI. This is always a possibility, but it does seem kind of cumbersome to users, so I wanted to explore the other option first, not knowing how deep into the rabbit hole that would lead.

(2) Allow flat union discriminators to be optional. Since both AES and luks encryption in practice only allow a key-secret option, we can add a new runtime encryption pseudo-type, which is “from-image” that only allows such a key-secret option. Now, when the user does not specify encrypt.format, the format is assumed to be “from-image” which means that you can specify a key-secret and qcow/qcow2 will figure out what encryption to use.
This allows to retain compatibility with the current interface, but it also means this series exploded, and maybe the idea is universally hated.
One issue I can see with this approach is how it would fare with new encryption types added; specifically, what if we add some encryption type that does not have a key-secret but something else? Do we add that something else to from-image and make everything optional? Or do we require users to explicitly specify the encryption type for such a new type when it is not compatible with from-image? In any case, one thing we have to watch out for from now on is that drivers must not allow options to be optional that are mandatory in the QAPI schema.
More generally, we have to make sure that the driver’s interface on -drive is compatible to QAPI. But I guess everybody knows that, so it’s good to be a bit more specific and say that this doesn’t just mean that options must match. Mandatoriness must match also.

So, OK, obviously I chose solution (2) for now. What this means is that in this series:

- The QAPI code generator is modified to allow optional discriminators for flat unions. In such a case, a default-variant must be supplied to choose when the discriminator value is not present on the wire.
- Accordingly, documentation, tests, and introspection are adjusted.

- This is used to make qcow’s and qcow2’s encrypt.format parameter optional. It now defaults to “from-image” which is a new pseudo-format that allows a key-secret to be given, and otherwise leaves it to the format driver to determine the encryption format.

- qdict_stringify_for_keyval() is added, as already proposed/hinted at in the rbd QAPI discussion. This includes movement of many block-specific QDict function declarations into an own header, as suggested by Markus.

- json:{} filenames are attempted to be typed correctly when they are generated, by running bs->full_open_options through a healthy mix of qdict_flatten(), qdict_stringify_for_keyval(), qdict_crumple(), the keyval input visitor for BlockdevOptions, and the output visitor. This may not always work but I hope it usually will. Fingers crossed. (At least it won’t make things worse.)

- Tests, tests, tests.

(Yes, I know that “In this series tests, tests, tests.” is not a
sentence.)

Max Reitz (13):
qapi: Add default-variant for flat unions docs/qapi: Document optional discriminators
tests: Add QAPI optional discriminator tests
qapi: Formalize qcow2 encryption probing
qapi: Formalize qcow encryption probing
block: Add block-specific QDict header
qdict: Add qdict_stringify_for_keyval()
tests: Add qdict_stringify_for_keyval() test
qdict: Make qdict_flatten() shallow-clone-friendly
tests: Add QDict clone-flatten test
block: Try to create well typed json:{} filenames
iotests: Test internal option typing
iotests: qcow2's encrypt.format is now optional

 docs/devel/qapi-code-gen.txt                       | 21 +++++-
 tests/Makefile.include                             |  3 +
 qapi/block-core.json                               | 72 ++++++++++++++++--
 qapi/introspect.json                               |  8 ++
 ...ion-optional-discriminator-invalid-default.json | 12 +++
 ...at-union-optional-discriminator-no-default.json | 11 +++
 .../flat-union-optional-discriminator.json         |  4 +-
 .../flat-union-superfluous-default-variant.json    | 11 +++
 include/block/qdict.h                              | 37 +++++++++
 include/qapi/qmp/qdict.h                           | 17 -----
 block.c                                            | 70 ++++++++++++++++-
 block/gluster.c                                    |  1 +
 block/iscsi.c                                      |  1 +
 block/nbd.c                                        |  1 +
 block/nfs.c                                        |  1 +
 block/parallels.c                                  |  1 +
 block/qcow.c                                       |  1 +
 block/qcow2.c                                      |  1 +
 block/qed.c                                        |  1 +
 block/quorum.c                                     |  1 +
 block/rbd.c                                        |  1 +
 block/sheepdog.c                                   |  1 +
 block/snapshot.c                                   |  1 +
 block/ssh.c                                        |  1 +
 block/vhdx.c                                       |  1 +
 block/vpc.c                                        |  1 +
 block/vvfat.c                                      |  1 +
 block/vxhs.c                                       |  1 +
 blockdev.c                                         |  1 +
 qobject/qdict.c                                    | 81 ++++++++++++++++++--
 tests/check-qdict.c                                | 88 ++++++++++++++++++++++
 tests/check-qobject.c                              |  1 +
 tests/test-replication.c                           |  1 +
 util/qemu-config.c                                 |  1 +
 scripts/qapi/common.py                             | 57 +++++++++++---
 scripts/qapi/doc.py                                |  8 +-
 scripts/qapi/introspect.py                         | 10 ++-
 scripts/qapi/visit.py                              | 33 +++++++-
 ...nion-optional-discriminator-invalid-default.err |  1 +
 ...ion-optional-discriminator-invalid-default.exit |  1 +
 ...nion-optional-discriminator-invalid-default.out |  0
 ...lat-union-optional-discriminator-no-default.err |  1 +
 ...at-union-optional-discriminator-no-default.exit |  1 +
 ...lat-union-optional-discriminator-no-default.out |  0
 .../flat-union-optional-discriminator.err          |  1 -
 .../flat-union-optional-discriminator.exit         |  2 +-
 .../flat-union-optional-discriminator.out          | 15 ++++
 .../flat-union-superfluous-default-variant.err     |  1 +
 .../flat-union-superfluous-default-variant.exit    |  1 +
 .../flat-union-superfluous-default-variant.out     |  0
 tests/qapi-schema/test-qapi.py                     |  2 +
 tests/qemu-iotests/059.out                         |  2 +-
 tests/qemu-iotests/087                             |  2 -
 tests/qemu-iotests/089                             | 25 ++++++
 tests/qemu-iotests/089.out                         |  9 +++
 tests/qemu-iotests/099.out                         |  4 +-
 tests/qemu-iotests/110.out                         |  2 +-
 tests/qemu-iotests/198.out                         |  4 +-
 tests/qemu-iotests/207.out                         | 10 +--
 59 files changed, 580 insertions(+), 68 deletions(-)
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.json
 create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.json
 create mode 100644 include/block/qdict.h
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.out
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.err
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.exit
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.out
 create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.err
 create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.exit
 create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.out
Signature · 1 line
2.14.3
author
#1 lore raw

PATCH 1/13 qapi: Add default-variant for flat unions

This patch allows specifying a discriminator that is an optional member of the base struct. In such a case, a default value must be provided that is used when no value is given.

---
 qapi/introspect.json           |  8 ++++++
 scripts/qapi/common.py         | 57 ++++++++++++++++++++++++++++++++++--------
 scripts/qapi/doc.py            |  8 ++++--
 scripts/qapi/introspect.py     | 10 +++++---
 scripts/qapi/visit.py          | 33 ++++++++++++++++++++++--
 tests/qapi-schema/test-qapi.py |  2 ++
 6 files changed, 101 insertions(+), 17 deletions(-)
qapi/introspect.json +8 -0
index c7f67b7d78..2d7b1e3745 100644
@@ -168,6 +168,13 @@
 # @tag: the name of the member serving as type tag.
 #       An element of @members with this name must exist.
 #
+# @default-variant: if the @tag element of @members is optional, this
+#                   is the default value for choosing a variant.  Its
+#                   value must be a valid value for @tag.
+#                   Present exactly when @tag is present and the
+#                   associated element of @members is optional.
+#                   (Since: 2.13)
+#
 # @variants: variant members, i.e. additional members that
 #            depend on the type tag's value.  Present exactly when
 #            @tag is present.  The variants are in no particular order,
@@ -181,6 +188,7 @@
 { 'struct': 'SchemaInfoObject',
   'data': { 'members': [ 'SchemaInfoObjectMember' ],
             '*tag': 'str',
+            '*default-variant': 'str',
             '*variants': [ 'SchemaInfoObjectVariant' ] } }
 
 ##
scripts/qapi/common.py +47 -10

118 lines not on this page · raw message

scripts/qapi/doc.py +6 -2
index 9b312b2c51..91204dc4c6 100644
@@ -160,8 +160,12 @@ def texi_members(doc, what, base, variants, member_func):
         items += '@item The members of @code{%s}\n' % base.doc_type()
     if variants:
         for v in variants.variants:
-            when = ' when @code{%s} is @t{"%s"}' % (
-                variants.tag_member.name, v.name)
+            if v.name == variants.default_tag_value:
+                when = ' when @code{%s} is @t{"%s"} or not given' % (
+                    variants.tag_member.name, v.name)
+            else:
+                when = ' when @code{%s} is @t{"%s"}' % (
+                    variants.tag_member.name, v.name)
             if v.type.is_implicit():
                 assert not v.type.base and not v.type.variants
                 for m in v.type.local_members:
scripts/qapi/introspect.py +7 -3
index f9e67e8227..2d1d4e320a 100644
@@ -142,9 +142,12 @@ const QLitObject %(c_name)s = %(c_string)s;
             ret['default'] = None
         return ret
 
-    def _gen_variants(self, tag_name, variants):
-        return {'tag': tag_name,
-                'variants': [self._gen_variant(v) for v in variants]}
+    def _gen_variants(self, tag_name, default_variant, variants):
+        ret = {'tag': tag_name,
+               'variants': [self._gen_variant(v) for v in variants]}
+        if default_variant:
+            ret['default-variant'] = default_variant
+        return ret
 
     def _gen_variant(self, variant):
         return {'case': variant.name, 'type': self._use_type(variant.type)}
@@ -163,6 +166,7 @@ const QLitObject %(c_name)s = %(c_string)s;
         obj = {'members': [self._gen_member(m) for m in members]}
         if variants:
             obj.update(self._gen_variants(variants.tag_member.name,
+                                          variants.default_tag_value,
                                           variants.variants))
         self._gen_qlit(name, 'object', obj)
 
scripts/qapi/visit.py +31 -2
index 5d72d8936c..ecffc46bd3 100644
@@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members, variants):
 void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
 {
     Error *err = NULL;
-
 ''',
                 c_name=c_name(name))
 
+    if variants:
+        ret += mcgen('''
+    %(c_type)s %(c_name)s;
+''',
+                     c_type=variants.tag_member.type.c_name(),
+                     c_name=c_name(variants.tag_member.name))
+
+    ret += mcgen('''
+
+''')
+
     if base:
         ret += mcgen('''
     visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, &err);
@@ -75,8 +85,27 @@ void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
 ''')
 
     if variants:
+        if variants.default_tag_value is None:
+            ret += mcgen('''
+    %(c_name)s = obj->%(c_name)s;
+''',
+                         c_name=c_name(variants.tag_member.name))
+        else:
+            ret += mcgen('''
+    if (obj->has_%(c_name)s) {
+        %(c_name)s = obj->%(c_name)s;
+    } else {
+        %(c_name)s = %(enum_const)s;
+    }
+''',
+                         c_name=c_name(variants.tag_member.name),
+                         enum_const=c_enum_const(
+                             variants.tag_member.type.name,
+                             variants.default_tag_value,
+                             variants.tag_member.type.prefix))
+
         ret += mcgen('''
-    switch (obj->%(c_name)s) {
+    switch (%(c_name)s) {
 ''',
                      c_name=c_name(variants.tag_member.name))
 
tests/qapi-schema/test-qapi.py +2 -0
index c1a144ba29..f2a072b92e 100644
@@ -56,6 +56,8 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
     def _print_variants(variants):
         if variants:
             print('    tag %s' % variants.tag_member.name)
+            if variants.default_tag_value:
+                print('    default variant: %s' % variants.default_tag_value)
             for v in variants.variants:
                 print('    case %s: %s' % (v.name, v.type.name))
 
Signature · 1 line
2.14.3
Reviewed-by on v1 1/13
#2 lore raw

Re: [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions

Quoted 5 lines · Max Reitz

This patch allows specifying a discriminator that is an optional member of the base struct. In such a case, a default value must be provided that is used when no value is given.

Quoted 8 lines
---
  qapi/introspect.json           |  8 ++++++
  scripts/qapi/common.py         | 57 ++++++++++++++++++++++++++++++++++--------
  scripts/qapi/doc.py            |  8 ++++--
  scripts/qapi/introspect.py     | 10 +++++---
  scripts/qapi/visit.py          | 33 ++++++++++++++++++++++--
  tests/qapi-schema/test-qapi.py |  2 ++
  6 files changed, 101 insertions(+), 17 deletions(-)

I've been threatening that we might need this for some time, so I'm glad to see it being implemented. We'll see if the tests in 2 and 3 cover the code added here.

qapi/introspect.json line 173

Quoted 12 lines
qapi/introspect.json +3 -0
index c7f67b7d78..2d7b1e3745 100644
@@ -168,6 +168,13 @@
  # @tag: the name of the member serving as type tag.
  #       An element of @members with this name must exist.
  #
+# @default-variant: if the @tag element of @members is optional, this
+#                   is the default value for choosing a variant.  Its
+#                   value must be a valid value for @tag.

Perhaps s/must/will/ as this struct is used for output (and therefore we always meet the condition, rather than the user having to do something correctly).

Nice that you remembered introspection.

Quoted 37 lines
qapi/introspect.json +5 -0
+#                   Present exactly when @tag is present and the
+#                   associated element of @members is optional.
+#                   (Since: 2.13)
+#
  # @variants: variant members, i.e. additional members that
  #            depend on the type tag's value.  Present exactly when
  #            @tag is present.  The variants are in no particular order,
@@ -181,6 +188,7 @@
  { 'struct': 'SchemaInfoObject',
    'data': { 'members': [ 'SchemaInfoObjectMember' ],
              '*tag': 'str',
+            '*default-variant': 'str',
              '*variants': [ 'SchemaInfoObjectVariant' ] } }
 
  ##
scripts/qapi/common.py +3 -2
index a032cec375..fbf0244f73 100644
@@ -721,6 +721,7 @@ def check_union(expr, info):
      name = expr['union']
      base = expr.get('base')
      discriminator = expr.get('discriminator')
+    default_variant = expr.get('default-variant')
      members = expr['data']
 
      # Two types of unions, determined by discriminator.
@@ -745,16 +746,37 @@ def check_union(expr, info):
          base_members = find_base_members(base)
          assert base_members is not None
 
-        # The value of member 'discriminator' must name a non-optional
-        # member of the base struct.
+        # The value of member 'discriminator' must name a member of
+        # the base struct.
          check_name(info, "Discriminator of flat union '%s'" % name,
                     discriminator)

scripts/qapi/common.py def check_union(expr, info): · line 756

Quoted 10 lines
scripts/qapi/common.py +4 -6
-        discriminator_type = base_members.get(discriminator)
-        if not discriminator_type:
-            raise QAPISemError(info,
-                               "Discriminator '%s' is not a member of base "
-                               "struct '%s'"
-                               % (discriminator, base))
+        if default_variant is None:
+            discriminator_type = base_members.get(discriminator)
+            if not discriminator_type:
+                if base_members.get('*' + discriminator) is None:

Unrelated to your patch, but this reminds me - we already had a question on list about whether we should allow ANY member of the base struct, rather than the members directly declared in the struct. (The use case was that if you have a grandparent struct, then an intermediate base struct, the code as written does not permit the discriminator to come from the grandparent struct, which is awkward - but if I also recall correctly, the question came up in relation to the query-cpus-fast command, where we ended up not refactoring things after all due to deprecation of query-cpus).

Quoted 13 lines
scripts/qapi/common.py +13 -0
+                    raise QAPISemError(info,
+                                       "Discriminator '%s' is not a member of "
+                                       "base struct '%s'"
+                                       % (discriminator, base))
+                else:
+                    raise QAPISemError(info,
+                                       "Default variant must be specified for "
+                                       "optional discriminator '%s'"
+                                       % discriminator)
+        else:
+            discriminator_type = base_members.get('*' + discriminator)
+            if not discriminator_type:
+                if base_members.get(discriminator) is None:

scripts/qapi/common.py def check_union(expr, info): · line 779

Quoted 10 lines
scripts/qapi/common.py +10 -0
+                    raise QAPISemError(info,
+                                       "Discriminator '%s' is not a member of "
+                                       "base struct '%s'"
+                                       % (discriminator, base))
+                else:
+                    raise QAPISemError(info,
+                                       "Must not specify a default variant for "
+                                       "non-optional discriminator '%s'"
+                                       % discriminator)
+

Looks like you've got good error messages for all the cases of enforcing the rule that default-variant and optional member must be used together.

Quoted 31 lines
scripts/qapi/common.py +10 -1
          enum_define = enum_types.get(discriminator_type)
          allow_metas = ['struct']
          # Do not allow string discriminator
@@ -763,6 +785,15 @@ def check_union(expr, info):
                                 "Discriminator '%s' must be of enumeration "
                                 "type" % discriminator)
 
+        if default_variant is not None:
+            # Must be a value of the enumeration
+            if default_variant not in enum_define['data']:
+                raise QAPISemError(info,
+                                   "Default variant '%s' of flat union '%s' is "
+                                   "not part of '%s'"
+                                   % (default_variant, name,
+                                      discriminator_type))
+
      # Check every branch; don't allow an empty union
      if len(members) == 0:
          raise QAPISemError(info, "Union '%s' cannot have empty 'data'" % name)
@@ -909,7 +940,7 @@ def check_exprs(exprs):
          elif 'union' in expr:
              meta = 'union'
              check_keys(expr_elem, 'union', ['data'],
-                       ['base', 'discriminator'])
+                       ['base', 'discriminator', 'default-variant'])
              union_types[expr[meta]] = expr
          elif 'alternate' in expr:
              meta = 'alternate'
@@ -1335,12 +1366,14 @@ class QAPISchemaObjectTypeMember(QAPISchemaMember):
 
 

scripts/qapi/common.py class QAPISchemaObjectTypeMember(QAPISchemaMember): · line 1376

Quoted 10 lines
scripts/qapi/common.py +3 -1
  class QAPISchemaObjectTypeVariants(object):
-    def __init__(self, tag_name, tag_member, variants):
+    def __init__(self, tag_name, tag_member, default_tag_value, variants):
          # Flat unions pass tag_name but not tag_member.
          # Simple unions and alternates pass tag_member but not tag_name.
          # After check(), tag_member is always set, and tag_name remains
          # a reliable witness of being used by a flat union.
          assert bool(tag_member) != bool(tag_name)
+        # default_tag_value is only passed for flat unions.
+        assert bool(tag_name) or not bool(default_tag_value)

Looks correct.

Quoted 40 lines
scripts/qapi/common.py +4 -0
          assert (isinstance(tag_name, str) or
                  isinstance(tag_member, QAPISchemaObjectTypeMember))
          assert len(variants) > 0
@@ -1348,6 +1381,7 @@ class QAPISchemaObjectTypeVariants(object):
              assert isinstance(v, QAPISchemaObjectTypeVariant)
          self._tag_name = tag_name
          self.tag_member = tag_member
+        self.default_tag_value = default_tag_value
          self.variants = variants
 
      def set_owner(self, name):
@@ -1637,6 +1671,7 @@ class QAPISchema(object):
          data = expr['data']
          base = expr.get('base')
          tag_name = expr.get('discriminator')
+        default_tag_value = expr.get('default-variant')
          tag_member = None
          if isinstance(base, dict):
              base = (self._make_implicit_object_type(
@@ -1656,6 +1691,7 @@ class QAPISchema(object):
              QAPISchemaObjectType(name, info, doc, base, members,
                                   QAPISchemaObjectTypeVariants(tag_name,
                                                                tag_member,
+                                                              default_tag_value,
                                                                variants)))
 
      def _def_alternate_type(self, expr, info, doc):
@@ -1668,6 +1704,7 @@ class QAPISchema(object):
              QAPISchemaAlternateType(name, info, doc,
                                      QAPISchemaObjectTypeVariants(None,
                                                                   tag_member,
+                                                                 None,
                                                                   variants)))
 
      def _def_command(self, expr, info, doc):
scripts/qapi/doc.py +0 -0
index 9b312b2c51..91204dc4c6 100644

scripts/qapi/doc.py def texi_members(doc, what, base, variants, member_func): · line 165

Quoted 8 lines
scripts/qapi/doc.py +3 -2
@@ -160,8 +160,12 @@ def texi_members(doc, what, base, variants, member_func):
          items += '@item The members of @code{%s}\n' % base.doc_type()
      if variants:
          for v in variants.variants:
-            when = ' when @code{%s} is @t{"%s"}' % (
-                variants.tag_member.name, v.name)
+            if v.name == variants.default_tag_value:
+                when = ' when @code{%s} is @t{"%s"} or not given' % (
+                    variants.tag_member.name, v.name)

Nice, you got the generated docs as well as introspection.

Quoted 47 lines
scripts/qapi/doc.py +3 -0
+            else:
+                when = ' when @code{%s} is @t{"%s"}' % (
+                    variants.tag_member.name, v.name)
              if v.type.is_implicit():
                  assert not v.type.base and not v.type.variants
                  for m in v.type.local_members:
scripts/qapi/introspect.py +7 -3
index f9e67e8227..2d1d4e320a 100644
@@ -142,9 +142,12 @@ const QLitObject %(c_name)s = %(c_string)s;
              ret['default'] = None
          return ret
 
-    def _gen_variants(self, tag_name, variants):
-        return {'tag': tag_name,
-                'variants': [self._gen_variant(v) for v in variants]}
+    def _gen_variants(self, tag_name, default_variant, variants):
+        ret = {'tag': tag_name,
+               'variants': [self._gen_variant(v) for v in variants]}
+        if default_variant:
+            ret['default-variant'] = default_variant
+        return ret
 
      def _gen_variant(self, variant):
          return {'case': variant.name, 'type': self._use_type(variant.type)}
@@ -163,6 +166,7 @@ const QLitObject %(c_name)s = %(c_string)s;
          obj = {'members': [self._gen_member(m) for m in members]}
          if variants:
              obj.update(self._gen_variants(variants.tag_member.name,
+                                          variants.default_tag_value,
                                            variants.variants))
          self._gen_qlit(name, 'object', obj)
 
scripts/qapi/visit.py +1 -1
index 5d72d8936c..ecffc46bd3 100644
@@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members, variants):
  void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
  {
      Error *err = NULL;
-
  ''',
                  c_name=c_name(name))
 
+    if variants:

scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 56

Quoted 10 lines
scripts/qapi/visit.py +10 -0
+        ret += mcgen('''
+    %(c_type)s %(c_name)s;
+''',
+                     c_type=variants.tag_member.type.c_name(),
+                     c_name=c_name(variants.tag_member.name))
+
+    ret += mcgen('''
+
+''')
+

Creating a temp variable makes it easier to handle the default...

Quoted 19 lines
scripts/qapi/visit.py +12 -0
      if base:
          ret += mcgen('''
      visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, &err);
@@ -75,8 +85,27 @@ void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
  ''')
 
      if variants:
+        if variants.default_tag_value is None:
+            ret += mcgen('''
+    %(c_name)s = obj->%(c_name)s;
+''',
+                         c_name=c_name(variants.tag_member.name))
+        else:
+            ret += mcgen('''
+    if (obj->has_%(c_name)s) {
+        %(c_name)s = obj->%(c_name)s;
+    } else {
+        %(c_name)s = %(enum_const)s;
+    }

scripts/qapi/visit.py void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp) · line 108

Quoted 10 lines
scripts/qapi/visit.py +8 -1
+''',
+                         c_name=c_name(variants.tag_member.name),
+                         enum_const=c_enum_const(
+                             variants.tag_member.type.name,
+                             variants.default_tag_value,
+                             variants.tag_member.type.prefix))
+
          ret += mcgen('''
-    switch (obj->%(c_name)s) {
+    switch (%(c_name)s) {

...compared to the old code that just inlined the one thing that used to be assigned to what is now the temporary variable.

It might be possible to inline things so that the generated code reads either:

switch (obj->discriminator) {
switch (!obj->has_discriminator ? DEFAULT : obj->discriminator) {

but I don't think it's worth the effort; and the temporary variable is fine even though it makes the generated file bigger.

Quoted 9 lines
scripts/qapi/visit.py +0 -0
  ''',
                       c_name=c_name(variants.tag_member.name))
 
tests/qapi-schema/test-qapi.py +0 -0
index c1a144ba29..f2a072b92e 100644

tests/qapi-schema/test-qapi.py class QAPISchemaTestVisitor(QAPISchemaVisitor): · line 63

Quoted 8 lines
tests/qapi-schema/test-qapi.py +2 -0
@@ -56,6 +56,8 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
      def _print_variants(variants):
          if variants:
              print('    tag %s' % variants.tag_member.name)
+            if variants.default_tag_value:
+                print('    default variant: %s' % variants.default_tag_value)
              for v in variants.variants:
                  print('    case %s: %s' % (v.name, v.type.name))
 

Looks good!

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#3 lore raw

Re: [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions

On 05/10/2018 08:12 AM, Eric Blake wrote:

Oh, I just had a thought:

Quoted 15 lines
Quoted 7 lines

+++ b/scripts/qapi/visit.py

(quoted diff) +0 -0
@@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members, 
 

      if variants:
+        if variants.default_tag_value is None:
+            ret += mcgen('''
+    %(c_name)s = obj->%(c_name)s;

Quoted 8 lines

+''',
+                         c_name=c_name(variants.tag_member.name)) +        else:
+            ret += mcgen('''
+    if (obj->has_%(c_name)s) {
+        %(c_name)s = obj->%(c_name)s;
+    } else {
+        %(c_name)s = %(enum_const)s;

In this branch of code, is it worth also generating:

%has_(c_name)s = true;

That way, the rest of the C code does not have to check has_discriminator, because the process of assigning the default will ensure that has_discriminator is always true later on. It does have the effect that output would never omit the discriminator - but that might be a good thing: if we ever have an output union that used to have a mandatory discriminator and want to now make it optional, we don't want to break older clients that expected the discriminator to be present. It does obscure whether input relied on the default, but I don't think that hurts.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#4 lore raw

Re: [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions

Quoted 20 lines · Eric Blake

On 05/10/2018 08:12 AM, Eric Blake wrote:

Oh, I just had a thought:

Quoted 15 lines
Quoted 7 lines

+++ b/scripts/qapi/visit.py

(quoted diff) +0 -0
@@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members, 
 

      if variants:
+        if variants.default_tag_value is None:
+            ret += mcgen('''
+    %(c_name)s = obj->%(c_name)s;

Quoted 8 lines

+''',
+                         c_name=c_name(variants.tag_member.name)) +        else:
+            ret += mcgen('''
+    if (obj->has_%(c_name)s) {
+        %(c_name)s = obj->%(c_name)s;
+    } else {
+        %(c_name)s = %(enum_const)s;

Quoted 3 lines

In this branch of code, is it worth also generating:

%has_(c_name)s = true;

You mean obj->has_%(c_name)s, and then also set obj->%(c_name)s?

Quoted 9 lines

That way, the rest of the C code does not have to check has_discriminator, because the process of assigning the default will ensure that has_discriminator is always true later on.  It does have the effect that output would never omit the discriminator - but that might be a good thing: if we ever have an output union that used to have a mandatory discriminator and want to now make it optional, we don't want to break older clients that expected the discriminator to be present. It does obscure whether input relied on the default, but I don't think that hurts.

Also, it would make code a bit simpler because it can cover the !has_X case under X == default_X.

But OTOH, you could make that case for any optional value -- except where "is missing" has special value.

And that's the tricky part: I think it's hard to say that a missing discriminator never has special meaning. We only need the default-variant so we know which variant of the union to pick; but we don't know that the fact that the discriminator value was missing does not have special meaning.

Of course, we can simply foreclose that by setting it here.

I don't have money in this game, so I suppose it's yours and Markus's call. :-)

Max

#5 lore raw

Re: [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions

Quoted 21 lines · Max Reitz
Quoted 20 lines · Eric Blake

On 05/10/2018 08:12 AM, Eric Blake wrote:

Oh, I just had a thought:

Quoted 15 lines
Quoted 15 lines

+++ b/scripts/qapi/visit.py

scripts/qapi/visit.py +11 -0
@@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members,
 
       if variants:
+        if variants.default_tag_value is None:
+            ret += mcgen('''
+    %(c_name)s = obj->%(c_name)s;
+''',
+                         c_name=c_name(variants.tag_member.name))
+        else:
+            ret += mcgen('''
+    if (obj->has_%(c_name)s) {
+        %(c_name)s = obj->%(c_name)s;
+    } else {
+        %(c_name)s = %(enum_const)s;
Quoted 5 lines
Quoted 3 lines

In this branch of code, is it worth also generating:

%has_(c_name)s = true;

You mean obj->has_%(c_name)s, and then also set obj->%(c_name)s?

Umm, yeah ;)

In fact, I think it is as simple as:

if variants:

    if variants.default_tag_value:
        ret += mcgen('''
if (!obj->has_%(c_name)s) {
    obj->has_%(c_name)s = true;
    obj->%(c_name)s = %(enum_const)s;
}
''')
    ret += mcgen('''

switch (obj->%(c_name)s) {
...

and you are back to not needing a temporary variable.

Quoted 11 lines
Quoted 10 lines

That way, the rest of the C code does not have to check has_discriminator, because the process of assigning the default will ensure that has_discriminator is always true later on.  It does have the effect that output would never omit the discriminator - but that might be a good thing: if we ever have an output union that used to have a mandatory discriminator and want to now make it optional, we don't want to break older clients that expected the discriminator to be present. It does obscure whether input relied on the default, but I don't think that hurts.

Quoted 5 lines

Also, it would make code a bit simpler because it can cover the !has_X case under X == default_X.

But OTOH, you could make that case for any optional value -- except where "is missing" has special value.

My preference - special-casing "is missing" is prone to abuse. I don't want to support that if we can at all avoid it. The sane semantics is that the default is populated as soon as we detect that something is missing, and whether the user relies on the default by leaving the discriminator absent, or explicitly supplies the discriminator set to the default, the behavior should always be the same.

Quoted 3 lines

And that's the tricky part: I think it's hard to say that a missing discriminator never has special meaning.

It won't, if we decide right now that we don't want to let it :)

Quoted 6 lines

We only need the
default-variant so we know which variant of the union to pick; but we don't know that the fact that the discriminator value was missing does not have special meaning.

Of course, we can simply foreclose that by setting it here.

And that's the way I'm leaning.

Quoted 3 lines

I don't have money in this game, so I suppose it's yours and Markus's call. :-)

Markus, what's your preference?

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#6 lore raw

Re: [Qemu-devel] [PATCH 01/13] qapi: Add default-variant for flat unions

In reply to Eric Blake
Quoted 21 lines · Eric Blake
Quoted 13 lines · Max Reitz

This patch allows specifying a discriminator that is an optional member of the base struct.  In such a case, a default value must be provided that is used when no value is given.

---
  qapi/introspect.json           |  8 ++++++ scripts/qapi/common.py         | 57
++++++++++++++++++++++++++++++++++--------
  scripts/qapi/doc.py            |  8 ++++-- scripts/qapi/introspect.py     | 10 +++++--- scripts/qapi/visit.py          | 33 ++++++++++++++++++++++-- tests/qapi-schema/test-qapi.py |  2 ++

Quoted 1 line
  6 files changed, 101 insertions(+), 17 deletions(-)

I've been threatening that we might need this for some time, so I'm glad to see it being implemented.  We'll see if the tests in 2 and 3 cover the code added here.

Quoted 1 line
Quoted 15 lines

qapi/introspect.json line 173

Quoted 11 lines
qapi/introspect.json +3 -0
index c7f67b7d78..2d7b1e3745 100644
@@ -168,6 +168,13 @@
   # @tag: the name of the member serving as type tag.
   #       An element of @members with this name must exist.
   #
+# @default-variant: if the @tag element of @members is optional, this
+#                   is the default value for choosing a variant.  Its
+#                   value must be a valid value for @tag.

Perhaps s/must/will/ as this struct is used for output (and therefore we always meet the condition, rather than the user having to do something correctly).

I mostly copied from the other descriptions which seemed to prefer "must", but I'm happy with either.

Quoted 1 line

Nice that you remembered introspection.

I didn't, because I had no idea how introspection works exactly before this series. But one of the tests broke, thus telling me I might have forgotten something. :-)

Quoted 7 lines
Quoted 7 lines
qapi/introspect.json +4 -0
+#                   Present exactly when @tag is present and the
+#                   associated element of @members is optional.
+#                   (Since: 2.13)
+#
   # @variants: variant members, i.e. additional members that
   #            depend on the type tag's value.  Present exactly when
   #            @tag is present.  The variants are in no particular order,

[...]

Quoted 60 lines
Quoted 19 lines

  diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py index 5d72d8936c..ecffc46bd3 100644

scripts/qapi/visit.py +0 -0
@@ -40,10 +40,20 @@ def gen_visit_object_members(name, base, members,

variants):
  void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s *obj, Error **errp)
  {
      Error *err = NULL;
-
  ''',
                  c_name=c_name(name))
  +    if variants:
+        ret += mcgen('''
+    %(c_type)s %(c_name)s;
+''',
+                     c_type=variants.tag_member.type.c_name(), +                     c_name=c_name(variants.tag_member.name))

scripts/qapi/visit.py def gen_visit_object_members(name, base, members, variants): · line 56

Quoted 5 lines
scripts/qapi/visit.py +5 -0
+
+    ret += mcgen('''
+
+''')
+

Creating a temp variable makes it easier to handle the default...

Quoted 21 lines

      if base:
          ret += mcgen(''' visit_type_%(c_type)s_members(v, (%(c_type)s *)obj, &err);

scripts/qapi/visit.py +0 -0
@@ -75,8 +85,27 @@ void visit_type_%(c_name)s_members(Visitor *v,
 %(c_name)s *obj, Error **errp)
   ''')

        if variants:
+        if variants.default_tag_value is None:
+            ret += mcgen('''
+    %(c_name)s = obj->%(c_name)s;
+''',
+                         c_name=c_name(variants.tag_member.name)) +        else:
+            ret += mcgen('''
+    if (obj->has_%(c_name)s) {
+        %(c_name)s = obj->%(c_name)s;
+    } else {
+        %(c_name)s = %(enum_const)s;
+    }
+''',
+                         c_name=c_name(variants.tag_member.name),

Quoted 8 lines

+                         enum_const=c_enum_const(
+                             variants.tag_member.type.name, +                             variants.default_tag_value, +                             variants.tag_member.type.prefix)) +
          ret += mcgen('''
-    switch (obj->%(c_name)s) {
+    switch (%(c_name)s) {

...compared to the old code that just inlined the one thing that used to be assigned to what is now the temporary variable.

Quoted 8 lines

It might be possible to inline things so that the generated code reads either:

switch (obj->discriminator) {
switch (!obj->has_discriminator ? DEFAULT : obj->discriminator) {

but I don't think it's worth the effort; and the temporary variable is fine even though it makes the generated file bigger.

I don't really mind either way, but depending on the default value and the discriminator name, using ?: may lead to a rather long line.

Quoted 19 lines
Quoted 9 lines

  ''',
                       c_name=c_name(variants.tag_member.name)) diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index c1a144ba29..f2a072b92e 100644

tests/qapi-schema/test-qapi.py +0 -0
@@ -56,6 +56,8 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):

      def _print_variants(variants):

Quoted 8 lines

          if variants:
              print('    tag %s' % variants.tag_member.name) +            if variants.default_tag_value:
+                print('    default variant: %s' % variants.default_tag_value)
              for v in variants.variants:
                  print('    case %s: %s' % (v.name, v.type.name))

Looks good!

Quoted 1 line

Phew. :-)

Thanks!

Max

author
#7 lore raw

PATCH 2/13 docs/qapi: Document optional discriminators

In reply to Max Reitz
---
 docs/devel/qapi-code-gen.txt | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)
docs/devel/qapi-code-gen.txt +19 -2
index b9b6eabd08..35b9f0d99d 100644
@@ -472,8 +472,8 @@ All branches of the union must be complex types, and the top-level
 members of the union dictionary on the wire will be combination of
 members from both the base type and the appropriate branch type (when
 merging two dictionaries, there must be no keys in common).  The
-'discriminator' member must be the name of a non-optional enum-typed
-member of the base struct.
+'discriminator' member must be the name of an enum-typed member of the
+base struct.
 
 The following example enhances the above simple union example by
 adding an optional common member 'read-only', renaming the
@@ -502,6 +502,23 @@ the enum).  In the resulting generated C data types, a flat union is
 represented as a struct with the base members included directly, and
 then a union of structures for each branch of the struct.
 
+If the discriminator points to an optional member of the base struct,
+its default value must be specified as a 'default-variant'.  In the
+following example, the above BlockDriver struct is changed so it
+defaults to the 'file' driver if that field is omitted on the wire:
+
+ { 'union': 'BlockdevOptions',
+   'base': { '*driver': 'BlockdevDriver', '*read-only': 'bool' },
+   'discriminator': 'driver',
+   'default-variant': 'file',
+   'data': { 'file': 'BlockdevOptionsFile',
+             'qcow2': 'BlockdevOptionsQcow2' } }
+
+Now the 'file' JSON object can be abbreviated to:
+
+ { "read-only": "true",
+   "filename": "/some/place/my-image" }
+
 A simple union can always be re-written as a flat union where the base
 class has a single member named 'type', and where each branch of the
 union has a struct with a single member named 'data'.  That is,
Signature · 1 line
2.14.3
Reviewed-by on v1 2/13
#8 lore raw

Re: [Qemu-devel] [PATCH 02/13] docs/qapi: Document optional discriminators

Quoted 5 lines · Max Reitz
---
  docs/devel/qapi-code-gen.txt | 21 +++++++++++++++++++--
  1 file changed, 19 insertions(+), 2 deletions(-)

I might have squashed this in with patch 1, but I can live with it separate.

Quoted 11 lines
docs/devel/qapi-code-gen.txt +7 -0
@@ -502,6 +502,23 @@ the enum).  In the resulting generated C data types, a flat union is
  represented as a struct with the base members included directly, and
  then a union of structures for each branch of the struct.
 
+If the discriminator points to an optional member of the base struct,
+its default value must be specified as a 'default-variant'.  In the
+following example, the above BlockDriver struct is changed so it
+defaults to the 'file' driver if that field is omitted on the wire:
+
+ { 'union': 'BlockdevOptions',
+   'base': { '*driver': 'BlockdevDriver', '*read-only': 'bool' },

docs/devel/qapi-code-gen.txt the enum). In the resulting generated C data types, a flat union is · line 521

Quoted 10 lines
docs/devel/qapi-code-gen.txt +10 -0
+   'discriminator': 'driver',
+   'default-variant': 'file',
+   'data': { 'file': 'BlockdevOptionsFile',
+             'qcow2': 'BlockdevOptionsQcow2' } }
+
+Now the 'file' JSON object can be abbreviated to:
+
+ { "read-only": "true",
+   "filename": "/some/place/my-image" }
+
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#9 lore raw

PATCH 3/13 tests: Add QAPI optional discriminator tests

In reply to Max Reitz

There already is an optional discriminator test, although it also noted the discriminator name itself as optional. Changing that, and adding a default-variant field, makes that test pass instead of failing.

Besides this one, a number of tests adding various error cases are added.

---
 tests/Makefile.include                                    |  3 +++
 ...flat-union-optional-discriminator-invalid-default.json | 12 ++++++++++++
 .../flat-union-optional-discriminator-no-default.json     | 11 +++++++++++
 tests/qapi-schema/flat-union-optional-discriminator.json  |  4 ++--
 .../flat-union-superfluous-default-variant.json           | 11 +++++++++++
 .../flat-union-optional-discriminator-invalid-default.err |  1 +
 ...flat-union-optional-discriminator-invalid-default.exit |  1 +
 .../flat-union-optional-discriminator-invalid-default.out |  0
 .../flat-union-optional-discriminator-no-default.err      |  1 +
 .../flat-union-optional-discriminator-no-default.exit     |  1 +
 .../flat-union-optional-discriminator-no-default.out      |  0
 tests/qapi-schema/flat-union-optional-discriminator.err   |  1 -
 tests/qapi-schema/flat-union-optional-discriminator.exit  |  2 +-
 tests/qapi-schema/flat-union-optional-discriminator.out   | 15 +++++++++++++++
 .../flat-union-superfluous-default-variant.err            |  1 +
 .../flat-union-superfluous-default-variant.exit           |  1 +
 .../flat-union-superfluous-default-variant.out            |  0
 17 files changed, 61 insertions(+), 4 deletions(-)
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.json
 create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.json
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.out
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.err
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.exit
 create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.out
 create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.err
 create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.exit
 create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.out
tests/Makefile.include +3 -0
index 3b9a5e31a2..d01df12b09 100644
@@ -500,7 +500,10 @@ qapi-schema += flat-union-invalid-branch-key.json
 qapi-schema += flat-union-invalid-discriminator.json
 qapi-schema += flat-union-no-base.json
 qapi-schema += flat-union-optional-discriminator.json
+qapi-schema += flat-union-optional-discriminator-no-default.json
+qapi-schema += flat-union-optional-discriminator-invalid-default.json
 qapi-schema += flat-union-string-discriminator.json
+qapi-schema += flat-union-superfluous-default-variant.json
 qapi-schema += funny-char.json
 qapi-schema += ident-with-escape.json
 qapi-schema += include-before-err.json
tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json new file +12 -0
new file mode 100644
index 0000000000..015a47ba52
@@ -0,0 +1,12 @@
+# default-variant must refer to an actual value of the discriminator's
+# enum
+{ 'enum': 'Enum', 'data': [ 'one', 'two' ] }
+{ 'struct': 'Base',
+  'data': { '*switch': 'Enum' } }
+{ 'struct': 'Branch', 'data': { 'name': 'str' } }
+{ 'union': 'MyUnion',
+  'base': 'Base',
+  'discriminator': 'switch',
+  'default-variant': 'three',
+  'data': { 'one': 'Branch',
+            'two': 'Branch' } }
tests/qapi-schema/flat-union-optional-discriminator-no-default.json new file +11 -0
new file mode 100644
index 0000000000..0d612f5a62
@@ -0,0 +1,11 @@
+# Using an optional discriminator requires specifying a default
+# variant
+{ 'enum': 'Enum', 'data': [ 'one', 'two' ] }
+{ 'struct': 'Base',
+  'data': { '*switch': 'Enum' } }
+{ 'struct': 'Branch', 'data': { 'name': 'str' } }
+{ 'union': 'MyUnion',
+  'base': 'Base',
+  'discriminator': 'switch',
+  'data': { 'one': 'Branch',
+            'two': 'Branch' } }
tests/qapi-schema/flat-union-optional-discriminator.json +2 -2
index 08a8f7ef8b..68e87083a1 100644
@@ -1,10 +1,10 @@
-# we require the discriminator to be non-optional
 { 'enum': 'Enum', 'data': [ 'one', 'two' ] }
 { 'struct': 'Base',
   'data': { '*switch': 'Enum' } }
 { 'struct': 'Branch', 'data': { 'name': 'str' } }
 { 'union': 'MyUnion',
   'base': 'Base',
-  'discriminator': '*switch',
+  'discriminator': 'switch',
+  'default-variant': 'one',
   'data': { 'one': 'Branch',
             'two': 'Branch' } }
tests/qapi-schema/flat-union-superfluous-default-variant.json new file +11 -0
new file mode 100644
index 0000000000..8558165868
@@ -0,0 +1,11 @@
+# default-variant only makes sense with an optional discriminator
+{ 'enum': 'Enum', 'data': [ 'one', 'two' ] }
+{ 'struct': 'Base',
+  'data': { 'switch': 'Enum' } }
+{ 'struct': 'Branch', 'data': { 'name': 'str' } }
+{ 'union': 'MyUnion',
+  'base': 'Base',
+  'discriminator': 'switch',
+  'default-variant': 'one',
+  'data': { 'one': 'Branch',
+            'two': 'Branch' } }
tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err new file +1 -0
new file mode 100644
index 0000000000..b6bd3423d6
@@ -0,0 +1 @@
+tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json:7: Default variant 'three' of flat union 'MyUnion' is not part of 'Enum'
tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit new file +1 -0
new file mode 100644
index 0000000000..d00491fd7e
@@ -0,0 +1 @@
+1
tests/qapi-schema/flat-union-optional-discriminator-invalid-default.out new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
tests/qapi-schema/flat-union-optional-discriminator-no-default.err new file +1 -0
new file mode 100644
index 0000000000..1342efd9e8
@@ -0,0 +1 @@
+tests/qapi-schema/flat-union-optional-discriminator-no-default.json:7: Default variant must be specified for optional discriminator 'switch'
tests/qapi-schema/flat-union-optional-discriminator-no-default.exit new file +1 -0
new file mode 100644
index 0000000000..d00491fd7e
@@ -0,0 +1 @@
+1
tests/qapi-schema/flat-union-optional-discriminator-no-default.out new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
tests/qapi-schema/flat-union-optional-discriminator.err +0 -1
index aaabedb3bd..e69de29bb2 100644
@@ -1 +0,0 @@
-tests/qapi-schema/flat-union-optional-discriminator.json:6: Discriminator of flat union 'MyUnion' does not allow optional name '*switch'
tests/qapi-schema/flat-union-optional-discriminator.exit +1 -1
index d00491fd7e..573541ac97 100644
@@ -1 +1 @@
-1
+0
tests/qapi-schema/flat-union-optional-discriminator.out +15 -0
index e69de29bb2..d3a72f63c2 100644
@@ -0,0 +1,15 @@
+object q_empty
+enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+    prefix QTYPE
+module flat-union-optional-discriminator.json
+enum Enum ['one', 'two']
+object Base
+    member switch: Enum optional=True
+object Branch
+    member name: str optional=False
+object MyUnion
+    base Base
+    tag switch
+    default variant: one
+    case one: Branch
+    case two: Branch
tests/qapi-schema/flat-union-superfluous-default-variant.err new file +1 -0
new file mode 100644
index 0000000000..5230bbf645
@@ -0,0 +1 @@
+tests/qapi-schema/flat-union-superfluous-default-variant.json:6: Must not specify a default variant for non-optional discriminator 'switch'
tests/qapi-schema/flat-union-superfluous-default-variant.exit new file +1 -0
new file mode 100644
index 0000000000..d00491fd7e
@@ -0,0 +1 @@
+1
tests/qapi-schema/flat-union-superfluous-default-variant.out new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
Signature · 1 line
2.14.3
#10 lore raw

Re: [Qemu-devel] [PATCH 03/13] tests: Add QAPI optional discriminator tests

Quoted 3 lines · Max Reitz

There already is an optional discriminator test, although it also noted the discriminator name itself as optional. Changing that, and adding a default-variant field, makes that test pass instead of failing.

I'm not sure I agree with that one. I think that you should instead add a positive test of a default variant into qapi-schema-test.json, especially since that is the one positive test where we also ensure that the C compiler is happy with the generated code (other positive tests prove that the generator produced something without error, but not that what it produced could be compiled).

Quoted 13 lines
  17 files changed, 61 insertions(+), 4 deletions(-)

create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.json create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.json create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.out create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.err create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.exit create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.out create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.err create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.exit create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.out

Patch 1 converted 2 error messages into 6, where 2 of them look identical:

Quoted 6 lines

Review

Quoted 6 lines
(quoted diff) +2 -2
-        # The value of member 'discriminator' must name a non-optional
-        # member of the base struct.
+        # The value of member 'discriminator' must name a member of
+        # the base struct.
         check_name(info, "Discriminator of flat union '%s'" % name,
                    discriminator)

[0] check_name() checks that 'discriminator':'*name' is rejected - this check is unchanged

Quoted 6 lines

Review

Quoted 6 lines
(quoted diff) +0 -6
-        discriminator_type = base_members.get(discriminator)
-        if not discriminator_type:
-            raise QAPISemError(info,
-                               "Discriminator '%s' is not a member of base "
-                               "struct '%s'"
-                               % (discriminator, base))

The old code ensured that 'discriminator':'name' finds 'name' as a mandatory field in the base type; which changed into:

Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+        if default_variant is None:
+            discriminator_type = base_members.get(discriminator)
+            if not discriminator_type:
+                if base_members.get('*' + discriminator) is None:
+                    raise QAPISemError(info,
+                                       "Discriminator '%s' is not a member of "
+                                       "base struct '%s'"
+                                       % (discriminator, base))

[1] the discriminator type must be a member field (we didn't find either a mandatory or an optional field)

Quoted 5 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+                else:
+                    raise QAPISemError(info,
+                                       "Default variant must be specified for "
+                                       "optional discriminator '%s'"
+                                       % discriminator)

[2] without default_variant, the member field must be mandatory

Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+        else:
+            discriminator_type = base_members.get('*' + discriminator)
+            if not discriminator_type:
+                if base_members.get(discriminator) is None:
+                    raise QAPISemError(info,
+                                       "Discriminator '%s' is not a member of "
+                                       "base struct '%s'"
+                                       % (discriminator, base))

[3] the discriminator type must be a member field (we didn't find either a mandatory or an optional field), identical message to [1]

Quoted 5 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+                else:
+                    raise QAPISemError(info,
+                                       "Must not specify a default variant for "
+                                       "non-optional discriminator '%s'"
+                                       % discriminator)

[4] with default_variant, the member field must be optional

Quoted 8 lines
Quoted 8 lines
(quoted diff) +1 -0
+
         enum_define = enum_types.get(discriminator_type)
         allow_metas = ['struct']
         # Do not allow string discriminator
@@ -763,6 +785,15 @@ def check_union(expr, info):
                                "Discriminator '%s' must be of enumeration "
                                "type" % discriminator)
 
Quoted 9 lines

Review def check_union(expr, info): · line 796

Quoted 9 lines
(quoted diff) +9 -0
+        if default_variant is not None:
+            # Must be a value of the enumeration
+            if default_variant not in enum_define['data']:
+                raise QAPISemError(info,
+                                   "Default variant '%s' of flat union '%s' is "
+                                   "not part of '%s'"
+                                   % (default_variant, name,
+                                      discriminator_type))
+

[5] the default discriminator value must belong to the discriminator's type

Of those 6 messages, you now have coverage of:

tests/Makefile.include qapi-schema += flat-union-invalid-branch-key.json · line 500

Quoted 4 lines

+++ b/tests/Makefile.include

tests/Makefile.include +0 -0
@@ -500,7 +500,10 @@ qapi-schema += flat-union-invalid-branch-key.json
 qapi-schema += flat-union-invalid-discriminator.json

[1] no change to that test

tests/Makefile.include qapi-schema += flat-union-invalid-branch-key.json · line 502

Quoted 2 lines
tests/Makefile.include +0 -0
 qapi-schema += flat-union-no-base.json
 qapi-schema += flat-union-optional-discriminator.json

the old test for [0] - you turned it into success. See more below...

tests/Makefile.include qapi-schema += flat-union-invalid-branch-key.json · line 503

Quoted 1 line
tests/Makefile.include +1 -0
+qapi-schema += flat-union-optional-discriminator-no-default.json

new test for [2]

tests/Makefile.include qapi-schema += flat-union-invalid-branch-key.json · line 504

Quoted 1 line
tests/Makefile.include +1 -0
+qapi-schema += flat-union-optional-discriminator-invalid-default.json

new test for [5]

tests/Makefile.include qapi-schema += flat-union-invalid-branch-key.json · line 506

Quoted 2 lines
tests/Makefile.include +1 -0
 qapi-schema += flat-union-string-discriminator.json
+qapi-schema += flat-union-superfluous-default-variant.json

new test for [4]

I don't know if it is worth trying to explicitly test for [3], given that it is identical to [1], but the loss of the test for [0] bothers me.

tests/qapi-schema/flat-union-optional-discriminator.err line 1

Quoted 3 lines
tests/qapi-schema/flat-union-optional-discriminator.err +1 -1
+++ b/tests/qapi-schema/flat-union-optional-discriminator.err
@@ -1 +0,0 @@
-tests/qapi-schema/flat-union-optional-discriminator.json:6: Discriminator of flat union 'MyUnion' does not allow optional name '*switch'

This used to be covered by [0] - but you rewrote the test by changing 'discriminator':'*switch' to 'discriminator':'switch'.

I'd argue that you WANT to keep 'discriminator':'*switch' in this test, to keep the testing of error [0] (even when the discriminator type is optional, we spell it without leading '*' in the 'union' description), and then only qapi-schema-test.json gets a positive test, which also gives us the additional coverage that the C compiler likes our generated code.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#11 lore raw

Re: [Qemu-devel] [PATCH 03/13] tests: Add QAPI optional discriminator tests

Quoted 11 lines · Eric Blake
Quoted 3 lines · Max Reitz

There already is an optional discriminator test, although it also noted the discriminator name itself as optional.  Changing that, and adding a default-variant field, makes that test pass instead of failing.

I'm not sure I agree with that one.  I think that you should instead add a positive test of a default variant into qapi-schema-test.json, especially since that is the one positive test where we also ensure that the C compiler is happy with the generated code (other positive tests prove that the generator produced something without error, but not that what it produced could be compiled).

Oh, sure.

Quoted 155 lines
Quoted 17 lines
  17 files changed, 61 insertions(+), 4 deletions(-)

  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.json create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.json create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.json create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.err create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.exit create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-invalid-default.out create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.err create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.exit

Quoted 8 lines

  create mode 100644 tests/qapi-schema/flat-union-optional-discriminator-no-default.out create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.err create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.exit create mode 100644 tests/qapi-schema/flat-union-superfluous-default-variant.out

Patch 1 converted 2 error messages into 6, where 2 of them look identical:

Quoted 6 lines
Quoted 6 lines

-        # The value of member 'discriminator' must name a non-optional -        # member of the base struct.
+        # The value of member 'discriminator' must name a member of +        # the base struct.
         check_name(info, "Discriminator of flat union '%s'" % name, discriminator)

[0] check_name() checks that 'discriminator':'*name' is rejected - this check is unchanged

Quoted 7 lines
Quoted 7 lines

-        discriminator_type = base_members.get(discriminator) -        if not discriminator_type:
-            raise QAPISemError(info,
-                               "Discriminator '%s' is not a member of base "
-                               "struct '%s'"
-                               % (discriminator, base))

The old code ensured that 'discriminator':'name' finds 'name' as a mandatory field in the base type; which changed into:

Quoted 9 lines
Quoted 9 lines

+        if default_variant is None:
+            discriminator_type = base_members.get(discriminator) +            if not discriminator_type:
+                if base_members.get('*' + discriminator) is None:
+                    raise QAPISemError(info,
+                                       "Discriminator '%s' is not a member of "
+                                       "base struct '%s'" +                                       % (discriminator, base))

[1] the discriminator type must be a member field (we didn't find either a mandatory or an optional field)

Quoted 6 lines
Quoted 6 lines

+                else:
+                    raise QAPISemError(info,
+                                       "Default variant must be specified for "
+                                       "optional discriminator '%s'" +                                       % discriminator)

[2] without default_variant, the member field must be mandatory

Quoted 9 lines
Quoted 9 lines

+        else:
+            discriminator_type = base_members.get('*' + discriminator) +            if not discriminator_type:
+                if base_members.get(discriminator) is None:
+                    raise QAPISemError(info,
+                                       "Discriminator '%s' is not a member of "
+                                       "base struct '%s'"
+                                       % (discriminator, base))

[3] the discriminator type must be a member field (we didn't find either a mandatory or an optional field), identical message to [1]

Quoted 7 lines
Quoted 7 lines

+                else:
+                    raise QAPISemError(info,
+                                       "Must not specify a default variant for "
+                                       "non-optional discriminator '%s'"
+                                       % discriminator)

[4] with default_variant, the member field must be optional

Quoted 11 lines
Quoted 11 lines

+
         enum_define = enum_types.get(discriminator_type) allow_metas = ['struct']
         # Do not allow string discriminator

(quoted diff) +0 -0
@@ -763,6 +785,15 @@ def check_union(expr, info):

                                "Discriminator '%s' must be of enumeration "
                                "type" % discriminator)
+        if default_variant is not None:
+            # Must be a value of the enumeration

Quoted 8 lines
Quoted 8 lines

+            if default_variant not in enum_define['data']:
+                raise QAPISemError(info,
+                                   "Default variant '%s' of flat union '%s' is "
+                                   "not part of '%s'" +                                   % (default_variant, name, +                                      discriminator_type)) +

[5] the default discriminator value must belong to the discriminator's type

Of those 6 messages, you now have coverage of:

Quoted 4 lines

+++ b/tests/Makefile.include

(quoted diff) +0 -0
@@ -500,7 +500,10 @@ qapi-schema += flat-union-invalid-branch-key.json

 qapi-schema += flat-union-invalid-discriminator.json

[1] no change to that test

Quoted 2 lines

 qapi-schema += flat-union-no-base.json qapi-schema += flat-union-optional-discriminator.json

the old test for [0] - you turned it into success. See more below...

Quoted 1 line

+qapi-schema += flat-union-optional-discriminator-no-default.json

new test for [2]

Quoted 1 line

+qapi-schema += flat-union-optional-discriminator-invalid-default.json

new test for [5]

Quoted 2 lines

 qapi-schema += flat-union-string-discriminator.json +qapi-schema += flat-union-superfluous-default-variant.json

new test for [4]

I don't know if it is worth trying to explicitly test for [3], given that it is identical to [1], but the loss of the test for [0] bothers me.

Quoted 5 lines

+++ b/tests/qapi-schema/flat-union-optional-discriminator.err

(quoted diff) +0 -1
@@ -1 +0,0 @@
-tests/qapi-schema/flat-union-optional-discriminator.json:6:

Discriminator of flat union 'MyUnion' does not allow optional name '*switch'

This used to be covered by [0] - but you rewrote the test by changing 'discriminator':'*switch' to 'discriminator':'switch'.

Quoted 6 lines

I'd argue that you WANT to keep 'discriminator':'*switch' in this test, to keep the testing of error [0] (even when the discriminator type is optional, we spell it without leading '*' in the 'union' description), and then only qapi-schema-test.json gets a positive test, which also gives us the additional coverage that the C compiler likes our generated code.

OK, but I'd like to rename the test, then.

Max

author
#12 lore raw

PATCH 4/13 qapi: Formalize qcow2 encryption probing

In reply to Max Reitz

Currently, you can give no encryption format for a qcow2 file while still passing a key-secret. That does not conform to the schema, so this patch changes the schema to allow it.

---
 qapi/block-core.json | 44 ++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 40 insertions(+), 4 deletions(-)
qapi/block-core.json +40 -4
index 71c9ab8538..092a1aba2d 100644
@@ -43,6 +43,19 @@
 { 'struct': 'ImageInfoSpecificQCow2EncryptionBase',
   'data': { 'format': 'BlockdevQcow2EncryptionFormat'}}
 
+##
+# @ImageInfoSpecificQCow2EncryptionNoInfo:
+#
+# Only used for the qcow2 encryption format "from-image" in which the
+# actual encryption format is determined from the image header.
+# Therefore, this encryption format will never be reported in
+# ImageInfoSpecificQCow2Encryption.
+#
+# Since: 2.13
+##
+{ 'struct': 'ImageInfoSpecificQCow2EncryptionNoInfo',
+  'data': { } }
+
 ##
 # @ImageInfoSpecificQCow2Encryption:
 #
@@ -52,7 +65,8 @@
   'base': 'ImageInfoSpecificQCow2EncryptionBase',
   'discriminator': 'format',
   'data': { 'aes': 'QCryptoBlockInfoQCow',
-            'luks': 'QCryptoBlockInfoLUKS' } }
+            'luks': 'QCryptoBlockInfoLUKS',
+            'from-image': 'ImageInfoSpecificQCow2EncryptionNoInfo' } }
 
 ##
 # @ImageInfoSpecificQCow2:
@@ -2739,10 +2753,30 @@
 # @BlockdevQcow2EncryptionFormat:
 # @aes: AES-CBC with plain64 initialization venctors
 #
+# @from-image:      Determine the encryption format from the image
+#                   header.  This only allows the use of the
+#                   key-secret option.  (Since: 2.13)
+#
 # Since: 2.10
 ##
 { 'enum': 'BlockdevQcow2EncryptionFormat',
-  'data': [ 'aes', 'luks' ] }
+  'data': [ 'aes', 'luks', 'from-image' ] }
+
+##
+# @BlockdevQcow2EncryptionSecret:
+#
+# Allows specifying a key-secret without specifying the exact
+# encryption format, which is determined automatically from the image
+# header.
+#
+# @key-secret:      The ID of a QCryptoSecret object providing the
+#                   decryption key.  Mandatory except when probing
+#                   image for metadata only.
+#
+# Since: 2.13
+##
+{ 'struct': 'BlockdevQcow2EncryptionSecret',
+  'data': { '*key-secret': 'str' } }
 
 ##
 # @BlockdevQcow2Encryption:
@@ -2750,10 +2784,12 @@
 # Since: 2.10
 ##
 { 'union': 'BlockdevQcow2Encryption',
-  'base': { 'format': 'BlockdevQcow2EncryptionFormat' },
+  'base': { '*format': 'BlockdevQcow2EncryptionFormat' },
   'discriminator': 'format',
+  'default-variant': 'from-image',
   'data': { 'aes': 'QCryptoBlockOptionsQCow',
-            'luks': 'QCryptoBlockOptionsLUKS'} }
+            'luks': 'QCryptoBlockOptionsLUKS',
+            'from-image': 'BlockdevQcow2EncryptionSecret' } }
 
 ##
 # @BlockdevOptionsQcow2:
Signature · 1 line
2.14.3
#13 lore raw

Re: [Qemu-devel] [PATCH 04/13] qapi: Formalize qcow2 encryption probing

Quoted 77 lines · Max Reitz

Currently, you can give no encryption format for a qcow2 file while still passing a key-secret. That does not conform to the schema, so this patch changes the schema to allow it.

---
 qapi/block-core.json | 44 ++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 40 insertions(+), 4 deletions(-)
qapi/block-core.json +36 -2
index 71c9ab8538..092a1aba2d 100644
@@ -43,6 +43,19 @@
 { 'struct': 'ImageInfoSpecificQCow2EncryptionBase',
   'data': { 'format': 'BlockdevQcow2EncryptionFormat'}}
 
+##
+# @ImageInfoSpecificQCow2EncryptionNoInfo:
+#
+# Only used for the qcow2 encryption format "from-image" in which the
+# actual encryption format is determined from the image header.
+# Therefore, this encryption format will never be reported in
+# ImageInfoSpecificQCow2Encryption.
+#
+# Since: 2.13
+##
+{ 'struct': 'ImageInfoSpecificQCow2EncryptionNoInfo',
+  'data': { } }
+
 ##
 # @ImageInfoSpecificQCow2Encryption:
 #
@@ -52,7 +65,8 @@
   'base': 'ImageInfoSpecificQCow2EncryptionBase',
   'discriminator': 'format',
   'data': { 'aes': 'QCryptoBlockInfoQCow',
-            'luks': 'QCryptoBlockInfoLUKS' } }
+            'luks': 'QCryptoBlockInfoLUKS',
+            'from-image': 'ImageInfoSpecificQCow2EncryptionNoInfo' } }
 
 ##
 # @ImageInfoSpecificQCow2:
@@ -2739,10 +2753,30 @@
 # @BlockdevQcow2EncryptionFormat:
 # @aes: AES-CBC with plain64 initialization venctors
 #
+# @from-image:      Determine the encryption format from the image
+#                   header.  This only allows the use of the
+#                   key-secret option.  (Since: 2.13)
+#
 # Since: 2.10
 ##
 { 'enum': 'BlockdevQcow2EncryptionFormat',
-  'data': [ 'aes', 'luks' ] }
+  'data': [ 'aes', 'luks', 'from-image' ] }
+
+##
+# @BlockdevQcow2EncryptionSecret:
+#
+# Allows specifying a key-secret without specifying the exact
+# encryption format, which is determined automatically from the image
+# header.
+#
+# @key-secret:      The ID of a QCryptoSecret object providing the
+#                   decryption key.  Mandatory except when probing
+#                   image for metadata only.
+#
+# Since: 2.13
+##
+{ 'struct': 'BlockdevQcow2EncryptionSecret',
+  'data': { '*key-secret': 'str' } }
 
 ##
 # @BlockdevQcow2Encryption:
@@ -2750,10 +2784,12 @@
 # Since: 2.10

qapi/block-core.json line 2792

Quoted 10 lines
qapi/block-core.json +4 -2
 ##
 { 'union': 'BlockdevQcow2Encryption',
-  'base': { 'format': 'BlockdevQcow2EncryptionFormat' },
+  'base': { '*format': 'BlockdevQcow2EncryptionFormat' },
   'discriminator': 'format',
+  'default-variant': 'from-image',
   'data': { 'aes': 'QCryptoBlockOptionsQCow',
-            'luks': 'QCryptoBlockOptionsLUKS'} }
+            'luks': 'QCryptoBlockOptionsLUKS',
+            'from-image': 'BlockdevQcow2EncryptionSecret' } }

Bike-shedding on name, how about "auto" or "probe" ?

IIUC, this schema addition means the QAPI parser now allows

encrypt.format=from-image,encrypt.key-secret=sec0,...other opts...

but the code will not accept "from-image" as a valid string.

eg qcow2_update_options_prepare() will do

    case QCOW_CRYPT_AES:
        if (encryptfmt && !g_str_equal(encryptfmt, "aes")) {
            error_setg(errp,
                       "Header reported 'aes' encryption format but "
                       "options specify '%s'", encryptfmt);
            ret = -EINVAL;
            goto fail;
        }

       ...snip....

    case QCOW_CRYPT_LUKS:
        if (encryptfmt && !g_str_equal(encryptfmt, "luks")) {
            error_setg(errp,
                       "Header reported 'luks' encryption format but "
                       "options specify '%s'", encryptfmt);
            ret = -EINVAL;
            goto fail;
        }

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#14 lore raw

Re: [Qemu-devel] [PATCH 04/13] qapi: Formalize qcow2 encryption probing

Quoted 20 lines · Daniel P. Berrangé
Quoted 10 lines · Max Reitz

Currently, you can give no encryption format for a qcow2 file while still passing a key-secret. That does not conform to the schema, so this patch changes the schema to allow it.

---
  qapi/block-core.json | 44 ++++++++++++++++++++++++++++++++++++++++----
  1 file changed, 40 insertions(+), 4 deletions(-)

{ 'union': 'BlockdevQcow2Encryption',

qapi/block-core.json line 2792

Quoted 8 lines
qapi/block-core.json +4 -2
-  'base': { 'format': 'BlockdevQcow2EncryptionFormat' },
+  'base': { '*format': 'BlockdevQcow2EncryptionFormat' },
    'discriminator': 'format',
+  'default-variant': 'from-image',
    'data': { 'aes': 'QCryptoBlockOptionsQCow',
-            'luks': 'QCryptoBlockOptionsLUKS'} }
+            'luks': 'QCryptoBlockOptionsLUKS',
+            'from-image': 'BlockdevQcow2EncryptionSecret' } }
Quoted 1 line

Bike-shedding on name, how about "auto" or "probe" ?

Either of those sounds nicer to me; 'auto' might be better in the context of creation (that way, we can state that creating a NEW image with x-blockdev-create maps 'auto' to 'luks'; while connecting to an EXISTING image maps 'auto' to either 'aes' or 'luks' as appropriate).

Quoted 4 lines

IIUC, this schema addition means the QAPI parser now allows

encrypt.format=from-image,encrypt.key-secret=sec0,...other opts...

Yes. You could, perhaps, add a special case on the command line parsing code to reject an explicit use of format=from-image, but the QMP should not reject an explicit discriminator.

Hmm, it plays in with my comment on 1/13 - should the QMP parser automatically set has_discriminator to true when it supplies the default? If it does, you lose the ability to see whether the user supplied an explicit encrypt.format=from-image (or the equivalent when using QMP instead of the command line), if you wanted to enforce that the user MUST omit format when relying on the from-image variant.

I don't see a problem in allowing the user to explicitly specify the name of the default branch, but I _do_ think the patch is incomplete for not handling the new QCOW_CRYPT_FROM_IMAGE case and converting it as soon as possible back into one of the other two preferred enum values.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#15 lore raw

Re: [Qemu-devel] [PATCH 04/13] qapi: Formalize qcow2 encryption probing

In reply to Daniel P. Berrangé
Quoted 74 lines · Daniel P. Berrangé
Quoted 73 lines · Max Reitz

Currently, you can give no encryption format for a qcow2 file while still passing a key-secret. That does not conform to the schema, so this patch changes the schema to allow it.

---
 qapi/block-core.json | 44 ++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 40 insertions(+), 4 deletions(-)
qapi/block-core.json +36 -2
index 71c9ab8538..092a1aba2d 100644
@@ -43,6 +43,19 @@
 { 'struct': 'ImageInfoSpecificQCow2EncryptionBase',
   'data': { 'format': 'BlockdevQcow2EncryptionFormat'}}
 
+##
+# @ImageInfoSpecificQCow2EncryptionNoInfo:
+#
+# Only used for the qcow2 encryption format "from-image" in which the
+# actual encryption format is determined from the image header.
+# Therefore, this encryption format will never be reported in
+# ImageInfoSpecificQCow2Encryption.
+#
+# Since: 2.13
+##
+{ 'struct': 'ImageInfoSpecificQCow2EncryptionNoInfo',
+  'data': { } }
+
 ##
 # @ImageInfoSpecificQCow2Encryption:
 #
@@ -52,7 +65,8 @@
   'base': 'ImageInfoSpecificQCow2EncryptionBase',
   'discriminator': 'format',
   'data': { 'aes': 'QCryptoBlockInfoQCow',
-            'luks': 'QCryptoBlockInfoLUKS' } }
+            'luks': 'QCryptoBlockInfoLUKS',
+            'from-image': 'ImageInfoSpecificQCow2EncryptionNoInfo' } }
 
 ##
 # @ImageInfoSpecificQCow2:
@@ -2739,10 +2753,30 @@
 # @BlockdevQcow2EncryptionFormat:
 # @aes: AES-CBC with plain64 initialization venctors
 #
+# @from-image:      Determine the encryption format from the image
+#                   header.  This only allows the use of the
+#                   key-secret option.  (Since: 2.13)
+#
 # Since: 2.10
 ##
 { 'enum': 'BlockdevQcow2EncryptionFormat',
-  'data': [ 'aes', 'luks' ] }
+  'data': [ 'aes', 'luks', 'from-image' ] }
+
+##
+# @BlockdevQcow2EncryptionSecret:
+#
+# Allows specifying a key-secret without specifying the exact
+# encryption format, which is determined automatically from the image
+# header.
+#
+# @key-secret:      The ID of a QCryptoSecret object providing the
+#                   decryption key.  Mandatory except when probing
+#                   image for metadata only.
+#
+# Since: 2.13
+##
+{ 'struct': 'BlockdevQcow2EncryptionSecret',
+  'data': { '*key-secret': 'str' } }
 
Quoted 16 lines
Quoted 4 lines
qapi/block-core.json +0 -0
 ##
 # @BlockdevQcow2Encryption:
@@ -2750,10 +2784,12 @@
 # Since: 2.10

qapi/block-core.json line 2792

Quoted 10 lines
qapi/block-core.json +4 -2
 ##
 { 'union': 'BlockdevQcow2Encryption',
-  'base': { 'format': 'BlockdevQcow2EncryptionFormat' },
+  'base': { '*format': 'BlockdevQcow2EncryptionFormat' },
   'discriminator': 'format',
+  'default-variant': 'from-image',
   'data': { 'aes': 'QCryptoBlockOptionsQCow',
-            'luks': 'QCryptoBlockOptionsLUKS'} }
+            'luks': 'QCryptoBlockOptionsLUKS',
+            'from-image': 'BlockdevQcow2EncryptionSecret' } }

Bike-shedding on name, how about "auto" or "probe" ?

Sure. I like "probe" a bit better than "auto", although "auto" is what we usually have... But I think "probe" is still a bit better.

Quoted 6 lines

IIUC, this schema addition means the QAPI parser now allows

encrypt.format=from-image,encrypt.key-secret=sec0,...other opts...

but the code will not accept "from-image" as a valid string.

Ah, right, I forgot that. Will fix.

Thanks for reviewing!

Max

Quoted 26 lines

eg qcow2_update_options_prepare() will do

    case QCOW_CRYPT_AES:
        if (encryptfmt && !g_str_equal(encryptfmt, "aes")) {
            error_setg(errp,
                       "Header reported 'aes' encryption format but "
                       "options specify '%s'", encryptfmt);
            ret = -EINVAL;
            goto fail;
        }

       ...snip....

    case QCOW_CRYPT_LUKS:
        if (encryptfmt && !g_str_equal(encryptfmt, "luks")) {
            error_setg(errp,
                       "Header reported 'luks' encryption format but "
                       "options specify '%s'", encryptfmt);
            ret = -EINVAL;
            goto fail;
        }

Regards,
Daniel

author
#16 lore raw

PATCH 5/13 qapi: Formalize qcow encryption probing

In reply to Max Reitz

Currently, you can give no encryption format for a qcow file while still passing a key-secret. That does not conform to the schema, so this patch changes the schema to allow it.

---
 qapi/block-core.json | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)
qapi/block-core.json +25 -3
index 092a1aba2d..d98af0a71d 100644
@@ -2712,15 +2712,35 @@
   'data': { 'flags': 'Qcow2OverlapCheckFlags',
             'mode':  'Qcow2OverlapCheckMode' } }
 
+##
+# @BlockdevQcowEncryptionSecret:
+#
+# Allows specifying a key-secret without specifying the exact
+# encryption format, which is determined automatically from the image
+# header.
+#
+# @key-secret:      The ID of a QCryptoSecret object providing the
+#                   decryption key.  Mandatory except when probing
+#                   image for metadata only.
+#
+# Since: 2.13
+##
+{ 'struct': 'BlockdevQcowEncryptionSecret',
+  'data': { '*key-secret': 'str' } }
+
 ##
 # @BlockdevQcowEncryptionFormat:
 #
 # @aes: AES-CBC with plain64 initialization vectors
 #
+# @from-image:      Determine the encryption format from the image
+#                   header.  This only allows the use of the
+#                   key-secret option.  (Since: 2.13)
+#
 # Since: 2.10
 ##
 { 'enum': 'BlockdevQcowEncryptionFormat',
-  'data': [ 'aes' ] }
+  'data': [ 'aes', 'from-image' ] }
 
 ##
 # @BlockdevQcowEncryption:
@@ -2728,9 +2748,11 @@
 # Since: 2.10
 ##
 { 'union': 'BlockdevQcowEncryption',
-  'base': { 'format': 'BlockdevQcowEncryptionFormat' },
+  'base': { '*format': 'BlockdevQcowEncryptionFormat' },
   'discriminator': 'format',
-  'data': { 'aes': 'QCryptoBlockOptionsQCow' } }
+  'default-variant': 'from-image',
+  'data': { 'aes': 'QCryptoBlockOptionsQCow',
+            'from-image': 'BlockdevQcowEncryptionSecret' } }
 
 ##
 # @BlockdevOptionsQcow:
Signature · 1 line
2.14.3
#17 lore raw

Re: [Qemu-devel] [PATCH 05/13] qapi: Formalize qcow encryption probing

Quoted 12 lines · Max Reitz

Currently, you can give no encryption format for a qcow file while still passing a key-secret. That does not conform to the schema, so this patch changes the schema to allow it.

---

##
# @BlockdevQcowEncryptionFormat:
#
# @aes: AES-CBC with plain64 initialization vectors #

qapi/block-core.json line 2743

Quoted 9 lines
qapi/block-core.json +5 -1
+# @from-image:      Determine the encryption format from the image
+#                   header.  This only allows the use of the
+#                   key-secret option.  (Since: 2.13)
+#
  # Since: 2.10
  ##
  { 'enum': 'BlockdevQcowEncryptionFormat',
-  'data': [ 'aes' ] }
+  'data': [ 'aes', 'from-image' ] }

Overkill. Why not just:

qapi/block-core.json line 2753

Quoted 12 lines

##
# @BlockdevQcowEncryption:

qapi/block-core.json +2 -2
@@ -2728,9 +2748,11 @@
  # Since: 2.10
  ##
  { 'union': 'BlockdevQcowEncryption',
-  'base': { 'format': 'BlockdevQcowEncryptionFormat' },
+  'base': { '*format': 'BlockdevQcowEncryptionFormat' },
    'discriminator': 'format',
-  'data': { 'aes': 'QCryptoBlockOptionsQCow' } }
+  'default-variant': 'from-image',

'default-variant': 'aes'

qapi/block-core.json line 2754

Quoted 1 line
qapi/block-core.json +1 -0
+  'data': { 'aes': 'QCryptoBlockOptionsQCow',

and call it good, because there are no other options to pick from, so 'from-image' would always resolve to 'aes' anyway.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#18 lore raw

Re: [Qemu-devel] [PATCH 05/13] qapi: Formalize qcow encryption probing

Quoted 39 lines · Eric Blake
Quoted 12 lines · Max Reitz

Currently, you can give no encryption format for a qcow file while still passing a key-secret. That does not conform to the schema, so this patch changes the schema to allow it.

---

##
# @BlockdevQcowEncryptionFormat:
#
# @aes: AES-CBC with plain64 initialization vectors #

qapi/block-core.json line 2743

Quoted 9 lines
qapi/block-core.json +5 -1
+# @from-image:      Determine the encryption format from the image
+#                   header.  This only allows the use of the
+#                   key-secret option.  (Since: 2.13)
+#
  # Since: 2.10
  ##
  { 'enum': 'BlockdevQcowEncryptionFormat',
-  'data': [ 'aes' ] }
+  'data': [ 'aes', 'from-image' ] }

Overkill. Why not just:

qapi/block-core.json line 2753

Quoted 11 lines

##
# @BlockdevQcowEncryption:

qapi/block-core.json +2 -2
@@ -2728,9 +2748,11 @@
  # Since: 2.10
  ##
  { 'union': 'BlockdevQcowEncryption',
-  'base': { 'format': 'BlockdevQcowEncryptionFormat' },
+  'base': { '*format': 'BlockdevQcowEncryptionFormat' },
    'discriminator': 'format',
-  'data': { 'aes': 'QCryptoBlockOptionsQCow' } }
+  'default-variant': 'from-image',

'default-variant': 'aes'

Quoted 4 lines

qapi/block-core.json line 2754

Quoted 1 line
qapi/block-core.json +1 -0
+  'data': { 'aes': 'QCryptoBlockOptionsQCow',

and call it good, because there are no other options to pick from, so 'from-image' would always resolve to 'aes' anyway.

Sounds reasonable because qcowv1 is a dead format we don't intend to add more features to.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
author
#19 lore raw

Re: [Qemu-devel] [PATCH 05/13] qapi: Formalize qcow encryption probing

In reply to Eric Blake
Quoted 39 lines · Eric Blake
Quoted 16 lines · Max Reitz

Currently, you can give no encryption format for a qcow file while still passing a key-secret.  That does not conform to the schema, so this patch changes the schema to allow it.

---

  ##
  # @BlockdevQcowEncryptionFormat:
  #
  # @aes: AES-CBC with plain64 initialization vectors #

qapi/block-core.json +4 -0
+# @from-image:      Determine the encryption format from the image
+#                   header.  This only allows the use of the
+#                   key-secret option.  (Since: 2.13)
+#
Quoted 5 lines

  # Since: 2.10
  ##
  { 'enum': 'BlockdevQcowEncryptionFormat',
-  'data': [ 'aes' ] }
+  'data': [ 'aes', 'from-image' ] }

Overkill.  Why not just:

qapi/block-core.json line 2753

Quoted 11 lines

    ##
  # @BlockdevQcowEncryption:

qapi/block-core.json +2 -2
@@ -2728,9 +2748,11 @@
   # Since: 2.10
   ##
   { 'union': 'BlockdevQcowEncryption',
-  'base': { 'format': 'BlockdevQcowEncryptionFormat' },
+  'base': { '*format': 'BlockdevQcowEncryptionFormat' },
     'discriminator': 'format',
-  'data': { 'aes': 'QCryptoBlockOptionsQCow' } }
+  'default-variant': 'from-image',

'default-variant': 'aes'

Quoted 4 lines

qapi/block-core.json line 2754

Quoted 1 line
qapi/block-core.json +1 -0
+  'data': { 'aes': 'QCryptoBlockOptionsQCow',

and call it good, because there are no other options to pick from, so 'from-image' would always resolve to 'aes' anyway.

Hmmmm. Yes. :-)

Max

author
#20 lore raw

PATCH 6/13 block: Add block-specific QDict header

In reply to Max Reitz

There are numerous QDict functions that have been introduced for and are used only by the block layer. Move their declarations into an own header file to reflect that.

While qdict_extract_subqdict() is in fact used outside of the block layer (in util/qemu-config.c), it is still a function related very closely to how the block layer works with nested QDicts, namely by sometimes flattening them. Therefore, its declaration is put into this header as well and util/qemu-config.c includes it with a comment stating exactly which function it needs.

---
 include/block/qdict.h    | 35 +++++++++++++++++++++++++++++++++++
 include/qapi/qmp/qdict.h | 17 -----------------
 block.c                  |  1 +
 block/gluster.c          |  1 +
 block/iscsi.c            |  1 +
 block/nbd.c              |  1 +
 block/nfs.c              |  1 +
 block/parallels.c        |  1 +
 block/qcow.c             |  1 +
 block/qcow2.c            |  1 +
 block/qed.c              |  1 +
 block/quorum.c           |  1 +
 block/rbd.c              |  1 +
 block/sheepdog.c         |  1 +
 block/snapshot.c         |  1 +
 block/ssh.c              |  1 +
 block/vhdx.c             |  1 +
 block/vpc.c              |  1 +
 block/vvfat.c            |  1 +
 block/vxhs.c             |  1 +
 blockdev.c               |  1 +
 qobject/qdict.c          |  1 +
 tests/check-qdict.c      |  1 +
 tests/check-qobject.c    |  1 +
 tests/test-replication.c |  1 +
 util/qemu-config.c       |  1 +
 26 files changed, 59 insertions(+), 17 deletions(-)
 create mode 100644 include/block/qdict.h
include/block/qdict.h new file +35 -0
new file mode 100644
index 0000000000..825e096a72
@@ -0,0 +1,35 @@
+/*
+ * Special QDict functions used by the block layer
+ *
+ * Copyright (c) 2018 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ */
+
+#ifndef BLOCK_QDICT_H
+#define BLOCK_QDICT_H
+
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qobject.h"
+#include "qapi/error.h"
+
+
+void qdict_copy_default(QDict *dst, QDict *src, const char *key);
+void qdict_set_default_str(QDict *dst, const char *key, const char *val);
+
+void qdict_join(QDict *dest, QDict *src, bool overwrite);
+
+void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
+void qdict_array_split(QDict *src, QList **dst);
+int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, Error **errp);
+void qdict_flatten(QDict *qdict);
+
+typedef struct QDictRenames {
+    const char *from;
+    const char *to;
+} QDictRenames;
+bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp);
+
+#endif
include/qapi/qmp/qdict.h +0 -17
index 921a28d2d3..7f3ec10a10 100644
@@ -67,23 +67,6 @@ int64_t qdict_get_try_int(const QDict *qdict, const char *key,
 bool qdict_get_try_bool(const QDict *qdict, const char *key, bool def_value);
 const char *qdict_get_try_str(const QDict *qdict, const char *key);
 
-void qdict_copy_default(QDict *dst, QDict *src, const char *key);
-void qdict_set_default_str(QDict *dst, const char *key, const char *val);
-
 QDict *qdict_clone_shallow(const QDict *src);
-void qdict_flatten(QDict *qdict);
-
-void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
-void qdict_array_split(QDict *src, QList **dst);
-int qdict_array_entries(QDict *src, const char *subqdict);
-QObject *qdict_crumple(const QDict *src, Error **errp);
-
-void qdict_join(QDict *dest, QDict *src, bool overwrite);
-
-typedef struct QDictRenames {
-    const char *from;
-    const char *to;
-} QDictRenames;
-bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp);
 
 #endif /* QDICT_H */
block.c +1 -0
index 676e57f562..3c3e8fd11d 100644
@@ -27,6 +27,7 @@
 #include "block/block_int.h"
 #include "block/blockjob.h"
 #include "block/nbd.h"
+#include "block/qdict.h"
 #include "qemu/error-report.h"
 #include "module_block.h"
 #include "qemu/module.h"
block/gluster.c +1 -0
index 9900b6420c..b5fe7f3e87 100644
@@ -11,6 +11,7 @@
 #include "qemu/osdep.h"
 #include <glusterfs/api/glfs.h>
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qerror.h"
block/iscsi.c +1 -0
index 3fd7203916..d70e8aea84 100644
@@ -33,6 +33,7 @@
 #include "qemu/bitops.h"
 #include "qemu/bitmap.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "scsi/constants.h"
 #include "qemu/iov.h"
 #include "qemu/option.h"
block/nbd.c +1 -0
index 3e1693cc55..f499830410 100644
@@ -28,6 +28,7 @@
 
 #include "qemu/osdep.h"
 #include "block/nbd-client.h"
+#include "block/qdict.h"
 #include "qapi/error.h"
 #include "qemu/uri.h"
 #include "block/block_int.h"
block/nfs.c +1 -0
index 66fddf12d4..5159ef0aa7 100644
@@ -29,6 +29,7 @@
 #include "qemu/error-report.h"
 #include "qapi/error.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "trace.h"
 #include "qemu/iov.h"
 #include "qemu/option.h"
block/parallels.c +1 -0
index 6e9c37f44e..c1d9643498 100644
@@ -31,6 +31,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
block/qcow.c +1 -0
index 3ba2ca25ea..e34eedc903 100644
@@ -26,6 +26,7 @@
 #include "qapi/error.h"
 #include "qemu/error-report.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
block/qcow2.c +1 -0
index 6d532470a8..416ab54298 100644
@@ -24,6 +24,7 @@
 
 #include "qemu/osdep.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include <zlib.h>
block/qed.c +1 -0
index 65cfe92393..324a953cbc 100644
@@ -13,6 +13,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "block/qdict.h"
 #include "qapi/error.h"
 #include "qemu/timer.h"
 #include "qemu/bswap.h"
block/quorum.c +1 -0
index e448d7e384..0d90a02583 100644
@@ -17,6 +17,7 @@
 #include "qemu/cutils.h"
 #include "qemu/option.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "qapi/error.h"
 #include "qapi/qapi-events-block.h"
 #include "qapi/qmp/qdict.h"
block/rbd.c +1 -0
index a16431e267..ad5fd30b43 100644
@@ -18,6 +18,7 @@
 #include "qemu/error-report.h"
 #include "qemu/option.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "crypto/secret.h"
 #include "qemu/cutils.h"
 #include "qapi/qmp/qstring.h"
block/sheepdog.c +1 -0
index 4237132419..bf7ab896d2 100644
@@ -24,6 +24,7 @@
 #include "qemu/option.h"
 #include "qemu/sockets.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "sysemu/block-backend.h"
 #include "qemu/bitops.h"
 #include "qemu/cutils.h"
block/snapshot.c +1 -0
index 2953d96c06..f9903bc94e 100644
@@ -25,6 +25,7 @@
 #include "qemu/osdep.h"
 #include "block/snapshot.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "qapi/error.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qerror.h"
block/ssh.c +1 -0
index 4c4fa3ccfc..eec37dd27c 100644
@@ -28,6 +28,7 @@
 #include <libssh2_sftp.h>
 
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "qapi/error.h"
 #include "qemu/error-report.h"
 #include "qemu/option.h"
block/vhdx.c +1 -0
index 0b1e21c750..b409acfeb2 100644
@@ -18,6 +18,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
block/vpc.c +1 -0
index 0ebfcd3cc8..41c8c980f1 100644
@@ -26,6 +26,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "sysemu/block-backend.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
block/vvfat.c +1 -0
index 662dca0114..4595f335b8 100644
@@ -27,6 +27,7 @@
 #include <dirent.h>
 #include "qapi/error.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "qemu/module.h"
 #include "qemu/option.h"
 #include "qemu/bswap.h"
block/vxhs.c +1 -0
index 339e23218d..0cb0a007e9 100644
@@ -12,6 +12,7 @@
 #include <qnio/qnio_api.h>
 #include <sys/param.h>
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qstring.h"
blockdev.c +1 -0
index 3808b1fc00..19c04d9276 100644
@@ -35,6 +35,7 @@
 #include "sysemu/blockdev.h"
 #include "hw/block/block.h"
 #include "block/blockjob.h"
+#include "block/qdict.h"
 #include "block/throttle-groups.h"
 #include "monitor/monitor.h"
 #include "qemu/error-report.h"
qobject/qdict.c +1 -0
index 22800eeceb..0554c64553 100644
@@ -11,6 +11,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "block/qdict.h"
 #include "qapi/qmp/qnum.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qbool.h"
tests/check-qdict.c +1 -0
index eba5d3528e..93e2112b6d 100644
@@ -11,6 +11,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "block/qdict.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qlist.h"
 #include "qapi/qmp/qnum.h"
tests/check-qobject.c +1 -0
index 5cb08fcb63..16ccbde82c 100644
@@ -8,6 +8,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "block/qdict.h"
 #include "qapi/qmp/qbool.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qlist.h"
tests/test-replication.c +1 -0
index 68c0d04f2a..c8165ae954 100644
@@ -15,6 +15,7 @@
 #include "qemu/option.h"
 #include "replication.h"
 #include "block/block_int.h"
+#include "block/qdict.h"
 #include "sysemu/block-backend.h"
 
 #define IMG_SIZE (64 * 1024 * 1024)
util/qemu-config.c +1 -0
index 14d84022dc..9d2e278e29 100644
@@ -1,4 +1,5 @@
 #include "qemu/osdep.h"
+#include "block/qdict.h" /* for qdict_extract_subqdict() */
 #include "qapi/error.h"
 #include "qapi/qapi-commands-misc.h"
 #include "qapi/qmp/qdict.h"
Signature · 1 line
2.14.3
Reviewed-by on v1 6/13
#21 lore raw

Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header

Quoted 2 lines · Max Reitz

There are numerous QDict functions that have been introduced for and are used only by the block layer. Move their declarations into an own

s/an own/their own/

Quoted 15 lines

header file to reflect that.

While qdict_extract_subqdict() is in fact used outside of the block layer (in util/qemu-config.c), it is still a function related very closely to how the block layer works with nested QDicts, namely by sometimes flattening them. Therefore, its declaration is put into this header as well and util/qemu-config.c includes it with a comment stating exactly which function it needs.

---

+++ b/include/block/qdict.h

include/block/qdict.h line 4

Quoted 4 lines
include/block/qdict.h new file +4 -0
@@ -0,0 +1,35 @@
+/*
+ * Special QDict functions used by the block layer
+ *
+ * Copyright (c) 2018 Red Hat, Inc.

You are extracting this from qdict.h which has:
* Copyright (C) 2009 Red Hat Inc.

Is it worth listing 2009-2018, instead of just this year?

include/block/qdict.h line 19

Quoted 3 lines
include/block/qdict.h new file +3 -0
+
+void qdict_copy_default(QDict *dst, QDict *src, const char *key);
+void qdict_set_default_str(QDict *dst, const char *key, const char *val);

These two might be useful outside of the block layer; we can move them back in a later patch if so. But for now, I agree with your choice of moving them.

Review line 9

Quoted 2 lines
(quoted diff) +2 -0
+
+void qdict_join(QDict *dest, QDict *src, bool overwrite);

This is borderline whether it would be useful outside of the block layer.

include/block/qdict.h line 27

Quoted 6 lines
include/block/qdict.h new file +6 -0
+
+void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
+void qdict_array_split(QDict *src, QList **dst);
+int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, Error **errp);
+void qdict_flatten(QDict *qdict);

And these are definitely hacks that the block layer relies on, where hopefully someday long term we can rewrite the block layer to use QAPI types directly instead of constant reconversion between QemuOpts and QDict and QAPI types.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#22 lore raw

Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header

Quoted 29 lines · Eric Blake
Quoted 2 lines · Max Reitz

There are numerous QDict functions that have been introduced for and are used only by the block layer.  Move their declarations into an own

s/an own/their own/

Quoted 15 lines

header file to reflect that.

While qdict_extract_subqdict() is in fact used outside of the block layer (in util/qemu-config.c), it is still a function related very closely to how the block layer works with nested QDicts, namely by sometimes flattening them.  Therefore, its declaration is put into this header as well and util/qemu-config.c includes it with a comment stating exactly which function it needs.

---

+++ b/include/block/qdict.h

include/block/qdict.h line 4

Quoted 4 lines
include/block/qdict.h new file +4 -0
@@ -0,0 +1,35 @@
+/*
+ * Special QDict functions used by the block layer
+ *
+ * Copyright (c) 2018 Red Hat, Inc.

You are extracting this from qdict.h which has:
 * Copyright (C) 2009 Red Hat Inc.

Quoted 1 line

Is it worth listing 2009-2018, instead of just this year?

I don't know, is it? I don't know whether it makes a real difference.

Quoted 9 lines
Quoted 4 lines
include/block/qdict.h new file +3 -0
+
+void qdict_copy_default(QDict *dst, QDict *src, const char *key);
+void qdict_set_default_str(QDict *dst, const char *key, const char

*val);

These two might be useful outside of the block layer; we can move them back in a later patch if so.  But for now, I agree with your choice of moving them.

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+void qdict_join(QDict *dest, QDict *src, bool overwrite);

This is borderline whether it would be useful outside of the block layer.

I decided I wanted to move the *_default* functions, and if I did that, I would have to move this one as well. I decided I can't be biased just because I wrote this one. :-)

But in general I'd say if anyone wants to use any of these functions outside of the block layer, they are welcome to move them back to the main header, provided they have a good reason to do so. I suppose a good reason for using qdict_join() may indeed turn up sooner or later.

Max

Quoted 14 lines

include/block/qdict.h line 27

Quoted 6 lines
include/block/qdict.h new file +6 -0
+
+void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
+void qdict_array_split(QDict *src, QList **dst);
+int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, Error **errp);
+void qdict_flatten(QDict *qdict);

And these are definitely hacks that the block layer relies on, where hopefully someday long term we can rewrite the block layer to use QAPI types directly instead of constant reconversion between QemuOpts and QDict and QAPI types.

Reviewed-by
#23 lore raw

Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header

Quoted 30 lines · Max Reitz
Quoted 29 lines · Eric Blake
Quoted 2 lines · Max Reitz

There are numerous QDict functions that have been introduced for and are used only by the block layer.  Move their declarations into an own

s/an own/their own/

Quoted 15 lines

header file to reflect that.

While qdict_extract_subqdict() is in fact used outside of the block layer (in util/qemu-config.c), it is still a function related very closely to how the block layer works with nested QDicts, namely by sometimes flattening them.  Therefore, its declaration is put into this header as well and util/qemu-config.c includes it with a comment stating exactly which function it needs.

---

+++ b/include/block/qdict.h

include/block/qdict.h line 4

Quoted 4 lines
include/block/qdict.h new file +4 -0
@@ -0,0 +1,35 @@
+/*
+ * Special QDict functions used by the block layer
+ *
+ * Copyright (c) 2018 Red Hat, Inc.

You are extracting this from qdict.h which has:
 * Copyright (C) 2009 Red Hat Inc.

Quoted 3 lines
Quoted 1 line

Is it worth listing 2009-2018, instead of just this year?

I don't know, is it? I don't know whether it makes a real difference.

Where's your taste for pedantry? Here's mine: please make it 2013-2018, because the oldest code moved is from 2013.

Quoted 11 lines
Quoted 11 lines

include/block/qdict.h line 15

Quoted 11 lines
include/block/qdict.h new file +11 -0
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ */
+
+#ifndef BLOCK_QDICT_H
+#define BLOCK_QDICT_H
+
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qobject.h"
+#include "qapi/error.h"

Only the first #include is necessary, due to qemu/typedefs.h. Please drop the other two.

Quoted 15 lines
Quoted 10 lines
Quoted 5 lines
include/block/qdict.h new file +4 -0
+
+
+void qdict_copy_default(QDict *dst, QDict *src, const char *key);
+void qdict_set_default_str(QDict *dst, const char *key, const char

*val);

These two might be useful outside of the block layer; we can move them back in a later patch if so.  But for now, I agree with your choice of moving them.

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+void qdict_join(QDict *dest, QDict *src, bool overwrite);

This is borderline whether it would be useful outside of the block layer.

Quoted 8 lines

I decided I wanted to move the *_default* functions, and if I did that, I would have to move this one as well. I decided I can't be biased just because I wrote this one. :-)

But in general I'd say if anyone wants to use any of these functions outside of the block layer, they are welcome to move them back to the main header, provided they have a good reason to do so. I suppose a good reason for using qdict_join() may indeed turn up sooner or later.

I like it this way.

With the trivial changes I asked for:

#24 lore raw

Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header

In reply to Max Reitz

MAINTAINERS files include/block/qdict.h under "Block layer core". Let's split qobject/qdict.c as well, so it's there, too. Obvious patch appended. Feel free to squash it into yours without giving me credit for it.

PS: I tried to move block-qdict.c to block/, but totally failed at persuading Make to link it into all programs that need it. Oh well.

Quoted 1 line

From 4614533375729c84c0599e1fed44420b29a180a6 Mon Sep 17 00:00:00 2001

From: Markus Armbruster <[email protected]>
Date: Wed, 6 Jun 2018 15:11:54 +0200
Subject: [PATCH] qobject: Move block-specific qdict code to block-qdict.c

---
 MAINTAINERS           |   1 +
 qobject/Makefile.objs |   1 +
 qobject/block-qdict.c | 637 ++++++++++++++++++++++++++++++++++++++++++
 qobject/qdict.c       | 629 -----------------------------------------
 4 files changed, 639 insertions(+), 629 deletions(-)
 create mode 100644 qobject/block-qdict.c
MAINTAINERS +1 -0
index 41cd3736a9..9f9835e052 100644
@@ -1365,6 +1365,7 @@ F: qemu-img*
 F: qemu-io*
 F: tests/qemu-iotests/
 F: util/qemu-progress.c
+F: qobject/block-qdict.c
 T: git git://repo.or.cz/qemu/kevin.git block
 
 Block I/O path
qobject/Makefile.objs +1 -0
index 002d25873a..7b12c9cacf 100644
@@ -1,2 +1,3 @@
 util-obj-y = qnull.o qnum.o qstring.o qdict.o qlist.o qbool.o qlit.o
 util-obj-y += qjson.o qobject.o json-lexer.o json-streamer.o json-parser.o
+util-obj-y += block-qdict.o
qobject/block-qdict.c new file +637 -0

637 lines not on this page · raw message

qobject/qdict.c +0 -629

643 lines not on this page · raw message

Signature · 1 line
2.17.1
#25 lore raw

Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header

Quoted 4 lines · Markus Armbruster

MAINTAINERS files include/block/qdict.h under "Block layer core". Let's split qobject/qdict.c as well, so it's there, too. Obvious patch appended. Feel free to squash it into yours without giving me credit for it.

Hmm, we should split tests/check-qdict.c for the same reason. Updated patch appended.

Quoted 5 lines

PS: I tried to move block-qdict.c to block/, but totally failed at persuading Make to link it into all programs that need it. Oh well.

From bdb7f6581a3066db853bca6c81cc2c42f6cec8e9 Mon Sep 17 00:00:00 2001

From: Markus Armbruster <[email protected]>
Date: Wed, 6 Jun 2018 15:11:54 +0200
Subject: [PATCH] qobject: Move block-specific qdict code to block-qdict.c

---
 MAINTAINERS               |   1 +
 qobject/Makefile.objs     |   1 +
 qobject/block-qdict.c     | 637 ++++++++++++++++++++++++++++++++++++
 qobject/qdict.c           | 629 ------------------------------------
 tests/Makefile.include    |   4 +
 tests/check-block-qdict.c | 657 ++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c       | 642 -------------------------------------
 7 files changed, 1300 insertions(+), 1271 deletions(-)
 create mode 100644 qobject/block-qdict.c
 create mode 100644 tests/check-block-qdict.c
MAINTAINERS +1 -0
index 41cd3736a9..9f9835e052 100644
@@ -1365,6 +1365,7 @@ F: qemu-img*
 F: qemu-io*
 F: tests/qemu-iotests/
 F: util/qemu-progress.c
+F: qobject/block-qdict.c
 T: git git://repo.or.cz/qemu/kevin.git block
 
 Block I/O path
qobject/Makefile.objs +1 -0
index 002d25873a..7b12c9cacf 100644
@@ -1,2 +1,3 @@
 util-obj-y = qnull.o qnum.o qstring.o qdict.o qlist.o qbool.o qlit.o
 util-obj-y += qjson.o qobject.o json-lexer.o json-streamer.o json-parser.o
+util-obj-y += block-qdict.o
qobject/block-qdict.c new file +637 -0

637 lines not on this page · raw message

qobject/qdict.c +0 -629

643 lines not on this page · raw message

tests/Makefile.include +4 -0
index 86f90c0cb0..fd0274b5dc 100644
@@ -40,6 +40,8 @@ SYSEMU_TARGET_LIST := $(subst -softmmu.mak,,$(notdir \
 
 check-unit-y = tests/check-qdict$(EXESUF)
 gcov-files-check-qdict-y = qobject/qdict.c
+check-unit-y = tests/check-block-qdict$(EXESUF)
+gcov-files-check-block-qdict-y = qobject/block-qdict.c
 check-unit-y += tests/test-char$(EXESUF)
 gcov-files-check-qdict-y = chardev/char.c
 check-unit-y += tests/check-qnum$(EXESUF)
@@ -582,6 +584,7 @@ GENERATED_FILES += tests/test-qapi-types.h tests/test-qapi-visit.h \
 test-obj-y = tests/check-qnum.o tests/check-qstring.o tests/check-qdict.o \
 	tests/check-qlist.o tests/check-qnull.o tests/check-qobject.o \
 	tests/check-qjson.o tests/check-qlit.o \
+	tests/check-block-qtest.o \
 	tests/test-coroutine.o tests/test-string-output-visitor.o \
 	tests/test-string-input-visitor.o tests/test-qobject-output-visitor.o \
 	tests/test-clone-visitor.o \
@@ -612,6 +615,7 @@ test-block-obj-y = $(block-obj-y) $(test-io-obj-y) tests/iothread.o
 tests/check-qnum$(EXESUF): tests/check-qnum.o $(test-util-obj-y)
 tests/check-qstring$(EXESUF): tests/check-qstring.o $(test-util-obj-y)
 tests/check-qdict$(EXESUF): tests/check-qdict.o $(test-util-obj-y)
+tests/check-block-qdict$(EXESUF): tests/check-block-qdict.o $(test-util-obj-y)
 tests/check-qlist$(EXESUF): tests/check-qlist.o $(test-util-obj-y)
 tests/check-qnull$(EXESUF): tests/check-qnull.o $(test-util-obj-y)
 tests/check-qobject$(EXESUF): tests/check-qobject.o $(test-util-obj-y)
tests/check-block-qdict.c new file +657 -0

657 lines not on this page · raw message

tests/check-qdict.c +0 -642

676 lines not on this page · raw message

Signature · 1 line
2.17.1
#26 lore raw

Re: [Qemu-devel] [PATCH 06/13] block: Add block-specific QDict header

Quoted 37 lines · Markus Armbruster
Quoted 4 lines · Markus Armbruster

MAINTAINERS files include/block/qdict.h under "Block layer core". Let's split qobject/qdict.c as well, so it's there, too. Obvious patch appended. Feel free to squash it into yours without giving me credit for it.

Hmm, we should split tests/check-qdict.c for the same reason. Updated patch appended.

Quoted 2 lines

PS: I tried to move block-qdict.c to block/, but totally failed at persuading Make to link it into all programs that need it. Oh well.

From bdb7f6581a3066db853bca6c81cc2c42f6cec8e9 Mon Sep 17 00:00:00 2001
From: Markus Armbruster <[email protected]>
Date: Wed, 6 Jun 2018 15:11:54 +0200
Subject: [PATCH] qobject: Move block-specific qdict code to block-qdict.c

---
 MAINTAINERS               |   1 +
 qobject/Makefile.objs     |   1 +
 qobject/block-qdict.c     | 637 ++++++++++++++++++++++++++++++++++++
 qobject/qdict.c           | 629 ------------------------------------
 tests/Makefile.include    |   4 +
 tests/check-block-qdict.c | 657 ++++++++++++++++++++++++++++++++++++++
 tests/check-qdict.c       | 642 -------------------------------------
 7 files changed, 1300 insertions(+), 1271 deletions(-)
 create mode 100644 qobject/block-qdict.c
 create mode 100644 tests/check-block-qdict.c
MAINTAINERS +0 -0
index 41cd3736a9..9f9835e052 100644

MAINTAINERS F: qemu-img* · line 1371

Quoted 7 lines
MAINTAINERS +1 -0
@@ -1365,6 +1365,7 @@ F: qemu-img*
 F: qemu-io*
 F: tests/qemu-iotests/
 F: util/qemu-progress.c
+F: qobject/block-qdict.c
 T: git git://repo.or.cz/qemu/kevin.git block
 
 Block I/O path

Forgot to update MAINTAINERS for tests/check-block-qdict.c. Fixup:

MAINTAINERS +1 -0
index 9f9835e052..21f4bc1806 100644
@@ -1366,6 +1366,7 @@ F: qemu-io*
 F: tests/qemu-iotests/
 F: util/qemu-progress.c
 F: qobject/block-qdict.c
+F: test/check-block-qdict.c
 T: git git://repo.or.cz/qemu/kevin.git block
 
 Block I/O path

[...]

author
#27 lore raw

PATCH 7/13 qdict: Add qdict_stringify_for_keyval()

In reply to Max Reitz

The purpose of this function is to prepare a QDict for consumption by the keyval visitor, which only accepts strings and QNull.

---
 include/block/qdict.h |  2 ++
 qobject/qdict.c       | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 59 insertions(+)
include/block/qdict.h +2 -0
index 825e096a72..05a24677b5 100644
@@ -32,4 +32,6 @@ typedef struct QDictRenames {
 } QDictRenames;
 bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp);
 
+void qdict_stringify_for_keyval(QDict *qdict);
+
 #endif
qobject/qdict.c +57 -0

60 lines not on this page · raw message

Signature · 1 line
2.14.3
Reviewed-by on v1 7/13
#28 lore raw

Re: [Qemu-devel] [PATCH 07/13] qdict: Add qdict_stringify_for_keyval()

Quoted 10 lines · Max Reitz

The purpose of this function is to prepare a QDict for consumption by the keyval visitor, which only accepts strings and QNull.

---
  include/block/qdict.h |  2 ++
  qobject/qdict.c       | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++
  2 files changed, 59 insertions(+)
Quoted 7 lines

+/**
+ * Convert all values in a QDict so it can be consumed by the keyval
+ * input visitor. QNull values are left as-is, all other values are
+ * converted to strings.
+ *
+ * @qdict must be flattened, i.e. it may not contain any nested QDicts
+ * or QLists.

Maybe s/flattened/flattened already/

or would it be worth letting this function PERFORM the flattening step automatically?

Quoted 18 lines

+ */

qobject/qdict.c +17 -0
+void qdict_stringify_for_keyval(QDict *qdict)
+{
+    const QDictEntry *e;
+
+    for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) {
+        QString *new_value = NULL;
+
+        switch (qobject_type(e->value)) {
+        case QTYPE_QNULL:
+            /* leave as-is */
+            break;
+
+        case QTYPE_QNUM: {
+            char *str = qnum_to_string(qobject_to(QNum, e->value));
+            new_value = qstring_from_str(str);
+            g_free(str);
+            break;

qobject/qdict.c bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp) · line 1125

Quoted 10 lines
qobject/qdict.c +10 -0
+        }
+
+        case QTYPE_QSTRING:
+            /* leave as-is */
+            break;
+
+        case QTYPE_QDICT:
+        case QTYPE_QLIST:
+            /* @qdict must be flattened */
+            abort();

Matches your documentation about requiring it to be already flattened.

Quoted 6 lines
qobject/qdict.c +6 -0
+
+        case QTYPE_QBOOL:
+            if (qbool_get_bool(qobject_to(QBool, e->value))) {
+                new_value = qstring_from_str("on");
+            } else {
+                new_value = qstring_from_str("off");

qobject/qdict.c bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp) · line 1141

Quoted 10 lines
qobject/qdict.c +10 -0
+            }
+            break;
+
+        case QTYPE_NONE:
+        case QTYPE__MAX:
+            abort();
+        }
+
+        if (new_value) {
+            QDictEntry *mut_e = (QDictEntry *)e;

A bit of a shame that you have to cast away const. It took me a couple reads to figure out this meant 'mutable_e'. But in the end, the code looks correct.

qobject/qdict.c bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp) · line 1143

Quoted 2 lines
qobject/qdict.c +2 -0
+            qobject_unref(mut_e->value);
+            mut_e->value = QOBJECT(new_value);

If we're okay requiring the caller to pre-flatten before calling this, instead of offering to do it here,

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#29 lore raw

Re: [Qemu-devel] [PATCH 07/13] qdict: Add qdict_stringify_for_keyval()

Quoted 20 lines · Eric Blake
Quoted 11 lines · Max Reitz

The purpose of this function is to prepare a QDict for consumption by the keyval visitor, which only accepts strings and QNull.

---
  include/block/qdict.h |  2 ++
  qobject/qdict.c       | 57
+++++++++++++++++++++++++++++++++++++++++++++++++++

  2 files changed, 59 insertions(+)
Quoted 7 lines

+/**
+ * Convert all values in a QDict so it can be consumed by the keyval
+ * input visitor.  QNull values are left as-is, all other values are
+ * converted to strings.
+ *
+ * @qdict must be flattened, i.e. it may not contain any nested QDicts
+ * or QLists.

Quoted 4 lines

Maybe s/flattened/flattened already/

or would it be worth letting this function PERFORM the flattening step automatically?

Possibly, but I don't think it would be any more efficient, so I'd rather leave it up to the caller to do it explicitly than to do it here and maybe surprise someone.

(Indeed I personally prefer to be surprised by an abort() than by some unintended data change.)

Max

Quoted 59 lines
Quoted 18 lines

+ */

qobject/qdict.c +17 -0
+void qdict_stringify_for_keyval(QDict *qdict)
+{
+    const QDictEntry *e;
+
+    for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) {
+        QString *new_value = NULL;
+
+        switch (qobject_type(e->value)) {
+        case QTYPE_QNULL:
+            /* leave as-is */
+            break;
+
+        case QTYPE_QNUM: {
+            char *str = qnum_to_string(qobject_to(QNum, e->value));
+            new_value = qstring_from_str(str);
+            g_free(str);
+            break;

qobject/qdict.c bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp) · line 1125

Quoted 10 lines
qobject/qdict.c +10 -0
+        }
+
+        case QTYPE_QSTRING:
+            /* leave as-is */
+            break;
+
+        case QTYPE_QDICT:
+        case QTYPE_QLIST:
+            /* @qdict must be flattened */
+            abort();

Matches your documentation about requiring it to be already flattened.

Quoted 6 lines
qobject/qdict.c +6 -0
+
+        case QTYPE_QBOOL:
+            if (qbool_get_bool(qobject_to(QBool, e->value))) {
+                new_value = qstring_from_str("on");
+            } else {
+                new_value = qstring_from_str("off");

qobject/qdict.c bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp) · line 1141

Quoted 10 lines
qobject/qdict.c +10 -0
+            }
+            break;
+
+        case QTYPE_NONE:
+        case QTYPE__MAX:
+            abort();
+        }
+
+        if (new_value) {
+            QDictEntry *mut_e = (QDictEntry *)e;

A bit of a shame that you have to cast away const. It took me a couple reads to figure out this meant 'mutable_e'.  But in the end, the code looks correct.

Quoted 2 lines

+            qobject_unref(mut_e->value);
+            mut_e->value = QOBJECT(new_value);

If we're okay requiring the caller to pre-flatten before calling this, instead of offering to do it here,

author
#30 lore raw

PATCH 8/13 tests: Add qdict_stringify_for_keyval() test

In reply to Max Reitz
---
 tests/check-qdict.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
tests/check-qdict.c +54 -0
index 93e2112b6d..ef5d17f815 100644
@@ -973,6 +973,57 @@ static void qdict_stress_test(void)
     qobject_unref(qdict);
 }
 
+static void qdict_stringify_for_keyval_test(void)
+{
+    QDict *dict = qdict_new();
+
+    /*
+     * Test stringification of:
+     *
+     * {
+     *     "a": "null",
+     *     "b": 42,
+     *     "c": -23,
+     *     "d": false,
+     *     "e": null,
+     *     "f": "",
+     *     "g": 0.5,
+     *     "h": 0xffffffffffffffff,
+     *     "i": true,
+     *     "j": 0
+     *  }
+     *
+     *  Note that null is not transformed into a string and remains a
+     *  QNull.
+     */
+
+    qdict_put_str(dict, "a", "null");
+    qdict_put_int(dict, "b", 42);
+    qdict_put_int(dict, "c", -23);
+    qdict_put_bool(dict, "d", false);
+    qdict_put_null(dict, "e");
+    qdict_put_str(dict, "f", "");
+    qdict_put(dict, "g", qnum_from_double(0.5));
+    qdict_put(dict, "h", qnum_from_uint(0xffffffffffffffffull));
+    qdict_put_bool(dict, "i", true);
+    qdict_put_int(dict, "j", 0);
+
+    qdict_stringify_for_keyval(dict);
+
+    g_assert(!strcmp(qdict_get_str(dict, "a"), "null"));
+    g_assert(!strcmp(qdict_get_str(dict, "b"), "42"));
+    g_assert(!strcmp(qdict_get_str(dict, "c"), "-23"));
+    g_assert(!strcmp(qdict_get_str(dict, "d"), "off"));
+    g_assert(qobject_type(qdict_get(dict, "e")) == QTYPE_QNULL);
+    g_assert(!strcmp(qdict_get_str(dict, "f"), ""));
+    g_assert(!strcmp(qdict_get_str(dict, "g"), "0.5"));
+    g_assert(!strcmp(qdict_get_str(dict, "h"), "18446744073709551615"));
+    g_assert(!strcmp(qdict_get_str(dict, "i"), "on"));
+    g_assert(!strcmp(qdict_get_str(dict, "j"), "0"));
+
+    qobject_unref(dict);
+}
+
 int main(int argc, char **argv)
 {
     g_test_init(&argc, &argv, NULL);
@@ -1010,6 +1061,9 @@ int main(int argc, char **argv)
 
     g_test_add_func("/public/rename_keys", qdict_rename_keys_test);
 
+    g_test_add_func("/public/stringify_for_keyval",
+                    qdict_stringify_for_keyval_test);
+
     /* The Big one */
     if (g_test_slow()) {
         g_test_add_func("/stress/test", qdict_stress_test);
Signature · 1 line
2.14.3
Reviewed-by on v1 8/13
#31 lore raw

Re: [Qemu-devel] [PATCH 08/13] tests: Add qdict_stringify_for_keyval() test

Quoted 14 lines · Max Reitz
---
  tests/check-qdict.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 54 insertions(+)
tests/check-qdict.c +8 -0
+static void qdict_stringify_for_keyval_test(void)
+{
+    QDict *dict = qdict_new();
+
+    /*
+     * Test stringification of:
+     *
+     * {

tests/check-qdict.c static void qdict_stress_test(void) · line 993

Quoted 10 lines
tests/check-qdict.c +10 -0
+     *     "a": "null",
+     *     "b": 42,
+     *     "c": -23,
+     *     "d": false,
+     *     "e": null,
+     *     "f": "",
+     *     "g": 0.5,
+     *     "h": 0xffffffffffffffff,
+     *     "i": true,
+     *     "j": 0

Is it worth testing fun things like '-0.0'?

tests/check-qdict.c static void qdict_stress_test(void) · line 1017

Quoted 5 lines
tests/check-qdict.c +5 -0
+    g_assert(!strcmp(qdict_get_str(dict, "a"), "null"));
+    g_assert(!strcmp(qdict_get_str(dict, "b"), "42"));
+    g_assert(!strcmp(qdict_get_str(dict, "c"), "-23"));
+    g_assert(!strcmp(qdict_get_str(dict, "d"), "off"));
+    g_assert(qobject_type(qdict_get(dict, "e")) == QTYPE_QNULL);

Is it worth shortening this line to:
g_assert(qobject_to(QNull, qdict_get(dict, "e")));

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#32 lore raw

Re: [Qemu-devel] [PATCH 08/13] tests: Add qdict_stringify_for_keyval() test

Quoted 20 lines · Eric Blake
Quoted 19 lines · Max Reitz

---
  tests/check-qdict.c | 54
+++++++++++++++++++++++++++++++++++++++++++++++++++++

  1 file changed, 54 insertions(+)
tests/check-qdict.c +12 -0
+static void qdict_stringify_for_keyval_test(void)
+{
+    QDict *dict = qdict_new();
+
+    /*
+     * Test stringification of:
+     *
+     * {
+     *     "a": "null",
+     *     "b": 42,
+     *     "c": -23,
+     *     "d": false,
Quoted 8 lines
Quoted 6 lines

+     *     "e": null,
+     *     "f": "",
+     *     "g": 0.5,
+     *     "h": 0xffffffffffffffff,
+     *     "i": true,
+     *     "j": 0

Is it worth testing fun things like '-0.0'?

Sure, why not. Maybe even infinity, although I'm not quite sure the input visitor can handle it...

Quoted 8 lines
Quoted 5 lines

+    g_assert(!strcmp(qdict_get_str(dict, "a"), "null")); +    g_assert(!strcmp(qdict_get_str(dict, "b"), "42")); +    g_assert(!strcmp(qdict_get_str(dict, "c"), "-23")); +    g_assert(!strcmp(qdict_get_str(dict, "d"), "off")); +    g_assert(qobject_type(qdict_get(dict, "e")) == QTYPE_QNULL);

Is it worth shortening this line to:
g_assert(qobject_to(QNull, qdict_get(dict, "e")));

I think explicitly checking the type is a bit more expressive.

Max

Quoted 1 line
#33 lore raw

Re: [Qemu-devel] [PATCH 08/13] tests: Add qdict_stringify_for_keyval() test

Quoted 8 lines · Max Reitz
Quoted 5 lines
Quoted 3 lines

+     *     "h": 0xffffffffffffffff,
+     *     "i": true,
+     *     "j": 0

Is it worth testing fun things like '-0.0'?

Sure, why not. Maybe even infinity, although I'm not quite sure the input visitor can handle it...

JSON can't handle Inf or NaN, even if the input visitor can. So probably best to not worry about those.

Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

+    g_assert(!strcmp(qdict_get_str(dict, "a"), "null")); +    g_assert(!strcmp(qdict_get_str(dict, "b"), "42")); +    g_assert(!strcmp(qdict_get_str(dict, "c"), "-23")); +    g_assert(!strcmp(qdict_get_str(dict, "d"), "off")); +    g_assert(qobject_type(qdict_get(dict, "e")) == QTYPE_QNULL);

Is it worth shortening this line to:
g_assert(qobject_to(QNull, qdict_get(dict, "e")));

I think explicitly checking the type is a bit more expressive.

Okay (qobject_to() checks the type under the hood, and returns non-NULL only when it was the right type - but I see your point about that being a bit more magic)

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#34 lore raw

PATCH 9/13 qdict: Make qdict_flatten() shallow-clone-friendly

In reply to Max Reitz

In its current form, qdict_flatten() removes all entries from nested QDicts that are moved to the root QDict. It is completely sufficient to remove all old entries from the root QDict, however. If the nested dicts have a refcount of 1, this will automatically delete them, too. And if they have a greater refcount, we probably do not want to modify them in the first place.

The latter observation means that it was currently (in general) impossible to qdict_flatten() a shallowly cloned dict because that would empty nested QDicts in the original dict as well. This patch changes this, so you can now use qdict_flatten(qdict_shallow_clone(dict)) to get a flattened copy without disturbing the original.

---
 qobject/qdict.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
qobject/qdict.c +16 -7
index 83678c4951..63dcb43e73 100644
@@ -537,7 +537,7 @@ static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix)
     QObject *value;
     const QDictEntry *entry, *next;
     char *new_key;
-    bool delete;
+    bool copied_value;
 
     entry = qdict_first(qdict);
 
@@ -546,7 +546,7 @@ static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix)
         next = qdict_next(qdict, entry);
         value = qdict_entry_value(entry);
         new_key = NULL;
-        delete = false;
+        copied_value = false;
 
         if (prefix) {
             new_key = g_strdup_printf("%s.%s", prefix, entry->key);
@@ -557,23 +557,32 @@ static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix)
              * itself disappears. */
             qdict_flatten_qdict(qobject_to(QDict, value), target,
                                 new_key ? new_key : entry->key);
-            delete = true;
+            copied_value = true;
         } else if (qobject_type(value) == QTYPE_QLIST) {
             qdict_flatten_qlist(qobject_to(QList, value), target,
                                 new_key ? new_key : entry->key);
-            delete = true;
+            copied_value = true;
         } else if (prefix) {
             /* All other objects are moved to the target unchanged. */
             qdict_put_obj(target, new_key, qobject_ref(value));
-            delete = true;
+            copied_value = true;
         }
 
         g_free(new_key);
 
-        if (delete) {
+        if (copied_value && qdict == target) {
+            /* If we have copied a value, and if we are on the root
+             * level, we need to remove the old entry.  Otherwise, we
+             * do not, because by removing these entries on the root
+             * level, the reference counts of nested dicts and listed
+             * will be reduced automatically.  In fact, we probably do
+             * not want to modify nested dicts and lists with
+             * refcounts greater than 1 anyway. */
             qdict_del(qdict, entry->key);
 
-            /* Restart loop after modifying the iterated QDict */
+            /* Restart loop after modifying the iterated QDict.  We
+             * only need to do this if qdict == target, because
+             * otherwise copying the value did not affect qdict. */
             entry = qdict_first(qdict);
             continue;
         }
Signature · 1 line
2.14.3
Reviewed-by on v1 9/13
#35 lore raw

Re: [Qemu-devel] [PATCH 09/13] qdict: Make qdict_flatten() shallow-clone-friendly

Quoted 22 lines · Max Reitz

In its current form, qdict_flatten() removes all entries from nested QDicts that are moved to the root QDict. It is completely sufficient to remove all old entries from the root QDict, however. If the nested dicts have a refcount of 1, this will automatically delete them, too. And if they have a greater refcount, we probably do not want to modify them in the first place.

The latter observation means that it was currently (in general) impossible to qdict_flatten() a shallowly cloned dict because that would empty nested QDicts in the original dict as well. This patch changes this, so you can now use qdict_flatten(qdict_shallow_clone(dict)) to get a flattened copy without disturbing the original.

---
  qobject/qdict.c | 23 ++++++++++++++++-------
  1 file changed, 16 insertions(+), 7 deletions(-)
          g_free(new_key);

qobject/qdict.c static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix) · line 577

Quoted 6 lines
qobject/qdict.c +5 -1
-        if (delete) {
+        if (copied_value && qdict == target) {
+            /* If we have copied a value, and if we are on the root
+             * level, we need to remove the old entry.  Otherwise, we
+             * do not, because by removing these entries on the root
+             * level, the reference counts of nested dicts and listed

s/listed/lists/

qobject/qdict.c static void qdict_flatten_qdict(QDict *qdict, QDict *target, const char *prefix) · line 588

Quoted 13 lines
qobject/qdict.c +6 -1
+             * will be reduced automatically.  In fact, we probably do
+             * not want to modify nested dicts and lists with
+             * refcounts greater than 1 anyway. */
              qdict_del(qdict, entry->key);
 
-            /* Restart loop after modifying the iterated QDict */
+            /* Restart loop after modifying the iterated QDict.  We
+             * only need to do this if qdict == target, because
+             * otherwise copying the value did not affect qdict. */
              entry = qdict_first(qdict);
              continue;
          }

With the typo fix,

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#36 lore raw

PATCH 10/13 tests: Add QDict clone-flatten test

In reply to Max Reitz

This new test verifies that qdict_flatten() does not modify a shallow clone of the given QDict.

---
 tests/check-qdict.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
tests/check-qdict.c +33 -0
index ef5d17f815..7960b4a385 100644
@@ -329,6 +329,38 @@ static void qdict_flatten_test(void)
     qobject_unref(dict3);
 }
 
+static void qdict_clone_flatten_test(void)
+{
+    QDict *dict1 = qdict_new();
+    QDict *dict2 = qdict_new();
+    QDict *cloned_dict1;
+
+    /*
+     * Test that we can clone and flatten
+     *    { "a": { "b": 42 } }
+     * without modifying the clone.
+     */
+
+    qdict_put_int(dict2, "b", 42);
+    qdict_put(dict1, "a", dict2);
+
+    cloned_dict1 = qdict_clone_shallow(dict1);
+
+    qdict_flatten(dict1);
+
+    g_assert(qdict_size(dict1) == 1);
+    g_assert(qdict_get_int(dict1, "a.b") == 42);
+
+    g_assert(qdict_size(cloned_dict1) == 1);
+    g_assert(qdict_get_qdict(cloned_dict1, "a") == dict2);
+
+    g_assert(qdict_size(dict2) == 1);
+    g_assert(qdict_get_int(dict2, "b") == 42);
+
+    qobject_unref(dict1);
+    qobject_unref(cloned_dict1);
+}
+
 static void qdict_array_split_test(void)
 {
     QDict *test_dict = qdict_new();
@@ -1045,6 +1077,7 @@ int main(int argc, char **argv)
     g_test_add_func("/public/to_qdict", qobject_to_qdict_test);
     g_test_add_func("/public/iterapi", qdict_iterapi_test);
     g_test_add_func("/public/flatten", qdict_flatten_test);
+    g_test_add_func("/public/clone-flatten", qdict_clone_flatten_test);
     g_test_add_func("/public/array_split", qdict_array_split_test);
     g_test_add_func("/public/array_entries", qdict_array_entries_test);
     g_test_add_func("/public/join", qdict_join_test);
Signature · 1 line
2.14.3
Reviewed-by on v1 10/13
#37 lore raw

Re: [Qemu-devel] [PATCH 10/13] tests: Add QDict clone-flatten test

Quoted 8 lines · Max Reitz

This new test verifies that qdict_flatten() does not modify a shallow clone of the given QDict.

---
  tests/check-qdict.c | 33 +++++++++++++++++++++++++++++++++
  1 file changed, 33 insertions(+)

I'm not sure I even want to know how long it took you to debug the crash that you obviously hit before adding the fix in 9/13 plus this test ;)

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#38 lore raw

Re: [Qemu-devel] [PATCH 10/13] tests: Add QDict clone-flatten test

Quoted 4 lines · Eric Blake
Quoted 3 lines · Max Reitz

This new test verifies that qdict_flatten() does not modify a shallow clone of the given QDict.

Quoted 8 lines
Quoted 5 lines

---
  tests/check-qdict.c | 33 +++++++++++++++++++++++++++++++++

  1 file changed, 33 insertions(+)

I'm not sure I even want to know how long it took you to debug the crash that you obviously hit before adding the fix in 9/13 plus this test ;)

Thank you very much, I made myself forget about that trauma already.

In short, I wondered why the whole thing worked for null-co directly:

driver=null-co,size=512
=> {"driver": "null-co", "size": 512}

But not for null-co through raw:

driver=raw,file.driver=null-co,file.size=512
=> {"file": {}}

(Or something like that, I don't remember exactly.)

With some debugging sprinkled into block.c, I could see that the correct options were there on the null-co level... But for some reason they disappeared one level above.

Then I recalled that dict cloning is just a shallow cloning and looked for the culprit...

Of course, in reality, much more cursing was involved.

Max

author
#39 lore raw

PATCH 11/13 block: Try to create well typed json:{} filenames

In reply to Max Reitz

By applying a health mix of qdict_flatten(), qdict_crumple(), qdict_stringify_for_keyval(), the keyval input visitor, and the QObject output visitor (not necessarily in that order), we can at least try to bring bs->full_open_options into accordance with the QAPI schema. This may not always work (there are some options left that have not been QAPI-fied yet), but in practice it usually will.

In any case, sometimes emitting wrongly typed json:{} filenames is better than doing it effectively half the time.

This affects some iotests because json:{} filenames are now usually crumpled.

---
 block.c                    | 69 +++++++++++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/059.out |  2 +-
 tests/qemu-iotests/099.out |  4 +--
 tests/qemu-iotests/110.out |  2 +-
 tests/qemu-iotests/198.out |  4 +--
 tests/qemu-iotests/207.out | 10 +++----
 6 files changed, 79 insertions(+), 12 deletions(-)
block.c +68 -1
index 3c3e8fd11d..9e4a6c0d30 100644
@@ -36,6 +36,7 @@
 #include "qapi/qmp/qjson.h"
 #include "qapi/qmp/qnull.h"
 #include "qapi/qmp/qstring.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/qobject-output-visitor.h"
 #include "qapi/qapi-visit-block-core.h"
 #include "sysemu/block-backend.h"
@@ -5143,6 +5144,59 @@ static bool append_open_options(QDict *d, BlockDriverState *bs)
     return found_any;
 }
 
+/**
+ * Take a blockdev @options QDict and convert its values to the
+ * correct type.
+ *
+ * Fail if @options does not match the QAPI schema of BlockdevOptions.
+ *
+ * In case of failure, return NULL and set @errp.
+ *
+ * In case of success, return a correctly typed new QDict.
+ */
+static QDict *bdrv_type_blockdev_opts(const QDict *options, Error **errp)
+{
+    Visitor *v;
+    BlockdevOptions *blockdev_options;
+    QObject *typed_opts, *crumpled_opts;
+    QDict *string_options;
+    Error *local_err = NULL;
+
+    string_options = qdict_clone_shallow(options);
+
+    qdict_flatten(string_options);
+    qdict_stringify_for_keyval(string_options);
+    crumpled_opts = qdict_crumple(string_options, errp);
+    qobject_unref(string_options);
+    if (!crumpled_opts) {
+        error_prepend(errp, "Failed to crumple options: ");
+        return NULL;
+    }
+
+    v = qobject_input_visitor_new_keyval(crumpled_opts);
+    visit_type_BlockdevOptions(v, NULL, &blockdev_options, &local_err);
+    visit_free(v);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        error_prepend(errp, "Not a valid BlockdevOptions object: ");
+        return NULL;
+    }
+
+    v = qobject_output_visitor_new(&typed_opts);
+    visit_type_BlockdevOptions(v, NULL, &blockdev_options, &local_err);
+    if (!local_err) {
+        visit_complete(v, &typed_opts);
+    }
+    visit_free(v);
+    qapi_free_BlockdevOptions(blockdev_options);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return NULL;
+    }
+
+    return qobject_to(QDict, typed_opts);
+}
+
 /* Updates the following BDS fields:
  *  - exact_filename: A filename which may be used for opening a block device
  *                    which (mostly) equals the given BDS (even without any
@@ -5244,10 +5298,23 @@ void bdrv_refresh_filename(BlockDriverState *bs)
     if (bs->exact_filename[0]) {
         pstrcpy(bs->filename, sizeof(bs->filename), bs->exact_filename);
     } else if (bs->full_open_options) {
-        QString *json = qobject_to_json(QOBJECT(bs->full_open_options));
+        QString *json;
+        QDict *typed_opts, *json_opts;
+
+        typed_opts = bdrv_type_blockdev_opts(bs->full_open_options, NULL);
+
+        /* We cannot be certain that bs->full_open_options matches
+         * BlockdevOptions, so bdrv_type_blockdev_opts() may fail.
+         * That is not fatal, we can just emit bs->full_open_options
+         * directly -- qemu will accept that, even if it does not
+         * match the schema. */
+        json_opts = typed_opts ?: bs->full_open_options;
+
+        json = qobject_to_json(QOBJECT(json_opts));
         snprintf(bs->filename, sizeof(bs->filename), "json:%s",
                  qstring_get_str(json));
         qobject_unref(json);
+        qobject_unref(typed_opts);
     }
 }
 
tests/qemu-iotests/059.out +1 -1
index f6dce7947c..0238b9e9a8 100644
@@ -2050,7 +2050,7 @@ wrote 512/512 bytes at offset 10240
 
 === Testing monolithicFlat with internally generated JSON file name ===
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 subformat=monolithicFlat
-can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "read_aio"}'
+can't open: Cannot use relative extent paths with VMDK descriptor file 'json:{"inject-error": [{"event": "read_aio"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}'
 
 === Testing version 3 ===
 image: TEST_DIR/iotest-version3.IMGFMT
tests/qemu-iotests/099.out +2 -2
index 8cce627529..0a9c434148 100644
@@ -12,11 +12,11 @@ blkverify:TEST_DIR/t.IMGFMT.compare:TEST_DIR/t.IMGFMT
 
 === Testing JSON filename for blkdebug ===
 
-json:{"driver": "IMGFMT", "file": {"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "l1_update"}}
+json:{"driver": "IMGFMT", "file": {"inject-error": [{"event": "l1_update"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}}
 
 === Testing indirectly enforced JSON filename ===
 
-json:{"driver": "raw", "file": {"test": {"driver": "IMGFMT", "file": {"image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "inject-error.0.event": "l1_update"}}, "driver": "blkverify", "raw": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.compare"}}}
+json:{"driver": "raw", "file": {"test": {"driver": "IMGFMT", "file": {"inject-error": [{"event": "l1_update"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}}, "driver": "blkverify", "raw": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.compare"}}}
 
 === Testing plain filename for blkdebug ===
 
tests/qemu-iotests/110.out +1 -1
index b3584ff87f..fa19315dfb 100644
@@ -11,7 +11,7 @@ backing file: t.IMGFMT.base (actual path: TEST_DIR/t.IMGFMT.base)
 
 === Non-reconstructable filename ===
 
-image: json:{"driver": "IMGFMT", "file": {"set-state.0.event": "read_aio", "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug", "set-state.0.new_state": 42}}
+image: json:{"driver": "IMGFMT", "file": {"set-state": [{"new_state": 42, "event": "read_aio"}], "image": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}, "driver": "blkdebug"}}
 file format: IMGFMT
 virtual size: 64M (67108864 bytes)
 backing file: t.IMGFMT.base (cannot determine actual path)
tests/qemu-iotests/198.out +2 -2
index adb805cce9..2a02077d0b 100644
@@ -32,7 +32,7 @@ read 16777216/16777216 bytes at offset 0
 16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 == checking image base ==
-image: json:{"encrypt.key-secret": "sec0", "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.base"}}
+image: json:{"driver": "IMGFMT", "encrypt": {"key-secret": "sec0"}, "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT.base"}}
 file format: IMGFMT
 virtual size: 16M (16777216 bytes)
 Format specific information:
@@ -74,7 +74,7 @@ Format specific information:
         master key iters: 1024
 
 == checking image layer ==
-image: json:{"encrypt.key-secret": "sec1", "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}}
+image: json:{"driver": "IMGFMT", "encrypt": {"key-secret": "sec1"}, "file": {"driver": "file", "filename": "TEST_DIR/t.IMGFMT"}}
 file format: IMGFMT
 virtual size: 16M (16777216 bytes)
 backing file: TEST_DIR/t.IMGFMT.base
tests/qemu-iotests/207.out +5 -5
index 417deee970..effb74bf02 100644
@@ -9,7 +9,7 @@ QMP_VERSION
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}}
 
-image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}}
+image: json:{"driver": "IMGFMT", "file": {"driver": "ssh", "path": "TEST_DIR/t.IMGFMT", "server": { "port": "22", "host": "127.0.0.1" }}}
 file format: IMGFMT
 virtual size: 4.0M (4194304 bytes)
 
@@ -26,7 +26,7 @@ QMP_VERSION
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}}
 
-image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}}
+image: json:{"driver": "IMGFMT", "file": {"driver": "ssh", "path": "TEST_DIR/t.IMGFMT", "server": { "port": "22", "host": "127.0.0.1" }}}
 file format: IMGFMT
 virtual size: 8.0M (8388608 bytes)
 Testing:
@@ -36,7 +36,7 @@ QMP_VERSION
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}}
 
-image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}}
+image: json:{"driver": "IMGFMT", "file": {"driver": "ssh", "path": "TEST_DIR/t.IMGFMT", "server": { "port": "22", "host": "127.0.0.1" }}}
 file format: IMGFMT
 virtual size: 4.0M (4194304 bytes)
 Testing:
@@ -47,7 +47,7 @@ QMP_VERSION
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}}
 
-image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}}
+image: json:{"driver": "IMGFMT", "file": {"driver": "ssh", "path": "TEST_DIR/t.IMGFMT", "server": { "port": "22", "host": "127.0.0.1" }}}
 file format: IMGFMT
 virtual size: 8.0M (8388608 bytes)
 Testing:
@@ -58,7 +58,7 @@ QMP_VERSION
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}}
 
-image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}}
+image: json:{"driver": "IMGFMT", "file": {"driver": "ssh", "path": "TEST_DIR/t.IMGFMT", "server": { "port": "22", "host": "127.0.0.1" }}}
 file format: IMGFMT
 virtual size: 4.0M (4194304 bytes)
 
Signature · 1 line
2.14.3
author
#40 lore raw

PATCH 12/13 iotests: Test internal option typing

In reply to Max Reitz

It would be nice if qemu used the correct types for blockdev options internally, even if the user specified string values (either through -drive or by being not so nice and using json:{} with string values). This patch adds a test verifying that fact.

---
 tests/qemu-iotests/089     | 25 +++++++++++++++++++++++++
 tests/qemu-iotests/089.out |  9 +++++++++
 2 files changed, 34 insertions(+)
tests/qemu-iotests/089 +25 -0
index aa1ba4a98e..0682d08f39 100755
@@ -36,6 +36,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 # get standard environment, filters and checks
 . ./common.rc
 . ./common.filter
+. ./common.qemu
 
 _supported_fmt qcow2
 _supported_proto file
@@ -145,6 +146,30 @@ $QEMU_IO -c "open -o driver=qcow2 json:{\"file.filename\":\"$TEST_IMG\"}" \
 $QEMU_IO -c "open -o driver=qcow2 json:{\"driver\":\"raw\",\"file.filename\":\"$TEST_IMG\"}" \
          -c "info" 2>&1 | _filter_img_info
 
+echo
+echo "=== Testing option typing ==="
+echo
+
+# json:{} accepts both strings and correctly typed values (even mixed,
+# although we probably do not want to support that...), but when
+# creating a json:{} filename, it should be correctly typed.
+# Therefore, both of these should make the "size" value an integer.
+
+TEST_IMG="json:{'driver': 'null-co', 'size':  42 }" _img_info | grep '^image'
+TEST_IMG="json:{'driver': 'null-co', 'size': '42'}" _img_info | grep '^image'
+
+echo
+
+# This should even work when some driver abuses bs->options to store
+# non-QAPI options (and the given -drive options are not complete)
+# (Watch for whether file.align appears as an int or a string)
+qemu_comm_method=monitor _launch_qemu \
+    -drive if=none,id=drv0,node-name=node0,format=raw,file=blkdebug::null-co://,file.align=512
+
+_send_qemu_cmd $QEMU_HANDLE 'info block' 'json:'
+
+_cleanup_qemu
+
 
 # success, all done
 echo "*** done"
tests/qemu-iotests/089.out +9 -0
index 89e3e4340a..35ffbabe77 100644
@@ -49,4 +49,13 @@ vm state offset: 512 MiB
 format name: IMGFMT
 cluster size: 64 KiB
 vm state offset: 512 MiB
+
+=== Testing option typing ===
+
+image: json:{"driver": "null-co", "size": 42}
+image: json:{"driver": "null-co", "size": 42}
+
+QEMU X.Y.Z monitor - type 'help' for more information
+(qemu) info block
+drv0 (node0): json:{"driver": "raw", "file": {"image": {"driver": "null-co"}, "driver": "blkdebug", "align": 512}} (raw)
 *** done
Signature · 1 line
2.14.3
author
#41 lore raw

PATCH 13/13 iotests: qcow2's encrypt.format is now optional

In reply to Max Reitz

Remove the encrypt.format option from the two blockdev-add test cases for encrypted qcow2 images in 087 to explicitly test that this parameter is now optional.

---
 tests/qemu-iotests/087 | 2 --
 1 file changed, 2 deletions(-)
tests/qemu-iotests/087 +0 -2
index 2561a14456..1eba457985 100755
@@ -150,7 +150,6 @@ run_qemu <<EOF
           "filename": "$TEST_IMG"
       },
       "encrypt": {
-          "format": "aes",
           "key-secret": "sec0"
       }
     }
@@ -183,7 +182,6 @@ run_qemu <<EOF
           "filename": "$TEST_IMG"
       },
       "encrypt": {
-        "format": "luks",
         "key-secret": "sec0"
       }
     }
Signature · 1 line
2.14.3