Skip to contentSkip to conversation

Exposing backing-chain allocation over NBD

PATCH Superseded Eric Blake 4 participants 31 messages Oct 23, 2020 – Oct 26, 2020 Block layer

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

v5 cover Exposing backing-chain allocation over NBD

v4 was here:
https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg02708.html

Since then:
- rebase to master
- patches 1, 2, and 12 are new based on Vladimir's observation of QAPI_LIST_ADD
- patches 10-11 are new based on prior discussion on exposing actual depth in addition to a tri-state encoding
- patch 3 has a nasty bug fixed that was causing iotest failures
- patch 6 updated to take advantage of patch 2
- other minor tweaks based on review comments

001/12:[down] 'qapi: Move GenericList to qapi/util.h' 002/12:[down] 'qapi: Make QAPI_LIST_ADD() public'
003/12:[0002] [FC] 'nbd: Utilize QAPI_CLONE for type conversion' 004/12:[0010] [FC] 'nbd: Add new qemu:allocation-depth metadata context' 005/12:[----] [--] 'nbd: Add 'qemu-nbd -A' to expose allocation depth' 006/12:[0014] [FC] 'nbd: Update qapi to support exporting multiple bitmaps' 007/12:[----] [--] 'nbd: Simplify qemu bitmap context name' 008/12:[----] [--] 'nbd: Refactor counting of metadata contexts' 009/12:[0017] [FC] 'nbd: Allow export of multiple bitmaps for one device' 010/12:[down] 'block: Return depth level during bdrv_is_allocated_above' 011/12:[down] 'nbd: Expose actual depth in qemu:allocation-depth' 012/12:[down] 'qapi: Use QAPI_LIST_ADD() where possible'

Eric Blake (12):
qapi: Move GenericList to qapi/util.h
qapi: Make QAPI_LIST_ADD() public
nbd: Utilize QAPI_CLONE for type conversion
nbd: Add new qemu:allocation-depth metadata context
nbd: Add 'qemu-nbd -A' to expose allocation depth
nbd: Update qapi to support exporting multiple bitmaps
nbd: Simplify qemu bitmap context name
nbd: Refactor counting of metadata contexts
nbd: Allow export of multiple bitmaps for one device
block: Return depth level during bdrv_is_allocated_above
nbd: Expose actual depth in qemu:allocation-depth
qapi: Use QAPI_LIST_ADD() where possible

 docs/devel/writing-qmp-commands.txt |  13 +-
 docs/interop/nbd.txt                |  31 +++-
 docs/system/deprecated.rst          |   4 +-
 docs/tools/qemu-nbd.rst             |   8 +-
 qapi/block-core.json                |   7 +-
 qapi/block-export.json              |  22 ++-
 include/qapi/visitor.h              |   9 +-
 hw/net/rocker/rocker_fp.h           |   2 +-
 include/block/nbd.h                 |  14 +-
 include/qapi/util.h                 |  16 ++
 block/io.c                          |  11 +-
 block.c                             |  15 +-
 block/commit.c                      |   2 +-
 block/gluster.c                     |  19 +--
 block/mirror.c                      |   2 +-
 block/stream.c                      |   2 +-
 blockdev-nbd.c                      |  28 ++--
 chardev/char.c                      |  21 ++-
 hw/core/machine.c                   |   6 +-
 hw/net/rocker/rocker.c              |   8 +-
 hw/net/rocker/rocker_fp.c           |  14 +-
 hw/net/virtio-net.c                 |  21 +--
 migration/migration.c               |   7 +-
 migration/postcopy-ram.c            |   7 +-
 monitor/hmp-cmds.c                  |  11 +-
 nbd/server.c                        | 222 ++++++++++++++++++++++------
 qemu-img.c                          |   5 +-
 qemu-nbd.c                          |  30 ++--
 qga/commands-posix.c                |  13 +-
 qga/commands-win32.c                |  17 +--
 qga/commands.c                      |   6 +-
 qom/qom-qmp-cmds.c                  |  29 +---
 target/arm/helper.c                 |   6 +-
 target/arm/monitor.c                |  13 +-
 target/i386/cpu.c                   |   6 +-
 target/mips/helper.c                |   6 +-
 target/s390x/cpu_models.c           |  12 +-
 tests/test-clone-visitor.c          |   7 +-
 tests/test-qobject-output-visitor.c |  42 +++---
 tests/test-visitor-serialization.c  |   5 +-
 trace/qmp.c                         |  22 ++-
 ui/vnc.c                            |  21 +--
 util/qemu-config.c                  |  14 +-
 target/ppc/translate_init.c.inc     |  12 +-
 tests/qemu-iotests/291              |   6 +-
 tests/qemu-iotests/309              |  73 +++++++++
 tests/qemu-iotests/309.out          |  22 +++
 tests/qemu-iotests/group            |   1 +
 48 files changed, 529 insertions(+), 361 deletions(-)
 create mode 100755 tests/qemu-iotests/309
 create mode 100644 tests/qemu-iotests/309.out
Signature · 1 line
2.29.0
author
#1 lore raw

PATCH v5 1/12 qapi: Move GenericList to qapi/util.h

Placing GenericList in util.h will make it easier for the next patch to promote QAPI_LIST_ADD() into a public macro without requiring more files to include the unrelated visitor.h.

However, we can't also move GenericAlternate; this is because it would introduce a circular dependency: qapi-builtin-types.h needs a complete definition of QEnumLookup (so it includes qapi/util.h), and GenericAlternate needs a complete definition of QType (declared in qapi-builtin-types.h). Leaving GenericAlternate in visitor.h breaks the cycle, and doesn't matter since we don't have any further planned uses for that type outside of visitors.

---
 include/qapi/visitor.h | 9 +--------
 include/qapi/util.h    | 8 ++++++++
 2 files changed, 9 insertions(+), 8 deletions(-)
include/qapi/visitor.h +1 -8
index ebc19ede7fff..8c2e1c29ad8b 100644
@@ -16,6 +16,7 @@
 #define QAPI_VISITOR_H
 
 #include "qapi/qapi-builtin-types.h"
+#include "qapi/util.h"
 
 /*
  * The QAPI schema defines both a set of C data types, and a QMP wire
@@ -228,14 +229,6 @@
 
 /*** Useful types ***/
 
-/* This struct is layout-compatible with all other *List structs
- * created by the QAPI generator.  It is used as a typical
- * singly-linked list. */
-typedef struct GenericList {
-    struct GenericList *next;
-    char padding[];
-} GenericList;
-
 /* This struct is layout-compatible with all Alternate types
  * created by the QAPI generator. */
 typedef struct GenericAlternate {
include/qapi/util.h +8 -0
index a7c3c6414874..50201896c7a4 100644
@@ -11,6 +11,14 @@
 #ifndef QAPI_UTIL_H
 #define QAPI_UTIL_H
 
+/* This struct is layout-compatible with all other *List structs
+ * created by the QAPI generator.  It is used as a typical
+ * singly-linked list. */
+typedef struct GenericList {
+    struct GenericList *next;
+    char padding[];
+} GenericList;
+
 typedef struct QEnumLookup {
     const char *const *array;
     int size;
Signature · 1 line
2.29.0
Reviewed-by on v5 1/12
#2 lore raw

Re: [PATCH v5 01/12] qapi: Move GenericList to qapi/util.h

Quoted 12 lines · Eric Blake

Placing GenericList in util.h will make it easier for the next patch to promote QAPI_LIST_ADD() into a public macro without requiring more files to include the unrelated visitor.h.

However, we can't also move GenericAlternate; this is because it would introduce a circular dependency: qapi-builtin-types.h needs a complete definition of QEnumLookup (so it includes qapi/util.h), and GenericAlternate needs a complete definition of QType (declared in qapi-builtin-types.h). Leaving GenericAlternate in visitor.h breaks the cycle, and doesn't matter since we don't have any further planned uses for that type outside of visitors.

Quoted 2 lines
Quoted 37 lines
---
  include/qapi/visitor.h | 9 +--------
  include/qapi/util.h    | 8 ++++++++
  2 files changed, 9 insertions(+), 8 deletions(-)
include/qapi/visitor.h +1 -8
index ebc19ede7fff..8c2e1c29ad8b 100644
@@ -16,6 +16,7 @@
  #define QAPI_VISITOR_H
 
  #include "qapi/qapi-builtin-types.h"
+#include "qapi/util.h"
 
  /*
   * The QAPI schema defines both a set of C data types, and a QMP wire
@@ -228,14 +229,6 @@
 
  /*** Useful types ***/
 
-/* This struct is layout-compatible with all other *List structs
- * created by the QAPI generator.  It is used as a typical
- * singly-linked list. */
-typedef struct GenericList {
-    struct GenericList *next;
-    char padding[];
-} GenericList;
-
  /* This struct is layout-compatible with all Alternate types
   * created by the QAPI generator. */
  typedef struct GenericAlternate {
include/qapi/util.h +0 -0
index a7c3c6414874..50201896c7a4 100644

include/qapi/util.h line 16

Quoted 6 lines
include/qapi/util.h +3 -0
@@ -11,6 +11,14 @@
  #ifndef QAPI_UTIL_H
  #define QAPI_UTIL_H
 
+/* This struct is layout-compatible with all other *List structs
+ * created by the QAPI generator.  It is used as a typical
+ * singly-linked list. */

doesn't checkpatch complain for comment style?

Quoted 9 lines
include/qapi/util.h +5 -0
+typedef struct GenericList {
+    struct GenericList *next;
+    char padding[];
+} GenericList;
+
  typedef struct QEnumLookup {
      const char *const *array;
      int size;
Signature · 2 lines
Best regards,
Vladimir
#3 lore raw

Re: [PATCH v5 01/12] qapi: Move GenericList to qapi/util.h

In reply to Eric Blake
Quoted 3 lines · Eric Blake

Placing GenericList in util.h will make it easier for the next patch to promote QAPI_LIST_ADD() into a public macro without requiring more files to include the unrelated visitor.h.

Is this true?

You don't actually need GenericList to make use of QAPI_LIST_ADD(), do you? Any QAPI list type should do.

Quoted 9 lines

However, we can't also move GenericAlternate; this is because it would introduce a circular dependency: qapi-builtin-types.h needs a complete definition of QEnumLookup (so it includes qapi/util.h), and GenericAlternate needs a complete definition of QType (declared in qapi-builtin-types.h). Leaving GenericAlternate in visitor.h breaks the cycle, and doesn't matter since we don't have any further planned uses for that type outside of visitors.

I did suggest to consider moving GenericList and GenericAlternate next to QAPI_LIST_ADD(), because they're (loosely) related. We can't move GenericAlternate. Moving only GenericList brings GenericList and QAPI_LIST_ADD() together, but separates the more closely related GenericList and GenericAlternate. Meh.

I'd leave it put.

Quoted 11 lines
---
 include/qapi/visitor.h | 9 +--------
 include/qapi/util.h    | 8 ++++++++
 2 files changed, 9 insertions(+), 8 deletions(-)
include/qapi/visitor.h +0 -0
index ebc19ede7fff..8c2e1c29ad8b 100644

include/qapi/visitor.h line 19

Quoted 4 lines
include/qapi/visitor.h +1 -0
@@ -16,6 +16,7 @@
 #define QAPI_VISITOR_H
 
 #include "qapi/qapi-builtin-types.h"
+#include "qapi/util.h"

Not necessary, qapi-builtin-types.h must include it for QEnumLookup.

Quoted 36 lines
include/qapi/visitor.h +0 -8
 /*
  * The QAPI schema defines both a set of C data types, and a QMP wire
@@ -228,14 +229,6 @@
 
 /*** Useful types ***/
 
-/* This struct is layout-compatible with all other *List structs
- * created by the QAPI generator.  It is used as a typical
- * singly-linked list. */
-typedef struct GenericList {
-    struct GenericList *next;
-    char padding[];
-} GenericList;
-
 /* This struct is layout-compatible with all Alternate types
  * created by the QAPI generator. */
 typedef struct GenericAlternate {
include/qapi/util.h +8 -0
index a7c3c6414874..50201896c7a4 100644
@@ -11,6 +11,14 @@
 #ifndef QAPI_UTIL_H
 #define QAPI_UTIL_H
 
+/* This struct is layout-compatible with all other *List structs
+ * created by the QAPI generator.  It is used as a typical
+ * singly-linked list. */
+typedef struct GenericList {
+    struct GenericList *next;
+    char padding[];
+} GenericList;
+
 typedef struct QEnumLookup {
     const char *const *array;
     int size;
author
#4 lore raw

Re: [PATCH v5 01/12] qapi: Move GenericList to qapi/util.h

Quoted 10 lines · Markus Armbruster
Quoted 3 lines · Eric Blake

Placing GenericList in util.h will make it easier for the next patch to promote QAPI_LIST_ADD() into a public macro without requiring more files to include the unrelated visitor.h.

Is this true?

You don't actually need GenericList to make use of QAPI_LIST_ADD(), do you? Any QAPI list type should do.

Correct, compilation still works if I drop this patch.

Quoted 11 lines
Quoted 10 lines

However, we can't also move GenericAlternate; this is because it would introduce a circular dependency: qapi-builtin-types.h needs a complete definition of QEnumLookup (so it includes qapi/util.h), and GenericAlternate needs a complete definition of QType (declared in qapi-builtin-types.h). Leaving GenericAlternate in visitor.h breaks the cycle, and doesn't matter since we don't have any further planned uses for that type outside of visitors.

Quoted 7 lines

I did suggest to consider moving GenericList and GenericAlternate next to QAPI_LIST_ADD(), because they're (loosely) related. We can't move GenericAlternate. Moving only GenericList brings GenericList and QAPI_LIST_ADD() together, but separates the more closely related GenericList and GenericAlternate. Meh.

I'd leave it put.

Agreed. Dropping this patch in v6.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#5 lore raw

PATCH v5 2/12 qapi: Make QAPI_LIST_ADD() public

In reply to Eric Blake

We have a useful macro for inserting at the front of any QAPI-generated list; move it from block.c to qapi/util.h so more places can use it, including one earlier place in block.c.

---
 include/qapi/util.h |  8 ++++++++
 block.c             | 15 +++------------
 2 files changed, 11 insertions(+), 12 deletions(-)
include/qapi/util.h +8 -0
index 50201896c7a4..b6083055ce69 100644
@@ -30,4 +30,12 @@ int qapi_enum_parse(const QEnumLookup *lookup, const char *buf,
 
 int parse_qapi_name(const char *name, bool complete);
 
+/* For any GenericList @list, insert @element at the front. */
+#define QAPI_LIST_ADD(list, element) do { \
+    typeof(list) _tmp = g_new(typeof(*(list)), 1); \
+    _tmp->value = (element); \
+    _tmp->next = (list); \
+    (list) = _tmp; \
+} while (0)
+
 #endif
block.c +3 -12
index 430edf79bb10..45bd79299611 100644
@@ -39,6 +39,7 @@
 #include "qapi/qmp/qstring.h"
 #include "qapi/qobject-output-visitor.h"
 #include "qapi/qapi-visit-block-core.h"
+#include "qapi/util.h"
 #include "sysemu/block-backend.h"
 #include "sysemu/sysemu.h"
 #include "qemu/notify.h"
@@ -5211,7 +5212,7 @@ BlockDriverState *bdrv_find_node(const char *node_name)
 BlockDeviceInfoList *bdrv_named_nodes_list(bool flat,
                                            Error **errp)
 {
-    BlockDeviceInfoList *list, *entry;
+    BlockDeviceInfoList *list;
     BlockDriverState *bs;
 
     list = NULL;
@@ -5221,22 +5222,12 @@ BlockDeviceInfoList *bdrv_named_nodes_list(bool flat,
             qapi_free_BlockDeviceInfoList(list);
             return NULL;
         }
-        entry = g_malloc0(sizeof(*entry));
-        entry->value = info;
-        entry->next = list;
-        list = entry;
+        QAPI_LIST_ADD(list, info);
     }
 
     return list;
 }
 
-#define QAPI_LIST_ADD(list, element) do { \
-    typeof(list) _tmp = g_new(typeof(*(list)), 1); \
-    _tmp->value = (element); \
-    _tmp->next = (list); \
-    (list) = _tmp; \
-} while (0)
-
 typedef struct XDbgBlockGraphConstructor {
     XDbgBlockGraph *graph;
     GHashTable *graph_nodes;
Signature · 1 line
2.29.0
Reviewed-by on v5 2/12
#6 lore raw

Re: [PATCH v5 02/12] qapi: Make QAPI_LIST_ADD() public

Quoted 6 lines · Eric Blake

We have a useful macro for inserting at the front of any QAPI-generated list; move it from block.c to qapi/util.h so more places can use it, including one earlier place in block.c.

Signature · 2 lines
Best regards,
Vladimir
Reviewed-by on v5 2/12
#7 lore raw

Re: [PATCH v5 02/12] qapi: Make QAPI_LIST_ADD() public

In reply to Eric Blake
Quoted 16 lines · Eric Blake

We have a useful macro for inserting at the front of any QAPI-generated list; move it from block.c to qapi/util.h so more places can use it, including one earlier place in block.c.

---
 include/qapi/util.h |  8 ++++++++
 block.c             | 15 +++------------
 2 files changed, 11 insertions(+), 12 deletions(-)
include/qapi/util.h +0 -0
index 50201896c7a4..b6083055ce69 100644

include/qapi/util.h int qapi_enum_parse(const QEnumLookup *lookup, const char *buf, · line 35

Quoted 6 lines
include/qapi/util.h +3 -0
@@ -30,4 +30,12 @@ int qapi_enum_parse(const QEnumLookup *lookup, const char *buf,
 
 int parse_qapi_name(const char *name, bool complete);
 
+/* For any GenericList @list, insert @element at the front. */
+#define QAPI_LIST_ADD(list, element) do { \
+    typeof(list) _tmp = g_new(typeof(*(list)), 1); \

g_new(typeof(*p), 1) is an rather roundabout way to say g_malloc(sizeof(*p). Yes, it returns typeof(p) instead of void *, but the chance of this catching mistakes here rounds to zero.

Not this patch's problem. Ignore me :)

Quoted 30 lines
include/qapi/util.h +5 -0
+    _tmp->value = (element); \
+    _tmp->next = (list); \
+    (list) = _tmp; \
+} while (0)
+
 #endif
block.c +2 -1
index 430edf79bb10..45bd79299611 100644
@@ -39,6 +39,7 @@
 #include "qapi/qmp/qstring.h"
 #include "qapi/qobject-output-visitor.h"
 #include "qapi/qapi-visit-block-core.h"
+#include "qapi/util.h"
 #include "sysemu/block-backend.h"
 #include "sysemu/sysemu.h"
 #include "qemu/notify.h"
@@ -5211,7 +5212,7 @@ BlockDriverState *bdrv_find_node(const char *node_name)
 BlockDeviceInfoList *bdrv_named_nodes_list(bool flat,
                                            Error **errp)
 {
-    BlockDeviceInfoList *list, *entry;
+    BlockDeviceInfoList *list;
     BlockDriverState *bs;
 
     list = NULL;
@@ -5221,22 +5222,12 @@ BlockDeviceInfoList *bdrv_named_nodes_list(bool flat,
             qapi_free_BlockDeviceInfoList(list);
             return NULL;

block.c BlockDeviceInfoList *bdrv_named_nodes_list(bool flat, · line 5229

Quoted 10 lines
block.c +1 -4
         }
-        entry = g_malloc0(sizeof(*entry));
-        entry->value = info;
-        entry->next = list;
-        list = entry;
+        QAPI_LIST_ADD(list, info);
     }
 
     return list;
 }

PATCH 12 has more. I wonder whether PATCH 12 should be squashed into this one. You decide.

block.c BlockDeviceInfoList *bdrv_named_nodes_list(bool flat, · line 5233

Quoted 11 lines
block.c +0 -7
-#define QAPI_LIST_ADD(list, element) do { \
-    typeof(list) _tmp = g_new(typeof(*(list)), 1); \
-    _tmp->value = (element); \
-    _tmp->next = (list); \
-    (list) = _tmp; \
-} while (0)
-
 typedef struct XDbgBlockGraphConstructor {
     XDbgBlockGraph *graph;
     GHashTable *graph_nodes;
author
#8 lore raw

Re: [PATCH v5 02/12] qapi: Make QAPI_LIST_ADD() public

Quoted 29 lines · Markus Armbruster
Quoted 16 lines · Eric Blake

We have a useful macro for inserting at the front of any QAPI-generated list; move it from block.c to qapi/util.h so more places can use it, including one earlier place in block.c.

---
 include/qapi/util.h |  8 ++++++++
 block.c             | 15 +++------------
 2 files changed, 11 insertions(+), 12 deletions(-)
include/qapi/util.h +0 -0
index 50201896c7a4..b6083055ce69 100644

include/qapi/util.h int qapi_enum_parse(const QEnumLookup *lookup, const char *buf, · line 35

Quoted 6 lines
include/qapi/util.h +3 -0
@@ -30,4 +30,12 @@ int qapi_enum_parse(const QEnumLookup *lookup, const char *buf,
 
 int parse_qapi_name(const char *name, bool complete);
 
+/* For any GenericList @list, insert @element at the front. */
+#define QAPI_LIST_ADD(list, element) do { \
+    typeof(list) _tmp = g_new(typeof(*(list)), 1); \

g_new(typeof(*p), 1) is an rather roundabout way to say g_malloc(sizeof(*p). Yes, it returns typeof(p) instead of void *, but the chance of this catching mistakes here rounds to zero.

Quoted 1 line

Not this patch's problem. Ignore me :)

typeof is a gcc/clang extension; using sizeof makes the code slightly more portable (but doesn't affect qemu's usage). I'm happy to change it, although it would require more commit message finesse since that becomes more than just code motion.

Quoted 11 lines
Quoted 11 lines
include/qapi/util.h +1 -4
@@ -5221,22 +5222,12 @@ BlockDeviceInfoList *bdrv_named_nodes_list(bool flat,
             qapi_free_BlockDeviceInfoList(list);
             return NULL;
         }
-        entry = g_malloc0(sizeof(*entry));
-        entry->value = info;
-        entry->next = list;
-        list = entry;
+        QAPI_LIST_ADD(list, info);
     }
 
Quoted 5 lines

include/qapi/util.h BlockDeviceInfoList *bdrv_named_nodes_list(bool flat, · line 5229

Quoted 2 lines
include/qapi/util.h +0 -0
     return list;
 }

PATCH 12 has more. I wonder whether PATCH 12 should be squashed into this one. You decide.

Patch 12 has a LOT more. And we're really close to soft freeze. I kept them separate to minimize the risk of landing my QAPI changes (4/12) before 5.2 (that MUST happen, or we've locked in a problematic API with block-export-add), vs. cleanup changes that may require review from a lot more areas in the tree. Given the timeline, I prefer to keep them separate for v6.

I also still wonder if it is possible to make Coccinelle identify candidates, rather than my manual audit that produced patch 12.

But for v6, I _will_ update the commit message to mention that more conversions are possible, and saved for a later patch.

Quoted 2 lines

Thanks; I think I can keep this even for v6, since all I am changing is an enhanced commit message.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#9 lore raw

Re: [PATCH v5 02/12] qapi: Make QAPI_LIST_ADD() public

Quoted 5 lines · Eric Blake
Quoted 2 lines

PATCH 12 has more. I wonder whether PATCH 12 should be squashed into this one. You decide.

Patch 12 has a LOT more. And we're really close to soft freeze. I kept them separate to minimize the risk of landing my QAPI changes (4/12)

Correction - the QAPI change was 6/12 of v5 (whereas it moves earlier in the series in my upcoming v6 posting)

Quoted 3 lines

before 5.2 (that MUST happen, or we've locked in a problematic API with block-export-add), vs. cleanup changes that may require review from a lot more areas in the tree.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#10 lore raw

PATCH v5 3/12 nbd: Utilize QAPI_CLONE for type conversion

In reply to Eric Blake

Rather than open-coding the translation from the deprecated NbdServerAddOptions type to the preferred BlockExportOptionsNbd, it's better to utilize QAPI_CLONE_MEMBERS. This solves a couple of issues:
first, if we do any more refactoring of the base type (which an upcoming patch plans to do), we don't have to revisit the open-coding. Second, our assignment to arg->name is fishy: the generated QAPI code currently does not visit it if arg->has_name is false, but if it DID visit it, we would have introduced a double-free situation when arg is finally freed.

---
 blockdev-nbd.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)
blockdev-nbd.c +6 -9
index 8174023e5c47..cee9134b12eb 100644
@@ -14,6 +14,8 @@
 #include "sysemu/block-backend.h"
 #include "hw/block/block.h"
 #include "qapi/error.h"
+#include "qapi/clone-visitor.h"
+#include "qapi/qapi-visit-block-export.h"
 #include "qapi/qapi-commands-block-export.h"
 #include "block/nbd.h"
 #include "io/channel-socket.h"
@@ -195,7 +197,8 @@ void qmp_nbd_server_add(NbdServerAddOptions *arg, Error **errp)
      * the device name as a default here for compatibility.
      */
     if (!arg->has_name) {
-        arg->name = arg->device;
+        arg->has_name = true;
+        arg->name = g_strdup(arg->device);
     }
 
     export_opts = g_new(BlockExportOptions, 1);
@@ -205,15 +208,9 @@ void qmp_nbd_server_add(NbdServerAddOptions *arg, Error **errp)
         .node_name              = g_strdup(bdrv_get_node_name(bs)),
         .has_writable           = arg->has_writable,
         .writable               = arg->writable,
-        .u.nbd = {
-            .has_name           = true,
-            .name               = g_strdup(arg->name),
-            .has_description    = arg->has_description,
-            .description        = g_strdup(arg->description),
-            .has_bitmap         = arg->has_bitmap,
-            .bitmap             = g_strdup(arg->bitmap),
-        },
     };
+    QAPI_CLONE_MEMBERS(BlockExportOptionsNbd, &export_opts->u.nbd,
+                       qapi_NbdServerAddOptions_base(arg));
 
     /*
      * nbd-server-add doesn't complain when a read-only device should be
Signature · 1 line
2.29.0
Reviewed-by on v5 3/12
#11 lore raw

Re: [PATCH v5 03/12] nbd: Utilize QAPI_CLONE for type conversion

Quoted 11 lines · Eric Blake

Rather than open-coding the translation from the deprecated NbdServerAddOptions type to the preferred BlockExportOptionsNbd, it's better to utilize QAPI_CLONE_MEMBERS. This solves a couple of issues:
first, if we do any more refactoring of the base type (which an upcoming patch plans to do), we don't have to revisit the open-coding. Second, our assignment to arg->name is fishy: the generated QAPI code currently does not visit it if arg->has_name is false, but if it DID visit it, we would have introduced a double-free situation when arg is finally freed.

Signature · 2 lines
Best regards,
Vladimir
Reviewed-by on v5 3/12
#12 lore raw

Re: [PATCH v5 03/12] nbd: Utilize QAPI_CLONE for type conversion

In reply to Eric Blake
Quoted 29 lines · Eric Blake

Rather than open-coding the translation from the deprecated NbdServerAddOptions type to the preferred BlockExportOptionsNbd, it's better to utilize QAPI_CLONE_MEMBERS. This solves a couple of issues:
first, if we do any more refactoring of the base type (which an upcoming patch plans to do), we don't have to revisit the open-coding. Second, our assignment to arg->name is fishy: the generated QAPI code currently does not visit it if arg->has_name is false, but if it DID visit it, we would have introduced a double-free situation when arg is finally freed.

---
 blockdev-nbd.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)
blockdev-nbd.c +2 -0
index 8174023e5c47..cee9134b12eb 100644
@@ -14,6 +14,8 @@
 #include "sysemu/block-backend.h"
 #include "hw/block/block.h"
 #include "qapi/error.h"
+#include "qapi/clone-visitor.h"
+#include "qapi/qapi-visit-block-export.h"
 #include "qapi/qapi-commands-block-export.h"
 #include "block/nbd.h"
 #include "io/channel-socket.h"

blockdev-nbd.c void qmp_nbd_server_add(NbdServerAddOptions *arg, Error **errp) · line 202

Quoted 7 lines
blockdev-nbd.c +2 -1
@@ -195,7 +197,8 @@ void qmp_nbd_server_add(NbdServerAddOptions *arg, Error **errp)
      * the device name as a default here for compatibility.
      */
     if (!arg->has_name) {
-        arg->name = arg->device;
+        arg->has_name = true;
+        arg->name = g_strdup(arg->device);
     }

This is the fix you mentioned in the commit message.

blockdev-nbd.c void qmp_nbd_server_add(NbdServerAddOptions *arg, Error **errp)

Quoted 3 lines
blockdev-nbd.c +0 -0
     export_opts = g_new(BlockExportOptions, 1);
@@ -205,15 +208,9 @@ void qmp_nbd_server_add(NbdServerAddOptions *arg, Error **errp)
       *export_opts = (BlockExportOptions) {
           .type                   = BLOCK_EXPORT_TYPE_NBD,
           .id                     = g_strdup(arg->name),

blockdev-nbd.c void qmp_nbd_server_add(NbdServerAddOptions *arg, Error **errp) · line 210

Quoted 3 lines
blockdev-nbd.c +0 -0
         .node_name              = g_strdup(bdrv_get_node_name(bs)),
         .has_writable           = arg->has_writable,
         .writable               = arg->writable,

Explicit initialization of all the common members, except for @writethrough. @writethrough is optional, so not mentioning it makes it absent. I don't mind.

blockdev-nbd.c void qmp_nbd_server_add(NbdServerAddOptions *arg, Error **errp) · line 214

Quoted 7 lines
blockdev-nbd.c +0 -7
-        .u.nbd = {
-            .has_name           = true,
-            .name               = g_strdup(arg->name),
-            .has_description    = arg->has_description,
-            .description        = g_strdup(arg->description),
-            .has_bitmap         = arg->has_bitmap,
-            .bitmap             = g_strdup(arg->bitmap),

Explicit initialization of all the variant members: copy of @arg.

blockdev-nbd.c void qmp_nbd_server_add(NbdServerAddOptions *arg, Error **errp) · line 213

Quoted 4 lines
blockdev-nbd.c +2 -1
-        },
     };
+    QAPI_CLONE_MEMBERS(BlockExportOptionsNbd, &export_opts->u.nbd,
+                       qapi_NbdServerAddOptions_base(arg));

Another (and better) way to copy.

blockdev-nbd.c void qmp_nbd_server_add(NbdServerAddOptions *arg, Error **errp) · line 216

Quoted 3 lines
blockdev-nbd.c +0 -0
     /*
      * nbd-server-add doesn't complain when a read-only device should be
author
#13 lore raw

PATCH v5 4/12 nbd: Add new qemu:allocation-depth metadata context

In reply to Eric Blake

'qemu-img map' provides a way to determine which extents of an image come from the top layer vs. inherited from a backing chain. This is useful information worth exposing over NBD. There is a proposal to add a QMP command block-dirty-bitmap-populate which can create a dirty bitmap that reflects allocation information, at which point the qemu:dirty-bitmap:NAME metadata context can expose that information via the creation of a temporary bitmap, but we can shorten the effort by adding a new qemu:allocation-depth metadata context that does the same thing without an intermediate bitmap (this patch does not eliminate the need for that proposal, as it will have other uses as well).

For this patch, I just encoded a tri-state value (DEPTH_CODE_MASK:
unallocated, from this layer, from any of the backing layers); an obvious extension would be to provide the actual depth in bits 31-4 (a new DEPTH_RAW_MASK 0xfffffff0) while keeping bits 1-0 as a tri-state (leaving bits 3-2 unused, for ease of reading depth from a hex number). But adding this extension would require bdrv_is_allocated_above to return a depth number.

While documenting things, remember that although the NBD protocol has NBD_OPT_SET_META_CONTEXT, the rest of its documentation refers to 'metadata context', which is a more apt description of what is actually being used by NBD_CMD_BLOCK_STATUS: the user is requesting metadata by passing one or more context names. So I also touched up some existing wording to prefer the term 'metadata context' where it makes sense.

Note that this patch does not actually enable any way to request a server to enable this context; that will come in the next patch.

---
 docs/interop/nbd.txt | 27 ++++++++++---
 include/block/nbd.h  | 12 ++++--
 nbd/server.c         | 92 ++++++++++++++++++++++++++++++++++++++++----
 3 files changed, 116 insertions(+), 15 deletions(-)
docs/interop/nbd.txt +22 -5
index f3b3cacc9621..7e948bd42218 100644
@@ -17,19 +17,35 @@ namespace "qemu".
 
 == "qemu" namespace ==
 
-The "qemu" namespace currently contains only one type of context,
-related to exposing the contents of a dirty bitmap alongside the
-associated disk contents.  That context has the following form:
+The "qemu" namespace currently contains two available metadata context
+types.  The first is related to exposing the contents of a dirty
+bitmap alongside the associated disk contents.  That metadata context
+is named with the following form:
 
     qemu:dirty-bitmap:<dirty-bitmap-export-name>
 
 Each dirty-bitmap metadata context defines only one flag for extents
 in reply for NBD_CMD_BLOCK_STATUS:
 
-    bit 0: NBD_STATE_DIRTY, means that the extent is "dirty"
+    bit 0: NBD_STATE_DIRTY, set when the extent is "dirty"
+
+The second is related to exposing the source of various extents within
+the image, with a single metadata context named:
+
+    qemu:allocation-depth
+
+In the allocation depth context, bits 0 and 1 form a tri-state value:
+
+    bits 0-1: 00: NBD_STATE_DEPTH_UNALLOC, the extent is unallocated
+              01: NBD_STATE_DEPTH_LOCAL, the extent is allocated in the
+                  top layer of the image
+              10: NBD_STATE_DEPTH_BACKING, the extent is inherited from a
+                  backing layer
+              11: invalid, never returned
 
 For NBD_OPT_LIST_META_CONTEXT the following queries are supported
-in addition to "qemu:dirty-bitmap:<dirty-bitmap-export-name>":
+in addition to the specific "qemu:allocation-depth" and
+"qemu:dirty-bitmap:<dirty-bitmap-export-name>":
 
 * "qemu:" - returns list of all available metadata contexts in the
             namespace.
@@ -55,3 +71,4 @@ the operation of that feature.
 NBD_CMD_BLOCK_STATUS for "qemu:dirty-bitmap:", NBD_CMD_CACHE
 * 4.2: NBD_FLAG_CAN_MULTI_CONN for shareable read-only exports,
 NBD_CMD_FLAG_FAST_ZERO
+* 5.2: NBD_CMD_BLOCK_STATUS for "qemu:allocation-depth"
include/block/nbd.h +9 -3
index 3dd9a04546ec..956687f5c368 100644
@@ -1,5 +1,5 @@
 /*
- *  Copyright (C) 2016-2019 Red Hat, Inc.
+ *  Copyright (C) 2016-2020 Red Hat, Inc.
  *  Copyright (C) 2005  Anthony Liguori <[email protected]>
  *
  *  Network Block Device
@@ -47,7 +47,7 @@ typedef struct NBDOptionReply NBDOptionReply;
 typedef struct NBDOptionReplyMetaContext {
     NBDOptionReply h; /* h.type = NBD_REP_META_CONTEXT, h.length > 4 */
     uint32_t context_id;
-    /* meta context name follows */
+    /* metadata context name follows */
 } QEMU_PACKED NBDOptionReplyMetaContext;
 
 /* Transmission phase structs
@@ -229,7 +229,7 @@ enum {
 #define NBD_MAX_BUFFER_SIZE (32 * 1024 * 1024)
 
 /*
- * Maximum size of a protocol string (export name, meta context name,
+ * Maximum size of a protocol string (export name, metadata context name,
  * etc.).  Use malloc rather than stack allocation for storage of a
  * string.
  */
@@ -259,6 +259,12 @@ enum {
 /* Extent flags for qemu:dirty-bitmap in NBD_REPLY_TYPE_BLOCK_STATUS */
 #define NBD_STATE_DIRTY (1 << 0)
 
+/* Extent flags for qemu:allocation-depth in NBD_REPLY_TYPE_BLOCK_STATUS */
+#define NBD_STATE_DEPTH_CODE_MASK    0x3
+#define NBD_STATE_DEPTH_UNALLOC      0x0
+#define NBD_STATE_DEPTH_LOCAL        0x1
+#define NBD_STATE_DEPTH_BACKING      0x2
+
 static inline bool nbd_reply_type_is_error(int type)
 {
     return type & (1 << 15);
nbd/server.c +85 -7

188 lines not on this page · raw message

Signature · 1 line
2.29.0
author
#14 lore raw

PATCH v5 5/12 nbd: Add 'qemu-nbd -A' to expose allocation depth

In reply to Eric Blake

Allow the server to expose an additional metacontext to be requested by savvy clients. qemu-nbd adds a new option -A to expose the qemu:allocation-depth metacontext through NBD_CMD_BLOCK_STATUS; this can also be set via QMP when using block-export-add.

qemu as client can be hacked into viewing this new context by using the now-misnamed x-dirty-bitmap option when creating an NBD blockdev (even though our x- naming means we could rename it, I did not think it worth breaking back-compat of tools that have been using it while waiting for a better solution). It is worth noting the decoding of how such context information will appear in 'qemu-img map --output=json':

NBD_STATE_DEPTH_UNALLOC => "zero":false, "data":true
NBD_STATE_DEPTH_LOCAL => "zero":false, "data":false
NBD_STATE_DEPTH_BACKING => "zero":true, "data":true

libnbd as client is probably a nicer way to get at the information without having to decipher such hacks in qemu as client. ;)

---
 docs/tools/qemu-nbd.rst    |  8 ++++-
 qapi/block-core.json       |  7 ++--
 qapi/block-export.json     | 12 +++++--
 nbd/server.c               |  2 ++
 qemu-nbd.c                 | 26 +++++++++-----
 tests/qemu-iotests/309     | 73 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/309.out | 22 ++++++++++++
 tests/qemu-iotests/group   |  1 +
 8 files changed, 136 insertions(+), 15 deletions(-)
 create mode 100755 tests/qemu-iotests/309
 create mode 100644 tests/qemu-iotests/309.out
docs/tools/qemu-nbd.rst +7 -1
index 667861cb22e9..fe41336dc550 100644
@@ -72,10 +72,16 @@ driver options if ``--image-opts`` is specified.
 
   Export the disk as read-only.
 
+.. option:: -A, --allocation-depth
+
+  Expose allocation depth information via the
+  ``qemu:allocation-depth`` metadata context accessible through
+  NBD_OPT_SET_META_CONTEXT.
+
 .. option:: -B, --bitmap=NAME
 
   If *filename* has a qcow2 persistent bitmap *NAME*, expose
-  that bitmap via the ``qemu:dirty-bitmap:NAME`` context
+  that bitmap via the ``qemu:dirty-bitmap:NAME`` metadata context
   accessible through NBD_OPT_SET_META_CONTEXT.
 
 .. option:: -s, --snapshot
qapi/block-core.json +5 -2
index ee5ebef7f2ce..b6368f5fd9a1 100644
@@ -3883,9 +3883,12 @@
 #
 # @tls-creds: TLS credentials ID
 #
-# @x-dirty-bitmap: A "qemu:dirty-bitmap:NAME" string to query in place of
+# @x-dirty-bitmap: A metadata context name such as "qemu:dirty-bitmap:NAME"
+#                  or "qemu:allocation-depth" to query in place of the
 #                  traditional "base:allocation" block status (see
-#                  NBD_OPT_LIST_META_CONTEXT in the NBD protocol) (since 3.0)
+#                  NBD_OPT_LIST_META_CONTEXT in the NBD protocol; and
+#                  yes, naming this option x-context would have made
+#                  more sense) (since 3.0)
 #
 # @reconnect-delay: On an unexpected disconnect, the nbd client tries to
 #                   connect again until succeeding or encountering a serious
qapi/block-export.json +9 -3
index 65804834d905..893d5cde5dfe 100644
@@ -75,14 +75,20 @@
 #               (Since 5.0)
 #
 # @bitmap: Also export the dirty bitmap reachable from @device, so the
-#          NBD client can use NBD_OPT_SET_META_CONTEXT with
-#          "qemu:dirty-bitmap:NAME" to inspect the bitmap. (since 4.0)
+#          NBD client can use NBD_OPT_SET_META_CONTEXT with the
+#          metadata context name "qemu:dirty-bitmap:NAME" to inspect the
+#          bitmap. (since 4.0)
+#
+# @allocation-depth: Also export the allocation depth map for @device, so
+#                    the NBD client can use NBD_OPT_SET_META_CONTEXT with
+#                    the metadata context name "qemu:allocation-depth" to
+#                    inspect allocation details. (since 5.2)
 #
 # Since: 5.0
 ##
 { 'struct': 'BlockExportOptionsNbd',
   'data': { '*name': 'str', '*description': 'str',
-            '*bitmap': 'str' } }
+            '*bitmap': 'str', '*allocation-depth': 'bool' } }
 
 ##
 # @NbdServerAddOptions:
nbd/server.c +2 -0
index ae6f8a8e5429..30cfe0eee467 100644
@@ -1597,6 +1597,8 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
         assert(strlen(exp->export_bitmap_context) < NBD_MAX_STRING_SIZE);
     }
 
+    exp->allocation_depth = arg->allocation_depth;
+
     blk_add_aio_context_notifier(blk, blk_aio_attached, blk_aio_detach, exp);
 
     QTAILQ_INSERT_TAIL(&exports, exp, next);
qemu-nbd.c +17 -9
index bc644a0670b6..847fde435a7f 100644
@@ -99,6 +99,7 @@ static void usage(const char *name)
 "\n"
 "Exposing part of the image:\n"
 "  -o, --offset=OFFSET       offset into the image\n"
+"  -A, --allocation-depth    expose the allocation depth\n"
 "  -B, --bitmap=NAME         expose a persistent dirty bitmap\n"
 "\n"
 "General purpose options:\n"
@@ -519,7 +520,7 @@ int main(int argc, char **argv)
     char *device = NULL;
     QemuOpts *sn_opts = NULL;
     const char *sn_id_or_name = NULL;
-    const char *sopt = "hVb:o:p:rsnc:dvk:e:f:tl:x:T:D:B:L";
+    const char *sopt = "hVb:o:p:rsnc:dvk:e:f:tl:x:T:D:AB:L";
     struct option lopt[] = {
         { "help", no_argument, NULL, 'h' },
         { "version", no_argument, NULL, 'V' },
@@ -528,6 +529,7 @@ int main(int argc, char **argv)
         { "socket", required_argument, NULL, 'k' },
         { "offset", required_argument, NULL, 'o' },
         { "read-only", no_argument, NULL, 'r' },
+        { "allocation-depth", no_argument, NULL, 'A' },
         { "bitmap", required_argument, NULL, 'B' },
         { "connect", required_argument, NULL, 'c' },
         { "disconnect", no_argument, NULL, 'd' },
@@ -569,6 +571,7 @@ int main(int argc, char **argv)
     QDict *options = NULL;
     const char *export_name = NULL; /* defaults to "" later for server mode */
     const char *export_description = NULL;
+    bool alloc_depth = false;
     const char *bitmap = NULL;
     const char *tlscredsid = NULL;
     bool imageOpts = false;
@@ -694,6 +697,9 @@ int main(int argc, char **argv)
             readonly = true;
             flags &= ~BDRV_O_RDWR;
             break;
+        case 'A':
+            alloc_depth = true;
+            break;
         case 'B':
             bitmap = optarg;
             break;
@@ -791,8 +797,8 @@ int main(int argc, char **argv)
             exit(EXIT_FAILURE);
         }
         if (export_name || export_description || dev_offset ||
-            device || disconnect || fmt || sn_id_or_name || bitmap ||
-            seen_aio || seen_discard || seen_cache) {
+            device || disconnect || fmt || sn_id_or_name || alloc_depth ||
+            bitmap || seen_aio || seen_discard || seen_cache) {
             error_report("List mode is incompatible with per-device settings");
             exit(EXIT_FAILURE);
         }
@@ -1072,12 +1078,14 @@ int main(int argc, char **argv)
         .has_writable       = true,
         .writable           = !readonly,
         .u.nbd = {
-            .has_name           = true,
-            .name               = g_strdup(export_name),
-            .has_description    = !!export_description,
-            .description        = g_strdup(export_description),
-            .has_bitmap         = !!bitmap,
-            .bitmap             = g_strdup(bitmap),
+            .has_name             = true,
+            .name                 = g_strdup(export_name),
+            .has_description      = !!export_description,
+            .description          = g_strdup(export_description),
+            .has_bitmap           = !!bitmap,
+            .bitmap               = g_strdup(bitmap),
+            .has_allocation_depth = alloc_depth,
+            .allocation_depth     = alloc_depth,
         },
     };
     blk_exp_add(export_opts, &error_fatal);
tests/qemu-iotests/309 new file +73 -0

73 lines not on this page · raw message

tests/qemu-iotests/309.out new file +22 -0
new file mode 100644
index 000000000000..db75bb6b0df9
@@ -0,0 +1,22 @@
+QA output created by 309
+
+=== Initial image setup ===
+
+Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=4194304
+wrote 2097152/2097152 bytes at offset 0
+2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=4194304 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
+wrote 2097152/2097152 bytes at offset 1048576
+2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+=== Check allocation over NBD ===
+
+[{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680},
+{ "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680},
+{ "start": 3145728, "length": 1048576, "depth": 1, "zero": true, "data": false}]
+[{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET},
+{ "start": 3145728, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": OFFSET}]
+[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": true, "offset": OFFSET},
+{ "start": 1048576, "length": 2097152, "depth": 0, "zero": false, "data": false},
+{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
+*** done
tests/qemu-iotests/group +1 -0
index 343298928350..2960dff72864 100644
@@ -315,3 +315,4 @@
 304 rw quick
 305 rw quick
 307 rw quick export
+309 rw auto quick
Signature · 1 line
2.29.0
author
#15 lore raw

PATCH v5 6/12 nbd: Update qapi to support exporting multiple bitmaps

In reply to Eric Blake

Since 'nbd-server-add' is deprecated, and 'block-export-add' is new to 5.2, we can still tweak the interface. Allowing 'bitmaps':['str'] is nicer than 'bitmap':'str'. This wires up the qapi and qemu-nbd changes to permit passing multiple bitmaps as distinct metadata contexts that the NBD client may request, but the actual support for more than one will require a further patch to the server.

---
 docs/system/deprecated.rst |  4 +++-
 qapi/block-export.json     | 18 ++++++++++++------
 blockdev-nbd.c             | 13 +++++++++++++
 nbd/server.c               | 19 +++++++++++++------
 qemu-nbd.c                 | 10 +++++-----
 5 files changed, 46 insertions(+), 18 deletions(-)
docs/system/deprecated.rst +3 -1
index 905628f3a0cb..d6cd027ac740 100644
@@ -268,7 +268,9 @@ the 'wait' field, which is only applicable to sockets in server mode
 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''
 
 Use the more generic commands ``block-export-add`` and ``block-export-del``
-instead.
+instead.  As part of this deprecation, it is now preferred to export a
+list of dirty bitmaps via ``bitmaps``, rather than a single bitmap via
+``bitmap``.
 
 Human Monitor Protocol (HMP) commands
 -------------------------------------
qapi/block-export.json +12 -6
index 893d5cde5dfe..c7c749d61097 100644
@@ -74,10 +74,10 @@
 # @description: Free-form description of the export, up to 4096 bytes.
 #               (Since 5.0)
 #
-# @bitmap: Also export the dirty bitmap reachable from @device, so the
-#          NBD client can use NBD_OPT_SET_META_CONTEXT with the
-#          metadata context name "qemu:dirty-bitmap:NAME" to inspect the
-#          bitmap. (since 4.0)
+# @bitmaps: Also export each of the named dirty bitmaps reachable from
+#           @device, so the NBD client can use NBD_OPT_SET_META_CONTEXT with
+#           the metadata context name "qemu:dirty-bitmap:BITMAP" to inspect
+#           each bitmap. (since 5.2)
 #
 # @allocation-depth: Also export the allocation depth map for @device, so
 #                    the NBD client can use NBD_OPT_SET_META_CONTEXT with
@@ -88,7 +88,7 @@
 ##
 { 'struct': 'BlockExportOptionsNbd',
   'data': { '*name': 'str', '*description': 'str',
-            '*bitmap': 'str', '*allocation-depth': 'bool' } }
+            '*bitmaps': ['str'], '*allocation-depth': 'bool' } }
 
 ##
 # @NbdServerAddOptions:
@@ -100,12 +100,18 @@
 # @writable: Whether clients should be able to write to the device via the
 #            NBD connection (default false).
 #
+# @bitmap: Also export a single dirty bitmap reachable from @device, so the
+#          NBD client can use NBD_OPT_SET_META_CONTEXT with the metadata
+#          context name "qemu:dirty-bitmap:BITMAP" to inspect the bitmap
+#          (since 4.0).  Mutually exclusive with @bitmaps, and newer
+#          clients should use that instead.
+#
 # Since: 5.0
 ##
 { 'struct': 'NbdServerAddOptions',
   'base': 'BlockExportOptionsNbd',
   'data': { 'device': 'str',
-            '*writable': 'bool' } }
+            '*writable': 'bool', '*bitmap': 'str' } }
 
 ##
 # @nbd-server-add:
blockdev-nbd.c +13 -0
index cee9134b12eb..cfd46223bf4d 100644
@@ -192,6 +192,19 @@ void qmp_nbd_server_add(NbdServerAddOptions *arg, Error **errp)
         return;
     }
 
+    /*
+     * New code should use the list 'bitmaps'; but until this code is
+     * gone, we must support the older single 'bitmap'.  Use only one.
+     */
+    if (arg->has_bitmap) {
+        if (arg->has_bitmaps) {
+            error_setg(errp, "Can't mix 'bitmap' and 'bitmaps'");
+            return;
+        }
+        arg->has_bitmaps = true;
+        QAPI_LIST_ADD(arg->bitmaps, g_strdup(arg->bitmap));
+    }
+
     /*
      * block-export-add would default to the node-name, but we may have to use
      * the device name as a default here for compatibility.
nbd/server.c +13 -6
index 30cfe0eee467..884ffa00f1bd 100644
@@ -1495,6 +1495,7 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
     uint64_t perm, shared_perm;
     bool readonly = !exp_args->writable;
     bool shared = !exp_args->writable;
+    strList *bitmaps;
     int ret;
 
     assert(exp_args->type == BLOCK_EXPORT_TYPE_NBD);
@@ -1556,12 +1557,18 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
     }
     exp->size = QEMU_ALIGN_DOWN(size, BDRV_SECTOR_SIZE);
 
-    if (arg->bitmap) {
+    /* XXX Allow more than one bitmap */
+    if (arg->bitmaps && arg->bitmaps->next) {
+        error_setg(errp, "multiple bitmaps per export not supported yet");
+        return -EOPNOTSUPP;
+    }
+    for (bitmaps = arg->bitmaps; bitmaps; bitmaps = bitmaps->next) {
+        const char *bitmap = bitmaps->value;
         BlockDriverState *bs = blk_bs(blk);
         BdrvDirtyBitmap *bm = NULL;
 
         while (bs) {
-            bm = bdrv_find_dirty_bitmap(bs, arg->bitmap);
+            bm = bdrv_find_dirty_bitmap(bs, bitmap);
             if (bm != NULL) {
                 break;
             }
@@ -1571,7 +1578,7 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
 
         if (bm == NULL) {
             ret = -ENOENT;
-            error_setg(errp, "Bitmap '%s' is not found", arg->bitmap);
+            error_setg(errp, "Bitmap '%s' is not found", bitmap);
             goto fail;
         }
 
@@ -1585,15 +1592,15 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
             ret = -EINVAL;
             error_setg(errp,
                        "Enabled bitmap '%s' incompatible with readonly export",
-                       arg->bitmap);
+                       bitmap);
             goto fail;
         }
 
         bdrv_dirty_bitmap_set_busy(bm, true);
         exp->export_bitmap = bm;
-        assert(strlen(arg->bitmap) <= BDRV_BITMAP_MAX_NAME_SIZE);
+        assert(strlen(bitmap) <= BDRV_BITMAP_MAX_NAME_SIZE);
         exp->export_bitmap_context = g_strdup_printf("qemu:dirty-bitmap:%s",
-                                                     arg->bitmap);
+                                                     bitmap);
         assert(strlen(exp->export_bitmap_context) < NBD_MAX_STRING_SIZE);
     }
 
qemu-nbd.c +5 -5
index 847fde435a7f..5473821216f7 100644
@@ -572,7 +572,7 @@ int main(int argc, char **argv)
     const char *export_name = NULL; /* defaults to "" later for server mode */
     const char *export_description = NULL;
     bool alloc_depth = false;
-    const char *bitmap = NULL;
+    strList *bitmaps = NULL;
     const char *tlscredsid = NULL;
     bool imageOpts = false;
     bool writethrough = true;
@@ -701,7 +701,7 @@ int main(int argc, char **argv)
             alloc_depth = true;
             break;
         case 'B':
-            bitmap = optarg;
+            QAPI_LIST_ADD(bitmaps, g_strdup(optarg));
             break;
         case 'k':
             sockpath = optarg;
@@ -798,7 +798,7 @@ int main(int argc, char **argv)
         }
         if (export_name || export_description || dev_offset ||
             device || disconnect || fmt || sn_id_or_name || alloc_depth ||
-            bitmap || seen_aio || seen_discard || seen_cache) {
+            bitmaps || seen_aio || seen_discard || seen_cache) {
             error_report("List mode is incompatible with per-device settings");
             exit(EXIT_FAILURE);
         }
@@ -1082,8 +1082,8 @@ int main(int argc, char **argv)
             .name                 = g_strdup(export_name),
             .has_description      = !!export_description,
             .description          = g_strdup(export_description),
-            .has_bitmap           = !!bitmap,
-            .bitmap               = g_strdup(bitmap),
+            .has_bitmaps          = !!bitmaps,
+            .bitmaps              = bitmaps,
             .has_allocation_depth = alloc_depth,
             .allocation_depth     = alloc_depth,
         },
Signature · 1 line
2.29.0
#16 lore raw

Re: [PATCH v5 06/12] nbd: Update qapi to support exporting multiple bitmaps

Quoted 38 lines · Eric Blake

Since 'nbd-server-add' is deprecated, and 'block-export-add' is new to 5.2, we can still tweak the interface. Allowing 'bitmaps':['str'] is nicer than 'bitmap':'str'. This wires up the qapi and qemu-nbd changes to permit passing multiple bitmaps as distinct metadata contexts that the NBD client may request, but the actual support for more than one will require a further patch to the server.

---
 docs/system/deprecated.rst |  4 +++-
 qapi/block-export.json     | 18 ++++++++++++------
 blockdev-nbd.c             | 13 +++++++++++++
 nbd/server.c               | 19 +++++++++++++------
 qemu-nbd.c                 | 10 +++++-----
 5 files changed, 46 insertions(+), 18 deletions(-)
docs/system/deprecated.rst +3 -1
index 905628f3a0cb..d6cd027ac740 100644
@@ -268,7 +268,9 @@ the 'wait' field, which is only applicable to sockets in server mode
 ''''''''''''''''''''''''''''''''''''''''''''''''''''''''
 
 Use the more generic commands ``block-export-add`` and ``block-export-del``
-instead.
+instead.  As part of this deprecation, it is now preferred to export a
+list of dirty bitmaps via ``bitmaps``, rather than a single bitmap via
+``bitmap``.
 
 Human Monitor Protocol (HMP) commands
 -------------------------------------
qapi/block-export.json +0 -0
index 893d5cde5dfe..c7c749d61097 100644
@@ -74,10 +74,10 @@
 # @description: Free-form description of the export, up to 4096 bytes.

qapi/block-export.json line 80

Quoted 10 lines
qapi/block-export.json +4 -4
 #               (Since 5.0)
 #
-# @bitmap: Also export the dirty bitmap reachable from @device, so the
-#          NBD client can use NBD_OPT_SET_META_CONTEXT with the
-#          metadata context name "qemu:dirty-bitmap:NAME" to inspect the
-#          bitmap. (since 4.0)
+# @bitmaps: Also export each of the named dirty bitmaps reachable from
+#           @device, so the NBD client can use NBD_OPT_SET_META_CONTEXT with
+#           the metadata context name "qemu:dirty-bitmap:BITMAP" to inspect
+#           each bitmap. (since 5.2)

Given unsynchronised release cycles between qemu and management apps it's not cool to deprecate an interface without having at least one release where the replacement interface is already stable.

This means that any project wanting to stay up to date will either have to use deprecated interfaces for at least one release and develop the replacement only when there's a stable interface or hope that they don't have to change the interfaces too often.

This specifically impacts libvirt as we have validators which notify us that a deprecated interface is used and we want to stay in sync, so that there's one less group of users to worry about at the point qemu will want to delete the interface.

qapi/block-export.json line 91

Quoted 8 lines
qapi/block-export.json +1 -1
 # @allocation-depth: Also export the allocation depth map for @device, so
 #                    the NBD client can use NBD_OPT_SET_META_CONTEXT with
@@ -88,7 +88,7 @@
 ##
 { 'struct': 'BlockExportOptionsNbd',
   'data': { '*name': 'str', '*description': 'str',
-            '*bitmap': 'str', '*allocation-depth': 'bool' } }
+            '*bitmaps': ['str'], '*allocation-depth': 'bool' } }

This adds 'bitmaps' also to nbd-server-add, which should not happen. You probably want to stop using 'base' for 'NbdServerAddOptions' and just duplicate everything else.

qapi/block-export.json line 107

Quoted 10 lines
qapi/block-export.json +5 -0
 # @NbdServerAddOptions:
@@ -100,12 +100,18 @@
 # @writable: Whether clients should be able to write to the device via the
 #            NBD connection (default false).
 #
+# @bitmap: Also export a single dirty bitmap reachable from @device, so the
+#          NBD client can use NBD_OPT_SET_META_CONTEXT with the metadata
+#          context name "qemu:dirty-bitmap:BITMAP" to inspect the bitmap
+#          (since 4.0).  Mutually exclusive with @bitmaps, and newer
+#          clients should use that instead.

This doesn't make sense, nbd-server-add never had @bitmaps. Also adding features to a deprecated interface doesn't IMO make sense if you want to motivate users switch to thne new one.

Quoted 11 lines
qapi/block-export.json +2 -1
+#
 # Since: 5.0
 ##
 { 'struct': 'NbdServerAddOptions',
   'base': 'BlockExportOptionsNbd',
   'data': { 'device': 'str',
-            '*writable': 'bool' } }
+            '*writable': 'bool', '*bitmap': 'str' } }
 
 ##
 # @nbd-server-add:
author
#17 lore raw

Re: [PATCH v5 06/12] nbd: Update qapi to support exporting multiple bitmaps

Quoted 21 lines · Peter Krempa
Quoted 20 lines · Eric Blake

Since 'nbd-server-add' is deprecated, and 'block-export-add' is new to 5.2, we can still tweak the interface. Allowing 'bitmaps':['str'] is nicer than 'bitmap':'str'. This wires up the qapi and qemu-nbd changes to permit passing multiple bitmaps as distinct metadata contexts that the NBD client may request, but the actual support for more than one will require a further patch to the server.

---

+++ b/qapi/block-export.json

qapi/block-export.json +0 -4
@@ -74,10 +74,10 @@
 # @description: Free-form description of the export, up to 4096 bytes.
 #               (Since 5.0)
 #
-# @bitmap: Also export the dirty bitmap reachable from @device, so the
-#          NBD client can use NBD_OPT_SET_META_CONTEXT with the
-#          metadata context name "qemu:dirty-bitmap:NAME" to inspect the
-#          bitmap. (since 4.0)
Quoted 9 lines

qapi/block-export.json line 80

Quoted 4 lines
qapi/block-export.json +4 -0
+# @bitmaps: Also export each of the named dirty bitmaps reachable from
+#           @device, so the NBD client can use NBD_OPT_SET_META_CONTEXT with
+#           the metadata context name "qemu:dirty-bitmap:BITMAP" to inspect
+#           each bitmap. (since 5.2)

Given unsynchronised release cycles between qemu and management apps it's not cool to deprecate an interface without having at least one release where the replacement interface is already stable.

That's why I'm trying as hard as possible to get the block-export-add interface perfect in its 5.2 release; if we agree that allowing qemu to expose more than one bitmap is beneficial (and I argue that it is), then the new interface MUST support that from the get-go, and not something where we release it with 5.2 having 'bitmap' and 6.0 adding 'bitmaps'.

Quoted 9 lines

This means that any project wanting to stay up to date will either have to use deprecated interfaces for at least one release and develop the replacement only when there's a stable interface or hope that they don't have to change the interfaces too often.

This specifically impacts libvirt as we have validators which notify us that a deprecated interface is used and we want to stay in sync, so that there's one less group of users to worry about at the point qemu will want to delete the interface.

The deprecated interface is nbd-server-add; for _that_ interface, the 'bitmap' parameter will continue to work until nbd-server-add is removed in 6.1 (so you have all of 5.2 and 6.0 to switch from nbd-server-add to block-export-add).

What this patch does, then, is alter the deprecation from merely changing the command from nbd-server-add to block-export-add with all parameter names remaining the same, to instead changing both the command name and the 'bitmap'=>'bitmaps' parameter. But I agree that libvirt wants to do an all-or-none conversion: there is no reason for libvirt to use block-export-add until 5.2 is actually released, at which point we have locked in the new interface; and this patch is a demonstration that we are still debating about a tweak to that interface before it becomes locked in.

Quoted 13 lines

qapi/block-export.json line 91

Quoted 9 lines
qapi/block-export.json +1 -1
 # @allocation-depth: Also export the allocation depth map for @device, so
 #                    the NBD client can use NBD_OPT_SET_META_CONTEXT with
@@ -88,7 +88,7 @@
 ##
 { 'struct': 'BlockExportOptionsNbd',
   'data': { '*name': 'str', '*description': 'str',
-            '*bitmap': 'str', '*allocation-depth': 'bool' } }
+            '*bitmaps': ['str'], '*allocation-depth': 'bool' } }

This adds 'bitmaps' also to nbd-server-add, which should not happen. You probably want to stop using 'base' for 'NbdServerAddOptions' and just duplicate everything else.

I can respin this to NOT add 'bitmaps' to the legacy 'nbd-server-add', if you think that would be better. It is more complex in the QAPI code, but not too much more difficulty in the glue code; and the glue code all goes away in 6.1 when the deprecation cycle ends.

Quoted 15 lines

qapi/block-export.json line 107

Quoted 11 lines
qapi/block-export.json +5 -0
 # @NbdServerAddOptions:
@@ -100,12 +100,18 @@
 # @writable: Whether clients should be able to write to the device via the
 #            NBD connection (default false).
 #
+# @bitmap: Also export a single dirty bitmap reachable from @device, so the
+#          NBD client can use NBD_OPT_SET_META_CONTEXT with the metadata
+#          context name "qemu:dirty-bitmap:BITMAP" to inspect the bitmap
+#          (since 4.0).  Mutually exclusive with @bitmaps, and newer
+#          clients should use that instead.

This doesn't make sense, nbd-server-add never had @bitmaps. Also adding features to a deprecated interface doesn't IMO make sense if you want to motivate users switch to thne new one.

Fair enough. v6 coming up later today.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#18 lore raw

PATCH v5 7/12 nbd: Simplify qemu bitmap context name

In reply to Eric Blake

Each dirty bitmap already knows its name; by reducing the scope of the places where we construct "qemu:dirty-bitmap:NAME" strings, tracking the name is more localized, and there are fewer per-export fields to worry about. This in turn will make it easier for an upcoming patch to export more than one bitmap at once.

---
 nbd/server.c | 19 +++++++++----------
 1 file changed, 9 insertions(+), 10 deletions(-)
nbd/server.c +9 -10
index 884ffa00f1bd..05a8154975a1 100644
@@ -97,7 +97,6 @@ struct NBDExport {
 
     bool allocation_depth;
     BdrvDirtyBitmap *export_bitmap;
-    char *export_bitmap_context;
 };
 
 static QTAILQ_HEAD(, NBDExport) exports = QTAILQ_HEAD_INITIALIZER(exports);
@@ -882,14 +881,15 @@ static bool nbd_meta_qemu_query(NBDClient *client, NBDExportMetaContexts *meta,
     }
 
     if (nbd_strshift(&query, "dirty-bitmap:")) {
+        const char *bm_name;
+
         trace_nbd_negotiate_meta_query_parse("dirty-bitmap:");
         if (!meta->exp->export_bitmap) {
             trace_nbd_negotiate_meta_query_skip("no dirty-bitmap exported");
             return true;
         }
-        if (nbd_meta_empty_or_pattern(client,
-                                      meta->exp->export_bitmap_context +
-                                      strlen("qemu:dirty-bitmap:"), query)) {
+        bm_name = bdrv_dirty_bitmap_name(meta->exp->export_bitmap);
+        if (nbd_meta_empty_or_pattern(client, bm_name, query)) {
             meta->bitmap = true;
         }
         return true;
@@ -1025,8 +1025,11 @@ static int nbd_negotiate_meta_queries(NBDClient *client,
     }
 
     if (meta->bitmap) {
-        ret = nbd_negotiate_send_meta_context(client,
-                                              meta->exp->export_bitmap_context,
+        const char *bm_name = bdrv_dirty_bitmap_name(meta->exp->export_bitmap);
+        g_autofree char *context = g_strdup_printf("qemu:dirty-bitmap:%s",
+                                                   bm_name);
+
+        ret = nbd_negotiate_send_meta_context(client, context,
                                               NBD_META_ID_DIRTY_BITMAP,
                                               errp);
         if (ret < 0) {
@@ -1599,9 +1602,6 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
         bdrv_dirty_bitmap_set_busy(bm, true);
         exp->export_bitmap = bm;
         assert(strlen(bitmap) <= BDRV_BITMAP_MAX_NAME_SIZE);
-        exp->export_bitmap_context = g_strdup_printf("qemu:dirty-bitmap:%s",
-                                                     bitmap);
-        assert(strlen(exp->export_bitmap_context) < NBD_MAX_STRING_SIZE);
     }
 
     exp->allocation_depth = arg->allocation_depth;
@@ -1681,7 +1681,6 @@ static void nbd_export_delete(BlockExport *blk_exp)
 
     if (exp->export_bitmap) {
         bdrv_dirty_bitmap_set_busy(exp->export_bitmap, false);
-        g_free(exp->export_bitmap_context);
     }
 }
 
Signature · 1 line
2.29.0
author
#19 lore raw

PATCH v5 8/12 nbd: Refactor counting of metadata contexts

In reply to Eric Blake

Rather than open-code the count of negotiated contexts at several sites, embed it directly into the struct.

---
 nbd/server.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
nbd/server.c +11 -12
index 05a8154975a1..27d943529409 100644
@@ -106,8 +106,7 @@ static QTAILQ_HEAD(, NBDExport) exports = QTAILQ_HEAD_INITIALIZER(exports);
  * NBD_OPT_LIST_META_CONTEXT. */
 typedef struct NBDExportMetaContexts {
     NBDExport *exp;
-    bool valid; /* means that negotiation of the option finished without
-                   errors */
+    size_t count; /* number of negotiated contexts */
     bool base_allocation; /* export base:allocation context (block status) */
     bool allocation_depth; /* export qemu:allocation-depth */
     bool bitmap; /* export qemu:dirty-bitmap:<export bitmap name> */
@@ -448,7 +447,9 @@ static int nbd_negotiate_handle_list(NBDClient *client, Error **errp)
 
 static void nbd_check_meta_export(NBDClient *client)
 {
-    client->export_meta.valid &= client->exp == client->export_meta.exp;
+    if (client->exp != client->export_meta.exp) {
+        client->export_meta.count = 0;
+    }
 }
 
 /* Send a reply to NBD_OPT_EXPORT_NAME.
@@ -956,6 +957,7 @@ static int nbd_negotiate_meta_queries(NBDClient *client,
     NBDExportMetaContexts local_meta;
     uint32_t nb_queries;
     int i;
+    size_t count = 0;
 
     if (!client->structured_reply) {
         return nbd_opt_invalid(client, errp,
@@ -1013,6 +1015,7 @@ static int nbd_negotiate_meta_queries(NBDClient *client,
         if (ret < 0) {
             return ret;
         }
+        count++;
     }
 
     if (meta->allocation_depth) {
@@ -1022,6 +1025,7 @@ static int nbd_negotiate_meta_queries(NBDClient *client,
         if (ret < 0) {
             return ret;
         }
+        count++;
     }
 
     if (meta->bitmap) {
@@ -1035,11 +1039,12 @@ static int nbd_negotiate_meta_queries(NBDClient *client,
         if (ret < 0) {
             return ret;
         }
+        count++;
     }
 
     ret = nbd_negotiate_send_rep(client, NBD_REP_ACK, errp);
     if (ret == 0) {
-        meta->valid = true;
+        meta->count = count;
     }
 
     return ret;
@@ -2400,15 +2405,9 @@ static coroutine_fn int nbd_handle_request(NBDClient *client,
             return nbd_send_generic_reply(client, request->handle, -EINVAL,
                                           "need non-zero length", errp);
         }
-        if (client->export_meta.valid &&
-            (client->export_meta.base_allocation ||
-             client->export_meta.allocation_depth ||
-             client->export_meta.bitmap))
-        {
+        if (client->export_meta.count) {
             bool dont_fragment = request->flags & NBD_CMD_FLAG_REQ_ONE;
-            int contexts_remaining = client->export_meta.base_allocation +
-                client->export_meta.allocation_depth +
-                client->export_meta.bitmap;
+            int contexts_remaining = client->export_meta.count;
 
             if (client->export_meta.base_allocation) {
                 ret = nbd_co_send_block_status(client, request->handle,
Signature · 1 line
2.29.0
author
#20 lore raw

PATCH v5 9/12 nbd: Allow export of multiple bitmaps for one device

In reply to Eric Blake

With this, 'qemu-nbd -B b0 -B b1 -f qcow2 img.qcow2' can let you sniff out multiple bitmaps from one server. qemu-img as client can still only read one bitmap per client connection, but other NBD clients (hello libnbd) can now read multiple bitmaps in a single pass.

---
 nbd/server.c           | 100 ++++++++++++++++++++++++++++-------------
 tests/qemu-iotests/291 |   6 +--
 2 files changed, 72 insertions(+), 34 deletions(-)
nbd/server.c +70 -30
index 27d943529409..53526090b0a2 100644
@@ -28,6 +28,7 @@
 
 #define NBD_META_ID_BASE_ALLOCATION 0
 #define NBD_META_ID_ALLOCATION_DEPTH 1
+/* Dirty bitmaps use 'NBD_META_ID_DIRTY_BITMAP + i', so keep this id last. */
 #define NBD_META_ID_DIRTY_BITMAP 2
 
 /*
@@ -96,7 +97,8 @@ struct NBDExport {
     Notifier eject_notifier;
 
     bool allocation_depth;
-    BdrvDirtyBitmap *export_bitmap;
+    BdrvDirtyBitmap **export_bitmaps;
+    size_t nr_export_bitmaps;
 };
 
 static QTAILQ_HEAD(, NBDExport) exports = QTAILQ_HEAD_INITIALIZER(exports);
@@ -109,7 +111,10 @@ typedef struct NBDExportMetaContexts {
     size_t count; /* number of negotiated contexts */
     bool base_allocation; /* export base:allocation context (block status) */
     bool allocation_depth; /* export qemu:allocation-depth */
-    bool bitmap; /* export qemu:dirty-bitmap:<export bitmap name> */
+    bool *bitmaps; /*
+                    * export qemu:dirty-bitmap:<export bitmap name>,
+                    * sized by exp->nr_export_bitmaps
+                    */
 } NBDExportMetaContexts;
 
 struct NBDClient {
@@ -861,6 +866,8 @@ static bool nbd_meta_base_query(NBDClient *client, NBDExportMetaContexts *meta,
 static bool nbd_meta_qemu_query(NBDClient *client, NBDExportMetaContexts *meta,
                                 const char *query)
 {
+    size_t i;
+
     if (!nbd_strshift(&query, "qemu:")) {
         return false;
     }
@@ -869,7 +876,7 @@ static bool nbd_meta_qemu_query(NBDClient *client, NBDExportMetaContexts *meta,
     if (!*query) {
         if (client->opt == NBD_OPT_LIST_META_CONTEXT) {
             meta->allocation_depth = meta->exp->allocation_depth;
-            meta->bitmap = !!meta->exp->export_bitmap;
+            memset(meta->bitmaps, 1, meta->exp->nr_export_bitmaps);
         }
         trace_nbd_negotiate_meta_query_parse("empty");
         return true;
@@ -882,17 +889,26 @@ static bool nbd_meta_qemu_query(NBDClient *client, NBDExportMetaContexts *meta,
     }
 
     if (nbd_strshift(&query, "dirty-bitmap:")) {
-        const char *bm_name;
-
         trace_nbd_negotiate_meta_query_parse("dirty-bitmap:");
-        if (!meta->exp->export_bitmap) {
-            trace_nbd_negotiate_meta_query_skip("no dirty-bitmap exported");
+        if (!*query) {
+            if (client->opt == NBD_OPT_LIST_META_CONTEXT) {
+                memset(meta->bitmaps, 1, meta->exp->nr_export_bitmaps);
+            }
+            trace_nbd_negotiate_meta_query_parse("empty");
             return true;
         }
-        bm_name = bdrv_dirty_bitmap_name(meta->exp->export_bitmap);
-        if (nbd_meta_empty_or_pattern(client, bm_name, query)) {
-            meta->bitmap = true;
+
+        for (i = 0; i < meta->exp->nr_export_bitmaps; i++) {
+            const char *bm_name;
+
+            bm_name = bdrv_dirty_bitmap_name(meta->exp->export_bitmaps[i]);
+            if (strcmp(bm_name, query) == 0) {
+                meta->bitmaps[i] = true;
+                trace_nbd_negotiate_meta_query_parse(query);
+                return true;
+            }
         }
+        trace_nbd_negotiate_meta_query_skip("no dirty-bitmap match");
         return true;
     }
 
@@ -954,9 +970,10 @@ static int nbd_negotiate_meta_queries(NBDClient *client,
 {
     int ret;
     g_autofree char *export_name = NULL;
-    NBDExportMetaContexts local_meta;
+    g_autofree bool *bitmaps = NULL;
+    NBDExportMetaContexts local_meta = {0};
     uint32_t nb_queries;
-    int i;
+    size_t i;
     size_t count = 0;
 
     if (!client->structured_reply) {
@@ -971,6 +988,7 @@ static int nbd_negotiate_meta_queries(NBDClient *client,
         meta = &local_meta;
     }
 
+    g_free(meta->bitmaps);
     memset(meta, 0, sizeof(*meta));
 
     ret = nbd_opt_read_name(client, &export_name, NULL, errp);
@@ -985,6 +1003,10 @@ static int nbd_negotiate_meta_queries(NBDClient *client,
         return nbd_opt_drop(client, NBD_REP_ERR_UNKNOWN, errp,
                             "export '%s' not present", sane_name);
     }
+    meta->bitmaps = g_new0(bool, meta->exp->nr_export_bitmaps);
+    if (client->opt == NBD_OPT_LIST_META_CONTEXT) {
+        bitmaps = meta->bitmaps;
+    }
 
     ret = nbd_opt_read(client, &nb_queries, sizeof(nb_queries), false, errp);
     if (ret <= 0) {
@@ -998,7 +1020,7 @@ static int nbd_negotiate_meta_queries(NBDClient *client,
         /* enable all known contexts */
         meta->base_allocation = true;
         meta->allocation_depth = meta->exp->allocation_depth;
-        meta->bitmap = !!meta->exp->export_bitmap;
+        memset(meta->bitmaps, 1, meta->exp->nr_export_bitmaps);
     } else {
         for (i = 0; i < nb_queries; ++i) {
             ret = nbd_negotiate_meta_query(client, meta, errp);
@@ -1028,13 +1050,19 @@ static int nbd_negotiate_meta_queries(NBDClient *client,
         count++;
     }
 
-    if (meta->bitmap) {
-        const char *bm_name = bdrv_dirty_bitmap_name(meta->exp->export_bitmap);
-        g_autofree char *context = g_strdup_printf("qemu:dirty-bitmap:%s",
-                                                   bm_name);
+    for (i = 0; i < meta->exp->nr_export_bitmaps; i++) {
+        const char *bm_name;
+        g_autofree char *context = NULL;
+
+        if (!meta->bitmaps[i]) {
+            continue;
+        }
+
+        bm_name = bdrv_dirty_bitmap_name(meta->exp->export_bitmaps[i]);
+        context = g_strdup_printf("qemu:dirty-bitmap:%s", bm_name);
 
         ret = nbd_negotiate_send_meta_context(client, context,
-                                              NBD_META_ID_DIRTY_BITMAP,
+                                              NBD_META_ID_DIRTY_BITMAP + i,
                                               errp);
         if (ret < 0) {
             return ret;
@@ -1388,6 +1416,7 @@ void nbd_client_put(NBDClient *client)
             QTAILQ_REMOVE(&client->exp->clients, client, next);
             blk_exp_unref(&client->exp->common);
         }
+        g_free(client->export_meta.bitmaps);
         g_free(client);
     }
 }
@@ -1504,6 +1533,7 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
     bool readonly = !exp_args->writable;
     bool shared = !exp_args->writable;
     strList *bitmaps;
+    size_t i;
     int ret;
 
     assert(exp_args->type == BLOCK_EXPORT_TYPE_NBD);
@@ -1565,12 +1595,12 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
     }
     exp->size = QEMU_ALIGN_DOWN(size, BDRV_SECTOR_SIZE);
 
-    /* XXX Allow more than one bitmap */
-    if (arg->bitmaps && arg->bitmaps->next) {
-        error_setg(errp, "multiple bitmaps per export not supported yet");
-        return -EOPNOTSUPP;
-    }
     for (bitmaps = arg->bitmaps; bitmaps; bitmaps = bitmaps->next) {
+        exp->nr_export_bitmaps++;
+    }
+    exp->export_bitmaps = g_new0(BdrvDirtyBitmap *, exp->nr_export_bitmaps);
+    for (i = 0, bitmaps = arg->bitmaps; bitmaps;
+         i++, bitmaps = bitmaps->next) {
         const char *bitmap = bitmaps->value;
         BlockDriverState *bs = blk_bs(blk);
         BdrvDirtyBitmap *bm = NULL;
@@ -1604,11 +1634,15 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
             goto fail;
         }
 
-        bdrv_dirty_bitmap_set_busy(bm, true);
-        exp->export_bitmap = bm;
+        exp->export_bitmaps[i] = bm;
         assert(strlen(bitmap) <= BDRV_BITMAP_MAX_NAME_SIZE);
     }
 
+    /* Mark bitmaps busy in a separate loop, to simplify roll-back concerns. */
+    for (i = 0; i < exp->nr_export_bitmaps; i++) {
+        bdrv_dirty_bitmap_set_busy(exp->export_bitmaps[i], true);
+    }
+
     exp->allocation_depth = arg->allocation_depth;
 
     blk_add_aio_context_notifier(blk, blk_aio_attached, blk_aio_detach, exp);
@@ -1618,6 +1652,7 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
     return 0;
 
 fail:
+    g_free(exp->export_bitmaps);
     g_free(exp->name);
     g_free(exp->description);
     return ret;
@@ -1667,6 +1702,7 @@ static void nbd_export_request_shutdown(BlockExport *blk_exp)
 
 static void nbd_export_delete(BlockExport *blk_exp)
 {
+    size_t i;
     NBDExport *exp = container_of(blk_exp, NBDExport, common);
 
     assert(exp->name == NULL);
@@ -1684,8 +1720,8 @@ static void nbd_export_delete(BlockExport *blk_exp)
                                         blk_aio_detach, exp);
     }
 
-    if (exp->export_bitmap) {
-        bdrv_dirty_bitmap_set_busy(exp->export_bitmap, false);
+    for (i = 0; i < exp->nr_export_bitmaps; i++) {
+        bdrv_dirty_bitmap_set_busy(exp->export_bitmaps[i], false);
     }
 }
 
@@ -2332,6 +2368,7 @@ static coroutine_fn int nbd_handle_request(NBDClient *client,
     int flags;
     NBDExport *exp = client->exp;
     char *msg;
+    size_t i;
 
     switch (request->type) {
     case NBD_CMD_CACHE:
@@ -2435,12 +2472,15 @@ static coroutine_fn int nbd_handle_request(NBDClient *client,
                 }
             }
 
-            if (client->export_meta.bitmap) {
+            for (i = 0; i < client->exp->nr_export_bitmaps; i++) {
+                if (!client->export_meta.bitmaps[i]) {
+                    continue;
+                }
                 ret = nbd_co_send_bitmap(client, request->handle,
-                                         client->exp->export_bitmap,
+                                         client->exp->export_bitmaps[i],
                                          request->from, request->len,
                                          dont_fragment, !--contexts_remaining,
-                                         NBD_META_ID_DIRTY_BITMAP, errp);
+                                         NBD_META_ID_DIRTY_BITMAP + i, errp);
                 if (ret < 0) {
                     return ret;
                 }
tests/qemu-iotests/291 +2 -4
index 4f837b205655..37848ac60bba 100755
@@ -107,16 +107,14 @@ echo
 # x-dirty-bitmap is a hack for reading bitmaps; it abuses block status to
 # report "data":false for portions of the bitmap which are set
 IMG="driver=nbd,server.type=unix,server.path=$nbd_unix_socket"
-nbd_server_start_unix_socket -r -f qcow2 -B b0 "$TEST_IMG"
+nbd_server_start_unix_socket -r -f qcow2 \
+    -B b0 -B b1 -B b2 -B b3 "$TEST_IMG"
 $QEMU_IMG map --output=json --image-opts \
     "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b0" | _filter_qemu_img_map
-nbd_server_start_unix_socket -r -f qcow2 -B b1 "$TEST_IMG"
 $QEMU_IMG map --output=json --image-opts \
     "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b1" | _filter_qemu_img_map
-nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG"
 $QEMU_IMG map --output=json --image-opts \
     "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map
-nbd_server_start_unix_socket -r -f qcow2 -B b3 "$TEST_IMG"
 $QEMU_IMG map --output=json --image-opts \
     "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b3" | _filter_qemu_img_map
 
Signature · 1 line
2.29.0
author
#21 lore raw

PATCH v5 10/12 block: Return depth level during bdrv_is_allocated_above

In reply to Eric Blake

When checking for allocation across a chain, it's already easy to count the depth within the chain at which the allocation is found. Instead of throwing that information away, return it to the caller. Existing callers only cared about allocated/non-allocated, but having a depth available will be used by NBD in the next patch.

Note that the previous code (since commit 188a7bbf94 in 2012) was
lazy: for each layer deeper in the backing chain, it was issuing a bdrv_is_allocated request on the original 'bytes' amount, rather than on any smaller amount 'n' learned from an upper layer. These semantics happened to work, since if you have:

Base <- Active
XX-- -XX-

the consecutive results are offset 0: '11' with *pnum 2, followed by offset 2: '1' with *pnum 1, followed by offset 3: '0' with *pnum 1; the resulting sequence 1110 matches reality (the first three clusters are indeed allocated somewhere in the given range). But post-patch, we correctly give the results offset 0: '2' with *pnum 1, followed by offset 1: '11' with *pnum 2, followed by offset 3: '0' with *pnum 1 (2110), without over-reporting the length of contributions from the backing layers.

---
 block/io.c     | 11 +++++++----
 block/commit.c |  2 +-
 block/mirror.c |  2 +-
 block/stream.c |  2 +-
 4 files changed, 10 insertions(+), 7 deletions(-)
block/io.c +7 -4
index 54f0968aee27..4a4fa1c9ab1b 100644
@@ -2414,8 +2414,9 @@ int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset,
 /*
  * Given an image chain: ... -> [BASE] -> [INTER1] -> [INTER2] -> [TOP]
  *
- * Return 1 if (a prefix of) the given range is allocated in any image
- * between BASE and TOP (BASE is only included if include_base is set).
+ * Return a positive depth if (a prefix of) the given range is allocated
+ * in any image between BASE and TOP (BASE is only included if include_base
+ * is set).  Depth 1 is TOP, 2 is the first backing layer, and so forth.
  * BASE can be NULL to check if the given offset is allocated in any
  * image of the chain.  Return 0 otherwise, or negative errno on
  * failure.
@@ -2435,6 +2436,7 @@ int bdrv_is_allocated_above(BlockDriverState *top,
 {
     BlockDriverState *intermediate;
     int ret;
+    int depth = 0;
     int64_t n = bytes;
 
     assert(base || !include_base);
@@ -2444,14 +2446,15 @@ int bdrv_is_allocated_above(BlockDriverState *top,
         int64_t pnum_inter;
         int64_t size_inter;
 
+        depth++;
         assert(intermediate);
-        ret = bdrv_is_allocated(intermediate, offset, bytes, &pnum_inter);
+        ret = bdrv_is_allocated(intermediate, offset, n, &pnum_inter);
         if (ret < 0) {
             return ret;
         }
         if (ret) {
             *pnum = pnum_inter;
-            return 1;
+            return depth;
         }
 
         size_inter = bdrv_getlength(intermediate);
block/commit.c +1 -1
index 1e85c306cc41..71db7ba7472e 100644
@@ -156,7 +156,7 @@ static int coroutine_fn commit_run(Job *job, Error **errp)
         /* Copy if allocated above the base */
         ret = bdrv_is_allocated_above(blk_bs(s->top), s->base_overlay, true,
                                       offset, COMMIT_BUFFER_SIZE, &n);
-        copy = (ret == 1);
+        copy = (ret > 0);
         trace_commit_one_iteration(s, offset, n, ret);
         if (copy) {
             assert(n < SIZE_MAX);
block/mirror.c +1 -1
index 26acf4af6fb7..8e1ad6eceb57 100644
@@ -846,7 +846,7 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
         }
 
         assert(count);
-        if (ret == 1) {
+        if (ret > 0) {
             bdrv_set_dirty_bitmap(s->dirty_bitmap, offset, count);
         }
         offset += count;
block/stream.c +1 -1
index 8ce6729a33da..236384f2f739 100644
@@ -167,7 +167,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
                 n = len - offset;
             }
 
-            copy = (ret == 1);
+            copy = (ret > 0);
         }
         trace_stream_one_iteration(s, offset, n, ret);
         if (copy) {
Signature · 1 line
2.29.0
#22 lore raw

Re: [PATCH v5 10/12] block: Return depth level during bdrv_is_allocated_above

Quoted 24 lines · Eric Blake

When checking for allocation across a chain, it's already easy to count the depth within the chain at which the allocation is found. Instead of throwing that information away, return it to the caller. Existing callers only cared about allocated/non-allocated, but having a depth available will be used by NBD in the next patch.

Note that the previous code (since commit 188a7bbf94 in 2012) was
lazy: for each layer deeper in the backing chain, it was issuing a bdrv_is_allocated request on the original 'bytes' amount, rather than on any smaller amount 'n' learned from an upper layer. These semantics happened to work, since if you have:

Base <- Active
XX-- -XX-

the consecutive results are offset 0: '11' with *pnum 2, followed by offset 2: '1' with *pnum 1, followed by offset 3: '0' with *pnum 1; the resulting sequence 1110 matches reality (the first three clusters are indeed allocated somewhere in the given range). But post-patch, we correctly give the results offset 0: '2' with *pnum 1, followed by offset 1: '11' with *pnum 2, followed by offset 3: '0' with *pnum 1 (2110), without over-reporting the length of contributions from the backing layers.

Quoted 1 line

This conflicts with block-status/is-allocated fix/merge patches in Stefan's "[PULL v3 00/28] Block patches"

Signature · 2 lines
Best regards,
Vladimir
#23 lore raw

Re: [PATCH v5 10/12] block: Return depth level during bdrv_is_allocated_above

Quoted 25 lines · Vladimir Sementsov-Ogievskiy
Quoted 24 lines · Eric Blake

When checking for allocation across a chain, it's already easy to count the depth within the chain at which the allocation is found. Instead of throwing that information away, return it to the caller. Existing callers only cared about allocated/non-allocated, but having a depth available will be used by NBD in the next patch.

Note that the previous code (since commit 188a7bbf94 in 2012) was
lazy: for each layer deeper in the backing chain, it was issuing a bdrv_is_allocated request on the original 'bytes' amount, rather than on any smaller amount 'n' learned from an upper layer.  These semantics happened to work, since if you have:

Base <- Active
XX--    -XX-

the consecutive results are offset 0: '11' with *pnum 2, followed by offset 2: '1' with *pnum 1, followed by offset 3: '0' with *pnum 1; the resulting sequence 1110 matches reality (the first three clusters are indeed allocated somewhere in the given range).  But post-patch, we correctly give the results offset 0: '2' with *pnum 1, followed by offset 1: '11' with *pnum 2, followed by offset 3: '0' with *pnum 1 (2110), without over-reporting the length of contributions from the backing layers.

Quoted 4 lines
Quoted 1 line

This conflicts with block-status/is-allocated fix/merge patches in Stefan's "[PULL v3 00/28] Block patches"

the patches applied to master branch

Signature · 2 lines
Best regards,
Vladimir
author
#24 lore raw

PATCH v5 11/12 nbd: Expose actual depth in qemu:allocation-depth

In reply to Eric Blake

Preserve the tri-state encoding in the low bits, as that still remains a valuable way to utilize qemu-img map with x-dirty-bitmap for accessing quick information without needing a third-party NBD client. But now that the block layer gives us an actual depth, we can easily expose it in the remaining bits of our metadata context (leaving two bits reserved, to make it easier to read depth out of a raw hex number). This assumes no one runs a backing chain larger than 256M elements.

iotest 309 remains unchanged (an example of the above-mentioned x-dirty-bitmap hack); actually testing the new bits requires libnbd or a similar client, and I didn't want to make iotests depend on libnbd at this point in time; rather, see the libnbd project for interop tests that exercise this new feature.

---
 docs/interop/nbd.txt |  6 +++++-
 include/block/nbd.h  |  2 ++
 nbd/server.c         | 27 +++++++++++++++------------
 3 files changed, 22 insertions(+), 13 deletions(-)
docs/interop/nbd.txt +5 -1
index 7e948bd42218..d90723ffe991 100644
@@ -34,7 +34,8 @@ the image, with a single metadata context named:
 
     qemu:allocation-depth
 
-In the allocation depth context, bits 0 and 1 form a tri-state value:
+In the allocation depth context, bits 0 and 1 form a tri-state value,
+along with 28 bits giving an actual depth:
 
     bits 0-1: 00: NBD_STATE_DEPTH_UNALLOC, the extent is unallocated
               01: NBD_STATE_DEPTH_LOCAL, the extent is allocated in the
@@ -42,6 +43,9 @@ In the allocation depth context, bits 0 and 1 form a tri-state value:
               10: NBD_STATE_DEPTH_BACKING, the extent is inherited from a
                   backing layer
               11: invalid, never returned
+    bits 2-3: reserved, always 0
+    bits 4-31: NBD_STATE_DEPTH_RAW, the backing layer depth (0 if
+               UNALLOC, 1 for LOCAL, 2 or more for BACKING)
 
 For NBD_OPT_LIST_META_CONTEXT the following queries are supported
 in addition to the specific "qemu:allocation-depth" and
include/block/nbd.h +2 -0
index 956687f5c368..3c0692aec642 100644
@@ -264,6 +264,8 @@ enum {
 #define NBD_STATE_DEPTH_UNALLOC      0x0
 #define NBD_STATE_DEPTH_LOCAL        0x1
 #define NBD_STATE_DEPTH_BACKING      0x2
+#define NBD_STATE_DEPTH_RAW_MASK     0xfffffff0
+#define NBD_STATE_DEPTH_RAW_SHIFT    4
 
 static inline bool nbd_reply_type_is_error(int type)
 {
nbd/server.c +15 -12
index 53526090b0a2..afa79e63a7a6 100644
@@ -2037,22 +2037,25 @@ static int blockalloc_to_extents(BlockDriverState *bs, uint64_t offset,
     while (bytes) {
         uint32_t flags;
         int64_t num;
-        int ret = bdrv_is_allocated(bs, offset, bytes, &num);
+        int depth = bdrv_is_allocated_above(bs, NULL, false, offset, bytes,
+                                            &num);
 
-        if (ret < 0) {
-            return ret;
-        }
-
-        if (ret == 1) {
+        switch (depth) {
+        case 0:
+            flags = NBD_STATE_DEPTH_UNALLOC;
+            break;
+        case 1:
             flags = NBD_STATE_DEPTH_LOCAL;
-        } else {
-            ret = bdrv_is_allocated_above(bs, NULL, false, offset, num,
-                                          &num);
-            if (ret < 0) {
-                return ret;
+            break;
+        default:
+            if (depth < 0) {
+                return depth;
             }
-            flags = ret ? NBD_STATE_DEPTH_BACKING : NBD_STATE_DEPTH_UNALLOC;
+            flags = NBD_STATE_DEPTH_BACKING;
+            break;
         }
+        assert(depth <= UINT32_MAX >> NBD_STATE_DEPTH_RAW_SHIFT);
+        flags |= depth << NBD_STATE_DEPTH_RAW_SHIFT;
 
         if (nbd_extent_array_add(ea, num, flags) < 0) {
             return 0;
Signature · 1 line
2.29.0
#25 lore raw

Re: [PATCH v5 11/12] nbd: Expose actual depth in qemu:allocation-depth

Quoted 3 lines · Eric Blake

Preserve the tri-state encoding in the low bits, as that still remains a valuable way to utilize qemu-img map with x-dirty-bitmap for accessing quick information without needing a third-party NBD client.

Hmm.. that doesn't sound as a good reason for redundant information in the protocol. Previously good reason was additional effort needed to implement sever part, but you've implemented it. And if we export depth anyway, it seems better to hack a bit nbd_client_co_block_status to convert extent.flags appropriately if metadata context is "qemu:allocation-depth" (to keep x-dirty-bitmap working), than have a workaround at the protocol layer.

Quoted 99 lines

But now that the block layer gives us an actual depth, we can easily expose it in the remaining bits of our metadata context (leaving two bits reserved, to make it easier to read depth out of a raw hex number). This assumes no one runs a backing chain larger than 256M elements.

iotest 309 remains unchanged (an example of the above-mentioned x-dirty-bitmap hack); actually testing the new bits requires libnbd or a similar client, and I didn't want to make iotests depend on libnbd at this point in time; rather, see the libnbd project for interop tests that exercise this new feature.

---
  docs/interop/nbd.txt |  6 +++++-
  include/block/nbd.h  |  2 ++
  nbd/server.c         | 27 +++++++++++++++------------
  3 files changed, 22 insertions(+), 13 deletions(-)
docs/interop/nbd.txt +5 -1
index 7e948bd42218..d90723ffe991 100644
@@ -34,7 +34,8 @@ the image, with a single metadata context named:
 
      qemu:allocation-depth
 
-In the allocation depth context, bits 0 and 1 form a tri-state value:
+In the allocation depth context, bits 0 and 1 form a tri-state value,
+along with 28 bits giving an actual depth:
 
      bits 0-1: 00: NBD_STATE_DEPTH_UNALLOC, the extent is unallocated
                01: NBD_STATE_DEPTH_LOCAL, the extent is allocated in the
@@ -42,6 +43,9 @@ In the allocation depth context, bits 0 and 1 form a tri-state value:
                10: NBD_STATE_DEPTH_BACKING, the extent is inherited from a
                    backing layer
                11: invalid, never returned
+    bits 2-3: reserved, always 0
+    bits 4-31: NBD_STATE_DEPTH_RAW, the backing layer depth (0 if
+               UNALLOC, 1 for LOCAL, 2 or more for BACKING)
 
  For NBD_OPT_LIST_META_CONTEXT the following queries are supported
  in addition to the specific "qemu:allocation-depth" and
include/block/nbd.h +2 -0
index 956687f5c368..3c0692aec642 100644
@@ -264,6 +264,8 @@ enum {
  #define NBD_STATE_DEPTH_UNALLOC      0x0
  #define NBD_STATE_DEPTH_LOCAL        0x1
  #define NBD_STATE_DEPTH_BACKING      0x2
+#define NBD_STATE_DEPTH_RAW_MASK     0xfffffff0
+#define NBD_STATE_DEPTH_RAW_SHIFT    4
 
  static inline bool nbd_reply_type_is_error(int type)
  {
nbd/server.c +15 -12
index 53526090b0a2..afa79e63a7a6 100644
@@ -2037,22 +2037,25 @@ static int blockalloc_to_extents(BlockDriverState *bs, uint64_t offset,
      while (bytes) {
          uint32_t flags;
          int64_t num;
-        int ret = bdrv_is_allocated(bs, offset, bytes, &num);
+        int depth = bdrv_is_allocated_above(bs, NULL, false, offset, bytes,
+                                            &num);
 
-        if (ret < 0) {
-            return ret;
-        }
-
-        if (ret == 1) {
+        switch (depth) {
+        case 0:
+            flags = NBD_STATE_DEPTH_UNALLOC;
+            break;
+        case 1:
              flags = NBD_STATE_DEPTH_LOCAL;
-        } else {
-            ret = bdrv_is_allocated_above(bs, NULL, false, offset, num,
-                                          &num);
-            if (ret < 0) {
-                return ret;
+            break;
+        default:
+            if (depth < 0) {
+                return depth;
              }
-            flags = ret ? NBD_STATE_DEPTH_BACKING : NBD_STATE_DEPTH_UNALLOC;
+            flags = NBD_STATE_DEPTH_BACKING;
+            break;
          }
+        assert(depth <= UINT32_MAX >> NBD_STATE_DEPTH_RAW_SHIFT);
+        flags |= depth << NBD_STATE_DEPTH_RAW_SHIFT;
 
          if (nbd_extent_array_add(ea, num, flags) < 0) {
              return 0;
Signature · 2 lines
Best regards,
Vladimir
author
#26 lore raw

Re: [PATCH v5 11/12] nbd: Expose actual depth in qemu:allocation-depth

Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Quoted 3 lines · Eric Blake

Preserve the tri-state encoding in the low bits, as that still remains a valuable way to utilize qemu-img map with x-dirty-bitmap for accessing quick information without needing a third-party NBD client.

Quoted 7 lines

Hmm.. that doesn't sound as a good reason for redundant information in the protocol. Previously good reason was additional effort needed to implement sever part, but you've implemented it. And if we export depth anyway, it seems better to hack a bit nbd_client_co_block_status to convert extent.flags appropriately if metadata context is "qemu:allocation-depth" (to keep x-dirty-bitmap working), than have a workaround at the protocol layer.

I'm happy to respin this to expose JUST a depth rather than redundant information, but time is short if we want it in 5.2 (as soft freeze is this week). I'll see what I can get to today; I'll rearrange the series to put multiple bitmap exports first (as that appears ready), while saving 'qemu-nbd -A' until we're happy with the qemu:allocation-depth semantics. After all, once we release something, we've committed to that user interface.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#27 lore raw

PATCH v5 12/12 qapi: Use QAPI_LIST_ADD() where possible

In reply to Eric Blake

Anywhere we create a list of just one item or by prepending items (typically because order doesn't matter), we can use the now-public macro. But places where we must keep the list in order by appending remain open-coded.

---
 docs/devel/writing-qmp-commands.txt | 13 +++------
 hw/net/rocker/rocker_fp.h           |  2 +-
 block/gluster.c                     | 19 +++++--------
 chardev/char.c                      | 21 +++++++--------
 hw/core/machine.c                   |  6 +----
 hw/net/rocker/rocker.c              |  8 +++---
 hw/net/rocker/rocker_fp.c           | 14 +++++-----
 hw/net/virtio-net.c                 | 21 +++++----------
 migration/migration.c               |  7 ++---
 migration/postcopy-ram.c            |  7 ++---
 monitor/hmp-cmds.c                  | 11 ++++----
 qemu-img.c                          |  5 ++--
 qga/commands-posix.c                | 13 +++------
 qga/commands-win32.c                | 17 +++---------
 qga/commands.c                      |  6 +----
 qom/qom-qmp-cmds.c                  | 29 ++++++--------------
 target/arm/helper.c                 |  6 +----
 target/arm/monitor.c                | 13 ++-------
 target/i386/cpu.c                   |  6 +----
 target/mips/helper.c                |  6 +----
 target/s390x/cpu_models.c           | 12 ++-------
 tests/test-clone-visitor.c          |  7 ++---
 tests/test-qobject-output-visitor.c | 42 ++++++++++++++---------------
 tests/test-visitor-serialization.c  |  5 +---
 trace/qmp.c                         | 22 +++++++--------
 ui/vnc.c                            | 21 +++++----------
 util/qemu-config.c                  | 14 +++-------
 target/ppc/translate_init.c.inc     | 12 ++-------
 28 files changed, 119 insertions(+), 246 deletions(-)
docs/devel/writing-qmp-commands.txt +4 -9
index 46a6c48683f5..3e11eeaa1893 100644
@@ -531,15 +531,10 @@ TimerAlarmMethodList *qmp_query_alarm_methods(Error **errp)
     bool current = true;
 
     for (p = alarm_timers; p->name; p++) {
-        TimerAlarmMethodList *info = g_malloc0(sizeof(*info));
-        info->value = g_malloc0(sizeof(*info->value));
-        info->value->method_name = g_strdup(p->name);
-        info->value->current = current;
-
-        current = false;
-
-        info->next = method_list;
-        method_list = info;
+	TimerAlarmMethod *value = g_new0(TimerAlarmMethod, 1);
+        value->method_name = g_strdup(p->name);
+        value->current = current;
+        QAPI_LIST_ADD(method_list, value);
     }
 
     return method_list;
hw/net/rocker/rocker_fp.h +1 -1
index dbe1dd329a4b..4cb0bb9ccf81 100644
@@ -28,7 +28,7 @@ int fp_port_eg(FpPort *port, const struct iovec *iov, int iovcnt);
 
 char *fp_port_get_name(FpPort *port);
 bool fp_port_get_link_up(FpPort *port);
-void fp_port_get_info(FpPort *port, RockerPortList *info);
+void fp_port_get_info(FpPort *port, RockerPort *info);
 void fp_port_get_macaddr(FpPort *port, MACAddr *macaddr);
 void fp_port_set_macaddr(FpPort *port, MACAddr *macaddr);
 uint8_t fp_port_get_learning(FpPort *port);
block/gluster.c +7 -12
index 4f1448e2bc88..cf446c23f85d 100644
@@ -359,8 +359,8 @@ static int qemu_gluster_parse_uri(BlockdevOptionsGluster *gconf,
         return -EINVAL;
     }
 
-    gconf->server = g_new0(SocketAddressList, 1);
-    gconf->server->value = gsconf = g_new0(SocketAddress, 1);
+    gsconf = g_new0(SocketAddress, 1);
+    QAPI_LIST_ADD(gconf->server, gsconf);
 
     /* transport */
     if (!uri->scheme || !strcmp(uri->scheme, "gluster")) {
@@ -514,7 +514,7 @@ static int qemu_gluster_parse_json(BlockdevOptionsGluster *gconf,
 {
     QemuOpts *opts;
     SocketAddress *gsconf = NULL;
-    SocketAddressList *curr = NULL;
+    SocketAddressList **curr;
     QDict *backing_options = NULL;
     Error *local_err = NULL;
     char *str = NULL;
@@ -547,6 +547,7 @@ static int qemu_gluster_parse_json(BlockdevOptionsGluster *gconf,
     }
     gconf->path = g_strdup(ptr);
     qemu_opts_del(opts);
+    curr = &gconf->server;
 
     for (i = 0; i < num_servers; i++) {
         str = g_strdup_printf(GLUSTER_OPT_SERVER_PATTERN"%d.", i);
@@ -655,15 +656,9 @@ static int qemu_gluster_parse_json(BlockdevOptionsGluster *gconf,
             qemu_opts_del(opts);
         }
 
-        if (gconf->server == NULL) {
-            gconf->server = g_new0(SocketAddressList, 1);
-            gconf->server->value = gsconf;
-            curr = gconf->server;
-        } else {
-            curr->next = g_new0(SocketAddressList, 1);
-            curr->next->value = gsconf;
-            curr = curr->next;
-        }
+        *curr = g_new0(SocketAddressList, 1);
+        (*curr)->value = gsconf;
+        curr = &(*curr)->next;
         gsconf = NULL;
 
         qobject_unref(backing_options);
chardev/char.c +9 -12
index 78553125d311..8dd7ef4c5935 100644
@@ -776,15 +776,14 @@ static int qmp_query_chardev_foreach(Object *obj, void *data)
 {
     Chardev *chr = CHARDEV(obj);
     ChardevInfoList **list = data;
-    ChardevInfoList *info = g_malloc0(sizeof(*info));
+    ChardevInfo *value;
 
-    info->value = g_malloc0(sizeof(*info->value));
-    info->value->label = g_strdup(chr->label);
-    info->value->filename = g_strdup(chr->filename);
-    info->value->frontend_open = chr->be && chr->be->fe_open;
+    value = g_malloc0(sizeof(*value));
+    value->label = g_strdup(chr->label);
+    value->filename = g_strdup(chr->filename);
+    value->frontend_open = chr->be && chr->be->fe_open;
 
-    info->next = *list;
-    *list = info;
+    QAPI_LIST_ADD(*list, value);
 
     return 0;
 }
@@ -803,12 +802,10 @@ static void
 qmp_prepend_backend(const char *name, void *opaque)
 {
     ChardevBackendInfoList **list = opaque;
-    ChardevBackendInfoList *info = g_malloc0(sizeof(*info));
+    ChardevBackendInfo *value = g_new0(ChardevBackendInfo, 1);
 
-    info->value = g_malloc0(sizeof(*info->value));
-    info->value->name = g_strdup(name);
-    info->next = *list;
-    *list = info;
+    value->name = g_strdup(name);
+    QAPI_LIST_ADD(*list, value);
 }
 
 ChardevBackendInfoList *qmp_query_chardev_backends(Error **errp)
hw/core/machine.c +1 -5
index d740a7e96371..a972d2445a20 100644
@@ -510,11 +510,7 @@ static void machine_set_nvdimm_persistence(Object *obj, const char *value,
 
 void machine_class_allow_dynamic_sysbus_dev(MachineClass *mc, const char *type)
 {
-    strList *item = g_new0(strList, 1);
-
-    item->value = g_strdup(type);
-    item->next = mc->allowed_dynamic_sysbus_devices;
-    mc->allowed_dynamic_sysbus_devices = item;
+    QAPI_LIST_ADD(mc->allowed_dynamic_sysbus_devices, g_strdup(type));
 }
 
 static void validate_sysbus_device(SysBusDevice *sbdev, void *opaque)
hw/net/rocker/rocker.c +3 -5
index 1af1e6fa2f9b..a1137e11ff48 100644
@@ -127,13 +127,11 @@ RockerPortList *qmp_query_rocker_ports(const char *name, Error **errp)
     }
 
     for (i = r->fp_ports - 1; i >= 0; i--) {
-        RockerPortList *info = g_malloc0(sizeof(*info));
-        info->value = g_malloc0(sizeof(*info->value));
+        RockerPort *value = g_malloc0(sizeof(*value));
         struct fp_port *port = r->fp_port[i];
 
-        fp_port_get_info(port, info);
-        info->next = list;
-        list = info;
+        fp_port_get_info(port, value);
+        QAPI_LIST_ADD(list, value);
     }
 
     return list;
hw/net/rocker/rocker_fp.c +7 -7
index 4aa7da79b81d..a616e709292e 100644
@@ -51,14 +51,14 @@ bool fp_port_get_link_up(FpPort *port)
     return !qemu_get_queue(port->nic)->link_down;
 }
 
-void fp_port_get_info(FpPort *port, RockerPortList *info)
+void fp_port_get_info(FpPort *port, RockerPort *value)
 {
-    info->value->name = g_strdup(port->name);
-    info->value->enabled = port->enabled;
-    info->value->link_up = fp_port_get_link_up(port);
-    info->value->speed = port->speed;
-    info->value->duplex = port->duplex;
-    info->value->autoneg = port->autoneg;
+    value->name = g_strdup(port->name);
+    value->enabled = port->enabled;
+    value->link_up = fp_port_get_link_up(port);
+    value->speed = port->speed;
+    value->duplex = port->duplex;
+    value->autoneg = port->autoneg;
 }
 
 void fp_port_get_macaddr(FpPort *port, MACAddr *macaddr)
hw/net/virtio-net.c +7 -14
index 277289d56e76..6b13d3ca3c8f 100644
@@ -437,17 +437,14 @@ static void rxfilter_notify(NetClientState *nc)
 
 static intList *get_vlan_table(VirtIONet *n)
 {
-    intList *list, *entry;
+    intList *list;
     int i, j;
 
     list = NULL;
     for (i = 0; i < MAX_VLAN >> 5; i++) {
         for (j = 0; n->vlans[i] && j <= 0x1f; j++) {
             if (n->vlans[i] & (1U << j)) {
-                entry = g_malloc0(sizeof(*entry));
-                entry->value = (i << 5) + j;
-                entry->next = list;
-                list = entry;
+                QAPI_LIST_ADD(list, (i << 5) + j);
             }
         }
     }
@@ -460,7 +457,7 @@ static RxFilterInfo *virtio_net_query_rxfilter(NetClientState *nc)
     VirtIONet *n = qemu_get_nic_opaque(nc);
     VirtIODevice *vdev = VIRTIO_DEVICE(n);
     RxFilterInfo *info;
-    strList *str_list, *entry;
+    strList *str_list;
     int i;
 
     info = g_malloc0(sizeof(*info));
@@ -491,19 +488,15 @@ static RxFilterInfo *virtio_net_query_rxfilter(NetClientState *nc)
 
     str_list = NULL;
     for (i = 0; i < n->mac_table.first_multi; i++) {
-        entry = g_malloc0(sizeof(*entry));
-        entry->value = qemu_mac_strdup_printf(n->mac_table.macs + i * ETH_ALEN);
-        entry->next = str_list;
-        str_list = entry;
+        QAPI_LIST_ADD(str_list,
+                      qemu_mac_strdup_printf(n->mac_table.macs + i * ETH_ALEN));
     }
     info->unicast_table = str_list;
 
     str_list = NULL;
     for (i = n->mac_table.first_multi; i < n->mac_table.in_use; i++) {
-        entry = g_malloc0(sizeof(*entry));
-        entry->value = qemu_mac_strdup_printf(n->mac_table.macs + i * ETH_ALEN);
-        entry->next = str_list;
-        str_list = entry;
+        QAPI_LIST_ADD(str_list,
+                      qemu_mac_strdup_printf(n->mac_table.macs + i * ETH_ALEN));
     }
     info->multicast_table = str_list;
     info->vlan_table = get_vlan_table(n);
migration/migration.c +2 -5
index 0575ecb37953..7b849e795699 100644
@@ -378,12 +378,9 @@ int migration_incoming_enable_colo(void)
 void migrate_add_address(SocketAddress *address)
 {
     MigrationIncomingState *mis = migration_incoming_get_current();
-    SocketAddressList *addrs;
 
-    addrs = g_new0(SocketAddressList, 1);
-    addrs->next = mis->socket_address_list;
-    mis->socket_address_list = addrs;
-    addrs->value = QAPI_CLONE(SocketAddress, address);
+    QAPI_LIST_ADD(mis->socket_address_list,
+                  QAPI_CLONE(SocketAddress, address));
 }
 
 void qemu_start_incoming_migration(const char *uri, Error **errp)
migration/postcopy-ram.c +2 -5
index 0a2f88a87d06..18ac7e06c581 100644
@@ -145,14 +145,11 @@ static struct PostcopyBlocktimeContext *blocktime_context_new(void)
 static uint32List *get_vcpu_blocktime_list(PostcopyBlocktimeContext *ctx)
 {
     MachineState *ms = MACHINE(qdev_get_machine());
-    uint32List *list = NULL, *entry = NULL;
+    uint32List *list = NULL;
     int i;
 
     for (i = ms->smp.cpus - 1; i >= 0; i--) {
-        entry = g_new0(uint32List, 1);
-        entry->value = ctx->vcpu_blocktime[i];
-        entry->next = list;
-        list = entry;
+        QAPI_LIST_ADD(list, ctx->vcpu_blocktime[i]);
     }
 
     return list;
monitor/hmp-cmds.c +6 -5
index 9789f4277f50..629c3d1bf741 100644
@@ -1248,7 +1248,8 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
     const char *cap = qdict_get_str(qdict, "capability");
     bool state = qdict_get_bool(qdict, "state");
     Error *err = NULL;
-    MigrationCapabilityStatusList *caps = g_malloc0(sizeof(*caps));
+    MigrationCapabilityStatusList *caps = NULL;
+    MigrationCapabilityStatus *value = NULL;
     int val;
 
     val = qapi_enum_parse(&MigrationCapability_lookup, cap, -1, &err);
@@ -1256,10 +1257,10 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict)
         goto end;
     }
 
-    caps->value = g_malloc0(sizeof(*caps->value));
-    caps->value->capability = val;
-    caps->value->state = state;
-    caps->next = NULL;
+    value = g_malloc0(sizeof(*value));
+    value->capability = val;
+    value->state = state;
+    QAPI_LIST_ADD(caps, value);
     qmp_migrate_set_capabilities(caps, &err);
 
 end:
qemu-img.c +2 -3
index 2103507936ea..4cfa8bccc5e7 100644
@@ -1643,14 +1643,13 @@ static void do_dirty_bitmap_merge(const char *dst_node, const char *dst_name,
                                   Error **errp)
 {
     BlockDirtyBitmapMergeSource *merge_src;
-    BlockDirtyBitmapMergeSourceList *list;
+    BlockDirtyBitmapMergeSourceList *list = NULL;
 
     merge_src = g_new0(BlockDirtyBitmapMergeSource, 1);
     merge_src->type = QTYPE_QDICT;
     merge_src->u.external.node = g_strdup(src_node);
     merge_src->u.external.name = g_strdup(src_name);
-    list = g_new0(BlockDirtyBitmapMergeSourceList, 1);
-    list->value = merge_src;
+    QAPI_LIST_ADD(list, merge_src);
     qmp_block_dirty_bitmap_merge(dst_node, dst_name, list, errp);
     qapi_free_BlockDirtyBitmapMergeSourceList(list);
 }
qga/commands-posix.c +3 -10
index 3bffee99d4c9..06540425ded2 100644
@@ -1211,7 +1211,7 @@ GuestFilesystemInfoList *qmp_guest_get_fsinfo(Error **errp)
 {
     FsMountList mounts;
     struct FsMount *mount;
-    GuestFilesystemInfoList *new, *ret = NULL;
+    GuestFilesystemInfoList *ret = NULL;
     Error *local_err = NULL;
 
     QTAILQ_INIT(&mounts);
@@ -1224,10 +1224,7 @@ GuestFilesystemInfoList *qmp_guest_get_fsinfo(Error **errp)
     QTAILQ_FOREACH(mount, &mounts, next) {
         g_debug("Building guest fsinfo for '%s'", mount->dirname);
 
-        new = g_malloc0(sizeof(*ret));
-        new->value = build_guest_fsinfo(mount, &local_err);
-        new->next = ret;
-        ret = new;
+        QAPI_LIST_ADD(ret, build_guest_fsinfo(mount, &local_err));
         if (local_err) {
             error_propagate(errp, local_err);
             qapi_free_GuestFilesystemInfoList(ret);
@@ -1493,7 +1490,6 @@ GuestFilesystemTrimResponse *
 qmp_guest_fstrim(bool has_minimum, int64_t minimum, Error **errp)
 {
     GuestFilesystemTrimResponse *response;
-    GuestFilesystemTrimResultList *list;
     GuestFilesystemTrimResult *result;
     int ret = 0;
     FsMountList mounts;
@@ -1517,10 +1513,7 @@ qmp_guest_fstrim(bool has_minimum, int64_t minimum, Error **errp)
         result = g_malloc0(sizeof(*result));
         result->path = g_strdup(mount->dirname);
 
-        list = g_malloc0(sizeof(*list));
-        list->value = result;
-        list->next = response->paths;
-        response->paths = list;
+        QAPI_LIST_ADD(response->paths, result);
 
         fd = qemu_open_old(mount->dirname, O_RDONLY);
         if (fd == -1) {
qga/commands-win32.c +4 -13
index 0c3c05484f5f..cc5736c3bba8 100644
@@ -926,10 +926,8 @@ static GuestDiskAddressList *build_guest_disk_info(char *guid, Error **errp)
                 error_free(local_err);
                 goto out;
             }
-            list = g_malloc0(sizeof(*list));
-            list->value = disk;
+            QAPI_LIST_ADD(list, disk);
             disk = NULL;
-            list->next = NULL;
             goto out;
         } else {
             error_setg_win32(errp, GetLastError(),
@@ -1064,7 +1062,7 @@ free:
 GuestFilesystemInfoList *qmp_guest_get_fsinfo(Error **errp)
 {
     HANDLE vol_h;
-    GuestFilesystemInfoList *new, *ret = NULL;
+    GuestFilesystemInfoList *ret = NULL;
     char guid[256];
 
     vol_h = FindFirstVolume(guid, sizeof(guid));
@@ -1082,10 +1080,7 @@ GuestFilesystemInfoList *qmp_guest_get_fsinfo(Error **errp)
             error_free(local_err);
             continue;
         }
-        new = g_malloc(sizeof(*ret));
-        new->value = info;
-        new->next = ret;
-        ret = new;
+        QAPI_LIST_ADD(ret, info);
     } while (FindNextVolume(vol_h, guid, sizeof(guid)));
 
     if (GetLastError() != ERROR_NO_MORE_FILES) {
@@ -1268,11 +1263,7 @@ qmp_guest_fstrim(bool has_minimum, int64_t minimum, Error **errp)
 
         res->path = path;
 
-        list = g_new0(GuestFilesystemTrimResultList, 1);
-        list->value = res;
-        list->next = resp->paths;
-
-        resp->paths = list;
+        QAPI_LIST_ADD(resp->paths, res);
 
         memset(argv, 0, sizeof(argv));
         argv[0] = (gchar *)"defrag.exe";
qga/commands.c +1 -5
index 3dcd5fbe5c4d..27118df6caea 100644
@@ -66,17 +66,13 @@ static void qmp_command_info(const QmpCommand *cmd, void *opaque)
 {
     GuestAgentInfo *info = opaque;
     GuestAgentCommandInfo *cmd_info;
-    GuestAgentCommandInfoList *cmd_info_list;
 
     cmd_info = g_new0(GuestAgentCommandInfo, 1);
     cmd_info->name = g_strdup(qmp_command_name(cmd));
     cmd_info->enabled = qmp_command_is_enabled(cmd);
     cmd_info->success_response = qmp_has_success_response(cmd);
 
-    cmd_info_list = g_new0(GuestAgentCommandInfoList, 1);
-    cmd_info_list->value = cmd_info;
-    cmd_info_list->next = info->supported_commands;
-    info->supported_commands = cmd_info_list;
+    QAPI_LIST_ADD(info->supported_commands, cmd_info);
 }
 
 struct GuestAgentInfo *qmp_guest_info(Error **errp)
qom/qom-qmp-cmds.c +8 -21
index 310ab2d0481d..5ac9272ffeea 100644
@@ -46,14 +46,12 @@ ObjectPropertyInfoList *qmp_qom_list(const char *path, Error **errp)
 
     object_property_iter_init(&iter, obj);
     while ((prop = object_property_iter_next(&iter))) {
-        ObjectPropertyInfoList *entry = g_malloc0(sizeof(*entry));
+        ObjectPropertyInfo *value = g_malloc0(sizeof(ObjectPropertyInfo));
 
-        entry->value = g_malloc0(sizeof(ObjectPropertyInfo));
-        entry->next = props;
-        props = entry;
+        QAPI_LIST_ADD(props, value);
 
-        entry->value->name = g_strdup(prop->name);
-        entry->value->type = g_strdup(prop->type);
+        value->name = g_strdup(prop->name);
+        value->type = g_strdup(prop->type);
     }
 
     return props;
@@ -90,7 +88,7 @@ QObject *qmp_qom_get(const char *path, const char *property, Error **errp)
 
 static void qom_list_types_tramp(ObjectClass *klass, void *data)
 {
-    ObjectTypeInfoList *e, **pret = data;
+    ObjectTypeInfoList **pret = data;
     ObjectTypeInfo *info;
     ObjectClass *parent = object_class_get_parent(klass);
 
@@ -102,10 +100,7 @@ static void qom_list_types_tramp(ObjectClass *klass, void *data)
         info->parent = g_strdup(object_class_get_name(parent));
     }
 
-    e = g_malloc0(sizeof(*e));
-    e->value = info;
-    e->next = *pret;
-    *pret = e;
+    QAPI_LIST_ADD(*pret, info);
 }
 
 ObjectTypeInfoList *qmp_qom_list_types(bool has_implements,
@@ -155,7 +150,6 @@ ObjectPropertyInfoList *qmp_device_list_properties(const char *typename,
     object_property_iter_init(&iter, obj);
     while ((prop = object_property_iter_next(&iter))) {
         ObjectPropertyInfo *info;
-        ObjectPropertyInfoList *entry;
 
         /* Skip Object and DeviceState properties */
         if (strcmp(prop->name, "type") == 0 ||
@@ -181,10 +175,7 @@ ObjectPropertyInfoList *qmp_device_list_properties(const char *typename,
         info->default_value = qobject_ref(prop->defval);
         info->has_default_value = !!info->default_value;
 
-        entry = g_malloc0(sizeof(*entry));
-        entry->value = info;
-        entry->next = prop_list;
-        prop_list = entry;
+        QAPI_LIST_ADD(prop_list, info);
     }
 
     object_unref(obj);
@@ -222,7 +213,6 @@ ObjectPropertyInfoList *qmp_qom_list_properties(const char *typename,
     }
     while ((prop = object_property_iter_next(&iter))) {
         ObjectPropertyInfo *info;
-        ObjectPropertyInfoList *entry;
 
         info = g_malloc0(sizeof(*info));
         info->name = g_strdup(prop->name);
@@ -230,10 +220,7 @@ ObjectPropertyInfoList *qmp_qom_list_properties(const char *typename,
         info->has_description = !!prop->description;
         info->description = g_strdup(prop->description);
 
-        entry = g_malloc0(sizeof(*entry));
-        entry->value = info;
-        entry->next = prop_list;
-        prop_list = entry;
+        QAPI_LIST_ADD(prop_list, info);
     }
 
     object_unref(obj);
target/arm/helper.c +1 -5
index 97bb6b8c01b4..df150f3c3eeb 100644
@@ -8293,7 +8293,6 @@ static void arm_cpu_add_definition(gpointer data, gpointer user_data)
 {
     ObjectClass *oc = data;
     CpuDefinitionInfoList **cpu_list = user_data;
-    CpuDefinitionInfoList *entry;
     CpuDefinitionInfo *info;
     const char *typename;
 
@@ -8303,10 +8302,7 @@ static void arm_cpu_add_definition(gpointer data, gpointer user_data)
                            strlen(typename) - strlen("-" TYPE_ARM_CPU));
     info->q_typename = g_strdup(typename);
 
-    entry = g_malloc0(sizeof(*entry));
-    entry->value = info;
-    entry->next = *cpu_list;
-    *cpu_list = entry;
+    QAPI_LIST_ADD(*cpu_list, info);
 }
 
 CpuDefinitionInfoList *qmp_query_cpu_definitions(Error **errp)
target/arm/monitor.c +2 -11
index 169d8a64b651..771101656535 100644
@@ -42,15 +42,6 @@ static GICCapability *gic_cap_new(int version)
     return cap;
 }
 
-static GICCapabilityList *gic_cap_list_add(GICCapabilityList *head,
-                                           GICCapability *cap)
-{
-    GICCapabilityList *item = g_new0(GICCapabilityList, 1);
-    item->value = cap;
-    item->next = head;
-    return item;
-}
-
 static inline void gic_cap_kvm_probe(GICCapability *v2, GICCapability *v3)
 {
 #ifdef CONFIG_KVM
@@ -84,8 +75,8 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
 
     gic_cap_kvm_probe(v2, v3);
 
-    head = gic_cap_list_add(head, v2);
-    head = gic_cap_list_add(head, v3);
+    QAPI_LIST_ADD(head, v2);
+    QAPI_LIST_ADD(head, v3);
 
     return head;
 }
target/i386/cpu.c +1 -5
index 0d8606958e9e..9ae6661f97e3 100644
@@ -4984,7 +4984,6 @@ static void x86_cpu_definition_entry(gpointer data, gpointer user_data)
     ObjectClass *oc = data;
     X86CPUClass *cc = X86_CPU_CLASS(oc);
     CpuDefinitionInfoList **cpu_list = user_data;
-    CpuDefinitionInfoList *entry;
     CpuDefinitionInfo *info;
 
     info = g_malloc0(sizeof(*info));
@@ -5009,10 +5008,7 @@ static void x86_cpu_definition_entry(gpointer data, gpointer user_data)
         info->has_alias_of = !!info->alias_of;
     }
 
-    entry = g_malloc0(sizeof(*entry));
-    entry->value = info;
-    entry->next = *cpu_list;
-    *cpu_list = entry;
+    QAPI_LIST_ADD(*cpu_list, info);
 }
 
 CpuDefinitionInfoList *qmp_query_cpu_definitions(Error **errp)
target/mips/helper.c +1 -5
index afd78b1990be..036bacc24b22 100644
@@ -1502,7 +1502,6 @@ static void mips_cpu_add_definition(gpointer data, gpointer user_data)
 {
     ObjectClass *oc = data;
     CpuDefinitionInfoList **cpu_list = user_data;
-    CpuDefinitionInfoList *entry;
     CpuDefinitionInfo *info;
     const char *typename;
 
@@ -1512,10 +1511,7 @@ static void mips_cpu_add_definition(gpointer data, gpointer user_data)
                            strlen(typename) - strlen("-" TYPE_MIPS_CPU));
     info->q_typename = g_strdup(typename);
 
-    entry = g_malloc0(sizeof(*entry));
-    entry->value = info;
-    entry->next = *cpu_list;
-    *cpu_list = entry;
+    QAPI_LIST_ADD(*cpu_list, info);
 }
 
 CpuDefinitionInfoList *qmp_query_cpu_definitions(Error **errp)
target/s390x/cpu_models.c +2 -10
index ca484bfda7be..89218bde049f 100644
@@ -423,7 +423,6 @@ static void create_cpu_model_list(ObjectClass *klass, void *opaque)
 {
     struct CpuDefinitionInfoListData *cpu_list_data = opaque;
     CpuDefinitionInfoList **cpu_list = &cpu_list_data->list;
-    CpuDefinitionInfoList *entry;
     CpuDefinitionInfo *info;
     char *name = g_strdup(object_class_get_name(klass));
     S390CPUClass *scc = S390_CPU_CLASS(klass);
@@ -450,10 +449,7 @@ static void create_cpu_model_list(ObjectClass *klass, void *opaque)
         object_unref(obj);
     }
 
-    entry = g_new0(CpuDefinitionInfoList, 1);
-    entry->value = info;
-    entry->next = *cpu_list;
-    *cpu_list = entry;
+    QAPI_LIST_ADD(*cpu_list, info);
 }
 
 CpuDefinitionInfoList *qmp_query_cpu_definitions(Error **errp)
@@ -620,12 +616,8 @@ CpuModelExpansionInfo *qmp_query_cpu_model_expansion(CpuModelExpansionType type,
 static void list_add_feat(const char *name, void *opaque)
 {
     strList **last = (strList **) opaque;
-    strList *entry;
 
-    entry = g_new0(strList, 1);
-    entry->value = g_strdup(name);
-    entry->next = *last;
-    *last = entry;
+    QAPI_LIST_ADD(*last, g_strdup(name));
 }
 
 CpuModelCompareInfo *qmp_query_cpu_model_comparison(CpuModelInfo *infoa,
tests/test-clone-visitor.c +2 -5
index 5e1e8b2f5e8a..03788d355130 100644
@@ -65,16 +65,13 @@ static void test_clone_alternate(void)
 
 static void test_clone_list_union(void)
 {
-    uint8List *src, *dst;
+    uint8List *src = NULL, *dst;
     uint8List *tmp = NULL;
     int i;
 
     /* Build list in reverse */
     for (i = 10; i; i--) {
-        src = g_new0(uint8List, 1);
-        src->next = tmp;
-        src->value = i;
-        tmp = src;
+        QAPI_LIST_ADD(src, i);
     }
 
     dst = QAPI_CLONE(uint8List, src);
tests/test-qobject-output-visitor.c +20 -22

67 lines not on this page · raw message

tests/test-visitor-serialization.c +1 -4
index 1c5a8b94ea87..efbf744fcf25 100644
@@ -704,10 +704,7 @@ static void test_nested_struct_list(gconstpointer opaque)
     int i = 0;
 
     for (i = 0; i < 8; i++) {
-        tmp = g_new0(UserDefTwoList, 1);
-        tmp->value = nested_struct_create();
-        tmp->next = listp;
-        listp = tmp;
+        QAPI_LIST_ADD(listp, nested_struct_create());
     }
 
     ops->serialize(listp, &serialize_data, visit_nested_struct_list,
trace/qmp.c +10 -12
index 38246e1aa692..8755835edabc 100644
@@ -92,39 +92,37 @@ TraceEventInfoList *qmp_trace_event_get_state(const char *name,
     /* Get states (all errors checked above) */
     trace_event_iter_init(&iter, name);
     while ((ev = trace_event_iter_next(&iter)) != NULL) {
-        TraceEventInfoList *elem;
+        TraceEventInfo *value;
         bool is_vcpu = trace_event_is_vcpu(ev);
         if (has_vcpu && !is_vcpu) {
             continue;
         }
 
-        elem = g_new(TraceEventInfoList, 1);
-        elem->value = g_new(TraceEventInfo, 1);
-        elem->value->vcpu = is_vcpu;
-        elem->value->name = g_strdup(trace_event_get_name(ev));
+        value = g_new(TraceEventInfo, 1);
+        value->vcpu = is_vcpu;
+        value->name = g_strdup(trace_event_get_name(ev));
 
         if (!trace_event_get_state_static(ev)) {
-            elem->value->state = TRACE_EVENT_STATE_UNAVAILABLE;
+            value->state = TRACE_EVENT_STATE_UNAVAILABLE;
         } else {
             if (has_vcpu) {
                 if (is_vcpu) {
                     if (trace_event_get_vcpu_state_dynamic(cpu, ev)) {
-                        elem->value->state = TRACE_EVENT_STATE_ENABLED;
+                        value->state = TRACE_EVENT_STATE_ENABLED;
                     } else {
-                        elem->value->state = TRACE_EVENT_STATE_DISABLED;
+                        value->state = TRACE_EVENT_STATE_DISABLED;
                     }
                 }
                 /* else: already skipped above */
             } else {
                 if (trace_event_get_state_dynamic(ev)) {
-                    elem->value->state = TRACE_EVENT_STATE_ENABLED;
+                    value->state = TRACE_EVENT_STATE_ENABLED;
                 } else {
-                    elem->value->state = TRACE_EVENT_STATE_DISABLED;
+                    value->state = TRACE_EVENT_STATE_DISABLED;
                 }
             }
         }
-        elem->next = events;
-        events = elem;
+        QAPI_LIST_ADD(events, value);
     }
 
     return events;
ui/vnc.c +6 -15
index f006aa1afdb2..f39cfc952906 100644
@@ -365,14 +365,11 @@ static VncDisplay *vnc_display_find(const char *id)
 
 static VncClientInfoList *qmp_query_client_list(VncDisplay *vd)
 {
-    VncClientInfoList *cinfo, *prev = NULL;
+    VncClientInfoList *prev = NULL;
     VncState *client;
 
     QTAILQ_FOREACH(client, &vd->clients, next) {
-        cinfo = g_new0(VncClientInfoList, 1);
-        cinfo->value = qmp_query_vnc_client(client);
-        cinfo->next = prev;
-        prev = cinfo;
+        QAPI_LIST_ADD(prev, qmp_query_vnc_client(client));
     }
     return prev;
 }
@@ -453,7 +450,6 @@ static VncServerInfo2List *qmp_query_server_entry(QIOChannelSocket *ioc,
                                                   int subauth,
                                                   VncServerInfo2List *prev)
 {
-    VncServerInfo2List *list;
     VncServerInfo2 *info;
     Error *err = NULL;
     SocketAddress *addr;
@@ -476,10 +472,8 @@ static VncServerInfo2List *qmp_query_server_entry(QIOChannelSocket *ioc,
     qmp_query_auth(auth, subauth, &info->auth,
                    &info->vencrypt, &info->has_vencrypt);
 
-    list = g_new0(VncServerInfo2List, 1);
-    list->value = info;
-    list->next = prev;
-    return list;
+    QAPI_LIST_ADD(prev, info);
+    return prev;
 }
 
 static void qmp_query_auth(int auth, int subauth,
@@ -554,7 +548,7 @@ static void qmp_query_auth(int auth, int subauth,
 
 VncInfo2List *qmp_query_vnc_servers(Error **errp)
 {
-    VncInfo2List *item, *prev = NULL;
+    VncInfo2List *prev = NULL;
     VncInfo2 *info;
     VncDisplay *vd;
     DeviceState *dev;
@@ -583,10 +577,7 @@ VncInfo2List *qmp_query_vnc_servers(Error **errp)
                 vd->ws_subauth, info->server);
         }
 
-        item = g_new0(VncInfo2List, 1);
-        item->value = info;
-        item->next = prev;
-        prev = item;
+        QAPI_LIST_ADD(prev, info);
     }
     return prev;
 }
util/qemu-config.c +4 -10
index 660f47b0050f..495ada45f3df 100644
@@ -55,7 +55,7 @@ QemuOpts *qemu_find_opts_singleton(const char *group)
 
 static CommandLineParameterInfoList *query_option_descs(const QemuOptDesc *desc)
 {
-    CommandLineParameterInfoList *param_list = NULL, *entry;
+    CommandLineParameterInfoList *param_list = NULL;
     CommandLineParameterInfo *info;
     int i;
 
@@ -87,10 +87,7 @@ static CommandLineParameterInfoList *query_option_descs(const QemuOptDesc *desc)
             info->q_default = g_strdup(desc[i].def_value_str);
         }
 
-        entry = g_malloc0(sizeof(*entry));
-        entry->value = info;
-        entry->next = param_list;
-        param_list = entry;
+        QAPI_LIST_ADD(param_list, info);
     }
 
     return param_list;
@@ -246,7 +243,7 @@ CommandLineOptionInfoList *qmp_query_command_line_options(bool has_option,
                                                           const char *option,
                                                           Error **errp)
 {
-    CommandLineOptionInfoList *conf_list = NULL, *entry;
+    CommandLineOptionInfoList *conf_list = NULL;
     CommandLineOptionInfo *info;
     int i;
 
@@ -262,10 +259,7 @@ CommandLineOptionInfoList *qmp_query_command_line_options(bool has_option,
                 info->parameters =
                     query_option_descs(vm_config_groups[i]->desc);
             }
-            entry = g_malloc0(sizeof(*entry));
-            entry->value = info;
-            entry->next = conf_list;
-            conf_list = entry;
+            QAPI_LIST_ADD(conf_list, info);
         }
     }
 
target/ppc/translate_init.c.inc +2 -10
index bb66526280ef..5795d0e5af2c 100644
@@ -10621,7 +10621,6 @@ static void ppc_cpu_defs_entry(gpointer data, gpointer user_data)
     ObjectClass *oc = data;
     CpuDefinitionInfoList **first = user_data;
     const char *typename;
-    CpuDefinitionInfoList *entry;
     CpuDefinitionInfo *info;
 
     typename = object_class_get_name(oc);
@@ -10629,10 +10628,7 @@ static void ppc_cpu_defs_entry(gpointer data, gpointer user_data)
     info->name = g_strndup(typename,
                            strlen(typename) - strlen(POWERPC_CPU_TYPE_SUFFIX));
 
-    entry = g_malloc0(sizeof(*entry));
-    entry->value = info;
-    entry->next = *first;
-    *first = entry;
+    QAPI_LIST_ADD(*first, info);
 }
 
 CpuDefinitionInfoList *qmp_query_cpu_definitions(Error **errp)
@@ -10648,7 +10644,6 @@ CpuDefinitionInfoList *qmp_query_cpu_definitions(Error **errp)
     for (i = 0; ppc_cpu_aliases[i].alias != NULL; i++) {
         PowerPCCPUAlias *alias = &ppc_cpu_aliases[i];
         ObjectClass *oc;
-        CpuDefinitionInfoList *entry;
         CpuDefinitionInfo *info;
 
         oc = ppc_cpu_class_by_name(alias->model);
@@ -10660,10 +10655,7 @@ CpuDefinitionInfoList *qmp_query_cpu_definitions(Error **errp)
         info->name = g_strdup(alias->alias);
         info->q_typename = g_strdup(object_class_get_name(oc));
 
-        entry = g_malloc0(sizeof(*entry));
-        entry->value = info;
-        entry->next = cpu_list;
-        cpu_list = entry;
+        QAPI_LIST_ADD(cpu_list, info);
     }
 
     return cpu_list;
Signature · 1 line
2.29.0
author
#28 lore raw

Re: [PATCH v5 12/12] qapi: Use QAPI_LIST_ADD() where possible

Quoted 15 lines · Eric Blake

Anywhere we create a list of just one item or by prepending items (typically because order doesn't matter), we can use the now-public macro. But places where we must keep the list in order by appending remain open-coded.

---
 docs/devel/writing-qmp-commands.txt | 13 +++------

+++ b/docs/devel/writing-qmp-commands.txt

docs/devel/writing-qmp-commands.txt +0 -1
@@ -531,15 +531,10 @@ TimerAlarmMethodList *qmp_query_alarm_methods(Error **errp)
     bool current = true;
 
     for (p = alarm_timers; p->name; p++) {
-        TimerAlarmMethodList *info = g_malloc0(sizeof(*info));

docs/devel/writing-qmp-commands.txt TimerAlarmMethodList *qmp_query_alarm_methods(Error **errp) · line 535

Quoted 10 lines
docs/devel/writing-qmp-commands.txt +2 -8
-        info->value = g_malloc0(sizeof(*info->value));
-        info->value->method_name = g_strdup(p->name);
-        info->value->current = current;
-
-        current = false;
-
-        info->next = method_list;
-        method_list = info;
+	TimerAlarmMethod *value = g_new0(TimerAlarmMethod, 1);
+        value->method_name = g_strdup(p->name);

Oops, tab damage.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#29 lore raw
In reply to Eric Blake
Quoted 12 lines · Eric Blake

v4 was here:
https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg02708.html

Since then:
- rebase to master
- patches 1, 2, and 12 are new based on Vladimir's observation of QAPI_LIST_ADD
- patches 10-11 are new based on prior discussion on exposing actual depth in addition to a tri-state encoding
- patch 3 has a nasty bug fixed that was causing iotest failures
- patch 6 updated to take advantage of patch 2
- other minor tweaks based on review comments

Quoted 12 lines

001/12:[down] 'qapi: Move GenericList to qapi/util.h' 002/12:[down] 'qapi: Make QAPI_LIST_ADD() public'
003/12:[0002] [FC] 'nbd: Utilize QAPI_CLONE for type conversion' 004/12:[0010] [FC] 'nbd: Add new qemu:allocation-depth metadata context' 005/12:[----] [--] 'nbd: Add 'qemu-nbd -A' to expose allocation depth' 006/12:[0014] [FC] 'nbd: Update qapi to support exporting multiple bitmaps' 007/12:[----] [--] 'nbd: Simplify qemu bitmap context name' 008/12:[----] [--] 'nbd: Refactor counting of metadata contexts' 009/12:[0017] [FC] 'nbd: Allow export of multiple bitmaps for one device' 010/12:[down] 'block: Return depth level during bdrv_is_allocated_above' 011/12:[down] 'nbd: Expose actual depth in qemu:allocation-depth' 012/12:[down] 'qapi: Use QAPI_LIST_ADD() where possible'

and I meant to add:

Also available at:
https://repo.or.cz/qemu/ericb.git/shortlog/refs/tags/nbd-alloc-depth-v5

patch 12 is the largest; it may be worth splitting that by maintainer, or even pushing it off post-5.2. Logically, it can go in anywhere after patch 2, but by putting it last, I'm hoping to send a pull request for soft freeze next week for patches 1-11 for sure, and only include 12 if we get enough positive review in time. I did not try to see if Coccinelle could make the work done in patch 12 more automatable.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#30 lore raw
Quoted 26 lines · Eric Blake
Quoted 12 lines · Eric Blake

v4 was here:
https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg02708.html

Since then:
- rebase to master
- patches 1, 2, and 12 are new based on Vladimir's observation of QAPI_LIST_ADD
- patches 10-11 are new based on prior discussion on exposing actual depth in addition to a tri-state encoding
- patch 3 has a nasty bug fixed that was causing iotest failures
- patch 6 updated to take advantage of patch 2
- other minor tweaks based on review comments

Quoted 12 lines

001/12:[down] 'qapi: Move GenericList to qapi/util.h' 002/12:[down] 'qapi: Make QAPI_LIST_ADD() public'
003/12:[0002] [FC] 'nbd: Utilize QAPI_CLONE for type conversion' 004/12:[0010] [FC] 'nbd: Add new qemu:allocation-depth metadata context' 005/12:[----] [--] 'nbd: Add 'qemu-nbd -A' to expose allocation depth' 006/12:[0014] [FC] 'nbd: Update qapi to support exporting multiple bitmaps' 007/12:[----] [--] 'nbd: Simplify qemu bitmap context name' 008/12:[----] [--] 'nbd: Refactor counting of metadata contexts' 009/12:[0017] [FC] 'nbd: Allow export of multiple bitmaps for one device' 010/12:[down] 'block: Return depth level during bdrv_is_allocated_above' 011/12:[down] 'nbd: Expose actual depth in qemu:allocation-depth' 012/12:[down] 'qapi: Use QAPI_LIST_ADD() where possible'

Quoted 6 lines

and I meant to add:

Also available at:
https://repo.or.cz/qemu/ericb.git/shortlog/refs/tags/nbd-alloc-depth-v5

patch 12 is the largest; it may be worth splitting that by maintainer,

I wouldn't split. The various parts will trickle in separately, and you may well end up holding leftovers.

Quoted 5 lines

or even pushing it off post-5.2. Logically, it can go in anywhere after patch 2, but by putting it last, I'm hoping to send a pull request for soft freeze next week for patches 1-11 for sure, and only include 12 if we get enough positive review in time. I did not try to see if Coccinelle could make the work done in patch 12 more automatable.

PATCH 12 serves a useful purpose even if we can't get it in right away:
it convinces me that making QAPI_LIST_ADD() public makes sense.