Skip to contentSkip to conversation

qemu-img: Add convert --bitmaps

PATCH Superseded Eric Blake 2 participants 30 messages May 13, 2020 – May 18, 2020

Subsystems (2)

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

v4 cover qemu-img: Add convert --bitmaps

v3 was here:
https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg02139.html
original cover letter here:
https://lists.gnu.org/archive/html/qemu-devel/2020-04/msg03464.html

Since then:
- add R-b where appropriate
- patch 3: rename new driver hook [Max]
- patch 6: typo and grammar fix [Max]
- patch 7: add 'measure --bitmaps' convenience option [Nir]
- patch 7: more documentation, enhance test to cover more cases
- patch 8: rebase to earlier changes, ensure that both 'measure --bitmaps' and 'convert --bitmaps' pass or fail on the same criteria

001/9:[----] [--] 'docs: Sort sections on qemu-img subcommand parameters' 002/9:[----] [--] 'qemu-img: Fix stale comments on doc location' 003/9:[0017] [FC] 'block: Make it easier to learn which BDS support bitmaps' 004/9:[----] [--] 'blockdev: Promote several bitmap functions to non-static' 005/9:[----] [--] 'blockdev: Split off basic bitmap operations for qemu-img' 006/9:[0009] [FC] 'qemu-img: Add bitmap sub-command' 007/9:[0084] [FC] 'qcow2: Expose bitmaps' size during measure' 008/9:[0008] [FC] 'qemu-img: Add convert --bitmaps option' 009/9:[----] [--] 'iotests: Add test 291 to for qemu-img bitmap coverage'

Series can also be downloaded at:
https://repo.or.cz/qemu/ericb.git/shortlog/refs/tags/qemu-img-bitmaps-v4

Eric Blake (9):
docs: Sort sections on qemu-img subcommand parameters
qemu-img: Fix stale comments on doc location
block: Make it easier to learn which BDS support bitmaps
blockdev: Promote several bitmap functions to non-static
blockdev: Split off basic bitmap operations for qemu-img
qemu-img: Add bitmap sub-command
qcow2: Expose bitmaps' size during measure
qemu-img: Add convert --bitmaps option
iotests: Add test 291 to for qemu-img bitmap coverage

 docs/tools/qemu-img.rst          |  88 +++++---
 Makefile.objs                    |   3 +-
 qapi/block-core.json             |  15 +-
 block/qcow2.h                    |   1 +
 include/block/block_int.h        |  13 ++
 include/block/dirty-bitmap.h     |   1 +
 block/crypto.c                   |   2 +-
 block/dirty-bitmap.c             |   9 +
 block/monitor/bitmap-qmp-cmds.c  | 323 +++++++++++++++++++++++++++
 block/qcow2-bitmap.c             |   7 +
 block/qcow2.c                    |  39 +++-
 block/raw-format.c               |   2 +-
 blockdev.c                       | 303 +------------------------
 qemu-img.c                       | 366 ++++++++++++++++++++++++++++++-
 MAINTAINERS                      |   1 +
 block/monitor/Makefile.objs      |   1 +
 qemu-img-cmds.hx                 |  17 +-
 tests/qemu-iotests/178.out.qcow2 |  16 ++
 tests/qemu-iotests/190           |  57 ++++-
 tests/qemu-iotests/190.out       |  35 ++-
 tests/qemu-iotests/291           | 112 ++++++++++
 tests/qemu-iotests/291.out       |  78 +++++++
 tests/qemu-iotests/group         |   1 +
 23 files changed, 1142 insertions(+), 348 deletions(-)
 create mode 100644 block/monitor/bitmap-qmp-cmds.c
 create mode 100755 tests/qemu-iotests/291
 create mode 100644 tests/qemu-iotests/291.out
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH v4 1/9 docs: Sort sections on qemu-img subcommand parameters

We already list the subcommand summaries alphabetically, we should do the same for the documentation related to subcommand-specific parameters.

---
 docs/tools/qemu-img.rst | 48 ++++++++++++++++++++---------------------
 1 file changed, 24 insertions(+), 24 deletions(-)
docs/tools/qemu-img.rst +24 -24
index 0080f83a76c9..7d08c48d308f 100644
@@ -142,30 +142,6 @@ by the used format or see the format descriptions below for details.
   the documentation of the emulator's ``-drive cache=...`` option for allowed
   values.
 
-Parameters to snapshot subcommand:
-
-.. program:: qemu-img-snapshot
-
-.. option:: snapshot
-
-  Is the name of the snapshot to create, apply or delete
-
-.. option:: -a
-
-  Applies a snapshot (revert disk to saved state)
-
-.. option:: -c
-
-  Creates a snapshot
-
-.. option:: -d
-
-  Deletes a snapshot
-
-.. option:: -l
-
-  Lists all snapshots in the given image
-
 Parameters to compare subcommand:
 
 .. program:: qemu-img-compare
@@ -245,6 +221,30 @@ Parameters to dd subcommand:
 
   Sets the number of input blocks to skip
 
+Parameters to snapshot subcommand:
+
+.. program:: qemu-img-snapshot
+
+.. option:: snapshot
+
+  Is the name of the snapshot to create, apply or delete
+
+.. option:: -a
+
+  Applies a snapshot (revert disk to saved state)
+
+.. option:: -c
+
+  Creates a snapshot
+
+.. option:: -d
+
+  Deletes a snapshot
+
+.. option:: -l
+
+  Lists all snapshots in the given image
+
 Command description:
 
 .. program:: qemu-img-commands
Signature · 1 line
2.26.2
Reviewed-by on v4 1/9
#2 lore raw

Re: [PATCH v4 1/9] docs: Sort sections on qemu-img subcommand parameters

Quoted 6 lines · Eric Blake

We already list the subcommand summaries alphabetically, we should do the same for the documentation related to subcommand-specific parameters.

Signature · 2 lines
Best regards,
Vladimir
author
#3 lore raw

PATCH v4 2/9 qemu-img: Fix stale comments on doc location

In reply to Eric Blake

Missed in commit e13c59fa.

---
 qemu-img.c       | 2 +-
 qemu-img-cmds.hx | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
qemu-img.c +1 -1
index 6a4327aaba56..b6e8af9202a5 100644
@@ -108,7 +108,7 @@ static void QEMU_NORETURN unrecognized_option(const char *option)
     error_exit("unrecognized option '%s'", option);
 }
 
-/* Please keep in synch with qemu-img.texi */
+/* Please keep in synch with docs/tools/qemu-img.rst */
 static void QEMU_NORETURN help(void)
 {
     const char *help_msg =
qemu-img-cmds.hx +1 -1
index c9c54de1df40..e0886437b1f2 100644
@@ -7,7 +7,7 @@ HXCOMM command structures and help message.
 HXCOMM HXCOMM can be used for comments, discarded from both rST and C
 
 HXCOMM When amending the rST sections, please remember to copy the usage
-HXCOMM over to the per-command sections in qemu-img.texi.
+HXCOMM over to the per-command sections in docs/tools/qemu-img.rst.
 
 DEF("amend", img_amend,
     "amend [--object objectdef] [--image-opts] [-p] [-q] [-f fmt] [-t cache] -o options filename")
Signature · 1 line
2.26.2
author
#5 lore raw

PATCH v4 3/9 block: Make it easier to learn which BDS support bitmaps

In reply to Eric Blake

Upcoming patches will enhance bitmap support in qemu-img, but in doing so, it turns out to be nice to suppress output when persistent bitmaps make no sense (such as on a qcow2 v2 image). Add a hook to make this easier to query.

This patch adds a new callback .bdrv_supports_persistent_dirty_bitmap, rather than trying to shoehorn the answer in via existing callbacks. In particular, while it might have been possible to overload .bdrv_co_can_store_new_dirty_bitmap to special-case a NULL input to answer whether any persistent bitmaps are supported, that is at odds with whether a particular bitmap can be stored (for example, even on an image that supports persistent bitmaps but has currently filled up the maximum number of bitmaps, attempts to store another one should fail); and the new functionality doesn't require coroutine safety. Similarly, we could have added one more piece of information to .bdrv_get_info, but then again, most callers to that function tend to already discard extraneous information, and making it a catch-all rather than a series of dedicated scalar queries hasn't really simplified life.

In the future, when we improve the ability to look up bitmaps through a filter, we will probably also want to teach the block layer to automatically let filters pass this request on through.

---
 block/qcow2.h                | 1 +
 include/block/block_int.h    | 1 +
 include/block/dirty-bitmap.h | 1 +
 block/dirty-bitmap.c         | 9 +++++++++
 block/qcow2-bitmap.c         | 7 +++++++
 block/qcow2.c                | 2 ++
 6 files changed, 21 insertions(+)
block/qcow2.h +1 -0
index f4de0a27d5c3..c94beb7f8716 100644
@@ -764,6 +764,7 @@ bool qcow2_co_can_store_new_dirty_bitmap(BlockDriverState *bs,
 int qcow2_co_remove_persistent_dirty_bitmap(BlockDriverState *bs,
                                             const char *name,
                                             Error **errp);
+bool qcow2_supports_persistent_dirty_bitmap(BlockDriverState *bs);
 
 ssize_t coroutine_fn
 qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
include/block/block_int.h +1 -0
index df6d0273d679..5bcd6aa39f6c 100644
@@ -560,6 +560,7 @@ struct BlockDriver {
                              uint64_t parent_perm, uint64_t parent_shared,
                              uint64_t *nperm, uint64_t *nshared);
 
+    bool (*bdrv_supports_persistent_dirty_bitmap)(BlockDriverState *bs);
     bool (*bdrv_co_can_store_new_dirty_bitmap)(BlockDriverState *bs,
                                                const char *name,
                                                uint32_t granularity,
include/block/dirty-bitmap.h +1 -0
index 8a1002941892..5a8d52e4deaf 100644
@@ -16,6 +16,7 @@ typedef enum BitmapCheckFlags {
 
 #define BDRV_BITMAP_MAX_NAME_SIZE 1023
 
+bool bdrv_supports_persistent_dirty_bitmap(BlockDriverState *bs);
 BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
                                           uint32_t granularity,
                                           const char *name,
block/dirty-bitmap.c +9 -0
index 063793e31606..f9bfc77985e8 100644
@@ -478,6 +478,15 @@ int bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char *name,
     }
 }
 
+bool
+bdrv_supports_persistent_dirty_bitmap(BlockDriverState *bs)
+{
+    if (bs->drv && bs->drv->bdrv_supports_persistent_dirty_bitmap) {
+        return bs->drv->bdrv_supports_persistent_dirty_bitmap(bs);
+    }
+    return false;
+}
+
 static bool coroutine_fn
 bdrv_co_can_store_new_dirty_bitmap(BlockDriverState *bs, const char *name,
                                    uint32_t granularity, Error **errp)
block/qcow2-bitmap.c +7 -0
index cb06954b4a5a..1cf6d2ab77a3 100644
@@ -1748,3 +1748,10 @@ fail:
                   name, bdrv_get_device_or_node_name(bs));
     return false;
 }
+
+bool qcow2_supports_persistent_dirty_bitmap(BlockDriverState *bs)
+{
+    BDRVQcow2State *s = bs->opaque;
+
+    return s->qcow_version >= 3;
+}
block/qcow2.c +2 -0
index 1ad95ff04851..1c8f3ab8ae68 100644
@@ -5663,6 +5663,8 @@ BlockDriver bdrv_qcow2 = {
     .bdrv_detach_aio_context  = qcow2_detach_aio_context,
     .bdrv_attach_aio_context  = qcow2_attach_aio_context,
 
+    .bdrv_supports_persistent_dirty_bitmap =
+            qcow2_supports_persistent_dirty_bitmap,
     .bdrv_co_can_store_new_dirty_bitmap = qcow2_co_can_store_new_dirty_bitmap,
     .bdrv_co_remove_persistent_dirty_bitmap =
             qcow2_co_remove_persistent_dirty_bitmap,
Signature · 1 line
2.26.2
Reviewed-by on v4 3/9
#6 lore raw

Re: [PATCH v4 3/9] block: Make it easier to learn which BDS support bitmaps

Quoted 20 lines · Eric Blake

Upcoming patches will enhance bitmap support in qemu-img, but in doing so, it turns out to be nice to suppress output when persistent bitmaps make no sense (such as on a qcow2 v2 image). Add a hook to make this easier to query.

This patch adds a new callback .bdrv_supports_persistent_dirty_bitmap, rather than trying to shoehorn the answer in via existing callbacks. In particular, while it might have been possible to overload .bdrv_co_can_store_new_dirty_bitmap to special-case a NULL input to answer whether any persistent bitmaps are supported, that is at odds with whether a particular bitmap can be stored (for example, even on an image that supports persistent bitmaps but has currently filled up the maximum number of bitmaps, attempts to store another one should fail); and the new functionality doesn't require coroutine safety. Similarly, we could have added one more piece of information to .bdrv_get_info, but then again, most callers to that function tend to already discard extraneous information, and making it a catch-all rather than a series of dedicated scalar queries hasn't really simplified life.

Quoted 3 lines

In the future, when we improve the ability to look up bitmaps through a filter, we will probably also want to teach the block layer to automatically let filters pass this request on through.

Hm. I think that bitmap at filter bs is a valid thing (moreover I have a plan to use it for one issue), so I'm not sure that it's good idea to do any generic logic around bitmaps work through filters, better to always address the exact node you mean..

Quoted 2 lines
Signature · 2 lines
Best regards,
Vladimir
author
#7 lore raw

Re: [PATCH v4 3/9] block: Make it easier to learn which BDS support bitmaps

Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Quoted 3 lines

In the future, when we improve the ability to look up bitmaps through a filter, we will probably also want to teach the block layer to automatically let filters pass this request on through.

Hm. I think that bitmap at filter bs is a valid thing (moreover I have a plan to use it for one issue), so I'm not sure that it's good idea to do any generic logic around bitmaps work through filters, better to always address the exact node you mean..

In the immediate term, the only user of this new function is qemu-img, and qemu-img is not setting up filters, so whether this callback can see through filters is moot (that is, we are always addressing the exact node the user passed on the command line, for whatever bitmap operations qemu-img does). In the long term, being able to see bitmaps through filters is better addressed after we finally include Max's series that include filter handling overall.

Quoted 5 lines
Quoted 3 lines

Thanks.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#8 lore raw

PATCH v4 4/9 blockdev: Promote several bitmap functions to non-static

In reply to Eric Blake

The next patch will split blockdev.c, which will require accessing some previously-static functions from more than one .c file. But part of promoting a function to public is picking a naming scheme that does not reek of exposing too many internals (two of the three functions were named starting with 'do_'). To make future code motion easier, perform the function rename and non-static promotion into its own patch.

---
 include/block/block_int.h | 12 +++++++++++
 blockdev.c                | 45 ++++++++++++++++-----------------------
 2 files changed, 30 insertions(+), 27 deletions(-)
include/block/block_int.h +12 -0
index 5bcd6aa39f6c..cd85899ea2b5 100644
@@ -1344,4 +1344,16 @@ int coroutine_fn bdrv_co_create_opts_simple(BlockDriver *drv,
                                             Error **errp);
 extern QemuOptsList bdrv_create_opts_simple;
 
+BdrvDirtyBitmap *block_dirty_bitmap_lookup(const char *node,
+                                           const char *name,
+                                           BlockDriverState **pbs,
+                                           Error **errp);
+BdrvDirtyBitmap *block_dirty_bitmap_merge(const char *node, const char *target,
+                                          BlockDirtyBitmapMergeSourceList *bms,
+                                          HBitmap **backup, Error **errp);
+BdrvDirtyBitmap *block_dirty_bitmap_remove(const char *node, const char *name,
+                                           bool release,
+                                           BlockDriverState **bitmap_bs,
+                                           Error **errp);
+
 #endif /* BLOCK_INT_H */
blockdev.c +18 -27

95 lines not on this page · raw message

Signature · 1 line
2.26.2
Reviewed-by on v4 4/9
#9 lore raw

Re: [PATCH v4 4/9] blockdev: Promote several bitmap functions to non-static

Quoted 10 lines · Eric Blake

The next patch will split blockdev.c, which will require accessing some previously-static functions from more than one .c file. But part of promoting a function to public is picking a naming scheme that does not reek of exposing too many internals (two of the three functions were named starting with 'do_'). To make future code motion easier, perform the function rename and non-static promotion into its own patch.

Signature · 2 lines
Best regards,
Vladimir
#10 lore raw

Re: [PATCH v4 4/9] blockdev: Promote several bitmap functions to non-static

In reply to Eric Blake

blockdev.c void qmp_block_dirty_bitmap_disable(const char *node, const char *name, · line 2608

Quoted 6 lines · Eric Blake
blockdev.c +3 -1
-        HBitmap **backup, Error **errp)
+BdrvDirtyBitmap *block_dirty_bitmap_merge(const char *node, const char *target,
+                                          BlockDirtyBitmapMergeSourceList *bitmaps,
+                                          HBitmap **backup, Error **errp)
  {
      BlockDriverState *bs;

s/bitmaps/bms/ to match declaration and fit into 80 characters

Signature · 2 lines
Best regards,
Vladimir
author
#11 lore raw

Re: [PATCH v4 4/9] blockdev: Promote several bitmap functions to non-static

Quoted 11 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Eric Blake

-        HBitmap **backup, Error **errp) +BdrvDirtyBitmap *block_dirty_bitmap_merge(const char *node, const char *target,
+
BlockDirtyBitmapMergeSourceList *bitmaps,
+                                          HBitmap **backup, Error **errp)
  {
      BlockDriverState *bs;

Quoted 1 line

s/bitmaps/bms/ to match declaration and fit into 80 characters

Can do, although it has ripple effects to 5/9 (as I wanted that to be pure code motion).

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#12 lore raw

PATCH v4 5/9 blockdev: Split off basic bitmap operations for qemu-img

In reply to Eric Blake

Upcoming patches want to add some basic bitmap manipulation abilities to qemu-img. But blockdev.o is too heavyweight to link into qemu-img (among other things, it would drag in block jobs and transaction support - qemu-img does offline manipulation, where atomicity is less important because there are no concurrent modifications to compete with), so it's time to split off the bare bones of what we will need into a new file block/monitor/bitmap-qmp-cmds.o.

This is sufficient to expose 6 QMP commands for use by qemu-img (add, remove, clear, enable, disable, merge), as well as move the three helper functions touched in the previous patch. Regarding MAINTAINERS, the new file is automatically part of block core, but also makes sense as related to other dirty bitmap files.

---
 Makefile.objs                   |   3 +-
 block/monitor/bitmap-qmp-cmds.c | 323 ++++++++++++++++++++++++++++++++
 blockdev.c                      | 284 ----------------------------
 MAINTAINERS                     |   1 +
 block/monitor/Makefile.objs     |   1 +
 5 files changed, 326 insertions(+), 286 deletions(-)
 create mode 100644 block/monitor/bitmap-qmp-cmds.c
Makefile.objs +1 -2
index a7c967633acf..99774cfd2545 100644
@@ -13,9 +13,8 @@ chardev-obj-y = chardev/
 
 authz-obj-y = authz/
 
-block-obj-y = nbd/
+block-obj-y = block/ block/monitor/ nbd/ scsi/
 block-obj-y += block.o blockjob.o job.o
-block-obj-y += block/ scsi/
 block-obj-y += qemu-io-cmds.o
 block-obj-$(CONFIG_REPLICATION) += replication.o
 
block/monitor/bitmap-qmp-cmds.c new file +323 -0

323 lines not on this page · raw message

blockdev.c +0 -284

296 lines not on this page · raw message

MAINTAINERS +1 -0
index 1f84e3ae2c6a..6a8dc1e69d42 100644
@@ -2007,6 +2007,7 @@ L: [email protected]
 S: Supported
 F: include/qemu/hbitmap.h
 F: include/block/dirty-bitmap.h
+F: block/monitor/bitmap-qmp-cmds.c
 F: block/dirty-bitmap.c
 F: block/qcow2-bitmap.c
 F: migration/block-dirty-bitmap.c
block/monitor/Makefile.objs +1 -0
index 0a74f9a8b5b7..39acf8502224 100644
@@ -1 +1,2 @@
 common-obj-y += block-hmp-cmds.o
+block-obj-y += bitmap-qmp-cmds.o
Signature · 1 line
2.26.2
Reviewed-by on v4 5/9
#13 lore raw

Re: [PATCH v4 5/9] blockdev: Split off basic bitmap operations for qemu-img

block/monitor/bitmap-qmp-cmds.c

Quoted 45 lines · Eric Blake

Upcoming patches want to add some basic bitmap manipulation abilities to qemu-img. But blockdev.o is too heavyweight to link into qemu-img (among other things, it would drag in block jobs and transaction support - qemu-img does offline manipulation, where atomicity is less important because there are no concurrent modifications to compete with), so it's time to split off the bare bones of what we will need into a new file block/monitor/bitmap-qmp-cmds.o.

This is sufficient to expose 6 QMP commands for use by qemu-img (add, remove, clear, enable, disable, merge), as well as move the three helper functions touched in the previous patch. Regarding MAINTAINERS, the new file is automatically part of block core, but also makes sense as related to other dirty bitmap files.

---
  Makefile.objs                   |   3 +-
  block/monitor/bitmap-qmp-cmds.c | 323 ++++++++++++++++++++++++++++++++
  blockdev.c                      | 284 ----------------------------
  MAINTAINERS                     |   1 +
  block/monitor/Makefile.objs     |   1 +
  5 files changed, 326 insertions(+), 286 deletions(-)

create mode 100644 block/monitor/bitmap-qmp-cmds.c

Makefile.objs +1 -2
index a7c967633acf..99774cfd2545 100644
@@ -13,9 +13,8 @@ chardev-obj-y = chardev/
 
  authz-obj-y = authz/
 
-block-obj-y = nbd/
+block-obj-y = block/ block/monitor/ nbd/ scsi/
  block-obj-y += block.o blockjob.o job.o
-block-obj-y += block/ scsi/
  block-obj-y += qemu-io-cmds.o
  block-obj-$(CONFIG_REPLICATION) += replication.o
 
block/monitor/bitmap-qmp-cmds.c new file +0 -0
new file mode 100644
index 000000000000..748e1e682483

Hmm, shouldn't transaction bitmap actions be moved here too? May be, not in these series..

block/monitor/bitmap-qmp-cmds.c line 2

Quoted 3 lines
block/monitor/bitmap-qmp-cmds.c new file +2 -0
@@ -0,0 +1,323 @@
+/*
+ * QEMU host block device bitmaps

A bit conflicts with tha fact that they are not of block-device level but of node-level.

May be just "Block dirty bitmap qmp commands" ?

block/monitor/bitmap-qmp-cmds.c line 4

Quoted 2 lines
block/monitor/bitmap-qmp-cmds.c new file +2 -0
+ *
+ * Copyright (c) 2003-2008 Fabrice Bellard

Does it really apply here? block-dirty-bitmap-add was added in 2015.. May be Red Hat and Virtuozzo copyrights would be more appropriate.

Quoted 25 lines
block/monitor/bitmap-qmp-cmds.c new file +25 -0
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later.  See the COPYING file in the top-level directory.
+ *
+ * This file incorporates work covered by the following copyright and
+ * permission notice:
+ *
+ * Copyright (c) 2003-2008 Fabrice Bellard
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN

block/monitor/bitmap-qmp-cmds.c line 39

Quoted 10 lines
block/monitor/bitmap-qmp-cmds.c new file +10 -0
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+
+#include "sysemu/blockdev.h"
+#include "block/block.h"
+#include "block/block_int.h"
+#include "qapi/qapi-commands-block.h"
+#include "qapi/error.h"

compiles for with only four:

  #include "qemu/osdep.h"
                                                                                 
  #include "block/block_int.h"
  #include "qapi/qapi-commands-block.h"
  #include "qapi/error.h"

with at least extra includes dropped:

Signature · 2 lines
Best regards,
Vladimir
author
#14 lore raw

Re: [PATCH v4 5/9] blockdev: Split off basic bitmap operations for qemu-img

Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Eric Blake

Upcoming patches want to add some basic bitmap manipulation abilities to qemu-img.  But blockdev.o is too heavyweight to link into qemu-img (among other things, it would drag in block jobs and transaction support - qemu-img does offline manipulation, where atomicity is less important because there are no concurrent modifications to compete with), so it's time to split off the bare bones of what we will need into a new file block/monitor/bitmap-qmp-cmds.o.

This is sufficient to expose 6 QMP commands for use by qemu-img (add, remove, clear, enable, disable, merge), as well as move the three helper functions touched in the previous patch.  Regarding MAINTAINERS, the new file is automatically part of block core, but also makes sense as related to other dirty bitmap files.

Quoted 7 lines
Quoted 5 lines

---

+++ b/block/monitor/bitmap-qmp-cmds.c

Hmm, shouldn't transaction bitmap actions be moved here too? May be, not in these series..

No, the very reason that we split this file off of blockdev.c is that transactions are NOT needed in qemu-img. You have to have separate .o files for what qemu-img needs, vs. what the rest of qemu proper needs, and transactions are only needed in qemu proper at the moment. If we ever need transactions in Kevin's qemu-storage-daemon but not all of blockdev.c, then we'd need yet another .c file independent from either blockdev.c or this patch's new bitmap-qmp-cmds.c.

Quoted 8 lines

block/monitor/bitmap-qmp-cmds.c line 2

Quoted 4 lines
block/monitor/bitmap-qmp-cmds.c new file +2 -0
@@ -0,0 +1,323 @@
+/*
+ * QEMU host block device bitmaps

A bit conflicts with tha fact that they are not of block-device level but of node-level.

May be just "Block dirty bitmap qmp commands" ?

Copy-and-paste from blockdev.c, tweaked by adding one word. Your wording is also fine.

Quoted 5 lines

Review line 4

Quoted 3 lines
(quoted diff) +2 -0
+ *
+ * Copyright (c) 2003-2008 Fabrice Bellard

Does it really apply here? block-dirty-bitmap-add was added in 2015.. May be Red Hat and Virtuozzo copyrights would be more appropriate.

When splitting a file, the safest course of action is to preserve ALL copyright from the original file into the new file.

Adding additional copyright lines is okay as part of submitting new functionality, but in this case, I don't feel comfortable adding Red Hat copyright (my split isn't adding new functionality), and I don't have authorization to add Virtuozzo copyright (as that is not my employer).

Quoted 10 lines

block/monitor/bitmap-qmp-cmds.c line 39

Quoted 7 lines
block/monitor/bitmap-qmp-cmds.c new file +7 -0
+#include "qemu/osdep.h"
+
+#include "sysemu/blockdev.h"
+#include "block/block.h"
+#include "block/block_int.h"
+#include "qapi/qapi-commands-block.h"
+#include "qapi/error.h"

compiles for with only four:

Quoted 4 lines

  #include "qemu/osdep.h"
  #include "block/block_int.h"
  #include "qapi/qapi-commands-block.h"
  #include "qapi/error.h"

Thanks. I blame rebase; an earlier version used a different header in patch 4/9; when I moved things to block_int.h in that patch, I didn't realize that this patch could be improved.

Quoted 4 lines

with at least extra includes dropped:

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#15 lore raw

PATCH v4 6/9 qemu-img: Add bitmap sub-command

In reply to Eric Blake

Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'. A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file). Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.

While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file merges. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go (although you can still use more than one --merge in the command line); in part because qemu-img is offline and therefore atomicity is not an issue.

Upcoming patches will add iotest coverage of these commands while also testing other features.

---
 docs/tools/qemu-img.rst |  24 ++++
 qemu-img.c              | 254 ++++++++++++++++++++++++++++++++++++++++
 qemu-img-cmds.hx        |   7 ++
 3 files changed, 285 insertions(+)
docs/tools/qemu-img.rst +24 -0
index 7d08c48d308f..219483cec279 100644
@@ -281,6 +281,30 @@ Command description:
   For write tests, by default a buffer filled with zeros is written. This can be
   overridden with a pattern byte specified by *PATTERN*.
 
+.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAP
+
+  Perform one or more modifications of the persistent bitmap *BITMAP*
+  in the disk image *FILENAME*.  The various modifications are:
+
+  ``--add`` to create *BITMAP*, enabled to record future edits.
+
+  ``--remove`` to remove *BITMAP*.
+
+  ``--clear`` to clear *BITMAP*.
+
+  ``--enable`` to change *BITMAP* to start recording future edits.
+
+  ``--disable`` to change *BITMAP* to stop recording future edits.
+
+  ``--merge`` to merge the contents of *SOURCE_BITMAP* into *BITMAP*.
+
+  Additional options include ``-g`` which sets a non-default
+  *GRANULARITY* for ``--add``, and ``-b`` and ``-F`` which select an
+  alternative source file for all *SOURCE* bitmaps used by
+  ``--merge``.
+
+  To see what bitmaps are present in an image, use ``qemu-img info``.
+
 .. option:: check [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [--output=OFMT] [-r [leaks | all]] [-T SRC_CACHE] [-U] FILENAME
 
   Perform a consistency check on the disk image *FILENAME*. The command can
qemu-img.c +254 -0

278 lines not on this page · raw message

qemu-img-cmds.hx +7 -0
index e0886437b1f2..011688245668 100644
@@ -20,6 +20,13 @@ DEF("bench", img_bench,
 SRST
 .. option:: bench [-c COUNT] [-d DEPTH] [-f FMT] [--flush-interval=FLUSH_INTERVAL] [-i AIO] [-n] [--no-drain] [-o OFFSET] [--pattern=PATTERN] [-q] [-s BUFFER_SIZE] [-S STEP_SIZE] [-t CACHE] [-w] [-U] FILENAME
 ERST
+
+DEF("bitmap", img_bitmap,
+    "bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b source_file [-F source_fmt]] [-g granularity] [--object objectdef] [--image-opts | -f fmt] filename bitmap")
+SRST
+.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAP
+ERST
+
 DEF("check", img_check,
     "check [--object objectdef] [--image-opts] [-q] [-f fmt] [--output=ofmt] [-r [leaks | all]] [-T src_cache] [-U] filename")
 SRST
Signature · 1 line
2.26.2
#16 lore raw

Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command

Quoted 20 lines · Eric Blake

Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'. A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file). Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.

While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file merges. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go (although you can still use more than one --merge in the command line); in part because qemu-img is offline and therefore atomicity is not an issue.

Quoted 5 lines

Upcoming patches will add iotest coverage of these commands while also testing other features.

I'm sorry for asking it only now on v4.. But still. Why do we need it? We can instead run qemu binary (or even new qemu-storage-daemon) and just use existing qmp commands. Is there a real benefit in developing qemu-img, maintaining two interfaces for the same thing? Of-course, just run qmp commands from terminal is a lot less comfortable than just a qemu img command.. But may be we need some wrapper, which make it simple to run one qmp command on an image?

It's simple to make a python wrapper working like

qemu-qmp block-dirty-bitmap-add '{node: self, name: bitmap0, persistent: true}' /path/to/x.qcow2

Signature · 2 lines
Best regards,
Vladimir
author
#17 lore raw

Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command

Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Quoted 12 lines · Eric Blake

Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist).  Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'.  A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file).  Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.

Quoted 1 line

I'm sorry for asking it only now on v4.. But still. Why do we need it?

Ease of use.

Quoted 1 line

We can instead run qemu binary (or even new qemu-storage-daemon) and just use existing qmp commands. Is there a real benefit in developing qemu-img, maintaining two interfaces for the same thing?

If it makes someone's life easier, and is not hard to maintain, then yes. A command line interface that calls into QMP is not hard to maintain. And _I_ certainly found it easier to write iotests with this patch in place, so it already has at least one client.

Quoted 5 lines

Of-course, just run qmp commands from terminal is a lot less comfortable than just a qemu img command.. But may be we need some wrapper, which make it simple to run one qmp command on an image?

It's simple to make a python wrapper working like

qemu-qmp block-dirty-bitmap-add '{node: self, name: bitmap0, persistent: true}' /path/to/x.qcow2

This _IS_ such a wrapper. The whole point of this patch is that it is now simpler to run one (or more) QMP command on an offline image from the command line. Just because I wrote it in C instead of python, and attached it to an existing tool instead of writing a new tool, doesn't change the fact that it is just a wrapper around the existing QMP commands.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#18 lore raw

Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command

Quoted 24 lines · Eric Blake
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Quoted 12 lines · Eric Blake

Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist).  Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'.  A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file).  Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.

Quoted 1 line

I'm sorry for asking it only now on v4.. But still. Why do we need it?

Ease of use.

Quoted 1 line

We can instead run qemu binary (or even new qemu-storage-daemon) and just use existing qmp commands. Is there a real benefit in developing qemu-img, maintaining two interfaces for the same thing?

If it makes someone's life easier, and is not hard to maintain, then yes.  A command line interface that calls into QMP is not hard to maintain.  And _I_ certainly found it easier to write iotests with this patch in place, so it already has at least one client.

Quoted 8 lines
Quoted 5 lines

Of-course, just run qmp commands from terminal is a lot less comfortable than just a qemu img command.. But may be we need some wrapper, which make it simple to run one qmp command on an image?

It's simple to make a python wrapper working like

qemu-qmp block-dirty-bitmap-add '{node: self, name: bitmap0, persistent: true}' /path/to/x.qcow2

This _IS_ such a wrapper.  The whole point of this patch is that it is now simpler to run one (or more) QMP command on an offline image from the command line.  Just because I wrote it in C instead of python, and attached it to an existing tool instead of writing a new tool, doesn't change the fact that it is just a wrapper around the existing QMP commands.

OK, that's right.

The thing that I didn't like is that we have to make cli-to-qapi interface mapping by hand. But I see now that interface you implementing is prepared to make several actions with same bitmap-name, which can't be achieved with some kind of automatic interface matching anyway, so my proposal don't match your needs, sorry for my haste)

Signature · 2 lines
Best regards,
Vladimir
Reviewed-by on v4 6/9
#19 lore raw

Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command

In reply to Eric Blake
Quoted 260 lines · Eric Blake

Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'. A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file). Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.

While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file merges. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go (although you can still use more than one --merge in the command line); in part because qemu-img is offline and therefore atomicity is not an issue.

Upcoming patches will add iotest coverage of these commands while also testing other features.

---
  docs/tools/qemu-img.rst |  24 ++++
  qemu-img.c              | 254 ++++++++++++++++++++++++++++++++++++++++
  qemu-img-cmds.hx        |   7 ++
  3 files changed, 285 insertions(+)
docs/tools/qemu-img.rst +24 -0
index 7d08c48d308f..219483cec279 100644
@@ -281,6 +281,30 @@ Command description:
    For write tests, by default a buffer filled with zeros is written. This can be
    overridden with a pattern byte specified by *PATTERN*.
 
+.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAP
+
+  Perform one or more modifications of the persistent bitmap *BITMAP*
+  in the disk image *FILENAME*.  The various modifications are:
+
+  ``--add`` to create *BITMAP*, enabled to record future edits.
+
+  ``--remove`` to remove *BITMAP*.
+
+  ``--clear`` to clear *BITMAP*.
+
+  ``--enable`` to change *BITMAP* to start recording future edits.
+
+  ``--disable`` to change *BITMAP* to stop recording future edits.
+
+  ``--merge`` to merge the contents of *SOURCE_BITMAP* into *BITMAP*.
+
+  Additional options include ``-g`` which sets a non-default
+  *GRANULARITY* for ``--add``, and ``-b`` and ``-F`` which select an
+  alternative source file for all *SOURCE* bitmaps used by
+  ``--merge``.
+
+  To see what bitmaps are present in an image, use ``qemu-img info``.
+
  .. option:: check [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [--output=OFMT] [-r [leaks | all]] [-T SRC_CACHE] [-U] FILENAME
 
    Perform a consistency check on the disk image *FILENAME*. The command can
qemu-img.c +165 -0
index b6e8af9202a5..8c99e68ba8aa 100644
@@ -28,6 +28,7 @@
  #include "qemu-common.h"
  #include "qemu-version.h"
  #include "qapi/error.h"
+#include "qapi/qapi-commands-block-core.h"
  #include "qapi/qapi-visit-block-core.h"
  #include "qapi/qobject-output-visitor.h"
  #include "qapi/qmp/qjson.h"
@@ -71,6 +72,12 @@ enum {
      OPTION_SHRINK = 266,
      OPTION_SALVAGE = 267,
      OPTION_TARGET_IS_ZERO = 268,
+    OPTION_ADD = 269,
+    OPTION_REMOVE = 270,
+    OPTION_CLEAR = 271,
+    OPTION_ENABLE = 272,
+    OPTION_DISABLE = 273,
+    OPTION_MERGE = 274,
  };
 
  typedef enum OutputFormat {
@@ -169,6 +176,14 @@ static void QEMU_NORETURN help(void)
             "  '-n' skips the target volume creation (useful if the volume is created\n"
             "       prior to running qemu-img)\n"
             "\n"
+           "Parameters to bitmap subcommand:\n"
+           "  'bitmap' is the name of the bitmap to manipulate, through one or more\n"
+           "       actions from '--add', '--remove', '--clear', '--enable', '--disable',\n"
+           "       or '--merge source'\n"
+           "  '-g granularity' sets the granularity for '--add' actions\n"
+           "  '-b source' and '-F src_fmt' tell '--merge' actions to find the source\n"
+           "       bitmaps from an alternative file\n"
+           "\n"
             "Parameters to check subcommand:\n"
             "  '-r' tries to repair any inconsistencies that are found during the check.\n"
             "       '-r leaks' repairs only cluster leaks, whereas '-r all' fixes all\n"
@@ -4461,6 +4476,245 @@ out:
      return 0;
  }
 
+enum ImgBitmapAct {
+    BITMAP_ADD,
+    BITMAP_REMOVE,
+    BITMAP_CLEAR,
+    BITMAP_ENABLE,
+    BITMAP_DISABLE,
+    BITMAP_MERGE,
+};
+typedef struct ImgBitmapAction {
+    enum ImgBitmapAct act;
+    const char *src; /* only used for merge */
+    QSIMPLEQ_ENTRY(ImgBitmapAction) next;
+} ImgBitmapAction;
+
+static int img_bitmap(int argc, char **argv)
+{
+    Error *err = NULL;
+    int c, ret = -1;
+    QemuOpts *opts = NULL;
+    const char *fmt = NULL, *src_fmt = NULL, *src_filename = NULL;
+    const char *filename, *bitmap;
+    BlockBackend *blk = NULL, *src = NULL;
+    BlockDriverState *bs = NULL, *src_bs = NULL;
+    bool image_opts = false;
+    int64_t granularity = 0;
+    bool add = false, merge = false;
+    QSIMPLEQ_HEAD(, ImgBitmapAction) actions;
+    ImgBitmapAction *act, *act_next;
+    const char *op;
+
+    QSIMPLEQ_INIT(&actions);
+
+    for (;;) {
+        static const struct option long_options[] = {
+            {"help", no_argument, 0, 'h'},
+            {"object", required_argument, 0, OPTION_OBJECT},
+            {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
+            {"add", no_argument, 0, OPTION_ADD},
+            {"remove", no_argument, 0, OPTION_REMOVE},
+            {"clear", no_argument, 0, OPTION_CLEAR},
+            {"enable", no_argument, 0, OPTION_ENABLE},
+            {"disable", no_argument, 0, OPTION_DISABLE},
+            {"merge", required_argument, 0, OPTION_MERGE},
+            {"granularity", required_argument, 0, 'g'},
+            {"source-file", required_argument, 0, 'b'},
+            {"source-format", required_argument, 0, 'F'},
+            {0, 0, 0, 0}
+        };
+        c = getopt_long(argc, argv, ":b:f:F:g:h", long_options, NULL);
+        if (c == -1) {
+            break;
+        }
+
+        switch (c) {
+        case ':':
+            missing_argument(argv[optind - 1]);
+            break;
+        case '?':
+            unrecognized_option(argv[optind - 1]);
+            break;
+        case 'h':
+            help();
+            break;
+        case 'b':
+            src_filename = optarg;
+            break;
+        case 'f':
+            fmt = optarg;
+            break;
+        case 'F':
+            src_fmt = optarg;
+            break;
+        case 'g':
+            granularity = cvtnum(optarg);
+            if (granularity < 0) {
+                error_report("Invalid granularity specified");
+                return 1;
+            }
+            break;
+        case OPTION_ADD:
+            act = g_new0(ImgBitmapAction, 1);
+            act->act = BITMAP_ADD;
+            QSIMPLEQ_INSERT_TAIL(&actions, act, next);
+            add = true;
+            break;
+        case OPTION_REMOVE:
+            act = g_new0(ImgBitmapAction, 1);
+            act->act = BITMAP_REMOVE;
+            QSIMPLEQ_INSERT_TAIL(&actions, act, next);
+            break;
+        case OPTION_CLEAR:
+            act = g_new0(ImgBitmapAction, 1);
+            act->act = BITMAP_CLEAR;
+            QSIMPLEQ_INSERT_TAIL(&actions, act, next);
+            break;
+        case OPTION_ENABLE:
+            act = g_new0(ImgBitmapAction, 1);
+            act->act = BITMAP_ENABLE;
+            QSIMPLEQ_INSERT_TAIL(&actions, act, next);
+            break;
+        case OPTION_DISABLE:
+            act = g_new0(ImgBitmapAction, 1);
+            act->act = BITMAP_DISABLE;
+            QSIMPLEQ_INSERT_TAIL(&actions, act, next);
+            break;
+        case OPTION_MERGE:
+            act = g_new0(ImgBitmapAction, 1);
+            act->act = BITMAP_MERGE;
+            act->src = optarg;
+            QSIMPLEQ_INSERT_TAIL(&actions, act, next);
+            merge = true;
+            break;
+        case OPTION_OBJECT:
+            opts = qemu_opts_parse_noisily(&qemu_object_opts, optarg, true);
+            if (!opts) {
+                goto out;
+            }
+            break;
+        case OPTION_IMAGE_OPTS:
+            image_opts = true;
+            break;
+        }
+    }
+
+    if (qemu_opts_foreach(&qemu_object_opts,
+                          user_creatable_add_opts_foreach,
+                          qemu_img_object_print_help, &error_fatal)) {
+        goto out;
+    }
+
+    if (QSIMPLEQ_EMPTY(&actions)) {
+        error_report("Need at least one of --add, --remove, --clear, "
+                     "--enable, --disable, or --merge");
+        goto out;
+    }
+
+    if (granularity && !add) {
+        error_report("granularity only supported with --add");
+        goto out;
+    }
+    if (src_fmt && !src_filename) {
+        error_report("-F only supported with -b");
+        goto out;
+    }
+    if (src_filename && !merge) {
+        error_report("Merge bitmap source file only supported with "
+                     "--merge");
+        goto out;
+    }
+

qemu-img.c out: · line 4638

Quoted 10 lines
qemu-img.c +10 -0
+    if (optind != argc - 2) {
+        error_report("Expecting filename and bitmap name");
+        goto out;
+    }
+
+    filename = argv[optind];
+    bitmap = argv[optind + 1];
+
+    blk = img_open(image_opts, filename, fmt, BDRV_O_RDWR, false, false,
+                   false);

fit in one line

qemu-img.c out: · line 4645

Quoted 7 lines
qemu-img.c +7 -0
+    if (!blk) {
+        goto out;
+    }
+    bs = blk_bs(blk);
+    if (src_filename) {
+        src = img_open(NULL, src_filename, src_fmt, 0, false, false,
+                       false);

s/NULL/false/

also, fit in one line

Quoted 43 lines
qemu-img.c +43 -0
+        if (!src) {
+            goto out;
+        }
+        src_bs = blk_bs(src);
+    } else {
+        src_bs = bs;
+    }
+
+    QSIMPLEQ_FOREACH_SAFE(act, &actions, next, act_next) {
+        switch (act->act) {
+        case BITMAP_ADD:
+            qmp_block_dirty_bitmap_add(bs->node_name, bitmap,
+                                       !!granularity, granularity, true, true,
+                                       false, false, &err);
+            op = "add";
+            break;
+        case BITMAP_REMOVE:
+            qmp_block_dirty_bitmap_remove(bs->node_name, bitmap, &err);
+            op = "remove";
+            break;
+        case BITMAP_CLEAR:
+            qmp_block_dirty_bitmap_clear(bs->node_name, bitmap, &err);
+            op = "clear";
+            break;
+        case BITMAP_ENABLE:
+            qmp_block_dirty_bitmap_enable(bs->node_name, bitmap, &err);
+            op = "enable";
+            break;
+        case BITMAP_DISABLE:
+            qmp_block_dirty_bitmap_disable(bs->node_name, bitmap, &err);
+            op = "disable";
+            break;
+        case BITMAP_MERGE: {
+            BlockDirtyBitmapMergeSource *merge_src;
+            BlockDirtyBitmapMergeSourceList *list;
+
+            merge_src = g_new0(BlockDirtyBitmapMergeSource, 1);
+            merge_src->type = QTYPE_QDICT;
+            merge_src->u.external.node = g_strdup(src_bs->node_name);
+            merge_src->u.external.name = g_strdup(act->src);
+            list = g_new0(BlockDirtyBitmapMergeSourceList, 1);
+            list->value = merge_src;
+            qmp_block_dirty_bitmap_merge(bs->node_name, bitmap, list, &err);

qemu-img.c out: · line 4698

Quoted 10 lines
qemu-img.c +10 -0
+            qapi_free_BlockDirtyBitmapMergeSourceList(list);
+            op = "merge";
+            break;
+        }
+        default:
+            g_assert_not_reached();
+        }
+
+        if (err) {
+            error_reportf_err(err, "Operation %s on bitmap %s failed",

s/failed/failed: /

qemu-img.c out: · line 4700

Quoted 2 lines
qemu-img.c +2 -0
+                              op, bitmap);
+            ret = -1;

dead assignment: you never set ret after first initialization to -1.

Quoted 5 lines
qemu-img.c +5 -0
+            goto out;
+        }
+        g_free(act);
+    }
+

qemu-img.c out: · line 4715

Quoted 10 lines
qemu-img.c +10 -0
+    ret = 0;
+
+ out:
+    blk_unref(src);
+    blk_unref(blk);
+    qemu_opts_del(opts);
+    if (ret) {
+        return 1;
+    }
+    return 0;

Hmm, as it's the only usage of ret, you may initialize it to 1 at function start, and here just "return ret;"

Quoted 10 lines
qemu-img.c +2 -0
+}
+
  #define C_BS      01
  #define C_COUNT   02
  #define C_IF      04
qemu-img-cmds.hx +0 -0
index e0886437b1f2..011688245668 100644

qemu-img-cmds.hx DEF("bench", img_bench, · line 27

Quoted 8 lines
qemu-img-cmds.hx +5 -0
@@ -20,6 +20,13 @@ DEF("bench", img_bench,
  SRST
  .. option:: bench [-c COUNT] [-d DEPTH] [-f FMT] [--flush-interval=FLUSH_INTERVAL] [-i AIO] [-n] [--no-drain] [-o OFFSET] [--pattern=PATTERN] [-q] [-s BUFFER_SIZE] [-S STEP_SIZE] [-t CACHE] [-w] [-U] FILENAME
  ERST
+
+DEF("bitmap", img_bitmap,
+    "bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b source_file [-F source_fmt]] [-g granularity] [--object objectdef] [--image-opts | -f fmt] filename bitmap")
+SRST
+.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAP

Not about this patch, but it's a pity that we have triple duplications (triplications ?) of such lines..

qemu-img-cmds.hx DEF("bench", img_bench, · line 32

Quoted 6 lines
qemu-img-cmds.hx +2 -0
+ERST
+
  DEF("check", img_check,
      "check [--object objectdef] [--image-opts] [-q] [-f fmt] [--output=ofmt] [-r [leaks | all]] [-T src_cache] [-U] filename")
  SRST

With at least s/NULL/false/ and s/failed/failed: / (or with all my tiny suggestions):

Signature · 2 lines
Best regards,
Vladimir
author
#20 lore raw

Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command

Quoted 22 lines · Vladimir Sementsov-Ogievskiy
Quoted 20 lines · Eric Blake

Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist).  Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'.  A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file).  Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.

While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file merges.  Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go (although you can still use more than one --merge in the command line); in part because qemu-img is offline and therefore atomicity is not an issue.

Quoted 5 lines
Quoted 3 lines

+
+    blk = img_open(image_opts, filename, fmt, BDRV_O_RDWR, false, false, +                   false);

fit in one line

That line would be exactly 80; I tend to wrap at 79 or earlier rather than exactly on 80.

Quoted 10 lines
Quoted 8 lines

+    if (!blk) {
+        goto out;
+    }
+    bs = blk_bs(blk);
+    if (src_filename) {
+        src = img_open(NULL, src_filename, src_fmt, 0, false, false, +                       false);

s/NULL/false/

D'oh. And yet it still compiles. Fixing.

Quoted 2 lines

also, fit in one line

Yes, this one's shorter. Fixing.

Quoted 10 lines
Quoted 3 lines

+
+        if (err) {
+            error_reportf_err(err, "Operation %s on bitmap %s failed",

s/failed/failed: /

Quoted 2 lines

+                              op, bitmap);
+            ret = -1;

dead assignment: you never set ret after first initialization to -1.

Fixing both.

Quoted 10 lines
Quoted 10 lines

+            goto out;
+        }
+        g_free(act);
+    }

(quoted diff) +5 -0
+
+    ret = 0;
+
+ out:
+    blk_unref(src);
Quoted 8 lines
Quoted 6 lines

+    blk_unref(blk);
+    qemu_opts_del(opts);
+    if (ret) {
+        return 1;
+    }
+    return 0;

Hmm, as it's the only usage of ret, you may initialize it to 1 at function start, and here just "return ret;"

Yep, done.

Quoted 10 lines
Quoted 8 lines

+DEF("bitmap", img_bitmap,
+    "bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b source_file [-F source_fmt]] [-g granularity] [--object objectdef] [--image-opts | -f fmt] filename bitmap") +SRST
+.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAP

Not about this patch, but it's a pity that we have triple duplications (triplications ?) of such lines..

Yes, it is annoying. But as you say, it's a cleanup for another day, for someone who is interested.

Quoted 3 lines

With at least s/NULL/false/ and s/failed/failed: / (or with all my tiny suggestions):

Thanks; squashing in:

qemu-img.c +4 -9
index 8c99e68ba8aa..f940137cb0e5 100644
@@ -4493,7 +4493,7 @@ typedef struct ImgBitmapAction {
 static int img_bitmap(int argc, char **argv)
 {
     Error *err = NULL;
-    int c, ret = -1;
+    int c, ret = 1;
     QemuOpts *opts = NULL;
     const char *fmt = NULL, *src_fmt = NULL, *src_filename = NULL;
     const char *filename, *bitmap;
@@ -4641,8 +4641,7 @@ static int img_bitmap(int argc, char **argv)
     }
     bs = blk_bs(blk);
     if (src_filename) {
-        src = img_open(NULL, src_filename, src_fmt, 0, false, false,
-                       false);
+        src = img_open(false, src_filename, src_fmt, 0, false, false, false);
         if (!src) {
             goto out;
         }
@@ -4695,9 +4694,8 @@ static int img_bitmap(int argc, char **argv)
         }
 
         if (err) {
-            error_reportf_err(err, "Operation %s on bitmap %s failed",
+            error_reportf_err(err, "Operation %s on bitmap %s failed: ",
                               op, bitmap);
-            ret = -1;
             goto out;
         }
         g_free(act);
@@ -4709,10 +4707,7 @@ static int img_bitmap(int argc, char **argv)
     blk_unref(src);
     blk_unref(blk);
     qemu_opts_del(opts);
-    if (ret) {
-        return 1;
-    }
-    return 0;
+    return ret;
 }
 
 #define C_BS      01
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#21 lore raw

Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command

Quoted 136 lines · Eric Blake
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
Quoted 20 lines · Eric Blake

Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist).  Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'.  A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file).  Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.

While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file merges.  Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go (although you can still use more than one --merge in the command line); in part because qemu-img is offline and therefore atomicity is not an issue.

Quoted 5 lines
Quoted 3 lines

+
+    blk = img_open(image_opts, filename, fmt, BDRV_O_RDWR, false, false, +                   false);

fit in one line

That line would be exactly 80; I tend to wrap at 79 or earlier rather than exactly on 80.

Quoted 10 lines
Quoted 8 lines

+    if (!blk) {
+        goto out;
+    }
+    bs = blk_bs(blk);
+    if (src_filename) {
+        src = img_open(NULL, src_filename, src_fmt, 0, false, false, +                       false);

s/NULL/false/

D'oh.  And yet it still compiles.  Fixing.

Quoted 2 lines

also, fit in one line

Yes, this one's shorter.  Fixing.

Quoted 10 lines
Quoted 3 lines

+
+        if (err) {
+            error_reportf_err(err, "Operation %s on bitmap %s failed",

s/failed/failed: /

Quoted 2 lines

+                              op, bitmap);
+            ret = -1;

dead assignment: you never set ret after first initialization to -1.

Fixing both.

Quoted 10 lines
Quoted 10 lines

+            goto out;
+        }
+        g_free(act);
+    }

(quoted diff) +5 -0
+
+    ret = 0;
+
+ out:
+    blk_unref(src);
Quoted 8 lines
Quoted 6 lines

+    blk_unref(blk);
+    qemu_opts_del(opts);
+    if (ret) {
+        return 1;
+    }
+    return 0;

Hmm, as it's the only usage of ret, you may initialize it to 1 at function start, and here just "return ret;"

Yep, done.

Quoted 6 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+DEF("bitmap", img_bitmap,
+    "bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b source_file [-F source_fmt]] [-g granularity] [--object objectdef] [--image-opts | -f fmt] filename bitmap")
+SRST
+.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAP

Not about this patch, but it's a pity that we have triple duplications (triplications ?) of such lines..

Yes, it is annoying.  But as you say, it's a cleanup for another day, for someone who is interested.

Quoted 3 lines

With at least s/NULL/false/ and s/failed/failed: / (or with all my tiny suggestions):

Thanks; squashing in:

qemu-img.c +3 -5
index 8c99e68ba8aa..f940137cb0e5 100644
@@ -4493,7 +4493,7 @@ typedef struct ImgBitmapAction {
  static int img_bitmap(int argc, char **argv)
  {
      Error *err = NULL;
-    int c, ret = -1;
+    int c, ret = 1;
      QemuOpts *opts = NULL;
      const char *fmt = NULL, *src_fmt = NULL, *src_filename = NULL;
      const char *filename, *bitmap;
@@ -4641,8 +4641,7 @@ static int img_bitmap(int argc, char **argv)
      }
      bs = blk_bs(blk);
      if (src_filename) {
-        src = img_open(NULL, src_filename, src_fmt, 0, false, false,
-                       false);
+        src = img_open(false, src_filename, src_fmt, 0, false, false, false);
          if (!src) {
              goto out;
          }
@@ -4695,9 +4694,8 @@ static int img_bitmap(int argc, char **argv)
          }
 
          if (err) {
-            error_reportf_err(err, "Operation %s on bitmap %s failed",
+            error_reportf_err(err, "Operation %s on bitmap %s failed: ",
                                op, bitmap);
-            ret = -1;
              goto out;
          }
          g_free(act);
@@ -4709,10 +4707,7 @@ static int img_bitmap(int argc, char **argv)
      blk_unref(src);

qemu-img.c static int img_bitmap(int argc, char **argv) · line 4713

Quoted 10 lines
qemu-img.c +1 -4
      blk_unref(blk);
      qemu_opts_del(opts);
-    if (ret) {
-        return 1;
-    }
-    return 0;
+    return ret;
  }
 
  #define C_BS      01

Great, keep my r-b with this.

Signature · 2 lines
Best regards,
Vladimir
author
#22 lore raw

PATCH v4 7/9 qcow2: Expose bitmaps' size during measure

In reply to Eric Blake

It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional QMP field, present when measuring an existing image and output format that both support bitmaps. Update iotest 178 and 190 to updated output, as well as new coverage in 190 demonstrating non-zero values made possible with the recently-added qemu-img bitmap command.

On the command-line side, 'qemu-img measure' gains a new --bitmaps flag. When present, the bitmap size is rolled into the two existing measures (or errors if either the source image or destination format lacks bitmaps); when absent, there is never an error (for back-compat), but the output will instead include a new line item for bitmaps (which you would have to manually add), with that line being omitted in the same cases where passing --bitmaps would error.

The behavior chosen here is symmetrical with the upcoming 'qemu-img convert --bitmaps' being added in the next patch: that is, either both commands will succeed (your qemu-img was new enough to do bitmap manipulations, AND you correctly measured and copied the bitmaps, even if that measurement was 0 because there was nothing to copy) or both fail (either your qemu-img is too old to understand --bitmaps, or it understands it but your choice of images do not support seamless transition of bitmaps because either source, destination, or both lack bitmap support).

The addition of a new field demonstrates why we should always zero-initialize qapi C structs; while the qcow2 driver still fully populates all fields, the raw and crypto drivers had to be tweaked to avoid uninitialized data.

See also: https://bugzilla.redhat.com/1779904

---
 docs/tools/qemu-img.rst          | 10 +++++-
 qapi/block-core.json             | 15 ++++++---
 block/crypto.c                   |  2 +-
 block/qcow2.c                    | 37 +++++++++++++++++++--
 block/raw-format.c               |  2 +-
 qemu-img.c                       | 25 ++++++++++++++
 qemu-img-cmds.hx                 |  4 +--
 tests/qemu-iotests/178.out.qcow2 | 16 +++++++++
 tests/qemu-iotests/190           | 57 ++++++++++++++++++++++++++++++--
 tests/qemu-iotests/190.out       | 35 +++++++++++++++++++-
 10 files changed, 187 insertions(+), 16 deletions(-)
docs/tools/qemu-img.rst +9 -1
index 219483cec279..b6f87ec6d3c0 100644
@@ -593,7 +593,7 @@ Command description:
   For more information, consult ``include/block/block.h`` in QEMU's
   source code.
 
-.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [-l SNAPSHOT_PARAM] FILENAME]
+.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [--bitmaps] [-l SNAPSHOT_PARAM] FILENAME]
 
   Calculate the file size required for a new image.  This information
   can be used to size logical volumes or SAN LUNs appropriately for
@@ -616,6 +616,7 @@ Command description:
 
     required size: 524288
     fully allocated size: 1074069504
+    bitmaps: 0
 
   The ``required size`` is the file size of the new image.  It may be smaller
   than the virtual disk size if the image format supports compact representation.
@@ -625,6 +626,13 @@ Command description:
   occupy with the exception of internal snapshots, dirty bitmaps, vmstate data,
   and other advanced image format features.
 
+  The ``bitmaps size`` is the additional size required if the
+  destination supports persistent bitmaps, in order to additionally
+  copy bitmaps in addition to the guest-visible data.  If the
+  ``--bitmaps`` option was in use, the bitmap size is folded into the
+  required and fully-allocated size for convenience, rather than being
+  a separate line item.
+
 .. option:: snapshot [--object OBJECTDEF] [--image-opts] [-U] [-q] [-l | -a SNAPSHOT | -c SNAPSHOT | -d SNAPSHOT] FILENAME
 
   List, apply, create or delete snapshots in image *FILENAME*.
qapi/block-core.json +10 -5
index 943df1926a91..65280eb9847d 100644
@@ -633,18 +633,23 @@
 # efficiently so file size may be smaller than virtual disk size.
 #
 # The values are upper bounds that are guaranteed to fit the new image file.
-# Subsequent modification, such as internal snapshot or bitmap creation, may
-# require additional space and is not covered here.
+# Subsequent modification, such as internal snapshot or further bitmap
+# creation, may require additional space and is not covered here.
 #
-# @required: Size required for a new image file, in bytes.
+# @required: Size required for a new image file, in bytes, when copying just
+#            guest-visible contents.
 #
 # @fully-allocated: Image file size, in bytes, once data has been written
-#                   to all sectors.
+#                   to all sectors, when copying just guest-visible contents.
+#
+# @bitmaps: Additional size required if all the top-level bitmap metadata in
+#           the source image were to be copied to the destination, present
+#           when the destination supports persistent bitmaps. (since 5.1)
 #
 # Since: 2.10
 ##
 { 'struct': 'BlockMeasureInfo',
-  'data': {'required': 'int', 'fully-allocated': 'int'} }
+  'data': {'required': 'int', 'fully-allocated': 'int', '*bitmaps': 'int'} }
 
 ##
 # @query-block:
block/crypto.c +1 -1
index 6b21d6bf6c01..eadbcb248563 100644
@@ -552,7 +552,7 @@ static BlockMeasureInfo *block_crypto_measure(QemuOpts *opts,
      * Unallocated blocks are still encrypted so allocation status makes no
      * difference to the file size.
      */
-    info = g_new(BlockMeasureInfo, 1);
+    info = g_new0(BlockMeasureInfo, 1);
     info->fully_allocated = luks_payload_size + size;
     info->required = luks_payload_size + size;
     return info;
block/qcow2.c +34 -3
index 1c8f3ab8ae68..be0950016365 100644
@@ -4721,6 +4721,7 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
     PreallocMode prealloc;
     bool has_backing_file;
     bool has_luks;
+    uint64_t bitmaps_size = 0; /* size occupied by bitmaps in in_bs */
 
     /* Parse image creation options */
     cluster_size = qcow2_opt_get_cluster_size_del(opts, &local_err);
@@ -4796,13 +4797,38 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
 
     /* Account for input image */
     if (in_bs) {
+        BdrvDirtyBitmap *bm;
+        size_t bitmap_dir_size = 0;
         int64_t ssize = bdrv_getlength(in_bs);
+
         if (ssize < 0) {
             error_setg_errno(&local_err, -ssize,
                              "Unable to get image virtual_size");
             goto err;
         }
 
+        FOR_EACH_DIRTY_BITMAP(in_bs, bm) {
+            if (bdrv_dirty_bitmap_get_persistence(bm)) {
+                const char *name = bdrv_dirty_bitmap_name(bm);
+                uint32_t granularity = bdrv_dirty_bitmap_granularity(bm);
+                uint64_t bmbits = DIV_ROUND_UP(bdrv_dirty_bitmap_size(bm),
+                                               granularity);
+                uint64_t bmclusters = DIV_ROUND_UP(DIV_ROUND_UP(bmbits,
+                                                                CHAR_BIT),
+                                                   cluster_size);
+
+                /* Assume the entire bitmap is allocated */
+                bitmaps_size += bmclusters * cluster_size;
+                /* Also reserve space for the bitmap table entries */
+                bitmaps_size += ROUND_UP(bmclusters * sizeof(uint64_t),
+                                         cluster_size);
+                /* And space for contribution to bitmap directory size */
+                bitmap_dir_size += ROUND_UP(strlen(name) + 24,
+                                            sizeof(uint64_t));
+            }
+        }
+        bitmaps_size += ROUND_UP(bitmap_dir_size, cluster_size);
+
         virtual_size = ROUND_UP(ssize, cluster_size);
 
         if (has_backing_file) {
@@ -4849,16 +4875,21 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
         required = virtual_size;
     }
 
-    info = g_new(BlockMeasureInfo, 1);
+    info = g_new0(BlockMeasureInfo, 1);
     info->fully_allocated =
         qcow2_calc_prealloc_size(virtual_size, cluster_size,
                                  ctz32(refcount_bits)) + luks_payload_size;
 
-    /* Remove data clusters that are not required.  This overestimates the
+    /*
+     * Remove data clusters that are not required.  This overestimates the
      * required size because metadata needed for the fully allocated file is
-     * still counted.
+     * still counted.  Show bitmaps only if both source and destination
+     * would support them.
      */
     info->required = info->fully_allocated - virtual_size + required;
+    info->has_bitmaps = version >= 3 && in_bs &&
+        bdrv_supports_persistent_dirty_bitmap(in_bs);
+    info->bitmaps = bitmaps_size;
     return info;
 
 err:
block/raw-format.c +1 -1
index 9108e4369628..a134b1954ca2 100644
@@ -346,7 +346,7 @@ static BlockMeasureInfo *raw_measure(QemuOpts *opts, BlockDriverState *in_bs,
                             BDRV_SECTOR_SIZE);
     }
 
-    info = g_new(BlockMeasureInfo, 1);
+    info = g_new0(BlockMeasureInfo, 1);
     info->required = required;
 
     /* Unallocated sectors count towards the file size in raw images */
qemu-img.c +25 -0
index 8c99e68ba8aa..0a326993d7ac 100644
@@ -78,6 +78,7 @@ enum {
     OPTION_ENABLE = 272,
     OPTION_DISABLE = 273,
     OPTION_MERGE = 274,
+    OPTION_BITMAPS = 275,
 };
 
 typedef enum OutputFormat {
@@ -5096,6 +5097,7 @@ static int img_measure(int argc, char **argv)
         {"output", required_argument, 0, OPTION_OUTPUT},
         {"size", required_argument, 0, OPTION_SIZE},
         {"force-share", no_argument, 0, 'U'},
+        {"bitmaps", no_argument, 0, OPTION_BITMAPS},
         {0, 0, 0, 0}
     };
     OutputFormat output_format = OFORMAT_HUMAN;
@@ -5112,6 +5114,7 @@ static int img_measure(int argc, char **argv)
     QemuOpts *sn_opts = NULL;
     QemuOptsList *create_opts = NULL;
     bool image_opts = false;
+    bool bitmaps = false;
     uint64_t img_size = UINT64_MAX;
     BlockMeasureInfo *info = NULL;
     Error *local_err = NULL;
@@ -5192,6 +5195,9 @@ static int img_measure(int argc, char **argv)
             img_size = (uint64_t)sval;
         }
         break;
+        case OPTION_BITMAPS:
+            bitmaps = true;
+            break;
         }
     }
 
@@ -5220,6 +5226,10 @@ static int img_measure(int argc, char **argv)
         error_report("Either --size N or one filename must be specified.");
         goto out;
     }
+    if (!filename && bitmaps) {
+        error_report("--bitmaps is only supported with a filename.");
+        goto out;
+    }
 
     if (filename) {
         in_blk = img_open(image_opts, filename, fmt, 0,
@@ -5275,9 +5285,24 @@ static int img_measure(int argc, char **argv)
         goto out;
     }
 
+    if (bitmaps) {
+        if (!info->has_bitmaps) {
+            error_report("no bitmaps measured, either source or destination "
+                         "format lacks bitmap support");
+            goto out;
+        } else {
+            info->required += info->bitmaps;
+            info->fully_allocated += info->bitmaps;
+            info->has_bitmaps = false;
+        }
+    }
+
     if (output_format == OFORMAT_HUMAN) {
         printf("required size: %" PRIu64 "\n", info->required);
         printf("fully allocated size: %" PRIu64 "\n", info->fully_allocated);
+        if (info->has_bitmaps) {
+            printf("bitmaps size: %" PRIu64 "\n", info->bitmaps);
+        }
     } else {
         dump_json_block_measure_info(info);
     }
qemu-img-cmds.hx +2 -2
index 011688245668..bfcd9f32dddf 100644
@@ -76,9 +76,9 @@ SRST
 ERST
 
 DEF("measure", img_measure,
-"measure [--output=ofmt] [-O output_fmt] [-o options] [--size N | [--object objectdef] [--image-opts] [-f fmt] [-l snapshot_param] filename]")
+"measure [--output=ofmt] [-O output_fmt] [-o options] [--size N | [--object objectdef] [--image-opts] [-f fmt] [--bitmaps] [-l snapshot_param] filename]")
 SRST
-.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [-l SNAPSHOT_PARAM] FILENAME]
+.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [--bitmaps] [-l SNAPSHOT_PARAM] FILENAME]
 ERST
 
 DEF("snapshot", img_snapshot,
tests/qemu-iotests/178.out.qcow2 +16 -0
index f59bf4b2fbc4..0c6ca5e05713 100644
@@ -37,6 +37,7 @@ qemu-img: The image size is too large (try using a larger cluster size)
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
 required size: 196608
 fully allocated size: 196608
+bitmaps size: 0
 
 converted image file size in bytes: 196608
 
@@ -45,6 +46,7 @@ converted image file size in bytes: 196608
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
 required size: 393216
 fully allocated size: 1074135040
+bitmaps size: 0
 wrote 512/512 bytes at offset 512
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 wrote 65536/65536 bytes at offset 65536
@@ -53,6 +55,7 @@ wrote 64512/64512 bytes at offset 134217728
 63 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 required size: 589824
 fully allocated size: 1074135040
+bitmaps size: 0
 
 converted image file size in bytes: 524288
 
@@ -60,6 +63,7 @@ converted image file size in bytes: 524288
 
 required size: 524288
 fully allocated size: 1074135040
+bitmaps size: 0
 
 converted image file size in bytes: 458752
 
@@ -67,16 +71,19 @@ converted image file size in bytes: 458752
 
 required size: 1074135040
 fully allocated size: 1074135040
+bitmaps size: 0
 
 == qcow2 input image and LUKS encryption ==
 
 required size: 2686976
 fully allocated size: 1076232192
+bitmaps size: 0
 
 == qcow2 input image and preallocation (human) ==
 
 required size: 1074135040
 fully allocated size: 1074135040
+bitmaps size: 0
 
 converted image file size in bytes: 1074135040
 
@@ -87,6 +94,7 @@ wrote 8388608/8388608 bytes at offset 0
 8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 required size: 8716288
 fully allocated size: 8716288
+bitmaps size: 0
 
 converted image file size in bytes: 8716288
 
@@ -173,6 +181,7 @@ qemu-img: The image size is too large (try using a larger cluster size)
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0
 {
+    "bitmaps": 0,
     "required": 196608,
     "fully-allocated": 196608
 }
@@ -183,6 +192,7 @@ converted image file size in bytes: 196608
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
 {
+    "bitmaps": 0,
     "required": 393216,
     "fully-allocated": 1074135040
 }
@@ -193,6 +203,7 @@ wrote 65536/65536 bytes at offset 65536
 wrote 64512/64512 bytes at offset 134217728
 63 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 {
+    "bitmaps": 0,
     "required": 589824,
     "fully-allocated": 1074135040
 }
@@ -202,6 +213,7 @@ converted image file size in bytes: 524288
 == qcow2 input image with internal snapshot (json) ==
 
 {
+    "bitmaps": 0,
     "required": 524288,
     "fully-allocated": 1074135040
 }
@@ -211,6 +223,7 @@ converted image file size in bytes: 458752
 == qcow2 input image and a backing file (json) ==
 
 {
+    "bitmaps": 0,
     "required": 1074135040,
     "fully-allocated": 1074135040
 }
@@ -218,6 +231,7 @@ converted image file size in bytes: 458752
 == qcow2 input image and LUKS encryption ==
 
 {
+    "bitmaps": 0,
     "required": 2686976,
     "fully-allocated": 1076232192
 }
@@ -225,6 +239,7 @@ converted image file size in bytes: 458752
 == qcow2 input image and preallocation (json) ==
 
 {
+    "bitmaps": 0,
     "required": 1074135040,
     "fully-allocated": 1074135040
 }
@@ -237,6 +252,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=8388608
 wrote 8388608/8388608 bytes at offset 0
 8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 {
+    "bitmaps": 0,
     "required": 8716288,
     "fully-allocated": 8716288
 }
tests/qemu-iotests/190 +55 -2

75 lines not on this page · raw message

tests/qemu-iotests/190.out +34 -1
index d001942002db..5c35f9268068 100644
@@ -1,11 +1,44 @@
 QA output created by 190
-== Huge file ==
+== Huge file without bitmaps ==
 
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2199023255552
 required size: 2199023255552
 fully allocated size: 2199023255552
 required size: 335806464
 fully allocated size: 2199359062016
+bitmaps size: 0
 required size: 18874368
 fully allocated size: 2199042129920
+bitmaps size: 0
+
+== Huge file with bitmaps ==
+
+qemu-img: --bitmaps is only supported with a filename.
+required size: 2199023255552
+fully allocated size: 2199023255552
+qemu-img: no bitmaps measured, either source or destination format lacks bitmap support
+required size: 7012352
+fully allocated size: 17170432
+qemu-img: no bitmaps measured, either source or destination format lacks bitmap support
+required size: 335806464
+fully allocated size: 2199359062016
+qemu-img: no bitmaps measured, either source or destination format lacks bitmap support
+
+expected bitmap 537198592
+required size: 335806464
+fully allocated size: 2199359062016
+bitmaps size: 537198592
+required size: 873005056
+fully allocated size: 2199896260608
+
+expected bitmap 545259520
+{
+    "bitmaps": 545259520,
+    "required": 18874368,
+    "fully-allocated": 2199042129920
+}
+{
+    "required": 564133888,
+    "fully-allocated": 2199587389440
+}
 *** done
Signature · 1 line
2.26.2
#23 lore raw

Re: [PATCH v4 7/9] qcow2: Expose bitmaps' size during measure

Quoted 70 lines · Eric Blake

It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional QMP field, present when measuring an existing image and output format that both support bitmaps. Update iotest 178 and 190 to updated output, as well as new coverage in 190 demonstrating non-zero values made possible with the recently-added qemu-img bitmap command.

On the command-line side, 'qemu-img measure' gains a new --bitmaps flag. When present, the bitmap size is rolled into the two existing measures (or errors if either the source image or destination format lacks bitmaps); when absent, there is never an error (for back-compat), but the output will instead include a new line item for bitmaps (which you would have to manually add), with that line being omitted in the same cases where passing --bitmaps would error.

The behavior chosen here is symmetrical with the upcoming 'qemu-img convert --bitmaps' being added in the next patch: that is, either both commands will succeed (your qemu-img was new enough to do bitmap manipulations, AND you correctly measured and copied the bitmaps, even if that measurement was 0 because there was nothing to copy) or both fail (either your qemu-img is too old to understand --bitmaps, or it understands it but your choice of images do not support seamless transition of bitmaps because either source, destination, or both lack bitmap support).

The addition of a new field demonstrates why we should always zero-initialize qapi C structs; while the qcow2 driver still fully populates all fields, the raw and crypto drivers had to be tweaked to avoid uninitialized data.

See also: https://bugzilla.redhat.com/1779904

---
  docs/tools/qemu-img.rst          | 10 +++++-
  qapi/block-core.json             | 15 ++++++---
  block/crypto.c                   |  2 +-
  block/qcow2.c                    | 37 +++++++++++++++++++--
  block/raw-format.c               |  2 +-
  qemu-img.c                       | 25 ++++++++++++++
  qemu-img-cmds.hx                 |  4 +--
  tests/qemu-iotests/178.out.qcow2 | 16 +++++++++
  tests/qemu-iotests/190           | 57 ++++++++++++++++++++++++++++++--
  tests/qemu-iotests/190.out       | 35 +++++++++++++++++++-
  10 files changed, 187 insertions(+), 16 deletions(-)
docs/tools/qemu-img.rst +2 -1
index 219483cec279..b6f87ec6d3c0 100644
@@ -593,7 +593,7 @@ Command description:
    For more information, consult ``include/block/block.h`` in QEMU's
    source code.
 
-.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [-l SNAPSHOT_PARAM] FILENAME]
+.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [--bitmaps] [-l SNAPSHOT_PARAM] FILENAME]
 
    Calculate the file size required for a new image.  This information
    can be used to size logical volumes or SAN LUNs appropriately for
@@ -616,6 +616,7 @@ Command description:
 
      required size: 524288
      fully allocated size: 1074069504
+    bitmaps: 0
 
    The ``required size`` is the file size of the new image.  It may be smaller
    than the virtual disk size if the image format supports compact representation.

docs/tools/qemu-img.rst Command description: · line 629

Quoted 4 lines
docs/tools/qemu-img.rst +1 -0
@@ -625,6 +626,13 @@ Command description:
    occupy with the exception of internal snapshots, dirty bitmaps, vmstate data,
    and other advanced image format features.
 
+  The ``bitmaps size`` is the additional size required if the

you called it "bitmaps" in example output above. Should it be consistent? Either "``bitmaps``" here, or "bitmaps size: 0" above?

Quoted 19 lines
docs/tools/qemu-img.rst +6 -0
+  destination supports persistent bitmaps, in order to additionally
+  copy bitmaps in addition to the guest-visible data.  If the
+  ``--bitmaps`` option was in use, the bitmap size is folded into the
+  required and fully-allocated size for convenience, rather than being
+  a separate line item.
+
  .. option:: snapshot [--object OBJECTDEF] [--image-opts] [-U] [-q] [-l | -a SNAPSHOT | -c SNAPSHOT | -d SNAPSHOT] FILENAME
 
    List, apply, create or delete snapshots in image *FILENAME*.
qapi/block-core.json +0 -2
index 943df1926a91..65280eb9847d 100644
@@ -633,18 +633,23 @@
  # efficiently so file size may be smaller than virtual disk size.
  #
  # The values are upper bounds that are guaranteed to fit the new image file.
-# Subsequent modification, such as internal snapshot or bitmap creation, may
-# require additional space and is not covered here.

qapi/block-core.json line 643

Quoted 10 lines
qapi/block-core.json +5 -2
+# Subsequent modification, such as internal snapshot or further bitmap
+# creation, may require additional space and is not covered here.
  #
-# @required: Size required for a new image file, in bytes.
+# @required: Size required for a new image file, in bytes, when copying just
+#            guest-visible contents.
  #
  # @fully-allocated: Image file size, in bytes, once data has been written
-#                   to all sectors.
+#                   to all sectors, when copying just guest-visible contents.

"copying just guest-visible" sounds like something less than "all fully-allocated sectors".. But I don't have better suggestion.. Just, "not including bitmaps" sounds weird too.

Quoted 60 lines
qapi/block-core.json +5 -1
+#
+# @bitmaps: Additional size required if all the top-level bitmap metadata in
+#           the source image were to be copied to the destination, present
+#           when the destination supports persistent bitmaps. (since 5.1)
  #
  # Since: 2.10
  ##
  { 'struct': 'BlockMeasureInfo',
-  'data': {'required': 'int', 'fully-allocated': 'int'} }
+  'data': {'required': 'int', 'fully-allocated': 'int', '*bitmaps': 'int'} }
 
  ##
  # @query-block:
block/crypto.c +1 -1
index 6b21d6bf6c01..eadbcb248563 100644
@@ -552,7 +552,7 @@ static BlockMeasureInfo *block_crypto_measure(QemuOpts *opts,
       * Unallocated blocks are still encrypted so allocation status makes no
       * difference to the file size.
       */
-    info = g_new(BlockMeasureInfo, 1);
+    info = g_new0(BlockMeasureInfo, 1);
      info->fully_allocated = luks_payload_size + size;
      info->required = luks_payload_size + size;
      return info;
block/qcow2.c +12 -0
index 1c8f3ab8ae68..be0950016365 100644
@@ -4721,6 +4721,7 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
      PreallocMode prealloc;
      bool has_backing_file;
      bool has_luks;
+    uint64_t bitmaps_size = 0; /* size occupied by bitmaps in in_bs */
 
      /* Parse image creation options */
      cluster_size = qcow2_opt_get_cluster_size_del(opts, &local_err);
@@ -4796,13 +4797,38 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
 
      /* Account for input image */
      if (in_bs) {
+        BdrvDirtyBitmap *bm;
+        size_t bitmap_dir_size = 0;
          int64_t ssize = bdrv_getlength(in_bs);
+
          if (ssize < 0) {
              error_setg_errno(&local_err, -ssize,
                               "Unable to get image virtual_size");
              goto err;
          }
 
+        FOR_EACH_DIRTY_BITMAP(in_bs, bm) {
+            if (bdrv_dirty_bitmap_get_persistence(bm)) {
+                const char *name = bdrv_dirty_bitmap_name(bm);
+                uint32_t granularity = bdrv_dirty_bitmap_granularity(bm);
+                uint64_t bmbits = DIV_ROUND_UP(bdrv_dirty_bitmap_size(bm),
+                                               granularity);
+                uint64_t bmclusters = DIV_ROUND_UP(DIV_ROUND_UP(bmbits,
+                                                                CHAR_BIT),

block/qcow2.c static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, · line 4827

Quoted 10 lines
block/qcow2.c +10 -0
+                                                   cluster_size);
+
+                /* Assume the entire bitmap is allocated */
+                bitmaps_size += bmclusters * cluster_size;
+                /* Also reserve space for the bitmap table entries */
+                bitmaps_size += ROUND_UP(bmclusters * sizeof(uint64_t),
+                                         cluster_size);
+                /* And space for contribution to bitmap directory size */
+                bitmap_dir_size += ROUND_UP(strlen(name) + 24,
+                                            sizeof(uint64_t));

Could we instead reuse code from qcow2_co_can_store_new_dirty_bitmap(), which calls calc_dir_entry_size() for this thing?
Possibly, make a function qcow2_measure_bitmaps in block/qcow2-bitmaps.c with this FOR_EACH? All details about qcow2 bitmap structures sounds better in block/qcow2-bitmaps.c

Quoted 19 lines
block/qcow2.c +6 -2
+            }
+        }
+        bitmaps_size += ROUND_UP(bitmap_dir_size, cluster_size);
+
          virtual_size = ROUND_UP(ssize, cluster_size);
 
          if (has_backing_file) {
@@ -4849,16 +4875,21 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
          required = virtual_size;
      }
 
-    info = g_new(BlockMeasureInfo, 1);
+    info = g_new0(BlockMeasureInfo, 1);
      info->fully_allocated =
          qcow2_calc_prealloc_size(virtual_size, cluster_size,
                                   ctz32(refcount_bits)) + luks_payload_size;
 
-    /* Remove data clusters that are not required.  This overestimates the
+    /*

block/qcow2.c static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, · line 4892

Quoted 10 lines
block/qcow2.c +6 -1
+     * Remove data clusters that are not required.  This overestimates the
       * required size because metadata needed for the fully allocated file is
-     * still counted.
+     * still counted.  Show bitmaps only if both source and destination
+     * would support them.
       */
      info->required = info->fully_allocated - virtual_size + required;
+    info->has_bitmaps = version >= 3 && in_bs &&
+        bdrv_supports_persistent_dirty_bitmap(in_bs);
+    info->bitmaps = bitmaps_size;

AFAIK, in QAPI, if has_<something> field is false, than <something> must be zero. Maybe, it's only about nested structured fields, not about simple numbers, but I think it's better keep bitmaps 0 in case when has_bitmaps is false.

Also, it seems a bit better to check version earlier, and don't do all the calculations, if we are not going to use them.. But it's a rare backward-compatibility case, I don't care.

Quoted 68 lines
block/qcow2.c +0 -0
      return info;
 
  err:
block/raw-format.c +1 -1
index 9108e4369628..a134b1954ca2 100644
@@ -346,7 +346,7 @@ static BlockMeasureInfo *raw_measure(QemuOpts *opts, BlockDriverState *in_bs,
                              BDRV_SECTOR_SIZE);
      }
 
-    info = g_new(BlockMeasureInfo, 1);
+    info = g_new0(BlockMeasureInfo, 1);
      info->required = required;
 
      /* Unallocated sectors count towards the file size in raw images */
qemu-img.c +10 -0
index 8c99e68ba8aa..0a326993d7ac 100644
@@ -78,6 +78,7 @@ enum {
      OPTION_ENABLE = 272,
      OPTION_DISABLE = 273,
      OPTION_MERGE = 274,
+    OPTION_BITMAPS = 275,
  };
 
  typedef enum OutputFormat {
@@ -5096,6 +5097,7 @@ static int img_measure(int argc, char **argv)
          {"output", required_argument, 0, OPTION_OUTPUT},
          {"size", required_argument, 0, OPTION_SIZE},
          {"force-share", no_argument, 0, 'U'},
+        {"bitmaps", no_argument, 0, OPTION_BITMAPS},
          {0, 0, 0, 0}
      };
      OutputFormat output_format = OFORMAT_HUMAN;
@@ -5112,6 +5114,7 @@ static int img_measure(int argc, char **argv)
      QemuOpts *sn_opts = NULL;
      QemuOptsList *create_opts = NULL;
      bool image_opts = false;
+    bool bitmaps = false;
      uint64_t img_size = UINT64_MAX;
      BlockMeasureInfo *info = NULL;
      Error *local_err = NULL;
@@ -5192,6 +5195,9 @@ static int img_measure(int argc, char **argv)
              img_size = (uint64_t)sval;
          }
          break;
+        case OPTION_BITMAPS:
+            bitmaps = true;
+            break;
          }
      }
 
@@ -5220,6 +5226,10 @@ static int img_measure(int argc, char **argv)
          error_report("Either --size N or one filename must be specified.");
          goto out;
      }
+    if (!filename && bitmaps) {
+        error_report("--bitmaps is only supported with a filename.");
+        goto out;
+    }
 
      if (filename) {
          in_blk = img_open(image_opts, filename, fmt, 0,
@@ -5275,9 +5285,24 @@ static int img_measure(int argc, char **argv)
          goto out;
      }

qemu-img.c static int img_measure(int argc, char **argv) · line 5296

Quoted 10 lines
qemu-img.c +9 -0
 
+    if (bitmaps) {
+        if (!info->has_bitmaps) {
+            error_report("no bitmaps measured, either source or destination "
+                         "format lacks bitmap support");
+            goto out;
+        } else {
+            info->required += info->bitmaps;
+            info->fully_allocated += info->bitmaps;
+            info->has_bitmaps = false;

And here, I think better to zero info->bitmaps as well.

Quoted 12 lines
qemu-img.c +6 -0
+        }
+    }
+
      if (output_format == OFORMAT_HUMAN) {
          printf("required size: %" PRIu64 "\n", info->required);
          printf("fully allocated size: %" PRIu64 "\n", info->fully_allocated);
+        if (info->has_bitmaps) {
+            printf("bitmaps size: %" PRIu64 "\n", info->bitmaps);
+        }
      } else {
          dump_json_block_measure_info(info);
      }

[..]

Quoted 23 lines
tests/qemu-iotests/190 +2 -2
@@ -2,7 +2,7 @@
  #
  # qemu-img measure sub-command tests on huge qcow2 files
  #
-# Copyright (C) 2017 Red Hat, Inc.
+# Copyright (C) 2017-2020 Red Hat, Inc.
  #
  # This program is free software; you can redistribute it and/or modify
  # it under the terms of the GNU General Public License as published by
@@ -42,7 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
  _supported_fmt qcow2
  _supported_proto file
 
-echo "== Huge file =="
+echo "== Huge file without bitmaps =="
  echo
 
  _make_test_img -o 'cluster_size=2M' 2T
@@ -51,6 +51,59 @@ $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG"
  $QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG"
  $QEMU_IMG measure -O qcow2 -o cluster_size=2M -f qcow2 "$TEST_IMG"

tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 62

Quoted 10 lines
tests/qemu-iotests/190 +9 -0
 
+echo
+echo "== Huge file with bitmaps =="
+echo
+
+$QEMU_IMG bitmap --add --granularity 512 -f qcow2 "$TEST_IMG" b1
+$QEMU_IMG bitmap --add -g 2M -f qcow2 "$TEST_IMG" b2
+
+# No bitmap without a source
+$QEMU_IMG measure --bitmaps -O qcow2 --size 10M

should this be ored to 'echo "unexpected success"' as following failures?

Quoted 12 lines
tests/qemu-iotests/190 +12 -0
+# No bitmap output, since raw does not support it
+$QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" ||
+    echo "unexpected failure"
+$QEMU_IMG measure --bitmaps -O raw -f qcow2 "$TEST_IMG" &&
+    echo "unexpected success"
+# No bitmap output, since no bitmaps on raw source
+$QEMU_IMG measure -O qcow2 -f raw "$TEST_IMG" ||
+    echo "unexpected failure"
+$QEMU_IMG measure --bitmaps -O qcow2 -f raw "$TEST_IMG" &&
+    echo "unexpected success"
+# No bitmap output, since v2 does not support it
+$QEMU_IMG measure -O qcow2 -o compat=0.10 -f qcow2 "$TEST_IMG" ||

tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 84

Quoted 10 lines
tests/qemu-iotests/190 +10 -0
+    echo "unexpected failure"
+$QEMU_IMG measure --bitmaps -O qcow2 -o compat=0.10 -f qcow2 "$TEST_IMG" &&
+    echo "unexpected success"
+
+# Compute expected output:
+echo
+val2T=$((2*1024*1024*1024*1024))
+cluster=$((64*1024))
+b1clusters=$(( (val2T/512/8 + cluster - 1) / cluster ))
+b2clusters=$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster ))

comment on the following calculations won't hurt, at least something like
"bitmap clusters + bitmap tables + bitmaps directory"

Quoted 38 lines
tests/qemu-iotests/190 +22 -0
+echo expected bitmap $((b1clusters * cluster +
+			(b1clusters * 8 + cluster - 1) / cluster * cluster +
+			b2clusters * cluster +
+			(b2clusters * 8 + cluster - 1) / cluster * cluster +
+			cluster))
+$QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG"
+$QEMU_IMG measure --bitmaps -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG"
+
+# Compute expected output:
+echo
+cluster=$((2*1024*1024))
+b1clusters=$(( (val2T/512/8 + cluster - 1) / cluster ))
+b2clusters=$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster ))
+echo expected bitmap $((b1clusters * cluster +
+			(b1clusters * 8 + cluster - 1) / cluster * cluster +
+			b2clusters * cluster +
+			(b2clusters * 8 + cluster - 1) / cluster * cluster +
+			cluster))
+$QEMU_IMG measure --output=json -O qcow2 -o cluster_size=2M -f qcow2 "$TEST_IMG"
+$QEMU_IMG measure --output=json --bitmaps -O qcow2 -o cluster_size=2M \
+     -f qcow2 "$TEST_IMG"
+
  # success, all done
  echo "*** done"
  rm -f $seq.full
tests/qemu-iotests/190.out +1 -1
index d001942002db..5c35f9268068 100644
@@ -1,11 +1,44 @@
  QA output created by 190
-== Huge file ==
+== Huge file without bitmaps ==
 
  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2199023255552
  required size: 2199023255552
  fully allocated size: 2199023255552
  required size: 335806464

tests/qemu-iotests/190.out line 17

Quoted 10 lines
tests/qemu-iotests/190.out +7 -0
  fully allocated size: 2199359062016
+bitmaps size: 0
  required size: 18874368
  fully allocated size: 2199042129920
+bitmaps size: 0
+
+== Huge file with bitmaps ==
+
+qemu-img: --bitmaps is only supported with a filename.
+required size: 2199023255552> +fully allocated size: 2199023255552

- same as for "without bitmaps" sections, good

tests/qemu-iotests/190.out line 24

Quoted 6 lines
tests/qemu-iotests/190.out +6 -0
+qemu-img: no bitmaps measured, either source or destination format lacks bitmap support
+required size: 7012352
+fully allocated size: 17170432
+qemu-img: no bitmaps measured, either source or destination format lacks bitmap support
+required size: 335806464
+fully allocated size: 2199359062016

matches "without bitmaps" section (hmm, v2 image needs same space? Ok)

tests/qemu-iotests/190.out line 29

Quoted 5 lines
tests/qemu-iotests/190.out +5 -0
+qemu-img: no bitmaps measured, either source or destination format lacks bitmap support
+
+expected bitmap 537198592
+required size: 335806464
+fully allocated size: 2199359062016

matches "without bitmaps" section.

tests/qemu-iotests/190.out line 30

Quoted 1 line
tests/qemu-iotests/190.out +1 -0
+bitmaps size: 537198592

match expectations

tests/qemu-iotests/190.out line 32

Quoted 2 lines
tests/qemu-iotests/190.out +2 -0
+required size: 873005056
+fully allocated size: 2199896260608

matches "without bitmaps" section + expected

tests/qemu-iotests/190.out line 36

Quoted 4 lines
tests/qemu-iotests/190.out +4 -0
+
+expected bitmap 545259520
+{
+    "bitmaps": 545259520,

match expected

tests/qemu-iotests/190.out line 38

Quoted 2 lines
tests/qemu-iotests/190.out +2 -0
+    "required": 18874368,
+    "fully-allocated": 2199042129920

matches "without bitmaps" section.

tests/qemu-iotests/190.out line 43

Quoted 5 lines
tests/qemu-iotests/190.out +5 -0
+}
+{
+    "required": 564133888,
+    "fully-allocated": 2199587389440
+}

matches "without bitmaps" section + expected

Quoted 2 lines
tests/qemu-iotests/190.out +0 -0
  *** done
Signature · 2 lines
Best regards,
Vladimir
author
#24 lore raw

Re: [PATCH v4 7/9] qcow2: Expose bitmaps' size during measure

Quoted 22 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines · Eric Blake

It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data.  Report this value as an additional QMP field, present when measuring an existing image and output format that both support bitmaps.  Update iotest 178 and 190 to updated output, as well as new coverage in 190 demonstrating non-zero values made possible with the recently-added qemu-img bitmap command.

(quoted diff) +0 -0
@@ -616,6 +616,7 @@ Command description:
 

      required size: 524288
      fully allocated size: 1074069504
+    bitmaps: 0

    The ``required size`` is the file size of the new image.  It may be smaller than the virtual disk size if the image format supports compact representation.

docs/tools/qemu-img.rst +0 -0
@@ -625,6 +626,13 @@ Command description:
     occupy with the exception of internal snapshots, dirty bitmaps, vmstate data,

    and other advanced image format features.

Quoted 3 lines
Quoted 1 line

+  The ``bitmaps size`` is the additional size required if the

you called it "bitmaps" in example output above. Should it be consistent? Either "``bitmaps``" here, or "bitmaps size: 0" above?

"bitmaps size: 0" is better. Will fix the description above.

Quoted 15 lines
Quoted 15 lines

+++ b/qapi/block-core.json

qapi/block-core.json +0 -1
@@ -633,18 +633,23 @@
   # efficiently so file size may be smaller than virtual disk size.
   #
   # The values are upper bounds that are guaranteed to fit the new image file.
-# Subsequent modification, such as internal snapshot or bitmap 

creation, may

qapi/block-core.json +2 -1
-# require additional space and is not covered here.
+# Subsequent modification, such as internal snapshot or further bitmap
+# creation, may require additional space and is not covered here.

  #

(quoted diff) +1 -1
-# @required: Size required for a new image file, in bytes.
+# @required: Size required for a new image file, in bytes, when 

copying just
+#            guest-visible contents.

Quoted 8 lines
Quoted 5 lines

  #
  # @fully-allocated: Image file size, in bytes, once data has been written

(quoted diff) +1 -1
-#                   to all sectors.
+#                   to all sectors, when copying just guest-visible 

contents.

"copying just guest-visible" sounds like something less than "all fully-allocated sectors".. But I don't have better suggestion.. Just, "not including bitmaps" sounds weird too.

If we ever add support for copying internal snapshots, that would not be included either. Maybe "copying just allocated guest-visible contents" for @required, and no change to the wording for @fully-allocated.

Quoted 27 lines
Quoted 23 lines
(quoted diff) +16 -0
@@ -4796,13 +4797,38 @@ static BlockMeasureInfo 
 *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
 
 
+        FOR_EACH_DIRTY_BITMAP(in_bs, bm) {
+            if (bdrv_dirty_bitmap_get_persistence(bm)) {
+                const char *name = bdrv_dirty_bitmap_name(bm);
+                uint32_t granularity = 
 bdrv_dirty_bitmap_granularity(bm);
+                uint64_t bmbits = 
 DIV_ROUND_UP(bdrv_dirty_bitmap_size(bm),
+                                               granularity);
+                uint64_t bmclusters = DIV_ROUND_UP(DIV_ROUND_UP(bmbits,
+                                                                
 CHAR_BIT),
+                                                   cluster_size);
+
+                /* Assume the entire bitmap is allocated */
+                bitmaps_size += bmclusters * cluster_size;
+                /* Also reserve space for the bitmap table entries */
+                bitmaps_size += ROUND_UP(bmclusters * sizeof(uint64_t),
+                                         cluster_size);
+                /* And space for contribution to bitmap directory 
Quoted 3 lines

size */
+                bitmap_dir_size += ROUND_UP(strlen(name) + 24, +                                            sizeof(uint64_t));

Quoted 2 lines

Could we instead reuse code from qcow2_co_can_store_new_dirty_bitmap(), which calls calc_dir_entry_size() for this thing?
Possibly, make a function qcow2_measure_bitmaps in block/qcow2-bitmaps.c with this FOR_EACH? All details about qcow2 bitmap structures sounds better in block/qcow2-bitmaps.c

Could do. Sounds like I'm better off submitting a v5 for this patch, although I'll go ahead and stage 1-6 for pull request today to minimize future rebase churn.

Quoted 5 lines
Quoted 3 lines

+    info->has_bitmaps = version >= 3 && in_bs && +        bdrv_supports_persistent_dirty_bitmap(in_bs); +    info->bitmaps = bitmaps_size;

AFAIK, in QAPI, if has_<something> field is false, than <something> must be zero. Maybe, it's only about nested structured fields, not about simple numbers, but I think it's better keep bitmaps 0 in case when has_bitmaps is false.

During creation (including when parsing QMP from the user over the monitor), everything is indeed guaranteed to be zero-initialized. But we don't have any requirement that things remain zero-initialized even when has_FOO is false; at the same time, it's easy enough to make this code conditional.

Quoted 2 lines

Also, it seems a bit better to check version earlier, and don't do all the calculations, if we are not going to use them.. But it's a rare backward-compatibility case, I don't care.

I'll see how easy or hard it is for my v5 patch.

Quoted 4 lines
Quoted 4 lines
qemu-img.c +0 -0
@@ -5275,9 +5285,24 @@ static int img_measure(int argc, char **argv)
           goto out;
       }
 
Quoted 12 lines
Quoted 10 lines
qemu-img.c +3 -0
+    if (bitmaps) {
+        if (!info->has_bitmaps) {
+            error_report("no bitmaps measured, either source or 

destination "
+                         "format lacks bitmap support"); +            goto out;
+        } else {
+            info->required += info->bitmaps;
+            info->fully_allocated += info->bitmaps; +            info->has_bitmaps = false;

And here, I think better to zero info->bitmaps as well.

Here, the object is going to be subsequently freed; I'm less worried about wasting time doing local cleanups than I am about the earlier case about letting an object escape immediate scope in a different state than the usual preconditions.

Quoted 8 lines

tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 62

Quoted 5 lines

+$QEMU_IMG bitmap --add --granularity 512 -f qcow2 "$TEST_IMG" b1 +$QEMU_IMG bitmap --add -g 2M -f qcow2 "$TEST_IMG" b2

tests/qemu-iotests/190 +3 -0
+
+# No bitmap without a source
+$QEMU_IMG measure --bitmaps -O qcow2 --size 10M

should this be ored to  'echo "unexpected success"' as following failures?

Can't hurt.

Quoted 9 lines

tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 84

Quoted 6 lines
tests/qemu-iotests/190 +6 -0
+# Compute expected output:
+echo
+val2T=$((2*1024*1024*1024*1024))
+cluster=$((64*1024))
+b1clusters=$(( (val2T/512/8 + cluster - 1) / cluster ))
+b2clusters=$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster ))

comment on the following calculations won't hurt, at least something like
 "bitmap clusters + bitmap tables + bitmaps directory"

Sure.

Quoted 10 lines
Quoted 10 lines

+echo expected bitmap $((b1clusters * cluster +
+            (b1clusters * 8 + cluster - 1) / cluster * cluster + +            b2clusters * cluster +
+            (b2clusters * 8 + cluster - 1) / cluster * cluster + +            cluster))
+$QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG" +$QEMU_IMG measure --bitmaps -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG"
+

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#25 lore raw

Re: [PATCH v4 7/9] qcow2: Expose bitmaps' size during measure

Quoted 50 lines · Eric Blake
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines · Eric Blake

It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data.  Report this value as an additional QMP field, present when measuring an existing image and output format that both support bitmaps.  Update iotest 178 and 190 to updated output, as well as new coverage in 190 demonstrating non-zero values made possible with the recently-added qemu-img bitmap command.

(quoted diff) +0 -0
@@ -616,6 +616,7 @@ Command description:
 

      required size: 524288
      fully allocated size: 1074069504
+    bitmaps: 0

    The ``required size`` is the file size of the new image.  It may be smaller than the virtual disk size if the image format supports compact representation.

docs/tools/qemu-img.rst +0 -0
@@ -625,6 +626,13 @@ Command description:
     occupy with the exception of internal snapshots, dirty bitmaps, vmstate data,

    and other advanced image format features.

Quoted 3 lines
Quoted 1 line

+  The ``bitmaps size`` is the additional size required if the

you called it "bitmaps" in example output above. Should it be consistent? Either "``bitmaps``" here, or "bitmaps size: 0" above?

"bitmaps size: 0" is better. Will fix the description above.

Quoted 12 lines
Quoted 12 lines

+++ b/qapi/block-core.json

qapi/block-core.json +3 -3
@@ -633,18 +633,23 @@
   # efficiently so file size may be smaller than virtual disk size.
   #
   # The values are upper bounds that are guaranteed to fit the new image file.
-# Subsequent modification, such as internal snapshot or bitmap creation, may
-# require additional space and is not covered here.
+# Subsequent modification, such as internal snapshot or further bitmap
+# creation, may require additional space and is not covered here.
   #
-# @required: Size required for a new image file, in bytes.
+# @required: Size required for a new image file, in bytes, when copying just
Quoted 8 lines

qapi/block-core.json line 643

Quoted 5 lines
qapi/block-core.json +2 -1
+#            guest-visible contents.
   #
   # @fully-allocated: Image file size, in bytes, once data has been written
-#                   to all sectors.
+#                   to all sectors, when copying just guest-visible contents.

"copying just guest-visible" sounds like something less than "all fully-allocated sectors".. But I don't have better suggestion.. Just, "not including bitmaps" sounds weird too.

Quoted 1 line

If we ever add support for copying internal snapshots, that would not be included either.  Maybe "copying just allocated guest-visible contents" for @required, and no change to the wording for @fully-allocated.

Not sure is it better or not, so, I'm OK with any variant.

Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
block/qcow2.c +18 -0
@@ -4796,13 +4797,38 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,
 
 
+        FOR_EACH_DIRTY_BITMAP(in_bs, bm) {
+            if (bdrv_dirty_bitmap_get_persistence(bm)) {
+                const char *name = bdrv_dirty_bitmap_name(bm);
+                uint32_t granularity = bdrv_dirty_bitmap_granularity(bm);
+                uint64_t bmbits = DIV_ROUND_UP(bdrv_dirty_bitmap_size(bm),
+                                               granularity);
+                uint64_t bmclusters = DIV_ROUND_UP(DIV_ROUND_UP(bmbits,
+ CHAR_BIT),
+                                                   cluster_size);
+
+                /* Assume the entire bitmap is allocated */
+                bitmaps_size += bmclusters * cluster_size;
+                /* Also reserve space for the bitmap table entries */
+                bitmaps_size += ROUND_UP(bmclusters * sizeof(uint64_t),
+                                         cluster_size);
+                /* And space for contribution to bitmap directory size */
+                bitmap_dir_size += ROUND_UP(strlen(name) + 24,
+                                            sizeof(uint64_t));
Quoted 4 lines
Quoted 2 lines

Could we instead reuse code from qcow2_co_can_store_new_dirty_bitmap(), which calls calc_dir_entry_size() for this thing?
Possibly, make a function qcow2_measure_bitmaps in block/qcow2-bitmaps.c with this FOR_EACH? All details about qcow2 bitmap structures sounds better in block/qcow2-bitmaps.c

Could do.  Sounds like I'm better off submitting a v5 for this patch, although I'll go ahead and stage 1-6 for pull request today to minimize future rebase churn.

Thanks!

Quoted 69 lines
Quoted 7 lines

block/qcow2.c static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, · line 4892

Quoted 5 lines
block/qcow2.c +3 -0
+    info->has_bitmaps = version >= 3 && in_bs &&
+        bdrv_supports_persistent_dirty_bitmap(in_bs);
+    info->bitmaps = bitmaps_size;

AFAIK, in QAPI, if has_<something> field is false, than <something> must be zero. Maybe, it's only about nested structured fields, not about simple numbers, but I think it's better keep bitmaps 0 in case when has_bitmaps is false.

During creation (including when parsing QMP from the user over the monitor), everything is indeed guaranteed to be zero-initialized.  But we don't have any requirement that things remain zero-initialized even when has_FOO is false; at the same time, it's easy enough to make this code conditional.

Quoted 2 lines

Also, it seems a bit better to check version earlier, and don't do all the calculations, if we are not going to use them.. But it's a rare backward-compatibility case, I don't care.

I'll see how easy or hard it is for my v5 patch.

Quoted 4 lines
Quoted 4 lines
qemu-img.c +0 -0
@@ -5275,9 +5285,24 @@ static int img_measure(int argc, char **argv)
           goto out;
       }
 
Quoted 11 lines

qemu-img.c static int img_measure(int argc, char **argv) · line 5296

Quoted 9 lines
qemu-img.c +9 -0
+    if (bitmaps) {
+        if (!info->has_bitmaps) {
+            error_report("no bitmaps measured, either source or destination "
+                         "format lacks bitmap support");
+            goto out;
+        } else {
+            info->required += info->bitmaps;
+            info->fully_allocated += info->bitmaps;
+            info->has_bitmaps = false;

And here, I think better to zero info->bitmaps as well.

Here, the object is going to be subsequently freed; I'm less worried about wasting time doing local cleanups than I am about the earlier case about letting an object escape immediate scope in a different state than the usual preconditions.

Quoted 8 lines

tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 62

Quoted 5 lines
tests/qemu-iotests/190 +5 -0
+$QEMU_IMG bitmap --add --granularity 512 -f qcow2 "$TEST_IMG" b1
+$QEMU_IMG bitmap --add -g 2M -f qcow2 "$TEST_IMG" b2
+
+# No bitmap without a source
+$QEMU_IMG measure --bitmaps -O qcow2 --size 10M

should this be ored to  'echo "unexpected success"' as following failures?

Can't hurt.

Quoted 9 lines

tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 84

Quoted 6 lines
tests/qemu-iotests/190 +6 -0
+# Compute expected output:
+echo
+val2T=$((2*1024*1024*1024*1024))
+cluster=$((64*1024))
+b1clusters=$(( (val2T/512/8 + cluster - 1) / cluster ))
+b2clusters=$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster ))

comment on the following calculations won't hurt, at least something like "bitmap clusters + bitmap tables + bitmaps directory"

Sure.

Quoted 9 lines
Quoted 9 lines
tests/qemu-iotests/190 +8 -0
+echo expected bitmap $((b1clusters * cluster +
+            (b1clusters * 8 + cluster - 1) / cluster * cluster +
+            b2clusters * cluster +
+            (b2clusters * 8 + cluster - 1) / cluster * cluster +
+            cluster))
+$QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG"
+$QEMU_IMG measure --bitmaps -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG"
+
Signature · 2 lines
Best regards,
Vladimir
author
#26 lore raw

PATCH v4 8/9 qemu-img: Add convert --bitmaps option

In reply to Eric Blake

Make it easier to copy all the persistent bitmaps of (the top layer of) a source image along with its guest-visible contents, by adding a boolean flag for use with qemu-img convert. This is basically shorthand, as the same effect could be accomplished with a series of 'qemu-img bitmap --add' and 'qemu-img bitmap --merge -b source' commands, or by QMP commands.

Note that this command will fail in the same scenarios where 'qemu-img measure --bitmaps' fails, when either the source or the destanation lacks persistent bitmap support altogether.

See also https://bugzilla.redhat.com/show_bug.cgi?id=1779893

While touching this, clean up a couple coding issues spotted in the same function: an extra blank line, and merging back-to-back 'if (!skip_create)' blocks.

---
 docs/tools/qemu-img.rst |  6 ++-
 qemu-img.c              | 85 +++++++++++++++++++++++++++++++++++++++--
 qemu-img-cmds.hx        |  4 +-
 3 files changed, 89 insertions(+), 6 deletions(-)
docs/tools/qemu-img.rst +5 -1
index b6f87ec6d3c0..25156f489696 100644
@@ -162,6 +162,10 @@ Parameters to convert subcommand:
 
 .. program:: qemu-img-convert
 
+.. option:: --bitmaps
+
+  Additionally copy all persistent bitmaps from the top layer of the source
+
 .. option:: -n
 
   Skip the creation of the target volume
@@ -397,7 +401,7 @@ Command description:
   4
     Error on reading data
 
-.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
+.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
 
   Convert the disk image *FILENAME* or a snapshot *SNAPSHOT_PARAM*
   to disk image *OUTPUT_FILENAME* using format *OUTPUT_FMT*. It can
qemu-img.c +82 -3

146 lines not on this page · raw message

qemu-img-cmds.hx +2 -2
index bfcd9f32dddf..2791c4f58ddd 100644
@@ -46,9 +46,9 @@ SRST
 ERST
 
 DEF("convert", img_convert,
-    "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
+    "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
 SRST
-.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
+.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
 ERST
 
 DEF("create", img_create,
Signature · 1 line
2.26.2
#27 lore raw

Re: [PATCH v4 8/9] qemu-img: Add convert --bitmaps option

Quoted 24 lines · Eric Blake

Make it easier to copy all the persistent bitmaps of (the top layer of) a source image along with its guest-visible contents, by adding a boolean flag for use with qemu-img convert. This is basically shorthand, as the same effect could be accomplished with a series of 'qemu-img bitmap --add' and 'qemu-img bitmap --merge -b source' commands, or by QMP commands.

Note that this command will fail in the same scenarios where 'qemu-img measure --bitmaps' fails, when either the source or the destanation lacks persistent bitmap support altogether.

See also https://bugzilla.redhat.com/show_bug.cgi?id=1779893

While touching this, clean up a couple coding issues spotted in the same function: an extra blank line, and merging back-to-back 'if (!skip_create)' blocks.

---
  docs/tools/qemu-img.rst |  6 ++-
  qemu-img.c              | 85 +++++++++++++++++++++++++++++++++++++++--
  qemu-img-cmds.hx        |  4 +-
  3 files changed, 89 insertions(+), 6 deletions(-)

[..]

Quoted 23 lines
qemu-img.c +22 -0
+static int convert_copy_bitmaps(BlockDriverState *src, BlockDriverState *dst)
+{
+    BdrvDirtyBitmap *bm;
+    Error *err = NULL;
+    BlockDirtyBitmapMergeSource *merge;
+    BlockDirtyBitmapMergeSourceList *list;
+
+    FOR_EACH_DIRTY_BITMAP(src, bm) {
+        const char *name;
+
+        if (!bdrv_dirty_bitmap_get_persistence(bm)) {
+            continue;
+        }
+        name = bdrv_dirty_bitmap_name(bm);
+        qmp_block_dirty_bitmap_add(dst->node_name, name,
+                                   true, bdrv_dirty_bitmap_granularity(bm),
+                                   true, true,
+                                   true, !bdrv_dirty_bitmap_enabled(bm),
+                                   &err);
+        if (err) {
+            error_reportf_err(err, "Failed to create bitmap %s: ", name);
+            return -1;

qemu-img.c static int convert_do_copy(ImgConvertState *s) · line 2144

Quoted 10 lines
qemu-img.c +10 -0
+        }
+
+        merge = g_new0(BlockDirtyBitmapMergeSource, 1);
+        merge->type = QTYPE_QDICT;
+        merge->u.external.node = g_strdup(src->node_name);
+        merge->u.external.name = g_strdup(name);
+        list = g_new0(BlockDirtyBitmapMergeSourceList, 1);
+        list->value = merge;
+        qmp_block_dirty_bitmap_merge(dst->node_name, name, list, &err);
+        qapi_free_BlockDirtyBitmapMergeSourceList(list);

^ duplicated code chunk with "case BITMAP_MERGE". Small helper function will not hurt.

Quoted 57 lines
qemu-img.c +21 -1
+        if (err) {
+            error_reportf_err(err, "Failed to populate bitmap %s: ", name);
+            return -1;
+        }
+    }
+
+    return 0;
+}
+
  #define MAX_BUF_SECTORS 32768
 
  static int img_convert(int argc, char **argv)
@@ -2130,6 +2172,8 @@ static int img_convert(int argc, char **argv)
      int64_t ret = -EINVAL;
      bool force_share = false;
      bool explict_min_sparse = false;
+    bool bitmaps = false;
+    size_t nbitmaps = 0;
 
      ImgConvertState s = (ImgConvertState) {
          /* Need at least 4k of zeros for sparse detection */
@@ -2149,6 +2193,7 @@ static int img_convert(int argc, char **argv)
              {"target-image-opts", no_argument, 0, OPTION_TARGET_IMAGE_OPTS},
              {"salvage", no_argument, 0, OPTION_SALVAGE},
              {"target-is-zero", no_argument, 0, OPTION_TARGET_IS_ZERO},
+            {"bitmaps", no_argument, 0, OPTION_BITMAPS},
              {0, 0, 0, 0}
          };
          c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU",
@@ -2272,6 +2317,9 @@ static int img_convert(int argc, char **argv)
               */
              s.has_zero_init = true;
              break;
+        case OPTION_BITMAPS:
+            bitmaps = true;
+            break;
          }
      }
 
@@ -2333,7 +2381,6 @@ static int img_convert(int argc, char **argv)
          goto fail_getopt;
      }
 
-
      /* ret is still -EINVAL until here */
      ret = bdrv_parse_cache_mode(src_cache, &src_flags, &src_writethrough);
      if (ret < 0) {
@@ -2493,6 +2540,27 @@ static int img_convert(int argc, char **argv)
          }
      }
 
+    /* Determine how many bitmaps need copying */
+    if (bitmaps) {
+        BdrvDirtyBitmap *bm;
+
+        if (s.src_num > 1) {
+            error_report("Copying bitmaps only possible with single source");

qemu-img.c static int img_convert(int argc, char **argv) · line 2558

Quoted 10 lines
qemu-img.c +10 -0
+            ret = -1;
+            goto out;
+        }
+        if (!bdrv_supports_persistent_dirty_bitmap(blk_bs(s.src[0]))) {
+            error_report("Source lacks bitmap support");
+            ret = -1;
+            goto out;
+        }
+        FOR_EACH_DIRTY_BITMAP(blk_bs(s.src[0]), bm) {
+            if (bdrv_dirty_bitmap_get_persistence(bm)) {

Just note, that we can't have not persistent bitmaps here.. But the check doesn't hurt.

Quoted 62 lines
qemu-img.c +18 -2
+                nbitmaps++;
+            }
+        }
+    }
+
      /*
       * The later open call will need any decryption secrets, and
       * bdrv_create() will purge "opts", so extract them now before
@@ -2501,9 +2569,7 @@ static int img_convert(int argc, char **argv)
      if (!skip_create) {
          open_opts = qdict_new();
          qemu_opt_foreach(opts, img_add_key_secrets, open_opts, &error_abort);
-    }
 
-    if (!skip_create) {
          /* Create the new image */
          ret = bdrv_create(drv, out_filename, opts, &local_err);
          if (ret < 0) {
@@ -2541,6 +2607,13 @@ static int img_convert(int argc, char **argv)
      }
      out_bs = blk_bs(s.target);
 
+    if (nbitmaps > 0 && !bdrv_supports_persistent_dirty_bitmap(out_bs)) {
+        error_report("Format driver '%s' does not support bitmaps",
+                     out_fmt);
+        ret = -1;
+        goto out;
+    }
+
      if (s.compressed && !block_driver_can_compress(out_bs->drv)) {
          error_report("Compression not supported for this file format");
          ret = -1;
@@ -2600,6 +2673,12 @@ static int img_convert(int argc, char **argv)
      }
 
      ret = convert_do_copy(&s);
+
+    /* Now copy the bitmaps */
+    if (nbitmaps > 0 && ret == 0) {
+        ret = convert_copy_bitmaps(blk_bs(s.src[0]), out_bs);
+    }
+
  out:
      if (!ret) {
          qemu_progress_print(100, 0);
qemu-img-cmds.hx +2 -2
index bfcd9f32dddf..2791c4f58ddd 100644
@@ -46,9 +46,9 @@ SRST
  ERST
 
  DEF("convert", img_convert,
-    "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
+    "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
  SRST
-.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
+.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
  ERST
 
  DEF("create", img_create,
Signature · 2 lines
Best regards,
Vladimir
author
#28 lore raw

PATCH v4 9/9 iotests: Add test 291 to for qemu-img bitmap coverage

In reply to Eric Blake

Add a new test covering the 'qemu-img bitmap' subcommand, as well as 'qemu-img convert --bitmaps', both added in recent patches.

---
 tests/qemu-iotests/291     | 112 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/291.out |  78 ++++++++++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 191 insertions(+)
 create mode 100755 tests/qemu-iotests/291
 create mode 100644 tests/qemu-iotests/291.out
tests/qemu-iotests/291 new file +112 -0
new file mode 100755
index 000000000000..3ca83b9cd1f7
@@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+#
+# Test qemu-img bitmap handling
+#
+# Copyright (C) 2018-2020 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+    nbd_server_stop
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.nbd
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+_require_command QEMU_NBD
+
+echo
+echo "=== Initial image setup ==="
+echo
+
+# Create backing image with one bitmap
+TEST_IMG="$TEST_IMG.base" _make_test_img 10M
+$QEMU_IMG bitmap --add -f $IMGFMT "$TEST_IMG.base" b0
+$QEMU_IO -c 'w 3M 1M' -f $IMGFMT "$TEST_IMG.base" | _filter_qemu_io
+
+# Create initial image and populate two bitmaps: one active, one inactive.
+ORIG_IMG=$TEST_IMG
+TEST_IMG=$TEST_IMG.orig
+_make_test_img -b "$ORIG_IMG.base" -F $IMGFMT 10M
+$QEMU_IO -c 'w 0 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
+$QEMU_IMG bitmap --add -g 512k -f $IMGFMT "$TEST_IMG" b1
+$QEMU_IMG bitmap --add --disable -f $IMGFMT "$TEST_IMG" b2
+$QEMU_IO -c 'w 3M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
+$QEMU_IMG bitmap --clear -f $IMGFMT "$TEST_IMG" b1
+$QEMU_IO -c 'w 1M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
+$QEMU_IMG bitmap --disable -f $IMGFMT "$TEST_IMG" b1
+$QEMU_IMG bitmap --enable -f $IMGFMT "$TEST_IMG" b2
+$QEMU_IO -c 'w 2M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io
+
+echo
+echo "=== Bitmap preservation not possible to non-qcow2 ==="
+echo
+
+TEST_IMG=$ORIG_IMG
+$QEMU_IMG convert --bitmaps -O raw "$TEST_IMG.orig" "$TEST_IMG" &&
+    echo "unexpected success"
+
+echo
+echo "=== Convert with bitmap preservation ==="
+echo
+
+# Only bitmaps from the active layer are copied
+$QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG.orig" "$TEST_IMG"
+$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific
+# But we can also merge in bitmaps from other layers.  This test is a bit
+# contrived to cover more code paths, in reality, you could merge directly
+# into b0 without going through tmp
+$QEMU_IMG bitmap --add --disable -f $IMGFMT "$TEST_IMG" b0
+$QEMU_IMG bitmap --add --merge b0 -b "$TEST_IMG.base" -F $IMGFMT \
+     -f $IMGFMT "$TEST_IMG" tmp
+$QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0
+$QEMU_IMG bitmap --remove --image-opts \
+    driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp
+$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific
+
+echo
+echo "=== Check bitmap contents ==="
+echo
+
+# x-dirty-bitmap is a hack for reading bitmaps; it abuses block status to
+# report "data":false for portions of the bitmap which are set
+IMG="driver=nbd,server.type=unix,server.path=$nbd_unix_socket"
+nbd_server_start_unix_socket -r -f qcow2 -B b0 "$TEST_IMG"
+$QEMU_IMG map --output=json --image-opts \
+    "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b0" | _filter_qemu_img_map
+nbd_server_start_unix_socket -r -f qcow2 -B b1 "$TEST_IMG"
+$QEMU_IMG map --output=json --image-opts \
+    "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b1" | _filter_qemu_img_map
+nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG"
+$QEMU_IMG map --output=json --image-opts \
+    "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map
+
+# success, all done
+echo '*** done'
+rm -f $seq.full
+status=0
tests/qemu-iotests/291.out new file +78 -0

78 lines not on this page · raw message

tests/qemu-iotests/group +1 -0
index fe649c5b736e..206a23292688 100644
@@ -298,4 +298,5 @@
 288 quick
 289 rw quick
 290 rw auto quick
+291 rw quick
 292 rw auto quick
Signature · 1 line
2.26.2
Reviewed-by on v4 9/9
#29 lore raw

Re: [PATCH v4 9/9] iotests: Add test 291 to for qemu-img bitmap coverage

Quoted 5 lines · Eric Blake

Add a new test covering the 'qemu-img bitmap' subcommand, as well as 'qemu-img convert --bitmaps', both added in recent patches.

Quoted 17 lines
---
  tests/qemu-iotests/291     | 112 +++++++++++++++++++++++++++++++++++++
  tests/qemu-iotests/291.out |  78 ++++++++++++++++++++++++++
  tests/qemu-iotests/group   |   1 +
  3 files changed, 191 insertions(+)

create mode 100755 tests/qemu-iotests/291
create mode 100644 tests/qemu-iotests/291.out

tests/qemu-iotests/291 new file +3 -0
new file mode 100755
index 000000000000..3ca83b9cd1f7
@@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+#
+# Test qemu-img bitmap handling

[..]

Quoted 3 lines

+TEST_IMG="$TEST_IMG.base" _make_test_img 10M
+$QEMU_IMG bitmap --add -f $IMGFMT "$TEST_IMG.base" b0 +$QEMU_IO -c 'w 3M 1M' -f $IMGFMT "$TEST_IMG.base" | _filter_qemu_io

Actually '-f $IMGFMT' option is already inside $QEMU_IO. Still it's a kind of standard for iotests to duplicate -f option, so I don't care. I hope one day we'll restrict it in qemu-io interface :)

Signature · 2 lines
Best regards,
Vladimir