Skip to contentSkip to conversation

qemu-img: Add convert --bitmaps

PATCH Superseded Eric Blake 2 participants 6 messages Apr 16, 2020 Kbuild

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

cover qemu-img: Add convert --bitmaps

Without this series, the process for copying one qcow2 image to another including all of its bitmaps involves running qemu and doing the copying by hand with a series of QMP commands. This makes the process a bit more convenient.

I still think that someday we will need a 'qemu-img bitmap' with various subcommands for manipulating bitmaps within an offline image, but in the meantime, this seems like a useful addition on its own.

Series can also be downloaded at:
https://repo.or.cz/qemu/ericb.git/shortlog/refs/tags/qemu-img-bitmaps-v1

Eric Blake (3):
blockdev: Split off basic bitmap operations for qemu-img
qemu-img: Add convert --bitmaps option
iotests: Add test 291 to for qemu-img convert --bitmaps

 docs/tools/qemu-img.rst    |   6 +-
 Makefile.objs              |   2 +-
 include/sysemu/blockdev.h  |  10 ++
 blockbitmaps.c             | 217 +++++++++++++++++++++++++++++++++++++
 blockdev.c                 | 184 -------------------------------
 qemu-img.c                 |  81 +++++++++++++-
 MAINTAINERS                |   1 +
 qemu-img-cmds.hx           |   4 +-
 tests/qemu-iotests/291     | 143 ++++++++++++++++++++++++
 tests/qemu-iotests/291.out |  56 ++++++++++
 tests/qemu-iotests/group   |   1 +
 11 files changed, 514 insertions(+), 191 deletions(-)
 create mode 100644 blockbitmaps.c
 create mode 100755 tests/qemu-iotests/291
 create mode 100644 tests/qemu-iotests/291.out
Signature · 1 line
2.26.0
author
#1 lore raw

PATCH 1/3 blockdev: Split off basic bitmap operations for qemu-img

The next patch wants to teach qemu how to copy a bitmap from one qcow2 file to another. But blockdev.o is too heavyweight to link into qemu-img, so it's time to split off the bare bones of what we will need into a new file blockbitmaps.o. Transactions are not needed in qemu-img (if things fail while creating the new image, the fix is to delete the botched copy, rather than worrying about atomic rollback).

For now, I stuck to just the minimum code motion (add and merge); we could instead decide to move everything bitmap-related that does not also pull in transactions (delete, enable, disable).

---
 Makefile.objs             |   2 +-
 include/sysemu/blockdev.h |  10 ++
 blockbitmaps.c            | 217 ++++++++++++++++++++++++++++++++++++++
 blockdev.c                | 184 --------------------------------
 MAINTAINERS               |   1 +
 5 files changed, 229 insertions(+), 185 deletions(-)
 create mode 100644 blockbitmaps.c
Makefile.objs +1 -1
index a7c967633acf..44e30fa9a6e3 100644
@@ -14,7 +14,7 @@ chardev-obj-y = chardev/
 authz-obj-y = authz/
 
 block-obj-y = nbd/
-block-obj-y += block.o blockjob.o job.o
+block-obj-y += block.o blockbitmaps.o blockjob.o job.o
 block-obj-y += block/ scsi/
 block-obj-y += qemu-io-cmds.o
 block-obj-$(CONFIG_REPLICATION) += replication.o
include/sysemu/blockdev.h +10 -0
index a86d99b3d875..95cfeb29bc0a 100644
@@ -57,4 +57,14 @@ QemuOpts *drive_add(BlockInterfaceType type, int index, const char *file,
 DriveInfo *drive_new(QemuOpts *arg, BlockInterfaceType block_default_type,
                      Error **errp);
 
+BdrvDirtyBitmap *block_dirty_bitmap_lookup(const char *node,
+                                           const char *name,
+                                           BlockDriverState **pbs,
+                                           Error **errp);
+BdrvDirtyBitmap *do_block_dirty_bitmap_merge(const char *node,
+                                             const char *target,
+                                             BlockDirtyBitmapMergeSourceList *bitmaps,
+                                             HBitmap **backup, Error **errp);
+
+
 #endif
blockbitmaps.c new file +217 -0

217 lines not on this page · raw message

blockdev.c +0 -184

208 lines not on this page · raw message

MAINTAINERS +1 -0
index 8cbc1fac2bfc..769cd357d281 100644
@@ -1989,6 +1989,7 @@ T: git https://github.com/jnsnow/qemu.git jobs
 Block QAPI, monitor, command line
 M: Markus Armbruster <[email protected]>
 S: Supported
+F: blockbitmaps.c
 F: blockdev.c
 F: blockdev-hmp-cmds.c
 F: block/qapi.c
Signature · 1 line
2.26.0
author
#2 lore raw

PATCH 2/3 qemu-img: Add convert --bitmaps option

In reply to Eric Blake

Make it easier to copy all the persistent bitmaps of a source image along with the contents, by adding a boolean flag for use with qemu-img convert.

See also https://bugzilla.redhat.com/show_bug.cgi?id=1779893

---
 docs/tools/qemu-img.rst |  6 ++-
 qemu-img.c              | 81 +++++++++++++++++++++++++++++++++++++++--
 qemu-img-cmds.hx        |  4 +-
 3 files changed, 85 insertions(+), 6 deletions(-)
docs/tools/qemu-img.rst +5 -1
index 0080f83a76c9..8c4d85e0b835 100644
@@ -186,6 +186,10 @@ Parameters to convert subcommand:
 
 .. program:: qemu-img-convert
 
+.. option:: --bitmaps
+
+  Additionally copy all bitmaps
+
 .. option:: -n
 
   Skip the creation of the target volume
@@ -373,7 +377,7 @@ Command description:
   4
     Error on reading data
 
-.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
+.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
 
   Convert the disk image *FILENAME* or a snapshot *SNAPSHOT_PARAM*
   to disk image *OUTPUT_FILENAME* using format *OUTPUT_FMT*. It can
qemu-img.c +78 -3

148 lines not on this page · raw message

qemu-img-cmds.hx +2 -2
index c9c54de1df40..37cb36335218 100644
@@ -39,9 +39,9 @@ SRST
 ERST
 
 DEF("convert", img_convert,
-    "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
+    "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
 SRST
-.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
+.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
 ERST
 
 DEF("create", img_create,
Signature · 1 line
2.26.0
author
#3 lore raw

PATCH 3/3 iotests: Add test 291 to for qemu-img convert --bitmaps

In reply to Eric Blake

Add a new test covering the feature added in the previous patch.

---
 tests/qemu-iotests/291     | 143 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/291.out |  56 +++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 200 insertions(+)
 create mode 100755 tests/qemu-iotests/291
 create mode 100644 tests/qemu-iotests/291.out
tests/qemu-iotests/291 new file +143 -0
new file mode 100755
index 000000000000..dfdcc8e352c8
@@ -0,0 +1,143 @@
+#!/usr/bin/env bash
+#
+# Test qemu-img convert --bitmaps
+#
+# Copyright (C) 2018-2020 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+    nbd_server_stop
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.nbd
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+_require_command QEMU_NBD
+
+do_run_qemu()
+{
+    echo Testing: "$@"
+    $QEMU -nographic -qmp stdio -serial none "$@"
+    echo
+}
+
+run_qemu()
+{
+    do_run_qemu "$@" 2>&1 | _filter_testdir | _filter_qmp \
+                          | _filter_qemu | _filter_imgfmt \
+                          | _filter_actual_image_size | _filter_qemu_io
+}
+
+# Create initial image and populate two bitmaps: one active, one inactive
+_make_test_img 10M
+run_qemu <<EOF
+{ "execute": "qmp_capabilities" }
+{ "execute": "blockdev-add",
+  "arguments": {
+    "driver": "$IMGFMT",
+    "node-name": "n",
+    "file": {
+      "driver": "file",
+      "filename": "$TEST_IMG"
+    }
+  }
+}
+{ "execute": "human-monitor-command",
+    "arguments": {
+        "command-line": 'qemu-io n "write 0 1M"'
+    }
+}
+{ "execute": "block-dirty-bitmap-add",
+  "arguments": {
+    "node": "n",
+    "name": "b1",
+    "persistent": true,
+    "granularity": 65536
+  }
+}
+{ "execute": "human-monitor-command",
+    "arguments": {
+        "command-line": 'qemu-io n "write 1M 1M"'
+    }
+}
+{ "execute": "transaction",
+  "arguments": {
+    "actions": [
+      { "type": "block-dirty-bitmap-disable",
+        "data": {
+          "node": "n",
+          "name": "b1"
+        } },
+      { "type": "block-dirty-bitmap-add",
+        "data": {
+          "node": "n",
+          "name": "b2",
+          "persistent": true
+       } }
+    ]
+  }
+}
+{ "execute": "human-monitor-command",
+    "arguments": {
+        "command-line": 'qemu-io n "write 2M 1M"'
+    }
+}
+{ "execute": "quit" }
+EOF
+
+echo
+echo "=== Bitmap preservation not possible to non-qcow2 ==="
+echo
+
+mv "$TEST_IMG" "$TEST_IMG.orig"
+$QEMU_IMG convert --bitmaps -O raw "$TEST_IMG.orig" "$TEST_IMG"
+
+echo
+echo "=== convert with bitmap preservation ==="
+echo
+
+$QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG.orig" "$TEST_IMG"
+$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific
+
+echo
+echo "=== check bitmap contents ==="
+echo
+
+IMG="driver=nbd,server.type=unix,server.path=$nbd_unix_socket"
+nbd_server_start_unix_socket -r -f qcow2 -B b1 "$TEST_IMG"
+$QEMU_IMG map --output=json --image-opts \
+    "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b1" | _filter_qemu_img_map
+nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG"
+$QEMU_IMG map --output=json --image-opts \
+    "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map
+
+# success, all done
+echo '*** done'
+rm -f $seq.full
+status=0
tests/qemu-iotests/291.out new file +56 -0

56 lines not on this page · raw message

tests/qemu-iotests/group +1 -0
index 435dccd5af90..8e9b9513a091 100644
@@ -297,3 +297,4 @@
 288 quick
 289 rw quick
 290 rw auto quick
+291 rw quick
Signature · 1 line
2.26.0
#4 lore raw
In reply to Eric Blake
Quoted 5 lines · Eric Blake

Without this series, the process for copying one qcow2 image to another including all of its bitmaps involves running qemu and doing the copying by hand with a series of QMP commands. This makes the process a bit more convenient.

This seems good for copying an image chain from one storage to another, but I think we need a similar --bitmaps option to qemu-img measure to make this really useful.

Here is example use case showing how qemu-img measure is related:

Source chain:
/dev/vg1/base
/dev/vg1/top

Destination chain:
/dev/vg2/base
/dev/vg2/top

We create empty lvs with the same name on destination storage (/dev/vg2).

We measure the base lv using qemu-img measure for creating the target lv:

qemu-img measure -f qcow2 -O qcow2 /dev/vg1/base lvcreate -L required_size /dev/vg2/base qemu-img create -f qcow2 /dev/vg2/base 10g

For the top lv we use the current size of the source lv - I think we should measure it instead but
I'm not sure if qemu-img measure supports measuring a single image in a chain (maybe -o backing_file?).

lvcreate -L current_size /dev/vg2/top
qemu-img create -f qcow2 -b /dev/vg2/base -F qcow2 /dev/vg2/top 10g

And then convert the lvs one by one:

    qemu-img convert -f qcow2 -O qcow2 -n --bitmaps /dev/vg1/base /dev/vg2/base
    qemu-img convert -f qcow2 -O qcow2 -n --bitmaps -B /dev/vg2/base

/dev/vg1/top /dev/vg2/top

The first copy may fail with ENOSPC since qemu-img measure of the base does not consider the
bitmaps in the required size.

So I think we need to add a similar --bitmaps option to qemu-img measure, hopefully reusing the
same code to find and estimate the size of the bitmaps.

Maybe we can estimate the size using qemu-img info --bitmaps, but I think the right way to
do this is in qemu-img measure.

We have also another use case when we collapsed an image chain to single image:

Source chain:
/dev/vg1/base
/dev/vg1/top

Destination:
/dev/vg2/collapsed

In this case we measure the size of the entire chain (/dev/vg1/base <- /dev/vg1/top) and create
/dev/vg2/collapsed in the correct size, and then we convert the chain using:

qemu-img convert /dev/vg1/top /dev/vg2/collapsed

Currently we use this for exporting images, for example when creating templates, or as a simple
backup. In this case we don't need to copy the bitmaps in the target image - this is a new image
not used by any VM. Copying the bitmaps may also be non-trivial since we may have the bitmaps
with the same names in several layers (e.g. result of live snapshot).

So I think using --bitmaps should be disabled when doing this kind of convert. We can handle this
on our side easily, but I think this should fail or log a warning on qemu-img, or require merging of
bitmaps with same names during the copy. I did not check if you already handle this.

Finally we also have a use case when we copy the chain as is to new or same storage, but
we create a new vm. In this case I don't think the backup history makes sense for the new
vm, so we don't need to copy the bitmaps.

I will review the rest of the patches next week and can maybe give this some testing.

Nir

Quoted 31 lines

I still think that someday we will need a 'qemu-img bitmap' with various subcommands for manipulating bitmaps within an offline image, but in the meantime, this seems like a useful addition on its own.

Series can also be downloaded at:
https://repo.or.cz/qemu/ericb.git/shortlog/refs/tags/qemu-img-bitmaps-v1

Eric Blake (3):
blockdev: Split off basic bitmap operations for qemu-img
qemu-img: Add convert --bitmaps option
iotests: Add test 291 to for qemu-img convert --bitmaps

 docs/tools/qemu-img.rst    |   6 +-
 Makefile.objs              |   2 +-
 include/sysemu/blockdev.h  |  10 ++
 blockbitmaps.c             | 217 +++++++++++++++++++++++++++++++++++++
 blockdev.c                 | 184 -------------------------------
 qemu-img.c                 |  81 +++++++++++++-
 MAINTAINERS                |   1 +
 qemu-img-cmds.hx           |   4 +-
 tests/qemu-iotests/291     | 143 ++++++++++++++++++++++++
 tests/qemu-iotests/291.out |  56 ++++++++++
 tests/qemu-iotests/group   |   1 +
 11 files changed, 514 insertions(+), 191 deletions(-)
 create mode 100644 blockbitmaps.c
 create mode 100755 tests/qemu-iotests/291
 create mode 100644 tests/qemu-iotests/291.out

--
2.26.0

author
#5 lore raw

(adding Markus for a CLI question, look for [*])

Quoted 25 lines · Nir Soffer
Quoted 5 lines · Eric Blake

Without this series, the process for copying one qcow2 image to another including all of its bitmaps involves running qemu and doing the copying by hand with a series of QMP commands. This makes the process a bit more convenient.

This seems good for copying an image chain from one storage to another, but I think we need a similar --bitmaps option to qemu-img measure to make this really useful.

Here is example use case showing how qemu-img measure is related:

Source chain:
/dev/vg1/base
/dev/vg1/top

Destination chain:
/dev/vg2/base
/dev/vg2/top

We create empty lvs with the same name on destination storage (/dev/vg2).

We measure the base lv using qemu-img measure for creating the target lv:

Quoted 8 lines

qemu-img measure -f qcow2 -O qcow2 /dev/vg1/base lvcreate -L required_size /dev/vg2/base qemu-img create -f qcow2 /dev/vg2/base 10g

For the top lv we use the current size of the source lv - I think we should measure it instead but
I'm not sure if qemu-img measure supports measuring a single image in a chain (maybe -o backing_file?).

qemu-measure --image-opts should be able to measure a single image by specifying image opts that purposefully treat the image as standalone rather than with its normal backing file included. Let's see if I can whip up an example:

$ qemu-img create -f qcow2 img.base 100M
Formatting 'img.base', fmt=qcow2 size=104857600 cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-io -f qcow2 -c 'w 0 25m' img.base
wrote 26214400/26214400 bytes at offset 0
25 MiB, 1 ops; 00.24 sec (103.405 MiB/sec and 4.1362 ops/sec)
$ qemu-img create -f qcow2 -F qcow2 -b img.base img.top
Formatting 'img.top', fmt=qcow2 size=104857600 backing_file=img.base backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-io -f qcow2 -c 'w 25m 25m' img.top
wrote 26214400/26214400 bytes at offset 26214400
25 MiB, 1 ops; 00.24 sec (103.116 MiB/sec and 4.1247 ops/sec)
$ qemu-img measure -f qcow2 -O qcow2 img.base
required size: 26542080
fully allocated size: 105185280
required size: 52756480
fully allocated size: 105185280

Okay, I can reproduce what you are seeing - measuring the top image defaults to measuring the full allocation of the entire chain, rather than the allocation of just the top image. And now with --image-opts to the rescue:

$ qemu-img measure --image-opts -O qcow2 driver=qcow2,backing=,file.driver=file,file.filename=img.top
qemu-img: warning: Use of "backing": "" is deprecated; use "backing": null instead
required size: 26542080
fully allocated size: 105185280

There you go - by forcing qemu to treat the overlay image as though it had no backing, you can then measure that image in isolation.

(*) Hmm - that warning about backing="" being deprecated is annoying, but I don't know any other way to use dotted command line syntax and still express that we want a QMP null. I tried to see if I could inject an alternative backing driver, such as null-co, but was met with errors:

$ ./qemu-img measure --image-opts -O qcow2 driver=qcow2,backing.driver=null-co,file.driver=file,file.filename=img.top
qemu-img: Could not open 'driver=qcow2,backing.driver=null-co,file.driver=file,file.filename=img.top': Could not open backing file: The only allowed filename for this driver is 'null-co://'
$ ./qemu-img measure --image-opts -O qcow2 driver=qcow2,backing.driver=null-co,backing.file=null-co://,file.driver=file,file.filename=img.top
qemu-img: Could not open 'driver=qcow2,backing.driver=null-co,backing.file=null-co://,file.driver=file,file.filename=img.top': Could not open backing file: The only allowed filename for this driver is 'null-co://'
$ ./qemu-img measure --image-opts -O qcow2 driver=qcow2,backing.driver=null-co,backing.file.filename=null-co://,file.driver=file,file.filename=img.top
qemu-img: Could not open 'driver=qcow2,backing.driver=null-co,backing.file.filename=null-co://,file.driver=file,file.filename=img.top': Could not open backing file: Block protocol 'null-co' doesn't support the option 'file.filename'

We don't want to support "" in the QMP syntax forever, but if the CLI syntax has to handle the empty string specially in order to get null passed to the QMP code, then so be it.

I also tried, but failed, to use JSON syntax. I don't know why we haven't wired up --image-opts to use JSON syntax yet.

$ qemu-img measure --image-opts -O qcow2 '{"driver":"qcow2", "backing":null,
  "file":{"driver":"file", "filename":"img.top"}}'
qemu-img: Could not open '{"driver":"qcow2", "backing":null,
  "file":{"driver":"file", "filename":"img.top"}}': Cannot find 
device={"driver":"qcow2" nor node_name={"driver":"qcow2"

I guess there's always the pseudo-json protocol:

$ qemu-img measure -O qcow2 'json:{"driver":"qcow2", "backing":null,
  "file":{"driver":"file", "filename":"img.top"}}'
required size: 26542080
fully allocated size: 105185280
Quoted 6 lines

lvcreate -L current_size /dev/vg2/top
qemu-img create -f qcow2 -b /dev/vg2/base -F qcow2 /dev/vg2/top 10g

And then convert the lvs one by one:

Quoted 7 lines
     qemu-img convert -f qcow2 -O qcow2 -n --bitmaps /dev/vg1/base /dev/vg2/base
     qemu-img convert -f qcow2 -O qcow2 -n --bitmaps -B /dev/vg2/base

/dev/vg1/top /dev/vg2/top

The first copy may fail with ENOSPC since qemu-img measure of the base does not consider the
bitmaps in the required size.

Yes, that's a good argument for adding 'qemu-img measure --bitmaps'. In the meantime...

Quoted 6 lines

So I think we need to add a similar --bitmaps option to qemu-img measure, hopefully reusing the
same code to find and estimate the size of the bitmaps.

Maybe we can estimate the size using qemu-img info --bitmaps,

...you are correct, this works today. Well, 'qemu-img info --bitmaps' doesn't exist, but 'qemu-img info' does output the number of bitmaps, as well as their granularity, and if you also assume that each bitmap is sized to match the virtual image size, you can compute the estimated space occupied by those bitmaps.

Quoted 3 lines

but I
think the right way to
do this is in qemu-img measure.

Yes, even though we have existing multi-step code, being able to do it in a single step is justification for the improvement (although you may still end up having to code the multi-step mode yourself if you can't guarantee new-enough qemu-img - the addition of 'qemu-img measure --bitmaps' won't land until at least 4 months from now with qemu 5.1). Not to mention that the estimation computation (image size / granularity rounded up to the next cluster size, summed over all bitmaps) is hairy enough that it shouldn't have to be reimplemented by multiple layers of software.

Quoted 7 lines

We have also another use case when we collapsed an image chain to single image:

Source chain:
/dev/vg1/base
/dev/vg1/top

Quoted 8 lines

Destination:
/dev/vg2/collapsed

In this case we measure the size of the entire chain (/dev/vg1/base <- /dev/vg1/top) and create
/dev/vg2/collapsed in the correct size, and then we convert the chain using:

qemu-img convert /dev/vg1/top /dev/vg2/collapsed

When I submitted my v1 patch, I had only tested with a single source image to a single destination. But now that you mention flattening, it's easy enough for me to test of what happens (tl;dr: my patch only looks at bitmaps in the active layer of the source):

$ qemu-img create -f qcow2 img.base 100M
Formatting 'img.base', fmt=qcow2 size=104857600 cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-img create -f qcow2 -F qcow2 -b img.base img.top
Formatting 'img.top', fmt=qcow2 size=104857600 backing_file=img.base backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-kvm --nographic --nodefaults --qmp stdio
{'execute':'qmp_capabilities'}
{'execute':'blockdev-add','arguments':{'driver':'qcow2',
 'node-name':'base','file':{'driver':'file','filename':'img.base'}}}
{'execute':'blockdev-add','arguments':{'driver':'qcow2',
 'node-name':'top','file':{'driver':'file','filename':'img.top'},
 'backing':'base'}}
{'execute':'block-dirty-bitmap-add','arguments':{'node':'base',
 'name':'b1','persistent':true}}
{'execute':'block-dirty-bitmap-add','arguments':{'node':'base',
 'name':'b2','persistent':true}}
{'execute':'block-dirty-bitmap-add','arguments':{'node':'top',
 'name':'b2','persistent':true}}
{'execute':'block-dirty-bitmap-add','arguments':{'node':'top',
 'name':'b3','persistent':true}}
{'execute':'quit'}
$ qemu-img convert -f qcow2 -O qcow2 --bitmaps img.top img.flat
$ qemu-img info img.flat
image: img.flat
file format: qcow2
virtual size: 100 MiB (104857600 bytes)
disk size: 208 KiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    bitmaps:
        [0]:
            flags:
                [0]: auto
            name: b3
            granularity: 65536
        [1]:
            flags:
                [0]: auto
            name: b2
            granularity: 65536
    refcount bits: 16
    corrupt: false

It only copied the bitmaps from the active layer, not all bitmaps from all layers. If we want more complicated handling (such as whether to pull in bitmaps deeper in the backing chain, rename bitmaps, policies on merging duplicated bitmap names across nodes vs. failure, etc), we really should be designing 'qemu-img bitmap' that gives full command-line control over all sorts of bitmap operations, rather than trying to further overload 'qemu-img convert'.

Quoted 9 lines

Currently we use this for exporting images, for example when creating templates, or as a simple
backup. In this case we don't need to copy the bitmaps in the target image - this is a new image
not used by any VM. Copying the bitmaps may also be non-trivial since we may have the bitmaps
with the same names in several layers (e.g. result of live snapshot).

Quoted 6 lines

So I think using --bitmaps should be disabled when doing this kind of convert. We can handle this
on our side easily, but I think this should fail or log a warning on qemu-img, or require merging of
bitmaps with same names during the copy. I did not check if you already handle this.

My patch only copies top-most bitmaps. If you are using qemu-img convert to flatten a chain, we'll need something else to control what to do with bitmaps across that chain.

Quoted 10 lines

Finally we also have a use case when we copy the chain as is to new or same storage, but
we create a new vm. In this case I don't think the backup history makes sense for the new
vm, so we don't need to copy the bitmaps.

I will review the rest of the patches next week and can maybe give this some testing.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org