qemu-img: Add convert --bitmaps
Subsystems (2)
v2 cover qemu-img: Add convert --bitmaps
Without this series, the process for copying one qcow2 image to another including all of its bitmaps involves running qemu and doing the copying by hand with a series of QMP commands. This makes the process a bit more convenient.
Series can also be downloaded at:
https://repo.or.cz/qemu/ericb.git/shortlog/refs/tags/qemu-img-bitmaps-v2
Since v1:
- patches 1, 3 are new
- patch 2 (old 1/3) moves more operations into new file
- patch 4 no longer a separate thread (was at v2 in isolation)
- patches 4, 6 rebased to take advantage of 3
- patch 5, 6 (old 2-3/3) have other minor tweaks
Eric Blake (6):
docs: Sort sections on qemu-img subcommand parameters
blockdev: Split off basic bitmap operations for qemu-img
qemu-img: Add bitmap sub-command
qcow2: Expose bitmaps' size during measure
qemu-img: Add convert --bitmaps option
iotests: Add test 291 to for qemu-img bitmap coverage
docs/tools/qemu-img.rst | 78 ++++++--- Makefile.objs | 2 +- qapi/block-core.json | 15 +- include/sysemu/blockdev.h | 14 ++ block/crypto.c | 2 +- block/qcow2.c | 29 +++- block/raw-format.c | 2 +- blockbitmaps.c | 324 +++++++++++++++++++++++++++++++++++++ blockdev.c | 293 --------------------------------- qemu-img.c | 281 +++++++++++++++++++++++++++++++- MAINTAINERS | 1 + qemu-img-cmds.hx | 11 +- tests/qemu-iotests/190 | 15 +- tests/qemu-iotests/190.out | 13 +- tests/qemu-iotests/291 | 103 ++++++++++++ tests/qemu-iotests/291.out | 78 +++++++++ tests/qemu-iotests/group | 1 + 17 files changed, 927 insertions(+), 335 deletions(-) create mode 100644 blockbitmaps.c create mode 100755 tests/qemu-iotests/291 create mode 100644 tests/qemu-iotests/291.out
Signature · 1 line
2.26.2
PATCH v2 1/6 docs: Sort sections on qemu-img subcommand parameters
We already list the subcommand summaries alphabetically, we should do the same for the documentation related to subcommand-specific parameters.
--- docs/tools/qemu-img.rst | 48 ++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 24 deletions(-)
docs/tools/qemu-img.rst +24 -24
@@ -142,30 +142,6 @@ by the used format or see the format descriptions below for details. the documentation of the emulator's ``-drive cache=...`` option for allowed values. -Parameters to snapshot subcommand: - -.. program:: qemu-img-snapshot - -.. option:: snapshot - - Is the name of the snapshot to create, apply or delete - -.. option:: -a - - Applies a snapshot (revert disk to saved state) - -.. option:: -c - - Creates a snapshot - -.. option:: -d - - Deletes a snapshot - -.. option:: -l - - Lists all snapshots in the given image - Parameters to compare subcommand: .. program:: qemu-img-compare
@@ -245,6 +221,30 @@ Parameters to dd subcommand: Sets the number of input blocks to skip +Parameters to snapshot subcommand: + +.. program:: qemu-img-snapshot + +.. option:: snapshot + + Is the name of the snapshot to create, apply or delete + +.. option:: -a + + Applies a snapshot (revert disk to saved state) + +.. option:: -c + + Creates a snapshot + +.. option:: -d + + Deletes a snapshot + +.. option:: -l + + Lists all snapshots in the given image + Command description: .. program:: qemu-img-commands
Signature · 1 line
2.26.2
Re: [PATCH v2 1/6] docs: Sort sections on qemu-img subcommand parameters
Quoted 8 lines · Eric Blake
We already list the subcommand summaries alphabetically, we should do the same for the documentation related to subcommand-specific parameters.
Signed-off-by Eric Blake <[email protected]>--- docs/tools/qemu-img.rst | 48 ++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 24 deletions(-)
PATCH v2 2/6 blockdev: Split off basic bitmap operations for qemu-img
In reply to Eric BlakeUpcoming patches want to add some basic bitmap manipulation abilities to qemu-img. But blockdev.o is too heavyweight to link into qemu-img (among other things, it would drag in block jobs and transaction support - qemu-img does offline manipulation, where atomicity is less important because there are no concurrent modifications to compete with), so it's time to split off the bare bones of what we will need into a new file blockbitmaps.o.
In addition to exposing 6 QMP commands for use by qemu-img (add, remove, clear, enable, disable, merge), this also has to export three previously-static functions for use by blockdev.c transactions.
--- Makefile.objs | 2 +- include/sysemu/blockdev.h | 14 ++ blockbitmaps.c | 324 ++++++++++++++++++++++++++++++++++++++ blockdev.c | 293 ---------------------------------- MAINTAINERS | 1 + 5 files changed, 340 insertions(+), 294 deletions(-) create mode 100644 blockbitmaps.c
Makefile.objs +1 -1
@@ -14,7 +14,7 @@ chardev-obj-y = chardev/ authz-obj-y = authz/ block-obj-y = nbd/ -block-obj-y += block.o blockjob.o job.o +block-obj-y += block.o blockbitmaps.o blockjob.o job.o block-obj-y += block/ scsi/ block-obj-y += qemu-io-cmds.o block-obj-$(CONFIG_REPLICATION) += replication.o
include/sysemu/blockdev.h +14 -0
@@ -57,4 +57,18 @@ QemuOpts *drive_add(BlockInterfaceType type, int index, const char *file,DriveInfo *drive_new (QemuOpts *arg,BlockInterfaceType block_default_type,Error **errp); +BdrvDirtyBitmap *block_dirty_bitmap_lookup (const char *node, +const char *name, +BlockDriverState **pbs, +Error **errp); +BdrvDirtyBitmap *do_block_dirty_bitmap_remove (const char *node, +const char *name,bool release, +BlockDriverState **bitmap_bs, +Error **errp); +BdrvDirtyBitmap *do_block_dirty_bitmap_merge (const char *node, +const char *target, +BlockDirtyBitmapMergeSourceList *bitmaps, +HBitmap **backup,Error **errp); + +#endif
blockbitmaps.c new file +324 -0
324 lines not on this page · raw message
blockdev.c +0 -293
317 lines not on this page · raw message
MAINTAINERS +1 -0
@@ -1989,6 +1989,7 @@ T: git https://github.com/jnsnow/qemu.git jobs Block QAPI, monitor, command line M: Markus Armbruster <[email protected]> S: Supported +F: blockbitmaps.c F: blockdev.c F: blockdev-hmp-cmds.c F: block/qapi.c
Signature · 1 line
2.26.2
Re: [PATCH v2 2/6] blockdev: Split off basic bitmap operations for qemu-img
Quoted 12 lines · Eric Blake
Upcoming patches want to add some basic bitmap manipulation abilities to qemu-img. But blockdev.o is too heavyweight to link into qemu-img (among other things, it would drag in block jobs and transaction support - qemu-img does offline manipulation, where atomicity is less important because there are no concurrent modifications to compete with), so it's time to split off the bare bones of what we will need into a new file blockbitmaps.o.
In addition to exposing 6 QMP commands for use by qemu-img (add, remove, clear, enable, disable, merge), this also has to export three previously-static functions for use by
blockdev.ctransactions.
Quoted 5 lines
Signed-off-by Eric Blake <[email protected]>--- Makefile.objs | 2 +- include/sysemu/blockdev.h | 14 ++ blockbitmaps.c | 324 ++++++++++++++++++++++++++++++++++++++
Hm. Can we get a better name? blockdev-bitmaps.c, for example?
Quoted 4 lines
blockdev.c | 293 ---------------------------------- MAINTAINERS | 1 + 5 files changed, 340 insertions(+), 294 deletions(-) create mode 100644 blockbitmaps.c
[...]
Quoted 10 lines
include/sysemu/blockdev.h +2
-0@@ -57,4 +57,18 @@ QemuOpts *drive_add(BlockInterfaceType type, int index, const char *file,DriveInfo *drive_new (QemuOpts *arg,BlockInterfaceType block_default_type,Error **errp); +BdrvDirtyBitmap *block_dirty_bitmap_lookup(const char *node, + const char *name,
include/sysemu/blockdev.h QemuOpts *drive_add(BlockInterfaceType type, int index, const char *file, · line 71
Quoted 10 lines
include/sysemu/blockdev.h +10
-0+ BlockDriverState **pbs, + Error **errp); +BdrvDirtyBitmap *do_block_dirty_bitmap_remove (const char *node, +const char *name,bool release, +BlockDriverState **bitmap_bs, +Error **errp); +BdrvDirtyBitmap *do_block_dirty_bitmap_merge (const char *node, +const char *target, +BlockDirtyBitmapMergeSourceList *bitmaps, +HBitmap **backup,Error **errp);
Putting do_* functions into a normal header seems a bit weird. Would these fit better into block_int.h?
Quoted 2 lines
include/sysemu/blockdev.h +1
-0+#endif
[...]
MAINTAINERS T: git https://github.com/jnsnow/qemu.git jobs · line 1992
Quoted 9 lines
MAINTAINERS +1
-0@@ -1989,6 +1989,7 @@ T: git https://github.com/jnsnow/qemu.git jobs Block QAPI, monitor, command line M: Markus Armbruster <[email protected]> S: Supported +F: blockbitmaps.c
The natural choice for something split off of blockdev.c, but I wonder if the Dirty Bitmaps section wouldn’t be a better fit.
Max
Quoted 4 lines
MAINTAINERS +0
-0F: blockdev.c F: blockdev-hmp-cmds.c F: block/qapi.c
Re: [PATCH v2 2/6] blockdev: Split off basic bitmap operations for qemu-img
Quoted 13 lines · Max Reitz
Quoted 12 lines · Eric Blake
Upcoming patches want to add some basic bitmap manipulation abilities to qemu-img. But blockdev.o is too heavyweight to link into qemu-img (among other things, it would drag in block jobs and transaction support - qemu-img does offline manipulation, where atomicity is less important because there are no concurrent modifications to compete with), so it's time to split off the bare bones of what we will need into a new file blockbitmaps.o.
In addition to exposing 6 QMP commands for use by qemu-img (add, remove, clear, enable, disable, merge), this also has to export three previously-static functions for use by
blockdev.ctransactions.
Quoted 7 lines
Quoted 5 lines
Signed-off-by Eric Blake <[email protected]>--- Makefile.objs | 2 +- include/sysemu/blockdev.h | 14 ++ blockbitmaps.c | 324 ++++++++++++++++++++++++++++++++++++++Hm. Can we get a better name?
blockdev-bitmaps.c, for example?
Sure, I'm open to bike-shed suggestions. I'd also _really_ love to make the new file NOT live in the top-level, but that's a harder task that I'm not sure how to do (it's easy to tweak Makefile.objs for another file in the same directory, but harder to see through the magic to figure out how to relocate things).
Quoted 16 lines
Quoted 5 lines
blockdev.c | 293 ---------------------------------- MAINTAINERS | 1 + 5 files changed, 340 insertions(+), 294 deletions(-)create mode 100644
blockbitmaps.c[...]
Quoted 8 lines
include/sysemu/blockdev.h +0
-0@@ -57,4 +57,18 @@ QemuOpts *drive_add(BlockInterfaceType type, int index, const char *file,DriveInfo *drive_new (QemuOpts *arg,BlockInterfaceType block_default_type,Error **errp);
Quoted 15 lines
include/sysemu/blockdev.h QemuOpts *drive_add(BlockInterfaceType type, int index, const char *file, · line 71
Quoted 12 lines
include/sysemu/blockdev.h +12
-0+BdrvDirtyBitmap *block_dirty_bitmap_lookup (const char *node, +const char *name, +BlockDriverState **pbs, +Error **errp); +BdrvDirtyBitmap *do_block_dirty_bitmap_remove (const char *node, +const char *name,bool release, +BlockDriverState **bitmap_bs, +Error **errp); +BdrvDirtyBitmap *do_block_dirty_bitmap_merge (const char *node, +const char *target, +BlockDirtyBitmapMergeSourceList *bitmaps, +HBitmap **backup,Error **errp);Putting do_* functions into a normal header seems a bit weird. Would these fit better into
block_int.h?
I don't care which header gets them, as long as things compile. block_int.h seems reasonable, so I can try that. I also wonder if I should rename them at all (at which point, maybe I split this into two patches - one doing the rename, the other doing the file motion).
Quoted 9 lines
include/sysemu/blockdev.h T: git https://github.com/jnsnow/qemu.git jobs · line 1992
Quoted 6 lines
include/sysemu/blockdev.h +2
-0+++ b/MAINTAINERS @@ -1989,6 +1989,7 @@ T: git https://github.com/jnsnow/qemu.git jobs Block QAPI, monitor, command line M: Markus Armbruster <[email protected]> S: Supported +F: blockbitmaps.cThe natural choice for something split off of
blockdev.c, but I wonder if the Dirty Bitmaps section wouldn’t be a better fit.
Or maybe even both?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v2 2/6] blockdev: Split off basic bitmap operations for qemu-img
Am 30.04.2020 um 16:50 hat Eric Blake geschrieben:
Quoted 20 lines · Max Reitz
Quoted 19 lines · Eric Blake
Quoted 17 lines
Upcoming patches want to add some basic bitmap manipulation abilities to qemu-img. But blockdev.o is too heavyweight to link into qemu-img (among other things, it would drag in block jobs and transaction support - qemu-img does offline manipulation, where atomicity is less important because there are no concurrent modifications to compete with), so it's time to split off the bare bones of what we will need into a new file blockbitmaps.o.
In addition to exposing 6 QMP commands for use by qemu-img (add, remove, clear, enable, disable, merge), this also has to export three previously-static functions for use by
blockdev.ctransactions.Signed-off-by Eric Blake <[email protected]>--- Makefile.objs | 2 +- include/sysemu/blockdev.h | 14 ++ blockbitmaps.c | 324 ++++++++++++++++++++++++++++++++++++++
Quoted 7 lines
Quoted 1 line
Hm. Can we get a better name?
blockdev-bitmaps.c, for example?Sure, I'm open to bike-shed suggestions. I'd also _really_ love to make the new file NOT live in the top-level, but that's a harder task that I'm not sure how to do (it's easy to tweak Makefile.objs for another file in the same directory, but harder to see through the magic to figure out how to relocate things).
Yes, please move it somewhere else. I'd suggest something like block/monitor/bitmap-qmp-cmds.c for the QMP command handlers, and if there are functions that are more generally useful, block/bitmaps.c.
Instead of modifying the top-level Makefile.objs, you would just edit block/monitor/Makefile.objs instead and add the filename there. I don't think you need to understand any magic apart from knowing that is exists and does what you would expect.
Kevin
Re: [PATCH v2 2/6] blockdev: Split off basic bitmap operations for qemu-img
Quoted 13 lines · Kevin Wolf
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
--- Makefile.objs | 2 +- include/sysemu/blockdev.h | 14 ++ blockbitmaps.c | 324 ++++++++++++++++++++++++++++++++++++++Quoted 7 lines
Quoted 1 line
Hm. Can we get a better name?
blockdev-bitmaps.c, for example?Sure, I'm open to bike-shed suggestions. I'd also _really_ love to make the new file NOT live in the top-level, but that's a harder task that I'm not sure how to do (it's easy to tweak Makefile.objs for another file in the same directory, but harder to see through the magic to figure out how to relocate things).
Quoted 8 lines
Yes, please move it somewhere else. I'd suggest something like
block/monitor/bitmap-qmp-cmds.cfor the QMP command handlers, and if there are functions that are more generally useful,block/bitmaps.c.Instead of modifying the top-level Makefile.objs, you would just edit
block/monitor/Makefile.objsinstead and add the filename there. I don't think you need to understand any magic apart from knowing that is exists and does what you would expect.
Well, you still have to modify the top-level Makefile.objs to tell it to look in the right subdirectories, but I got it figured out finally. v3 will use block/monitor/bitmap-qmp-cmds.c.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v2 3/6 qemu-img: Add bitmap sub-command
In reply to Eric BlakeInclude actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'. Merge can work either from another bitmap in the same image, or from a bitmap in a distinct image.
While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file bitmap merge. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go; in part to simplify the command line, and in part because an offline image can achieve the same effect by multiple qemu-img bitmap --merge calls. We can enhance that if needed in the future (the same way that 'qemu-img convert' has a mode that concatenates multiple sources into one destination).
Upcoming patches will add iotest coverage of these commands while also testing other features.
--- docs/tools/qemu-img.rst | 24 +++++ qemu-img.c | 198 ++++++++++++++++++++++++++++++++++++++++ qemu-img-cmds.hx | 7 ++ 3 files changed, 229 insertions(+)
docs/tools/qemu-img.rst +24 -0
@@ -281,6 +281,30 @@ Command description: For write tests, by default a buffer filled with zeros is written. This can be overridden with a pattern byte specified by *PATTERN*. +.. option:: bitmap {--add [-g GRANULARITY] [--disabled] | --remove | --clear | --enable | --disable | --merge SOURCE_BITMAP [-b SOURCE_FILE [-F SOURCE_FMT]]} [--object OBJECTDEF] [--image-opts] [-f FMT] FILENAME BITMAP + + Perform a modification of the persistent bitmap *BITMAP* in the disk + image *FILENAME*. The various modifications are: + + ``--add`` to create *BITMAP*, with additional options ``-g`` to + specify a non-default *GRANULARITY*, or whether the bitmap should be + ``--disabled`` instead of enabled. + + ``--remove`` to remove *BITMAP*. + + ``--clear`` to clear *BITMAP*. + + ``--enable`` to change *BITMAP* to start recording future edits. + + ``--disable`` to change *BITMAP* to stop recording future edits. + + ``--merge`` to merge the contents of *SOURCE_BITMAP* into *BITMAP*. + This defaults to requiring a source bitmap from the same *FILENAME*, + but can also be used for cross-image merge by supplying ``-b`` to + specify a different *SOURCE_FILE*. + + To see what bitmaps are present in an image, use ``qemu-img info``. + .. option:: check [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [--output=OFMT] [-r [leaks | all]] [-T SRC_CACHE] [-U] FILENAME Perform a consistency check on the disk image *FILENAME*. The command can
qemu-img.c +198 -0
216 lines not on this page · raw message
qemu-img-cmds.hx +7 -0
@@ -20,6 +20,13 @@ DEF("bench", img_bench, SRST .. option:: bench [-c COUNT] [-d DEPTH] [-f FMT] [--flush-interval=FLUSH_INTERVAL] [-i AIO] [-n] [--no-drain] [-o OFFSET] [--pattern=PATTERN] [-q] [-s BUFFER_SIZE] [-S STEP_SIZE] [-t CACHE] [-w] [-U] FILENAME ERST + +DEF("bitmap", img_bitmap, + "bitmap {--add [-g granularity] [--disabled] | --remove | --clear | --enable | --disable | --merge source_bitmap [-b source_file [-F source_fmt]]} [--object objectdef] [--image-opts] [-f fmt] filename bitmap") +SRST +.. option:: bitmap {--add [-g GRANULARITY] [--disabled] | --remove | --clear | --enable | --disable | --merge SOURCE_BITMAP [-b SOURCE_FILE [-F SOURCE_FMT]]} [--object OBJECTDEF] [--image-opts] [-f FMT] FILENAME BITMAP +ERST + DEF("check", img_check, "check [--object objectdef] [--image-opts] [-q] [-f fmt] [--output=ofmt] [-r [leaks | all]] [-T src_cache] [-U] filename") SRST
Signature · 1 line
2.26.2
Re: [PATCH v2 3/6] qemu-img: Add bitmap sub-command
Quoted 4 lines · Eric Blake
Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place,
Well, ideally, all of qemu-img is just fluff because “the same can be accomplished by launching qemu and issuing the equivalent QMP commands”. :)
Quoted 3 lines
but it matches what QMP commands exist). Listing is omitted,
because it does not require a bitmap name and because it was already possible with 'qemu-img info'.
Fair enough, although it can be said that qemu-img info’s output is qcow2-specific. It might be nice to have some definitely format-independent output. (But we don’t have persistent bitmaps in anything but qcow2 yet (or do we in NBD?), so I don’t expect anyone to care much.)
Quoted 28 lines
Merge can work either from another
bitmap in the same image, or from a bitmap in a distinct image.While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file bitmap merge. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go; in part to simplify the command line, and in part because an offline image can achieve the same effect by multiple qemu-img bitmap --merge calls. We can enhance that if needed in the future (the same way that 'qemu-img convert' has a mode that concatenates multiple sources into one destination).
Upcoming patches will add iotest coverage of these commands while also testing other features.
Signed-off-by Eric Blake <[email protected]>--- docs/tools/qemu-img.rst | 24 +++++ qemu-img.c | 198 ++++++++++++++++++++++++++++++++++++++++ qemu-img-cmds.hx | 7 ++ 3 files changed, 229 insertions(+)docs/tools/qemu-img.rst +0
-0
docs/tools/qemu-img.rst Command description: · line 284
Quoted 4 lines
docs/tools/qemu-img.rst +1
-0@@ -281,6 +281,30 @@ Command description: For write tests, by default a buffer filled with zeros is written. This can be overridden with a pattern byte specified by *PATTERN*. +.. option:: bitmap {--add [-g GRANULARITY] [--disabled] | --remove | --clear | --enable | --disable | --merge SOURCE_BITMAP [-b SOURCE_FILE [-F SOURCE_FMT]]} [--object OBJECTDEF] [--image-opts] [-f FMT] FILENAME BITMAP
So I can do multiple operations in one roll, but they all use the same BITMAP? Sounds a bit weird. It actually took me a while to understands this, because I thought for sure that each command would take a bitmap name. (And was ready to complain that it looked like they don’t, but, well, that’s because they don’t.)
Although I suppose some practical example like
$ qemu-img bitmap --add --merge sbmap --disable foo.qcow2 nbmap
does make sense.[1]
Would
$ qemu-img bitmap --add nbmap --merge nbmap sbmap --enable nbmap \
foo.qcow2make more sense? Doesn’t really look like it, because at that point you just have to ask why not just call qemu-img bitmap multiple times.
*shrug*
[1] However, that leads me to ask: Why does --add need a --disabled option? Isn’t that equivalent to --add --disable?
Quoted 59 lines
docs/tools/qemu-img.rst +23
-0+ + Perform a modification of the persistent bitmap *BITMAP* in the disk + image *FILENAME*. The various modifications are: + + ``--add`` to create *BITMAP*, with additional options ``-g`` to + specify a non-default *GRANULARITY*, or whether the bitmap should be + ``--disabled`` instead of enabled. + + ``--remove`` to remove *BITMAP*. + + ``--clear`` to clear *BITMAP*. + + ``--enable`` to change *BITMAP* to start recording future edits. + + ``--disable`` to change *BITMAP* to stop recording future edits. + + ``--merge`` to merge the contents of *SOURCE_BITMAP* into *BITMAP*. + This defaults to requiring a source bitmap from the same *FILENAME*, + but can also be used for cross-image merge by supplying ``-b`` to + specify a different *SOURCE_FILE*. + + To see what bitmaps are present in an image, use ``qemu-img info``. + .. option:: check [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [--output=OFMT] [-r [leaks | all]] [-T SRC_CACHE] [-U] FILENAME Perform a consistency check on the disk image *FILENAME*. The command canqemu-img.c +11
-0@@ -28,6 +28,7 @@#include "qemu-common.h" #include "qemu-version.h" #include "qapi/error.h" +#include "qapi/qapi-commands-block-core.h" #include "qapi/qapi-visit-block-core.h" #include "qapi/qobject-output-visitor.h" #include "qapi/qmp/qjson.h" @@ -71,6 +72,12 @@ enum {OPTION_SHRINK =266 ,OPTION_SALVAGE =267 ,OPTION_TARGET_IS_ZERO =268 , +OPTION_ADD =269 , +OPTION_REMOVE =270 , +OPTION_CLEAR =271 , +OPTION_ENABLE =272 , +OPTION_DISABLE =273 , + OPTION_MERGE = 274, }; typedef enum OutputFormat {@@ -4438,6 +4445,197 @@ out:return 0 ; } +static int img_bitmap (int argc,char **argv) +{ +Error *err =NULL ; +int c, ret =-1 ;
qemu-img.c out: · line 4461
Quoted 10 lines
qemu-img.c +10
-0+QemuOpts *opts =NULL ; +const char *fmt =NULL , *src_fmt =NULL , *src_filename =NULL ; +const char *filename, *bitmap; +BlockBackend *blk =NULL , *src =NULL ; +BlockDriverState *bs =NULL , *src_bs =NULL ; +bool image_opts = false; +unsigned long granularity =0 ; +bool add = false, remove = false, clear = false; +bool enable = false, disable = false, add_disabled = false; +const char *merge =NULL ;
So actually we can’t do one operation multiple times in one roll.
qemu-img.c out: · line 4473
Quoted 12 lines
qemu-img.c +12
-0+ +for (;;) { +static const struct option long_options[] = { + {"help" , no_argument,0 ,'h' }, + {"object" , required_argument,0 ,OPTION_OBJECT }, + {"image-opts" , no_argument,0 ,OPTION_IMAGE_OPTS }, + {"add" , no_argument,0 ,OPTION_ADD }, + {"remove" , no_argument,0 ,OPTION_REMOVE }, + {"clear" , no_argument,0 ,OPTION_CLEAR }, + {"enable" , no_argument,0 ,OPTION_ENABLE }, + {"disable" , no_argument,0 ,OPTION_DISABLE }, + {"disabled" , no_argument,0 ,OPTION_DISABLE },
So if --disable and --disabled are exactly the same, I really don’t know why --disabled even exists.
Quoted 70 lines
qemu-img.c +70
-0+ {"merge", required_argument, 0, OPTION_MERGE}, + {"granularity", required_argument, 0, 'g'}, + {"source-file", required_argument, 0, 'b'}, + {"source-format", required_argument, 0, 'F'}, + {0, 0, 0, 0} + }; + c =getopt_long (argc, argv,":b:f:F:g:h" , long_options,NULL ); +if (c ==-1 ) { +break ; + } + +switch (c) { +case ':' : +missing_argument (argv[optind -1 ]); +break ; +case '?' : +unrecognized_option (argv[optind -1 ]); +break ; +case 'h' : +help (); +break ; +case 'b' : + src_filename = optarg; +break ; +case 'f' : + fmt = optarg; +break ; +case 'F' : + src_fmt = optarg; +break ; +case 'g' : +if (qemu_strtosz (optarg,NULL , &granularity)) { +error_report ("Invalid granularity specified" ); +return 1 ; + } +break ; +case OPTION_ADD : + add = true; +break ; +case OPTION_REMOVE : + remove = true; +break ; +case OPTION_CLEAR : + clear = true; +break ; +case OPTION_ENABLE : + enable = true; +break ; +case OPTION_DISABLE : + disable = true; +break ; +case OPTION_MERGE : + merge = optarg; +break ; +case OPTION_OBJECT : + opts =qemu_opts_parse_noisily (&qemu_object_opts, + optarg, true); +if (!opts) { + goto out; + } +break ; +case OPTION_IMAGE_OPTS : + image_opts = true; +break ; + } + } + +if (qemu_opts_foreach (&qemu_object_opts, + user_creatable_add_opts_foreach, + qemu_img_object_print_help, &error_fatal)) {
qemu-img.c out: · line 4553
Quoted 10 lines
qemu-img.c +10
-0+ goto out; + } + +if (add && disable) { + disable = false; + add_disabled = true; + } +if (add + remove + clear + enable + disable + !!merge !=1 ) { +error_report ("Need exactly one mode of --add, --remove, --clear, " +"--enable, --disable, or --merge" );
Aha. So you can actually only do a single operation.
That means the doc shouldn’t use {}, in my opinion, because that to me means repetition (thanks to EBNF). It definitely served to confuse me greatly until this point.
I also don’t see why we would disallow multiple operations in one go. The --add --merge combination seems useful to me, and I don’t see a problem in implementing it.
I don’t know why we don’t just create a list of operations to execute, based on the order given in the argument list, and then execute them in order. That would even allow multiple --merge operations.
If we don’t want that (because we don’t want argument order to matter), we could still allow all operations to be done at least once and always execute them in the same order, e.g.:
(1) add
(2) clear
(3) merge
(4) disable
(5) enable
(6) remove
qemu-img.c out: · line 4565
Quoted 12 lines
qemu-img.c +12
-0+ goto out; + } +if (granularity && !add) { +error_report ("granularity only supported with --add" ); + goto out; + } +if (src_fmt && !src_filename) { +error_report ("-F only supported with -b" ); + goto out; + } +if (src_filename && !merge) { +error_report ("alternate source file only supported with --merge" );
“alternate” sounds strange. Why not be more precise and call it a “Merge bitmap source file” or “File to source merge bitmap from”?
Quoted 47 lines
qemu-img.c +47
-0+ goto out; + } + +if (optind != argc -2 ) { +error_report ("Expecting filename and bitmap name" ); + goto out; + } + + filename = argv[optind]; + bitmap = argv[optind +1 ]; + + blk =img_open (image_opts, filename, fmt,BDRV_O_RDWR , false, false, + false); +if (!blk) { + goto out; + } + bs =blk_bs (blk); + +if (add) { +qmp_block_dirty_bitmap_add (bs->node_name, bitmap, + !!granularity, granularity, true, true, + true, add_disabled, &err); + }else if (remove) { +qmp_block_dirty_bitmap_remove (bs->node_name, bitmap, &err); + }else if (clear) { +qmp_block_dirty_bitmap_clear (bs->node_name, bitmap, &err); + }else if (enable) { +qmp_block_dirty_bitmap_enable (bs->node_name, bitmap, &err); + }else if (disable) { +qmp_block_dirty_bitmap_disable (bs->node_name, bitmap, &err); + }else if (merge) { +BlockDirtyBitmapMergeSource *merge_src; +BlockDirtyBitmapMergeSourceList *list; + +if (src_filename) { + src =img_open (NULL , src_filename, src_fmt,0 , false, false, + false); +if (!src) { + goto out; + } + src_bs =blk_bs (src); + }else { + src_bs = bs; + } + + merge_src =g_new0 (BlockDirtyBitmapMergeSource,1 ); + merge_src->type =QTYPE_QDICT ;
qemu-img.c out: · line 4622
Quoted 10 lines
qemu-img.c +10
-0+ merge_src->u.external.node =g_strdup (src_bs->node_name); + merge_src->u.external.name =g_strdup (merge); + list =g_new0 (BlockDirtyBitmapMergeSourceList,1 ); + list->value = merge_src; +qmp_block_dirty_bitmap_merge (bs->node_name, bitmap, list, &err); +qapi_free_BlockDirtyBitmapMergeSourceList (list); + } + +if (err) { +error_reportf_err (err,"Bitmap %s operation failed" , bitmap);
Wouldn’t “Operation on bitmap %s failed” work better?
Max
Quoted 19 lines
qemu-img.c +16
-0+ ret =-1 ; + goto out; + } + + ret =0 ; + + out: +blk_unref (src); +blk_unref (blk); +qemu_opts_del (opts); +if (ret) { +return 1 ; + } +return 0 ; +} +#define C_BS 01#define C_COUNT 02#define C_IF 04
Re: [PATCH v2 3/6] qemu-img: Add bitmap sub-command
Quoted 13 lines · Max Reitz
Quoted 4 lines · Eric Blake
Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place,
Well, ideally, all of qemu-img is just fluff because “the same can be accomplished by launching qemu and issuing the equivalent QMP commands”. :)
Quoted 3 lines
but it matches what QMP commands exist). Listing is omitted,
because it does not require a bitmap name and because it was already possible with 'qemu-img info'.
Quoted 5 lines
Fair enough, although it can be said that qemu-img info’s output is qcow2-specific. It might be nice to have some definitely format-independent output. (But we don’t have persistent bitmaps in anything but qcow2 yet (or do we in NBD?), so I don’t expect anyone to care much.)
We can add a list subcommand later if it is still desired. I agree that a tabular format:
name enabled granularity bitmap1 false 65536 bitmap2 true 512
in isolation is easier to read than:
bitmaps:
[0]:
flags:
name: bitmap1
granularity: 65536
[1]:
flags:
[0]: auto
name: bitmap2
granularity: 512embedded inside even more information.
Quoted 32 lines
Quoted 32 lines
Merge can work either from another
bitmap in the same image, or from a bitmap in a distinct image.While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file bitmap merge. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go; in part to simplify the command line, and in part because an offline image can achieve the same effect by multiple qemu-img bitmap --merge calls. We can enhance that if needed in the future (the same way that 'qemu-img convert' has a mode that concatenates multiple sources into one destination).
Upcoming patches will add iotest coverage of these commands while also testing other features.
Signed-off-by Eric Blake <[email protected]>--- docs/tools/qemu-img.rst | 24 +++++ qemu-img.c | 198 ++++++++++++++++++++++++++++++++++++++++ qemu-img-cmds.hx | 7 ++ 3 files changed, 229 insertions(+)docs/tools/qemu-img.rst +0
-0@@ -281,6 +281,30 @@ Command description: For write tests, by default a buffer filled with zeros is written. This can be overridden with a pattern byte specified by *PATTERN*.
Quoted 7 lines
docs/tools/qemu-img.rst Command description: · line 284
Quoted 1 line
docs/tools/qemu-img.rst +1
-0+.. option:: bitmap {--add [-g GRANULARITY] [--disabled] | --remove | --clear | --enable | --disable | --merge SOURCE_BITMAP [-b SOURCE_FILE [-F SOURCE_FMT]]} [--object OBJECTDEF] [--image-opts] [-f FMT] FILENAME BITMAPSo I can do multiple operations in one roll, but they all use the same BITMAP? Sounds a bit weird. It actually took me a while to understands this, because I thought for sure that each command would take a bitmap name. (And was ready to complain that it looked like they don’t, but, well, that’s because they don’t.)
All of the operations take one bitmap name (the final BITMAP). Additionally, the --merge operation takes a second bitmap name (SOURCE_BITMAP). None of the other operations need a second bitmap name, so only --merge requires an option argument. As written, the { a | b | c } implies that operations are mutually exclusive: you can only request one operation per qemu-img invocation.
Quoted 8 lines
Although I suppose some practical example like
$ qemu-img bitmap --add --merge sbmap --disable foo.qcow2 nbmapdoes make sense.[1]
Quoted 6 lines
Would
$ qemu-img bitmap --add nbmap --merge nbmap sbmap --enable nbmap \ foo.qcow2make more sense?
That would be more transactional, and more effort to implement. My argument is that you should instead write:
$ qemu-img bitmap --add foo.qcow2 nbmap $ qemu-img bitmap --merge sbmap foo.qcow2 nbmap $ qemu-img bitmap --enable foo.qcow2 nbmap
where I only have to implement one operation per qemu-img.
Quoted 8 lines
Doesn’t really look like it, because at that point you
just have to ask why not just call qemu-img bitmap multiple times.*shrug*
[1] However, that leads me to ask: Why does --add need a --disabled option? Isn’t that equivalent to --add --disable?
The QMP command for add has an optional 'disabled' parameter, which I exposed here. Alternatively, we could indeed NOT expose that parameter through qemu-img, but require two separate qemu-img bitmap commands to add and then disable things. Atomicity is not a concern here like it was in QMP. Removing that sugar does simplify things slightly.
Quoted 8 lines
Quoted 8 lines
docs/tools/qemu-img.rst +8
-0+static int img_bitmap(int argc, char **argv) +{ + Error *err = NULL; + int c, ret = -1; + QemuOpts *opts = NULL; + const char *fmt = NULL, *src_fmt = NULL, *src_filename = NULL; + const char *filename, *bitmap; + BlockBackend *blk = NULL, *src = NULL;
Quoted 8 lines
docs/tools/qemu-img.rst Command description: · line 298
Quoted 6 lines
docs/tools/qemu-img.rst +6
-0+ BlockDriverState *bs = NULL, *src_bs = NULL; + bool image_opts = false; + unsigned long granularity = 0; + bool add = false, remove = false, clear = false; + bool enable = false, disable = false, add_disabled = false; + const char *merge = NULL;So actually we can’t do one operation multiple times in one roll.
Correct. At least, not as I wrote it.
Quoted 14 lines
docs/tools/qemu-img.rst Command description: · line 310
Quoted 13 lines
docs/tools/qemu-img.rst +12
-0+ + for (;;) { + static const struct option long_options[] = { + {"help", no_argument, 0, 'h'}, + {"object", required_argument, 0, OPTION_OBJECT}, + {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS}, + {"add", no_argument, 0, OPTION_ADD}, + {"remove", no_argument, 0, OPTION_REMOVE}, + {"clear", no_argument, 0, OPTION_CLEAR}, + {"enable", no_argument, 0, OPTION_ENABLE}, + {"disable", no_argument, 0, OPTION_DISABLE}, + {"disabled", no_argument, 0, OPTION_DISABLE},
Quoted 2 lines
So if --disable and --disabled are exactly the same, I really don’t know why --disabled even exists.
Logically, '--add --disabled' matches the name of the QMP command with its optional 'disabled' parameter, while --disable matches the name of the QMP command. We don't have to have the alias; and in fact, if you agree that supporting '--add --disabled' is too much sugar (since we don't care about atomicity the way QMP did), then life gets simpler to drop --disabled altogether.
Quoted 8 lines
docs/tools/qemu-img.rst Command description:
Quoted 7 lines
docs/tools/qemu-img.rst +7
-0+ if (add && disable) { + disable = false; + add_disabled = true; + } + if (add + remove + clear + enable + disable + !!merge != 1) { + error_report("Need exactly one mode of --add, --remove, --clear, " + "--enable, --disable, or --merge");
Quoted 5 lines
Aha. So you can actually only do a single operation.
That means the doc shouldn’t use {}, in my opinion, because that to me means repetition (thanks to EBNF). It definitely served to confuse me greatly until this point.
In command line syntax, I'm most used to seeing repetition as '...', optional as [], and mutually-exclusive choice as {|}. Yes, that's different than EBNF.
Quoted 8 lines
I also don’t see why we would disallow multiple operations in one go. The --add --merge combination seems useful to me, and I don’t see a problem in implementing it.
I don’t know why we don’t just create a list of operations to execute, based on the order given in the argument list, and then execute them in order. That would even allow multiple --merge operations.
If I understand, you're asking why we can't do:
qemu-img bitmap foo.qcow2 --add b1 --merge sb b1
in one operation.
That changes the syntax entirely, compared to what I implemented. You'd have to have an argument to every option, AND figure out how to specify TWO arguments to the --merge option. Might be doable, but seems hairy.
Quoted 10 lines
If we don’t want that (because we don’t want argument order to matter), we could still allow all operations to be done at least once and always execute them in the same order, e.g.:
(1) add
(2) clear
(3) merge
(4) disable
(5) enable
(6) remove
I still find it simpler to do exactly one operation per invocation.
Quoted 14 lines
docs/tools/qemu-img.rst
Quoted 13 lines
docs/tools/qemu-img.rst +12
-0+ goto out; + } + if (granularity && !add) { + error_report("granularity only supported with --add"); + goto out; + } + if (src_fmt && !src_filename) { + error_report("-F only supported with -b"); + goto out; + } + if (src_filename && !merge) { + error_report("alternate source file only supported with --merge");
Quoted 2 lines
“alternate” sounds strange. Why not be more precise and call it a “Merge bitmap source file” or “File to source merge bitmap from”?
"Merge bitmap source file" sounds fine.
Quoted 5 lines
docs/tools/qemu-img.rst
Quoted 3 lines
docs/tools/qemu-img.rst +3
-0+ + if (err) { + error_reportf_err(err, "Bitmap %s operation failed", bitmap);Wouldn’t “Operation on bitmap %s failed” work better?
Yes.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v2 3/6] qemu-img: Add bitmap sub-command
Quoted 87 lines · Eric Blake
Quoted 14 lines · Max Reitz
Quoted 4 lines · Eric Blake
Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place,
Well, ideally, all of qemu-img is just fluff because “the same can be accomplished by launching qemu and issuing the equivalent QMP commands”. :)
Quoted 3 lines
but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'.
Quoted 5 lines
Fair enough, although it can be said that qemu-img info’s output is qcow2-specific. It might be nice to have some definitely format-independent output. (But we don’t have persistent bitmaps in anything but qcow2 yet (or do we in NBD?), so I don’t expect anyone to care much.)
We can add a list subcommand later if it is still desired. I agree that a tabular format:
name enabled granularity
bitmap1 false 65536
bitmap2 true 512in isolation is easier to read than:
bitmaps:
[0]:
flags:
name: bitmap1
granularity: 65536
[1]:
flags:
[0]: auto
name: bitmap2
granularity: 512embedded inside even more information.
Quoted 33 lines
Quoted 33 lines
Merge can work either from another bitmap in the same image, or from a bitmap in a distinct image.
While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file bitmap merge. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go; in part to simplify the command line, and in part because an offline image can achieve the same effect by multiple qemu-img bitmap --merge calls. We can enhance that if needed in the future (the same way that 'qemu-img convert' has a mode that concatenates multiple sources into one destination).
Upcoming patches will add iotest coverage of these commands while also testing other features.
Signed-off-by Eric Blake <[email protected]>---
docs/tools/qemu-img.rst| 24 +++++
qemu-img.c | 198 ++++++++++++++++++++++++++++++++++++++++ qemu-img-cmds.hx | 7 ++3 files changed, 229 insertions(+)docs/tools/qemu-img.rst +0
-0@@ -281,6 +281,30 @@ Command description:For write tests, by default a buffer filled with zeros is written. This can be
overridden with a pattern byte specified by *PATTERN*.Quoted 10 lines
Quoted 4 lines
+.. option:: bitmap {--add [-g GRANULARITY] [--disabled] | --remove | --clear | --enable | --disable | --merge
SOURCE_BITMAP[-bSOURCE_FILE[-FSOURCE_FMT]]} [--object OBJECTDEF] [--image-opts] [-f FMT] FILENAME BITMAPSo I can do multiple operations in one roll, but they all use the same BITMAP? Sounds a bit weird. It actually took me a while to understands this, because I thought for sure that each command would take a bitmap name. (And was ready to complain that it looked like they don’t, but, well, that’s because they don’t.)
Quoted 6 lines
All of the operations take one bitmap name (the final BITMAP). Additionally, the --merge operation takes a second bitmap name (SOURCE_BITMAP). None of the other operations need a second bitmap name, so only --merge requires an option argument. As written, the { a | b | c } implies that operations are mutually exclusive: you can only request one operation per qemu-img invocation.
Well, as I found out later it’s supposed to imply that. I always expect {} to mean repetition.
Quoted 7 lines
Quoted 7 lines
Although I suppose some practical example like
$ qemu-img bitmap --add --merge sbmap --disable foo.qcow2 nbmapdoes make sense.[1]
Quoted 8 lines
Quoted 6 lines
Would
$ qemu-img bitmap --add nbmap --merge nbmap sbmap --enable nbmap \ foo.qcow2make more sense?
That would be more transactional, and more effort to implement.
As a user, I wouldn’t expect it to be a transaction, but just executed in order.
Quoted 7 lines
My argument is that you should instead write:
$ qemu-img bitmap --add foo.qcow2 nbmap $ qemu-img bitmap --merge sbmap foo.qcow2 nbmap $ qemu-img bitmap --enable foo.qcow2 nbmapwhere I only have to implement one operation per qemu-img.
I don’t know about the “have to”, because from an algorithm standpoint, doing so would be trivial. (That is, collecting the operations into a list, along with their specific arguments, and then execute the list in a loop.)
I can see that doing this in C is more difficult than it would be in nicer languages, and so not actually trivial.
But allowing all switches at most once without mutual exclusion still seems simple. The question is mostly whether the implementation would match what we can expect users to expect.
[...]
Quoted 9 lines
Quoted 2 lines
So if --disable and --disabled are exactly the same, I really don’t know why --disabled even exists.
Logically, '--add --disabled' matches the name of the QMP command with its optional 'disabled' parameter, while --disable matches the name of the QMP command. We don't have to have the alias; and in fact, if you agree that supporting '--add --disabled' is too much sugar (since we don't care about atomicity the way QMP did), then life gets simpler to drop --disabled altogether.
I find it makes the interface unnecessarily complex, as does requiring mutual exclusion.
If we want mutual exclusion, I can see that a separate --disabled for --add makes sense.
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
+ if (add && disable) {
+ disable = false;
+add_disabled= true;
+ }
+ if (add + remove + clear + enable + disable + !!merge != 1) { + error_report("Need exactly one mode of --add, --remove, --clear, "
+ "--enable, --disable, or --merge");Aha. So you can actually only do a single operation.
Quoted 7 lines
Quoted 3 lines
That means the doc shouldn’t use {}, in my opinion, because that to me means repetition (thanks to EBNF). It definitely served to confuse me greatly until this point.
In command line syntax, I'm most used to seeing repetition as '...', optional as [], and mutually-exclusive choice as {|}. Yes, that's different than EBNF.
It’s confusing is what it is, and unnecessarily so. The | already signifies exclusion: Say there are -a and -b, if they’re not mutually exclusive, then the doc describe them as “[-a] [-b]”; if they are, it’d be “-a | -b”. Maybe “(-a | -b)”.
I can’t remember having seen {|} for mutual exclusivity before, but then again, it doesn’t happen often, I suppose. git for one thing seems to use (|).
(Regex also uses {} for repetition, even if in a different way from EBNF.)
I’ve never seen “...” used for switches, only for free-form arguments like filenames. (Because normally, a switch can be specified only once, or it wouldn’t be a “switch”.)
Quoted 10 lines
Quoted 7 lines
I also don’t see why we would disallow multiple operations in one go. The --add --merge combination seems useful to me, and I don’t see a problem in implementing it.
I don’t know why we don’t just create a list of operations to execute, based on the order given in the argument list, and then execute them in order. That would even allow multiple --merge operations.
If I understand, you're asking why we can't do:
Quoted 7 lines
qemu-img bitmap foo.qcow2 --add b1 --merge sb b1
in one operation.
That changes the syntax entirely, compared to what I implemented. You'd have to have an argument to every option, AND figure out how to specify TWO arguments to the --merge option. Might be doable, but seems hairy.
Just “qemu-img bitmap --add --merge sb foo.qcow2 b1” would be enough.
Quoted 11 lines
Quoted 9 lines
If we don’t want that (because we don’t want argument order to matter), we could still allow all operations to be done at least once and always execute them in the same order, e.g.:
(1) add
(2) clear
(3) merge
(4) disable
(5) enable
(6) removeI still find it simpler to do exactly one operation per invocation.
I feel like that’s mostly because of our coding style allowing an “else if” to end and start a block on the same line.
(That is, I feel like if we allowed multiple operations in a single go, the only difference would be that we wouldn’t have to check for mutual exclusivity, and that all “} else if (cond) {”s would have to be turned into “} if (cond) {”s. And reordered.)
It’s possible that I mostly feel compelled to vote for non-exclusivity because it would be clear then how to document the interface (i.e., “[--add] [--clear] [...]”) and I wouldn’t have to explain my utter confusion at the sight of {}.
Max
Re: [PATCH v2 3/6] qemu-img: Add bitmap sub-command
Quoted 22 lines · Max Reitz
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
+.. option:: bitmap {--add [-g GRANULARITY] [--disabled] | --remove | --clear | --enable | --disable | --merge
SOURCE_BITMAP[-bSOURCE_FILE[-FSOURCE_FMT]]} [--object OBJECTDEF] [--image-opts] [-f FMT] FILENAME BITMAPSo I can do multiple operations in one roll, but they all use the same BITMAP? Sounds a bit weird. It actually took me a while to understands this, because I thought for sure that each command would take a bitmap name. (And was ready to complain that it looked like they don’t, but, well, that’s because they don’t.)
Quoted 6 lines
All of the operations take one bitmap name (the final BITMAP). Additionally, the --merge operation takes a second bitmap name (SOURCE_BITMAP). None of the other operations need a second bitmap name, so only --merge requires an option argument. As written, the { a | b | c } implies that operations are mutually exclusive: you can only request one operation per qemu-img invocation.
Well, as I found out later it’s supposed to imply that. I always expect {} to mean repetition.
Quoted 8 lines
Quoted 3 lines
In command line syntax, I'm most used to seeing repetition as '...', optional as [], and mutually-exclusive choice as {|}. Yes, that's different than EBNF.
It’s confusing is what it is, and unnecessarily so. The | already signifies exclusion: Say there are -a and -b, if they’re not mutually exclusive, then the doc describe them as “[-a] [-b]”; if they are, it’d be “-a | -b”. Maybe “(-a | -b)”.
Is s/{/(/ is all the more that would be needed to take this patch as-is?
Or should I really try and re-write it to take a list of operations,
and iterate through the list until the first failure?
(At any rate, I'll probably end up trying that anyways, just for comparison in the lines of code...)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v2 4/6 qcow2: Expose bitmaps' size during measure
In reply to Eric BlakeIt's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional field. Update iotest 190 to cover it and a portion of the just-added qemu-img bitmap command.
The addition of a new field demonstrates why we should always zero-initialize qapi C structs; while the qcow2 driver still fully populates all fields, the raw and crypto drivers had to be tweaked.
See also: https://bugzilla.redhat.com/1779904
--- qapi/block-core.json | 15 ++++++++++----- block/crypto.c | 2 +- block/qcow2.c | 29 ++++++++++++++++++++++++++++- block/raw-format.c | 2 +- qemu-img.c | 3 +++ tests/qemu-iotests/190 | 15 +++++++++++++-- tests/qemu-iotests/190.out | 13 ++++++++++++- 7 files changed, 68 insertions(+), 11 deletions(-)
qapi/block-core.json +10 -5
@@ -633,18 +633,23 @@ # efficiently so file size may be smaller than virtual disk size. # # The values are upper bounds that are guaranteed to fit the new image file. -# Subsequent modification, such as internal snapshot or bitmap creation, may -# require additional space and is not covered here. +# Subsequent modification, such as internal snapshot or further bitmap +# creation, may require additional space and is not covered here. # -# @required: Size required for a new image file, in bytes. +# @required: Size required for a new image file, in bytes, when copying just +# guest-visible contents. # # @fully-allocated: Image file size, in bytes, once data has been written -# to all sectors. +# to all sectors, when copying just guest-visible contents. +# +# @bitmaps: Additional size required for bitmap metadata not directly used +# for guest contents, when that metadata can be copied in addition +# to guest contents. (since 5.1) # # Since: 2.10 ## { 'struct': 'BlockMeasureInfo', - 'data': {'required': 'int', 'fully-allocated': 'int'} } + 'data': {'required': 'int', 'fully-allocated': 'int', '*bitmaps': 'int'} } ## # @query-block:
block/crypto.c +1 -1
@@ -535,7 +535,7 @@ static BlockMeasureInfo *block_crypto_measure(QemuOpts *opts,* Unallocated blocks are still encrypted so allocation status makes no * difference to the file size. */ - info =g_new (BlockMeasureInfo,1 ); + info =g_new0 (BlockMeasureInfo,1 ); info->fully_allocated = luks_payload_size + size; info->required = luks_payload_size + size;return info;
block/qcow2.c +28 -1
@@ -4657,6 +4657,7 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,PreallocMode prealloc;bool has_backing_file;bool has_luks; +uint64_t bitmaps_size =0 ;/* size occupied by bitmaps in in_bs */ /* Parse image creation options */ cluster_size =qcow2_opt_get_cluster_size_del (opts, &local_err);
@@ -4732,6 +4733,8 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,/* Account for input image */ if (in_bs) { +BdrvDirtyBitmap *bm; +size_t bitmap_overhead =0 ;int64_t ssize =bdrv_getlength (in_bs);if (ssize <0 ) { error_setg_errno(&local_err, -ssize,
@@ -4739,6 +4742,28 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, goto err; } + FOR_EACH_DIRTY_BITMAP(in_bs, bm) { +if (bdrv_dirty_bitmap_get_persistence (bm)) { +const char *name =bdrv_dirty_bitmap_name (bm); +uint32_t granularity =bdrv_dirty_bitmap_granularity (bm); +uint64_t bmbits =DIV_ROUND_UP (bdrv_dirty_bitmap_size (bm), + granularity); +uint64_t bmclusters =DIV_ROUND_UP (DIV_ROUND_UP (bmbits, +CHAR_BIT ), + cluster_size); + +/* Assume the entire bitmap is allocated */ + bitmaps_size += bmclusters * cluster_size; +/* Also reserve space for the bitmap table entries */ + bitmaps_size +=ROUND_UP (bmclusters *sizeof (uint64_t ), + cluster_size); +/* Guess at contribution to bitmap directory size */ + bitmap_overhead +=ROUND_UP (strlen (name) +24 , +sizeof (uint64_t )); + } + } + bitmaps_size +=ROUND_UP (bitmap_overhead, cluster_size); + virtual_size =ROUND_UP (ssize, cluster_size);if (has_backing_file) {
@@ -4785,7 +4810,7 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, required = virtual_size; } - info =g_new (BlockMeasureInfo,1 ); + info =g_new0 (BlockMeasureInfo,1 ); info->fully_allocated =qcow2_calc_prealloc_size (virtual_size, cluster_size,ctz32 (refcount_bits)) + luks_payload_size;
@@ -4795,6 +4820,8 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,* still counted. */ info->required = info->fully_allocated - virtual_size + required; + info->has_bitmaps = !!bitmaps_size; + info->bitmaps = bitmaps_size;return info; err:
block/raw-format.c +1 -1
@@ -346,7 +346,7 @@ static BlockMeasureInfo *raw_measure(QemuOpts *opts, BlockDriverState *in_bs, BDRV_SECTOR_SIZE); } - info =g_new (BlockMeasureInfo,1 ); + info =g_new0 (BlockMeasureInfo,1 ); info->required = required;/* Unallocated sectors count towards the file size in raw images */
qemu-img.c +3 -0
@@ -5207,6 +5207,9 @@ static int img_measure(int argc, char **argv)if (output_format ==OFORMAT_HUMAN ) {printf ("required size: %" PRIu64"\n" , info->required);printf ("fully allocated size: %" PRIu64"\n" , info->fully_allocated); +if (info->has_bitmaps) { +printf ("bitmaps size: %" PRIu64"\n" , info->bitmaps); + } }else {dump_json_block_measure_info (info); }
tests/qemu-iotests/190 +13 -2
@@ -2,7 +2,7 @@ # # qemu-img measure sub-command tests on huge qcow2 files # -# Copyright (C) 2017 Red Hat, Inc. +# Copyright (C) 2017-2020 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by
@@ -42,7 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -echo "== Huge file ==" +echo "== Huge file without bitmaps ==" echo _make_test_img -o 'cluster_size=2M' 2T
@@ -51,6 +51,17 @@ $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" $QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG" $QEMU_IMG measure -O qcow2 -o cluster_size=2M -f qcow2 "$TEST_IMG" +echo +echo "== Huge file with bitmaps ==" +echo + +$QEMU_IMG bitmap --add --granularity 512 -f qcow2 "$TEST_IMG" b1 +$QEMU_IMG bitmap --add -g 2M -f qcow2 "$TEST_IMG" b2 + +$QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" +$QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG" +$QEMU_IMG measure -O qcow2 -o cluster_size=2M -f qcow2 "$TEST_IMG" + # success, all done echo "*** done" rm -f $seq.full
tests/qemu-iotests/190.out +12 -1
@@ -1,5 +1,5 @@ QA output created by 190 -== Huge file == +== Huge file without bitmaps == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2199023255552 required size: 2199023255552
@@ -8,4 +8,15 @@ required size: 335806464 fully allocated size: 2199359062016 required size: 18874368 fully allocated size: 2199042129920 + +== Huge file with bitmaps == + +required size: 2199023255552 +fully allocated size: 2199023255552 +required size: 335806464 +fully allocated size: 2199359062016 +bitmaps size: 537198592 +required size: 18874368 +fully allocated size: 2199042129920 +bitmaps size: 545259520 *** done
Signature · 1 line
2.26.2
Re: [PATCH v2 4/6] qcow2: Expose bitmaps' size during measure
Quoted 36 lines · Eric Blake
It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional field. Update iotest 190 to cover it and a portion of the just-added qemu-img bitmap command.
The addition of a new field demonstrates why we should always zero-initialize qapi C structs; while the qcow2 driver still fully populates all fields, the raw and crypto drivers had to be tweaked.
See also: https://bugzilla.redhat.com/1779904
Reported-by Nir Soffer <[email protected]>Signed-off-by Eric Blake <[email protected]>--- qapi/block-core.json | 15 ++++++++++----- block/crypto.c | 2 +- block/qcow2.c | 29 ++++++++++++++++++++++++++++- block/raw-format.c | 2 +- qemu-img.c | 3 +++ tests/qemu-iotests/190 | 15 +++++++++++++-- tests/qemu-iotests/190.out | 13 ++++++++++++- 7 files changed, 68 insertions(+), 11 deletions(-)qapi/block-core.json +2
-2@@ -633,18 +633,23 @@ # efficiently so file size may be smaller than virtual disk size. # # The values are upper bounds that are guaranteed to fit the new image file. -# Subsequent modification, such as internal snapshot or bitmap creation, may -# require additional space and is not covered here. +# Subsequent modification, such as internal snapshot or further bitmap +# creation, may require additional space and is not covered here. #
qapi/block-core.json line 646
Quoted 10 lines
qapi/block-core.json +6
-2-# @required: Size required for a new image file, in bytes. +# @required: Size required for a new image file, in bytes, when copying just +# guest-visible contents. # # @fully-allocated: Image file size, in bytes, once data has been written -# to all sectors. +# to all sectors, when copying just guest-visible contents. +# +# @bitmaps: Additional size required for bitmap metadata not directly used +# for guest contents,
Not sure how I feel about the “not directly used for guest contents”, because it feels a bit superfluous, and it sounds like there might be bitmap data that is directly used for guest contents.
qapi/block-core.json line 652
Quoted 8 lines
qapi/block-core.json +2
-1when that metadata can be copied in addition +# to guest contents. (since 5.1) # # Since: 2.10 ## { 'struct': 'BlockMeasureInfo', - 'data': {'required': 'int', 'fully-allocated': 'int'} } + 'data': {'required': 'int', 'fully-allocated': 'int', '*bitmaps': 'int'} }
Why is @bitmaps optional? I.e., what does absence mean, besides “not supported yet”?
Right now, absence means anything in “format doesn’t support bitmaps, so nothing can be copied”, “no input image given, so there’s no data on bitmaps”, to “0 bytes to copy”.
I think in the latter case we should emit it as 0, maybe even in the former case, because I think the fact that there won’t be any bitmap data to copy might be interesting. (And it’s also definitely 0, not just “don’t know”.)
I suppose absence does make sense in case the user didn’t specify an input image, because then we just really don’t know.
[...]
Quoted 4 lines
block/qcow2.c +0
-0
[...]
block/qcow2.c static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, · line 4752
Quoted 12 lines
block/qcow2.c +8
-0@@ -4739,6 +4742,28 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, goto err; } + FOR_EACH_DIRTY_BITMAP(in_bs, bm) { +if (bdrv_dirty_bitmap_get_persistence (bm)) { +const char *name =bdrv_dirty_bitmap_name (bm); +uint32_t granularity =bdrv_dirty_bitmap_granularity (bm); +uint64_t bmbits =DIV_ROUND_UP (bdrv_dirty_bitmap_size (bm), + granularity); + uint64_t bmclusters = DIV_ROUND_UP(DIV_ROUND_UP(bmbits, + CHAR_BIT),
I suppose if we allowed CHAR_BIT to be anything but 8, it would be wrong to use it here. So maybe just a plain 8 would be more correct; although I suppose CHAR_BIT kind of describes what constant we want here.
block/qcow2.c static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, · line 4762
Quoted 10 lines
block/qcow2.c +10
-0+ cluster_size); + +/* Assume the entire bitmap is allocated */ + bitmaps_size += bmclusters * cluster_size; +/* Also reserve space for the bitmap table entries */ + bitmaps_size +=ROUND_UP (bmclusters *sizeof (uint64_t ), + cluster_size); +/* Guess at contribution to bitmap directory size */ + bitmap_overhead +=ROUND_UP (strlen (name) +24 , +sizeof (uint64_t ));
Seems not just a guess to me, but actually correct. Maybe bitmap_overhead should be called bitmap_directory_size (or bitmap_dir_size, or bmap_dir_size, or whatever (but probably not bds! :)), because that’s what it is.
Quoted 7 lines
block/qcow2.c +4
-0+ } + } + bitmaps_size +=ROUND_UP (bitmap_overhead, cluster_size); + virtual_size =ROUND_UP (ssize, cluster_size);if (has_backing_file) {
[...]
Quoted 16 lines
tests/qemu-iotests/190.out +2
-1@@ -1,5 +1,5 @@ QA output created by 190 -== Huge file == +== Huge file without bitmaps == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2199023255552 required size: 2199023255552@@ -8,4 +8,15 @@ required size: 335806464 fully allocated size: 2199359062016 required size: 18874368 fully allocated size: 2199042129920 +
tests/qemu-iotests/190.out required size: 335806464 · line 21
Quoted 10 lines
tests/qemu-iotests/190.out +10
-0+== Huge file with bitmaps == + +required size: 2199023255552 +fully allocated size: 2199023255552 +required size: 335806464 +fully allocated size: 2199359062016 +bitmaps size: 537198592 +required size: 18874368 +fully allocated size: 2199042129920 +bitmaps size: 545259520
Looks correct.
(It might be nicer to calculate the reference value in the script, because this way I had to verify it by hand, but, well, now I did verify it, so...)
Max
Re: [PATCH v2 4/6] qcow2: Expose bitmaps' size during measure
Quoted 16 lines · Max Reitz
Quoted 14 lines · Eric Blake
It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional field. Update iotest 190 to cover it and a portion of the just-added qemu-img bitmap command.
The addition of a new field demonstrates why we should always zero-initialize qapi C structs; while the qcow2 driver still fully populates all fields, the raw and crypto drivers had to be tweaked.
See also: https://bugzilla.redhat.com/1779904
Reported-by Nir Soffer <[email protected]>Signed-off-by Eric Blake <[email protected]>---
Quoted 10 lines
qapi/block-core.json
Quoted 6 lines
# @fully-allocated: Image file size, in bytes, once data has been written
qapi/block-core.json +4
-1-# to all sectors. +# to all sectors, when copying just guest-visible contents. +# +# @bitmaps: Additional size required for bitmap metadata not directly used +# for guest contents,Not sure how I feel about the “not directly used for guest contents”, because it feels a bit superfluous, and it sounds like there might be bitmap data that is directly used for guest contents.
Hmm. I was trying to make it obvious that bitmap size is separate from fully-allocated (and not double-counted), but you may have a point that just using "Additional size required for bitmap metadata, when that metadata can be copied in addition..." would work.
Quoted 17 lines
qapi/block-core.json line 652
Quoted 9 lines
when that metadata can be copied in addition
qapi/block-core.json +2
-1+# to guest contents. (since 5.1) # # Since: 2.10 ## { 'struct': 'BlockMeasureInfo', - 'data': {'required': 'int', 'fully-allocated': 'int'} } + 'data': {'required': 'int', 'fully-allocated': 'int', '*bitmaps': 'int'} }Why is @bitmaps optional? I.e., what does absence mean, besides “not supported yet”?
Right now, absence means anything in “format doesn’t support bitmaps, so nothing can be copied”, “no input image given, so there’s no data on bitmaps”, to “0 bytes to copy”.
Quoted 7 lines
I think in the latter case we should emit it as 0, maybe even in the former case, because I think the fact that there won’t be any bitmap data to copy might be interesting. (And it’s also definitely 0, not just “don’t know”.)
I suppose absence does make sense in case the user didn’t specify an input image, because then we just really don’t know.
The patch will require a tweak to report an explicit 0 (when there is nothing to be copied), but doing that makes sense (explicit 0 for v3 qcow2 images, and maybe even for v2, but omitted for other formats that have no bitmap support).
Quoted 4 lines
Quoted 4 lines
block/qcow2.c +0
-0@@ -4739,6 +4742,28 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, goto err; }
Quoted 12 lines
block/qcow2.c static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, · line 4752
Quoted 8 lines
block/qcow2.c +8
-0+ FOR_EACH_DIRTY_BITMAP(in_bs, bm) { +if (bdrv_dirty_bitmap_get_persistence (bm)) { +const char *name =bdrv_dirty_bitmap_name (bm); +uint32_t granularity =bdrv_dirty_bitmap_granularity (bm); +uint64_t bmbits =DIV_ROUND_UP (bdrv_dirty_bitmap_size (bm), + granularity); + uint64_t bmclusters = DIV_ROUND_UP(DIV_ROUND_UP(bmbits, + CHAR_BIT),I suppose if we allowed
CHAR_BITto be anything but 8, it would be wrong to use it here. So maybe just a plain 8 would be more correct; although I supposeCHAR_BITkind of describes what constant we want here.
POSIX requires CHAR_BIT to be 8. (C99 allows some odd machines where CHAR_BIT is 9, 16, or even 32, but we don't ever try to port to such machines).
Quoted 13 lines
tests/qemu-iotests/190.out required size: 335806464 · line 21
Quoted 10 lines
tests/qemu-iotests/190.out +10
-0+== Huge file with bitmaps == + +required size: 2199023255552 +fully allocated size: 2199023255552 +required size: 335806464 +fully allocated size: 2199359062016 +bitmaps size: 537198592 +required size: 18874368 +fully allocated size: 2199042129920 +bitmaps size: 545259520Looks correct.
Quoted 3 lines
(It might be nicer to calculate the reference value in the script, because this way I had to verify it by hand, but, well, now I did verify it, so...)
Feels like duplicate work, which would require tweaking in two spots if we ever change our implementation or the qcow2 format is further enhanced, but it would also make it obvious that we are aware of the impact of such future changes. Okay, I'll see what I can add.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v2 5/6 qemu-img: Add convert --bitmaps option
In reply to Eric BlakeMake it easier to copy all the persistent bitmaps of a source image along with the contents, by adding a boolean flag for use with qemu-img convert. This is basically shorthand, as the same effect could be accomplished with a series of 'qemu-img bitmap --add' and 'qemu-img bitmap --merge -b source' commands, or by QMP commands.
See also https://bugzilla.redhat.com/show_bug.cgi?id=1779893
--- docs/tools/qemu-img.rst | 6 +++- qemu-img.c | 80 +++++++++++++++++++++++++++++++++++++++-- qemu-img-cmds.hx | 4 +-- 3 files changed, 84 insertions(+), 6 deletions(-)
docs/tools/qemu-img.rst +5 -1
@@ -162,6 +162,10 @@ Parameters to convert subcommand: .. program:: qemu-img-convert +.. option:: --bitmaps + + Additionally copy all bitmaps + .. option:: -n Skip the creation of the target volume
@@ -397,7 +401,7 @@ Command description: 4 Error on reading data -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME Convert the disk image *FILENAME* or a snapshot *SNAPSHOT_PARAM* to disk image *OUTPUT_FILENAME* using format *OUTPUT_FMT*. It can
qemu-img.c +77 -3
141 lines not on this page · raw message
qemu-img-cmds.hx +2 -2
@@ -46,9 +46,9 @@ SRST ERST DEF("convert", img_convert, - "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") + "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") SRST -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME ERST DEF("create", img_create,
Signature · 1 line
2.26.2
Re: [PATCH v2 5/6] qemu-img: Add convert --bitmaps option
Quoted 14 lines · Eric Blake
Make it easier to copy all the persistent bitmaps of a source image along with the contents, by adding a boolean flag for use with qemu-img convert. This is basically shorthand, as the same effect could be accomplished with a series of 'qemu-img bitmap --add' and 'qemu-img bitmap --merge -b source' commands, or by QMP commands.
See also https://bugzilla.redhat.com/show_bug.cgi?id=1779893
Signed-off-by Eric Blake <[email protected]>--- docs/tools/qemu-img.rst | 6 +++- qemu-img.c | 80 +++++++++++++++++++++++++++++++++++++++-- qemu-img-cmds.hx | 4 +-- 3 files changed, 84 insertions(+), 6 deletions(-)
[...]
Quoted 4 lines
qemu-img.c +0
-0
[...]
Quoted 8 lines
qemu-img.c +0
-1@@ -2293,7 +2342,6 @@ static int img_convert(int argc, char **argv) goto fail_getopt; } -/* ret is still -EINVAL until here */ ret =bdrv_parse_cache_mode (src_cache, &src_flags, &src_writethrough);if (ret <0 ) {
[...]
qemu-img.c static int img_convert(int argc, char **argv) · line 2466
Quoted 7 lines
qemu-img.c +0
-2@@ -2461,9 +2531,7 @@ static int img_convert(int argc, char **argv)if (!skip_create) { open_opts =qdict_new ();qemu_opt_foreach (opts, img_add_key_secrets, open_opts, &error_abort); - } -if (!skip_create) {
I mean, cleanups are always welcome, but I feel compelled[1] to complain that it’s done in this patch without mentioning it in the commit message.
[1] By social norms, not by my personal opinion.
I feel good about all changes, though, so:
PATCH v2 6/6 iotests: Add test 291 to for qemu-img bitmap coverage
In reply to Eric BlakeAdd a new test covering the 'qemu-img bitmap' subcommand, as well as 'qemu-img convert --bitmaps', both added in recent patches.
--- tests/qemu-iotests/291 | 103 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/291.out | 78 ++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 182 insertions(+) create mode 100755 tests/qemu-iotests/291 create mode 100644 tests/qemu-iotests/291.out
tests/qemu-iotests/291 new file +103 -0
@@ -0,0 +1,103 @@ +#!/usr/bin/env bash +# +# Test qemu-img bitmap handling +# +# Copyright (C) 2018-2020 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +seq="$(basename $0)" +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img + nbd_server_stop +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter +. ./common.nbd + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux +_require_command QEMU_NBD + +echo +echo "=== Initial image setup ===" +echo + +# Create backing image with one bitmap +TEST_IMG="$TEST_IMG.base" _make_test_img 10M +$QEMU_IMG bitmap --add -f $IMGFMT "$TEST_IMG.base" b0 +$QEMU_IO -c 'w 3M 1M' -f $IMGFMT "$TEST_IMG.base" | _filter_qemu_io +# Create initial image and populate two bitmaps: one active, one inactive. +_make_test_img -b "$TEST_IMG.base" -F $IMGFMT 10M +$QEMU_IO -c 'w 0 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG bitmap --add -g 512k -f $IMGFMT "$TEST_IMG" b1 +$QEMU_IMG bitmap --add --disabled -f $IMGFMT "$TEST_IMG" b2 +$QEMU_IO -c 'w 3M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG bitmap --clear -f $IMGFMT "$TEST_IMG" b1 +$QEMU_IO -c 'w 1M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG bitmap --disable -f $IMGFMT "$TEST_IMG" b1 +$QEMU_IMG bitmap --enable -f $IMGFMT "$TEST_IMG" b2 +$QEMU_IO -c 'w 2M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io + +echo +echo "=== Bitmap preservation not possible to non-qcow2 ===" +echo + +mv "$TEST_IMG" "$TEST_IMG.orig" +$QEMU_IMG convert --bitmaps -O raw "$TEST_IMG.orig" "$TEST_IMG" + +echo +echo "=== Convert with bitmap preservation ===" +echo + +# Only bitmaps from the active layer are copied +$QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG.orig" "$TEST_IMG" +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific +# But we can also merge in bitmaps from other layers +$QEMU_IMG bitmap --add --disabled -f $IMGFMT "$TEST_IMG" b0 +$QEMU_IMG bitmap --add -f $IMGFMT "$TEST_IMG" tmp +$QEMU_IMG bitmap --merge b0 -b "$TEST_IMG.base" -F $IMGFMT "$TEST_IMG" tmp +$QEMU_IMG bitmap --merge tmp "$TEST_IMG" b0 +$QEMU_IMG bitmap --remove -f $IMGFMT "$TEST_IMG" tmp +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific + +echo +echo "=== Check bitmap contents ===" +echo + +IMG="driver=nbd,server.type=unix,server.path=$nbd_unix_socket" +nbd_server_start_unix_socket -r -f qcow2 -B b0 "$TEST_IMG" +$QEMU_IMG map --output=json --image-opts \ + "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b0" | _filter_qemu_img_map +nbd_server_start_unix_socket -r -f qcow2 -B b1 "$TEST_IMG" +$QEMU_IMG map --output=json --image-opts \ + "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b1" | _filter_qemu_img_map +nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG" +$QEMU_IMG map --output=json --image-opts \ + "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map + +# success, all done +echo '*** done' +rm -f $seq.full +status=0
tests/qemu-iotests/291.out new file +78 -0
78 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -297,3 +297,4 @@ 288 quick 289 rw quick 290 rw auto quick +291 rw quick
Signature · 1 line
2.26.2
Re: [PATCH v2 6/6] iotests: Add test 291 to for qemu-img bitmap coverage
Quoted 17 lines · Eric Blake
Add a new test covering the 'qemu-img bitmap' subcommand, as well as 'qemu-img convert --bitmaps', both added in recent patches.
Signed-off-by Eric Blake <[email protected]>--- tests/qemu-iotests/291 | 103 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/291.out | 78 ++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 182 insertions(+) create mode 100755 tests/qemu-iotests/291 create mode 100644 tests/qemu-iotests/291.outtests/qemu-iotests/291 new file +0
-0
[...]
tests/qemu-iotests/291 line 67
Quoted 5 lines
tests/qemu-iotests/291 new file +5
-0+echo +echo "=== Bitmap preservation not possible to non-qcow2 ===" +echo + +mv "$TEST_IMG" "$TEST_IMG.orig"
“mv” doesn’t work images with external data files.
(ORIG_IMG=$TEST_IMG; TEST_IMG="$TEST_IMG".orig should work)
tests/qemu-iotests/291 line 74
Quoted 7 lines
+$QEMU_IMG convert --bitmaps -O raw "$TEST_IMG.orig" "$TEST_IMG"
tests/qemu-iotests/291 new file +6
-0+ +echo +echo "=== Convert with bitmap preservation ===" +echo + +# Only bitmaps from the active layer are copied
That’s kind of obvious when you think about (whenever an image is attached to a VM, only the active layer’s bitmaps are visible, not those from the backing chain), but maybe this should be noted in the documentation?
Quoted 8 lines
+$QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG.orig" "$TEST_IMG" +$QEMU_IMG info "$TEST_IMG" |
_filter_img_info--format-specific +# But we can also merge in bitmaps from other layers +$QEMU_IMG bitmap --add --disabled -f $IMGFMT "$TEST_IMG" b0 +$QEMU_IMG bitmap --add -f $IMGFMT "$TEST_IMG" tmp
+$QEMU_IMG bitmap --merge b0 -b "$TEST_IMG.base" -F $IMGFMT "$TEST_IMG" tmp +$QEMU_IMG bitmap --merge tmp "$TEST_IMG" b0
+$QEMU_IMG bitmap --remove -f $IMGFMT "$TEST_IMG" tmp
Why do we need tmp here? Can’t we just merge base’s b0 directly into $TEST_IMG’s b0?
[...]
Quoted 5 lines
tests/qemu-iotests/291.out new file +0
-0
[...]
tests/qemu-iotests/291.out line 77
Quoted 11 lines
+=== Check bitmap contents ===
tests/qemu-iotests/291.out new file +10
-0+ +[{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 4194304, "length": 6291456, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] +[{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 2097152, "length": 8388608, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] +[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]
Am I looking at this wrong or does the bitmap data seem to be inverted?
Everywhere where I’d expect the bitmaps to be cleared, this map reports
data=true, whereas where I’d expect them to be set, it reports data=false.
I suppose that’s intentional, but can you explain this behavior to me?
Max
Quoted 1 line
+*** done
Re: [PATCH v2 6/6] iotests: Add test 291 to for qemu-img bitmap coverage
Quoted 14 lines · Max Reitz
tests/qemu-iotests/291 line 67
Quoted 10 lines · Eric Blake
Add a new test covering the 'qemu-img bitmap' subcommand, as well as 'qemu-img convert --bitmaps', both added in recent patches.
Signed-off-by Eric Blake <[email protected]>tests/qemu-iotests/291 new file +5
-0+echo +echo "=== Bitmap preservation not possible to non-qcow2 ===" +echo + +mv "$TEST_IMG" "$TEST_IMG.orig"“mv” doesn’t work images with external data files.
Quoted 1 line
(ORIG_IMG=$TEST_IMG; TEST_IMG="$TEST_IMG".orig should work)
Good idea.
Quoted 13 lines
tests/qemu-iotests/291 line 74
Quoted 8 lines
+$QEMU_IMG convert --bitmaps -O raw "$TEST_IMG.orig" "$TEST_IMG"
tests/qemu-iotests/291 new file +6
-0+ +echo +echo "=== Convert with bitmap preservation ===" +echo + +# Only bitmaps from the active layer are copiedThat’s kind of obvious when you think about (whenever an image is attached to a VM, only the active layer’s bitmaps are visible, not those from the backing chain), but maybe this should be noted in the documentation?
As part of integrating bitmaps with external snapshots, libvirt actually depends on being able to see bitmaps from the backing chain - but as bitmaps are always referenced as a 'node name, bitmap name' tuple, this is indeed doable.
Quoted 10 lines
Quoted 9 lines
+$QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG.orig" "$TEST_IMG" +$QEMU_IMG info "$TEST_IMG" |
_filter_img_info--format-specific +# But we can also merge in bitmaps from other layers +$QEMU_IMG bitmap --add --disabled -f $IMGFMT "$TEST_IMG" b0 +$QEMU_IMG bitmap --add -f $IMGFMT "$TEST_IMG" tmp
+$QEMU_IMG bitmap --merge b0 -b "$TEST_IMG.base" -F $IMGFMT "$TEST_IMG" tmp +$QEMU_IMG bitmap --merge tmp "$TEST_IMG" b0
+$QEMU_IMG bitmap --remove -f $IMGFMT "$TEST_IMG" tmp
Quoted 2 lines
Why do we need tmp here? Can’t we just merge base’s b0 directly into $TEST_IMG’s b0?
Yes, we could. But then I wouldn't cover as many bitmap subcommands. Adding a comment about why the example is contrived (for maximal coverage) is a good idea.
Quoted 16 lines
tests/qemu-iotests/291.out line 77
Quoted 11 lines
+=== Check bitmap contents ===
tests/qemu-iotests/291.out new file +10
-0+ +[{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 4194304, "length": 6291456, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] +[{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 2097152, "length": 8388608, "depth": 0, "zero": false, "data": true, "offset": OFFSET}] +[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": OFFSET}, +{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": false}, +{ "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": true, "offset": OFFSET}]Am I looking at this wrong or does the bitmap data seem to be inverted?
Everywhere where I’d expect the bitmaps to be cleared, this map reports
data=true, whereas where I’d expect them to be set, it reports data=false.
Quoted 1 line
I suppose that’s intentional, but can you explain this behavior to me?
This is an artifact of how x-dirty-bitmap works (it has the x- prefix because it is a hack, but we don't have anything better for reading out bitmap contents). The NBD spec returns block status as a 32-bit value for a 'metadata context'; normally, we use context 'base:allocation' context where bit 0 is set for holes or clear for allocated, and bit 1 is set for reads-as-zero or clear for unknown contents (favoring all-0 as the most-common case). But with x-dirty-bitmap, we are instead abusing NBD to query the 'qemu:dirty-bitmap:FOO' context, where bit 0 is set for anywhere the bitmap has a 1, yet feed that information into the pre-existing qemu code for handling block status. So qemu-img map is reporting "data":true for what it thinks is the normal 0-for-allocated, and "data":false for 1-for-sparse, and we just have to translate that back into an understanding of what the bitmap reported. Yes, a comment would be helpful.
I would _really_ love to enhance 'qemu-img map' to output image-specific metadata _in addition_ to the existing "zero" and "data" fields (by having qemu-img read two NBD contexts at once: both base:allocation and qemu:dirty-bitmap:FOO), at which point we can drop the x- prefix and avoid the abuse of qemu's internals by overwriting the block_status code. But that's a bigger project.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-mingw@fedora build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===
LINK qemu-edid.exe
LINK qemu-ga.exe/tmp/qemu-test/src/qemu-img.c: In function 'img_bitmap':/tmp/qemu-test/src/qemu-img.c:4579:44: error: passing argument 3 of 'qemu_strtosz' from incompatible pointer type [-Werror=incompatible-pointer-types]
if (qemu_strtosz(optarg, NULL, &granularity)) {
^~~~~~~~~~~~
In file included from /tmp/qemu-test/src/qemu-img.c:37:
---
int qemu_strtosz(const char *nptr, const char **end, uint64_t *result);
~~~~~~~~~~^~~~~~cc1: all warnings being treated as errors
make: *** [/tmp/qemu-test/src/rules.mak:69: qemu-img.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
File "./tests/docker/docker.py", line 664, in <module>
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=e06abfa47fc2454097264ac821c5dceb', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-d9esijcl/src/docker-src.2020-04-21-18.18.59.19445:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=e06abfa47fc2454097264ac821c5dceb
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-d9esijcl/src'
make: *** [docker-run-test-mingw@fedora] Error 2
real 3m19.767s
user 0m8.957s
The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
PATCH 1/1 fixup! qemu-img: Add bitmap sub-command
---
Squash this into patch 3/6 to fix docker-test-mingw@fedora
qemu-img.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
qemu-img.c +1 -1
@@ -4529,7 +4529,7 @@ static int img_bitmap(int argc, char **argv)BlockBackend *blk =NULL , *src =NULL ;BlockDriverState *bs =NULL , *src_bs =NULL ;bool image_opts = false; -unsigned long granularity =0 ; +uint64_t granularity =0 ;bool add = false, remove = false, clear = false;bool enable = false, disable = false, add_disabled = false;const char *merge =NULL ;
Signature · 1 line
2.26.2