qemu-img: Add convert --bitmaps
Subsystems (2)
v4 cover qemu-img: Add convert --bitmaps
v3 was here:
https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg02139.html
original cover letter here:
https://lists.gnu.org/archive/html/qemu-devel/2020-04/msg03464.html
Since then:
- add R-b where appropriate
- patch 3: rename new driver hook [Max]
- patch 6: typo and grammar fix [Max]
- patch 7: add 'measure --bitmaps' convenience option [Nir]
- patch 7: more documentation, enhance test to cover more cases
- patch 8: rebase to earlier changes, ensure that both 'measure --bitmaps' and 'convert --bitmaps' pass or fail on the same criteria
001/9:[----] [--] 'docs: Sort sections on qemu-img subcommand parameters' 002/9:[----] [--] 'qemu-img: Fix stale comments on doc location' 003/9:[0017] [FC] 'block: Make it easier to learn which BDS support bitmaps' 004/9:[----] [--] 'blockdev: Promote several bitmap functions to non-static' 005/9:[----] [--] 'blockdev: Split off basic bitmap operations for qemu-img' 006/9:[0009] [FC] 'qemu-img: Add bitmap sub-command' 007/9:[0084] [FC] 'qcow2: Expose bitmaps' size during measure' 008/9:[0008] [FC] 'qemu-img: Add convert --bitmaps option' 009/9:[----] [--] 'iotests: Add test 291 to for qemu-img bitmap coverage'
Series can also be downloaded at:
https://repo.or.cz/qemu/ericb.git/shortlog/refs/tags/qemu-img-bitmaps-v4
Eric Blake (9):
docs: Sort sections on qemu-img subcommand parameters
qemu-img: Fix stale comments on doc location
block: Make it easier to learn which BDS support bitmaps
blockdev: Promote several bitmap functions to non-static
blockdev: Split off basic bitmap operations for qemu-img
qemu-img: Add bitmap sub-command
qcow2: Expose bitmaps' size during measure
qemu-img: Add convert --bitmaps option
iotests: Add test 291 to for qemu-img bitmap coverage
docs/tools/qemu-img.rst | 88 +++++--- Makefile.objs | 3 +- qapi/block-core.json | 15 +- block/qcow2.h | 1 + include/block/block_int.h | 13 ++ include/block/dirty-bitmap.h | 1 + block/crypto.c | 2 +- block/dirty-bitmap.c | 9 + block/monitor/bitmap-qmp-cmds.c | 323 +++++++++++++++++++++++++++ block/qcow2-bitmap.c | 7 + block/qcow2.c | 39 +++- block/raw-format.c | 2 +- blockdev.c | 303 +------------------------ qemu-img.c | 366 ++++++++++++++++++++++++++++++- MAINTAINERS | 1 + block/monitor/Makefile.objs | 1 + qemu-img-cmds.hx | 17 +- tests/qemu-iotests/178.out.qcow2 | 16 ++ tests/qemu-iotests/190 | 57 ++++- tests/qemu-iotests/190.out | 35 ++- tests/qemu-iotests/291 | 112 ++++++++++ tests/qemu-iotests/291.out | 78 +++++++ tests/qemu-iotests/group | 1 + 23 files changed, 1142 insertions(+), 348 deletions(-) create mode 100644 block/monitor/bitmap-qmp-cmds.c create mode 100755 tests/qemu-iotests/291 create mode 100644 tests/qemu-iotests/291.out
Signature · 1 line
2.26.2
PATCH v4 1/9 docs: Sort sections on qemu-img subcommand parameters
We already list the subcommand summaries alphabetically, we should do the same for the documentation related to subcommand-specific parameters.
--- docs/tools/qemu-img.rst | 48 ++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 24 deletions(-)
docs/tools/qemu-img.rst +24 -24
@@ -142,30 +142,6 @@ by the used format or see the format descriptions below for details. the documentation of the emulator's ``-drive cache=...`` option for allowed values. -Parameters to snapshot subcommand: - -.. program:: qemu-img-snapshot - -.. option:: snapshot - - Is the name of the snapshot to create, apply or delete - -.. option:: -a - - Applies a snapshot (revert disk to saved state) - -.. option:: -c - - Creates a snapshot - -.. option:: -d - - Deletes a snapshot - -.. option:: -l - - Lists all snapshots in the given image - Parameters to compare subcommand: .. program:: qemu-img-compare
@@ -245,6 +221,30 @@ Parameters to dd subcommand: Sets the number of input blocks to skip +Parameters to snapshot subcommand: + +.. program:: qemu-img-snapshot + +.. option:: snapshot + + Is the name of the snapshot to create, apply or delete + +.. option:: -a + + Applies a snapshot (revert disk to saved state) + +.. option:: -c + + Creates a snapshot + +.. option:: -d + + Deletes a snapshot + +.. option:: -l + + Lists all snapshots in the given image + Command description: .. program:: qemu-img-commands
Signature · 1 line
2.26.2
Re: [PATCH v4 1/9] docs: Sort sections on qemu-img subcommand parameters
Quoted 6 lines · Eric Blake
We already list the subcommand summaries alphabetically, we should do the same for the documentation related to subcommand-specific parameters.
Signed-off-by Eric Blake<[email protected]>Reviewed-by Max Reitz<[email protected]>
Signature · 2 lines
Best regards, Vladimir
PATCH v4 2/9 qemu-img: Fix stale comments on doc location
In reply to Eric BlakeMissed in commit e13c59fa.
--- qemu-img.c | 2 +- qemu-img-cmds.hx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
qemu-img.c +1 -1
@@ -108,7 +108,7 @@ static void QEMU_NORETURN unrecognized_option(const char *option)error_exit ("unrecognized option '%s'" , option); } -/* Please keep in synch with qemu-img.texi */ +/* Please keep in synch with docs/tools/qemu-img.rst */ static void QEMU_NORETURN help(void) { const char *help_msg =
qemu-img-cmds.hx +1 -1
@@ -7,7 +7,7 @@ HXCOMM command structures and help message. HXCOMM HXCOMM can be used for comments, discarded from both rST and C HXCOMM When amending the rST sections, please remember to copy the usage -HXCOMM over to the per-command sections in qemu-img.texi. +HXCOMM over to the per-command sections in docs/tools/qemu-img.rst. DEF("amend", img_amend, "amend [--object objectdef] [--image-opts] [-p] [-q] [-f fmt] [-t cache] -o options filename")
Signature · 1 line
2.26.2
Re: [PATCH v4 2/9] qemu-img: Fix stale comments on doc location
Quoted 4 lines · Eric Blake
Missed in commit
e13c59fa.Signed-off-by Eric Blake<[email protected]>Reviewed-by Max Reitz<[email protected]>
Signature · 2 lines
Best regards, Vladimir
PATCH v4 3/9 block: Make it easier to learn which BDS support bitmaps
In reply to Eric BlakeUpcoming patches will enhance bitmap support in qemu-img, but in doing so, it turns out to be nice to suppress output when persistent bitmaps make no sense (such as on a qcow2 v2 image). Add a hook to make this easier to query.
This patch adds a new callback .bdrv_supports_persistent_dirty_bitmap, rather than trying to shoehorn the answer in via existing callbacks. In particular, while it might have been possible to overload .bdrv_co_can_store_new_dirty_bitmap to special-case a NULL input to answer whether any persistent bitmaps are supported, that is at odds with whether a particular bitmap can be stored (for example, even on an image that supports persistent bitmaps but has currently filled up the maximum number of bitmaps, attempts to store another one should fail); and the new functionality doesn't require coroutine safety. Similarly, we could have added one more piece of information to .bdrv_get_info, but then again, most callers to that function tend to already discard extraneous information, and making it a catch-all rather than a series of dedicated scalar queries hasn't really simplified life.
In the future, when we improve the ability to look up bitmaps through a filter, we will probably also want to teach the block layer to automatically let filters pass this request on through.
--- block/qcow2.h | 1 + include/block/block_int.h | 1 + include/block/dirty-bitmap.h | 1 + block/dirty-bitmap.c | 9 +++++++++ block/qcow2-bitmap.c | 7 +++++++ block/qcow2.c | 2 ++ 6 files changed, 21 insertions(+)
block/qcow2.h +1 -0
@@ -764,6 +764,7 @@ bool qcow2_co_can_store_new_dirty_bitmap(BlockDriverState *bs,int qcow2_co_remove_persistent_dirty_bitmap (BlockDriverState *bs,const char *name,Error **errp); +bool qcow2_supports_persistent_dirty_bitmap (BlockDriverState *bs); ssize_t coroutine_fn qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
include/block/block_int.h +1 -0
@@ -560,6 +560,7 @@ struct BlockDriver { uint64_t parent_perm, uint64_t parent_shared, uint64_t *nperm, uint64_t *nshared); +bool (*bdrv_supports_persistent_dirty_bitmap)(BlockDriverState *bs); bool (*bdrv_co_can_store_new_dirty_bitmap)(BlockDriverState *bs, const char *name, uint32_t granularity,
include/block/dirty-bitmap.h +1 -0
@@ -16,6 +16,7 @@ typedef enum BitmapCheckFlags {#define BDRV_BITMAP_MAX_NAME_SIZE 1023 +bool bdrv_supports_persistent_dirty_bitmap (BlockDriverState *bs); BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs, uint32_t granularity, const char *name,
block/dirty-bitmap.c +9 -0
@@ -478,6 +478,15 @@ int bdrv_remove_persistent_dirty_bitmap(BlockDriverState *bs, const char *name, } } +bool +bdrv_supports_persistent_dirty_bitmap (BlockDriverState *bs) +{ +if (bs->drv && bs->drv->bdrv_supports_persistent_dirty_bitmap) { +return bs->drv->bdrv_supports_persistent_dirty_bitmap (bs); + } +return false; +} + static bool coroutine_fnbdrv_co_can_store_new_dirty_bitmap (BlockDriverState *bs,const char *name,uint32_t granularity,Error **errp)
block/qcow2-bitmap.c +7 -0
@@ -1748,3 +1748,10 @@ fail: name, bdrv_get_device_or_node_name(bs));return false; } + +bool qcow2_supports_persistent_dirty_bitmap (BlockDriverState *bs) +{ +BDRVQcow2State *s = bs->opaque; + +return s->qcow_version >=3 ; +}
block/qcow2.c +2 -0
@@ -5663,6 +5663,8 @@ BlockDriver bdrv_qcow2 = { .bdrv_detach_aio_context = qcow2_detach_aio_context, .bdrv_attach_aio_context = qcow2_attach_aio_context, + .bdrv_supports_persistent_dirty_bitmap = + qcow2_supports_persistent_dirty_bitmap, .bdrv_co_can_store_new_dirty_bitmap = qcow2_co_can_store_new_dirty_bitmap, .bdrv_co_remove_persistent_dirty_bitmap = qcow2_co_remove_persistent_dirty_bitmap,
Signature · 1 line
2.26.2
Re: [PATCH v4 3/9] block: Make it easier to learn which BDS support bitmaps
Quoted 20 lines · Eric Blake
Upcoming patches will enhance bitmap support in qemu-img, but in doing so, it turns out to be nice to suppress output when persistent bitmaps make no sense (such as on a qcow2 v2 image). Add a hook to make this easier to query.
This patch adds a new callback .bdrv_supports_persistent_dirty_bitmap, rather than trying to shoehorn the answer in via existing callbacks. In particular, while it might have been possible to overload .bdrv_co_can_store_new_dirty_bitmap to special-case a NULL input to answer whether any persistent bitmaps are supported, that is at odds with whether a particular bitmap can be stored (for example, even on an image that supports persistent bitmaps but has currently filled up the maximum number of bitmaps, attempts to store another one should fail); and the new functionality doesn't require coroutine safety. Similarly, we could have added one more piece of information to .bdrv_get_info, but then again, most callers to that function tend to already discard extraneous information, and making it a catch-all rather than a series of dedicated scalar queries hasn't really simplified life.
Quoted 3 lines
In the future, when we improve the ability to look up bitmaps through a filter, we will probably also want to teach the block layer to automatically let filters pass this request on through.
Hm. I think that bitmap at filter bs is a valid thing (moreover I have a plan to use it for one issue), so I'm not sure that it's good idea to do any generic logic around bitmaps work through filters, better to always address the exact node you mean..
Quoted 2 lines
Signed-off-by Eric Blake <[email protected]>
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v4 3/9] block: Make it easier to learn which BDS support bitmaps
Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Quoted 3 lines
In the future, when we improve the ability to look up bitmaps through a filter, we will probably also want to teach the block layer to automatically let filters pass this request on through.
Hm. I think that bitmap at filter bs is a valid thing (moreover I have a plan to use it for one issue), so I'm not sure that it's good idea to do any generic logic around bitmaps work through filters, better to always address the exact node you mean..
In the immediate term, the only user of this new function is qemu-img, and qemu-img is not setting up filters, so whether this callback can see through filters is moot (that is, we are always addressing the exact node the user passed on the command line, for whatever bitmap operations qemu-img does). In the long term, being able to see bitmaps through filters is better addressed after we finally include Max's series that include filter handling overall.
Quoted 5 lines
Quoted 3 lines
Signed-off-by Eric Blake <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
Thanks.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v4 4/9 blockdev: Promote several bitmap functions to non-static
In reply to Eric BlakeThe next patch will split blockdev.c, which will require accessing some previously-static functions from more than one .c file. But part of promoting a function to public is picking a naming scheme that does not reek of exposing too many internals (two of the three functions were named starting with 'do_'). To make future code motion easier, perform the function rename and non-static promotion into its own patch.
--- include/block/block_int.h | 12 +++++++++++ blockdev.c | 45 ++++++++++++++++----------------------- 2 files changed, 30 insertions(+), 27 deletions(-)
include/block/block_int.h +12 -0
@@ -1344,4 +1344,16 @@ int coroutine_fn bdrv_co_create_opts_simple(BlockDriver *drv, Error **errp);extern QemuOptsList bdrv_create_opts_simple; +BdrvDirtyBitmap *block_dirty_bitmap_lookup (const char *node, +const char *name, +BlockDriverState **pbs, +Error **errp); +BdrvDirtyBitmap *block_dirty_bitmap_merge (const char *node,const char *target, +BlockDirtyBitmapMergeSourceList *bms, +HBitmap **backup,Error **errp); +BdrvDirtyBitmap *block_dirty_bitmap_remove (const char *node,const char *name, +bool release, +BlockDriverState **bitmap_bs, +Error **errp); +#endif /* BLOCK_INT_H */
blockdev.c +18 -27
95 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH v4 4/9] blockdev: Promote several bitmap functions to non-static
Quoted 10 lines · Eric Blake
The next patch will split
blockdev.c, which will require accessing some previously-static functions from more than one .c file. But part of promoting a function to public is picking a naming scheme that does not reek of exposing too many internals (two of the three functions were named starting with 'do_'). To make future code motion easier, perform the function rename and non-static promotion into its own patch.Signed-off-by Eric Blake<[email protected]>Reviewed-by Max Reitz<[email protected]>
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v4 4/9] blockdev: Promote several bitmap functions to non-static
In reply to Eric Blakeblockdev.c void qmp_block_dirty_bitmap_disable(const char *node, const char *name, · line 2608
Quoted 6 lines · Eric Blake
blockdev.c +3
-1- HBitmap **backup, Error **errp) +BdrvDirtyBitmap *block_dirty_bitmap_merge (const char *node,const char *target, +BlockDirtyBitmapMergeSourceList *bitmaps, +HBitmap **backup,Error **errp) {BlockDriverState *bs;
s/bitmaps/bms/ to match declaration and fit into 80 characters
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v4 4/9] blockdev: Promote several bitmap functions to non-static
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Eric Blake
- HBitmap **backup, Error **errp) +BdrvDirtyBitmap *block_dirty_bitmap_merge(const char *node, const char *target,
+
BlockDirtyBitmapMergeSourceList *bitmaps,
+ HBitmap **backup, Error **errp)
{
BlockDriverState *bs;
Quoted 1 line
s/bitmaps/bms/ to match declaration and fit into 80 characters
Can do, although it has ripple effects to 5/9 (as I wanted that to be pure code motion).
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v4 5/9 blockdev: Split off basic bitmap operations for qemu-img
In reply to Eric BlakeUpcoming patches want to add some basic bitmap manipulation abilities to qemu-img. But blockdev.o is too heavyweight to link into qemu-img (among other things, it would drag in block jobs and transaction support - qemu-img does offline manipulation, where atomicity is less important because there are no concurrent modifications to compete with), so it's time to split off the bare bones of what we will need into a new file block/monitor/bitmap-qmp-cmds.o.
This is sufficient to expose 6 QMP commands for use by qemu-img (add, remove, clear, enable, disable, merge), as well as move the three helper functions touched in the previous patch. Regarding MAINTAINERS, the new file is automatically part of block core, but also makes sense as related to other dirty bitmap files.
--- Makefile.objs | 3 +- block/monitor/bitmap-qmp-cmds.c | 323 ++++++++++++++++++++++++++++++++ blockdev.c | 284 ---------------------------- MAINTAINERS | 1 + block/monitor/Makefile.objs | 1 + 5 files changed, 326 insertions(+), 286 deletions(-) create mode 100644 block/monitor/bitmap-qmp-cmds.c
Makefile.objs +1 -2
@@ -13,9 +13,8 @@ chardev-obj-y = chardev/ authz-obj-y = authz/ -block-obj-y = nbd/ +block-obj-y = block/ block/monitor/ nbd/ scsi/ block-obj-y += block.o blockjob.o job.o -block-obj-y += block/ scsi/ block-obj-y += qemu-io-cmds.o block-obj-$(CONFIG_REPLICATION) += replication.o
block/monitor/bitmap-qmp-cmds.c new file +323 -0
323 lines not on this page · raw message
blockdev.c +0 -284
296 lines not on this page · raw message
MAINTAINERS +1 -0
@@ -2007,6 +2007,7 @@ L: [email protected] S: Supported F: include/qemu/hbitmap.h F: include/block/dirty-bitmap.h +F: block/monitor/bitmap-qmp-cmds.c F: block/dirty-bitmap.c F: block/qcow2-bitmap.c F: migration/block-dirty-bitmap.c
block/monitor/Makefile.objs +1 -0
@@ -1 +1,2 @@ common-obj-y += block-hmp-cmds.o +block-obj-y += bitmap-qmp-cmds.o
Signature · 1 line
2.26.2
Re: [PATCH v4 5/9] blockdev: Split off basic bitmap operations for qemu-img
block/monitor/bitmap-qmp-cmds.c
Quoted 45 lines · Eric Blake
Upcoming patches want to add some basic bitmap manipulation abilities to qemu-img. But blockdev.o is too heavyweight to link into qemu-img (among other things, it would drag in block jobs and transaction support - qemu-img does offline manipulation, where atomicity is less important because there are no concurrent modifications to compete with), so it's time to split off the bare bones of what we will need into a new file
block/monitor/bitmap-qmp-cmds.o.This is sufficient to expose 6 QMP commands for use by qemu-img (add, remove, clear, enable, disable, merge), as well as move the three helper functions touched in the previous patch. Regarding
MAINTAINERS, the new file is automatically part of block core, but also makes sense as related to other dirty bitmap files.Signed-off-by Eric Blake <[email protected]>Reviewed-by Max Reitz <[email protected]>--- Makefile.objs | 3 +- block/monitor/bitmap-qmp-cmds.c | 323 ++++++++++++++++++++++++++++++++ blockdev.c | 284 ---------------------------- MAINTAINERS | 1 + block/monitor/Makefile.objs | 1 + 5 files changed, 326 insertions(+), 286 deletions(-)create mode 100644
block/monitor/bitmap-qmp-cmds.cMakefile.objs +1
-2@@ -13,9 +13,8 @@ chardev-obj-y = chardev/ authz-obj-y = authz/ -block-obj-y = nbd/ +block-obj-y = block/ block/monitor/ nbd/ scsi/ block-obj-y += block.o blockjob.o job.o -block-obj-y += block/ scsi/ block-obj-y += qemu-io-cmds.o block-obj-$(CONFIG_REPLICATION) += replication.oblock/monitor/bitmap-qmp-cmds.c new file +0
-0
Hmm, shouldn't transaction bitmap actions be moved here too? May be, not in these series..
block/monitor/bitmap-qmp-cmds.c line 2
Quoted 3 lines
block/monitor/bitmap-qmp-cmds.c new file +2
-0@@ -0,0 +1,323 @@ +/* +* QEMU host block device bitmaps
A bit conflicts with tha fact that they are not of block-device level but of node-level.
May be just "Block dirty bitmap qmp commands" ?
block/monitor/bitmap-qmp-cmds.c line 4
Quoted 2 lines
block/monitor/bitmap-qmp-cmds.c new file +2
-0+* +* Copyright (c) 2003-2008 Fabrice Bellard
Does it really apply here? block-dirty-bitmap-add was added in 2015.. May be Red Hat and Virtuozzo copyrights would be more appropriate.
Quoted 25 lines
block/monitor/bitmap-qmp-cmds.c new file +25
-0+* +* This work is licensed under the terms of the GNU GPL, version 2 or +* later. See the COPYING file in the top-level directory. +* +* This file incorporates work covered by the following copyright and +* permission notice: +* +* Copyright (c) 2003-2008 Fabrice Bellard +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to deal +* in the Software without restriction, including without limitation the rights +* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +* copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
block/monitor/bitmap-qmp-cmds.c line 39
Quoted 10 lines
block/monitor/bitmap-qmp-cmds.c new file +10
-0+* THE SOFTWARE. +*/ + +#include "qemu/osdep.h" + +#include "sysemu/blockdev.h" +#include "block/block.h" +#include "block/block_int.h" +#include "qapi/qapi-commands-block.h" +#include "qapi/error.h"
compiles for with only four:
#include "qemu/osdep.h"
#include "block/block_int.h"
#include "qapi/qapi-commands-block.h"
#include "qapi/error.h"with at least extra includes dropped:
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v4 5/9] blockdev: Split off basic bitmap operations for qemu-img
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Eric Blake
Upcoming patches want to add some basic bitmap manipulation abilities to qemu-img. But blockdev.o is too heavyweight to link into qemu-img (among other things, it would drag in block jobs and transaction support - qemu-img does offline manipulation, where atomicity is less important because there are no concurrent modifications to compete with), so it's time to split off the bare bones of what we will need into a new file
block/monitor/bitmap-qmp-cmds.o.This is sufficient to expose 6 QMP commands for use by qemu-img (add, remove, clear, enable, disable, merge), as well as move the three helper functions touched in the previous patch. Regarding
MAINTAINERS, the new file is automatically part of block core, but also makes sense as related to other dirty bitmap files.
Quoted 7 lines
Quoted 5 lines
Signed-off-by Eric Blake <[email protected]>Reviewed-by Max Reitz <[email protected]>---
+++
b/block/monitor/bitmap-qmp-cmds.cHmm, shouldn't transaction bitmap actions be moved here too? May be, not in these series..
No, the very reason that we split this file off of blockdev.c is that transactions are NOT needed in qemu-img. You have to have separate .o files for what qemu-img needs, vs. what the rest of qemu proper needs, and transactions are only needed in qemu proper at the moment. If we ever need transactions in Kevin's qemu-storage-daemon but not all of blockdev.c, then we'd need yet another .c file independent from either blockdev.c or this patch's new bitmap-qmp-cmds.c.
Quoted 8 lines
block/monitor/bitmap-qmp-cmds.c line 2
Quoted 4 lines
block/monitor/bitmap-qmp-cmds.c new file +2
-0@@ -0,0 +1,323 @@ +/* +* QEMU host block device bitmaps A bit conflicts with tha fact that they are not of block-device level but of node-level.
May be just "Block dirty bitmap qmp commands" ?
Copy-and-paste from blockdev.c, tweaked by adding one word. Your wording is also fine.
Quoted 5 lines
Review line 4
Quoted 3 lines
(quoted diff) +2
-0+ * + * Copyright (c) 2003-2008 Fabrice BellardDoes it really apply here? block-dirty-bitmap-add was added in 2015.. May be Red Hat and Virtuozzo copyrights would be more appropriate.
When splitting a file, the safest course of action is to preserve ALL copyright from the original file into the new file.
Adding additional copyright lines is okay as part of submitting new functionality, but in this case, I don't feel comfortable adding Red Hat copyright (my split isn't adding new functionality), and I don't have authorization to add Virtuozzo copyright (as that is not my employer).
Quoted 10 lines
block/monitor/bitmap-qmp-cmds.c line 39
Quoted 7 lines
block/monitor/bitmap-qmp-cmds.c new file +7
-0+#include "qemu/osdep.h" + +#include "sysemu/blockdev.h" +#include "block/block.h" +#include "block/block_int.h" +#include "qapi/qapi-commands-block.h" +#include "qapi/error.h" compiles for with only four:
Quoted 4 lines
#include "
qemu/osdep.h"
#include "block/block_int.h"
#include "qapi/qapi-commands-block.h"
#include "qapi/error.h"
Thanks. I blame rebase; an earlier version used a different header in patch 4/9; when I moved things to block_int.h in that patch, I didn't realize that this patch could be improved.
Quoted 4 lines
with at least extra includes dropped:
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v4 6/9 qemu-img: Add bitmap sub-command
In reply to Eric BlakeInclude actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'. A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file). Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.
While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file merges. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go (although you can still use more than one --merge in the command line); in part because qemu-img is offline and therefore atomicity is not an issue.
Upcoming patches will add iotest coverage of these commands while also testing other features.
--- docs/tools/qemu-img.rst | 24 ++++ qemu-img.c | 254 ++++++++++++++++++++++++++++++++++++++++ qemu-img-cmds.hx | 7 ++ 3 files changed, 285 insertions(+)
docs/tools/qemu-img.rst +24 -0
@@ -281,6 +281,30 @@ Command description: For write tests, by default a buffer filled with zeros is written. This can be overridden with a pattern byte specified by *PATTERN*. +.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAP + + Perform one or more modifications of the persistent bitmap *BITMAP* + in the disk image *FILENAME*. The various modifications are: + + ``--add`` to create *BITMAP*, enabled to record future edits. + + ``--remove`` to remove *BITMAP*. + + ``--clear`` to clear *BITMAP*. + + ``--enable`` to change *BITMAP* to start recording future edits. + + ``--disable`` to change *BITMAP* to stop recording future edits. + + ``--merge`` to merge the contents of *SOURCE_BITMAP* into *BITMAP*. + + Additional options include ``-g`` which sets a non-default + *GRANULARITY* for ``--add``, and ``-b`` and ``-F`` which select an + alternative source file for all *SOURCE* bitmaps used by + ``--merge``. + + To see what bitmaps are present in an image, use ``qemu-img info``. + .. option:: check [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [--output=OFMT] [-r [leaks | all]] [-T SRC_CACHE] [-U] FILENAME Perform a consistency check on the disk image *FILENAME*. The command can
qemu-img.c +254 -0
278 lines not on this page · raw message
qemu-img-cmds.hx +7 -0
@@ -20,6 +20,13 @@ DEF("bench", img_bench, SRST .. option:: bench [-c COUNT] [-d DEPTH] [-f FMT] [--flush-interval=FLUSH_INTERVAL] [-i AIO] [-n] [--no-drain] [-o OFFSET] [--pattern=PATTERN] [-q] [-s BUFFER_SIZE] [-S STEP_SIZE] [-t CACHE] [-w] [-U] FILENAME ERST + +DEF("bitmap", img_bitmap, + "bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b source_file [-F source_fmt]] [-g granularity] [--object objectdef] [--image-opts | -f fmt] filename bitmap") +SRST +.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAP +ERST + DEF("check", img_check, "check [--object objectdef] [--image-opts] [-q] [-f fmt] [--output=ofmt] [-r [leaks | all]] [-T src_cache] [-U] filename") SRST
Signature · 1 line
2.26.2
Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command
Quoted 20 lines · Eric Blake
Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'. A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file). Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.
While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file merges. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go (although you can still use more than one --merge in the command line); in part because qemu-img is offline and therefore atomicity is not an issue.
Quoted 5 lines
Upcoming patches will add iotest coverage of these commands while also testing other features.
Signed-off-by Eric Blake <[email protected]>Reviewed-by Max Reitz <[email protected]>
I'm sorry for asking it only now on v4.. But still. Why do we need it? We can instead run qemu binary (or even new qemu-storage-daemon) and just use existing qmp commands. Is there a real benefit in developing qemu-img, maintaining two interfaces for the same thing? Of-course, just run qmp commands from terminal is a lot less comfortable than just a qemu img command.. But may be we need some wrapper, which make it simple to run one qmp command on an image?
It's simple to make a python wrapper working like
qemu-qmp block-dirty-bitmap-add '{node: self, name: bitmap0, persistent: true}' /path/to/x.qcow2
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Quoted 12 lines · Eric Blake
Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'. A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file). Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.
Quoted 1 line
I'm sorry for asking it only now on v4.. But still. Why do we need it?
Ease of use.
Quoted 1 line
We can instead run qemu binary (or even new qemu-storage-daemon) and just use existing qmp commands. Is there a real benefit in developing qemu-img, maintaining two interfaces for the same thing?
If it makes someone's life easier, and is not hard to maintain, then yes. A command line interface that calls into QMP is not hard to maintain. And _I_ certainly found it easier to write iotests with this patch in place, so it already has at least one client.
Quoted 5 lines
Of-course, just run qmp commands from terminal is a lot less comfortable than just a qemu img command.. But may be we need some wrapper, which make it simple to run one qmp command on an image?
It's simple to make a python wrapper working like
qemu-qmp block-dirty-bitmap-add '{node: self, name: bitmap0, persistent: true}' /path/to/x.qcow2
This _IS_ such a wrapper. The whole point of this patch is that it is now simpler to run one (or more) QMP command on an offline image from the command line. Just because I wrote it in C instead of python, and attached it to an existing tool instead of writing a new tool, doesn't change the fact that it is just a wrapper around the existing QMP commands.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command
Quoted 24 lines · Eric Blake
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Quoted 12 lines · Eric Blake
Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'. A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file). Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.
Quoted 1 line
I'm sorry for asking it only now on v4.. But still. Why do we need it?
Ease of use.
Quoted 1 line
We can instead run qemu binary (or even new qemu-storage-daemon) and just use existing qmp commands. Is there a real benefit in developing qemu-img, maintaining two interfaces for the same thing?
If it makes someone's life easier, and is not hard to maintain, then yes. A command line interface that calls into QMP is not hard to maintain. And _I_ certainly found it easier to write iotests with this patch in place, so it already has at least one client.
Quoted 8 lines
Quoted 5 lines
Of-course, just run qmp commands from terminal is a lot less comfortable than just a qemu img command.. But may be we need some wrapper, which make it simple to run one qmp command on an image?
It's simple to make a python wrapper working like
qemu-qmp block-dirty-bitmap-add '{node: self, name: bitmap0, persistent: true}' /path/to/x.qcow2
This _IS_ such a wrapper. The whole point of this patch is that it is now simpler to run one (or more) QMP command on an offline image from the command line. Just because I wrote it in C instead of python, and attached it to an existing tool instead of writing a new tool, doesn't change the fact that it is just a wrapper around the existing QMP commands.
OK, that's right.
The thing that I didn't like is that we have to make cli-to-qapi interface mapping by hand. But I see now that interface you implementing is prepared to make several actions with same bitmap-name, which can't be achieved with some kind of automatic interface matching anyway, so my proposal don't match your needs, sorry for my haste)
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command
In reply to Eric BlakeQuoted 260 lines · Eric Blake
Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'. A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file). Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.
While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file merges. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go (although you can still use more than one --merge in the command line); in part because qemu-img is offline and therefore atomicity is not an issue.
Upcoming patches will add iotest coverage of these commands while also testing other features.
Signed-off-by Eric Blake <[email protected]>Reviewed-by Max Reitz <[email protected]>--- docs/tools/qemu-img.rst | 24 ++++ qemu-img.c | 254 ++++++++++++++++++++++++++++++++++++++++ qemu-img-cmds.hx | 7 ++ 3 files changed, 285 insertions(+)docs/tools/qemu-img.rst +24
-0@@ -281,6 +281,30 @@ Command description: For write tests, by default a buffer filled with zeros is written. This can be overridden with a pattern byte specified by *PATTERN*. +.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAP + + Perform one or more modifications of the persistent bitmap *BITMAP* + in the disk image *FILENAME*. The various modifications are: + + ``--add`` to create *BITMAP*, enabled to record future edits. + + ``--remove`` to remove *BITMAP*. + + ``--clear`` to clear *BITMAP*. + + ``--enable`` to change *BITMAP* to start recording future edits. + + ``--disable`` to change *BITMAP* to stop recording future edits. + + ``--merge`` to merge the contents of *SOURCE_BITMAP* into *BITMAP*. + + Additional options include ``-g`` which sets a non-default + *GRANULARITY* for ``--add``, and ``-b`` and ``-F`` which select an + alternative source file for all *SOURCE* bitmaps used by + ``--merge``. + + To see what bitmaps are present in an image, use ``qemu-img info``. + .. option:: check [--object OBJECTDEF] [--image-opts] [-q] [-f FMT] [--output=OFMT] [-r [leaks | all]] [-T SRC_CACHE] [-U] FILENAME Perform a consistency check on the disk image *FILENAME*. The command canqemu-img.c +165
-0@@ -28,6 +28,7 @@#include "qemu-common.h" #include "qemu-version.h" #include "qapi/error.h" +#include "qapi/qapi-commands-block-core.h" #include "qapi/qapi-visit-block-core.h" #include "qapi/qobject-output-visitor.h" #include "qapi/qmp/qjson.h" @@ -71,6 +72,12 @@ enum {OPTION_SHRINK =266 ,OPTION_SALVAGE =267 ,OPTION_TARGET_IS_ZERO =268 , +OPTION_ADD =269 , +OPTION_REMOVE =270 , +OPTION_CLEAR =271 , +OPTION_ENABLE =272 , +OPTION_DISABLE =273 , + OPTION_MERGE = 274, }; typedef enum OutputFormat {@@ -169,6 +176,14 @@ static void QEMU_NORETURN help(void)" '-n' skips the target volume creation (useful if the volume is created\n" " prior to running qemu-img)\n" "\n" +"Parameters to bitmap subcommand:\n" +" 'bitmap' is the name of the bitmap to manipulate, through one or more\n" +" actions from '--add', '--remove', '--clear', '--enable', '--disable',\n" +" or '--merge source'\n" +" '-g granularity' sets the granularity for '--add' actions\n" +" '-b source' and '-F src_fmt' tell '--merge' actions to find the source\n" +" bitmaps from an alternative file\n" +"\n" "Parameters to check subcommand:\n" " '-r' tries to repair any inconsistencies that are found during the check.\n" " '-r leaks' repairs only cluster leaks, whereas '-r all' fixes all\n" @@ -4461,6 +4476,245 @@ out:return 0 ; } +enum ImgBitmapAct { +BITMAP_ADD , +BITMAP_REMOVE , +BITMAP_CLEAR , +BITMAP_ENABLE , +BITMAP_DISABLE , +BITMAP_MERGE , +}; +typedef struct ImgBitmapAction { +enum ImgBitmapAct act; +const char *src;/* only used for merge */ +QSIMPLEQ_ENTRY (ImgBitmapAction) next; +}ImgBitmapAction ; + +static int img_bitmap (int argc,char **argv) +{ +Error *err =NULL ; +int c, ret =-1 ; +QemuOpts *opts =NULL ; +const char *fmt =NULL , *src_fmt =NULL , *src_filename =NULL ; +const char *filename, *bitmap; +BlockBackend *blk =NULL , *src =NULL ; +BlockDriverState *bs =NULL , *src_bs =NULL ; +bool image_opts = false; +int64_t granularity =0 ; +bool add = false, merge = false; + QSIMPLEQ_HEAD(, ImgBitmapAction) actions; +ImgBitmapAction *act, *act_next; +const char *op; + +QSIMPLEQ_INIT (&actions); + +for (;;) { +static const struct option long_options[] = { + {"help" , no_argument,0 ,'h' }, + {"object" , required_argument,0 ,OPTION_OBJECT }, + {"image-opts" , no_argument,0 ,OPTION_IMAGE_OPTS }, + {"add" , no_argument,0 ,OPTION_ADD }, + {"remove" , no_argument,0 ,OPTION_REMOVE }, + {"clear" , no_argument,0 ,OPTION_CLEAR }, + {"enable" , no_argument,0 ,OPTION_ENABLE }, + {"disable" , no_argument,0 ,OPTION_DISABLE }, + {"merge" , required_argument,0 ,OPTION_MERGE }, + {"granularity" , required_argument,0 ,'g' }, + {"source-file" , required_argument,0 ,'b' }, + {"source-format" , required_argument,0 ,'F' }, + {0 ,0 ,0 ,0 } + }; + c =getopt_long (argc, argv,":b:f:F:g:h" , long_options,NULL ); +if (c ==-1 ) { +break ; + } + +switch (c) { +case ':' : +missing_argument (argv[optind -1 ]); +break ; +case '?' : +unrecognized_option (argv[optind -1 ]); +break ; +case 'h' : +help (); +break ; +case 'b' : + src_filename = optarg; +break ; +case 'f' : + fmt = optarg; +break ; +case 'F' : + src_fmt = optarg; +break ; +case 'g' : + granularity =cvtnum (optarg); +if (granularity <0 ) { +error_report ("Invalid granularity specified" ); +return 1 ; + } +break ; +case OPTION_ADD : + act =g_new0 (ImgBitmapAction,1 ); + act->act =BITMAP_ADD ; +QSIMPLEQ_INSERT_TAIL (&actions, act, next); + add = true; +break ; +case OPTION_REMOVE : + act =g_new0 (ImgBitmapAction,1 ); + act->act =BITMAP_REMOVE ; +QSIMPLEQ_INSERT_TAIL (&actions, act, next); +break ; +case OPTION_CLEAR : + act =g_new0 (ImgBitmapAction,1 ); + act->act =BITMAP_CLEAR ; +QSIMPLEQ_INSERT_TAIL (&actions, act, next); +break ; +case OPTION_ENABLE : + act =g_new0 (ImgBitmapAction,1 ); + act->act =BITMAP_ENABLE ; +QSIMPLEQ_INSERT_TAIL (&actions, act, next); +break ; +case OPTION_DISABLE : + act =g_new0 (ImgBitmapAction,1 ); + act->act =BITMAP_DISABLE ; +QSIMPLEQ_INSERT_TAIL (&actions, act, next); +break ; +case OPTION_MERGE : + act =g_new0 (ImgBitmapAction,1 ); + act->act =BITMAP_MERGE ; + act->src = optarg; +QSIMPLEQ_INSERT_TAIL (&actions, act, next); + merge = true; +break ; +case OPTION_OBJECT : + opts =qemu_opts_parse_noisily (&qemu_object_opts, optarg, true); +if (!opts) { + goto out; + } +break ; +case OPTION_IMAGE_OPTS : + image_opts = true; +break ; + } + } + +if (qemu_opts_foreach (&qemu_object_opts, + user_creatable_add_opts_foreach, + qemu_img_object_print_help, &error_fatal)) { + goto out; + } + +if (QSIMPLEQ_EMPTY (&actions)) { +error_report ("Need at least one of --add, --remove, --clear, " +"--enable, --disable, or --merge" ); + goto out; + } + +if (granularity && !add) { +error_report ("granularity only supported with --add" ); + goto out; + } +if (src_fmt && !src_filename) { +error_report ("-F only supported with -b" ); + goto out; + } +if (src_filename && !merge) { +error_report ("Merge bitmap source file only supported with " +"--merge" ); + goto out; + } +
qemu-img.c out: · line 4638
Quoted 10 lines
qemu-img.c +10
-0+if (optind != argc -2 ) { +error_report ("Expecting filename and bitmap name" ); + goto out; + } + + filename = argv[optind]; + bitmap = argv[optind +1 ]; + + blk =img_open (image_opts, filename, fmt,BDRV_O_RDWR , false, false, + false);
fit in one line
qemu-img.c out: · line 4645
Quoted 7 lines
qemu-img.c +7
-0+if (!blk) { + goto out; + } + bs =blk_bs (blk); +if (src_filename) { + src =img_open (NULL , src_filename, src_fmt,0 , false, false, + false);
s/NULL/false/
also, fit in one line
Quoted 43 lines
qemu-img.c +43
-0+if (!src) { + goto out; + } + src_bs =blk_bs (src); + }else { + src_bs = bs; + } + + QSIMPLEQ_FOREACH_SAFE(act, &actions, next, act_next) { +switch (act->act) { +case BITMAP_ADD : +qmp_block_dirty_bitmap_add (bs->node_name, bitmap, + !!granularity, granularity, true, true, + false, false, &err); + op ="add" ; +break ; +case BITMAP_REMOVE : +qmp_block_dirty_bitmap_remove (bs->node_name, bitmap, &err); + op ="remove" ; +break ; +case BITMAP_CLEAR : +qmp_block_dirty_bitmap_clear (bs->node_name, bitmap, &err); + op ="clear" ; +break ; +case BITMAP_ENABLE : +qmp_block_dirty_bitmap_enable (bs->node_name, bitmap, &err); + op ="enable" ; +break ; +case BITMAP_DISABLE : +qmp_block_dirty_bitmap_disable (bs->node_name, bitmap, &err); + op ="disable" ; +break ; +case BITMAP_MERGE : { +BlockDirtyBitmapMergeSource *merge_src; +BlockDirtyBitmapMergeSourceList *list; + + merge_src =g_new0 (BlockDirtyBitmapMergeSource,1 ); + merge_src->type =QTYPE_QDICT ; + merge_src->u.external.node =g_strdup (src_bs->node_name); + merge_src->u.external.name =g_strdup (act->src); + list =g_new0 (BlockDirtyBitmapMergeSourceList,1 ); + list->value = merge_src; +qmp_block_dirty_bitmap_merge (bs->node_name, bitmap, list, &err);
qemu-img.c out: · line 4698
Quoted 10 lines
qemu-img.c +10
-0+qapi_free_BlockDirtyBitmapMergeSourceList (list); + op ="merge" ; +break ; + } +default : +g_assert_not_reached (); + } + +if (err) { + error_reportf_err(err, "Operation %s on bitmap %s failed",
s/failed/failed: /
qemu-img.c out: · line 4700
Quoted 2 lines
qemu-img.c +2
-0+ op, bitmap); + ret =-1 ;
dead assignment: you never set ret after first initialization to -1.
Quoted 5 lines
qemu-img.c +5
-0+ goto out; + } +g_free (act); + } +
qemu-img.c out: · line 4715
Quoted 10 lines
qemu-img.c +10
-0+ ret =0 ; + + out: +blk_unref (src); +blk_unref (blk); +qemu_opts_del (opts); +if (ret) { +return 1 ; + } +return 0 ;
Hmm, as it's the only usage of ret, you may initialize it to 1 at function start, and here just "return ret;"
Quoted 10 lines
qemu-img.c +2
-0+} +#define C_BS 01#define C_COUNT 02#define C_IF 04qemu-img-cmds.hx +0
-0
qemu-img-cmds.hx DEF("bench", img_bench, · line 27
Quoted 8 lines
qemu-img-cmds.hx +5
-0@@ -20,6 +20,13 @@ DEF("bench", img_bench, SRST .. option:: bench [-c COUNT] [-d DEPTH] [-f FMT] [--flush-interval=FLUSH_INTERVAL] [-i AIO] [-n] [--no-drain] [-o OFFSET] [--pattern=PATTERN] [-q] [-s BUFFER_SIZE] [-S STEP_SIZE] [-t CACHE] [-w] [-U] FILENAME ERST + +DEF("bitmap", img_bitmap, + "bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b source_file [-F source_fmt]] [-g granularity] [--object objectdef] [--image-opts | -f fmt] filename bitmap") +SRST +.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAP
Not about this patch, but it's a pity that we have triple duplications (triplications ?) of such lines..
qemu-img-cmds.hx DEF("bench", img_bench, · line 32
Quoted 6 lines
qemu-img-cmds.hx +2
-0+ERST + DEF("check", img_check, "check [--object objectdef] [--image-opts] [-q] [-f fmt] [--output=ofmt] [-r [leaks | all]] [-T src_cache] [-U] filename") SRST
With at least s/NULL/false/ and s/failed/failed: / (or with all my tiny suggestions):
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
Quoted 20 lines · Eric Blake
Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'. A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file). Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.
While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file merges. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go (although you can still use more than one --merge in the command line); in part because qemu-img is offline and therefore atomicity is not an issue.
Quoted 5 lines
Quoted 3 lines
+
+ blk =img_open(image_opts, filename, fmt, BDRV_O_RDWR, false, false, + false);fit in one line
That line would be exactly 80; I tend to wrap at 79 or earlier rather than exactly on 80.
Quoted 10 lines
Quoted 8 lines
+ if (!blk) {
+ goto out;
+ }
+ bs =blk_bs(blk);
+ if (src_filename) {
+ src =img_open(NULL, src_filename, src_fmt, 0, false, false, + false);s/NULL/false/
D'oh. And yet it still compiles. Fixing.
Quoted 2 lines
also, fit in one line
Yes, this one's shorter. Fixing.
Quoted 10 lines
Quoted 3 lines
+
+ if (err) {
+ error_reportf_err(err, "Operation %s on bitmap %s failed",s/failed/failed: /
Quoted 2 lines
+ op, bitmap);
+ ret = -1;dead assignment: you never set ret after first initialization to -1.
Fixing both.
Quoted 10 lines
Quoted 10 lines
+ goto out;
+ }
+g_free(act);
+ }(quoted diff) +5
-0+ + ret = 0; + + out: + blk_unref(src);
Quoted 8 lines
Quoted 6 lines
+
blk_unref(blk);
+qemu_opts_del(opts);
+ if (ret) {
+ return 1;
+ }
+ return 0;Hmm, as it's the only usage of ret, you may initialize it to 1 at function start, and here just "return ret;"
Yep, done.
Quoted 10 lines
Quoted 8 lines
+DEF("bitmap",
img_bitmap,
+ "bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-bsource_file[-Fsource_fmt]] [-g granularity] [--object objectdef] [--image-opts | -f fmt] filename bitmap") +SRST
+.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-bSOURCE_FILE[-FSOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAPNot about this patch, but it's a pity that we have triple duplications (triplications ?) of such lines..
Yes, it is annoying. But as you say, it's a cleanup for another day, for someone who is interested.
Quoted 3 lines
With at least s/NULL/false/ and s/failed/failed: / (or with all my tiny suggestions):
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
Thanks; squashing in:
qemu-img.c +4 -9
@@ -4493,7 +4493,7 @@ typedef struct ImgBitmapAction {static int img_bitmap (int argc,char **argv) {Error *err =NULL ; -int c, ret =-1 ; +int c, ret =1 ;QemuOpts *opts =NULL ;const char *fmt =NULL , *src_fmt =NULL , *src_filename =NULL ;const char *filename, *bitmap;
@@ -4641,8 +4641,7 @@ static int img_bitmap(int argc, char **argv) } bs =blk_bs (blk);if (src_filename) { - src =img_open (NULL , src_filename, src_fmt,0 , false, false, - false); + src =img_open (false, src_filename, src_fmt,0 , false, false, false);if (!src) { goto out; }
@@ -4695,9 +4694,8 @@ static int img_bitmap(int argc, char **argv) }if (err) { -error_reportf_err (err,"Operation %s on bitmap %s failed" , +error_reportf_err (err,"Operation %s on bitmap %s failed: " , op, bitmap); - ret =-1 ; goto out; }g_free (act);
@@ -4709,10 +4707,7 @@ static int img_bitmap(int argc, char **argv)blk_unref (src);blk_unref (blk);qemu_opts_del (opts); -if (ret) { -return 1 ; - } -return 0 ; +return ret; }#define C_BS 01
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v4 6/9] qemu-img: Add bitmap sub-command
Quoted 136 lines · Eric Blake
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
Quoted 20 lines · Eric Blake
Include actions for --add, --remove, --clear, --enable, --disable, and --merge (note that --clear is a bit of fluff, because the same can be accomplished by removing a bitmap and then adding a new one in its place, but it matches what QMP commands exist). Listing is omitted, because it does not require a bitmap name and because it was already possible with 'qemu-img info'. A single command line can play one or more bitmap commands in sequence on the same bitmap name (although all added bitmaps share the same granularity, and and all merged bitmaps come from the same source file). Merge defaults to other bitmaps in the primary image, but can also be told to merge bitmaps from a distinct image.
While this supports --image-opts for the file being modified, I did not think it worth the extra complexity to support that for the source file in a cross-file merges. Likewise, I chose to have --merge only take a single source rather than following the QMP support for multiple merges in one go (although you can still use more than one --merge in the command line); in part because qemu-img is offline and therefore atomicity is not an issue.
Quoted 5 lines
Quoted 3 lines
+
+ blk =img_open(image_opts, filename, fmt, BDRV_O_RDWR, false, false, + false);fit in one line
That line would be exactly 80; I tend to wrap at 79 or earlier rather than exactly on 80.
Quoted 10 lines
Quoted 8 lines
+ if (!blk) {
+ goto out;
+ }
+ bs =blk_bs(blk);
+ if (src_filename) {
+ src =img_open(NULL, src_filename, src_fmt, 0, false, false, + false);s/NULL/false/
D'oh. And yet it still compiles. Fixing.
Quoted 2 lines
also, fit in one line
Yes, this one's shorter. Fixing.
Quoted 10 lines
Quoted 3 lines
+
+ if (err) {
+ error_reportf_err(err, "Operation %s on bitmap %s failed",s/failed/failed: /
Quoted 2 lines
+ op, bitmap);
+ ret = -1;dead assignment: you never set ret after first initialization to -1.
Fixing both.
Quoted 10 lines
Quoted 10 lines
+ goto out;
+ }
+g_free(act);
+ }(quoted diff) +5
-0+ + ret = 0; + + out: + blk_unref(src);Quoted 8 lines
Quoted 6 lines
+
blk_unref(blk);
+qemu_opts_del(opts);
+ if (ret) {
+ return 1;
+ }
+ return 0;Hmm, as it's the only usage of ret, you may initialize it to 1 at function start, and here just "return ret;"
Yep, done.
Quoted 6 lines
Review
Quoted 4 lines
(quoted diff) +4
-0+DEF("bitmap", img_bitmap, + "bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b source_file [-F source_fmt]] [-g granularity] [--object objectdef] [--image-opts | -f fmt] filename bitmap") +SRST +.. option:: bitmap (--merge SOURCE | --add | --remove | --clear | --enable | --disable)... [-b SOURCE_FILE [-F SOURCE_FMT]] [-g GRANULARITY] [--object OBJECTDEF] [--image-opts | -f FMT] FILENAME BITMAPNot about this patch, but it's a pity that we have triple duplications (triplications ?) of such lines..
Yes, it is annoying. But as you say, it's a cleanup for another day, for someone who is interested.
Quoted 3 lines
With at least s/NULL/false/ and s/failed/failed: / (or with all my tiny suggestions):
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Thanks; squashing in:
qemu-img.c +3
-5@@ -4493,7 +4493,7 @@ typedef struct ImgBitmapAction { static int img_bitmap(int argc, char **argv) { Error *err = NULL; - int c, ret = -1; + int c, ret = 1; QemuOpts *opts = NULL; const char *fmt = NULL, *src_fmt = NULL, *src_filename = NULL; const char *filename, *bitmap;@@ -4641,8 +4641,7 @@ static int img_bitmap(int argc, char **argv) } bs = blk_bs(blk); if (src_filename) { - src = img_open(NULL, src_filename, src_fmt, 0, false, false, - false); + src = img_open(false, src_filename, src_fmt, 0, false, false, false); if (!src) { goto out; }@@ -4695,9 +4694,8 @@ static int img_bitmap(int argc, char **argv) } if (err) { - error_reportf_err(err, "Operation %s on bitmap %s failed", + error_reportf_err(err, "Operation %s on bitmap %s failed: ", op, bitmap); - ret = -1; goto out; } g_free(act);@@ -4709,10 +4707,7 @@ static int img_bitmap(int argc, char **argv) blk_unref(src);
qemu-img.c static int img_bitmap(int argc, char **argv) · line 4713
Quoted 10 lines
qemu-img.c +1
-4blk_unref(blk); qemu_opts_del(opts); - if (ret) { - return 1; - } - return 0; + return ret; } #define C_BS 01
Great, keep my r-b with this.
Signature · 2 lines
Best regards, Vladimir
PATCH v4 7/9 qcow2: Expose bitmaps' size during measure
In reply to Eric BlakeIt's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional QMP field, present when measuring an existing image and output format that both support bitmaps. Update iotest 178 and 190 to updated output, as well as new coverage in 190 demonstrating non-zero values made possible with the recently-added qemu-img bitmap command.
On the command-line side, 'qemu-img measure' gains a new --bitmaps flag. When present, the bitmap size is rolled into the two existing measures (or errors if either the source image or destination format lacks bitmaps); when absent, there is never an error (for back-compat), but the output will instead include a new line item for bitmaps (which you would have to manually add), with that line being omitted in the same cases where passing --bitmaps would error.
The behavior chosen here is symmetrical with the upcoming 'qemu-img convert --bitmaps' being added in the next patch: that is, either both commands will succeed (your qemu-img was new enough to do bitmap manipulations, AND you correctly measured and copied the bitmaps, even if that measurement was 0 because there was nothing to copy) or both fail (either your qemu-img is too old to understand --bitmaps, or it understands it but your choice of images do not support seamless transition of bitmaps because either source, destination, or both lack bitmap support).
The addition of a new field demonstrates why we should always zero-initialize qapi C structs; while the qcow2 driver still fully populates all fields, the raw and crypto drivers had to be tweaked to avoid uninitialized data.
See also: https://bugzilla.redhat.com/1779904
--- docs/tools/qemu-img.rst | 10 +++++- qapi/block-core.json | 15 ++++++--- block/crypto.c | 2 +- block/qcow2.c | 37 +++++++++++++++++++-- block/raw-format.c | 2 +- qemu-img.c | 25 ++++++++++++++ qemu-img-cmds.hx | 4 +-- tests/qemu-iotests/178.out.qcow2 | 16 +++++++++ tests/qemu-iotests/190 | 57 ++++++++++++++++++++++++++++++-- tests/qemu-iotests/190.out | 35 +++++++++++++++++++- 10 files changed, 187 insertions(+), 16 deletions(-)
docs/tools/qemu-img.rst +9 -1
@@ -593,7 +593,7 @@ Command description: For more information, consult ``include/block/block.h`` in QEMU's source code. -.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [-l SNAPSHOT_PARAM] FILENAME] +.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [--bitmaps] [-l SNAPSHOT_PARAM] FILENAME] Calculate the file size required for a new image. This information can be used to size logical volumes or SAN LUNs appropriately for
@@ -616,6 +616,7 @@ Command description: required size: 524288 fully allocated size: 1074069504 + bitmaps: 0 The ``required size`` is the file size of the new image. It may be smaller than the virtual disk size if the image format supports compact representation.
@@ -625,6 +626,13 @@ Command description: occupy with the exception of internal snapshots, dirty bitmaps, vmstate data, and other advanced image format features. + The ``bitmaps size`` is the additional size required if the + destination supports persistent bitmaps, in order to additionally + copy bitmaps in addition to the guest-visible data. If the + ``--bitmaps`` option was in use, the bitmap size is folded into the + required and fully-allocated size for convenience, rather than being + a separate line item. + .. option:: snapshot [--object OBJECTDEF] [--image-opts] [-U] [-q] [-l | -a SNAPSHOT | -c SNAPSHOT | -d SNAPSHOT] FILENAME List, apply, create or delete snapshots in image *FILENAME*.
qapi/block-core.json +10 -5
@@ -633,18 +633,23 @@ # efficiently so file size may be smaller than virtual disk size. # # The values are upper bounds that are guaranteed to fit the new image file. -# Subsequent modification, such as internal snapshot or bitmap creation, may -# require additional space and is not covered here. +# Subsequent modification, such as internal snapshot or further bitmap +# creation, may require additional space and is not covered here. # -# @required: Size required for a new image file, in bytes. +# @required: Size required for a new image file, in bytes, when copying just +# guest-visible contents. # # @fully-allocated: Image file size, in bytes, once data has been written -# to all sectors. +# to all sectors, when copying just guest-visible contents. +# +# @bitmaps: Additional size required if all the top-level bitmap metadata in +# the source image were to be copied to the destination, present +# when the destination supports persistent bitmaps. (since 5.1) # # Since: 2.10 ## { 'struct': 'BlockMeasureInfo', - 'data': {'required': 'int', 'fully-allocated': 'int'} } + 'data': {'required': 'int', 'fully-allocated': 'int', '*bitmaps': 'int'} } ## # @query-block:
block/crypto.c +1 -1
@@ -552,7 +552,7 @@ static BlockMeasureInfo *block_crypto_measure(QemuOpts *opts,* Unallocated blocks are still encrypted so allocation status makes no * difference to the file size. */ - info =g_new (BlockMeasureInfo,1 ); + info =g_new0 (BlockMeasureInfo,1 ); info->fully_allocated = luks_payload_size + size; info->required = luks_payload_size + size;return info;
block/qcow2.c +34 -3
@@ -4721,6 +4721,7 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,PreallocMode prealloc;bool has_backing_file;bool has_luks; +uint64_t bitmaps_size =0 ;/* size occupied by bitmaps in in_bs */ /* Parse image creation options */ cluster_size =qcow2_opt_get_cluster_size_del (opts, &local_err);
@@ -4796,13 +4797,38 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,/* Account for input image */ if (in_bs) { +BdrvDirtyBitmap *bm; +size_t bitmap_dir_size =0 ;int64_t ssize =bdrv_getlength (in_bs); +if (ssize <0 ) {error_setg_errno (&local_err, -ssize,"Unable to get image virtual_size" ); goto err; } + FOR_EACH_DIRTY_BITMAP(in_bs, bm) { +if (bdrv_dirty_bitmap_get_persistence (bm)) { +const char *name =bdrv_dirty_bitmap_name (bm); +uint32_t granularity =bdrv_dirty_bitmap_granularity (bm); +uint64_t bmbits =DIV_ROUND_UP (bdrv_dirty_bitmap_size (bm), + granularity); +uint64_t bmclusters =DIV_ROUND_UP (DIV_ROUND_UP (bmbits, +CHAR_BIT ), + cluster_size); + +/* Assume the entire bitmap is allocated */ + bitmaps_size += bmclusters * cluster_size; +/* Also reserve space for the bitmap table entries */ + bitmaps_size +=ROUND_UP (bmclusters *sizeof (uint64_t ), + cluster_size); +/* And space for contribution to bitmap directory size */ + bitmap_dir_size +=ROUND_UP (strlen (name) +24 , +sizeof (uint64_t )); + } + } + bitmaps_size +=ROUND_UP (bitmap_dir_size, cluster_size); + virtual_size =ROUND_UP (ssize, cluster_size);if (has_backing_file) {
@@ -4849,16 +4875,21 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, required = virtual_size; } - info =g_new (BlockMeasureInfo,1 ); + info =g_new0 (BlockMeasureInfo,1 ); info->fully_allocated =qcow2_calc_prealloc_size (virtual_size, cluster_size,ctz32 (refcount_bits)) + luks_payload_size; -/* Remove data clusters that are not required. This overestimates the +/* +* Remove data clusters that are not required. This overestimates the * required size because metadata needed for the fully allocated file is -* still counted. +* still counted. Show bitmaps only if both source and destination +* would support them. */ info->required = info->fully_allocated - virtual_size + required; + info->has_bitmaps = version >=3 && in_bs && +bdrv_supports_persistent_dirty_bitmap (in_bs); + info->bitmaps = bitmaps_size;return info; err:
block/raw-format.c +1 -1
@@ -346,7 +346,7 @@ static BlockMeasureInfo *raw_measure(QemuOpts *opts, BlockDriverState *in_bs, BDRV_SECTOR_SIZE); } - info =g_new (BlockMeasureInfo,1 ); + info =g_new0 (BlockMeasureInfo,1 ); info->required = required;/* Unallocated sectors count towards the file size in raw images */
qemu-img.c +25 -0
@@ -78,6 +78,7 @@ enum {OPTION_ENABLE =272 ,OPTION_DISABLE =273 ,OPTION_MERGE =274 , + OPTION_BITMAPS = 275, }; typedef enum OutputFormat {
@@ -5096,6 +5097,7 @@ static int img_measure(int argc, char **argv) {"output", required_argument, 0, OPTION_OUTPUT}, {"size", required_argument, 0, OPTION_SIZE}, {"force-share", no_argument, 0, 'U'}, + {"bitmaps", no_argument, 0, OPTION_BITMAPS}, {0, 0, 0, 0} };OutputFormat output_format =OFORMAT_HUMAN ;
@@ -5112,6 +5114,7 @@ static int img_measure(int argc, char **argv)QemuOpts *sn_opts =NULL ;QemuOptsList *create_opts =NULL ;bool image_opts = false; +bool bitmaps = false;uint64_t img_size =UINT64_MAX ;BlockMeasureInfo *info =NULL ;Error *local_err =NULL ;
@@ -5192,6 +5195,9 @@ static int img_measure(int argc, char **argv) img_size = (uint64_t )sval; }break ; +case OPTION_BITMAPS : + bitmaps = true; +break ; } }
@@ -5220,6 +5226,10 @@ static int img_measure(int argc, char **argv)error_report ("Either --size N or one filename must be specified." ); goto out; } +if (!filename && bitmaps) { +error_report ("--bitmaps is only supported with a filename." ); + goto out; + }if (filename) { in_blk = img_open(image_opts, filename, fmt, 0,
@@ -5275,9 +5285,24 @@ static int img_measure(int argc, char **argv) goto out; } +if (bitmaps) { +if (!info->has_bitmaps) { +error_report ("no bitmaps measured, either source or destination " +"format lacks bitmap support" ); + goto out; + }else { + info->required += info->bitmaps; + info->fully_allocated += info->bitmaps; + info->has_bitmaps = false; + } + } +if (output_format ==OFORMAT_HUMAN ) {printf ("required size: %" PRIu64"\n" , info->required);printf ("fully allocated size: %" PRIu64"\n" , info->fully_allocated); +if (info->has_bitmaps) { +printf ("bitmaps size: %" PRIu64"\n" , info->bitmaps); + } }else {dump_json_block_measure_info (info); }
qemu-img-cmds.hx +2 -2
@@ -76,9 +76,9 @@ SRST ERST DEF("measure", img_measure, -"measure [--output=ofmt] [-O output_fmt] [-o options] [--size N | [--object objectdef] [--image-opts] [-f fmt] [-l snapshot_param] filename]") +"measure [--output=ofmt] [-O output_fmt] [-o options] [--size N | [--object objectdef] [--image-opts] [-f fmt] [--bitmaps] [-l snapshot_param] filename]") SRST -.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [-l SNAPSHOT_PARAM] FILENAME] +.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [--bitmaps] [-l SNAPSHOT_PARAM] FILENAME] ERST DEF("snapshot", img_snapshot,
tests/qemu-iotests/178.out.qcow2 +16 -0
@@ -37,6 +37,7 @@ qemu-img: The image size is too large (try using a larger cluster size) Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0 required size: 196608 fully allocated size: 196608 +bitmaps size: 0 converted image file size in bytes: 196608
@@ -45,6 +46,7 @@ converted image file size in bytes: 196608 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 required size: 393216 fully allocated size: 1074135040 +bitmaps size: 0 wrote 512/512 bytes at offset 512 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) wrote 65536/65536 bytes at offset 65536
@@ -53,6 +55,7 @@ wrote 64512/64512 bytes at offset 134217728 63 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) required size: 589824 fully allocated size: 1074135040 +bitmaps size: 0 converted image file size in bytes: 524288
@@ -60,6 +63,7 @@ converted image file size in bytes: 524288 required size: 524288 fully allocated size: 1074135040 +bitmaps size: 0 converted image file size in bytes: 458752
@@ -67,16 +71,19 @@ converted image file size in bytes: 458752 required size: 1074135040 fully allocated size: 1074135040 +bitmaps size: 0 == qcow2 input image and LUKS encryption == required size: 2686976 fully allocated size: 1076232192 +bitmaps size: 0 == qcow2 input image and preallocation (human) == required size: 1074135040 fully allocated size: 1074135040 +bitmaps size: 0 converted image file size in bytes: 1074135040
@@ -87,6 +94,7 @@ wrote 8388608/8388608 bytes at offset 0 8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) required size: 8716288 fully allocated size: 8716288 +bitmaps size: 0 converted image file size in bytes: 8716288
@@ -173,6 +181,7 @@ qemu-img: The image size is too large (try using a larger cluster size) Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=0 { + "bitmaps": 0, "required": 196608, "fully-allocated": 196608 }
@@ -183,6 +192,7 @@ converted image file size in bytes: 196608 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 { + "bitmaps": 0, "required": 393216, "fully-allocated": 1074135040 }
@@ -193,6 +203,7 @@ wrote 65536/65536 bytes at offset 65536 wrote 64512/64512 bytes at offset 134217728 63 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) { + "bitmaps": 0, "required": 589824, "fully-allocated": 1074135040 }
@@ -202,6 +213,7 @@ converted image file size in bytes: 524288 == qcow2 input image with internal snapshot (json) == { + "bitmaps": 0, "required": 524288, "fully-allocated": 1074135040 }
@@ -211,6 +223,7 @@ converted image file size in bytes: 458752 == qcow2 input image and a backing file (json) == { + "bitmaps": 0, "required": 1074135040, "fully-allocated": 1074135040 }
@@ -218,6 +231,7 @@ converted image file size in bytes: 458752 == qcow2 input image and LUKS encryption == { + "bitmaps": 0, "required": 2686976, "fully-allocated": 1076232192 }
@@ -225,6 +239,7 @@ converted image file size in bytes: 458752 == qcow2 input image and preallocation (json) == { + "bitmaps": 0, "required": 1074135040, "fully-allocated": 1074135040 }
@@ -237,6 +252,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=8388608 wrote 8388608/8388608 bytes at offset 0 8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) { + "bitmaps": 0, "required": 8716288, "fully-allocated": 8716288 }
tests/qemu-iotests/190 +55 -2
75 lines not on this page · raw message
tests/qemu-iotests/190.out +34 -1
@@ -1,11 +1,44 @@ QA output created by 190 -== Huge file == +== Huge file without bitmaps == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2199023255552 required size: 2199023255552 fully allocated size: 2199023255552 required size: 335806464 fully allocated size: 2199359062016 +bitmaps size: 0 required size: 18874368 fully allocated size: 2199042129920 +bitmaps size: 0 + +== Huge file with bitmaps == + +qemu-img: --bitmaps is only supported with a filename. +required size: 2199023255552 +fully allocated size: 2199023255552 +qemu-img: no bitmaps measured, either source or destination format lacks bitmap support +required size: 7012352 +fully allocated size: 17170432 +qemu-img: no bitmaps measured, either source or destination format lacks bitmap support +required size: 335806464 +fully allocated size: 2199359062016 +qemu-img: no bitmaps measured, either source or destination format lacks bitmap support + +expected bitmap 537198592 +required size: 335806464 +fully allocated size: 2199359062016 +bitmaps size: 537198592 +required size: 873005056 +fully allocated size: 2199896260608 + +expected bitmap 545259520 +{ + "bitmaps": 545259520, + "required": 18874368, + "fully-allocated": 2199042129920 +} +{ + "required": 564133888, + "fully-allocated": 2199587389440 +} *** done
Signature · 1 line
2.26.2
Re: [PATCH v4 7/9] qcow2: Expose bitmaps' size during measure
Quoted 70 lines · Eric Blake
It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional QMP field, present when measuring an existing image and output format that both support bitmaps. Update iotest 178 and 190 to updated output, as well as new coverage in 190 demonstrating non-zero values made possible with the recently-added qemu-img bitmap command.
On the command-line side, 'qemu-img measure' gains a new --bitmaps flag. When present, the bitmap size is rolled into the two existing measures (or errors if either the source image or destination format lacks bitmaps); when absent, there is never an error (for back-compat), but the output will instead include a new line item for bitmaps (which you would have to manually add), with that line being omitted in the same cases where passing --bitmaps would error.
The behavior chosen here is symmetrical with the upcoming 'qemu-img convert --bitmaps' being added in the next patch: that is, either both commands will succeed (your qemu-img was new enough to do bitmap manipulations, AND you correctly measured and copied the bitmaps, even if that measurement was 0 because there was nothing to copy) or both fail (either your qemu-img is too old to understand --bitmaps, or it understands it but your choice of images do not support seamless transition of bitmaps because either source, destination, or both lack bitmap support).
The addition of a new field demonstrates why we should always zero-initialize qapi C structs; while the qcow2 driver still fully populates all fields, the raw and crypto drivers had to be tweaked to avoid uninitialized data.
See also: https://bugzilla.redhat.com/1779904
Reported-by Nir Soffer <[email protected]>Signed-off-by Eric Blake <[email protected]>--- docs/tools/qemu-img.rst | 10 +++++- qapi/block-core.json | 15 ++++++--- block/crypto.c | 2 +- block/qcow2.c | 37 +++++++++++++++++++-- block/raw-format.c | 2 +- qemu-img.c | 25 ++++++++++++++ qemu-img-cmds.hx | 4 +-- tests/qemu-iotests/178.out.qcow2 | 16 +++++++++ tests/qemu-iotests/190 | 57 ++++++++++++++++++++++++++++++-- tests/qemu-iotests/190.out | 35 +++++++++++++++++++- 10 files changed, 187 insertions(+), 16 deletions(-)docs/tools/qemu-img.rst +2
-1@@ -593,7 +593,7 @@ Command description: For more information, consult ``include/block/block.h`` in QEMU's source code. -.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [-l SNAPSHOT_PARAM] FILENAME] +.. option:: measure [--output=OFMT] [-O OUTPUT_FMT] [-o OPTIONS] [--size N | [--object OBJECTDEF] [--image-opts] [-f FMT] [--bitmaps] [-l SNAPSHOT_PARAM] FILENAME] Calculate the file size required for a new image. This information can be used to size logical volumes or SAN LUNs appropriately for@@ -616,6 +616,7 @@ Command description: required size: 524288 fully allocated size: 1074069504 + bitmaps: 0 The ``required size`` is the file size of the new image. It may be smaller than the virtual disk size if the image format supports compact representation.
docs/tools/qemu-img.rst Command description: · line 629
Quoted 4 lines
docs/tools/qemu-img.rst +1
-0@@ -625,6 +626,13 @@ Command description: occupy with the exception of internal snapshots, dirty bitmaps, vmstate data, and other advanced image format features. + The ``bitmaps size`` is the additional size required if the
you called it "bitmaps" in example output above. Should it be consistent? Either "``bitmaps``" here, or "bitmaps size: 0" above?
Quoted 19 lines
docs/tools/qemu-img.rst +6
-0+ destination supports persistent bitmaps, in order to additionally + copy bitmaps in addition to the guest-visible data. If the + ``--bitmaps`` option was in use, the bitmap size is folded into the + required and fully-allocated size for convenience, rather than being + a separate line item. + .. option:: snapshot [--object OBJECTDEF] [--image-opts] [-U] [-q] [-l | -a SNAPSHOT | -c SNAPSHOT | -d SNAPSHOT] FILENAME List, apply, create or delete snapshots in image *FILENAME*.qapi/block-core.json +0
-2@@ -633,18 +633,23 @@ # efficiently so file size may be smaller than virtual disk size. # # The values are upper bounds that are guaranteed to fit the new image file. -# Subsequent modification, such as internal snapshot or bitmap creation, may -# require additional space and is not covered here.
qapi/block-core.json line 643
Quoted 10 lines
qapi/block-core.json +5
-2+# Subsequent modification, such as internal snapshot or further bitmap +# creation, may require additional space and is not covered here. # -# @required: Size required for a new image file, in bytes. +# @required: Size required for a new image file, in bytes, when copying just +# guest-visible contents. # # @fully-allocated: Image file size, in bytes, once data has been written -# to all sectors. +# to all sectors, when copying just guest-visible contents.
"copying just guest-visible" sounds like something less than "all fully-allocated sectors".. But I don't have better suggestion.. Just, "not including bitmaps" sounds weird too.
Quoted 60 lines
qapi/block-core.json +5
-1+# +# @bitmaps: Additional size required if all the top-level bitmap metadata in +# the source image were to be copied to the destination, present +# when the destination supports persistent bitmaps. (since 5.1) # # Since: 2.10 ## { 'struct': 'BlockMeasureInfo', - 'data': {'required': 'int', 'fully-allocated': 'int'} } + 'data': {'required': 'int', 'fully-allocated': 'int', '*bitmaps': 'int'} } ## # @query-block:block/crypto.c +1
-1@@ -552,7 +552,7 @@ static BlockMeasureInfo *block_crypto_measure(QemuOpts *opts,* Unallocated blocks are still encrypted so allocation status makes no * difference to the file size. */ - info =g_new (BlockMeasureInfo,1 ); + info =g_new0 (BlockMeasureInfo,1 ); info->fully_allocated = luks_payload_size + size; info->required = luks_payload_size + size;return info;block/qcow2.c +12
-0@@ -4721,6 +4721,7 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,PreallocMode prealloc;bool has_backing_file;bool has_luks; +uint64_t bitmaps_size =0 ;/* size occupied by bitmaps in in_bs */ /* Parse image creation options */ cluster_size =qcow2_opt_get_cluster_size_del (opts, &local_err);@@ -4796,13 +4797,38 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs,/* Account for input image */ if (in_bs) { +BdrvDirtyBitmap *bm; +size_t bitmap_dir_size =0 ;int64_t ssize =bdrv_getlength (in_bs); +if (ssize <0 ) {error_setg_errno (&local_err, -ssize,"Unable to get image virtual_size" ); goto err; } + FOR_EACH_DIRTY_BITMAP(in_bs, bm) { +if (bdrv_dirty_bitmap_get_persistence (bm)) { +const char *name =bdrv_dirty_bitmap_name (bm); +uint32_t granularity =bdrv_dirty_bitmap_granularity (bm); +uint64_t bmbits =DIV_ROUND_UP (bdrv_dirty_bitmap_size (bm), + granularity); + uint64_t bmclusters = DIV_ROUND_UP(DIV_ROUND_UP(bmbits, + CHAR_BIT),
block/qcow2.c static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, · line 4827
Quoted 10 lines
block/qcow2.c +10
-0+ cluster_size); + +/* Assume the entire bitmap is allocated */ + bitmaps_size += bmclusters * cluster_size; +/* Also reserve space for the bitmap table entries */ + bitmaps_size +=ROUND_UP (bmclusters *sizeof (uint64_t ), + cluster_size); +/* And space for contribution to bitmap directory size */ + bitmap_dir_size +=ROUND_UP (strlen (name) +24 , +sizeof (uint64_t ));
Could we instead reuse code from qcow2_co_can_store_new_dirty_bitmap(), which calls calc_dir_entry_size() for this thing?
Possibly, make a function qcow2_measure_bitmaps in block/qcow2-bitmaps.c with this FOR_EACH? All details about qcow2 bitmap structures sounds better in block/qcow2-bitmaps.c
Quoted 19 lines
block/qcow2.c +6
-2+ } + } + bitmaps_size +=ROUND_UP (bitmap_dir_size, cluster_size); + virtual_size =ROUND_UP (ssize, cluster_size);if (has_backing_file) {@@ -4849,16 +4875,21 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, required = virtual_size; } - info =g_new (BlockMeasureInfo,1 ); + info =g_new0 (BlockMeasureInfo,1 ); info->fully_allocated =qcow2_calc_prealloc_size (virtual_size, cluster_size,ctz32 (refcount_bits)) + luks_payload_size; -/* Remove data clusters that are not required. This overestimates the +/*
block/qcow2.c static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, · line 4892
Quoted 10 lines
block/qcow2.c +6
-1+* Remove data clusters that are not required. This overestimates the * required size because metadata needed for the fully allocated file is -* still counted. +* still counted. Show bitmaps only if both source and destination +* would support them. */ info->required = info->fully_allocated - virtual_size + required; + info->has_bitmaps = version >=3 && in_bs && +bdrv_supports_persistent_dirty_bitmap (in_bs); + info->bitmaps = bitmaps_size;
AFAIK, in QAPI, if has_<something> field is false, than <something> must be zero. Maybe, it's only about nested structured fields, not about simple numbers, but I think it's better keep bitmaps 0 in case when has_bitmaps is false.
Also, it seems a bit better to check version earlier, and don't do all the calculations, if we are not going to use them.. But it's a rare backward-compatibility case, I don't care.
Quoted 68 lines
block/qcow2.c +0
-0return info; err:block/raw-format.c +1
-1@@ -346,7 +346,7 @@ static BlockMeasureInfo *raw_measure(QemuOpts *opts, BlockDriverState *in_bs, BDRV_SECTOR_SIZE); } - info =g_new (BlockMeasureInfo,1 ); + info =g_new0 (BlockMeasureInfo,1 ); info->required = required;/* Unallocated sectors count towards the file size in raw images */ qemu-img.c +10
-0@@ -78,6 +78,7 @@ enum {OPTION_ENABLE =272 ,OPTION_DISABLE =273 ,OPTION_MERGE =274 , + OPTION_BITMAPS = 275, }; typedef enum OutputFormat {@@ -5096,6 +5097,7 @@ static int img_measure(int argc, char **argv) {"output", required_argument, 0, OPTION_OUTPUT}, {"size", required_argument, 0, OPTION_SIZE}, {"force-share", no_argument, 0, 'U'}, + {"bitmaps", no_argument, 0, OPTION_BITMAPS}, {0, 0, 0, 0} };OutputFormat output_format =OFORMAT_HUMAN ;@@ -5112,6 +5114,7 @@ static int img_measure(int argc, char **argv)QemuOpts *sn_opts =NULL ;QemuOptsList *create_opts =NULL ;bool image_opts = false; +bool bitmaps = false;uint64_t img_size =UINT64_MAX ;BlockMeasureInfo *info =NULL ;Error *local_err =NULL ;@@ -5192,6 +5195,9 @@ static int img_measure(int argc, char **argv) img_size = (uint64_t )sval; }break ; +case OPTION_BITMAPS : + bitmaps = true; +break ; } }@@ -5220,6 +5226,10 @@ static int img_measure(int argc, char **argv)error_report ("Either --size N or one filename must be specified." ); goto out; } +if (!filename && bitmaps) { +error_report ("--bitmaps is only supported with a filename." ); + goto out; + }if (filename) { in_blk = img_open(image_opts, filename, fmt, 0,@@ -5275,9 +5285,24 @@ static int img_measure(int argc, char **argv) goto out; }
qemu-img.c static int img_measure(int argc, char **argv) · line 5296
Quoted 10 lines
qemu-img.c +9
-0+if (bitmaps) { +if (!info->has_bitmaps) { +error_report ("no bitmaps measured, either source or destination " +"format lacks bitmap support" ); + goto out; + }else { + info->required += info->bitmaps; + info->fully_allocated += info->bitmaps; + info->has_bitmaps = false;
And here, I think better to zero info->bitmaps as well.
Quoted 12 lines
qemu-img.c +6
-0+ } + } +if (output_format ==OFORMAT_HUMAN ) {printf ("required size: %" PRIu64"\n" , info->required);printf ("fully allocated size: %" PRIu64"\n" , info->fully_allocated); +if (info->has_bitmaps) { +printf ("bitmaps size: %" PRIu64"\n" , info->bitmaps); + } }else {dump_json_block_measure_info (info); }
[..]
Quoted 23 lines
tests/qemu-iotests/190 +2
-2@@ -2,7 +2,7 @@ # # qemu-img measure sub-command tests on huge qcow2 files # -# Copyright (C) 2017 Red Hat, Inc. +# Copyright (C) 2017-2020 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by@@ -42,7 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -echo "== Huge file ==" +echo "== Huge file without bitmaps ==" echo _make_test_img -o 'cluster_size=2M' 2T@@ -51,6 +51,59 @@ $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" $QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG" $QEMU_IMG measure -O qcow2 -o cluster_size=2M -f qcow2 "$TEST_IMG"
tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 62
Quoted 10 lines
tests/qemu-iotests/190 +9
-0+echo +echo "== Huge file with bitmaps ==" +echo + +$QEMU_IMG bitmap --add --granularity 512 -f qcow2 "$TEST_IMG" b1 +$QEMU_IMG bitmap --add -g 2M -f qcow2 "$TEST_IMG" b2 + +# No bitmap without a source +$QEMU_IMG measure --bitmaps -O qcow2 --size 10M
should this be ored to 'echo "unexpected success"' as following failures?
Quoted 12 lines
tests/qemu-iotests/190 +12
-0+# No bitmap output, since raw does not support it +$QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" || + echo "unexpected failure" +$QEMU_IMG measure --bitmaps -O raw -f qcow2 "$TEST_IMG" && + echo "unexpected success" +# No bitmap output, since no bitmaps on raw source +$QEMU_IMG measure -O qcow2 -f raw "$TEST_IMG" || + echo "unexpected failure" +$QEMU_IMG measure --bitmaps -O qcow2 -f raw "$TEST_IMG" && + echo "unexpected success" +# No bitmap output, since v2 does not support it +$QEMU_IMG measure -O qcow2 -o compat=0.10 -f qcow2 "$TEST_IMG" ||
tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 84
Quoted 10 lines
tests/qemu-iotests/190 +10
-0+ echo "unexpected failure" +$QEMU_IMG measure --bitmaps -O qcow2 -o compat=0.10 -f qcow2 "$TEST_IMG" && + echo "unexpected success" + +# Compute expected output: +echo +val2T=$((2*1024*1024*1024*1024)) +cluster=$((64*1024)) +b1clusters=$(( (val2T/512/8 + cluster - 1) / cluster )) +b2clusters=$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster ))
comment on the following calculations won't hurt, at least something like
"bitmap clusters + bitmap tables + bitmaps directory"
Quoted 38 lines
tests/qemu-iotests/190 +22
-0+echo expected bitmap $((b1clusters * cluster + + (b1clusters * 8 + cluster - 1) / cluster * cluster + + b2clusters * cluster + + (b2clusters * 8 + cluster - 1) / cluster * cluster + + cluster)) +$QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG" +$QEMU_IMG measure --bitmaps -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG" + +# Compute expected output: +echo +cluster=$((2*1024*1024)) +b1clusters=$(( (val2T/512/8 + cluster - 1) / cluster )) +b2clusters=$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster )) +echo expected bitmap $((b1clusters * cluster + + (b1clusters * 8 + cluster - 1) / cluster * cluster + + b2clusters * cluster + + (b2clusters * 8 + cluster - 1) / cluster * cluster + + cluster)) +$QEMU_IMG measure --output=json -O qcow2 -o cluster_size=2M -f qcow2 "$TEST_IMG" +$QEMU_IMG measure --output=json --bitmaps -O qcow2 -o cluster_size=2M \ + -f qcow2 "$TEST_IMG" + # success, all done echo "*** done" rm -f $seq.fulltests/qemu-iotests/190.out +1
-1@@ -1,11 +1,44 @@ QA output created by 190 -== Huge file == +== Huge file without bitmaps == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2199023255552 required size: 2199023255552 fully allocated size: 2199023255552 required size: 335806464
tests/qemu-iotests/190.out line 17
Quoted 10 lines
tests/qemu-iotests/190.out +7
-0fully allocated size: 2199359062016 +bitmaps size: 0 required size: 18874368 fully allocated size: 2199042129920 +bitmaps size: 0 + +== Huge file with bitmaps == + +qemu-img: --bitmaps is only supported with a filename. +required size: 2199023255552> +fully allocated size: 2199023255552
- same as for "without bitmaps" sections, good
tests/qemu-iotests/190.out line 24
Quoted 6 lines
tests/qemu-iotests/190.out +6
-0+qemu-img: no bitmaps measured, either source or destination format lacks bitmap support +required size: 7012352 +fully allocated size: 17170432 +qemu-img: no bitmaps measured, either source or destination format lacks bitmap support +required size: 335806464 +fully allocated size: 2199359062016
matches "without bitmaps" section (hmm, v2 image needs same space? Ok)
tests/qemu-iotests/190.out line 29
Quoted 5 lines
tests/qemu-iotests/190.out +5
-0+qemu-img: no bitmaps measured, either source or destination format lacks bitmap support + +expected bitmap 537198592 +required size: 335806464 +fully allocated size: 2199359062016
matches "without bitmaps" section.
tests/qemu-iotests/190.out line 30
Quoted 1 line
tests/qemu-iotests/190.out +1
-0+bitmaps size: 537198592
match expectations
tests/qemu-iotests/190.out line 32
Quoted 2 lines
tests/qemu-iotests/190.out +2
-0+required size: 873005056 +fully allocated size: 2199896260608
matches "without bitmaps" section + expected
tests/qemu-iotests/190.out line 36
Quoted 4 lines
tests/qemu-iotests/190.out +4
-0+ +expected bitmap 545259520 +{ + "bitmaps": 545259520,
match expected
tests/qemu-iotests/190.out line 38
Quoted 2 lines
tests/qemu-iotests/190.out +2
-0+ "required": 18874368, + "fully-allocated": 2199042129920
matches "without bitmaps" section.
tests/qemu-iotests/190.out line 43
Quoted 5 lines
tests/qemu-iotests/190.out +5
-0+} +{ + "required": 564133888, + "fully-allocated": 2199587389440 +}
matches "without bitmaps" section + expected
Quoted 2 lines
tests/qemu-iotests/190.out +0
-0*** done
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v4 7/9] qcow2: Expose bitmaps' size during measure
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines · Eric Blake
It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional QMP field, present when measuring an existing image and output format that both support bitmaps. Update iotest 178 and 190 to updated output, as well as new coverage in 190 demonstrating non-zero values made possible with the recently-added qemu-img bitmap command.
(quoted diff) +0
-0@@ -616,6 +616,7 @@ Command description:required size: 524288
fully allocated size: 1074069504
+ bitmaps: 0The ``required size`` is the file size of the new image. It may be smaller than the virtual disk size if the image format supports compact representation.
docs/tools/qemu-img.rst +0
-0@@ -625,6 +626,13 @@ Command description: occupy with the exception of internal snapshots, dirty bitmaps, vmstate data,and other advanced image format features.
Quoted 3 lines
Quoted 1 line
+ The ``bitmaps size`` is the additional size required if the
you called it "bitmaps" in example output above. Should it be consistent? Either "``bitmaps``" here, or "bitmaps size: 0" above?
"bitmaps size: 0" is better. Will fix the description above.
Quoted 15 lines
Quoted 15 lines
+++
b/qapi/block-core.jsonqapi/block-core.json +0
-1@@ -633,18 +633,23 @@ # efficiently so file size may be smaller than virtual disk size. # # The values are upper bounds that are guaranteed to fit the new image file. -# Subsequent modification, such as internal snapshot or bitmapcreation, may
qapi/block-core.json +2
-1-# require additional space and is not covered here. +# Subsequent modification, such as internal snapshot or further bitmap +# creation, may require additional space and is not covered here.#
(quoted diff) +1
-1-# @required: Size required for a new image file, in bytes. +# @required: Size required for a new image file, in bytes, whencopying just
+# guest-visible contents.
Quoted 8 lines
Quoted 5 lines
#
# @fully-allocated: Image file size, in bytes, once data has been written(quoted diff) +1
-1-# to all sectors. +# to all sectors, when copying just guest-visiblecontents.
"copying just guest-visible" sounds like something less than "all fully-allocated sectors".. But I don't have better suggestion.. Just, "not including bitmaps" sounds weird too.
If we ever add support for copying internal snapshots, that would not be included either. Maybe "copying just allocated guest-visible contents" for @required, and no change to the wording for @fully-allocated.
Quoted 27 lines
Quoted 23 lines
(quoted diff) +16
-0@@ -4796,13 +4797,38 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, + FOR_EACH_DIRTY_BITMAP(in_bs, bm) { + if (bdrv_dirty_bitmap_get_persistence(bm)) { + const char *name = bdrv_dirty_bitmap_name(bm); + uint32_t granularity = bdrv_dirty_bitmap_granularity(bm); + uint64_t bmbits = DIV_ROUND_UP(bdrv_dirty_bitmap_size(bm), + granularity); + uint64_t bmclusters = DIV_ROUND_UP(DIV_ROUND_UP(bmbits, + CHAR_BIT), + cluster_size); + + /* Assume the entire bitmap is allocated */ + bitmaps_size += bmclusters * cluster_size; + /* Also reserve space for the bitmap table entries */ + bitmaps_size += ROUND_UP(bmclusters * sizeof(uint64_t), + cluster_size); + /* And space for contribution to bitmap directoryQuoted 3 lines
size */
+bitmap_dir_size+= ROUND_UP(strlen(name) + 24, +sizeof(uint64_t));
Quoted 2 lines
Could we instead reuse code from
qcow2_co_can_store_new_dirty_bitmap(), which callscalc_dir_entry_size()for this thing?
Possibly, make a functionqcow2_measure_bitmapsinblock/qcow2-bitmaps.cwith thisFOR_EACH? All details about qcow2 bitmap structures sounds better inblock/qcow2-bitmaps.c
Could do. Sounds like I'm better off submitting a v5 for this patch, although I'll go ahead and stage 1-6 for pull request today to minimize future rebase churn.
Quoted 5 lines
Quoted 3 lines
+
info->has_bitmaps= version >= 3 &&in_bs&& +bdrv_supports_persistent_dirty_bitmap(in_bs); +info->bitmaps=bitmaps_size;AFAIK, in QAPI, if has_<something> field is false, than <something> must be zero. Maybe, it's only about nested structured fields, not about simple numbers, but I think it's better keep bitmaps 0 in case when
has_bitmapsis false.
During creation (including when parsing QMP from the user over the monitor), everything is indeed guaranteed to be zero-initialized. But we don't have any requirement that things remain zero-initialized even when has_FOO is false; at the same time, it's easy enough to make this code conditional.
Quoted 2 lines
Also, it seems a bit better to check version earlier, and don't do all the calculations, if we are not going to use them.. But it's a rare backward-compatibility case, I don't care.
I'll see how easy or hard it is for my v5 patch.
Quoted 4 lines
Quoted 4 lines
qemu-img.c +0
-0@@ -5275,9 +5285,24 @@ static int img_measure(int argc, char **argv) goto out; }
Quoted 12 lines
Quoted 10 lines
qemu-img.c +3
-0+ if (bitmaps) { + if (!info->has_bitmaps) { + error_report("no bitmaps measured, either source ordestination "
+ "format lacks bitmap support"); + goto out;
+ } else {
+info->required+=info->bitmaps;
+info->fully_allocated+=info->bitmaps; +info->has_bitmaps= false;And here, I think better to zero
info->bitmapsas well.
Here, the object is going to be subsequently freed; I'm less worried about wasting time doing local cleanups than I am about the earlier case about letting an object escape immediate scope in a different state than the usual preconditions.
Quoted 8 lines
tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 62
Quoted 5 lines
+$QEMU_IMG bitmap --add --granularity 512 -f qcow2 "$TEST_IMG" b1 +$QEMU_IMG bitmap --add -g 2M -f qcow2 "$TEST_IMG" b2
tests/qemu-iotests/190 +3
-0+ +# No bitmap without a source +$QEMU_IMG measure --bitmaps -O qcow2 --size 10Mshould this be ored to 'echo "unexpected success"' as following failures?
Can't hurt.
Quoted 9 lines
tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 84
Quoted 6 lines
tests/qemu-iotests/190 +6
-0+# Compute expected output: +echo +val2T=$((2*1024*1024*1024*1024)) +cluster=$((64*1024)) +b1clusters=$(( (val2T/512/8 + cluster - 1) / cluster )) +b2clusters=$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster ))comment on the following calculations won't hurt, at least something like
"bitmap clusters + bitmap tables + bitmaps directory"
Sure.
Quoted 10 lines
Quoted 10 lines
+echo expected bitmap $((b1clusters * cluster +
+ (b1clusters * 8 + cluster - 1) / cluster * cluster + + b2clusters * cluster +
+ (b2clusters * 8 + cluster - 1) / cluster * cluster + + cluster))
+$QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG" +$QEMU_IMG measure --bitmaps -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG"
+
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v4 7/9] qcow2: Expose bitmaps' size during measure
Quoted 50 lines · Eric Blake
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
Quoted 21 lines · Eric Blake
It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional QMP field, present when measuring an existing image and output format that both support bitmaps. Update iotest 178 and 190 to updated output, as well as new coverage in 190 demonstrating non-zero values made possible with the recently-added qemu-img bitmap command.
(quoted diff) +0
-0@@ -616,6 +616,7 @@ Command description:required size: 524288
fully allocated size: 1074069504
+ bitmaps: 0The ``required size`` is the file size of the new image. It may be smaller than the virtual disk size if the image format supports compact representation.
docs/tools/qemu-img.rst +0
-0@@ -625,6 +626,13 @@ Command description: occupy with the exception of internal snapshots, dirty bitmaps, vmstate data,and other advanced image format features.
Quoted 3 lines
Quoted 1 line
+ The ``bitmaps size`` is the additional size required if the
you called it "bitmaps" in example output above. Should it be consistent? Either "``bitmaps``" here, or "bitmaps size: 0" above?
"bitmaps size: 0" is better. Will fix the description above.
Quoted 12 lines
Quoted 12 lines
+++
b/qapi/block-core.jsonqapi/block-core.json +3
-3@@ -633,18 +633,23 @@ # efficiently so file size may be smaller than virtual disk size. # # The values are upper bounds that are guaranteed to fit the new image file. -# Subsequent modification, such as internal snapshot or bitmap creation, may -# require additional space and is not covered here. +# Subsequent modification, such as internal snapshot or further bitmap +# creation, may require additional space and is not covered here. # -# @required: Size required for a new image file, in bytes. +# @required: Size required for a new image file, in bytes, when copying justQuoted 8 lines
qapi/block-core.json line 643
Quoted 5 lines
qapi/block-core.json +2
-1+# guest-visible contents. # # @fully-allocated: Image file size, in bytes, once data has been written -# to all sectors. +# to all sectors, when copying just guest-visible contents."copying just guest-visible" sounds like something less than "all fully-allocated sectors".. But I don't have better suggestion.. Just, "not including bitmaps" sounds weird too.
Quoted 1 line
If we ever add support for copying internal snapshots, that would not be included either. Maybe "copying just allocated guest-visible contents" for @required, and no change to the wording for @fully-allocated.
Not sure is it better or not, so, I'm OK with any variant.
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
block/qcow2.c +18
-0@@ -4796,13 +4797,38 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, + FOR_EACH_DIRTY_BITMAP(in_bs, bm) { + if (bdrv_dirty_bitmap_get_persistence(bm)) { + const char *name = bdrv_dirty_bitmap_name(bm); + uint32_t granularity = bdrv_dirty_bitmap_granularity(bm); + uint64_t bmbits = DIV_ROUND_UP(bdrv_dirty_bitmap_size(bm), + granularity); + uint64_t bmclusters = DIV_ROUND_UP(DIV_ROUND_UP(bmbits, +CHAR_BIT ), + cluster_size); + + /* Assume the entire bitmap is allocated */ + bitmaps_size += bmclusters * cluster_size; + /* Also reserve space for the bitmap table entries */ + bitmaps_size += ROUND_UP(bmclusters * sizeof(uint64_t), + cluster_size); + /* And space for contribution to bitmap directory size */ + bitmap_dir_size += ROUND_UP(strlen(name) + 24, + sizeof(uint64_t));
Quoted 4 lines
Quoted 2 lines
Could we instead reuse code from
qcow2_co_can_store_new_dirty_bitmap(), which callscalc_dir_entry_size()for this thing?
Possibly, make a functionqcow2_measure_bitmapsinblock/qcow2-bitmaps.cwith thisFOR_EACH? All details about qcow2 bitmap structures sounds better inblock/qcow2-bitmaps.cCould do. Sounds like I'm better off submitting a v5 for this patch, although I'll go ahead and stage 1-6 for pull request today to minimize future rebase churn.
Thanks!
Quoted 69 lines
Quoted 7 lines
block/qcow2.c static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, · line 4892
Quoted 5 lines
block/qcow2.c +3
-0+ info->has_bitmaps = version >= 3 && in_bs && + bdrv_supports_persistent_dirty_bitmap(in_bs); + info->bitmaps = bitmaps_size;AFAIK, in QAPI, if has_<something> field is false, than <something> must be zero. Maybe, it's only about nested structured fields, not about simple numbers, but I think it's better keep bitmaps 0 in case when
has_bitmapsis false.During creation (including when parsing QMP from the user over the monitor), everything is indeed guaranteed to be zero-initialized. But we don't have any requirement that things remain zero-initialized even when
has_FOOis false; at the same time, it's easy enough to make this code conditional.Quoted 2 lines
Also, it seems a bit better to check version earlier, and don't do all the calculations, if we are not going to use them.. But it's a rare backward-compatibility case, I don't care.
I'll see how easy or hard it is for my v5 patch.
Quoted 4 lines
Quoted 4 lines
qemu-img.c +0
-0@@ -5275,9 +5285,24 @@ static int img_measure(int argc, char **argv) goto out; }Quoted 11 lines
qemu-img.c static int img_measure(int argc, char **argv) · line 5296
Quoted 9 lines
qemu-img.c +9
-0+ if (bitmaps) { + if (!info->has_bitmaps) { + error_report("no bitmaps measured, either source or destination " + "format lacks bitmap support"); + goto out; + } else { + info->required += info->bitmaps; + info->fully_allocated += info->bitmaps; + info->has_bitmaps = false;And here, I think better to zero
info->bitmapsas well.Here, the object is going to be subsequently freed; I'm less worried about wasting time doing local cleanups than I am about the earlier case about letting an object escape immediate scope in a different state than the usual preconditions.
Quoted 8 lines
tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 62
Quoted 5 lines
tests/qemu-iotests/190 +5
-0+$QEMU_IMG bitmap --add --granularity 512 -f qcow2 "$TEST_IMG" b1 +$QEMU_IMG bitmap --add -g 2M -f qcow2 "$TEST_IMG" b2 + +# No bitmap without a source +$QEMU_IMG measure --bitmaps -O qcow2 --size 10Mshould this be ored to 'echo "unexpected success"' as following failures?
Can't hurt.
Quoted 9 lines
tests/qemu-iotests/190 $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" · line 84
Quoted 6 lines
tests/qemu-iotests/190 +6
-0+# Compute expected output: +echo +val2T=$((2*1024*1024*1024*1024)) +cluster=$((64*1024)) +b1clusters=$(( (val2T/512/8 + cluster - 1) / cluster )) +b2clusters=$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster ))comment on the following calculations won't hurt, at least something like "bitmap clusters + bitmap tables + bitmaps directory"
Sure.
Quoted 9 lines
Quoted 9 lines
tests/qemu-iotests/190 +8
-0+echo expected bitmap $((b1clusters * cluster + + (b1clusters * 8 + cluster - 1) / cluster * cluster + + b2clusters * cluster + + (b2clusters * 8 + cluster - 1) / cluster * cluster + + cluster)) +$QEMU_IMG measure -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG" +$QEMU_IMG measure --bitmaps -O qcow2 -o cluster_size=64k -f qcow2 "$TEST_IMG" +
Signature · 2 lines
Best regards, Vladimir
PATCH v4 8/9 qemu-img: Add convert --bitmaps option
In reply to Eric BlakeMake it easier to copy all the persistent bitmaps of (the top layer of) a source image along with its guest-visible contents, by adding a boolean flag for use with qemu-img convert. This is basically shorthand, as the same effect could be accomplished with a series of 'qemu-img bitmap --add' and 'qemu-img bitmap --merge -b source' commands, or by QMP commands.
Note that this command will fail in the same scenarios where 'qemu-img measure --bitmaps' fails, when either the source or the destanation lacks persistent bitmap support altogether.
See also https://bugzilla.redhat.com/show_bug.cgi?id=1779893
While touching this, clean up a couple coding issues spotted in the same function: an extra blank line, and merging back-to-back 'if (!skip_create)' blocks.
--- docs/tools/qemu-img.rst | 6 ++- qemu-img.c | 85 +++++++++++++++++++++++++++++++++++++++-- qemu-img-cmds.hx | 4 +- 3 files changed, 89 insertions(+), 6 deletions(-)
docs/tools/qemu-img.rst +5 -1
@@ -162,6 +162,10 @@ Parameters to convert subcommand: .. program:: qemu-img-convert +.. option:: --bitmaps + + Additionally copy all persistent bitmaps from the top layer of the source + .. option:: -n Skip the creation of the target volume
@@ -397,7 +401,7 @@ Command description: 4 Error on reading data -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME Convert the disk image *FILENAME* or a snapshot *SNAPSHOT_PARAM* to disk image *OUTPUT_FILENAME* using format *OUTPUT_FMT*. It can
qemu-img.c +82 -3
146 lines not on this page · raw message
qemu-img-cmds.hx +2 -2
@@ -46,9 +46,9 @@ SRST ERST DEF("convert", img_convert, - "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") + "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") SRST -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME ERST DEF("create", img_create,
Signature · 1 line
2.26.2
Re: [PATCH v4 8/9] qemu-img: Add convert --bitmaps option
Quoted 24 lines · Eric Blake
Make it easier to copy all the persistent bitmaps of (the top layer of) a source image along with its guest-visible contents, by adding a boolean flag for use with qemu-img convert. This is basically shorthand, as the same effect could be accomplished with a series of 'qemu-img bitmap --add' and 'qemu-img bitmap --merge -b source' commands, or by QMP commands.
Note that this command will fail in the same scenarios where 'qemu-img measure --bitmaps' fails, when either the source or the destanation lacks persistent bitmap support altogether.
See also https://bugzilla.redhat.com/show_bug.cgi?id=1779893
While touching this, clean up a couple coding issues spotted in the same function: an extra blank line, and merging back-to-back 'if (!skip_create)' blocks.
Signed-off-by Eric Blake <[email protected]>--- docs/tools/qemu-img.rst | 6 ++- qemu-img.c | 85 +++++++++++++++++++++++++++++++++++++++-- qemu-img-cmds.hx | 4 +- 3 files changed, 89 insertions(+), 6 deletions(-)
[..]
Quoted 23 lines
qemu-img.c +22
-0+static int convert_copy_bitmaps (BlockDriverState *src,BlockDriverState *dst) +{ +BdrvDirtyBitmap *bm; +Error *err =NULL ; +BlockDirtyBitmapMergeSource *merge; +BlockDirtyBitmapMergeSourceList *list; + + FOR_EACH_DIRTY_BITMAP(src, bm) { +const char *name; + +if (!bdrv_dirty_bitmap_get_persistence (bm)) { +continue ; + } + name =bdrv_dirty_bitmap_name (bm); +qmp_block_dirty_bitmap_add (dst->node_name, name, + true,bdrv_dirty_bitmap_granularity (bm), + true, true, + true, !bdrv_dirty_bitmap_enabled (bm), + &err); +if (err) { +error_reportf_err (err,"Failed to create bitmap %s: " , name); +return -1 ;
qemu-img.c static int convert_do_copy(ImgConvertState *s) · line 2144
Quoted 10 lines
qemu-img.c +10
-0+ } + + merge =g_new0 (BlockDirtyBitmapMergeSource,1 ); + merge->type =QTYPE_QDICT ; + merge->u.external.node =g_strdup (src->node_name); + merge->u.external.name =g_strdup (name); + list =g_new0 (BlockDirtyBitmapMergeSourceList,1 ); + list->value = merge; +qmp_block_dirty_bitmap_merge (dst->node_name, name, list, &err); +qapi_free_BlockDirtyBitmapMergeSourceList (list);
^ duplicated code chunk with "case BITMAP_MERGE". Small helper function will not hurt.
Quoted 57 lines
qemu-img.c +21
-1+if (err) { +error_reportf_err (err,"Failed to populate bitmap %s: " , name); +return -1 ; + } + } + +return 0 ; +} +#define MAX_BUF_SECTORS 32768static int img_convert (int argc,char **argv)@@ -2130,6 +2172,8 @@ static int img_convert(int argc, char **argv)int64_t ret = -EINVAL ;bool force_share = false;bool explict_min_sparse = false; +bool bitmaps = false; +size_t nbitmaps =0 ;ImgConvertState s = (ImgConvertState ) {/* Need at least 4k of zeros for sparse detection */ @@ -2149,6 +2193,7 @@ static int img_convert(int argc, char **argv) {"target-image-opts", no_argument, 0, OPTION_TARGET_IMAGE_OPTS}, {"salvage", no_argument, 0, OPTION_SALVAGE}, {"target-is-zero", no_argument, 0, OPTION_TARGET_IS_ZERO}, + {"bitmaps", no_argument, 0, OPTION_BITMAPS}, {0, 0, 0, 0} }; c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU",@@ -2272,6 +2317,9 @@ static int img_convert(int argc, char **argv)*/ s.has_zero_init = true;break ; +case OPTION_BITMAPS : + bitmaps = true; +break ; } }@@ -2333,7 +2381,6 @@ static int img_convert(int argc, char **argv) goto fail_getopt; } -/* ret is still -EINVAL until here */ ret =bdrv_parse_cache_mode (src_cache, &src_flags, &src_writethrough);if (ret <0 ) {@@ -2493,6 +2540,27 @@ static int img_convert(int argc, char **argv) } } +/* Determine how many bitmaps need copying */ +if (bitmaps) { +BdrvDirtyBitmap *bm; + +if (s.src_num >1 ) { +error_report ("Copying bitmaps only possible with single source" );
qemu-img.c static int img_convert(int argc, char **argv) · line 2558
Quoted 10 lines
qemu-img.c +10
-0+ ret =-1 ; + goto out; + } +if (!bdrv_supports_persistent_dirty_bitmap (blk_bs (s.src[0 ]))) { +error_report ("Source lacks bitmap support" ); + ret =-1 ; + goto out; + } + FOR_EACH_DIRTY_BITMAP(blk_bs(s.src[0]), bm) { +if (bdrv_dirty_bitmap_get_persistence (bm)) {
Just note, that we can't have not persistent bitmaps here.. But the check doesn't hurt.
Quoted 62 lines
qemu-img.c +18
-2+ nbitmaps++; + } + } + } +/* * The later open call will need any decryption secrets, and * bdrv_create() will purge "opts", so extract them now before @@ -2501,9 +2569,7 @@ static int img_convert(int argc, char **argv)if (!skip_create) { open_opts =qdict_new ();qemu_opt_foreach (opts, img_add_key_secrets, open_opts, &error_abort); - } -if (!skip_create) {/* Create the new image */ ret =bdrv_create (drv, out_filename, opts, &local_err);if (ret <0 ) {@@ -2541,6 +2607,13 @@ static int img_convert(int argc, char **argv) } out_bs =blk_bs (s.target); +if (nbitmaps >0 && !bdrv_supports_persistent_dirty_bitmap (out_bs)) { +error_report ("Format driver '%s' does not support bitmaps" , + out_fmt); + ret =-1 ; + goto out; + } +if (s.compressed && !block_driver_can_compress (out_bs->drv)) {error_report ("Compression not supported for this file format" ); ret =-1 ;@@ -2600,6 +2673,12 @@ static int img_convert(int argc, char **argv) } ret =convert_do_copy (&s); + +/* Now copy the bitmaps */ +if (nbitmaps >0 && ret ==0 ) { + ret =convert_copy_bitmaps (blk_bs (s.src[0 ]), out_bs); + } + out:if (!ret) {qemu_progress_print (100 ,0 );qemu-img-cmds.hx +2
-2@@ -46,9 +46,9 @@ SRST ERST DEF("convert", img_convert, - "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") + "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename") SRST -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME ERST DEF("create", img_create,
Signature · 2 lines
Best regards, Vladimir
PATCH v4 9/9 iotests: Add test 291 to for qemu-img bitmap coverage
In reply to Eric BlakeAdd a new test covering the 'qemu-img bitmap' subcommand, as well as 'qemu-img convert --bitmaps', both added in recent patches.
--- tests/qemu-iotests/291 | 112 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/291.out | 78 ++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 191 insertions(+) create mode 100755 tests/qemu-iotests/291 create mode 100644 tests/qemu-iotests/291.out
tests/qemu-iotests/291 new file +112 -0
@@ -0,0 +1,112 @@ +#!/usr/bin/env bash +# +# Test qemu-img bitmap handling +# +# Copyright (C) 2018-2020 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +seq="$(basename $0)" +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img + nbd_server_stop +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter +. ./common.nbd + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux +_require_command QEMU_NBD + +echo +echo "=== Initial image setup ===" +echo + +# Create backing image with one bitmap +TEST_IMG="$TEST_IMG.base" _make_test_img 10M +$QEMU_IMG bitmap --add -f $IMGFMT "$TEST_IMG.base" b0 +$QEMU_IO -c 'w 3M 1M' -f $IMGFMT "$TEST_IMG.base" | _filter_qemu_io + +# Create initial image and populate two bitmaps: one active, one inactive. +ORIG_IMG=$TEST_IMG +TEST_IMG=$TEST_IMG.orig +_make_test_img -b "$ORIG_IMG.base" -F $IMGFMT 10M +$QEMU_IO -c 'w 0 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG bitmap --add -g 512k -f $IMGFMT "$TEST_IMG" b1 +$QEMU_IMG bitmap --add --disable -f $IMGFMT "$TEST_IMG" b2 +$QEMU_IO -c 'w 3M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG bitmap --clear -f $IMGFMT "$TEST_IMG" b1 +$QEMU_IO -c 'w 1M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG bitmap --disable -f $IMGFMT "$TEST_IMG" b1 +$QEMU_IMG bitmap --enable -f $IMGFMT "$TEST_IMG" b2 +$QEMU_IO -c 'w 2M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io + +echo +echo "=== Bitmap preservation not possible to non-qcow2 ===" +echo + +TEST_IMG=$ORIG_IMG +$QEMU_IMG convert --bitmaps -O raw "$TEST_IMG.orig" "$TEST_IMG" && + echo "unexpected success" + +echo +echo "=== Convert with bitmap preservation ===" +echo + +# Only bitmaps from the active layer are copied +$QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG.orig" "$TEST_IMG" +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific +# But we can also merge in bitmaps from other layers. This test is a bit +# contrived to cover more code paths, in reality, you could merge directly +# into b0 without going through tmp +$QEMU_IMG bitmap --add --disable -f $IMGFMT "$TEST_IMG" b0 +$QEMU_IMG bitmap --add --merge b0 -b "$TEST_IMG.base" -F $IMGFMT \ + -f $IMGFMT "$TEST_IMG" tmp +$QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0 +$QEMU_IMG bitmap --remove --image-opts \ + driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific + +echo +echo "=== Check bitmap contents ===" +echo + +# x-dirty-bitmap is a hack for reading bitmaps; it abuses block status to +# report "data":false for portions of the bitmap which are set +IMG="driver=nbd,server.type=unix,server.path=$nbd_unix_socket" +nbd_server_start_unix_socket -r -f qcow2 -B b0 "$TEST_IMG" +$QEMU_IMG map --output=json --image-opts \ + "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b0" | _filter_qemu_img_map +nbd_server_start_unix_socket -r -f qcow2 -B b1 "$TEST_IMG" +$QEMU_IMG map --output=json --image-opts \ + "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b1" | _filter_qemu_img_map +nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG" +$QEMU_IMG map --output=json --image-opts \ + "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map + +# success, all done +echo '*** done' +rm -f $seq.full +status=0
tests/qemu-iotests/291.out new file +78 -0
78 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -298,4 +298,5 @@ 288 quick 289 rw quick 290 rw auto quick +291 rw quick 292 rw auto quick
Signature · 1 line
2.26.2
Re: [PATCH v4 9/9] iotests: Add test 291 to for qemu-img bitmap coverage
Quoted 5 lines · Eric Blake
Add a new test covering the 'qemu-img bitmap' subcommand, as well as 'qemu-img convert --bitmaps', both added in recent patches.
Signed-off-by Eric Blake <[email protected]>Reviewed-by Max Reitz <[email protected]>
Quoted 17 lines
--- tests/qemu-iotests/291 | 112 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/291.out | 78 ++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 191 insertions(+)create mode 100755 tests/qemu-iotests/291
create mode 100644 tests/qemu-iotests/291.outtests/qemu-iotests/291 new file +3
-0@@ -0,0 +1,112 @@ +#!/usr/bin/env bash +# +# Test qemu-img bitmap handling
[..]
Quoted 3 lines
+TEST_IMG="$TEST_IMG.base"
_make_test_img10M
+$QEMU_IMG bitmap --add -f $IMGFMT "$TEST_IMG.base" b0 +$QEMU_IO -c 'w 3M 1M' -f $IMGFMT "$TEST_IMG.base" |_filter_qemu_io
Actually '-f $IMGFMT' option is already inside $QEMU_IO. Still it's a kind of standard for iotests to duplicate -f option, so I don't care. I hope one day we'll restrict it in qemu-io interface :)
Signature · 2 lines
Best regards, Vladimir