qcow2: Allow resize of images with internal snapshots
v4 cover qcow2: Allow resize of images with internal snapshots
Re-posting this to make Max' life easier when rebasing on top of Kevin's work.
Based-on: <[email protected]> [PATCH v7 00/10] block: Fix resize (extending) of short overlays
In v4:
- patch 1: fold in Max's touchups to my v3
- patch 1: resolve merge conflict on top of Kevin's block-next branch (truncate signature change)
- patch 2: resolve semantic conflict (truncate signature change)
001/3:[0004] [FC] 'block: Add blk_new_with_bs() helper' 002/3:[0002] [FC] 'qcow2: Allow resize of images with internal snapshots' 003/3:[----] [--] 'qcow2: Tweak comment about bitmaps vs. resize'
Eric Blake (3):
block: Add blk_new_with_bs() helper
qcow2: Allow resize of images with internal snapshots
qcow2: Tweak comment about bitmaps vs. resize
include/sysemu/block-backend.h | 2 ++ block/block-backend.c | 23 +++++++++++++++++ block/crypto.c | 9 +++---- block/parallels.c | 8 +++--- block/qcow.c | 8 +++--- block/qcow2-snapshot.c | 20 ++++++++++++--- block/qcow2.c | 45 +++++++++++++++++++++++----------- block/qed.c | 8 +++--- block/sheepdog.c | 10 ++++---- block/vdi.c | 8 +++--- block/vhdx.c | 8 +++--- block/vmdk.c | 9 +++---- block/vpc.c | 8 +++--- blockdev.c | 8 +++--- blockjob.c | 7 ++---- tests/qemu-iotests/061 | 35 ++++++++++++++++++++++++++ tests/qemu-iotests/061.out | 28 +++++++++++++++++++++ 17 files changed, 177 insertions(+), 67 deletions(-)
Signature · 1 line
2.26.2
PATCH v4 1/3 block: Add blk_new_with_bs() helper
There are several callers that need to create a new block backend from an existing BDS; make the task slightly easier with a common helper routine.
Message-Id: <[email protected]> [mreitz: Set @ret only in error paths, see
https://lists.nongnu.org/archive/html/qemu-block/2020-04/msg01216.html]
--- include/sysemu/block-backend.h | 2 ++ block/block-backend.c | 23 +++++++++++++++++++++++ block/crypto.c | 9 ++++----- block/parallels.c | 8 ++++---- block/qcow.c | 8 ++++---- block/qcow2.c | 18 ++++++++---------- block/qed.c | 8 ++++---- block/sheepdog.c | 10 +++++----- block/vdi.c | 8 ++++---- block/vhdx.c | 8 ++++---- block/vmdk.c | 9 ++++----- block/vpc.c | 8 ++++---- blockdev.c | 8 +++----- blockjob.c | 7 ++----- 14 files changed, 75 insertions(+), 59 deletions(-)
include/sysemu/block-backend.h +2 -0
@@ -77,6 +77,8 @@ typedef struct BlockBackendPublic { } BlockBackendPublic;BlockBackend *blk_new (AioContext *ctx,uint64_t perm,uint64_t shared_perm); +BlockBackend *blk_new_with_bs (BlockDriverState *bs,uint64_t perm, +uint64_t shared_perm,Error **errp);BlockBackend *blk_new_open (const char *filename,const char *reference,QDict *options,int flags,Error **errp);int blk_get_refcnt (BlockBackend *blk);
block/block-backend.c +23 -0
@@ -355,6 +355,29 @@ BlockBackend *blk_new(AioContext *ctx, uint64_t perm, uint64_t shared_perm)return blk; } +/* +* Create a new BlockBackend connected to an existing BlockDriverState. +* +* @perm is a bitmasks of BLK_PERM_* constants which describes the +* permissions to request for @bs that is attached to this +* BlockBackend. @shared_perm is a bitmask which describes which +* permissions may be granted to other users of the attached node. +* Both sets of permissions can be changed later using blk_set_perm(). +* +* Return the new BlockBackend on success, null on failure. +*/ +BlockBackend *blk_new_with_bs (BlockDriverState *bs,uint64_t perm, +uint64_t shared_perm,Error **errp) +{ +BlockBackend *blk =blk_new (bdrv_get_aio_context (bs), perm, shared_perm); + +if (blk_insert_bs (blk, bs, errp) <0 ) { +blk_unref (blk); +return NULL ; + } +return blk; +} +/* * Creates a new BlockBackend, opens a new BlockDriverState, and connects both. * The new BlockBackend is in the main AioContext.
block/crypto.c +4 -5
@@ -261,11 +261,10 @@ static int block_crypto_co_create_generic(BlockDriverState *bs,QCryptoBlock *crypto =NULL ;struct BlockCryptoCreateData data; - blk =blk_new (bdrv_get_aio_context (bs), -BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL ); - - ret =blk_insert_bs (blk, bs, errp); -if (ret <0 ) { + blk =blk_new_with_bs (bs,BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL , + errp); +if (!blk) { + ret = -EPERM ; goto cleanup; }
block/parallels.c +4 -4
@@ -559,10 +559,10 @@ static int coroutine_fn parallels_co_create(BlockdevCreateOptions* opts,return -EIO ; } - blk =blk_new (bdrv_get_aio_context (bs), -BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL ); - ret =blk_insert_bs (blk, bs, errp); -if (ret <0 ) { + blk =blk_new_with_bs (bs,BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL , + errp); +if (!blk) { + ret = -EPERM ; goto out; }blk_set_allow_write_beyond_eof (blk, true);
block/qcow.c +4 -4
@@ -849,10 +849,10 @@ static int coroutine_fn qcow_co_create(BlockdevCreateOptions *opts,return -EIO ; } - qcow_blk =blk_new (bdrv_get_aio_context (bs), -BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL ); - ret =blk_insert_bs (qcow_blk, bs, errp); -if (ret <0 ) { + qcow_blk =blk_new_with_bs (bs,BLK_PERM_WRITE |BLK_PERM_RESIZE , +BLK_PERM_ALL , errp); +if (!qcow_blk) { + ret = -EPERM ; goto exit; }blk_set_allow_write_beyond_eof (qcow_blk, true);
block/qcow2.c +8 -10
@@ -3405,10 +3405,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) }/* Create BlockBackend to write to the image */ - blk =blk_new (bdrv_get_aio_context (bs), -BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL ); - ret =blk_insert_bs (blk, bs, errp); -if (ret <0 ) { + blk =blk_new_with_bs (bs,BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL , + errp); +if (!blk) { + ret = -EPERM ; goto out; }blk_set_allow_write_beyond_eof (blk, true);
@@ -5412,12 +5412,10 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, }if (new_size) { -BlockBackend *blk =blk_new (bdrv_get_aio_context (bs), -BLK_PERM_RESIZE ,BLK_PERM_ALL ); - ret =blk_insert_bs (blk, bs, errp); -if (ret <0 ) { -blk_unref (blk); -return ret; +BlockBackend *blk =blk_new_with_bs (bs,BLK_PERM_RESIZE ,BLK_PERM_ALL , + errp); +if (!blk) { +return -EPERM ; }/*
block/qed.c +4 -4
@@ -651,10 +651,10 @@ static int coroutine_fn bdrv_qed_co_create(BlockdevCreateOptions *opts,return -EIO ; } - blk =blk_new (bdrv_get_aio_context (bs), -BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL ); - ret =blk_insert_bs (blk, bs, errp); -if (ret <0 ) { + blk =blk_new_with_bs (bs,BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL , + errp); +if (!blk) { + ret = -EPERM ; goto out; }blk_set_allow_write_beyond_eof (blk, true);
block/sheepdog.c +5 -5
@@ -1803,12 +1803,12 @@ static int sd_prealloc(BlockDriverState *bs, int64_t old_size, int64_t new_size,void *buf =NULL ;int ret; - blk =blk_new (bdrv_get_aio_context (bs), -BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE |BLK_PERM_RESIZE , -BLK_PERM_ALL ); + blk =blk_new_with_bs (bs, +BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE |BLK_PERM_RESIZE , +BLK_PERM_ALL , errp); - ret =blk_insert_bs (blk, bs, errp); -if (ret <0 ) { +if (!blk) { + ret = -EPERM ; goto out_with_err_set; }
block/vdi.c +4 -4
@@ -804,10 +804,10 @@ static int coroutine_fn vdi_co_do_create(BlockdevCreateOptions *create_options, goto exit; } - blk =blk_new (bdrv_get_aio_context (bs_file), -BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL ); - ret =blk_insert_bs (blk, bs_file, errp); -if (ret <0 ) { + blk =blk_new_with_bs (bs_file,BLK_PERM_WRITE |BLK_PERM_RESIZE , +BLK_PERM_ALL , errp); +if (!blk) { + ret = -EPERM ; goto exit; }
block/vhdx.c +4 -4
@@ -1984,10 +1984,10 @@ static int coroutine_fn vhdx_co_create(BlockdevCreateOptions *opts,return -EIO ; } - blk =blk_new (bdrv_get_aio_context (bs), -BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL ); - ret =blk_insert_bs (blk, bs, errp); -if (ret <0 ) { + blk =blk_new_with_bs (bs,BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL , + errp); +if (!blk) { + ret = -EPERM ; goto delete_and_exit; }blk_set_allow_write_beyond_eof (blk, true);
block/vmdk.c +4 -5
@@ -2717,11 +2717,10 @@ static BlockBackend *vmdk_co_create_cb(int64_t size, int idx,if (!bs) {return NULL ; } - blk =blk_new (bdrv_get_aio_context (bs), -BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE |BLK_PERM_RESIZE , -BLK_PERM_ALL ); -if (blk_insert_bs (blk, bs, errp)) { -bdrv_unref (bs); + blk =blk_new_with_bs (bs, +BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE |BLK_PERM_RESIZE , +BLK_PERM_ALL , errp); +if (!blk) {return NULL ; }blk_set_allow_write_beyond_eof (blk, true);
block/vpc.c +4 -4
@@ -1012,10 +1012,10 @@ static int coroutine_fn vpc_co_create(BlockdevCreateOptions *opts,return -EIO ; } - blk =blk_new (bdrv_get_aio_context (bs), -BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL ); - ret =blk_insert_bs (blk, bs, errp); -if (ret <0 ) { + blk =blk_new_with_bs (bs,BLK_PERM_WRITE |BLK_PERM_RESIZE ,BLK_PERM_ALL , + errp); +if (!blk) { + ret = -EPERM ; goto out; }blk_set_allow_write_beyond_eof (blk, true);
blockdev.c +3 -5
@@ -2711,7 +2711,6 @@ void qmp_block_resize(bool has_device, const char *device,BlockBackend *blk =NULL ;BlockDriverState *bs;AioContext *aio_context; -int ret; bs = bdrv_lookup_bs(has_device ? device : NULL, has_node_name ? node_name : NULL,
@@ -2734,14 +2733,13 @@ void qmp_block_resize(bool has_device, const char *device, goto out; } - blk =blk_new (bdrv_get_aio_context (bs),BLK_PERM_RESIZE ,BLK_PERM_ALL ); - ret =blk_insert_bs (blk, bs, errp); -if (ret <0 ) { + blk =blk_new_with_bs (bs,BLK_PERM_RESIZE ,BLK_PERM_ALL , errp); +if (!blk) { goto out; }bdrv_drained_begin (bs); - ret =blk_truncate (blk, size, false,PREALLOC_MODE_OFF ,0 , errp); +blk_truncate (blk, size, false,PREALLOC_MODE_OFF ,0 , errp);bdrv_drained_end (bs); out:
blockjob.c +2 -5
@@ -397,16 +397,13 @@ void *block_job_create(const char *job_id, const BlockJobDriver *driver, {BlockBackend *blk;BlockJob *job; -int ret;if (job_id ==NULL && !(flags &JOB_INTERNAL )) { job_id =bdrv_get_device_name (bs); } - blk =blk_new (bdrv_get_aio_context (bs), perm, shared_perm); - ret =blk_insert_bs (blk, bs, errp); -if (ret <0 ) { -blk_unref (blk); + blk =blk_new_with_bs (bs, perm, shared_perm, errp); +if (!blk) {return NULL ; }
Signature · 1 line
2.26.2
PATCH v4 2/3 qcow2: Allow resize of images with internal snapshots
In reply to Eric BlakeWe originally refused to allow resize of images with internal snapshots because the v2 image format did not require the tracking of snapshot size, making it impossible to safely revert to a snapshot with a different size than the current view of the image. But the snapshot size tracking was rectified in v3, and our recent fixes to qemu-img amend (see 0a85af35) guarantee that we always have a valid snapshot size. Thus, we no longer need to artificially limit image resizes, but it does become one more thing that would prevent a downgrade back to v2. And now that we support different-sized snapshots, it's also easy to fix reverting to a snapshot to apply the new size.
Upgrade iotest 61 to cover this (we previously had NO coverage of refusal to resize while snapshots exist). Note that the amend process can fail but still have effects: in particular, since we break things into upgrade, resize, downgrade, a failure during resize does not roll back changes made during upgrade, nor does failure in downgrade roll back a resize. But this situation is pre-existing even without this patch; and without journaling, the best we could do is minimize the chance of partial failure by collecting all changes prior to doing any writes - which adds a lot of complexity but could still fail with EIO. On the other hand, we are careful that even if we have partial modification but then fail, the image is left viable (that is, we are careful to sequence things so that after each successful cluster write, there may be transient leaked clusters but no corrupt metadata). And complicating the code to make it more transaction-like is not worth the effort: a user can always request multiple 'qemu-img amend' changing one thing each, if they need finer-grained control over detecting the first failure than what they get by letting qemu decide how to sequence multiple changes.
--- block/qcow2-snapshot.c | 20 ++++++++++++++++---- block/qcow2.c | 25 ++++++++++++++++++++++--- tests/qemu-iotests/061 | 35 +++++++++++++++++++++++++++++++++++ tests/qemu-iotests/061.out | 28 ++++++++++++++++++++++++++++ 4 files changed, 101 insertions(+), 7 deletions(-)
block/qcow2-snapshot.c +16 -4
@@ -23,6 +23,7 @@*/ #include "qemu/osdep.h" +#include "sysemu/block-backend.h" #include "qapi/error.h" #include "qcow2.h" #include "qemu/bswap.h"
@@ -775,10 +776,21 @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id) }if (sn->disk_size != bs->total_sectors *BDRV_SECTOR_SIZE ) { -error_report ("qcow2: Loading snapshots with different disk " -"size is not implemented" ); - ret = -ENOTSUP ; - goto fail; +BlockBackend *blk =blk_new_with_bs (bs,BLK_PERM_RESIZE ,BLK_PERM_ALL , + &local_err); +if (!blk) { +error_report_err (local_err); + ret = -ENOTSUP ; + goto fail; + } + + ret =blk_truncate (blk, sn->disk_size, true,PREALLOC_MODE_OFF ,0 , + &local_err); +blk_unref (blk); +if (ret <0 ) { +error_report_err (local_err); + goto fail; + } }/*
block/qcow2.c +22 -3
@@ -3989,9 +3989,12 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset,qemu_co_mutex_lock (&s->lock); -/* cannot proceed if image has snapshots */ -if (s->nb_snapshots) { -error_setg (errp,"Can't resize an image which has snapshots" ); +/* +* Even though we store snapshot size for all images, it was not +* required until v3, so it is not safe to proceed for v2. +*/ +if (s->nb_snapshots && s->qcow_version <3 ) { +error_setg (errp,"Can't resize a v2 image which has snapshots" ); ret = -ENOTSUP ; goto fail; }
@@ -5005,6 +5008,7 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,BDRVQcow2State *s = bs->opaque;int current_version = s->qcow_version;int ret; +int i;/* This is qcow2_downgrade(), not qcow2_upgrade() */ assert (target_version < current_version);
@@ -5022,6 +5026,21 @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version,return -ENOTSUP ; } +/* +* If any internal snapshot has a different size than the current +* image size, or VM state size that exceeds 32 bits, downgrading +* is unsafe. Even though we would still use v3-compliant output +* to preserve that data, other v2 programs might not realize +* those optional fields are important. +*/ +for (i =0 ; i < s->nb_snapshots; i++) { +if (s->snapshots[i].vm_state_size >UINT32_MAX || + s->snapshots[i].disk_size != bs->total_sectors *BDRV_SECTOR_SIZE ) { +error_setg (errp,"Internal snapshots prevent downgrade of image" ); +return -ENOTSUP ; + } + } +/* clear incompatible features */ if (s->incompatible_features &QCOW2_INCOMPAT_DIRTY ) { ret =qcow2_mark_clean (bs);
tests/qemu-iotests/061 +35 -0
@@ -111,6 +111,41 @@ $PYTHON qcow2.py "$TEST_IMG" dump-header $QEMU_IO -c "read -P 0x2a 42M 64k" "$TEST_IMG" | _filter_qemu_io _check_test_img +echo +echo "=== Testing resize with snapshots ===" +echo +_make_test_img -o "compat=0.10" 32M +$QEMU_IO -c "write -P 0x2a 24M 64k" "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG snapshot -c foo "$TEST_IMG" +$QEMU_IMG resize "$TEST_IMG" 64M && + echo "unexpected pass" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG amend -o "compat=1.1,size=128M" "$TEST_IMG" || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG snapshot -c bar "$TEST_IMG" +$QEMU_IMG resize --shrink "$TEST_IMG" 64M || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG amend -o "compat=0.10,size=32M" "$TEST_IMG" && + echo "unexpected pass" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG snapshot -a bar "$TEST_IMG" || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +$QEMU_IMG snapshot -d bar "$TEST_IMG" +$QEMU_IMG amend -o "compat=0.10,size=32M" "$TEST_IMG" || + echo "unexpected fail" +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep '^\(version\|size\|nb_snap\)' + +_check_test_img + + echo echo "=== Testing dirty lazy_refcounts=off ===" echo
tests/qemu-iotests/061.out +28 -0
@@ -271,6 +271,34 @@ read 65536/65536 bytes at offset 44040192 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) No errors were found on the image. +=== Testing resize with snapshots === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 +wrote 65536/65536 bytes at offset 25165824 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +qemu-img: Can't resize a v2 image which has snapshots +version 2 +size 33554432 +nb_snapshots 1 +version 3 +size 134217728 +nb_snapshots 1 +Image resized. +version 3 +size 67108864 +nb_snapshots 2 +qemu-img: Internal snapshots prevent downgrade of image +version 3 +size 33554432 +nb_snapshots 2 +version 3 +size 134217728 +nb_snapshots 2 +version 2 +size 33554432 +nb_snapshots 1 +No errors were found on the image. + === Testing dirty lazy_refcounts=off === Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
Signature · 1 line
2.26.2
PATCH v4 3/3 qcow2: Tweak comment about bitmaps vs. resize
In reply to Eric BlakeOur comment did not actually match the code. Rewrite the comment to be less sensitive to any future changes to qcow2-bitmap.c that might implement scenarios that we currently reject.
--- block/qcow2.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
block/qcow2.c +1 -1
@@ -3999,7 +3999,7 @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, goto fail; } -/* cannot proceed if image has bitmaps */ +/* See qcow2-bitmap.c for which bitmap scenarios prevent a resize. */ if (qcow2_truncate_bitmaps_check (bs, errp)) { ret = -ENOTSUP ; goto fail;
Signature · 1 line
2.26.2
Quoted 5 lines · Eric Blake
Re-posting this to make Max' life easier when rebasing on top of Kevin's work.
Based-on: <[email protected]> [PATCH v7 00/10] block: Fix resize (extending) of short overlays
Quoted 8 lines
In v4:
- patch 1: fold in Max's touchups to my v3
- patch 1: resolve merge conflict on top of Kevin's block-next branch (truncate signature change)
- patch 2: resolve semantic conflict (truncate signature change)001/3:[0004] [FC] 'block: Add
blk_new_with_bs()helper' 002/3:[0002] [FC] 'qcow2: Allow resize of images with internal snapshots' 003/3:[----] [--] 'qcow2: Tweak comment about bitmaps vs. resize'
Thanks, I’ve updated my branch:
https://git.xanclic.moe/XanClic/qemu/commits/branch/block-next
(A commit with two Message-Id tags is interesting :))
Max