qapi: block-dirty-bitmap-remove transaction action
v3 cover qapi: block-dirty-bitmap-remove transaction action
Hi, this is a proposal based off of Vladimir's patchset:
[Qemu-devel] [PATCH 0/4] qapi: block-dirty-bitmap-remove transaction action
===
V3:
===
001/3:[----] [--] 'blockdev: reduce aio_context locked sections in bitmap add/remove' 002/3:[0024] [FC] 'qapi: implement block-dirty-bitmap-remove transaction action' 003/3:[----] [--] 'iotests: test bitmap moving inside 254'
- Changed "squelch_persistence" to "skip_store"
- Use Max's suggestion for return expr
===
V2:
===
It replaces patches two and three with a modified patch (now patch 2) that foregoes the need for a hide()/unhide() bitmap API. I think it's suitable as a smaller alternative, but I'm not sure if it covers all of the use cases of the original series.
Patches 1 and 3 (formerly 4) included as-is.
John Snow (1):
qapi: implement block-dirty-bitmap-remove transaction action
Vladimir Sementsov-Ogievskiy (2):
blockdev: reduce aio_context locked sections in bitmap add/remove
iotests: test bitmap moving inside 254
block.c | 2 +- block/dirty-bitmap.c | 15 +++-- blockdev.c | 105 ++++++++++++++++++++++++++------- include/block/dirty-bitmap.h | 2 +- migration/block-dirty-bitmap.c | 2 +- qapi/transaction.json | 2 + tests/qemu-iotests/254 | 30 +++++++++- tests/qemu-iotests/254.out | 82 +++++++++++++++++++++++++ 8 files changed, 206 insertions(+), 34 deletions(-)
Signature · 1 line
2.21.0
PATCH v3 1/3 blockdev: reduce aio_context locked sections in bitmap add/remove
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Commit 0a6c86d024c52 returned these locks back to add/remove functionality, to protect from intersection of persistent bitmap related IO with other IO. But other bitmap-related functions called here are unrelated to the problem, and there are no needs to keep these calls inside critical sections.
--- blockdev.c | 30 +++++++++++++----------------- 1 file changed, 13 insertions(+), 17 deletions(-)
blockdev.c +13 -17
@@ -2811,7 +2811,6 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name, {BlockDriverState *bs;BdrvDirtyBitmap *bitmap; -AioContext *aio_context =NULL ;if (!name || name[0 ] =='\0' ) {error_setg (errp,"Bitmap name cannot be empty" );
@@ -2847,16 +2846,20 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name, }if (persistent) { - aio_context =bdrv_get_aio_context (bs); +AioContext *aio_context =bdrv_get_aio_context (bs); +bool ok; +aio_context_acquire (aio_context); -if (!bdrv_can_store_new_dirty_bitmap (bs, name, granularity, errp)) { - goto out; + ok =bdrv_can_store_new_dirty_bitmap (bs, name, granularity, errp); +aio_context_release (aio_context); +if (!ok) { +return ; } } bitmap =bdrv_create_dirty_bitmap (bs, granularity, name, errp);if (bitmap ==NULL ) { - goto out; +return ; }if (disabled) {
@@ -2864,10 +2867,6 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name, }bdrv_dirty_bitmap_set_persistence (bitmap, persistent); - out: -if (aio_context) { -aio_context_release (aio_context); - } } void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
@@ -2875,8 +2874,6 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name, {BlockDriverState *bs;BdrvDirtyBitmap *bitmap; -Error *local_err =NULL ; -AioContext *aio_context =NULL ; bitmap =block_dirty_bitmap_lookup (node, name, &bs, errp);if (!bitmap || !bs) {
@@ -2889,20 +2886,19 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name, }if (bdrv_dirty_bitmap_get_persistence (bitmap)) { - aio_context =bdrv_get_aio_context (bs); +AioContext *aio_context =bdrv_get_aio_context (bs); +Error *local_err =NULL ; +aio_context_acquire (aio_context);bdrv_remove_persistent_dirty_bitmap (bs, name, &local_err); +aio_context_release (aio_context);if (local_err !=NULL ) {error_propagate (errp, local_err); - goto out; +return ; } }bdrv_release_dirty_bitmap (bs, bitmap); - out: -if (aio_context) { -aio_context_release (aio_context); - } }/**
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v3 1/3] blockdev: reduce aio_context locked sections in bitmap add/remove
Quoted 8 lines · John Snow
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Commit
0a6c86d024c52returned these locks back to add/remove functionality, to protect from intersection of persistent bitmap related IO with other IO. But other bitmap-related functions called here are unrelated to the problem, and there are no needs to keep these calls inside critical sections.
Quoted 6 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by John Snow <[email protected]>Signed-off-by John Snow <[email protected]>--- blockdev.c | 30 +++++++++++++----------------- 1 file changed, 13 insertions(+), 17 deletions(-)
PATCH v3 2/3 qapi: implement block-dirty-bitmap-remove transaction action
In reply to John SnowIt is used to do transactional movement of the bitmap (which is possible in conjunction with merge command). Transactional bitmap movement is needed in scenarios with external snapshot, when we don't want to leave copy of the bitmap in the base image.
--- block.c | 2 +- block/dirty-bitmap.c | 15 +++---- blockdev.c | 79 +++++++++++++++++++++++++++++++--- include/block/dirty-bitmap.h | 2 +- migration/block-dirty-bitmap.c | 2 +- qapi/transaction.json | 2 + 6 files changed, 85 insertions(+), 17 deletions(-)
block.c +1 -1
@@ -5316,7 +5316,7 @@ static void coroutine_fn bdrv_co_invalidate_cache(BlockDriverState *bs,for (bm =bdrv_dirty_bitmap_next (bs,NULL ); bm; bm =bdrv_dirty_bitmap_next (bs, bm)) { -bdrv_dirty_bitmap_set_migration (bm, false); +bdrv_dirty_bitmap_skip_store (bm, false); } ret =refresh_total_sectors (bs, bs->total_sectors);
block/dirty-bitmap.c +7 -8
@@ -48,10 +48,9 @@ struct BdrvDirtyBitmap {bool inconsistent;/* bitmap is persistent, but inconsistent. It cannot be used at all in any way, except a QMP user can remove it. */ -bool migration;/* Bitmap is selected for migration, it should -not be stored on the next inactivation -(persistent flag doesn't matter until next -invalidation).*/ +bool skip_store;/* We are either migrating or deleting this +* bitmap; it should not be stored on the next +* inactivation. */ QLIST_ENTRY (BdrvDirtyBitmap) list; };
@@ -757,16 +756,16 @@ void bdrv_dirty_bitmap_set_inconsistent(BdrvDirtyBitmap *bitmap) }/* Called with BQL taken. */ -void bdrv_dirty_bitmap_set_migration (BdrvDirtyBitmap *bitmap,bool migration) +void bdrv_dirty_bitmap_skip_store (BdrvDirtyBitmap *bitmap,bool skip) {qemu_mutex_lock (bitmap->mutex); - bitmap->migration = migration; + bitmap->skip_store = skip;qemu_mutex_unlock (bitmap->mutex); }bool bdrv_dirty_bitmap_get_persistence (BdrvDirtyBitmap *bitmap) { -return bitmap->persistent && !bitmap->migration; +return bitmap->persistent && !bitmap->skip_store; }bool bdrv_dirty_bitmap_inconsistent (const BdrvDirtyBitmap *bitmap)
@@ -778,7 +777,7 @@ bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs) {BdrvDirtyBitmap *bm; QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) { -if (bm->persistent && !bm->readonly && !bm->migration) { +if (bm->persistent && !bm->readonly && !bm->skip_store) {return true; } }
blockdev.c +73 -6
118 lines not on this page · raw message
include/block/dirty-bitmap.h +1 -1
@@ -83,7 +83,7 @@ void bdrv_dirty_bitmap_set_inconsistent(BdrvDirtyBitmap *bitmap);void bdrv_dirty_bitmap_set_busy (BdrvDirtyBitmap *bitmap,bool busy);void bdrv_merge_dirty_bitmap (BdrvDirtyBitmap *dest,const BdrvDirtyBitmap *src,HBitmap **backup,Error **errp); -void bdrv_dirty_bitmap_set_migration (BdrvDirtyBitmap *bitmap,bool migration); +void bdrv_dirty_bitmap_skip_store (BdrvDirtyBitmap *bitmap,bool skip);/* Functions that require manual locking. */ void bdrv_dirty_bitmap_lock (BdrvDirtyBitmap *bitmap);
migration/block-dirty-bitmap.c +1 -1
@@ -326,7 +326,7 @@ static int init_dirty_bitmap_migration(void)/* unset migration flags here, to not roll back it */ QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) { -bdrv_dirty_bitmap_set_migration (dbms->bitmap, true); +bdrv_dirty_bitmap_skip_store (dbms->bitmap, true); }if (QSIMPLEQ_EMPTY (&dirty_bitmap_mig_state.dbms_list)) {
qapi/transaction.json +2 -0
@@ -45,6 +45,7 @@ # # - @abort: since 1.6 # - @block-dirty-bitmap-add: since 2.5 +# - @block-dirty-bitmap-remove: since 4.1 # - @block-dirty-bitmap-clear: since 2.5 # - @block-dirty-bitmap-enable: since 4.0 # - @block-dirty-bitmap-disable: since 4.0
@@ -61,6 +62,7 @@ 'data': { 'abort': 'Abort', 'block-dirty-bitmap-add': 'BlockDirtyBitmapAdd', + 'block-dirty-bitmap-remove': 'BlockDirtyBitmap', 'block-dirty-bitmap-clear': 'BlockDirtyBitmap', 'block-dirty-bitmap-enable': 'BlockDirtyBitmap', 'block-dirty-bitmap-disable': 'BlockDirtyBitmap',
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v3 2/3] qapi: implement block-dirty-bitmap-remove transaction action
Quoted 15 lines · John Snow
It is used to do transactional movement of the bitmap (which is possible in conjunction with merge command). Transactional bitmap movement is needed in scenarios with external snapshot, when we don't want to leave copy of the bitmap in the base image.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by John Snow <[email protected]>--- block.c | 2 +- block/dirty-bitmap.c | 15 +++---- blockdev.c | 79 +++++++++++++++++++++++++++++++--- include/block/dirty-bitmap.h | 2 +- migration/block-dirty-bitmap.c | 2 +- qapi/transaction.json | 2 + 6 files changed, 85 insertions(+), 17 deletions(-)
[...]
qapi/transaction.json line 48
Quoted 9 lines
qapi/transaction.json +1
-0@@ -45,6 +45,7 @@ # # - @abort: since 1.6 # - @block-dirty-bitmap-add: since 2.5 +# - @block-dirty-bitmap-remove: since 4.1
Optimistic.
But anyway:
Quoted 3 lines
qapi/transaction.json +0
-0# - @block-dirty-bitmap-clear: since 2.5 # - @block-dirty-bitmap-enable: since 4.0 # - @block-dirty-bitmap-disable: since 4.0
Re: [Qemu-devel] [PATCH v3 2/3] qapi: implement block-dirty-bitmap-remove transaction action
Quoted 31 lines · Max Reitz
Quoted 15 lines · John Snow
It is used to do transactional movement of the bitmap (which is possible in conjunction with merge command). Transactional bitmap movement is needed in scenarios with external snapshot, when we don't want to leave copy of the bitmap in the base image.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by John Snow <[email protected]>--- block.c | 2 +- block/dirty-bitmap.c | 15 +++---- blockdev.c | 79 +++++++++++++++++++++++++++++++--- include/block/dirty-bitmap.h | 2 +- migration/block-dirty-bitmap.c | 2 +- qapi/transaction.json | 2 + 6 files changed, 85 insertions(+), 17 deletions(-)[...]
qapi/transaction.json line 48
Quoted 9 lines
qapi/transaction.json +1
-0@@ -45,6 +45,7 @@ # # - @abort: since 1.6 # - @block-dirty-bitmap-add: since 2.5 +# - @block-dirty-bitmap-remove: since 4.1Optimistic.
Quoted 8 lines
But anyway:
Reviewed-by Max Reitz <[email protected]>Quoted 3 lines
qapi/transaction.json +0
-0# - @block-dirty-bitmap-clear: since 2.5 # - @block-dirty-bitmap-enable: since 4.0 # - @block-dirty-bitmap-disable: since 4.0
Hah, oops. OK, I'll adjust that (and your comment in patch 3) and stage them.
Thank you!
Re: [Qemu-devel] [PATCH v3 2/3] qapi: implement block-dirty-bitmap-remove transaction action
In reply to John SnowQuoted 105 lines · John Snow
It is used to do transactional movement of the bitmap (which is possible in conjunction with merge command). Transactional bitmap movement is needed in scenarios with external snapshot, when we don't want to leave copy of the bitmap in the base image.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by John Snow <[email protected]>--- block.c | 2 +- block/dirty-bitmap.c | 15 +++---- blockdev.c | 79 +++++++++++++++++++++++++++++++--- include/block/dirty-bitmap.h | 2 +- migration/block-dirty-bitmap.c | 2 +- qapi/transaction.json | 2 + 6 files changed, 85 insertions(+), 17 deletions(-)block.c +1
-1@@ -5316,7 +5316,7 @@ static void coroutine_fn bdrv_co_invalidate_cache(BlockDriverState *bs,for (bm =bdrv_dirty_bitmap_next (bs,NULL ); bm; bm =bdrv_dirty_bitmap_next (bs, bm)) { -bdrv_dirty_bitmap_set_migration (bm, false); +bdrv_dirty_bitmap_skip_store (bm, false); } ret =refresh_total_sectors (bs, bs->total_sectors);block/dirty-bitmap.c +7
-8@@ -48,10 +48,9 @@ struct BdrvDirtyBitmap {bool inconsistent;/* bitmap is persistent, but inconsistent. It cannot be used at all in any way, except a QMP user can remove it. */ -bool migration;/* Bitmap is selected for migration, it should -not be stored on the next inactivation -(persistent flag doesn't matter until next -invalidation).*/ +bool skip_store;/* We are either migrating or deleting this +* bitmap; it should not be stored on the next +* inactivation. */ QLIST_ENTRY (BdrvDirtyBitmap) list; };@@ -757,16 +756,16 @@ void bdrv_dirty_bitmap_set_inconsistent(BdrvDirtyBitmap *bitmap) }/* Called with BQL taken. */ -void bdrv_dirty_bitmap_set_migration (BdrvDirtyBitmap *bitmap,bool migration) +void bdrv_dirty_bitmap_skip_store (BdrvDirtyBitmap *bitmap,bool skip) {qemu_mutex_lock (bitmap->mutex); - bitmap->migration = migration; + bitmap->skip_store = skip;qemu_mutex_unlock (bitmap->mutex); }bool bdrv_dirty_bitmap_get_persistence (BdrvDirtyBitmap *bitmap) { -return bitmap->persistent && !bitmap->migration; +return bitmap->persistent && !bitmap->skip_store; }bool bdrv_dirty_bitmap_inconsistent (const BdrvDirtyBitmap *bitmap)@@ -778,7 +777,7 @@ bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs) {BdrvDirtyBitmap *bm; QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) { -if (bm->persistent && !bm->readonly && !bm->migration) { +if (bm->persistent && !bm->readonly && !bm->skip_store) {return true; } }blockdev.c +21
-0@@ -2134,6 +2134,51 @@ static void block_dirty_bitmap_merge_prepare(BlkActionState *common, errp); } +static BdrvDirtyBitmap *do_block_dirty_bitmap_remove ( +const char *node,const char *name,bool release, +BlockDriverState **bitmap_bs,Error **errp); + +static void block_dirty_bitmap_remove_prepare (BlkActionState *common, +Error **errp) +{ +BlockDirtyBitmap *action; +BlockDirtyBitmapState *state =DO_UPCAST (BlockDirtyBitmapState, + common, common); + +if (action_check_completion_mode (common, errp) <0 ) { +return ; + } + + action = common->action->u.block_dirty_bitmap_remove.data; + + state->bitmap =do_block_dirty_bitmap_remove (action->node, action->name, + false, &state->bs, errp); +if (state->bitmap) { +bdrv_dirty_bitmap_skip_store (state->bitmap, true);
blockdev.c static void block_dirty_bitmap_merge_prepare(BlkActionState *common, · line 2167
Quoted 10 lines
blockdev.c +10
-0+bdrv_dirty_bitmap_set_busy (state->bitmap, true); + } +} + +static void block_dirty_bitmap_remove_abort (BlkActionState *common) +{ +BlockDirtyBitmapState *state =DO_UPCAST (BlockDirtyBitmapState, + common, common); + +if (state->bitmap) {
Hmm, interesting, I thought, abort should not be called, if prepare failed, so the following may be done unconditionally?
blockdev.c static void block_dirty_bitmap_merge_prepare(BlkActionState *common, · line 2172
Quoted 5 lines
blockdev.c +5
-0+bdrv_dirty_bitmap_skip_store (state->bitmap, false); +bdrv_dirty_bitmap_set_busy (state->bitmap, false); + } +} +
[..]
OK, I agree, good idea to reuse BUSY and migration field here and avoid new API. Now release-prepare is less "release", but I don't see any real problems with it. Maybe, we need something to be noted in docs.
Hmm, as we are not in a hurry now, we may wait for Nikolay to check this on his scenarios.
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v3 2/3] qapi: implement block-dirty-bitmap-remove transaction action
Quoted 111 lines · Vladimir Sementsov-Ogievskiy
Quoted 110 lines · John Snow
It is used to do transactional movement of the bitmap (which is possible in conjunction with merge command). Transactional bitmap movement is needed in scenarios with external snapshot, when we don't want to leave copy of the bitmap in the base image.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by John Snow <[email protected]>--- block.c | 2 +- block/dirty-bitmap.c | 15 +++---- blockdev.c | 79 +++++++++++++++++++++++++++++++--- include/block/dirty-bitmap.h | 2 +- migration/block-dirty-bitmap.c | 2 +- qapi/transaction.json | 2 + 6 files changed, 85 insertions(+), 17 deletions(-)block.c +1
-1@@ -5316,7 +5316,7 @@ static void coroutine_fn bdrv_co_invalidate_cache(BlockDriverState *bs,for (bm =bdrv_dirty_bitmap_next (bs,NULL ); bm; bm =bdrv_dirty_bitmap_next (bs, bm)) { -bdrv_dirty_bitmap_set_migration (bm, false); +bdrv_dirty_bitmap_skip_store (bm, false); } ret =refresh_total_sectors (bs, bs->total_sectors);block/dirty-bitmap.c +7
-8@@ -48,10 +48,9 @@ struct BdrvDirtyBitmap {bool inconsistent;/* bitmap is persistent, but inconsistent. It cannot be used at all in any way, except a QMP user can remove it. */ -bool migration;/* Bitmap is selected for migration, it should -not be stored on the next inactivation -(persistent flag doesn't matter until next -invalidation).*/ +bool skip_store;/* We are either migrating or deleting this +* bitmap; it should not be stored on the next +* inactivation. */ QLIST_ENTRY (BdrvDirtyBitmap) list; };@@ -757,16 +756,16 @@ void bdrv_dirty_bitmap_set_inconsistent(BdrvDirtyBitmap *bitmap) }/* Called with BQL taken. */ -void bdrv_dirty_bitmap_set_migration (BdrvDirtyBitmap *bitmap,bool migration) +void bdrv_dirty_bitmap_skip_store (BdrvDirtyBitmap *bitmap,bool skip) {qemu_mutex_lock (bitmap->mutex); - bitmap->migration = migration; + bitmap->skip_store = skip;qemu_mutex_unlock (bitmap->mutex); }bool bdrv_dirty_bitmap_get_persistence (BdrvDirtyBitmap *bitmap) { -return bitmap->persistent && !bitmap->migration; +return bitmap->persistent && !bitmap->skip_store; }bool bdrv_dirty_bitmap_inconsistent (const BdrvDirtyBitmap *bitmap)@@ -778,7 +777,7 @@ bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs) {BdrvDirtyBitmap *bm; QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) { -if (bm->persistent && !bm->readonly && !bm->migration) { +if (bm->persistent && !bm->readonly && !bm->skip_store) {return true; } }blockdev.c +26
-0@@ -2134,6 +2134,51 @@ static void block_dirty_bitmap_merge_prepare(BlkActionState *common, errp); } +static BdrvDirtyBitmap *do_block_dirty_bitmap_remove ( +const char *node,const char *name,bool release, +BlockDriverState **bitmap_bs,Error **errp); + +static void block_dirty_bitmap_remove_prepare (BlkActionState *common, +Error **errp) +{ +BlockDirtyBitmap *action; +BlockDirtyBitmapState *state =DO_UPCAST (BlockDirtyBitmapState, + common, common); + +if (action_check_completion_mode (common, errp) <0 ) { +return ; + } + + action = common->action->u.block_dirty_bitmap_remove.data; + + state->bitmap =do_block_dirty_bitmap_remove (action->node, action->name, + false, &state->bs, errp); +if (state->bitmap) { +bdrv_dirty_bitmap_skip_store (state->bitmap, true); +bdrv_dirty_bitmap_set_busy (state->bitmap, true); + } +} + +static void block_dirty_bitmap_remove_abort (BlkActionState *common)
Quoted 9 lines
blockdev.c static void block_dirty_bitmap_merge_prepare(BlkActionState *common, · line 2156
Quoted 5 lines
blockdev.c +5
-0+{ +BlockDirtyBitmapState *state =DO_UPCAST (BlockDirtyBitmapState, + common, common); + +if (state->bitmap) {Hmm, interesting, I thought, abort should not be called, if prepare failed, so the following may be done unconditionally?
I don't think so:
qmp_transaction:
...
state->ops->prepare(state, &local_err);
if (local_err) {
error_propagate(errp, local_err);
goto delete_and_fail;
}
...
delete_and_fail:
...
QTAILQ_FOREACH_REVERSE(state, &snap_bdrv_states, entry) {
if (state->ops->abort) {
state->ops->abort(state);
}
}Quoted 8 lines
Quoted 5 lines
blockdev.c +5
-0+bdrv_dirty_bitmap_skip_store (state->bitmap, false); +bdrv_dirty_bitmap_set_busy (state->bitmap, false); + } +} +[..]
Quoted 7 lines
OK, I agree, good idea to reuse BUSY and migration field here and avoid new API. Now release-prepare is less "release", but I don't see any real problems with it. Maybe, we need something to be noted in docs.
Hmm, as we are not in a hurry now, we may wait for Nikolay to check this on his scenarios.
OK, you have time. These patches are all sitting in my bitmaps branch and I haven't sent the PR for them yet.
--js
PATCH v3 3/3 iotests: test bitmap moving inside 254
In reply to John SnowFrom: Vladimir Sementsov-Ogievskiy <[email protected]>
Test persistent bitmap copying with and without removal of original bitmap.
--- tests/qemu-iotests/254 | 30 +++++++++++++- tests/qemu-iotests/254.out | 82 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 110 insertions(+), 2 deletions(-)
tests/qemu-iotests/254 +28 -2
@@ -1,6 +1,6 @@ #!/usr/bin/env python # -# Test external snapshot with bitmap copying. +# Test external snapshot with bitmap copying and moving. # # Copyright (c) 2019 Virtuozzo International GmbH. All rights reserved. #
@@ -32,6 +32,10 @@ vm = iotests.VM().add_drive(disk, opts='node-name=base') vm.launch() vm.qmp_log('block-dirty-bitmap-add', node='drive0', name='bitmap0') +vm.qmp_log('block-dirty-bitmap-add', node='drive0', name='bitmap1', + persistent=True) +vm.qmp_log('block-dirty-bitmap-add', node='drive0', name='bitmap2', + persistent=True) vm.hmp_qemu_io('drive0', 'write 0 512K')
@@ -39,16 +43,38 @@ vm.qmp_log('transaction', indent=2, actions=[ {'type': 'blockdev-snapshot-sync', 'data': {'device': 'drive0', 'snapshot-file': top, 'snapshot-node-name': 'snap'}}, + + # copy non-persistent bitmap0 {'type': 'block-dirty-bitmap-add', 'data': {'node': 'snap', 'name': 'bitmap0'}}, {'type': 'block-dirty-bitmap-merge', 'data': {'node': 'snap', 'target': 'bitmap0', - 'bitmaps': [{'node': 'base', 'name': 'bitmap0'}]}} + 'bitmaps': [{'node': 'base', 'name': 'bitmap0'}]}}, + + # copy persistent bitmap1, original will be saved to base image + {'type': 'block-dirty-bitmap-add', + 'data': {'node': 'snap', 'name': 'bitmap1', 'persistent': True}}, + {'type': 'block-dirty-bitmap-merge', + 'data': {'node': 'snap', 'target': 'bitmap1', + 'bitmaps': [{'node': 'base', 'name': 'bitmap1'}]}}, + + # move persistent bitmap1, original will be removed and not saved + # to base image + {'type': 'block-dirty-bitmap-add', + 'data': {'node': 'snap', 'name': 'bitmap2', 'persistent': True}}, + {'type': 'block-dirty-bitmap-merge', + 'data': {'node': 'snap', 'target': 'bitmap2', + 'bitmaps': [{'node': 'base', 'name': 'bitmap2'}]}}, + {'type': 'block-dirty-bitmap-remove', + 'data': {'node': 'base', 'name': 'bitmap2'}} ], filters=[iotests.filter_qmp_testfiles]) result = vm.qmp('query-block')['return'][0] log("query-block: device = {}, node-name = {}, dirty-bitmaps:".format( result['device'], result['inserted']['node-name'])) log(result['dirty-bitmaps'], indent=2) +log("\nbitmaps in backing image:") +log(result['inserted']['image']['backing-image']['format-specific'] \ + ['data']['bitmaps'], indent=2) vm.shutdown()
tests/qemu-iotests/254.out +82 -0
102 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v3 3/3] iotests: test bitmap moving inside 254
Quoted 16 lines · John Snow
From: Vladimir Sementsov-Ogievskiy <[email protected]>
Test persistent bitmap copying with and without removal of original bitmap.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/254 | 30 +++++++++++++- tests/qemu-iotests/254.out | 82 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 110 insertions(+), 2 deletions(-)tests/qemu-iotests/254 +0
-0
[...]
Quoted 11 lines
tests/qemu-iotests/254 +2
-1@@ -39,16 +43,38 @@ vm.qmp_log('transaction', indent=2, actions=[ {'type': 'blockdev-snapshot-sync', 'data': {'device': 'drive0', 'snapshot-file': top, 'snapshot-node-name': 'snap'}}, + + # copy non-persistent bitmap0 {'type': 'block-dirty-bitmap-add', 'data': {'node': 'snap', 'name': 'bitmap0'}}, {'type': 'block-dirty-bitmap-merge', 'data': {'node': 'snap', 'target': 'bitmap0', - 'bitmaps': [{'node': 'base', 'name': 'bitmap0'}]}}
tests/qemu-iotests/254 vm.qmp_log('transaction', indent=2, actions=[ · line 61
Quoted 10 lines
tests/qemu-iotests/254 +10
-0+ 'bitmaps': [{'node': 'base', 'name': 'bitmap0'}]}}, + + # copy persistent bitmap1, original will be saved to base image + {'type': 'block-dirty-bitmap-add', + 'data': {'node': 'snap', 'name': 'bitmap1', 'persistent': True}}, + {'type': 'block-dirty-bitmap-merge', + 'data': {'node': 'snap', 'target': 'bitmap1', + 'bitmaps': [{'node': 'base', 'name': 'bitmap1'}]}}, + + # move persistent bitmap1, original will be removed and not saved
*bitmap2
Quoted 9 lines
tests/qemu-iotests/254 +8
-0+ # to base image + {'type': 'block-dirty-bitmap-add', + 'data': {'node': 'snap', 'name': 'bitmap2', 'persistent': True}}, + {'type': 'block-dirty-bitmap-merge', + 'data': {'node': 'snap', 'target': 'bitmap2', + 'bitmaps': [{'node': 'base', 'name': 'bitmap2'}]}}, + {'type': 'block-dirty-bitmap-remove', + 'data': {'node': 'base', 'name': 'bitmap2'}} ], filters=[iotests.filter_qmp_testfiles])
John Snow
AppliedQuoted 32 lines · John Snow
Hi, this is a proposal based off of Vladimir's patchset:
[Qemu-devel] [PATCH 0/4] qapi: block-dirty-bitmap-remove transaction action===
V3:
===001/3:[----] [--] 'blockdev: reduce
aio_contextlocked sections in bitmap add/remove' 002/3:[0024] [FC] 'qapi: implement block-dirty-bitmap-remove transaction action' 003/3:[----] [--] 'iotests: test bitmap moving inside 254'- Changed "
squelch_persistence" to "skip_store"
- Use Max's suggestion for return expr===
V2:
===It replaces patches two and three with a modified patch (now patch 2) that foregoes the need for a hide()/unhide() bitmap API. I think it's suitable as a smaller alternative, but I'm not sure if it covers all of the use cases of the original series.
Patches 1 and 3 (formerly 4) included as-is.
John Snow (1):
qapi: implement block-dirty-bitmap-remove transaction actionVladimir Sementsov-Ogievskiy (2):
blockdev: reduceaio_contextlocked sections in bitmap add/remove
iotests: test bitmap moving inside 254
Quoted 10 lines
block.c | 2 +- block/dirty-bitmap.c | 15 +++-- blockdev.c | 105 ++++++++++++++++++++++++++------- include/block/dirty-bitmap.h | 2 +- migration/block-dirty-bitmap.c | 2 +- qapi/transaction.json | 2 + tests/qemu-iotests/254 | 30 +++++++++- tests/qemu-iotests/254.out | 82 +++++++++++++++++++++++++ 8 files changed, 206 insertions(+), 34 deletions(-)
Thanks, applied to my bitmaps tree:
https://github.com/jnsnow/qemu/commits/bitmaps
https://github.com/jnsnow/qemu.git
--js
(Vladimir: if this isn't amenable to you, it's going in for 4.2, so we have until the next freeze to change it. Let me know, OK?)
Quoted 48 lines · John Snow
Quoted 32 lines · John Snow
Hi, this is a proposal based off of Vladimir's patchset:
[Qemu-devel] [PATCH 0/4] qapi: block-dirty-bitmap-remove transaction action===
V3:
===001/3:[----] [--] 'blockdev: reduce
aio_contextlocked sections in bitmap add/remove' 002/3:[0024] [FC] 'qapi: implement block-dirty-bitmap-remove transaction action' 003/3:[----] [--] 'iotests: test bitmap moving inside 254'- Changed "
squelch_persistence" to "skip_store"
- Use Max's suggestion for return expr===
V2:
===It replaces patches two and three with a modified patch (now patch 2) that foregoes the need for a hide()/unhide() bitmap API. I think it's suitable as a smaller alternative, but I'm not sure if it covers all of the use cases of the original series.
Patches 1 and 3 (formerly 4) included as-is.
John Snow (1):
qapi: implement block-dirty-bitmap-remove transaction actionVladimir Sementsov-Ogievskiy (2):
blockdev: reduceaio_contextlocked sections in bitmap add/remove
iotests: test bitmap moving inside 254Quoted 10 lines
block.c | 2 +- block/dirty-bitmap.c | 15 +++-- blockdev.c | 105 ++++++++++++++++++++++++++------- include/block/dirty-bitmap.h | 2 +- migration/block-dirty-bitmap.c | 2 +- qapi/transaction.json | 2 + tests/qemu-iotests/254 | 30 +++++++++- tests/qemu-iotests/254.out | 82 +++++++++++++++++++++++++ 8 files changed, 206 insertions(+), 34 deletions(-)Thanks, applied to my bitmaps tree:
Quoted 9 lines
https://github.com/jnsnow/qemu/commits/bitmaps
https://github.com/jnsnow/qemu.git--js
(Vladimir: if this isn't amenable to you, it's going in for 4.2, so we have until the next freeze to change it. Let me know, OK?)
And finally I'm here :)
Thanks a lot for doing this job and for your explanations in other threads which I'm reading today and sorry for the delay! I'll look through these series soon.
Actually, my second child (girl:) was born a month ago, and then her elder brother was ill, so I took two weeks sick leave after about two weeks vacation and forget about work for a month.
Hmm. And Nikolay, who doing libvirt part is on vocation now (I started bitmap remove transaction series by his request), so I don't know about the end of the story with release and this functionality..
Anyway, it's cool, thanks!
Signature · 2 lines
Best regards, Vladimir
Quoted 69 lines · Vladimir Sementsov-Ogievskiy
Quoted 48 lines · John Snow
Quoted 32 lines · John Snow
Hi, this is a proposal based off of Vladimir's patchset:
[Qemu-devel] [PATCH 0/4] qapi: block-dirty-bitmap-remove transaction action===
V3:
===001/3:[----] [--] 'blockdev: reduce
aio_contextlocked sections in bitmap add/remove' 002/3:[0024] [FC] 'qapi: implement block-dirty-bitmap-remove transaction action' 003/3:[----] [--] 'iotests: test bitmap moving inside 254'- Changed "
squelch_persistence" to "skip_store"
- Use Max's suggestion for return expr===
V2:
===It replaces patches two and three with a modified patch (now patch 2) that foregoes the need for a hide()/unhide() bitmap API. I think it's suitable as a smaller alternative, but I'm not sure if it covers all of the use cases of the original series.
Patches 1 and 3 (formerly 4) included as-is.
John Snow (1):
qapi: implement block-dirty-bitmap-remove transaction actionVladimir Sementsov-Ogievskiy (2):
blockdev: reduceaio_contextlocked sections in bitmap add/remove
iotests: test bitmap moving inside 254Quoted 10 lines
block.c | 2 +- block/dirty-bitmap.c | 15 +++-- blockdev.c | 105 ++++++++++++++++++++++++++------- include/block/dirty-bitmap.h | 2 +- migration/block-dirty-bitmap.c | 2 +- qapi/transaction.json | 2 + tests/qemu-iotests/254 | 30 +++++++++- tests/qemu-iotests/254.out | 82 +++++++++++++++++++++++++ 8 files changed, 206 insertions(+), 34 deletions(-)Thanks, applied to my bitmaps tree:
Quoted 9 lines
https://github.com/jnsnow/qemu/commits/bitmaps
https://github.com/jnsnow/qemu.git--js
(Vladimir: if this isn't amenable to you, it's going in for 4.2, so we have until the next freeze to change it. Let me know, OK?)
And finally I'm here :)
Thanks a lot for doing this job and for your explanations in other threads which I'm reading today and sorry for the delay! I'll look through these series soon.
Actually, my second child (girl:) was born a month ago, and then her elder brother was ill, so I took two weeks sick leave after about two weeks vacation and forget about work for a month.
Quoted 6 lines
Hmm. And Nikolay, who doing libvirt part is on vocation now (I started bitmap remove transaction series by his request), so I don't know about the end of the story with release and this functionality..
Anyway, it's cool, thanks!
Wow!
Congratulations Vladimir!
Quoted 76 lines · John Snow
Quoted 73 lines · Vladimir Sementsov-Ogievskiy
Quoted 48 lines · John Snow
Quoted 32 lines · John Snow
Hi, this is a proposal based off of Vladimir's patchset:
[Qemu-devel] [PATCH 0/4] qapi: block-dirty-bitmap-remove transaction action===
V3:
===001/3:[----] [--] 'blockdev: reduce
aio_contextlocked sections in bitmap add/remove' 002/3:[0024] [FC] 'qapi: implement block-dirty-bitmap-remove transaction action' 003/3:[----] [--] 'iotests: test bitmap moving inside 254'- Changed "
squelch_persistence" to "skip_store"
- Use Max's suggestion for return expr===
V2:
===It replaces patches two and three with a modified patch (now patch 2) that foregoes the need for a hide()/unhide() bitmap API. I think it's suitable as a smaller alternative, but I'm not sure if it covers all of the use cases of the original series.
Patches 1 and 3 (formerly 4) included as-is.
John Snow (1):
qapi: implement block-dirty-bitmap-remove transaction actionVladimir Sementsov-Ogievskiy (2):
blockdev: reduceaio_contextlocked sections in bitmap add/remove
iotests: test bitmap moving inside 254Quoted 10 lines
block.c | 2 +- block/dirty-bitmap.c | 15 +++-- blockdev.c | 105 ++++++++++++++++++++++++++------- include/block/dirty-bitmap.h | 2 +- migration/block-dirty-bitmap.c | 2 +- qapi/transaction.json | 2 + tests/qemu-iotests/254 | 30 +++++++++- tests/qemu-iotests/254.out | 82 +++++++++++++++++++++++++ 8 files changed, 206 insertions(+), 34 deletions(-)Thanks, applied to my bitmaps tree:
Quoted 9 lines
https://github.com/jnsnow/qemu/commits/bitmaps
https://github.com/jnsnow/qemu.git--js
(Vladimir: if this isn't amenable to you, it's going in for 4.2, so we have until the next freeze to change it. Let me know, OK?)
And finally I'm here :)
Thanks a lot for doing this job and for your explanations in other threads which I'm reading today and sorry for the delay! I'll look through these series soon.
Actually, my second child (girl:) was born a month ago, and then her elder brother was ill, so I took two weeks sick leave after about two weeks vacation and forget about work for a month.
Hmm. And Nikolay, who doing libvirt part is on vocation now (I started bitmap remove transaction series by his request), so I don't know about the end of the story with release and this functionality..
Quoted 7 lines
Quoted 2 lines
Anyway, it's cool, thanks!
Wow!
Congratulations Vladimir!
Thank you!
Signature · 2 lines
Best regards, Vladimir