bitmaps: remove x- prefix from QMP api
v6 cover bitmaps: remove x- prefix from QMP api
Fix some outstanding bugs, change the design of an API element, remove the x- prefix to signify stability, and add iotests.
V6:
- Cover letter touchups
- 09: Rewrite qmp_filter to cope with lists or dicts.
- 11: small touchups.
V5:
002: New bugfix.
003: I forgot to actually capture state->bitmap anywhere,
which is needed for restoration...
007: Better commit message
- use .items() instead of .keys() to save a lookup [Vladimir]
- use a sequence of tuples to preserve ordering in the OrderedDict constructor [Vladimir]
- Move the sort_keys boolean up from patch 010
008: Better commit message
009: Better commit message
010: Moved the sort_keys function up to patch 007
011: Expanded this test considerably:
- query_bitmaps can now show empty results,
and prefixes results with "bitmaps:" in the log
- logging declarations are one line [Vladimir]
- Added a bad version of the bitmap handoff transaction [Eric]
- Added a bad version of the bitmap merge transaction, which revealed a problem that patch 02 now addresses [Eric]
- Added bitmap removal / cleanup [Eric]
- Added newline at end of file. [Eric]
V4:
- Removed patches 1-5 which have been staged
- Rewrite qmp_log entirely, split into three patches
- Pretty-printing has been extended to log() as well as qmp_log()
- Adjust iotest 236 to be format generic instead of qcow2 [Vladimir]
- Adjust iotest 236 to not reduplicate serialization work [Vladimir]
- Many other small touchups
V3:
- Reworked qmp_log to pretty-print the outgoing command, too [Vladimir]
- Modified test to log only bitmap information [Vladimir]
- Test disable/enable transaction toggle [Eric]
John Snow (11):
blockdev: abort transactions in reverse order block/dirty-bitmap: remove assertion from restore
blockdev: n-ary bitmap merge
block: remove 'x' prefix from experimental bitmap APIs iotests.py: don't abort if IMGKEYSECRET is undefined
iotests: add filter_generated_node_ids
iotests: add qmp recursive sorting function
iotests: remove default filters from qmp_log
iotests: change qmp_log filters to expect QMP objects only
iotests: implement pretty-print for log and qmp_log
iotests: add iotest 236 for testing bitmap merge
block/dirty-bitmap.c | 1 - blockdev.c | 107 +++++++---- qapi/block-core.json | 56 +++--- qapi/transaction.json | 12 +- tests/qemu-iotests/206 | 8 +- tests/qemu-iotests/223 | 4 +- tests/qemu-iotests/236 | 161 ++++++++++++++++ tests/qemu-iotests/236.out | 351 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 64 ++++++- 10 files changed, 677 insertions(+), 88 deletions(-) create mode 100755 tests/qemu-iotests/236 create mode 100644 tests/qemu-iotests/236.out
Signature · 1 line
2.17.2
PATCH v6 1/11 blockdev: abort transactions in reverse order
Presently, we abort transactions in the same order they were processed in. Bitmap commands, though, attempt to restore backup data structures on abort.
That's not valid, they need to be aborted in reverse chronological order.
Replace the QSIMPLEQ data structure with a QTAILQ one, so we can iterate in reverse for the abort phase of the transaction.
--- blockdev.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
blockdev.c +7 -7
@@ -1339,7 +1339,7 @@ struct BlkActionState {const BlkActionOps *ops;JobTxn *block_job_txn;TransactionProperties *txn_props; -QSIMPLEQ_ENTRY (BlkActionState) entry; +QTAILQ_ENTRY (BlkActionState) entry; };/* internal snapshot private data */
@@ -2266,8 +2266,8 @@ void qmp_transaction(TransactionActionList *dev_list,BlkActionState *state, *next;Error *local_err =NULL ; - QSIMPLEQ_HEAD(snap_bdrv_states, BlkActionState) snap_bdrv_states; -QSIMPLEQ_INIT (&snap_bdrv_states); + QTAILQ_HEAD(snap_bdrv_states, BlkActionState) snap_bdrv_states; +QTAILQ_INIT (&snap_bdrv_states);/* Does this transaction get canceled as a group on failure? * If not, we don't really need to make a JobTxn.
@@ -2298,7 +2298,7 @@ void qmp_transaction(TransactionActionList *dev_list, state->action = dev_info; state->block_job_txn = block_job_txn; state->txn_props = props; -QSIMPLEQ_INSERT_TAIL (&snap_bdrv_states, state, entry); +QTAILQ_INSERT_TAIL (&snap_bdrv_states, state, entry); state->ops->prepare (state, &local_err);if (local_err) {
@@ -2307,7 +2307,7 @@ void qmp_transaction(TransactionActionList *dev_list, } } - QSIMPLEQ_FOREACH(state, &snap_bdrv_states, entry) { + QTAILQ_FOREACH(state, &snap_bdrv_states, entry) {if (state->ops->commit) { state->ops->commit (state); }
@@ -2318,13 +2318,13 @@ void qmp_transaction(TransactionActionList *dev_list, delete_and_fail:/* failure, and it is all-or-none; roll back all operations */ - QSIMPLEQ_FOREACH(state, &snap_bdrv_states, entry) { + QTAILQ_FOREACH_REVERSE(state, &snap_bdrv_states, snap_bdrv_states, entry) {if (state->ops->abort) { state->ops->abort (state); } } exit: - QSIMPLEQ_FOREACH_SAFE(state, &snap_bdrv_states, entry, next) { + QTAILQ_FOREACH_SAFE(state, &snap_bdrv_states, entry, next) {if (state->ops->clean) { state->ops->clean (state); }
Signature · 1 line
2.17.2
Re: [Qemu-devel] [PATCH v6 01/11] blockdev: abort transactions in reverse order
Quoted 7 lines · John Snow
Presently, we abort transactions in the same order they were processed in. Bitmap commands, though, attempt to restore backup data structures on abort.
That's not valid, they need to be aborted in reverse chronological order.
Replace the QSIMPLEQ data structure with a QTAILQ one, so we can iterate in reverse for the abort phase of the transaction.
This patch conflicts with Paolo's improvements to QTAILQ that just landed; the resolution should be obvious, but at this point, you'll want to post a v7.
Or did you want me to try and fix the conflict, and take the series through my NBD tree since I have patches based on top of it?
Quoted 8 lines
Signed-off-by John Snow <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- blockdev.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v6 01/11] blockdev: abort transactions in reverse order
Quoted 9 lines · Eric Blake
Quoted 7 lines · John Snow
Presently, we abort transactions in the same order they were processed in. Bitmap commands, though, attempt to restore backup data structures on abort.
That's not valid, they need to be aborted in reverse chronological order.
Replace the QSIMPLEQ data structure with a QTAILQ one, so we can iterate in reverse for the abort phase of the transaction.
Quoted 6 lines
This patch conflicts with Paolo's improvements to QTAILQ that just landed; the resolution should be obvious, but at this point, you'll want to post a v7.
Or did you want me to try and fix the conflict, and take the series through my NBD tree since I have patches based on top of it?
Actually, I went ahead and queued this series in my NBD tree after making the fixes, so I can get the removal of x- in place sooner for easier integration testing on my libvirt code.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v6 2/11 block/dirty-bitmap: remove assertion from restore
In reply to John SnowWhen making a backup of a dirty bitmap (for transactions), we want to restore that backup whether or not the bitmap is enabled.
It is perfectly valid to write into bitmaps that are disabled. It is only illegitimate for the guest to have done so.
Remove this assertion.
--- block/dirty-bitmap.c | 1 - 1 file changed, 1 deletion(-)
block/dirty-bitmap.c +0 -1
@@ -625,7 +625,6 @@ void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap **out)void bdrv_restore_dirty_bitmap (BdrvDirtyBitmap *bitmap,HBitmap *backup) {HBitmap *tmp = bitmap->bitmap; -assert (bdrv_dirty_bitmap_enabled (bitmap));assert (!bdrv_dirty_bitmap_readonly (bitmap)); bitmap->bitmap = backup;hbitmap_free (tmp);
Signature · 1 line
2.17.2
PATCH v6 3/11 blockdev: n-ary bitmap merge
In reply to John SnowEspecially outside of transactions, it is helpful to provide all-or-nothing semantics for bitmap merges. This facilitates the coalescing of multiple bitmaps into a single target for the "checkpoint" interpretation when assembling bitmaps that represent arbitrary points in time from component bitmaps.
This is an incompatible change from the preliminary version of the API.
--- blockdev.c | 75 ++++++++++++++++++++++++++++++-------------- qapi/block-core.json | 22 ++++++------- 2 files changed, 62 insertions(+), 35 deletions(-)
blockdev.c +51 -24
@@ -2119,33 +2119,28 @@ static void block_dirty_bitmap_disable_abort(BlkActionState *common) } } +static BdrvDirtyBitmap *do_block_dirty_bitmap_merge (const char *node, +const char *target, +strList *bitmaps, +HBitmap **backup, +Error **errp); +static void block_dirty_bitmap_merge_prepare (BlkActionState *common,Error **errp) {BlockDirtyBitmapMerge *action;BlockDirtyBitmapState *state =DO_UPCAST (BlockDirtyBitmapState, common, common); -BdrvDirtyBitmap *merge_source;if (action_check_completion_mode (common, errp) <0 ) {return ; } action = common->action->u.x_block_dirty_bitmap_merge.data; - state->bitmap =block_dirty_bitmap_lookup (action->node, - action->dst_name, - &state->bs, - errp); -if (!state->bitmap) { -return ; - } - merge_source =bdrv_find_dirty_bitmap (state->bs, action->src_name); -if (!merge_source) { -return ; - } - -bdrv_merge_dirty_bitmap (state->bitmap, merge_source, &state->backup, errp); + state->bitmap =do_block_dirty_bitmap_merge (action->node, action->target, + action->bitmaps, &state->backup, + errp); }static void abort_prepare (BlkActionState *common,Error **errp)
@@ -2977,24 +2972,56 @@ void qmp_x_block_dirty_bitmap_disable(const char *node, const char *name,bdrv_disable_dirty_bitmap (bitmap); } -void qmp_x_block_dirty_bitmap_merge (const char *node,const char *dst_name, -const char *src_name,Error **errp) +static BdrvDirtyBitmap *do_block_dirty_bitmap_merge (const char *node, +const char *target, +strList *bitmaps, +HBitmap **backup, +Error **errp) {BlockDriverState *bs; -BdrvDirtyBitmap *dst, *src; +BdrvDirtyBitmap *dst, *src, *anon; +strList *lst; +Error *local_err =NULL ; - dst =block_dirty_bitmap_lookup (node, dst_name, &bs, errp); + dst =block_dirty_bitmap_lookup (node, target, &bs, errp);if (!dst) { -return ; +return NULL ; } - src =bdrv_find_dirty_bitmap (bs, src_name); -if (!src) { -error_setg (errp,"Dirty bitmap '%s' not found" , src_name); -return ; + anon =bdrv_create_dirty_bitmap (bs,bdrv_dirty_bitmap_granularity (dst), +NULL , errp); +if (!anon) { +return NULL ; } -bdrv_merge_dirty_bitmap (dst, src,NULL , errp); +for (lst = bitmaps; lst; lst = lst->next) { + src =bdrv_find_dirty_bitmap (bs, lst->value); +if (!src) { +error_setg (errp,"Dirty bitmap '%s' not found" , lst->value); + dst =NULL ; + goto out; + } + +bdrv_merge_dirty_bitmap (anon, src,NULL , &local_err); +if (local_err) { +error_propagate (errp, local_err); + dst =NULL ; + goto out; + } + } + +/* Merge into dst; dst is unchanged on failure. */ +bdrv_merge_dirty_bitmap (dst, anon, backup, errp); + + out: +bdrv_release_dirty_bitmap (bs, anon); +return dst; +} + +void qmp_x_block_dirty_bitmap_merge (const char *node,const char *target, +strList *bitmaps,Error **errp) +{ +do_block_dirty_bitmap_merge (node, target, bitmaps,NULL , errp); } BlockDirtyBitmapSha256 *qmp_x_debug_block_dirty_bitmap_sha256(const char *node,
qapi/block-core.json +11 -11
@@ -1821,14 +1821,14 @@ # # @node: name of device/node which the bitmap is tracking # -# @dst_name: name of the destination dirty bitmap +# @target: name of the destination dirty bitmap # -# @src_name: name of the source dirty bitmap +# @bitmaps: name(s) of the source dirty bitmap(s) # # Since: 3.0 ## { 'struct': 'BlockDirtyBitmapMerge', - 'data': { 'node': 'str', 'dst_name': 'str', 'src_name': 'str' } } + 'data': { 'node': 'str', 'target': 'str', 'bitmaps': ['str'] } } ## # @block-dirty-bitmap-add:
@@ -1943,23 +1943,23 @@ ## # @x-block-dirty-bitmap-merge: # -# FIXME: Rename @src_name and @dst_name to src-name and dst-name. -# -# Merge @src_name dirty bitmap to @dst_name dirty bitmap. @src_name dirty -# bitmap is unchanged. On error, @dst_name is unchanged. +# Merge dirty bitmaps listed in @bitmaps to the @target dirty bitmap. +# The @bitmaps dirty bitmaps are unchanged. +# On error, @target is unchanged. # # Returns: nothing on success # If @node is not a valid block device, DeviceNotFound -# If @dst_name or @src_name is not found, GenericError -# If bitmaps has different sizes or granularities, GenericError +# If any bitmap in @bitmaps or @target is not found, GenericError +# If any of the bitmaps have different sizes or granularities, +# GenericError # # Since: 3.0 # # Example: # # -> { "execute": "x-block-dirty-bitmap-merge", -# "arguments": { "node": "drive0", "dst_name": "bitmap0", -# "src_name": "bitmap1" } } +# "arguments": { "node": "drive0", "target": "bitmap0", +# "bitmaps": ["bitmap1"] } } # <- { "return": {} } # ##
Signature · 1 line
2.17.2
PATCH v6 4/11 block: remove 'x' prefix from experimental bitmap APIs
In reply to John SnowThe 'x' prefix was added because I was uncertain of the direction we'd take for the libvirt API. With the general approach solidified, I feel comfortable committing to this API for 4.0.
--- blockdev.c | 22 +++++++++++----------- qapi/block-core.json | 34 +++++++++++++++++----------------- qapi/transaction.json | 12 ++++++------ tests/qemu-iotests/223 | 4 ++-- 4 files changed, 36 insertions(+), 36 deletions(-)
blockdev.c +11 -11
@@ -1963,7 +1963,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState *common, action->has_granularity, action->granularity, action->has_persistent, action->persistent, action->has_autoload, action->autoload, - action->has_x_disabled, action->x_disabled, + action->has_disabled, action->disabled, &local_err);if (!local_err) {
@@ -2048,7 +2048,7 @@ static void block_dirty_bitmap_enable_prepare(BlkActionState *common,return ; } - action = common->action->u.x_block_dirty_bitmap_enable.data; + action = common->action->u.block_dirty_bitmap_enable.data; state->bitmap = block_dirty_bitmap_lookup(action->node, action->name, NULL,
@@ -2089,7 +2089,7 @@ static void block_dirty_bitmap_disable_prepare(BlkActionState *common,return ; } - action = common->action->u.x_block_dirty_bitmap_disable.data; + action = common->action->u.block_dirty_bitmap_disable.data; state->bitmap = block_dirty_bitmap_lookup(action->node, action->name, NULL,
@@ -2136,7 +2136,7 @@ static void block_dirty_bitmap_merge_prepare(BlkActionState *common,return ; } - action = common->action->u.x_block_dirty_bitmap_merge.data; + action = common->action->u.block_dirty_bitmap_merge.data; state->bitmap = do_block_dirty_bitmap_merge(action->node, action->target, action->bitmaps, &state->backup,
@@ -2204,17 +2204,17 @@ static const BlkActionOps actions[] = { .commit = block_dirty_bitmap_free_backup, .abort = block_dirty_bitmap_restore, }, - [TRANSACTION_ACTION_KIND_X_BLOCK_DIRTY_BITMAP_ENABLE] = { + [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_ENABLE] = { .instance_size = sizeof(BlockDirtyBitmapState), .prepare = block_dirty_bitmap_enable_prepare, .abort = block_dirty_bitmap_enable_abort, }, - [TRANSACTION_ACTION_KIND_X_BLOCK_DIRTY_BITMAP_DISABLE] = { + [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_DISABLE] = { .instance_size = sizeof(BlockDirtyBitmapState), .prepare = block_dirty_bitmap_disable_prepare, .abort = block_dirty_bitmap_disable_abort, }, - [TRANSACTION_ACTION_KIND_X_BLOCK_DIRTY_BITMAP_MERGE] = { + [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_MERGE] = { .instance_size = sizeof(BlockDirtyBitmapState), .prepare = block_dirty_bitmap_merge_prepare, .commit = block_dirty_bitmap_free_backup,
@@ -2930,7 +2930,7 @@ void qmp_block_dirty_bitmap_clear(const char *node, const char *name,bdrv_clear_dirty_bitmap (bitmap,NULL ); } -void qmp_x_block_dirty_bitmap_enable (const char *node,const char *name, +void qmp_block_dirty_bitmap_enable (const char *node,const char *name,Error **errp) {BlockDriverState *bs;
@@ -2951,7 +2951,7 @@ void qmp_x_block_dirty_bitmap_enable(const char *node, const char *name,bdrv_enable_dirty_bitmap (bitmap); } -void qmp_x_block_dirty_bitmap_disable (const char *node,const char *name, +void qmp_block_dirty_bitmap_disable (const char *node,const char *name,Error **errp) {BlockDriverState *bs;
@@ -3018,8 +3018,8 @@ static BdrvDirtyBitmap *do_block_dirty_bitmap_merge(const char *node,return dst; } -void qmp_x_block_dirty_bitmap_merge (const char *node,const char *target, -strList *bitmaps,Error **errp) +void qmp_block_dirty_bitmap_merge (const char *node,const char *target, +strList *bitmaps,Error **errp) {do_block_dirty_bitmap_merge (node, target, bitmaps,NULL , errp); }
qapi/block-core.json +17 -17
@@ -1806,15 +1806,15 @@ # Currently, all dirty tracking bitmaps are loaded from Qcow2 on # open. # -# @x-disabled: the bitmap is created in the disabled state, which means that -# it will not track drive changes. The bitmap may be enabled with -# x-block-dirty-bitmap-enable. Default is false. (Since: 3.0) +# @disabled: the bitmap is created in the disabled state, which means that +# it will not track drive changes. The bitmap may be enabled with +# block-dirty-bitmap-enable. Default is false. (Since: 4.0) # # Since: 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', - '*persistent': 'bool', '*autoload': 'bool', '*x-disabled': 'bool' } } + '*persistent': 'bool', '*autoload': 'bool', '*disabled': 'bool' } } ## # @BlockDirtyBitmapMerge:
@@ -1825,7 +1825,7 @@ # # @bitmaps: name(s) of the source dirty bitmap(s) # -# Since: 3.0 +# Since: 4.0 ## { 'struct': 'BlockDirtyBitmapMerge', 'data': { 'node': 'str', 'target': 'str', 'bitmaps': ['str'] } }
@@ -1899,7 +1899,7 @@ 'data': 'BlockDirtyBitmap' } ## -# @x-block-dirty-bitmap-enable: +# @block-dirty-bitmap-enable: # # Enables a dirty bitmap so that it will begin tracking disk changes. #
@@ -1907,20 +1907,20 @@ # If @node is not a valid block device, DeviceNotFound # If @name is not found, GenericError with an explanation # -# Since: 3.0 +# Since: 4.0 # # Example: # -# -> { "execute": "x-block-dirty-bitmap-enable", +# -> { "execute": "block-dirty-bitmap-enable", # "arguments": { "node": "drive0", "name": "bitmap0" } } # <- { "return": {} } # ## - { 'command': 'x-block-dirty-bitmap-enable', + { 'command': 'block-dirty-bitmap-enable', 'data': 'BlockDirtyBitmap' } ## -# @x-block-dirty-bitmap-disable: +# @block-dirty-bitmap-disable: # # Disables a dirty bitmap so that it will stop tracking disk changes. #
@@ -1928,20 +1928,20 @@ # If @node is not a valid block device, DeviceNotFound # If @name is not found, GenericError with an explanation # -# Since: 3.0 +# Since: 4.0 # # Example: # -# -> { "execute": "x-block-dirty-bitmap-disable", +# -> { "execute": "block-dirty-bitmap-disable", # "arguments": { "node": "drive0", "name": "bitmap0" } } # <- { "return": {} } # ## - { 'command': 'x-block-dirty-bitmap-disable', + { 'command': 'block-dirty-bitmap-disable', 'data': 'BlockDirtyBitmap' } ## -# @x-block-dirty-bitmap-merge: +# @block-dirty-bitmap-merge: # # Merge dirty bitmaps listed in @bitmaps to the @target dirty bitmap. # The @bitmaps dirty bitmaps are unchanged.
@@ -1953,17 +1953,17 @@ # If any of the bitmaps have different sizes or granularities, # GenericError # -# Since: 3.0 +# Since: 4.0 # # Example: # -# -> { "execute": "x-block-dirty-bitmap-merge", +# -> { "execute": "block-dirty-bitmap-merge", # "arguments": { "node": "drive0", "target": "bitmap0", # "bitmaps": ["bitmap1"] } } # <- { "return": {} } # ## - { 'command': 'x-block-dirty-bitmap-merge', + { 'command': 'block-dirty-bitmap-merge', 'data': 'BlockDirtyBitmapMerge' } ##
qapi/transaction.json +6 -6
@@ -46,9 +46,9 @@ # - @abort: since 1.6 # - @block-dirty-bitmap-add: since 2.5 # - @block-dirty-bitmap-clear: since 2.5 -# - @x-block-dirty-bitmap-enable: since 3.0 -# - @x-block-dirty-bitmap-disable: since 3.0 -# - @x-block-dirty-bitmap-merge: since 3.1 +# - @block-dirty-bitmap-enable: since 4.0 +# - @block-dirty-bitmap-disable: since 4.0 +# - @block-dirty-bitmap-merge: since 4.0 # - @blockdev-backup: since 2.3 # - @blockdev-snapshot: since 2.5 # - @blockdev-snapshot-internal-sync: since 1.7
@@ -62,9 +62,9 @@ 'abort': 'Abort', 'block-dirty-bitmap-add': 'BlockDirtyBitmapAdd', 'block-dirty-bitmap-clear': 'BlockDirtyBitmap', - 'x-block-dirty-bitmap-enable': 'BlockDirtyBitmap', - 'x-block-dirty-bitmap-disable': 'BlockDirtyBitmap', - 'x-block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge', + 'block-dirty-bitmap-enable': 'BlockDirtyBitmap', + 'block-dirty-bitmap-disable': 'BlockDirtyBitmap', + 'block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge', 'blockdev-backup': 'BlockdevBackup', 'blockdev-snapshot': 'BlockdevSnapshot', 'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal',
tests/qemu-iotests/223 +2 -2
@@ -112,9 +112,9 @@ _send_qemu_cmd $QEMU_HANDLE '{"execute":"qmp_capabilities"}' "return" _send_qemu_cmd $QEMU_HANDLE '{"execute":"blockdev-add", "arguments":{"driver":"qcow2", "node-name":"n", "file":{"driver":"file", "filename":"'"$TEST_IMG"'"}}}' "return" -_send_qemu_cmd $QEMU_HANDLE '{"execute":"x-block-dirty-bitmap-disable", +_send_qemu_cmd $QEMU_HANDLE '{"execute":"block-dirty-bitmap-disable", "arguments":{"node":"n", "name":"b"}}' "return" -_send_qemu_cmd $QEMU_HANDLE '{"execute":"x-block-dirty-bitmap-disable", +_send_qemu_cmd $QEMU_HANDLE '{"execute":"block-dirty-bitmap-disable", "arguments":{"node":"n", "name":"b2"}}' "return" _send_qemu_cmd $QEMU_HANDLE '{"execute":"nbd-server-start", "arguments":{"addr":{"type":"unix",
Signature · 1 line
2.17.2
Re: [Qemu-devel] [PATCH v6 04/11] block: remove 'x' prefix from experimental bitmap APIs
Quoted 7 lines · John Snow
The 'x' prefix was added because I was uncertain of the direction we'd take for the libvirt API. With the general approach solidified, I feel comfortable committing to this API for 4.0.
Signed-off-by John Snow <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
Quoted 6 lines
--- blockdev.c | 22 +++++++++++----------- qapi/block-core.json | 34 +++++++++++++++++----------------- qapi/transaction.json | 12 ++++++------ tests/qemu-iotests/223 | 4 ++-- 4 files changed, 36 insertions(+), 36 deletions(-)
Misses the conversion of x-nbd-server-add-bitmap in qapi/block.json (could be a separate patch), but that's also one of the commands that libvirt will need to use alongside all the bitmap changes.
Speaking of x-nbd-server-add-bitmap, does it actually have to be a separate command, or should it just be an additional parameter to the existing nbd-server-add? There are two directions this could go:
As a single command, it enforces that there is no window of time where the export is available without the dirty bitmap. However, it also means we can only export at most one bitmap, unless we make the parameter take an array of bitmaps to export. Fewer commands also means less roll-back scenarios in libvirt (no need to worry about the export being added but the bitmap failing).
As a separate command, we could in the future allow the server to export multiple bitmaps at once by calling the add-bitmap command more than once; but right now, the code base does not allow multiple bitmap exports (a second bitmap export attempt fails).
I'm leaning towards the former (drop x-nbd-server-add-bitmap altogether, and instead add a bitmap parameter to nbd-server-add). Also, I'm leaning towards locking in that we only ever export one qemu:dirty-bitmap:FOO bitmap per export, and thus not worry about an array. Besides, if we make the command take an array now but enforce that the array has at most one element, then later relax it to take more than one element, there is no way to introspect that change; but if we make the command take a single string now, and later have a strong reason why exporting more than one bitmap at once makes sense, we could still maintain backcompat by using a QAPI alternate type between a string and an array, which would be introspectible.
I guess I should propose a patch for that, then...
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v6 04/11] block: remove 'x' prefix from experimental bitmap APIs
Quoted 47 lines · Eric Blake
Quoted 7 lines · John Snow
The 'x' prefix was added because I was uncertain of the direction we'd take for the libvirt API. With the general approach solidified, I feel comfortable committing to this API for 4.0.
Signed-off-by John Snow <[email protected]>Reviewed-by Eric Blake <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Quoted 6 lines
--- blockdev.c | 22 +++++++++++----------- qapi/block-core.json | 34 +++++++++++++++++----------------- qapi/transaction.json | 12 ++++++------ tests/qemu-iotests/223 | 4 ++-- 4 files changed, 36 insertions(+), 36 deletions(-)Misses the conversion of x-nbd-server-add-bitmap in
qapi/block.json(could be a separate patch), but that's also one of the commands that libvirt will need to use alongside all the bitmap changes.Speaking of x-nbd-server-add-bitmap, does it actually have to be a separate command, or should it just be an additional parameter to the existing nbd-server-add? There are two directions this could go:
As a single command, it enforces that there is no window of time where the export is available without the dirty bitmap. However, it also means we can only export at most one bitmap, unless we make the parameter take an array of bitmaps to export. Fewer commands also means less roll-back scenarios in libvirt (no need to worry about the export being added but the bitmap failing).
As a separate command, we could in the future allow the server to export multiple bitmaps at once by calling the add-bitmap command more than once; but right now, the code base does not allow multiple bitmap exports (a second bitmap export attempt fails).
I'm leaning towards the former (drop x-nbd-server-add-bitmap altogether, and instead add a bitmap parameter to nbd-server-add). Also, I'm leaning towards locking in that we only ever export one qemu:dirty-bitmap:FOO bitmap per export, and thus not worry about an array. Besides, if we make the command take an array now but enforce that the array has at most one element, then later relax it to take more than one element, there is no way to introspect that change; but if we make the command take a single string now, and later have a strong reason why exporting more than one bitmap at once makes sense, we could still maintain backcompat by using a QAPI alternate type between a string and an array, which would be introspectible.
Quoted 2 lines
I guess I should propose a patch for that, then...
Reasonable, I've no objections.
Signature · 2 lines
Best regards, Vladimir
PATCH v6 5/11 iotests.py: don't abort if IMGKEYSECRET is undefined
In reply to John SnowInstead of using os.environ[], use .get with a default of empty string to match the setup in check to allow us to import the iotests module (for debugging, say) without needing a crafted environment just to import the module.
--- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +1 -1
@@ -63,7 +63,7 @@ socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper') debug = False luks_default_secret_object = 'secret,id=keysec0,data=' + \ - os.environ['IMGKEYSECRET'] + os.environ.get('IMGKEYSECRET', '') luks_default_key_secret_opt = 'key-secret=keysec0'
Signature · 1 line
2.17.2
PATCH v6 6/11 iotests: add filter_generated_node_ids
In reply to John SnowTo mimic the common filter of the same name, but for the python tests.
--- tests/qemu-iotests/iotests.py | 3 +++ 1 file changed, 3 insertions(+)
tests/qemu-iotests/iotests.py +3 -0
@@ -239,6 +239,9 @@ def filter_testfiles(msg): prefix = os.path.join(test_dir, "%s-" % (os.getpid())) return msg.replace(prefix, 'TEST_DIR/PID-') +def filter_generated_node_ids(msg): + return re.sub("#block[0-9]+", "NODE_NAME", msg) + def filter_img_info(output, filename): lines = [] for line in output.split('\n'):
Signature · 1 line
2.17.2
PATCH v6 7/11 iotests: add qmp recursive sorting function
In reply to John SnowPython before 3.6 does not sort dictionaries (including kwargs). Therefore, printing QMP objects involves sorting the keys to have a predictable ordering in the iotests output. This means that iotests output will sometimes show arguments in an order not specified by the test author.
Presently, we accomplish this by using json.dumps' sort_keys argument, where we only serialize the arguments dictionary, but not the command.
However, if we want to pretty-print QMP objects being sent to the QEMU process, we need to build the entire command before logging it. Ordinarily, this would then involve "arguments" being sorted above "execute", which would necessitate a rather ugly and harder-to-read change to many iotests outputs.
To facilitate pretty-printing AND maintaining predictable output AND having "arguments" sort after "execute", add a custom sort function that takes a dictionary and recursively builds an OrderedDict that maintains the specific key order we wish to see in iotests output.
The qmp_log function uses this to build a QMP object that keeps "execute" above "arguments", but sorts all keys and keys in any subdicts in "arguments" lexicographically to maintain consistent iotests output, with no incompatible changes to any current test.
--- tests/qemu-iotests/iotests.py | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-)
tests/qemu-iotests/iotests.py +20 -4
@@ -30,6 +30,7 @@ import signal import logging import atexit import io +from collections import OrderedDict sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scripts')) import qtest
@@ -75,6 +76,16 @@ def qemu_img(*args): sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args)))) return exitcode +def ordered_kwargs(kwargs): + # kwargs prior to 3.6 are not ordered, so: + od = OrderedDict() + for k, v in sorted(kwargs.items()): + if isinstance(v, dict): + od[k] = ordered_kwargs(v) + else: + od[k] = v + return od + def qemu_img_create(*args): args = list(args)
@@ -257,8 +268,10 @@ def filter_img_info(output, filename): def log(msg, filters=[]): for flt in filters: msg = flt(msg) - if type(msg) is dict or type(msg) is list: - print(json.dumps(msg, sort_keys=True)) + if isinstance(msg, dict) or isinstance(msg, list): + # Don't sort if it's already sorted + do_sort = not isinstance(msg, OrderedDict) + print(json.dumps(msg, sort_keys=do_sort)) else: print(msg)
@@ -448,8 +461,11 @@ class VM(qtest.QEMUQtestMachine): return result def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs): - logmsg = '{"execute": "%s", "arguments": %s}' % \ - (cmd, json.dumps(kwargs, sort_keys=True)) + full_cmd = OrderedDict(( + ("execute", cmd), + ("arguments", ordered_kwargs(kwargs)) + )) + logmsg = json.dumps(full_cmd) log(logmsg, filters) result = self.qmp(cmd, **kwargs) log(json.dumps(result, sort_keys=True), filters)
Signature · 1 line
2.17.2
PATCH v6 8/11 iotests: remove default filters from qmp_log
In reply to John SnowSeveral places in iotests deal with serializing objects into JSON strings, but to add pretty-printing it seems desirable to localize all of those cases.
log() seems like a good candidate for that centralized behavior. log() can already serialize json objects, but when it does so, it assumes filters=[] operates on QMP objects, not strings.
qmp_log currently operates by dumping outgoing and incoming QMP objects into strings and filtering them assuming that filters=[] are string filters.
To have qmp_log use log's serialization, qmp_log will need to accept only qmp filters, not text filters.
However, only a single caller of qmp_log actually requires any filters at all. I remove the default filter and add it explicitly to the caller in preparation for refactoring qmp_log to use rich filters instead.
test 206 is amended to name the filter explicitly and the default is removed.
--- tests/qemu-iotests/206 | 8 ++++++-- tests/qemu-iotests/iotests.py | 2 +- 2 files changed, 7 insertions(+), 3 deletions(-)
tests/qemu-iotests/206 +6 -2
@@ -26,7 +26,9 @@ from iotests import imgfmt iotests.verify_image_format(supported_fmts=['qcow2']) def blockdev_create(vm, options): - result = vm.qmp_log('blockdev-create', job_id='job0', options=options) + result = vm.qmp_log('blockdev-create', + filters=[iotests.filter_testfiles], + job_id='job0', options=options) if 'return' in result: assert result['return'] == {}
@@ -52,7 +54,9 @@ with iotests.FilePath('t.qcow2') as disk_path, \ 'filename': disk_path, 'size': 0 }) - vm.qmp_log('blockdev-add', driver='file', filename=disk_path, + vm.qmp_log('blockdev-add', + filters=[iotests.filter_testfiles], + driver='file', filename=disk_path, node_name='imgfile') blockdev_create(vm, { 'driver': imgfmt,
tests/qemu-iotests/iotests.py +1 -1
@@ -460,7 +460,7 @@ class VM(qtest.QEMUQtestMachine): result.append(filter_qmp_event(ev)) return result - def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs): + def qmp_log(self, cmd, filters=[], **kwargs): full_cmd = OrderedDict(( ("execute", cmd), ("arguments", ordered_kwargs(kwargs))
Signature · 1 line
2.17.2
PATCH v6 9/11 iotests: change qmp_log filters to expect QMP objects only
In reply to John SnowAs laid out in the previous commit's message:
``` Several places in iotests deal with serializing objects into JSON strings, but to add pretty-printing it seems desirable to localize all of those cases. log() seems like a good candidate for that centralized behavior. log() can already serialize json objects, but when it does so, it assumes filters=[] operates on QMP objects, not strings. qmp_log currently operates by dumping outgoing and incoming QMP objects into strings and filtering them assuming that filters=[] are string filters. ```
Therefore:
Change qmp_log to treat filters as if they're always qmp object filters, then change the logging call to rely on log()'s ability to serialize QMP objects, so we're not duplicating that effort.
Add a qmp version of filter_testfiles and adjust the only caller using it for qmp_log to use the qmp version.
--- tests/qemu-iotests/206 | 4 ++-- tests/qemu-iotests/iotests.py | 28 +++++++++++++++++++++++++--- 2 files changed, 27 insertions(+), 5 deletions(-)
tests/qemu-iotests/206 +2 -2
@@ -27,7 +27,7 @@ iotests.verify_image_format(supported_fmts=['qcow2']) def blockdev_create(vm, options): result = vm.qmp_log('blockdev-create', - filters=[iotests.filter_testfiles], + filters=[iotests.filter_qmp_testfiles], job_id='job0', options=options) if 'return' in result:
@@ -55,7 +55,7 @@ with iotests.FilePath('t.qcow2') as disk_path, \ 'size': 0 }) vm.qmp_log('blockdev-add', - filters=[iotests.filter_testfiles], + filters=[iotests.filter_qmp_testfiles], driver='file', filename=disk_path, node_name='imgfile')
tests/qemu-iotests/iotests.py +25 -3
@@ -246,10 +246,33 @@ def filter_qmp_event(event): event['timestamp']['microseconds'] = 'USECS' return event +def filter_qmp(qmsg, filter_fn): + '''Given a string filter, filter a QMP object's values. + filter_fn takes a (key, value) pair.''' + # Iterate through either lists or dicts; + if isinstance(qmsg, list): + items = enumerate(qmsg) + else: + items = qmsg.items() + + for k, v in items: + if isinstance(v, list) or isinstance(v, dict): + qmsg[k] = filter_qmp(v, filter_fn) + else: + qmsg[k] = filter_fn(k, v) + return qmsg + def filter_testfiles(msg): prefix = os.path.join(test_dir, "%s-" % (os.getpid())) return msg.replace(prefix, 'TEST_DIR/PID-') +def filter_qmp_testfiles(qmsg): + def _filter(key, value): + if key == 'filename' or key == 'backing-file': + return filter_testfiles(value) + return value + return filter_qmp(qmsg, _filter) + def filter_generated_node_ids(msg): return re.sub("#block[0-9]+", "NODE_NAME", msg)
@@ -465,10 +488,9 @@ class VM(qtest.QEMUQtestMachine): ("execute", cmd), ("arguments", ordered_kwargs(kwargs)) )) - logmsg = json.dumps(full_cmd) - log(logmsg, filters) + log(full_cmd, filters) result = self.qmp(cmd, **kwargs) - log(json.dumps(result, sort_keys=True), filters) + log(result, filters) return result def run_job(self, job, auto_finalize=True, auto_dismiss=False):
Signature · 1 line
2.17.2
Re: [Qemu-devel] [PATCH v6 09/11] iotests: change qmp_log filters to expect QMP objects only
Quoted 66 lines · John Snow
As laid out in the previous commit's message:
``` Several places in iotests deal with serializing objects into JSON strings, but to add pretty-printing it seems desirable to localize all of those cases. log() seems like a good candidate for that centralized behavior. log() can already serialize json objects, but when it does so, it assumes filters=[] operates on QMP objects, not strings. qmp_log currently operates by dumping outgoing and incoming QMP objects into strings and filtering them assuming that filters=[] are string filters. ```Therefore:
Change
qmp_logto treat filters as if they're always qmp object filters, then change the logging call to rely on log()'s ability to serialize QMP objects, so we're not duplicating that effort.Add a qmp version of
filter_testfilesand adjust the only caller using it forqmp_logto use the qmp version.Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/206 | 4 ++-- tests/qemu-iotests/iotests.py | 28 +++++++++++++++++++++++++--- 2 files changed, 27 insertions(+), 5 deletions(-)tests/qemu-iotests/206 +2
-2@@ -27,7 +27,7 @@ iotests.verify_image_format(supported_fmts=['qcow2']) def blockdev_create(vm, options): result = vm.qmp_log('blockdev-create', - filters=[iotests.filter_testfiles], + filters=[iotests.filter_qmp_testfiles], job_id='job0', options=options) if 'return' in result:@@ -55,7 +55,7 @@ with iotests.FilePath('t.qcow2') as disk_path, \ 'size': 0 }) vm.qmp_log('blockdev-add', - filters=[iotests.filter_testfiles], + filters=[iotests.filter_qmp_testfiles], driver='file', filename=disk_path, node_name='imgfile')tests/qemu-iotests/iotests.py +5
-0@@ -246,10 +246,33 @@ def filter_qmp_event(event): event['timestamp']['microseconds'] = 'USECS' return event +def filter_qmp(qmsg, filter_fn): + '''Given a string filter, filter a QMP object's values. + filter_fn takes a (key, value) pair.''' + # Iterate through either lists or dicts; + if isinstance(qmsg, list):
tests/qemu-iotests/iotests.py def filter_qmp_event(event): · line 263
Quoted 10 lines
tests/qemu-iotests/iotests.py +10
-0+ items = enumerate(qmsg) + else: + items = qmsg.items() + + for k, v in items: + if isinstance(v, list) or isinstance(v, dict): + qmsg[k] = filter_qmp(v, filter_fn) + else: + qmsg[k] = filter_fn(k, v) + return qmsg
Hmm. This made me check, is enumerate applicable to dicts, and, yes it is.
so, it may be as easy as
for k, v in enumerate(qmsg):
if isinstance(v, list) or isinstance(v, dict):
qmsg[k] = filter_qmp(v, filter_fn)
else:
qmsg[k] = filter_fn(k, v)
with this:
But does it what you want?
So, for lists of strings, filter_fn will get pair of index in array and value?
On the other hand, we can adjust the behavior as we want when we introduce such filters.
Quoted 29 lines
tests/qemu-iotests/iotests.py +10
-3+ def filter_testfiles(msg): prefix = os.path.join(test_dir, "%s-" % (os.getpid())) return msg.replace(prefix, 'TEST_DIR/PID-') +def filter_qmp_testfiles(qmsg): + def _filter(key, value): + if key == 'filename' or key == 'backing-file': + return filter_testfiles(value) + return value + return filter_qmp(qmsg, _filter) + def filter_generated_node_ids(msg): return re.sub("#block[0-9]+", "NODE_NAME", msg)@@ -465,10 +488,9 @@ class VM(qtest.QEMUQtestMachine): ("execute", cmd), ("arguments", ordered_kwargs(kwargs)) )) - logmsg = json.dumps(full_cmd) - log(logmsg, filters) + log(full_cmd, filters) result = self.qmp(cmd, **kwargs) - log(json.dumps(result, sort_keys=True), filters) + log(result, filters) return result def run_job(self, job, auto_finalize=True, auto_dismiss=False):
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 09/11] iotests: change qmp_log filters to expect QMP objects only
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
Hmm. This made me check, is enumerate applicable to dicts, and, yes it is.
enumerate on dicts gives you a numerical index paired with the key, so... k is the numerical index and v is the key.
Quoted 5 lines
so, it may be as easy as
for k, v in enumerate(qmsg):
ifisinstance(v, list)orisinstance(v, dict):
qmsg[k] =filter_qmp(v, filter_fn)
^ this will break if qmsg was a dict.
Quoted 4 lines
else: qmsg[k] = filter_fn(k, v)with this:
Quoted 8 lines
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>But does it what you want?
So, for lists of strings,filter_fnwill get pair of index in array and value?
On the other hand, we can adjust the behavior as we want when we introduce such filters.
I'm not sure what I want...
Named index is the best, but isn't always available because like you say, the object we are passed is not always a dictionary. We could theoretically have a list of lists somewhere. What key would I pass then? It's meaningless.
I could change qmp_filter to pass the k,v to the filter when it finds the "last dictionary," i.e. if it recursively finds no more dictionaries underneath, it passes the value along regardless of whatever it finds. This is a bit more complex because I think I'd need a co-mutual recursive function to "travel" each value and report if there are further dictionaries down below.
Or, I could just leave the filter as-is where it only ever gets one, non-dict/non-list item and receives a key. Sometimes the key will now be a numerical index which is likely not useful...
I might stage this as-is for now. Let's revisit it before 4.0 rc0.
--js
Re: [Qemu-devel] [PATCH v6 09/11] iotests: change qmp_log filters to expect QMP objects only
Quoted 9 lines · John Snow
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
Hmm. This made me check, is enumerate applicable to dicts, and, yes it is.
enumerate on dicts gives you a numerical index paired with the key, so... k is the numerical index and v is the key.
hmm, yes, I was wrong. Ok, than I'm fine with your code. Take my r-b.
Quoted 44 lines
Quoted 6 lines
so, it may be as easy as
for k, v in enumerate(qmsg):
ifisinstance(v, list)orisinstance(v, dict):
qmsg[k] =filter_qmp(v, filter_fn)^ this will break if qmsg was a dict.
Quoted 4 lines
else: qmsg[k] = filter_fn(k, v)with this:
Quoted 8 lines
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>But does it what you want?
So, for lists of strings,filter_fnwill get pair of index in array and value?
On the other hand, we can adjust the behavior as we want when we introduce such filters.I'm not sure what I want...
Named index is the best, but isn't always available because like you say, the object we are passed is not always a dictionary. We could theoretically have a list of lists somewhere. What key would I pass then? It's meaningless.
I could change
qmp_filterto pass the k,v to the filter when it finds the "last dictionary," i.e. if it recursively finds no more dictionaries underneath, it passes the value along regardless of whatever it finds. This is a bit more complex because I think I'd need a co-mutual recursive function to "travel" each value and report if there are further dictionaries down below.Or, I could just leave the filter as-is where it only ever gets one, non-dict/non-list item and receives a key. Sometimes the key will now be a numerical index which is likely not useful...
I might stage this as-is for now. Let's revisit it before 4.0 rc0.
--js
Signature · 2 lines
Best regards, Vladimir
PATCH v6 10/11 iotests: implement pretty-print for log and qmp_log
In reply to John SnowIf iotests have lines exceeding >998 characters long, git doesn't want to send it plaintext to the list. We can solve this by allowing the iotests to use pretty printed QMP output that we can match against instead.
As a bonus, it's much nicer for human eyes too.
--- tests/qemu-iotests/iotests.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
tests/qemu-iotests/iotests.py +10 -5
@@ -288,13 +288,18 @@ def filter_img_info(output, filename): lines.append(line) return '\n'.join(lines) -def log(msg, filters=[]): +def log(msg, filters=[], indent=None): + '''Logs either a string message or a JSON serializable message (like QMP). + If indent is provided, JSON serializable messages are pretty-printed.''' for flt in filters: msg = flt(msg) if isinstance(msg, dict) or isinstance(msg, list): + # Python < 3.4 needs to know not to add whitespace when pretty-printing: + separators = (', ', ': ') if indent is None else (',', ': ') # Don't sort if it's already sorted do_sort = not isinstance(msg, OrderedDict) - print(json.dumps(msg, sort_keys=do_sort)) + print(json.dumps(msg, sort_keys=do_sort, + indent=indent, separators=separators)) else: print(msg)
@@ -483,14 +488,14 @@ class VM(qtest.QEMUQtestMachine): result.append(filter_qmp_event(ev)) return result - def qmp_log(self, cmd, filters=[], **kwargs): + def qmp_log(self, cmd, filters=[], indent=None, **kwargs): full_cmd = OrderedDict(( ("execute", cmd), ("arguments", ordered_kwargs(kwargs)) )) - log(full_cmd, filters) + log(full_cmd, filters, indent=indent) result = self.qmp(cmd, **kwargs) - log(result, filters) + log(result, filters, indent=indent) return result def run_job(self, job, auto_finalize=True, auto_dismiss=False):
Signature · 1 line
2.17.2
PATCH v6 11/11 iotests: add iotest 236 for testing bitmap merge
In reply to John SnowNew interface, new smoke test.
--- tests/qemu-iotests/236 | 161 +++++++++++++++++ tests/qemu-iotests/236.out | 351 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 513 insertions(+) create mode 100755 tests/qemu-iotests/236 create mode 100644 tests/qemu-iotests/236.out
tests/qemu-iotests/236 new file +161 -0
@@ -0,0 +1,161 @@ +#!/usr/bin/env python +# +# Test bitmap merges. +# +# Copyright (c) 2018 John Snow for Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# [email protected] + +import iotests +from iotests import log + +iotests.verify_image_format(supported_fmts=['generic']) +size = 64 * 1024 * 1024 +granularity = 64 * 1024 + +patterns = [("0x5d", "0", "64k"), + ("0xd5", "1M", "64k"), + ("0xdc", "32M", "64k"), + ("0xcd", "0x3ff0000", "64k")] # 64M - 64K + +overwrite = [("0xab", "0", "64k"), # Full overwrite + ("0xad", "0x00f8000", "64k"), # Partial-left (1M-32K) + ("0x1d", "0x2008000", "64k"), # Partial-right (32M+32K) + ("0xea", "0x3fe0000", "64k")] # Adjacent-left (64M - 128K) + +def query_bitmaps(vm): + res = vm.qmp("query-block") + return { "bitmaps": { device['device']: device.get('dirty-bitmaps', []) for + device in res['return'] } } + +with iotests.FilePath('img') as img_path, \ + iotests.VM() as vm: + + log('--- Preparing image & VM ---\n') + iotests.qemu_img_create('-f', iotests.imgfmt, img_path, str(size)) + vm.add_drive(img_path) + vm.launch() + + log('\n--- Adding preliminary bitmaps A & B ---\n') + vm.qmp_log("block-dirty-bitmap-add", node="drive0", + name="bitmapA", granularity=granularity) + vm.qmp_log("block-dirty-bitmap-add", node="drive0", + name="bitmapB", granularity=granularity) + + # Dirties 4 clusters. count=262144 + log('\n--- Emulating writes ---\n') + for p in patterns: + cmd = "write -P%s %s %s" % p + log(cmd) + log(vm.hmp_qemu_io("drive0", cmd)) + + log(query_bitmaps(vm), indent=2) + + log('\n--- Submitting & Aborting Transaction ---\n') + vm.qmp_log("transaction", indent=2, actions=[ + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapB" }}, + { "type": "block-dirty-bitmap-add", + "data": { "node": "drive0", "name": "bitmapC", + "granularity": granularity }}, + { "type": "block-dirty-bitmap-clear", + "data": { "node": "drive0", "name": "bitmapA" }}, + { "type": "abort", "data": {}} + ]) + log(query_bitmaps(vm), indent=2) + + log('\n--- Disabling B & Adding C ---\n') + vm.qmp_log("transaction", indent=2, actions=[ + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapB" }}, + { "type": "block-dirty-bitmap-add", + "data": { "node": "drive0", "name": "bitmapC", + "granularity": granularity }}, + # Purely extraneous, but test that it works: + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapC" }}, + { "type": "block-dirty-bitmap-enable", + "data": { "node": "drive0", "name": "bitmapC" }}, + ]) + + log('\n--- Emulating further writes ---\n') + # Dirties 6 clusters, 3 of which are new in contrast to "A". + # A = 64 * 1024 * (4 + 3) = 458752 + # C = 64 * 1024 * 6 = 393216 + for p in overwrite: + cmd = "write -P%s %s %s" % p + log(cmd) + log(vm.hmp_qemu_io("drive0", cmd)) + + log('\n--- Disabling A & C ---\n') + vm.qmp_log("transaction", indent=2, actions=[ + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapA" }}, + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapC" }} + ]) + + # A: 7 clusters + # B: 4 clusters + # C: 6 clusters + log(query_bitmaps(vm), indent=2) + + log('\n--- Submitting & Aborting Merge Transaction ---\n') + vm.qmp_log("transaction", indent=2, actions=[ + { "type": "block-dirty-bitmap-add", + "data": { "node": "drive0", "name": "bitmapD", + "disabled": True, "granularity": granularity }}, + { "type": "block-dirty-bitmap-merge", + "data": { "node": "drive0", "target": "bitmapD", + "bitmaps": ["bitmapB", "bitmapC"] }}, + { "type": "abort", "data": {}} + ]) + log(query_bitmaps(vm), indent=2) + + log('\n--- Creating D as a merge of B & C ---\n') + # Good hygiene: create a disabled bitmap as a merge target. + vm.qmp_log("transaction", indent=2, actions=[ + { "type": "block-dirty-bitmap-add", + "data": { "node": "drive0", "name": "bitmapD", + "disabled": True, "granularity": granularity }}, + { "type": "block-dirty-bitmap-merge", + "data": { "node": "drive0", "target": "bitmapD", + "bitmaps": ["bitmapB", "bitmapC"] }} + ]) + + # A and D should now both have 7 clusters apiece. + # B and C remain unchanged with 4 and 6 respectively. + log(query_bitmaps(vm), indent=2) + + # A and D should be equivalent. + # Some formats round the size of the disk, so don't print the checksums. + check_a = vm.qmp('x-debug-block-dirty-bitmap-sha256', + node="drive0", name="bitmapA")['return']['sha256'] + check_d = vm.qmp('x-debug-block-dirty-bitmap-sha256', + node="drive0", name="bitmapD")['return']['sha256'] + assert(check_a == check_d) + + log('\n--- Removing bitmaps A, B, C, and D ---\n') + vm.qmp_log("block-dirty-bitmap-remove", node="drive0", name="bitmapA") + vm.qmp_log("block-dirty-bitmap-remove", node="drive0", name="bitmapB") + vm.qmp_log("block-dirty-bitmap-remove", node="drive0", name="bitmapC") + vm.qmp_log("block-dirty-bitmap-remove", node="drive0", name="bitmapD") + + log('\n--- Final Query ---\n') + log(query_bitmaps(vm), indent=2) + + log('\n--- Done ---\n') + vm.shutdown()
tests/qemu-iotests/236.out new file +351 -0
351 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -233,3 +233,4 @@ 233 auto quick 234 auto quick migration 235 auto quick +236 auto quick
Signature · 1 line
2.17.2
Re: [Qemu-devel] [PATCH v6 11/11] iotests: add iotest 236 for testing bitmap merge
Quoted 18 lines · John Snow
New interface, new smoke test.
Signed-off-by John Snow <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Eric Blake <[email protected]>--- tests/qemu-iotests/236 | 161 +++++++++++++++++ tests/qemu-iotests/236.out | 351 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 513 insertions(+) create mode 100755 tests/qemu-iotests/236 create mode 100644 tests/qemu-iotests/236.outtests/qemu-iotests/236 new file +0
-0
tests/qemu-iotests/236
Quoted 2 lines
tests/qemu-iotests/236 new file +1
-0+ log('\n--- Submitting & Aborting Merge Transaction ---\n')
You updated the test but forgot to update 236.out:
236 - output mismatch (see 236.out.bad)
--- /home/eblake/qemu/tests/qemu-iotests/236.out 2019-01-08
17:16:10.938311529 -0600
+++ /home/eblake/qemu/tests/qemu-iotests/236.out.bad 2019-01-08
20:46:21.240645478 -0600
@@ -37,7 +37,7 @@
}
}
---- Submitting Bad Transaction ---
+--- Submitting & Aborting Transaction ---
{
"execute": "transaction",
@@ -201,7 +201,7 @@
}
}
---- Submitting Bad Merge ---
+--- Submitting & Aborting Merge Transaction ---
{
"execute": "transaction",
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org