backup-top filter driver for backup
v15 cover backup-top filter driver for backup
Hi all!
These series introduce backup-top driver. It's a filter-node, which do copy-before-write operation. Mirror uses filter-node for handling guest writes, let's move to filter-node (from write-notifiers) for backup too.
Preparation patches are queued in Max's block branch:
Based-on: https://github.com/XanClic/qemu.git block
v15: use BdrvChildren in block-copy
01-03: new
04-05: a lot of changes, such as
04:
- add new parameters for creation
- prepare bcs creation
- add target child
- refactor bdrv_backup_top_append
- drop r-b
05:
- move block-copy to use BdrvChildren
- drop extra style fixing hunks
- iotest 141 output changed
v14: Drop range locks, keep old good in-flight requests waiting
12: new patch
14: use old request-waiting scheme instead of range locks
Vladimir Sementsov-Ogievskiy (5):
block/backup: move in-flight requests handling from backup to
block-copy
block/backup: move write_flags calculation inside backup_job_create block/block-copy: split block_copy_set_callbacks function
block: introduce backup-top filter driver
block/backup: use backup-top instead of write notifiers
qapi/block-core.json | 8 +- block/backup-top.h | 41 +++++ include/block/block-copy.h | 31 +++- include/block/block_int.h | 1 + block/backup-top.c | 276 +++++++++++++++++++++++++++++++++ block/backup.c | 147 +++++------------- block/block-copy.c | 140 +++++++++-------- block/replication.c | 2 +- blockdev.c | 1 + block/Makefile.objs | 1 + tests/qemu-iotests/056 | 8 +- tests/qemu-iotests/141.out | 2 +- tests/qemu-iotests/257 | 7 +- tests/qemu-iotests/257.out | 306 ++++++++++++++++++------------------- 14 files changed, 628 insertions(+), 343 deletions(-) create mode 100644 block/backup-top.h create mode 100644 block/backup-top.c
Signature · 1 line
2.21.0
PATCH v15 1/5 block/backup: move in-flight requests handling from backup to block-copy
Move synchronization mechanism to block-copy, to be able to use one block-copy instance from backup job and backup-top filter in parallel.
--- include/block/block-copy.h | 8 ++++++ block/backup.c | 52 -------------------------------------- block/block-copy.c | 43 +++++++++++++++++++++++++++++++ 3 files changed, 51 insertions(+), 52 deletions(-)
include/block/block-copy.h +8 -0
@@ -17,6 +17,13 @@#include "block/block.h" +typedef struct BlockCopyInFlightReq { +int64_t start_byte; +int64_t end_byte; +QLIST_ENTRY (BlockCopyInFlightReq) list; +CoQueue wait_queue;/* coroutines blocked on this request */ +}BlockCopyInFlightReq ; +typedef void (*ProgressBytesCallbackFunc )(int64_t bytes,void *opaque);typedef void (*ProgressResetCallbackFunc )(void *opaque); typedef struct BlockCopyState {
@@ -27,6 +34,7 @@ typedef struct BlockCopyState {bool use_copy_range;int64_t copy_range_size;uint64_t len; + QLIST_HEAD(, BlockCopyInFlightReq) inflight_reqs;BdrvRequestFlags write_flags;
block/backup.c +0 -52
91 lines not on this page · raw message
block/block-copy.c +43 -0
72 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [PATCH v15 1/5] block/backup: move in-flight requests handling from backup to block-copy
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
Move synchronization mechanism to block-copy, to be able to use one block-copy instance from backup job and backup-top filter in parallel.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 8 ++++++ block/backup.c | 52 -------------------------------------- block/block-copy.c | 43 +++++++++++++++++++++++++++++++ 3 files changed, 51 insertions(+), 52 deletions(-)
PATCH v15 2/5 block/backup: move write_flags calculation inside backup_job_create
In reply to Vladimir Sementsov-OgievskiyThis is logic-less refactoring, which simplifies further patch, as we'll need write_flags for backup-top filter creation and backup-top should be created before block job creation.
--- block/backup.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
block/backup.c +15 -15
@@ -446,20 +446,6 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, goto error; } -/* job->len is fixed, so we can't allow resize */ - job =block_job_create (job_id, &backup_job_driver, txn, bs,0 ,BLK_PERM_ALL , - speed, creation_flags, cb, opaque, errp); -if (!job) { - goto error; - } - - job->source_bs = bs; - job->on_source_error = on_source_error; - job->on_target_error = on_target_error; - job->sync_mode = sync_mode; - job->sync_bitmap = sync_bitmap; - job->bitmap_mode = bitmap_mode; -/* * If source is in backing chain of target assume that target is going to be * used for "image fleecing", i.e. it should represent a kind of snapshot of
@@ -477,6 +463,20 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, write_flags = (bdrv_chain_contains (target, bs) ?BDRV_REQ_SERIALISING :0 ) | (compress ?BDRV_REQ_WRITE_COMPRESSED :0 ), +/* job->len is fixed, so we can't allow resize */ + job =block_job_create (job_id, &backup_job_driver, txn, bs,0 ,BLK_PERM_ALL , + speed, creation_flags, cb, opaque, errp); +if (!job) { + goto error; + } + + job->source_bs = bs; + job->on_source_error = on_source_error; + job->on_target_error = on_target_error; + job->sync_mode = sync_mode; + job->sync_bitmap = sync_bitmap; + job->bitmap_mode = bitmap_mode; + job->bcs =block_copy_state_new (bs, target, cluster_size, write_flags, backup_progress_bytes_callback, backup_progress_reset_callback, job, errp);
@@ -485,11 +485,11 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, } job->cluster_size = cluster_size; + job->len = len;/* Required permissions are already taken by block-copy-state target */ block_job_add_bdrv (&job->common,"target" , target,0 ,BLK_PERM_ALL , &error_abort); - job->len = len;return &job->common;
Signature · 1 line
2.21.0
Re: [PATCH v15 2/5] block/backup: move write_flags calculation inside backup_job_create
Quoted 8 lines · Vladimir Sementsov-Ogievskiy
This is logic-less refactoring, which simplifies further patch, as we'll need
write_flagsfor backup-top filter creation and backup-top should be created before block job creation.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/backup.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
PATCH v15 3/5 block/block-copy: split block_copy_set_callbacks function
In reply to Vladimir Sementsov-OgievskiySplit block_copy_set_callbacks out of block_copy_state_new. It's needed for further commit: block-copy will use BdrvChildren of backup-top filter, so it will be created from backup-top filter creation function. But callbacks will still belong to backup job and will be set in separate.
--- include/block/block-copy.h | 13 +++++++++---- block/backup.c | 6 ++++-- block/block-copy.c | 24 +++++++++++++++--------- 3 files changed, 28 insertions(+), 15 deletions(-)
include/block/block-copy.h +9 -4
@@ -66,12 +66,17 @@ typedef struct BlockCopyState {void *progress_opaque; } BlockCopyState; -BlockCopyState *block_copy_state_new ( -BlockDriverState *source,BlockDriverState *target, -int64_t cluster_size,BdrvRequestFlags write_flags, +BlockCopyState *block_copy_state_new (BlockDriverState *source, +BlockDriverState *target, +int64_t cluster_size, +BdrvRequestFlags write_flags, +Error **errp); + +void block_copy_set_callbacks ( +BlockCopyState *s,ProgressBytesCallbackFunc progress_bytes_callback,ProgressResetCallbackFunc progress_reset_callback, -void *progress_opaque,Error **errp); +void *progress_opaque);void block_copy_state_free (BlockCopyState *s);
block/backup.c +4 -2
@@ -478,8 +478,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, job->bitmap_mode = bitmap_mode; job->bcs =block_copy_state_new (bs, target, cluster_size, write_flags, - backup_progress_bytes_callback, - backup_progress_reset_callback, job, errp); + errp);if (!job->bcs) { goto error; }
@@ -487,6 +486,9 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, job->cluster_size = cluster_size; job->len = len; +block_copy_set_callbacks (job->bcs, backup_progress_bytes_callback, + backup_progress_reset_callback, job); +/* Required permissions are already taken by block-copy-state target */ block_job_add_bdrv (&job->common,"target" , target,0 ,BLK_PERM_ALL , &error_abort);
block/block-copy.c +15 -9
@@ -66,12 +66,10 @@ void block_copy_state_free(BlockCopyState *s)g_free (s); } -BlockCopyState *block_copy_state_new ( -BlockDriverState *source,BlockDriverState *target, -int64_t cluster_size,BdrvRequestFlags write_flags, -ProgressBytesCallbackFunc progress_bytes_callback, -ProgressResetCallbackFunc progress_reset_callback, -void *progress_opaque,Error **errp) +BlockCopyState *block_copy_state_new (BlockDriverState *source, +BlockDriverState *target, +int64_t cluster_size, +BdrvRequestFlags write_flags,Error **errp) {BlockCopyState *s;int ret;
@@ -95,9 +93,6 @@ BlockCopyState *block_copy_state_new( .cluster_size = cluster_size, .len = bdrv_dirty_bitmap_size(copy_bitmap), .write_flags = write_flags, - .progress_bytes_callback = progress_bytes_callback, - .progress_reset_callback = progress_reset_callback, - .progress_opaque = progress_opaque, }; s->copy_range_size = QEMU_ALIGN_DOWN(MIN(blk_get_max_transfer(s->source),
@@ -144,6 +139,17 @@ fail:return NULL ; } +void block_copy_set_callbacks ( +BlockCopyState *s, +ProgressBytesCallbackFunc progress_bytes_callback, +ProgressResetCallbackFunc progress_reset_callback, +void *progress_opaque) +{ + s->progress_bytes_callback = progress_bytes_callback; + s->progress_reset_callback = progress_reset_callback; + s->progress_opaque = progress_opaque; +} +/* * Copy range to target with a bounce buffer and return the bytes copied. If * error occurred, return a negative error number
Signature · 1 line
2.21.0
Re: [PATCH v15 3/5] block/block-copy: split block_copy_set_callbacks function
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Split
block_copy_set_callbacksout ofblock_copy_state_new. It's needed for further commit: block-copy will use BdrvChildren of backup-top filter, so it will be created from backup-top filter creation function. But callbacks will still belong to backup job and will be set in separate.
Quoted 6 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 13 +++++++++---- block/backup.c | 6 ++++-- block/block-copy.c | 24 +++++++++++++++--------- 3 files changed, 28 insertions(+), 15 deletions(-)
PATCH v15 4/5 block: introduce backup-top filter driver
In reply to Vladimir Sementsov-OgievskiyBackup-top filter caches write operations and does copy-before-write operations.
The driver will be used in backup instead of write-notifiers.
--- block/backup-top.h | 41 +++++++ block/backup-top.c | 281 ++++++++++++++++++++++++++++++++++++++++++++ block/Makefile.objs | 1 + 3 files changed, 323 insertions(+) create mode 100644 block/backup-top.h create mode 100644 block/backup-top.c
block/backup-top.h new file +41 -0
@@ -0,0 +1,41 @@ +/* +* backup-top filter driver +* +* The driver performs Copy-Before-Write (CBW) operation: it is injected above +* some node, and before each write it copies _old_ data to the target node. +* +* Copyright (c) 2018-2019 Virtuozzo International GmbH. +* +* Author: +* Sementsov-Ogievskiy Vladimir <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#ifndef BACKUP_TOP_H +#define BACKUP_TOP_H + +#include "block/block_int.h" +#include "block/block-copy.h" + +BlockDriverState *bdrv_backup_top_append (BlockDriverState *source, +BlockDriverState *target, +const char *filter_node_name, +uint64_t cluster_size, +BdrvRequestFlags write_flags, +BlockCopyState **bcs, +Error **errp); +void bdrv_backup_top_drop (BlockDriverState *bs); + +#endif /* BACKUP_TOP_H */
block/backup-top.c new file +281 -0
281 lines not on this page · raw message
block/Makefile.objs +1 -0
@@ -42,6 +42,7 @@ block-obj-y += block-copy.o block-obj-y += crypto.o block-obj-y += aio_task.o +block-obj-y += backup-top.o common-obj-y += stream.o
Signature · 1 line
2.21.0
Re: [PATCH v15 4/5] block: introduce backup-top filter driver
Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Backup-top filter caches write operations and does copy-before-write operations.
The driver will be used in backup instead of write-notifiers.
Quoted 8 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/backup-top.h | 41 +++++++ block/backup-top.c | 281 ++++++++++++++++++++++++++++++++++++++++++++ block/Makefile.objs | 1 + 3 files changed, 323 insertions(+) create mode 100644 block/backup-top.h create mode 100644 block/backup-top.c
PATCH v15 5/5 block/backup: use backup-top instead of write notifiers
In reply to Vladimir Sementsov-OgievskiyDrop write notifiers and use filter node instead.
= Changes =
1. Add filter-node-name argument for backup qmp api. We have to do it in this commit, as 257 needs to be fixed.
2. There are no more write notifiers here, so is_write_notifier parameter is dropped from block-copy paths.
3. To sync with in-flight requests at job finish we now have drained removing of the filter, we don't need rw-lock.
4. Block-copy is now using BdrvChildren instead of BlockBackends
5. As backup-top owns these children, we also move block-copy state into backup-top's ownership.
= Iotest changes =
56: op-blocker doesn't shoot now, as we set it on source, but then check on filter, when trying to start second backup.
To keep the test we instead can catch another collision: both jobs will get 'drive0' job-id, as job-id parameter is unspecified. To prevent interleaving with file-posix locks (as they are dependent on config) let's use another target for second backup.
Also, it's obvious now that we'd like to drop this op-blocker at all and add a test-case for two backups from one node (to different destinations) actually works. But not in these series.
141: Output changed: prepatch, "Node is in use" comes from bdrv_has_blk check inside qmp_blockdev_del. But we've dropped block-copy blk objects, so no more blk objects on source bs (job blk is on backup-top filter bs). New message is from op-blocker, which is the next check in qmp_blockdev_add.
257: The test wants to emulate guest write during backup. They should go to filter node, not to original source node, of course. Therefore we need to specify filter node name and use it.
--- qapi/block-core.json | 8 +- include/block/block-copy.h | 14 +- include/block/block_int.h | 1 + block/backup-top.c | 21 +-- block/backup.c | 73 +++------ block/block-copy.c | 81 +++------- block/replication.c | 2 +- blockdev.c | 1 + tests/qemu-iotests/056 | 8 +- tests/qemu-iotests/141.out | 2 +- tests/qemu-iotests/257 | 7 +- tests/qemu-iotests/257.out | 306 ++++++++++++++++++------------------- 12 files changed, 237 insertions(+), 287 deletions(-)
qapi/block-core.json +7 -1
@@ -1450,6 +1450,11 @@ # list without user intervention. # Defaults to true. (Since 2.12) # +# @filter-node-name: the node name that should be assigned to the +# filter driver that the backup job inserts into the graph +# above node specified by @drive. If this option is not given, +# a node name is autogenerated. (Since: 4.2) +# # Note: @on-source-error and @on-target-error only affect background # I/O. If an error occurs during a guest write request, the device's # rerror/werror actions will be used.
@@ -1463,7 +1468,8 @@ '*compress': 'bool', '*on-source-error': 'BlockdevOnError', '*on-target-error': 'BlockdevOnError', - '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } } + '*auto-finalize': 'bool', '*auto-dismiss': 'bool', + '*filter-node-name': 'str' } } ## # @DriveBackup:
include/block/block-copy.h +9 -5
@@ -27,8 +27,13 @@ typedef struct BlockCopyInFlightReq {typedef void (*ProgressBytesCallbackFunc )(int64_t bytes,void *opaque);typedef void (*ProgressResetCallbackFunc )(void *opaque); typedef struct BlockCopyState { - BlockBackend *source; - BlockBackend *target; + /* + * BdrvChild objects are not owned or managed by block-copy. They are + * provided by block-copy user and user is responsible for appropriate + * permissions on these children. + */ + BdrvChild *source; + BdrvChild *target; BdrvDirtyBitmap *copy_bitmap; int64_t cluster_size; bool use_copy_range;
@@ -66,8 +71,7 @@ typedef struct BlockCopyState {void *progress_opaque; } BlockCopyState; -BlockCopyState *block_copy_state_new (BlockDriverState *source, -BlockDriverState *target, +BlockCopyState *block_copy_state_new (BdrvChild *source,BdrvChild *target,int64_t cluster_size,BdrvRequestFlags write_flags,Error **errp);
@@ -84,6 +88,6 @@ int64_t block_copy_reset_unallocated(BlockCopyState *s, int64_t offset, int64_t *count); int coroutine_fn block_copy(BlockCopyState *s, int64_t start, uint64_t bytes, -bool *error_is_read,bool is_write_notifier); +bool *error_is_read);#endif /* BLOCK_COPY_H */
include/block/block_int.h +1 -0
@@ -1197,6 +1197,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,BdrvDirtyBitmap *sync_bitmap, BitmapSyncMode bitmap_mode, bool compress, + const char *filter_node_name, BlockdevOnError on_source_error, BlockdevOnError on_target_error, int creation_flags,
block/backup-top.c +8 -13
@@ -50,12 +50,11 @@ static coroutine_fn int backup_top_co_preadv( static coroutine_fn int backup_top_cbw(BlockDriverState *bs, uint64_t offset,uint64_t bytes) { -/* -* Here we'd like to use block_copy(), but block-copy need to be moved to -* use BdrvChildren to correctly use it in backup-top filter. It's a TODO. -*/ +BDRVBackupTopState *s = bs->opaque; +uint64_t end =QEMU_ALIGN_UP (offset + bytes, s->bcs->cluster_size); +uint64_t off =QEMU_ALIGN_DOWN (offset, s->bcs->cluster_size); -abort (); +return block_copy (s->bcs, off, end - off,NULL ); } static int coroutine_fn backup_top_co_pdiscard(BlockDriverState *bs,
@@ -228,14 +227,10 @@ BlockDriverState *bdrv_backup_top_append(BlockDriverState *source, goto failed_after_append; } -/* -* TODO: Create block-copy-state here (which will utilize @cluster_size and -* @write_flags parameters which are unused now). For this, block-copy -* should be refactored to use BdrvChildren. -*/ - state->bcs =NULL ; -if (!state->bcs) { -error_setg (&local_err,"Cannot create block-copy-state" ); + state->bcs =block_copy_state_new (top->backing, state->target, + cluster_size, write_flags, &local_err); +if (local_err) { +error_prepend (&local_err,"Cannot create block-copy-state: " ); goto failed_after_append; } *bcs = state->bcs;
block/backup.c +25 -48
186 lines not on this page · raw message
block/block-copy.c +22 -59
185 lines not on this page · raw message
block/replication.c +1 -1
@@ -543,7 +543,7 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode, s->backup_job =backup_job_create (NULL , s->secondary_disk->bs, s->hidden_disk->bs, -0 ,MIRROR_SYNC_MODE_NONE ,NULL ,0 , false, +0 ,MIRROR_SYNC_MODE_NONE ,NULL ,0 , false,NULL ,BLOCKDEV_ON_ERROR_REPORT ,BLOCKDEV_ON_ERROR_REPORT ,JOB_INTERNAL , backup_job_completed, bs,NULL , &local_err);
blockdev.c +1 -0
@@ -3601,6 +3601,7 @@ static BlockJob *do_backup_common(BackupCommon *backup, job =backup_job_create (backup->job_id, bs, target_bs, backup->speed, backup->sync, bmap, backup->bitmap_mode, backup->compress, + backup->filter_node_name, backup->on_source_error, backup->on_target_error, job_flags,NULL ,NULL , txn, errp);
tests/qemu-iotests/056 +5 -3
@@ -133,6 +133,7 @@ class BackupTest(iotests.QMPTestCase): self.vm = iotests.VM() self.test_img = img_create('test') self.dest_img = img_create('dest') + self.dest_img2 = img_create('dest2') self.ref_img = img_create('ref') self.vm.add_drive(self.test_img) self.vm.launch()
@@ -141,6 +142,7 @@ class BackupTest(iotests.QMPTestCase): self.vm.shutdown() try_remove(self.test_img) try_remove(self.dest_img) + try_remove(self.dest_img2) try_remove(self.ref_img) def hmp_io_writes(self, drive, patterns):
@@ -253,9 +255,9 @@ class BackupTest(iotests.QMPTestCase): res = self.vm.qmp('query-block-jobs') self.assert_qmp(res, 'return[0]/status', 'concluded') # Leave zombie job un-dismissed, observe a failure: - res = self.qmp_backup_and_wait(serror="Node 'drive0' is busy: block device is in use by block job: backup", + res = self.qmp_backup_and_wait(serror="Job ID 'drive0' already in use", device='drive0', format=iotests.imgfmt, - sync='full', target=self.dest_img, + sync='full', target=self.dest_img2, auto_dismiss=False) self.assertEqual(res, False) # OK, dismiss the zombie.
@@ -265,7 +267,7 @@ class BackupTest(iotests.QMPTestCase): self.assert_qmp(res, 'return', []) # Ensure it's really gone. self.qmp_backup_and_wait(device='drive0', format=iotests.imgfmt, - sync='full', target=self.dest_img, + sync='full', target=self.dest_img2, auto_dismiss=False) def dismissal_failure(self, dismissal_opt):
tests/qemu-iotests/141.out +1 -1
@@ -10,7 +10,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/m. Formatting 'TEST_DIR/o.IMGFMT', fmt=IMGFMT size=1048576 backing_file=TEST_DIR/t.IMGFMT backing_fmt=IMGFMT {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "job0"}} -{"error": {"class": "GenericError", "desc": "Node drv0 is in use"}} +{"error": {"class": "GenericError", "desc": "Node 'drv0' is busy: node is used as backing hd of 'NODE_NAME'"}} {"return": {}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "aborting", "id": "job0"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_CANCELLED", "data": {"device": "job0", "len": 1048576, "offset": 0, "speed": 0, "type": "backup"}}
tests/qemu-iotests/257 +4 -3
@@ -190,6 +190,7 @@ def blockdev_backup(vm, device, target, sync, **kwargs): device=device, target=target, sync=sync, + filter_node_name='backup-top', **kwargs) return result
@@ -216,7 +217,7 @@ def backup(drive, n, filepath, sync, **kwargs): job_id=job_id, **kwargs) return job_id -def perform_writes(drive, n): +def perform_writes(drive, n, filter_node_name=None): log("--- Write #{:d} ---\n".format(n)) for pattern in GROUPS[n].patterns: cmd = "write -P{:s} 0x{:07x} 0x{:x}".format(
@@ -224,7 +225,7 @@ def perform_writes(drive, n): pattern.offset, pattern.size) log(cmd) - log(drive.vm.hmp_qemu_io(drive.node, cmd)) + log(drive.vm.hmp_qemu_io(filter_node_name or drive.node, cmd)) bitmaps = drive.vm.query_bitmaps() log({'bitmaps': bitmaps}, indent=2) log('')
@@ -355,7 +356,7 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): """Issue writes while the job is open to test bitmap divergence.""" # Note: when `failure` is 'intermediate', this isn't called. log('') - bitmaps = perform_writes(drive0, 2) + bitmaps = perform_writes(drive0, 2, filter_node_name='backup-top') # Named bitmap (static, should be unchanged) ebitmap.compare(vm.get_bitmap(drive0.node, 'bitmap0', bitmaps=bitmaps))
tests/qemu-iotests/257.out +153 -153
1,043 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [PATCH v15 5/5] block/backup: use backup-top instead of write notifiers
Quoted 56 lines · Vladimir Sementsov-Ogievskiy
Drop write notifiers and use filter node instead.
= Changes =
1. Add filter-node-name argument for backup qmp api. We have to do it in this commit, as 257 needs to be fixed.
2. There are no more write notifiers here, so
is_write_notifierparameter is dropped from block-copy paths.3. To sync with in-flight requests at job finish we now have drained removing of the filter, we don't need rw-lock.
4. Block-copy is now using BdrvChildren instead of BlockBackends
5. As backup-top owns these children, we also move block-copy state into backup-top's ownership.
= Iotest changes =
56: op-blocker doesn't shoot now, as we set it on source, but then check on filter, when trying to start second backup.
To keep the test we instead can catch another collision: both jobs will get 'drive0' job-id, as job-id parameter is unspecified. To prevent interleaving with file-posix locks (as they are dependent on config) let's use another target for second backup.Also, it's obvious now that we'd like to drop this op-blocker at all and add a test-case for two backups from one node (to different destinations) actually works. But not in these series.
141: Output changed: prepatch, "Node is in use" comes from
bdrv_has_blkcheck insideqmp_blockdev_del. But we've dropped block-copy blk objects, so no more blk objects on source bs (job blk is on backup-top filter bs). New message is from op-blocker, which is the next check inqmp_blockdev_add.257: The test wants to emulate guest write during backup. They should go to filter node, not to original source node, of course. Therefore we need to specify filter node name and use it.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- qapi/block-core.json | 8 +- include/block/block-copy.h | 14 +- include/block/block_int.h | 1 + block/backup-top.c | 21 +-- block/backup.c | 73 +++------ block/block-copy.c | 81 +++------- block/replication.c | 2 +- blockdev.c | 1 + tests/qemu-iotests/056 | 8 +- tests/qemu-iotests/141.out | 2 +- tests/qemu-iotests/257 | 7 +- tests/qemu-iotests/257.out | 306 ++++++++++++++++++------------------- 12 files changed, 237 insertions(+), 287 deletions(-)
[...]
Quoted 4 lines
block/block-copy.c +0
-0
[...]
block/block-copy.c static int coroutine_fn block_copy_with_offload(BlockCopyState *s, · line 187
Quoted 8 lines
block/block-copy.c +2
-2@@ -218,8 +183,8 @@ static int coroutine_fn block_copy_with_offload(BlockCopyState *s, nr_clusters =DIV_ROUND_UP (nbytes, s->cluster_size);bdrv_reset_dirty_bitmap (s->copy_bitmap, start, s->cluster_size * nr_clusters); - ret =blk_co_copy_range (s->source, start, s->target, start, nbytes, - read_flags, s->write_flags); + ret =bdrv_co_copy_range (s->source, start, s->target, start, nbytes, +0 , s->write_flags);
The indentation’s off here. I’ll fix it.
Max Reitz
AppliedQuoted 47 lines · Vladimir Sementsov-Ogievskiy
Hi all!
These series introduce backup-top driver. It's a filter-node, which do copy-before-write operation. Mirror uses filter-node for handling guest writes, let's move to filter-node (from write-notifiers) for backup too.
Preparation patches are queued in Max's block branch:
Based-on: https://github.com/XanClic/qemu.git block
v15: use BdrvChildren in block-copy
01-03: new
04-05: a lot of changes, such as
04:
- add new parameters for creation
- prepare bcs creation
- add target child
- refactorbdrv_backup_top_append
- drop r-b
05:
- move block-copy to use BdrvChildren
- drop extra style fixing hunks
- iotest 141 output changedv14: Drop range locks, keep old good in-flight requests waiting
12: new patch
14: use old request-waiting scheme instead of range locksVladimir Sementsov-Ogievskiy (5):
block/backup: move in-flight requests handling from backup to
block-copy
block/backup: movewrite_flagscalculation insidebackup_job_createblock/block-copy: splitblock_copy_set_callbacksfunction
block: introduce backup-top filter driver
block/backup: use backup-top instead of write notifiersqapi/block-core.json | 8 +- block/backup-top.h | 41 +++++ include/block/block-copy.h | 31 +++- include/block/block_int.h | 1 + block/backup-top.c | 276 +++++++++++++++++++++++++++++++++ block/backup.c | 147 +++++------------- block/block-copy.c | 140 +++++++++-------- block/replication.c | 2 +- blockdev.c | 1 +
Quoted 8 lines
block/Makefile.objs | 1 + tests/qemu-iotests/056 | 8 +- tests/qemu-iotests/141.out | 2 +- tests/qemu-iotests/257 | 7 +- tests/qemu-iotests/257.out | 306 ++++++++++++++++++------------------- 14 files changed, 628 insertions(+), 343 deletions(-) create mode 100644 block/backup-top.h create mode 100644 block/backup-top.c
Thanks, applied to my block branch:
https://git.xanclic.moe/XanClic/qemu/commits/branch/block
Max
Quoted 57 lines · Max Reitz
Quoted 53 lines · Vladimir Sementsov-Ogievskiy
Hi all!
These series introduce backup-top driver. It's a filter-node, which do copy-before-write operation. Mirror uses filter-node for handling guest writes, let's move to filter-node (from write-notifiers) for backup too.
Preparation patches are queued in Max's block branch:
Based-on: https://github.com/XanClic/qemu.git block
v15: use BdrvChildren in block-copy
01-03: new
04-05: a lot of changes, such as
04:
- add new parameters for creation
- prepare bcs creation
- add target child
- refactorbdrv_backup_top_append
- drop r-b
05:
- move block-copy to use BdrvChildren
- drop extra style fixing hunks
- iotest 141 output changedv14: Drop range locks, keep old good in-flight requests waiting
12: new patch
14: use old request-waiting scheme instead of range locksVladimir Sementsov-Ogievskiy (5):
block/backup: move in-flight requests handling from backup to
block-copy
block/backup: movewrite_flagscalculation insidebackup_job_createblock/block-copy: splitblock_copy_set_callbacksfunction
block: introduce backup-top filter driver
block/backup: use backup-top instead of write notifiersqapi/block-core.json | 8 +- block/backup-top.h | 41 +++++ include/block/block-copy.h | 31 +++- include/block/block_int.h | 1 + block/backup-top.c | 276 +++++++++++++++++++++++++++++++++ block/backup.c | 147 +++++------------- block/block-copy.c | 140 +++++++++-------- block/replication.c | 2 +- blockdev.c | 1 + block/Makefile.objs | 1 + tests/qemu-iotests/056 | 8 +- tests/qemu-iotests/141.out | 2 +- tests/qemu-iotests/257 | 7 +- tests/qemu-iotests/257.out | 306 ++++++++++++++++++------------------- 14 files changed, 628 insertions(+), 343 deletions(-)Quoted 2 lines
create mode 100644
block/backup-top.h
create mode 100644block/backup-top.c
Quoted 6 lines
Thanks, applied to my block branch:
https://git.xanclic.moe/XanClic/qemu/commits/branch/block
Max
Yay!! Thank you!
Signature · 2 lines
Best regards, Vladimir