push backup with fleecing
v2 cover push backup with fleecing
Hi all!
Here is push-backup with fleecing. What is it:
1. Make fleecing scheme
guest blk
|
|root
v
copy-before-write filter -------> temp qcow2
| | |file | backing V |
active disk <-------------------------
2. Start backup job from temp qcow2 to final remote target (NBD or something)
Benefit in comparison with simple backup job: for remote final target write operations are not very fast. And guest have to wait for copy-before-write operations. With fleecing scheme target for copy-before-write operations is local qcow2 file with faster access than actual backup target. So, guest is less disturbed by copy-before-write operations.
Based-on: <[email protected]>
([PATCH v6 00/33] block: publish backup-top filter)
v2:
01: changed to simply check s->target->perm
06: make explicit immutable-source parameter instead of auto-detecting
Vladimir Sementsov-Ogievskiy (6):
block/block-copy: use write-unchanged for fleecing scheme block/copy-before-write: require BLK_PERM_WRITE_UNCHANGED for fleecing
block: share writes on backing child of fleecing node
block: blk_root(): return non-const pointer
qapi: backup: add immutable-source paramter iotests/image-fleecing: test push backup with fleecing
qapi/block-core.json | 12 ++- block/copy-before-write.h | 1 + include/block/block_int.h | 1 + include/sysemu/block-backend.h | 2 +- block.c | 3 +- block/backup.c | 71 ++++++++++++- block/block-backend.c | 2 +- block/block-copy.c | 18 +++- block/copy-before-write.c | 48 ++++++++- block/replication.c | 2 +- blockdev.c | 1 + tests/qemu-iotests/tests/image-fleecing | 105 +++++++++++++++----- tests/qemu-iotests/tests/image-fleecing.out | 61 ++++++++++++ 13 files changed, 287 insertions(+), 40 deletions(-)
Signature · 1 line
2.29.2
PATCH v2 1/6 block/block-copy: use write-unchanged for fleecing scheme
We are going to use fleecing scheme for push-backup, so that copy-before-write filter does copy before write operations to temporary image and backup job copies data from (immutable from backup's point of view) temporary image to actual backup target. For this to work properly, backup job should unshare writes on immutable source node. copy-before-write filter should do write-unchanged operations for this (they are really unchanged, as source is a backing of temporary node).
So, we want to teach block-copy to do WRITE_UNCHANGED operations in case of fleecing. We can detect fleecing on initialization but that's not safe enough: block graph may change. We can freeze the backing chain of target to avoid changes but that's not flexible. We can detect fleecing before each write but that's an overhead.
Actually, we'll have to detect fleecing in copy-before-write filter anyway, to chose correct permissions on target node in further patch. So, we can just add a possibility to control adding BDRV_REQ_WRITE_UNCHANGED flag in block_copy, like block_copy_set_write_unchanged(). Block-copy doesn't own source and target child and relies on correct permission handling by block-copy user.
Finally, let's go the simplest way: just do WRITE_UNCHANGED if s->target has only that permission currently.
--- block/block-copy.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
block/block-copy.c +14 -4
@@ -114,7 +114,10 @@ typedef struct BlockCopyState {/* * BdrvChild objects are not owned or managed by block-copy. They are * provided by block-copy user and user is responsible for appropriate -* permissions on these children. +* permissions on these children. Note also that block-copy will +* automatically add BDRV_REQ_WRITE_UNCHANGED flag to write operations if +* target child has BLK_PERM_WRITE_UNCHANGED permission but doesn't have +* BLK_PERM_WRITE permission. */ BdrvChild *source;BdrvChild *target;
@@ -508,6 +511,7 @@ static int coroutine_fn block_copy_do_copy(BlockCopyState *s,int ret;int64_t nbytes =MIN (offset + bytes, s->len) - offset;void *bounce_buffer =NULL ; +BdrvRequestFlags write_flags = s->write_flags;assert (offset >=0 && bytes >0 &&INT64_MAX - offset >= bytes);assert (QEMU_IS_ALIGNED (offset, s->cluster_size));
@@ -517,9 +521,15 @@ static int coroutine_fn block_copy_do_copy(BlockCopyState *s, offset + bytes == QEMU_ALIGN_UP(s->len, s->cluster_size));assert (nbytes <INT_MAX ); +if ((s->target->perm & (BLK_PERM_WRITE |BLK_PERM_WRITE_UNCHANGED )) == +BLK_PERM_WRITE_UNCHANGED ) + { + write_flags |=BDRV_REQ_WRITE_UNCHANGED ; + } +switch (*method) {case COPY_WRITE_ZEROES : - ret =bdrv_co_pwrite_zeroes (s->target, offset, nbytes, s->write_flags & + ret =bdrv_co_pwrite_zeroes (s->target, offset, nbytes, write_flags & ~BDRV_REQ_WRITE_COMPRESSED );if (ret <0 ) {trace_block_copy_write_zeroes_fail (s, offset, ret);
@@ -530,7 +540,7 @@ static int coroutine_fn block_copy_do_copy(BlockCopyState *s,case COPY_RANGE_SMALL :case COPY_RANGE_FULL : ret =bdrv_co_copy_range (s->source, offset, s->target, offset, nbytes, -0 , s->write_flags); +0 , write_flags);if (ret >=0 ) {/* Successful copy-range, increase chunk size. */ *method =COPY_RANGE_FULL ;
@@ -563,7 +573,7 @@ static int coroutine_fn block_copy_do_copy(BlockCopyState *s, } ret =bdrv_co_pwrite (s->target, offset, nbytes, bounce_buffer, - s->write_flags); + write_flags);if (ret <0 ) {trace_block_copy_write_fail (s, offset, ret); *error_is_read = false;
Signature · 1 line
2.29.2
PATCH v2 2/6 block/copy-before-write: require BLK_PERM_WRITE_UNCHANGED for fleecing
In reply to Vladimir Sementsov-OgievskiyNow block-copy detects fleecing scheme and do write-unchanged operations if detected. So, let's require appropriate permissions.
--- block/copy-before-write.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
block/copy-before-write.c +10 -1
@@ -113,6 +113,14 @@ static void cbw_refresh_filename(BlockDriverState *bs) bs->file->bs->filename); } +static bool cbw_is_fleecing (BlockDriverState *bs) +{ +BDRVCopyBeforeWriteState *s = bs->opaque; + +return bs->file && s->target && +bdrv_skip_filters (bs) ==bdrv_backing_chain_next (s->target->bs); +} + static void cbw_child_perm(BlockDriverState *bs, BdrvChild *c, BdrvChildRole role, BlockReopenQueue *reopen_queue,
@@ -129,7 +137,8 @@ static void cbw_child_perm(BlockDriverState *bs, BdrvChild *c,* only upfront. */ *nshared =BLK_PERM_ALL & ~BLK_PERM_RESIZE ; - *nperm =BLK_PERM_WRITE ; + *nperm = +cbw_is_fleecing (bs) ?BLK_PERM_WRITE_UNCHANGED :BLK_PERM_WRITE ; }else {/* Source child */ bdrv_default_perms(bs, c, role, reopen_queue,
Signature · 1 line
2.29.2
PATCH v2 3/6 block: share writes on backing child of fleecing node
In reply to Vladimir Sementsov-OgievskiyBy default, we share writes on backing child only if our parents share write permission on us.
Still, with fleecing scheme we want to be able to unshare writes on fleecing node, which is a kind of immutable snapshot (copy-before-write operations are write-unchanged). So, let's detect fleecing node and share writes on its backing child. (we should share them, otherwise copy-before-write filter can't write to its file child).
With fleecing scheme we are sure, that writes to backing child goes through copy-before-write filter, so we are safe to share them.
--- block/copy-before-write.h | 1 + block.c | 3 ++- block/copy-before-write.c | 37 +++++++++++++++++++++++++++++++++++++ 3 files changed, 40 insertions(+), 1 deletion(-)
block/copy-before-write.h +1 -0
@@ -35,5 +35,6 @@ BlockDriverState *bdrv_cbw_append(BlockDriverState *source, BlockCopyState **bcs, Error **errp);void bdrv_cbw_drop (BlockDriverState *bs); +bool bdrv_is_fleecing_node (BlockDriverState *bs);#endif /* COPY_BEFORE_WRITE_H */
block.c +2 -1
@@ -50,6 +50,7 @@#include "qemu/cutils.h" #include "qemu/id.h" #include "block/coroutines.h" +#include "block/copy-before-write.h" #ifdef CONFIG_BSD #include <sys/ioctl.h>
@@ -2507,7 +2508,7 @@ static void bdrv_default_perms_for_cow(BlockDriverState *bs, BdrvChild *c,* writable and resizable backing file. * TODO Require !(perm & BLK_PERM_CONSISTENT_READ), too? */ -if (shared &BLK_PERM_WRITE ) { +if (shared &BLK_PERM_WRITE ||bdrv_is_fleecing_node (bs)) { shared =BLK_PERM_WRITE |BLK_PERM_RESIZE ; }else { shared =0 ;
block/copy-before-write.c +37 -0
@@ -257,6 +257,43 @@ void bdrv_cbw_drop(BlockDriverState *bs)bdrv_unref (bs); } +/* +* Detect is bs a fleecing node in some fleecing sceheme like: +* +* copy-before-write -- target --> fleecing-node +* | | +* | file | backing +* active-node <--------------------- +* +* In this case, fleecing-node can (and should) safely share writes on its +* backing child. +*/ +bool bdrv_is_fleecing_node (BlockDriverState *bs) +{ +BdrvChild *parent; +BlockDriverState *parent_bs; +BDRVCopyBeforeWriteState *s; + + QLIST_FOREACH(parent, &bs->parents, next_parent) { +if (parent->klass != &child_of_bds) { +continue ; + } + + parent_bs = parent->opaque; +if (parent_bs->drv != &bdrv_cbw_filter) { +continue ; + } + + s = parent_bs->opaque; + +if (s->target && s->target->bs == bs &&cbw_is_fleecing (parent_bs)) { +return true; + } + } + +return false; +} +static void cbw_init (void ) {bdrv_register (&bdrv_cbw_filter);
Signature · 1 line
2.29.2
PATCH v2 4/6 block: blk_root(): return non-const pointer
In reply to Vladimir Sementsov-OgievskiyIn the following patch we'll want to pass blk children to block-copy. Const pointers are not enough. So, return non const pointer from blk_root().
--- include/sysemu/block-backend.h | 2 +- block/block-backend.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
include/sysemu/block-backend.h +1 -1
@@ -271,7 +271,7 @@ int coroutine_fn blk_co_copy_range(BlockBackend *blk_in, int64_t off_in, int bytes, BdrvRequestFlags read_flags, BdrvRequestFlags write_flags); -const BdrvChild *blk_root (BlockBackend *blk); +BdrvChild *blk_root (BlockBackend *blk);int blk_make_empty (BlockBackend *blk,Error **errp);
block/block-backend.c +1 -1
@@ -2463,7 +2463,7 @@ int coroutine_fn blk_co_copy_range(BlockBackend *blk_in, int64_t off_in, bytes, read_flags, write_flags); } -const BdrvChild *blk_root (BlockBackend *blk) +BdrvChild *blk_root (BlockBackend *blk) {return blk->root; }
Signature · 1 line
2.29.2
PATCH v2 5/6 qapi: backup: add immutable-source paramter
In reply to Vladimir Sementsov-OgievskiyWe are on the way to implement internal-backup with fleecing scheme, which includes backup job copying from temporary node (which is target of copy-before-write filter) to final target of backup. This job doesn't need own filter, as temporary node is a kind of snapshot, it's immutable.
Let's add a parameter for backup to not insert filter and handle guest writes to source node but instead unshare writes on source. This way backup job become a simple copying process.
--- qapi/block-core.json | 12 ++++++- include/block/block_int.h | 1 + block/backup.c | 71 ++++++++++++++++++++++++++++++++++++--- block/replication.c | 2 +- blockdev.c | 1 + 5 files changed, 80 insertions(+), 7 deletions(-)
qapi/block-core.json +11 -1
@@ -1391,6 +1391,15 @@ # above node specified by @drive. If this option is not given, # a node name is autogenerated. (Since: 4.2) # +# @immutable-source: If true, assume source is immutable and don't insert filter +# as no copy-before-write operations are needed. It will +# fail if there are existing writers on source node, as well, +# any attempt to add writer to source node during backup will +# fail. @filter-node-name must not be set. +# If false, insert copy-before-write filter above source node +# (see also @filter-node-name parameter). +# Default is false. (Since 6.2) +# # @x-perf: Performance options. (Since 6.0) # # Note: @on-source-error and @on-target-error only affect background
@@ -1407,7 +1416,8 @@ '*on-source-error': 'BlockdevOnError', '*on-target-error': 'BlockdevOnError', '*auto-finalize': 'bool', '*auto-dismiss': 'bool', - '*filter-node-name': 'str', '*x-perf': 'BackupPerf' } } + '*filter-node-name': 'str', '*immutable-source': 'bool', + '*x-perf': 'BackupPerf' } } ## # @DriveBackup:
include/block/block_int.h +1 -0
@@ -1284,6 +1284,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,BitmapSyncMode bitmap_mode, bool compress, const char *filter_node_name, + bool immutable_source, BackupPerf *perf, BlockdevOnError on_source_error, BlockdevOnError on_target_error,
block/backup.c +66 -5
125 lines not on this page · raw message
block/replication.c +1 -1
@@ -566,7 +566,7 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode, s->backup_job =backup_job_create (NULL , s->secondary_disk->bs, s->hidden_disk->bs,0 ,MIRROR_SYNC_MODE_NONE ,NULL ,0 , false,NULL , - &perf, + false, &perf,BLOCKDEV_ON_ERROR_REPORT ,BLOCKDEV_ON_ERROR_REPORT ,JOB_INTERNAL , backup_job_completed, bs,NULL , &local_err);
blockdev.c +1 -0
@@ -2911,6 +2911,7 @@ static BlockJob *do_backup_common(BackupCommon *backup, backup->sync, bmap, backup->bitmap_mode, backup->compress, backup->filter_node_name, + backup->immutable_source, &perf, backup->on_source_error, backup->on_target_error,
Signature · 1 line
2.29.2
PATCH v2 6/6 iotests/image-fleecing: test push backup with fleecing
In reply to Vladimir Sementsov-OgievskiyAdd test for push backup with fleecing:
- start fleecing with copy-before-write filter
- start a backup job from temporary fleecing node to actual backup target
--- tests/qemu-iotests/tests/image-fleecing | 105 +++++++++++++++----- tests/qemu-iotests/tests/image-fleecing.out | 61 ++++++++++++ 2 files changed, 141 insertions(+), 25 deletions(-)
tests/qemu-iotests/tests/image-fleecing +80 -25
@@ -48,12 +48,20 @@ remainder = [('0xd5', '0x108000', '32k'), # Right-end of partial-left [1] ('0xdc', '32M', '32k'), # Left-end of partial-right [2] ('0xcd', '0x3ff0000', '64k')] # patterns[3] -def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm): +def do_test(vm, use_cbw, base_img_path, fleece_img_path, nbd_sock_path=None, + target_img_path=None): + push_backup = target_img_path is not None + assert (nbd_sock_path is not None) != push_backup + if push_backup: + assert use_cbw + log('--- Setting up images ---') log('') assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0 assert qemu_img('create', '-f', 'qcow2', fleece_img_path, '64M') == 0 + if push_backup: + assert qemu_img('create', '-f', 'qcow2', target_img_path, '64M') == 0 for p in patterns: qemu_io('-f', iotests.imgfmt,
@@ -108,25 +116,43 @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm): target=tmp_node, sync='none')) - log('') - log('--- Setting up NBD Export ---') - log('') + if push_backup: + log('') + log('--- Starting actual backup ---') + log('') + + log(vm.qmp('blockdev-add', **{ + 'driver': iotests.imgfmt, + 'node-name': 'target', + 'file': { + 'driver': 'file', + 'filename': target_img_path + } + })) + log(vm.qmp('blockdev-backup', device=tmp_node, + sync='full', target='target', + immutable_source=True, + job_id='push-backup', speed=1)) + else: + log('') + log('--- Setting up NBD Export ---') + log('') - nbd_uri = 'nbd+unix:///%s?socket=%s' % (tmp_node, nbd_sock_path) - log(vm.qmp('nbd-server-start', - {'addr': { 'type': 'unix', - 'data': { 'path': nbd_sock_path } } })) + nbd_uri = 'nbd+unix:///%s?socket=%s' % (tmp_node, nbd_sock_path) + log(vm.qmp('nbd-server-start', + {'addr': { 'type': 'unix', + 'data': { 'path': nbd_sock_path } } })) - log(vm.qmp('nbd-server-add', device=tmp_node)) + log(vm.qmp('nbd-server-add', device=tmp_node)) - log('') - log('--- Sanity Check ---') - log('') + log('') + log('--- Sanity Check ---') + log('') - for p in patterns + zeroes: - cmd = 'read -P%s %s %s' % p - log(cmd) - assert qemu_io_silent('-r', '-f', 'raw', '-c', cmd, nbd_uri) == 0 + for p in patterns + zeroes: + cmd = 'read -P%s %s %s' % p + log(cmd) + assert qemu_io_silent('-r', '-f', 'raw', '-c', cmd, nbd_uri) == 0 log('') log('--- Testing COW ---')
@@ -137,6 +163,20 @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm): log(cmd) log(vm.hmp_qemu_io(qom_path, cmd, qdev=True)) + if push_backup: + # Check that previous operations were done during backup, not after + result = vm.qmp('query-block-jobs') + if len(result['return']) != 1: + log('Backup finished too fast, COW is not tested') + + result = vm.qmp('block-job-set-speed', device='push-backup', speed=0) + assert result == {'return': {}} + + log(vm.event_wait(name='BLOCK_JOB_COMPLETED', + match={'data': {'device': 'push-backup'}}), + filters=[iotests.filter_qmp_event]) + log(vm.qmp('blockdev-del', node_name='target')) + log('') log('--- Verifying Data ---') log('')
@@ -144,7 +184,10 @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm): for p in patterns + zeroes: cmd = 'read -P%s %s %s' % p log(cmd) - assert qemu_io_silent('-r', '-f', 'raw', '-c', cmd, nbd_uri) == 0 + if push_backup: + assert qemu_io_silent('-r', '-c', cmd, target_img_path) == 0 + else: + assert qemu_io_silent('-r', '-f', 'raw', '-c', cmd, nbd_uri) == 0 log('') log('--- Cleanup ---')
@@ -159,7 +202,9 @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm): assert e is not None log(e, filters=[iotests.filter_qmp_event]) - log(vm.qmp('nbd-server-stop')) + if not push_backup: + log(vm.qmp('nbd-server-stop')) + log(vm.qmp('blockdev-del', node_name=tmp_node)) vm.shutdown()
@@ -175,18 +220,28 @@ def do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm): log('') log('Done') - -def test(use_cbw): +def test(use_cbw, nbd_sock_path=None, target_img_path=None): with iotests.FilePath('base.img') as base_img_path, \ iotests.FilePath('fleece.img') as fleece_img_path, \ - iotests.FilePath('nbd.sock', - base_dir=iotests.sock_dir) as nbd_sock_path, \ iotests.VM() as vm: - do_test(use_cbw, base_img_path, fleece_img_path, nbd_sock_path, vm) + do_test(vm, use_cbw, base_img_path, fleece_img_path, nbd_sock_path, + target_img_path) + +def test_pull(use_cbw): + with iotests.FilePath('nbd.sock', + base_dir=iotests.sock_dir) as nbd_sock_path: + test(use_cbw, nbd_sock_path, None) + +def test_push(): + with iotests.FilePath('target.img') as target_img_path: + test(True, None, target_img_path) log('=== Test backup(sync=none) based fleecing ===\n') -test(False) +test_pull(False) log('=== Test filter based fleecing ===\n') -test(True) +test_pull(True) + +log('=== Test push backup with fleecing ===\n') +test_push()
tests/qemu-iotests/tests/image-fleecing.out +61 -0
64 lines not on this page · raw message
Signature · 1 line
2.29.2
Consider that as RFC.
I'm preparing an alternative and more efficient fleecing scheme, based on special block driver, not on backing link.
So, this will be rebased, and probably some permission-related difficulties may be avoided.
Quoted 56 lines · Vladimir Sementsov-Ogievskiy
Hi all!
Here is push-backup with fleecing. What is it:
1. Make fleecing scheme
guest blk
|
|root
v
copy-before-write filter -------> temp qcow2| | |file | backing V |active disk <-------------------------
2. Start backup job from temp qcow2 to final remote target (NBD or something)
Benefit in comparison with simple backup job: for remote final target write operations are not very fast. And guest have to wait for copy-before-write operations. With fleecing scheme target for copy-before-write operations is local qcow2 file with faster access than actual backup target. So, guest is less disturbed by copy-before-write operations.
Based-on: <[email protected]>
([PATCH v6 00/33] block: publish backup-top filter)v2:
01: changed to simply checks->target->perm
06: make explicit immutable-source parameter instead of auto-detectingVladimir Sementsov-Ogievskiy (6):
block/block-copy: use write-unchanged for fleecing schemeblock/copy-before-write: requireBLK_PERM_WRITE_UNCHANGEDfor fleecing
block: share writes on backing child of fleecing node
block:blk_root(): return non-const pointer
qapi: backup: add immutable-source paramter iotests/image-fleecing: test push backup with fleecingqapi/block-core.json | 12 ++- block/copy-before-write.h | 1 + include/block/block_int.h | 1 + include/sysemu/block-backend.h | 2 +- block.c | 3 +- block/backup.c | 71 ++++++++++++- block/block-backend.c | 2 +- block/block-copy.c | 18 +++- block/copy-before-write.c | 48 ++++++++- block/replication.c | 2 +- blockdev.c | 1 + tests/qemu-iotests/tests/image-fleecing | 105 +++++++++++++++----- tests/qemu-iotests/tests/image-fleecing.out | 61 ++++++++++++ 13 files changed, 287 insertions(+), 40 deletions(-)
Signature · 2 lines
Best regards, Vladimir