copy-before-write: on-cbw-error and cbw-timeout
Subsystems (2)
v3 cover copy-before-write: on-cbw-error and cbw-timeout
Hi all!
v3:
01: refactor options parsing
02: - wording, grammar
- early return from cbw_do_copy_before_write() when snapshot_error is set
- drop premature optimization around waiting for in-flight requests when we set snapshot_error
03: Hanna's r-b
05: - add callback to block_copy() interface
- add forgotten g_free(call_state)
- do cancel block-copy call on timeout
06: - wording
- increase in_flight for block_copy(), to handle timed-out block_copy() on drain
07: - less strict limit for bps-write, as now we honestly wait for cancelled
block-copy call on final drain
Here are two new options for copy-before-write filter:
on-cbw-error allows to alter the behavior on copy-before-write operation
failure: not break guest write but break the snapshot (and therefore backup process)
cbw-timeout allows to limit cbw operation by some timeout.
So, for example, using cbw-timeout=60 and on-cbw-error=break-snapshot you can be sure that guest write will not stuck for more than 60 seconds and will never fail due to backup problems.
Based-on: <[email protected]>
( [PATCH for-7.1 0/2] throttle-groups: use QEMU_CLOCK_REALTIME )
Vladimir Sementsov-Ogievskiy (7):
block/copy-before-write: refactor option parsing block/copy-before-write: add on-cbw-error open parameter
iotests: add copy-before-write: on-cbw-error tests
util: add qemu-co-timeout
block/block-copy: block_copy(): add timeout_ns parameter block/copy-before-write: implement cbw-timeout option
iotests: copy-before-write: add cases for cbw-timeout option
qapi/block-core.json | 31 ++- include/block/block-copy.h | 4 +- include/qemu/coroutine.h | 13 ++ block/block-copy.c | 33 ++- block/copy-before-write.c | 110 +++++++--- util/qemu-co-timeout.c | 89 ++++++++ tests/qemu-iotests/tests/copy-before-write | 206 ++++++++++++++++++ .../qemu-iotests/tests/copy-before-write.out | 5 + util/meson.build | 1 + 9 files changed, 453 insertions(+), 39 deletions(-) create mode 100644 util/qemu-co-timeout.c create mode 100755 tests/qemu-iotests/tests/copy-before-write create mode 100644 tests/qemu-iotests/tests/copy-before-write.out
Signature · 1 line
2.35.1
PATCH v3 1/7 block/copy-before-write: refactor option parsing
We are going to add one more option of enum type. Let's refactor option parsing so that we can simply work with BlockdevOptionsCbw object.
--- block/copy-before-write.c | 55 ++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 27 deletions(-)
block/copy-before-write.c +28 -27
@@ -24,6 +24,7 @@*/ #include "qemu/osdep.h" +#include "qapi/qmp/qjson.h" #include "sysemu/block-backend.h" #include "qemu/cutils.h"
@@ -328,46 +329,34 @@ static void cbw_child_perm(BlockDriverState *bs, BdrvChild *c, } } -static bool cbw_parse_bitmap_option (QDict *options,BdrvDirtyBitmap **bitmap, -Error **errp) +static BlockdevOptions *cbw_parse_options (QDict *options,Error **errp) { -QDict *bitmap_qdict =NULL ; -BlockDirtyBitmap *bmp_param =NULL ; +BlockdevOptions *opts =NULL ;Visitor *v =NULL ; -bool ret = false; - *bitmap =NULL ; +qdict_put_str (options,"driver" ,"copy-before-write" ); -qdict_extract_subqdict (options, &bitmap_qdict,"bitmap." ); -if (!qdict_size (bitmap_qdict)) { - ret = true; - goto out; - } - - v =qobject_input_visitor_new_flat_confused (bitmap_qdict, errp); + v =qobject_input_visitor_new_flat_confused (options, errp);if (!v) { goto out; } -visit_type_BlockDirtyBitmap (v,NULL , &bmp_param, errp); -if (!bmp_param) { +visit_type_BlockdevOptions (v,NULL , &opts, errp); +if (!opts) { goto out; } - *bitmap =block_dirty_bitmap_lookup (bmp_param->node, bmp_param->name,NULL , - errp); -if (!*bitmap) { - goto out; - } - - ret = true; +/* +* Delete options which we are going to parse through BlockdevOptions +* object for original options. +*/ +qdict_extract_subqdict (options,NULL ,"bitmap" ); out: -qapi_free_BlockDirtyBitmap (bmp_param);visit_free (v); -qobject_unref (bitmap_qdict); +qdict_del (options,"driver" ); -return ret; +return opts; } static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
@@ -376,6 +365,14 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,BDRVCopyBeforeWriteState *s = bs->opaque;BdrvDirtyBitmap *bitmap =NULL ;int64_t cluster_size; +g_autoptr (BlockdevOptions) full_opts =NULL ; +BlockdevOptionsCbw *opts; + + full_opts =cbw_parse_options (options, errp); +if (!full_opts) { +return -EINVAL ; + } + opts = &full_opts->u.copy_before_write; bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds, BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY,
@@ -390,8 +387,12 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,return -EINVAL ; } -if (!cbw_parse_bitmap_option (options, &bitmap, errp)) { -return -EINVAL ; +if (opts->has_bitmap) { + bitmap =block_dirty_bitmap_lookup (opts->bitmap->node, + opts->bitmap->name,NULL , errp); +if (!bitmap) { +return -EINVAL ; + } } bs->total_sectors = bs->file->bs->total_sectors;
Signature · 1 line
2.35.1
Re: [PATCH v3 1/7] block/copy-before-write: refactor option parsing
Quoted 12 lines · Vladimir Sementsov-Ogievskiy
We are going to add one more option of enum type. Let's refactor option parsing so that we can simply work with BlockdevOptionsCbw object.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/copy-before-write.c | 55 ++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 27 deletions(-)block/copy-before-write.c +0
-0
[...]
block/copy-before-write.c static int cbw_open(BlockDriverState *bs, QDict *options, int flags, · line 375
Quoted 12 lines
block/copy-before-write.c +8
-0@@ -376,6 +365,14 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,BDRVCopyBeforeWriteState *s = bs->opaque;BdrvDirtyBitmap *bitmap =NULL ;int64_t cluster_size; +g_autoptr (BlockdevOptions) full_opts =NULL ; +BlockdevOptionsCbw *opts; + + full_opts =cbw_parse_options (options, errp); +if (!full_opts) { +return -EINVAL ; + } + opts = &full_opts->u.copy_before_write;
I would prefer an assert(full_opts->driver == BLOCKDEV_DRIVER_COPY_BEFORE_WRITE); here, but, either way:
PATCH v3 2/7 block/copy-before-write: add on-cbw-error open parameter
In reply to Vladimir Sementsov-OgievskiyCurrently, behavior on copy-before-write operation failure is simple:
report error to the guest.
Let's implement alternative behavior: break the whole copy-before-write process (and corresponding backup job or NBD client) but keep guest working. It's needed if we consider guest stability as more important.
The realisation is simple: on copy-before-write failure we set s->snapshot_ret and continue guest operations. s->snapshot_ret being set will lead to all further snapshot API requests. Note that all in-flight snapshot-API requests may still success: we do wait for them on BREAK_SNAPSHOT-failure path in cbw_do_copy_before_write().
--- qapi/block-core.json | 25 ++++++++++++++++++++++++- block/copy-before-write.c | 32 ++++++++++++++++++++++++++++++-- 2 files changed, 54 insertions(+), 3 deletions(-)
qapi/block-core.json +24 -1
@@ -4163,6 +4163,25 @@ 'base': 'BlockdevOptionsGenericFormat', 'data': { '*bottom': 'str' } } +## +# @OnCbwError: +# +# An enumeration of possible behaviors for copy-before-write operation +# failures. +# +# @break-guest-write: report the error to the guest. This way, the guest +# will not be able to overwrite areas that cannot be +# backed up, so the backup process remains valid. +# +# @break-snapshot: continue guest write. Doing so will make the provided +# snapshot state invalid and any backup or export +# process based on it will finally fail. +# +# Since: 7.1 +## +{ 'enum': 'OnCbwError', + 'data': [ 'break-guest-write', 'break-snapshot' ] } + ## # @BlockdevOptionsCbw: #
@@ -4184,11 +4203,15 @@ # modifications (or removing) of specified bitmap doesn't # influence the filter. (Since 7.0) # +# @on-cbw-error: Behavior on failure of copy-before-write operation. +# Default is @break-guest-write. (Since 7.0) +# # Since: 6.2 ## { 'struct': 'BlockdevOptionsCbw', 'base': 'BlockdevOptionsGenericFormat', - 'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap' } } + 'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap', + '*on-cbw-error': 'OnCbwError' } } ## # @BlockdevOptions:
block/copy-before-write.c +30 -2
@@ -41,6 +41,7 @@ typedef struct BDRVCopyBeforeWriteState { BlockCopyState *bcs; BdrvChild *target; + OnCbwError on_cbw_error; /* * @lock: protects access to @access_bitmap, @done_bitmap and
@@ -65,6 +66,14 @@ typedef struct BDRVCopyBeforeWriteState {* node. These areas must not be rewritten by guest. */ BlockReqList frozen_read_reqs; + +/* +* @snapshot_error is normally zero. But on first copy-before-write failure +* when @on_cbw_error == ON_CBW_ERROR_BREAK_SNAPSHOT, @snapshot_error takes +* value of this error (<0). After that all in-flight and further +* snapshot-API requests will fail with that error. +*/ +int snapshot_error; } BDRVCopyBeforeWriteState; static coroutine_fn int cbw_co_preadv(
@@ -95,16 +104,27 @@ static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs,return 0 ; } +if (s->snapshot_error) { +return 0 ; + } + off =QEMU_ALIGN_DOWN (offset, cluster_size); end =QEMU_ALIGN_UP (offset + bytes, cluster_size); ret =block_copy (s->bcs, off, end - off, true); -if (ret <0 ) { +if (ret <0 && s->on_cbw_error ==ON_CBW_ERROR_BREAK_GUEST_WRITE ) {return ret; } WITH_QEMU_LOCK_GUARD(&s->lock) { -bdrv_set_dirty_bitmap (s->done_bitmap, off, end - off); +if (ret <0 ) { +assert (s->on_cbw_error ==ON_CBW_ERROR_BREAK_SNAPSHOT ); +if (!s->snapshot_error) { + s->snapshot_error = ret; + } + }else { +bdrv_set_dirty_bitmap (s->done_bitmap, off, end - off); + }reqlist_wait_all (&s->frozen_read_reqs, off, end - off, &s->lock); }
@@ -176,6 +196,11 @@ static BlockReq *cbw_snapshot_read_lock(BlockDriverState *bs,QEMU_LOCK_GUARD (&s->lock); +if (s->snapshot_error) { +g_free (req); +return NULL ; + } +if (bdrv_dirty_bitmap_next_zero (s->access_bitmap, offset, bytes) !=-1 ) {g_free (req);return NULL ;
@@ -351,6 +376,7 @@ static BlockdevOptions *cbw_parse_options(QDict *options, Error **errp)* object for original options. */ qdict_extract_subqdict (options,NULL ,"bitmap" ); +qdict_del (options,"on-cbw-error" ); out:visit_free (v);
@@ -394,6 +420,8 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,return -EINVAL ; } } + s->on_cbw_error = opts->has_on_cbw_error ? opts->on_cbw_error : +ON_CBW_ERROR_BREAK_GUEST_WRITE ; bs->total_sectors = bs->file->bs->total_sectors; bs->supported_write_flags = BDRV_REQ_WRITE_UNCHANGED |
Signature · 1 line
2.35.1
Re: [PATCH v3 2/7] block/copy-before-write: add on-cbw-error open parameter
Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Currently, behavior on copy-before-write operation failure is simple:
report error to the guest.Let's implement alternative behavior: break the whole copy-before-write process (and corresponding backup job or NBD client) but keep guest working. It's needed if we consider guest stability as more important.
The realisation is simple: on copy-before-write failure we set
s->snapshot_retand continue guest operations.s->snapshot_retbeing set will lead to all further snapshot API requests. Note that all in-flight snapshot-API requests may still success: we do wait for them on BREAK_SNAPSHOT-failure path incbw_do_copy_before_write().Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- qapi/block-core.json | 25 ++++++++++++++++++++++++- block/copy-before-write.c | 32 ++++++++++++++++++++++++++++++-- 2 files changed, 54 insertions(+), 3 deletions(-)qapi/block-core.json +0
-0
[...]
qapi/block-core.json line 4207
Quoted 6 lines
qapi/block-core.json +2
-0@@ -4184,11 +4203,15 @@ # modifications (or removing) of specified bitmap doesn't # influence the filter. (Since 7.0) # +# @on-cbw-error: Behavior on failure of copy-before-write operation. +# Default is @break-guest-write. (Since 7.0)
With *7.1:
Quoted 11 lines
qapi/block-core.json +3
-1+# # Since: 6.2 ## { 'struct': 'BlockdevOptionsCbw', 'base': 'BlockdevOptionsGenericFormat', - 'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap' } } + 'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap', + '*on-cbw-error': 'OnCbwError' } } ## # @BlockdevOptions:
PATCH v3 3/7 iotests: add copy-before-write: on-cbw-error tests
In reply to Vladimir Sementsov-OgievskiyAdd tests for new option of copy-before-write filter: on-cbw-error.
--- tests/qemu-iotests/tests/copy-before-write | 128 ++++++++++++++++++ .../qemu-iotests/tests/copy-before-write.out | 5 + 2 files changed, 133 insertions(+) create mode 100755 tests/qemu-iotests/tests/copy-before-write create mode 100644 tests/qemu-iotests/tests/copy-before-write.out
tests/qemu-iotests/tests/copy-before-write new file +128 -0
@@ -0,0 +1,128 @@ +#!/usr/bin/env python3 +# group: auto backup +# +# Copyright (c) 2022 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import re + +import iotests +from iotests import qemu_img_create, qemu_io + + +temp_img = os.path.join(iotests.test_dir, 'temp') +source_img = os.path.join(iotests.test_dir, 'source') +size = '1M' + + +class TestCbwError(iotests.QMPTestCase): + def tearDown(self): + self.vm.shutdown() + os.remove(temp_img) + os.remove(source_img) + + def setUp(self): + qemu_img_create('-f', iotests.imgfmt, source_img, size) + qemu_img_create('-f', iotests.imgfmt, temp_img, size) + qemu_io('-c', 'write 0 1M', source_img) + + self.vm = iotests.VM() + self.vm.launch() + + def do_cbw_error(self, on_cbw_error): + result = self.vm.qmp('blockdev-add', { + 'node-name': 'cbw', + 'driver': 'copy-before-write', + 'on-cbw-error': on_cbw_error, + 'file': { + 'driver': iotests.imgfmt, + 'file': { + 'driver': 'file', + 'filename': source_img, + } + }, + 'target': { + 'driver': iotests.imgfmt, + 'file': { + 'driver': 'blkdebug', + 'image': { + 'driver': 'file', + 'filename': temp_img + }, + 'inject-error': [ + { + 'event': 'write_aio', + 'errno': 5, + 'immediately': False, + 'once': True + } + ] + } + } + }) + self.assert_qmp(result, 'return', {}) + + result = self.vm.qmp('blockdev-add', { + 'node-name': 'access', + 'driver': 'snapshot-access', + 'file': 'cbw' + }) + self.assert_qmp(result, 'return', {}) + + result = self.vm.hmp_qemu_io('cbw', 'write 0 1M') + self.assert_qmp(result, 'return', '') + + result = self.vm.hmp_qemu_io('access', 'read 0 1M') + self.assert_qmp(result, 'return', '') + + self.vm.shutdown() + log = self.vm.get_log() + log = re.sub(r'^\[I \d+\.\d+\] OPENED\n', '', log) + log = re.sub(r'\[I \+\d+\.\d+\] CLOSED\n?$', '', log) + log = iotests.filter_qemu_io(log) + return log + + def test_break_snapshot_on_cbw_error(self): + """break-snapshot behavior: + Guest write succeed, but further snapshot-read fails, as snapshot is + broken. + """ + log = self.do_cbw_error('break-snapshot') + + self.assertEqual(log, """\ +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read failed: Permission denied +""") + + def test_break_guest_write_on_cbw_error(self): + """break-guest-write behavior: + Guest write fails, but snapshot-access continues working and further + snapshot-read succeeds. + """ + log = self.do_cbw_error('break-guest-write') + + self.assertEqual(log, """\ +write failed: Input/output error +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +""") + + +if __name__ == '__main__': + iotests.main(supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/tests/copy-before-write.out new file +5 -0
@@ -0,0 +1,5 @@ +.. +---------------------------------------------------------------------- +Ran 2 tests + +OK
Signature · 1 line
2.35.1
PATCH v3 4/7 util: add qemu-co-timeout
In reply to Vladimir Sementsov-OgievskiyAdd new API, to make a time limited call of the coroutine.
--- include/qemu/coroutine.h | 13 ++++++ util/qemu-co-timeout.c | 89 ++++++++++++++++++++++++++++++++++++++++ util/meson.build | 1 + 3 files changed, 103 insertions(+) create mode 100644 util/qemu-co-timeout.c
include/qemu/coroutine.h +13 -0
@@ -316,6 +316,19 @@ static inline void coroutine_fn qemu_co_sleep_ns(QEMUClockType type, int64_t ns)qemu_co_sleep_ns_wakeable (&w, type, ns); } +typedef void CleanupFunc (void *opaque); +/** +* Run entry in a coroutine and start timer. Wait for entry to finish or for +* timer to elapse, what happen first. If entry finished, return 0, if timer +* elapsed earlier, return -ETIMEDOUT. +* +* Be careful, entry execution is not canceled, user should handle it somehow. +* If @clean is provided, it's called after coroutine finish if timeout +* happened. +*/ +int coroutine_fn qemu_co_timeout(CoroutineEntry *entry, void *opaque, +uint64_t timeout_ns,CleanupFunc clean); +/** * Wake a coroutine if it is sleeping in qemu_co_sleep_ns. The timer will be * deleted. @sleep_state must be the variable whose address was given to
util/qemu-co-timeout.c new file +89 -0
89 lines not on this page · raw message
util/meson.build +1 -0
@@ -83,6 +83,7 @@ if have_block util_ss.add(files('block-helpers.c')) util_ss.add(files('qemu-coroutine-sleep.c')) util_ss.add(files('qemu-co-shared-resource.c')) + util_ss.add(files('qemu-co-timeout.c')) util_ss.add(files('thread-pool.c', 'qemu-timer.c')) util_ss.add(files('readline.c')) util_ss.add(files('throttle.c'))
Signature · 1 line
2.35.1
Re: [PATCH v3 4/7] util: add qemu-co-timeout
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
Add new API, to make a time limited call of the coroutine.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/qemu/coroutine.h | 13 ++++++ util/qemu-co-timeout.c | 89 ++++++++++++++++++++++++++++++++++++++++ util/meson.build | 1 + 3 files changed, 103 insertions(+)create mode 100644
util/qemu-co-timeout.c
PATCH v3 5/7 block/block-copy: block_copy(): add timeout_ns parameter
In reply to Vladimir Sementsov-OgievskiyAdd possibility to limit block_copy() call in time. To be used in the next commit.
As timed-out block_copy() call will continue in background anyway (we can't immediately cancel IO operation), it's important also give user a possibility to pass a callback, to do some additional actions on block-copy call finish.
--- include/block/block-copy.h | 4 +++- block/block-copy.c | 33 ++++++++++++++++++++++++++------- block/copy-before-write.c | 2 +- 3 files changed, 30 insertions(+), 9 deletions(-)
include/block/block-copy.h +3 -1
@@ -40,7 +40,9 @@ int64_t block_copy_reset_unallocated(BlockCopyState *s, int64_t offset, int64_t *count); int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, -bool ignore_ratelimit); +bool ignore_ratelimit,uint64_t timeout_ns, +BlockCopyAsyncCallbackFunc cb, +void *cb_opaque);/* * Run block-copy in a coroutine, create corresponding BlockCopyCallState
block/block-copy.c +26 -7
@@ -883,23 +883,42 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)return ret; } +static void coroutine_fn block_copy_async_co_entry(void *opaque) +{ +block_copy_common (opaque); +} + int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, -bool ignore_ratelimit) +bool ignore_ratelimit,uint64_t timeout_ns, +BlockCopyAsyncCallbackFunc cb, +void *cb_opaque) { -BlockCopyCallState call_state = { +int ret; +BlockCopyCallState *call_state =g_new (BlockCopyCallState,1 ); + + *call_state = (BlockCopyCallState ) { .s = s, .offset = start, .bytes = bytes, .ignore_ratelimit = ignore_ratelimit, .max_workers =BLOCK_COPY_MAX_WORKERS , + .cb = cb, + .cb_opaque = cb_opaque, }; -return block_copy_common (&call_state); -} + ret =qemu_co_timeout (block_copy_async_co_entry, call_state, timeout_ns, + g_free); +if (ret <0 ) { +assert (ret == -ETIMEDOUT ); +block_copy_call_cancel (call_state); +/* call_state will be freed by running coroutine. */ +return ret; + } -static void coroutine_fn block_copy_async_co_entry(void *opaque) -{ -block_copy_common (opaque); + ret = call_state->ret; +g_free (call_state); + +return ret; } BlockCopyCallState *block_copy_async(BlockCopyState *s,
block/copy-before-write.c +1 -1
@@ -111,7 +111,7 @@ static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs, off =QEMU_ALIGN_DOWN (offset, cluster_size); end =QEMU_ALIGN_UP (offset + bytes, cluster_size); - ret =block_copy (s->bcs, off, end - off, true); + ret =block_copy (s->bcs, off, end - off, true,0 ,NULL ,NULL );if (ret <0 && s->on_cbw_error ==ON_CBW_ERROR_BREAK_GUEST_WRITE ) {return ret; }
Signature · 1 line
2.35.1
Re: [PATCH v3 5/7] block/block-copy: block_copy(): add timeout_ns parameter
Quoted 8 lines · Vladimir Sementsov-Ogievskiy
Add possibility to limit
block_copy()call in time. To be used in the next commit.As timed-out
block_copy()call will continue in background anyway (we can't immediately cancel IO operation), it's important also give user a possibility to pass a callback, to do some additional actions on block-copy call finish.
Quoted 6 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 4 +++- block/block-copy.c | 33 ++++++++++++++++++++++++++------- block/copy-before-write.c | 2 +- 3 files changed, 30 insertions(+), 9 deletions(-)
PATCH v3 6/7 block/copy-before-write: implement cbw-timeout option
In reply to Vladimir Sementsov-OgievskiyIn some scenarios, when copy-before-write operations lasts too long time, it's better to cancel it.
Most useful would be to use the new option together with on-cbw-error=break-snapshot: this way if cbw operation takes too long time we'll just cancel backup process but do not disturb the guest too much.
Note the tricky point of realization: we keep additional point in bs->in_fligth during block_copy operation even if it's timed-out. Background "cancelled" block_copy operations will finish at some point and will want to access state. We should care to not free the state in .bdrv_close() earlier.
--- qapi/block-core.json | 8 +++++++- block/copy-before-write.c | 23 ++++++++++++++++++++++- 2 files changed, 29 insertions(+), 2 deletions(-)
qapi/block-core.json +7 -1
@@ -4206,12 +4206,18 @@ # @on-cbw-error: Behavior on failure of copy-before-write operation. # Default is @break-guest-write. (Since 7.0) # +# @cbw-timeout: Zero means no limit. Non-zero sets the timeout in seconds +# for copy-before-write operation. When a timeout occurs, +# the respective copy-before-write operation will fail, and +# the @on-cbw-error parameter will decide how this failure +# is handled. Default 0. (Since 7.1) +# # Since: 6.2 ## { 'struct': 'BlockdevOptionsCbw', 'base': 'BlockdevOptionsGenericFormat', 'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap', - '*on-cbw-error': 'OnCbwError' } } + '*on-cbw-error': 'OnCbwError', '*cbw-timeout': 'uint32' } } ## # @BlockdevOptions:
block/copy-before-write.c +22 -1
@@ -42,6 +42,7 @@ typedef struct BDRVCopyBeforeWriteState {BlockCopyState *bcs;BdrvChild *target;OnCbwError on_cbw_error; +uint32_t cbw_timeout_ns;/* * @lock: protects access to @access_bitmap, @done_bitmap and
@@ -83,6 +84,14 @@ static coroutine_fn int cbw_co_preadv(return bdrv_co_preadv (bs->file, offset, bytes, qiov, flags); } +static void block_copy_cb (void *opaque) +{ +BlockDriverState *bs = opaque; + + bs->in_flight--; +aio_wait_kick (); +} +/* * Do copy-before-write operation. *
@@ -111,7 +120,16 @@ static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs, off =QEMU_ALIGN_DOWN (offset, cluster_size); end =QEMU_ALIGN_UP (offset + bytes, cluster_size); - ret =block_copy (s->bcs, off, end - off, true,0 ,NULL ,NULL ); +/* +* Increase in_flight, so that in case of timed-out block-copy, the +* remaining background block_copy() request (which can't be immediately +* cancelled by timeout) is presented in bs->in_flight. This way we are +* sure that on bs close() we'll previously wait for all timed-out but yet +* running block_copy calls. +*/ + bs->in_flight++; + ret =block_copy (s->bcs, off, end - off, true, s->cbw_timeout_ns, + block_copy_cb, bs);if (ret <0 && s->on_cbw_error ==ON_CBW_ERROR_BREAK_GUEST_WRITE ) {return ret; }
@@ -377,6 +395,7 @@ static BlockdevOptions *cbw_parse_options(QDict *options, Error **errp)*/ qdict_extract_subqdict (options,NULL ,"bitmap" );qdict_del (options,"on-cbw-error" ); +qdict_del (options,"cbw-timeout" ); out:visit_free (v);
@@ -422,6 +441,8 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags, } s->on_cbw_error = opts->has_on_cbw_error ? opts->on_cbw_error :ON_CBW_ERROR_BREAK_GUEST_WRITE ; + s->cbw_timeout_ns = opts->has_cbw_timeout ? + opts->cbw_timeout *NANOSECONDS_PER_SECOND :0 ; bs->total_sectors = bs->file->bs->total_sectors; bs->supported_write_flags = BDRV_REQ_WRITE_UNCHANGED |
Signature · 1 line
2.35.1
Re: [PATCH v3 6/7] block/copy-before-write: implement cbw-timeout option
Quoted 10 lines · Vladimir Sementsov-Ogievskiy
In some scenarios, when copy-before-write operations lasts too long time, it's better to cancel it.
Most useful would be to use the new option together with on-cbw-error=break-snapshot: this way if cbw operation takes too long time we'll just cancel backup process but do not disturb the guest too much.
Note the tricky point of realization: we keep additional point in
bs->in_fligthduringblock_copyoperation even if it's timed-out.
*flight
Quoted 9 lines
Background "cancelled"
block_copyoperations will finish at some point and will want to access state. We should care to not free the state in .bdrv_close() earlier.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- qapi/block-core.json | 8 +++++++- block/copy-before-write.c | 23 ++++++++++++++++++++++- 2 files changed, 29 insertions(+), 2 deletions(-)
PATCH v3 7/7 iotests: copy-before-write: add cases for cbw-timeout option
In reply to Vladimir Sementsov-OgievskiyAdd two simple test-cases: timeout failure with break-snapshot-on-cbw-error behavior and similar with break-guest-write-on-cbw-error behavior.
--- tests/qemu-iotests/tests/copy-before-write | 78 +++++++++++++++++++ .../qemu-iotests/tests/copy-before-write.out | 4 +- 2 files changed, 80 insertions(+), 2 deletions(-)
tests/qemu-iotests/tests/copy-before-write +78 -0
@@ -122,6 +122,84 @@ read 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) """) + def do_cbw_timeout(self, on_cbw_error): + result = self.vm.qmp('object-add', { + 'qom-type': 'throttle-group', + 'id': 'group0', + 'limits': {'bps-write': 300 * 1024} + }) + self.assert_qmp(result, 'return', {}) + + result = self.vm.qmp('blockdev-add', { + 'node-name': 'cbw', + 'driver': 'copy-before-write', + 'on-cbw-error': on_cbw_error, + 'cbw-timeout': 1, + 'file': { + 'driver': iotests.imgfmt, + 'file': { + 'driver': 'file', + 'filename': source_img, + } + }, + 'target': { + 'driver': 'throttle', + 'throttle-group': 'group0', + 'file': { + 'driver': 'qcow2', + 'file': { + 'driver': 'file', + 'filename': temp_img + } + } + } + }) + self.assert_qmp(result, 'return', {}) + + result = self.vm.qmp('blockdev-add', { + 'node-name': 'access', + 'driver': 'snapshot-access', + 'file': 'cbw' + }) + self.assert_qmp(result, 'return', {}) + + result = self.vm.hmp_qemu_io('cbw', 'write 0 512K') + self.assert_qmp(result, 'return', '') + + # We need second write to trigger throttling + result = self.vm.hmp_qemu_io('cbw', 'write 512K 512K') + self.assert_qmp(result, 'return', '') + + result = self.vm.hmp_qemu_io('access', 'read 0 1M') + self.assert_qmp(result, 'return', '') + + self.vm.shutdown() + log = self.vm.get_log() + log = re.sub(r'^\[I \d+\.\d+\] OPENED\n', '', log) + log = re.sub(r'\[I \+\d+\.\d+\] CLOSED\n?$', '', log) + log = iotests.filter_qemu_io(log) + return log + + def test_timeout_break_guest(self): + log = self.do_cbw_timeout('break-guest-write') + self.assertEqual(log, """\ +wrote 524288/524288 bytes at offset 0 +512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +write failed: Connection timed out +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +""") + + def test_timeout_break_snapshot(self): + log = self.do_cbw_timeout('break-snapshot') + self.assertEqual(log, """\ +wrote 524288/524288 bytes at offset 0 +512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 524288/524288 bytes at offset 524288 +512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read failed: Permission denied +""") + if __name__ == '__main__': iotests.main(supported_fmts=['qcow2'],
tests/qemu-iotests/tests/copy-before-write.out +2 -2
@@ -1,5 +1,5 @@ -.. +.... ---------------------------------------------------------------------- -Ran 2 tests +Ran 4 tests OK
Signature · 1 line
2.35.1
Re: [PATCH v3 7/7] iotests: copy-before-write: add cases for cbw-timeout option
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Add two simple test-cases: timeout failure with break-snapshot-on-cbw-error behavior and similar with break-guest-write-on-cbw-error behavior.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/tests/copy-before-write | 78 +++++++++++++++++++ .../qemu-iotests/tests/copy-before-write.out | 4 +- 2 files changed, 80 insertions(+), 2 deletions(-)tests/qemu-iotests/tests/copy-before-write +0
-0
tests/qemu-iotests/tests/copy-before-write read 1048576/1048576 bytes at offset 0 · line 129
Quoted 8 lines
tests/qemu-iotests/tests/copy-before-write +5
-0@@ -122,6 +122,84 @@ read 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) """) + def do_cbw_timeout(self, on_cbw_error): + result = self.vm.qmp('object-add', { + 'qom-type': 'throttle-group', + 'id': 'group0', + 'limits': {'bps-write': 300 * 1024}
Hm, yes, I can’t find a way to make this work without your other series. For some reason, not even -accel tcg helps; and using qtest to advance the virtual clock doesn’t really help because the qemu-io commands block while the request is throttled.
One thing that should work would be to run everything in a qemu-storage-daemon instance, and then having qemu-io access an NBD export...
Re: [PATCH v3 7/7] iotests: copy-before-write: add cases for cbw-timeout option
Quoted 26 lines · Hanna Reitz
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Add two simple test-cases: timeout failure with break-snapshot-on-cbw-error behavior and similar with break-guest-write-on-cbw-error behavior.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
tests/qemu-iotests/tests/copy-before-write | 78 +++++++++++++++++++ .../qemu-iotests/tests/copy-before-write.out | 4 +-2 files changed, 80 insertions(+), 2 deletions(-)tests/qemu-iotests/tests/copy-before-write +0
-0tests/qemu-iotests/tests/copy-before-write read 1048576/1048576 bytes at offset 0 · line 128
Quoted 7 lines
tests/qemu-iotests/tests/copy-before-write +5
-0@@ -122,6 +122,84 @@ read 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) """) + def do_cbw_timeout(self, on_cbw_error): + result = self.vm.qmp('object-add', { + 'qom-type': 'throttle-group', + 'id': 'group0', + 'limits': {'bps-write': 300 * 1024}Hm, yes, I can’t find a way to make this work without your other series. For some reason, not even -accel tcg helps; and using qtest to advance the virtual clock doesn’t really help because the qemu-io commands block while the request is throttled.
Quoted 2 lines
One thing that should work would be to run everything in a qemu-storage-daemon instance, and then having qemu-io access an NBD export...
Simple switch to QEMUMachine helps. I'll resend soon.
Signature · 2 lines
Best regards, Vladimir