Skip to contentSkip to conversation

copy-before-write: on-cbw-error and cbw-timeout

PATCH In review Vladimir Sementsov-Ogievskiy 3 participants 15 messages Apr 7, 2022 – Jun 28, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v4 cover copy-before-write: on-cbw-error and cbw-timeout

Hi all!

v4: Now based on master
01: add assertion and r-b
02: s/7.0/7.1/ and r-b
03: switch to QEMUMachine, touch-up pylintrc, drop r-b 04,05,06: add r-b
07: switch to QEMUMachine

Here are two new options for copy-before-write filter:

on-cbw-error allows to alter the behavior on copy-before-write operation
failure: not break guest write but break the snapshot (and therefore backup process)

cbw-timeout allows to limit cbw operation by some timeout.

So, for example, using cbw-timeout=60 and on-cbw-error=break-snapshot you can be sure that guest write will not stuck for more than 60 seconds and will never fail due to backup problems.

Vladimir Sementsov-Ogievskiy (7):
block/copy-before-write: refactor option parsing block/copy-before-write: add on-cbw-error open parameter
iotests: add copy-before-write: on-cbw-error tests
util: add qemu-co-timeout
block/block-copy: block_copy(): add timeout_ns parameter block/copy-before-write: implement cbw-timeout option
iotests: copy-before-write: add cases for cbw-timeout option

 qapi/block-core.json                          |  31 ++-
 include/block/block-copy.h                    |   4 +-
 include/qemu/coroutine.h                      |  13 ++
 block/block-copy.c                            |  33 ++-
 block/copy-before-write.c                     | 111 ++++++---
 util/qemu-co-timeout.c                        |  89 ++++++++
 tests/qemu-iotests/pylintrc                   |   5 +
 tests/qemu-iotests/tests/copy-before-write    | 213 ++++++++++++++++++
 .../qemu-iotests/tests/copy-before-write.out  |   5 +
 util/meson.build                              |   1 +
 10 files changed, 466 insertions(+), 39 deletions(-)
 create mode 100644 util/qemu-co-timeout.c
 create mode 100755 tests/qemu-iotests/tests/copy-before-write
 create mode 100644 tests/qemu-iotests/tests/copy-before-write.out
Signature · 1 line
2.35.1
#1 lore raw

PATCH v4 1/7 block/copy-before-write: refactor option parsing

We are going to add one more option of enum type. Let's refactor option parsing so that we can simply work with BlockdevOptionsCbw object.

---
 block/copy-before-write.c | 56 ++++++++++++++++++++-------------------
 1 file changed, 29 insertions(+), 27 deletions(-)
block/copy-before-write.c +29 -27
index a8a06fdc09..e29c46cd7a 100644
@@ -24,6 +24,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "qapi/qmp/qjson.h"
 
 #include "sysemu/block-backend.h"
 #include "qemu/cutils.h"
@@ -328,46 +329,34 @@ static void cbw_child_perm(BlockDriverState *bs, BdrvChild *c,
     }
 }
 
-static bool cbw_parse_bitmap_option(QDict *options, BdrvDirtyBitmap **bitmap,
-                                    Error **errp)
+static BlockdevOptions *cbw_parse_options(QDict *options, Error **errp)
 {
-    QDict *bitmap_qdict = NULL;
-    BlockDirtyBitmap *bmp_param = NULL;
+    BlockdevOptions *opts = NULL;
     Visitor *v = NULL;
-    bool ret = false;
 
-    *bitmap = NULL;
+    qdict_put_str(options, "driver", "copy-before-write");
 
-    qdict_extract_subqdict(options, &bitmap_qdict, "bitmap.");
-    if (!qdict_size(bitmap_qdict)) {
-        ret = true;
-        goto out;
-    }
-
-    v = qobject_input_visitor_new_flat_confused(bitmap_qdict, errp);
+    v = qobject_input_visitor_new_flat_confused(options, errp);
     if (!v) {
         goto out;
     }
 
-    visit_type_BlockDirtyBitmap(v, NULL, &bmp_param, errp);
-    if (!bmp_param) {
+    visit_type_BlockdevOptions(v, NULL, &opts, errp);
+    if (!opts) {
         goto out;
     }
 
-    *bitmap = block_dirty_bitmap_lookup(bmp_param->node, bmp_param->name, NULL,
-                                        errp);
-    if (!*bitmap) {
-        goto out;
-    }
-
-    ret = true;
+    /*
+     * Delete options which we are going to parse through BlockdevOptions
+     * object for original options.
+     */
+    qdict_extract_subqdict(options, NULL, "bitmap");
 
 out:
-    qapi_free_BlockDirtyBitmap(bmp_param);
     visit_free(v);
-    qobject_unref(bitmap_qdict);
+    qdict_del(options, "driver");
 
-    return ret;
+    return opts;
 }
 
 static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
@@ -376,6 +365,15 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
     BDRVCopyBeforeWriteState *s = bs->opaque;
     BdrvDirtyBitmap *bitmap = NULL;
     int64_t cluster_size;
+    g_autoptr(BlockdevOptions) full_opts = NULL;
+    BlockdevOptionsCbw *opts;
+
+    full_opts = cbw_parse_options(options, errp);
+    if (!full_opts) {
+        return -EINVAL;
+    }
+    assert(full_opts->driver == BLOCKDEV_DRIVER_COPY_BEFORE_WRITE);
+    opts = &full_opts->u.copy_before_write;
 
     bs->file = bdrv_open_child(NULL, options, "file", bs, &child_of_bds,
                                BDRV_CHILD_FILTERED | BDRV_CHILD_PRIMARY,
@@ -390,8 +388,12 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
         return -EINVAL;
     }
 
-    if (!cbw_parse_bitmap_option(options, &bitmap, errp)) {
-        return -EINVAL;
+    if (opts->has_bitmap) {
+        bitmap = block_dirty_bitmap_lookup(opts->bitmap->node,
+                                           opts->bitmap->name, NULL, errp);
+        if (!bitmap) {
+            return -EINVAL;
+        }
     }
 
     bs->total_sectors = bs->file->bs->total_sectors;
Signature · 1 line
2.35.1
#2 lore raw

PATCH v4 2/7 block/copy-before-write: add on-cbw-error open parameter

In reply to Vladimir Sementsov-Ogievskiy

Currently, behavior on copy-before-write operation failure is simple:
report error to the guest.

Let's implement alternative behavior: break the whole copy-before-write process (and corresponding backup job or NBD client) but keep guest working. It's needed if we consider guest stability as more important.

The realisation is simple: on copy-before-write failure we set s->snapshot_ret and continue guest operations. s->snapshot_ret being set will lead to all further snapshot API requests. Note that all in-flight snapshot-API requests may still success: we do wait for them on BREAK_SNAPSHOT-failure path in cbw_do_copy_before_write().

---
 qapi/block-core.json      | 25 ++++++++++++++++++++++++-
 block/copy-before-write.c | 32 ++++++++++++++++++++++++++++++--
 2 files changed, 54 insertions(+), 3 deletions(-)
qapi/block-core.json +24 -1
index beeb91952a..6b870b2f37 100644
@@ -4163,6 +4163,25 @@
   'base': 'BlockdevOptionsGenericFormat',
   'data': { '*bottom': 'str' } }
 
+##
+# @OnCbwError:
+#
+# An enumeration of possible behaviors for copy-before-write operation
+# failures.
+#
+# @break-guest-write: report the error to the guest. This way, the guest
+#                     will not be able to overwrite areas that cannot be
+#                     backed up, so the backup process remains valid.
+#
+# @break-snapshot: continue guest write. Doing so will make the provided
+#                  snapshot state invalid and any backup or export
+#                  process based on it will finally fail.
+#
+# Since: 7.1
+##
+{ 'enum': 'OnCbwError',
+  'data': [ 'break-guest-write', 'break-snapshot' ] }
+
 ##
 # @BlockdevOptionsCbw:
 #
@@ -4184,11 +4203,15 @@
 #          modifications (or removing) of specified bitmap doesn't
 #          influence the filter. (Since 7.0)
 #
+# @on-cbw-error: Behavior on failure of copy-before-write operation.
+#                Default is @break-guest-write. (Since 7.1)
+#
 # Since: 6.2
 ##
 { 'struct': 'BlockdevOptionsCbw',
   'base': 'BlockdevOptionsGenericFormat',
-  'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap' } }
+  'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap',
+            '*on-cbw-error': 'OnCbwError' } }
 
 ##
 # @BlockdevOptions:
block/copy-before-write.c +30 -2
index e29c46cd7a..c8a11a09d2 100644
@@ -41,6 +41,7 @@
 typedef struct BDRVCopyBeforeWriteState {
     BlockCopyState *bcs;
     BdrvChild *target;
+    OnCbwError on_cbw_error;
 
     /*
      * @lock: protects access to @access_bitmap, @done_bitmap and
@@ -65,6 +66,14 @@ typedef struct BDRVCopyBeforeWriteState {
      * node. These areas must not be rewritten by guest.
      */
     BlockReqList frozen_read_reqs;
+
+    /*
+     * @snapshot_error is normally zero. But on first copy-before-write failure
+     * when @on_cbw_error == ON_CBW_ERROR_BREAK_SNAPSHOT, @snapshot_error takes
+     * value of this error (<0). After that all in-flight and further
+     * snapshot-API requests will fail with that error.
+     */
+    int snapshot_error;
 } BDRVCopyBeforeWriteState;
 
 static coroutine_fn int cbw_co_preadv(
@@ -95,16 +104,27 @@ static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs,
         return 0;
     }
 
+    if (s->snapshot_error) {
+        return 0;
+    }
+
     off = QEMU_ALIGN_DOWN(offset, cluster_size);
     end = QEMU_ALIGN_UP(offset + bytes, cluster_size);
 
     ret = block_copy(s->bcs, off, end - off, true);
-    if (ret < 0) {
+    if (ret < 0 && s->on_cbw_error == ON_CBW_ERROR_BREAK_GUEST_WRITE) {
         return ret;
     }
 
     WITH_QEMU_LOCK_GUARD(&s->lock) {
-        bdrv_set_dirty_bitmap(s->done_bitmap, off, end - off);
+        if (ret < 0) {
+            assert(s->on_cbw_error == ON_CBW_ERROR_BREAK_SNAPSHOT);
+            if (!s->snapshot_error) {
+                s->snapshot_error = ret;
+            }
+        } else {
+            bdrv_set_dirty_bitmap(s->done_bitmap, off, end - off);
+        }
         reqlist_wait_all(&s->frozen_read_reqs, off, end - off, &s->lock);
     }
 
@@ -176,6 +196,11 @@ static BlockReq *cbw_snapshot_read_lock(BlockDriverState *bs,
 
     QEMU_LOCK_GUARD(&s->lock);
 
+    if (s->snapshot_error) {
+        g_free(req);
+        return NULL;
+    }
+
     if (bdrv_dirty_bitmap_next_zero(s->access_bitmap, offset, bytes) != -1) {
         g_free(req);
         return NULL;
@@ -351,6 +376,7 @@ static BlockdevOptions *cbw_parse_options(QDict *options, Error **errp)
      * object for original options.
      */
     qdict_extract_subqdict(options, NULL, "bitmap");
+    qdict_del(options, "on-cbw-error");
 
 out:
     visit_free(v);
@@ -395,6 +421,8 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
             return -EINVAL;
         }
     }
+    s->on_cbw_error = opts->has_on_cbw_error ? opts->on_cbw_error :
+            ON_CBW_ERROR_BREAK_GUEST_WRITE;
 
     bs->total_sectors = bs->file->bs->total_sectors;
     bs->supported_write_flags = BDRV_REQ_WRITE_UNCHANGED |
Signature · 1 line
2.35.1
#3 lore raw

PATCH v4 3/7 iotests: add copy-before-write: on-cbw-error tests

In reply to Vladimir Sementsov-Ogievskiy

Add tests for new option of copy-before-write filter: on-cbw-error.

Note that we use QEMUMachine instead of VM class, because in further commit we'll want to use throttling which doesn't work with -accel qtest used by VM.

We also touch pylintrc to not break iotest 297.

---
 tests/qemu-iotests/pylintrc                   |   5 +
 tests/qemu-iotests/tests/copy-before-write    | 132 ++++++++++++++++++
 .../qemu-iotests/tests/copy-before-write.out  |   5 +
 3 files changed, 142 insertions(+)
 create mode 100755 tests/qemu-iotests/tests/copy-before-write
 create mode 100644 tests/qemu-iotests/tests/copy-before-write.out
tests/qemu-iotests/pylintrc +5 -0
index 32ab77b8bb..f4f823a991 100644
@@ -51,3 +51,8 @@ notes=FIXME,
 
 # Maximum number of characters on a single line.
 max-line-length=79
+
+
+[SIMILARITIES]
+
+min-similarity-lines=6
tests/qemu-iotests/tests/copy-before-write new file +132 -0

132 lines not on this page · raw message

tests/qemu-iotests/tests/copy-before-write.out new file +5 -0
new file mode 100644
index 0000000000..fbc63e62f8
@@ -0,0 +1,5 @@
+..
+----------------------------------------------------------------------
+Ran 2 tests
+
+OK
Signature · 1 line
2.35.1
Reviewed-by on v4 3/7
#4 lore raw

Re: [PATCH v4 3/7] iotests: add copy-before-write: on-cbw-error tests

Quoted 8 lines · Vladimir Sementsov-Ogievskiy

Add tests for new option of copy-before-write filter: on-cbw-error.

Note that we use QEMUMachine instead of VM class, because in further commit we'll want to use throttling which doesn't work with -accel qtest used by VM.

We also touch pylintrc to not break iotest 297.

Quoted 8 lines
---
  tests/qemu-iotests/pylintrc                   |   5 +
  tests/qemu-iotests/tests/copy-before-write    | 132 ++++++++++++++++++
  .../qemu-iotests/tests/copy-before-write.out  |   5 +
  3 files changed, 142 insertions(+)

create mode 100755 tests/qemu-iotests/tests/copy-before-write create mode 100644 tests/qemu-iotests/tests/copy-before-write.out

#5 lore raw

Re: [PATCH v4 3/7] iotests: add copy-before-write: on-cbw-error tests

In reply to Vladimir Sementsov-Ogievskiy

I had a long and not fun debugging session through gitlab pipelines with this:)

The problem is that pure QEMUMachine doesn't work on arm in gitlab. And we have to specify at least machine. And we don't want qtest, as described in commit message.

So, the following fix helps:

Quoted 38 lines · Vladimir Sementsov-Ogievskiy

Add tests for new option of copy-before-write filter: on-cbw-error.

Note that we use QEMUMachine instead of VM class, because in further commit we'll want to use throttling which doesn't work with -accel qtest used by VM.

We also touch pylintrc to not break iotest 297.

---
  tests/qemu-iotests/pylintrc                   |   5 +
  tests/qemu-iotests/tests/copy-before-write    | 132 ++++++++++++++++++
  .../qemu-iotests/tests/copy-before-write.out  |   5 +
  3 files changed, 142 insertions(+)

create mode 100755 tests/qemu-iotests/tests/copy-before-write create mode 100644 tests/qemu-iotests/tests/copy-before-write.out

tests/qemu-iotests/pylintrc +5 -0
index 32ab77b8bb..f4f823a991 100644
@@ -51,3 +51,8 @@ notes=FIXME,
 
  # Maximum number of characters on a single line.
  max-line-length=79
+
+
+[SIMILARITIES]
+
+min-similarity-lines=6
tests/qemu-iotests/tests/copy-before-write new file +2 -0
new file mode 100755
index 0000000000..6c7638965e
@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+# group: auto backup

[..]

tests/qemu-iotests/tests/copy-before-write line 45

Quoted 7 lines
tests/qemu-iotests/tests/copy-before-write new file +7 -0
+
+    def setUp(self):
+        qemu_img_create('-f', iotests.imgfmt, source_img, size)
+        qemu_img_create('-f', iotests.imgfmt, temp_img, size)
+        qemu_io('-c', 'write 0 1M', source_img)
+
+        self.vm = QEMUMachine(iotests.qemu_prog)

Will fix to be:

+ opts = ['-nodefaults', '-display', 'none', '-machine', 'none']
+ self.vm = QEMUMachine(iotests.qemu_prog, opts,
+ base_temp_dir=iotests.test_dir,
+ sock_dir=iotests.sock_dir)

tests/qemu-iotests/tests/copy-before-write

Quoted 2 lines
tests/qemu-iotests/tests/copy-before-write new file +2 -0
+        self.vm.launch()
+

So, if no objections, I'm going to resend a PULL request (v1 was "[PULL 00/10] Block jobs & NBD patches") with this fix and small improvement in 06.

Signature · 2 lines
Best regards,
Vladimir
#6 lore raw

PATCH v4 4/7 util: add qemu-co-timeout

In reply to Vladimir Sementsov-Ogievskiy

Add new API, to make a time limited call of the coroutine.

---
 include/qemu/coroutine.h | 13 ++++++
 util/qemu-co-timeout.c   | 89 ++++++++++++++++++++++++++++++++++++++++
 util/meson.build         |  1 +
 3 files changed, 103 insertions(+)
 create mode 100644 util/qemu-co-timeout.c
include/qemu/coroutine.h +13 -0
index c828a95ee0..8704b05da8 100644
@@ -316,6 +316,19 @@ static inline void coroutine_fn qemu_co_sleep_ns(QEMUClockType type, int64_t ns)
     qemu_co_sleep_ns_wakeable(&w, type, ns);
 }
 
+typedef void CleanupFunc(void *opaque);
+/**
+ * Run entry in a coroutine and start timer. Wait for entry to finish or for
+ * timer to elapse, what happen first. If entry finished, return 0, if timer
+ * elapsed earlier, return -ETIMEDOUT.
+ *
+ * Be careful, entry execution is not canceled, user should handle it somehow.
+ * If @clean is provided, it's called after coroutine finish if timeout
+ * happened.
+ */
+int coroutine_fn qemu_co_timeout(CoroutineEntry *entry, void *opaque,
+                                 uint64_t timeout_ns, CleanupFunc clean);
+
 /**
  * Wake a coroutine if it is sleeping in qemu_co_sleep_ns. The timer will be
  * deleted. @sleep_state must be the variable whose address was given to
util/qemu-co-timeout.c new file +89 -0

89 lines not on this page · raw message

util/meson.build +1 -0
index f6ee74ad0c..249891db72 100644
@@ -83,6 +83,7 @@ if have_block
   util_ss.add(files('block-helpers.c'))
   util_ss.add(files('qemu-coroutine-sleep.c'))
   util_ss.add(files('qemu-co-shared-resource.c'))
+  util_ss.add(files('qemu-co-timeout.c'))
   util_ss.add(files('thread-pool.c', 'qemu-timer.c'))
   util_ss.add(files('readline.c'))
   util_ss.add(files('throttle.c'))
Signature · 1 line
2.35.1
#7 lore raw

PATCH v4 5/7 block/block-copy: block_copy(): add timeout_ns parameter

In reply to Vladimir Sementsov-Ogievskiy

Add possibility to limit block_copy() call in time. To be used in the next commit.

As timed-out block_copy() call will continue in background anyway (we can't immediately cancel IO operation), it's important also give user a possibility to pass a callback, to do some additional actions on block-copy call finish.

---
 include/block/block-copy.h |  4 +++-
 block/block-copy.c         | 33 ++++++++++++++++++++++++++-------
 block/copy-before-write.c  |  2 +-
 3 files changed, 30 insertions(+), 9 deletions(-)
include/block/block-copy.h +3 -1
index 68bbd344b2..ba0b425d78 100644
@@ -40,7 +40,9 @@ int64_t block_copy_reset_unallocated(BlockCopyState *s,
                                      int64_t offset, int64_t *count);
 
 int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes,
-                            bool ignore_ratelimit);
+                            bool ignore_ratelimit, uint64_t timeout_ns,
+                            BlockCopyAsyncCallbackFunc cb,
+                            void *cb_opaque);
 
 /*
  * Run block-copy in a coroutine, create corresponding BlockCopyCallState
block/block-copy.c +26 -7
index ec46775ea5..bb947afdda 100644
@@ -883,23 +883,42 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)
     return ret;
 }
 
+static void coroutine_fn block_copy_async_co_entry(void *opaque)
+{
+    block_copy_common(opaque);
+}
+
 int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes,
-                            bool ignore_ratelimit)
+                            bool ignore_ratelimit, uint64_t timeout_ns,
+                            BlockCopyAsyncCallbackFunc cb,
+                            void *cb_opaque)
 {
-    BlockCopyCallState call_state = {
+    int ret;
+    BlockCopyCallState *call_state = g_new(BlockCopyCallState, 1);
+
+    *call_state = (BlockCopyCallState) {
         .s = s,
         .offset = start,
         .bytes = bytes,
         .ignore_ratelimit = ignore_ratelimit,
         .max_workers = BLOCK_COPY_MAX_WORKERS,
+        .cb = cb,
+        .cb_opaque = cb_opaque,
     };
 
-    return block_copy_common(&call_state);
-}
+    ret = qemu_co_timeout(block_copy_async_co_entry, call_state, timeout_ns,
+                          g_free);
+    if (ret < 0) {
+        assert(ret == -ETIMEDOUT);
+        block_copy_call_cancel(call_state);
+        /* call_state will be freed by running coroutine. */
+        return ret;
+    }
 
-static void coroutine_fn block_copy_async_co_entry(void *opaque)
-{
-    block_copy_common(opaque);
+    ret = call_state->ret;
+    g_free(call_state);
+
+    return ret;
 }
 
 BlockCopyCallState *block_copy_async(BlockCopyState *s,
block/copy-before-write.c +1 -1
index c8a11a09d2..fc13c7cd44 100644
@@ -111,7 +111,7 @@ static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs,
     off = QEMU_ALIGN_DOWN(offset, cluster_size);
     end = QEMU_ALIGN_UP(offset + bytes, cluster_size);
 
-    ret = block_copy(s->bcs, off, end - off, true);
+    ret = block_copy(s->bcs, off, end - off, true, 0, NULL, NULL);
     if (ret < 0 && s->on_cbw_error == ON_CBW_ERROR_BREAK_GUEST_WRITE) {
         return ret;
     }
Signature · 1 line
2.35.1
#8 lore raw

PATCH v4 6/7 block/copy-before-write: implement cbw-timeout option

In reply to Vladimir Sementsov-Ogievskiy

In some scenarios, when copy-before-write operations lasts too long time, it's better to cancel it.

Most useful would be to use the new option together with on-cbw-error=break-snapshot: this way if cbw operation takes too long time we'll just cancel backup process but do not disturb the guest too much.

Note the tricky point of realization: we keep additional point in bs->in_flight during block_copy operation even if it's timed-out. Background "cancelled" block_copy operations will finish at some point and will want to access state. We should care to not free the state in .bdrv_close() earlier.

---
 qapi/block-core.json      |  8 +++++++-
 block/copy-before-write.c | 23 ++++++++++++++++++++++-
 2 files changed, 29 insertions(+), 2 deletions(-)
qapi/block-core.json +7 -1
index 6b870b2f37..682b599a4a 100644
@@ -4206,12 +4206,18 @@
 # @on-cbw-error: Behavior on failure of copy-before-write operation.
 #                Default is @break-guest-write. (Since 7.1)
 #
+# @cbw-timeout: Zero means no limit. Non-zero sets the timeout in seconds
+#               for copy-before-write operation. When a timeout occurs,
+#               the respective copy-before-write operation will fail, and
+#               the @on-cbw-error parameter will decide how this failure
+#               is handled. Default 0. (Since 7.1)
+#
 # Since: 6.2
 ##
 { 'struct': 'BlockdevOptionsCbw',
   'base': 'BlockdevOptionsGenericFormat',
   'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap',
-            '*on-cbw-error': 'OnCbwError' } }
+            '*on-cbw-error': 'OnCbwError', '*cbw-timeout': 'uint32' } }
 
 ##
 # @BlockdevOptions:
block/copy-before-write.c +22 -1
index fc13c7cd44..1bc2e7f9ba 100644
@@ -42,6 +42,7 @@ typedef struct BDRVCopyBeforeWriteState {
     BlockCopyState *bcs;
     BdrvChild *target;
     OnCbwError on_cbw_error;
+    uint32_t cbw_timeout_ns;
 
     /*
      * @lock: protects access to @access_bitmap, @done_bitmap and
@@ -83,6 +84,14 @@ static coroutine_fn int cbw_co_preadv(
     return bdrv_co_preadv(bs->file, offset, bytes, qiov, flags);
 }
 
+static void block_copy_cb(void *opaque)
+{
+    BlockDriverState *bs = opaque;
+
+    bs->in_flight--;
+    aio_wait_kick();
+}
+
 /*
  * Do copy-before-write operation.
  *
@@ -111,7 +120,16 @@ static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs,
     off = QEMU_ALIGN_DOWN(offset, cluster_size);
     end = QEMU_ALIGN_UP(offset + bytes, cluster_size);
 
-    ret = block_copy(s->bcs, off, end - off, true, 0, NULL, NULL);
+    /*
+     * Increase in_flight, so that in case of timed-out block-copy, the
+     * remaining background block_copy() request (which can't be immediately
+     * cancelled by timeout) is presented in bs->in_flight. This way we are
+     * sure that on bs close() we'll previously wait for all timed-out but yet
+     * running block_copy calls.
+     */
+    bs->in_flight++;
+    ret = block_copy(s->bcs, off, end - off, true, s->cbw_timeout_ns,
+                     block_copy_cb, bs);
     if (ret < 0 && s->on_cbw_error == ON_CBW_ERROR_BREAK_GUEST_WRITE) {
         return ret;
     }
@@ -377,6 +395,7 @@ static BlockdevOptions *cbw_parse_options(QDict *options, Error **errp)
      */
     qdict_extract_subqdict(options, NULL, "bitmap");
     qdict_del(options, "on-cbw-error");
+    qdict_del(options, "cbw-timeout");
 
 out:
     visit_free(v);
@@ -423,6 +442,8 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags,
     }
     s->on_cbw_error = opts->has_on_cbw_error ? opts->on_cbw_error :
             ON_CBW_ERROR_BREAK_GUEST_WRITE;
+    s->cbw_timeout_ns = opts->has_cbw_timeout ?
+        opts->cbw_timeout * NANOSECONDS_PER_SECOND : 0;
 
     bs->total_sectors = bs->file->bs->total_sectors;
     bs->supported_write_flags = BDRV_REQ_WRITE_UNCHANGED |
Signature · 1 line
2.35.1
#9 lore raw

Re: [PATCH v4 6/7] block/copy-before-write: implement cbw-timeout option

While debugging my "[PULL 00/10] Block jobs & NBD patches", I found that we have bdrv_dec_in_flight() and bdrv_inc_in_flight().

So, this should be fixed:

Quoted 8 lines · Vladimir Sementsov-Ogievskiy

In some scenarios, when copy-before-write operations lasts too long time, it's better to cancel it.

Most useful would be to use the new option together with on-cbw-error=break-snapshot: this way if cbw operation takes too long time we'll just cancel backup process but do not disturb the guest too much.

[..]

block/copy-before-write.c static coroutine_fn int cbw_co_preadv( · line 92

Quoted 7 lines
block/copy-before-write.c +6 -0
+static void block_copy_cb(void *opaque)
+{
+    BlockDriverState *bs = opaque;
+
+    bs->in_flight--;
+    aio_wait_kick();

Just bdrv_dec_in_flight(bs), which includes aio_wait_kick().

Quoted 8 lines
block/copy-before-write.c +2 -0
+}
+
  /*
   * Do copy-before-write operation.
   *
@@ -111,7 +120,16 @@ static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs,
      off = QEMU_ALIGN_DOWN(offset, cluster_size);
      end = QEMU_ALIGN_UP(offset + bytes, cluster_size);

block/copy-before-write.c static coroutine_fn int cbw_do_copy_before_write(BlockDriverState *bs, · line 130

Quoted 10 lines
block/copy-before-write.c +8 -1
 
-    ret = block_copy(s->bcs, off, end - off, true, 0, NULL, NULL);
+    /*
+     * Increase in_flight, so that in case of timed-out block-copy, the
+     * remaining background block_copy() request (which can't be immediately
+     * cancelled by timeout) is presented in bs->in_flight. This way we are
+     * sure that on bs close() we'll previously wait for all timed-out but yet
+     * running block_copy calls.
+     */
+    bs->in_flight++;

bdrv_inc_in_flight(bs)

block/copy-before-write.c static BlockdevOptions *cbw_parse_options(QDict *options, Error **errp) · line 399

Quoted 11 lines
block/copy-before-write.c +3 -0
+    ret = block_copy(s->bcs, off, end - off, true, s->cbw_timeout_ns,
+                     block_copy_cb, bs);
      if (ret < 0 && s->on_cbw_error == ON_CBW_ERROR_BREAK_GUEST_WRITE) {
          return ret;
      }
@@ -377,6 +395,7 @@ static BlockdevOptions *cbw_parse_options(QDict *options, Error **errp)
       */
      qdict_extract_subqdict(options, NULL, "bitmap");
      qdict_del(options, "on-cbw-error");
+    qdict_del(options, "cbw-timeout");
 

I'm going to resend "[PULL 00/10] Block jobs & NBD patches" with this fix and with fix in 03, if no objections.

Signature · 2 lines
Best regards,
Vladimir
#10 lore raw

PATCH v4 7/7 iotests: copy-before-write: add cases for cbw-timeout option

In reply to Vladimir Sementsov-Ogievskiy

Add two simple test-cases: timeout failure with break-snapshot-on-cbw-error behavior and similar with break-guest-write-on-cbw-error behavior.

---
 tests/qemu-iotests/tests/copy-before-write    | 81 +++++++++++++++++++
 .../qemu-iotests/tests/copy-before-write.out  |  4 +-
 2 files changed, 83 insertions(+), 2 deletions(-)
tests/qemu-iotests/tests/copy-before-write +81 -0
index 6c7638965e..f01f26f01c 100755
@@ -126,6 +126,87 @@ read 1048576/1048576 bytes at offset 0
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 """)
 
+    def do_cbw_timeout(self, on_cbw_error):
+        result = self.vm.qmp('object-add', {
+            'qom-type': 'throttle-group',
+            'id': 'group0',
+            'limits': {'bps-write': 300 * 1024}
+        })
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp('blockdev-add', {
+            'node-name': 'cbw',
+            'driver': 'copy-before-write',
+            'on-cbw-error': on_cbw_error,
+            'cbw-timeout': 1,
+            'file': {
+                'driver': iotests.imgfmt,
+                'file': {
+                    'driver': 'file',
+                    'filename': source_img,
+                }
+            },
+            'target': {
+                'driver': 'throttle',
+                'throttle-group': 'group0',
+                'file': {
+                    'driver': 'qcow2',
+                    'file': {
+                        'driver': 'file',
+                        'filename': temp_img
+                    }
+                }
+            }
+        })
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp('blockdev-add', {
+            'node-name': 'access',
+            'driver': 'snapshot-access',
+            'file': 'cbw'
+        })
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp('human-monitor-command',
+                             command_line='qemu-io cbw "write 0 512K"')
+        self.assert_qmp(result, 'return', '')
+
+        # We need second write to trigger throttling
+        result = self.vm.qmp('human-monitor-command',
+                             command_line='qemu-io cbw "write 512K 512K"')
+        self.assert_qmp(result, 'return', '')
+
+        result = self.vm.qmp('human-monitor-command',
+                             command_line='qemu-io access "read 0 1M"')
+        self.assert_qmp(result, 'return', '')
+
+        self.vm.shutdown()
+        log = self.vm.get_log()
+        log = re.sub(r'^\[I \d+\.\d+\] OPENED\n', '', log)
+        log = re.sub(r'\[I \+\d+\.\d+\] CLOSED\n?$', '', log)
+        log = iotests.filter_qemu_io(log)
+        return log
+
+    def test_timeout_break_guest(self):
+        log = self.do_cbw_timeout('break-guest-write')
+        self.assertEqual(log, """\
+wrote 524288/524288 bytes at offset 0
+512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+write failed: Connection timed out
+read 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+""")
+
+    def test_timeout_break_snapshot(self):
+        log = self.do_cbw_timeout('break-snapshot')
+        self.assertEqual(log, """\
+wrote 524288/524288 bytes at offset 0
+512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 524288/524288 bytes at offset 524288
+512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read failed: Permission denied
+""")
+
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=['qcow2'],
tests/qemu-iotests/tests/copy-before-write.out +2 -2
index fbc63e62f8..89968f35d7 100644
@@ -1,5 +1,5 @@
-..
+....
 ----------------------------------------------------------------------
-Ran 2 tests
+Ran 4 tests
 
 OK
Signature · 1 line
2.35.1
Reviewed-by on v4 7/7
#11 lore raw

Re: [PATCH v4 7/7] iotests: copy-before-write: add cases for cbw-timeout option

Quoted 9 lines · Vladimir Sementsov-Ogievskiy

Add two simple test-cases: timeout failure with break-snapshot-on-cbw-error behavior and similar with break-guest-write-on-cbw-error behavior.

---
  tests/qemu-iotests/tests/copy-before-write    | 81 +++++++++++++++++++
  .../qemu-iotests/tests/copy-before-write.out  |  4 +-
  2 files changed, 83 insertions(+), 2 deletions(-)
#12 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Quoted 42 lines · Vladimir Sementsov-Ogievskiy

Hi all!

v4: Now based on master
01: add assertion and r-b
02: s/7.0/7.1/ and r-b
03: switch to QEMUMachine, touch-up pylintrc, drop r-b 04,05,06: add r-b
07: switch to QEMUMachine

Here are two new options for copy-before-write filter:

on-cbw-error allows to alter the behavior on copy-before-write operation
failure: not break guest write but break the snapshot (and therefore backup process)

cbw-timeout allows to limit cbw operation by some timeout.

So, for example, using cbw-timeout=60 and on-cbw-error=break-snapshot you can be sure that guest write will not stuck for more than 60 seconds and will never fail due to backup problems.

Vladimir Sementsov-Ogievskiy (7):
block/copy-before-write: refactor option parsing block/copy-before-write: add on-cbw-error open parameter
iotests: add copy-before-write: on-cbw-error tests
util: add qemu-co-timeout
block/block-copy: block_copy(): add timeout_ns parameter block/copy-before-write: implement cbw-timeout option
iotests: copy-before-write: add cases for cbw-timeout option

  qapi/block-core.json                          |  31 ++-
  include/block/block-copy.h                    |   4 +-
  include/qemu/coroutine.h                      |  13 ++
  block/block-copy.c                            |  33 ++-
  block/copy-before-write.c                     | 111 ++++++---
  util/qemu-co-timeout.c                        |  89 ++++++++
  tests/qemu-iotests/pylintrc                   |   5 +
  tests/qemu-iotests/tests/copy-before-write    | 213 ++++++++++++++++++
  .../qemu-iotests/tests/copy-before-write.out  |   5 +
  util/meson.build                              |   1 +
  10 files changed, 466 insertions(+), 39 deletions(-)
Quoted 4 lines

create mode 100644 util/qemu-co-timeout.c
create mode 100755 tests/qemu-iotests/tests/copy-before-write create mode 100644 tests/qemu-iotests/tests/copy-before-write.out

Thanks for review, applied to my new block branch at https://gitlab.com/vsementsov/qemu.git

Signature · 2 lines
Best regards,
Vladimir
#13 lore raw
Quoted 48 lines · Vladimir Sementsov-Ogievskiy
Quoted 41 lines · Vladimir Sementsov-Ogievskiy

Hi all!

v4: Now based on master
01: add assertion and r-b
02: s/7.0/7.1/ and r-b
03: switch to QEMUMachine, touch-up pylintrc,  drop r-b 04,05,06: add r-b
07: switch to QEMUMachine

Here are two new options for copy-before-write filter:

on-cbw-error allows to alter the behavior on copy-before-write operation
failure: not break guest write but break the snapshot (and therefore backup process)

cbw-timeout allows to limit cbw operation by some timeout.

So, for example, using cbw-timeout=60 and on-cbw-error=break-snapshot you can be sure that guest write will not stuck for more than 60 seconds and will never fail due to backup problems.

Vladimir Sementsov-Ogievskiy (7):
   block/copy-before-write: refactor option parsing block/copy-before-write: add on-cbw-error open parameter
   iotests: add copy-before-write: on-cbw-error tests
   util: add qemu-co-timeout
   block/block-copy: block_copy(): add timeout_ns parameter block/copy-before-write: implement cbw-timeout option
   iotests: copy-before-write: add cases for cbw-timeout option

  qapi/block-core.json                          |  31 ++- include/block/block-copy.h                    |   4 +- include/qemu/coroutine.h                      |  13 ++ block/block-copy.c                            |  33 ++- block/copy-before-write.c                     | 111 ++++++--- util/qemu-co-timeout.c                        |  89 ++++++++ tests/qemu-iotests/pylintrc                   |   5 + tests/qemu-iotests/tests/copy-before-write    | 213 ++++++++++++++++++ .../qemu-iotests/tests/copy-before-write.out  |   5 + util/meson.build                              |   1 +

Quoted 5 lines
  10 files changed, 466 insertions(+), 39 deletions(-)

  create mode 100644 util/qemu-co-timeout.c
  create mode 100755 tests/qemu-iotests/tests/copy-before-write create mode 100644 tests/qemu-iotests/tests/copy-before-write.out

Quoted 2 lines

Thanks for review, applied to my new block branch at https://gitlab.com/vsementsov/qemu.git

Or not. I still need an acc for QAPI interface (Eric or Markus could you please look?).

Also, may be I should rename qemu-co-timeout.c to qemu-coroutine-timeout.c, to match "F: util/*coroutine*" in MAINTAINERS.. Stefan, Kevin, could you please look at it?

Signature · 2 lines
Best regards,
Vladimir
#14 lore raw
Quoted 49 lines · Vladimir Sementsov-Ogievskiy
Quoted 48 lines · Vladimir Sementsov-Ogievskiy
Quoted 46 lines · Vladimir Sementsov-Ogievskiy

Hi all!

v4: Now based on master
01: add assertion and r-b
02: s/7.0/7.1/ and r-b
03: switch to QEMUMachine, touch-up pylintrc,  drop r-b 04,05,06: add r-b
07: switch to QEMUMachine

Here are two new options for copy-before-write filter:

on-cbw-error allows to alter the behavior on copy-before-write operation
failure: not break guest write but break the snapshot (and therefore backup process)

cbw-timeout allows to limit cbw operation by some timeout.

So, for example, using cbw-timeout=60 and on-cbw-error=break-snapshot you can be sure that guest write will not stuck for more than 60 seconds and will never fail due to backup problems.

Vladimir Sementsov-Ogievskiy (7):
   block/copy-before-write: refactor option parsing block/copy-before-write: add on-cbw-error open parameter
   iotests: add copy-before-write: on-cbw-error tests
   util: add qemu-co-timeout
   block/block-copy: block_copy(): add timeout_ns parameter block/copy-before-write: implement cbw-timeout option
   iotests: copy-before-write: add cases for cbw-timeout option

  qapi/block-core.json                          |  31 ++- include/block/block-copy.h                    |   4 +- include/qemu/coroutine.h                      |  13 ++ block/block-copy.c                            |  33 ++- block/copy-before-write.c                     | 111 ++++++--- util/qemu-co-timeout.c                        |  89 ++++++++ tests/qemu-iotests/pylintrc                   |   5 + tests/qemu-iotests/tests/copy-before-write    | 213 ++++++++++++++++++ .../qemu-iotests/tests/copy-before-write.out  |   5 + util/meson.build                              |   1 +

  10 files changed, 466 insertions(+), 39 deletions(-)

  create mode 100644 util/qemu-co-timeout.c
  create mode 100755 tests/qemu-iotests/tests/copy-before-write create mode 100644 tests/qemu-iotests/tests/copy-before-write.out

Quoted 8 lines
Quoted 2 lines

Thanks for review, applied to my new block branch at https://gitlab.com/vsementsov/qemu.git

Or not. I still need an acc for QAPI interface (Eric or Markus could you please look?).

Also, may be I should rename qemu-co-timeout.c to qemu-coroutine-timeout.c, to match "F: util/*coroutine*" in MAINTAINERS.. Stefan, Kevin, could you please look at it?

OK, I think, I can stage it, if no more comments. API changes are quite usual and new qemu_co_timeout is isolated. Applied to my block branch https://gitlab.com/vsementsov/qemu/-/commits/block

I think, I'll prepare a pull request on Monday, and include also my "[PATCH] MAINTAINERS: update Vladimir's address and repositories" if Stefan don't send it earlier.

Signature · 2 lines
Best regards,
Vladimir