Skip to contentSkip to conversation

active-mirror: support unaligned guest operations

PATCH Superseded Vladimir Sementsov-Ogievskiy 2 participants 25 messages Sep 12, 2019 – Oct 11, 2019 Block layer

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover active-mirror: support unaligned guest operations

Commit 9adc1cb49af8d fixed a bug about unaligned (to dirty bitmap granularity) guest writes (and discards) by simply requesting corresponding alignment on mirror-top filter. However forcing large alignment obviously decreases performance of unaligned requests.

So it's time for a new solution which is in 03. And 04 reverts 9adc1cb49af8d.

Vladimir Sementsov-Ogievskiy (4):
block/mirror: simplify do_sync_target_write block/block-backend: add blk_co_pwritev_part block/mirror: support unaligned write in active mirror Revert "mirror: Only mirror granularity-aligned chunks"

 include/sysemu/block-backend.h |   4 +
 block/block-backend.c          |  17 +++-
 block/mirror.c                 | 153 +++++++++++++--------------------
 3 files changed, 78 insertions(+), 96 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

PATCH 1/4 block/mirror: simplify do_sync_target_write

do_sync_target_write is called from bdrv_mirror_top_do_write after write/discard operation, all inside active_write/active_write_settle protecting us from mirror iteration. So the whole area is dirty for sure, no reason to examine dirty bitmap.

---
 block/mirror.c | 95 +++++++++++++++-----------------------------------
 1 file changed, 28 insertions(+), 67 deletions(-)
block/mirror.c +28 -67
index 853e2c7510..d176bf5920 100644
@@ -1203,84 +1203,45 @@ do_sync_target_write(MirrorBlockJob *job, MirrorMethod method,
                      uint64_t offset, uint64_t bytes,
                      QEMUIOVector *qiov, int flags)
 {
-    QEMUIOVector target_qiov;
-    uint64_t dirty_offset = offset;
-    uint64_t dirty_bytes;
-
-    if (qiov) {
-        qemu_iovec_init(&target_qiov, qiov->niov);
-    }
-
-    while (true) {
-        bool valid_area;
-        int ret;
-
-        bdrv_dirty_bitmap_lock(job->dirty_bitmap);
-        dirty_bytes = MIN(offset + bytes - dirty_offset, INT_MAX);
-        valid_area = bdrv_dirty_bitmap_next_dirty_area(job->dirty_bitmap,
-                                                       &dirty_offset,
-                                                       &dirty_bytes);
-        if (!valid_area) {
-            bdrv_dirty_bitmap_unlock(job->dirty_bitmap);
-            break;
-        }
+    int ret;
 
-        bdrv_reset_dirty_bitmap_locked(job->dirty_bitmap,
-                                       dirty_offset, dirty_bytes);
-        bdrv_dirty_bitmap_unlock(job->dirty_bitmap);
+    bdrv_reset_dirty_bitmap(job->dirty_bitmap, offset, bytes);
 
-        job_progress_increase_remaining(&job->common.job, dirty_bytes);
+    job_progress_increase_remaining(&job->common.job, bytes);
 
-        assert(dirty_offset - offset <= SIZE_MAX);
-        if (qiov) {
-            qemu_iovec_reset(&target_qiov);
-            qemu_iovec_concat(&target_qiov, qiov,
-                              dirty_offset - offset, dirty_bytes);
-        }
-
-        switch (method) {
-        case MIRROR_METHOD_COPY:
-            ret = blk_co_pwritev(job->target, dirty_offset, dirty_bytes,
-                                 qiov ? &target_qiov : NULL, flags);
-            break;
+    switch (method) {
+    case MIRROR_METHOD_COPY:
+        ret = blk_co_pwritev(job->target, offset, bytes, qiov, flags);
+        break;
 
-        case MIRROR_METHOD_ZERO:
-            assert(!qiov);
-            ret = blk_co_pwrite_zeroes(job->target, dirty_offset, dirty_bytes,
-                                       flags);
-            break;
+    case MIRROR_METHOD_ZERO:
+        assert(!qiov);
+        ret = blk_co_pwrite_zeroes(job->target, offset, bytes, flags);
+        break;
 
-        case MIRROR_METHOD_DISCARD:
-            assert(!qiov);
-            ret = blk_co_pdiscard(job->target, dirty_offset, dirty_bytes);
-            break;
+    case MIRROR_METHOD_DISCARD:
+        assert(!qiov);
+        ret = blk_co_pdiscard(job->target, offset, bytes);
+        break;
 
-        default:
-            abort();
-        }
+    default:
+        abort();
+    }
 
-        if (ret >= 0) {
-            job_progress_update(&job->common.job, dirty_bytes);
-        } else {
-            BlockErrorAction action;
+    if (ret >= 0) {
+        job_progress_update(&job->common.job, bytes);
+    } else {
+        BlockErrorAction action;
 
-            bdrv_set_dirty_bitmap(job->dirty_bitmap, dirty_offset, dirty_bytes);
-            job->actively_synced = false;
+        bdrv_set_dirty_bitmap(job->dirty_bitmap, offset, bytes);
+        job->actively_synced = false;
 
-            action = mirror_error_action(job, false, -ret);
-            if (action == BLOCK_ERROR_ACTION_REPORT) {
-                if (!job->ret) {
-                    job->ret = ret;
-                }
-                break;
+        action = mirror_error_action(job, false, -ret);
+        if (action == BLOCK_ERROR_ACTION_REPORT) {
+            if (!job->ret) {
+                job->ret = ret;
             }
         }
-
-        dirty_offset += dirty_bytes;
-    }
-
-    if (qiov) {
-        qemu_iovec_destroy(&target_qiov);
     }
 }
 
Signature · 1 line
2.21.0
Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH 1/4] block/mirror: simplify do_sync_target_write

Quoted 9 lines · Vladimir Sementsov-Ogievskiy

do_sync_target_write is called from bdrv_mirror_top_do_write after write/discard operation, all inside active_write/active_write_settle protecting us from mirror iteration. So the whole area is dirty for sure, no reason to examine dirty bitmap.

---
 block/mirror.c | 95 +++++++++++++++-----------------------------------
 1 file changed, 28 insertions(+), 67 deletions(-)
#3 lore raw

PATCH 2/4 block/block-backend: add blk_co_pwritev_part

In reply to Vladimir Sementsov-Ogievskiy

Add blk write function with qiov_offset parameter. It's needed for the following commit.

---
 include/sysemu/block-backend.h |  4 ++++
 block/block-backend.c          | 17 +++++++++++++----
 2 files changed, 17 insertions(+), 4 deletions(-)
include/sysemu/block-backend.h +4 -0
index 368d53af77..73f2cef7fe 100644
@@ -121,6 +121,10 @@ void blk_set_dev_ops(BlockBackend *blk, const BlockDevOps *ops, void *opaque);
 int coroutine_fn blk_co_preadv(BlockBackend *blk, int64_t offset,
                                unsigned int bytes, QEMUIOVector *qiov,
                                BdrvRequestFlags flags);
+int coroutine_fn blk_co_pwritev_part(BlockBackend *blk, int64_t offset,
+                                     unsigned int bytes,
+                                     QEMUIOVector *qiov, size_t qiov_offset,
+                                     BdrvRequestFlags flags);
 int coroutine_fn blk_co_pwritev(BlockBackend *blk, int64_t offset,
                                unsigned int bytes, QEMUIOVector *qiov,
                                BdrvRequestFlags flags);
block/block-backend.c +13 -4
index 1c605d5444..b3d00478af 100644
@@ -1176,9 +1176,10 @@ int coroutine_fn blk_co_preadv(BlockBackend *blk, int64_t offset,
     return ret;
 }
 
-int coroutine_fn blk_co_pwritev(BlockBackend *blk, int64_t offset,
-                                unsigned int bytes, QEMUIOVector *qiov,
-                                BdrvRequestFlags flags)
+int coroutine_fn blk_co_pwritev_part(BlockBackend *blk, int64_t offset,
+                                     unsigned int bytes,
+                                     QEMUIOVector *qiov, size_t qiov_offset,
+                                     BdrvRequestFlags flags)
 {
     int ret;
     BlockDriverState *bs;
@@ -1205,11 +1206,19 @@ int coroutine_fn blk_co_pwritev(BlockBackend *blk, int64_t offset,
         flags |= BDRV_REQ_FUA;
     }
 
-    ret = bdrv_co_pwritev(blk->root, offset, bytes, qiov, flags);
+    ret = bdrv_co_pwritev_part(blk->root, offset, bytes, qiov, qiov_offset,
+                               flags);
     bdrv_dec_in_flight(bs);
     return ret;
 }
 
+int coroutine_fn blk_co_pwritev(BlockBackend *blk, int64_t offset,
+                                unsigned int bytes, QEMUIOVector *qiov,
+                                BdrvRequestFlags flags)
+{
+    return blk_co_pwritev_part(blk, offset, bytes, qiov, 0, flags);
+}
+
 typedef struct BlkRwCo {
     BlockBackend *blk;
     int64_t offset;
Signature · 1 line
2.21.0
Reviewed-by on v1 2/4
#4 lore raw

Re: [PATCH 2/4] block/block-backend: add blk_co_pwritev_part

Quoted 8 lines · Vladimir Sementsov-Ogievskiy

Add blk write function with qiov_offset parameter. It's needed for the following commit.

---
 include/sysemu/block-backend.h |  4 ++++
 block/block-backend.c          | 17 +++++++++++++----
 2 files changed, 17 insertions(+), 4 deletions(-)

I don’t know about the next commit, but:

#5 lore raw

PATCH 3/4 block/mirror: support unaligned write in active mirror

In reply to Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

If unaligned padding is not dirty, we just write it, no reason to touch dirty bitmap if we succeed (on failure we'll set the whole region ofcourse, but we loss "synchronicity" on failure anyway).

Note: we need to disable dirty_bitmap, otherwise we will not be able to see in do_sync_target_write bitmap state before current operation. We may of course check dirty bitmap before the operation in bdrv_mirror_top_do_write and remember it, but we don't need active dirty bitmap for write-blocking mirror anyway.

New code-path is unused until the following commit reverts 9adc1cb49af8d.

---
 block/mirror.c | 39 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)
block/mirror.c +38 -1
index d176bf5920..d192f6a96b 100644
@@ -1204,6 +1204,39 @@ do_sync_target_write(MirrorBlockJob *job, MirrorMethod method,
                      QEMUIOVector *qiov, int flags)
 {
     int ret;
+    size_t qiov_offset = 0;
+
+    if (!QEMU_IS_ALIGNED(offset, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset)) {
+            /*
+             * Dirty unaligned padding
+             * 1. It's already dirty, no damage to "actively_synced" if we just
+             *    skip unaligned part.
+             * 2. If we copy it, we can't reset corresponding bit in
+             *    dirty_bitmap as there may be some "dirty" bytes still not
+             *    copied.
+             * So, just ignore it.
+             */
+            qiov_offset = QEMU_ALIGN_UP(offset, job->granularity) - offset;
+            if (bytes <= qiov_offset) {
+                /* nothing to do after shrink */
+                return;
+            }
+            offset += qiov_offset;
+            bytes -= qiov_offset;
+    }
+
+    if (!QEMU_IS_ALIGNED(offset + bytes, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset + bytes - 1))
+    {
+        uint64_t tail = (offset + bytes) % job->granularity;
+
+        if (bytes <= tail) {
+            /* nothing to do after shrink */
+            return;
+        }
+        bytes -= tail;
+    }
 
     bdrv_reset_dirty_bitmap(job->dirty_bitmap, offset, bytes);
 
@@ -1211,7 +1244,8 @@ do_sync_target_write(MirrorBlockJob *job, MirrorMethod method,
 
     switch (method) {
     case MIRROR_METHOD_COPY:
-        ret = blk_co_pwritev(job->target, offset, bytes, qiov, flags);
+        ret = blk_co_pwritev_part(job->target, offset, bytes,
+                                  qiov, qiov_offset, flags);
         break;
 
     case MIRROR_METHOD_ZERO:
@@ -1640,6 +1674,9 @@ static BlockJob *mirror_start_job(
     if (!s->dirty_bitmap) {
         goto fail;
     }
+    if (s->copy_mode == MIRROR_COPY_MODE_WRITE_BLOCKING) {
+        bdrv_disable_dirty_bitmap(s->dirty_bitmap);
+    }
 
     ret = block_job_add_bdrv(&s->common, "source", bs, 0,
                              BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE |
Signature · 1 line
2.21.0
#6 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 10 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

Quoted 6 lines

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

But that’s not what active mirror is for. The point of active mirror is that it must converge because every guest write will contribute towards that goal.

If you skip active mirroring for unaligned guest writes, they will not contribute towards converging, but in fact lead to the opposite.

Max

#7 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 18 lines · Max Reitz
Quoted 10 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

Quoted 6 lines

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

Quoted 7 lines

But that’s not what active mirror is for. The point of active mirror is that it must converge because every guest write will contribute towards that goal.

If you skip active mirroring for unaligned guest writes, they will not contribute towards converging, but in fact lead to the opposite.

The will not contribute only if region is already dirty. Actually, after first iteration of mirroring (copying the whole disk), all following writes will contribute, so the whole process must converge. It is a bit similar with running one mirror loop in normal mode, and then enable write-blocking.

Signature · 2 lines
Best regards,
Vladimir
#8 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Quoted 22 lines · Max Reitz
Quoted 10 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

Quoted 6 lines

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

But that’s not what active mirror is for.  The point of active mirror is that it must converge because every guest write will contribute towards that goal.

Quoted 9 lines
Quoted 3 lines

If you skip active mirroring for unaligned guest writes, they will not contribute towards converging, but in fact lead to the opposite.

The will not contribute only if region is already dirty. Actually, after first iteration of mirroring (copying the whole disk), all following writes will contribute, so the whole process must converge. It is a bit similar with running one mirror loop in normal mode, and then enable write-blocking.

In other words, we don't need "all guest writes contribute" to converge, "all guest writes don't create new dirty bits" is enough, as we have parallel mirror iteration which contiguously handles dirty bits.

Signature · 2 lines
Best regards,
Vladimir
#9 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 35 lines · Vladimir Sementsov-Ogievskiy
Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Quoted 22 lines · Max Reitz
Quoted 10 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

Quoted 6 lines

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

But that’s not what active mirror is for.  The point of active mirror is that it must converge because every guest write will contribute towards that goal.

Quoted 9 lines
Quoted 3 lines

If you skip active mirroring for unaligned guest writes, they will not contribute towards converging, but in fact lead to the opposite.

The will not contribute only if region is already dirty. Actually, after first iteration of mirroring (copying the whole disk), all following writes will contribute, so the whole process must converge. It is a bit similar with running one mirror loop in normal mode, and then enable write-blocking.

Quoted 3 lines

In other words, we don't need "all guest writes contribute" to converge, "all guest writes don't create new dirty bits" is enough, as we have parallel mirror iteration which contiguously handles dirty bits.

Hm, in a sense. But it does mean that guest writes will not contribute to convergence.

And that’s against the current definition of write-blocking, which does state that “when data is written to the source, write it (synchronously) to the target as well”.

Max

#10 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 40 lines · Max Reitz
Quoted 35 lines · Vladimir Sementsov-Ogievskiy
Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Quoted 22 lines · Max Reitz
Quoted 10 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

Quoted 6 lines

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

But that’s not what active mirror is for.  The point of active mirror is that it must converge because every guest write will contribute towards that goal.

Quoted 9 lines
Quoted 3 lines

If you skip active mirroring for unaligned guest writes, they will not contribute towards converging, but in fact lead to the opposite.

The will not contribute only if region is already dirty. Actually, after first iteration of mirroring (copying the whole disk), all following writes will contribute, so the whole process must converge. It is a bit similar with running one mirror loop in normal mode, and then enable write-blocking.

Quoted 3 lines

In other words, we don't need "all guest writes contribute" to converge, "all guest writes don't create new dirty bits" is enough, as we have parallel mirror iteration which contiguously handles dirty bits.

Quoted 7 lines

Hm, in a sense. But it does mean that guest writes will not contribute to convergence.

And that’s against the current definition of write-blocking, which does state that “when data is written to the source, write it (synchronously) to the target as well”.

Hmm, understand. But IMHO our proposed behavior is better in general. Do you think it's a problem to change spec now?
If yes, I'll resend with an option

Signature · 2 lines
Best regards,
Vladimir
#11 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 44 lines · Vladimir Sementsov-Ogievskiy
Quoted 43 lines · Max Reitz
Quoted 35 lines · Vladimir Sementsov-Ogievskiy
Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Quoted 22 lines · Max Reitz
Quoted 10 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

Quoted 6 lines

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

But that’s not what active mirror is for.  The point of active mirror is that it must converge because every guest write will contribute towards that goal.

Quoted 9 lines
Quoted 3 lines

If you skip active mirroring for unaligned guest writes, they will not contribute towards converging, but in fact lead to the opposite.

The will not contribute only if region is already dirty. Actually, after first iteration of mirroring (copying the whole disk), all following writes will contribute, so the whole process must converge. It is a bit similar with running one mirror loop in normal mode, and then enable write-blocking.

Quoted 3 lines

In other words, we don't need "all guest writes contribute" to converge, "all guest writes don't create new dirty bits" is enough, as we have parallel mirror iteration which contiguously handles dirty bits.

Hm, in a sense. But it does mean that guest writes will not contribute to convergence.

Quoted 8 lines
Quoted 4 lines

And that’s against the current definition of write-blocking, which does state that “when data is written to the source, write it (synchronously) to the target as well”.

Hmm, understand. But IMHO our proposed behavior is better in general. Do you think it's a problem to change spec now?
If yes, I'll resend with an option

Well, the thing is that I’d find it weird if write-blocking wasn’t blocking in all cases. And in my opinion, it makes more sense for active mirror if all writes actively contributed to convergence.

Max

#12 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 50 lines · Max Reitz
Quoted 49 lines · Vladimir Sementsov-Ogievskiy
Quoted 47 lines · Max Reitz
Quoted 35 lines · Vladimir Sementsov-Ogievskiy
Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Quoted 22 lines · Max Reitz
Quoted 10 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

Quoted 6 lines

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

But that’s not what active mirror is for.  The point of active mirror is that it must converge because every guest write will contribute towards that goal.

Quoted 9 lines
Quoted 3 lines

If you skip active mirroring for unaligned guest writes, they will not contribute towards converging, but in fact lead to the opposite.

The will not contribute only if region is already dirty. Actually, after first iteration of mirroring (copying the whole disk), all following writes will contribute, so the whole process must converge. It is a bit similar with running one mirror loop in normal mode, and then enable write-blocking.

Quoted 3 lines

In other words, we don't need "all guest writes contribute" to converge, "all guest writes don't create new dirty bits" is enough, as we have parallel mirror iteration which contiguously handles dirty bits.

Hm, in a sense. But it does mean that guest writes will not contribute to convergence.

And that’s against the current definition of write-blocking, which does state that “when data is written to the source, write it (synchronously) to the target as well”.

Quoted 8 lines
Quoted 3 lines

Hmm, understand. But IMHO our proposed behavior is better in general. Do you think it's a problem to change spec now?
If yes, I'll resend with an option

Well, the thing is that I’d find it weird if write-blocking wasn’t blocking in all cases. And in my opinion, it makes more sense for active mirror if all writes actively contributed to convergence.

Why? What is the benefit in it?
What is "all writes actively contributed to convergence" for user?

I think for user there may be the following criteria:

1. guaranteed converge, with any guest write load. Both current and my proposed variants are OK.

2. Less impact on guest.
Obviously my proposed variant is better

3. Total time of mirroring
Seems, current may be a bit better, but I don't think that unaligned tails gives significant impact.

===

So, assume I want [1]+[2]. And possibly
2.2: Even less impact on guest: ignore not only unaligned tails if they are already dirty, but full synchronous mirror operation if area is already dirty.

How should I call this? Should it be separate mode, or option for write-blocking?

Signature · 2 lines
Best regards,
Vladimir
#13 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 55 lines · Vladimir Sementsov-Ogievskiy
Quoted 54 lines · Max Reitz
Quoted 52 lines · Vladimir Sementsov-Ogievskiy
Quoted 40 lines · Max Reitz
Quoted 35 lines · Vladimir Sementsov-Ogievskiy
Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Quoted 22 lines · Max Reitz
Quoted 10 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

Quoted 6 lines

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

But that’s not what active mirror is for.  The point of active mirror is that it must converge because every guest write will contribute towards that goal.

Quoted 9 lines
Quoted 3 lines

If you skip active mirroring for unaligned guest writes, they will not contribute towards converging, but in fact lead to the opposite.

The will not contribute only if region is already dirty. Actually, after first iteration of mirroring (copying the whole disk), all following writes will contribute, so the whole process must converge. It is a bit similar with running one mirror loop in normal mode, and then enable write-blocking.

Quoted 3 lines

In other words, we don't need "all guest writes contribute" to converge, "all guest writes don't create new dirty bits" is enough, as we have parallel mirror iteration which contiguously handles dirty bits.

Quoted 7 lines

Hm, in a sense. But it does mean that guest writes will not contribute to convergence.

And that’s against the current definition of write-blocking, which does state that “when data is written to the source, write it (synchronously) to the target as well”.

Hmm, understand. But IMHO our proposed behavior is better in general. Do you think it's a problem to change spec now?
If yes, I'll resend with an option

Quoted 7 lines
Quoted 4 lines

Well, the thing is that I’d find it weird if write-blocking wasn’t blocking in all cases. And in my opinion, it makes more sense for active mirror if all writes actively contributed to convergence.

Why? What is the benefit in it?
What is "all writes actively contributed to convergence" for user?

One thing I wonder about is whether it’s really guaranteed that the background job will run under full I/O load, and how often it runs.

I fear that with your model, the background job might starve and the mirror may take a very long time. It won’t diverge, but it also won’t really converge.

The advantage of letting all writes block is that even under full I/O load, the mirror job will progress at a steady pace.

Quoted 12 lines

I think for user there may be the following criteria:

1. guaranteed converge, with any guest write load. Both current and my proposed variants are OK.

2. Less impact on guest.
Obviously my proposed variant is better

3. Total time of mirroring
Seems, current may be a bit better, but I don't think that unaligned tails gives significant impact.

Quoted 7 lines

===

So, assume I want [1]+[2]. And possibly
2.2: Even less impact on guest: ignore not only unaligned tails if they are already dirty, but full synchronous mirror operation if area is already dirty.

How should I call this? Should it be separate mode, or option for write-blocking?

I don’t know whether it makes sense to add a separate mode or a separate option just for this difference. I don’t think anyone would choose the non-default option.

But I do think there’s quite a bit of difference in how the job behaves still... I don’t know. :-/

Max

#14 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 61 lines · Max Reitz
Quoted 60 lines · Vladimir Sementsov-Ogievskiy
Quoted 58 lines · Max Reitz
Quoted 44 lines · Vladimir Sementsov-Ogievskiy
Quoted 43 lines · Max Reitz
Quoted 35 lines · Vladimir Sementsov-Ogievskiy
Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Quoted 22 lines · Max Reitz
Quoted 10 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

Quoted 6 lines

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

But that’s not what active mirror is for.  The point of active mirror is that it must converge because every guest write will contribute towards that goal.

Quoted 9 lines
Quoted 3 lines

If you skip active mirroring for unaligned guest writes, they will not contribute towards converging, but in fact lead to the opposite.

The will not contribute only if region is already dirty. Actually, after first iteration of mirroring (copying the whole disk), all following writes will contribute, so the whole process must converge. It is a bit similar with running one mirror loop in normal mode, and then enable write-blocking.

Quoted 3 lines

In other words, we don't need "all guest writes contribute" to converge, "all guest writes don't create new dirty bits" is enough, as we have parallel mirror iteration which contiguously handles dirty bits.

Hm, in a sense. But it does mean that guest writes will not contribute to convergence.

Quoted 8 lines
Quoted 4 lines

And that’s against the current definition of write-blocking, which does state that “when data is written to the source, write it (synchronously) to the target as well”.

Hmm, understand. But IMHO our proposed behavior is better in general. Do you think it's a problem to change spec now?
If yes, I'll resend with an option

Well, the thing is that I’d find it weird if write-blocking wasn’t blocking in all cases. And in my opinion, it makes more sense for active mirror if all writes actively contributed to convergence.

Quoted 8 lines
Quoted 2 lines

Why? What is the benefit in it?
What is "all writes actively contributed to convergence" for user?

One thing I wonder about is whether it’s really guaranteed that the background job will run under full I/O load, and how often it runs.

I fear that with your model, the background job might starve and the mirror may take a very long time.

Hmmm. I think mirror job is in same context as guest writes, and goes through same IO api.. Why it will starve? (I understand that my words are not an evidence...).

Quoted 2 lines

It won’t diverge, but it also won’t
really converge.

But same will be with current behavior: guest is not guaranteed to write to all parts of disk. And in most scenarios it doesn't. So, if mirror job starve because of huge guest IO load, we will not converge anyway.

So, background process is necessary thing for converge anyway.

Quoted 22 lines

The advantage of letting all writes block is that even under full I/O load, the mirror job will progress at a steady pace.

Quoted 18 lines

I think for user there may be the following criteria:

1. guaranteed converge, with any guest write load. Both current and my proposed variants are OK.

2. Less impact on guest.
Obviously my proposed variant is better

3. Total time of mirroring
Seems, current may be a bit better, but I don't think that unaligned tails gives significant impact.

===

So, assume I want [1]+[2]. And possibly
2.2: Even less impact on guest: ignore not only unaligned tails if they are already dirty, but full synchronous mirror operation if area is already dirty.

Quoted 5 lines
Quoted 1 line

How should I call this? Should it be separate mode, or option for write-blocking?

I don’t know whether it makes sense to add a separate mode or a separate option just for this difference. I don’t think anyone would choose the non-default option.

At least Virtuozzo will choose :)

Quoted 6 lines

But I do think there’s quite a bit of difference in how the job behaves still... I don’t know. :-/

Max

Signature · 2 lines
Best regards,
Vladimir
#15 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 68 lines · Vladimir Sementsov-Ogievskiy
Quoted 67 lines · Max Reitz
Quoted 55 lines · Vladimir Sementsov-Ogievskiy
Quoted 54 lines · Max Reitz
Quoted 52 lines · Vladimir Sementsov-Ogievskiy
Quoted 40 lines · Max Reitz
Quoted 35 lines · Vladimir Sementsov-Ogievskiy
Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Quoted 22 lines · Max Reitz
Quoted 10 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

Quoted 6 lines

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

But that’s not what active mirror is for.  The point of active mirror is that it must converge because every guest write will contribute towards that goal.

Quoted 9 lines
Quoted 3 lines

If you skip active mirroring for unaligned guest writes, they will not contribute towards converging, but in fact lead to the opposite.

The will not contribute only if region is already dirty. Actually, after first iteration of mirroring (copying the whole disk), all following writes will contribute, so the whole process must converge. It is a bit similar with running one mirror loop in normal mode, and then enable write-blocking.

Quoted 3 lines

In other words, we don't need "all guest writes contribute" to converge, "all guest writes don't create new dirty bits" is enough, as we have parallel mirror iteration which contiguously handles dirty bits.

Quoted 7 lines

Hm, in a sense. But it does mean that guest writes will not contribute to convergence.

And that’s against the current definition of write-blocking, which does state that “when data is written to the source, write it (synchronously) to the target as well”.

Hmm, understand. But IMHO our proposed behavior is better in general. Do you think it's a problem to change spec now?
If yes, I'll resend with an option

Quoted 7 lines
Quoted 4 lines

Well, the thing is that I’d find it weird if write-blocking wasn’t blocking in all cases. And in my opinion, it makes more sense for active mirror if all writes actively contributed to convergence.

Why? What is the benefit in it?
What is "all writes actively contributed to convergence" for user?

One thing I wonder about is whether it’s really guaranteed that the background job will run under full I/O load, and how often it runs.

Quoted 6 lines
Quoted 2 lines

I fear that with your model, the background job might starve and the mirror may take a very long time.

Hmmm. I think mirror job is in same context as guest writes, and goes through same IO api.. Why it will starve? (I understand that my words are not an evidence...).

I thought that maybe if the disk is read to write and write all the time, there’d be no time for the mirror coroutine.

Quoted 8 lines
Quoted 2 lines

It won’t diverge, but it also won’t
really converge.

But same will be with current behavior: guest is not guaranteed to write to all parts of disk. And in most scenarios it doesn't. So, if mirror job starve because of huge guest IO load, we will not converge anyway.

So, background process is necessary thing for converge anyway.

Good point. That convinces me.

Quoted 21 lines
Quoted 21 lines

The advantage of letting all writes block is that even under full I/O load, the mirror job will progress at a steady pace.

Quoted 18 lines

I think for user there may be the following criteria:

1. guaranteed converge, with any guest write load. Both current and my proposed variants are OK.

2. Less impact on guest.
Obviously my proposed variant is better

3. Total time of mirroring
Seems, current may be a bit better, but I don't think that unaligned tails gives significant impact.

===

So, assume I want [1]+[2]. And possibly
2.2: Even less impact on guest: ignore not only unaligned tails if they are already dirty, but full synchronous mirror operation if area is already dirty.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

How should I call this? Should it be separate mode, or option for write-blocking?

I don’t know whether it makes sense to add a separate mode or a separate option just for this difference. I don’t think anyone would choose the non-default option.

At least Virtuozzo will choose :)

But if Virtuozzo knows exactly what to choose, then we can just implement only that behavior.

Max

#16 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 113 lines · Max Reitz
Quoted 68 lines · Vladimir Sementsov-Ogievskiy
Quoted 67 lines · Max Reitz
Quoted 55 lines · Vladimir Sementsov-Ogievskiy
Quoted 54 lines · Max Reitz
Quoted 52 lines · Vladimir Sementsov-Ogievskiy
Quoted 40 lines · Max Reitz
Quoted 35 lines · Vladimir Sementsov-Ogievskiy
Quoted 23 lines · Vladimir Sementsov-Ogievskiy
Quoted 22 lines · Max Reitz
Quoted 10 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

Quoted 6 lines

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

But that’s not what active mirror is for.  The point of active mirror is that it must converge because every guest write will contribute towards that goal.

Quoted 9 lines
Quoted 3 lines

If you skip active mirroring for unaligned guest writes, they will not contribute towards converging, but in fact lead to the opposite.

The will not contribute only if region is already dirty. Actually, after first iteration of mirroring (copying the whole disk), all following writes will contribute, so the whole process must converge. It is a bit similar with running one mirror loop in normal mode, and then enable write-blocking.

Quoted 3 lines

In other words, we don't need "all guest writes contribute" to converge, "all guest writes don't create new dirty bits" is enough, as we have parallel mirror iteration which contiguously handles dirty bits.

Quoted 7 lines

Hm, in a sense. But it does mean that guest writes will not contribute to convergence.

And that’s against the current definition of write-blocking, which does state that “when data is written to the source, write it (synchronously) to the target as well”.

Hmm, understand. But IMHO our proposed behavior is better in general. Do you think it's a problem to change spec now?
If yes, I'll resend with an option

Quoted 7 lines
Quoted 4 lines

Well, the thing is that I’d find it weird if write-blocking wasn’t blocking in all cases. And in my opinion, it makes more sense for active mirror if all writes actively contributed to convergence.

Why? What is the benefit in it?
What is "all writes actively contributed to convergence" for user?

One thing I wonder about is whether it’s really guaranteed that the background job will run under full I/O load, and how often it runs.

Quoted 6 lines
Quoted 2 lines

I fear that with your model, the background job might starve and the mirror may take a very long time.

Hmmm. I think mirror job is in same context as guest writes, and goes through same IO api.. Why it will starve? (I understand that my words are not an evidence...).

I thought that maybe if the disk is read to write and write all the time, there’d be no time for the mirror coroutine.

Quoted 8 lines
Quoted 2 lines

It won’t diverge, but it also won’t
really converge.

But same will be with current behavior: guest is not guaranteed to write to all parts of disk. And in most scenarios it doesn't. So, if mirror job starve because of huge guest IO load, we will not converge anyway.

So, background process is necessary thing for converge anyway.

Good point. That convinces me.

Quoted 23 lines
Quoted 23 lines

The advantage of letting all writes block is that even under full I/O load, the mirror job will progress at a steady pace.

Quoted 19 lines

I think for user there may be the following criteria:

1. guaranteed converge, with any guest write load. Both current and my proposed variants are OK.

2. Less impact on guest.
Obviously my proposed variant is better

3. Total time of mirroring
Seems, current may be a bit better, but I don't think that unaligned tails gives significant impact.

===

So, assume I want [1]+[2]. And possibly
2.2: Even less impact on guest: ignore not only unaligned tails if they are already dirty, but full synchronous mirror operation if area is already dirty.

How should I call this? Should it be separate mode, or option for write-blocking?

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

I don’t know whether it makes sense to add a separate mode or a separate option just for this difference. I don’t think anyone would choose the non-default option.

At least Virtuozzo will choose :)

But if Virtuozzo knows exactly what to choose, then we can just implement only that behavior.

Then, welcome to these series)

Still [2.2] is arguable: if we skip some dirty areas on synchronous copy, guest will be a bit happier, but we miss a chance to skip read part of copying for the mirror part. But this may be postponed.

Signature · 2 lines
Best regards,
Vladimir
#17 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

In reply to Vladimir Sementsov-Ogievskiy
Quoted 70 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

If unaligned padding is not dirty, we just write it, no reason to touch dirty bitmap if we succeed (on failure we'll set the whole region ofcourse, but we loss "synchronicity" on failure anyway).

Note: we need to disable dirty_bitmap, otherwise we will not be able to see in do_sync_target_write bitmap state before current operation. We may of course check dirty bitmap before the operation in bdrv_mirror_top_do_write and remember it, but we don't need active dirty bitmap for write-blocking mirror anyway.

New code-path is unused until the following commit reverts 9adc1cb49af8d.

---
 block/mirror.c | 39 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)
block/mirror.c +26 -0
index d176bf5920..d192f6a96b 100644
@@ -1204,6 +1204,39 @@ do_sync_target_write(MirrorBlockJob *job, MirrorMethod method,
                      QEMUIOVector *qiov, int flags)
 {
     int ret;
+    size_t qiov_offset = 0;
+
+    if (!QEMU_IS_ALIGNED(offset, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset)) {
+            /*
+             * Dirty unaligned padding
+             * 1. It's already dirty, no damage to "actively_synced" if we just
+             *    skip unaligned part.
+             * 2. If we copy it, we can't reset corresponding bit in
+             *    dirty_bitmap as there may be some "dirty" bytes still not
+             *    copied.
+             * So, just ignore it.
+             */
+            qiov_offset = QEMU_ALIGN_UP(offset, job->granularity) - offset;
+            if (bytes <= qiov_offset) {
+                /* nothing to do after shrink */
+                return;
+            }
+            offset += qiov_offset;
+            bytes -= qiov_offset;
+    }
+
+    if (!QEMU_IS_ALIGNED(offset + bytes, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset + bytes - 1))
+    {
+        uint64_t tail = (offset + bytes) % job->granularity;

block/mirror.c do_sync_target_write(MirrorBlockJob *job, MirrorMethod method, · line 1242

Quoted 10 lines
block/mirror.c +7 -0
+
+        if (bytes <= tail) {
+            /* nothing to do after shrink */
+            return;
+        }
+        bytes -= tail;
+    }
 
     bdrv_reset_dirty_bitmap(job->dirty_bitmap, offset, bytes);
 

The bdrv_set_dirty_bitmap() in the error case below needs to use the original offset/bytes, I suppose.

Apart from that, looks good to me.

Max

#18 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 79 lines · Max Reitz
Quoted 78 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

If unaligned padding is not dirty, we just write it, no reason to touch dirty bitmap if we succeed (on failure we'll set the whole region ofcourse, but we loss "synchronicity" on failure anyway).

Note: we need to disable dirty_bitmap, otherwise we will not be able to see in do_sync_target_write bitmap state before current operation. We may of course check dirty bitmap before the operation in bdrv_mirror_top_do_write and remember it, but we don't need active dirty bitmap for write-blocking mirror anyway.

New code-path is unused until the following commit reverts 9adc1cb49af8d.

---
  block/mirror.c | 39 ++++++++++++++++++++++++++++++++++++++-
  1 file changed, 38 insertions(+), 1 deletion(-)
block/mirror.c +33 -0
index d176bf5920..d192f6a96b 100644
@@ -1204,6 +1204,39 @@ do_sync_target_write(MirrorBlockJob *job, MirrorMethod method,
                       QEMUIOVector *qiov, int flags)
  {
      int ret;
+    size_t qiov_offset = 0;
+
+    if (!QEMU_IS_ALIGNED(offset, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset)) {
+            /*
+             * Dirty unaligned padding
+             * 1. It's already dirty, no damage to "actively_synced" if we just
+             *    skip unaligned part.
+             * 2. If we copy it, we can't reset corresponding bit in
+             *    dirty_bitmap as there may be some "dirty" bytes still not
+             *    copied.
+             * So, just ignore it.
+             */
+            qiov_offset = QEMU_ALIGN_UP(offset, job->granularity) - offset;
+            if (bytes <= qiov_offset) {
+                /* nothing to do after shrink */
+                return;
+            }
+            offset += qiov_offset;
+            bytes -= qiov_offset;
+    }
+
+    if (!QEMU_IS_ALIGNED(offset + bytes, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset + bytes - 1))
+    {
+        uint64_t tail = (offset + bytes) % job->granularity;
+
+        if (bytes <= tail) {
+            /* nothing to do after shrink */
+            return;
+        }
+        bytes -= tail;
+    }
 
Quoted 5 lines

block/mirror.c do_sync_target_write(MirrorBlockJob *job, MirrorMethod method, · line 1242

Quoted 2 lines
block/mirror.c +0 -0
      bdrv_reset_dirty_bitmap(job->dirty_bitmap, offset, bytes);
 

The bdrv_set_dirty_bitmap() in the error case below needs to use the original offset/bytes, I suppose.

No, because we shrink tail only if it is already dirty. And we've locked the region for in-flight operation, so nobody can clear the bitmap in a meantime.

But still, here is something to do:

for not-shrinked tails, if any, we should:
1. align down for reset
2. align up for set on failure

Quoted 5 lines

Apart from that, looks good to me.

Max

Signature · 2 lines
Best regards,
Vladimir
#19 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 80 lines · Vladimir Sementsov-Ogievskiy
Quoted 79 lines · Max Reitz
Quoted 78 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

If unaligned padding is not dirty, we just write it, no reason to touch dirty bitmap if we succeed (on failure we'll set the whole region ofcourse, but we loss "synchronicity" on failure anyway).

Note: we need to disable dirty_bitmap, otherwise we will not be able to see in do_sync_target_write bitmap state before current operation. We may of course check dirty bitmap before the operation in bdrv_mirror_top_do_write and remember it, but we don't need active dirty bitmap for write-blocking mirror anyway.

New code-path is unused until the following commit reverts 9adc1cb49af8d.

---
  block/mirror.c | 39 ++++++++++++++++++++++++++++++++++++++-
  1 file changed, 38 insertions(+), 1 deletion(-)
block/mirror.c +33 -0
index d176bf5920..d192f6a96b 100644
@@ -1204,6 +1204,39 @@ do_sync_target_write(MirrorBlockJob *job, MirrorMethod method,
                       QEMUIOVector *qiov, int flags)
  {
      int ret;
+    size_t qiov_offset = 0;
+
+    if (!QEMU_IS_ALIGNED(offset, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset)) {
+            /*
+             * Dirty unaligned padding
+             * 1. It's already dirty, no damage to "actively_synced" if we just
+             *    skip unaligned part.
+             * 2. If we copy it, we can't reset corresponding bit in
+             *    dirty_bitmap as there may be some "dirty" bytes still not
+             *    copied.
+             * So, just ignore it.
+             */
+            qiov_offset = QEMU_ALIGN_UP(offset, job->granularity) - offset;
+            if (bytes <= qiov_offset) {
+                /* nothing to do after shrink */
+                return;
+            }
+            offset += qiov_offset;
+            bytes -= qiov_offset;
+    }
+
+    if (!QEMU_IS_ALIGNED(offset + bytes, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset + bytes - 1))
+    {
+        uint64_t tail = (offset + bytes) % job->granularity;
+
+        if (bytes <= tail) {
+            /* nothing to do after shrink */
+            return;
+        }
+        bytes -= tail;
+    }
 
Quoted 8 lines
Quoted 5 lines

block/mirror.c do_sync_target_write(MirrorBlockJob *job, MirrorMethod method, · line 1242

Quoted 2 lines
block/mirror.c +0 -0
      bdrv_reset_dirty_bitmap(job->dirty_bitmap, offset, bytes);
 

The bdrv_set_dirty_bitmap() in the error case below needs to use the original offset/bytes, I suppose.

No, because we shrink tail only if it is already dirty. And we've locked the region for in-flight operation, so nobody can clear the bitmap in a meantime.

True. But wouldn’t it be simpler to understand to just use the original offsets?

Quoted 5 lines

But still, here is something to do:

for not-shrinked tails, if any, we should:
1. align down for reset
2. align up for set on failure

Well, the align up is done automatically, and I think that’s pretty self-explanatory.

Max

Quoted 7 lines
Quoted 5 lines

Apart from that, looks good to me.

Max

#20 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

Quoted 93 lines · Max Reitz
Quoted 80 lines · Vladimir Sementsov-Ogievskiy
Quoted 79 lines · Max Reitz
Quoted 78 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

If unaligned padding is not dirty, we just write it, no reason to touch dirty bitmap if we succeed (on failure we'll set the whole region ofcourse, but we loss "synchronicity" on failure anyway).

Note: we need to disable dirty_bitmap, otherwise we will not be able to see in do_sync_target_write bitmap state before current operation. We may of course check dirty bitmap before the operation in bdrv_mirror_top_do_write and remember it, but we don't need active dirty bitmap for write-blocking mirror anyway.

New code-path is unused until the following commit reverts 9adc1cb49af8d.

---
   block/mirror.c | 39 ++++++++++++++++++++++++++++++++++++++-
   1 file changed, 38 insertions(+), 1 deletion(-)
block/mirror.c +33 -0
index d176bf5920..d192f6a96b 100644
@@ -1204,6 +1204,39 @@ do_sync_target_write(MirrorBlockJob *job, MirrorMethod method,
                        QEMUIOVector *qiov, int flags)
   {
       int ret;
+    size_t qiov_offset = 0;
+
+    if (!QEMU_IS_ALIGNED(offset, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset)) {
+            /*
+             * Dirty unaligned padding
+             * 1. It's already dirty, no damage to "actively_synced" if we just
+             *    skip unaligned part.
+             * 2. If we copy it, we can't reset corresponding bit in
+             *    dirty_bitmap as there may be some "dirty" bytes still not
+             *    copied.
+             * So, just ignore it.
+             */
+            qiov_offset = QEMU_ALIGN_UP(offset, job->granularity) - offset;
+            if (bytes <= qiov_offset) {
+                /* nothing to do after shrink */
+                return;
+            }
+            offset += qiov_offset;
+            bytes -= qiov_offset;
+    }
+
+    if (!QEMU_IS_ALIGNED(offset + bytes, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset + bytes - 1))
+    {
+        uint64_t tail = (offset + bytes) % job->granularity;
+
+        if (bytes <= tail) {
+            /* nothing to do after shrink */
+            return;
+        }
+        bytes -= tail;
+    }
 
Quoted 8 lines
Quoted 5 lines

block/mirror.c do_sync_target_write(MirrorBlockJob *job, MirrorMethod method, · line 1242

Quoted 2 lines
block/mirror.c +0 -0
       bdrv_reset_dirty_bitmap(job->dirty_bitmap, offset, bytes);
 

The bdrv_set_dirty_bitmap() in the error case below needs to use the original offset/bytes, I suppose.

No, because we shrink tail only if it is already dirty. And we've locked the region for in-flight operation, so nobody can clear the bitmap in a meantime.

True. But wouldn’t it be simpler to understand to just use the original offsets?

Quoted 8 lines
Quoted 5 lines

But still, here is something to do:

for not-shrinked tails, if any, we should:
1. align down for reset
2. align up for set on failure

Well, the align up is done automatically, and I think that’s pretty self-explanatory.

Patch to make hbitmap_reset very strict (assert on analigned except for the very end of the bitmap, if orig_size is unaligned) is queued by John. So, I've made explicit alignment in v2.

Quoted 10 lines
Quoted 8 lines
Quoted 6 lines

Apart from that, looks good to me.

Max

Signature · 2 lines
Best regards,
Vladimir
#21 lore raw

Re: [PATCH 3/4] block/mirror: support unaligned write in active mirror

In reply to Max Reitz
Quoted 87 lines · Max Reitz
Quoted 86 lines · Vladimir Sementsov-Ogievskiy
Quoted 84 lines · Max Reitz
Quoted 70 lines · Vladimir Sementsov-Ogievskiy

Prior 9adc1cb49af8d do_sync_target_write had a bug: it reset aligned-up region in the dirty bitmap, which means that we may not copy some bytes and assume them copied, which actually leads to producing corrupted target.

So 9adc1cb49af8d forced dirty bitmap granularity to be request_alignment for mirror-top filter, so we are not working with unaligned requests. However forcing large alignment obviously decreases performance of unaligned requests.

This commit provides another solution for the problem: if unaligned padding is already dirty, we can safely ignore it, as
1. It's dirty, it will be copied by mirror_iteration anyway
2. It's dirty, so skipping it now we don't increase dirtiness of the bitmap and therefore don't damage "synchronicity" of the write-blocking mirror.

If unaligned padding is not dirty, we just write it, no reason to touch dirty bitmap if we succeed (on failure we'll set the whole region ofcourse, but we loss "synchronicity" on failure anyway).

Note: we need to disable dirty_bitmap, otherwise we will not be able to see in do_sync_target_write bitmap state before current operation. We may of course check dirty bitmap before the operation in bdrv_mirror_top_do_write and remember it, but we don't need active dirty bitmap for write-blocking mirror anyway.

New code-path is unused until the following commit reverts 9adc1cb49af8d.

---
   block/mirror.c | 39 ++++++++++++++++++++++++++++++++++++++-
   1 file changed, 38 insertions(+), 1 deletion(-)
block/mirror.c +26 -0
index d176bf5920..d192f6a96b 100644
@@ -1204,6 +1204,39 @@ do_sync_target_write(MirrorBlockJob *job, MirrorMethod method,
                        QEMUIOVector *qiov, int flags)
   {
       int ret;
+    size_t qiov_offset = 0;
+
+    if (!QEMU_IS_ALIGNED(offset, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset)) {
+            /*
+             * Dirty unaligned padding
+             * 1. It's already dirty, no damage to "actively_synced" if we just
+             *    skip unaligned part.
+             * 2. If we copy it, we can't reset corresponding bit in
+             *    dirty_bitmap as there may be some "dirty" bytes still not
+             *    copied.
+             * So, just ignore it.
+             */
+            qiov_offset = QEMU_ALIGN_UP(offset, job->granularity) - offset;
+            if (bytes <= qiov_offset) {
+                /* nothing to do after shrink */
+                return;
+            }
+            offset += qiov_offset;
+            bytes -= qiov_offset;
+    }
+
+    if (!QEMU_IS_ALIGNED(offset + bytes, job->granularity) &&
+        bdrv_dirty_bitmap_get(job->dirty_bitmap, offset + bytes - 1))
+    {
+        uint64_t tail = (offset + bytes) % job->granularity;

block/mirror.c do_sync_target_write(MirrorBlockJob *job, MirrorMethod method, · line 1242

Quoted 10 lines
block/mirror.c +7 -0
+
+        if (bytes <= tail) {
+            /* nothing to do after shrink */
+            return;
+        }
+        bytes -= tail;
+    }
 
       bdrv_reset_dirty_bitmap(job->dirty_bitmap, offset, bytes);
 

The bdrv_set_dirty_bitmap() in the error case below needs to use the original offset/bytes, I suppose.

Quoted 5 lines
Quoted 2 lines

No, because we shrink tail only if it is already dirty. And we've locked the region for in-flight operation, so nobody can clear the bitmap in a meantime.

True. But wouldn’t it be simpler to understand to just use the original offsets?

then we need to store them.. And if we do it, I'd prefer to add an assertion that shrunk tails are still dirty instead.

Quoted 21 lines
Quoted 6 lines

But still, here is something to do:

for not-shrinked tails, if any, we should:
1. align down for reset
2. align up for set on failure

Well, the align up is done automatically, and I think that’s pretty self-explanatory.

Max

Quoted 7 lines
Quoted 5 lines

Apart from that, looks good to me.

Max

Signature · 2 lines
Best regards,
Vladimir
#22 lore raw

PATCH 4/4 Revert "mirror: Only mirror granularity-aligned chunks"

In reply to Vladimir Sementsov-Ogievskiy

This reverts commit 9adc1cb49af8d4e54f57980b1eed5c0a4b2dafa6.
"mirror: Only mirror granularity-aligned chunks"

Since previous commit unaligned chunks are supported by do_sync_target_write.

---
 block/mirror.c | 29 -----------------------------
 1 file changed, 29 deletions(-)
block/mirror.c +0 -29
index d192f6a96b..4626b91541 100644
@@ -1482,15 +1482,6 @@ static void bdrv_mirror_top_child_perm(BlockDriverState *bs, BdrvChild *c,
     *nshared = BLK_PERM_ALL;
 }
 
-static void bdrv_mirror_top_refresh_limits(BlockDriverState *bs, Error **errp)
-{
-    MirrorBDSOpaque *s = bs->opaque;
-
-    if (s && s->job && s->job->copy_mode == MIRROR_COPY_MODE_WRITE_BLOCKING) {
-        bs->bl.request_alignment = s->job->granularity;
-    }
-}
-
 /* Dummy node that provides consistent read to its users without requiring it
  * from its backing file and that allows writes on the backing file chain. */
 static BlockDriver bdrv_mirror_top = {
@@ -1503,7 +1494,6 @@ static BlockDriver bdrv_mirror_top = {
     .bdrv_co_block_status       = bdrv_co_block_status_from_backing,
     .bdrv_refresh_filename      = bdrv_mirror_top_refresh_filename,
     .bdrv_child_perm            = bdrv_mirror_top_child_perm,
-    .bdrv_refresh_limits        = bdrv_mirror_top_refresh_limits,
 };
 
 static BlockJob *mirror_start_job(
@@ -1651,25 +1641,6 @@ static BlockJob *mirror_start_job(
         s->should_complete = true;
     }
 
-    /*
-     * Must be called before we start tracking writes, but after
-     *
-     *     ((MirrorBlockJob *)
-     *         ((MirrorBDSOpaque *)
-     *             mirror_top_bs->opaque
-     *         )->job
-     *     )->copy_mode
-     *
-     * has the correct value.
-     * (We start tracking writes as of the following
-     * bdrv_create_dirty_bitmap() call.)
-     */
-    bdrv_refresh_limits(mirror_top_bs, &local_err);
-    if (local_err) {
-        error_propagate(errp, local_err);
-        goto fail;
-    }
-
     s->dirty_bitmap = bdrv_create_dirty_bitmap(bs, granularity, NULL, errp);
     if (!s->dirty_bitmap) {
         goto fail;
Signature · 1 line
2.21.0
Reviewed-by on v1 4/4
#23 lore raw

Re: [PATCH 4/4] Revert "mirror: Only mirror granularity-aligned chunks"

Quoted 10 lines · Vladimir Sementsov-Ogievskiy

This reverts commit 9adc1cb49af8d4e54f57980b1eed5c0a4b2dafa6.
"mirror: Only mirror granularity-aligned chunks"

Since previous commit unaligned chunks are supported by do_sync_target_write.

---
 block/mirror.c | 29 -----------------------------
 1 file changed, 29 deletions(-)
#24 lore raw
In reply to Vladimir Sementsov-Ogievskiy

ping

Quoted 19 lines · Vladimir Sementsov-Ogievskiy

Commit 9adc1cb49af8d fixed a bug about unaligned (to dirty bitmap granularity) guest writes (and discards) by simply requesting corresponding alignment on mirror-top filter. However forcing large alignment obviously decreases performance of unaligned requests.

So it's time for a new solution which is in 03. And 04 reverts 9adc1cb49af8d.

Vladimir Sementsov-Ogievskiy (4):
block/mirror: simplify do_sync_target_write block/block-backend: add blk_co_pwritev_part block/mirror: support unaligned write in active mirror Revert "mirror: Only mirror granularity-aligned chunks"

  include/sysemu/block-backend.h |   4 +
  block/block-backend.c          |  17 +++-
  block/mirror.c                 | 153 +++++++++++++--------------------
  3 files changed, 78 insertions(+), 96 deletions(-)
Signature · 2 lines
Best regards,
Vladimir