Skip to contentSkip to conversation

mirror: Fix hang (operation waiting for itself/circular dependency)

PATCH Superseded Kevin Wolf 2 participants 6 messages Mar 25, 2020 Block layer

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
author
#1 lore raw

PATCH 1/2 Revert "mirror: Don't let an operation wait for itself"

In reply to Kevin Wolf

This reverts commit 7e6c4ff792734e196c8ca82564c56b5e7c6288ca.

The fix was incomplete as it only protected against requests waiting for themselves, but not against requests waiting for each other. We need a different solution.

---
 block/mirror.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
block/mirror.c +9 -12
index 447051dbc6..393131b135 100644
@@ -283,14 +283,11 @@ static int mirror_cow_align(MirrorBlockJob *s, int64_t *offset,
 }
 
 static inline void coroutine_fn
-mirror_wait_for_any_operation(MirrorBlockJob *s, MirrorOp *self, bool active)
+mirror_wait_for_any_operation(MirrorBlockJob *s, bool active)
 {
     MirrorOp *op;
 
     QTAILQ_FOREACH(op, &s->ops_in_flight, next) {
-        if (self == op) {
-            continue;
-        }
         /* Do not wait on pseudo ops, because it may in turn wait on
          * some other operation to start, which may in fact be the
          * caller of this function.  Since there is only one pseudo op
@@ -305,10 +302,10 @@ mirror_wait_for_any_operation(MirrorBlockJob *s, MirrorOp *self, bool active)
 }
 
 static inline void coroutine_fn
-mirror_wait_for_free_in_flight_slot(MirrorBlockJob *s, MirrorOp *self)
+mirror_wait_for_free_in_flight_slot(MirrorBlockJob *s)
 {
     /* Only non-active operations use up in-flight slots */
-    mirror_wait_for_any_operation(s, self, false);
+    mirror_wait_for_any_operation(s, false);
 }
 
 /* Perform a mirror copy operation.
@@ -351,7 +348,7 @@ static void coroutine_fn mirror_co_read(void *opaque)
 
     while (s->buf_free_count < nb_chunks) {
         trace_mirror_yield_in_flight(s, op->offset, s->in_flight);
-        mirror_wait_for_free_in_flight_slot(s, op);
+        mirror_wait_for_free_in_flight_slot(s);
     }
 
     /* Now make a QEMUIOVector taking enough granularity-sized chunks
@@ -558,7 +555,7 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)
 
         while (s->in_flight >= MAX_IN_FLIGHT) {
             trace_mirror_yield_in_flight(s, offset, s->in_flight);
-            mirror_wait_for_free_in_flight_slot(s, pseudo_op);
+            mirror_wait_for_free_in_flight_slot(s);
         }
 
         if (s->ret < 0) {
@@ -612,7 +609,7 @@ static void mirror_free_init(MirrorBlockJob *s)
 static void coroutine_fn mirror_wait_for_all_io(MirrorBlockJob *s)
 {
     while (s->in_flight > 0) {
-        mirror_wait_for_free_in_flight_slot(s, NULL);
+        mirror_wait_for_free_in_flight_slot(s);
     }
 }
 
@@ -809,7 +806,7 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
             if (s->in_flight >= MAX_IN_FLIGHT) {
                 trace_mirror_yield(s, UINT64_MAX, s->buf_free_count,
                                    s->in_flight);
-                mirror_wait_for_free_in_flight_slot(s, NULL);
+                mirror_wait_for_free_in_flight_slot(s);
                 continue;
             }
 
@@ -962,7 +959,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
         /* Do not start passive operations while there are active
          * writes in progress */
         while (s->in_active_write_counter) {
-            mirror_wait_for_any_operation(s, NULL, true);
+            mirror_wait_for_any_operation(s, true);
         }
 
         if (s->ret < 0) {
@@ -988,7 +985,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
             if (s->in_flight >= MAX_IN_FLIGHT || s->buf_free_count == 0 ||
                 (cnt == 0 && s->in_flight > 0)) {
                 trace_mirror_yield(s, cnt, s->buf_free_count, s->in_flight);
-                mirror_wait_for_free_in_flight_slot(s, NULL);
+                mirror_wait_for_free_in_flight_slot(s);
                 continue;
             } else if (cnt != 0) {
                 delay_ns = mirror_iteration(s);
Signature · 1 line
2.20.1

Kevin Wolf

author
#0 lore raw

cover mirror: Fix hang (operation waiting for itself/circular dependency)

The recent fix didn't actually fix the whole problem. Operations can't only wait for themselves, but we can also end up with circular dependencies like two operations waiting for each other to complete.

This reverts the first fix and implements another approach.

Kevin Wolf (2):
Revert "mirror: Don't let an operation wait for itself"
mirror: Wait only for in-flight operations

 block/mirror.c | 29 ++++++++++++++++-------------
 1 file changed, 16 insertions(+), 13 deletions(-)
Signature · 1 line
2.20.1
author
#3 lore raw

PATCH 2/2 mirror: Wait only for in-flight operations

mirror_wait_for_free_in_flight_slot() just picks a random operation to wait for. However, a MirrorOp is already in s->ops_in_flight when mirror_co_read() waits for free slots, so if not enough slots are immediately available, an operation can end up waiting for itself, or two or more operations can wait for each other to complete, which results in a hang.

Fix this by adding a flag to MirrorOp that tells us if the request is already in flight (and therefore occupies slots that it will later free), and picking only such operations for waiting.

---
 block/mirror.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
block/mirror.c +7 -1
index 393131b135..7fef52ded2 100644
@@ -102,6 +102,7 @@ struct MirrorOp {
 
     bool is_pseudo_op;
     bool is_active_write;
+    bool is_in_flight;
     CoQueue waiting_requests;
     Coroutine *co;
 
@@ -293,7 +294,9 @@ mirror_wait_for_any_operation(MirrorBlockJob *s, bool active)
          * caller of this function.  Since there is only one pseudo op
          * at any given time, we will always find some real operation
          * to wait on. */
-        if (!op->is_pseudo_op && op->is_active_write == active) {
+        if (!op->is_pseudo_op && op->is_in_flight &&
+            op->is_active_write == active)
+        {
             qemu_co_queue_wait(&op->waiting_requests, NULL);
             return;
         }
@@ -367,6 +370,7 @@ static void coroutine_fn mirror_co_read(void *opaque)
     /* Copy the dirty cluster.  */
     s->in_flight++;
     s->bytes_in_flight += op->bytes;
+    op->is_in_flight = true;
     trace_mirror_one_iteration(s, op->offset, op->bytes);
 
     ret = bdrv_co_preadv(s->mirror_top_bs->backing, op->offset, op->bytes,
@@ -382,6 +386,7 @@ static void coroutine_fn mirror_co_zero(void *opaque)
     op->s->in_flight++;
     op->s->bytes_in_flight += op->bytes;
     *op->bytes_handled = op->bytes;
+    op->is_in_flight = true;
 
     ret = blk_co_pwrite_zeroes(op->s->target, op->offset, op->bytes,
                                op->s->unmap ? BDRV_REQ_MAY_UNMAP : 0);
@@ -396,6 +401,7 @@ static void coroutine_fn mirror_co_discard(void *opaque)
     op->s->in_flight++;
     op->s->bytes_in_flight += op->bytes;
     *op->bytes_handled = op->bytes;
+    op->is_in_flight = true;
 
     ret = blk_co_pdiscard(op->s->target, op->offset, op->bytes);
     mirror_write_complete(op, ret);
Signature · 1 line
2.20.1
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] Revert "mirror: Don't let an operation wait for itself"

In reply to Kevin Wolf
Quoted 10 lines · Kevin Wolf

This reverts commit 7e6c4ff792734e196c8ca82564c56b5e7c6288ca.

The fix was incomplete as it only protected against requests waiting for themselves, but not against requests waiting for each other. We need a different solution.

---
  block/mirror.c | 21 +++++++++------------
  1 file changed, 9 insertions(+), 12 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v1 2/2
#4 lore raw

Re: [PATCH 2/2] mirror: Wait only for in-flight operations

In reply to Kevin Wolf
Quoted 30 lines · Kevin Wolf

mirror_wait_for_free_in_flight_slot() just picks a random operation to wait for. However, a MirrorOp is already in s->ops_in_flight when mirror_co_read() waits for free slots, so if not enough slots are immediately available, an operation can end up waiting for itself, or two or more operations can wait for each other to complete, which results in a hang.

Fix this by adding a flag to MirrorOp that tells us if the request is already in flight (and therefore occupies slots that it will later free), and picking only such operations for waiting.

---
  block/mirror.c | 8 +++++++-
  1 file changed, 7 insertions(+), 1 deletion(-)
block/mirror.c +1 -0
index 393131b135..7fef52ded2 100644
@@ -102,6 +102,7 @@ struct MirrorOp {
 
      bool is_pseudo_op;
      bool is_active_write;
+    bool is_in_flight;
      CoQueue waiting_requests;
      Coroutine *co;
 

block/mirror.c mirror_wait_for_any_operation(MirrorBlockJob *s, bool active) · line 300

Quoted 8 lines
block/mirror.c +3 -1
@@ -293,7 +294,9 @@ mirror_wait_for_any_operation(MirrorBlockJob *s, bool active)
           * caller of this function.  Since there is only one pseudo op
           * at any given time, we will always find some real operation
           * to wait on. */
-        if (!op->is_pseudo_op && op->is_active_write == active) {
+        if (!op->is_pseudo_op && op->is_in_flight &&
+            op->is_active_write == active)
+        {
              qemu_co_queue_wait(&op->waiting_requests, NULL);

Looks like a one-way transition - op->is_in_flight always starts as false, and only ever gets set to true (once the op is finished, op is no longer needed). And being more selective on what you wait for here does look like it should work in more cases than what patch 1 reverted.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#5 lore raw
In reply to Kevin Wolf

Am 25.03.2020 um 18:23 hat Kevin Wolf geschrieben:

Quoted 5 lines

The recent fix didn't actually fix the whole problem. Operations can't only wait for themselves, but we can also end up with circular dependencies like two operations waiting for each other to complete.

This reverts the first fix and implements another approach.

Hm, somehow this seems to break iotests 151. I don't actually understand the backtrace, because that's during job initialisation, so my changes shouldn't have had any effect yet:

(gdb) bt

#0  0x00007fba6d85057f in raise () at /lib64/libc.so.6
#1  0x00007fba6d83a895 in abort () at /lib64/libc.so.6
#2  0x00005624d94d109a in bitmap_new (nbits=<optimized out>) at /home/kwolf/source/qemu/include/qemu/bitmap.h:103
#3  0x00005624d94d109a in mirror_run (job=0x5624dc8d5560, errp=<optimized out>) at block/mirror.c:922
#4  0x00005624d988053f in job_co_entry (opaque=0x5624dc8d5560) at job.c:878
#5  0x00005624d998d3bb in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at util/coroutine-ucontext.c:115
#6  0x00007fba6d866250 in __start_context () at /lib64/libc.so.6
#7  0x00007fffa2d48130 in  ()
#8  0x0000000000000000 in  ()

Something to check tomorrow.

Kevin