mirror: Fix hang (operation waiting for itself/circular dependency)
v2 cover mirror: Fix hang (operation waiting for itself/circular dependency)
The recent fix didn't actually fix the whole problem. Operations can't only wait for themselves, but we can also end up with circular dependencies like two operations waiting for each other to complete.
This reverts the first fix and implements another approach.
v2:
- Mark active mirror operations as in-flight, too
Kevin Wolf (2):
Revert "mirror: Don't let an operation wait for itself"
mirror: Wait only for in-flight operations
block/mirror.c | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-)
Signature · 1 line
2.20.1
PATCH v2 1/2 Revert "mirror: Don't let an operation wait for itself"
This reverts commit 7e6c4ff792734e196c8ca82564c56b5e7c6288ca.
The fix was incomplete as it only protected against requests waiting for themselves, but not against requests waiting for each other. We need a different solution.
--- block/mirror.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)
block/mirror.c +9 -12
@@ -283,14 +283,11 @@ static int mirror_cow_align(MirrorBlockJob *s, int64_t *offset, } static inline void coroutine_fn -mirror_wait_for_any_operation (MirrorBlockJob *s,MirrorOp *self,bool active) +mirror_wait_for_any_operation (MirrorBlockJob *s,bool active) {MirrorOp *op; QTAILQ_FOREACH(op, &s->ops_in_flight, next) { -if (self == op) { -continue ; - }/* Do not wait on pseudo ops, because it may in turn wait on * some other operation to start, which may in fact be the * caller of this function. Since there is only one pseudo op
@@ -305,10 +302,10 @@ mirror_wait_for_any_operation(MirrorBlockJob *s, MirrorOp *self, bool active) } static inline void coroutine_fn -mirror_wait_for_free_in_flight_slot (MirrorBlockJob *s,MirrorOp *self) +mirror_wait_for_free_in_flight_slot (MirrorBlockJob *s) {/* Only non-active operations use up in-flight slots */ -mirror_wait_for_any_operation (s, self, false); +mirror_wait_for_any_operation (s, false); }/* Perform a mirror copy operation.
@@ -351,7 +348,7 @@ static void coroutine_fn mirror_co_read(void *opaque)while (s->buf_free_count < nb_chunks) {trace_mirror_yield_in_flight (s, op->offset, s->in_flight); -mirror_wait_for_free_in_flight_slot (s, op); +mirror_wait_for_free_in_flight_slot (s); }/* Now make a QEMUIOVector taking enough granularity-sized chunks
@@ -558,7 +555,7 @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s)while (s->in_flight >=MAX_IN_FLIGHT ) {trace_mirror_yield_in_flight (s, offset, s->in_flight); -mirror_wait_for_free_in_flight_slot (s, pseudo_op); +mirror_wait_for_free_in_flight_slot (s); }if (s->ret <0 ) {
@@ -612,7 +609,7 @@ static void mirror_free_init(MirrorBlockJob *s) static void coroutine_fn mirror_wait_for_all_io(MirrorBlockJob *s) {while (s->in_flight >0 ) { -mirror_wait_for_free_in_flight_slot (s,NULL ); +mirror_wait_for_free_in_flight_slot (s); } }
@@ -809,7 +806,7 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)if (s->in_flight >=MAX_IN_FLIGHT ) {trace_mirror_yield (s,UINT64_MAX , s->buf_free_count, s->in_flight); -mirror_wait_for_free_in_flight_slot (s,NULL ); +mirror_wait_for_free_in_flight_slot (s);continue ; }
@@ -962,7 +959,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)/* Do not start passive operations while there are active * writes in progress */ while (s->in_active_write_counter) { -mirror_wait_for_any_operation (s,NULL , true); +mirror_wait_for_any_operation (s, true); }if (s->ret <0 ) {
@@ -988,7 +985,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)if (s->in_flight >=MAX_IN_FLIGHT || s->buf_free_count ==0 || (cnt ==0 && s->in_flight >0 )) {trace_mirror_yield (s, cnt, s->buf_free_count, s->in_flight); -mirror_wait_for_free_in_flight_slot (s,NULL ); +mirror_wait_for_free_in_flight_slot (s);continue ; }else if (cnt !=0 ) { delay_ns =mirror_iteration (s);
Signature · 1 line
2.20.1
PATCH v2 2/2 mirror: Wait only for in-flight operations
In reply to Kevin Wolfmirror_wait_for_free_in_flight_slot() just picks a random operation to wait for. However, a MirrorOp is already in s->ops_in_flight when mirror_co_read() waits for free slots, so if not enough slots are immediately available, an operation can end up waiting for itself, or two or more operations can wait for each other to complete, which results in a hang.
Fix this by adding a flag to MirrorOp that tells us if the request is already in flight (and therefore occupies slots that it will later free), and picking only such operations for waiting.
--- block/mirror.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
block/mirror.c +8 -1
@@ -102,6 +102,7 @@ struct MirrorOp {bool is_pseudo_op;bool is_active_write; +bool is_in_flight;CoQueue waiting_requests;Coroutine *co;
@@ -293,7 +294,9 @@ mirror_wait_for_any_operation(MirrorBlockJob *s, bool active)* caller of this function. Since there is only one pseudo op * at any given time, we will always find some real operation * to wait on. */ -if (!op->is_pseudo_op && op->is_active_write == active) { +if (!op->is_pseudo_op && op->is_in_flight && + op->is_active_write == active) + {qemu_co_queue_wait (&op->waiting_requests,NULL );return ; }
@@ -367,6 +370,7 @@ static void coroutine_fn mirror_co_read(void *opaque)/* Copy the dirty cluster. */ s->in_flight++; s->bytes_in_flight += op->bytes; + op->is_in_flight = true;trace_mirror_one_iteration (s, op->offset, op->bytes); ret = bdrv_co_preadv(s->mirror_top_bs->backing, op->offset, op->bytes,
@@ -382,6 +386,7 @@ static void coroutine_fn mirror_co_zero(void *opaque) op->s->in_flight++; op->s->bytes_in_flight += op->bytes; *op->bytes_handled = op->bytes; + op->is_in_flight = true; ret =blk_co_pwrite_zeroes (op->s->target, op->offset, op->bytes, op->s->unmap ?BDRV_REQ_MAY_UNMAP :0 );
@@ -396,6 +401,7 @@ static void coroutine_fn mirror_co_discard(void *opaque) op->s->in_flight++; op->s->bytes_in_flight += op->bytes; *op->bytes_handled = op->bytes; + op->is_in_flight = true; ret =blk_co_pdiscard (op->s->target, op->offset, op->bytes);mirror_write_complete (op, ret);
@@ -1318,6 +1324,7 @@ static MirrorOp *coroutine_fn active_write_prepare(MirrorBlockJob *s, .offset = offset, .bytes = bytes, .is_active_write = true, + .is_in_flight = true, };qemu_co_queue_init (&op->waiting_requests);QTAILQ_INSERT_TAIL (&s->ops_in_flight, op, next);
Signature · 1 line
2.20.1
Re: [PATCH v2 2/2] mirror: Wait only for in-flight operations
Quoted 21 lines · Kevin Wolf
mirror_wait_for_free_in_flight_slot()just picks a random operation to wait for. However, a MirrorOp is already ins->ops_in_flightwhenmirror_co_read()waits for free slots, so if not enough slots are immediately available, an operation can end up waiting for itself, or two or more operations can wait for each other to complete, which results in a hang.Fix this by adding a flag to MirrorOp that tells us if the request is already in flight (and therefore occupies slots that it will later free), and picking only such operations for waiting.
Signed-off-by Kevin Wolf <[email protected]>--- block/mirror.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)block/mirror.c +0
-0
[...]
block/mirror.c static MirrorOp *coroutine_fn active_write_prepare(MirrorBlockJob *s, · line 1330
Quoted 9 lines
block/mirror.c +1
-0@@ -1318,6 +1324,7 @@ static MirrorOp *coroutine_fn active_write_prepare(MirrorBlockJob *s, .offset = offset, .bytes = bytes, .is_active_write = true, + .is_in_flight = true, };qemu_co_queue_init (&op->waiting_requests);QTAILQ_INSERT_TAIL (&s->ops_in_flight, op, next);
There is a mirror_wait_on_conflicts() call after this. I was a bit worried about dependencies there. But I don’t think there’s any problem, because mirror_wait_for_any_operation() is only called by:
(1) mirror_wait_for_free_in_flight_slot(), which makes it look for non-active operations only, and
(2) mirror_run(), which specifically waits for all active operations to settle, so it makes sense to wait for all of them, even when they are still doing their own dependency-waiting.
But still, I’m not sure whether this is conceptually the best thing to do. I think what we actually want is for mirror_wait_for_free_in_flight_slot() to only wait for in-flight operations; but the call in mirror_run() that waits for active-mirror operations wants to wait for all active-mirror operations, not just the ones that are in flight.
So I think conceptually it would make more sense to set is_in_flight only after mirror_wait_on_conflicts(), and ensure that the mirror_wait_for_any_operation() call from mirror_run() ignores is_in_flight. E.g. by having another parameter “bool in_flight” for mirror_wait_for_any_operation() that chooses whether to check is_in_flight or whether to ignore it.
In practice, @in_flight would always be the same as @active, but they are different things. But that would mean we would always ignore is_in_flight for active-mirror operations.
In practice, there’s no difference to what this patch does, i.e. to just let active-mirror operations have is_in_flight to be always true and let mirror_wait_for_any_operation() check is_in_flight unconditionally.
So I don’t know. Maybe this is a start:
Max
Quoted 4 lines · Kevin Wolf
The recent fix didn't actually fix the whole problem. Operations can't only wait for themselves, but we can also end up with circular dependencies like two operations waiting for each other to complete.
Quoted 8 lines
This reverts the first fix and implements another approach.
v2:
- Mark active mirror operations as in-flight, tooKevin Wolf (2):
Revert "mirror: Don't let an operation wait for itself"
mirror: Wait only for in-flight operations
Series:
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org