Skip to contentSkip to conversation

mirror: Confirm we're quiesced only if the job is paused or cancelled

PATCH Superseded Sergio Lopez 2 participants 3 messages Mar 7, 2019 – Mar 8, 2019 Block layer

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 mirror: Confirm we're quiesced only if the job is paused or cancelled

While child_job_drained_begin() calls to job_pause(), the job doesn't actually transition between states until it runs again and reaches a pause point. This means bdrv_drained_begin() may return with some jobs using the node still having 'busy == true'.

As a consequence, block_job_detach_aio_context() may get into a deadlock, waiting for the job to be actually paused, while the coroutine servicing the job is yielding and doesn't get the opportunity to get scheduled again. This situation can be reproduced by issuing a 'block-commit' immediately followed by a 'device_del'.

To ensure bdrv_drained_begin() only returns when the jobs have been paused, we change mirror_drained_poll() to only confirm it's quiesced when job->paused == true and there aren't any in-flight requests, except if we reached that point by a drained section initiated by the mirror/commit job itself.

The other block jobs shouldn't need any changes, as the default drained_poll() behavior is to only confirm it's quiesced if the job is not busy or completed.

---
v2
- Fix typo (thanks to Eric Blake)

---
 block/mirror.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
block/mirror.c +17 -0
index 726d3c27fb..1a1fb174b6 100644
@@ -80,6 +80,7 @@ typedef struct MirrorBlockJob {
     bool initial_zeroing_ongoing;
     int in_active_write_counter;
     bool prepared;
+    bool in_drain;
 } MirrorBlockJob;
 
 typedef struct MirrorBDSOpaque {
@@ -679,9 +680,11 @@ static int mirror_exit_common(Job *job)
 
         /* The mirror job has no requests in flight any more, but we need to
          * drain potential other users of the BDS before changing the graph. */
+        s->in_drain = true;
         bdrv_drained_begin(target_bs);
         bdrv_replace_node(to_replace, target_bs, &local_err);
         bdrv_drained_end(target_bs);
+        s->in_drain = false;
         if (local_err) {
             error_report_err(local_err);
             ret = -EPERM;
@@ -717,6 +720,7 @@ static int mirror_exit_common(Job *job)
     bs_opaque->job = NULL;
 
     bdrv_drained_end(src);
+    s->in_drain = false;
     bdrv_unref(mirror_top_bs);
     bdrv_unref(src);
 
@@ -1000,10 +1004,12 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
              */
             trace_mirror_before_drain(s, cnt);
 
+            s->in_drain = true;
             bdrv_drained_begin(bs);
             cnt = bdrv_get_dirty_count(s->dirty_bitmap);
             if (cnt > 0 || mirror_flush(s) < 0) {
                 bdrv_drained_end(bs);
+                s->in_drain = false;
                 continue;
             }
 
@@ -1051,6 +1057,7 @@ immediate_exit:
     bdrv_dirty_iter_free(s->dbi);
 
     if (need_drain) {
+        s->in_drain = true;
         bdrv_drained_begin(bs);
     }
 
@@ -1119,6 +1126,16 @@ static void coroutine_fn mirror_pause(Job *job)
 static bool mirror_drained_poll(BlockJob *job)
 {
     MirrorBlockJob *s = container_of(job, MirrorBlockJob, common);
+
+    /* If the job isn't paused nor cancelled, we can't be sure that it won't
+     * issue more requests. We make an exception if we've reached this point
+     * from one of our own drain sections, to avoid a deadlock waiting for
+     * ourselves.
+     */
+    if (!s->common.job.paused && !s->common.job.cancelled && !s->in_drain) {
+        return true;
+    }
+
     return !!s->in_flight;
 }
 
Signature · 1 line
2.20.1
#1 lore raw

Am 07.03.2019 um 19:54 hat Sergio Lopez geschrieben:

Quoted 44 lines

While child_job_drained_begin() calls to job_pause(), the job doesn't actually transition between states until it runs again and reaches a pause point. This means bdrv_drained_begin() may return with some jobs using the node still having 'busy == true'.

As a consequence, block_job_detach_aio_context() may get into a deadlock, waiting for the job to be actually paused, while the coroutine servicing the job is yielding and doesn't get the opportunity to get scheduled again. This situation can be reproduced by issuing a 'block-commit' immediately followed by a 'device_del'.

To ensure bdrv_drained_begin() only returns when the jobs have been paused, we change mirror_drained_poll() to only confirm it's quiesced when job->paused == true and there aren't any in-flight requests, except if we reached that point by a drained section initiated by the mirror/commit job itself.

The other block jobs shouldn't need any changes, as the default drained_poll() behavior is to only confirm it's quiesced if the job is not busy or completed.

---
v2
- Fix typo (thanks to Eric Blake)

---
 block/mirror.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
block/mirror.c +1 -0
index 726d3c27fb..1a1fb174b6 100644
@@ -80,6 +80,7 @@ typedef struct MirrorBlockJob {
     bool initial_zeroing_ongoing;
     int in_active_write_counter;
     bool prepared;
+    bool in_drain;
 } MirrorBlockJob;
 
 typedef struct MirrorBDSOpaque {
@@ -679,9 +680,11 @@ static int mirror_exit_common(Job *job)
 

block/mirror.c static int mirror_exit_common(Job *job) · line 690

Quoted 10 lines
block/mirror.c +2 -0
         /* The mirror job has no requests in flight any more, but we need to
          * drain potential other users of the BDS before changing the graph. */
+        s->in_drain = true;
         bdrv_drained_begin(target_bs);
         bdrv_replace_node(to_replace, target_bs, &local_err);
         bdrv_drained_end(target_bs);
+        s->in_drain = false;
         if (local_err) {
             error_report_err(local_err);
             ret = -EPERM;

I think this hunk is wrong because this is nested: s->in_drain is already true before this block, so we're setting it to false too early. We can either drop is completely or just assert(s->in_drain).

Other than this, the patch looks good to me.

Kevin

#2 lore raw
Quoted 45 lines · Kevin Wolf

Am 07.03.2019 um 19:54 hat Sergio Lopez geschrieben:

Quoted 44 lines

While child_job_drained_begin() calls to job_pause(), the job doesn't actually transition between states until it runs again and reaches a pause point. This means bdrv_drained_begin() may return with some jobs using the node still having 'busy == true'.

As a consequence, block_job_detach_aio_context() may get into a deadlock, waiting for the job to be actually paused, while the coroutine servicing the job is yielding and doesn't get the opportunity to get scheduled again. This situation can be reproduced by issuing a 'block-commit' immediately followed by a 'device_del'.

To ensure bdrv_drained_begin() only returns when the jobs have been paused, we change mirror_drained_poll() to only confirm it's quiesced when job->paused == true and there aren't any in-flight requests, except if we reached that point by a drained section initiated by the mirror/commit job itself.

The other block jobs shouldn't need any changes, as the default drained_poll() behavior is to only confirm it's quiesced if the job is not busy or completed.

---
v2
- Fix typo (thanks to Eric Blake)

---
 block/mirror.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
block/mirror.c +1 -0
index 726d3c27fb..1a1fb174b6 100644
@@ -80,6 +80,7 @@ typedef struct MirrorBlockJob {
     bool initial_zeroing_ongoing;
     int in_active_write_counter;
     bool prepared;
+    bool in_drain;
 } MirrorBlockJob;
 
 typedef struct MirrorBDSOpaque {
@@ -679,9 +680,11 @@ static int mirror_exit_common(Job *job)
 
Quoted 14 lines

block/mirror.c static int mirror_exit_common(Job *job) · line 690

Quoted 10 lines
block/mirror.c +2 -0
         /* The mirror job has no requests in flight any more, but we need to
          * drain potential other users of the BDS before changing the graph. */
+        s->in_drain = true;
         bdrv_drained_begin(target_bs);
         bdrv_replace_node(to_replace, target_bs, &local_err);
         bdrv_drained_end(target_bs);
+        s->in_drain = false;
         if (local_err) {
             error_report_err(local_err);
             ret = -EPERM;

I think this hunk is wrong because this is nested: s->in_drain is already true before this block, so we're setting it to false too early. We can either drop is completely or just assert(s->in_drain).

You're right, I'll send a v3 with an assert instead of touching the value there.

Thanks!
Sergio (slp).