Skip to contentSkip to conversation

Fir a race between request queue cleanup and running queues

PATCH In review Bart Van Assche 2 participants 8 messages Apr 1, 2019 – Apr 3, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Fir a race between request queue cleanup and running queues

Hi Jens,

This patch series fixes a recently reported race between request queue cleanup and running request queues. The approach followed is not to run the queue anymore after a request queue has been marked "dead". Please consider this patch series for kernel v5.1.

Thanks,

Bart.

Bart Van Assche (4):
block: Move the percpu_ref_exit(&q->q_usage_counter) call into

    __blk_release_queue()

block: Fix a race between request queue freezing and running queues
block: Fix a comment in blk_cleanup_queue()
block: Make blk_cleanup_queue() faster

 block/blk-core.c  | 17 ++---------------
 block/blk-mq.c    |  8 ++++++++
 block/blk-sysfs.c |  1 +
 3 files changed, 11 insertions(+), 15 deletions(-)
Signature · 1 line
2.21.0.196.g041f5ea1cf98
#1 lore raw

PATCH 1/4 block: Move the percpu_ref_exit(&q->q_usage_counter) call into __blk_release_queue()

This patch makes it safe to access the queue usage counter from inside blk_mq_run_hw_queues(). Other than deferring q_usage_counter freeing, this patch does not change any functionality.

Cc Christoph Hellwig <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc James Smart <[email protected]>
Cc Ming Lei <[email protected]>
Cc Jianchao Wang <[email protected]>
Cc Dongli Zhang <[email protected]>
---
 block/blk-core.c  | 2 --
 block/blk-sysfs.c | 1 +
 2 files changed, 1 insertion(+), 2 deletions(-)
block/blk-core.c +0 -2
index 4673ebe42255..fdcf6de2243f 100644
@@ -377,8 +377,6 @@ void blk_cleanup_queue(struct request_queue *q)
 	if (queue_is_mq(q))
 		blk_mq_free_queue(q);
 
-	percpu_ref_exit(&q->q_usage_counter);
-
 	/* @q is and will stay empty, shutdown and put */
 	blk_put_queue(q);
 }
block/blk-sysfs.c +1 -0
index 422327089e0f..b8b3d69cd0a5 100644
@@ -870,6 +870,7 @@ static void __blk_release_queue(struct work_struct *work)
 
 	bioset_exit(&q->bio_split);
 
+	percpu_ref_exit(&q->q_usage_counter);
 	ida_simple_remove(&blk_queue_ida, q->id);
 	call_rcu(&q->rcu_head, blk_free_queue_rcu);
 }
Signature · 1 line
2.21.0.196.g041f5ea1cf98
#2 lore raw

PATCH 2/4 block: Fix a race between request queue freezing and running queues

In reply to Bart Van Assche

Any request queue data structure may change while a queue is frozen. Hence make sure that blk_mq_run_hw_queues() does not access any hw queue while a request queue is frozen.

After blk_cleanup_queue() has marked a queue as dead it is no longer safe to access the hardware queue data structures. This patch avoids that blk_mq_run_hw_queues() crashes when called during or after blk_cleanup_queue() has freed the hardware queues. This patch is a variant of a patch posted by Hannes Reinecke ("[PATCH] block: don't call blk_mq_run_hw_queues() for dead or dying queues "). This patch is similar in nature to commit c246e80d8673 ("block: Avoid that request_fn is invoked on a dead queue"; v3.8). An example of a crash that is fixed by this patch:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff8135a10b>] sbitmap_any_bit_set+0xb/0x30
Call Trace:
 [<ffffffff81303a88>] blk_mq_run_hw_queues+0x48/0x90
 [<ffffffff813053cc>] blk_mq_requeue_work+0x10c/0x120
 [<ffffffff81098cb4>] process_one_work+0x154/0x410
 [<ffffffff81099896>] worker_thread+0x116/0x4a0
 [<ffffffff8109edb9>] kthread+0xc9/0xe0
 [<ffffffff81619b05>] ret_from_fork+0x55/0x80
Cc Christoph Hellwig <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc James Smart <[email protected]>
Cc Ming Lei <[email protected]>
Cc Jianchao Wang <[email protected]>
Cc Dongli Zhang <[email protected]>
Fixes a063057d7c73 ("block: Fix a race between request queue removal and the block cgroup controller") # v4.17.
---
 block/blk-mq.c | 8 ++++++++
 1 file changed, 8 insertions(+)
block/blk-mq.c +8 -0
index 3ff3d7b49969..652d0c6d5945 100644
@@ -1499,12 +1499,20 @@ void blk_mq_run_hw_queues(struct request_queue *q, bool async)
 	struct blk_mq_hw_ctx *hctx;
 	int i;
 
+	/*
+	 * Do not run any hardware queues if the queue is frozen or if a
+	 * concurrent blk_cleanup_queue() call is removing any data
+	 * structures used by this function.
+	 */
+	if (!percpu_ref_tryget(&q->q_usage_counter))
+		return;
 	queue_for_each_hw_ctx(q, hctx, i) {
 		if (blk_mq_hctx_stopped(hctx))
 			continue;
 
 		blk_mq_run_hw_queue(hctx, async);
 	}
+	percpu_ref_put(&q->q_usage_counter);
 }
 EXPORT_SYMBOL(blk_mq_run_hw_queues);
 
Signature · 1 line
2.21.0.196.g041f5ea1cf98
#3 lore raw

Re: [PATCH 2/4] block: Fix a race between request queue freezing and running queues

Quoted 52 lines · Bart Van Assche

Any request queue data structure may change while a queue is frozen. Hence make sure that blk_mq_run_hw_queues() does not access any hw queue while a request queue is frozen.

After blk_cleanup_queue() has marked a queue as dead it is no longer safe to access the hardware queue data structures. This patch avoids that blk_mq_run_hw_queues() crashes when called during or after blk_cleanup_queue() has freed the hardware queues. This patch is a variant of a patch posted by Hannes Reinecke ("[PATCH] block: don't call blk_mq_run_hw_queues() for dead or dying queues "). This patch is similar in nature to commit c246e80d8673 ("block: Avoid that request_fn is invoked on a dead queue"; v3.8). An example of a crash that is fixed by this patch:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff8135a10b>] sbitmap_any_bit_set+0xb/0x30
Call Trace:
 [<ffffffff81303a88>] blk_mq_run_hw_queues+0x48/0x90
 [<ffffffff813053cc>] blk_mq_requeue_work+0x10c/0x120
 [<ffffffff81098cb4>] process_one_work+0x154/0x410
 [<ffffffff81099896>] worker_thread+0x116/0x4a0
 [<ffffffff8109edb9>] kthread+0xc9/0xe0
 [<ffffffff81619b05>] ret_from_fork+0x55/0x80
Cc Christoph Hellwig <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc James Smart <[email protected]>
Cc Ming Lei <[email protected]>
Cc Jianchao Wang <[email protected]>
Cc Dongli Zhang <[email protected]>
Fixes a063057d7c73 ("block: Fix a race between request queue removal and the block cgroup controller") # v4.17.
---
 block/blk-mq.c | 8 ++++++++
 1 file changed, 8 insertions(+)
block/blk-mq.c +6 -0
index 3ff3d7b49969..652d0c6d5945 100644
@@ -1499,12 +1499,20 @@ void blk_mq_run_hw_queues(struct request_queue *q, bool async)
 	struct blk_mq_hw_ctx *hctx;
 	int i;
 
+	/*
+	 * Do not run any hardware queues if the queue is frozen or if a
+	 * concurrent blk_cleanup_queue() call is removing any data
+	 * structures used by this function.
+	 */
+	if (!percpu_ref_tryget(&q->q_usage_counter))

block/blk-mq.c void blk_mq_run_hw_queues(struct request_queue *q, bool async) · line 1517

Quoted 10 lines
block/blk-mq.c +2 -0
+		return;
 	queue_for_each_hw_ctx(q, hctx, i) {
 		if (blk_mq_hctx_stopped(hctx))
 			continue;
 
 		blk_mq_run_hw_queue(hctx, async);
 	}
+	percpu_ref_put(&q->q_usage_counter);
 }
 EXPORT_SYMBOL(blk_mq_run_hw_queues);

I don't see it is necessary to add percpu_ref_tryget()/percpu_ref_put() in the fast path if we simply release all hctx resource in hctx's release handler by the following patch:

https://lore.kernel.org/linux-block/[email protected]/T/#u

Even we can kill the percpu_ref_tryget_live()/percpu_ref_put() in scsi_end_request().

Thanks,
Ming

#4 lore raw

Re: [PATCH 2/4] block: Fix a race between request queue freezing and running queues

Quoted 19 lines · Ming Lei
Quoted 5 lines · Bart Van Assche
block/blk-mq.c +1 -1
index 3ff3d7b49969..652d0c6d5945 100644
--- a/block/blk-mq.c
+-+-+- b/block/blk-mq.c

@@-1499,12 +-1499,20 @@void blk_mq_run_hw_queues(struct request_queue *q, bool async)

Quoted 9 lines
	struct blk_mq_hw_ctx *hctx;>> 	int i;>> 
+-	/*>>+-	 *Do not run any hardware queues if the queue is frozen or if a
+-	 *concurrent blk_cleanup_queue() call is removing any data
+-	 *structures used by this function.
+-	 */
+-	if (!percpu_ref_tryget(&q->q_usage_counter))
+-		return;>> 	queue_for_each_hw_ctx(q, hctx, i) {>> 		if (blk_mq_hctx_stopped(hctx))
			continue;>> 
		blk_mq_run_hw_queue(hctx, async);>> 	}>>+-	percpu_ref_put(&q->q_usage_counter);>> }>> EXPORT_SYMBOL(blk_mq_run_hw_queues);>

I don't see it is necessary to add percpu_ref_tryget()/percpu_ref_put() in the fast path if we simply release all hctx resource in hctx's release handler by the following patch:

Quoted 4 lines

https://lore.kernel.org/linux-block/[email protected]/T/#u

Even we can kill the percpu_ref_tryget_live()/percpu_ref_put() in scsi_end_request().

The above approach has the advantages of being easy to review and to maintain.

Patch "[PATCH V2 1/3]blk-mq: free hw queue's resource in hctx's release handler"makes the block layer more complicated because it introduces a new state for hardware queues: block driver cleanup has happened (set->ops->exit_hctx(...)) but
the hardware queues are still in use by the block layer core.

Let's see what other reviewers think.

Bart.

#5 lore raw

Re: [PATCH 2/4] block: Fix a race between request queue freezing and running queues

Quoted 33 lines · Bart Van Assche
Quoted 32 lines · Ming Lei
Quoted 14 lines · Bart Van Assche
block/blk-mq.c +6 -0
index 3ff3d7b49969..652d0c6d5945 100644
@@ -1499,12 +1499,20 @@ void blk_mq_run_hw_queues(struct request_queue *q, bool async)
 	struct blk_mq_hw_ctx *hctx;
 	int i;
 
+	/*
+	 * Do not run any hardware queues if the queue is frozen or if a
+	 * concurrent blk_cleanup_queue() call is removing any data
+	 * structures used by this function.
+	 */
+	if (!percpu_ref_tryget(&q->q_usage_counter))

block/blk-mq.c void blk_mq_run_hw_queues(struct request_queue *q, bool async) · line 1517

Quoted 10 lines
block/blk-mq.c +2 -0
+		return;
 	queue_for_each_hw_ctx(q, hctx, i) {
 		if (blk_mq_hctx_stopped(hctx))
 			continue;
 
 		blk_mq_run_hw_queue(hctx, async);
 	}
+	percpu_ref_put(&q->q_usage_counter);
 }
 EXPORT_SYMBOL(blk_mq_run_hw_queues);

I don't see it is necessary to add percpu_ref_tryget()/percpu_ref_put() in the fast path if we simply release all hctx resource in hctx's release handler by the following patch:

https://lore.kernel.org/linux-block/[email protected]/T/#u

Quoted 8 lines
Quoted 2 lines

Even we can kill the percpu_ref_tryget_live()/percpu_ref_put() in scsi_end_request().

The above approach has the advantages of being easy to review and to maintain.

Patch "[PATCH V2 1/3] blk-mq: free hw queue's resource in hctx's release handler" makes the block layer more complicated because it introduces a new state for hardware queues: block driver cleanup has happened (set->ops->exit_hctx(...)) but

We are done with driver after blk_freeze_queue() and blk_sync_queue(), then call .exit_hctx() to say good bye with driver, I don't see it causes any issue.

Quoted 1 line

the hardware queues are still in use by the block layer core.

Block layer has the correct in-memory state to work well, and no driver activity is involved too.

Thanks,
Ming

#6 lore raw

PATCH 3/4 block: Fix a comment in blk_cleanup_queue()

In reply to Bart Van Assche

Change a reference to the legacy block layer into a reference to blk-mq.

Cc Christoph Hellwig <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc James Smart <[email protected]>
Cc Ming Lei <[email protected]>
Cc Jianchao Wang <[email protected]>
Cc Dongli Zhang <[email protected]>
---
 block/blk-core.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
block/blk-core.c +2 -1
index fdcf6de2243f..bfdbdbb8ec65 100644
@@ -339,7 +339,8 @@ void blk_cleanup_queue(struct request_queue *q)
 
 	/*
 	 * Drain all requests queued before DYING marking. Set DEAD flag to
-	 * prevent that q->request_fn() gets invoked after draining finished.
+	 * prevent that blk_mq_run_hw_queues() accesses the hardware queues
+	 * after draining finished.
 	 */
 	blk_freeze_queue(q);
 
Signature · 1 line
2.21.0.196.g041f5ea1cf98
#7 lore raw

PATCH 4/4 block: Make blk_cleanup_queue() faster

In reply to Bart Van Assche

Since blk_cleanup_queue() waits until q->q_usage_counter drops to zero before setting the "dead" flag, it is guaranteed that all requests that were in progress when blk_cleanup_queue() was called have finished before the "dead" flag is set. This means it is not possible that any .queue_rq() call is ongoing while the "dead" flag is set. Hence remove the code that waits for ongoing .queue_rq() calls to finish after the "dead" flag has been set. See also commit c2856ae2f315 ("blk-mq: quiesce queue before freeing queue") # v4.16.

Cc Christoph Hellwig <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc James Smart <[email protected]>
Cc Ming Lei <[email protected]>
Cc Jianchao Wang <[email protected]>
Cc Dongli Zhang <[email protected]>
---
 block/blk-core.c | 12 ------------
 1 file changed, 12 deletions(-)
block/blk-core.c +0 -12
index bfdbdbb8ec65..2921af6f8d33 100644
@@ -348,18 +348,6 @@ void blk_cleanup_queue(struct request_queue *q)
 
 	blk_queue_flag_set(QUEUE_FLAG_DEAD, q);
 
-	/*
-	 * make sure all in-progress dispatch are completed because
-	 * blk_freeze_queue() can only complete all requests, and
-	 * dispatch may still be in-progress since we dispatch requests
-	 * from more than one contexts.
-	 *
-	 * We rely on driver to deal with the race in case that queue
-	 * initialization isn't done.
-	 */
-	if (queue_is_mq(q) && blk_queue_init_done(q))
-		blk_mq_quiesce_queue(q);
-
 	/* for synchronous bio-based driver finish in-flight integrity i/o */
 	blk_flush_integrity();
 
Signature · 1 line
2.21.0.196.g041f5ea1cf98