Fir a race between request queue cleanup and running queues
Subsystems (2)
cover Fir a race between request queue cleanup and running queues
Hi Jens,
This patch series fixes a recently reported race between request queue cleanup and running request queues. The approach followed is not to run the queue anymore after a request queue has been marked "dead". Please consider this patch series for kernel v5.1.
Thanks,
Bart.
Bart Van Assche (4):
block: Move the percpu_ref_exit(&q->q_usage_counter) call into
__blk_release_queue()
block: Fix a race between request queue freezing and running queues
block: Fix a comment in blk_cleanup_queue()
block: Make blk_cleanup_queue() faster
block/blk-core.c | 17 ++--------------- block/blk-mq.c | 8 ++++++++ block/blk-sysfs.c | 1 + 3 files changed, 11 insertions(+), 15 deletions(-)
Signature · 1 line
2.21.0.196.g041f5ea1cf98
PATCH 1/4 block: Move the percpu_ref_exit(&q->q_usage_counter) call into __blk_release_queue()
This patch makes it safe to access the queue usage counter from inside blk_mq_run_hw_queues(). Other than deferring q_usage_counter freeing, this patch does not change any functionality.
--- block/blk-core.c | 2 -- block/blk-sysfs.c | 1 + 2 files changed, 1 insertion(+), 2 deletions(-)
block/blk-core.c +0 -2
@@ -377,8 +377,6 @@ void blk_cleanup_queue(struct request_queue *q)if (queue_is_mq (q))blk_mq_free_queue (q); -percpu_ref_exit (&q->q_usage_counter); -/* @q is and will stay empty, shutdown and put */ blk_put_queue (q); }
block/blk-sysfs.c +1 -0
@@ -870,6 +870,7 @@ static void __blk_release_queue(struct work_struct *work)bioset_exit (&q->bio_split); +percpu_ref_exit (&q->q_usage_counter);ida_simple_remove (&blk_queue_ida, q->id);call_rcu (&q->rcu_head, blk_free_queue_rcu); }
Signature · 1 line
2.21.0.196.g041f5ea1cf98
PATCH 2/4 block: Fix a race between request queue freezing and running queues
In reply to Bart Van AsscheAny request queue data structure may change while a queue is frozen. Hence make sure that blk_mq_run_hw_queues() does not access any hw queue while a request queue is frozen.
After blk_cleanup_queue() has marked a queue as dead it is no longer safe to access the hardware queue data structures. This patch avoids that blk_mq_run_hw_queues() crashes when called during or after blk_cleanup_queue() has freed the hardware queues. This patch is a variant of a patch posted by Hannes Reinecke ("[PATCH] block: don't call blk_mq_run_hw_queues() for dead or dying queues "). This patch is similar in nature to commit c246e80d8673 ("block: Avoid that request_fn is invoked on a dead queue"; v3.8). An example of a crash that is fixed by this patch:
BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff8135a10b>] sbitmap_any_bit_set+0xb/0x30 Call Trace: [<ffffffff81303a88>] blk_mq_run_hw_queues+0x48/0x90 [<ffffffff813053cc>] blk_mq_requeue_work+0x10c/0x120 [<ffffffff81098cb4>] process_one_work+0x154/0x410 [<ffffffff81099896>] worker_thread+0x116/0x4a0 [<ffffffff8109edb9>] kthread+0xc9/0xe0 [<ffffffff81619b05>] ret_from_fork+0x55/0x80
--- block/blk-mq.c | 8 ++++++++ 1 file changed, 8 insertions(+)
block/blk-mq.c +8 -0
@@ -1499,12 +1499,20 @@ void blk_mq_run_hw_queues(struct request_queue *q, bool async)struct blk_mq_hw_ctx *hctx;int i; +/* +* Do not run any hardware queues if the queue is frozen or if a +* concurrent blk_cleanup_queue() call is removing any data +* structures used by this function. +*/ +if (!percpu_ref_tryget (&q->q_usage_counter)) +return ;queue_for_each_hw_ctx (q, hctx, i) {if (blk_mq_hctx_stopped (hctx))continue ;blk_mq_run_hw_queue (hctx, async); } +percpu_ref_put (&q->q_usage_counter); }EXPORT_SYMBOL (blk_mq_run_hw_queues);
Signature · 1 line
2.21.0.196.g041f5ea1cf98
Re: [PATCH 2/4] block: Fix a race between request queue freezing and running queues
Quoted 52 lines · Bart Van Assche
Any request queue data structure may change while a queue is frozen. Hence make sure that
blk_mq_run_hw_queues()does not access any hw queue while a request queue is frozen.After
blk_cleanup_queue()has marked a queue as dead it is no longer safe to access the hardware queue data structures. This patch avoids thatblk_mq_run_hw_queues()crashes when called during or afterblk_cleanup_queue()has freed the hardware queues. This patch is a variant of a patch posted by Hannes Reinecke ("[PATCH] block: don't callblk_mq_run_hw_queues()for dead or dying queues "). This patch is similar in nature to commitc246e80d8673("block: Avoid thatrequest_fnis invoked on a dead queue"; v3.8). An example of a crash that is fixed by this patch:BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff8135a10b>] sbitmap_any_bit_set+0xb/0x30 Call Trace: [<ffffffff81303a88>] blk_mq_run_hw_queues+0x48/0x90 [<ffffffff813053cc>] blk_mq_requeue_work+0x10c/0x120 [<ffffffff81098cb4>] process_one_work+0x154/0x410 [<ffffffff81099896>] worker_thread+0x116/0x4a0 [<ffffffff8109edb9>] kthread+0xc9/0xe0 [<ffffffff81619b05>] ret_from_fork+0x55/0x80Cc Christoph Hellwig <[email protected]>Cc Hannes Reinecke <[email protected]>Cc James Smart <[email protected]>Cc Ming Lei <[email protected]>Cc Jianchao Wang <[email protected]>Cc Dongli Zhang <[email protected]>Cc <[email protected]>Fixes a063057d7c73 ("block: Fix a race between request queue removal and the block cgroup controller") # v4.17.Reported-by James Smart <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- block/blk-mq.c | 8 ++++++++ 1 file changed, 8 insertions(+)block/blk-mq.c +6
-0@@ -1499,12 +1499,20 @@ void blk_mq_run_hw_queues(struct request_queue *q, bool async)struct blk_mq_hw_ctx *hctx;int i; +/* +* Do not run any hardware queues if the queue is frozen or if a +* concurrent blk_cleanup_queue() call is removing any data +* structures used by this function. +*/ +if (!percpu_ref_tryget (&q->q_usage_counter))
block/blk-mq.c void blk_mq_run_hw_queues(struct request_queue *q, bool async) · line 1517
Quoted 10 lines
block/blk-mq.c +2
-0+return ;queue_for_each_hw_ctx (q, hctx, i) {if (blk_mq_hctx_stopped (hctx))continue ;blk_mq_run_hw_queue (hctx, async); } +percpu_ref_put (&q->q_usage_counter); }EXPORT_SYMBOL (blk_mq_run_hw_queues);
I don't see it is necessary to add percpu_ref_tryget()/percpu_ref_put() in the fast path if we simply release all hctx resource in hctx's release handler by the following patch:
https://lore.kernel.org/linux-block/[email protected]/T/#u
Even we can kill the percpu_ref_tryget_live()/percpu_ref_put() in scsi_end_request().
Thanks,
Ming
Re: [PATCH 2/4] block: Fix a race between request queue freezing and running queues
Quoted 19 lines · Ming Lei
Quoted 5 lines · Bart Van Assche
block/blk-mq.c +1
-1--- a/block/blk-mq.c +-+-+- b/block/blk-mq.c@@-1499,12 +-1499,20 @@void
blk_mq_run_hw_queues(struct request_queue *q, bool async)Quoted 9 lines
struct blk_mq_hw_ctx *hctx;>> int i;>> +- /*>>+- *Do not run any hardware queues if the queue is frozen or if a +- *concurrent blk_cleanup_queue() call is removing any data +- *structures used by this function. +- */ +- if (!percpu_ref_tryget(&q->q_usage_counter)) +- return;>> queue_for_each_hw_ctx(q, hctx, i) {>> if (blk_mq_hctx_stopped(hctx)) continue;>> blk_mq_run_hw_queue(hctx, async);>> }>>+- percpu_ref_put(&q->q_usage_counter);>> }>> EXPORT_SYMBOL(blk_mq_run_hw_queues);>I don't see it is necessary to add percpu_ref_tryget()/percpu_ref_put() in the fast path if we simply release all hctx resource in hctx's release handler by the following patch:
Quoted 4 lines
https://lore.kernel.org/linux-block/[email protected]/T/#u
Even we can kill the percpu_ref_tryget_live()/percpu_ref_put() in
scsi_end_request().
The above approach has the advantages of being easy to review and to maintain.
Patch "[PATCH V2 1/3]blk-mq: free hw queue's resource in hctx's release handler"makes the block layer more complicated because it introduces a new state for hardware queues: block driver cleanup has happened (set->ops->exit_hctx(...)) but
the hardware queues are still in use by the block layer core.
Let's see what other reviewers think.
Bart.
Re: [PATCH 2/4] block: Fix a race between request queue freezing and running queues
Quoted 33 lines · Bart Van Assche
Quoted 32 lines · Ming Lei
Quoted 14 lines · Bart Van Assche
block/blk-mq.c +6
-0@@ -1499,12 +1499,20 @@ void blk_mq_run_hw_queues(struct request_queue *q, bool async)struct blk_mq_hw_ctx *hctx;int i; +/* +* Do not run any hardware queues if the queue is frozen or if a +* concurrent blk_cleanup_queue() call is removing any data +* structures used by this function. +*/ +if (!percpu_ref_tryget (&q->q_usage_counter))block/blk-mq.c void blk_mq_run_hw_queues(struct request_queue *q, bool async) · line 1517
Quoted 10 lines
block/blk-mq.c +2
-0+return ;queue_for_each_hw_ctx (q, hctx, i) {if (blk_mq_hctx_stopped (hctx))continue ;blk_mq_run_hw_queue (hctx, async); } +percpu_ref_put (&q->q_usage_counter); }EXPORT_SYMBOL (blk_mq_run_hw_queues);I don't see it is necessary to add percpu_ref_tryget()/percpu_ref_put() in the fast path if we simply release all hctx resource in hctx's release handler by the following patch:
Quoted 8 lines
Quoted 2 lines
Even we can kill the percpu_ref_tryget_live()/percpu_ref_put() in
scsi_end_request().The above approach has the advantages of being easy to review and to maintain.
Patch "[PATCH V2 1/3] blk-mq: free hw queue's resource in hctx's release handler" makes the block layer more complicated because it introduces a new state for hardware queues: block driver cleanup has happened (
set->ops->exit_hctx(...)) but
We are done with driver after blk_freeze_queue() and blk_sync_queue(), then call .exit_hctx() to say good bye with driver, I don't see it causes any issue.
Quoted 1 line
the hardware queues are still in use by the block layer core.
Block layer has the correct in-memory state to work well, and no driver activity is involved too.
Thanks,
Ming
PATCH 3/4 block: Fix a comment in blk_cleanup_queue()
In reply to Bart Van AsscheChange a reference to the legacy block layer into a reference to blk-mq.
--- block/blk-core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
block/blk-core.c +2 -1
@@ -339,7 +339,8 @@ void blk_cleanup_queue(struct request_queue *q)/* * Drain all requests queued before DYING marking. Set DEAD flag to -* prevent that q->request_fn() gets invoked after draining finished. +* prevent that blk_mq_run_hw_queues() accesses the hardware queues +* after draining finished. */ blk_freeze_queue (q);
Signature · 1 line
2.21.0.196.g041f5ea1cf98
PATCH 4/4 block: Make blk_cleanup_queue() faster
In reply to Bart Van AsscheSince blk_cleanup_queue() waits until q->q_usage_counter drops to zero before setting the "dead" flag, it is guaranteed that all requests that were in progress when blk_cleanup_queue() was called have finished before the "dead" flag is set. This means it is not possible that any .queue_rq() call is ongoing while the "dead" flag is set. Hence remove the code that waits for ongoing .queue_rq() calls to finish after the "dead" flag has been set. See also commit c2856ae2f315 ("blk-mq: quiesce queue before freeing queue") # v4.16.
--- block/blk-core.c | 12 ------------ 1 file changed, 12 deletions(-)
block/blk-core.c +0 -12
@@ -348,18 +348,6 @@ void blk_cleanup_queue(struct request_queue *q)blk_queue_flag_set (QUEUE_FLAG_DEAD , q); -/* -* make sure all in-progress dispatch are completed because -* blk_freeze_queue() can only complete all requests, and -* dispatch may still be in-progress since we dispatch requests -* from more than one contexts. -* -* We rely on driver to deal with the race in case that queue -* initialization isn't done. -*/ -if (queue_is_mq (q) &&blk_queue_init_done (q)) -blk_mq_quiesce_queue (q); -/* for synchronous bio-based driver finish in-flight integrity i/o */ blk_flush_integrity ();
Signature · 1 line
2.21.0.196.g041f5ea1cf98