blk-mq: allow to run queue if queue refcount is held
Subsystems (3)
v2 cover blk-mq: allow to run queue if queue refcount is held
Hi,
Since 45a9c9d909b2 ("blk-mq: Fix a use-after-free"), run queue isn't allowed during cleanup queue even though queue refcount is held.
This change has caused lots of kernel oops triggered in run queue path, turns out it isn't easy to fix them all.
So move freeing of hw queue resources into hctx's release handler, then the above issue is fixed. Meantime, this way is safe given freeing hw queue resource doesn't require tags.
V2:
- moving freeing hw queue resources into hctx's release handler
Ming Lei (3):
blk-mq: free hw queue's resource in hctx's release handler
block: don't drain in-progress dispatch in blk_cleanup_queue()
SCSI: don't grab queue usage counter before run queue
block/blk-core.c | 14 +------------- block/blk-mq-sysfs.c | 6 ++++++ block/blk-mq.c | 8 ++------ block/blk-mq.h | 2 +- drivers/scsi/scsi_lib.c | 7 ------- 5 files changed, 10 insertions(+), 27 deletions(-)
Signature · 1 line
2.9.5
PATCH v2 1/3 blk-mq: free hw queue's resource in hctx's release handler
Once blk_cleanup_queue() returns, tags shouldn't be used any more, because blk_mq_free_tag_set() may be called. Commit 45a9c9d909b2 ("blk-mq: Fix a use-after-free") fixes this issue exactly.
However, that commit introduces another issue. Before 45a9c9d909b2, we are allowed to run queue during cleaning up queue if the queue's kobj refcount is held. After that commit, queue can't be run during queue cleaning up, otherwise oops can be triggered easily because some fields of hctx are freed by blk_mq_free_queue() in blk_cleanup_queue().
We have invented ways for addressing this kind of issue before, such as:
8dc765d438f1 ("SCSI: fix queue cleanup race before queue initialization is done") c2856ae2f315 ("blk-mq: quiesce queue before freeing queue")
But still can't cover all cases, recently James reports another such kind of issue:
https://marc.info/?l=linux-scsi&m=155389088124782&w=2
This issue can be quite hard to address by previous way, given scsi_run_queue() may run requeues for other LUNs.
Fixes the above issue by freeing hctx's resources in its release handler, and this way is safe becasue tags isn't needed for freeing such hctx resource.
This approach follows typical design pattern wrt. kobject's release handler.
--- block/blk-core.c | 2 +- block/blk-mq-sysfs.c | 6 ++++++ block/blk-mq.c | 8 ++------ block/blk-mq.h | 2 +- 4 files changed, 10 insertions(+), 8 deletions(-)
block/blk-core.c +1 -1
@@ -375,7 +375,7 @@ void blk_cleanup_queue(struct request_queue *q)blk_exit_queue (q);if (queue_is_mq (q)) -blk_mq_free_queue (q); +blk_mq_exit_queue (q);percpu_ref_exit (&q->q_usage_counter);
block/blk-mq-sysfs.c +6 -0
@@ -10,6 +10,7 @@#include <linux/smp.h> #include <linux/blk-mq.h> +#include "blk.h" #include "blk-mq.h" #include "blk-mq-tag.h"
@@ -33,6 +34,11 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj) { struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx, kobj); + +if (hctx->flags &BLK_MQ_F_BLOCKING ) +cleanup_srcu_struct (hctx->srcu); +blk_free_flush_queue (hctx->fq); +sbitmap_free (&hctx->ctx_map);free_cpumask_var (hctx->cpumask);kfree (hctx->ctxs);kfree (hctx);
block/blk-mq.c +2 -6
@@ -2243,12 +2243,7 @@ static void blk_mq_exit_hctx(struct request_queue *q,if (set->ops->exit_hctx) set->ops->exit_hctx (hctx, hctx_idx); -if (hctx->flags &BLK_MQ_F_BLOCKING ) -cleanup_srcu_struct (hctx->srcu); -blk_mq_remove_cpuhp (hctx); -blk_free_flush_queue (hctx->fq); -sbitmap_free (&hctx->ctx_map); } static void blk_mq_exit_hw_queues(struct request_queue *q,
@@ -2881,7 +2876,8 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, }EXPORT_SYMBOL (blk_mq_init_allocated_queue); -void blk_mq_free_queue (struct request_queue *q) +/* tags can _not_ be used after returning from blk_mq_exit_queue */ +void blk_mq_exit_queue (struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set;
block/blk-mq.h +1 -1
@@ -37,7 +37,7 @@ struct blk_mq_ctx {struct kobject kobj; }____cacheline_aligned_in_smp ; -void blk_mq_free_queue (struct request_queue *q); +void blk_mq_exit_queue (struct request_queue *q);int blk_mq_update_nr_requests (struct request_queue *q,unsigned int nr);void blk_mq_wake_waiters (struct request_queue *q);bool blk_mq_dispatch_rq_list (struct request_queue *,struct list_head *,bool );
Signature · 1 line
2.9.5
PATCH v2 2/3 block: don't drain in-progress dispatch in blk_cleanup_queue()
In reply to Ming LeiNow freeing hw queue resource is moved to hctx's release handler, we don't need to worry about the race between blk_cleanup_queue and run queue any more.
So don't drain in-progress dispatch in blk_cleanup_queue().
This is basically revert of c2856ae2f315 ("blk-mq: quiesce queue before freeing queue").
--- block/blk-core.c | 12 ------------ 1 file changed, 12 deletions(-)
block/blk-core.c +0 -12
@@ -347,18 +347,6 @@ void blk_cleanup_queue(struct request_queue *q)blk_queue_flag_set (QUEUE_FLAG_DEAD , q); -/* -* make sure all in-progress dispatch are completed because -* blk_freeze_queue() can only complete all requests, and -* dispatch may still be in-progress since we dispatch requests -* from more than one contexts. -* -* We rely on driver to deal with the race in case that queue -* initialization isn't done. -*/ -if (queue_is_mq (q) &&blk_queue_init_done (q)) -blk_mq_quiesce_queue (q); -/* for synchronous bio-based driver finish in-flight integrity i/o */ blk_flush_integrity ();
Signature · 1 line
2.9.5
Re: [PATCH V2 2/3] block: don't drain in-progress dispatch in blk_cleanup_queue()
Quoted 24 lines · Ming Lei
Now freeing hw queue resource is moved to hctx's release handler, we don't need to worry about the race between
blk_cleanup_queueand run queue any more.So don't drain in-progress dispatch in
blk_cleanup_queue().This is basically revert of
c2856ae2f315("blk-mq: quiesce queue before freeing queue".Cc: Dongli Zhang <[email protected]>>Cc: James Smart <[email protected]>>Cc: Bart Van Assche <[email protected]>>Cc: [email protected],
Cc: Martin K . Petersen <[email protected]>
Cc: Christoph Hellwig <[email protected]>
Cc: James E . J . Bottomley <[email protected]>
Cc: jianchao wang <[email protected]>>Signed-off-by: Ming Lei <[email protected]>>---block/blk-core.c|12 ------------1 file changed, 12 deletions(-)block/blk-core.c
Quoted 1 line
block/blk-core.c +0
-0index
b3bbf8a5110d..491dc0295778100644
---a/block/blk-core.c
+-+-+-b/block/blk-core.c
@@-347,18 +-347,6 @@voidblk_cleanup_queue(struct request_queue *q)
blk_queue_flag_set(QUEUE_FLAG_DEAD, q);>(quoted diff) +0
-1- /*>- *make sure all in-progress dispatch are completed because
Review
Quoted 10 lines
(quoted diff) +0
-8- *blk_freeze_queue() can only complete all requests, and - *dispatch may still be in-progress since we dispatch requests - *from more than one contexts. - *>- *We rely on driver to deal with the race in case that queue - *initialization isn't done. - */ - if (queue_is_mq(q) &&blk_queue_init_done(q)) - blk_mq_quiesce_queue(q);>- /*for synchronous bio-based driver finish in-flight integrity i/o */ blk_flush_integrity();
Many block drivers clean up resources immediately after blk_cleanup_queue() returns. Not waiting for ongoing .queue_rq() calls to finish is wrong because it can cause block drivers to destroy resources that are in use by a concurrent .queue_rq() call.
Bart.
Re: [PATCH V2 2/3] block: don't drain in-progress dispatch in blk_cleanup_queue()
Quoted 42 lines · Bart Van Assche
Quoted 41 lines · Ming Lei
Now freeing hw queue resource is moved to hctx's release handler, we don't need to worry about the race between
blk_cleanup_queueand run queue any more.So don't drain in-progress dispatch in
blk_cleanup_queue().This is basically revert of
c2856ae2f315("blk-mq: quiesce queue before freeing queue").Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-core.c | 12 ------------ 1 file changed, 12 deletions(-)block/blk-core.c +0
-11@@ -347,18 +347,6 @@ void blk_cleanup_queue(struct request_queue *q)blk_queue_flag_set (QUEUE_FLAG_DEAD , q); -/* -* make sure all in-progress dispatch are completed because -* blk_freeze_queue() can only complete all requests, and -* dispatch may still be in-progress since we dispatch requests -* from more than one contexts. -* -* We rely on driver to deal with the race in case that queue -* initialization isn't done. -*/ -if (queue_is_mq (q) &&blk_queue_init_done (q)) -blk_mq_quiesce_queue (q);
Quoted 8 lines
block/blk-core.c void blk_cleanup_queue(struct request_queue *q) · line 351
Quoted 3 lines
block/blk-core.c +0
-1-/* for synchronous bio-based driver finish in-flight integrity i/o */ blk_flush_integrity ();Many block drivers clean up resources immediately after
blk_cleanup_queue()returns. Not waiting for ongoing .queue_rq() calls to finish is wrong because it can cause block drivers to destroy resources that are in use by a concurrent .queue_rq() call.
blk_freeze_queue() has returned, so there can't be any in-flight IOs and .queue_rq(), but there might be run queue activities, which do not call into driver.
Thanks,
Ming
PATCH v2 3/3 SCSI: don't grab queue usage counter before run queue
In reply to Ming LeiNow freeing hw queue resource is moved to hctx's release handler, we don't need to worry about the possible race between blk_cleanup_queue and run queue.
So don't grab the queue usage counter before run queue in scsi_end_request().
This is basically revert of 8dc765d438f1 ("SCSI: fix queue cleanup race before queue initialization is done").
--- drivers/scsi/scsi_lib.c | 7 ------- 1 file changed, 7 deletions(-)
drivers/scsi/scsi_lib.c +0 -7
@@ -604,12 +604,6 @@ static bool scsi_end_request(struct request *req, blk_status_t error,*/ scsi_mq_uninit_cmd (cmd); -/* -* queue is still alive, so grab the ref for preventing it -* from being cleaned up during running queue. -*/ -percpu_ref_get (&q->q_usage_counter); -__blk_mq_end_request (req, error); if (scsi_target(sdev)->single_lun ||
@@ -618,7 +612,6 @@ static bool scsi_end_request(struct request *req, blk_status_t error,else blk_mq_run_hw_queues (q, true); -percpu_ref_put (&q->q_usage_counter);put_device (&sdev->sdev_gendev);return false; }
Signature · 1 line
2.9.5