blk-mq: fix races related with freeing queue
Subsystems (3)
v4 cover blk-mq: fix races related with freeing queue
Hi,
Since 45a9c9d909b2 ("blk-mq: Fix a use-after-free"), run queue isn't allowed during cleanup queue even though queue refcount is held.
This change has caused lots of kernel oops triggered in run queue path, turns out it isn't easy to fix them all.
So move freeing of hw queue resources into hctx's release handler, then the above issue is fixed. Meantime, this way is safe given freeing hw queue resource doesn't require tags.
V3 covers more races.
V4:
- add patch for fixing potential use-after-free in blk_mq_update_nr_hw_queues
- fix comment in the last patch
V3:
- cancel q->requeue_work in queue's release handler
- cancel hctx->run_work in hctx's release handler
- add patch 1 for fixing race in plug code path
- the last patch is added for avoiding to grab SCSI's refcont in IO path
V2:
- moving freeing hw queue resources into hctx's release handler
Ming Lei (7):
blk-mq: grab .q_usage_counter when queuing request from plug code path
blk-mq: move cancel of requeue_work into blk_mq_release
blk-mq: quiesce queue before updating nr_hw_queues
blk-mq: free hw queue's resource in hctx's release handler
blk-mq: move cancel of hctx->run_work into blk_mq_hw_sysfs_release
block: don't drain in-progress dispatch in blk_cleanup_queue()
SCSI: don't hold device refcount in IO path
block/blk-core.c | 23 +---------------------- block/blk-mq-sysfs.c | 8 ++++++++ block/blk-mq.c | 25 +++++++++++++++++-------- block/blk-mq.h | 2 +- drivers/scsi/scsi_lib.c | 28 ++-------------------------- 5 files changed, 29 insertions(+), 57 deletions(-)
Signature · 1 line
2.9.5
PATCH v4 1/7 blk-mq: grab .q_usage_counter when queuing request from plug code path
Just like aio/io_uring, we need to grab 2 refcount for queuing one request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed in generic_make_request() serves for submission. In theroy, this refcount should have been released after the sumission(async run queue) is done. blk_freeze_queue() works with blk_sync_queue() together for avoiding race between cleanup queue and IO submission, given async run queue activities are canceled because hctx->run_work is scheduled with the refcount held, so it is fine to not hold the refcount when running the run queue work function for dispatch IO.
However, if request is staggered into plug list, and finally queued from plug code path, the refcount in submission side is actually missed. And we may start to run queue after queue is removed because the queue's kobject refcount isn't guaranteed to be grabbed in flushing plug list context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queue blk_mq_run_hw_queue
Because of concurrent run queue, all requests inserted above may be completed before calling the above blk_mq_run_hw_queue. Then queue can be freed during the above blk_mq_run_hw_queue().
Fixes the issue by grab .q_usage_counter before calling blk_mq_sched_insert_requests() in blk_mq_flush_plug_list(). This way is safe because the queue is absolutely alive before inserting request.
--- block/blk-mq.c | 6 ++++++ 1 file changed, 6 insertions(+)
block/blk-mq.c +6 -0
@@ -1728,9 +1728,12 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) {if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); } this_q = rq->q;
@@ -1749,8 +1752,11 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)*/ if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); } }
Signature · 1 line
2.9.5
Re: [PATCH V4 1/7] blk-mq: grab .q_usage_counter when queuing request from plug code path
Quoted 44 lines · Ming Lei
Just like aio/io_uring, we need to grab 2 refcount for queuing one request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed in
generic_make_request()serves for submission. In theroy, this refcount should have been released after the sumission(async run queue) is done.blk_freeze_queue()works withblk_sync_queue()together for avoiding race between cleanup queue and IO submission, given async run queue activities are canceled becausehctx->run_workis scheduled with the refcount held, so it is fine to not hold the refcount when running the run queue work function for dispatch IO.However, if request is staggered into plug list, and finally queued from plug code path, the refcount in submission side is actually missed. And we may start to run queue after queue is removed because the queue's kobject refcount isn't guaranteed to be grabbed in flushing plug list context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queueblk_mq_run_hw_queueBecause of concurrent run queue, all requests inserted above may be completed before calling the above
blk_mq_run_hw_queue. Then queue can be freed during the aboveblk_mq_run_hw_queue().Fixes the issue by grab .q_usage_counter before calling
blk_mq_sched_insert_requests()inblk_mq_flush_plug_list(). This way is safe because the queue is absolutely alive before inserting request.Cc: Dongli Zhang <[email protected]>>Cc: James Smart <[email protected]>>Cc: Bart Van Assche <[email protected]>>Cc: [email protected],
Cc: Martin K . Petersen <[email protected]>
Cc: Christoph Hellwig <[email protected]>
Cc: James E . J . Bottomley <[email protected]>
Cc: jianchao wang <[email protected]>>Signed-off-by: Ming Lei <[email protected]>>---block/blk-mq.c|6 +-+-+-+-+-+-1 file changed, 6 insertions(+-)block/blk-mq.c +1
-1--- a/block/blk-mq.c +-+-+- b/block/blk-mq.c@@-1728,9 +-1728,12 @@void
blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)
Quoted 9 lines
if (rq->mq_hctx !=this_hctx ||rq->mq_ctx !=this_ctx) {> if (this_hctx) {> trace_block_unplug(this_q, depth, !from_schedule);>+-
+- percpu_ref_get(&this_q->q_usage_counter);> blk_mq_sched_insert_requests(this_hctx, this_ctx,
&rq_list,
from_schedule);>+- percpu_ref_put(&this_q->q_usage_counter);> }>
this_q =rq->q;>@@-1749,8 +-1752,11 @@void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)
*/
if (this_hctx) {> trace_block_unplug(this_q, depth, !from_schedule);>+-
+- percpu_ref_get(&this_q->q_usage_counter);> blk_mq_sched_insert_requests(this_hctx, this_ctx, &rq_list,
from_schedule);>+- percpu_ref_put(&this_q->q_usage_counter);> }> }Although this patch looks fine to me: have you considered to insert one percpu_ref_get() call at the start of blk_mq_flush_plug_list() and one percpu_ref_put() call at the end of the same function?
Thanks,
Bart.
Re: [PATCH V4 1/7] blk-mq: grab .q_usage_counter when queuing request from plug code path
Quoted 62 lines · Bart Van Assche
Quoted 60 lines · Ming Lei
Just like aio/io_uring, we need to grab 2 refcount for queuing one request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed in
generic_make_request()serves for submission. In theroy, this refcount should have been released after the sumission(async run queue) is done.blk_freeze_queue()works withblk_sync_queue()together for avoiding race between cleanup queue and IO submission, given async run queue activities are canceled becausehctx->run_workis scheduled with the refcount held, so it is fine to not hold the refcount when running the run queue work function for dispatch IO.However, if request is staggered into plug list, and finally queued from plug code path, the refcount in submission side is actually missed. And we may start to run queue after queue is removed because the queue's kobject refcount isn't guaranteed to be grabbed in flushing plug list context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queueblk_mq_run_hw_queueBecause of concurrent run queue, all requests inserted above may be completed before calling the above
blk_mq_run_hw_queue. Then queue can be freed during the aboveblk_mq_run_hw_queue().Fixes the issue by grab .q_usage_counter before calling
blk_mq_sched_insert_requests()inblk_mq_flush_plug_list(). This way is safe because the queue is absolutely alive before inserting request.Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 6 ++++++ 1 file changed, 6 insertions(+)block/blk-mq.c +3
-0@@ -1728,9 +1728,12 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) {if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); }
Quoted 16 lines
block/blk-mq.c void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) · line 1761
Quoted 12 lines
block/blk-mq.c +3
-0this_q = rq->q;@@ -1749,8 +1752,11 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)*/ if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); } }Although this patch looks fine to me: have you considered to insert one
percpu_ref_get()call at the start ofblk_mq_flush_plug_list()and onepercpu_ref_put()call at the end of the same function?
Requests from different request queues can be added to the same per-task plug list, so we can't do that way simply.
Thanks,
Ming Lei
Re: [PATCH V4 1/7] blk-mq: grab .q_usage_counter when queuing request from plug code path
In reply to Ming LeiQuoted 23 lines · Ming Lei
Just like aio/io_uring, we need to grab 2 refcount for queuing one request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed in
generic_make_request()serves for submission. In theroy, this refcount should have been released after the sumission(async run queue) is done.blk_freeze_queue()works withblk_sync_queue()together for avoiding race between cleanup queue and IO submission, given async run queue activities are canceled becausehctx->run_workis scheduled with the refcount held, so it is fine to not hold the refcount when running the run queue work function for dispatch IO.However, if request is staggered into plug list, and finally queued from plug code path, the refcount in submission side is actually missed. And we may start to run queue after queue is removed because the queue's kobject refcount isn't guaranteed to be grabbed in flushing plug list context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queueblk_mq_run_hw_queue
Quoted 7 lines
Because of concurrent run queue, all requests inserted above may be completed before calling the above
blk_mq_run_hw_queue. Then queue can be freed during the aboveblk_mq_run_hw_queue().Fixes the issue by grab .q_usage_counter before calling
blk_mq_sched_insert_requests()inblk_mq_flush_plug_list(). This way is safe because the queue is absolutely alive before inserting request.
Re: [PATCH V4 1/7] blk-mq: grab .q_usage_counter when queuing request from plug code path
In reply to Ming LeiHi Ming,
Quoted 49 lines · Ming Lei
Just like aio/io_uring, we need to grab 2 refcount for queuing one request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed in
generic_make_request()serves for submission. In theroy, this refcount should have been released after the sumission(async run queue) is done.blk_freeze_queue()works withblk_sync_queue()together for avoiding race between cleanup queue and IO submission, given async run queue activities are canceled becausehctx->run_workis scheduled with the refcount held, so it is fine to not hold the refcount when running the run queue work function for dispatch IO.However, if request is staggered into plug list, and finally queued from plug code path, the refcount in submission side is actually missed. And we may start to run queue after queue is removed because the queue's kobject refcount isn't guaranteed to be grabbed in flushing plug list context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queueblk_mq_run_hw_queueBecause of concurrent run queue, all requests inserted above may be completed before calling the above
blk_mq_run_hw_queue. Then queue can be freed during the aboveblk_mq_run_hw_queue().Fixes the issue by grab .q_usage_counter before calling
blk_mq_sched_insert_requests()inblk_mq_flush_plug_list(). This way is safe because the queue is absolutely alive before inserting request.Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 6 ++++++ 1 file changed, 6 insertions(+)block/blk-mq.c +0
-0
block/blk-mq.c void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) · line 1732
Quoted 5 lines
block/blk-mq.c +2
-0@@ -1728,9 +1728,12 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) {if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);
Sorry to bother but I would just like to double confirm the reason to use "percpu_ref_get()" here which does not check whether the queue has been frozen.
Is it because there is assumption that any direct/indirect caller of blk_mq_flush_plug_list() much have already grabbed q_usage_counter, which is similar to blk_queue_enter_live()?
Thank you very much!
Dongli Zhang
Re: [PATCH V4 1/7] blk-mq: grab .q_usage_counter when queuing request from plug code path
Quoted 61 lines · Dongli Zhang
Hi Ming,
Quoted 49 lines · Ming Lei
Just like aio/io_uring, we need to grab 2 refcount for queuing one request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed in
generic_make_request()serves for submission. In theroy, this refcount should have been released after the sumission(async run queue) is done.blk_freeze_queue()works withblk_sync_queue()together for avoiding race between cleanup queue and IO submission, given async run queue activities are canceled becausehctx->run_workis scheduled with the refcount held, so it is fine to not hold the refcount when running the run queue work function for dispatch IO.However, if request is staggered into plug list, and finally queued from plug code path, the refcount in submission side is actually missed. And we may start to run queue after queue is removed because the queue's kobject refcount isn't guaranteed to be grabbed in flushing plug list context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queueblk_mq_run_hw_queueBecause of concurrent run queue, all requests inserted above may be completed before calling the above
blk_mq_run_hw_queue. Then queue can be freed during the aboveblk_mq_run_hw_queue().Fixes the issue by grab .q_usage_counter before calling
blk_mq_sched_insert_requests()inblk_mq_flush_plug_list(). This way is safe because the queue is absolutely alive before inserting request.Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 6 ++++++ 1 file changed, 6 insertions(+)block/blk-mq.c +0
-0block/blk-mq.c void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) · line 1732
Quoted 5 lines
block/blk-mq.c +2
-0@@ -1728,9 +1728,12 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) {if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);Sorry to bother but I would just like to double confirm the reason to use "
percpu_ref_get()" here which does not check whether the queue has been frozen.
Quoted 3 lines
Is it because there is assumption that any direct/indirect caller of
blk_mq_flush_plug_list()much have already grabbedq_usage_counter, which is similar toblk_queue_enter_live()?
Because there is request in the plug list to be queued.
Thanks,
Ming
PATCH v4 2/7 blk-mq: move cancel of requeue_work into blk_mq_release
In reply to Ming LeiWith holding queue's kobject refcount, it is safe for driver to schedule requeue. However, blk_mq_kick_requeue_list() may be called after blk_sync_queue() is done because of concurrent requeue activities, then requeue work may not be completed when freeing queue, and kernel oops is triggered.
So moving the cancel of requeue_work into blk_mq_release() for avoiding race between requeue and freeing queue.
--- block/blk-core.c | 1 - block/blk-mq.c | 2 ++ 2 files changed, 2 insertions(+), 1 deletion(-)
block/blk-core.c +0 -1
@@ -237,7 +237,6 @@ void blk_sync_queue(struct request_queue *q)struct blk_mq_hw_ctx *hctx;int i; -cancel_delayed_work_sync (&q->requeue_work);queue_for_each_hw_ctx (q, hctx, i)cancel_delayed_work_sync (&hctx->run_work); }
block/blk-mq.c +2 -0
@@ -2626,6 +2626,8 @@ void blk_mq_release(struct request_queue *q)struct blk_mq_hw_ctx *hctx;unsigned int i; +cancel_delayed_work_sync (&q->requeue_work); +/* hctx kobj stays in hctx */ queue_for_each_hw_ctx (q, hctx, i) {if (!hctx)
Signature · 1 line
2.9.5
PATCH v4 3/7 blk-mq: quiesce queue before updating nr_hw_queues
In reply to Ming LeiInside __blk_mq_update_nr_hw_queues(), only request queues are frozen before updating nr_hw_queues.
However, even though blk_mq_freeze_queue() is returned, there might be run queue activity not completed, then use-after-free may be triggered on hctx and its fields.
Fix this issue by really quiescing queue via blk_mq_quiesce_queue() and blk_sync_queue() for making sure no any run queue activity pending before releasing hctx.
--- block/blk-mq.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
block/blk-mq.c +7 -2
@@ -3224,8 +3224,11 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,if (nr_hw_queues <1 || nr_hw_queues == set->nr_hw_queues)return ; -list_for_each_entry (q, &set->tag_list, tag_set_list) +list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_freeze_queue (q); +blk_mq_quiesce_queue (q); +blk_sync_queue (q); + }/* * Sync with blk_mq_queue_tag_busy_iter. */
@@ -3269,8 +3272,10 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,list_for_each_entry (q, &set->tag_list, tag_set_list)blk_mq_elv_switch_back (&head, q); -list_for_each_entry (q, &set->tag_list, tag_set_list) +list_for_each_entry (q, &set->tag_list, tag_set_list) { +blk_mq_unquiesce_queue (q);blk_mq_unfreeze_queue (q); + } }void blk_mq_update_nr_hw_queues (struct blk_mq_tag_set *set,int nr_hw_queues)
Signature · 1 line
2.9.5
Re: [PATCH V4 3/7] blk-mq: quiesce queue before updating nr_hw_queues
Quoted 13 lines · Ming Lei
block/blk-mq.c +1
-1--- a/block/blk-mq.c +-+-+- b/block/blk-mq.c@@-3224,8 +-3224,11 @@static void __blk_mq_update_nr_hw_queues(struct
blk_mq_tag_set*set,if (nr_hw_queues <1 ||nr_hw_queues ==set->nr_hw_queues) return;> - list_for_each_entry(q, &set->tag_list, tag_set_list) +- list_for_each_entry(q, &set->tag_list, tag_set_list) {> blk_mq_freeze_queue(q);>+- blk_mq_quiesce_queue(q);>+- blk_sync_queue(q);>+- }> /*> *Sync with blk_mq_queue_tag_busy_iter. */ @@-3269,8 +-3272,10 @@static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, list_for_each_entry(q, &set->tag_list, tag_set_list) blk_mq_elv_switch_back(&head, q);>
Quoted 3 lines
-
list_for_each_entry(q, &set->tag_list, tag_set_list)
+-list_for_each_entry(q, &set->tag_list, tag_set_list){>+-blk_mq_unquiesce_queue(q);> blk_mq_unfreeze_queue(q);>+- }> }>
voidblk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues)
Are you sure this patch is sufficient? What prevents that blk_mq_run_hw_queues() gets called after the blk_mq_quiesce() and blk_sync_queue() calls have finished and before the queue is unfrozen?
Bart.
Re: [PATCH V4 3/7] blk-mq: quiesce queue before updating nr_hw_queues
Quoted 30 lines · Bart Van Assche
Quoted 28 lines · Ming Lei
block/blk-mq.c +7
-2@@ -3224,8 +3224,11 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,if (nr_hw_queues <1 || nr_hw_queues == set->nr_hw_queues)return ; -list_for_each_entry (q, &set->tag_list, tag_set_list) +list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_freeze_queue (q); +blk_mq_quiesce_queue (q); +blk_sync_queue (q); + }/* * Sync with blk_mq_queue_tag_busy_iter. */ @@ -3269,8 +3272,10 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,list_for_each_entry (q, &set->tag_list, tag_set_list)blk_mq_elv_switch_back (&head, q); -list_for_each_entry (q, &set->tag_list, tag_set_list) +list_for_each_entry (q, &set->tag_list, tag_set_list) { +blk_mq_unquiesce_queue (q);blk_mq_unfreeze_queue (q); + } }
Quoted 5 lines
block/blk-mq.c static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, · line 3281
Quoted 1 line
block/blk-mq.c +0
-0void blk_mq_update_nr_hw_queues (struct blk_mq_tag_set *set,int nr_hw_queues)Are you sure this patch is sufficient? What prevents that
blk_mq_run_hw_queues()gets called after theblk_mq_quiesce()andblk_sync_queue()calls have finished and before the queue is unfrozen?
blk_queue_quiesced(hctx->queue) is supposed to prevents blk_mq_run_hw_queues() from being called after the queue is quiesced.
But there is another bug in blk_mq_quiesce() which shouldn't depend on hctx_lock(hctx, &srcu_idx)
for this check, will fix blk_mq_quiesce() in V5.
Thanks,
Ming Lei
Re: [PATCH V4 3/7] blk-mq: quiesce queue before updating nr_hw_queues
In reply to Ming LeiHi Ming
Why not add percpu_ref_tryget/put pair into run queue and requeue work, or before queue the work ?
Then freezing queue could really implement to freeze the queue and there will be no any queue activity after freeze, including run queue and requeue work.
Thanks
Jianchao
Quoted 55 lines · Ming Lei
Inside
__blk_mq_update_nr_hw_queues(), only request queues are frozen before updatingnr_hw_queues.However, even though
blk_mq_freeze_queue()is returned, there might be run queue activity not completed, then use-after-free may be triggered on hctx and its fields.Fix this issue by really quiescing queue via
blk_mq_quiesce_queue()andblk_sync_queue()for making sure no any run queue activity pending before releasing hctx.Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Reported-by Bart Van Assche <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)block/blk-mq.c +7
-2@@ -3224,8 +3224,11 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,if (nr_hw_queues <1 || nr_hw_queues == set->nr_hw_queues)return ; -list_for_each_entry (q, &set->tag_list, tag_set_list) +list_for_each_entry (q, &set->tag_list, tag_set_list) {blk_mq_freeze_queue (q); +blk_mq_quiesce_queue (q); +blk_sync_queue (q); + }/* * Sync with blk_mq_queue_tag_busy_iter. */ @@ -3269,8 +3272,10 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,list_for_each_entry (q, &set->tag_list, tag_set_list)blk_mq_elv_switch_back (&head, q); -list_for_each_entry (q, &set->tag_list, tag_set_list) +list_for_each_entry (q, &set->tag_list, tag_set_list) { +blk_mq_unquiesce_queue (q);blk_mq_unfreeze_queue (q); + } }void blk_mq_update_nr_hw_queues (struct blk_mq_tag_set *set,int nr_hw_queues)
Re: [PATCH V4 3/7] blk-mq: quiesce queue before updating nr_hw_queues
Quoted 5 lines · jianchao.wang
Hi Ming
Why not add percpu_ref_tryget/put pair into run queue and requeue work, or before queue the work ?
If following this direction, most of block layer API might need the pair.
Also Jens has complained in another thread, the pair may introduce 1.2% performance
loss, then we should avoid it in the fast path, such as blk_mq_run_hw_queue().
Given it is required that request queue is alive from kobject view before calling almost
every block layer API, this lifetime issue should be addressed easily by moving the hctx
free into queue's release handler.
Quoted 3 lines
Then freezing queue could really implement to freeze the queue and there will be no any queue activity after freeze, including run queue and requeue work.
The queue activity is just block layer internal thing, not related with driver, so not a big
deal by dealing with freeing hctx resources in release handler.
Thanks,
Ming Lei
PATCH v4 4/7 blk-mq: free hw queue's resource in hctx's release handler
In reply to Ming LeiOnce blk_cleanup_queue() returns, tags shouldn't be used any more, because blk_mq_free_tag_set() may be called. Commit 45a9c9d909b2 ("blk-mq: Fix a use-after-free") fixes this issue exactly.
However, that commit introduces another issue. Before 45a9c9d909b2, we are allowed to run queue during cleaning up queue if the queue's kobj refcount is held. After that commit, queue can't be run during queue cleaning up, otherwise oops can be triggered easily because some fields of hctx are freed by blk_mq_free_queue() in blk_cleanup_queue().
We have invented ways for addressing this kind of issue before, such as:
8dc765d438f1 ("SCSI: fix queue cleanup race before queue initialization is done") c2856ae2f315 ("blk-mq: quiesce queue before freeing queue")
But still can't cover all cases, recently James reports another such kind of issue:
https://marc.info/?l=linux-scsi&m=155389088124782&w=2
This issue can be quite hard to address by previous way, given scsi_run_queue() may run requeues for other LUNs.
Fixes the above issue by freeing hctx's resources in its release handler, and this way is safe becasue tags isn't needed for freeing such hctx resource.
This approach follows typical design pattern wrt. kobject's release handler.
--- block/blk-core.c | 2 +- block/blk-mq-sysfs.c | 6 ++++++ block/blk-mq.c | 8 ++------ block/blk-mq.h | 2 +- 4 files changed, 10 insertions(+), 8 deletions(-)
block/blk-core.c +1 -1
@@ -374,7 +374,7 @@ void blk_cleanup_queue(struct request_queue *q)blk_exit_queue (q);if (queue_is_mq (q)) -blk_mq_free_queue (q); +blk_mq_exit_queue (q);percpu_ref_exit (&q->q_usage_counter);
block/blk-mq-sysfs.c +6 -0
@@ -10,6 +10,7 @@#include <linux/smp.h> #include <linux/blk-mq.h> +#include "blk.h" #include "blk-mq.h" #include "blk-mq-tag.h"
@@ -33,6 +34,11 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj) { struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx, kobj); + +if (hctx->flags &BLK_MQ_F_BLOCKING ) +cleanup_srcu_struct (hctx->srcu); +blk_free_flush_queue (hctx->fq); +sbitmap_free (&hctx->ctx_map);free_cpumask_var (hctx->cpumask);kfree (hctx->ctxs);kfree (hctx);
block/blk-mq.c +2 -6
@@ -2259,12 +2259,7 @@ static void blk_mq_exit_hctx(struct request_queue *q,if (set->ops->exit_hctx) set->ops->exit_hctx (hctx, hctx_idx); -if (hctx->flags &BLK_MQ_F_BLOCKING ) -cleanup_srcu_struct (hctx->srcu); -blk_mq_remove_cpuhp (hctx); -blk_free_flush_queue (hctx->fq); -sbitmap_free (&hctx->ctx_map); } static void blk_mq_exit_hw_queues(struct request_queue *q,
@@ -2899,7 +2894,8 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, }EXPORT_SYMBOL (blk_mq_init_allocated_queue); -void blk_mq_free_queue (struct request_queue *q) +/* tags can _not_ be used after returning from blk_mq_exit_queue */ +void blk_mq_exit_queue (struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set;
block/blk-mq.h +1 -1
@@ -37,7 +37,7 @@ struct blk_mq_ctx {struct kobject kobj; }____cacheline_aligned_in_smp ; -void blk_mq_free_queue (struct request_queue *q); +void blk_mq_exit_queue (struct request_queue *q);int blk_mq_update_nr_requests (struct request_queue *q,unsigned int nr);void blk_mq_wake_waiters (struct request_queue *q);bool blk_mq_dispatch_rq_list (struct request_queue *,struct list_head *,bool );
Signature · 1 line
2.9.5
PATCH v4 5/7 blk-mq: move cancel of hctx->run_work into blk_mq_hw_sysfs_release
In reply to Ming Leihctx is always released after requeue is freed.
With holding queue's kobject refcount, it is safe for driver to run queue, so one run queue might be scheduled after blk_sync_queue() is done.
So moving the cancel of hctx->run_work into blk_mq_hw_sysfs_release() for avoiding run released queue.
--- block/blk-core.c | 8 -------- block/blk-mq-sysfs.c | 2 ++ 2 files changed, 2 insertions(+), 8 deletions(-)
block/blk-core.c +0 -8
@@ -232,14 +232,6 @@ void blk_sync_queue(struct request_queue *q) {del_timer_sync (&q->timeout);cancel_work_sync (&q->timeout_work); - -if (queue_is_mq (q)) { -struct blk_mq_hw_ctx *hctx; -int i; - -queue_for_each_hw_ctx (q, hctx, i) -cancel_delayed_work_sync (&hctx->run_work); - } }EXPORT_SYMBOL (blk_sync_queue);
block/blk-mq-sysfs.c +2 -0
@@ -35,6 +35,8 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj) struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx, kobj); +cancel_delayed_work_sync (&hctx->run_work); +if (hctx->flags &BLK_MQ_F_BLOCKING )cleanup_srcu_struct (hctx->srcu);blk_free_flush_queue (hctx->fq);
Signature · 1 line
2.9.5
PATCH v4 6/7 block: don't drain in-progress dispatch in blk_cleanup_queue()
In reply to Ming LeiNow freeing hw queue resource is moved to hctx's release handler, we don't need to worry about the race between blk_cleanup_queue and run queue any more.
So don't drain in-progress dispatch in blk_cleanup_queue().
This is basically revert of c2856ae2f315 ("blk-mq: quiesce queue before freeing queue").
--- block/blk-core.c | 12 ------------ 1 file changed, 12 deletions(-)
block/blk-core.c +0 -12
@@ -338,18 +338,6 @@ void blk_cleanup_queue(struct request_queue *q)blk_queue_flag_set (QUEUE_FLAG_DEAD , q); -/* -* make sure all in-progress dispatch are completed because -* blk_freeze_queue() can only complete all requests, and -* dispatch may still be in-progress since we dispatch requests -* from more than one contexts. -* -* We rely on driver to deal with the race in case that queue -* initialization isn't done. -*/ -if (queue_is_mq (q) &&blk_queue_init_done (q)) -blk_mq_quiesce_queue (q); -/* for synchronous bio-based driver finish in-flight integrity i/o */ blk_flush_integrity ();
Signature · 1 line
2.9.5
PATCH v4 7/7 SCSI: don't hold device refcount in IO path
In reply to Ming Leiscsi_device's refcount is always grabbed in IO path.
Turns out it isn't necessary, because blk_queue_cleanup() will drain any in-flight IOs, then cancel timeout/requeue work, and SCSI's requeue_work is canceled too in __scsi_remove_device().
Also scsi_device won't go away until blk_cleanup_queue() is done.
So don't hold the refcount in IO path, especially the refcount isn't required in IO path since blk_queue_enter() / blk_queue_exit() is introduced in the legacy block layer.
--- drivers/scsi/scsi_lib.c | 28 ++-------------------------- 1 file changed, 2 insertions(+), 26 deletions(-)
drivers/scsi/scsi_lib.c +2 -26
@@ -141,8 +141,6 @@ scsi_set_blocked(struct scsi_cmnd *cmd, int reason)static void scsi_mq_requeue_cmd (struct scsi_cmnd *cmd) { -struct scsi_device *sdev = cmd->device; -if (cmd->request->rq_flags &RQF_DONTPREP ) { cmd->request->rq_flags &= ~RQF_DONTPREP ;scsi_mq_uninit_cmd (cmd);
@@ -150,7 +148,6 @@ static void scsi_mq_requeue_cmd(struct scsi_cmnd *cmd)WARN_ON_ONCE (true); }blk_mq_requeue_request (cmd->request, true); -put_device (&sdev->sdev_gendev); }/**
@@ -189,19 +186,7 @@ static void __scsi_queue_insert(struct scsi_cmnd *cmd, int reason, bool unbusy)*/ cmd->result =0 ; -/* -* Before a SCSI command is dispatched, -* get_device(&sdev->sdev_gendev) is called and the host, -* target and device busy counters are increased. Since -* requeuing a request causes these actions to be repeated and -* since scsi_device_unbusy() has already been called, -* put_device(&device->sdev_gendev) must still be called. Call -* put_device() after blk_mq_requeue_request() to avoid that -* removal of the SCSI device can start before requeueing has -* happened. -*/ blk_mq_requeue_request (cmd->request, true); -put_device (&device->sdev_gendev); }/*
@@ -619,7 +604,6 @@ static bool scsi_end_request(struct request *req, blk_status_t error,blk_mq_run_hw_queues (q, true);percpu_ref_put (&q->q_usage_counter); -put_device (&sdev->sdev_gendev);return false; }
@@ -1613,7 +1597,6 @@ static void scsi_mq_put_budget(struct blk_mq_hw_ctx *hctx)struct scsi_device *sdev = q->queuedata;atomic_dec (&sdev->device_busy); -put_device (&sdev->sdev_gendev); }static bool scsi_mq_get_budget (struct blk_mq_hw_ctx *hctx)
@@ -1621,16 +1604,9 @@ static bool scsi_mq_get_budget(struct blk_mq_hw_ctx *hctx)struct request_queue *q = hctx->queue;struct scsi_device *sdev = q->queuedata; -if (!get_device (&sdev->sdev_gendev)) - goto out; -if (!scsi_dev_queue_ready (q, sdev)) - goto out_put_device; - -return true; +if (scsi_dev_queue_ready (q, sdev)) +return true; -out_put_device: -put_device (&sdev->sdev_gendev); -out:if (atomic_read (&sdev->device_busy) ==0 && !scsi_device_blocked (sdev))blk_mq_delay_run_hw_queue (hctx,SCSI_QUEUE_DELAY );return false;
Signature · 1 line
2.9.5
Re: [PATCH V4 7/7] SCSI: don't hold device refcount in IO path
Quoted 53 lines · Ming Lei
scsi_device's refcount is always grabbed in IO path.
Turns out it isn't necessary, because
blk_queue_cleanup()will drain any in-flight IOs, then cancel timeout/requeue work, and SCSI'srequeue_workis canceled too in__scsi_remove_device().Also
scsi_devicewon't go away untilblk_cleanup_queue()is done.So don't hold the refcount in IO path, especially the refcount isn't required in IO path since
blk_queue_enter()/blk_queue_exit()is introduced in the legacy block layer.Cc: Dongli Zhang <[email protected]>>Cc: James Smart <[email protected]>>Cc: Bart Van Assche <[email protected]>>Cc: [email protected],
Cc: Martin K . Petersen <[email protected]>
Cc: Christoph Hellwig <[email protected]>
Cc: James E . J . Bottomley <[email protected]>
Cc: jianchao wang <[email protected]>>Signed-off-by: Ming Lei <[email protected]>>---drivers/scsi/scsi_lib.c|28 +-+---------------------------1 file changed, 2 insertions(+-), 26 deletions(-)drivers/scsi/scsi_lib.c +1
-1--- a/drivers/scsi/scsi_lib.c +-+-+- b/drivers/scsi/scsi_lib.c@@-141,8 +-141,6 @@scsi_set_blocked(struct
scsi_cmnd*cmd, int reason)static void
scsi_mq_requeue_cmd(struct scsi_cmnd *cmd)
{>- structscsi_device*sdev =cmd->device;>-if (cmd->request->rq_flags &RQF_DONTPREP) {> cmd->request->rq_flags &=~RQF_DONTPREP;> scsi_mq_uninit_cmd(cmd);>@@-150,7 +-148,6 @@static void scsi_mq_requeue_cmd(struct scsi_cmnd *cmd) WARN_ON_ONCE(true);> }> blk_mq_requeue_request(cmd->request, true);>- put_device(&sdev->sdev_gendev);> }> /**>@@-189,19 +-186,7 @@static void __scsi_queue_insert(struct scsi_cmnd *cmd, int reason, bool unbusy) */ cmd->result =0;>drivers/scsi/scsi_lib.c +0
-10-/*>- *Before a SCSI command is dispatched, -*get_device(&sdev->sdev_gendev) is called and the host, -*target and device busy counters are increased. Since -*requeuing a request causes these actions to be repeated and -*since scsi_device_unbusy() has already been called, -*put_device(&device->sdev_gendev) must still be called. Call -*put_device() after blk_mq_requeue_request() to avoid that -*removal of the SCSI device can start before requeueing has -*happened. -*/ blk_mq_requeue_request(cmd->request, true);>- put_device(&device->sdev_gendev);> }>/*>@@-619,7 +-604,6 @@static bool scsi_end_request(struct request *req,
blk_status_terror,blk_mq_run_hw_queues(q, true);> percpu_ref_put(&q->q_usage_counter);>- put_device(&sdev->sdev_gendev);> return false;> }> @@-1613,7 +-1597,6 @@static void scsi_mq_put_budget(struct blk_mq_hw_ctx *hctx) struct scsi_device *sdev =q->queuedata;> atomic_dec(&sdev->device_busy);>- put_device(&sdev->sdev_gendev);> }> static bool scsi_mq_get_budget(struct blk_mq_hw_ctx *hctx) @@-1621,16 +-1604,9 @@static bool scsi_mq_get_budget(struct blk_mq_hw_ctx *hctx) struct request_queue *q =hctx->queue;> struct scsi_device *sdev =q->queuedata;>
drivers/scsi/scsi_lib.c
Quoted 9 lines
drivers/scsi/scsi_lib.c +1
-6-if (!get_device (&sdev->sdev_gendev)) - goto out;>- if (!scsi_dev_queue_ready(q, sdev)) - goto out_put_device;>- - return true;>+- if (scsi_dev_queue_ready(q, sdev)) +- return true;> -out_put_device: - put_device(&sdev->sdev_gendev);>-out:if (atomic_read (&sdev->device_busy) ==0 &&!scsi_device_blocked (sdev)) blk_mq_delay_run_hw_queue(hctx, SCSI_QUEUE_DELAY);> return false;