blk-mq: fix races related with freeing queue
Subsystems (3)
v5 cover blk-mq: fix races related with freeing queue
Hi,
Since 45a9c9d909b2 ("blk-mq: Fix a use-after-free"), run queue isn't allowed during cleanup queue even though queue refcount is held.
This change has caused lots of kernel oops triggered in run queue path, turns out it isn't easy to fix them all.
So move freeing of hw queue resources into hctx's release handler, then the above issue is fixed. Meantime, this way is safe given freeing hw queue resource doesn't require tags.
V3 covers more races.
V5:
- refactor blk_mq_alloc_and_init_hctx()
- fix race related updating nr_hw_queues by always freeing hctx after request queue is released
V4:
- add patch for fixing potential use-after-free in blk_mq_update_nr_hw_queues
- fix comment in the last patch
V3:
- cancel q->requeue_work in queue's release handler
- cancel hctx->run_work in hctx's release handler
- add patch 1 for fixing race in plug code path
- the last patch is added for avoiding to grab SCSI's refcont in IO path
V2:
- moving freeing hw queue resources into hctx's release handler
Ming Lei (9):
blk-mq: grab .q_usage_counter when queuing request from plug code path
blk-mq: move cancel of requeue_work into blk_mq_release
blk-mq: free hw queue's resource in hctx's release handler
blk-mq: move all hctx alloction & initialization into
__blk_mq_alloc_and_init_hctx
blk-mq: split blk_mq_alloc_and_init_hctx into two parts
blk-mq: always free hctx after request queue is freed
blk-mq: move cancel of hctx->run_work into blk_mq_hw_sysfs_release
block: don't drain in-progress dispatch in blk_cleanup_queue()
SCSI: don't hold device refcount in IO path
block/blk-core.c | 23 +----- block/blk-mq-sysfs.c | 8 +++ block/blk-mq.c | 188 ++++++++++++++++++++++++++++-------------------- block/blk-mq.h | 2 +- drivers/scsi/scsi_lib.c | 28 +------- include/linux/blk-mq.h | 2 + include/linux/blkdev.h | 7 ++ 7 files changed, 132 insertions(+), 126 deletions(-)
Signature · 1 line
2.9.5
PATCH v5 1/9 blk-mq: grab .q_usage_counter when queuing request from plug code path
Just like aio/io_uring, we need to grab 2 refcount for queuing one request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed in generic_make_request() serves for submission. In theroy, this refcount should have been released after the sumission(async run queue) is done. blk_freeze_queue() works with blk_sync_queue() together for avoiding race between cleanup queue and IO submission, given async run queue activities are canceled because hctx->run_work is scheduled with the refcount held, so it is fine to not hold the refcount when running the run queue work function for dispatch IO.
However, if request is staggered into plug list, and finally queued from plug code path, the refcount in submission side is actually missed. And we may start to run queue after queue is removed because the queue's kobject refcount isn't guaranteed to be grabbed in flushing plug list context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queue blk_mq_run_hw_queue
Because of concurrent run queue, all requests inserted above may be completed before calling the above blk_mq_run_hw_queue. Then queue can be freed during the above blk_mq_run_hw_queue().
Fixes the issue by grab .q_usage_counter before calling blk_mq_sched_insert_requests() in blk_mq_flush_plug_list(). This way is safe because the queue is absolutely alive before inserting request.
--- block/blk-mq.c | 6 ++++++ 1 file changed, 6 insertions(+)
block/blk-mq.c +6 -0
@@ -1728,9 +1728,12 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) {if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); } this_q = rq->q;
@@ -1749,8 +1752,11 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)*/ if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); } }
Signature · 1 line
2.9.5
Re: [PATCH V5 1/9] blk-mq: grab .q_usage_counter when queuing request from plug code path
Looks good,
Signature · 6 lines
Johannes Thumshirn SUSE Labs Filesystems [email protected] +49 911 74053 689 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
Re: [PATCH V5 1/9] blk-mq: grab .q_usage_counter when queuing request from plug code path
In reply to Ming LeiQuoted 65 lines · Ming Lei
Just like aio/io_uring, we need to grab 2 refcount for queuing one request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed in
generic_make_request()serves for submission. In theroy, this refcount should have been released after the sumission(async run queue) is done.blk_freeze_queue()works withblk_sync_queue()together for avoiding race between cleanup queue and IO submission, given async run queue activities are canceled becausehctx->run_workis scheduled with the refcount held, so it is fine to not hold the refcount when running the run queue work function for dispatch IO.However, if request is staggered into plug list, and finally queued from plug code path, the refcount in submission side is actually missed. And we may start to run queue after queue is removed because the queue's kobject refcount isn't guaranteed to be grabbed in flushing plug list context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queueblk_mq_run_hw_queueBecause of concurrent run queue, all requests inserted above may be completed before calling the above
blk_mq_run_hw_queue. Then queue can be freed during the aboveblk_mq_run_hw_queue().Fixes the issue by grab .q_usage_counter before calling
blk_mq_sched_insert_requests()inblk_mq_flush_plug_list(). This way is safe because the queue is absolutely alive before inserting request.Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Reviewed-by Bart Van Assche <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 6 ++++++ 1 file changed, 6 insertions(+)block/blk-mq.c +3
-0@@ -1728,9 +1728,12 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) {if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); } this_q = rq->q;@@ -1749,8 +1752,11 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)*/
block/blk-mq.c void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) · line 1762
Quoted 10 lines
block/blk-mq.c +3
-0if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); } }
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
PATCH v5 2/9 blk-mq: move cancel of requeue_work into blk_mq_release
In reply to Ming LeiWith holding queue's kobject refcount, it is safe for driver to schedule requeue. However, blk_mq_kick_requeue_list() may be called after blk_sync_queue() is done because of concurrent requeue activities, then requeue work may not be completed when freeing queue, and kernel oops is triggered.
So moving the cancel of requeue_work into blk_mq_release() for avoiding race between requeue and freeing queue.
--- block/blk-core.c | 1 - block/blk-mq.c | 2 ++ 2 files changed, 2 insertions(+), 1 deletion(-)
block/blk-core.c +0 -1
@@ -237,7 +237,6 @@ void blk_sync_queue(struct request_queue *q)struct blk_mq_hw_ctx *hctx;int i; -cancel_delayed_work_sync (&q->requeue_work);queue_for_each_hw_ctx (q, hctx, i)cancel_delayed_work_sync (&hctx->run_work); }
block/blk-mq.c +2 -0
@@ -2626,6 +2626,8 @@ void blk_mq_release(struct request_queue *q)struct blk_mq_hw_ctx *hctx;unsigned int i; +cancel_delayed_work_sync (&q->requeue_work); +/* hctx kobj stays in hctx */ queue_for_each_hw_ctx (q, hctx, i) {if (!hctx)
Signature · 1 line
2.9.5
Re: [PATCH V5 2/9] blk-mq: move cancel of requeue_work into blk_mq_release
Looks good,
Signature · 6 lines
Johannes Thumshirn SUSE Labs Filesystems [email protected] +49 911 74053 689 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg) Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
PATCH v5 3/9 blk-mq: free hw queue's resource in hctx's release handler
In reply to Ming LeiOnce blk_cleanup_queue() returns, tags shouldn't be used any more, because blk_mq_free_tag_set() may be called. Commit 45a9c9d909b2 ("blk-mq: Fix a use-after-free") fixes this issue exactly.
However, that commit introduces another issue. Before 45a9c9d909b2, we are allowed to run queue during cleaning up queue if the queue's kobj refcount is held. After that commit, queue can't be run during queue cleaning up, otherwise oops can be triggered easily because some fields of hctx are freed by blk_mq_free_queue() in blk_cleanup_queue().
We have invented ways for addressing this kind of issue before, such as:
8dc765d438f1 ("SCSI: fix queue cleanup race before queue initialization is done") c2856ae2f315 ("blk-mq: quiesce queue before freeing queue")
But still can't cover all cases, recently James reports another such kind of issue:
https://marc.info/?l=linux-scsi&m=155389088124782&w=2
This issue can be quite hard to address by previous way, given scsi_run_queue() may run requeues for other LUNs.
Fixes the above issue by freeing hctx's resources in its release handler, and this way is safe becasue tags isn't needed for freeing such hctx resource.
This approach follows typical design pattern wrt. kobject's release handler.
--- block/blk-core.c | 2 +- block/blk-mq-sysfs.c | 6 ++++++ block/blk-mq.c | 8 ++------ block/blk-mq.h | 2 +- 4 files changed, 10 insertions(+), 8 deletions(-)
block/blk-core.c +1 -1
@@ -374,7 +374,7 @@ void blk_cleanup_queue(struct request_queue *q)blk_exit_queue (q);if (queue_is_mq (q)) -blk_mq_free_queue (q); +blk_mq_exit_queue (q);percpu_ref_exit (&q->q_usage_counter);
block/blk-mq-sysfs.c +6 -0
@@ -10,6 +10,7 @@#include <linux/smp.h> #include <linux/blk-mq.h> +#include "blk.h" #include "blk-mq.h" #include "blk-mq-tag.h"
@@ -33,6 +34,11 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj) { struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx, kobj); + +if (hctx->flags &BLK_MQ_F_BLOCKING ) +cleanup_srcu_struct (hctx->srcu); +blk_free_flush_queue (hctx->fq); +sbitmap_free (&hctx->ctx_map);free_cpumask_var (hctx->cpumask);kfree (hctx->ctxs);kfree (hctx);
block/blk-mq.c +2 -6
@@ -2259,12 +2259,7 @@ static void blk_mq_exit_hctx(struct request_queue *q,if (set->ops->exit_hctx) set->ops->exit_hctx (hctx, hctx_idx); -if (hctx->flags &BLK_MQ_F_BLOCKING ) -cleanup_srcu_struct (hctx->srcu); -blk_mq_remove_cpuhp (hctx); -blk_free_flush_queue (hctx->fq); -sbitmap_free (&hctx->ctx_map); } static void blk_mq_exit_hw_queues(struct request_queue *q,
@@ -2899,7 +2894,8 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, }EXPORT_SYMBOL (blk_mq_init_allocated_queue); -void blk_mq_free_queue (struct request_queue *q) +/* tags can _not_ be used after returning from blk_mq_exit_queue */ +void blk_mq_exit_queue (struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set;
block/blk-mq.h +1 -1
@@ -37,7 +37,7 @@ struct blk_mq_ctx {struct kobject kobj; }____cacheline_aligned_in_smp ; -void blk_mq_free_queue (struct request_queue *q); +void blk_mq_exit_queue (struct request_queue *q);int blk_mq_update_nr_requests (struct request_queue *q,unsigned int nr);void blk_mq_wake_waiters (struct request_queue *q);bool blk_mq_dispatch_rq_list (struct request_queue *,struct list_head *,bool );
Signature · 1 line
2.9.5
Re: [PATCH V5 3/9] blk-mq: free hw queue's resource in hctx's release handler
Quoted 117 lines · Ming Lei
Once
blk_cleanup_queue()returns, tags shouldn't be used any more, becauseblk_mq_free_tag_set()may be called. Commit45a9c9d909b2("blk-mq: Fix a use-after-free") fixes this issue exactly.However, that commit introduces another issue. Before
45a9c9d909b2, we are allowed to run queue during cleaning up queue if the queue's kobj refcount is held. After that commit, queue can't be run during queue cleaning up, otherwise oops can be triggered easily because some fields of hctx are freed byblk_mq_free_queue()inblk_cleanup_queue().We have invented ways for addressing this kind of issue before, such as:
8dc765d438f1("SCSI: fix queue cleanup race before queue initialization is done")c2856ae2f315("blk-mq: quiesce queue before freeing queue")But still can't cover all cases, recently James reports another such kind of issue:
https://marc.info/?l=linux-scsi&m=155389088124782&w=2This issue can be quite hard to address by previous way, given
scsi_run_queue()may run requeues for other LUNs.Fixes the above issue by freeing hctx's resources in its release handler, and this way is safe becasue tags isn't needed for freeing such hctx resource.
This approach follows typical design pattern wrt. kobject's release handler.
Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Reported-by James Smart <[email protected]>Fixes 45a9c9d909b2 ("blk-mq: Fix a use-after-free")Signed-off-by Ming Lei <[email protected]>--- block/blk-core.c | 2 +- block/blk-mq-sysfs.c | 6 ++++++ block/blk-mq.c | 8 ++------ block/blk-mq.h | 2 +- 4 files changed, 10 insertions(+), 8 deletions(-)block/blk-core.c +1
-1@@ -374,7 +374,7 @@ void blk_cleanup_queue(struct request_queue *q)blk_exit_queue (q);if (queue_is_mq (q)) -blk_mq_free_queue (q); +blk_mq_exit_queue (q);percpu_ref_exit (&q->q_usage_counter);block/blk-mq-sysfs.c +6
-0@@ -10,6 +10,7 @@#include <linux/smp.h> #include <linux/blk-mq.h> +#include "blk.h" #include "blk-mq.h" #include "blk-mq-tag.h" @@ -33,6 +34,11 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj) { struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx, kobj); + +if (hctx->flags &BLK_MQ_F_BLOCKING ) +cleanup_srcu_struct (hctx->srcu); +blk_free_flush_queue (hctx->fq); +sbitmap_free (&hctx->ctx_map);free_cpumask_var (hctx->cpumask);kfree (hctx->ctxs);kfree (hctx);block/blk-mq.c +2
-6@@ -2259,12 +2259,7 @@ static void blk_mq_exit_hctx(struct request_queue *q,if (set->ops->exit_hctx) set->ops->exit_hctx (hctx, hctx_idx); -if (hctx->flags &BLK_MQ_F_BLOCKING ) -cleanup_srcu_struct (hctx->srcu); -blk_mq_remove_cpuhp (hctx); -blk_free_flush_queue (hctx->fq); -sbitmap_free (&hctx->ctx_map); } static void blk_mq_exit_hw_queues(struct request_queue *q,@@ -2899,7 +2894,8 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, }EXPORT_SYMBOL (blk_mq_init_allocated_queue); -void blk_mq_free_queue (struct request_queue *q) +/* tags can _not_ be used after returning from blk_mq_exit_queue */ +void blk_mq_exit_queue (struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set;block/blk-mq.h +0
-0
block/blk-mq.h struct blk_mq_ctx { · line 43
Quoted 8 lines
block/blk-mq.h +1
-1@@ -37,7 +37,7 @@ struct blk_mq_ctx {struct kobject kobj; }____cacheline_aligned_in_smp ; -void blk_mq_free_queue (struct request_queue *q); +void blk_mq_exit_queue (struct request_queue *q);int blk_mq_update_nr_requests (struct request_queue *q,unsigned int nr);void blk_mq_wake_waiters (struct request_queue *q);bool blk_mq_dispatch_rq_list (struct request_queue *,struct list_head *,bool );
Hmm. While this is a good point, I'm somehow not convinced that it fixes the mentioned problem.
For the sbitmap_any_set() case, the problem appears to be the racy interaction between nvme_scan_queues() and nvme_reset_ctrl().
Both can (and do) run in parallel, and as nvme_scan_queue() is not able to update the nvme namespace list race-free (nvme_ns_remove() will kill the namespace before removing it from the queue) we'll hit issues where nvme_start_queues() is called on namespaces which are halfway through nvme_ns_remove(), causing all sorts of issues.
See "[PATCHv2 RFC] nvme: use nvme_set_queue_dying() during namespace rescanning" for my attempt to fix this.
That patch is actually not complete; there is another one under testing currently to not remove namespaces at all during nvme_reset_ctrl(), but results are not in yet.
Otherwise:
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
Re: [PATCH V5 3/9] blk-mq: free hw queue's resource in hctx's release handler
Quoted 110 lines · Hannes Reinecke
Quoted 109 lines · Ming Lei
Once
blk_cleanup_queue()returns, tags shouldn't be used any more, becauseblk_mq_free_tag_set()may be called. Commit45a9c9d909b2("blk-mq: Fix a use-after-free") fixes this issue exactly.However, that commit introduces another issue. Before
45a9c9d909b2, we are allowed to run queue during cleaning up queue if the queue's kobj refcount is held. After that commit, queue can't be run during queue cleaning up, otherwise oops can be triggered easily because some fields of hctx are freed byblk_mq_free_queue()inblk_cleanup_queue().We have invented ways for addressing this kind of issue before, such as:
8dc765d438f1("SCSI: fix queue cleanup race before queue initialization is done")c2856ae2f315("blk-mq: quiesce queue before freeing queue")But still can't cover all cases, recently James reports another such kind of issue:
https://marc.info/?l=linux-scsi&m=155389088124782&w=2This issue can be quite hard to address by previous way, given
scsi_run_queue()may run requeues for other LUNs.Fixes the above issue by freeing hctx's resources in its release handler, and this way is safe becasue tags isn't needed for freeing such hctx resource.
This approach follows typical design pattern wrt. kobject's release handler.
Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Reported-by James Smart <[email protected]>Fixes 45a9c9d909b2 ("blk-mq: Fix a use-after-free")Signed-off-by Ming Lei <[email protected]>--- block/blk-core.c | 2 +- block/blk-mq-sysfs.c | 6 ++++++ block/blk-mq.c | 8 ++------ block/blk-mq.h | 2 +- 4 files changed, 10 insertions(+), 8 deletions(-)block/blk-core.c +1
-1@@ -374,7 +374,7 @@ void blk_cleanup_queue(struct request_queue *q)blk_exit_queue (q);if (queue_is_mq (q)) -blk_mq_free_queue (q); +blk_mq_exit_queue (q);percpu_ref_exit (&q->q_usage_counter);block/blk-mq-sysfs.c +6
-0@@ -10,6 +10,7 @@#include <linux/smp.h> #include <linux/blk-mq.h> +#include "blk.h" #include "blk-mq.h" #include "blk-mq-tag.h" @@ -33,6 +34,11 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj) { struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx, kobj); + +if (hctx->flags &BLK_MQ_F_BLOCKING ) +cleanup_srcu_struct (hctx->srcu); +blk_free_flush_queue (hctx->fq); +sbitmap_free (&hctx->ctx_map);free_cpumask_var (hctx->cpumask);kfree (hctx->ctxs);kfree (hctx);block/blk-mq.c +2
-6@@ -2259,12 +2259,7 @@ static void blk_mq_exit_hctx(struct request_queue *q,if (set->ops->exit_hctx) set->ops->exit_hctx (hctx, hctx_idx); -if (hctx->flags &BLK_MQ_F_BLOCKING ) -cleanup_srcu_struct (hctx->srcu); -blk_mq_remove_cpuhp (hctx); -blk_free_flush_queue (hctx->fq); -sbitmap_free (&hctx->ctx_map); } static void blk_mq_exit_hw_queues(struct request_queue *q,@@ -2899,7 +2894,8 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, }EXPORT_SYMBOL (blk_mq_init_allocated_queue); -void blk_mq_free_queue (struct request_queue *q) +/* tags can _not_ be used after returning from blk_mq_exit_queue */ +void blk_mq_exit_queue (struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set;block/blk-mq.h +0
-0@@ -37,7 +37,7 @@ struct blk_mq_ctx {struct kobject kobj; }____cacheline_aligned_in_smp ;
Quoted 8 lines
block/blk-mq.h struct blk_mq_ctx {
Quoted 6 lines
block/blk-mq.h +1
-1-void blk_mq_free_queue (struct request_queue *q); +void blk_mq_exit_queue (struct request_queue *q);int blk_mq_update_nr_requests (struct request_queue *q,unsigned int nr);void blk_mq_wake_waiters (struct request_queue *q);bool blk_mq_dispatch_rq_list (struct request_queue *,struct list_head *,bool );Hmm. While this is a good point, I'm somehow not convinced that it fixes the mentioned problem.
This patch fixes the following generic issue in block layer:
1) usually any block layer API may be called safely when the request queue's refcount is held
2) when blk_cleanup_queue() is run, no use-after-free oops can be triggered on the code path in 1)
Quoted 8 lines
For the
sbitmap_any_set()case, the problem appears to be the racy interaction betweennvme_scan_queues()andnvme_reset_ctrl(). Both can (and do) run in parallel, and asnvme_scan_queue()is not able to update the nvme namespace list race-free (nvme_ns_remove()will kill the namespace before removing it from the queue) we'll hit issues wherenvme_start_queues()is called on namespaces which are halfway throughnvme_ns_remove(), causing all sorts of issues.
This is one race between nvme reset and scan, and especially the refcount of ns queue isn't held before calling nvme_stop_queues() and nvme_start_queues().
The above situation is one NVMe specific issue and doesn't belong to what this patch tries to address.
Quoted 8 lines
See "[PATCHv2 RFC] nvme: use
nvme_set_queue_dying()during namespace rescanning" for my attempt to fix this.
That patch is actually not complete; there is another one under testing currently to not remove namespaces at all duringnvme_reset_ctrl(), but results are not in yet.Otherwise:
Looks the above 'Otherwise' shouldn't be needed, as I explained.
Quoted 2 lines
Reviewed-by Hannes Reinecke <[email protected]>
Thanks!
Signature · 1 line
Ming
PATCH v5 4/9 blk-mq: move all hctx alloction & initialization into __blk_mq_alloc_and_init_hctx
In reply to Ming LeiMove all hctx related allocation and initialization into __blk_mq_alloc_and_init_hctx, and prepare for splitting blk_mq_alloc_and_init_hctx into real two functions, one is for allocate everything, and another is for initializing everyting.
--- block/blk-mq.c | 93 +++++++++++++++++++++++++++------------------------------- 1 file changed, 44 insertions(+), 49 deletions(-)
block/blk-mq.c +44 -49
@@ -2276,15 +2276,45 @@ static void blk_mq_exit_hw_queues(struct request_queue *q, } } -static int blk_mq_init_hctx (struct request_queue *q, -struct blk_mq_tag_set *set, -struct blk_mq_hw_ctx *hctx,unsigned hctx_idx) +static int blk_mq_hw_ctx_size (struct blk_mq_tag_set *tag_set) { -int node; +int hw_ctx_size =sizeof (struct blk_mq_hw_ctx ); + +BUILD_BUG_ON (ALIGN (offsetof(struct blk_mq_hw_ctx , srcu), + __alignof__(struct blk_mq_hw_ctx )) != +sizeof (struct blk_mq_hw_ctx )); + +if (tag_set->flags &BLK_MQ_F_BLOCKING ) + hw_ctx_size +=sizeof (struct srcu_struct ); + +return hw_ctx_size; +} + +static struct blk_mq_hw_ctx * +__blk_mq_alloc_and_init_hctx (struct request_queue *q, +struct blk_mq_tag_set *set, +unsigned hctx_idx,int node) +{ +struct blk_mq_hw_ctx *hctx; + + hctx =kzalloc_node (blk_mq_hw_ctx_size (set), +GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, + node); +if (!hctx) + goto fail_alloc_hctx; + +if (!zalloc_cpumask_var_node (&hctx->cpumask, +GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, + node)) + goto free_hctx; + +atomic_set (&hctx->nr_active,0 ); + hctx->numa_node = node; + hctx->queue_num = hctx_idx; - node = hctx->numa_node;if (node ==NUMA_NO_NODE ) - node = hctx->numa_node = set->numa_node; + hctx->numa_node = set->numa_node; + node = hctx->numa_node;INIT_DELAYED_WORK (&hctx->run_work, blk_mq_run_work_fn);spin_lock_init (&hctx->lock);
@@ -2329,8 +2359,9 @@ static int blk_mq_init_hctx(struct request_queue *q,if (hctx->flags &BLK_MQ_F_BLOCKING )init_srcu_struct (hctx->srcu); +blk_mq_hctx_kobj_init (hctx); -return 0 ; +return hctx; free_fq:kfree (hctx->fq);
@@ -2343,7 +2374,11 @@ static int blk_mq_init_hctx(struct request_queue *q,kfree (hctx->ctxs); unregister_cpu_notifier:blk_mq_remove_cpuhp (hctx); -return -1 ; +free_cpumask_var (hctx->cpumask); + free_hctx: +kfree (hctx); + fail_alloc_hctx: +return NULL ; } static void blk_mq_init_cpu_queues(struct request_queue *q,
@@ -2689,51 +2724,11 @@ struct request_queue *blk_mq_init_sq_queue(struct blk_mq_tag_set *set, }EXPORT_SYMBOL (blk_mq_init_sq_queue); -static int blk_mq_hw_ctx_size (struct blk_mq_tag_set *tag_set) -{ -int hw_ctx_size =sizeof (struct blk_mq_hw_ctx ); - -BUILD_BUG_ON (ALIGN (offsetof(struct blk_mq_hw_ctx , srcu), - __alignof__(struct blk_mq_hw_ctx )) != -sizeof (struct blk_mq_hw_ctx )); - -if (tag_set->flags &BLK_MQ_F_BLOCKING ) - hw_ctx_size +=sizeof (struct srcu_struct ); - -return hw_ctx_size; -} -static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx (struct blk_mq_tag_set *set,struct request_queue *q,int hctx_idx,int node) { -struct blk_mq_hw_ctx *hctx; - - hctx =kzalloc_node (blk_mq_hw_ctx_size (set), -GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, - node); -if (!hctx) -return NULL ; - -if (!zalloc_cpumask_var_node (&hctx->cpumask, -GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, - node)) { -kfree (hctx); -return NULL ; - } - -atomic_set (&hctx->nr_active,0 ); - hctx->numa_node = node; - hctx->queue_num = hctx_idx; - -if (blk_mq_init_hctx (q, set, hctx, hctx_idx)) { -free_cpumask_var (hctx->cpumask); -kfree (hctx); -return NULL ; - } -blk_mq_hctx_kobj_init (hctx); - -return hctx; +return __blk_mq_alloc_and_init_hctx (q, set, hctx_idx, node); } static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
Signature · 1 line
2.9.5
Re: [PATCH V5 4/9] blk-mq: move all hctx alloction & initialization into __blk_mq_alloc_and_init_hctx
Quoted 14 lines · Ming Lei
Move all hctx related allocation and initialization into
__blk_mq_alloc_and_init_hctx, and prepare for splittingblk_mq_alloc_and_init_hctxinto real two functions, one is for allocate everything, and another is for initializing everyting.Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Signed-off-by Ming Lei <[email protected]>
Quoted 4 lines
--- block/blk-mq.c | 93 +++++++++++++++++++++++++++------------------------------- 1 file changed, 44 insertions(+), 49 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
PATCH v5 5/9 blk-mq: split blk_mq_alloc_and_init_hctx into two parts
In reply to Ming LeiSplit blk_mq_alloc_and_init_hctx into two parts, and one is blk_mq_alloc_hctx() which is for allocating all hctx resources, another is blk_mq_init_hctx() which is for initializing hctx, and serves as counter-part of blk_mq_exit_hctx().
--- block/blk-mq.c | 69 ++++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 45 insertions(+), 24 deletions(-)
block/blk-mq.c +45 -24
@@ -2290,10 +2290,36 @@ static int blk_mq_hw_ctx_size(struct blk_mq_tag_set *tag_set)return hw_ctx_size; } +static int blk_mq_init_hctx (struct request_queue *q, +struct blk_mq_tag_set *set, +struct blk_mq_hw_ctx *hctx, +unsigned hctx_idx) +{ + hctx->queue_num = hctx_idx; + +cpuhp_state_add_instance_nocalls (CPUHP_BLK_MQ_DEAD , &hctx->cpuhp_dead); + + hctx->tags = set->tags[hctx_idx]; + +if (set->ops->init_hctx && + set->ops->init_hctx (hctx, set->driver_data, hctx_idx)) + goto fail; + +if (blk_mq_init_request (set, hctx->fq->flush_rq, hctx_idx, + hctx->numa_node)) + goto exit_hctx; +return 0 ; + exit_hctx: +if (set->ops->exit_hctx) + set->ops->exit_hctx (hctx, hctx_idx); + fail: +return -1 ; +} +static struct blk_mq_hw_ctx * -__blk_mq_alloc_and_init_hctx (struct request_queue *q, -struct blk_mq_tag_set *set, -unsigned hctx_idx,int node) +blk_mq_alloc_hctx (struct request_queue *q, +struct blk_mq_tag_set *set, +unsigned hctx_idx,int node) {struct blk_mq_hw_ctx *hctx;
@@ -2310,8 +2336,6 @@ __blk_mq_alloc_and_init_hctx(struct request_queue *q,atomic_set (&hctx->nr_active,0 ); hctx->numa_node = node; - hctx->queue_num = hctx_idx; -if (node ==NUMA_NO_NODE ) hctx->numa_node = set->numa_node; node = hctx->numa_node;
@@ -2322,10 +2346,6 @@ __blk_mq_alloc_and_init_hctx(struct request_queue *q, hctx->queue = q; hctx->flags = set->flags & ~BLK_MQ_F_TAG_SHARED ; -cpuhp_state_add_instance_nocalls (CPUHP_BLK_MQ_DEAD , &hctx->cpuhp_dead); - - hctx->tags = set->tags[hctx_idx]; -/* * Allocate space for all possible cpus to avoid allocation at * runtime
@@ -2338,24 +2358,16 @@ __blk_mq_alloc_and_init_hctx(struct request_queue *q,if (sbitmap_init_node (&hctx->ctx_map, nr_cpu_ids,ilog2 (8 ),GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, node)) goto free_ctxs; - hctx->nr_ctx =0 ;spin_lock_init (&hctx->dispatch_wait_lock);init_waitqueue_func_entry (&hctx->dispatch_wait, blk_mq_dispatch_wake);INIT_LIST_HEAD (&hctx->dispatch_wait.entry); -if (set->ops->init_hctx && - set->ops->init_hctx (hctx, set->driver_data, hctx_idx)) - goto free_bitmap; - hctx->fq =blk_alloc_flush_queue (q, hctx->numa_node, set->cmd_size,GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY);if (!hctx->fq) - goto exit_hctx; - -if (blk_mq_init_request (set, hctx->fq->flush_rq, hctx_idx, node)) - goto free_fq; + goto free_bitmap;if (hctx->flags &BLK_MQ_F_BLOCKING )init_srcu_struct (hctx->srcu);
@@ -2363,11 +2375,6 @@ __blk_mq_alloc_and_init_hctx(struct request_queue *q,return hctx; - free_fq: -kfree (hctx->fq); - exit_hctx: -if (set->ops->exit_hctx) - set->ops->exit_hctx (hctx, hctx_idx); free_bitmap:sbitmap_free (&hctx->ctx_map); free_ctxs:
@@ -2728,7 +2735,21 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx( struct blk_mq_tag_set *set, struct request_queue *q, int hctx_idx, int node) { - return __blk_mq_alloc_and_init_hctx(q, set, hctx_idx, node); + struct blk_mq_hw_ctx *hctx; + + hctx =blk_mq_alloc_hctx (q, set, hctx_idx, node); +if (!hctx) + goto fail; + +if (blk_mq_init_hctx (q, set, hctx, hctx_idx)) + goto free_hctx; + +return hctx; + + free_hctx: +kobject_put (&hctx->kobj); + fail: +return NULL ; } static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
Signature · 1 line
2.9.5
Re: [PATCH V5 5/9] blk-mq: split blk_mq_alloc_and_init_hctx into two parts
Quoted 14 lines · Ming Lei
Split
blk_mq_alloc_and_init_hctxinto two parts, and one isblk_mq_alloc_hctx()which is for allocating all hctx resources, another isblk_mq_init_hctx()which is for initializing hctx, and serves as counter-part ofblk_mq_exit_hctx().Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Signed-off-by Ming Lei <[email protected]>
Quoted 4 lines
--- block/blk-mq.c | 69 ++++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 45 insertions(+), 24 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
PATCH v5 6/9 blk-mq: always free hctx after request queue is freed
In reply to Ming LeiIn normal queue cleanup path, hctx is released after request queue is freed, see blk_mq_release().
However, in __blk_mq_update_nr_hw_queues(), hctx may be freed because of hw queues shrinking. This way is easy to cause use-after-free,
because: one implicit rule is that it is safe to call almost all block layer APIs if the request queue is alive; and one hctx may be retrieved by one API, then the hctx can be freed by blk_mq_update_nr_hw_queues(); finally use-after-free is triggered.
Fixes this issue by always freeing hctx after releasing request queue. If some hctxs are removed in blk_mq_update_nr_hw_queues(), introduce a per-queue list to hold them, then try to resuse these hctxs if numa node is matched.
--- block/blk-mq.c | 40 +++++++++++++++++++++++++++------------- include/linux/blk-mq.h | 2 ++ include/linux/blkdev.h | 7 +++++++ 3 files changed, 36 insertions(+), 13 deletions(-)
block/blk-mq.c +27 -13
@@ -2260,6 +2260,10 @@ static void blk_mq_exit_hctx(struct request_queue *q, set->ops->exit_hctx (hctx, hctx_idx);blk_mq_remove_cpuhp (hctx); + +spin_lock (&q->dead_hctx_lock); +list_add (&hctx->hctx_list, &q->dead_hctx_list); +spin_unlock (&q->dead_hctx_lock); } static void blk_mq_exit_hw_queues(struct request_queue *q,
@@ -2660,15 +2664,13 @@ static int blk_mq_alloc_ctxs(struct request_queue *q)*/ void blk_mq_release (struct request_queue *q) { -struct blk_mq_hw_ctx *hctx; -unsigned int i; +struct blk_mq_hw_ctx *hctx, *next;cancel_delayed_work_sync (&q->requeue_work); -/* hctx kobj stays in hctx */ -queue_for_each_hw_ctx (q, hctx, i) { -if (!hctx) -continue ; +/* all hctx are in .dead_hctx_list now */ +list_for_each_entry_safe (hctx, next, &q->dead_hctx_list, hctx_list) { +list_del_init (&hctx->hctx_list);kobject_put (&hctx->kobj); }
@@ -2735,9 +2737,22 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx( struct blk_mq_tag_set *set, struct request_queue *q, int hctx_idx, int node) { - struct blk_mq_hw_ctx *hctx; + struct blk_mq_hw_ctx *hctx = NULL, *tmp; + +/* reuse dead hctx first */ +spin_lock (&q->dead_hctx_lock); +list_for_each_entry (tmp, &q->dead_hctx_list, hctx_list) { +if (tmp->numa_node == node) { + hctx = tmp; +break ; + } + } +if (hctx) +list_del_init (&hctx->hctx_list); +spin_unlock (&q->dead_hctx_lock); - hctx =blk_mq_alloc_hctx (q, set, hctx_idx, node); +if (!hctx) + hctx =blk_mq_alloc_hctx (q, set, hctx_idx, node);if (!hctx) goto fail;
@@ -2775,10 +2790,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, hctx =blk_mq_alloc_and_init_hctx (set, q, i, node);if (hctx) { -if (hctxs[i]) { +if (hctxs[i])blk_mq_exit_hctx (q, set, hctxs[i], i); -kobject_put (&hctxs[i]->kobj); - } hctxs[i] = hctx; }else {if (hctxs[i])
@@ -2809,9 +2822,7 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,if (hctx->tags)blk_mq_free_map_and_requests (set, j);blk_mq_exit_hctx (q, set, hctx, j); -kobject_put (&hctx->kobj); hctxs[j] =NULL ; - } }mutex_unlock (&q->sysfs_lock);
@@ -2854,6 +2865,9 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,if (!q->queue_hw_ctx) goto err_sys_init; +INIT_LIST_HEAD (&q->dead_hctx_list); +spin_lock_init (&q->dead_hctx_lock); +blk_mq_realloc_hw_ctxs (set, q);if (!q->nr_hw_queues) goto err_hctxs;
include/linux/blk-mq.h +2 -0
@@ -70,6 +70,8 @@ struct blk_mq_hw_ctx {struct dentry *sched_debugfs_dir;#endif +struct list_head hctx_list; +/* Must be the last member - see also blk_mq_hw_ctx_size(). */ struct srcu_struct srcu[0 ]; };
include/linux/blkdev.h +7 -0
@@ -535,6 +535,13 @@ struct request_queue {struct mutex sysfs_lock; +/* +* for reusing dead hctx instance in case of updating +* nr_hw_queues +*/ +struct list_head dead_hctx_list; +spinlock_t dead_hctx_lock; +atomic_t mq_freeze_depth;#if defined(CONFIG_BLK_DEV_BSG )
Signature · 1 line
2.9.5
Re: [PATCH V5 6/9] blk-mq: always free hctx after request queue is freed
Quoted 144 lines · Ming Lei
In normal queue cleanup path, hctx is released after request queue is freed, see
blk_mq_release().However, in
__blk_mq_update_nr_hw_queues(), hctx may be freed because of hw queues shrinking. This way is easy to cause use-after-free,
because: one implicit rule is that it is safe to call almost all block layer APIs if the request queue is alive; and one hctx may be retrieved by one API, then the hctx can be freed byblk_mq_update_nr_hw_queues(); finally use-after-free is triggered.Fixes this issue by always freeing hctx after releasing request queue. If some hctxs are removed in
blk_mq_update_nr_hw_queues(), introduce a per-queue list to hold them, then try to resuse these hctxs if numa node is matched.Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 40 +++++++++++++++++++++++++++------------- include/linux/blk-mq.h | 2 ++ include/linux/blkdev.h | 7 +++++++ 3 files changed, 36 insertions(+), 13 deletions(-)block/blk-mq.c +27
-13@@ -2260,6 +2260,10 @@ static void blk_mq_exit_hctx(struct request_queue *q, set->ops->exit_hctx (hctx, hctx_idx);blk_mq_remove_cpuhp (hctx); + +spin_lock (&q->dead_hctx_lock); +list_add (&hctx->hctx_list, &q->dead_hctx_list); +spin_unlock (&q->dead_hctx_lock); } static void blk_mq_exit_hw_queues(struct request_queue *q,@@ -2660,15 +2664,13 @@ static int blk_mq_alloc_ctxs(struct request_queue *q)*/ void blk_mq_release (struct request_queue *q) { -struct blk_mq_hw_ctx *hctx; -unsigned int i; +struct blk_mq_hw_ctx *hctx, *next;cancel_delayed_work_sync (&q->requeue_work); -/* hctx kobj stays in hctx */ -queue_for_each_hw_ctx (q, hctx, i) { -if (!hctx) -continue ; +/* all hctx are in .dead_hctx_list now */ +list_for_each_entry_safe (hctx, next, &q->dead_hctx_list, hctx_list) { +list_del_init (&hctx->hctx_list);kobject_put (&hctx->kobj); }@@ -2735,9 +2737,22 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx( struct blk_mq_tag_set *set, struct request_queue *q, int hctx_idx, int node) { - struct blk_mq_hw_ctx *hctx; + struct blk_mq_hw_ctx *hctx = NULL, *tmp; + +/* reuse dead hctx first */ +spin_lock (&q->dead_hctx_lock); +list_for_each_entry (tmp, &q->dead_hctx_list, hctx_list) { +if (tmp->numa_node == node) { + hctx = tmp; +break ; + } + } +if (hctx) +list_del_init (&hctx->hctx_list); +spin_unlock (&q->dead_hctx_lock); - hctx =blk_mq_alloc_hctx (q, set, hctx_idx, node); +if (!hctx) + hctx =blk_mq_alloc_hctx (q, set, hctx_idx, node);if (!hctx) goto fail;@@ -2775,10 +2790,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, hctx =blk_mq_alloc_and_init_hctx (set, q, i, node);if (hctx) { -if (hctxs[i]) { +if (hctxs[i])blk_mq_exit_hctx (q, set, hctxs[i], i); -kobject_put (&hctxs[i]->kobj); - } hctxs[i] = hctx; }else {if (hctxs[i])@@ -2809,9 +2822,7 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,if (hctx->tags)blk_mq_free_map_and_requests (set, j);blk_mq_exit_hctx (q, set, hctx, j); -kobject_put (&hctx->kobj); hctxs[j] =NULL ; - } }mutex_unlock (&q->sysfs_lock);@@ -2854,6 +2865,9 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,if (!q->queue_hw_ctx) goto err_sys_init; +INIT_LIST_HEAD (&q->dead_hctx_list); +spin_lock_init (&q->dead_hctx_lock); +blk_mq_realloc_hw_ctxs (set, q);if (!q->nr_hw_queues) goto err_hctxs;include/linux/blk-mq.h +2
-0@@ -70,6 +70,8 @@ struct blk_mq_hw_ctx {struct dentry *sched_debugfs_dir;#endif +struct list_head hctx_list; +/* Must be the last member - see also blk_mq_hw_ctx_size(). */ struct srcu_struct srcu[0 ]; };include/linux/blkdev.h +0
-0@@ -535,6 +535,13 @@ struct request_queue {struct mutex sysfs_lock;
include/linux/blkdev.h struct request_queue { · line 547
Quoted 10 lines
include/linux/blkdev.h +7
-0+/* +* for reusing dead hctx instance in case of updating +* nr_hw_queues +*/ +struct list_head dead_hctx_list; +spinlock_t dead_hctx_lock; +atomic_t mq_freeze_depth;#if defined(CONFIG_BLK_DEV_BSG )
I actually had been looking into this, too, but couldn't convince myself that the code really is a problem. Did you see this happening in real life?
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
Re: [PATCH V5 6/9] blk-mq: always free hctx after request queue is freed
Quoted 137 lines · Hannes Reinecke
Quoted 136 lines · Ming Lei
In normal queue cleanup path, hctx is released after request queue is freed, see
blk_mq_release().However, in
__blk_mq_update_nr_hw_queues(), hctx may be freed because of hw queues shrinking. This way is easy to cause use-after-free,
because: one implicit rule is that it is safe to call almost all block layer APIs if the request queue is alive; and one hctx may be retrieved by one API, then the hctx can be freed byblk_mq_update_nr_hw_queues(); finally use-after-free is triggered.Fixes this issue by always freeing hctx after releasing request queue. If some hctxs are removed in
blk_mq_update_nr_hw_queues(), introduce a per-queue list to hold them, then try to resuse these hctxs if numa node is matched.Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 40 +++++++++++++++++++++++++++------------- include/linux/blk-mq.h | 2 ++ include/linux/blkdev.h | 7 +++++++ 3 files changed, 36 insertions(+), 13 deletions(-)block/blk-mq.c +27
-13@@ -2260,6 +2260,10 @@ static void blk_mq_exit_hctx(struct request_queue *q, set->ops->exit_hctx (hctx, hctx_idx);blk_mq_remove_cpuhp (hctx); + +spin_lock (&q->dead_hctx_lock); +list_add (&hctx->hctx_list, &q->dead_hctx_list); +spin_unlock (&q->dead_hctx_lock); } static void blk_mq_exit_hw_queues(struct request_queue *q,@@ -2660,15 +2664,13 @@ static int blk_mq_alloc_ctxs(struct request_queue *q)*/ void blk_mq_release (struct request_queue *q) { -struct blk_mq_hw_ctx *hctx; -unsigned int i; +struct blk_mq_hw_ctx *hctx, *next;cancel_delayed_work_sync (&q->requeue_work); -/* hctx kobj stays in hctx */ -queue_for_each_hw_ctx (q, hctx, i) { -if (!hctx) -continue ; +/* all hctx are in .dead_hctx_list now */ +list_for_each_entry_safe (hctx, next, &q->dead_hctx_list, hctx_list) { +list_del_init (&hctx->hctx_list);kobject_put (&hctx->kobj); }@@ -2735,9 +2737,22 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx( struct blk_mq_tag_set *set, struct request_queue *q, int hctx_idx, int node) { - struct blk_mq_hw_ctx *hctx; + struct blk_mq_hw_ctx *hctx = NULL, *tmp; + +/* reuse dead hctx first */ +spin_lock (&q->dead_hctx_lock); +list_for_each_entry (tmp, &q->dead_hctx_list, hctx_list) { +if (tmp->numa_node == node) { + hctx = tmp; +break ; + } + } +if (hctx) +list_del_init (&hctx->hctx_list); +spin_unlock (&q->dead_hctx_lock); - hctx =blk_mq_alloc_hctx (q, set, hctx_idx, node); +if (!hctx) + hctx =blk_mq_alloc_hctx (q, set, hctx_idx, node);if (!hctx) goto fail;@@ -2775,10 +2790,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, hctx =blk_mq_alloc_and_init_hctx (set, q, i, node);if (hctx) { -if (hctxs[i]) { +if (hctxs[i])blk_mq_exit_hctx (q, set, hctxs[i], i); -kobject_put (&hctxs[i]->kobj); - } hctxs[i] = hctx; }else {if (hctxs[i])@@ -2809,9 +2822,7 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,if (hctx->tags)blk_mq_free_map_and_requests (set, j);blk_mq_exit_hctx (q, set, hctx, j); -kobject_put (&hctx->kobj); hctxs[j] =NULL ; - } }mutex_unlock (&q->sysfs_lock);@@ -2854,6 +2865,9 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,if (!q->queue_hw_ctx) goto err_sys_init; +INIT_LIST_HEAD (&q->dead_hctx_list); +spin_lock_init (&q->dead_hctx_lock); +blk_mq_realloc_hw_ctxs (set, q);if (!q->nr_hw_queues) goto err_hctxs;include/linux/blk-mq.h +2
-0@@ -70,6 +70,8 @@ struct blk_mq_hw_ctx {struct dentry *sched_debugfs_dir;#endif +struct list_head hctx_list; +/* Must be the last member - see also blk_mq_hw_ctx_size(). */ struct srcu_struct srcu[0 ]; };include/linux/blkdev.h +5
-0@@ -535,6 +535,13 @@ struct request_queue {struct mutex sysfs_lock; +/* +* for reusing dead hctx instance in case of updating +* nr_hw_queues +*/ +struct list_head dead_hctx_list;
Quoted 8 lines
include/linux/blkdev.h struct request_queue {
Quoted 5 lines
include/linux/blkdev.h +2
-0+spinlock_t dead_hctx_lock; +atomic_t mq_freeze_depth;#if defined(CONFIG_BLK_DEV_BSG )I actually had been looking into this, too, but couldn't convince myself that the code really is a problem.
Did you see this happening in real life?
User-after-free on hctx can be triggered if the hctx is freed in blk_mq_update_nr_hw_queues().
This patch aligns to normal cleanup queue model, and will avoid this kind of issue.
Updating nr_hw_queues is only triggered on very specific situation, so in practice, it isn't easy to trigger.
However, if we play CPU hotplug & timeout test on nvme-loop, it should have been easy to trigger, but nvme-loop's reset/timeout handler is simply broken. The same test can be done on nvme-pci too, just needs real hardware and environment and takes time...
Thanks,
Ming
PATCH v5 7/9 blk-mq: move cancel of hctx->run_work into blk_mq_hw_sysfs_release
In reply to Ming Leihctx is always released after requeue is freed.
With holding queue's kobject refcount, it is safe for driver to run queue, so one run queue might be scheduled after blk_sync_queue() is done.
So moving the cancel of hctx->run_work into blk_mq_hw_sysfs_release() for avoiding run released queue.
--- block/blk-core.c | 8 -------- block/blk-mq-sysfs.c | 2 ++ 2 files changed, 2 insertions(+), 8 deletions(-)
block/blk-core.c +0 -8
@@ -232,14 +232,6 @@ void blk_sync_queue(struct request_queue *q) {del_timer_sync (&q->timeout);cancel_work_sync (&q->timeout_work); - -if (queue_is_mq (q)) { -struct blk_mq_hw_ctx *hctx; -int i; - -queue_for_each_hw_ctx (q, hctx, i) -cancel_delayed_work_sync (&hctx->run_work); - } }EXPORT_SYMBOL (blk_sync_queue);
block/blk-mq-sysfs.c +2 -0
@@ -35,6 +35,8 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj) struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx, kobj); +cancel_delayed_work_sync (&hctx->run_work); +if (hctx->flags &BLK_MQ_F_BLOCKING )cleanup_srcu_struct (hctx->srcu);blk_free_flush_queue (hctx->fq);
Signature · 1 line
2.9.5
Re: [PATCH V5 7/9] blk-mq: move cancel of hctx->run_work into blk_mq_hw_sysfs_release
Quoted 18 lines · Ming Lei
hctx is always released after requeue is freed.
With holding queue's kobject refcount, it is safe for driver to run queue, so one run queue might be scheduled after
blk_sync_queue()is done.So moving the cancel of
hctx->run_workintoblk_mq_hw_sysfs_release()for avoiding run released queue.Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Reviewed-by Bart Van Assche <[email protected]>Signed-off-by Ming Lei <[email protected]>
Quoted 5 lines
--- block/blk-core.c | 8 -------- block/blk-mq-sysfs.c | 2 ++ 2 files changed, 2 insertions(+), 8 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
PATCH v5 8/9 block: don't drain in-progress dispatch in blk_cleanup_queue()
In reply to Ming LeiNow freeing hw queue resource is moved to hctx's release handler, we don't need to worry about the race between blk_cleanup_queue and run queue any more.
So don't drain in-progress dispatch in blk_cleanup_queue().
This is basically revert of c2856ae2f315 ("blk-mq: quiesce queue before freeing queue").
--- block/blk-core.c | 12 ------------ 1 file changed, 12 deletions(-)
block/blk-core.c +0 -12
@@ -338,18 +338,6 @@ void blk_cleanup_queue(struct request_queue *q)blk_queue_flag_set (QUEUE_FLAG_DEAD , q); -/* -* make sure all in-progress dispatch are completed because -* blk_freeze_queue() can only complete all requests, and -* dispatch may still be in-progress since we dispatch requests -* from more than one contexts. -* -* We rely on driver to deal with the race in case that queue -* initialization isn't done. -*/ -if (queue_is_mq (q) &&blk_queue_init_done (q)) -blk_mq_quiesce_queue (q); -/* for synchronous bio-based driver finish in-flight integrity i/o */ blk_flush_integrity ();
Signature · 1 line
2.9.5
Re: [PATCH V5 8/9] block: don't drain in-progress dispatch in blk_cleanup_queue()
Quoted 19 lines · Ming Lei
Now freeing hw queue resource is moved to hctx's release handler, we don't need to worry about the race between
blk_cleanup_queueand run queue any more.So don't drain in-progress dispatch in
blk_cleanup_queue().This is basically revert of
c2856ae2f315("blk-mq: quiesce queue before freeing queue").Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Reviewed-by Bart Van Assche <[email protected]>Signed-off-by Ming Lei <[email protected]>
Quoted 4 lines
--- block/blk-core.c | 12 ------------ 1 file changed, 12 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
PATCH v5 9/9 SCSI: don't hold device refcount in IO path
In reply to Ming Leiscsi_device's refcount is always grabbed in IO path.
Turns out it isn't necessary, because blk_queue_cleanup() will drain any in-flight IOs, then cancel timeout/requeue work, and SCSI's requeue_work is canceled too in __scsi_remove_device().
Also scsi_device won't go away until blk_cleanup_queue() is done.
So don't hold the refcount in IO path, especially the refcount isn't required in IO path since blk_queue_enter() / blk_queue_exit() is introduced in the legacy block layer.
--- drivers/scsi/scsi_lib.c | 28 ++-------------------------- 1 file changed, 2 insertions(+), 26 deletions(-)
drivers/scsi/scsi_lib.c +2 -26
@@ -141,8 +141,6 @@ scsi_set_blocked(struct scsi_cmnd *cmd, int reason)static void scsi_mq_requeue_cmd (struct scsi_cmnd *cmd) { -struct scsi_device *sdev = cmd->device; -if (cmd->request->rq_flags &RQF_DONTPREP ) { cmd->request->rq_flags &= ~RQF_DONTPREP ;scsi_mq_uninit_cmd (cmd);
@@ -150,7 +148,6 @@ static void scsi_mq_requeue_cmd(struct scsi_cmnd *cmd)WARN_ON_ONCE (true); }blk_mq_requeue_request (cmd->request, true); -put_device (&sdev->sdev_gendev); }/**
@@ -189,19 +186,7 @@ static void __scsi_queue_insert(struct scsi_cmnd *cmd, int reason, bool unbusy)*/ cmd->result =0 ; -/* -* Before a SCSI command is dispatched, -* get_device(&sdev->sdev_gendev) is called and the host, -* target and device busy counters are increased. Since -* requeuing a request causes these actions to be repeated and -* since scsi_device_unbusy() has already been called, -* put_device(&device->sdev_gendev) must still be called. Call -* put_device() after blk_mq_requeue_request() to avoid that -* removal of the SCSI device can start before requeueing has -* happened. -*/ blk_mq_requeue_request (cmd->request, true); -put_device (&device->sdev_gendev); }/*
@@ -619,7 +604,6 @@ static bool scsi_end_request(struct request *req, blk_status_t error,blk_mq_run_hw_queues (q, true);percpu_ref_put (&q->q_usage_counter); -put_device (&sdev->sdev_gendev);return false; }
@@ -1613,7 +1597,6 @@ static void scsi_mq_put_budget(struct blk_mq_hw_ctx *hctx)struct scsi_device *sdev = q->queuedata;atomic_dec (&sdev->device_busy); -put_device (&sdev->sdev_gendev); }static bool scsi_mq_get_budget (struct blk_mq_hw_ctx *hctx)
@@ -1621,16 +1604,9 @@ static bool scsi_mq_get_budget(struct blk_mq_hw_ctx *hctx)struct request_queue *q = hctx->queue;struct scsi_device *sdev = q->queuedata; -if (!get_device (&sdev->sdev_gendev)) - goto out; -if (!scsi_dev_queue_ready (q, sdev)) - goto out_put_device; - -return true; +if (scsi_dev_queue_ready (q, sdev)) +return true; -out_put_device: -put_device (&sdev->sdev_gendev); -out:if (atomic_read (&sdev->device_busy) ==0 && !scsi_device_blocked (sdev))blk_mq_delay_run_hw_queue (hctx,SCSI_QUEUE_DELAY );return false;
Signature · 1 line
2.9.5
Re: [PATCH V5 9/9] SCSI: don't hold device refcount in IO path
Quoted 22 lines · Ming Lei
scsi_device's refcount is always grabbed in IO path.
Turns out it isn't necessary, because
blk_queue_cleanup()will drain any in-flight IOs, then cancel timeout/requeue work, and SCSI'srequeue_workis canceled too in__scsi_remove_device().Also
scsi_devicewon't go away untilblk_cleanup_queue()is done.So don't hold the refcount in IO path, especially the refcount isn't required in IO path since
blk_queue_enter()/blk_queue_exit()is introduced in the legacy block layer.Cc Dongli Zhang <[email protected]>Cc James Smart <[email protected]>Cc Bart Van Assche <[email protected]>Cc Martin K . Petersen <[email protected]>,Cc Christoph Hellwig <[email protected]>,Cc James E . J . Bottomley <[email protected]>,Cc jianchao wang <[email protected]>Reviewed-by Bart Van Assche <[email protected]>Signed-off-by Ming Lei <[email protected]>
Quoted 4 lines
--- drivers/scsi/scsi_lib.c | 28 ++-------------------------- 1 file changed, 2 insertions(+), 26 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
Re: [PATCH V5 9/9] SCSI: don't hold device refcount in IO path
In reply to Ming LeiMing,
Quoted 11 lines
scsi_device's refcount is always grabbed in IO path.
Turns out it isn't necessary, because
blk_queue_cleanup()will drain any in-flight IOs, then cancel timeout/requeue work, and SCSI'srequeue_workis canceled too in__scsi_remove_device().Also
scsi_devicewon't go away untilblk_cleanup_queue()is done.So don't hold the refcount in IO path, especially the refcount isn't required in IO path since
blk_queue_enter()/blk_queue_exit()is introduced in the legacy block layer.
This all looks good to me. I assume it will go through Jens' tree?
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Re: [PATCH V5 9/9] SCSI: don't hold device refcount in IO path
Quoted 11 lines · Martin K. Petersen
Ming,
Quoted 8 lines
scsi_device's refcount is always grabbed in IO path.
Turns out it isn't necessary, because
blk_queue_cleanup()will drain any in-flight IOs, then cancel timeout/requeue work, and SCSI'srequeue_workis canceled too in__scsi_remove_device().Also
scsi_devicewon't go away untilblk_cleanup_queue()is done.
Quoted 7 lines
Quoted 3 lines
So don't hold the refcount in IO path, especially the refcount isn't required in IO path since
blk_queue_enter()/blk_queue_exit()is introduced in the legacy block layer.This all looks good to me. I assume it will go through Jens' tree?
Hi Nartin,
This patch doesn't depend on patch 1~8, so please take it via scsi tree if you are fine.
Thanks,
Ming
Re: [PATCH V5 9/9] SCSI: don't hold device refcount in IO path
Quoted 17 lines · Ming Lei
Quoted 15 lines · Martin K. Petersen
Ming,
Quoted 11 lines
scsi_device's refcount is always grabbed in IO path.
Turns out it isn't necessary, because
blk_queue_cleanup()will drain any in-flight IOs, then cancel timeout/requeue work, and SCSI'srequeue_workis canceled too in__scsi_remove_device().Also
scsi_devicewon't go away untilblk_cleanup_queue()is done.So don't hold the refcount in IO path, especially the refcount isn't required in IO path since
blk_queue_enter()/blk_queue_exit()is introduced in the legacy block layer.
Quoted 5 lines
Quoted 3 lines
This all looks good to me. I assume it will go through Jens' tree?
Hi Nartin,
Martin, sorry for the typo, :-(
Thanks,
Ming Lei
Martin K. Petersen
AppliedRe: [PATCH V5 9/9] SCSI: don't hold device refcount in IO path
In reply to Ming LeiMing,
Quoted 2 lines
This patch doesn't depend on patch 1~8, so please take it via scsi tree if you are fine.
Applied, thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering