blk-mq: fix races related with freeing queue
Subsystems (4)
v7 cover blk-mq: fix races related with freeing queue
Hi,
Since 45a9c9d909b2 ("blk-mq: Fix a use-after-free"), run queue isn't allowed during cleanup queue even though queue refcount is held.
This change has caused lots of kernel oops triggered in run queue path, turns out it isn't easy to fix them all.
So move freeing of hw queue resources into hctx's release handler, then the above issue is fixed. Meantime, this way is safe given freeing hw queue resource doesn't require tags.
V3 covers more races.
V7:
- add reviewed-by and tested-by tag
- rename "dead_hctx" as "unused_hctx"
- check if there are live hctx in queue's release handler
- only patch 6 is modified
V6:
- remove previous SCSI patch which will be routed via SCSI tree
- add reviewed-by tag
- fix one related NVMe scan vs reset race
V5:
- refactor blk_mq_alloc_and_init_hctx()
- fix race related updating nr_hw_queues by always freeing hctx after request queue is released
V4:
- add patch for fixing potential use-after-free in blk_mq_update_nr_hw_queues
- fix comment in the last patch
V3:
- cancel q->requeue_work in queue's release handler
- cancel hctx->run_work in hctx's release handler
- add patch 1 for fixing race in plug code path
- the last patch is added for avoiding to grab SCSI's refcont in IO path
V2:
- moving freeing hw queue resources into hctx's release handler
Ming Lei (9):
blk-mq: grab .q_usage_counter when queuing request from plug code path
blk-mq: move cancel of requeue_work into blk_mq_release
blk-mq: free hw queue's resource in hctx's release handler
blk-mq: move all hctx alloction & initialization into
__blk_mq_alloc_and_init_hctx
blk-mq: split blk_mq_alloc_and_init_hctx into two parts
blk-mq: always free hctx after request queue is freed
blk-mq: move cancel of hctx->run_work into blk_mq_hw_sysfs_release
block: don't drain in-progress dispatch in blk_cleanup_queue()
nvme: hold request queue's refcount in ns's whole lifetime
block/blk-core.c | 23 +----- block/blk-mq-sysfs.c | 8 ++ block/blk-mq.c | 199 ++++++++++++++++++++++++++++------------------- block/blk-mq.h | 2 +- drivers/nvme/host/core.c | 10 ++- include/linux/blk-mq.h | 2 + include/linux/blkdev.h | 7 ++ 7 files changed, 148 insertions(+), 103 deletions(-)
Signature · 1 line
2.9.5
PATCH v7 1/9 blk-mq: grab .q_usage_counter when queuing request from plug code path
Just like aio/io_uring, we need to grab 2 refcount for queuing one request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed in generic_make_request() serves for submission. In theroy, this refcount should have been released after the sumission(async run queue) is done. blk_freeze_queue() works with blk_sync_queue() together for avoiding race between cleanup queue and IO submission, given async run queue activities are canceled because hctx->run_work is scheduled with the refcount held, so it is fine to not hold the refcount when running the run queue work function for dispatch IO.
However, if request is staggered into plug list, and finally queued from plug code path, the refcount in submission side is actually missed. And we may start to run queue after queue is removed because the queue's kobject refcount isn't guaranteed to be grabbed in flushing plug list context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queue blk_mq_run_hw_queue
Because of concurrent run queue, all requests inserted above may be completed before calling the above blk_mq_run_hw_queue. Then queue can be freed during the above blk_mq_run_hw_queue().
Fixes the issue by grab .q_usage_counter before calling blk_mq_sched_insert_requests() in blk_mq_flush_plug_list(). This way is safe because the queue is absolutely alive before inserting request.
--- block/blk-mq.c | 6 ++++++ 1 file changed, 6 insertions(+)
block/blk-mq.c +6 -0
@@ -1736,9 +1736,12 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) {if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); } this_q = rq->q;
@@ -1757,8 +1760,11 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)*/ if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); } }
Signature · 1 line
2.9.5
Christoph Hellwig
maintainerRe: [PATCH V7 1/9] blk-mq: grab .q_usage_counter when queuing request from plug code path
Quoted 13 lines · Ming Lei
block/blk-mq.c +3
-0if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) {if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); } this_q = rq->q;
block/blk-mq.c void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) · line 1767
Quoted 8 lines
block/blk-mq.c +3
-0@@ -1757,8 +1760,11 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)*/ if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter);
Why don't we push this into blk_mq_sched_insert_requests? Yes, it would need a request_queue argument, but that still seems saner than duplicating it in both callers.
Re: [PATCH V7 1/9] blk-mq: grab .q_usage_counter when queuing request from plug code path
Quoted 12 lines · Christoph Hellwig
Quoted 11 lines · Ming Lei
block/blk-mq.c +3
-0if (rq->mq_hctx != this_hctx || rq->mq_ctx != this_ctx) {if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter); }
Quoted 14 lines
block/blk-mq.c void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) · line 1767
Quoted 10 lines
block/blk-mq.c +3
-0this_q = rq->q;@@ -1757,8 +1760,11 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)*/ if (this_hctx) {trace_block_unplug (this_q, depth, !from_schedule); + +percpu_ref_get (&this_q->q_usage_counter);blk_mq_sched_insert_requests (this_hctx, this_ctx, &rq_list, from_schedule); +percpu_ref_put (&this_q->q_usage_counter);Why don't we push this into
blk_mq_sched_insert_requests? Yes, it would need arequest_queueargument, but that still seems saner than duplicating it in both callers.
It isn't in other callers of blk_mq_sched_insert_requests(), it is just needed in some corner case like flush plug context.
For other callers of blk_mq_sched_insert_requests(), it is guaranteed that request queue's ref is held.
Thanks,
Ming
Christoph Hellwig
maintainerRe: [PATCH V7 1/9] blk-mq: grab .q_usage_counter when queuing request from plug code path
Quoted 5 lines · Ming Lei
It isn't in other callers of
blk_mq_sched_insert_requests(), it is just needed in some corner case like flush plug context.For other callers of
blk_mq_sched_insert_requests(), it is guaranteed that request queue's ref is held.
In both Linus' tree and Jens' for-5.2 tree I only see these two callers of blk_mq_sched_insert_requests. What am I missing?
Re: [PATCH V7 1/9] blk-mq: grab .q_usage_counter when queuing request from plug code path
Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Ming Lei
It isn't in other callers of
blk_mq_sched_insert_requests(), it is just needed in some corner case like flush plug context.For other callers of
blk_mq_sched_insert_requests(), it is guaranteed that request queue's ref is held.In both Linus' tree and Jens' for-5.2 tree I only see these two callers of
blk_mq_sched_insert_requests. What am I missing?
OK, what I meant is that the issue is flush plug context specific. And I didn't notice that this function is only called by blk_mq_flush_plug_list().
Then it is fine to move the percpu_ref_get()/percpu_ref_put() into the function, better with comment on this issue.
Thanks,
Ming
PATCH v7 2/9 blk-mq: move cancel of requeue_work into blk_mq_release
In reply to Ming LeiWith holding queue's kobject refcount, it is safe for driver to schedule requeue. However, blk_mq_kick_requeue_list() may be called after blk_sync_queue() is done because of concurrent requeue activities, then requeue work may not be completed when freeing queue, and kernel oops is triggered.
So moving the cancel of requeue_work into blk_mq_release() for avoiding race between requeue and freeing queue.
--- block/blk-core.c | 1 - block/blk-mq.c | 2 ++ 2 files changed, 2 insertions(+), 1 deletion(-)
block/blk-core.c +0 -1
@@ -237,7 +237,6 @@ void blk_sync_queue(struct request_queue *q)struct blk_mq_hw_ctx *hctx;int i; -cancel_delayed_work_sync (&q->requeue_work);queue_for_each_hw_ctx (q, hctx, i)cancel_delayed_work_sync (&hctx->run_work); }
block/blk-mq.c +2 -0
@@ -2640,6 +2640,8 @@ void blk_mq_release(struct request_queue *q)struct blk_mq_hw_ctx *hctx;unsigned int i; +cancel_delayed_work_sync (&q->requeue_work); +/* hctx kobj stays in hctx */ queue_for_each_hw_ctx (q, hctx, i) {if (!hctx)
Signature · 1 line
2.9.5
Re: [PATCH V7 2/9] blk-mq: move cancel of requeue_work into blk_mq_release
Quoted 8 lines · Ming Lei
With holding queue's kobject refcount, it is safe for driver to schedule requeue. However,
blk_mq_kick_requeue_list()may be called afterblk_sync_queue()is done because of concurrent requeue activities, then requeue work may not be completed when freeing queue, and kernel oops is triggered.So moving the cancel of
requeue_workintoblk_mq_release()for avoiding race between requeue and freeing queue.
Looks good,
PATCH v7 3/9 blk-mq: free hw queue's resource in hctx's release handler
In reply to Ming LeiOnce blk_cleanup_queue() returns, tags shouldn't be used any more, because blk_mq_free_tag_set() may be called. Commit 45a9c9d909b2 ("blk-mq: Fix a use-after-free") fixes this issue exactly.
However, that commit introduces another issue. Before 45a9c9d909b2, we are allowed to run queue during cleaning up queue if the queue's kobj refcount is held. After that commit, queue can't be run during queue cleaning up, otherwise oops can be triggered easily because some fields of hctx are freed by blk_mq_free_queue() in blk_cleanup_queue().
We have invented ways for addressing this kind of issue before, such as:
8dc765d438f1 ("SCSI: fix queue cleanup race before queue initialization is done") c2856ae2f315 ("blk-mq: quiesce queue before freeing queue")
But still can't cover all cases, recently James reports another such kind of issue:
https://marc.info/?l=linux-scsi&m=155389088124782&w=2
This issue can be quite hard to address by previous way, given scsi_run_queue() may run requeues for other LUNs.
Fixes the above issue by freeing hctx's resources in its release handler, and this way is safe becasue tags isn't needed for freeing such hctx resource.
This approach follows typical design pattern wrt. kobject's release handler.
--- block/blk-core.c | 2 +- block/blk-mq-sysfs.c | 6 ++++++ block/blk-mq.c | 8 ++------ block/blk-mq.h | 2 +- 4 files changed, 10 insertions(+), 8 deletions(-)
block/blk-core.c +1 -1
@@ -374,7 +374,7 @@ void blk_cleanup_queue(struct request_queue *q)blk_exit_queue (q);if (queue_is_mq (q)) -blk_mq_free_queue (q); +blk_mq_exit_queue (q);percpu_ref_exit (&q->q_usage_counter);
block/blk-mq-sysfs.c +6 -0
@@ -10,6 +10,7 @@#include <linux/smp.h> #include <linux/blk-mq.h> +#include "blk.h" #include "blk-mq.h" #include "blk-mq-tag.h"
@@ -33,6 +34,11 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj) { struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx, kobj); + +if (hctx->flags &BLK_MQ_F_BLOCKING ) +cleanup_srcu_struct (hctx->srcu); +blk_free_flush_queue (hctx->fq); +sbitmap_free (&hctx->ctx_map);free_cpumask_var (hctx->cpumask);kfree (hctx->ctxs);kfree (hctx);
block/blk-mq.c +2 -6
@@ -2273,12 +2273,7 @@ static void blk_mq_exit_hctx(struct request_queue *q,if (set->ops->exit_hctx) set->ops->exit_hctx (hctx, hctx_idx); -if (hctx->flags &BLK_MQ_F_BLOCKING ) -cleanup_srcu_struct (hctx->srcu); -blk_mq_remove_cpuhp (hctx); -blk_free_flush_queue (hctx->fq); -sbitmap_free (&hctx->ctx_map); } static void blk_mq_exit_hw_queues(struct request_queue *q,
@@ -2913,7 +2908,8 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, }EXPORT_SYMBOL (blk_mq_init_allocated_queue); -void blk_mq_free_queue (struct request_queue *q) +/* tags can _not_ be used after returning from blk_mq_exit_queue */ +void blk_mq_exit_queue (struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set;
block/blk-mq.h +1 -1
@@ -37,7 +37,7 @@ struct blk_mq_ctx {struct kobject kobj; }____cacheline_aligned_in_smp ; -void blk_mq_free_queue (struct request_queue *q); +void blk_mq_exit_queue (struct request_queue *q);int blk_mq_update_nr_requests (struct request_queue *q,unsigned int nr);void blk_mq_wake_waiters (struct request_queue *q);bool blk_mq_dispatch_rq_list (struct request_queue *,struct list_head *,bool );
Signature · 1 line
2.9.5
Re: [PATCH V7 3/9] blk-mq: free hw queue's resource in hctx's release handler
Looks good,
PATCH v7 4/9 blk-mq: move all hctx alloction & initialization into __blk_mq_alloc_and_init_hctx
In reply to Ming LeiMove all hctx related allocation and initialization into __blk_mq_alloc_and_init_hctx, and prepare for splitting blk_mq_alloc_and_init_hctx into real two functions, one is for allocate everything, and another is for initializing everyting.
--- block/blk-mq.c | 93 +++++++++++++++++++++++++++------------------------------- 1 file changed, 44 insertions(+), 49 deletions(-)
block/blk-mq.c +44 -49
@@ -2290,15 +2290,45 @@ static void blk_mq_exit_hw_queues(struct request_queue *q, } } -static int blk_mq_init_hctx (struct request_queue *q, -struct blk_mq_tag_set *set, -struct blk_mq_hw_ctx *hctx,unsigned hctx_idx) +static int blk_mq_hw_ctx_size (struct blk_mq_tag_set *tag_set) { -int node; +int hw_ctx_size =sizeof (struct blk_mq_hw_ctx ); + +BUILD_BUG_ON (ALIGN (offsetof(struct blk_mq_hw_ctx , srcu), + __alignof__(struct blk_mq_hw_ctx )) != +sizeof (struct blk_mq_hw_ctx )); + +if (tag_set->flags &BLK_MQ_F_BLOCKING ) + hw_ctx_size +=sizeof (struct srcu_struct ); + +return hw_ctx_size; +} + +static struct blk_mq_hw_ctx * +__blk_mq_alloc_and_init_hctx (struct request_queue *q, +struct blk_mq_tag_set *set, +unsigned hctx_idx,int node) +{ +struct blk_mq_hw_ctx *hctx; + + hctx =kzalloc_node (blk_mq_hw_ctx_size (set), +GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, + node); +if (!hctx) + goto fail_alloc_hctx; + +if (!zalloc_cpumask_var_node (&hctx->cpumask, +GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, + node)) + goto free_hctx; + +atomic_set (&hctx->nr_active,0 ); + hctx->numa_node = node; + hctx->queue_num = hctx_idx; - node = hctx->numa_node;if (node ==NUMA_NO_NODE ) - node = hctx->numa_node = set->numa_node; + hctx->numa_node = set->numa_node; + node = hctx->numa_node;INIT_DELAYED_WORK (&hctx->run_work, blk_mq_run_work_fn);spin_lock_init (&hctx->lock);
@@ -2343,8 +2373,9 @@ static int blk_mq_init_hctx(struct request_queue *q,if (hctx->flags &BLK_MQ_F_BLOCKING )init_srcu_struct (hctx->srcu); +blk_mq_hctx_kobj_init (hctx); -return 0 ; +return hctx; free_fq:blk_free_flush_queue (hctx->fq);
@@ -2357,7 +2388,11 @@ static int blk_mq_init_hctx(struct request_queue *q,kfree (hctx->ctxs); unregister_cpu_notifier:blk_mq_remove_cpuhp (hctx); -return -1 ; +free_cpumask_var (hctx->cpumask); + free_hctx: +kfree (hctx); + fail_alloc_hctx: +return NULL ; } static void blk_mq_init_cpu_queues(struct request_queue *q,
@@ -2703,51 +2738,11 @@ struct request_queue *blk_mq_init_sq_queue(struct blk_mq_tag_set *set, }EXPORT_SYMBOL (blk_mq_init_sq_queue); -static int blk_mq_hw_ctx_size (struct blk_mq_tag_set *tag_set) -{ -int hw_ctx_size =sizeof (struct blk_mq_hw_ctx ); - -BUILD_BUG_ON (ALIGN (offsetof(struct blk_mq_hw_ctx , srcu), - __alignof__(struct blk_mq_hw_ctx )) != -sizeof (struct blk_mq_hw_ctx )); - -if (tag_set->flags &BLK_MQ_F_BLOCKING ) - hw_ctx_size +=sizeof (struct srcu_struct ); - -return hw_ctx_size; -} -static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx (struct blk_mq_tag_set *set,struct request_queue *q,int hctx_idx,int node) { -struct blk_mq_hw_ctx *hctx; - - hctx =kzalloc_node (blk_mq_hw_ctx_size (set), -GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, - node); -if (!hctx) -return NULL ; - -if (!zalloc_cpumask_var_node (&hctx->cpumask, -GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, - node)) { -kfree (hctx); -return NULL ; - } - -atomic_set (&hctx->nr_active,0 ); - hctx->numa_node = node; - hctx->queue_num = hctx_idx; - -if (blk_mq_init_hctx (q, set, hctx, hctx_idx)) { -free_cpumask_var (hctx->cpumask); -kfree (hctx); -return NULL ; - } -blk_mq_hctx_kobj_init (hctx); - -return hctx; +return __blk_mq_alloc_and_init_hctx (q, set, hctx_idx, node); } static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
Signature · 1 line
2.9.5
Christoph Hellwig
maintainerRe: [PATCH V7 4/9] blk-mq: move all hctx alloction & initialization into __blk_mq_alloc_and_init_hctx
Quoted 50 lines · Ming Lei
Move all hctx related allocation and initialization into
__blk_mq_alloc_and_init_hctx, and prepare for splittingblk_mq_alloc_and_init_hctxinto real two functions, one is for allocate everything, and another is for initializing everyting.Cc Dongli Zhang <dongli.zhang at oracle.com>Cc James Smart <james.smart at broadcom.com>Cc Bart Van Assche <bart.vanassche at wdc.com>Cc linux-scsi at vger.kernel.org,Cc Martin K . Petersen <martin.petersen at oracle.com>,Cc Christoph Hellwig <hch at lst.de>,Cc James E . J . Bottomley <jejb at linux.vnet.ibm.com>,Reviewed-by Hannes Reinecke <hare at suse.com>Tested-by James Smart <james.smart at broadcom.com>Signed-off-by Ming Lei <ming.lei at redhat.com>--- block/blk-mq.c | 93 +++++++++++++++++++++++++++------------------------------- 1 file changed, 44 insertions(+), 49 deletions(-)block/blk-mq.c +18
-4@@ -2290,15 +2290,45 @@ static void blk_mq_exit_hw_queues(struct request_queue *q, } } -static int blk_mq_init_hctx (struct request_queue *q, -struct blk_mq_tag_set *set, -struct blk_mq_hw_ctx *hctx,unsigned hctx_idx) +static int blk_mq_hw_ctx_size (struct blk_mq_tag_set *tag_set) { -int node; +int hw_ctx_size =sizeof (struct blk_mq_hw_ctx ); + +BUILD_BUG_ON (ALIGN (offsetof(struct blk_mq_hw_ctx , srcu), + __alignof__(struct blk_mq_hw_ctx )) != +sizeof (struct blk_mq_hw_ctx )); + +if (tag_set->flags &BLK_MQ_F_BLOCKING ) + hw_ctx_size +=sizeof (struct srcu_struct ); + +return hw_ctx_size; +} + +static struct blk_mq_hw_ctx * +__blk_mq_alloc_and_init_hctx (struct request_queue *q, +struct blk_mq_tag_set *set, +unsigned hctx_idx,int node) +{
block/blk-mq.c static void blk_mq_exit_hw_queues(struct request_queue *q, · line 2321
Quoted 10 lines
block/blk-mq.c +10
-0+struct blk_mq_hw_ctx *hctx; + + hctx =kzalloc_node (blk_mq_hw_ctx_size (set), +GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, + node); +if (!hctx) + goto fail_alloc_hctx; + + if (!zalloc_cpumask_var_node(&hctx->cpumask, + GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY,
Maybe throw in a
const gfp_t gfp = GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY;
as the first line to shorten these two a bit..
block/blk-mq.c struct request_queue *blk_mq_init_sq_queue(struct blk_mq_tag_set *set, · line 2746
Quoted 6 lines
block/blk-mq.c +1
-0static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx (struct blk_mq_tag_set *set,struct request_queue *q,int hctx_idx,int node) { +return __blk_mq_alloc_and_init_hctx (q, set, hctx_idx, node); }
I really don't the point of this wrapper. If we need it later we should introduce it there.
Re: [PATCH V7 4/9] blk-mq: move all hctx alloction & initialization into __blk_mq_alloc_and_init_hctx
Quoted 68 lines · Christoph Hellwig
Quoted 50 lines · Ming Lei
Move all hctx related allocation and initialization into
__blk_mq_alloc_and_init_hctx, and prepare for splittingblk_mq_alloc_and_init_hctxinto real two functions, one is for allocate everything, and another is for initializing everyting.Cc Dongli Zhang <dongli.zhang at oracle.com>Cc James Smart <james.smart at broadcom.com>Cc Bart Van Assche <bart.vanassche at wdc.com>Cc linux-scsi at vger.kernel.org,Cc Martin K . Petersen <martin.petersen at oracle.com>,Cc Christoph Hellwig <hch at lst.de>,Cc James E . J . Bottomley <jejb at linux.vnet.ibm.com>,Reviewed-by Hannes Reinecke <hare at suse.com>Tested-by James Smart <james.smart at broadcom.com>Signed-off-by Ming Lei <ming.lei at redhat.com>--- block/blk-mq.c | 93 +++++++++++++++++++++++++++------------------------------- 1 file changed, 44 insertions(+), 49 deletions(-)block/blk-mq.c +18
-4@@ -2290,15 +2290,45 @@ static void blk_mq_exit_hw_queues(struct request_queue *q, } } -static int blk_mq_init_hctx (struct request_queue *q, -struct blk_mq_tag_set *set, -struct blk_mq_hw_ctx *hctx,unsigned hctx_idx) +static int blk_mq_hw_ctx_size (struct blk_mq_tag_set *tag_set) { -int node; +int hw_ctx_size =sizeof (struct blk_mq_hw_ctx ); + +BUILD_BUG_ON (ALIGN (offsetof(struct blk_mq_hw_ctx , srcu), + __alignof__(struct blk_mq_hw_ctx )) != +sizeof (struct blk_mq_hw_ctx )); + +if (tag_set->flags &BLK_MQ_F_BLOCKING ) + hw_ctx_size +=sizeof (struct srcu_struct ); + +return hw_ctx_size; +} + +static struct blk_mq_hw_ctx * +__blk_mq_alloc_and_init_hctx (struct request_queue *q, +struct blk_mq_tag_set *set, +unsigned hctx_idx,int node) +{block/blk-mq.c static void blk_mq_exit_hw_queues(struct request_queue *q, · line 2321
Quoted 10 lines
block/blk-mq.c +10
-0+struct blk_mq_hw_ctx *hctx; + + hctx =kzalloc_node (blk_mq_hw_ctx_size (set), +GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, + node); +if (!hctx) + goto fail_alloc_hctx; + + if (!zalloc_cpumask_var_node(&hctx->cpumask, + GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY,Maybe throw in a
const gfp_t gfp = GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY;as the first line to shorten these two a bit..
Quoted 9 lines
block/blk-mq.c struct request_queue *blk_mq_init_sq_queue(struct blk_mq_tag_set *set, · line 2746
Quoted 6 lines
block/blk-mq.c +1
-0static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx (struct blk_mq_tag_set *set,struct request_queue *q,int hctx_idx,int node) { +return __blk_mq_alloc_and_init_hctx (q, set, hctx_idx, node); }I really don't the point of this wrapper. If we need it later we should introduce it there.
It prepares for the following patch, otherwise the whole patch may become difficult to review.
thanks,
Ming
Christoph Hellwig
maintainerRe: [PATCH V7 4/9] blk-mq: move all hctx alloction & initialization into __blk_mq_alloc_and_init_hctx
Quoted 2 lines · Ming Lei
It prepares for the following patch, otherwise the whole patch may become difficult to review.
Honestly, I'd find it much easier to review if this was folded into the next patch. That way we only move things around once instead of twice.
Re: [PATCH V7 4/9] blk-mq: move all hctx alloction & initialization into __blk_mq_alloc_and_init_hctx
In reply to Christoph HellwigQuoted 64 lines · Christoph Hellwig
Quoted 50 lines · Ming Lei
Move all hctx related allocation and initialization into
__blk_mq_alloc_and_init_hctx, and prepare for splittingblk_mq_alloc_and_init_hctxinto real two functions, one is for allocate everything, and another is for initializing everyting.Cc Dongli Zhang <dongli.zhang at oracle.com>Cc James Smart <james.smart at broadcom.com>Cc Bart Van Assche <bart.vanassche at wdc.com>Cc linux-scsi at vger.kernel.org,Cc Martin K . Petersen <martin.petersen at oracle.com>,Cc Christoph Hellwig <hch at lst.de>,Cc James E . J . Bottomley <jejb at linux.vnet.ibm.com>,Reviewed-by Hannes Reinecke <hare at suse.com>Tested-by James Smart <james.smart at broadcom.com>Signed-off-by Ming Lei <ming.lei at redhat.com>--- block/blk-mq.c | 93 +++++++++++++++++++++++++++------------------------------- 1 file changed, 44 insertions(+), 49 deletions(-)block/blk-mq.c +18
-4@@ -2290,15 +2290,45 @@ static void blk_mq_exit_hw_queues(struct request_queue *q, } } -static int blk_mq_init_hctx (struct request_queue *q, -struct blk_mq_tag_set *set, -struct blk_mq_hw_ctx *hctx,unsigned hctx_idx) +static int blk_mq_hw_ctx_size (struct blk_mq_tag_set *tag_set) { -int node; +int hw_ctx_size =sizeof (struct blk_mq_hw_ctx ); + +BUILD_BUG_ON (ALIGN (offsetof(struct blk_mq_hw_ctx , srcu), + __alignof__(struct blk_mq_hw_ctx )) != +sizeof (struct blk_mq_hw_ctx )); + +if (tag_set->flags &BLK_MQ_F_BLOCKING ) + hw_ctx_size +=sizeof (struct srcu_struct ); + +return hw_ctx_size; +} + +static struct blk_mq_hw_ctx * +__blk_mq_alloc_and_init_hctx (struct request_queue *q, +struct blk_mq_tag_set *set, +unsigned hctx_idx,int node) +{block/blk-mq.c static void blk_mq_exit_hw_queues(struct request_queue *q, · line 2321
Quoted 10 lines
block/blk-mq.c +10
-0+struct blk_mq_hw_ctx *hctx; + + hctx =kzalloc_node (blk_mq_hw_ctx_size (set), +GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, + node); +if (!hctx) + goto fail_alloc_hctx; + + if (!zalloc_cpumask_var_node(&hctx->cpumask, + GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY,Maybe throw in a
Quoted 3 lines
const gfp_t gfp = GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY;as the first line to shorten these two a bit..
That is fine, but could be a cleanup.
Quoted 10 lines
block/blk-mq.c struct request_queue *blk_mq_init_sq_queue(struct blk_mq_tag_set *set, · line 2746
Quoted 7 lines
block/blk-mq.c +1
-0static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx (struct blk_mq_tag_set *set,struct request_queue *q,int hctx_idx,int node) { +return __blk_mq_alloc_and_init_hctx (q, set, hctx_idx, node); }I really don't the point of this wrapper. If we need it later we should introduce it there.
This patch prepares for the conversion, especially for the next two.
Thanks,
Ming
PATCH v7 5/9 blk-mq: split blk_mq_alloc_and_init_hctx into two parts
In reply to Ming LeiSplit blk_mq_alloc_and_init_hctx into two parts, and one is blk_mq_alloc_hctx() which is for allocating all hctx resources, another is blk_mq_init_hctx() which is for initializing hctx, and serves as counter-part of blk_mq_exit_hctx().
--- block/blk-mq.c | 76 +++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 49 insertions(+), 27 deletions(-)
block/blk-mq.c +49 -27
@@ -2304,10 +2304,38 @@ static int blk_mq_hw_ctx_size(struct blk_mq_tag_set *tag_set)return hw_ctx_size; } +static int blk_mq_init_hctx (struct request_queue *q, +struct blk_mq_tag_set *set, +struct blk_mq_hw_ctx *hctx, +unsigned hctx_idx) +{ + hctx->queue_num = hctx_idx; + +cpuhp_state_add_instance_nocalls (CPUHP_BLK_MQ_DEAD , &hctx->cpuhp_dead); + + hctx->tags = set->tags[hctx_idx]; + +if (set->ops->init_hctx && + set->ops->init_hctx (hctx, set->driver_data, hctx_idx)) + goto unregister_cpu_notifier; + +if (blk_mq_init_request (set, hctx->fq->flush_rq, hctx_idx, + hctx->numa_node)) + goto exit_hctx; +return 0 ; + + exit_hctx: +if (set->ops->exit_hctx) + set->ops->exit_hctx (hctx, hctx_idx); + unregister_cpu_notifier: +blk_mq_remove_cpuhp (hctx); +return -1 ; +} +static struct blk_mq_hw_ctx * -__blk_mq_alloc_and_init_hctx (struct request_queue *q, -struct blk_mq_tag_set *set, -unsigned hctx_idx,int node) +blk_mq_alloc_hctx (struct request_queue *q, +struct blk_mq_tag_set *set, +unsigned hctx_idx,int node) {struct blk_mq_hw_ctx *hctx;
@@ -2324,8 +2352,6 @@ __blk_mq_alloc_and_init_hctx(struct request_queue *q,atomic_set (&hctx->nr_active,0 ); hctx->numa_node = node; - hctx->queue_num = hctx_idx; -if (node ==NUMA_NO_NODE ) hctx->numa_node = set->numa_node; node = hctx->numa_node;
@@ -2336,10 +2362,6 @@ __blk_mq_alloc_and_init_hctx(struct request_queue *q, hctx->queue = q; hctx->flags = set->flags & ~BLK_MQ_F_TAG_SHARED ; -cpuhp_state_add_instance_nocalls (CPUHP_BLK_MQ_DEAD , &hctx->cpuhp_dead); - - hctx->tags = set->tags[hctx_idx]; -/* * Allocate space for all possible cpus to avoid allocation at * runtime
@@ -2347,29 +2369,21 @@ __blk_mq_alloc_and_init_hctx(struct request_queue *q, hctx->ctxs =kmalloc_array_node (nr_cpu_ids,sizeof (void *),GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, node);if (!hctx->ctxs) - goto unregister_cpu_notifier; + goto free_cpumask;if (sbitmap_init_node (&hctx->ctx_map, nr_cpu_ids,ilog2 (8 ),GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, node)) goto free_ctxs; - hctx->nr_ctx =0 ;spin_lock_init (&hctx->dispatch_wait_lock);init_waitqueue_func_entry (&hctx->dispatch_wait, blk_mq_dispatch_wake);INIT_LIST_HEAD (&hctx->dispatch_wait.entry); -if (set->ops->init_hctx && - set->ops->init_hctx (hctx, set->driver_data, hctx_idx)) - goto free_bitmap; - hctx->fq =blk_alloc_flush_queue (q, hctx->numa_node, set->cmd_size,GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY);if (!hctx->fq) - goto exit_hctx; - -if (blk_mq_init_request (set, hctx->fq->flush_rq, hctx_idx, node)) - goto free_fq; + goto free_bitmap;if (hctx->flags &BLK_MQ_F_BLOCKING )init_srcu_struct (hctx->srcu);
@@ -2377,17 +2391,11 @@ __blk_mq_alloc_and_init_hctx(struct request_queue *q,return hctx; - free_fq: -blk_free_flush_queue (hctx->fq); - exit_hctx: -if (set->ops->exit_hctx) - set->ops->exit_hctx (hctx, hctx_idx); free_bitmap:sbitmap_free (&hctx->ctx_map); free_ctxs:kfree (hctx->ctxs); - unregister_cpu_notifier: -blk_mq_remove_cpuhp (hctx); + free_cpumask:free_cpumask_var (hctx->cpumask); free_hctx:kfree (hctx);
@@ -2742,7 +2750,21 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx( struct blk_mq_tag_set *set, struct request_queue *q, int hctx_idx, int node) { - return __blk_mq_alloc_and_init_hctx(q, set, hctx_idx, node); + struct blk_mq_hw_ctx *hctx; + + hctx =blk_mq_alloc_hctx (q, set, hctx_idx, node); +if (!hctx) + goto fail; + +if (blk_mq_init_hctx (q, set, hctx, hctx_idx)) + goto free_hctx; + +return hctx; + + free_hctx: +kobject_put (&hctx->kobj); + fail: +return NULL ; } static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
Signature · 1 line
2.9.5
PATCH v7 6/9 blk-mq: always free hctx after request queue is freed
In reply to Ming LeiIn normal queue cleanup path, hctx is released after request queue is freed, see blk_mq_release().
However, in __blk_mq_update_nr_hw_queues(), hctx may be freed because of hw queues shrinking. This way is easy to cause use-after-free,
because: one implicit rule is that it is safe to call almost all block layer APIs if the request queue is alive; and one hctx may be retrieved by one API, then the hctx can be freed by blk_mq_update_nr_hw_queues(); finally use-after-free is triggered.
Fixes this issue by always freeing hctx after releasing request queue. If some hctxs are removed in blk_mq_update_nr_hw_queues(), introduce a per-queue list to hold them, then try to resuse these hctxs if numa node is matched.
--- block/blk-mq.c | 46 +++++++++++++++++++++++++++++++++------------- include/linux/blk-mq.h | 2 ++ include/linux/blkdev.h | 7 +++++++ 3 files changed, 42 insertions(+), 13 deletions(-)
block/blk-mq.c +33 -13
@@ -2274,6 +2274,10 @@ static void blk_mq_exit_hctx(struct request_queue *q, set->ops->exit_hctx (hctx, hctx_idx);blk_mq_remove_cpuhp (hctx); + +spin_lock (&q->unused_hctx_lock); +list_add (&hctx->hctx_list, &q->unused_hctx_list); +spin_unlock (&q->unused_hctx_lock); } static void blk_mq_exit_hw_queues(struct request_queue *q,
@@ -2362,6 +2366,8 @@ blk_mq_alloc_hctx(struct request_queue *q, hctx->queue = q; hctx->flags = set->flags & ~BLK_MQ_F_TAG_SHARED ; +INIT_LIST_HEAD (&hctx->hctx_list); +/* * Allocate space for all possible cpus to avoid allocation at * runtime
@@ -2675,15 +2681,17 @@ static int blk_mq_alloc_ctxs(struct request_queue *q)*/ void blk_mq_release (struct request_queue *q) { -struct blk_mq_hw_ctx *hctx; -unsigned int i; +struct blk_mq_hw_ctx *hctx, *next; +int i;cancel_delayed_work_sync (&q->requeue_work); -/* hctx kobj stays in hctx */ -queue_for_each_hw_ctx (q, hctx, i) { -if (!hctx) -continue ; +queue_for_each_hw_ctx (q, hctx, i) +WARN_ON_ONCE (hctx &&list_empty (&hctx->hctx_list)); + +/* all hctx are in .unused_hctx_list now */ +list_for_each_entry_safe (hctx, next, &q->unused_hctx_list, hctx_list) { +list_del_init (&hctx->hctx_list);kobject_put (&hctx->kobj); }
@@ -2750,9 +2758,22 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx( struct blk_mq_tag_set *set, struct request_queue *q, int hctx_idx, int node) { - struct blk_mq_hw_ctx *hctx; + struct blk_mq_hw_ctx *hctx = NULL, *tmp; - hctx =blk_mq_alloc_hctx (q, set, hctx_idx, node); +/* reuse dead hctx first */ +spin_lock (&q->unused_hctx_lock); +list_for_each_entry (tmp, &q->unused_hctx_list, hctx_list) { +if (tmp->numa_node == node) { + hctx = tmp; +break ; + } + } +if (hctx) +list_del_init (&hctx->hctx_list); +spin_unlock (&q->unused_hctx_lock); + +if (!hctx) + hctx =blk_mq_alloc_hctx (q, set, hctx_idx, node);if (!hctx) goto fail;
@@ -2790,10 +2811,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, hctx =blk_mq_alloc_and_init_hctx (set, q, i, node);if (hctx) { -if (hctxs[i]) { +if (hctxs[i])blk_mq_exit_hctx (q, set, hctxs[i], i); -kobject_put (&hctxs[i]->kobj); - } hctxs[i] = hctx; }else {if (hctxs[i])
@@ -2824,9 +2843,7 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,if (hctx->tags)blk_mq_free_map_and_requests (set, j);blk_mq_exit_hctx (q, set, hctx, j); -kobject_put (&hctx->kobj); hctxs[j] =NULL ; - } }mutex_unlock (&q->sysfs_lock);
@@ -2869,6 +2886,9 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,if (!q->queue_hw_ctx) goto err_sys_init; +INIT_LIST_HEAD (&q->unused_hctx_list); +spin_lock_init (&q->unused_hctx_lock); +blk_mq_realloc_hw_ctxs (set, q);if (!q->nr_hw_queues) goto err_hctxs;
include/linux/blk-mq.h +2 -0
@@ -70,6 +70,8 @@ struct blk_mq_hw_ctx {struct dentry *sched_debugfs_dir;#endif +struct list_head hctx_list; +/* Must be the last member - see also blk_mq_hw_ctx_size(). */ struct srcu_struct srcu[0 ]; };
include/linux/blkdev.h +7 -0
@@ -535,6 +535,13 @@ struct request_queue {struct mutex sysfs_lock; +/* +* for reusing dead hctx instance in case of updating +* nr_hw_queues +*/ +struct list_head unused_hctx_list; +spinlock_t unused_hctx_lock; +atomic_t mq_freeze_depth;#if defined(CONFIG_BLK_DEV_BSG )
Signature · 1 line
2.9.5
Re: [PATCH V7 6/9] blk-mq: always free hctx after request queue is freed
Quoted 68 lines · Ming Lei
In normal queue cleanup path, hctx is released after request queue is freed, see
blk_mq_release().However, in
__blk_mq_update_nr_hw_queues(), hctx may be freed because of hw queues shrinking. This way is easy to cause use-after-free,
because: one implicit rule is that it is safe to call almost all block layer APIs if the request queue is alive; and one hctx may be retrieved by one API, then the hctx can be freed byblk_mq_update_nr_hw_queues(); finally use-after-free is triggered.Fixes this issue by always freeing hctx after releasing request queue. If some hctxs are removed in
blk_mq_update_nr_hw_queues(), introduce a per-queue list to hold them, then try to resuse these hctxs if numa node is matched.Cc Dongli Zhang <dongli.zhang at oracle.com>Cc James Smart <james.smart at broadcom.com>Cc Bart Van Assche <bart.vanassche at wdc.com>Cc linux-scsi at vger.kernel.org,Cc Martin K . Petersen <martin.petersen at oracle.com>,Cc Christoph Hellwig <hch at lst.de>,Cc James E . J . Bottomley <jejb at linux.vnet.ibm.com>,Tested-by James Smart <james.smart at broadcom.com>Signed-off-by Ming Lei <ming.lei at redhat.com>--- block/blk-mq.c | 46 +++++++++++++++++++++++++++++++++------------- include/linux/blk-mq.h | 2 ++ include/linux/blkdev.h | 7 +++++++ 3 files changed, 42 insertions(+), 13 deletions(-)block/blk-mq.c +8
-5@@ -2274,6 +2274,10 @@ static void blk_mq_exit_hctx(struct request_queue *q, set->ops->exit_hctx (hctx, hctx_idx);blk_mq_remove_cpuhp (hctx); + +spin_lock (&q->unused_hctx_lock); +list_add (&hctx->hctx_list, &q->unused_hctx_list); +spin_unlock (&q->unused_hctx_lock); } static void blk_mq_exit_hw_queues(struct request_queue *q,@@ -2362,6 +2366,8 @@ blk_mq_alloc_hctx(struct request_queue *q, hctx->queue = q; hctx->flags = set->flags & ~BLK_MQ_F_TAG_SHARED ; +INIT_LIST_HEAD (&hctx->hctx_list); +/* * Allocate space for all possible cpus to avoid allocation at * runtime @@ -2675,15 +2681,17 @@ static int blk_mq_alloc_ctxs(struct request_queue *q)*/ void blk_mq_release (struct request_queue *q) { -struct blk_mq_hw_ctx *hctx; -unsigned int i; +struct blk_mq_hw_ctx *hctx, *next; +int i;cancel_delayed_work_sync (&q->requeue_work); -/* hctx kobj stays in hctx */ -queue_for_each_hw_ctx (q, hctx, i) { -if (!hctx)
block/blk-mq.c static int blk_mq_alloc_ctxs(struct request_queue *q) · line 2697
Quoted 10 lines
block/blk-mq.c +6
-1-continue ; +queue_for_each_hw_ctx (q, hctx, i) +WARN_ON_ONCE (hctx &&list_empty (&hctx->hctx_list)); + +/* all hctx are in .unused_hctx_list now */ +list_for_each_entry_safe (hctx, next, &q->unused_hctx_list, hctx_list) { +list_del_init (&hctx->hctx_list);kobject_put (&hctx->kobj); }
I would consider it a bug if the hctx is still assigned to ->queue_hw_ctx at this point.
But that's a minor issue.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage hare at suse.com +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: F. Imend?rffer, J. Smithard, D. Upmanyu, G. Norton HRB 21284 (AG N?rnberg)
PATCH v7 7/9 blk-mq: move cancel of hctx->run_work into blk_mq_hw_sysfs_release
In reply to Ming Leihctx is always released after requeue is freed.
With holding queue's kobject refcount, it is safe for driver to run queue, so one run queue might be scheduled after blk_sync_queue() is done.
So moving the cancel of hctx->run_work into blk_mq_hw_sysfs_release() for avoiding run released queue.
--- block/blk-core.c | 8 -------- block/blk-mq-sysfs.c | 2 ++ 2 files changed, 2 insertions(+), 8 deletions(-)
block/blk-core.c +0 -8
@@ -232,14 +232,6 @@ void blk_sync_queue(struct request_queue *q) {del_timer_sync (&q->timeout);cancel_work_sync (&q->timeout_work); - -if (queue_is_mq (q)) { -struct blk_mq_hw_ctx *hctx; -int i; - -queue_for_each_hw_ctx (q, hctx, i) -cancel_delayed_work_sync (&hctx->run_work); - } }EXPORT_SYMBOL (blk_sync_queue);
block/blk-mq-sysfs.c +2 -0
@@ -35,6 +35,8 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj) struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx, kobj); +cancel_delayed_work_sync (&hctx->run_work); +if (hctx->flags &BLK_MQ_F_BLOCKING )cleanup_srcu_struct (hctx->srcu);blk_free_flush_queue (hctx->fq);
Signature · 1 line
2.9.5
PATCH v7 8/9 block: don't drain in-progress dispatch in blk_cleanup_queue()
In reply to Ming LeiNow freeing hw queue resource is moved to hctx's release handler, we don't need to worry about the race between blk_cleanup_queue and run queue any more.
So don't drain in-progress dispatch in blk_cleanup_queue().
This is basically revert of c2856ae2f315 ("blk-mq: quiesce queue before freeing queue").
--- block/blk-core.c | 12 ------------ 1 file changed, 12 deletions(-)
block/blk-core.c +0 -12
@@ -338,18 +338,6 @@ void blk_cleanup_queue(struct request_queue *q)blk_queue_flag_set (QUEUE_FLAG_DEAD , q); -/* -* make sure all in-progress dispatch are completed because -* blk_freeze_queue() can only complete all requests, and -* dispatch may still be in-progress since we dispatch requests -* from more than one contexts. -* -* We rely on driver to deal with the race in case that queue -* initialization isn't done. -*/ -if (queue_is_mq (q) &&blk_queue_init_done (q)) -blk_mq_quiesce_queue (q); -/* for synchronous bio-based driver finish in-flight integrity i/o */ blk_flush_integrity ();
Signature · 1 line
2.9.5
PATCH v7 9/9 nvme: hold request queue's refcount in ns's whole lifetime
In reply to Ming LeiHennes reported the following kernel oops:
There is a race condition between namespace rescanning and controller reset; during controller reset all namespaces are quiesed vie nams_stop_ctrl(), and after reset all namespaces are unquiesced again.
When namespace scanning was active by the time controller reset was triggered the rescan code will call nvme_ns_remove(), which then will cause a kernel crash in nvme_start_ctrl() as it'll trip over uninitialized namespaces.
Patch "blk-mq: free hw queue's resource in hctx's release handler" should make this issue quite difficult to trigger. However it can't kill the issue completely becasue pre-condition of that patch is to hold request queue's refcount before calling block layer API, and there is still a small window between blk_cleanup_queue() and removing the ns from the controller namspace list in nvme_ns_remove().
Hold request queue's refcount until the ns is freed, then the above race can be avoided completely. Given the 'namespaces_rwsem' is always held to retrieve ns for starting/stopping request queue, this lock can prevent namespaces from being freed.
--- drivers/nvme/host/core.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +9 -1
@@ -406,6 +406,7 @@ static void nvme_free_ns(struct kref *kref)nvme_nvm_unregister (ns);put_disk (ns->disk); +blk_put_queue (ns->queue);nvme_put_ns_head (ns->head);nvme_put_ctrl (ns->ctrl);kfree (ns);
@@ -3229,6 +3230,11 @@ static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) goto out_free_ns; } +if (!blk_get_queue (ns->queue)) { + ret = -ENXIO ; + goto out_free_queue; + } +blk_queue_flag_set (QUEUE_FLAG_NONROT , ns->queue);if (ctrl->ops->flags &NVME_F_PCI_P2PDMA )blk_queue_flag_set (QUEUE_FLAG_PCI_P2PDMA , ns->queue);
@@ -3245,7 +3251,7 @@ static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) id =nvme_identify_ns (ctrl, nsid);if (!id) { ret = -EIO ; - goto out_free_queue; + goto out_put_queue; }if (id->ncap ==0 ) {
@@ -3304,6 +3310,8 @@ static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)nvme_put_ns_head (ns->head); out_free_id:kfree (id); + out_put_queue: +blk_put_queue (ns->queue); out_free_queue:blk_cleanup_queue (ns->queue); out_free_ns:
Signature · 1 line
2.9.5
Christoph Hellwig
maintainerRe: [PATCH V7 9/9] nvme: hold request queue's refcount in ns's whole lifetime
Quoted 1 line · Ming Lei
Hennes reported the following kernel oops:
Hannes?
drivers/nvme/host/core.c static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) · line 3236
Quoted 4 lines
drivers/nvme/host/core.c +4
-0+if (!blk_get_queue (ns->queue)) { + ret = -ENXIO ; + goto out_free_queue; + }
If we always need to hold a reference, shouldn't blk_mq_init_queue return with that reference held (and yes, that means changes to every driver, but it seems like we need to audit all of them anyway..)
It seems like the queue lifetimes are a bit of a mess, and I'm not sure if this just papers over the problem.
Re: [PATCH V7 9/9] nvme: hold request queue's refcount in ns's whole lifetime
Quoted 5 lines · Christoph Hellwig
Quoted 1 line · Ming Lei
Hennes reported the following kernel oops:
Hannes?
Quoted 8 lines
drivers/nvme/host/core.c static int nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) · line 3236
Quoted 4 lines
drivers/nvme/host/core.c +4
-0+if (!blk_get_queue (ns->queue)) { + ret = -ENXIO ; + goto out_free_queue; + }If we always need to hold a reference, shouldn't
blk_mq_init_queuereturn with that reference held (and yes, that means changes to every driver, but it seems like we need to audit all of them anyway..)
The issue is driver(NVMe) specific, the race window is just between between blk_cleanup_queue() and removing the ns from the controller namspace list in nvme_ns_remove()
blk_mq_init_queue() does hold one refcount, and its counter-part is blk_cleanup_queue().
It is simply ugly to ask blk_mq_init_queue() to grab a refcnt for driver, then who is the counter-part for releasing the extra refcount?
Quoted 3 lines
It seems like the queue lifetimes are a bit of a mess, and I'm not sure if this just papers over the problem.
Could you explain a bit what the mess is?
Thanks,
Ming
Christoph Hellwig
maintainerRe: [PATCH V7 9/9] nvme: hold request queue's refcount in ns's whole lifetime
Quoted 3 lines · Ming Lei
The issue is driver(NVMe) specific, the race window is just between between
blk_cleanup_queue()and removing the ns from the controller namspace list innvme_ns_remove()
And I wouldn't be surprised if others have the same issue.
Quoted 6 lines
blk_mq_init_queue()does hold one refcount, and its counter-part isblk_cleanup_queue().It is simply ugly to ask
blk_mq_init_queue()to grab a refcnt for driver, then who is the counter-part for releasing the extra refcount?
Well, the problem is exactly that blk_cleanup_queue drops the reference. If move the blk_put_queue() call from the end of it to the callers the callers can keep the reference as long as they need them, and we wouldn't need an extra reference.
Re: [PATCH V7 9/9] nvme: hold request queue's refcount in ns's whole lifetime
Quoted 11 lines · Christoph Hellwig
Quoted 3 lines · Ming Lei
The issue is driver(NVMe) specific, the race window is just between between
blk_cleanup_queue()and removing the ns from the controller namspace list innvme_ns_remove()And I wouldn't be surprised if others have the same issue.
Quoted 4 lines
blk_mq_init_queue()does hold one refcount, and its counter-part isblk_cleanup_queue().
Quoted 7 lines
Quoted 2 lines
It is simply ugly to ask
blk_mq_init_queue()to grab a refcnt for driver, then who is the counter-part for releasing the extra refcount?Well, the problem is exactly that
blk_cleanup_queuedrops the reference. If move theblk_put_queue()call from the end of it to the callers the callers can keep the reference as long as they need them, and we wouldn't need an extra reference.
Hi Christoph,
There are more than hundred callers of blk_cleanup_queue() so that change would cause a lot of churn. Since blk_get_queue() and blk_put_queue() are available, how inserting a pair of calls to these functions where necessary?
Thanks,
Bart.
Re: [PATCH V7 9/9] nvme: hold request queue's refcount in ns's whole lifetime
Quoted 20 lines · Bart Van Assche
Quoted 11 lines · Christoph Hellwig
Quoted 3 lines · Ming Lei
The issue is driver(NVMe) specific, the race window is just between between
blk_cleanup_queue()and removing the ns from the controller namspace list innvme_ns_remove()And I wouldn't be surprised if others have the same issue.
Quoted 4 lines
blk_mq_init_queue()does hold one refcount, and its counter-part isblk_cleanup_queue().Quoted 7 lines
Quoted 2 lines
It is simply ugly to ask
blk_mq_init_queue()to grab a refcnt for driver, then who is the counter-part for releasing the extra refcount?Well, the problem is exactly that
blk_cleanup_queuedrops the reference. If move theblk_put_queue()call from the end of it to the callers the callers can keep the reference as long as they need them, and we wouldn't need an extra reference.
Quoted 5 lines
Hi Christoph,
There are more than hundred callers of
blk_cleanup_queue()so that change would cause a lot of churn. Sinceblk_get_queue()andblk_put_queue()are available, how inserting a pair of calls to these functions where necessary?
The problem is that queue might be used after blk_cleanup_queue() is returned by some drivers.
Gendisk is removed before cleanup queue, and the other activities on queue depends on driver itself. There can't be universal way to deal with that.
Thanks,
Ming
Re: [PATCH V7 9/9] nvme: hold request queue's refcount in ns's whole lifetime
In reply to Christoph HellwigQuoted 6 lines · Christoph Hellwig
Quoted 3 lines · Ming Lei
The issue is driver(NVMe) specific, the race window is just between between
blk_cleanup_queue()and removing the ns from the controller namspace list innvme_ns_remove()And I wouldn't be surprised if others have the same issue.
SCSI hasn't such issue, and loop/null/virtio-blk doesn't too.
Quoted 5 lines
Quoted 5 lines
blk_mq_init_queue()does hold one refcount, and its counter-part isblk_cleanup_queue().
Quoted 7 lines
Quoted 2 lines
It is simply ugly to ask
blk_mq_init_queue()to grab a refcnt for driver, then who is the counter-part for releasing the extra refcount?Well, the problem is exactly that
blk_cleanup_queuedrops the reference. If move theblk_put_queue()call from the end of it to the callers the callers can keep the reference as long as they need them, and we wouldn't need an extra reference.
It depends on driver.
Still no difference between your suggestion and the way in this patch, given driver specific change is a must. Even it is more clean to hold the queue refocunt by drivers explicitly because we usually do the get/put pair in one place, instead that getting refcnt in one subsystem, and doing put in another place.
I am going to drop this patch in V8 since my original plan is to fix block layer's race in this patchset.
Thanks,
Ming
Christoph Hellwig
maintainerRe: [PATCH V7 9/9] nvme: hold request queue's refcount in ns's whole lifetime
Quoted 8 lines · Ming Lei
Still no difference between your suggestion and the way in this patch, given driver specific change is a must. Even it is more clean to hold the queue refocunt by drivers explicitly because we usually do the get/put pair in one place, instead that getting refcnt in one subsystem, and doing put in another place.
I am going to drop this patch in V8 since my original plan is to fix block layer's race in this patchset.
Doing the untegistration and final reference drop is just a bad pattern, leading to possible bugs. I'm fine with you just dropping it for now, I guess I'll just send a scripted conversion moving the put to the drivers to Jens after -rc1 to sort it out. Probably including a rename of blk_cleanup_queue so that everyone notices the difference.