blk-mq: fix races related with freeing queue
Subsystems (3)
v9 cover blk-mq: fix races related with freeing queue
Hi,
Since 45a9c9d909b2 ("blk-mq: Fix a use-after-free"), run queue isn't allowed during cleanup queue even though queue refcount is held.
This change has caused lots of kernel oops triggered in run queue path, turns out it isn't easy to fix them all.
So move freeing of hw queue resources into hctx's release handler, then the above issue is fixed. Meantime, this way is safe given freeing hw queue resource doesn't require tags.
V3 covers more races.
V9:
- fix one issue in patch 1 of V8 by put the queue usage counter in early return branch
- make patch 4 of V8 a bit clean, as suggested by Christoph and Hannes
V8:
- merge the 4th and 5th of V7 into one patch, as suggested by Christoph
- drop the 9th patch
V7:
- add reviewed-by and tested-by tag
- rename "dead_hctx" as "unused_hctx"
- check if there are live hctx in queue's release handler
- only patch 6 is modified
V6:
- remove previous SCSI patch which will be routed via SCSI tree
- add reviewed-by tag
- fix one related NVMe scan vs reset race
V5:
- refactor blk_mq_alloc_and_init_hctx()
- fix race related updating nr_hw_queues by always freeing hctx after request queue is released
V4:
- add patch for fixing potential use-after-free in blk_mq_update_nr_hw_queues
- fix comment in the last patch
V3:
- cancel q->requeue_work in queue's release handler
- cancel hctx->run_work in hctx's release handler
- add patch 1 for fixing race in plug code path
- the last patch is added for avoiding to grab SCSI's refcont in IO path
V2:
- moving freeing hw queue resources into hctx's release handler
Ming Lei (7):
blk-mq: grab .q_usage_counter when queuing request from plug code path
blk-mq: move cancel of requeue_work into blk_mq_release
blk-mq: free hw queue's resource in hctx's release handler
blk-mq: split blk_mq_alloc_and_init_hctx into two parts
blk-mq: always free hctx after request queue is freed
blk-mq: move cancel of hctx->run_work into blk_mq_hw_sysfs_release
block: don't drain in-progress dispatch in blk_cleanup_queue()
block/blk-core.c | 23 +----- block/blk-mq-sched.c | 12 +++- block/blk-mq-sysfs.c | 8 +++ block/blk-mq.c | 189 ++++++++++++++++++++++++++++--------------------- block/blk-mq.h | 2 +- include/linux/blk-mq.h | 2 + include/linux/blkdev.h | 7 ++ 7 files changed, 139 insertions(+), 104 deletions(-)
Signature · 1 line
2.9.5
PATCH v9 1/7 blk-mq: grab .q_usage_counter when queuing request from plug code path
Just like aio/io_uring, we need to grab 2 refcount for queuing one request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed in generic_make_request() serves for submission. In theroy, this refcount should have been released after the sumission(async run queue) is done. blk_freeze_queue() works with blk_sync_queue() together for avoiding race between cleanup queue and IO submission, given async run queue activities are canceled because hctx->run_work is scheduled with the refcount held, so it is fine to not hold the refcount when running the run queue work function for dispatch IO.
However, if request is staggered into plug list, and finally queued from plug code path, the refcount in submission side is actually missed. And we may start to run queue after queue is removed because the queue's kobject refcount isn't guaranteed to be grabbed in flushing plug list context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queue blk_mq_run_hw_queue
Because of concurrent run queue, all requests inserted above may be completed before calling the above blk_mq_run_hw_queue. Then queue can be freed during the above blk_mq_run_hw_queue().
Fixes the issue by grab .q_usage_counter before calling blk_mq_sched_insert_requests() in blk_mq_flush_plug_list(). This way is safe because the queue is absolutely alive before inserting request.
--- block/blk-mq-sched.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
block/blk-mq-sched.c +11 -1
@@ -413,6 +413,14 @@ void blk_mq_sched_insert_requests(struct blk_mq_hw_ctx *hctx, struct list_head *list, bool run_queue_async) {struct elevator_queue *e; +struct request_queue *q = hctx->queue; + +/* +* blk_mq_sched_insert_requests() is called from flush plug +* context only, and hold one usage counter to prevent queue +* from being released. +*/ +percpu_ref_get (&q->q_usage_counter); e = hctx->queue->elevator;if (e && e->type->ops.insert_requests)
@@ -426,12 +434,14 @@ void blk_mq_sched_insert_requests(struct blk_mq_hw_ctx *hctx,if (!hctx->dispatch_busy && !e && !run_queue_async) {blk_mq_try_issue_list_directly (hctx, list);if (list_empty (list)) -return ; + goto out; }blk_mq_insert_requests (hctx, ctx, list); }blk_mq_run_hw_queue (hctx, run_queue_async); + out: +percpu_ref_put (&q->q_usage_counter); } static void blk_mq_sched_free_tags(struct blk_mq_tag_set *set,
Signature · 1 line
2.9.5
Re: [PATCH V9 1/7] blk-mq: grab .q_usage_counter when queuing request from plug code path
Quoted 23 lines · Ming Lei
Just like aio/io_uring, we need to grab 2 refcount for queuing one request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed in
generic_make_request()serves for submission. In theroy, this refcount should have been released after the sumission(async run queue) is done.blk_freeze_queue()works withblk_sync_queue()together for avoiding race between cleanup queue and IO submission, given async run queue activities are canceled becausehctx->run_workis scheduled with the refcount held, so it is fine to not hold the refcount when running the run queue work function for dispatch IO.However, if request is staggered into plug list, and finally queued from plug code path, the refcount in submission side is actually missed. And we may start to run queue after queue is removed because the queue's kobject refcount isn't guaranteed to be grabbed in flushing plug list context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queueblk_mq_run_hw_queue
Quoted 7 lines
Because of concurrent run queue, all requests inserted above may be completed before calling the above
blk_mq_run_hw_queue. Then queue can be freed during the aboveblk_mq_run_hw_queue().Fixes the issue by grab .q_usage_counter before calling
blk_mq_sched_insert_requests()inblk_mq_flush_plug_list(). This way is safe because the queue is absolutely alive before inserting request.
PATCH v9 2/7 blk-mq: move cancel of requeue_work into blk_mq_release
In reply to Ming LeiWith holding queue's kobject refcount, it is safe for driver to schedule requeue. However, blk_mq_kick_requeue_list() may be called after blk_sync_queue() is done because of concurrent requeue activities, then requeue work may not be completed when freeing queue, and kernel oops is triggered.
So moving the cancel of requeue_work into blk_mq_release() for avoiding race between requeue and freeing queue.
--- block/blk-core.c | 1 - block/blk-mq.c | 2 ++ 2 files changed, 2 insertions(+), 1 deletion(-)
block/blk-core.c +0 -1
@@ -237,7 +237,6 @@ void blk_sync_queue(struct request_queue *q)struct blk_mq_hw_ctx *hctx;int i; -cancel_delayed_work_sync (&q->requeue_work);queue_for_each_hw_ctx (q, hctx, i)cancel_delayed_work_sync (&hctx->run_work); }
block/blk-mq.c +2 -0
@@ -2634,6 +2634,8 @@ void blk_mq_release(struct request_queue *q)struct blk_mq_hw_ctx *hctx;unsigned int i; +cancel_delayed_work_sync (&q->requeue_work); +/* hctx kobj stays in hctx */ queue_for_each_hw_ctx (q, hctx, i) {if (!hctx)
Signature · 1 line
2.9.5
PATCH v9 3/7 blk-mq: free hw queue's resource in hctx's release handler
In reply to Ming LeiOnce blk_cleanup_queue() returns, tags shouldn't be used any more, because blk_mq_free_tag_set() may be called. Commit 45a9c9d909b2 ("blk-mq: Fix a use-after-free") fixes this issue exactly.
However, that commit introduces another issue. Before 45a9c9d909b2, we are allowed to run queue during cleaning up queue if the queue's kobj refcount is held. After that commit, queue can't be run during queue cleaning up, otherwise oops can be triggered easily because some fields of hctx are freed by blk_mq_free_queue() in blk_cleanup_queue().
We have invented ways for addressing this kind of issue before, such as:
8dc765d438f1 ("SCSI: fix queue cleanup race before queue initialization is done") c2856ae2f315 ("blk-mq: quiesce queue before freeing queue")
But still can't cover all cases, recently James reports another such kind of issue:
https://marc.info/?l=linux-scsi&m=155389088124782&w=2
This issue can be quite hard to address by previous way, given scsi_run_queue() may run requeues for other LUNs.
Fixes the above issue by freeing hctx's resources in its release handler, and this way is safe becasue tags isn't needed for freeing such hctx resource.
This approach follows typical design pattern wrt. kobject's release handler.
--- block/blk-core.c | 2 +- block/blk-mq-sysfs.c | 6 ++++++ block/blk-mq.c | 8 ++------ block/blk-mq.h | 2 +- 4 files changed, 10 insertions(+), 8 deletions(-)
block/blk-core.c +1 -1
@@ -374,7 +374,7 @@ void blk_cleanup_queue(struct request_queue *q)blk_exit_queue (q);if (queue_is_mq (q)) -blk_mq_free_queue (q); +blk_mq_exit_queue (q);percpu_ref_exit (&q->q_usage_counter);
block/blk-mq-sysfs.c +6 -0
@@ -10,6 +10,7 @@#include <linux/smp.h> #include <linux/blk-mq.h> +#include "blk.h" #include "blk-mq.h" #include "blk-mq-tag.h"
@@ -33,6 +34,11 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj) { struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx, kobj); + +if (hctx->flags &BLK_MQ_F_BLOCKING ) +cleanup_srcu_struct (hctx->srcu); +blk_free_flush_queue (hctx->fq); +sbitmap_free (&hctx->ctx_map);free_cpumask_var (hctx->cpumask);kfree (hctx->ctxs);kfree (hctx);
block/blk-mq.c +2 -6
@@ -2267,12 +2267,7 @@ static void blk_mq_exit_hctx(struct request_queue *q,if (set->ops->exit_hctx) set->ops->exit_hctx (hctx, hctx_idx); -if (hctx->flags &BLK_MQ_F_BLOCKING ) -cleanup_srcu_struct (hctx->srcu); -blk_mq_remove_cpuhp (hctx); -blk_free_flush_queue (hctx->fq); -sbitmap_free (&hctx->ctx_map); } static void blk_mq_exit_hw_queues(struct request_queue *q,
@@ -2907,7 +2902,8 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, }EXPORT_SYMBOL (blk_mq_init_allocated_queue); -void blk_mq_free_queue (struct request_queue *q) +/* tags can _not_ be used after returning from blk_mq_exit_queue */ +void blk_mq_exit_queue (struct request_queue *q) {struct blk_mq_tag_set *set = q->tag_set;
block/blk-mq.h +1 -1
@@ -37,7 +37,7 @@ struct blk_mq_ctx {struct kobject kobj; }____cacheline_aligned_in_smp ; -void blk_mq_free_queue (struct request_queue *q); +void blk_mq_exit_queue (struct request_queue *q);int blk_mq_update_nr_requests (struct request_queue *q,unsigned int nr);void blk_mq_wake_waiters (struct request_queue *q);bool blk_mq_dispatch_rq_list (struct request_queue *,struct list_head *,bool );
Signature · 1 line
2.9.5
PATCH v9 4/7 blk-mq: split blk_mq_alloc_and_init_hctx into two parts
In reply to Ming LeiSplit blk_mq_alloc_and_init_hctx into two parts, and one is blk_mq_alloc_hctx() for allocating all hctx resources, another is blk_mq_init_hctx() for initializing hctx, which serves as counter-part of blk_mq_exit_hctx().
--- block/blk-mq.c | 139 +++++++++++++++++++++++++++++++-------------------------- 1 file changed, 75 insertions(+), 64 deletions(-)
block/blk-mq.c +75 -64
@@ -2284,15 +2284,65 @@ static void blk_mq_exit_hw_queues(struct request_queue *q, } } +static int blk_mq_hw_ctx_size (struct blk_mq_tag_set *tag_set) +{ +int hw_ctx_size =sizeof (struct blk_mq_hw_ctx ); + +BUILD_BUG_ON (ALIGN (offsetof(struct blk_mq_hw_ctx , srcu), + __alignof__(struct blk_mq_hw_ctx )) != +sizeof (struct blk_mq_hw_ctx )); + +if (tag_set->flags &BLK_MQ_F_BLOCKING ) + hw_ctx_size +=sizeof (struct srcu_struct ); + +return hw_ctx_size; +} +static int blk_mq_init_hctx (struct request_queue *q,struct blk_mq_tag_set *set,struct blk_mq_hw_ctx *hctx,unsigned hctx_idx) { -int node; + hctx->queue_num = hctx_idx; + +cpuhp_state_add_instance_nocalls (CPUHP_BLK_MQ_DEAD , &hctx->cpuhp_dead); + + hctx->tags = set->tags[hctx_idx]; + +if (set->ops->init_hctx && + set->ops->init_hctx (hctx, set->driver_data, hctx_idx)) + goto unregister_cpu_notifier; - node = hctx->numa_node; +if (blk_mq_init_request (set, hctx->fq->flush_rq, hctx_idx, + hctx->numa_node)) + goto exit_hctx; +return 0 ; + + exit_hctx: +if (set->ops->exit_hctx) + set->ops->exit_hctx (hctx, hctx_idx); + unregister_cpu_notifier: +blk_mq_remove_cpuhp (hctx); +return -1 ; +} + +static struct blk_mq_hw_ctx * +blk_mq_alloc_hctx (struct request_queue *q,struct blk_mq_tag_set *set, +int node) +{ +struct blk_mq_hw_ctx *hctx; +gfp_t gfp =GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY; + + hctx =kzalloc_node (blk_mq_hw_ctx_size (set), gfp, node); +if (!hctx) + goto fail_alloc_hctx; + +if (!zalloc_cpumask_var_node (&hctx->cpumask, gfp, node)) + goto free_hctx; + +atomic_set (&hctx->nr_active,0 );if (node ==NUMA_NO_NODE ) - node = hctx->numa_node = set->numa_node; + node = set->numa_node; + hctx->numa_node = node;INIT_DELAYED_WORK (&hctx->run_work, blk_mq_run_work_fn);spin_lock_init (&hctx->lock);
@@ -2300,58 +2350,45 @@ static int blk_mq_init_hctx(struct request_queue *q, hctx->queue = q; hctx->flags = set->flags & ~BLK_MQ_F_TAG_SHARED ; -cpuhp_state_add_instance_nocalls (CPUHP_BLK_MQ_DEAD , &hctx->cpuhp_dead); - - hctx->tags = set->tags[hctx_idx]; -/* * Allocate space for all possible cpus to avoid allocation at * runtime */ hctx->ctxs =kmalloc_array_node (nr_cpu_ids,sizeof (void *), -GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, node); + gfp, node);if (!hctx->ctxs) - goto unregister_cpu_notifier; + goto free_cpumask;if (sbitmap_init_node (&hctx->ctx_map, nr_cpu_ids,ilog2 (8 ), -GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, node)) + gfp, node)) goto free_ctxs; - hctx->nr_ctx =0 ;spin_lock_init (&hctx->dispatch_wait_lock);init_waitqueue_func_entry (&hctx->dispatch_wait, blk_mq_dispatch_wake);INIT_LIST_HEAD (&hctx->dispatch_wait.entry); -if (set->ops->init_hctx && - set->ops->init_hctx (hctx, set->driver_data, hctx_idx)) - goto free_bitmap; - hctx->fq =blk_alloc_flush_queue (q, hctx->numa_node, set->cmd_size, -GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY); + gfp);if (!hctx->fq) - goto exit_hctx; - -if (blk_mq_init_request (set, hctx->fq->flush_rq, hctx_idx, node)) - goto free_fq; + goto free_bitmap;if (hctx->flags &BLK_MQ_F_BLOCKING )init_srcu_struct (hctx->srcu); +blk_mq_hctx_kobj_init (hctx); -return 0 ; +return hctx; - free_fq: -blk_free_flush_queue (hctx->fq); - exit_hctx: -if (set->ops->exit_hctx) - set->ops->exit_hctx (hctx, hctx_idx); free_bitmap:sbitmap_free (&hctx->ctx_map); free_ctxs:kfree (hctx->ctxs); - unregister_cpu_notifier: -blk_mq_remove_cpuhp (hctx); -return -1 ; + free_cpumask: +free_cpumask_var (hctx->cpumask); + free_hctx: +kfree (hctx); + fail_alloc_hctx: +return NULL ; } static void blk_mq_init_cpu_queues(struct request_queue *q,
@@ -2697,51 +2734,25 @@ struct request_queue *blk_mq_init_sq_queue(struct blk_mq_tag_set *set, }EXPORT_SYMBOL (blk_mq_init_sq_queue); -static int blk_mq_hw_ctx_size (struct blk_mq_tag_set *tag_set) -{ -int hw_ctx_size =sizeof (struct blk_mq_hw_ctx ); - -BUILD_BUG_ON (ALIGN (offsetof(struct blk_mq_hw_ctx , srcu), - __alignof__(struct blk_mq_hw_ctx )) != -sizeof (struct blk_mq_hw_ctx )); - -if (tag_set->flags &BLK_MQ_F_BLOCKING ) - hw_ctx_size +=sizeof (struct srcu_struct ); - -return hw_ctx_size; -} -static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx (struct blk_mq_tag_set *set,struct request_queue *q,int hctx_idx,int node) {struct blk_mq_hw_ctx *hctx; - hctx =kzalloc_node (blk_mq_hw_ctx_size (set), -GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, - node); + hctx =blk_mq_alloc_hctx (q, set, node);if (!hctx) -return NULL ; - -if (!zalloc_cpumask_var_node (&hctx->cpumask, -GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, - node)) { -kfree (hctx); -return NULL ; - } - -atomic_set (&hctx->nr_active,0 ); - hctx->numa_node = node; - hctx->queue_num = hctx_idx; + goto fail; -if (blk_mq_init_hctx (q, set, hctx, hctx_idx)) { -free_cpumask_var (hctx->cpumask); -kfree (hctx); -return NULL ; - } -blk_mq_hctx_kobj_init (hctx); +if (blk_mq_init_hctx (q, set, hctx, hctx_idx)) + goto free_hctx;return hctx; + + free_hctx: +kobject_put (&hctx->kobj); + fail: +return NULL ; } static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
Signature · 1 line
2.9.5
PATCH v9 5/7 blk-mq: always free hctx after request queue is freed
In reply to Ming LeiIn normal queue cleanup path, hctx is released after request queue is freed, see blk_mq_release().
However, in __blk_mq_update_nr_hw_queues(), hctx may be freed because of hw queues shrinking. This way is easy to cause use-after-free,
because: one implicit rule is that it is safe to call almost all block layer APIs if the request queue is alive; and one hctx may be retrieved by one API, then the hctx can be freed by blk_mq_update_nr_hw_queues(); finally use-after-free is triggered.
Fixes this issue by always freeing hctx after releasing request queue. If some hctxs are removed in blk_mq_update_nr_hw_queues(), introduce a per-queue list to hold them, then try to resuse these hctxs if numa node is matched.
--- block/blk-mq.c | 46 +++++++++++++++++++++++++++++++++------------- include/linux/blk-mq.h | 2 ++ include/linux/blkdev.h | 7 +++++++ 3 files changed, 42 insertions(+), 13 deletions(-)
block/blk-mq.c +33 -13
@@ -2268,6 +2268,10 @@ static void blk_mq_exit_hctx(struct request_queue *q, set->ops->exit_hctx (hctx, hctx_idx);blk_mq_remove_cpuhp (hctx); + +spin_lock (&q->unused_hctx_lock); +list_add (&hctx->hctx_list, &q->unused_hctx_list); +spin_unlock (&q->unused_hctx_lock); } static void blk_mq_exit_hw_queues(struct request_queue *q,
@@ -2350,6 +2354,8 @@ blk_mq_alloc_hctx(struct request_queue *q, struct blk_mq_tag_set *set, hctx->queue = q; hctx->flags = set->flags & ~BLK_MQ_F_TAG_SHARED ; +INIT_LIST_HEAD (&hctx->hctx_list); +/* * Allocate space for all possible cpus to avoid allocation at * runtime
@@ -2663,15 +2669,17 @@ static int blk_mq_alloc_ctxs(struct request_queue *q)*/ void blk_mq_release (struct request_queue *q) { -struct blk_mq_hw_ctx *hctx; -unsigned int i; +struct blk_mq_hw_ctx *hctx, *next; +int i;cancel_delayed_work_sync (&q->requeue_work); -/* hctx kobj stays in hctx */ -queue_for_each_hw_ctx (q, hctx, i) { -if (!hctx) -continue ; +queue_for_each_hw_ctx (q, hctx, i) +WARN_ON_ONCE (hctx &&list_empty (&hctx->hctx_list)); + +/* all hctx are in .unused_hctx_list now */ +list_for_each_entry_safe (hctx, next, &q->unused_hctx_list, hctx_list) { +list_del_init (&hctx->hctx_list);kobject_put (&hctx->kobj); }
@@ -2738,9 +2746,22 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx( struct blk_mq_tag_set *set, struct request_queue *q, int hctx_idx, int node) { - struct blk_mq_hw_ctx *hctx; + struct blk_mq_hw_ctx *hctx = NULL, *tmp; - hctx =blk_mq_alloc_hctx (q, set, node); +/* reuse dead hctx first */ +spin_lock (&q->unused_hctx_lock); +list_for_each_entry (tmp, &q->unused_hctx_list, hctx_list) { +if (tmp->numa_node == node) { + hctx = tmp; +break ; + } + } +if (hctx) +list_del_init (&hctx->hctx_list); +spin_unlock (&q->unused_hctx_lock); + +if (!hctx) + hctx =blk_mq_alloc_hctx (q, set, node);if (!hctx) goto fail;
@@ -2778,10 +2799,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, hctx =blk_mq_alloc_and_init_hctx (set, q, i, node);if (hctx) { -if (hctxs[i]) { +if (hctxs[i])blk_mq_exit_hctx (q, set, hctxs[i], i); -kobject_put (&hctxs[i]->kobj); - } hctxs[i] = hctx; }else {if (hctxs[i])
@@ -2812,9 +2831,7 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,if (hctx->tags)blk_mq_free_map_and_requests (set, j);blk_mq_exit_hctx (q, set, hctx, j); -kobject_put (&hctx->kobj); hctxs[j] =NULL ; - } }mutex_unlock (&q->sysfs_lock);
@@ -2857,6 +2874,9 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,if (!q->queue_hw_ctx) goto err_sys_init; +INIT_LIST_HEAD (&q->unused_hctx_list); +spin_lock_init (&q->unused_hctx_lock); +blk_mq_realloc_hw_ctxs (set, q);if (!q->nr_hw_queues) goto err_hctxs;
include/linux/blk-mq.h +2 -0
@@ -70,6 +70,8 @@ struct blk_mq_hw_ctx {struct dentry *sched_debugfs_dir;#endif +struct list_head hctx_list; +/* Must be the last member - see also blk_mq_hw_ctx_size(). */ struct srcu_struct srcu[0 ]; };
include/linux/blkdev.h +7 -0
@@ -535,6 +535,13 @@ struct request_queue {struct mutex sysfs_lock; +/* +* for reusing dead hctx instance in case of updating +* nr_hw_queues +*/ +struct list_head unused_hctx_list; +spinlock_t unused_hctx_lock; +atomic_t mq_freeze_depth;#if defined(CONFIG_BLK_DEV_BSG )
Signature · 1 line
2.9.5
PATCH v9 6/7 blk-mq: move cancel of hctx->run_work into blk_mq_hw_sysfs_release
In reply to Ming Leihctx is always released after requeue is freed.
With holding queue's kobject refcount, it is safe for driver to run queue, so one run queue might be scheduled after blk_sync_queue() is done.
So moving the cancel of hctx->run_work into blk_mq_hw_sysfs_release() for avoiding run released queue.
--- block/blk-core.c | 8 -------- block/blk-mq-sysfs.c | 2 ++ 2 files changed, 2 insertions(+), 8 deletions(-)
block/blk-core.c +0 -8
@@ -232,14 +232,6 @@ void blk_sync_queue(struct request_queue *q) {del_timer_sync (&q->timeout);cancel_work_sync (&q->timeout_work); - -if (queue_is_mq (q)) { -struct blk_mq_hw_ctx *hctx; -int i; - -queue_for_each_hw_ctx (q, hctx, i) -cancel_delayed_work_sync (&hctx->run_work); - } }EXPORT_SYMBOL (blk_sync_queue);
block/blk-mq-sysfs.c +2 -0
@@ -35,6 +35,8 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj) struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx, kobj); +cancel_delayed_work_sync (&hctx->run_work); +if (hctx->flags &BLK_MQ_F_BLOCKING )cleanup_srcu_struct (hctx->srcu);blk_free_flush_queue (hctx->fq);
Signature · 1 line
2.9.5
PATCH v9 7/7 block: don't drain in-progress dispatch in blk_cleanup_queue()
In reply to Ming LeiNow freeing hw queue resource is moved to hctx's release handler, we don't need to worry about the race between blk_cleanup_queue and run queue any more.
So don't drain in-progress dispatch in blk_cleanup_queue().
This is basically revert of c2856ae2f315 ("blk-mq: quiesce queue before freeing queue").
--- block/blk-core.c | 12 ------------ 1 file changed, 12 deletions(-)
block/blk-core.c +0 -12
@@ -338,18 +338,6 @@ void blk_cleanup_queue(struct request_queue *q)blk_queue_flag_set (QUEUE_FLAG_DEAD , q); -/* -* make sure all in-progress dispatch are completed because -* blk_freeze_queue() can only complete all requests, and -* dispatch may still be in-progress since we dispatch requests -* from more than one contexts. -* -* We rely on driver to deal with the race in case that queue -* initialization isn't done. -*/ -if (queue_is_mq (q) &&blk_queue_init_done (q)) -blk_mq_quiesce_queue (q); -/* for synchronous bio-based driver finish in-flight integrity i/o */ blk_flush_integrity ();
Signature · 1 line
2.9.5
Quoted 11 lines · Ming Lei
Hi,
Since
45a9c9d909b2("blk-mq: Fix a use-after-free"), run queue isn't allowed during cleanup queue even though queue refcount is held.This change has caused lots of kernel oops triggered in run queue path, turns out it isn't easy to fix them all.
So move freeing of hw queue resources into hctx's release handler, then the above issue is fixed. Meantime, this way is safe given freeing hw queue resource doesn't require tags.
Applied, thanks Ming.
Signature · 1 line
Jens Axboe