blk-mq: improvement on handling IO during CPU hotplug
Subsystems (4)
v2 cover blk-mq: improvement on handling IO during CPU hotplug
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very beginning:
The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"
But no drivers or blk-mq do that before one hctx becomes dead(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see blk_mq_hctx_notify_dead().
This patchset tries to address the issue by two stages:
1) add one new cpuhp state of CPUHP_AP_BLK_MQ_ONLINE
- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of CPUHP_BLK_MQ_DEAD after the hctx becomes dead
- steal bios from the request, and resubmit them via generic_make_request(), then these IO will be mapped to other live hctx for dispatch
Please comment & review, thanks!
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and Minwoo
Ming Lei (5):
blk-mq: add new state of BLK_MQ_S_INTERNAL_STOPPED
blk-mq: add blk-mq flag of BLK_MQ_F_NO_MANAGED_IRQ
blk-mq: stop to handle IO before hctx's all CPUs become offline
blk-mq: re-submit IO in case that hctx is dead
blk-mq: handle requests dispatched from IO scheduler in case that hctx
is dead
block/blk-mq-debugfs.c | 2 + block/blk-mq-tag.c | 2 +- block/blk-mq-tag.h | 2 + block/blk-mq.c | 143 +++++++++++++++++++++++++++++++++---- block/blk-mq.h | 3 +- drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 5 ++ include/linux/cpuhotplug.h | 1 + 9 files changed, 146 insertions(+), 16 deletions(-)
Signature · 1 line
2.20.1
PATCH v2 1/5 blk-mq: add new state of BLK_MQ_S_INTERNAL_STOPPED
Add a new hw queue state of BLK_MQ_S_INTERNAL_STOPPED, which prepares for stopping hw queue before all CPUs of this hctx become offline.
We can't reuse BLK_MQ_S_STOPPED because that state can be cleared during IO completion.
--- block/blk-mq-debugfs.c | 1 + block/blk-mq.h | 3 ++- include/linux/blk-mq.h | 3 +++ 3 files changed, 6 insertions(+), 1 deletion(-)
block/blk-mq-debugfs.c +1 -0
@@ -213,6 +213,7 @@ static const char *const hctx_state_name[] = {HCTX_STATE_NAME (STOPPED ),HCTX_STATE_NAME (TAG_ACTIVE ),HCTX_STATE_NAME (SCHED_RESTART ), + HCTX_STATE_NAME(INTERNAL_STOPPED), };#undef HCTX_STATE_NAME
block/blk-mq.h +2 -1
@@ -176,7 +176,8 @@ static inline struct blk_mq_tags *blk_mq_tags_from_data(struct blk_mq_alloc_datastatic inline bool blk_mq_hctx_stopped (struct blk_mq_hw_ctx *hctx) { -return test_bit (BLK_MQ_S_STOPPED , &hctx->state); +return test_bit (BLK_MQ_S_STOPPED , &hctx->state) || +test_bit (BLK_MQ_S_INTERNAL_STOPPED , &hctx->state); }static inline bool blk_mq_hw_queue_mapped (struct blk_mq_hw_ctx *hctx)
include/linux/blk-mq.h +3 -0
@@ -235,6 +235,9 @@ enum {BLK_MQ_S_TAG_ACTIVE =1 ,BLK_MQ_S_SCHED_RESTART =2 , +/* hw queue is internal stopped, driver do not use it */ +BLK_MQ_S_INTERNAL_STOPPED =3 , +BLK_MQ_MAX_DEPTH =10240 , BLK_MQ_CPU_WORK_BATCH = 8,
Signature · 1 line
2.20.1
PATCH v2 2/5 blk-mq: add blk-mq flag of BLK_MQ_F_NO_MANAGED_IRQ
In reply to Ming LeiWe will stop hw queue and wait for completion of in-flight requests when one hctx is becoming dead in the following patch. This way may cause dead-lock for some stacking blk-mq drivers, such as dm-rq and loop.
Add blk-mq flag of BLK_MQ_F_NO_MANAGED_IRQ and mark it for dm-rq and loop, so we needn't to wait for completion of in-flight requests of dm-rq & loop, then the potential dead-lock can be avoided.
--- block/blk-mq-debugfs.c | 1 + drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 1 + 4 files changed, 4 insertions(+), 2 deletions(-)
block/blk-mq-debugfs.c +1 -0
@@ -240,6 +240,7 @@ static const char *const hctx_flag_name[] = {HCTX_FLAG_NAME (TAG_SHARED ),HCTX_FLAG_NAME (BLOCKING ),HCTX_FLAG_NAME (NO_SCHED ), + HCTX_FLAG_NAME(NO_MANAGED_IRQ), };#undef HCTX_FLAG_NAME
drivers/block/loop.c +1 -1
@@ -1989,7 +1989,7 @@ static int loop_add(struct loop_device **l, int i) lo->tag_set.queue_depth =128 ; lo->tag_set.numa_node =NUMA_NO_NODE ; lo->tag_set.cmd_size =sizeof (struct loop_cmd ); - lo->tag_set.flags =BLK_MQ_F_SHOULD_MERGE ; + lo->tag_set.flags =BLK_MQ_F_SHOULD_MERGE |BLK_MQ_F_NO_MANAGED_IRQ ; lo->tag_set.driver_data = lo; err =blk_mq_alloc_tag_set (&lo->tag_set);
drivers/md/dm-rq.c +1 -1
@@ -547,7 +547,7 @@ int dm_mq_init_request_queue(struct mapped_device *md, struct dm_table *t) md->tag_set->ops = &dm_mq_ops; md->tag_set->queue_depth =dm_get_blk_mq_queue_depth (); md->tag_set->numa_node = md->numa_node_id; - md->tag_set->flags =BLK_MQ_F_SHOULD_MERGE ; + md->tag_set->flags =BLK_MQ_F_SHOULD_MERGE |BLK_MQ_F_NO_MANAGED_IRQ ; md->tag_set->nr_hw_queues =dm_get_blk_mq_nr_hw_queues (); md->tag_set->driver_data = md;
include/linux/blk-mq.h +1 -0
@@ -226,6 +226,7 @@ struct blk_mq_ops {enum {BLK_MQ_F_SHOULD_MERGE =1 <<0 ,BLK_MQ_F_TAG_SHARED =1 <<1 , +BLK_MQ_F_NO_MANAGED_IRQ =1 <<2 ,BLK_MQ_F_BLOCKING =1 <<5 ,BLK_MQ_F_NO_SCHED =1 <<6 ,BLK_MQ_F_ALLOC_POLICY_START_BIT =8 ,
Signature · 1 line
2.20.1
PATCH v2 3/5 blk-mq: stop to handle IO before hctx's all CPUs become offline
In reply to Ming LeiMost of blk-mq drivers depend on managed IRQ's auto-affinity to setup up queue mapping. Thomas mentioned the following point[1]:
"
That was the constraint of managed interrupts from the very beginning:
The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"
However, current blk-mq implementation doesn't quiesce hw queue before the last CPU in the hctx is shutdown. Even worse, CPUHP_BLK_MQ_DEAD is one cpuhp state handled after the CPU is down, so there isn't any chance to quiesce hctx for blk-mq wrt. CPU hotplug.
Add new cpuhp state of CPUHP_AP_BLK_MQ_ONLINE for blk-mq to stop queues and wait for completion of in-flight requests.
[1] https://lore.kernel.org/linux-block/[email protected]/
--- block/blk-mq-tag.c | 2 +- block/blk-mq-tag.h | 2 ++ block/blk-mq.c | 65 ++++++++++++++++++++++++++++++++++++++ include/linux/blk-mq.h | 1 + include/linux/cpuhotplug.h | 1 + 5 files changed, 70 insertions(+), 1 deletion(-)
block/blk-mq-tag.c +1 -1
@@ -325,7 +325,7 @@ static void bt_tags_for_each(struct blk_mq_tags *tags, struct sbitmap_queue *bt,* true to continue iterating tags, false to stop. * @priv: Will be passed as second argument to @fn. */ -static void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags, +void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags,busy_tag_iter_fn *fn,void *priv) {if (tags->nr_reserved_tags)
block/blk-mq-tag.h +2 -0
@@ -35,6 +35,8 @@ extern int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx,extern void blk_mq_tag_wakeup_all (struct blk_mq_tags *tags,bool );void blk_mq_queue_tag_busy_iter (struct request_queue *q,busy_iter_fn *fn,void *priv); +void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags, +busy_tag_iter_fn *fn,void *priv);static inline struct sbq_wait_state *bt_wait_ptr (struct sbitmap_queue *bt,struct blk_mq_hw_ctx *hctx)
block/blk-mq.c +65 -0
95 lines not on this page · raw message
include/linux/blk-mq.h +1 -0
@@ -58,6 +58,7 @@ struct blk_mq_hw_ctx {atomic_t nr_active; +struct hlist_node cpuhp_online;struct hlist_node cpuhp_dead;struct kobject kobj;
include/linux/cpuhotplug.h +1 -0
@@ -147,6 +147,7 @@ enum cpuhp_state {CPUHP_AP_SMPBOOT_THREADS ,CPUHP_AP_X86_VDSO_VMA_ONLINE ,CPUHP_AP_IRQ_AFFINITY_ONLINE , +CPUHP_AP_BLK_MQ_ONLINE ,CPUHP_AP_ARM_MVEBU_SYNC_CLOCKS ,CPUHP_AP_X86_INTEL_EPB_ONLINE , CPUHP_AP_PERF_ONLINE,
Signature · 1 line
2.20.1
Re: [PATCH V2 3/5] blk-mq: stop to handle IO before hctx's all CPUs become offline
Quoted 117 lines · Ming Lei
Most of blk-mq drivers depend on managed IRQ's auto-affinity to setup up queue mapping. Thomas mentioned the following point[1]:
"
That was the constraint of managed interrupts from the very beginning:The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"However, current blk-mq implementation doesn't quiesce hw queue before the last CPU in the hctx is shutdown. Even worse,
CPUHP_BLK_MQ_DEADis one cpuhp state handled after the CPU is down, so there isn't any chance to quiesce hctx for blk-mq wrt. CPU hotplug.Add new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINEfor blk-mq to stop queues and wait for completion of in-flight requests.[1] https://lore.kernel.org/linux-block/[email protected]/
Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Keith Busch <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-mq-tag.c | 2 +- block/blk-mq-tag.h | 2 ++ block/blk-mq.c | 65 ++++++++++++++++++++++++++++++++++++++ include/linux/blk-mq.h | 1 + include/linux/cpuhotplug.h | 1 + 5 files changed, 70 insertions(+), 1 deletion(-)block/blk-mq-tag.c +1
-1@@ -325,7 +325,7 @@ static void bt_tags_for_each(struct blk_mq_tags *tags, struct sbitmap_queue *bt,* true to continue iterating tags, false to stop. * @priv: Will be passed as second argument to @fn. */ -static void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags, +void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags,busy_tag_iter_fn *fn,void *priv) {if (tags->nr_reserved_tags)block/blk-mq-tag.h +2
-0@@ -35,6 +35,8 @@ extern int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx,extern void blk_mq_tag_wakeup_all (struct blk_mq_tags *tags,bool );void blk_mq_queue_tag_busy_iter (struct request_queue *q,busy_iter_fn *fn,void *priv); +void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags, +busy_tag_iter_fn *fn,void *priv);static inline struct sbq_wait_state *bt_wait_ptr (struct sbitmap_queue *bt,struct blk_mq_hw_ctx *hctx)block/blk-mq.c +48
-0@@ -2206,6 +2206,61 @@ int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags,return -ENOMEM ; } +static bool blk_mq_count_inflight_rq (struct request *rq,void *data, +bool reserved) +{ +unsigned *count = data; + +if ((blk_mq_rq_state (rq) ==MQ_RQ_IN_FLIGHT )) + (*count)++; + +return true; +} + +static unsigned blk_mq_tags_inflight_rqs (struct blk_mq_tags *tags) +{ +unsigned count =0 ; + +blk_mq_all_tag_busy_iter (tags, blk_mq_count_inflight_rq, &count); + +return count; +} + +static void blk_mq_drain_inflight_rqs (struct blk_mq_hw_ctx *hctx) +{ +while (1 ) { +if (!blk_mq_tags_inflight_rqs (hctx->tags)) +break ; +msleep (5 ); + } +} + +static int blk_mq_hctx_notify_online (unsigned int cpu,struct hlist_node *node) +{ +struct blk_mq_hw_ctx *hctx =hlist_entry_safe (node, + struct blk_mq_hw_ctx, cpuhp_online); +unsigned prev_cpu =-1 ; + +while (true) { +unsigned next_cpu =cpumask_next_and (prev_cpu, hctx->cpumask, + cpu_online_mask); + +if (next_cpu >= nr_cpu_ids) +break ; + +/* return if there is other online CPU on this hctx */ +if (next_cpu != cpu) +return 0 ; + + prev_cpu = next_cpu; + }
block/blk-mq.c int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, · line 2266
Quoted 10 lines
block/blk-mq.c +7
-0+ +set_bit (BLK_MQ_S_INTERNAL_STOPPED , &hctx->state); +blk_mq_drain_inflight_rqs (hctx); + +return 0 ; +} +/* * 'cpu' is going away. splice any existing rq_list entries from this * software queue to the hw queue dispatch list, and ensure that it
Isn't that inverted?
From the function I would assume it'll be called once the CPU is being set toe 'online', yet from the description I would have assumed the INTERNAL_STOPPED bit is set when the cpu goes offline.
Care to elaborate?
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
Re: [PATCH V2 3/5] blk-mq: stop to handle IO before hctx's all CPUs become offline
Quoted 122 lines · Hannes Reinecke
Quoted 121 lines · Ming Lei
Most of blk-mq drivers depend on managed IRQ's auto-affinity to setup up queue mapping. Thomas mentioned the following point[1]:
"
That was the constraint of managed interrupts from the very beginning:The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"However, current blk-mq implementation doesn't quiesce hw queue before the last CPU in the hctx is shutdown. Even worse,
CPUHP_BLK_MQ_DEADis one cpuhp state handled after the CPU is down, so there isn't any chance to quiesce hctx for blk-mq wrt. CPU hotplug.Add new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINEfor blk-mq to stop queues and wait for completion of in-flight requests.[1] https://lore.kernel.org/linux-block/[email protected]/
Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Keith Busch <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-mq-tag.c | 2 +- block/blk-mq-tag.h | 2 ++ block/blk-mq.c | 65 ++++++++++++++++++++++++++++++++++++++ include/linux/blk-mq.h | 1 + include/linux/cpuhotplug.h | 1 + 5 files changed, 70 insertions(+), 1 deletion(-)block/blk-mq-tag.c +1
-1@@ -325,7 +325,7 @@ static void bt_tags_for_each(struct blk_mq_tags *tags, struct sbitmap_queue *bt,* true to continue iterating tags, false to stop. * @priv: Will be passed as second argument to @fn. */ -static void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags, +void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags,busy_tag_iter_fn *fn,void *priv) {if (tags->nr_reserved_tags)block/blk-mq-tag.h +2
-0@@ -35,6 +35,8 @@ extern int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx,extern void blk_mq_tag_wakeup_all (struct blk_mq_tags *tags,bool );void blk_mq_queue_tag_busy_iter (struct request_queue *q,busy_iter_fn *fn,void *priv); +void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags, +busy_tag_iter_fn *fn,void *priv);static inline struct sbq_wait_state *bt_wait_ptr (struct sbitmap_queue *bt,struct blk_mq_hw_ctx *hctx)block/blk-mq.c +52
-0@@ -2206,6 +2206,61 @@ int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags,return -ENOMEM ; } +static bool blk_mq_count_inflight_rq (struct request *rq,void *data, +bool reserved) +{ +unsigned *count = data; + +if ((blk_mq_rq_state (rq) ==MQ_RQ_IN_FLIGHT )) + (*count)++; + +return true; +} + +static unsigned blk_mq_tags_inflight_rqs (struct blk_mq_tags *tags) +{ +unsigned count =0 ; + +blk_mq_all_tag_busy_iter (tags, blk_mq_count_inflight_rq, &count); + +return count; +} + +static void blk_mq_drain_inflight_rqs (struct blk_mq_hw_ctx *hctx) +{ +while (1 ) { +if (!blk_mq_tags_inflight_rqs (hctx->tags)) +break ; +msleep (5 ); + } +} + +static int blk_mq_hctx_notify_online (unsigned int cpu,struct hlist_node *node) +{ +struct blk_mq_hw_ctx *hctx =hlist_entry_safe (node, + struct blk_mq_hw_ctx, cpuhp_online); +unsigned prev_cpu =-1 ; + +while (true) { +unsigned next_cpu =cpumask_next_and (prev_cpu, hctx->cpumask, + cpu_online_mask); + +if (next_cpu >= nr_cpu_ids) +break ; + +/* return if there is other online CPU on this hctx */ +if (next_cpu != cpu) +return 0 ; + + prev_cpu = next_cpu; + } + +set_bit (BLK_MQ_S_INTERNAL_STOPPED , &hctx->state); +blk_mq_drain_inflight_rqs (hctx); +
Quoted 8 lines
block/blk-mq.c int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, · line 2266
Quoted 6 lines
block/blk-mq.c +3
-0+return 0 ; +} +/* * 'cpu' is going away. splice any existing rq_list entries from this * software queue to the hw queue dispatch list, and ensure that it Isn't that inverted?
The above comment is wrong, and you will see it is fixed in patch 4.
blk_mq_hctx_notify_dead() is called when the specified CPU is dead, and blk_mq_hctx_notify_online() is called before the CPU goes away during cpuhp path.
Quoted 2 lines
From the function I would assume it'll be called once the CPU is being set toe 'online', yet from the description I would have assumed the
No, blk_mq_init() only registers teardown callback for CPUHP_AP_BLK_MQ_ONLINE, that means blk_mq_hctx_notify_online() is only called when CPU is going away, still online, so we can stop queue for quiescing IO on this queue.
Quoted 2 lines
INTERNAL_STOPPEDbit is set when the cpu goes offline. Care to elaborate?
The idea is to quiesce queue in two stages:
1) in blk_mq_hctx_notify_online(), this CPU isn't dead yet, but is going to become dead, so we can wait for completion of in-flight IOs. Meantime stop the hw queue.
2) in blk_mq_hctx_notify_dead(), all CPUs of this hw queue have been dead, what we can do is to end the request and re-submit the IO, and new request will be mapped to another active hw queue because blk_mq_hctx_notify_dead() is always called from one online CPU.
Thanks,
Ming
PATCH v2 4/5 blk-mq: re-submit IO in case that hctx is dead
In reply to Ming LeiWhen all CPUs in one hctx are offline, we shouldn't run this hw queue for completing request any more.
So steal bios from the request, and resubmit them, and finally free the request in blk_mq_hctx_notify_dead().
--- block/blk-mq.c | 48 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 7 deletions(-)
block/blk-mq.c +41 -7
@@ -2261,10 +2261,30 @@ static int blk_mq_hctx_notify_online(unsigned int cpu, struct hlist_node *node)return 0 ; } +static void blk_mq_resubmit_io (struct request *rq) +{ +struct bio_list list; +struct bio *bio; + +bio_list_init (&list); +blk_steal_bios (&list, rq); + +while (true) { + bio =bio_list_pop (&list); +if (!bio) +break ; + +generic_make_request (bio); + } + +blk_mq_cleanup_rq (rq); +blk_mq_end_request (rq,0 ); +} +/* -* 'cpu' is going away. splice any existing rq_list entries from this -* software queue to the hw queue dispatch list, and ensure that it -* gets run. +* 'cpu' has gone away. If this hctx is dead, we can't dispatch request +* to the hctx any more, so steal bios from requests of this hctx, and +* re-submit them to the request queue, and free these requests finally. */ static int blk_mq_hctx_notify_dead (unsigned int cpu,struct hlist_node *node) {
@@ -2272,6 +2292,8 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)struct blk_mq_ctx *ctx;LIST_HEAD (tmp);enum hctx_type type; +bool hctx_dead; +struct request *rq; hctx = hlist_entry_safe(node, struct blk_mq_hw_ctx, cpuhp_dead); ctx =__blk_mq_get_ctx (hctx->queue, cpu);
@@ -2279,6 +2301,9 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)clear_bit (BLK_MQ_S_INTERNAL_STOPPED , &hctx->state); + hctx_dead =cpumask_first_and (hctx->cpumask, cpu_online_mask) >= + nr_cpu_ids; +spin_lock (&ctx->lock);if (!list_empty (&ctx->rq_lists[type])) {list_splice_init (&ctx->rq_lists[type], &tmp);
@@ -2289,11 +2314,20 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)if (list_empty (&tmp))return 0 ; -spin_lock (&hctx->lock); -list_splice_tail_init (&tmp, &hctx->dispatch); -spin_unlock (&hctx->lock); +if (!hctx_dead) { +spin_lock (&hctx->lock); +list_splice_tail_init (&tmp, &hctx->dispatch); +spin_unlock (&hctx->lock); +blk_mq_run_hw_queue (hctx, true); +return 0 ; + } + +while (!list_empty (&tmp)) { + rq = list_entry(tmp.next, struct request, queuelist); +list_del_init (&rq->queuelist); +blk_mq_resubmit_io (rq); + } -blk_mq_run_hw_queue (hctx, true);return 0 ; }
Signature · 1 line
2.20.1
Re: [PATCH V2 4/5] blk-mq: re-submit IO in case that hctx is dead
Quoted 89 lines · Ming Lei
When all CPUs in one hctx are offline, we shouldn't run this hw queue for completing request any more.
So steal bios from the request, and resubmit them, and finally free the request in
blk_mq_hctx_notify_dead().Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Keith Busch <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 48 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 7 deletions(-)block/blk-mq.c +36
-6@@ -2261,10 +2261,30 @@ static int blk_mq_hctx_notify_online(unsigned int cpu, struct hlist_node *node)return 0 ; } +static void blk_mq_resubmit_io (struct request *rq) +{ +struct bio_list list; +struct bio *bio; + +bio_list_init (&list); +blk_steal_bios (&list, rq); + +while (true) { + bio =bio_list_pop (&list); +if (!bio) +break ; + +generic_make_request (bio); + } + +blk_mq_cleanup_rq (rq); +blk_mq_end_request (rq,0 ); +} +/* -* 'cpu' is going away. splice any existing rq_list entries from this -* software queue to the hw queue dispatch list, and ensure that it -* gets run. +* 'cpu' has gone away. If this hctx is dead, we can't dispatch request +* to the hctx any more, so steal bios from requests of this hctx, and +* re-submit them to the request queue, and free these requests finally. */ static int blk_mq_hctx_notify_dead (unsigned int cpu,struct hlist_node *node) {@@ -2272,6 +2292,8 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)struct blk_mq_ctx *ctx;LIST_HEAD (tmp);enum hctx_type type; +bool hctx_dead; +struct request *rq; hctx = hlist_entry_safe(node, struct blk_mq_hw_ctx, cpuhp_dead); ctx =__blk_mq_get_ctx (hctx->queue, cpu);@@ -2279,6 +2301,9 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)clear_bit (BLK_MQ_S_INTERNAL_STOPPED , &hctx->state); + hctx_dead =cpumask_first_and (hctx->cpumask, cpu_online_mask) >= + nr_cpu_ids; +spin_lock (&ctx->lock);if (!list_empty (&ctx->rq_lists[type])) {list_splice_init (&ctx->rq_lists[type], &tmp);@@ -2289,11 +2314,20 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)if (list_empty (&tmp))return 0 ; -spin_lock (&hctx->lock); -list_splice_tail_init (&tmp, &hctx->dispatch); -spin_unlock (&hctx->lock); +if (!hctx_dead) { +spin_lock (&hctx->lock); +list_splice_tail_init (&tmp, &hctx->dispatch); +spin_unlock (&hctx->lock); +blk_mq_run_hw_queue (hctx, true); +return 0 ; + } +
block/blk-mq.c static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node) · line 2333
Quoted 10 lines
block/blk-mq.c +5
-1+while (!list_empty (&tmp)) { + rq = list_entry(tmp.next, struct request, queuelist); +list_del_init (&rq->queuelist); +blk_mq_resubmit_io (rq); + } -blk_mq_run_hw_queue (hctx, true);return 0 ; }
So what happens when all CPUs assigned to a hardware queue go offline? Wouldn't blk_steal_bios() etc resend the I/O to the same hw queue, causing an infinite loop?
Don't we have to rearrange the hardware queues here?
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
Re: [PATCH V2 4/5] blk-mq: re-submit IO in case that hctx is dead
Quoted 95 lines · Hannes Reinecke
Quoted 94 lines · Ming Lei
When all CPUs in one hctx are offline, we shouldn't run this hw queue for completing request any more.
So steal bios from the request, and resubmit them, and finally free the request in
blk_mq_hctx_notify_dead().Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Keith Busch <[email protected]>Signed-off-by Ming Lei <[email protected]>--- block/blk-mq.c | 48 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 7 deletions(-)block/blk-mq.c +41
-6@@ -2261,10 +2261,30 @@ static int blk_mq_hctx_notify_online(unsigned int cpu, struct hlist_node *node)return 0 ; } +static void blk_mq_resubmit_io (struct request *rq) +{ +struct bio_list list; +struct bio *bio; + +bio_list_init (&list); +blk_steal_bios (&list, rq); + +while (true) { + bio =bio_list_pop (&list); +if (!bio) +break ; + +generic_make_request (bio); + } + +blk_mq_cleanup_rq (rq); +blk_mq_end_request (rq,0 ); +} +/* -* 'cpu' is going away. splice any existing rq_list entries from this -* software queue to the hw queue dispatch list, and ensure that it -* gets run. +* 'cpu' has gone away. If this hctx is dead, we can't dispatch request +* to the hctx any more, so steal bios from requests of this hctx, and +* re-submit them to the request queue, and free these requests finally. */ static int blk_mq_hctx_notify_dead (unsigned int cpu,struct hlist_node *node) {@@ -2272,6 +2292,8 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)struct blk_mq_ctx *ctx;LIST_HEAD (tmp);enum hctx_type type; +bool hctx_dead; +struct request *rq; hctx = hlist_entry_safe(node, struct blk_mq_hw_ctx, cpuhp_dead); ctx =__blk_mq_get_ctx (hctx->queue, cpu);@@ -2279,6 +2301,9 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)clear_bit (BLK_MQ_S_INTERNAL_STOPPED , &hctx->state); + hctx_dead =cpumask_first_and (hctx->cpumask, cpu_online_mask) >= + nr_cpu_ids; +spin_lock (&ctx->lock);if (!list_empty (&ctx->rq_lists[type])) {list_splice_init (&ctx->rq_lists[type], &tmp);@@ -2289,11 +2314,20 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)if (list_empty (&tmp))return 0 ; -spin_lock (&hctx->lock); -list_splice_tail_init (&tmp, &hctx->dispatch); -spin_unlock (&hctx->lock); +if (!hctx_dead) { +spin_lock (&hctx->lock); +list_splice_tail_init (&tmp, &hctx->dispatch); +spin_unlock (&hctx->lock); +blk_mq_run_hw_queue (hctx, true); +return 0 ; + } + +while (!list_empty (&tmp)) { + rq = list_entry(tmp.next, struct request, queuelist); +list_del_init (&rq->queuelist); +blk_mq_resubmit_io (rq); + }
Quoted 8 lines
block/blk-mq.c static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node) · line 2333
Quoted 5 lines
block/blk-mq.c +0
-1-blk_mq_run_hw_queue (hctx, true);return 0 ; }So what happens when all CPUs assigned to a hardware queue go offline? Wouldn't
blk_steal_bios()etc resend the I/O to the same hw queue, causing an infinite loop?
No, blk_mq_hctx_notify_dead() is always called on one online CPU, so the I/O won't be remapped to same hw queue.
Quoted 2 lines
Don't we have to rearrange the hardware queues here?
No, we use static queue mapping for managed IRQ, all possible CPUs have been spread on all hw queues.
Thanks,
Ming
PATCH v2 5/5 blk-mq: handle requests dispatched from IO scheduler in case that hctx is dead
In reply to Ming LeiIf hctx becomes dead, all in-queue IO requests aimed at this hctx have to be re-submitted, so cover requests queued in scheduler queue.
--- block/blk-mq.c | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-)
block/blk-mq.c +25 -5
@@ -2294,6 +2294,7 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)enum hctx_type type;bool hctx_dead;struct request *rq; +struct elevator_queue *e; hctx = hlist_entry_safe(node, struct blk_mq_hw_ctx, cpuhp_dead); ctx =__blk_mq_get_ctx (hctx->queue, cpu);
@@ -2304,12 +2305,31 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node) hctx_dead =cpumask_first_and (hctx->cpumask, cpu_online_mask) >= nr_cpu_ids; -spin_lock (&ctx->lock); -if (!list_empty (&ctx->rq_lists[type])) { -list_splice_init (&ctx->rq_lists[type], &tmp); -blk_mq_hctx_clear_pending (hctx, ctx); + e = hctx->queue->elevator; +if (!e) { +spin_lock (&ctx->lock); +if (!list_empty (&ctx->rq_lists[type])) { +list_splice_init (&ctx->rq_lists[type], &tmp); +blk_mq_hctx_clear_pending (hctx, ctx); + } +spin_unlock (&ctx->lock); + }else if (hctx_dead) { +LIST_HEAD (sched_tmp); + +while ((rq = e->type->ops.dispatch_request (hctx))) { +if (rq->mq_hctx != hctx) +list_add (&rq->queuelist, &sched_tmp); +else +list_add (&rq->queuelist, &tmp); + } + +while (!list_empty (&sched_tmp)) { + rq = list_entry(sched_tmp.next, struct request, + queuelist); +list_del_init (&rq->queuelist); +blk_mq_sched_insert_request (rq, true, true, true); + } } -spin_unlock (&ctx->lock);if (list_empty (&tmp))return 0 ;
Signature · 1 line
2.20.1
Hi John,
Quoted 54 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very beginning:The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"But no drivers or blk-mq do that before one hctx becomes dead(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(), then these IO will be mapped to other live hctx for dispatchPlease comment & review, thanks!
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and MinwooMing Lei (5):
blk-mq: add new state ofBLK_MQ_S_INTERNAL_STOPPED
blk-mq: add blk-mq flag ofBLK_MQ_F_NO_MANAGED_IRQ
blk-mq: stop to handle IO before hctx's all CPUs become offline
blk-mq: re-submit IO in case that hctx is dead
blk-mq: handle requests dispatched from IO scheduler in case that hctx
is deadblock/blk-mq-debugfs.c | 2 + block/blk-mq-tag.c | 2 +- block/blk-mq-tag.h | 2 + block/blk-mq.c | 143 +++++++++++++++++++++++++++++++++---- block/blk-mq.h | 3 +- drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 5 ++ include/linux/cpuhotplug.h | 1 + 9 files changed, 146 insertions(+), 16 deletions(-)
Quoted 8 lines
Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Keith Busch <[email protected]>Signature · 1 line
2.20.1
Sorry for forgetting to Cc you.
Thanks,
Ming
Quoted 66 lines · Ming Lei
Hi John,
Quoted 54 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very beginning:The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"But no drivers or blk-mq do that before one hctx becomes dead(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(), then these IO will be mapped to other live hctx for dispatchPlease comment & review, thanks!
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and MinwooMing Lei (5):
blk-mq: add new state ofBLK_MQ_S_INTERNAL_STOPPED
blk-mq: add blk-mq flag ofBLK_MQ_F_NO_MANAGED_IRQ
blk-mq: stop to handle IO before hctx's all CPUs become offline
blk-mq: re-submit IO in case that hctx is dead
blk-mq: handle requests dispatched from IO scheduler in case that hctx
is deadblock/blk-mq-debugfs.c | 2 + block/blk-mq-tag.c | 2 +- block/blk-mq-tag.h | 2 + block/blk-mq.c | 143 +++++++++++++++++++++++++++++++++---- block/blk-mq.h | 3 +- drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 5 ++ include/linux/cpuhotplug.h | 1 + 9 files changed, 146 insertions(+), 16 deletions(-)Quoted 8 lines
Cc: Bart Van Assche <[email protected]>
Cc: Hannes Reinecke <[email protected]>
Cc: Christoph Hellwig <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Keith Busch <[email protected]>
--
2.20.1
Quoted 1 line
Sorry for forgetting to Cc you.
Already subscribed :)
I don't mean to hijack this thread, but JFYI we're getting around to test https://github.com/ming1/linux/commits/v5.2-rc-host-tags-V2 - unfortunately we're still seeing a performance regression. I can't see where it's coming from. We're double-checking the test though.
Thanks,
John
Quoted 7 lines
Thanks,
Ming.
Quoted 67 lines · John Garry
Quoted 66 lines · Ming Lei
Hi John,
Quoted 62 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very beginning:The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"But no drivers or blk-mq do that before one hctx becomes dead(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(), then these IO will be mapped to other live hctx for dispatchPlease comment & review, thanks!
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and MinwooMing Lei (5):
blk-mq: add new state ofBLK_MQ_S_INTERNAL_STOPPED
blk-mq: add blk-mq flag ofBLK_MQ_F_NO_MANAGED_IRQ
blk-mq: stop to handle IO before hctx's all CPUs become offline
blk-mq: re-submit IO in case that hctx is dead
blk-mq: handle requests dispatched from IO scheduler in case that hctx
is deadblock/blk-mq-debugfs.c | 2 + block/blk-mq-tag.c | 2 +- block/blk-mq-tag.h | 2 + block/blk-mq.c | 143 +++++++++++++++++++++++++++++++++---- block/blk-mq.h | 3 +- drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 5 ++ include/linux/cpuhotplug.h | 1 + 9 files changed, 146 insertions(+), 16 deletions(-)Cc: Bart Van Assche <[email protected]>
Cc: Hannes Reinecke <[email protected]>
Cc: Christoph Hellwig <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Keith Busch <[email protected]>
--
2.20.1
Quoted 8 lines
Quoted 1 line
Sorry for forgetting to Cc you.
Already subscribed :)
I don't mean to hijack this thread, but JFYI we're getting around to test
https://github.com/ming1/linux/commits/v5.2-rc-host-tags-V2 - unfortunately we're still seeing a performance regression. I can't see where it's coming from. We're double-checking the test though.
host-tag patchset is only for several particular drivers which use private reply queue as completion queue.
This patchset is for handling generic blk-mq CPU hotplug issue, and the several particular scsi drivers(hisi_sas_v3, hpsa, megaraid_sas and mp3sas) won't be covered so far.
I'd suggest to move on for generic blk-mq devices first given now blk-mq is the only request IO path now.
There are at least two choices for us to handle drivers/devices with private completion queue:
1) host-tags
- performance issue shouldn't be hard to solve, given it is same with with single tags in theory, and just corner cases is there.
What I am not glad with this approach is that blk-mq-tag code becomes mess.
2) private callback
- we could define private callback to drain each completion queue in driver simply.
- problem is that the four drivers have to duplicate the same job
Thanks,
Ming
Quoted 35 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very beginning:The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"But no drivers or blk-mq do that before one hctx becomes dead(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(), then these IO will be mapped to other live hctx for dispatchPlease comment & review, thanks!
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and Minwoo
Quoted 7 lines
Ming Lei (5):
blk-mq: add new state ofBLK_MQ_S_INTERNAL_STOPPED
blk-mq: add blk-mq flag ofBLK_MQ_F_NO_MANAGED_IRQ
blk-mq: stop to handle IO before hctx's all CPUs become offline
blk-mq: re-submit IO in case that hctx is dead
blk-mq: handle requests dispatched from IO scheduler in case that hctx
is dead
Hi Ming,
This looks to fix the hotplug issue for me.
Previously I could manufacture a scenario while running fio where I got IO timeouts, like this:
root@(none)$ echo 0 > ./sys/devices/system/cpu/cpu0/online [ 296.897627] process 891 (fio) no longer affine to cpu0 [ 296.898488] process 893 (fio) no longer affine to cpu0 [ 296.910270] process 890 (fio) no longer affine to cpu0 [ 296.927322] IRQ 775: no longer affine to CPU0 [ 296.932762] CPU0: shutdown [ 296.935469] psci: CPU0 killed. root@(none)$ [ 326.971962] sas: Enter sas_scsi_recover_host busy: 61 failed: 61 [ 326.977978] sas: sas_scsi_find_task: aborting task 0x00000000e2cdc79b root@(none)$ [ 333.047964] hisi_sas_v3_hw 0000:74:02.0: internal task abort: timeout and not done. [ 333.055616] hisi_sas_v3_hw 0000:74:02.0: abort task: internal abort (-5) [ 333.062306] sas: sas_scsi_find_task: querying task 0x00000000e2cdc79b [ 333.068776] sas: sas_scsi_find_task: task 0x00000000e2cdc79b not at LU [ 333.075295] sas: task 0x00000000e2cdc79b is not at LU: I_T recover [ 333.081464] sas: I_T nexus reset for dev 5000c500a7b95a49
Please notice the 30-second delay for the SCSI IO timeout.
And now I don't see it; here's a sample for irq shutdown:
root@(none)$ echo 0 > ./sys/devices/system/cpu/cpu0/online [ 344.608148] process 849 (fio) no longer affine to cpu0 [ 344.608639] process 848 (fio) no longer affine to cpu0 [ 344.609454] process 850 (fio) no longer affine to cpu0 [ 344.643481] process 847 (fio) no longer affine to cpu0 [ 346.213842] IRQ 775: no longer affine to CPU0 [ 346.219712] CPU0: shutdown [ 346.222425] psci: CPU0 killed.
Please notice the ~1.5s pause, which would be the queue draining.
So FWIW:
JFYI, I tested on 5.3-rc5 and cherry-picked https://github.com/ming1/linux/commit/0d2cd3c99bb0fe81d2c0ca5d68e02bdc4521d4d6 and "blk-mq: add callback of .cleanup_rq".
Cheers,
John
Quoted 18 lines
block/blk-mq-debugfs.c | 2 + block/blk-mq-tag.c | 2 +- block/blk-mq-tag.h | 2 + block/blk-mq.c | 143 +++++++++++++++++++++++++++++++++---- block/blk-mq.h | 3 +- drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 5 ++ include/linux/cpuhotplug.h | 1 + 9 files changed, 146 insertions(+), 16 deletions(-)Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Keith Busch <[email protected]>
Quoted 89 lines · John Garry
Quoted 40 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very
beginning:The driver/subsystem has to quiesce the interrupt line and the
associated
queue _before_ it gets shutdown in CPU unplug and not fiddle
with it
until it's restarted by the core when the CPU is plugged in again.
"But no drivers or blk-mq do that before one hctx becomes dead(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(),
then these IO will be mapped to other live hctx for dispatchPlease comment & review, thanks!
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and MinwooQuoted 7 lines
Ming Lei (5):
blk-mq: add new state ofBLK_MQ_S_INTERNAL_STOPPED
blk-mq: add blk-mq flag ofBLK_MQ_F_NO_MANAGED_IRQ
blk-mq: stop to handle IO before hctx's all CPUs become offline
blk-mq: re-submit IO in case that hctx is dead
blk-mq: handle requests dispatched from IO scheduler in case that hctx
is deadHi Ming,
This looks to fix the hotplug issue for me.
Previously I could manufacture a scenario while running fio where I got IO timeouts, like this:
root@(none)$ echo 0 > ./sys/devices/system/cpu/cpu0/online [ 296.897627] process 891 (fio) no longer affine to cpu0 [ 296.898488] process 893 (fio) no longer affine to cpu0 [ 296.910270] process 890 (fio) no longer affine to cpu0 [ 296.927322] IRQ 775: no longer affine to CPU0 [ 296.932762] CPU0: shutdown [ 296.935469] psci: CPU0 killed. root@(none)$ [ 326.971962] sas: Enter sas_scsi_recover_host busy: 61 failed: 61 [ 326.977978] sas: sas_scsi_find_task: aborting task 0x00000000e2cdc79b root@(none)$ [ 333.047964] hisi_sas_v3_hw 0000:74:02.0: internal task abort: timeout and not done. [ 333.055616] hisi_sas_v3_hw 0000:74:02.0: abort task: internal abort (-5) [ 333.062306] sas: sas_scsi_find_task: querying task 0x00000000e2cdc79b [ 333.068776] sas: sas_scsi_find_task: task 0x00000000e2cdc79b not at LU [ 333.075295] sas: task 0x00000000e2cdc79b is not at LU: I_T recover [ 333.081464] sas: I_T nexus reset for dev 5000c500a7b95a49Please notice the 30-second delay for the SCSI IO timeout.
And now I don't see it; here's a sample for irq shutdown:
root@(none)$ echo 0 > ./sys/devices/system/cpu/cpu0/online [ 344.608148] process 849 (fio) no longer affine to cpu0 [ 344.608639] process 848 (fio) no longer affine to cpu0 [ 344.609454] process 850 (fio) no longer affine to cpu0 [ 344.643481] process 847 (fio) no longer affine to cpu0 [ 346.213842] IRQ 775: no longer affine to CPU0 [ 346.219712] CPU0: shutdown [ 346.222425] psci: CPU0 killed.Please notice the ~1.5s pause, which would be the queue draining.
So FWIW:
Quoted 8 lines
Tested-by John Garry <[email protected]>JFYI, I tested on 5.3-rc5 and cherry-picked
https://github.com/ming1/linux/commit/0d2cd3c99bb0fe81d2c0ca5d68e02bdc4521d4d6
and "blk-mq: add callback of .cleanup_rq".Cheers,
John
Hi Jens,
I don't mean to be pushy, but can we consider to get these patches from Ming merged?
As above, I tested on my SCSI driver and it works. I also tested on an NVMe disk, and it solves the condition which generates this message:
root@(none)$ echo 0 > /sys/devices/system/cpu/cpu2/online [ 465.635960] CPU2: shutdown [ 465.638662] psci: CPU2 killed. [ 111.381653] nvme nvme0: I/O 705 QID 18 timeout, completion polled
(that's on top off v5.4-rc1)
Thanks,
John
Quoted 3 lines
Quoted 3 lines
block/blk-mq-debugfs.c | 2 +
Jens Axboe
maintainerQuoted 104 lines · John Garry
Quoted 89 lines · John Garry
Quoted 40 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very
beginning:The driver/subsystem has to quiesce the interrupt line and the
associated
queue _before_ it gets shutdown in CPU unplug and not fiddle
with it
until it's restarted by the core when the CPU is plugged in again.
"But no drivers or blk-mq do that before one hctx becomes dead(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(),
then these IO will be mapped to other live hctx for dispatchPlease comment & review, thanks!
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and MinwooQuoted 7 lines
Ming Lei (5):
blk-mq: add new state ofBLK_MQ_S_INTERNAL_STOPPED
blk-mq: add blk-mq flag ofBLK_MQ_F_NO_MANAGED_IRQ
blk-mq: stop to handle IO before hctx's all CPUs become offline
blk-mq: re-submit IO in case that hctx is dead
blk-mq: handle requests dispatched from IO scheduler in case that hctx
is deadHi Ming,
This looks to fix the hotplug issue for me.
Previously I could manufacture a scenario while running fio where I got IO timeouts, like this:
root@(none)$ echo 0 > ./sys/devices/system/cpu/cpu0/online [ 296.897627] process 891 (fio) no longer affine to cpu0 [ 296.898488] process 893 (fio) no longer affine to cpu0 [ 296.910270] process 890 (fio) no longer affine to cpu0 [ 296.927322] IRQ 775: no longer affine to CPU0 [ 296.932762] CPU0: shutdown [ 296.935469] psci: CPU0 killed. root@(none)$ [ 326.971962] sas: Enter sas_scsi_recover_host busy: 61 failed: 61 [ 326.977978] sas: sas_scsi_find_task: aborting task 0x00000000e2cdc79b root@(none)$ [ 333.047964] hisi_sas_v3_hw 0000:74:02.0: internal task abort: timeout and not done. [ 333.055616] hisi_sas_v3_hw 0000:74:02.0: abort task: internal abort (-5) [ 333.062306] sas: sas_scsi_find_task: querying task 0x00000000e2cdc79b [ 333.068776] sas: sas_scsi_find_task: task 0x00000000e2cdc79b not at LU [ 333.075295] sas: task 0x00000000e2cdc79b is not at LU: I_T recover [ 333.081464] sas: I_T nexus reset for dev 5000c500a7b95a49Please notice the 30-second delay for the SCSI IO timeout.
And now I don't see it; here's a sample for irq shutdown:
root@(none)$ echo 0 > ./sys/devices/system/cpu/cpu0/online [ 344.608148] process 849 (fio) no longer affine to cpu0 [ 344.608639] process 848 (fio) no longer affine to cpu0 [ 344.609454] process 850 (fio) no longer affine to cpu0 [ 344.643481] process 847 (fio) no longer affine to cpu0 [ 346.213842] IRQ 775: no longer affine to CPU0 [ 346.219712] CPU0: shutdown [ 346.222425] psci: CPU0 killed.Please notice the ~1.5s pause, which would be the queue draining.
So FWIW:
Quoted 8 lines
Tested-by John Garry <[email protected]>JFYI, I tested on 5.3-rc5 and cherry-picked
https://github.com/ming1/linux/commit/0d2cd3c99bb0fe81d2c0ca5d68e02bdc4521d4d6
and "blk-mq: add callback of .cleanup_rq".Cheers,
JohnHi Jens,
I don't mean to be pushy, but can we consider to get these patches from Ming merged?
Quoted 8 lines
As above, I tested on my SCSI driver and it works. I also tested on an NVMe disk, and it solves the condition which generates this message:
root@(none)$ echo 0 > /sys/devices/system/cpu/cpu2/online [ 465.635960] CPU2: shutdown [ 465.638662] psci: CPU2 killed. [ 111.381653] nvme nvme0: I/O 705 QID 18 timeout, completion polled(that's on top off v5.4-rc1)
Ming, can you repost the series?
Signature · 1 line
Jens Axboe
Quoted 112 lines · Jens Axboe
Quoted 111 lines · John Garry
Quoted 89 lines · John Garry
Quoted 40 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very
beginning:The driver/subsystem has to quiesce the interrupt line and the
associated
queue _before_ it gets shutdown in CPU unplug and not fiddle
with it
until it's restarted by the core when the CPU is plugged in again.
"But no drivers or blk-mq do that before one hctx becomes dead(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(),
then these IO will be mapped to other live hctx for dispatchPlease comment & review, thanks!
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and MinwooQuoted 7 lines
Ming Lei (5):
blk-mq: add new state ofBLK_MQ_S_INTERNAL_STOPPED
blk-mq: add blk-mq flag ofBLK_MQ_F_NO_MANAGED_IRQ
blk-mq: stop to handle IO before hctx's all CPUs become offline
blk-mq: re-submit IO in case that hctx is dead
blk-mq: handle requests dispatched from IO scheduler in case that hctx
is deadHi Ming,
This looks to fix the hotplug issue for me.
Previously I could manufacture a scenario while running fio where I got IO timeouts, like this:
root@(none)$ echo 0 > ./sys/devices/system/cpu/cpu0/online [ 296.897627] process 891 (fio) no longer affine to cpu0 [ 296.898488] process 893 (fio) no longer affine to cpu0 [ 296.910270] process 890 (fio) no longer affine to cpu0 [ 296.927322] IRQ 775: no longer affine to CPU0 [ 296.932762] CPU0: shutdown [ 296.935469] psci: CPU0 killed. root@(none)$ [ 326.971962] sas: Enter sas_scsi_recover_host busy: 61 failed: 61 [ 326.977978] sas: sas_scsi_find_task: aborting task 0x00000000e2cdc79b root@(none)$ [ 333.047964] hisi_sas_v3_hw 0000:74:02.0: internal task abort: timeout and not done. [ 333.055616] hisi_sas_v3_hw 0000:74:02.0: abort task: internal abort (-5) [ 333.062306] sas: sas_scsi_find_task: querying task 0x00000000e2cdc79b [ 333.068776] sas: sas_scsi_find_task: task 0x00000000e2cdc79b not at LU [ 333.075295] sas: task 0x00000000e2cdc79b is not at LU: I_T recover [ 333.081464] sas: I_T nexus reset for dev 5000c500a7b95a49Please notice the 30-second delay for the SCSI IO timeout.
And now I don't see it; here's a sample for irq shutdown:
root@(none)$ echo 0 > ./sys/devices/system/cpu/cpu0/online [ 344.608148] process 849 (fio) no longer affine to cpu0 [ 344.608639] process 848 (fio) no longer affine to cpu0 [ 344.609454] process 850 (fio) no longer affine to cpu0 [ 344.643481] process 847 (fio) no longer affine to cpu0 [ 346.213842] IRQ 775: no longer affine to CPU0 [ 346.219712] CPU0: shutdown [ 346.222425] psci: CPU0 killed.Please notice the ~1.5s pause, which would be the queue draining.
So FWIW:
Quoted 8 lines
Tested-by John Garry <[email protected]>JFYI, I tested on 5.3-rc5 and cherry-picked
https://github.com/ming1/linux/commit/0d2cd3c99bb0fe81d2c0ca5d68e02bdc4521d4d6
and "blk-mq: add callback of .cleanup_rq".Cheers,
JohnHi Jens,
I don't mean to be pushy, but can we consider to get these patches from Ming merged?
As above, I tested on my SCSI driver and it works. I also tested on an NVMe disk, and it solves the condition which generates this message:
root@(none)$ echo 0 > /sys/devices/system/cpu/cpu2/online [ 465.635960] CPU2: shutdown [ 465.638662] psci: CPU2 killed. [ 111.381653] nvme nvme0: I/O 705 QID 18 timeout, completion polled
Quoted 3 lines
Quoted 1 line
(that's on top off v5.4-rc1)
Ming, can you repost the series?
It has been resent out just now.
Thanks,
Ming