blk-mq: improvement CPU hotplug
Subsystems (3)
v9 cover blk-mq: improvement CPU hotplug
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very beginning:
The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"
But no drivers or blk-mq do that before one hctx becomes inactive(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see blk_mq_hctx_notify_dead().
This patchset tries to address the issue by two stages:
1) add one new cpuhp state of CPUHP_AP_BLK_MQ_ONLINE
- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of CPUHP_BLK_MQ_DEAD after the hctx becomes dead
- steal bios from the request, and resubmit them via generic_make_request(), then these IO will be mapped to other live hctx for dispatch
Thanks John Garry for running lots of tests on arm64 with this patchset and co-working on investigating all kinds of issues.
Thanks Christoph's review on V7.
Please comment & review, thanks!
https://github.com/ming1/linux/commits/v5.7-rc-blk-mq-improve-cpu-hotplug
V9:
- add Reviewed-by tag
- document more on memory barrier usage between getting driver tag and handling cpu offline(7/11)
- small code cleanup as suggested by Chritoph(7/11)
- rebase against for-5.8/block(1/11, 2/11)
V8:
- add patches to share code with blk_rq_prep_clone
- code re-organization as suggested by Christoph, most of them are in 04/11, 10/11
- add reviewed-by tag
V7:
- fix updating .nr_active in get_driver_tag
- add hctx->cpumask check in cpuhp handler
- only drain requests which tag is >= 0
- pass more aggressive cpuhotplug&io test
V6:
- simplify getting driver tag, so that we can drain in-flight requests correctly without using synchronize_rcu()
- handle re-submission of flush & passthrough request correctly
V5:
- rename BLK_MQ_S_INTERNAL_STOPPED as BLK_MQ_S_INACTIVE
- re-factor code for re-submit requests in cpu dead hotplug handler
- address requeue corner case
V4:
- resubmit IOs in dispatch list in case that this hctx is dead
V3:
- re-organize patch 2 & 3 a bit for addressing Hannes's comment
- fix patch 4 for avoiding potential deadlock, as found by Hannes
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and Minwoo
Ming Lei (11):
block: clone nr_integrity_segments and write_hint in blk_rq_prep_clone
block: add helper for copying request
blk-mq: mark blk_mq_get_driver_tag as static
blk-mq: assign rq->tag in blk_mq_get_driver_tag
blk-mq: support rq filter callback when iterating rqs
blk-mq: prepare for draining IO when hctx's all CPUs are offline
blk-mq: stop to handle IO and drain IO before hctx becomes inactive
block: add blk_end_flush_machinery
blk-mq: add blk_mq_hctx_handle_dead_cpu for handling cpu dead
blk-mq: re-submit IO in case that hctx is inactive
block: deactivate hctx when the hctx is actually inactive
block/blk-core.c | 27 ++- block/blk-flush.c | 141 ++++++++++++--- block/blk-mq-debugfs.c | 2 + block/blk-mq-tag.c | 39 ++-- block/blk-mq-tag.h | 4 + block/blk-mq.c | 352 +++++++++++++++++++++++++++++-------- block/blk-mq.h | 22 ++- block/blk.h | 11 +- drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 6 + include/linux/cpuhotplug.h | 1 + 12 files changed, 477 insertions(+), 132 deletions(-)
Signature · 1 line
2.25.2
PATCH v9 1/11 block: clone nr_integrity_segments and write_hint in blk_rq_prep_clone
So far blk_rq_prep_clone() is only used for setup one underlying cloned request from dm-rq request. block intetrity can be enabled for both dm-rq and the underlying queues, so it is reasonable to clone rq's nr_integrity_segments. Also write_hint is from bio, it should have been cloned too.
So clone nr_integrity_segments and write_hint in blk_rq_prep_clone.
--- block/blk-core.c | 4 ++++ 1 file changed, 4 insertions(+)
block/blk-core.c +4 -0
@@ -1630,8 +1630,12 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src, rq->rq_flags |=RQF_SPECIAL_PAYLOAD ; rq->special_vec = rq_src->special_vec; } +#ifdef CONFIG_BLK_DEV_INTEGRITY + rq->nr_integrity_segments = rq_src->nr_integrity_segments; +#endif rq->nr_phys_segments = rq_src->nr_phys_segments; rq->ioprio = rq_src->ioprio; + rq->write_hint = rq_src->write_hint;return 0 ;
Signature · 1 line
2.25.2
PATCH v9 2/11 block: add helper for copying request
In reply to Ming LeiAdd one new helper of blk_rq_copy_request() to copy request, and the helper will be used in this patch for re-submitting request, so make it as a block layer internal helper.
--- block/blk-core.c | 31 ++++++++++++++++++------------- block/blk.h | 2 ++ 2 files changed, 20 insertions(+), 13 deletions(-)
block/blk-core.c +18 -13
@@ -1581,6 +1581,23 @@ void blk_rq_unprep_clone(struct request *rq) }EXPORT_SYMBOL_GPL (blk_rq_unprep_clone); +void blk_rq_copy_request (struct request *rq,struct request *rq_src) +{ +/* Copy attributes of the original request to the clone request. */ + rq->__sector =blk_rq_pos (rq_src); + rq->__data_len =blk_rq_bytes (rq_src); +if (rq_src->rq_flags &RQF_SPECIAL_PAYLOAD ) { + rq->rq_flags |=RQF_SPECIAL_PAYLOAD ; + rq->special_vec = rq_src->special_vec; + } +#ifdef CONFIG_BLK_DEV_INTEGRITY + rq->nr_integrity_segments = rq_src->nr_integrity_segments; +#endif + rq->nr_phys_segments = rq_src->nr_phys_segments; + rq->ioprio = rq_src->ioprio; + rq->write_hint = rq_src->write_hint; +} +/** * blk_rq_prep_clone - Helper function to setup clone request * @rq: the request to be setup
@@ -1623,19 +1640,7 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src, rq->bio = rq->biotail = bio; } -/* Copy attributes of the original request to the clone request. */ - rq->__sector =blk_rq_pos (rq_src); - rq->__data_len =blk_rq_bytes (rq_src); -if (rq_src->rq_flags &RQF_SPECIAL_PAYLOAD ) { - rq->rq_flags |=RQF_SPECIAL_PAYLOAD ; - rq->special_vec = rq_src->special_vec; - } -#ifdef CONFIG_BLK_DEV_INTEGRITY - rq->nr_integrity_segments = rq_src->nr_integrity_segments; -#endif - rq->nr_phys_segments = rq_src->nr_phys_segments; - rq->ioprio = rq_src->ioprio; - rq->write_hint = rq_src->write_hint; +blk_rq_copy_request (rq, rq_src);return 0 ;
block/blk.h +2 -0
@@ -120,6 +120,8 @@ static inline void blk_rq_bio_prep(struct request *rq, struct bio *bio, rq->rq_disk = bio->bi_disk; } +void blk_rq_copy_request (struct request *rq,struct request *rq_src); +#ifdef CONFIG_BLK_DEV_INTEGRITY void blk_flush_integrity (void );bool __bio_integrity_endio (struct bio *);
Signature · 1 line
2.25.2
PATCH v9 3/11 blk-mq: mark blk_mq_get_driver_tag as static
In reply to Ming LeiNow all callers of blk_mq_get_driver_tag are in blk-mq.c, so mark it as static.
--- block/blk-mq.c | 2 +- block/blk-mq.h | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)
block/blk-mq.c +1 -1
@@ -1015,7 +1015,7 @@ static inline unsigned int queued_to_index(unsigned int queued)return min (BLK_MQ_MAX_DISPATCH_ORDER -1 ,ilog2 (queued) +1 ); } -bool blk_mq_get_driver_tag (struct request *rq) +static bool blk_mq_get_driver_tag (struct request *rq) {struct blk_mq_alloc_data data = { .q = rq->q,
block/blk-mq.h +0 -1
@@ -44,7 +44,6 @@ bool blk_mq_dispatch_rq_list(struct request_queue *, struct list_head *, bool);void blk_mq_add_to_requeue_list (struct request *rq,bool at_head,bool kick_requeue_list);void blk_mq_flush_busy_ctxs (struct blk_mq_hw_ctx *hctx,struct list_head *list); -bool blk_mq_get_driver_tag (struct request *rq);struct request *blk_mq_dequeue_from_ctx (struct blk_mq_hw_ctx *hctx,struct blk_mq_ctx *start);
Signature · 1 line
2.25.2
PATCH v9 4/11 blk-mq: assign rq->tag in blk_mq_get_driver_tag
In reply to Ming LeiEspecially for none elevator, rq->tag is assigned after the request is allocated, so there isn't any way to figure out if one request is in being dispatched. Also the code path wrt. driver tag becomes a bit difference between none and io scheduler.
When one hctx becomes inactive, we have to prevent any request from being dispatched to LLD. And get driver tag provides one perfect chance to do that. Meantime we can drain any such requests by checking if rq->tag is assigned.
So only assign rq->tag until blk_mq_get_driver_tag() is called.
This way also simplifies code of dealing with driver tag a lot.
--- block/blk-flush.c | 18 ++---------- block/blk-mq.c | 75 ++++++++++++++++++++++++----------------------- block/blk-mq.h | 21 +++++++------ block/blk.h | 5 ---- 4 files changed, 51 insertions(+), 68 deletions(-)
block/blk-flush.c +3 -15
@@ -236,13 +236,8 @@ static void flush_end_io(struct request *flush_rq, blk_status_t error) error = fq->rq_status; hctx = flush_rq->mq_hctx; -if (!q->elevator) { -blk_mq_tag_set_rq (hctx, flush_rq->tag, fq->orig_rq); - flush_rq->tag =-1 ; - }else { -blk_mq_put_driver_tag (flush_rq); - flush_rq->internal_tag =-1 ; - } + flush_rq->internal_tag =-1 ; +blk_mq_put_driver_tag (flush_rq); running = &fq->flush_queue[fq->flush_running_idx];BUG_ON (fq->flush_pending_idx == fq->flush_running_idx);
@@ -317,14 +312,7 @@ static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq, flush_rq->mq_ctx = first_rq->mq_ctx; flush_rq->mq_hctx = first_rq->mq_hctx; -if (!q->elevator) { - fq->orig_rq = first_rq; - flush_rq->tag = first_rq->tag; -blk_mq_tag_set_rq (flush_rq->mq_hctx, first_rq->tag, flush_rq); - }else { - flush_rq->internal_tag = first_rq->internal_tag; - } - + flush_rq->internal_tag = first_rq->internal_tag; flush_rq->cmd_flags =REQ_OP_FLUSH |REQ_PREFLUSH ; flush_rq->cmd_flags |= (flags &REQ_DRV ) | (flags &REQ_FAILFAST_MASK ); flush_rq->rq_flags |=RQF_FLUSH_SEQ ;
block/blk-mq.c +38 -37
110 lines not on this page · raw message
block/blk-mq.h +10 -11
@@ -196,26 +196,25 @@ static inline bool blk_mq_get_dispatch_budget(struct blk_mq_hw_ctx *hctx)return true; } -static inline void __blk_mq_put_driver_tag (struct blk_mq_hw_ctx *hctx, -struct request *rq) +static inline void blk_mq_put_driver_tag (struct request *rq) { -blk_mq_put_tag (hctx->tags, rq->mq_ctx, rq->tag); +struct blk_mq_hw_ctx *hctx = rq->mq_hctx; +int tag = rq->tag; + +if (tag <0 ) +return ; + rq->tag =-1 ; +if (hctx->sched_tags) +blk_mq_put_tag (hctx->tags, rq->mq_ctx, tag); +if (rq->rq_flags &RQF_MQ_INFLIGHT ) { rq->rq_flags &= ~RQF_MQ_INFLIGHT ;atomic_dec (&hctx->nr_active); } } -static inline void blk_mq_put_driver_tag (struct request *rq) -{ -if (rq->tag ==-1 || rq->internal_tag ==-1 ) -return ; - -__blk_mq_put_driver_tag (rq->mq_hctx, rq); -} -static inline void blk_mq_clear_mq_map (struct blk_mq_queue_map *qmap) {int cpu;
block/blk.h +0 -5
@@ -26,11 +26,6 @@ struct blk_flush_queue {struct list_head flush_data_in_flight;struct request *flush_rq; -/* -* flush_rq shares tag with this rq, both can't be active -* at the same time -*/ -struct request *orig_rq;struct lock_class_key key;spinlock_t mq_flush_lock; };
Signature · 1 line
2.25.2
PATCH v9 5/11 blk-mq: support rq filter callback when iterating rqs
In reply to Ming LeiNow request is thought as in-flight only when its state is updated as MQ_RQ_IN_FLIGHT, which is done by driver via blk_mq_start_request().
Actually from blk-mq's view, one rq can be thought as in-flight after its tag is >= 0.
Passing one rq filter callback so that we can iterating requests very flexible.
Implement blk_mq_all_tag_busy_iter() which accepts a 'busy_fn' argument to filter over which commands to iterate, and make the existing blk_mq_tag_busy_iter() a wrapper for the new function.
--- block/blk-mq-tag.c | 39 +++++++++++++++++++++++++++------------ block/blk-mq-tag.h | 4 ++++ 2 files changed, 31 insertions(+), 12 deletions(-)
block/blk-mq-tag.c +27 -12
@@ -255,6 +255,7 @@ static void bt_for_each(struct blk_mq_hw_ctx *hctx, struct sbitmap_queue *bt,struct bt_tags_iter_data {struct blk_mq_tags *tags;busy_tag_iter_fn *fn; +busy_rq_iter_fn *busy_rq_fn;void *data;bool reserved; };
@@ -274,7 +275,7 @@ static bool bt_tags_iter(struct sbitmap *bitmap, unsigned int bitnr, void *data)* test and set the bit before assining ->rqs[]. */ rq = tags->rqs[bitnr]; -if (rq &&blk_mq_request_started (rq)) +if (rq && iter_data->busy_rq_fn (rq, iter_data->data, reserved))return iter_data->fn (rq, iter_data->data, reserved);return true;
@@ -294,11 +295,13 @@ static bool bt_tags_iter(struct sbitmap *bitmap, unsigned int bitnr, void *data)* bitmap_tags member of struct blk_mq_tags. */ static void bt_tags_for_each (struct blk_mq_tags *tags,struct sbitmap_queue *bt, -busy_tag_iter_fn *fn,void *data,bool reserved) +busy_tag_iter_fn *fn,busy_rq_iter_fn *busy_rq_fn, +void *data,bool reserved) {struct bt_tags_iter_data iter_data = { .tags = tags, .fn = fn, + .busy_rq_fn = busy_rq_fn, .data = data, .reserved = reserved, };
@@ -310,19 +313,30 @@ static void bt_tags_for_each(struct blk_mq_tags *tags, struct sbitmap_queue *bt,/** * blk_mq_all_tag_busy_iter - iterate over all started requests in a tag map * @tags: Tag map to iterate over. -* @fn: Pointer to the function that will be called for each started -* request. @fn will be called as follows: @fn(rq, @priv, -* reserved) where rq is a pointer to a request. 'reserved' -* indicates whether or not @rq is a reserved request. Return -* true to continue iterating tags, false to stop. +* @fn: Pointer to the function that will be called for each request +* when .busy_rq_fn(rq) returns true. @fn will be called as +* follows: @fn(rq, @priv, reserved) where rq is a pointer to a +* request. 'reserved' indicates whether or not @rq is a reserved +* request. Return true to continue iterating tags, false to stop. +* @busy_rq_fn: Pointer to the function that will be called for each request, +* @busy_rq_fn's type is same with @fn. Only when @busy_rq_fn(rq, +* @priv, reserved) returns true, @fn will be called on this rq. * @priv: Will be passed as second argument to @fn. */ -static void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags, -busy_tag_iter_fn *fn,void *priv) +void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags, +busy_tag_iter_fn *fn,busy_rq_iter_fn *busy_rq_fn, +void *priv) {if (tags->nr_reserved_tags) -bt_tags_for_each (tags, &tags->breserved_tags, fn, priv, true); -bt_tags_for_each (tags, &tags->bitmap_tags, fn, priv, false); +bt_tags_for_each (tags, &tags->breserved_tags, fn, busy_rq_fn, + priv, true); +bt_tags_for_each (tags, &tags->bitmap_tags, fn, busy_rq_fn, priv, false); +} + +static bool blk_mq_default_busy_rq (struct request *rq,void *data, +bool reserved) +{ +return blk_mq_request_started (rq); }/**
@@ -342,7 +356,8 @@ void blk_mq_tagset_busy_iter(struct blk_mq_tag_set *tagset,for (i =0 ; i < tagset->nr_hw_queues; i++) {if (tagset->tags && tagset->tags[i]) -blk_mq_all_tag_busy_iter (tagset->tags[i], fn, priv); +blk_mq_all_tag_busy_iter (tagset->tags[i], fn, + blk_mq_default_busy_rq, priv); } }EXPORT_SYMBOL (blk_mq_tagset_busy_iter);
block/blk-mq-tag.h +4 -0
@@ -21,6 +21,7 @@ struct blk_mq_tags {struct list_head page_list; }; +typedef bool (busy_rq_iter_fn )(struct request *,void *,bool );extern struct blk_mq_tags *blk_mq_init_tags (unsigned int nr_tags,unsigned int reserved_tags,int node,int alloc_policy);extern void blk_mq_free_tags (struct blk_mq_tags *tags);
@@ -34,6 +35,9 @@ extern int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx,extern void blk_mq_tag_wakeup_all (struct blk_mq_tags *tags,bool );void blk_mq_queue_tag_busy_iter (struct request_queue *q,busy_iter_fn *fn,void *priv); +void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags, +busy_tag_iter_fn *fn,busy_rq_iter_fn *busy_rq_fn, +void *priv);static inline struct sbq_wait_state *bt_wait_ptr (struct sbitmap_queue *bt,struct blk_mq_hw_ctx *hctx)
Signature · 1 line
2.25.2
Re: [PATCH V9 05/11] blk-mq: support rq filter callback when iterating rqs
Quoted 20 lines · Ming Lei
Now request is thought as in-flight only when its state is updated as
MQ_RQ_IN_FLIGHT, which is done by driver viablk_mq_start_request().Actually from blk-mq's view, one rq can be thought as in-flight after its tag is >= 0.
Passing one rq filter callback so that we can iterating requests very flexible.
Implement
blk_mq_all_tag_busy_iter()which accepts a 'busy_fn' argument to filter over which commands to iterate, and make the existingblk_mq_tag_busy_iter()a wrapper for the new function.Cc John Garry <[email protected]>Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Thomas Gleixner <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Ming Lei <[email protected]>
Quoted 5 lines
--- block/blk-mq-tag.c | 39 +++++++++++++++++++++++++++------------ block/blk-mq-tag.h | 4 ++++ 2 files changed, 31 insertions(+), 12 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
PATCH v9 6/11 blk-mq: prepare for draining IO when hctx's all CPUs are offline
In reply to Ming LeiMost of blk-mq drivers depend on managed IRQ's auto-affinity to setup up queue mapping. Thomas mentioned the following point[1]:
"
That was the constraint of managed interrupts from the very beginning:
The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"
However, current blk-mq implementation doesn't quiesce hw queue before the last CPU in the hctx is shutdown. Even worse, CPUHP_BLK_MQ_DEAD is one cpuhp state handled after the CPU is down, so there isn't any chance to quiesce hctx for blk-mq wrt. CPU hotplug.
Add new cpuhp state of CPUHP_AP_BLK_MQ_ONLINE for blk-mq to stop queues and wait for completion of in-flight requests.
We will stop hw queue and wait for completion of in-flight requests when one hctx is becoming dead in the following patch. This way may cause dead-lock for some stacking blk-mq drivers, such as dm-rq and loop.
Add blk-mq flag of BLK_MQ_F_NO_MANAGED_IRQ and mark it for dm-rq and loop, so we needn't to wait for completion of in-flight requests from dm-rq & loop, then the potential dead-lock can be avoided.
[1] https://lore.kernel.org/linux-block/[email protected]/
--- block/blk-mq-debugfs.c | 1 + block/blk-mq.c | 19 +++++++++++++++++++ drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 3 +++ include/linux/cpuhotplug.h | 1 + 6 files changed, 26 insertions(+), 2 deletions(-)
block/blk-mq-debugfs.c +1 -0
@@ -239,6 +239,7 @@ static const char *const hctx_flag_name[] = {HCTX_FLAG_NAME (TAG_SHARED ),HCTX_FLAG_NAME (BLOCKING ),HCTX_FLAG_NAME (NO_SCHED ), + HCTX_FLAG_NAME(NO_MANAGED_IRQ), };#undef HCTX_FLAG_NAME
block/blk-mq.c +19 -0
@@ -2273,6 +2273,16 @@ int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags,return -ENOMEM ; } +static int blk_mq_hctx_notify_online (unsigned int cpu,struct hlist_node *node) +{ +return 0 ; +} + +static int blk_mq_hctx_notify_offline (unsigned int cpu,struct hlist_node *node) +{ +return 0 ; +} +/* * 'cpu' is going away. splice any existing rq_list entries from this * software queue to the hw queue dispatch list, and ensure that it
@@ -2309,6 +2319,9 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)static void blk_mq_remove_cpuhp (struct blk_mq_hw_ctx *hctx) { +if (!(hctx->flags &BLK_MQ_F_NO_MANAGED_IRQ )) +cpuhp_state_remove_instance_nocalls (CPUHP_AP_BLK_MQ_ONLINE , + &hctx->cpuhp_online);cpuhp_state_remove_instance_nocalls (CPUHP_BLK_MQ_DEAD , &hctx->cpuhp_dead); }
@@ -2368,6 +2381,9 @@ static int blk_mq_init_hctx(struct request_queue *q, { hctx->queue_num = hctx_idx; +if (!(hctx->flags &BLK_MQ_F_NO_MANAGED_IRQ )) +cpuhp_state_add_instance_nocalls (CPUHP_AP_BLK_MQ_ONLINE , + &hctx->cpuhp_online);cpuhp_state_add_instance_nocalls (CPUHP_BLK_MQ_DEAD , &hctx->cpuhp_dead); hctx->tags = set->tags[hctx_idx];
@@ -3621,6 +3637,9 @@ static int __init blk_mq_init(void) {cpuhp_setup_state_multi (CPUHP_BLK_MQ_DEAD ,"block/mq:dead" ,NULL , blk_mq_hctx_notify_dead); +cpuhp_setup_state_multi (CPUHP_AP_BLK_MQ_ONLINE ,"block/mq:online" , + blk_mq_hctx_notify_online, + blk_mq_hctx_notify_offline);return 0 ; }subsys_initcall (blk_mq_init);
drivers/block/loop.c +1 -1
@@ -2037,7 +2037,7 @@ static int loop_add(struct loop_device **l, int i) lo->tag_set.queue_depth =128 ; lo->tag_set.numa_node =NUMA_NO_NODE ; lo->tag_set.cmd_size =sizeof (struct loop_cmd ); - lo->tag_set.flags =BLK_MQ_F_SHOULD_MERGE ; + lo->tag_set.flags =BLK_MQ_F_SHOULD_MERGE |BLK_MQ_F_NO_MANAGED_IRQ ; lo->tag_set.driver_data = lo; err =blk_mq_alloc_tag_set (&lo->tag_set);
drivers/md/dm-rq.c +1 -1
@@ -547,7 +547,7 @@ int dm_mq_init_request_queue(struct mapped_device *md, struct dm_table *t) md->tag_set->ops = &dm_mq_ops; md->tag_set->queue_depth =dm_get_blk_mq_queue_depth (); md->tag_set->numa_node = md->numa_node_id; - md->tag_set->flags =BLK_MQ_F_SHOULD_MERGE ; + md->tag_set->flags =BLK_MQ_F_SHOULD_MERGE |BLK_MQ_F_NO_MANAGED_IRQ ; md->tag_set->nr_hw_queues =dm_get_blk_mq_nr_hw_queues (); md->tag_set->driver_data = md;
include/linux/blk-mq.h +3 -0
@@ -140,6 +140,8 @@ struct blk_mq_hw_ctx {*/ atomic_t nr_active; +/** @cpuhp_online: List to store request if CPU is going to die */ +struct hlist_node cpuhp_online;/** @cpuhp_dead: List to store request if some CPU die. */ struct hlist_node cpuhp_dead;/** @kobj: Kernel object for sysfs. */
@@ -391,6 +393,7 @@ struct blk_mq_ops {enum {BLK_MQ_F_SHOULD_MERGE =1 <<0 ,BLK_MQ_F_TAG_SHARED =1 <<1 , +BLK_MQ_F_NO_MANAGED_IRQ =1 <<2 ,BLK_MQ_F_BLOCKING =1 <<5 ,BLK_MQ_F_NO_SCHED =1 <<6 ,BLK_MQ_F_ALLOC_POLICY_START_BIT =8 ,
include/linux/cpuhotplug.h +1 -0
@@ -152,6 +152,7 @@ enum cpuhp_state {CPUHP_AP_SMPBOOT_THREADS ,CPUHP_AP_X86_VDSO_VMA_ONLINE ,CPUHP_AP_IRQ_AFFINITY_ONLINE , +CPUHP_AP_BLK_MQ_ONLINE ,CPUHP_AP_ARM_MVEBU_SYNC_CLOCKS ,CPUHP_AP_X86_INTEL_EPB_ONLINE , CPUHP_AP_PERF_ONLINE,
Signature · 1 line
2.25.2
PATCH v9 7/11 blk-mq: stop to handle IO and drain IO before hctx becomes inactive
In reply to Ming LeiBefore one CPU becomes offline, check if it is the last online CPU of hctx. If yes, mark this hctx as inactive, meantime wait for completion of all in-flight IOs originated from this hctx. Meantime check if this hctx has become inactive in blk_mq_get_driver_tag(), if yes, release the allocated tag.
This way guarantees that there isn't any inflight IO before shutdowning the managed IRQ line when all CPUs of this IRQ line is offline.
--- block/blk-mq-debugfs.c | 1 + block/blk-mq.c | 124 +++++++++++++++++++++++++++++++++++++---- include/linux/blk-mq.h | 3 + 3 files changed, 117 insertions(+), 11 deletions(-)
block/blk-mq-debugfs.c +1 -0
@@ -213,6 +213,7 @@ static const char *const hctx_state_name[] = {HCTX_STATE_NAME (STOPPED ),HCTX_STATE_NAME (TAG_ACTIVE ),HCTX_STATE_NAME (SCHED_RESTART ), + HCTX_STATE_NAME(INACTIVE), };#undef HCTX_STATE_NAME
block/blk-mq.c +113 -11
184 lines not on this page · raw message
include/linux/blk-mq.h +3 -0
@@ -403,6 +403,9 @@ enum {BLK_MQ_S_TAG_ACTIVE =1 ,BLK_MQ_S_SCHED_RESTART =2 , +/* hw queue is inactive after all its CPUs become offline */ +BLK_MQ_S_INACTIVE =3 , +BLK_MQ_MAX_DEPTH =10240 , BLK_MQ_CPU_WORK_BATCH = 8,
Signature · 1 line
2.25.2
PATCH v9 8/11 block: add blk_end_flush_machinery
In reply to Ming LeiFlush requests aren't same with normal FS request:
1) one dedicated per-hctx flush rq is pre-allocated for sending flush request
2) flush request si issued to hardware via one machinary so that flush merge can be applied
We can't simply re-submit flush rqs via blk_steal_bios(), so add blk_end_flush_machinery to collect flush requests which needs to be resubmitted:
- if one flush command without DATA is enough, send one flush, complete this kind of requests
- otherwise, add the request into a list and let caller re-submit it.
--- block/blk-flush.c | 123 +++++++++++++++++++++++++++++++++++++++++++--- block/blk.h | 4 ++ 2 files changed, 120 insertions(+), 7 deletions(-)
block/blk-flush.c +116 -7
@@ -170,10 +170,11 @@ static void blk_flush_complete_seq(struct request *rq,unsigned int cmd_flags;BUG_ON (rq->flush.seq & seq); - rq->flush.seq |= seq; +if (!error) + rq->flush.seq |= seq; cmd_flags = rq->cmd_flags; -if (likely (!error)) +if (likely (!error && !fq->flush_queue_terminating)) seq =blk_flush_cur_seq (rq);else seq =REQ_FSEQ_DONE ;
@@ -200,9 +201,15 @@ static void blk_flush_complete_seq(struct request *rq,* normal completion and end it. */ BUG_ON (!list_empty (&rq->queuelist)); -list_del_init (&rq->flush.list); -blk_flush_restore_request (rq); -blk_mq_end_request (rq, error); + +/* Terminating code will end the request from flush queue */ +if (likely (!fq->flush_queue_terminating)) { +list_del_init (&rq->flush.list); +blk_flush_restore_request (rq); +blk_mq_end_request (rq, error); + }else { +list_move_tail (&rq->flush.list, pending); + }break ;default :
@@ -279,7 +286,8 @@ static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq,struct request *flush_rq = fq->flush_rq;/* C1 described at the top of this file */ -if (fq->flush_pending_idx != fq->flush_running_idx ||list_empty (pending)) +if (fq->flush_pending_idx != fq->flush_running_idx || +list_empty (pending) || fq->flush_queue_terminating)return ;/* C2 and C3
@@ -331,7 +339,7 @@ static void mq_flush_data_end_io(struct request *rq, blk_status_t error)struct blk_flush_queue *fq =blk_get_flush_queue (q, ctx);if (q->elevator) { -WARN_ON (rq->tag <0 ); +WARN_ON (rq->tag <0 && !fq->flush_queue_terminating);blk_mq_put_driver_tag (rq); }
@@ -503,3 +511,104 @@ void blk_free_flush_queue(struct blk_flush_queue *fq)kfree (fq->flush_rq);kfree (fq); } + +static void __blk_end_queued_flush (struct blk_flush_queue *fq, +unsigned int queue_idx,struct list_head *resubmit_list, +struct list_head *flush_list) +{ +struct list_head *queue = &fq->flush_queue[queue_idx]; +struct request *rq, *nxt; + +list_for_each_entry_safe (rq, nxt, queue, flush.list) { +unsigned int seq =blk_flush_cur_seq (rq); + +list_del_init (&rq->flush.list); +blk_flush_restore_request (rq); +if (!blk_rq_sectors (rq) || seq ==REQ_FSEQ_POSTFLUSH ) +list_add_tail (&rq->queuelist, flush_list); +else +list_add_tail (&rq->queuelist, resubmit_list); + } +} + +static void blk_end_queued_flush (struct blk_flush_queue *fq, +struct list_head *resubmit_list,struct list_head *flush_list) +{ +unsigned long flags; + +spin_lock_irqsave (&fq->mq_flush_lock, flags); +__blk_end_queued_flush (fq,0 , resubmit_list, flush_list); +__blk_end_queued_flush (fq,1 , resubmit_list, flush_list); +spin_unlock_irqrestore (&fq->mq_flush_lock, flags); +} + +/* complete requests which just requires one flush command */ +static void blk_complete_flush_requests (struct blk_flush_queue *fq, +struct list_head *flush_list) +{ +struct block_device *bdev; +struct request *rq; +int error = -ENXIO ; + +if (list_empty (flush_list)) +return ; + + rq = list_first_entry(flush_list, struct request, queuelist); + +/* Send flush via one active hctx so we can move on */ + bdev =bdget_disk (rq->rq_disk,0 ); +if (bdev) { + error =blkdev_issue_flush (bdev,GFP_KERNEL ,NULL ); +bdput (bdev); + } + +while (!list_empty (flush_list)) { + rq = list_first_entry(flush_list, struct request, queuelist); +list_del_init (&rq->queuelist); +blk_mq_end_request (rq, error); + } +} + +/* +* Called when this hctx is inactive and all CPUs of this hctx is dead, +* otherwise don't reuse this function. +* +* Terminate this hw queue's flush machinery, and try to complete flush +* IO requests if possible, such as any flush IO without data, or flush +* data IO in POSTFLUSH stage. Otherwise, add the flush IOs into @list +* and let caller to re-submit them. +*/ +void blk_end_flush_machinery (struct blk_mq_hw_ctx *hctx, +struct list_head *in,struct list_head *out) +{ +LIST_HEAD (resubmit_list); +LIST_HEAD (flush_list); +struct blk_flush_queue *fq = hctx->fq; +struct request *rq, *nxt; +unsigned long flags; + +spin_lock_irqsave (&fq->mq_flush_lock, flags); + fq->flush_queue_terminating =1 ; +spin_unlock_irqrestore (&fq->mq_flush_lock, flags); + +/* End inflight flush requests */ +list_for_each_entry_safe (rq, nxt, in, queuelist) { +WARN_ON (!(rq->rq_flags &RQF_FLUSH_SEQ )); +list_del_init (&rq->queuelist); + rq->end_io (rq,BLK_STS_AGAIN ); + } + +/* End queued requests */ +blk_end_queued_flush (fq, &resubmit_list, &flush_list); + +/* Send flush and complete requests which just need one flush req */ +blk_complete_flush_requests (fq, &flush_list); + +spin_lock_irqsave (&fq->mq_flush_lock, flags); +/* reset flush queue so that it is ready to work next time */ + fq->flush_pending_idx = fq->flush_running_idx =0 ; + fq->flush_queue_terminating =0 ; +spin_unlock_irqrestore (&fq->mq_flush_lock, flags); + +list_splice_init (&resubmit_list, out); +}
block/blk.h +4 -0
@@ -20,6 +20,7 @@ struct blk_flush_queue { unsigned int flush_queue_delayed:1; unsigned int flush_pending_idx:1; unsigned int flush_running_idx:1; + unsigned int flush_queue_terminating:1;blk_status_t rq_status;unsigned long flush_pending_since;struct list_head flush_queue[2 ];
@@ -454,4 +455,7 @@ int __bio_add_pc_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, bool *same_page); +void blk_end_flush_machinery (struct blk_mq_hw_ctx *hctx, +struct list_head *in,struct list_head *out); +#endif /* BLK_INTERNAL_H */
Signature · 1 line
2.25.2
PATCH v9 9/11 blk-mq: add blk_mq_hctx_handle_dead_cpu for handling cpu dead
In reply to Ming LeiAdd helper of blk_mq_hctx_handle_dead_cpu for handling cpu dead, and prepare for handling inactive hctx.
No functional change.
--- block/blk-mq.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)
block/blk-mq.c +17 -12
@@ -2382,22 +2382,13 @@ static int blk_mq_hctx_notify_online(unsigned int cpu, struct hlist_node *node)return 0 ; } -/* -* 'cpu' is going away. splice any existing rq_list entries from this -* software queue to the hw queue dispatch list, and ensure that it -* gets run. -*/ -static int blk_mq_hctx_notify_dead (unsigned int cpu,struct hlist_node *node) +static void blk_mq_hctx_handle_dead_cpu (struct blk_mq_hw_ctx *hctx, +unsigned int cpu) { -struct blk_mq_hw_ctx *hctx =hlist_entry_safe (node, - struct blk_mq_hw_ctx, cpuhp_dead);struct blk_mq_ctx *ctx;LIST_HEAD (tmp);enum hctx_type type; -if (!cpumask_test_cpu (cpu, hctx->cpumask)) -return 0 ; - ctx =__blk_mq_get_ctx (hctx->queue, cpu); type = hctx->type;
@@ -2409,13 +2400,27 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)spin_unlock (&ctx->lock);if (list_empty (&tmp)) -return 0 ; +return ;spin_lock (&hctx->lock);list_splice_tail_init (&tmp, &hctx->dispatch);spin_unlock (&hctx->lock);blk_mq_run_hw_queue (hctx, true); +} + +/* +* 'cpu' is going away. splice any existing rq_list entries from this +* software queue to the hw queue dispatch list, and ensure that it +* gets run. +*/ +static int blk_mq_hctx_notify_dead (unsigned int cpu,struct hlist_node *node) +{ +struct blk_mq_hw_ctx *hctx =hlist_entry_safe (node, + struct blk_mq_hw_ctx, cpuhp_dead); + +if (cpumask_test_cpu (cpu, hctx->cpumask)) +blk_mq_hctx_handle_dead_cpu (hctx, cpu);return 0 ; }
Signature · 1 line
2.25.2
PATCH v9 10/11 blk-mq: re-submit IO in case that hctx is inactive
In reply to Ming LeiWhen all CPUs in one hctx are offline and this hctx becomes inactive, we shouldn't run this hw queue for completing request any more.
So allocate request from one live hctx, and clone & resubmit the request, either it is from sw queue or scheduler queue.
--- block/blk-mq.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 100 insertions(+), 5 deletions(-)
block/blk-mq.c +100 -5
@@ -2382,6 +2382,98 @@ static int blk_mq_hctx_notify_online(unsigned int cpu, struct hlist_node *node)return 0 ; } +static void blk_mq_resubmit_end_rq (struct request *rq,blk_status_t error) +{ +struct request *orig_rq = rq->end_io_data; + +blk_mq_cleanup_rq (orig_rq); +blk_mq_end_request (orig_rq, error); + +blk_put_request (rq); +} + +static void blk_mq_resubmit_rq (struct request *rq) +{ +struct request *nrq; +unsigned int flags =0 ; +struct blk_mq_hw_ctx *hctx = rq->mq_hctx; +struct blk_mq_tags *tags = rq->q->elevator ? hctx->sched_tags : + hctx->tags; +bool reserved =blk_mq_tag_is_reserved (tags, rq->internal_tag); + +if (rq->rq_flags &RQF_PREEMPT ) + flags |=BLK_MQ_REQ_PREEMPT ; +if (reserved) + flags |=BLK_MQ_REQ_RESERVED ; + +/* avoid allocation failure by clearing NOWAIT */ + nrq =blk_get_request (rq->q, rq->cmd_flags & ~REQ_NOWAIT , flags); +if (!nrq) +return ; + +blk_rq_copy_request (nrq, rq); + + nrq->timeout = rq->timeout; + nrq->rq_disk = rq->rq_disk; + nrq->part = rq->part; + +memcpy (blk_mq_rq_to_pdu (nrq),blk_mq_rq_to_pdu (rq), + rq->q->tag_set->cmd_size); + + nrq->end_io = blk_mq_resubmit_end_rq; + nrq->end_io_data = rq; + nrq->bio = rq->bio; + nrq->biotail = rq->biotail; + +if (blk_insert_cloned_request (nrq->q, nrq) !=BLK_STS_OK ) +blk_mq_request_bypass_insert (nrq, false, true); +} + +static void blk_mq_hctx_deactivate (struct blk_mq_hw_ctx *hctx) +{ +LIST_HEAD (sched); +LIST_HEAD (re_submit); +LIST_HEAD (flush_in); +LIST_HEAD (flush_out); +struct request *rq, *nxt; +struct elevator_queue *e = hctx->queue->elevator; + +if (!e) { +blk_mq_flush_busy_ctxs (hctx, &re_submit); + }else { +while ((rq = e->type->ops.dispatch_request (hctx))) { +if (rq->mq_hctx != hctx) +list_add (&rq->queuelist, &sched); +else +list_add (&rq->queuelist, &re_submit); + } + } +while (!list_empty (&sched)) { + rq = list_first_entry(&sched, struct request, queuelist); +list_del_init (&rq->queuelist); +blk_mq_sched_insert_request (rq, true, true, true); + } + +/* requests in dispatch list have to be re-submitted too */ +spin_lock (&hctx->lock); +list_splice_tail_init (&hctx->dispatch, &re_submit); +spin_unlock (&hctx->lock); + +/* blk_end_flush_machinery will cover flush request */ +list_for_each_entry_safe (rq, nxt, &re_submit, queuelist) { +if (rq->rq_flags &RQF_FLUSH_SEQ ) +list_move (&rq->queuelist, &flush_in); + } +blk_end_flush_machinery (hctx, &flush_in, &flush_out); +list_splice_tail (&flush_out, &re_submit); + +while (!list_empty (&re_submit)) { + rq = list_first_entry(&re_submit, struct request, queuelist); +list_del_init (&rq->queuelist); +blk_mq_resubmit_rq (rq); + } +} +static void blk_mq_hctx_handle_dead_cpu (struct blk_mq_hw_ctx *hctx,unsigned int cpu) {
@@ -2410,17 +2502,20 @@ static void blk_mq_hctx_handle_dead_cpu(struct blk_mq_hw_ctx *hctx, }/* -* 'cpu' is going away. splice any existing rq_list entries from this -* software queue to the hw queue dispatch list, and ensure that it -* gets run. +* @cpu has gone away. If this hctx is inactive, we can't dispatch request +* to the hctx any more, so clone and re-submit requests from this hctx */ static int blk_mq_hctx_notify_dead (unsigned int cpu,struct hlist_node *node) {struct blk_mq_hw_ctx *hctx =hlist_entry_safe (node, struct blk_mq_hw_ctx, cpuhp_dead); -if (cpumask_test_cpu (cpu, hctx->cpumask)) -blk_mq_hctx_handle_dead_cpu (hctx, cpu); +if (cpumask_test_cpu (cpu, hctx->cpumask)) { +if (test_bit (BLK_MQ_S_INACTIVE , &hctx->state)) +blk_mq_hctx_deactivate (hctx); +else +blk_mq_hctx_handle_dead_cpu (hctx, cpu); + }return 0 ; }
Signature · 1 line
2.25.2
PATCH v9 11/11 block: deactivate hctx when the hctx is actually inactive
In reply to Ming LeiRun queue on dead CPU still may be triggered in some corner case, such as one request is requeued after CPU hotplug is handled.
So handle this corner case during run queue.
--- block/blk-mq.c | 30 ++++++++++-------------------- 1 file changed, 10 insertions(+), 20 deletions(-)
block/blk-mq.c +10 -20
@@ -43,6 +43,8 @@static void blk_mq_poll_stats_start (struct request_queue *q);static void blk_mq_poll_stats_fn (struct blk_stat_callback *cb); +static void blk_mq_hctx_deactivate (struct blk_mq_hw_ctx *hctx); +static int blk_mq_poll_stats_bkt (const struct request *rq) {int ddir, sectors, bucket;
@@ -1373,28 +1375,16 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)int srcu_idx;/* -* We should be running this queue from one of the CPUs that -* are mapped to it. -* -* There are at least two related races now between setting -* hctx->next_cpu from blk_mq_hctx_next_cpu() and running -* __blk_mq_run_hw_queue(): -* -* - hctx->next_cpu is found offline in blk_mq_hctx_next_cpu(), -* but later it becomes online, then this warning is harmless -* at all -* -* - hctx->next_cpu is found online in blk_mq_hctx_next_cpu(), -* but later it becomes offline, then the warning can't be -* triggered, and we depend on blk-mq timeout handler to -* handle dispatched requests to this hctx +* BLK_MQ_S_INACTIVE may not deal with some requeue corner case: +* one request is requeued after cpu unplug is handled, so check +* if the hctx is actually inactive. If yes, deactive it and +* re-submit all requests in the queue. */ if (!cpumask_test_cpu (raw_smp_processor_id (), hctx->cpumask) && -cpu_online (hctx->next_cpu)) { -printk (KERN_WARNING "run queue from wrong CPU %d, hctx %s\n" , -raw_smp_processor_id (), -cpumask_empty (hctx->cpumask) ?"inactive" :"active" ); -dump_stack (); +cpumask_next_and (-1 , hctx->cpumask, cpu_online_mask) >= + nr_cpu_ids) { +blk_mq_hctx_deactivate (hctx); +return ; }/*
Signature · 1 line
2.25.2
Quoted 27 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very beginning:The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"But no drivers or blk-mq do that before one hctx becomes inactive(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(), then these IO will be mapped to other live hctx for dispatch
Quoted 7 lines
Thanks John Garry for running lots of tests on arm64 with this patchset and co-working on investigating all kinds of issues.
Thanks Christoph's review on V7.
Please comment & review, thanks!
Hi Ming,
Bad news, I see this on the first hotplug loop:
[66.628964] CPU2: shutdown [66.631663] psci: CPU2 killed (polled 0 ms) [66.681062] CPU3: shutdown [66.683761] psci: CPU3 killed (polled 0 ms) [66.717097] CPU4: shutdown [66.719796] psci: CPU4 killed (polled 0 ms) [66.753136] CPU5: shutdown [66.755834] psci: CPU5 killed (polled 0 ms) [66.801234] CPU6: shutdown=2746MiB/s,w=0KiB/s][r=703k,w=0 IOPS][eta 00m:55s] [66.803932] psci: CPU6 killed (polled 0 ms) [66.837170] irq_shutdown irq150 [66.840410] CPU7: shutdown [66.843112] psci: CPU7 killed (polled 0 ms) [66.885394] CPU8: shutdown [66.888092] psci: CPU8 killed (polled 0 ms) [66.925431] CPU9: shutdown [66.928128] psci: CPU9 killed (polled 0 ms) [66.965526] CPU10: shutdown [66.968311] psci: CPU10 killed (polled 0 ms) [67.025569] irq_shutdown irq151 [67.028808] CPU11: shutdown [67.031599] psci: CPU11 killed (polled 0 ms) [67.306393] CPU12: shutdown [67.309179] psci: CPU12 killed (polled 0 ms) [67.347266] CPU13: shutdown [67.350058] psci: CPU13 killed (polled 0 ms) [67.410910] CPU14: shutdown [67.413695] psci: CPU14 killed (polled 0 ms) [67.454453] irq_shutdown irq152 [67.457699] CPU15: shutdown [67.457799] ------------[ cut here ]------------ [67.460506] psci: CPU15 killed (polled 0 ms) [67.465091] refcount_t: underflow; use-after-free. [67.465112] WARNING: CPU: 0 PID: 557 at lib/refcount.c:28 refcount_warn_saturate+0x6c/0x13c [67.482468] Modules linked in: [67.485511] CPU: 0 PID: 557 Comm: irq/149-hisi_sa Not tainted 5.7.0-rc2-13190-gb615db31b5f4 #331 [67.494281] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 IT21 Nemo 2.0 RC0 04/18/2018 [67.503398] pstate: 20000005 (nzCv daif -PAN -UAO) [67.508176] pc : refcount_warn_saturate+0x6c/0x13c [67.512954] lr : refcount_warn_saturate+0x6c/0x13c [67.517730] sp : ffff80001fd3b920 [67.521031] x29: ffff80001fd3b920 x28: ffff001f9c549500 [67.526329] x27: ffff001f9c54956c x26: 000000000000d8d0 [67.531627] x25: ffff001fb08c7378 x24: 0000000000000000 [67.536925] x23: ffff001f9a6eef00 x22: 0000000000000000 [67.542223] x21: 0000000000001000 x20: ffff001f9ebd0918 [67.547521] x19: ffff001f9a6eef00 x18: 0000000000000000 [67.552820] x17: 000000007fffffff x16: 0000000000000311 [67.558118] x15: 000000000001ffff x14: 0000000000000001 [67.563416] x13: 000000000000001f x12: 0000000000000000 [67.568714] x11: ffff800011aca468 x10: 0000000000fddfa0 [67.574012] x9 : 0000000000000000 x8 : ffff041faa022098 [67.579310] x7 : 0000000000000000 x6 : ffff001ffbe871d0 [67.584608] x5 : 0000000000000001 x4 : 0000000000000000 [67.589906] x3 : 0000000000000000 x2 : 0000000000000007 [67.595204] x1 : 0000000100000001 x0 : 0000000000000026 [67.600503] Call trace: [67.602937] refcount_warn_saturate+0x6c/0x13c [67.607369] aio_complete_rw+0x350/0x384 [67.611279] blkdev_bio_end_io+0xc4/0x12c [67.615276] bio_endio+0x104/0x130 [67.618665] blk_update_request+0x98/0x37c [67.622748] blk_mq_end_request+0x24/0x138 [67.626831] blk_mq_resubmit_end_rq+0x40/0x58 [67.631174] __blk_mq_end_request+0xb0/0x10c [67.635432] scsi_end_request+0xdc/0x20c [67.639341] scsi_io_completion+0x5c/0x468 [67.643424] scsi_finish_command+0xcc/0x11c [67.647594] scsi_softirq_done+0x11c/0x144 [67.651676] blk_mq_complete_request+0x114/0x150 [67.656280] scsi_mq_done+0x40/0x74 [67.659756] sas_scsi_task_done+0x64/0xa4 [67.663753] slot_complete_v2_hw+0x1c4/0x470 [67.668010] cq_thread_v2_hw+0x194/0x250 [67.671920] irq_thread_fn+0x28/0x6c [67.675482] irq_thread+0x158/0x1e8 [67.678957] kthread+0x124/0x150 [67.682173] ret_from_fork+0x10/0x18 [67.685735] ---[ end trace 90afd0085490db4a ]--- [67.690361] Unable to handle kernel paging request at virtual address 0000000b0000000a [67.698279] Mem abort info: [67.701068]ESR = 0x96000004 [67.704121]EC = 0x25: DABT (current EL), IL = 32 bits [67.709425]SET = 0, FnV = 0 [67.712470]EA = 0, S1PTW = 0 [67.715608] Data abort info: [67.718476]ISV = 0, ISS = 0x00000004 [67.722307]CM = 0, WnR = 0 [67.725272] user pgtable: 4k pages, 48-bit VAs, pgdp=0000001fabf2c000 [67.731708] [0000000b0000000a] pgd=0000000000000000 [67.736580] Internal error: Oops: 96000004 [#1] PREEMPT SMP [67.742139] Modules linked in: [67.745181] CPU: 0 PID: 557 Comm: irq/149-hisi_sa Tainted: G W5.7.0-rc2-13190-gb615db31b5f4 #331 [67.755340] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 IT21 Nemo 2.0 RC0 04/18/2018 [67.764458] pstate: 60000005 (nZCv daif -PAN -UAO) [67.769235] pc : bio_check_pages_dirty+0x60/0x130 [67.773926] lr : blkdev_bio_end_io+0xe0/0x12c [67.778269] sp : ffff80001fd3b940 [67.781570] x29: ffff80001fd3b940 x28: ffff001f9c549500 [67.786869] x27: ffff001f9c54956c x26: 000000000000d8d0 [67.792167] x25: ffff001fb08c7378 x24: 0000000000000000 Jobs: 6 (f=6): [[ R ( 66)7].[7191466] x23: ffff001f9a6eef00 x22: 0000000000000001 [67.809706] x21: 0000000000000001 x20: ffff001f9ebd0918 [67.815005] x19: ffff001f9ebd0918 x18: 0000000000000000 [67.820302] x17: 000000007fffffff x16: 0000000000000311 [67.825601] x15: 000000000001ffff x14: 0000000000000001 [67.830899] x13: 000000000000001f x12: 0000000000000000 [67.836198] x11: ffff800011aca468 x10: 0000000000fddfa0 [67.841496] x9 : 0000000000001000 x8 : ffff001f9ebd1c00 [67.846794] x7 : 0000000000000000 x6 : 0000000000000001 [67.852092] x5 : 0000000000000ffe x4 : 0000000000000000 [67.857390] x3 : 0000000b0000000a x2 : ffff001fb4db9800 [67.862688] x1 : 0000000000000000 x0 : 0000000000000001 [67.867986] Call trace: [67.870420] bio_check_pages_dirty+0x60/0x130 [67.874764] blkdev_bio_end_io+0xe0/0x12c [67.878759] bio_endio+0x104/0x130 [67.882148] blk_update_request+0x98/0x37c [67.886231] blk_mq_end_request+0x24/0x138 [67.890314] blk_mq_resubmit_end_rq+0x40/0x58 [67.894656] __blk_mq_end_request+0xb0/0x10c [67.898913] scsi_end_request+0xdc/0x20c [67.902823] scsi_io_completion+0x5c/0x468 [67.906906] scsi_finish_command+0xcc/0x11c [67.911076] scsi_softirq_done+0x11c/0x144 [67.915159] blk_mq_complete_request+0x114/0x150 [67.919762] scsi_mq_done+0x40/0x74 [67.923238] sas_scsi_task_done+0x64/0xa4 [67.927234] slot_complete_v2_hw+0x1c4/0x470 [67.931491] cq_thread_v2_hw+0x194/0x250 [67.935400] irq_thread_fn+0x28/0x6c [67.938962] irq_thread+0x158/0x1e8 [67.942438] kthread+0x124/0x150 [67.945652] ret_from_fork+0x10/0x18 [67.949217] Code: 52800001 f9400443 37000683 aa0203e3 (f9400063) [67.955297] ---[ end trace 90afd0085490db4b ]--- [67.959919] note: irq/149-hisi_sa[557] exited with preempt_count 1 [67.966094] genirq: exiting task "irq/149-hisi_sa" (557) is an active IRQ thread (irq 149) [68.011756] CPU16: shutdown [68.014542] psci: CPU16 killed (polled 0 ms) [68.063856] CPU17: shutdown [68.066642] psci: CPU17 killed (polled 0 ms) [68.107841] CPU18: shutdown [68.110626] psci: CPU18 killed (polled 0 ms) [68.175801] irq_shutdown irq137 [68.179099] CPU19: shutdown [68.179289] Unable to handle kernel NULL pointer dereference at virtual address 000000000000000c [68.181894] psci: CPU19 killed (polled 0 ms) [68.190667] Mem abort info: [68.190669]ESR = 0x96000004 [68.190671]EC = 0x25: DABT (current EL), IL = 32 bits [68.190673]SET = 0, FnV = 0 [68.190676]EA = 0, S1PTW = 0 [68.212237] Data abort info: [68.215104]ISV = 0, ISS = 0x00000004 [68.218931]CM = 0, WnR = 0 [68.221892] user pgtable: 4k pages, 48-bit VAs, pgdp=0000001f9a7fa000 [68.228324] [000000000000000c] pgd=0000000000000000 [68.233200] Internal error: Oops: 96000004 [#2] PREEMPT SMP [68.238758] Modules linked in: [68.241802] CPU: 20 PID: 546 Comm: irq/138-hisi_sa Tainted: GD W5.7.0-rc2-13190-gb615db31b5f4 #331 [68.252047] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 IT21 Nemo 2.0 RC0 04/18/2018 [68.261165] pstate: 20000005 (nzCv daif -PAN -UAO) [68.265949] pc : bio_check_pages_dirty+0x80/0x130 [68.270641] lr : blkdev_bio_end_io+0xe0/0x12c [68.274984] sp : ffff80001fce3940 [68.278285] x29: ffff80001fce3940 x28: ffff001f78d27600 [68.283583] x27: ffff001f78d2766c x26: 0000000000002020 [68.288882] x25: ffff001fb08c34b8 x24: 0000000000000000 [68.294180] x23: ffff001f9d476cc0 x22: 0000000000000001 [68.299478] x21: 0000000000000001 x20: ffff001f8db52318 [68.304776] x19: ffff001f8db52318 x18: 0000000000000000 [68.310074] x17: 00003fffffffffff x16: 00000000000000f7 [68.315372] x15: 000000000001ffff x14: 0000000000000001 [68.320670] x13: 000000000000002e x12: 000000000000002e [68.325968] x11: 000000000000ffff x10: 0000000000013dee [68.331266] x9 : 0000000000001000 x8 : 0000000000000000 [68.336564] x7 : 0000000000000000 x6 : 0000000000000000 [68.341862] x5 : fffffdffbfeded88 x4 : 0000000000000000 [68.347160] x3 : 0000000000000000 x2 : ffff80001043da70 [68.352458] x1 : 0000000000000000 x0 : 0000000000000001 [68.357757] Call trace: [68.360191] bio_check_pages_dirty+0x80/0x130 [68.364534] blkdev_bio_end_io+0xe0/0x12c [68.368530] bio_endio+0x104/0x130 [68.371919] blk_update_request+0x98/0x37c [68.376004] blk_mq_end_request+0x24/0x138 [68.380087] blk_mq_resubmit_end_rq+0x40/0x58 [68.384430] __blk_mq_end_request+0xb0/0x10c [68.388688] scsi_end_request+0xdc/0x20c [68.392596] scsi_io_completion+0x5c/0x468 [68.396681] scsi_finish_command+0xcc/0x11c [68.400850] scsi_softirq_done+0x11c/0x144 [68.404933] blk_mq_complete_request+0x114/0x150 [68.409536] scsi_mq_done+0x40/0x74 [68.413012] sas_scsi_task_done+0x64/0xa4 [68.417010] slot_complete_v2_hw+0x1c4/0x470 [68.421267] cq_thread_v2_hw+0x194/0x250 [68.425177] irq_thread_fn+0x28/0x6c [68.428740] irq_thread+0x158/0x1e8 [68.432216] kthread+0x124/0x150 [68.435431] ret_from_fork+0x10/0x18 [68.438997] Code: f9403668 937c7cc7 8b070103 35fffd61 (b9400c65) [68.445077] ---[ end trace 90afd0085490db4c ]--- [68.449690] note: irq/138-hisi_sa[546] exited with preempt_count 1 [68.455866] genirq: exiting task "irq/138-hisi_sa" (546) is an active IRQ thread (irq 138) [68.504505] CPU20: shutdown [68.507305] psci: CPU20 killed (polled 0 ms) [68.556524] CPU21: shutdown [68.559312] psci: CPU21 killed (polled 0 ms) [68.592523] CPU22: shutdown
note: I'm using this branch, and we really should tag this or have an immutable branch so I know that I'm testing the same thing as in this series. I don't recognize the baseline here. I guess it's Jens' for-next.
Cheers,
John
Quoted 74 lines
V9:
- add Reviewed-by tag
- document more on memory barrier usage between getting driver tag and handling cpu offline(7/11)
- small code cleanup as suggested by Chritoph(7/11)
- rebase against for-5.8/block(1/11, 2/11)
V8:
- add patches to share code withblk_rq_prep_clone
- code re-organization as suggested by Christoph, most of them are in 04/11, 10/11
- add reviewed-by tagV7:
- fix updating .nr_active inget_driver_tag
- addhctx->cpumaskcheck in cpuhp handler
- only drain requests which tag is >= 0
- pass more aggressive cpuhotplug&io testV6:
- simplify getting driver tag, so that we can drain in-flight requests correctly without usingsynchronize_rcu()
- handle re-submission of flush & passthrough request correctlyV5:
- renameBLK_MQ_S_INTERNAL_STOPPEDasBLK_MQ_S_INACTIVE
- re-factor code for re-submit requests in cpu dead hotplug handler
- address requeue corner caseV4:
- resubmit IOs in dispatch list in case that this hctx is deadV3:
- re-organize patch 2 & 3 a bit for addressing Hannes's comment
- fix patch 4 for avoiding potential deadlock, as found by HannesV2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and MinwooMing Lei (11):
block: clonenr_integrity_segmentsandwrite_hintinblk_rq_prep_clone
block: add helper for copying request
blk-mq: markblk_mq_get_driver_tagas static
blk-mq: assignrq->taginblk_mq_get_driver_tag
blk-mq: support rq filter callback when iterating rqs
blk-mq: prepare for draining IO when hctx's all CPUs are offline
blk-mq: stop to handle IO and drain IO before hctx becomes inactive
block: addblk_end_flush_machinery
blk-mq: addblk_mq_hctx_handle_dead_cpufor handling cpu dead
blk-mq: re-submit IO in case that hctx is inactive
block: deactivate hctx when the hctx is actually inactiveblock/blk-core.c | 27 ++- block/blk-flush.c | 141 ++++++++++++--- block/blk-mq-debugfs.c | 2 + block/blk-mq-tag.c | 39 ++-- block/blk-mq-tag.h | 4 + block/blk-mq.c | 352 +++++++++++++++++++++++++++++-------- block/blk-mq.h | 22 ++- block/blk.h | 11 +- drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 6 + include/linux/cpuhotplug.h | 1 + 12 files changed, 477 insertions(+), 132 deletions(-)Cc John Garry <[email protected]>Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Thomas Gleixner <[email protected]>
Quoted 100 lines · John Garry
Quoted 26 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very beginning:
The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"But no drivers or blk-mq do that before one hctx becomes inactive(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(), then these IO will be mapped to other live hctx for dispatchQuoted 7 lines
Thanks John Garry for running lots of tests on arm64 with this patchset and co-working on investigating all kinds of issues.
Thanks Christoph's review on V7.
Please comment & review, thanks!
Hi Ming,
Bad news, I see this on the first hotplug loop:
[66.628964] CPU2: shutdown [66.631663] psci: CPU2 killed (polled 0 ms) [66.681062] CPU3: shutdown [66.683761] psci: CPU3 killed (polled 0 ms) [66.717097] CPU4: shutdown [66.719796] psci: CPU4 killed (polled 0 ms) [66.753136] CPU5: shutdown [66.755834] psci: CPU5 killed (polled 0 ms) [66.801234] CPU6: shutdown=2746MiB/s,w=0KiB/s][r=703k,w=0 IOPS][eta 00m:55s] [66.803932] psci: CPU6 killed (polled 0 ms) [66.837170] irq_shutdown irq150 [66.840410] CPU7: shutdown [66.843112] psci: CPU7 killed (polled 0 ms) [66.885394] CPU8: shutdown [66.888092] psci: CPU8 killed (polled 0 ms) [66.925431] CPU9: shutdown [66.928128] psci: CPU9 killed (polled 0 ms) [66.965526] CPU10: shutdown [66.968311] psci: CPU10 killed (polled 0 ms) [67.025569] irq_shutdown irq151 [67.028808] CPU11: shutdown [67.031599] psci: CPU11 killed (polled 0 ms) [67.306393] CPU12: shutdown [67.309179] psci: CPU12 killed (polled 0 ms) [67.347266] CPU13: shutdown [67.350058] psci: CPU13 killed (polled 0 ms) [67.410910] CPU14: shutdown [67.413695] psci: CPU14 killed (polled 0 ms) [67.454453] irq_shutdown irq152 [67.457699] CPU15: shutdown [67.457799] ------------[ cut here ]------------ [67.460506] psci: CPU15 killed (polled 0 ms) [67.465091] refcount_t: underflow; use-after-free. [67.465112] WARNING: CPU: 0 PID: 557 at lib/refcount.c:28 refcount_warn_saturate+0x6c/0x13c [67.482468] Modules linked in: [67.485511] CPU: 0 PID: 557 Comm: irq/149-hisi_sa Not tainted 5.7.0-rc2-13190-gb615db31b5f4 #331 [67.494281] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 IT21 Nemo 2.0 RC0 04/18/2018 [67.503398] pstate: 20000005 (nzCv daif -PAN -UAO) [67.508176] pc : refcount_warn_saturate+0x6c/0x13c [67.512954] lr : refcount_warn_saturate+0x6c/0x13c [67.517730] sp : ffff80001fd3b920 [67.521031] x29: ffff80001fd3b920 x28: ffff001f9c549500 [67.526329] x27: ffff001f9c54956c x26: 000000000000d8d0 [67.531627] x25: ffff001fb08c7378 x24: 0000000000000000 [67.536925] x23: ffff001f9a6eef00 x22: 0000000000000000 [67.542223] x21: 0000000000001000 x20: ffff001f9ebd0918 [67.547521] x19: ffff001f9a6eef00 x18: 0000000000000000 [67.552820] x17: 000000007fffffff x16: 0000000000000311 [67.558118] x15: 000000000001ffff x14: 0000000000000001 [67.563416] x13: 000000000000001f x12: 0000000000000000 [67.568714] x11: ffff800011aca468 x10: 0000000000fddfa0 [67.574012] x9 : 0000000000000000 x8 : ffff041faa022098 [67.579310] x7 : 0000000000000000 x6 : ffff001ffbe871d0 [67.584608] x5 : 0000000000000001 x4 : 0000000000000000 [67.589906] x3 : 0000000000000000 x2 : 0000000000000007 [67.595204] x1 : 0000000100000001 x0 : 0000000000000026 [67.600503] Call trace: [67.602937] refcount_warn_saturate+0x6c/0x13c
Quoted 8 lines
[67.607369] aio_complete_rw+0x350/0x384 [67.611279] blkdev_bio_end_io+0xc4/0x12c [67.615276] bio_endio+0x104/0x130 [67.618665] blk_update_request+0x98/0x37c [67.622748] blk_mq_end_request+0x24/0x138 [67.626831] blk_mq_resubmit_end_rq+0x40/0x58 [67.631174] __blk_mq_end_request+0xb0/0x10c [67.635432] scsi_end_request+0xdc/0x20c
Looks an old issue, I believe the following patch can fix the issue:
block/blk-mq.c +4 -0
@@ -2415,6 +2415,10 @@ static void blk_mq_resubmit_rq(struct request *rq) nrq->bio = rq->bio; nrq->biotail = rq->biotail; +/* Now all bios ownership is transfered to 'nrq' */ + rq->bio = rq->biotail =NULL ; + rq->__data_len =0 ; +if (blk_insert_cloned_request (nrq->q, nrq) !=BLK_STS_OK )blk_mq_request_bypass_insert (nrq, false, true); }
Thanks,
Ming
Quoted 16 lines · Ming Lei
Quoted 9 lines
[67.602937] refcount_warn_saturate+0x6c/0x13c [67.607369] aio_complete_rw+0x350/0x384 [67.611279] blkdev_bio_end_io+0xc4/0x12c [67.615276] bio_endio+0x104/0x130 [67.618665] blk_update_request+0x98/0x37c [67.622748] blk_mq_end_request+0x24/0x138 [67.626831] blk_mq_resubmit_end_rq+0x40/0x58 [67.631174] __blk_mq_end_request+0xb0/0x10c [67.635432] scsi_end_request+0xdc/0x20cLooks an old issue, I believe the following patch can fix the issue:
block/blk-mq.c +0
-0
block/blk-mq.c static void blk_mq_resubmit_rq(struct request *rq) · line 2424
Quoted 10 lines
block/blk-mq.c +4
-0@@ -2415,6 +2415,10 @@ static void blk_mq_resubmit_rq(struct request *rq) nrq->bio = rq->bio; nrq->biotail = rq->biotail; +/* Now all bios ownership is transfered to 'nrq' */ + rq->bio = rq->biotail =NULL ; + rq->__data_len =0 ; +if (blk_insert_cloned_request (nrq->q, nrq) !=BLK_STS_OK )blk_mq_request_bypass_insert (nrq, false, true); }
ok, looks much better.
I'll test this a bit more now.
cheers
John Garry
Tested-byQuoted 29 lines · John Garry
Quoted 16 lines · Ming Lei
Quoted 9 lines
[67.602937] refcount_warn_saturate+0x6c/0x13c [67.607369] aio_complete_rw+0x350/0x384 [67.611279] blkdev_bio_end_io+0xc4/0x12c [67.615276] bio_endio+0x104/0x130 [67.618665] blk_update_request+0x98/0x37c [67.622748] blk_mq_end_request+0x24/0x138 [67.626831] blk_mq_resubmit_end_rq+0x40/0x58 [67.631174] __blk_mq_end_request+0xb0/0x10c [67.635432] scsi_end_request+0xdc/0x20cLooks an old issue, I believe the following patch can fix the issue:
block/blk-mq.c +0
-0block/blk-mq.c static void blk_mq_resubmit_rq(struct request *rq) · line 2423
Quoted 9 lines
block/blk-mq.c +4
-0@@ -2415,6 +2415,10 @@ static void blk_mq_resubmit_rq(struct request *rq) nrq->bio = rq->bio; nrq->biotail = rq->biotail; + /* Now all bios ownership is transfered to 'nrq' */ + rq->bio = rq->biotail = NULL; + rq->__data_len = 0; + if (blk_insert_cloned_request(nrq->q, nrq) != BLK_STS_OK) blk_mq_request_bypass_insert(nrq, false, true); }ok, looks much better.
Quoted 1 line
I'll test this a bit more now.
looks pretty solid now. I assume we'll see a v10, so please add:
Quoted 4 lines
cheers
.