blk-mq: improvement CPU hotplug
Subsystems (3)
v5 cover blk-mq: improvement CPU hotplug
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very beginning:
The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"
But no drivers or blk-mq do that before one hctx becomes inactive(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see blk_mq_hctx_notify_dead().
This patchset tries to address the issue by two stages:
1) add one new cpuhp state of CPUHP_AP_BLK_MQ_ONLINE
- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of CPUHP_BLK_MQ_DEAD after the hctx becomes dead
- steal bios from the request, and resubmit them via generic_make_request(), then these IO will be mapped to other live hctx for dispatch
Please comment & review, thanks!
V5:
- rename BLK_MQ_S_INTERNAL_STOPPED as BLK_MQ_S_INACTIVE
- re-factor code for re-submit requests in cpu dead hotplug handler
- address requeue corner case
V4:
- resubmit IOs in dispatch list in case that this hctx is dead
V3:
- re-organize patch 2 & 3 a bit for addressing Hannes's comment
- fix patch 4 for avoiding potential deadlock, as found by Hannes
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and Minwoo
Ming Lei (6):
blk-mq: add new state of BLK_MQ_S_INACTIVE
blk-mq: prepare for draining IO when hctx's all CPUs are offline
blk-mq: stop to handle IO and drain IO before hctx becomes inactive
blk-mq: re-submit IO in case that hctx is inactive
blk-mq: handle requests dispatched from IO scheduler in case of
inactive hctx
block: deactivate hctx when all its CPUs are offline when running
queue
block/blk-mq-debugfs.c | 2 + block/blk-mq-tag.c | 2 +- block/blk-mq-tag.h | 2 + block/blk-mq.c | 172 +++++++++++++++++++++++++++++++------ block/blk-mq.h | 3 +- drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 6 ++ include/linux/cpuhotplug.h | 1 + 9 files changed, 163 insertions(+), 29 deletions(-)
Signature · 1 line
2.20.1
PATCH v5 1/6 blk-mq: add new state of BLK_MQ_S_INACTIVE
Add a new hw queue state of BLK_MQ_S_INACTIVE, which is set when all CPUs of this hctx are offline.
Prepares for stopping hw queue when all CPUs of this hctx become offline.
--- block/blk-mq-debugfs.c | 1 + block/blk-mq.h | 3 ++- include/linux/blk-mq.h | 3 +++ 3 files changed, 6 insertions(+), 1 deletion(-)
block/blk-mq-debugfs.c +1 -0
@@ -213,6 +213,7 @@ static const char *const hctx_state_name[] = {HCTX_STATE_NAME (STOPPED ),HCTX_STATE_NAME (TAG_ACTIVE ),HCTX_STATE_NAME (SCHED_RESTART ), + HCTX_STATE_NAME(INACTIVE), };#undef HCTX_STATE_NAME
block/blk-mq.h +2 -1
@@ -167,7 +167,8 @@ static inline struct blk_mq_tags *blk_mq_tags_from_data(struct blk_mq_alloc_datastatic inline bool blk_mq_hctx_stopped (struct blk_mq_hw_ctx *hctx) { -return test_bit (BLK_MQ_S_STOPPED , &hctx->state); +return test_bit (BLK_MQ_S_STOPPED , &hctx->state) || +test_bit (BLK_MQ_S_INACTIVE , &hctx->state); }static inline bool blk_mq_hw_queue_mapped (struct blk_mq_hw_ctx *hctx)
include/linux/blk-mq.h +3 -0
@@ -397,6 +397,9 @@ enum {BLK_MQ_S_TAG_ACTIVE =1 ,BLK_MQ_S_SCHED_RESTART =2 , +/* hw queue is inactive after all its CPUs become offline */ +BLK_MQ_S_INACTIVE =3 , +BLK_MQ_MAX_DEPTH =10240 , BLK_MQ_CPU_WORK_BATCH = 8,
Signature · 1 line
2.20.1
PATCH v5 2/6 blk-mq: prepare for draining IO when hctx's all CPUs are offline
In reply to Ming LeiMost of blk-mq drivers depend on managed IRQ's auto-affinity to setup up queue mapping. Thomas mentioned the following point[1]:
"
That was the constraint of managed interrupts from the very beginning:
The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"
However, current blk-mq implementation doesn't quiesce hw queue before the last CPU in the hctx is shutdown. Even worse, CPUHP_BLK_MQ_DEAD is one cpuhp state handled after the CPU is down, so there isn't any chance to quiesce hctx for blk-mq wrt. CPU hotplug.
Add new cpuhp state of CPUHP_AP_BLK_MQ_ONLINE for blk-mq to stop queues and wait for completion of in-flight requests.
We will stop hw queue and wait for completion of in-flight requests when one hctx is becoming dead in the following patch. This way may cause dead-lock for some stacking blk-mq drivers, such as dm-rq and loop.
Add blk-mq flag of BLK_MQ_F_NO_MANAGED_IRQ and mark it for dm-rq and loop, so we needn't to wait for completion of in-flight requests from dm-rq & loop, then the potential dead-lock can be avoided.
[1] https://lore.kernel.org/linux-block/[email protected]/
--- block/blk-mq-debugfs.c | 1 + block/blk-mq.c | 13 +++++++++++++ drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 3 +++ include/linux/cpuhotplug.h | 1 + 6 files changed, 20 insertions(+), 2 deletions(-)
block/blk-mq-debugfs.c +1 -0
@@ -240,6 +240,7 @@ static const char *const hctx_flag_name[] = {HCTX_FLAG_NAME (TAG_SHARED ),HCTX_FLAG_NAME (BLOCKING ),HCTX_FLAG_NAME (NO_SCHED ), + HCTX_FLAG_NAME(NO_MANAGED_IRQ), };#undef HCTX_FLAG_NAME
block/blk-mq.c +13 -0
@@ -2244,6 +2244,11 @@ int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags,return -ENOMEM ; } +static int blk_mq_hctx_notify_online (unsigned int cpu,struct hlist_node *node) +{ +return 0 ; +} +/* * 'cpu' is going away. splice any existing rq_list entries from this * software queue to the hw queue dispatch list, and ensure that it
@@ -2280,6 +2285,9 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node)static void blk_mq_remove_cpuhp (struct blk_mq_hw_ctx *hctx) { +if (!(hctx->flags &BLK_MQ_F_NO_MANAGED_IRQ )) +cpuhp_state_remove_instance_nocalls (CPUHP_AP_BLK_MQ_ONLINE , + &hctx->cpuhp_online);cpuhp_state_remove_instance_nocalls (CPUHP_BLK_MQ_DEAD , &hctx->cpuhp_dead); }
@@ -2339,6 +2347,9 @@ static int blk_mq_init_hctx(struct request_queue *q, { hctx->queue_num = hctx_idx; +if (!(hctx->flags &BLK_MQ_F_NO_MANAGED_IRQ )) +cpuhp_state_add_instance_nocalls (CPUHP_AP_BLK_MQ_ONLINE , + &hctx->cpuhp_online);cpuhp_state_add_instance_nocalls (CPUHP_BLK_MQ_DEAD , &hctx->cpuhp_dead); hctx->tags = set->tags[hctx_idx];
@@ -3585,6 +3596,8 @@ static int __init blk_mq_init(void) {cpuhp_setup_state_multi (CPUHP_BLK_MQ_DEAD ,"block/mq:dead" ,NULL , blk_mq_hctx_notify_dead); +cpuhp_setup_state_multi (CPUHP_AP_BLK_MQ_ONLINE ,"block/mq:online" , +NULL , blk_mq_hctx_notify_online);return 0 ; }subsys_initcall (blk_mq_init);
drivers/block/loop.c +1 -1
@@ -2012,7 +2012,7 @@ static int loop_add(struct loop_device **l, int i) lo->tag_set.queue_depth =128 ; lo->tag_set.numa_node =NUMA_NO_NODE ; lo->tag_set.cmd_size =sizeof (struct loop_cmd ); - lo->tag_set.flags =BLK_MQ_F_SHOULD_MERGE ; + lo->tag_set.flags =BLK_MQ_F_SHOULD_MERGE |BLK_MQ_F_NO_MANAGED_IRQ ; lo->tag_set.driver_data = lo; err =blk_mq_alloc_tag_set (&lo->tag_set);
drivers/md/dm-rq.c +1 -1
@@ -547,7 +547,7 @@ int dm_mq_init_request_queue(struct mapped_device *md, struct dm_table *t) md->tag_set->ops = &dm_mq_ops; md->tag_set->queue_depth =dm_get_blk_mq_queue_depth (); md->tag_set->numa_node = md->numa_node_id; - md->tag_set->flags =BLK_MQ_F_SHOULD_MERGE ; + md->tag_set->flags =BLK_MQ_F_SHOULD_MERGE |BLK_MQ_F_NO_MANAGED_IRQ ; md->tag_set->nr_hw_queues =dm_get_blk_mq_nr_hw_queues (); md->tag_set->driver_data = md;
include/linux/blk-mq.h +3 -0
@@ -140,6 +140,8 @@ struct blk_mq_hw_ctx {*/ atomic_t nr_active; +/** @cpuhp_online: List to store request if CPU is going to die */ +struct hlist_node cpuhp_online;/** @cpuhp_dead: List to store request if some CPU die. */ struct hlist_node cpuhp_dead;/** @kobj: Kernel object for sysfs. */
@@ -388,6 +390,7 @@ struct blk_mq_ops {enum {BLK_MQ_F_SHOULD_MERGE =1 <<0 ,BLK_MQ_F_TAG_SHARED =1 <<1 , +BLK_MQ_F_NO_MANAGED_IRQ =1 <<2 ,BLK_MQ_F_BLOCKING =1 <<5 ,BLK_MQ_F_NO_SCHED =1 <<6 ,BLK_MQ_F_ALLOC_POLICY_START_BIT =8 ,
include/linux/cpuhotplug.h +1 -0
@@ -149,6 +149,7 @@ enum cpuhp_state {CPUHP_AP_SMPBOOT_THREADS ,CPUHP_AP_X86_VDSO_VMA_ONLINE ,CPUHP_AP_IRQ_AFFINITY_ONLINE , +CPUHP_AP_BLK_MQ_ONLINE ,CPUHP_AP_ARM_MVEBU_SYNC_CLOCKS ,CPUHP_AP_X86_INTEL_EPB_ONLINE , CPUHP_AP_PERF_ONLINE,
Signature · 1 line
2.20.1
PATCH v5 3/6 blk-mq: stop to handle IO and drain IO before hctx becomes inactive
In reply to Ming LeiBefore one CPU becomes offline, check if it is the last online CPU of hctx. If yes, mark this hctx as inactive, meantime wait for completion of all in-flight IOs originated from this hctx.
This way guarantees that there isn't any inflight IO before shutdowning the managed IRQ line.
--- block/blk-mq-tag.c | 2 +- block/blk-mq-tag.h | 2 ++ block/blk-mq.c | 40 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+), 1 deletion(-)
block/blk-mq-tag.c +1 -1
@@ -317,7 +317,7 @@ static void bt_tags_for_each(struct blk_mq_tags *tags, struct sbitmap_queue *bt,* true to continue iterating tags, false to stop. * @priv: Will be passed as second argument to @fn. */ -static void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags, +void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags,busy_tag_iter_fn *fn,void *priv) {if (tags->nr_reserved_tags)
block/blk-mq-tag.h +2 -0
@@ -34,6 +34,8 @@ extern int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx,extern void blk_mq_tag_wakeup_all (struct blk_mq_tags *tags,bool );void blk_mq_queue_tag_busy_iter (struct request_queue *q,busy_iter_fn *fn,void *priv); +void blk_mq_all_tag_busy_iter (struct blk_mq_tags *tags, +busy_tag_iter_fn *fn,void *priv);static inline struct sbq_wait_state *bt_wait_ptr (struct sbitmap_queue *bt,struct blk_mq_hw_ctx *hctx)
block/blk-mq.c +40 -0
@@ -2244,8 +2244,46 @@ int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags,return -ENOMEM ; } +static bool blk_mq_count_inflight_rq (struct request *rq,void *data, +bool reserved) +{ +unsigned *count = data; + +if ((blk_mq_rq_state (rq) ==MQ_RQ_IN_FLIGHT )) + (*count)++; + +return true; +} + +static unsigned blk_mq_tags_inflight_rqs (struct blk_mq_tags *tags) +{ +unsigned count =0 ; + +blk_mq_all_tag_busy_iter (tags, blk_mq_count_inflight_rq, &count); + +return count; +} + +static void blk_mq_hctx_drain_inflight_rqs (struct blk_mq_hw_ctx *hctx) +{ +while (1 ) { +if (!blk_mq_tags_inflight_rqs (hctx->tags)) +break ; +msleep (5 ); + } +} +static int blk_mq_hctx_notify_online (unsigned int cpu,struct hlist_node *node) { +struct blk_mq_hw_ctx *hctx =hlist_entry_safe (node, + struct blk_mq_hw_ctx, cpuhp_online); + +if ((cpumask_next_and (-1 , hctx->cpumask, cpu_online_mask) == cpu) && + (cpumask_next_and (cpu, hctx->cpumask, cpu_online_mask) + >= nr_cpu_ids)) { +set_bit (BLK_MQ_S_INACTIVE , &hctx->state); +blk_mq_hctx_drain_inflight_rqs (hctx); + }return 0 ; }
@@ -2272,6 +2310,8 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node) }spin_unlock (&ctx->lock); +clear_bit (BLK_MQ_S_INACTIVE , &hctx->state); +if (list_empty (&tmp))return 0 ;
Signature · 1 line
2.20.1
PATCH v5 4/6 blk-mq: re-submit IO in case that hctx is inactive
In reply to Ming LeiWhen all CPUs in one hctx are offline and this hctx becomes inactive, we shouldn't run this hw queue for completing request any more.
So steal bios from the request, and resubmit them, and finally free the request in blk_mq_hctx_notify_dead().
--- block/blk-mq.c | 58 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 47 insertions(+), 11 deletions(-)
block/blk-mq.c +47 -11
@@ -2287,10 +2287,34 @@ static int blk_mq_hctx_notify_online(unsigned int cpu, struct hlist_node *node)return 0 ; } +static void blk_mq_resubmit_io (struct request *rq) +{ +struct bio_list list; +struct bio *bio; + +bio_list_init (&list); +blk_steal_bios (&list, rq); + +/* +* Free the old empty request before submitting bio for avoiding +* potential deadlock +*/ +blk_mq_cleanup_rq (rq); +blk_mq_end_request (rq,0 ); + +while (true) { + bio =bio_list_pop (&list); +if (!bio) +break ; + +generic_make_request (bio); + } +} +/* -* 'cpu' is going away. splice any existing rq_list entries from this -* software queue to the hw queue dispatch list, and ensure that it -* gets run. +* 'cpu' has gone away. If this hctx is inactive, we can't dispatch request +* to the hctx any more, so steal bios from requests of this hctx, and +* re-submit them to the request queue, and free these requests finally. */ static int blk_mq_hctx_notify_dead (unsigned int cpu,struct hlist_node *node) {
@@ -2310,16 +2334,28 @@ static int blk_mq_hctx_notify_dead(unsigned int cpu, struct hlist_node *node) }spin_unlock (&ctx->lock); -clear_bit (BLK_MQ_S_INACTIVE , &hctx->state); - -if (list_empty (&tmp)) -return 0 ; +if (!test_bit (BLK_MQ_S_INACTIVE , &hctx->state)) { +if (!list_empty (&tmp)) { +spin_lock (&hctx->lock); +list_splice_tail_init (&tmp, &hctx->dispatch); +spin_unlock (&hctx->lock); +blk_mq_run_hw_queue (hctx, true); + } + }else { +/* requests in dispatch list has to be re-submitted too */ +spin_lock (&hctx->lock); +list_splice_tail_init (&hctx->dispatch, &tmp); +spin_unlock (&hctx->lock); -spin_lock (&hctx->lock); -list_splice_tail_init (&tmp, &hctx->dispatch); -spin_unlock (&hctx->lock); +while (!list_empty (&tmp)) { +struct request *rq =list_entry (tmp.next, + struct request, queuelist); +list_del_init (&rq->queuelist); +blk_mq_resubmit_io (rq); + } +clear_bit (BLK_MQ_S_INACTIVE , &hctx->state); + } -blk_mq_run_hw_queue (hctx, true);return 0 ; }
Signature · 1 line
2.20.1
PATCH v5 5/6 blk-mq: handle requests dispatched from IO scheduler in case of inactive hctx
In reply to Ming LeiIf one hctx becomes inactive when its CPUs are all offline, all in-queue requests aimed at this hctx have to be re-submitted.
Re-submit requests from both sw queue or scheduler queue when the hctx is found as inactive.
--- block/blk-mq.c | 80 ++++++++++++++++++++++++++++++++------------------ 1 file changed, 52 insertions(+), 28 deletions(-)
block/blk-mq.c +52 -28
@@ -2311,6 +2311,42 @@ static void blk_mq_resubmit_io(struct request *rq) } } +static void blk_mq_hctx_deactivate (struct blk_mq_hw_ctx *hctx) +{ +LIST_HEAD (sched_tmp); +LIST_HEAD (re_submit); +struct request *rq; +struct elevator_queue *e = hctx->queue->elevator; + +if (!e) { +blk_mq_flush_busy_ctxs (hctx, &re_submit); + }else { +while ((rq = e->type->ops.dispatch_request (hctx))) { +if (rq->mq_hctx != hctx) +list_add (&rq->queuelist, &sched_tmp); +else +list_add (&rq->queuelist, &re_submit); + } + } +while (!list_empty (&sched_tmp)) { + rq = list_entry(sched_tmp.next, struct request, + queuelist); +list_del_init (&rq->queuelist); +blk_mq_sched_insert_request (rq, true, true, true); + } + +/* requests in dispatch list has to be re-submitted too */ +spin_lock (&hctx->lock); +list_splice_tail_init (&hctx->dispatch, &re_submit); +spin_unlock (&hctx->lock); + +while (!list_empty (&re_submit)) { + rq = list_entry(re_submit.next, struct request, queuelist); +list_del_init (&rq->queuelist); +blk_mq_resubmit_io (rq); + } +} +/* * 'cpu' has gone away. If this hctx is inactive, we can't dispatch request * to the hctx any more, so steal bios from requests of this hctx, and
@@ -2318,42 +2354,30 @@ static void blk_mq_resubmit_io(struct request *rq)*/ static int blk_mq_hctx_notify_dead (unsigned int cpu,struct hlist_node *node) { -struct blk_mq_hw_ctx *hctx; -struct blk_mq_ctx *ctx; -LIST_HEAD (tmp); -enum hctx_type type; - - hctx = hlist_entry_safe(node, struct blk_mq_hw_ctx, cpuhp_dead); - ctx =__blk_mq_get_ctx (hctx->queue, cpu); - type = hctx->type; +struct blk_mq_hw_ctx *hctx =hlist_entry_safe (node, + struct blk_mq_hw_ctx, cpuhp_dead); -spin_lock (&ctx->lock); -if (!list_empty (&ctx->rq_lists[type])) { -list_splice_init (&ctx->rq_lists[type], &tmp); -blk_mq_hctx_clear_pending (hctx, ctx); - } -spin_unlock (&ctx->lock); +if (test_bit (BLK_MQ_S_INACTIVE , &hctx->state)) { +blk_mq_hctx_deactivate (hctx); +clear_bit (BLK_MQ_S_INACTIVE , &hctx->state); + }else if (!hctx->queue->elevator) { +struct blk_mq_ctx *ctx =__blk_mq_get_ctx (hctx->queue, cpu); +enum hctx_type type = hctx->type; +LIST_HEAD (tmp); + +spin_lock (&ctx->lock); +if (!list_empty (&ctx->rq_lists[type])) { +list_splice_init (&ctx->rq_lists[type], &tmp); +blk_mq_hctx_clear_pending (hctx, ctx); + } +spin_unlock (&ctx->lock); -if (!test_bit (BLK_MQ_S_INACTIVE , &hctx->state)) {if (!list_empty (&tmp)) {spin_lock (&hctx->lock);list_splice_tail_init (&tmp, &hctx->dispatch);spin_unlock (&hctx->lock);blk_mq_run_hw_queue (hctx, true); } - }else { -/* requests in dispatch list has to be re-submitted too */ -spin_lock (&hctx->lock); -list_splice_tail_init (&hctx->dispatch, &tmp); -spin_unlock (&hctx->lock); - -while (!list_empty (&tmp)) { -struct request *rq =list_entry (tmp.next, - struct request, queuelist); -list_del_init (&rq->queuelist); -blk_mq_resubmit_io (rq); - } -clear_bit (BLK_MQ_S_INACTIVE , &hctx->state); }return 0 ;
Signature · 1 line
2.20.1
PATCH v5 6/6 block: deactivate hctx when all its CPUs are offline when running queue
In reply to Ming LeiWhen we select next cpu for running hw queue, all this hctx's CPUs may become offline, so deactivate this hctx at that time.
So we can fix some corner case which can't be covered by deactivating hctx in CPU hotplug handler, such as request may be requeued during CPU hotplug, which handler can't found the requeued rquests and re-submit them.
--- block/blk-mq.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
block/blk-mq.c +12 -3
@@ -43,6 +43,8 @@static void blk_mq_poll_stats_start (struct request_queue *q);static void blk_mq_poll_stats_fn (struct blk_stat_callback *cb); +static void blk_mq_hctx_deactivate (struct blk_mq_hw_ctx *hctx); +static int blk_mq_poll_stats_bkt (const struct request *rq) {int ddir, sectors, bucket;
@@ -1431,7 +1433,7 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)*/ hctx->next_cpu = next_cpu; hctx->next_cpu_batch =1 ; -return WORK_CPU_UNBOUND ; +return -1 ; } hctx->next_cpu = next_cpu;
@@ -1450,6 +1452,8 @@ static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)static void __blk_mq_delay_run_hw_queue (struct blk_mq_hw_ctx *hctx,bool async,unsigned long msecs) { +int cpu; +if (unlikely (blk_mq_hctx_stopped (hctx)))return ;
@@ -1464,8 +1468,13 @@ static void __blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async,put_cpu (); } -kblockd_mod_delayed_work_on (blk_mq_hctx_next_cpu (hctx), &hctx->run_work, -msecs_to_jiffies (msecs)); + cpu =blk_mq_hctx_next_cpu (hctx); + +if (likely (cpu >=0 )) +kblockd_mod_delayed_work_on (cpu, &hctx->run_work, +msecs_to_jiffies (msecs)); +else +blk_mq_hctx_deactivate (hctx); }/**
Signature · 1 line
2.20.1
Quoted 37 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very beginning:The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again.
"But no drivers or blk-mq do that before one hctx becomes inactive(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(), then these IO will be mapped to other live hctx for dispatchPlease comment & review, thanks!
V5:
- renameBLK_MQ_S_INTERNAL_STOPPEDasBLK_MQ_S_INACTIVE
- re-factor code for re-submit requests in cpu dead hotplug handler
- address requeue corner caseV4:
- resubmit IOs in dispatch list in case that this hctx is dead
Quoted 10 lines
V3:
- re-organize patch 2 & 3 a bit for addressing Hannes's comment
- fix patch 4 for avoiding potential deadlock, as found by HannesV2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and Minwoo
thanks Ming, we'll have a look at this new series and test it again.
John
Quoted 29 lines
Ming Lei (6):
blk-mq: add new state ofBLK_MQ_S_INACTIVE
blk-mq: prepare for draining IO when hctx's all CPUs are offline
blk-mq: stop to handle IO and drain IO before hctx becomes inactive
blk-mq: re-submit IO in case that hctx is inactive
blk-mq: handle requests dispatched from IO scheduler in case of
inactive hctx
block: deactivate hctx when all its CPUs are offline when running
queueblock/blk-mq-debugfs.c | 2 + block/blk-mq-tag.c | 2 +- block/blk-mq-tag.h | 2 + block/blk-mq.c | 172 +++++++++++++++++++++++++++++++------ block/blk-mq.h | 3 +- drivers/block/loop.c | 2 +- drivers/md/dm-rq.c | 2 +- include/linux/blk-mq.h | 6 ++ include/linux/cpuhotplug.h | 1 + 9 files changed, 163 insertions(+), 29 deletions(-)Cc John Garry <[email protected]>Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Keith Busch <[email protected]>
Quoted 42 lines · John Garry
Quoted 41 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very beginning:
The driver/subsystem has to quiesce the interrupt line and the associated queue _before_ it gets shutdown in CPU unplug and not fiddle with it until it's restarted by the core when the CPU is plugged in again. "But no drivers or blk-mq do that before one hctx becomes inactive(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(),
then these IO will be mapped to other live hctx for dispatchPlease comment & review, thanks!
V5:
- renameBLK_MQ_S_INTERNAL_STOPPEDasBLK_MQ_S_INACTIVE
- re-factor code for re-submit requests in cpu dead hotplug handler
- address requeue corner caseV4:
- resubmit IOs in dispatch list in case that this hctx is deadV3:
- re-organize patch 2 & 3 a bit for addressing Hannes's comment
- fix patch 4 for avoiding potential deadlock, as found by Hannes
Quoted 6 lines
Quoted 6 lines
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and Minwoo
Hi Ming,
Colleague Xiang Chen found this:
Starting 40 processes
Jobs: 40 (f=40)
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [8.3% done] [2899MB/0KB/0KB /s] [742K/0/0 iops] [eta 00m:33s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [11.1% done] [2892MB/0KB/0KB /s] [740K/0/0 iops] [eta 00m:32s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [13.9% done] [2887MB/0KB/0KB /s] [739K/0/0 iops] [eta 00m:31s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [16.7% done] [2850MB/0KB/0KB /s] [730K/0/0 iops] [eta 00m:30s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [19.4% done] [2834MB/0KB/0KB /s] [726K/0/0 iops] [eta 00m:29s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [22.2% done] [2827MB/0KB/0KB /s] [724K/0/0 iops] [eta 00m:28s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [25.0% done] [2804MB/0KB/0KB /s] [718K/0/0 iops] [eta 00m:27s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [27.8% done] [2783MB/0KB/0KB /s] [713K/0/0 iops] [eta 00m:26s]
[ 112.990292] CPU16: shutdown [ 112.993113] psci: CPU16 killed (polled 0 ms) [ 113.029429] CPU15: shutdown [ 113.032245] psci: CPU15 killed (polled 0 ms) [ 113.073649] CPU14: shutdown [ 113.076461] psci: CPU14 killed (polled 0 ms) Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRR[ 113.121258] CPU13: shutdown RRRRRRRRRR] [30.[ 113.124247] psci: CPU13 killed (polled 0 ms) 6% done] [2772MB/0KB/0KB /s] [710K/0/0 iops] [eta 00m:25s] [ 113.177108] IRQ 600: no longer affine to CPU12 [ 113.183185] CPU12: shutdown [ 113.186001] psci: CPU12 killed (polled 0 ms) [ 113.242555] CPU11: shutdown [ 113.245368] psci: CPU11 killed (polled 0 ms) [ 113.302598] CPU10: shutdown [ 113.305420] psci: CPU10 killed (polled 0 ms) [ 113.365757] CPU9: shutdown [ 113.368489] psci: CPU9 killed (polled 0 ms) [ 113.611904] IRQ 599: no longer affine to CPU8 [ 113.620611] CPU8: shutdown [ 113.623340] psci: CPU8 killed (polled 0 ms)
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [33.3% done] [2765MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:24s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [36.1% done] [2767MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:23s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [38.9% done] [2771MB/0KB/0KB /s] [709K/0/0 iops] [eta 00m:22s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [41.7% done] [2780MB/0KB/0KB /s] [712K/0/0 iops] [eta 00m:21s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [44.4% done] [2731MB/0KB/0KB /s] [699K/0/0 iops] [eta 00m:20s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [47.2% done] [2757MB/0KB/0KB /s] [706K/0/0 iops] [eta 00m:19s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [50.0% done] [2764MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:18s]
[ 120.762141] ------------[ cut here ]------------ [ 120.766760] WARNING: CPU: 0 PID: 10 at block/blk.h:299 generic_make_request_checks+0x480/0x4e8 [ 120.775348] Modules linked in: [ 120.778397] CPU: 0 PID: 10 Comm: ksoftirqd/0 Not tainted 5.5.0-
Then this:
1158048319d:18h:40m:34s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049605d:23h:03m:30s]
[ 141.915198] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 141.921280] rcu: 96-...0: (4 GPs behind) idle=af6/0/0x1 softirq=90/90 fqs=2626 [ 141.928658] rcu: 99-...0: (2 GPs behind) idle=b82/1/0x4000000000000000 softirq=82/82 fqs=2627 [ 141.937339] (detected by 17, t=5254 jiffies, g=3925, q=545) [ 141.942985] Task dump for CPU 96: [ 141.946292] swapper/96 R running task 0 0 1 0x0000002a [ 141.953321] Call trace: [ 141.955763] __switch_to+0xbc/0x218 [ 141.959244] 0x0 [ 141.961079] Task dump for CPU 99: [ 141.964385] kworker/99:1H R running task 0 3473 2 0x0000022a [ 141.971417] Workqueue: kblockd blk_mq_run_work_fn [ 141.976109] Call trace: [ 141.978550] __switch_to+0xbc/0x218 [ 141.982029] blk_mq_run_work_fn+0x1c/0x28 [ 141.986027] process_one_work+0x1e0/0x358 [ 141.990025] worker_thread+0x40/0x488 [ 141.993678] kthread+0x118/0x120 [ 141.996897] ret_from_fork+0x10/0x18
Fuller log at bottom.
We're basing the patchset on v5.5-rc6:
https://github.com/hisilicon/kernel-dev/tree/private-topic-sas-5.5-mq-debug-improve-cpu-hotplug-v5 (we have not tested my latest patch there, but I doubt that this is the problem).
Not sure if something missing which is in -next.
Cheers,
John
uptime: 0 days, 0 hours
Usage on /: 8% Swap usage: 0.0%
IP Addresses: 192.168.1.164
Euler:~ # export TMOUT=0
Euler:~ # fdisk -l | grep sd | wc -l
7
Euler:~ # ps -aux | grep irqbalance
root 3307 0.0 0.0 106076 1796 ttyS0 S+ 05:58 0:00 grep --color=auto irqbalance
Euler:~ # echo 15 > /proc/sys/kernel/printk
Euler:~ #
Euler:~ #
Euler:~ # ./hotplug_sas_d06.sh
Looping ... number 0
Creat 4k_read_depth2048_fiotest file sucessfully
cp: cannot stat ‘liba*’: No such file or directory
chmod: cannot access ‘fio’: No such file or directory
job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2048
...
job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2048
...
job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2048
...
job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=2048
...
fio-2.1.10
Starting 40 processes
Jobs: 40 (f=40)
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [8.3% done] [2899MB/0KB/0KB /s] [742K/0/0 iops] [eta 00m:33s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [11.1% done] [2892MB/0KB/0KB /s] [740K/0/0 iops] [eta 00m:32s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [13.9% done] [2887MB/0KB/0KB /s] [739K/0/0 iops] [eta 00m:31s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [16.7% done] [2850MB/0KB/0KB /s] [730K/0/0 iops] [eta 00m:30s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [19.4% done] [2834MB/0KB/0KB /s] [726K/0/0 iops] [eta 00m:29s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [22.2% done] [2827MB/0KB/0KB /s] [724K/0/0 iops] [eta 00m:28s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [25.0% done] [2804MB/0KB/0KB /s] [718K/0/0 iops] [eta 00m:27s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [27.8% done] [2783MB/0KB/0KB /s] [713K/0/0 iops] [eta 00m:26s]
[ 112.990292] CPU16: shutdown [ 112.993113] psci: CPU16 killed (polled 0 ms) [ 113.029429] CPU15: shutdown [ 113.032245] psci: CPU15 killed (polled 0 ms) [ 113.073649] CPU14: shutdown [ 113.076461] psci: CPU14 killed (polled 0 ms) Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRR[ 113.121258] CPU13: shutdown RRRRRRRRRR] [30.[ 113.124247] psci: CPU13 killed (polled 0 ms) 6% done] [2772MB/0KB/0KB /s] [710K/0/0 iops] [eta 00m:25s] [ 113.177108] IRQ 600: no longer affine to CPU12 [ 113.183185] CPU12: shutdown [ 113.186001] psci: CPU12 killed (polled 0 ms) [ 113.242555] CPU11: shutdown [ 113.245368] psci: CPU11 killed (polled 0 ms) [ 113.302598] CPU10: shutdown [ 113.305420] psci: CPU10 killed (polled 0 ms) [ 113.365757] CPU9: shutdown [ 113.368489] psci: CPU9 killed (polled 0 ms) [ 113.611904] IRQ 599: no longer affine to CPU8 [ 113.620611] CPU8: shutdown [ 113.623340] psci: CPU8 killed (polled 0 ms)
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [33.3% done] [2765MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:24s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [36.1% done] [2767MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:23s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [38.9% done] [2771MB/0KB/0KB /s] [709K/0/0 iops] [eta 00m:22s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [41.7% done] [2780MB/0KB/0KB /s] [712K/0/0 iops] [eta 00m:21s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [44.4% done] [2731MB/0KB/0KB /s] [699K/0/0 iops] [eta 00m:20s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [47.2% done] [2757MB/0KB/0KB /s] [706K/0/0 iops] [eta 00m:19s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [50.0% done] [2764MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:18s]
[ 120.762141] ------------[ cut here ]------------ [ 120.766760] WARNING: CPU: 0 PID: 10 at block/blk.h:299 generic_make_request_checks+0x480/0x4e8 [ 120.775348] Modules linked in: [ 120.778397] CPU: 0 PID: 10 Comm: ksoftirqd/0 Not tainted 5.5.0-rc6-31756-gdddc331 #333 [ 120.786282] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B160.02 01/15/2020 [ 120.795129] pstate: 80c00089 (Nzcv daIf +PAN +UAO) [ 120.799907] pc : generic_make_request_checks+0x480/0x4e8 [ 120.805205] lr : generic_make_request_checks+0x48/0x4e8 [ 120.810415] sp : ffff80001052b700 [ 120.813720] x29: ffff80001052b700 x28: 0000000000000dd0 [ 120.819017] x27: ffff00277b6c0d00 x26: ffff00277863ea18 [ 120.824313] x25: 0000000000000000 x24: ffff00277b6c0d6c [ 120.829612] x23: 0000000000000008 x22: ffff0027d66c7800 [ 120.834909] x21: ffff2027c7133790 x20: ffffc8fcee7c9000 [ 120.840205] x19: ffff00277863ea18 x18: 000000000000000e [ 120.845502] x17: 0000000000000007 x16: 0000000000000001 [ 120.850801] x15: 0000000000000019 x14: 0000000000000033 [ 120.856098] x13: 0000000000000001 x12: 00000000015cd30a [ 120.861394] x11: 0000000000000001 x10: 0000000000000040 [ 120.866692] x9 : ffffc8fcee7df550 x8 : ffffc8fcee3fb018 [ 120.871990] x7 : ffff0027d2010010 x6 : 00000006711b402b [ 120.877287] x5 : 00000000000000c0 x4 : 0000000000000006 [ 120.882585] x3 : ffffffffffffffff x2 : 00000000ffffffff [ 120.887882] x1 : 0000000000000080 x0 : 0000000000000080 [ 120.893181] Call trace: [ 120.895621] generic_make_request_checks+0x480/0x4e8 [ 120.900571] generic_make_request+0x2c/0x320 [ 120.904832] blk_mq_hctx_deactivate+0x1a0/0x2e8 [ 120.909352] __blk_mq_delay_run_hw_queue+0x220/0x230 [ 120.914303] blk_mq_run_hw_queue+0xa0/0x110 [ 120.918475] blk_mq_run_hw_queues+0x48/0x70 [ 120.922649] scsi_end_request+0x1d8/0x230 [ 120.926648] scsi_io_completion+0x70/0x520 [ 120.930735] scsi_finish_command+0xe0/0xf0 [ 120.934822] scsi_softirq_done+0x80/0x190 [ 120.938821] blk_mq_complete_request+0xc4/0x160 [ 120.943339] scsi_mq_done+0x70/0xc8 [ 120.946821] ata_scsi_qc_complete+0xa0/0x3f0 [ 120.951081] __ata_qc_complete+0xa4/0x150 [ 120.955079] ata_qc_complete+0xe8/0x200 [ 120.958906] sas_ata_task_done+0x194/0x2d0 [ 120.962992] cq_tasklet_v3_hw+0x12c/0x5e8 [ 120.966991] tasklet_action_common.isra.15+0x11c/0x190 [ 120.972115] tasklet_action+0x24/0x30 [ 120.975767] efi_header_end+0x114/0x234 [ 120.979592] run_ksoftirqd+0x38/0x48 [ 120.983159] smpboot_thread_fn+0x16c/0x270 [ 120.987245] kthread+0x118/0x120 [ 120.990465] ret_from_fork+0x10/0x18 [ 120.994031] ---[ end trace 0125724e2f3f2476 ]--- [ 120.998643] ------------[ cut here ]------------ [ 121.003250] WARNING: CPU: 0 PID: 10 at block/blk-mq.c:1365 __blk_mq_run_hw_queue+0xdc/0x128 [ 121.011576] Modules linked in: [ 121.014623] CPU: 0 PID: 10 Comm: ksoftirqd/0 Tainted: G W 5.5.0-rc6-31756-gdddc331 #333 [ 121.023902] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B160.02 01/15/2020 [ 121.032749] pstate: 00c00089 (nzcv daIf +PAN +UAO) [ 121.037526] pc : __blk_mq_run_hw_queue+0xdc/0x128 [ 121.042217] lr : __blk_mq_delay_run_hw_queue+0x1ac/0x230 [ 121.047515] sp : ffff80001052b540 [ 121.050820] x29: ffff80001052b540 x28: 0000000000000dd0 [ 121.056118] x27: ffff00277b6c0d00 x26: ffff00277863ea18 [ 121.061416] x25: 000000000000000c x24: 000000000000000a [ 121.066714] x23: 0000000000000001 x22: 0000000000000000 [ 121.072012] x21: ffffc8fcee7c9000 x20: 0000000000000000 [ 121.077310] x19: ffff0027cb7f8000 x18: 000000000000000e [ 121.082608] x17: 0000000000000007 x16: 0000000000000001 [ 121.087905] x15: 0000000000000019 x14: 0000000000000033 [ 121.093203] x13: 0000000000000001 x12: 00000000015cd30a [ 121.098501] x11: 0000000000000001 x10: 0000000000000040 [ 121.103799] x9 : 00000000ffff4840 x8 : ffffc8fcee3fb018 [ 121.109097] x7 : ffffc8fcee7c98e8 x6 : 0000000000000000 [ 121.114395] x5 : ffffc8fcee7c98c8 x4 : 0000000000000000 [ 121.119692] x3 : 0000000000000000 x2 : ffffc8fcee7c98c8 [ 121.124990] x1 : 00000000000000ff x0 : 0000000000000103 [ 121.130288] Call trace: [ 121.132729] __blk_mq_run_hw_queue+0xdc/0x128 [ 121.137073] __blk_mq_delay_run_hw_queue+0x1ac/0x230 [ 121.142025] blk_mq_run_hw_queue+0xa0/0x110 [ 121.146196] blk_mq_request_bypass_insert+0x54/0x68 [ 121.151060] __blk_mq_try_issue_directly+0x60/0x208 [ 121.155926] blk_mq_try_issue_directly+0x50/0xb8 [ 121.160529] blk_mq_make_request+0x3d0/0x420 [ 121.164787] generic_make_request+0xf4/0x320 [ 121.169046] blk_mq_hctx_deactivate+0x1a0/0x2e8 [ 121.173566] __blk_mq_delay_run_hw_queue+0x220/0x230 [ 121.178517] blk_mq_run_hw_queue+0xa0/0x110 [ 121.182688] blk_mq_run_hw_queues+0x48/0x70 [ 121.186860] scsi_end_request+0x1d8/0x230 [ 121.190858] scsi_io_completion+0x70/0x520 [ 121.194944] scsi_finish_command+0xe0/0xf0 [ 121.199029] scsi_softirq_done+0x80/0x190 [ 121.203027] blk_mq_complete_request+0xc4/0x160 [ 121.207545] scsi_mq_done+0x70/0xc8 [ 121.211024] ata_scsi_qc_complete+0xa0/0x3f0 [ 121.215283] __ata_qc_complete+0xa4/0x150 [ 121.219281] ata_qc_complete+0xe8/0x200 [ 121.223106] sas_ata_task_done+0x194/0x2d0 [ 121.227191] cq_tasklet_v3_hw+0x12c/0x5e8 [ 121.231190] tasklet_action_common.isra.15+0x11c/0x190 [ 121.236312] tasklet_action+0x24/0x30 [ 121.239965] efi_header_end+0x114/0x234 [ 121.243790] run_ksoftirqd+0x38/0x48 [ 121.247356] smpboot_thread_fn+0x16c/0x270 [ 121.251441] kthread+0x118/0x120 [ 121.254661] ret_from_fork+0x10/0x18 [ 121.258226] ---[ end trace 0125724e2f3f2477 ]--- [ 121.267999] BUG: scheduling while atomic: ksoftirqd/0/10/0x00000103 [ 121.274248] Modules linked in: [ 121.277295] CPU: 0 PID: 10 Comm: ksoftirqd/0 Tainted: G W 5.5.0-rc6-31756-gdddc331 #333 [ 121.286574] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B160.02 01/15/2020 [ 121.295420] Call trace: [ 121.297861] dump_backtrace+0x0/0x1a0 [ 121.301513] show_stack+0x14/0x20 [ 121.304822] dump_stack+0xbc/0x104 [ 121.308216] __schedule_bug+0x50/0x70 [ 121.311869] __schedule+0x534/0x598 [ 121.315349] schedule+0x74/0x100 [ 121.318569] io_schedule+0x18/0x98 [ 121.321963] blk_mq_get_tag+0x150/0x300 [ 121.325789] blk_mq_get_request+0x120/0x3a8 [ 121.329961] blk_mq_make_request+0xf8/0x420 [ 121.334134] generic_make_request+0xf4/0x320 [ 121.338392] blk_mq_hctx_deactivate+0x1a0/0x2e8 [ 121.342910] __blk_mq_delay_run_hw_queue+0x220/0x230 [ 121.347861] blk_mq_run_hw_queue+0xa0/0x110 [ 121.352034] blk_mq_run_hw_queues+0x48/0x70 [ 121.356205] scsi_end_request+0x1d8/0x230 [ 121.360204] scsi_io_completion+0x70/0x520 [ 121.364289] scsi_finish_command+0xe0/0xf0 [ 121.368373] scsi_softirq_done+0x80/0x190 [ 121.372372] blk_mq_complete_request+0xc4/0x160 [ 121.376891] scsi_mq_done+0x70/0xc8 [ 121.380371] ata_scsi_qc_complete+0xa0/0x3f0 [ 121.384629] __ata_qc_complete+0xa4/0x150 [ 121.388629] ata_qc_complete+0xe8/0x200 [ 121.392453] sas_ata_task_done+0x194/0x2d0 [ 121.396540] cq_tasklet_v3_hw+0x12c/0x5e8 [ 121.400540] tasklet_action_common.isra.15+0x11c/0x190 [ 121.405664] tasklet_action+0x24/0x30 [ 121.409316] efi_header_end+0x114/0x234 [ 121.413141] run_ksoftirqd+0x38/0x48 [ 121.416706] smpboot_thread_fn+0x16c/0x270 [ 121.420792] kthread+0x118/0x120 [ 121.424013] ret_from_fork+0x10/0x18 Jobs: 40 (f=40):[ 121.427627] NOHZ: local_softirq_pending 2c2 [RRRRRRRRRRRRRR[ 121.433311] NOHZ: local_softirq_pending 40 RRRRRRRRRRRRRRRR[ 121.438658] NOHZ: local_softirq_pending 40 RRRRRRRRRR] [52.[ 121.444103] NOHZ: local_softirq_pending 40 8% done] [1676MB[ 121.449564] NOHZ: local_softirq_pending 40 /0KB/0KB /s] [42[ 121.455028] NOHZ: local_softirq_pending 40 9K/0/0 iops] [et[ 121.460491] NOHZ: local_softirq_pending 40 a 00m:17s] [ 121.465948] NOHZ: local_softirq_pending 40 [ 121.470963] NOHZ: local_softirq_pending 40 [ 121.475055] NOHZ: local_softirq_pending 40
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [55.6% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:16s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [58.3% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:15s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [61.1% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:14s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [63.9% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:13s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [66.7% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:12s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [69.4% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:11s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [72.2% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:10s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [75.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:09s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [77.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:08s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [80.6% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:07s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [83.3% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:06s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [86.1% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:05s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [88.9% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:04s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [91.7% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:03s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [94.4% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:02s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [97.2% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:01s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158047033d:14h:17m:38s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158048319d:18h:40m:34s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049605d:23h:03m:30s]
[ 141.915198] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 141.921280] rcu: 96-...0: (4 GPs behind) idle=af6/0/0x1 softirq=90/90 fqs=2626 [ 141.928658] rcu: 99-...0: (2 GPs behind) idle=b82/1/0x4000000000000000 softirq=82/82 fqs=2627 [ 141.937339] (detected by 17, t=5254 jiffies, g=3925, q=545) [ 141.942985] Task dump for CPU 96: [ 141.946292] swapper/96 R running task 0 0 1 0x0000002a [ 141.953321] Call trace: [ 141.955763] __switch_to+0xbc/0x218 [ 141.959244] 0x0 [ 141.961079] Task dump for CPU 99: [ 141.964385] kworker/99:1H R running task 0 3473 2 0x0000022a [ 141.971417] Workqueue: kblockd blk_mq_run_work_fn [ 141.976109] Call trace: [ 141.978550] __switch_to+0xbc/0x218 [ 141.982029] blk_mq_run_work_fn+0x1c/0x28 [ 141.986027] process_one_work+0x1e0/0x358 [ 141.990025] worker_thread+0x40/0x488 [ 141.993678] kthread+0x118/0x120 [ 141.996897] ret_from_fork+0x10/0x18
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158042478d:14h:26m:49s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158043549d:01h:15m:55s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158044619d:12h:05m:01s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158045689d:22h:54m:06s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158046760d:09h:43m:12s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158047830d:20h:32m:18s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158048901d:07h:21m:24s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049971d:18h:10m:30s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158048794d:00h:27m:52s]
[ 151.263502] softirq: huh, entered softirq 6 TASKLET 000000002aed33dd with preempt_count 00000100, exited with fffffffe? Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049816d:15h:16m:18s] [ 151.312427] sas: Enter sas_scsi_recover_host busy: 192 failed: 192 [ 151.318613] sas: trying to find task 0x00000000294f58ee [ 151.323830] sas: sas_scsi_find_task: aborting task 0x00000000294f58ee
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158045974d:14h:49m:07s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158046897d:22h:56m:49s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158047821d:07h:04m:31s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158048744d:15h:12m:13s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158046537d:10h:31m:29s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158050141d:20h:12m:10s]
[ 157.407215] hisi_sas_v3_hw 0000:74:02.0: internal task abort: timeout and not done. [ 157.414855] hisi_sas_v3_hw 0000:74:02.0: abort task: internal abort failed [ 157.421720] hisi_sas_v3_hw 0000:74:02.0: abort task: rc=-5 [ 157.427202] sas: sas_scsi_find_task: querying task 0x00000000294f58ee [ 157.433634] sas: sas_scsi_find_task: task 0x00000000294f58ee failed to abort [ 157.440670] sas: task 0x00000000294f58ee is not at LU: I_T recover [ 157.446841] sas: I_T nexus reset for dev 500e004aaaaaaa02 [ 157.452239] hisi_sas_v3_hw 0000:74:02.0: internal task abort: executing internal task failed: -22 [ 157.461095] hisi_sas_v3_hw 0000:74:02.0: I_T nexus reset: internal abort (-22) [ 157.468310] hisi_sas_v3_hw 0000:74:02.0: internal task abort: executing internal task failed: -22 [ 157.477170] hisi_sas_v3_hw 0000:74:02.0: lu_reset: internal abort failed [ 157.483866] hisi_sas_v3_hw 0000:74:02.0: lu_reset: for device[3]:rc= -22 [ 157.490558] hisi_sas_v3_hw 0000:74:02.0: internal task abort: executing internal task failed: -22 [ 157.499417] hisi_sas_v3_hw 0000:74:02.0: I_T nexus reset: internal abort (-22) [ 157.506627] sas: clear nexus ha [ 157.516706] hisi_sas_v3_hw 0000:74:02.0: controller resetting... Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158047356d:20h:42m:24s] Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158048189d:11h:12m:35s] [ 159.318030] hisi_sas_v3_hw 0000:74:02.0: phyup: phy0 link_rate=11 [ 159.324114] hisi_sas_v3_hw 0000:74:02.0: phyup: phy1 link_rate=11 [ 159.330195] hisi_sas_v3_hw 0000:74:02.0: phyup: phy2 link_rate=11 [ 159.336274] hisi_sas_v3_hw 0000:74:02.0: phyup: phy3 link_rate=11 [ 159.342354] hisi_sas_v3_hw 0000:74:02.0: phyup: phy4 link_rate=11 [ 159.348433] hisi_sas_v3_hw 0000:74:02.0: phyup: phy5 link_rate=11 [ 159.354513] hisi_sas_v3_hw 0000:74:02.0: phyup: phy6 link_rate=11 [ 159.360593] hisi_sas_v3_hw 0000:74:02.0: phyup: phy7 link_rate=11 Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049022d:01h:42m:46s] [ 160.349201] hisi_sas_v3_hw 0000:74:02.0: controller reset complete [ 160.349208] sas: broadcast received: 0 [ 160.355378] hisi_sas_v3_hw 0000:74:02.0: dump count exceeded! [ 160.359138] sas: REVALIDATING DOMAIN on port 0, pid:937 [ 160.370508] sas: ex 500e004aaaaaaa1f phy01 change count has changed [ 160.376993] sas: ex 500e004aaaaaaa1f phy01 originated BROADCAST(CHANGE) [ 160.383603] sas: ex 500e004aaaaaaa1f rediscovering phy01 [ 160.389117] sas: ex 500e004aaaaaaa1f phy01:U:5 attached: 0000000000000000 (no device) [ 160.398068] sas: ex 500e004aaaaaaa1f phy16 originated BROADCAST(CHANGE) [ 160.404676] sas: ex 500e004aaaaaaa1f rediscovering phy16, part of a wide port with phy17 [ 160.412978] sas: ex 500e004aaaaaaa1f phy16 broadcast flutter [ 160.418751] sas: ex 500e004aaaaaaa1f phy17 originated BROADCAST(CHANGE) [ 160.425358] sas: ex 500e004aaaaaaa1f rediscovering phy17, part of a wide port with phy16 [ 160.433638] sas: ex 500e004aaaaaaa1f phy17 broadcast flutter [ 160.439421] sas: ex 500e004aaaaaaa1f phy18 originated BROADCAST(CHANGE) [ 160.446030] sas: ex 500e004aaaaaaa1f rediscovering phy18, part of a wide port with phy16 [ 160.454316] sas: ex 500e004aaaaaaa1f phy18 broadcast flutter [ 160.460105] sas: ex 500e004aaaaaaa1f phy19 originated BROADCAST(CHANGE) [ 160.466712] sas: ex 500e004aaaaaaa1f rediscovering phy19, part of a wide port with phy16 [ 160.475022] sas: ex 500e004aaaaaaa1f phy19 broadcast flutter [ 160.480801] sas: ex 500e004aaaaaaa1f phy20 originated BROADCAST(CHANGE) [ 160.487410] sas: ex 500e004aaaaaaa1f rediscovering phy20, part of a wide port with phy16 [ 160.495692] sas: ex 500e004aaaaaaa1f phy20 broadcast flutter [ 160.501421] sas: ex 500e004aaaaaaa1f phy21 originated BROADCAST(CHANGE) [ 160.508029] sas: ex 500e004aaaaaaa1f rediscovering phy21, part of a wide port with phy16 [ 160.516309] sas: ex 500e004aaaaaaa1f phy21 broadcast flutter [ 160.522048] sas: ex 500e004aaaaaaa1f phy22 originated BROADCAST(CHANGE) [ 160.528653] sas: ex 500e004aaaaaaa1f rediscovering phy22, part of a wide port with phy16 [ 160.536975] sas: ex 500e004aaaaaaa1f phy22 broadcast flutter [ 160.542699] sas: ex 500e004aaaaaaa1f phy23 originated BROADCAST(CHANGE) [ 160.549307] sas: ex 500e004aaaaaaa1f rediscovering phy23, part of a wide port with phy16 [ 160.557591] sas: ex 500e004aaaaaaa1f phy23 broadcast flutter [ 160.563318] sas: done REVALIDATING DOMAIN on port 0, pid:937, res 0x0
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049854d:16h:12m:57s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158048255d:18h:40m:08s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158050345d:15h:15m:03s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049839d:22h:28m:25s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158046025d:15h:04m:58s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158046744d:14h:10m:15s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158047614d:11h:05m:16s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049100d:12h:27m:26s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158048901d:11h:26m:07s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049620d:10h:31m:24s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158050339d:09h:36m:41s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049314d:16h:14m:45s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158047401d:22h:42m:56s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158048059d:06h:48m:09s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158048716d:14h:53m:22s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049373d:22h:58m:36s]
[ 176.479216] sas: clear nexus ha succeeded [ 176.483220] sas: --- Exit sas_eh_handle_sas_errors -- clear_q [ 176.488980] sas: ata2: end_device-0:0:4: cmd error handler [ 176.494464] sas: ata1: end_device-0:0:2: cmd error handler
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158050031d:07h:03m:49s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049507d:04h:34m:56s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158050077d:05h:12m:47s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049104d:11h:12m:35s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049724d:15h:20m:48s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158050344d:19h:29m:01s]
[ 182.751214] hisi_sas_v3_hw 0000:74:02.0: internal task abort: timeout and not done. [ 182.751217] hisi_sas_v3_hw 0000:74:02.0: dump count exceeded! [ 182.758857] hisi_sas_v3_hw 0000:74:02.0: lu_reset: internal abort failed [ 182.758858] hisi_sas_v3_hw 0000:74:02.0: lu_reset: for device[4]:rc= -5
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158047857d:06h:57m:44s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049091d:00h:22m:57s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049020d:21h:38m:22s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
Quoted 32 lines
Quoted 21 lines
Ming Lei (6):
blk-mq: add new state ofBLK_MQ_S_INACTIVE
blk-mq: prepare for draining IO when hctx's all CPUs are offline
blk-mq: stop to handle IO and drain IO before hctx becomes inactive
blk-mq: re-submit IO in case that hctx is inactive
blk-mq: handle requests dispatched from IO scheduler in case of inactive hctx
block: deactivate hctx when all its CPUs are offline when running queueblock/blk-mq-debugfs.c | 2 +
block/blk-mq-tag.c | 2 +-
block/blk-mq-tag.h | 2 +
block/blk-mq.c | 172 +++++++++++++++++++++++++++++++------ block/blk-mq.h | 3 +-
drivers/block/loop.c | 2 +-
drivers/md/dm-rq.c | 2 +-
include/linux/blk-mq.h | 6 ++include/linux/cpuhotplug.h| 1 +Quoted 9 lines
9 files changed, 163 insertions(+), 29 deletions(-)Cc John Garry <[email protected]>Cc Bart Van Assche <[email protected]>Cc Hannes Reinecke <[email protected]>Cc Christoph Hellwig <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Keith Busch <[email protected]>.
Quoted 139 lines · John Garry
Quoted 46 lines · John Garry
Quoted 45 lines · Ming Lei
Hi,
Thomas mentioned:
"
That was the constraint of managed interrupts from the very
beginning:
The driver/subsystem has to quiesce the interrupt line and the
associated
queue _before_ it gets shutdown in CPU unplug and not fiddle
with it
until it's restarted by the core when the CPU is plugged in again.
"But no drivers or blk-mq do that before one hctx becomes inactive(all CPUs for one hctx are offline), and even it is worse, blk-mq stills tries to run hw queue after hctx is dead, see
blk_mq_hctx_notify_dead().This patchset tries to address the issue by two stages:
1) add one new cpuhp state of
CPUHP_AP_BLK_MQ_ONLINE- mark the hctx as internal stopped, and drain all in-flight requests if the hctx is going to be dead.
2) re-submit IO in the state of
CPUHP_BLK_MQ_DEADafter the hctx becomes dead- steal bios from the request, and resubmit them via
generic_make_request(),
then these IO will be mapped to other live hctx for dispatchPlease comment & review, thanks!
V5:
- renameBLK_MQ_S_INTERNAL_STOPPEDasBLK_MQ_S_INACTIVE
- re-factor code for re-submit requests in cpu dead hotplug handler
- address requeue corner caseV4:
- resubmit IOs in dispatch list in case that this hctx is deadV3:
- re-organize patch 2 & 3 a bit for addressing Hannes's comment
- fix patch 4 for avoiding potential deadlock, as found by HannesQuoted 6 lines
Quoted 6 lines
V2:
- patch4 & patch 5 in V1 have been merged to block tree, so remove them
- address comments from John Garry and MinwooHi Ming,
Colleague Xiang Chen found this:
Starting 40 processes
Jobs: 40 (f=40)
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [8.3% done] [2899MB/0KB/0KB /s] [742K/0/0 iops] [eta 00m:33s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [11.1% done] [2892MB/0KB/0KB /s] [740K/0/0 iops] [eta 00m:32s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [13.9% done] [2887MB/0KB/0KB /s] [739K/0/0 iops] [eta 00m:31s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [16.7% done] [2850MB/0KB/0KB /s] [730K/0/0 iops] [eta 00m:30s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [19.4% done] [2834MB/0KB/0KB /s] [726K/0/0 iops] [eta 00m:29s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [22.2% done] [2827MB/0KB/0KB /s] [724K/0/0 iops] [eta 00m:28s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [25.0% done] [2804MB/0KB/0KB /s] [718K/0/0 iops] [eta 00m:27s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [27.8% done] [2783MB/0KB/0KB /s] [713K/0/0 iops] [eta 00m:26s][ 112.990292] CPU16: shutdown [ 112.993113] psci: CPU16 killed (polled 0 ms) [ 113.029429] CPU15: shutdown [ 113.032245] psci: CPU15 killed (polled 0 ms) [ 113.073649] CPU14: shutdown [ 113.076461] psci: CPU14 killed (polled 0 ms)Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRR[ 113.121258] CPU13:
shutdown
RRRRRRRRRR] [30.[ 113.124247] psci: CPU13 killed (polled 0 ms) 6% done] [2772MB/0KB/0KB /s] [710K/0/0 iops] [eta 00m:25s][ 113.177108] IRQ 600: no longer affine to CPU12 [ 113.183185] CPU12: shutdown [ 113.186001] psci: CPU12 killed (polled 0 ms) [ 113.242555] CPU11: shutdown [ 113.245368] psci: CPU11 killed (polled 0 ms) [ 113.302598] CPU10: shutdown [ 113.305420] psci: CPU10 killed (polled 0 ms) [ 113.365757] CPU9: shutdown [ 113.368489] psci: CPU9 killed (polled 0 ms) [ 113.611904] IRQ 599: no longer affine to CPU8 [ 113.620611] CPU8: shutdown [ 113.623340] psci: CPU8 killed (polled 0 ms)Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [33.3% done] [2765MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:24s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [36.1% done] [2767MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:23s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [38.9% done] [2771MB/0KB/0KB /s] [709K/0/0 iops] [eta 00m:22s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [41.7% done] [2780MB/0KB/0KB /s] [712K/0/0 iops] [eta 00m:21s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [44.4% done] [2731MB/0KB/0KB /s] [699K/0/0 iops] [eta 00m:20s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [47.2% done] [2757MB/0KB/0KB /s] [706K/0/0 iops] [eta 00m:19s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [50.0% done] [2764MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:18s][ 120.762141] ------------[ cut here ]------------ [ 120.766760] WARNING: CPU: 0 PID: 10 at block/blk.h:299 generic_make_request_checks+0x480/0x4e8 [ 120.775348] Modules linked in: [ 120.778397] CPU: 0 PID: 10 Comm: ksoftirqd/0 Not tainted 5.5.0-Then this:
1158048319d:18h:40m:34s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049605d:23h:03m:30s][ 141.915198] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 141.921280] rcu: 96-...0: (4 GPs behind) idle=af6/0/0x1 softirq=90/90 fqs=2626 [ 141.928658] rcu: 99-...0: (2 GPs behind) idle=b82/1/0x4000000000000000 softirq=82/82 fqs=2627 [ 141.937339] (detected by 17, t=5254 jiffies, g=3925, q=545) [ 141.942985] Task dump for CPU 96: [ 141.946292] swapper/96 R running task 0 0 1 0x0000002a [ 141.953321] Call trace: [ 141.955763] __switch_to+0xbc/0x218 [ 141.959244] 0x0 [ 141.961079] Task dump for CPU 99: [ 141.964385] kworker/99:1H R running task 0 3473 2
0x0000022a
Quoted 8 lines
[ 141.971417] Workqueue: kblockd blk_mq_run_work_fn [ 141.976109] Call trace: [ 141.978550] __switch_to+0xbc/0x218 [ 141.982029] blk_mq_run_work_fn+0x1c/0x28 [ 141.986027] process_one_work+0x1e0/0x358 [ 141.990025] worker_thread+0x40/0x488 [ 141.993678] kthread+0x118/0x120 [ 141.996897] ret_from_fork+0x10/0x18
Hi John,
Thanks for your test!
Could you test the following patchset and only the last one is changed?
https://github.com/ming1/linux/commits/my_for_5.6_block
Thanks,
Ming Lei
Quoted 8 lines
Quoted 7 lines
[ 141.976109] Call trace: [ 141.978550] __switch_to+0xbc/0x218 [ 141.982029] blk_mq_run_work_fn+0x1c/0x28 [ 141.986027] process_one_work+0x1e0/0x358 [ 141.990025] worker_thread+0x40/0x488 [ 141.993678] kthread+0x118/0x120 [ 141.996897] ret_from_fork+0x10/0x18
Quoted 4 lines
Hi John,
Thanks for your test!
Hi Ming,
Quoted 3 lines
Could you test the following patchset and only the last one is changed?
For SCSI testing, I will ask my colleague Xiang Chen to test when he returns to work. So I did not see this issue for my SCSI testing for your original v5, but I was only using 1x as opposed to maybe 20x SAS disks.
BTW, did you test NVMe? For some reason I could not trigger a scenario where we're draining the outstanding requests for a queue which is being deactivated - I mean, the queues were always already quiesced.
Thanks,
John
Quoted 20 lines · John Garry
Quoted 9 lines
Quoted 8 lines
[ 141.976109] Call trace: [ 141.978550] __switch_to+0xbc/0x218 [ 141.982029] blk_mq_run_work_fn+0x1c/0x28 [ 141.986027] process_one_work+0x1e0/0x358 [ 141.990025] worker_thread+0x40/0x488 [ 141.993678] kthread+0x118/0x120 [ 141.996897] ret_from_fork+0x10/0x18Quoted 4 lines
Hi John,
Thanks for your test!
Hi Ming,
Quoted 3 lines
Could you test the following patchset and only the last one is changed?
Quoted 7 lines
For SCSI testing, I will ask my colleague Xiang Chen to test when he returns to work. So I did not see this issue for my SCSI testing for your original v5, but I was only using 1x as opposed to maybe 20x SAS disks.
BTW, did you test NVMe? For some reason I could not trigger a scenario where we're draining the outstanding requests for a queue which is being deactivated - I mean, the queues were always already quiesced.
I run cpu hotplug test on both NVMe and SCSI in KVM, and fio just runs as expected.
NVMe is often 1:1 mapping, so it might be a bit difficult to trigger draining in-flight IOs.
Thanks,
Ming Lei
+- Keith
Quoted 29 lines · Ming Lei
Quoted 20 lines · John Garry
Quoted 9 lines
Quoted 8 lines
[ 141.976109] Call trace: [ 141.978550] __switch_to+0xbc/0x218 [ 141.982029] blk_mq_run_work_fn+0x1c/0x28 [ 141.986027] process_one_work+0x1e0/0x358 [ 141.990025] worker_thread+0x40/0x488 [ 141.993678] kthread+0x118/0x120 [ 141.996897] ret_from_fork+0x10/0x18Quoted 4 lines
Hi John,
Thanks for your test!
Hi Ming,
Quoted 3 lines
Could you test the following patchset and only the last one is changed?
Quoted 7 lines
For SCSI testing, I will ask my colleague Xiang Chen to test when he returns to work. So I did not see this issue for my SCSI testing for your original v5, but I was only using 1x as opposed to maybe 20x SAS disks.
BTW, did you test NVMe? For some reason I could not trigger a scenario where we're draining the outstanding requests for a queue which is being deactivated - I mean, the queues were always already quiesced.
Quoted 5 lines
I run cpu hotplug test on both NVMe and SCSI in KVM, and fio just runs as expected.
NVMe is often 1:1 mapping, so it might be a bit difficult to trigger draining in-flight IOs.
I generally have a 4:1 mapping for my NVMe cards - so many CPUs.
But more alarming is that I see this on every run:
[ 246.378744] psci: CPU6 killed (polled 0 ms) [ 246.427955] irq_migrate_all_off_this_cpu: 3 callbacks suppressed [ 246.427956] IRQ 822: no longer affine to CPU7 [ 246.438586] CPU7: shutdown [ 246.441300] psci: CPU7 killed (polled 0 ms) [ 247.015206] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible) Jobs: 6[ 267.503159] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 267.509244] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=2626 [ 267.518283] (detected by 0, t=5255 jiffies, g=14001, q=1787) [ 267.524016] Task dump for CPU 25: [ 267.527318] irqbalance R running task 0 1275 1 0x00000002 [ 267.534354] Call trace: [ 267.536790] __switch_to+0xbc/0x218 [ 267.540266] 0xffff800014c53d68 Jobs: 6 (f=6): [R(6)][96.0%][r=0KiB/s[ 278.195193] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff ,w=0KiB/s][r=0,w=0 IOPS][eta 00m:[ 278.205197] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff Jobs: 6 (f=6): [R(6)][0.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IO[ 330.523157] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 330.529242] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=10503 [ 330.538366] (detected by 0, t=21010 jiffies, g=14001, q=2318) [ 330.544186] Task dump for CPU 25: [ 330.547488] irqbalance R running task 0 1275 1 0x00000002 [ 330.554523] Call trace: [ 330.556958] __switch_to+0xbc/0x218 [ 330.560434] 0xffff800014c53d68 [ 393.543155] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 393.549242] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=18379 [ 393.558366] (detected by 8, t=36765 jiffies, g=14001, q=2385) [ 393.564187] Task dump for CPU 25: [ 393.567488] irqbalance R running task 0 1275 1 0x00000002 [ 393.574523] Call trace: [ 393.576960] __switch_to+0xbc/0x218 [ 393.580435] 0xffff800014c53d68 [ 456.567156] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 456.573246] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=26255 [ 456.582389] (detected by 90, t=52520 jiffies, g=14001, q=2397) [ 456.588296] Task dump for CPU 25: [ 456.591601] irqbalance R running task 0 1275 1 0x00000002 [ 456.598638] Call trace: [ 456.601078] __switch_to+0xbc/0x218 [ 456.604556] 0xffff800014c53d68 [ 519.587152] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 519.593235] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=34131 [ 519.602357] (detected by 90, t=68275 jiffies, g=14001, q=2405) [ 519.608264] Task dump for CPU 25: [ 519.611566] irqbalance R running task 0 1275 1 0x00000002 [ 519.618601] Call trace: [ 519.621036] __switch_to+0xbc/0x218 [ 519.624511] 0xffff800014c53d68 [ 582.603153] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 582.609236] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=42006 [ 582.618358] (detected by 90, t=84029 jiffies, g=14001, q=2409) [ 582.624265] Task dump for CPU 25: [ 582.627567] irqbalance R running task 0 1275 1 0x00000002 [ 582.634601] Call trace: [ 582.637036] __switch_to+0xbc/0x218 [ 582.640511] 0xffff800014c53d68
I tested v5.5 and I cannot see this, but your branch is based in 5.5-rc2 with lots of other stuff on top. The only thing I changed was nvme.use_threaded_interrupts = 1.
Note that I do see this with v5.5 for my hotplug test, as expected:
[ 280.221917] nvme nvme1: I/O 114 QID 19 timeout, completion polledta 01m:50s]
...so I'm not going insane.
Thanks,
John
Quoted 1 line · John Garry
+- Keith
+ Marc, on rare chance that this is related to the GIC
Quoted 99 lines · Ming Lei
Quoted 29 lines · John Garry
Quoted 20 lines
Quoted 9 lines
Quoted 8 lines
[ 141.976109] Call trace:
[ 141.978550]__switch_to+0xbc/0x218
[ 141.982029]blk_mq_run_work_fn+0x1c/0x28
[ 141.986027]process_one_work+0x1e0/0x358
[ 141.990025]worker_thread+0x40/0x488
[ 141.993678]kthread+0x118/0x120
[ 141.996897]ret_from_fork+0x10/0x18Quoted 4 lines
Hi John,
Thanks for your test!
Hi Ming,
Quoted 3 lines
Could you test the following patchset and only the last one is changed?
Quoted 7 lines
For SCSI testing, I will ask my colleague Xiang Chen to test when he returns to work. So I did not see this issue for my SCSI testing for your original v5, but I was only using 1x as opposed to maybe 20x SAS disks.
BTW, did you test NVMe? For some reason I could not trigger a scenario where we're draining the outstanding requests for a queue which is being deactivated - I mean, the queues were always already quiesced.
Quoted 5 lines
I run cpu hotplug test on both NVMe and SCSI in KVM, and fio just runs as expected.
NVMe is often 1:1 mapping, so it might be a bit difficult to trigger draining in-flight IOs.
I generally have a 4:1 mapping for my NVMe cards - so many CPUs.
But more alarming is that I see this on every run:
[ 246.378744] psci: CPU6 killed (polled 0 ms)
[ 246.427955]irq_migrate_all_off_this_cpu: 3 callbacks suppressed
[ 246.427956] IRQ 822: no longer affine to CPU7
[ 246.438586] CPU7: shutdown
[ 246.441300] psci: CPU7 killed (polled 0 ms)
[ 247.015206] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible)
Jobs: 6[ 267.503159] rcu: INFO:rcu_preemptdetected stalls on CPUs/tasks:
[ 267.509244] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=2626
[ 267.518283] (detected by 0, t=5255 jiffies, g=14001, q=1787)
[ 267.524016] Task dump for CPU 25:
[ 267.527318] irqbalance R running task 0 1275 10x00000002
[ 267.534354] Call trace:
[ 267.536790]__switch_to+0xbc/0x218
[ 267.540266]0xffff800014c53d68
Jobs: 6 (f=6): [R(6)][96.0%][r=0KiB/s[ 278.195193] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff ,w=0KiB/s][r=0,w=0 IOPS][eta 00m:[ 278.205197] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Jobs: 6 (f=6): [R(6)][0.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IO[ 330.523157] rcu: INFO:rcu_preemptdetected stalls on CPUs/tasks:
[ 330.529242] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=10503
[ 330.538366] (detected by 0, t=21010 jiffies, g=14001, q=2318)
[ 330.544186] Task dump for CPU 25:
[ 330.547488] irqbalance R running task 0 1275 10x00000002
[ 330.554523] Call trace:
[ 330.556958]__switch_to+0xbc/0x218
[ 330.560434]0xffff800014c53d68
[ 393.543155] rcu: INFO:rcu_preemptdetected stalls on CPUs/tasks:
[ 393.549242] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=18379
[ 393.558366] (detected by 8, t=36765 jiffies, g=14001, q=2385)
[ 393.564187] Task dump for CPU 25:
[ 393.567488] irqbalance R running task 0 1275 10x00000002
[ 393.574523] Call trace:
[ 393.576960]__switch_to+0xbc/0x218
[ 393.580435]0xffff800014c53d68
[ 456.567156] rcu: INFO:rcu_preemptdetected stalls on CPUs/tasks:
[ 456.573246] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=26255
[ 456.582389] (detected by 90, t=52520 jiffies, g=14001, q=2397)
[ 456.588296] Task dump for CPU 25:
[ 456.591601] irqbalance R running task 0 1275 10x00000002
[ 456.598638] Call trace:
[ 456.601078]__switch_to+0xbc/0x218
[ 456.604556]0xffff800014c53d68
[ 519.587152] rcu: INFO:rcu_preemptdetected stalls on CPUs/tasks:
[ 519.593235] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=34131
[ 519.602357] (detected by 90, t=68275 jiffies, g=14001, q=2405)
[ 519.608264] Task dump for CPU 25:
[ 519.611566] irqbalance R running task 0 1275 10x00000002
[ 519.618601] Call trace:
[ 519.621036]__switch_to+0xbc/0x218
[ 519.624511]0xffff800014c53d68
[ 582.603153] rcu: INFO:rcu_preemptdetected stalls on CPUs/tasks:
[ 582.609236] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=42006
[ 582.618358] (detected by 90, t=84029 jiffies, g=14001, q=2409)
[ 582.624265] Task dump for CPU 25:
[ 582.627567] irqbalance R running task 0 1275 10x00000002
[ 582.634601] Call trace:
[ 582.637036]__switch_to+0xbc/0x218
[ 582.640511]0xffff800014c53d68
Quoted 1 line
I tested v5.5 and I cannot see this, but your branch is based in 5.5-rc2 with lots of other stuff on top. The only thing I changed was nvme.use_threaded_interrupts = 1.
Uhh, now I do with v5.5:
[ 705.354601] CPU22: shutdown [ 705.357395] psci: CPU22 killed (polled 0 ms) [ 705.402837] CPU23: shutdown [ 705.405630] psci: CPU23 killed (polled 0 ms) [ 705.666716] CPU24: shutdown [ 705.669512] psci: CPU24 killed (polled 0 ms) [ 705.966753] CPU25: shutdown [ 705.969548] psci: CPU25 killed (polled 0 ms) [ 706.250771] CPU26: shutdown=2347MiB/s,w=0KiB/s][r=601k,w=0 IOPS][eta 00m:13s] [ 706.253565] psci: CPU26 killed (polled 0 ms) [ 706.514728] CPU27: shutdown [ 706.517523] psci: CPU27 killed (polled 0 ms) [ 706.826708] CPU28: shutdown [ 706.829502] psci: CPU28 killed (polled 0 ms) [ 707.130916] CPU29: shutdown=2134MiB/s,w=0KiB/s][r=546k,w=0 IOPS][eta 00m:12s] [ 707.133714] psci: CPU29 killed (polled 0 ms) [ 707.439066] CPU30: shutdown [ 707.441870] psci: CPU30 killed (polled 0 ms) [ 707.706727] CPU31: shutdown [ 707.709526] psci: CPU31 killed (polled 0 ms) [ 708.521853] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible) [ 728.741808] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:80d:00h:35m:42s] [ 728.747895] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=2626 [ 728.757197] (detected by 63, t=5255 jiffies, g=40989, q=1890) [ 728.763018] Task dump for CPU 48: [ 728.766321] irqbalance R running task 0 1272 1 0x00000002 [ 728.773358] Call trace: [ 728.775801] __switch_to+0xbc/0x218 [ 728.779283] gic_set_affinity+0x16c/0x1d8 [ 728.783282] irq_do_set_affinity+0x30/0xd0 [ 728.787365] irq_set_affinity_locked+0xc8/0xf0 [ 728.791796] __irq_set_affinity+0x4c/0x80 [ 728.795794] write_irq_affinity.isra.7+0x104/0x120 [ 728.800572] irq_affinity_proc_write+0x1c/0x28 [ 728.805008] proc_reg_write+0x78/0xb8 [ 728.808660] __vfs_write+0x18/0x38 [ 728.812050] vfs_write+0xb4/0x1e0 [ 728.815352] ksys_write+0x68/0xf8 [ 728.818655] __arm64_sys_write+0x18/0x20 [ 728.822567] el0_svc_common.constprop.2+0x64/0x160 [ 728.827345] el0_svc_handler+0x20/0x80 [ 728.831082] el0_sync_handler+0xe4/0x188 [ 728.834991] el0_sync+0x140/0x180 [ 738.993844] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 791.761810] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:63d:14h:24m:13s] [ 791.767897] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=10495 [ 791.777281] (detected by 54, t=21010 jiffies, g=40989, q=2396) [ 791.783189] Task dump for CPU 48: [ 791.786491] irqbalance R running task 0 1272 1 0x00000002 [ 791.793528] Call trace: [ 791.795964] __switch_to+0xbc/0x218 [ 791.799441] gic_set_affinity+0x16c/0x1d8 [ 791.803439] irq_do_set_affinity+0x30/0xd0 [ 791.807522] irq_set_affinity_locked+0xc8/0xf0 [ 791.811953] __irq_set_affinity+0x4c/0x80 [ 791.815949] write_irq_affinity.isra.7+0x104/0x120 [ 791.820727] irq_affinity_proc_write+0x1c/0x28 [ 791.825158] proc_reg_write+0x78/0xb8 [ 791.828808] __vfs_write+0x18/0x38 [ 791.832197] vfs_write+0xb4/0x1e0 [ 791.835500] ksys_write+0x68/0xf8 [ 791.838802] __arm64_sys_write+0x18/0x20 [ 791.842712] el0_svc_common.constprop.2+0x64/0x160 [ 791.847490] el0_svc_handler+0x20/0x80 [ 791.851226] el0_sync_handler+0xe4/0x188 [ 791.855135] el0_sync+0x140/0x180
Jobs: 6 (f=6): [R(6)][0.0%][r=0KiB/s
Quoted 6 lines
Note that I do see this with v5.5 for my hotplug test, as expected:
[ 280.221917] nvme nvme1: I/O 114 QID 19 timeout, completion polledta 01m:50s]...so I'm not going insane.
Thanks,John
Quoted 155 lines · John Garry
Quoted 1 line · John Garry
+- Keith
+ Marc, on rare chance that this is related to the GIC
Quoted 116 lines · Ming Lei
Quoted 30 lines · John Garry
Quoted 20 lines
Quoted 9 lines
Quoted 8 lines
[ 141.976109] Call trace: [ 141.978550] __switch_to+0xbc/0x218 [ 141.982029] blk_mq_run_work_fn+0x1c/0x28 [ 141.986027] process_one_work+0x1e0/0x358 [ 141.990025] worker_thread+0x40/0x488 [ 141.993678] kthread+0x118/0x120 [ 141.996897] ret_from_fork+0x10/0x18Quoted 4 lines
Hi John,
Thanks for your test!
Hi Ming,
Quoted 3 lines
Could you test the following patchset and only the last one is changed?
Quoted 8 lines
For SCSI testing, I will ask my colleague Xiang Chen to test when he returns to work. So I did not see this issue for my SCSI testing for your original v5, but I was only using 1x as opposed to maybe 20x SAS disks.
BTW, did you test NVMe? For some reason I could not trigger a scenario where we're draining the outstanding requests for a queue which is being deactivated - I mean, the queues were always already quiesced.
Quoted 5 lines
I run cpu hotplug test on both NVMe and SCSI in KVM, and fio just runs as expected.
NVMe is often 1:1 mapping, so it might be a bit difficult to trigger draining in-flight IOs.
I generally have a 4:1 mapping for my NVMe cards - so many CPUs.
But more alarming is that I see this on every run:
[ 246.378744] psci: CPU6 killed (polled 0 ms) [ 246.427955] irq_migrate_all_off_this_cpu: 3 callbacks suppressed [ 246.427956] IRQ 822: no longer affine to CPU7 [ 246.438586] CPU7: shutdown [ 246.441300] psci: CPU7 killed (polled 0 ms) [ 247.015206] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible) Jobs: 6[ 267.503159] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 267.509244] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=2626 [ 267.518283] (detected by 0, t=5255 jiffies, g=14001, q=1787) [ 267.524016] Task dump for CPU 25: [ 267.527318] irqbalance R running task 0 1275 1 0x00000002 [ 267.534354] Call trace: [ 267.536790] __switch_to+0xbc/0x218 [ 267.540266] 0xffff800014c53d68Jobs: 6 (f=6): [R(6)][96.0%][r=0KiB/s[ 278.195193] nvme nvme1:
controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff ,w=0KiB/s][r=0,w=0 IOPS][eta 00m:[ 278.205197] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Jobs: 6 (f=6): [R(6)][0.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IO[ 330.523157]rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 330.529242] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=10503 [ 330.538366] (detected by 0, t=21010 jiffies, g=14001, q=2318) [ 330.544186] Task dump for CPU 25: [ 330.547488] irqbalance R running task 0 1275 1 0x00000002 [ 330.554523] Call trace: [ 330.556958] __switch_to+0xbc/0x218 [ 330.560434] 0xffff800014c53d68 [ 393.543155] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 393.549242] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=18379 [ 393.558366] (detected by 8, t=36765 jiffies, g=14001, q=2385) [ 393.564187] Task dump for CPU 25: [ 393.567488] irqbalance R running task 0 1275 1 0x00000002 [ 393.574523] Call trace: [ 393.576960] __switch_to+0xbc/0x218 [ 393.580435] 0xffff800014c53d68 [ 456.567156] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 456.573246] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=26255 [ 456.582389] (detected by 90, t=52520 jiffies, g=14001, q=2397) [ 456.588296] Task dump for CPU 25: [ 456.591601] irqbalance R running task 0 1275 1 0x00000002 [ 456.598638] Call trace: [ 456.601078] __switch_to+0xbc/0x218 [ 456.604556] 0xffff800014c53d68 [ 519.587152] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 519.593235] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=34131 [ 519.602357] (detected by 90, t=68275 jiffies, g=14001, q=2405) [ 519.608264] Task dump for CPU 25: [ 519.611566] irqbalance R running task 0 1275 1 0x00000002 [ 519.618601] Call trace: [ 519.621036] __switch_to+0xbc/0x218 [ 519.624511] 0xffff800014c53d68 [ 582.603153] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 582.609236] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=42006 [ 582.618358] (detected by 90, t=84029 jiffies, g=14001, q=2409) [ 582.624265] Task dump for CPU 25: [ 582.627567] irqbalance R running task 0 1275 1 0x00000002 [ 582.634601] Call trace: [ 582.637036] __switch_to+0xbc/0x218 [ 582.640511] 0xffff800014c53d68Quoted 3 lines
I tested v5.5 and I cannot see this, but your branch is based in 5.5-rc2 with lots of other stuff on top. The only thing I changed was nvme.use_threaded_interrupts = 1.
Uhh, now I do with v5.5:
[ 705.354601] CPU22: shutdown [ 705.357395] psci: CPU22 killed (polled 0 ms) [ 705.402837] CPU23: shutdown [ 705.405630] psci: CPU23 killed (polled 0 ms) [ 705.666716] CPU24: shutdown [ 705.669512] psci: CPU24 killed (polled 0 ms) [ 705.966753] CPU25: shutdown [ 705.969548] psci: CPU25 killed (polled 0 ms) [ 706.250771] CPU26: shutdown=2347MiB/s,w=0KiB/s][r=601k,w=0 IOPS][eta 00m:13s] [ 706.253565] psci: CPU26 killed (polled 0 ms) [ 706.514728] CPU27: shutdown [ 706.517523] psci: CPU27 killed (polled 0 ms) [ 706.826708] CPU28: shutdown [ 706.829502] psci: CPU28 killed (polled 0 ms) [ 707.130916] CPU29: shutdown=2134MiB/s,w=0KiB/s][r=546k,w=0 IOPS][eta 00m:12s] [ 707.133714] psci: CPU29 killed (polled 0 ms) [ 707.439066] CPU30: shutdown [ 707.441870] psci: CPU30 killed (polled 0 ms) [ 707.706727] CPU31: shutdown [ 707.709526] psci: CPU31 killed (polled 0 ms) [ 708.521853] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible) [ 728.741808] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:80d:00h:35m:42s] [ 728.747895] rcu: 48-...0: (0 ticks this GP)
Quoted 8 lines
idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=2626
[ 728.757197] (detected by 63, t=5255 jiffies, g=40989, q=1890) [ 728.763018] Task dump for CPU 48: [ 728.766321] irqbalance R running task 0 1272 1 0x00000002 [ 728.773358] Call trace: [ 728.775801] __switch_to+0xbc/0x218 [ 728.779283] gic_set_affinity+0x16c/0x1d8
...
gic_set_affinity shouldn't have switched out, so looks like one gic issue.
BTW, you may try to run the test by disabling irqbalance.
Thanks,
Ming Lei
Quoted 164 lines · On Sat, 1 Feb 2020 09:31:17 +0800 Ming Lei <[email protected]> wrote:
Quoted 155 lines · John Garry
Quoted 1 line · John Garry
+- Keith
+ Marc, on rare chance that this is related to the GIC
Quoted 116 lines · Ming Lei
Quoted 30 lines · John Garry
Quoted 20 lines
Quoted 9 lines
Quoted 8 lines
[ 141.976109] Call trace: [ 141.978550] __switch_to+0xbc/0x218 [ 141.982029] blk_mq_run_work_fn+0x1c/0x28 [ 141.986027] process_one_work+0x1e0/0x358 [ 141.990025] worker_thread+0x40/0x488 [ 141.993678] kthread+0x118/0x120 [ 141.996897] ret_from_fork+0x10/0x18Quoted 4 lines
Hi John,
Thanks for your test!
Hi Ming,
Quoted 3 lines
Could you test the following patchset and only the last one is changed?
Quoted 8 lines
For SCSI testing, I will ask my colleague Xiang Chen to test when he returns to work. So I did not see this issue for my SCSI testing for your original v5, but I was only using 1x as opposed to maybe 20x SAS disks.
BTW, did you test NVMe? For some reason I could not trigger a scenario where we're draining the outstanding requests for a queue which is being deactivated - I mean, the queues were always already quiesced.
Quoted 5 lines
I run cpu hotplug test on both NVMe and SCSI in KVM, and fio just runs as expected.
NVMe is often 1:1 mapping, so it might be a bit difficult to trigger draining in-flight IOs.
I generally have a 4:1 mapping for my NVMe cards - so many CPUs.
But more alarming is that I see this on every run:
[ 246.378744] psci: CPU6 killed (polled 0 ms) [ 246.427955] irq_migrate_all_off_this_cpu: 3 callbacks suppressed [ 246.427956] IRQ 822: no longer affine to CPU7 [ 246.438586] CPU7: shutdown [ 246.441300] psci: CPU7 killed (polled 0 ms) [ 247.015206] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible) Jobs: 6[ 267.503159] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 267.509244] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=2626 [ 267.518283] (detected by 0, t=5255 jiffies, g=14001, q=1787) [ 267.524016] Task dump for CPU 25: [ 267.527318] irqbalance R running task 0 1275 1 0x00000002 [ 267.534354] Call trace: [ 267.536790] __switch_to+0xbc/0x218 [ 267.540266] 0xffff800014c53d68Jobs: 6 (f=6): [R(6)][96.0%][r=0KiB/s[ 278.195193] nvme nvme1:
controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff ,w=0KiB/s][r=0,w=0 IOPS][eta 00m:[ 278.205197] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Jobs: 6 (f=6): [R(6)][0.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IO[ 330.523157]rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 330.529242] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=10503 [ 330.538366] (detected by 0, t=21010 jiffies, g=14001, q=2318) [ 330.544186] Task dump for CPU 25: [ 330.547488] irqbalance R running task 0 1275 1 0x00000002 [ 330.554523] Call trace: [ 330.556958] __switch_to+0xbc/0x218 [ 330.560434] 0xffff800014c53d68 [ 393.543155] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 393.549242] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=18379 [ 393.558366] (detected by 8, t=36765 jiffies, g=14001, q=2385) [ 393.564187] Task dump for CPU 25: [ 393.567488] irqbalance R running task 0 1275 1 0x00000002 [ 393.574523] Call trace: [ 393.576960] __switch_to+0xbc/0x218 [ 393.580435] 0xffff800014c53d68 [ 456.567156] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 456.573246] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=26255 [ 456.582389] (detected by 90, t=52520 jiffies, g=14001, q=2397) [ 456.588296] Task dump for CPU 25: [ 456.591601] irqbalance R running task 0 1275 1 0x00000002 [ 456.598638] Call trace: [ 456.601078] __switch_to+0xbc/0x218 [ 456.604556] 0xffff800014c53d68 [ 519.587152] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 519.593235] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=34131 [ 519.602357] (detected by 90, t=68275 jiffies, g=14001, q=2405) [ 519.608264] Task dump for CPU 25: [ 519.611566] irqbalance R running task 0 1275 1 0x00000002 [ 519.618601] Call trace: [ 519.621036] __switch_to+0xbc/0x218 [ 519.624511] 0xffff800014c53d68 [ 582.603153] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 582.609236] rcu: 25-...0: (1 GPs behind) idle=11e/1/0x4000000000000000 softirq=1175/1177 fqs=42006 [ 582.618358] (detected by 90, t=84029 jiffies, g=14001, q=2409) [ 582.624265] Task dump for CPU 25: [ 582.627567] irqbalance R running task 0 1275 1 0x00000002 [ 582.634601] Call trace: [ 582.637036] __switch_to+0xbc/0x218 [ 582.640511] 0xffff800014c53d68Quoted 3 lines
I tested v5.5 and I cannot see this, but your branch is based in 5.5-rc2 with lots of other stuff on top. The only thing I changed was nvme.use_threaded_interrupts = 1.
Uhh, now I do with v5.5:
[ 705.354601] CPU22: shutdown [ 705.357395] psci: CPU22 killed (polled 0 ms) [ 705.402837] CPU23: shutdown [ 705.405630] psci: CPU23 killed (polled 0 ms) [ 705.666716] CPU24: shutdown [ 705.669512] psci: CPU24 killed (polled 0 ms) [ 705.966753] CPU25: shutdown [ 705.969548] psci: CPU25 killed (polled 0 ms) [ 706.250771] CPU26: shutdown=2347MiB/s,w=0KiB/s][r=601k,w=0 IOPS][eta 00m:13s] [ 706.253565] psci: CPU26 killed (polled 0 ms) [ 706.514728] CPU27: shutdown [ 706.517523] psci: CPU27 killed (polled 0 ms) [ 706.826708] CPU28: shutdown [ 706.829502] psci: CPU28 killed (polled 0 ms) [ 707.130916] CPU29: shutdown=2134MiB/s,w=0KiB/s][r=546k,w=0 IOPS][eta 00m:12s] [ 707.133714] psci: CPU29 killed (polled 0 ms) [ 707.439066] CPU30: shutdown [ 707.441870] psci: CPU30 killed (polled 0 ms) [ 707.706727] CPU31: shutdown [ 707.709526] psci: CPU31 killed (polled 0 ms) [ 708.521853] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible) [ 728.741808] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:80d:00h:35m:42s] [ 728.747895] rcu: 48-...0: (0 ticks this GP)Quoted 8 lines
idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=2626
[ 728.757197] (detected by 63, t=5255 jiffies, g=40989, q=1890) [ 728.763018] Task dump for CPU 48: [ 728.766321] irqbalance R running task 0 1272 1 0x00000002 [ 728.773358] Call trace: [ 728.775801] __switch_to+0xbc/0x218 [ 728.779283] gic_set_affinity+0x16c/0x1d8
Quoted 4 lines
...
gic_set_affinityshouldn't have switched out, so looks like one gic issue.
Given that gic_set_affinity doesn't sleep, this looks pretty unlikely. And __irq_set_affinity() holds a spinlock with irq disabled, so I can't really explain how you'd get there. I've just booted a lockdep enabled v5.5 on my D05, moved SPIs around (because that's the only way to reach this code), and nothing caught fire.
Either the stack trace isn't reliable (when I read things like "80d:00h:35m:42s" in the trace, I'm a bit suspicious), or CPU hotplug is doing something really funky here.
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Quoted 13 lines · Marc Zyngier
Quoted 4 lines · On Sat, 1 Feb 2020 09:31:17 +0800 Ming Lei <[email protected]> wrote:
On Sat, Feb 1, 2020 at 2:02 AM John Garry <[email protected]> wrote:
gic_set_affinityshouldn't have switched out, so looks like one gic issue.Given that
gic_set_affinitydoesn't sleep, this looks pretty unlikely. And__irq_set_affinity()holds a spinlock with irq disabled, so I can't really explain how you'd get there. I've just booted a lockdep enabled v5.5 on my D05, moved SPIs around (because that's the only way to reach this code), and nothing caught fire.
Quoted 3 lines
Either the stack trace isn't reliable (when I read things like "80d:00h:35m:42s" in the trace, I'm a bit suspicious), or CPU hotplug is doing something really funky here.
The hotplug code cannot end up in schedule either and it holds desc lock as normal affinity setting. The other backtrace is more complete,
[ 728.741808] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:80d:00h:35m:42s] [ 728.747895] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=2626 [ 728.757197] (detected by 63, t=5255 jiffies, g=40989, q=1890) [ 728.763018] Task dump for CPU 48: [ 728.766321] irqbalance R running task 0 1272 1 0x00000002 [ 728.773358] Call trace: [ 728.775801] __switch_to+0xbc/0x218 [ 728.779283] gic_set_affinity+0x16c/0x1d8 [ 728.783282] irq_do_set_affinity+0x30/0xd0 [ 728.787365] irq_set_affinity_locked+0xc8/0xf0 [ 728.791796] __irq_set_affinity+0x4c/0x80 [ 728.795794] write_irq_affinity.isra.7+0x104/0x120 [ 728.800572] irq_affinity_proc_write+0x1c/0x28 [ 728.805008] proc_reg_write+0x78/0xb8 [ 728.808660] __vfs_write+0x18/0x38 [ 728.812050] vfs_write+0xb4/0x1e0 [ 728.815352] ksys_write+0x68/0xf8 [ 728.818655] __arm64_sys_write+0x18/0x20 [ 728.822567] el0_svc_common.constprop.2+0x64/0x160 [ 728.827345] el0_svc_handler+0x20/0x80 [ 728.831082] el0_sync_handler+0xe4/0x188 [ 728.834991] el0_sync+0x140/0x180
But the __switch_to() there definitely does not make any sense at all.
Thanks,
tglx
Quoted 44 lines · Thomas Gleixner
Quoted 13 lines · Marc Zyngier
Quoted 4 lines · On Sat, 1 Feb 2020 09:31:17 +0800 Ming Lei <[email protected]> wrote:
On Sat, Feb 1, 2020 at 2:02 AM John Garry <[email protected]> wrote:
gic_set_affinityshouldn't have switched out, so looks like one gic issue.Given that
gic_set_affinitydoesn't sleep, this looks pretty unlikely. And__irq_set_affinity()holds a spinlock with irq disabled, so I can't really explain how you'd get there. I've just booted a lockdep enabled v5.5 on my D05, moved SPIs around (because that's the only way to reach this code), and nothing caught fire.Quoted 3 lines
Either the stack trace isn't reliable (when I read things like "80d:00h:35m:42s" in the trace, I'm a bit suspicious), or CPU hotplug is doing something really funky here.
The hotplug code cannot end up in schedule either and it holds desc lock as normal affinity setting. The other backtrace is more complete,
[ 728.741808] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:80d:00h:35m:42s] [ 728.747895] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=2626 [ 728.757197] (detected by 63, t=5255 jiffies, g=40989, q=1890) [ 728.763018] Task dump for CPU 48: [ 728.766321] irqbalance R running task 0 1272 1 0x00000002 [ 728.773358] Call trace: [ 728.775801] __switch_to+0xbc/0x218 [ 728.779283] gic_set_affinity+0x16c/0x1d8 [ 728.783282] irq_do_set_affinity+0x30/0xd0 [ 728.787365] irq_set_affinity_locked+0xc8/0xf0 [ 728.791796] __irq_set_affinity+0x4c/0x80 [ 728.795794] write_irq_affinity.isra.7+0x104/0x120 [ 728.800572] irq_affinity_proc_write+0x1c/0x28 [ 728.805008] proc_reg_write+0x78/0xb8 [ 728.808660] __vfs_write+0x18/0x38 [ 728.812050] vfs_write+0xb4/0x1e0 [ 728.815352] ksys_write+0x68/0xf8 [ 728.818655] __arm64_sys_write+0x18/0x20 [ 728.822567] el0_svc_common.constprop.2+0x64/0x160 [ 728.827345] el0_svc_handler+0x20/0x80 [ 728.831082] el0_sync_handler+0xe4/0x188 [ 728.834991] el0_sync+0x140/0x180
Quoted 1 line
But the
__switch_to()there definitely does not make any sense at all.
I have put a full kernel log here for another run (hang snippet at end), including scripts:
This does not look good:
Jobs: 6 ([ 519.858094] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
And some NVMe error also coincides with the hang. Another run has this:
[ 247.015206] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible)
I did test v5.4 previously and did not see this, but that would have included the v4 patchset in the $subject. I'll test v5.4 without that patchset now.
Thanks,
John
[ 487.442928] CPU32: shutdown=2149MiB/s,w=0KiB/s][r=550k,w=0 IOPS][eta 00m:11s] [ 487.445722] psci: CPU32 killed (polled 0 ms) [ 487.762957] CPU33: shutdown [ 487.765752] psci: CPU33 killed (polled 0 ms) [ 488.054863] CPU34: shutdown [ 488.057659] psci: CPU34 killed (polled 0 ms) [ 488.342910] CPU35: shutdown=2137MiB/s,w=0KiB/s][r=547k,w=0 IOPS][eta 00m:10s] [ 488.345704] psci: CPU35 killed (polled 0 ms) Jobs: 6[ 509.614051] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 509.620140] rcu: 0-...0: (1 GPs behind) idle=896/1/0x4000000000000000 softirq=7411/7412 fqs=2626 [ 509.629087] rcu: 36-...!: (0 ticks this GP) idle=ad2/1/0x4000000000000000 softirq=2631/2631 fqs=2627 [ 509.638398] (detected by 74, t=5257 jiffies, g=24921, q=1785) [ 509.644218] Task dump for CPU 0: [ 509.647435] irqbalance R running task 0 1308 1 0x00000002 [ 509.654472] Call trace: [ 509.656913] __switch_to+0xbc/0x218 [ 509.660392] 0xffff800017043d68 [ 509.663521] Task dump for CPU 36: [ 509.666824] cpuhp/36 R running task 0 192 2 0x0000002a [ 509.673860] Call trace: [ 509.676295] __switch_to+0xbc/0x218 [ 509.679776] page_wait_table+0x1500/0x1800 Jobs: 6 ([ 519.858094] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff Jobs: 6 (f=6[ 572.634049] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:6s] [ 572.640136] rcu: 0-...0: (1 GPs behind) idle=896/1/0x4000000000000000 softirq=7411/7412 fqs=10499 [ 572.649171] rcu: 36-...!: (0 ticks this GP) idle=ad2/1/0x4000000000000000 softirq=2631/2631 fqs=10500 [ 572.658561] (detected by 51, t=21012 jiffies, g=24921, q=2275) [ 572.664469] Task dump for CPU 0: [ 572.667685] irqbalance R running task 0 1308 1 0x00000002 [ 572.674723] Call trace: [ 572.677164] __switch_to+0xbc/0x218 [ 572.680641] 0xffff800017043d68 [ 572.683772] Task dump for CPU 36: [ 572.687076] cpuhp/36 R running task 0 192 2 0x0000002a [ 572.694112] Call trace: [ 572.696547] __switch_to+0xbc/0x218 [ 572.700027] page_wait_table+0x1500/0x1800 [ 635.654053] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 635.660141] rcu: 0-...0: (1 GPs behind) idle=896/1/0x4000000000000000 softirq=7411/7412 fqs=18372 [ 635.669175] rcu: 36-...!: (0 ticks this GP) idle=ad2/1/0x4000000000000000 softirq=2631/2631 fqs=18373 [ 635.678576] (detected by 92, t=36768 jiffies, g=24921, q=2279) [ 635.684482] Task dump for CPU 0: [ 635.687699] irqbalance R running task 0 1308 1 0x00000002 [ 635.694735] Call trace: [ 635.697172] __switch_to+0xbc/0x218 [ 635.700648] 0xffff800017043d68 [ 635.703778] Task dump for CPU 36: [ 635.707081] cpuhp/36 R running task 0 192 2 0x0000002a [ 635.714117] Call trace: [ 635.716552] __switch_to+0xbc/0x218 [ 635.720030] page_wait_table+0x1500/0x1800 [ 698.674054] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 698.680141] rcu: 0-...0: (1 GPs behind) idle=896/1/0x4000000000000000 softirq=7411/7412 fqs=26242 [ 698.689174] rcu: 36-...!: (0 ticks this GP) idle=ad2/1/0x4000000000000000 softirq=2631/2631 fqs=26242 [ 698.698563] (detected by 81, t=52522 jiffies, g=24921, q=2281) [ 698.704469] Task dump for CPU 0: [ 698.707684] irqbalance R running task 0 1308 1 0x00000002 [ 698.714721] Call trace: [ 698.717158] __switch_to+0xbc/0x218 [ 698.720634] 0xffff800017043d68 [ 698.723764] Task dump for CPU 36: [ 698.727066] cpuhp/36 R running task 0 192 2 0x0000002a [ 698.734101] Call trace: [ 698.736536] __switch_to+0xbc/0x218 [ 698.740013] page_wait_table+0x1500/0x1800 [ 761.694053] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 761.700136] rcu: 0-...0: (1 GPs behind) idle=896/1/0x4000000000000000 softirq=7411/7412 fqs=34111 [ 761.709169] rcu: 36-...!: (0 ticks this GP) idle=ad2/1/0x4000000000000000 44] __switch_to+0xbc/0x218 [ 761.740620] 0xffff800017043d68 [ 761.743749] Task dump for CPU 36: [ 761.747051] cpuhp/36 R running task 0 192 2 0x0000002a [ 761.754087] Call trace: [ 761.756521] __switch_to+0xbc/0x218 [ 761.759998] page_wait_table+0x1500/0x1800
Quoted 53 lines · John Garry
Quoted 44 lines · Thomas Gleixner
Quoted 13 lines · Marc Zyngier
Quoted 4 lines · On Sat, 1 Feb 2020 09:31:17 +0800 Ming Lei <[email protected]> wrote:
On Sat, Feb 1, 2020 at 2:02 AM John Garry <[email protected]> wrote:
gic_set_affinityshouldn't have switched out, so looks like one gic issue.Given that
gic_set_affinitydoesn't sleep, this looks pretty unlikely. And__irq_set_affinity()holds a spinlock with irq disabled, so I can't really explain how you'd get there. I've just booted a lockdep enabled v5.5 on my D05, moved SPIs around (because that's the only way to reach this code), and nothing caught fire.Quoted 3 lines
Either the stack trace isn't reliable (when I read things like "80d:00h:35m:42s" in the trace, I'm a bit suspicious), or CPU hotplug is doing something really funky here.
The hotplug code cannot end up in schedule either and it holds desc lock as normal affinity setting. The other backtrace is more complete,
[ 728.741808] rcu: INFO:
rcu_preemptdetected stalls on CPUs/tasks:80d:00h:35m:42s]
[ 728.747895] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=2626 [ 728.757197] (detected by 63, t=5255 jiffies, g=40989, q=1890)
[ 728.763018] Task dump for CPU 48:
[ 728.766321] irqbalance R running task 0 1272 10x00000002
[ 728.773358] Call trace:
[ 728.775801]__switch_to+0xbc/0x218
[ 728.779283]gic_set_affinity+0x16c/0x1d8
[ 728.783282]irq_do_set_affinity+0x30/0xd0
[ 728.787365]irq_set_affinity_locked+0xc8/0xf0
[ 728.791796]__irq_set_affinity+0x4c/0x80
[ 728.795794] write_irq_affinity.isra.7+0x104/0x120
[ 728.800572]irq_affinity_proc_write+0x1c/0x28
[ 728.805008]proc_reg_write+0x78/0xb8
[ 728.808660]__vfs_write+0x18/0x38
[ 728.812050]vfs_write+0xb4/0x1e0
[ 728.815352]ksys_write+0x68/0xf8
[ 728.818655]__arm64_sys_write+0x18/0x20
[ 728.822567] el0_svc_common.constprop.2+0x64/0x160
[ 728.827345]el0_svc_handler+0x20/0x80
[ 728.831082]el0_sync_handler+0xe4/0x188
[ 728.834991]el0_sync+0x140/0x180Quoted 1 line
But the
__switch_to()there definitely does not make any sense at all.I have put a full kernel log here for another run (hang snippet at end), including scripts:
This does not look good:
Quoted 8 lines
Jobs: 6 ([ 519.858094] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
And some NVMe error also coincides with the hang. Another run has this:
[ 247.015206] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible)
I did test v5.4 previously and did not see this, but that would have included the v4 patchset in the $subject. I'll test v5.4 without that patchset now.
So v5.4 does have this issue also:
[ 705.669512] psci: CPU24 killed (polled 0 ms) [ 705.966753] CPU25: shutdown [ 705.969548] psci: CPU25 killed (polled 0 ms) [ 706.250771] CPU26: shutdown=2347MiB/s,w=0KiB/s][r=601k,w=0 IOPS][eta 00m:13s] [ 706.253565] psci: CPU26 killed (polled 0 ms) [ 706.514728] CPU27: shutdown [ 706.517523] psci: CPU27 killed (polled 0 ms) [ 706.826708] CPU28: shutdown [ 706.829502] psci: CPU28 killed (polled 0 ms) [ 707.130916] CPU29: shutdown=2134MiB/s,w=0KiB/s][r=546k,w=0 IOPS][eta 00m:12s] [ 707.133714] psci: CPU29 killed (polled 0 ms) [ 707.439066] CPU30: shutdown [ 707.441870] psci: CPU30 killed (polled 0 ms) [ 707.706727] CPU31: shutdown [ 707.709526] psci: CPU31 killed (polled 0 ms) [ 708.521853] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible) [ 728.741808] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:80d:00h:35m:42s] [ 728.747895] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=2626 [ 728.757197] (detected by 63, t=5255 jiffies, g=40989, q=1890) [ 728.763018] Task dump for CPU 48: [ 728.766321] irqbalance R running task 0 1272 1 0x00000002 [ 728.773358] Call trace: [ 728.775801] __switch_to+0xbc/0x218 [ 728.779283] gic_set_affinity+0x16c/0x1d8 [ 728.783282] irq_do_set_affinity+0x30/0xd0 [ 728.787365] irq_set_affinity_locked+0xc8/0xf0 [ 728.791796] __irq_set_affinity+0x4c/0x80 [ 728.795794] write_irq_affinity.isra.7+0x104/0x120 [ 728.800572] irq_affinity_proc_write+0x1c/0x28 [ 728.805008] proc_reg_write+0x78/0xb8 [ 728.808660] __vfs_write+0x18/0x38 [ 728.812050] vfs_write+0xb4/0x1e0 [ 728.815352] ksys_write+0x68/0xf8 [ 728.818655] __arm64_sys_write+0x18/0x20 [ 728.822567] el0_svc_common.constprop.2+0x64/0x160 [ 728.827345] el0_svc_handler+0x20/0x80 [ 728.831082] el0_sync_handler+0xe4/0x188 [ 728.834991] el0_sync+0x140/0x180 [ 738.993844] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 791.761810] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:63d:14h:24m:13s] [ 791.767897] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=10495 [ 791.777281] (detected by 54, t=21010 jiffies, g=40989, q=2396) [ 791.783189] Task dump for CPU 48: [ 791.786491] irqbalance R running task 0 1272 1 0x00000002 [ 791.793528] Call trace: [ 791.795964] __switch_to+0xbc/0x218 [ 791.799441] gic_set_affinity+0x16c/0x1d8 [ 791.803439] irq_do_set_affinity+0x30/0xd0 [ 791.807522] irq_set_affinity_locked+0xc8/0xf0 [ 791.811953] __irq_set_affinity+0x4c/0x80 [ 791.815949] write_irq_affinity.isra.7+0x104/0x120 [ 791.820727] irq_affinity_proc_write+0x1c/0x28 [ 791.825158] proc_reg_write+0x78/0xb8 [ 791.828808] __vfs_write+0x18/0x38 [ 791.832197] vfs_write+0xb4/0x1e0 [ 791.835500] ksys_write+0x68/0xf8 [ 791.838802] __arm64_sys_write+0x18/0x20 [ 791.842712] el0_svc_common.constprop.2+0x64/0x160 [ 791.847490] el0_svc_handler+0x20/0x80 [ 791.851226] el0_sync_handler+0xe4/0x188 [ 791.855135] el0_sync+0x140/0x180
Jobs: 6 (f=6): [R(6)][0.0%][r=0KiB/s
Thanks,
John
Quoted 65 lines · John Garry
Quoted 63 lines · John Garry
Quoted 48 lines · Thomas Gleixner
Quoted 14 lines · Marc Zyngier
Quoted 5 lines · On Sat, 1 Feb 2020 09:31:17 +0800 Ming Lei <[email protected]> wrote:
On Sat, Feb 1, 2020 at 2:02 AM John Garry <[email protected]> wrote:
gic_set_affinityshouldn't have switched out, so looks like one gic issue.Given that
gic_set_affinitydoesn't sleep, this looks pretty unlikely. And__irq_set_affinity()holds a spinlock with irq disabled, so I can't really explain how you'd get there. I've just booted a lockdep enabled v5.5 on my D05, moved SPIs around (because that's the only way to reach this code), and nothing caught fire.Quoted 3 lines
Either the stack trace isn't reliable (when I read things like "80d:00h:35m:42s" in the trace, I'm a bit suspicious), or CPU hotplug is doing something really funky here.
The hotplug code cannot end up in schedule either and it holds desc lock as normal affinity setting. The other backtrace is more complete,
[ 728.741808] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:80d:00h:35m:42s] [ 728.747895] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=2626 [ 728.757197] (detected by 63, t=5255 jiffies, g=40989, q=1890) [ 728.763018] Task dump for CPU 48: [ 728.766321] irqbalance R running task 0 1272 1 0x00000002 [ 728.773358] Call trace: [ 728.775801] __switch_to+0xbc/0x218 [ 728.779283] gic_set_affinity+0x16c/0x1d8 [ 728.783282] irq_do_set_affinity+0x30/0xd0 [ 728.787365] irq_set_affinity_locked+0xc8/0xf0 [ 728.791796] __irq_set_affinity+0x4c/0x80 [ 728.795794] write_irq_affinity.isra.7+0x104/0x120 [ 728.800572] irq_affinity_proc_write+0x1c/0x28 [ 728.805008] proc_reg_write+0x78/0xb8 [ 728.808660] __vfs_write+0x18/0x38 [ 728.812050] vfs_write+0xb4/0x1e0 [ 728.815352] ksys_write+0x68/0xf8 [ 728.818655] __arm64_sys_write+0x18/0x20 [ 728.822567] el0_svc_common.constprop.2+0x64/0x160 [ 728.827345] el0_svc_handler+0x20/0x80 [ 728.831082] el0_sync_handler+0xe4/0x188 [ 728.834991] el0_sync+0x140/0x180Quoted 1 line
But the
__switch_to()there definitely does not make any sense at all.I have put a full kernel log here for another run (hang snippet at end), including scripts:
This does not look good:
Jobs: 6 ([ 519.858094] nvme nvme1: controller is down; will reset:
CSTS=0xffffffff, PCI_STATUS=0xffffAnd some NVMe error also coincides with the hang. Another run has this:
Quoted 8 lines
Quoted 6 lines
[ 247.015206] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible)I did test v5.4 previously and did not see this, but that would have included the v4 patchset in the $subject. I'll test v5.4 without that patchset now.
So v5.4 does have this issue also:
v5.5?
Quoted 72 lines
[ 705.669512] psci: CPU24 killed (polled 0 ms) [ 705.966753] CPU25: shutdown [ 705.969548] psci: CPU25 killed (polled 0 ms) [ 706.250771] CPU26: shutdown=2347MiB/s,w=0KiB/s][r=601k,w=0 IOPS][eta 00m:13s] [ 706.253565] psci: CPU26 killed (polled 0 ms) [ 706.514728] CPU27: shutdown [ 706.517523] psci: CPU27 killed (polled 0 ms) [ 706.826708] CPU28: shutdown [ 706.829502] psci: CPU28 killed (polled 0 ms) [ 707.130916] CPU29: shutdown=2134MiB/s,w=0KiB/s][r=546k,w=0 IOPS][eta 00m:12s] [ 707.133714] psci: CPU29 killed (polled 0 ms) [ 707.439066] CPU30: shutdown [ 707.441870] psci: CPU30 killed (polled 0 ms) [ 707.706727] CPU31: shutdown [ 707.709526] psci: CPU31 killed (polled 0 ms) [ 708.521853] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible) [ 728.741808] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:80d:00h:35m:42s] [ 728.747895] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=2626 [ 728.757197] (detected by 63, t=5255 jiffies, g=40989, q=1890) [ 728.763018] Task dump for CPU 48: [ 728.766321] irqbalance R running task 0 1272 1 0x00000002 [ 728.773358] Call trace: [ 728.775801] __switch_to+0xbc/0x218 [ 728.779283] gic_set_affinity+0x16c/0x1d8 [ 728.783282] irq_do_set_affinity+0x30/0xd0 [ 728.787365] irq_set_affinity_locked+0xc8/0xf0 [ 728.791796] __irq_set_affinity+0x4c/0x80 [ 728.795794] write_irq_affinity.isra.7+0x104/0x120 [ 728.800572] irq_affinity_proc_write+0x1c/0x28 [ 728.805008] proc_reg_write+0x78/0xb8 [ 728.808660] __vfs_write+0x18/0x38 [ 728.812050] vfs_write+0xb4/0x1e0 [ 728.815352] ksys_write+0x68/0xf8 [ 728.818655] __arm64_sys_write+0x18/0x20 [ 728.822567] el0_svc_common.constprop.2+0x64/0x160 [ 728.827345] el0_svc_handler+0x20/0x80 [ 728.831082] el0_sync_handler+0xe4/0x188 [ 728.834991] el0_sync+0x140/0x180 [ 738.993844] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 791.761810] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:63d:14h:24m:13s] [ 791.767897] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=10495 [ 791.777281] (detected by 54, t=21010 jiffies, g=40989, q=2396) [ 791.783189] Task dump for CPU 48: [ 791.786491] irqbalance R running task 0 1272 1 0x00000002 [ 791.793528] Call trace: [ 791.795964] __switch_to+0xbc/0x218 [ 791.799441] gic_set_affinity+0x16c/0x1d8 [ 791.803439] irq_do_set_affinity+0x30/0xd0 [ 791.807522] irq_set_affinity_locked+0xc8/0xf0 [ 791.811953] __irq_set_affinity+0x4c/0x80 [ 791.815949] write_irq_affinity.isra.7+0x104/0x120 [ 791.820727] irq_affinity_proc_write+0x1c/0x28 [ 791.825158] proc_reg_write+0x78/0xb8 [ 791.828808] __vfs_write+0x18/0x38 [ 791.832197] vfs_write+0xb4/0x1e0 [ 791.835500] ksys_write+0x68/0xf8 [ 791.838802] __arm64_sys_write+0x18/0x20 [ 791.842712] el0_svc_common.constprop.2+0x64/0x160 [ 791.847490] el0_svc_handler+0x20/0x80 [ 791.851226] el0_sync_handler+0xe4/0x188 [ 791.855135] el0_sync+0x140/0x180
Quoted 1 line
Jobs: 6 (f=6): [R(6)][0.0%][r=0KiB/s
Can you trigger it after disabling irqbalance?
Thanks,
Ming Lei
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Jobs: 6 ([ 519.858094] nvme nvme1: controller is down; will reset:
CSTS=0xffffffff, PCI_STATUS=0xffffAnd some NVMe error also coincides with the hang. Another run has this:
[ 247.015206] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible)
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I did test v5.4 previously and did not see this, but that would have included the v4 patchset in the $subject. I'll test v5.4 without that patchset now.
So v5.4 does have this issue also:
v5.5?
I am saying that both v5.4 and v5.5 have the issue. Below is the kernel hang snippet for v5.4.
Quoted 75 lines
Quoted 73 lines
[ 705.669512] psci: CPU24 killed (polled 0 ms) [ 705.966753] CPU25: shutdown [ 705.969548] psci: CPU25 killed (polled 0 ms) [ 706.250771] CPU26: shutdown=2347MiB/s,w=0KiB/s][r=601k,w=0 IOPS][eta 00m:13s] [ 706.253565] psci: CPU26 killed (polled 0 ms) [ 706.514728] CPU27: shutdown [ 706.517523] psci: CPU27 killed (polled 0 ms) [ 706.826708] CPU28: shutdown [ 706.829502] psci: CPU28 killed (polled 0 ms) [ 707.130916] CPU29: shutdown=2134MiB/s,w=0KiB/s][r=546k,w=0 IOPS][eta 00m:12s] [ 707.133714] psci: CPU29 killed (polled 0 ms) [ 707.439066] CPU30: shutdown [ 707.441870] psci: CPU30 killed (polled 0 ms) [ 707.706727] CPU31: shutdown [ 707.709526] psci: CPU31 killed (polled 0 ms) [ 708.521853] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible) [ 728.741808] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:80d:00h:35m:42s] [ 728.747895] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=2626 [ 728.757197] (detected by 63, t=5255 jiffies, g=40989, q=1890) [ 728.763018] Task dump for CPU 48: [ 728.766321] irqbalance R running task 0 1272 1 0x00000002 [ 728.773358] Call trace: [ 728.775801] __switch_to+0xbc/0x218 [ 728.779283] gic_set_affinity+0x16c/0x1d8 [ 728.783282] irq_do_set_affinity+0x30/0xd0 [ 728.787365] irq_set_affinity_locked+0xc8/0xf0 [ 728.791796] __irq_set_affinity+0x4c/0x80 [ 728.795794] write_irq_affinity.isra.7+0x104/0x120 [ 728.800572] irq_affinity_proc_write+0x1c/0x28 [ 728.805008] proc_reg_write+0x78/0xb8 [ 728.808660] __vfs_write+0x18/0x38 [ 728.812050] vfs_write+0xb4/0x1e0 [ 728.815352] ksys_write+0x68/0xf8 [ 728.818655] __arm64_sys_write+0x18/0x20 [ 728.822567] el0_svc_common.constprop.2+0x64/0x160 [ 728.827345] el0_svc_handler+0x20/0x80 [ 728.831082] el0_sync_handler+0xe4/0x188 [ 728.834991] el0_sync+0x140/0x180 [ 738.993844] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 791.761810] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:63d:14h:24m:13s] [ 791.767897] rcu: 48-...0: (0 ticks this GP) idle=b3e/1/0x4000000000000000 softirq=5548/5548 fqs=10495 [ 791.777281] (detected by 54, t=21010 jiffies, g=40989, q=2396) [ 791.783189] Task dump for CPU 48: [ 791.786491] irqbalance R running task 0 1272 1 0x00000002 [ 791.793528] Call trace: [ 791.795964] __switch_to+0xbc/0x218 [ 791.799441] gic_set_affinity+0x16c/0x1d8 [ 791.803439] irq_do_set_affinity+0x30/0xd0 [ 791.807522] irq_set_affinity_locked+0xc8/0xf0 [ 791.811953] __irq_set_affinity+0x4c/0x80 [ 791.815949] write_irq_affinity.isra.7+0x104/0x120 [ 791.820727] irq_affinity_proc_write+0x1c/0x28 [ 791.825158] proc_reg_write+0x78/0xb8 [ 791.828808] __vfs_write+0x18/0x38 [ 791.832197] vfs_write+0xb4/0x1e0 [ 791.835500] ksys_write+0x68/0xf8 [ 791.838802] __arm64_sys_write+0x18/0x20 [ 791.842712] el0_svc_common.constprop.2+0x64/0x160 [ 791.847490] el0_svc_handler+0x20/0x80 [ 791.851226] el0_sync_handler+0xe4/0x188 [ 791.855135] el0_sync+0x140/0x180Quoted 1 line
Jobs: 6 (f=6): [R(6)][0.0%][r=0KiB/s
Quoted 1 line
Can you trigger it after disabling irqbalance?
No, so tested by killing the irqbalance process and it ran for 25 minutes without issue.
Thanks,
John
On 2020-02-03 12:56, John Garry wrote:
[...]
Quoted 4 lines
Quoted 1 line
Can you trigger it after disabling irqbalance?
No, so tested by killing the irqbalance process and it ran for 25 minutes without issue.
OK, that's interesting.
Can you find you whether irqbalance tries to move an interrupt to an offlined CPU? Just putting a trace into git_set_affinity() should be enough.
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Quoted 6 lines · Marc Zyngier
On 2020-02-03 12:56, John Garry wrote:
[...]
Quoted 2 lines
Quoted 1 line
Can you trigger it after disabling irqbalance?
Quoted 8 lines
Quoted 2 lines
No, so tested by killing the irqbalance process and it ran for 25 minutes without issue.
OK, that's interesting.
Can you find you whether irqbalance tries to move an interrupt to an offlined CPU? Just putting a trace into
git_set_affinity()should be enough.
Hi Marc,
I should have mentioned this already, but this board is the same D06 which I reported had the CPU0 hotplug issue from broken FW, if you remember:
https://lore.kernel.org/linux-arm-kernel/[email protected]/
That's why I'm not including CPU0 in my hotplug testing. Other CPUs were fine. And it doesn't look like that issue.
Apart from that, I tried as you suggested, with this change:
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -1143,6 +1143,9 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
int enabled;
u64 val;
+ pr_err("%s irq%d mask_val=%*pbl cpu_online_mask=%*pbl force=%d cpumask_any_and=%d\n", __func__, d->irq, cpumask_pr_args(mask_val),cpumask_pr_args(cpu_online_mask), force, pumask_any_and(cpu_online_mask, mask_val));
if (force)
cpu = cpumask_first(mask_val);
else
@@ -1176,6 +1179,9 @@ static int gic_set_affinity(struct irq_data *d, const struct cpumask *mask_val,
irq_data_update_effective_affinity(d, cpumask_of(cpu));
+ pr_err("%s1 irq%d mask_val=%*pbl cpu_online_mask=%*pbl force=%d cpumask_any_and=%d cpu=%d\n", __func__, d->irq, cpumask_pr_args(mask_val), cpumask_pr_args(cpu_online_mask), force, cpumask_any_and(cpu_online_mask, mask_val), cpu);
+
return IRQ_SET_MASK_OK_DONE;
}
And see this:
polled 0 ms)
[ 947.551340] GICv3: gic_set_affinity irq5 mask_val=24-47 cpu_online_mask=0,44-95 force=0 cpumask_any_and=44 [ 947.560986] GICv3: gic_set_affinity1 irq5 mask_val=24-47 cpu_online_mask=0,44-95 force=0 cpumask_any_and=44 cpu=44 [ 947.571321] GICv3: gic_set_affinity irq8 mask_val=24-47 cpu_online_mask=0,44-95 force=0 cpumask_any_and=44 [ 947.580963] GICv3: gic_set_affinity1 irq8 mask_val=24-47 cpu_online_mask=0,44-95 force=0 cpumask_any_and=44 cpu=44 [ 947.591581] IRQ 819: no longer affine to CPU43 [ 947.596149] CPU43: shutdown [ 947.598945] psci: CPU43 killed (polled 0 ms) [ 947.607029] GICv3: gic_set_affinity irq5 mask_val=29-47 cpu_online_mask=0,44-95 force=0 cpumask_any_and=44 Jobs: 6 (f=6): [R(6)][0.0%][r=0KiB/[ 968.614971] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: [ 968.621062] rcu: 66-...0: (0 ticks this GP) idle=d9a/1/0x4000000000000000 softirq=3654/3654 fqs=2625 [ 968.630365] (detected by 69, t=5256 jiffies, g=51577, q=1884) [ 968.636187] Task dump for CPU 66: [ 968.639490] irqbalance R running task 0 1577 1 0x00000002 [ 968.646527] Call trace: [ 968.648970] __switch_to+0xbc/0x218 [ 968.652450] irq_do_set_affinity+0x30/0xd0 [ 968.656534] irq_set_affinity_locked+0xc8/0xf0 [ 968.660965] __irq_set_affinity+0x4c/0x80 [ 968.664963] write_irq_affinity.isra.7+0x104/0x120 [ 968.669741] irq_affinity_proc_write+0x1c/0x28 [ 968.674175] proc_reg_write+0x78/0xb8 [ 968.677827] __vfs_write+0x18/0x38 [ 968.681217] vfs_write+0xb4/0x1e0 [ 968.684519] ksys_write+0x68/0xf8 [ 968.687822] __arm64_sys_write+0x18/0x20 [ 968.691733] el0_svc_common.constprop.2+0x64/0x160 [ 968.696511] el0_svc_handler+0x20/0x80 [ 968.700247] el0_sync_handler+0xe4/0x188 [ 968.704157] el0_sync+0x140/0x180
and this on a 2nd run:
[ 215.468476] CPU42: shutdown [ 215.471272] psci: CPU42 killed (polled 0 ms) [ 215.723714] GICv3: gic_set_affinity irq5 mask_val=24-47 cpu_online_mask=0,44-95 force=0 cpumask_any_and=44 [ 215.733360] GICv3: gic_set_affinity1 irq5 mask_val=24-47 cpu_online_mask=0,44-95 force=0 cpumask_any_and=44 cpu=44 [ 215.743696] GICv3: gic_set_affinity irq8 mask_val=24-47 cpu_online_mask=0,44-95 force=0 cpumask_any_and=44 [ 215.753338] GICv3: gic_set_affinity1 irq8 mask_val=24-47 cpu_online_mask=0,44-95 force=0 cpumask_any_and=44 cpu=44 [ 215.763835] IRQ 426: no longer affine to CPU43 [ 215.768412] IRQ 819: no longer affine to CPU43 [ 215.773023] CPU43: shutdown Jobs: 6 (f=6): [R(6)][76.9%][r=13[ 215.775834] psci: CPU43 killed (polled 0 ms) [ 216.604779] GICv3: gic_set_affinity irq10 mask_val=53 cpu_online_mask=0,44-95 force=0 cpumask_any_and=53 [ 217.223461] pcieport 0000:00:08.0: can't change power state from D3cold to D0 (config space inaccessible) [ 237.615383] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:2d:17h:39m:00s] [ 237.621469] rcu: 58-...0: (1 GPs behind) idle=b5e/1/0x4000000000000000 softirq=1908/1908 fqs=2626 [ 237.630525] (detected by 44, t=5254 jiffies, g=12137, q=191) [ 237.636260] Task dump for CPU 58: [ 237.639563] irqbalance R running task 0 1567 1 0x00000002 [ 237.646599] Call trace: [ 237.649037] __switch_to+0xbc/0x218 [ 237.652513] 0xffff80001529bd68 [ 239.283412] nvme nvme1: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff [ 300.635382] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:03d:01h:02m:56s] [ 300.641466] rcu: 58-...0: (1 GPs behind) idle=b5e/1/0x4000000000000000 softirq=1908/1908 fqs=10503 [ 300.650589] (detected by 44, t=21010 jiffies, g=12137, q=698) [ 300.656410] Task dump for CPU 58: [ 300.659712] irqbalance R running task 0 1567 1 0x00000002 [ 300.666747] Call trace:
Info about irq5 and irq10 after booting:
john@ubuntu:~$ ls /proc/irq/5 affinity_hint effective_affinity_list smp_affinity spurious effective_affinity node smp_affinity_list uart-pl011 john@ubuntu:~$ more /proc/irq/5/smp_affinity_list 24-47 john@ubuntu:~$ ls /proc/irq/10 affinity_hint ehci_hcd:usb1 ohci_hcd:usb3 smp_affinity_list effective_affinity ehci_hcd:usb2 ohci_hcd:usb4 spurious effective_affinity_list node smp_affinity john@ubuntu:~$ more /proc/irq/10/smp_affinity_list 71
Thanks,
John
Quoted 7 lines · John Garry
Quoted 6 lines · Marc Zyngier
On 2020-02-03 12:56, John Garry wrote:
[...]
Quoted 2 lines
Quoted 1 line
Can you trigger it after disabling irqbalance?
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
No, so tested by killing the irqbalance process and it ran for 25 minutes without issue.
OK, that's interesting.
Can you find you whether irqbalance tries to move an interrupt to an offlined CPU? Just putting a trace into
git_set_affinity()should be enough.
Just an update here: I have tried this same test on a new model dev board and I don't experience the same issue. It's quite stable.
I'd like to get to the bottom of the issue reported, but I feel that the root cause may be a BIOS issue and I will get next to no BIOS support for that particular board. Hmmm.
Thanks,
John
Quoted 13 lines · John Garry
Quoted 12 lines · John Garry
Quoted 11 lines · Marc Zyngier
On 2020-02-03 12:56, John Garry wrote:
[...]
Quoted 4 lines
Quoted 1 line
Can you trigger it after disabling irqbalance?
No, so tested by killing the irqbalance process and it ran for 25 minutes without issue.
OK, that's interesting.
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
Can you find you whether irqbalance tries to move an interrupt to an offlined CPU? Just putting a trace into
git_set_affinity()should be enough.Just an update here: I have tried this same test on a new model dev board and I don't experience the same issue. It's quite stable.
Is it the exact same SoC? Or a revised version?
Quoted 3 lines
I'd like to get to the bottom of the issue reported, but I feel that the root cause may be a BIOS issue and I will get next to no BIOS support for that particular board. Hmmm.
I'd very much like to understand it too. Your latest log is even more puzzling, as the backtrace shows a switch_to() even earlier... The fact that this only happens on hotplug off tends to tell me that the firmware gets confused with PSCI OFF. You'd see something like that if a CPU was taken into the firmware (or powered-off) without the rest of the kernel knowing...
Could it be that PSCI powers off more than a single CPU at once?
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Quoted 21 lines · Ming Lei
Quoted 20 lines · John Garry
Quoted 11 lines
Quoted 7 lines
[ 141.976109] Call trace: [ 141.978550] __switch_to+0xbc/0x218 [ 141.982029] blk_mq_run_work_fn+0x1c/0x28 [ 141.986027] process_one_work+0x1e0/0x358 [ 141.990025] worker_thread+0x40/0x488 [ 141.993678] kthread+0x118/0x120 [ 141.996897] ret_from_fork+0x10/0x18Hi John,
Thanks for your test!
Hi Ming,
Quoted 3 lines
Could you test the following patchset and only the last one is changed?
For SCSI testing, I will ask my colleague Xiang Chen to test when he returns to work. So I did not see this issue for my SCSI testing for your original v5, but I was only using 1x as opposed to maybe 20x SAS disks.
Quoted 9 lines
Quoted 3 lines
BTW, did you test NVMe? For some reason I could not trigger a scenario where we're draining the outstanding requests for a queue which is being deactivated - I mean, the queues were always already quiesced.
I run cpu hotplug test on both NVMe and SCSI in KVM, and fio just runs as expected.
NVMe is often 1:1 mapping, so it might be a bit difficult to trigger draining in-flight IOs.
Hi Ming,
We got around to testing your my_for_5.6_block branch (Xiang Chen actually took the v5 series and applied the following on top only:
block: deactivate hctx when running queue in wrong CPU core
Revert "block: deactivate hctx when all its CPUs are offline when run…)
and we get this:
] IRQ 598: no longer affine to CPU4
[ 1077.396063] CPU4: shutdown [ 1077.398769] psci: CPU4 killed (polled 0 ms) [ 1077.457777] CPU3: shutdown [ 1077.460495] psci: CPU3 killed (polled 0 ms) [ 1077.499650] CPU2: shutdown [ 1077.502357] psci: CPU2 killed (polled 0 ms) [ 1077.546976] CPU1: shutdown [ 1077.549690] psci: CPU1 killed (polled 0 ms)
it's running b 0
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [38.9% done] [1201MB/0KB/0KB /s] [307K/0/0 iops] [eta 00m:22s] it's running b 1
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [41.7% done] [625.2MB/0KB/0KB /s] [160K/0/0 iops] [eta 00m:21s] it's running b 2
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [44.4% done] [637.1MB/0KB/0KB /s] [163K/0/0 iops] [eta 00m:20s] it's running b 3
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [47.2% done] [648.6MB/0KB/0KB /s] [166K/0/0 iops] [eta 00m:19s] it's running b 4
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [50.0% done] [672.8MB/0KB/0KB /s] [172K/0/0 iops] [eta 00m:18s] it's running b 5
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [52.8% done] [680.2MB/0KB/0KB /s] [174K/0/0 iops] [eta 00m:17s] it's running b 6
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [55.6% done] [674.7MB/0KB/0KB /s] [173K/0/0 iops] [eta 00m:16s] it's running b 7
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [58.3% done] [666.2MB/0KB/0KB /s] [171K/0/0 iops] [eta 00m:15s] it's running b 8
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [61.1% done] [668.7MB/0KB/0KB /s] [171K/0/0 iops] [eta 00m:14s] it's running b 9
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [63.9% done] [657.9MB/0KB/0KB /s] [168K/0/0 iops] [eta 00m:13s] it's running b 10
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [66.7% done] [659.6MB/0KB/0KB /s] [169K/0/0 iops] [eta 00m:12s] it's running b 11
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [69.4% done] [662.8MB/0KB/0KB /s] [170K/0/0 iops] [eta 00m:11s] it's running b 12
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [72.2% done] [669.8MB/0KB/0KB /s] [171K/0/0 iops] [eta 00m:10s] it's running b 13
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [75.0% done] [673.2MB/0KB/0KB /s] [172K/0/0 iops] [eta 00m:09s] it's running b 14
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [77.8% done] [650.5MB/0KB/0KB /s] [167K/0/0 iops] [eta 00m:08s] it's running b 15
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [80.6% done] [658.9MB/0KB/0KB /s] [169K/0/0 iops] [eta 00m:07s] it's running b 16
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [83.3% done] [670.3MB/0KB/0KB /s] [172K/0/0 iops] [eta 00m:06s] it's running b 17
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [86.1% done] [663.7MB/0KB/0KB /s] [170K/0/0 iops] [eta 00m:05s] it's running b 18
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [88.9% done] [657.9MB/0KB/0KB /s] [168K/0/0 iops] [eta 00m:04s] it's running b 19
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [91.7% done] [650.9MB/0KB/0KB /s] [167K/0/0 iops] [eta 00m:03s] it's running b 20
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [94.4% done] [646.1MB/0KB/0KB /s] [166K/0/0 iops] [eta 00m:02s] it's running b 21
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [97.2% done] [658.4MB/0KB/0KB /s] [169K/0/0 iops] [eta 00m:01s] it's running b 22
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [100.0% done] [649.4MB/0KB/0KB /s] [166K/0/0 iops] [eta 00m:00s] it's running b 23
Jobs: 1 (f=1): [______________________R_________________] [2.6% done] [402.5MB/0KB/0KB /s] [103K/0/0 iops] [eta 22m:44s] it's running b 24
Jobs: 1 (f=1): [______________________R_________________] [2.7% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 22m:43s] it's running b 25
Jobs: 1 (f=1): [______________________R_________________] [2.8% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 22m:42s] it's running b 26
Jobs: 1 (f=1): [______________________R_________________] [2.9% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 22m:41s] it's running b 27
Jobs: 1 (f=1): [______________________R_________________] [2.9% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 22m:40s]
[ 1105.419335] sas: Enter sas_scsi_recover_host busy: 1 failed: 1 [ 1105.425185] sas: trying to find task 0x00000000f1b865f3 [ 1105.430409] sas: sas_scsi_find_task: aborting task 0x00000000f1b865f3
not running b 28
#
Looks like the queues are not properly drained as we're getting a single IO timeout. I'll have a look when I get a chance.
Cheers,
John