Skip to contentSkip to conversation

blk-mq: Make running from the wrong CPU less scary

PATCH In review Daniel Wagner 2 participants 2 messages Nov 26, 2020 – Nov 27, 2020 Block layer

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 blk-mq: Make running from the wrong CPU less scary

The current warning looks aweful like a proper crash. This is confusing. There is not much information to gained from the stack trace anyway, let's drop it.

While at it print the cpumask as there might be additial helpful information when debugging the sitation.

---
Hi,

We got a report from a customer because he was concerned about the log entries. As it turns out, it fooled me too to be honest. What do you think about making it a bit less look-a-like a kernel oops?

smpboot: Booting Node 0 Processor 12 APIC 0x26
WARNING, didn't collect load info for all cpus, balancing is broken
run queue from wrong CPU 0, hctx active

 CPU: 0 PID: 42300 Comm: kworker/13:2H Kdump: loaded Tainted: G           OE  X    5.3.18-109.$
 Hardware name: IBM System x3650 M5 -[5462AC1]-/00KG915, BIOS -[TCE144J-3.11]- 12/03/2019      
 Workqueue: kblockd blk_mq_run_work_fn                                                         
 Call Trace:                                                                                   
  dump_stack+0x66/0x8b                                                                         
  __blk_mq_run_hw_queue+0xee/0x100                                                             
  process_one_work+0x1f4/0x3e0                                                                 
  worker_thread+0x2d/0x3e0                                                                     
  ? process_one_work+0x3e0/0x3e0                                                               
  kthread+0x10d/0x130                                                                          
  ? kthread_park+0xa0/0xa0                                                                     
  ret_from_fork+0x35/0x40                                                                      
 run queue from wrong CPU 0, hctx active                                                       
 CPU: 0 PID: 42300 Comm: kworker/13:2H Kdump: loaded Tainted: G           OE  X    5.3.18-109.$
 Hardware name: IBM System x3650 M5 -[5462AC1]-/00KG915, BIOS -[TCE144J-3.11]- 12/03/2019      
 Workqueue: kblockd blk_mq_run_work_fn    

Thanks,
Daniel

 block/blk-mq.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
block/blk-mq.c +2 -4
index 55bcee5dc032..0427b719d9c4 100644
@@ -1514,10 +1514,8 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
 	 */
 	if (!cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask) &&
 		cpu_online(hctx->next_cpu)) {
-		printk(KERN_WARNING "run queue from wrong CPU %d, hctx %s\n",
-			raw_smp_processor_id(),
-			cpumask_empty(hctx->cpumask) ? "inactive": "active");
-		dump_stack();
+		printk(KERN_WARNING "run queue from wrong CPU %d, hctx %*pbl\n",
+			raw_smp_processor_id(), cpumask_pr_args(hctx->cpumask));
 	}
 
 	/*
Signature · 1 line
2.16.4
#1 lore raw
Quoted 48 lines · Daniel Wagner

The current warning looks aweful like a proper crash. This is confusing. There is not much information to gained from the stack trace anyway, let's drop it.

While at it print the cpumask as there might be additial helpful information when debugging the sitation.

---
Hi,

We got a report from a customer because he was concerned about the log entries. As it turns out, it fooled me too to be honest. What do you think about making it a bit less look-a-like a kernel oops?

smpboot: Booting Node 0 Processor 12 APIC 0x26
WARNING, didn't collect load info for all cpus, balancing is broken
run queue from wrong CPU 0, hctx active

 CPU: 0 PID: 42300 Comm: kworker/13:2H Kdump: loaded Tainted: G           OE  X    5.3.18-109.$
 Hardware name: IBM System x3650 M5 -[5462AC1]-/00KG915, BIOS -[TCE144J-3.11]- 12/03/2019      
 Workqueue: kblockd blk_mq_run_work_fn                                                         
 Call Trace:                                                                                   
  dump_stack+0x66/0x8b                                                                         
  __blk_mq_run_hw_queue+0xee/0x100                                                             
  process_one_work+0x1f4/0x3e0                                                                 
  worker_thread+0x2d/0x3e0                                                                     
  ? process_one_work+0x3e0/0x3e0                                                               
  kthread+0x10d/0x130                                                                          
  ? kthread_park+0xa0/0xa0                                                                     
  ret_from_fork+0x35/0x40                                                                      
 run queue from wrong CPU 0, hctx active                                                       
 CPU: 0 PID: 42300 Comm: kworker/13:2H Kdump: loaded Tainted: G           OE  X    5.3.18-109.$
 Hardware name: IBM System x3650 M5 -[5462AC1]-/00KG915, BIOS -[TCE144J-3.11]- 12/03/2019      
 Workqueue: kblockd blk_mq_run_work_fn    

Thanks,
Daniel

 block/blk-mq.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
block/blk-mq.c +0 -0
index 55bcee5dc032..0427b719d9c4 100644

block/blk-mq.c static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx) · line 1519

Quoted 10 lines
block/blk-mq.c +2 -4
@@ -1514,10 +1514,8 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
 	 */
 	if (!cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask) &&
 		cpu_online(hctx->next_cpu)) {
-		printk(KERN_WARNING "run queue from wrong CPU %d, hctx %s\n",
-			raw_smp_processor_id(),
-			cpumask_empty(hctx->cpumask) ? "inactive": "active");
-		dump_stack();
+		printk(KERN_WARNING "run queue from wrong CPU %d, hctx %*pbl\n",
+			raw_smp_processor_id(), cpumask_pr_args(hctx->cpumask));
 	}

Now we have guaranteed that no any requests originated from one hctx exists when this hctx is going to offline, which is strong enough for killing the check.

The reason why such warning is triggered is that wq's cpu hot unplug is handled before blk-mq's handling.

I'd suggest to kill the whole branch in the fast path.

Thanks,
Ming