blk-mq and CPU hotplug error

Hello everyone!

I'm running Ubuntu 18.04 (4.15.0-12-generic) in KVM/QEMU (powerpc64le). Everything looks good until I try to hotplug CPUs in my VM. As soon as I do that I get the following error in my VM dmesg:

[  763.629425] WARNING: CPU: 34 PID: 2276 at /build/linux-zBVy54/linux-4.15.0/block/blk-mq.c:1211 __blk_mq_run_hw_queue+0xf0/0x140
...
[  763.629497] CPU: 34 PID: 2276 Comm: kworker/34:1H Not tainted 4.15.0-12-generic #13-Ubuntu
[  763.629501] Workqueue: kblockd blk_mq_run_work_fn

See that blk-mq complains about CPU 34. This CPU is one of the hotplugged CPUs.

Looking at mq/0/cpu_list, I see the my new CPUs aren't in that list:

# cat /sys/block/vda/mq/0/cpu_list
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31

But the real problem comes after it, if I try to run any command that performs intensive IO operations, like:

# dd if=/dev/random of=test.img bs=1M count=100

I start to get errors like:

[  967.661543] INFO: task kworker/u128:3:454 blocked for more than 120 seconds.
[  967.661656]       Tainted: G        W        4.15.0-12-generic #13-Ubuntu
[  967.661705] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  967.661762] kworker/u128:3  D    0   454      2 0x00000800
[  967.661802] Workqueue: writeback wb_workfn (flush-252:0)
[  967.661805] Call Trace:
[  967.661808] [c000000267e5f1d0] [0000002200000000] 0x2200000000 (unreliable)
[  967.661813] [c000000267e5f3a0] [c00000000001c220] __switch_to+0x2a0/0x4d0
[  967.661823] [c000000267e5f400] [c000000000d035e4] __schedule+0x2a4/0xaf0
[  967.661825] [c000000267e5f4d0] [c000000000d03e70] schedule+0x40/0xc0
[  967.661834] [c000000267e5f4f0] [c00000000014f77c] io_schedule+0x2c/0x50
[  967.661838] [c000000267e5f520] [c0000000002d810c] __lock_page+0x14c/0x1c0
[  967.661842] [c000000267e5f5c0] [c0000000004c9288] mpage_prepare_extent_to_map+0x2c8/0x3d0
[  967.661844] [c000000267e5f6c0] [c0000000004d16bc] ext4_writepages+0x49c/0xfd0
[  967.661847] [c000000267e5f870] [c0000000002f1a9c] do_writepages+0x4c/0x130
[  967.661849] [c000000267e5f8e0] [c00000000041a370] __writeback_single_inode+0x70/0x570
[  967.661852] [c000000267e5f940] [c00000000041af0c] writeback_sb_inodes+0x26c/0x5d0
[  967.661854] [c000000267e5fa50] [c00000000041b308] __writeback_inodes_wb+0x98/0x110
[  967.661856] [c000000267e5fab0] [c00000000041b72c] wb_writeback+0x29c/0x460
[  967.661859] [c000000267e5fb80] [c00000000041c38c] wb_workfn+0x1ec/0x600
[  967.661862] [c000000267e5fc90] [c000000000131538] process_one_work+0x298/0x5a0
[  967.661864] [c000000267e5fd20] [c0000000001318d8] worker_thread+0x98/0x630
[  967.661866] [c000000267e5fdc0] [c00000000013a508] kthread+0x1a8/0x1b0
[  967.661869] [c000000267e5fe30] [c00000000000b528] ret_from_kernel_thread+0x5c/0xb4
[  967.661872] INFO: task jbd2/vda2-8:579 blocked for more than 120 seconds.
[  967.661924]       Tainted: G        W        4.15.0-12-generic #13-Ubuntu
[  967.661973] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  967.662035] jbd2/vda2-8     D    0   579      2 0x00000800
[  967.662037] Call Trace:

...snip

Which leads to an unresponsive VM that forces me to reboot it. After searching the internet I changed the IO scheduler:

# cat /sys/block/vda/queue/scheduler
[none]

# modprobe bfq
# echo bfq > /sys/block/vda/queue/scheduler

# cat /sys/block/vda/queue/scheduler
[bfq] none

which seems to be enough (it still throws that WARNING, but no hung_task_timeout_secs so far).

Now I'm not sure on how to attack the problem. At first I thought that CPU hotplug IRQ should also update the cpu_list but now I think that the problem could be in the queue, maybe it should give up scheduling tasks on the failing CPU and try another one. What do you people think?

Any suggestion on how to debug it is welcome as well.

Thank you!

Jose R. Ziviani

Hi Jose

Please refer to the following commit:
20e4d813 blk-mq: simplify queue mapping & schedule with each possisble CPU 84676c1 genirq/affinity: assign vectors to all possible CPUs

Thanks
Jianchao