workqueue throwing cpu affinity warnings during CPU hotplug
Hi Tejun,
I just upgraded my test VMs from 6.12.0 to a current TOT kernel and I got several of these warnings whilst running fstests whilst running CPU hotplug online/offline concurrently with various tests:
[ 2508.109594] ------------[ cut here ]------------ [ 2508.115669] WARNING: CPU: 23 PID: 133 at kernel/kthread.c:76 kthread_set_per_cpu+0x33/0x50 [ 2508.125271] Modules linked in: [ 2508.131000] CPU: 23 UID: 0 PID: 133 Comm: cpuhp/23 Tainted: G D W 6.12.0-dgc+ #296 [ 2508.142954] Tainted: [D]=DIE, [W]=WARN [ 2508.148153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 2508.148158] RIP: 0010:kthread_set_per_cpu+0x33/0x50 [ 2508.169229] Code: f6 47 2e 20 74 24 48 8b 87 58 0a 00 00 48 85 c0 74 11 f6 47 2f 04 74 16 85 f6 78 18 89 70 08 f0 80 08 01 5d c3 cc cc cc cc cc <0f> 0b eb d8 0f 0b 85 f6 79 e8 f0 80 20 fe 5d c3 cc cc cc cc cc 0f [ 2508.187728] RSP: 0018:ffffc900068c3de8 EFLAGS: 00010246 [ 2508.192758] RAX: 0000000000000000 RBX: ffff8888d4295680 RCX: 0b0ee35baf810600 [ 2508.199999] RDX: 0000000000000040 RSI: 00000000ffffffff RDI: ffff888866aa0000 [ 2508.208341] RBP: ffffc900068c3de8 R08: 0000000000000001 R09: 0000000000000001 [ 2508.218878] R10: 000000005b5d0000 R11: ffffffff8118d8b0 R12: 000000000002fef0 [ 2508.226572] R13: ffff88901fbefb88 R14: ffff88901fbef8c0 R15: 0000000000000017 [ 2508.233346] FS: 0000000000000000(0000) GS:ffff88901fbc0000(0000) knlGS:0000000000000000 [ 2508.239695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2508.244123] CR2: 00007fbaf43c0004 CR3: 0000000003256000 CR4: 0000000000350ef0 [ 2508.249599] Call Trace: [ 2508.253909] <TASK> [ 2508.311972] unbind_worker+0x1b/0x70 [ 2508.315444] workqueue_offline_cpu+0xd8/0x1f0 [ 2508.319554] cpuhp_invoke_callback+0x13e/0x4f0 [ 2508.328936] cpuhp_thread_fun+0xda/0x120 [ 2508.332746] smpboot_thread_fn+0x132/0x1d0 [ 2508.336645] kthread+0x147/0x170 [ 2508.347646] ret_from_fork+0x3e/0x50 [ 2508.353845] ret_from_fork_asm+0x1a/0x30 [ 2508.357773] </TASK> [ 2508.357776] ---[ end trace 0000000000000000 ]---
I have also seen similar traces from the CPUs coming on-line:
[ 2535.818771] WARNING: CPU: 23 PID: 133 at kernel/kthread.c:76 kthread_set_per_cpu+0x33/0x50 .... [ 2535.969004] RIP: 0010:kthread_set_per_cpu+0x33/0x50 .... [ 2508.249599] Call Trace: [ 2508.253909] <TASK> [ 2535.969029] workqueue_online_cpu+0xe6/0x2f0 [ 2535.969032] cpuhp_invoke_callback+0x13e/0x4f0 [ 2535.969044] cpuhp_thread_fun+0xda/0x120 [ 2535.969047] smpboot_thread_fn+0x132/0x1d0 [ 2535.969053] kthread+0x147/0x170 [ 2535.969066] ret_from_fork+0x3e/0x50 [ 2535.969076] ret_from_fork_asm+0x1a/0x30 [ 2508.357773] </TASK>
I didn't see these on 6.12.0, so I'm guessing that there is something in the merge window that has started triggering this.
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Hello, Dave.
Sorry about the really late reply.
Quoted 8 lines · Dave Chinner
Hi Tejun,
I just upgraded my test VMs from 6.12.0 to a current TOT kernel and I got several of these warnings whilst running fstests whilst running CPU hotplug online/offline concurrently with various tests:
[ 2508.109594] ------------[ cut here ]------------ [ 2508.115669] WARNING: CPU: 23 PID: 133 at kernel/kthread.c:76 kthread_set_per_cpu+0x33/0x50
...
Quoted 11 lines
[ 2508.253909] <TASK> [ 2508.311972] unbind_worker+0x1b/0x70 [ 2508.315444] workqueue_offline_cpu+0xd8/0x1f0 [ 2508.319554] cpuhp_invoke_callback+0x13e/0x4f0 [ 2508.328936] cpuhp_thread_fun+0xda/0x120 [ 2508.332746] smpboot_thread_fn+0x132/0x1d0 [ 2508.336645] kthread+0x147/0x170 [ 2508.347646] ret_from_fork+0x3e/0x50 [ 2508.353845] ret_from_fork_asm+0x1a/0x30 [ 2508.357773] </TASK> [ 2508.357776] ---[ end trace 0000000000000000 ]---
So, this is kthread saying that the thread passed to it doesn't have PF_KTHREAD set. There hasn't been any related changes and the flag is never cleared once set, so I don't see how that could be for a kworker.
Quoted 16 lines
I have also seen similar traces from the CPUs coming on-line:
[ 2535.818771] WARNING: CPU: 23 PID: 133 at kernel/kthread.c:76 kthread_set_per_cpu+0x33/0x50 .... [ 2535.969004] RIP: 0010:kthread_set_per_cpu+0x33/0x50 .... [ 2508.249599] Call Trace: [ 2508.253909] <TASK> [ 2535.969029] workqueue_online_cpu+0xe6/0x2f0 [ 2535.969032] cpuhp_invoke_callback+0x13e/0x4f0 [ 2535.969044] cpuhp_thread_fun+0xda/0x120 [ 2535.969047] smpboot_thread_fn+0x132/0x1d0 [ 2535.969053] kthread+0x147/0x170 [ 2535.969066] ret_from_fork+0x3e/0x50 [ 2535.969076] ret_from_fork_asm+0x1a/0x30 [ 2508.357773] </TASK>
Yeah, this is the same.
Quoted 2 lines
I didn't see these on 6.12.0, so I'm guessing that there is something in the merge window that has started triggering this.
I tried a few mixtures of stress-ng + continuous hot [un]plugging but can't reproduce in the current linus#master. Do you still see this happening?
Thanks.
Signature · 1 line
tejun
Quoted 29 lines · Tejun Heo
Hello, Dave.
Sorry about the really late reply.
Quoted 8 lines · Dave Chinner
Hi Tejun,
I just upgraded my test VMs from 6.12.0 to a current TOT kernel and I got several of these warnings whilst running fstests whilst running CPU hotplug online/offline concurrently with various tests:
[ 2508.109594] ------------[ cut here ]------------ [ 2508.115669] WARNING: CPU: 23 PID: 133 at kernel/kthread.c:76 kthread_set_per_cpu+0x33/0x50...
Quoted 11 lines
[ 2508.253909] <TASK> [ 2508.311972] unbind_worker+0x1b/0x70 [ 2508.315444] workqueue_offline_cpu+0xd8/0x1f0 [ 2508.319554] cpuhp_invoke_callback+0x13e/0x4f0 [ 2508.328936] cpuhp_thread_fun+0xda/0x120 [ 2508.332746] smpboot_thread_fn+0x132/0x1d0 [ 2508.336645] kthread+0x147/0x170 [ 2508.347646] ret_from_fork+0x3e/0x50 [ 2508.353845] ret_from_fork_asm+0x1a/0x30 [ 2508.357773] </TASK> [ 2508.357776] ---[ end trace 0000000000000000 ]---So, this is kthread saying that the thread passed to it doesn't have
PF_KTHREADset. There hasn't been any related changes and the flag is never cleared once set, so I don't see how that could be for a kworker.
....
Quoted 5 lines
Quoted 2 lines
I didn't see these on 6.12.0, so I'm guessing that there is something in the merge window that has started triggering this.
I tried a few mixtures of stress-ng + continuous hot [un]plugging but can't reproduce in the current linus#master. Do you still see this happening?
I don't know. The machine that I was using to reproduce this has been down with hardware failure for the last two weeks. I might have it back up today, and then I'll be able to start exercising this again.
-Dave.
Signature · 2 lines
Dave Chinner [email protected]