[3.14] core onlining/hotplug regression
Hi Thomas et al,
On a larger x86 system with 1728 cores, 3.15(.6) asserts on smpboot_thread_fn's td->cpu != smp_processor_id() consistently after ~1500 cores are online.
Reverting the only directly related changes I could find [1,2] doesn't help. Debugging indicates there is a race where the created thread is quickly migrated to core 0 when this occurs, since smp_processor_id returns 0 in these cases. Thomas introduced a thread parked state to fix related issues a year back. Linux 3.14(.13) boots just nice.
Full boot output is at:
https://resources.numascale.com/linux-315-thread-mig.txt
Any theories so far? I'll start bisecting when I have full access to the system again in a week and I'll do some more debugging with intermittent access before then.
Thanks,
Daniel
-- [1]
commit 81c98869faa5f3a9457c93efef908ef476326b31
Author: Nishanth Aravamudan <[email protected]>
Date: Thu Apr 3 14:46:25 2014 -0700
kthread: ensure locality of task_struct allocations
-- [2]
commit 89f898c1e195fa6235c869bb457e500b7b3ac49d
Author: Igor Mammedov <[email protected]> Date: Thu Jun 5 15:42:43 2014 +0200
x86: Fix list/memory corruption on CPU hotplug
Signature · 2 lines
Daniel J Blueman Principal Software Engineer, Numascale
Quoted 9 lines · Daniel J Blueman
On a larger x86 system with 1728 cores, 3.15(.6) asserts on smpboot_thread_fn's
td->cpu!=smp_processor_id()consistently after ~1500 cores are online.Reverting the only directly related changes I could find [1,2] doesn't help. Debugging indicates there is a race where the created thread is quickly migrated to core 0 when this occurs, since
smp_processor_idreturns 0 in these cases. Thomas introduced a thread parked state to fix related issues a year back. Linux 3.14(.13) boots just nice.
Weird. Commits [1,2] are definitely not the culprits.
Quoted 2 lines
Full boot output is at:
https://resources.numascale.com/linux-315-thread-mig.txt
Not really helpful, as we don't see what causes it. We just see the wreckage.
Quoted 3 lines
Any theories so far? I'll start bisecting when I have full access to the system again in a week and I'll do some more debugging with intermittent access before then.
One thing you could try is enabling tracing.
"ftrace=function ftrace_dump_on_oops"
It'll take a looooong time to spill out the traces, but that should give us the root cause precisely.
Thanks,
tglx
Quoted 23 lines · Thomas Gleixner
Quoted 9 lines · Daniel J Blueman
On a larger x86 system with 1728 cores, 3.15(.6) asserts on smpboot_thread_fn's
td->cpu!=smp_processor_id()consistently after ~1500 cores are online.Reverting the only directly related changes I could find [1,2] doesn't help. Debugging indicates there is a race where the created thread is quickly migrated to core 0 when this occurs, since
smp_processor_idreturns 0 in these cases. Thomas introduced a thread parked state to fix related issues a year back. Linux 3.14(.13) boots just nice.Weird. Commits [1,2] are definitely not the culprits.
Quoted 2 lines
Full boot output is at:
https://resources.numascale.com/linux-315-thread-mig.txtNot really helpful, as we don't see what causes it. We just see the wreckage.
Quoted 3 lines
Any theories so far? I'll start bisecting when I have full access to the system again in a week and I'll do some more debugging with intermittent access before then.
Quoted 6 lines
One thing you could try is enabling tracing.
"ftrace=function
ftrace_dump_on_oops"It'll take a looooong time to spill out the traces, but that should give us the root cause precisely.
Good trick. I'll get this early next week and we'll see what's up.
Thanks,
Daniel
Signature · 2 lines
Daniel J Blueman Principal Software Engineer, Numascale
Quoted 23 lines · Thomas Gleixner
Quoted 9 lines · Daniel J Blueman
On a larger x86 system with 1728 cores, 3.15(.6) asserts on smpboot_thread_fn's
td->cpu!=smp_processor_id()consistently after ~1500 cores are online.Reverting the only directly related changes I could find [1,2] doesn't help. Debugging indicates there is a race where the created thread is quickly migrated to core 0 when this occurs, since
smp_processor_idreturns 0 in these cases. Thomas introduced a thread parked state to fix related issues a year back. Linux 3.14(.13) boots just nice.Weird. Commits [1,2] are definitely not the culprits.
Quoted 2 lines
Full boot output is at:
https://resources.numascale.com/linux-315-thread-mig.txtNot really helpful, as we don't see what causes it. We just see the wreckage.
Quoted 3 lines
Any theories so far? I'll start bisecting when I have full access to the system again in a week and I'll do some more debugging with intermittent access before then.
Quoted 6 lines
One thing you could try is enabling tracing.
"ftrace=function
ftrace_dump_on_oops"It'll take a looooong time to spill out the traces, but that should give us the root cause precisely.
It turns out that bisecting led to Lai's patch "Fix hotplug vs. set_cpus_allowed_ptr()" [1]. Reverting it prevents the smpboot.c BUG_ON(td->cpu != smp_processor_id()) in smpboot_thread_fn from tripping.
-- [1]
commit 6acbfb96976fc3350e30d964acb1dbbdf876d55e
Author: Lai Jiangshan <[email protected]> Date: Fri May 16 11:50:42 2014 +0800
sched: Fix hotplug vs. set_cpus_allowed_ptr()
Lai found that:
WARNING: CPU: 1 PID: 13 at arch/x86/kernel/smp.c:124
native_smp_send_reschedule+0x2d/0x4b()
...
migration_cpu_stop+0x1d/0x22 was caused by set_cpus_allowed_ptr() assuming that cpu_active_mask is always a sub-set of cpu_online_mask.
This isn't true since 5fbd036b552f ("sched: Cleanup cpu_active
madness").
So set active and online at the same time to avoid this particular problem.
Signature · 2 lines
Daniel J Blueman Principal Software Engineer, Numascale