Skip to contentSkip to conversation

[3.14] core onlining/hotplug regression

DISCUSSION Daniel J Blueman 2 participants 4 messages Jul 25, 2014 – Sep 13, 2014

lore
Share & tools
lore
#0 lore raw

Hi Thomas et al,

On a larger x86 system with 1728 cores, 3.15(.6) asserts on smpboot_thread_fn's td->cpu != smp_processor_id() consistently after ~1500 cores are online.

Reverting the only directly related changes I could find [1,2] doesn't help. Debugging indicates there is a race where the created thread is quickly migrated to core 0 when this occurs, since smp_processor_id returns 0 in these cases. Thomas introduced a thread parked state to fix related issues a year back. Linux 3.14(.13) boots just nice.

Full boot output is at:
https://resources.numascale.com/linux-315-thread-mig.txt

Any theories so far? I'll start bisecting when I have full access to the system again in a week and I'll do some more debugging with intermittent access before then.

Thanks,
Daniel

-- [1]

commit 81c98869faa5f3a9457c93efef908ef476326b31
Author: Nishanth Aravamudan <[email protected]>
Date: Thu Apr 3 14:46:25 2014 -0700
kthread: ensure locality of task_struct allocations

-- [2]

commit 89f898c1e195fa6235c869bb457e500b7b3ac49d

Author: Igor Mammedov <[email protected]>
Date:   Thu Jun 5 15:42:43 2014 +0200

x86: Fix list/memory corruption on CPU hotplug

Signature · 2 lines
Daniel J Blueman
Principal Software Engineer, Numascale
#1 lore raw
Quoted 9 lines · Daniel J Blueman

On a larger x86 system with 1728 cores, 3.15(.6) asserts on smpboot_thread_fn's td->cpu != smp_processor_id() consistently after ~1500 cores are online.

Reverting the only directly related changes I could find [1,2] doesn't help. Debugging indicates there is a race where the created thread is quickly migrated to core 0 when this occurs, since smp_processor_id returns 0 in these cases. Thomas introduced a thread parked state to fix related issues a year back. Linux 3.14(.13) boots just nice.

Weird. Commits [1,2] are definitely not the culprits.

Quoted 2 lines

Full boot output is at:
https://resources.numascale.com/linux-315-thread-mig.txt

Not really helpful, as we don't see what causes it. We just see the wreckage.

Quoted 3 lines

Any theories so far? I'll start bisecting when I have full access to the system again in a week and I'll do some more debugging with intermittent access before then.

One thing you could try is enabling tracing.

"ftrace=function ftrace_dump_on_oops"

It'll take a looooong time to spill out the traces, but that should give us the root cause precisely.

Thanks,

tglx

#2 lore raw
Quoted 23 lines · Thomas Gleixner
Quoted 9 lines · Daniel J Blueman

On a larger x86 system with 1728 cores, 3.15(.6) asserts on smpboot_thread_fn's td->cpu != smp_processor_id() consistently after ~1500 cores are online.

Reverting the only directly related changes I could find [1,2] doesn't help. Debugging indicates there is a race where the created thread is quickly migrated to core 0 when this occurs, since smp_processor_id returns 0 in these cases. Thomas introduced a thread parked state to fix related issues a year back. Linux 3.14(.13) boots just nice.

Weird. Commits [1,2] are definitely not the culprits.

Quoted 2 lines

Full boot output is at:
https://resources.numascale.com/linux-315-thread-mig.txt

Not really helpful, as we don't see what causes it. We just see the wreckage.

Quoted 3 lines

Any theories so far? I'll start bisecting when I have full access to the system again in a week and I'll do some more debugging with intermittent access before then.

Quoted 6 lines

One thing you could try is enabling tracing.

"ftrace=function ftrace_dump_on_oops"

It'll take a looooong time to spill out the traces, but that should give us the root cause precisely.

Good trick. I'll get this early next week and we'll see what's up.

Thanks,
Daniel

Signature · 2 lines
Daniel J Blueman
Principal Software Engineer, Numascale
#3 lore raw
In reply to Thomas Gleixner
Quoted 23 lines · Thomas Gleixner
Quoted 9 lines · Daniel J Blueman

On a larger x86 system with 1728 cores, 3.15(.6) asserts on smpboot_thread_fn's td->cpu != smp_processor_id() consistently after ~1500 cores are online.

Reverting the only directly related changes I could find [1,2] doesn't help. Debugging indicates there is a race where the created thread is quickly migrated to core 0 when this occurs, since smp_processor_id returns 0 in these cases. Thomas introduced a thread parked state to fix related issues a year back. Linux 3.14(.13) boots just nice.

Weird. Commits [1,2] are definitely not the culprits.

Quoted 2 lines

Full boot output is at:
https://resources.numascale.com/linux-315-thread-mig.txt

Not really helpful, as we don't see what causes it. We just see the wreckage.

Quoted 3 lines

Any theories so far? I'll start bisecting when I have full access to the system again in a week and I'll do some more debugging with intermittent access before then.

Quoted 6 lines

One thing you could try is enabling tracing.

"ftrace=function ftrace_dump_on_oops"

It'll take a looooong time to spill out the traces, but that should give us the root cause precisely.

It turns out that bisecting led to Lai's patch "Fix hotplug vs. set_cpus_allowed_ptr()" [1]. Reverting it prevents the smpboot.c BUG_ON(td->cpu != smp_processor_id()) in smpboot_thread_fn from tripping.

-- [1]

commit 6acbfb96976fc3350e30d964acb1dbbdf876d55e

Author: Lai Jiangshan <[email protected]>
Date:   Fri May 16 11:50:42 2014 +0800

sched: Fix hotplug vs. set_cpus_allowed_ptr()

Lai found that:

WARNING: CPU: 1 PID: 13 at arch/x86/kernel/smp.c:124

native_smp_send_reschedule+0x2d/0x4b()
      ...
      migration_cpu_stop+0x1d/0x22

was caused by set_cpus_allowed_ptr() assuming that cpu_active_mask is always a sub-set of cpu_online_mask.

This isn't true since 5fbd036b552f ("sched: Cleanup cpu_active
madness").

So set active and online at the same time to avoid this particular problem.

Fixes 5fbd036b552f ("sched: Cleanup cpu_active madness")
Cc Andrew Morton <[email protected]>
Cc Gautham R. Shenoy <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Michael wang <[email protected]>
Cc Paul Gortmaker <[email protected]>
Cc Rafael J. Wysocki <[email protected]>
Cc Srivatsa S. Bhat <[email protected]>
Cc Toshi Kani <[email protected]>
Signature · 2 lines
Daniel J Blueman
Principal Software Engineer, Numascale