x86: fix hang when AP bringup is too slow
v5 cover x86: fix hang when AP bringup is too slow
changes since v4:
* merge "[PATCH v4 1/5] x86: fix list corruption on CPU hotplug" and "[PATCH v4 2/5] x86: fix memory corruption in acpi_unmap_lsapic()" together
* "x86: initialize secondary CPU only if master CPU will wait for it:
- add 10 seconds timeout description into commit message
- add smp_mb() after clearing cpu_initialized_mask
changes since v3:
* put simple bugfixes first
* move common part of syncing with master CPU in cpu_init() for x32/64 variant into helper function
* cpu_init(): WARN_ON if cpu_initialized_mask is set
* fix panic on CPU unplug, caused by erroneous removing of "pr->dev = dev;" in drivers/acpi/acpi_processor.c
--
Hang is observed on virtual machines during CPU hotplug, especially in big guests with many CPUs. (It happens more often if host is over-committed).
Hang happens because master CPU timeouts on waiting till AP boots and 'cancels' CPU online operation assuming AP is not functional but AP may continue run wild later causing various hangs or panics in running kernel that is assuming that AP was offline.
This is an alternative approach, that instead of canceling in-progress AP bringup (https://lkml.org/lkml/2014/3/6/257),
removes timeouts so that AP bringup won't be affected by poor timing and syncs AP with master CPU at early startup making sure that AP won't run wild if master CPU doesn't expect AP to come online.
Series also fixes 3 bugs found during testing CPU bringup failure case.
--
Below is the detailed description of a more often happening hang:
---
Master CPU may timeout before cpu_callin_mask is set and cancel booting CPU, but being onlined CPU still continues to boot, sets cpu_active_mask (CPU_STARTING notifiers) and spins in check_tsc_sync_target() for master cpu to arrive. Following attempt to online another cpu hangs in stop_machine, initiated from here:smp_callin ->
smp_store_cpu_info ->
identify_secondary_cpu ->
mtrr_ap_init -> set_mtrr_from_inactive_cpustop_machine waits on completion of stop_work on all CPUs from cpu_active_mask including a failed CPU that spins in check_tsc_sync_target().
Igor Mammedov (4):
x86: fix list/memory corruption on CPU hotplug
acpi_processor: do not mark present at boot but not onlined CPU as
onlined
x86: log error on secondary CPU wakeup failure at ERR level
x86: initialize secondary CPU only if master CPU will wait for it
arch/x86/kernel/cpu/common.c | 27 ++++++---- arch/x86/kernel/smpboot.c | 104 +++++++++++++---------------------------- drivers/acpi/acpi_processor.c | 1 - 3 files changed, 48 insertions(+), 84 deletions(-)
PATCH v5 1/4 x86: fix list/memory corruption on CPU hotplug
currently if AP wake up is failed, master CPU marks AP as not present in do_boot_cpu() by calling set_cpu_present(cpu, false). That leads to following list corruption on the next physical CPU hotplug:
[ 418.107336] WARNING: CPU: 1 PID: 45 at lib/list_debug.c:33 __list_add+0xbe/0xd0() [ 418.115268] list_add corruption. prev->next should be next (ffff88003dc57600), but was ffff88003e20c3a0. (prev=ffff88003e20c3a0). [ 418.123693] Modules linked in: nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6t_REJECT ipt_REJECT cfg80211 xt_conntrack rfkill ee [ 418.138979] CPU: 1 PID: 45 Comm: kworker/u10:1 Not tainted 3.14.0-rc6+ #387 [ 418.149989] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [ 418.165750] Workqueue: kacpi_hotplug acpi_hotplug_work_fn [ 418.166433] 0000000000000021 ffff880038ca7988 ffffffff8159b22d 0000000000000021 [ 418.176460] ffff880038ca79d8 ffff880038ca79c8 ffffffff8106942c ffff880038ca79e8 [ 418.177453] ffff88003e20c3a0 ffff88003dc57600 ffff88003e20c3a0 00000000ffffffea [ 418.178445] Call Trace: [ 418.185811] [<ffffffff8159b22d>] dump_stack+0x49/0x5c [ 418.186440] [<ffffffff8106942c>] warn_slowpath_common+0x8c/0xc0 [ 418.187192] [<ffffffff81069516>] warn_slowpath_fmt+0x46/0x50 [ 418.191231] [<ffffffff8136ef51>] ? acpi_ns_get_node+0xb7/0xc7 [ 418.193889] [<ffffffff812f796e>] __list_add+0xbe/0xd0 [ 418.196649] [<ffffffff812e2aa9>] kobject_add_internal+0x79/0x200 [ 418.208610] [<ffffffff812e2e18>] kobject_add_varg+0x38/0x60 [ 418.213831] [<ffffffff812e2ef4>] kobject_add+0x44/0x70 [ 418.229961] [<ffffffff813e2c60>] device_add+0xd0/0x550 [ 418.234991] [<ffffffff813f0e95>] ? pm_runtime_init+0xe5/0xf0 [ 418.250226] [<ffffffff813e32be>] device_register+0x1e/0x30 [ 418.255296] [<ffffffff813e82a3>] register_cpu+0xe3/0x130 [ 418.266539] [<ffffffff81592be5>] arch_register_cpu+0x65/0x150 [ 418.285845] [<ffffffff81355c0d>] acpi_processor_hotadd_init+0x5a/0x9b
...
Which is caused by the fact that generic_processor_info() allocates logical CPU id by calling:
cpu = cpumask_next_zero(-1, cpu_present_mask);
which returns id of previously failed to wake up CPU, since its bit is cleared by do_boot_cpu() and as result register_cpu() tries to register another CPU with the same id as already present but failed to be onlined CPU.
Taking in account that AP will not do anything if master CPU failed to wake it up, there is no reason to mark that AP as not present and break next cpu hotplug attempts. As a side effect of not marking AP as not present, user would be allowed to online it again later.
--
Also fix memory corruption in acpi_unmap_lsapic()
if during CPU hotplug master CPU failed to wake up AP it set percpu x86_cpu_to_apicid to BAD_APICID=0xFFFF for AP.
However following attempt to unplug that CPU will lead to out of bound write access to __apicid_to_node[] which is 32768 items long on x86_64 kernel.
So with above fix of cpu_present_mask make sure that a present CPU has a valid APIC ID by not setting x86_cpu_to_apicid to BAD_APICID in do_boot_cpu() on failure and allow acpi_processor_remove()->acpi_unmap_lsapic() cleanly remove CPU.
---
v2:
- squash "[v4 2/5] x86: fix memory corruption in acpi_unmap_lsapic()" into "[v4 1/5] x86: fix list corruption on CPU hotplug"
--- arch/x86/kernel/smpboot.c | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-)
arch/x86/kernel/smpboot.c +0 -3
@@ -859,9 +859,6 @@ static int do_boot_cpu(int apicid, int cpu, struct task_struct *idle)/* was set by cpu_init() */ cpumask_clear_cpu (cpu, cpu_initialized_mask); - -set_cpu_present (cpu, false); -per_cpu (x86_cpu_to_apicid, cpu) =BAD_APICID ; }/* mark "stuck" area as not stuck */
Signature · 1 line
1.7.1
PATCH v5 2/4 acpi_processor: do not mark present at boot but not onlined CPU as onlined
In reply to Igor Mammedovacpi_processor_add() assumes that present at boot CPUs are always onlined, it is not so if a CPU failed to become onlined. As result acpi_processor_add() will mark such CPU device as onlined in sysfs and following attempts to online/offline it using /sys/device/system/cpu/cpuX/online attribute will fail.
Do not poke into device internals in acpi_processor_add() and touch "struct device { .offline }" attribute, since for CPUs onlined at boot it's set by:
topology_init() -> arch_register_cpu() -> register_cpu()
before ACPI device tree is parsed, and for hotplugged CPUs it's set when userspace onlines CPU via sysfs.
---
v2:
- fix regression in v1 leading to NULL pointer dereference on CPU unplug, do not remove "pr->dev = dev;"
--- drivers/acpi/acpi_processor.c | 1 - 1 files changed, 0 insertions(+), 1 deletions(-)
drivers/acpi/acpi_processor.c +0 -1
@@ -405,7 +405,6 @@ static int acpi_processor_add(struct acpi_device *device, goto err; pr->dev = dev; - dev->offline = pr->flags.need_hotplug_init;/* Trigger the processor driver's .probe() if present. */ if (device_attach (dev) >=0 )
Signature · 1 line
1.7.1
Re: [PATCH v5 2/4] acpi_processor: do not mark present at boot but not onlined CPU as onlined
Quoted 14 lines · Igor Mammedov
acpi_processor_add()assumes that present at boot CPUs are always onlined, it is not so if a CPU failed to become onlined. As resultacpi_processor_add()will mark such CPU device as onlined in sysfs and following attempts to online/offline it using/sys/device/system/cpu/cpuX/onlineattribute will fail.Do not poke into device internals in
acpi_processor_add()and touch "struct device { .offline }" attribute, since for CPUs onlined at boot it's set by:
topology_init()->arch_register_cpu()->register_cpu()
before ACPI device tree is parsed, and for hotplugged CPUs it's set when userspace onlines CPU via sysfs.
Quoted 2 lines
Signed-off-by Igor Mammedov <[email protected]>Acked-by Toshi Kani <[email protected]>
Would there be a problem if I applied this separately from the rest of the series?
Quoted 21 lines
---
v2:
- fix regression in v1 leading to NULL pointer dereference on CPU unplug, do not remove "pr->dev= dev;"--- drivers/acpi/acpi_processor.c | 1 - 1 files changed, 0 insertions(+), 1 deletions(-)drivers/acpi/acpi_processor.c +0
-1@@ -405,7 +405,6 @@ static int acpi_processor_add(struct acpi_device *device, goto err; pr->dev = dev; - dev->offline = pr->flags.need_hotplug_init;/* Trigger the processor driver's .probe() if present. */ if (device_attach (dev) >=0 )
Signature · 2 lines
I speak only for myself. Rafael J. Wysocki, Intel Open Source Technology Center.
Re: [PATCH v5 2/4] acpi_processor: do not mark present at boot but not onlined CPU as onlined
Quoted 15 lines · Rafael J. Wysocki
Quoted 14 lines · Igor Mammedov
acpi_processor_add()assumes that present at boot CPUs are always onlined, it is not so if a CPU failed to become onlined. As resultacpi_processor_add()will mark such CPU device as onlined in sysfs and following attempts to online/offline it using/sys/device/system/cpu/cpuX/onlineattribute will fail.Do not poke into device internals in
acpi_processor_add()and touch "struct device { .offline }" attribute, since for CPUs onlined at boot it's set by:
topology_init()->arch_register_cpu()->register_cpu()
before ACPI device tree is parsed, and for hotplugged CPUs it's set when userspace onlines CPU via sysfs.
Quoted 5 lines
Quoted 2 lines
Signed-off-by Igor Mammedov <[email protected]>Acked-by Toshi Kani <[email protected]>Would there be a problem if I applied this separately from the rest of the series?
If you push the fix upstream for v3.15 then it would be fine and I could base the other patches on top of your (soon to be upstream) commit.
Thanks,
Ingo
Re: [PATCH v5 2/4] acpi_processor: do not mark present at boot but not onlined CPU as onlined
Quoted 21 lines · Ingo Molnar
Quoted 18 lines · Rafael J. Wysocki
Quoted 16 lines · Igor Mammedov
acpi_processor_add()assumes that present at boot CPUs are always onlined, it is not so if a CPU failed to become onlined. As resultacpi_processor_add()will mark such CPU device as onlined in sysfs and following attempts to online/offline it using/sys/device/system/cpu/cpuX/onlineattribute will fail.Do not poke into device internals in
acpi_processor_add()and touch "struct device { .offline }" attribute, since for CPUs onlined at boot it's set by:
topology_init()->arch_register_cpu()->register_cpu()
before ACPI device tree is parsed, and for hotplugged CPUs it's set when userspace onlines CPU via sysfs.Signed-off-by Igor Mammedov <[email protected]>Acked-by Toshi Kani <[email protected]>
Quoted 6 lines
Quoted 2 lines
Would there be a problem if I applied this separately from the rest of the series?
If you push the fix upstream for v3.15 then it would be fine and I could base the other patches on top of your (soon to be upstream) commit.
OK, I can do that.
We also seem to need this in -stable, right?
Signature · 2 lines
I speak only for myself. Rafael J. Wysocki, Intel Open Source Technology Center.
Re: [PATCH v5 2/4] acpi_processor: do not mark present at boot but not onlined CPU as onlined
Quoted 25 lines · Rafael J. Wysocki
Quoted 24 lines · Ingo Molnar
Quoted 20 lines · Rafael J. Wysocki
Quoted 14 lines · Igor Mammedov
acpi_processor_add()assumes that present at boot CPUs are always onlined, it is not so if a CPU failed to become onlined. As resultacpi_processor_add()will mark such CPU device as onlined in sysfs and following attempts to online/offline it using/sys/device/system/cpu/cpuX/onlineattribute will fail.Do not poke into device internals in
acpi_processor_add()and touch "struct device { .offline }" attribute, since for CPUs onlined at boot it's set by:
topology_init()->arch_register_cpu()->register_cpu()
before ACPI device tree is parsed, and for hotplugged CPUs it's set when userspace onlines CPU via sysfs.Quoted 2 lines
Signed-off-by Igor Mammedov <[email protected]>Acked-by Toshi Kani <[email protected]>Would there be a problem if I applied this separately from the rest of the series?
Quoted 7 lines
Quoted 3 lines
If you push the fix upstream for v3.15 then it would be fine and I could base the other patches on top of your (soon to be upstream) commit.
OK, I can do that.
We also seem to need this in -stable, right?
Yeah, agreed.
Thanks,
Ingo
PATCH v5 3/4 x86: log error on secondary CPU wakeup failure at ERR level
In reply to Igor MammedovIf system is running without debug level logging, it will not log error if do_boot_cpu() failed to wakeup AP. It may lead to silent AP bringup failures at boot time.
Change message level to KERN_ERR to make error visible to user as it's done on other architectures.
--- arch/x86/kernel/smpboot.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
arch/x86/kernel/smpboot.c +1 -1
@@ -918,7 +918,7 @@ int native_cpu_up(unsigned int cpu, struct task_struct *tidle) err =do_boot_cpu (apicid, cpu, tidle);if (err) { -pr_debug ("do_boot_cpu failed %d\n" , err); +pr_err ("do_boot_cpu failed(%d) to wakeup CPU#%u\n" , err, cpu);return -EIO ; }
Signature · 1 line
1.7.1
PATCH v5 4/4 x86: initialize secondary CPU only if master CPU will wait for it
In reply to Igor MammedovHang is observed on virtual machines during CPU hotplug, especially in big guests with many CPUs. (It reproducible more often if host is over-committed).
It happens because master CPU gives up waiting on secondary CPU and allows it to run wild. As result AP causes locking or crashing system. For example as described here: https://lkml.org/lkml/2014/3/6/257
If master CPU have sent STARTUP IPI successfully, and AP signalled to master CPU that it's ready to start initialization, make master CPU wait indefinitely till AP is onlined.
To ensure that AP won't ever run wild, make it wait at early startup till master CPU confirms its intention to wait for AP. If AP doesn't respond in 10 seconds, the master CPU will timeout and cancel AP onlining.
---
v2:
- ammend comment in cpu_init()
v3:
- leave timeouts in do_boot_cpu(), so that master CPU won't hang if AP doesn't respond, use cpu_initialized_mask as a way for AP to signal to master CPU that it's ready to start initialzation.
v4:
- move commont code in cpu_init() for x32/x64 in shared helper function wait_formaster_cpu()
- add WARN_ON(cpumask_test_and_set_cpu(cpu, cpu_initialized_mask)) to wait_formaster_cpu()
v5:
- add smp_mb() after clearing cpu_initialized_mask in do_boot_cpu()
- add 10 sec timeout description into commit message.
--- arch/x86/kernel/cpu/common.c | 27 +++++++----- arch/x86/kernel/smpboot.c | 99 +++++++++++++----------------------------- 2 files changed, 47 insertions(+), 79 deletions(-)
arch/x86/kernel/cpu/common.c +16 -11
@@ -1221,6 +1221,17 @@ static void dbg_restore_debug_regs(void)#define dbg_restore_debug_regs ()#endif /* ! CONFIG_KGDB */ +static void wait_for_master_cpu (int cpu) +{ +/* +* wait for ACK from master CPU before continuing +* with AP initialization +*/ +WARN_ON (cpumask_test_and_set_cpu (cpu, cpu_initialized_mask)); +while (!cpumask_test_cpu (cpu, cpu_callout_mask)) +cpu_relax (); +} +/* * cpu_init() initializes state that is per-CPU. Some data is already * initialized (naturally) in the bootstrap process, such as the GDT
@@ -1236,16 +1247,17 @@ void cpu_init(void)struct task_struct *me;struct tss_struct *t;unsigned long v; -int cpu; +int cpu =stack_smp_processor_id ();int i; +wait_for_master_cpu (cpu); +/* * Load microcode on this cpu if a valid microcode is available. * This is early microcode loading procedure. */ load_ucode_ap (); - cpu =stack_smp_processor_id (); t = &per_cpu (init_tss, cpu); oist = &per_cpu (orig_ist, cpu);
@@ -1257,9 +1269,6 @@ void cpu_init(void) me = current; -if (cpumask_test_and_set_cpu (cpu, cpu_initialized_mask)) -panic ("CPU#%d already initialized!\n" , cpu); -pr_debug ("Initializing CPU#%d\n" , cpu);clear_in_cr4 (X86_CR4_VME |X86_CR4_PVI |X86_CR4_TSD |X86_CR4_DE );
@@ -1336,13 +1345,9 @@ void cpu_init(void)struct tss_struct *t = &per_cpu (init_tss, cpu);struct thread_struct *thread = &curr->thread; -show_ucode_info_early (); +wait_for_master_cpu (cpu); -if (cpumask_test_and_set_cpu (cpu, cpu_initialized_mask)) { -printk (KERN_WARNING "CPU#%d already initialized!\n" , cpu); -for (;;) -local_irq_enable (); - } +show_ucode_info_early ();printk (KERN_INFO "Initializing CPU#%d\n" , cpu);
arch/x86/kernel/smpboot.c +31 -68
144 lines not on this page · raw message
Signature · 1 line
1.7.1
Re: [PATCH v5 4/4] x86: initialize secondary CPU only if master CPU will wait for it
Quoted 19 lines · Igor Mammedov
Hang is observed on virtual machines during CPU hotplug, especially in big guests with many CPUs. (It reproducible more often if host is over-committed).
It happens because master CPU gives up waiting on secondary CPU and allows it to run wild. As result AP causes locking or crashing system. For example as described here: https://lkml.org/lkml/2014/3/6/257
If master CPU have sent STARTUP IPI successfully, and AP signalled to master CPU that it's ready to start initialization, make master CPU wait indefinitely till AP is onlined.
To ensure that AP won't ever run wild, make it wait at early startup till master CPU confirms its intention to wait for AP. If AP doesn't respond in 10 seconds, the master CPU will timeout and cancel AP onlining.
Quoted 1 line
Signed-off-by Igor Mammedov <[email protected]>
Thanks for the update.
-Toshi
Quoted 34 lines · On Mon, 5 May 2014 22:49:47 +0200 Igor Mammedov <[email protected]> wrote:
changes since v4:
* merge "[PATCH v4 1/5] x86: fix list corruption on CPU hotplug" and "[PATCH v4 2/5] x86: fix memory corruption inacpi_unmap_lsapic()" together
* "x86: initialize secondary CPU only if master CPU will wait for it:
- add 10 seconds timeout description into commit message
- addsmp_mb()after clearingcpu_initialized_maskchanges since v3:
* put simple bugfixes first
* move common part of syncing with master CPU incpu_init()for x32/64 variant into helper function
*cpu_init():WARN_ONifcpu_initialized_maskis set
* fix panic on CPU unplug, caused by erroneous removing of "pr->dev= dev;" indrivers/acpi/acpi_processor.c--
Hang is observed on virtual machines during CPU hotplug, especially in big guests with many CPUs. (It happens more often if host is over-committed).Hang happens because master CPU timeouts on waiting till AP boots and 'cancels' CPU online operation assuming AP is not functional but AP may continue run wild later causing various hangs or panics in running kernel that is assuming that AP was offline.
This is an alternative approach, that instead of canceling in-progress AP bringup (https://lkml.org/lkml/2014/3/6/257),
removes timeouts so that AP bringup won't be affected by poor timing and syncs AP with master CPU at early startup making sure that AP won't run wild if master CPU doesn't expect AP to come online.
Quoted 2 lines
Series also fixes 3 bugs found during testing CPU bringup failure case.
since 3.16 merge window is open now,
ping
Quoted 28 lines
--
Below is the detailed description of a more often happening hang:
---
Master CPU may timeout beforecpu_callin_maskis set and cancel booting CPU, but being onlined CPU still continues to boot, setscpu_active_mask(CPU_STARTINGnotifiers) and spins incheck_tsc_sync_target()for master cpu to arrive. Following attempt to online another cpu hangs instop_machine, initiated from here:smp_callin->
smp_store_cpu_info->identify_secondary_cpu -> mtrr_ap_init -> set_mtrr_from_inactive_cpu
stop_machinewaits on completion ofstop_workon all CPUs fromcpu_active_maskincluding a failed CPU that spins incheck_tsc_sync_target().Igor Mammedov (4):
x86: fix list/memory corruption on CPU hotplug
acpi_processor: do not mark present at boot but not onlined CPU as
onlined
x86: log error on secondary CPU wakeup failure at ERR level
x86: initialize secondary CPU only if master CPU will wait for itarch/x86/kernel/cpu/common.c | 27 ++++++---- arch/x86/kernel/smpboot.c | 104 +++++++++++++---------------------------- drivers/acpi/acpi_processor.c | 1 - 3 files changed, 48 insertions(+), 84 deletions(-)
Quoted 5 lines · Igor Mammedov
Quoted 2 lines
Series also fixes 3 bugs found during testing CPU bringup failure case.
since 3.16 merge window is open now,
ping
Mind resending the remaining patches on top of Linus's latest, which I suppose has one of the fixes already included via Rafael's tree?
Thanks,
Ingo
Quoted 11 lines · On Thu, 5 Jun 2014 14:29:40 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 5 lines · Igor Mammedov
Quoted 2 lines
Series also fixes 3 bugs found during testing CPU bringup failure case.
since 3.16 merge window is open now,
pingMind resending the remaining patches on top of Linus's latest, which I suppose has one of the fixes already included via Rafael's tree?
Sure, I'll rebase and repost it today.
Quoted 4 lines
Thanks,
Ingo