kdump: Enter 2nd kernel with BSP for enabling multiple CPUs
Subsystems (3)
cover kdump: Enter 2nd kernel with BSP for enabling multiple CPUs
Currently, booting up 2nd kernel with multiple CPUs fails in most cases since it enters 2nd kernel with AP if the crash happens on the AP. The problem is to signal startup IPI from AP to BSP. Typical result of the operation I saw is the machine hanging during the 2nd kernel boot.
To solve this issue, always enter 2nd kernel with BSP. To do this, I modify logic for shooting down CPUs. I use simple existing logic only in this mechanism, not complicating crash path to machine_kexec().
I did stress tests about 100 in total on the processors below:
Intel(R) Xeon(R) CPU E7- 4820 @ 2.00GHz Socket x 4, Core x 8, Thread x 16 (160 LCPUS in total)
Intel(R) Xeon(R) CPU E7- 8870 @ 2.40GHz Socket x 8, Core x 10, Thread x 20 (64 LCPUS in total)
* Motivation of enabling multiple CPUs on the 2nd kernel
This patch is aimed at doing parallel compression on the 2nd kernel. The machine that has more than tera bytes memory requires several hours to generate crash dump.
There are several ways to reduce generation time of crash time, but they have different pros and cons:
Fast I/O devices
pros
- Can obtain high-speed stably
cons
- Big financial cost for good performance I/O devices. It's difficult financially to prepare these for all environments as dump devices.
Filtering
pros
- No financial cost.
- Large reduction of crash dump size
cons
- Some data is definitely lost. So, we cannot use this on some situations:
1) High availability configuration where application triggers OS to crash and users want to debug the application later by retrieving the application's user process image from the system's crash dump.
2) KVM virtualization configuration where KVM host machine contains KVM guest machine images as user processes.
3) Page cache is needed for debugging filesystem related bugs.
Compression
pros
- No financial cost.
- No data lost.
cons
- Compression doesn't always reduce crash dump size.
- take heavy CPU time. Slow if CPU is weak in speed.
Machines with large memory tend to have a lot of CPUs. Parallel compression is sutable for parallel processing. My goal is to make compression as for free as possible.
* TODO
- Extend 512MB limit of reserved memory size for 2nd kernel for multiple CPUs.
- Intel microcode patch loading on the 2nd kenrel is slow for the 2nd and later CPUs: about one or more minutes per one CPU.
- There are a limited number of irq vectors for TLB flush IPI on
x86_64: 32 for recent 3.x kernels and 8 for around 2.6.x kernels. So compression doesn't scale if a lot of page reclaim happens when reading kernel image larger than memory. Special handling without page cache could be applicable to parallel dump mechanism, but more investigation is needed.
---
HATAYAMA Daisuke (2):
Enter 2nd kernel with BSP
Introduce crash ipi helpers to wait for APs to stop
arch/x86/include/asm/reboot.h | 4 +++ arch/x86/kernel/crash.c | 15 +++++++++- arch/x86/kernel/reboot.c | 63 +++++++++++++++++++++++++++++------------ 3 files changed, 62 insertions(+), 20 deletions(-)
Signature · 6 lines
HATAYAMA Daisuke _______________________________________________ kexec mailing list [email protected] http://lists.infradead.org/mailman/listinfo/kexec
PATCH 1/2 Introduce crash ipi helpers to wait for APs to stop
Introduce crash ipi helpers to use them from BSP and AP sides in common.
There's no logical change in this patch.
---
arch/x86/include/asm/reboot.h | 4 +++ arch/x86/kernel/reboot.c | 53 ++++++++++++++++++++++++++++++++--------- 2 files changed, 45 insertions(+), 12 deletions(-)
arch/x86/include/asm/reboot.h +4 -0
@@ -26,4 +26,8 @@ void machine_real_restart(unsigned int type);typedef void (*nmi_shootdown_cb )(int ,struct pt_regs *);void nmi_shootdown_cpus (nmi_shootdown_cb callback); +void crash_ipi_init (void ); +void crash_ipi_dec_and_halt (void ); +void crash_ipi_wait_for_APs (void ); +#endif /* _ASM_X86_REBOOT_H */
arch/x86/kernel/reboot.c +41 -12
87 lines not on this page · raw message
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH 2/2 Enter 2nd kernel with BSP
In reply to HATAYAMA DaisukeSplit logic into BSP's and AP's: BSP waits for AP halting.
Don't remove variable crashing_cpu for debugging use; useful for determining one what CPU crash happens.
---
arch/x86/kernel/crash.c | 15 ++++++++++++++- arch/x86/kernel/reboot.c | 16 ++++++---------- 2 files changed, 20 insertions(+), 11 deletions(-)
arch/x86/kernel/crash.c +14 -1
@@ -83,9 +83,14 @@ void native_machine_crash_shutdown(struct pt_regs *regs)* In practice this means shooting down the other cpus in * an SMP system. */ + +int cpu; +/* The kernel is broken so disable interrupts */ local_irq_disable (); +crash_ipi_init (); +kdump_nmi_shootdown_cpus ();/* Booting kdump kernel with VMX or SVM enabled won't work,
@@ -102,5 +107,13 @@ void native_machine_crash_shutdown(struct pt_regs *regs)#ifdef CONFIG_HPET_TIMER hpet_disable ();#endif -crash_save_cpu (regs,safe_smp_processor_id ()); + cpu =safe_smp_processor_id (); +crash_save_cpu (regs, cpu); + +if (cpu_physical_id (cpu) == boot_cpu_physical_apicid) { +crash_ipi_wait_for_APs (); +return ; + } + +crash_ipi_dec_and_halt (); }
arch/x86/kernel/reboot.c +6 -10
@@ -7,6 +7,7 @@#include <linux/sched.h> #include <linux/tboot.h> #include <linux/delay.h> +#include <linux/kexec.h> #include <acpi/reboot.h> #include <asm/io.h> #include <asm/apic.h>
@@ -800,16 +801,15 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs) cpu =raw_smp_processor_id (); -/* Don't do anything if this handler is invoked on crashing cpu. -* Otherwise, system will completely hang. Crashing cpu can get -* an NMI if system was initially booted with nmi_watchdog parameter. -*/ -if (cpu == crashing_cpu) -return NMI_HANDLED ;local_irq_disable ();shootdown_callback (cpu, regs); +if (cpu_physical_id (cpu) == boot_cpu_physical_apicid) { +crash_ipi_wait_for_APs (); +machine_kexec (kexec_crash_image); + } +crash_ipi_dec_and_halt ();return NMI_HANDLED ;
@@ -835,8 +835,6 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback) shootdown_callback = callback; -crash_ipi_init (); -/* Would it be better to replace the trap vector here? */ if (register_nmi_handler (NMI_LOCAL , crash_nmi_callback,NMI_FLAG_FIRST ,"crash" ))
@@ -848,8 +846,6 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback)smp_send_nmi_allbutself (); -crash_ipi_wait_for_APs (); - /* Leave the nmi callback set */ }#else /* !CONFIG_SMP */
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH 2/2] Enter 2nd kernel with BSP
Quoted 5 lines
Split logic into BSP's and AP's: BSP waits for AP halting.
Don't remove variable
crashing_cpufor debugging use; useful for determining one what CPU crash happens.
Fenghua has this patchkit to allow offlining the BSP. What happens then?
It would be good to understand why initializing APs from other APs should not work. Not aware of such a limitation in x86.
AFAIK when we online a random cpu after boot there is no effort to do it from BSP. Why does that work and not from the kexec boot?
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Re: [PATCH 2/2] Enter 2nd kernel with BSP
From: Andi Kleen <[email protected]>
Subject: Re: [PATCH 2/2] Enter 2nd kernel with BSP
Date: Mon, 23 Apr 2012 03:49:15 -0700
Quoted 5 lines
Quoted 4 lines
Split logic into BSP's and AP's: BSP waits for AP halting.
Don't remove variable
crashing_cpufor debugging use; useful for determining one what CPU crash happens.
Quoted 8 lines
Fenghua has this patchkit to allow offlining the BSP. What happens then?
It would be good to understand why initializing APs from other APs should not work. Not aware of such a limitation in x86.
AFAIK when we online a random cpu after boot there is no effort to do it from BSP. Why does that work and not from the kexec boot?
Hello Andi,
I have yet to test what I explain here, so I'm sorry I say these with ``would''.
APs to APs case would be entirely my misunderstanding. The problematic case would be APs to BSP case only.
native_cpu_up avoids to wake up cpu with boot_cpu_physical_apicid, but the boot_cpu_physical_apicid would be just a boot cpu now, which could be non-BSP on the 2nd kernel; equal to crashing cpu on the 1st kernel. It would need to check explicitly whether a given cpu is BSP or not and this would be better fix here. This is suggested by Eric.
Thanks.
HATAYAMA, Daisuke
RE: [PATCH 2/2] Enter 2nd kernel with BSP
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Split logic into BSP's and AP's: BSP waits for AP halting.
Don't remove variable
crashing_cpufor debugging use; useful for determining one what CPU crash happens.Fenghua has this patchkit to allow offlining the BSP. What happens
then?
I'm fixing one issue when doing stress BSP online/offline test. I'll send out a newer patch set pretty soon.
Thanks.
-Fenghua
Re: [PATCH 2/2] Enter 2nd kernel with BSP
In reply to HATAYAMA DaisukeQuoted 1 line
native_cpu_upavoids to wake up cpu withboot_cpu_physical_apicid, but
Ok so that check needs to go then. I wonder why it is there.
Quoted 4 lines
the
boot_cpu_physical_apicidwould be just a boot cpu now, which could be non-BSP on the 2nd kernel; equal to crashing cpu on the 1st kernel. It would need to check explicitly whether a given cpu is BSP or not and this would be better fix here. This is suggested by Eric.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [PATCH 2/2] Enter 2nd kernel with BSP
Quoted 3 lines · Andi Kleen
Quoted 1 line
native_cpu_upavoids to wake up cpu withboot_cpu_physical_apicid, butOk so that check needs to go then. I wonder why it is there.
As I remember we had a check to avoid waking up the current cpu we are running on, and there was something about the value for the current cpu coming from BIOS tables etc. Because of all of the silliness in how we get the apicid for the current cpu I seem to remember the case where we start all of our secondary processors to periodically regress.
Quoted 4 lines
Quoted 4 lines
the
boot_cpu_physical_apicidwould be just a boot cpu now, which could be non-BSP on the 2nd kernel; equal to crashing cpu on the 1st kernel. It would need to check explicitly whether a given cpu is BSP or not and this would be better fix here. This is suggested by Eric.
Yes. My two reasons for designing the crash path the way I did was that we might have offlined/hot-unplugged the boot cpu, or the bootcpu might have gotten itself into a terrible state. Since we want as much reliability in kexec on panic we don't bother trying to restore it.
The normal reboot path, and a normal kexec tries to reboot on the boot cpu primarily because there are a lot of BIOS's out there that if you talk to them on anything other than the boot cpu they get confused.
Eric
Re: [PATCH 2/2] Enter 2nd kernel with BSP
In reply to HATAYAMA DaisukeHATAYAMA Daisuke <[email protected]>
Quoted 4 lines · writes:
Split logic into BSP's and AP's: BSP waits for AP halting.
Don't remove variable
crashing_cpufor debugging use; useful for determining one what CPU crash happens.
Fenghua has this patchkit to allow offlining the BSP. What happens then?
It would be good to understand why initializing APs from other APs should not work. Not aware of such a limitation in x86.
AFAIK when we online a random cpu after boot there is no effort to do it from BSP. Why does that work and not from the kexec boot?
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
A message that is not in the archive (sent off-list or not yet received): [email protected]
From: "Eric W. Biederman" <[email protected]>
Subject: Re: [PATCH 0/2] kdump: Enter 2nd kernel with BSP for enabling multiple CPUs
Date: Sun, 15 Apr 2012 19:59:52 -0700
Quoted 9 lines
Quoted 3 lines · HATAYAMA Daisuke
Currently, booting up 2nd kernel with multiple CPUs fails in most cases since it enters 2nd kernel with AP if the crash happens on the AP. The problem is to signal startup IPI from AP to BSP.
If so, then we need to fix our code that sends startupIPIs. And perhaps the code that attempts to shutdown the other cpus.
Now maxcpus=1 is set at default, in which configuration, 2nd kernel doesn't try to wake up secondary and later CPUs. So there's no startup IPI on the 2nd kernel now.
Quoted 1 line
It is not ok to switch cpus during kdump (reducing the reliability) just so you can write crash dumps faster. Better would be to cope with secondary cpus not booting.
Even the current implememntation uses NMI to stop other CPUs. The reliability you are concerned about here is the possibility where the non-crashing BSP doesn't go into machine_kexec() due to some failures of interrupt processing, right?
Alternative idea is:
1) try to go into 2nd kernel with BSP,
2) after some seconds, then try to go into 2nd kernel with crashing CPU. Then, think of CPUs except for the crashing cpu as abnormal, and use crashing cpu only on the 2nd kernel.
This seems as reliable as the current one.
Quoted 3 lines
I do like the direction of pounding on things so we can get multiple cpus going in large configurations. Although I am surprised you are cpu bound and not disk bound in the time to write your crash dumps.
What do you indicate in the 1st sentence? I don't understand around ``pounding on thing'', sorry.
For the 2nd, it depends on data. If data is sparse enough, the data size is significantly reduced, and so IO size is also reduced. If data is randomized enough, compression takes much time and the data remains the same size, resulting in cpu bound processing.
Thanks.
HATAYAMA, Daisuke
So, the reason why you want to have multiple CPU's enabled in the 2nd kernel is to speed up the compression of the core dump?
The first question is that, why the speed is important? Given the fact that the whole kdump progress happens automatically nowdays, there should be very few guys waiting for a kdump to complete, so the speed is not that important.
Second, currently we use nr_cpus=1 for the 2nd kernel on RHEL6, to reduce the memory usage in the 2nd kernel. You mentioned 512M is a limit, but we want to make it even less, even 512M is not a good choice for us on x86. Bringing up more than 1 CPU will of course need more memory in the 2nd kernel.
The limit is not only the size, but also the max address of loading initrd, which is 896M on x86 IIRC. A contiguos memory area larger than 512M usually sit above 869M, due to the fragmentation in the lower memory, so I am afraid you need to do (much?) more work.
So, I am afraid you spend too much effort to fix a not-that-important issue, but I may miss something here...
Thanks.
Quoted 9 lines · On Mon, 16 Apr 2012 11:21:28 +0900 HATAYAMA Daisuke…
Currently, booting up 2nd kernel with multiple CPUs fails in most cases since it enters 2nd kernel with AP if the crash happens on the AP. The problem is to signal startup IPI from AP to BSP. Typical result of the operation I saw is the machine hanging during the 2nd kernel boot.
To solve this issue, always enter 2nd kernel with BSP. To do this, I modify logic for shooting down CPUs. I use simple existing logic only in this mechanism, not complicating crash path to
machine_kexec().
These patches looked pretty good. I seem to recall that Fenghua (from Intel) had an alternative solution for booting from AP. Unfortunately I can't find his mails in my kexec mailbox...
Anyway, what's the latest upstream status?
Petr
Quoted 84 lines
I did stress tests about 100 in total on the processors below:
Intel(R) Xeon(R) CPU E7- 4820 @ 2.00GHz Socket x 4, Core x 8, Thread x 16 (160 LCPUS in total)
Intel(R) Xeon(R) CPU E7- 8870 @ 2.40GHz Socket x 8, Core x 10, Thread x 20 (64 LCPUS in total)
* Motivation of enabling multiple CPUs on the 2nd kernel
This patch is aimed at doing parallel compression on the 2nd kernel. The machine that has more than tera bytes memory requires several hours to generate crash dump.
There are several ways to reduce generation time of crash time, but they have different pros and cons:
Fast I/O devices
pros
- Can obtain high-speed stably
cons
- Big financial cost for good performance I/O devices. It's difficult financially to prepare these for all environments as dump devices.Filtering
pros
- No financial cost.
- Large reduction of crash dump sizecons
- Some data is definitely lost. So, we cannot use this on some situations:1) High availability configuration where application triggers OS to crash and users want to debug the application later by retrieving the application's user process image from the system's crash dump.
2) KVM virtualization configuration where KVM host machine contains KVM guest machine images as user processes.
3) Page cache is needed for debugging filesystem related bugs.
Compression
pros
- No financial cost.
- No data lost.cons
- Compression doesn't always reduce crash dump size.
- take heavy CPU time. Slow if CPU is weak in speed.Machines with large memory tend to have a lot of CPUs. Parallel compression is sutable for parallel processing. My goal is to make compression as for free as possible.
* TODO
- Extend 512MB limit of reserved memory size for 2nd kernel for multiple CPUs.
- Intel microcode patch loading on the 2nd kenrel is slow for the 2nd and later CPUs: about one or more minutes per one CPU.
- There are a limited number of irq vectors for TLB flush IPI on
x86_64: 32 for recent 3.x kernels and 8 for around 2.6.x kernels. So compression doesn't scale if a lot of page reclaim happens when reading kernel image larger than memory. Special handling without page cache could be applicable to parallel dump mechanism, but more investigation is needed.---
HATAYAMA Daisuke (2):
Enter 2nd kernel with BSP
Introduce crash ipi helpers to wait for APs to stoparch/x86/include/asm/reboot.h | 4 +++ arch/x86/kernel/crash.c | 15 +++++++++- arch/x86/kernel/reboot.c | 63 +++++++++++++++++++++++++++++------------ 3 files changed, 62 insertions(+), 20 deletions(-)
Quoted 13 lines · Petr Tesarik
Quoted 9 lines · On Mon, 16 Apr 2012 11:21:28 +0900 HATAYAMA Daisuke…
Currently, booting up 2nd kernel with multiple CPUs fails in most cases since it enters 2nd kernel with AP if the crash happens on the AP. The problem is to signal startup IPI from AP to BSP. Typical result of the operation I saw is the machine hanging during the 2nd kernel boot.
To solve this issue, always enter 2nd kernel with BSP. To do this, I modify logic for shooting down CPUs. I use simple existing logic only in this mechanism, not complicating crash path to
machine_kexec().
Quoted 5 lines
These patches looked pretty good. I seem to recall that Fenghua (from Intel) had an alternative solution for booting from AP. Unfortunately I can't find his mails in my kexec mailbox...
Anyway, what's the latest upstream status?
It's still in experimental state.
The patch itself was nacked by Erick since switching the CPU that entered 2nd kenrel through NMI reduced reliability of kdump.
At the discussion of my 2nd patch set that tried to reset BSP flag at boot on the 2nd kernel, Erick suggested that BSP flag could be changed at runtime and then behaviour when INIT was received varied and first we should discuss how unsetting BSP flag affects system.
I'm now going in this direction and the patch I posted a month ago is:
[PATCH] x86, apic: Add unset_bsp parameter to unset BSP flag at boot time
https://lkml.org/lkml/2013/3/18/107
According to Fenghua, some kind of firmware assumes that BSP flag is being kept throughout system is running. I have yet to see difference of behaviour when unsetting BSP flag on top of the patch on my machine. I think this is system dependent and it might be better to assign each user to decide whether to unset BSP flag or not.
BTW, the work of software cpu hotplug for BSP by Fenghua is orthogonal to my case. His work is for system including firmware that is affected if BSP flag is unset and assumes healthy system that cpu#0 is always BSP. On the other hand, our case is for crash kernel and we can no longer assume cpu#0 is BSP and can no longer use NMI to wake up other CPUs since we cannot use logic that depends on the state of CPUs sleeping in the 1st kernel.
Signature · 2 lines
Thanks. HATAYAMA, Daisuke