Skip to contentSkip to conversation

kdump: Enter 2nd kernel with BSP for enabling multiple CPUs

PATCH In review HATAYAMA Daisuke 6 participants 13 messages Apr 16, 2012 – Apr 19, 2013 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover kdump: Enter 2nd kernel with BSP for enabling multiple CPUs

Currently, booting up 2nd kernel with multiple CPUs fails in most cases since it enters 2nd kernel with AP if the crash happens on the AP. The problem is to signal startup IPI from AP to BSP. Typical result of the operation I saw is the machine hanging during the 2nd kernel boot.

To solve this issue, always enter 2nd kernel with BSP. To do this, I modify logic for shooting down CPUs. I use simple existing logic only in this mechanism, not complicating crash path to machine_kexec().

I did stress tests about 100 in total on the processors below:

Intel(R) Xeon(R) CPU E7- 4820 @ 2.00GHz Socket x 4, Core x 8, Thread x 16 (160 LCPUS in total)

Intel(R) Xeon(R) CPU E7- 8870 @ 2.40GHz Socket x 8, Core x 10, Thread x 20 (64 LCPUS in total)

* Motivation of enabling multiple CPUs on the 2nd kernel

This patch is aimed at doing parallel compression on the 2nd kernel. The machine that has more than tera bytes memory requires several hours to generate crash dump.

There are several ways to reduce generation time of crash time, but they have different pros and cons:

Fast I/O devices
pros
- Can obtain high-speed stably
cons
- Big financial cost for good performance I/O devices. It's difficult financially to prepare these for all environments as dump devices.

Filtering
pros
- No financial cost.
- Large reduction of crash dump size

cons
- Some data is definitely lost. So, we cannot use this on some situations:

1) High availability configuration where application triggers OS to crash and users want to debug the application later by retrieving the application's user process image from the system's crash dump.

2) KVM virtualization configuration where KVM host machine contains KVM guest machine images as user processes.

3) Page cache is needed for debugging filesystem related bugs.

Compression
pros
- No financial cost.
- No data lost.

cons
- Compression doesn't always reduce crash dump size.
- take heavy CPU time. Slow if CPU is weak in speed.

Machines with large memory tend to have a lot of CPUs. Parallel compression is sutable for parallel processing. My goal is to make compression as for free as possible.

* TODO

- Extend 512MB limit of reserved memory size for 2nd kernel for multiple CPUs.

- Intel microcode patch loading on the 2nd kenrel is slow for the 2nd and later CPUs: about one or more minutes per one CPU.

- There are a limited number of irq vectors for TLB flush IPI on
x86_64: 32 for recent 3.x kernels and 8 for around 2.6.x kernels. So compression doesn't scale if a lot of page reclaim happens when reading kernel image larger than memory. Special handling without page cache could be applicable to parallel dump mechanism, but more investigation is needed.

---

HATAYAMA Daisuke (2):
Enter 2nd kernel with BSP
Introduce crash ipi helpers to wait for APs to stop

 arch/x86/include/asm/reboot.h |    4 +++
 arch/x86/kernel/crash.c       |   15 +++++++++-
 arch/x86/kernel/reboot.c      |   63 +++++++++++++++++++++++++++++------------
 3 files changed, 62 insertions(+), 20 deletions(-)
Signature · 6 lines
HATAYAMA Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#1 lore raw

PATCH 1/2 Introduce crash ipi helpers to wait for APs to stop

Introduce crash ipi helpers to use them from BSP and AP sides in common.

There's no logical change in this patch.

---

 arch/x86/include/asm/reboot.h |    4 +++
 arch/x86/kernel/reboot.c      |   53 ++++++++++++++++++++++++++++++++---------
 2 files changed, 45 insertions(+), 12 deletions(-)
arch/x86/include/asm/reboot.h +4 -0
index 92f29706..2f8e9e7 100644
@@ -26,4 +26,8 @@ void machine_real_restart(unsigned int type);
 typedef void (*nmi_shootdown_cb)(int, struct pt_regs*);
 void nmi_shootdown_cpus(nmi_shootdown_cb callback);
 
+void crash_ipi_init(void);
+void crash_ipi_dec_and_halt(void);
+void crash_ipi_wait_for_APs(void);
+
 #endif /* _ASM_X86_REBOOT_H */
arch/x86/kernel/reboot.c +41 -12

87 lines not on this page · raw message

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw

PATCH 2/2 Enter 2nd kernel with BSP

In reply to HATAYAMA Daisuke

Split logic into BSP's and AP's: BSP waits for AP halting.

Don't remove variable crashing_cpu for debugging use; useful for determining one what CPU crash happens.

---

 arch/x86/kernel/crash.c  |   15 ++++++++++++++-
 arch/x86/kernel/reboot.c |   16 ++++++----------
 2 files changed, 20 insertions(+), 11 deletions(-)
arch/x86/kernel/crash.c +14 -1
index 13ad899..c5c19fa 100644
@@ -83,9 +83,14 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 	 * In practice this means shooting down the other cpus in
 	 * an SMP system.
 	 */
+
+	int cpu;
+
 	/* The kernel is broken so disable interrupts */
 	local_irq_disable();
 
+	crash_ipi_init();
+
 	kdump_nmi_shootdown_cpus();
 
 	/* Booting kdump kernel with VMX or SVM enabled won't work,
@@ -102,5 +107,13 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 #ifdef CONFIG_HPET_TIMER
 	hpet_disable();
 #endif
-	crash_save_cpu(regs, safe_smp_processor_id());
+	cpu = safe_smp_processor_id();
+	crash_save_cpu(regs, cpu);
+
+	if (cpu_physical_id(cpu) == boot_cpu_physical_apicid) {
+		crash_ipi_wait_for_APs();
+		return;
+	}
+
+	crash_ipi_dec_and_halt();
 }
arch/x86/kernel/reboot.c +6 -10
index 6dd77a8..90354f9 100644
@@ -7,6 +7,7 @@
 #include <linux/sched.h>
 #include <linux/tboot.h>
 #include <linux/delay.h>
+#include <linux/kexec.h>
 #include <acpi/reboot.h>
 #include <asm/io.h>
 #include <asm/apic.h>
@@ -800,16 +801,15 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs)
 
 	cpu = raw_smp_processor_id();
 
-	/* Don't do anything if this handler is invoked on crashing cpu.
-	 * Otherwise, system will completely hang. Crashing cpu can get
-	 * an NMI if system was initially booted with nmi_watchdog parameter.
-	 */
-	if (cpu == crashing_cpu)
-		return NMI_HANDLED;
 	local_irq_disable();
 
 	shootdown_callback(cpu, regs);
 
+	if (cpu_physical_id(cpu) == boot_cpu_physical_apicid) {
+		crash_ipi_wait_for_APs();
+		machine_kexec(kexec_crash_image);
+	}
+
 	crash_ipi_dec_and_halt();
 
 	return NMI_HANDLED;
@@ -835,8 +835,6 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback)
 
 	shootdown_callback = callback;
 
-	crash_ipi_init();
-
 	/* Would it be better to replace the trap vector here? */
 	if (register_nmi_handler(NMI_LOCAL, crash_nmi_callback,
 				 NMI_FLAG_FIRST, "crash"))
@@ -848,8 +846,6 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback)
 
 	smp_send_nmi_allbutself();
 
-	crash_ipi_wait_for_APs();
-
 	/* Leave the nmi callback set */
 }
 #else /* !CONFIG_SMP */

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw

Re: [PATCH 2/2] Enter 2nd kernel with BSP

Quoted 5 lines

Split logic into BSP's and AP's: BSP waits for AP halting.

Don't remove variable crashing_cpu for debugging use; useful for determining one what CPU crash happens.

Fenghua has this patchkit to allow offlining the BSP. What happens then?

It would be good to understand why initializing APs from other APs should not work. Not aware of such a limitation in x86.

AFAIK when we online a random cpu after boot there is no effort to do it from BSP. Why does that work and not from the kexec boot?

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
#4 lore raw

Re: [PATCH 2/2] Enter 2nd kernel with BSP

From: Andi Kleen <[email protected]>
Subject: Re: [PATCH 2/2] Enter 2nd kernel with BSP
Date: Mon, 23 Apr 2012 03:49:15 -0700

Quoted 5 lines
Quoted 4 lines

Split logic into BSP's and AP's: BSP waits for AP halting.

Don't remove variable crashing_cpu for debugging use; useful for determining one what CPU crash happens.

Quoted 8 lines

Fenghua has this patchkit to allow offlining the BSP. What happens then?

It would be good to understand why initializing APs from other APs should not work. Not aware of such a limitation in x86.

AFAIK when we online a random cpu after boot there is no effort to do it from BSP. Why does that work and not from the kexec boot?

Hello Andi,

I have yet to test what I explain here, so I'm sorry I say these with ``would''.

APs to APs case would be entirely my misunderstanding. The problematic case would be APs to BSP case only.

native_cpu_up avoids to wake up cpu with boot_cpu_physical_apicid, but the boot_cpu_physical_apicid would be just a boot cpu now, which could be non-BSP on the 2nd kernel; equal to crashing cpu on the 1st kernel. It would need to check explicitly whether a given cpu is BSP or not and this would be better fix here. This is suggested by Eric.

Thanks.
HATAYAMA, Daisuke

#5 lore raw

RE: [PATCH 2/2] Enter 2nd kernel with BSP

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

Split logic into BSP's and AP's: BSP waits for AP halting.

Don't remove variable crashing_cpu for debugging use; useful for determining one what CPU crash happens.

Fenghua has this patchkit to allow offlining the BSP. What happens

then?

I'm fixing one issue when doing stress BSP online/offline test. I'll send out a newer patch set pretty soon.

Thanks.

-Fenghua

#6 lore raw

Re: [PATCH 2/2] Enter 2nd kernel with BSP

In reply to HATAYAMA Daisuke
Quoted 1 line

native_cpu_up avoids to wake up cpu with boot_cpu_physical_apicid, but

Ok so that check needs to go then. I wonder why it is there.

Quoted 4 lines

the boot_cpu_physical_apicid would be just a boot cpu now, which could be non-BSP on the 2nd kernel; equal to crashing cpu on the 1st kernel. It would need to check explicitly whether a given cpu is BSP or not and this would be better fix here. This is suggested by Eric.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#7 lore raw

Re: [PATCH 2/2] Enter 2nd kernel with BSP

Quoted 3 lines · Andi Kleen
Quoted 1 line

native_cpu_up avoids to wake up cpu with boot_cpu_physical_apicid, but

Ok so that check needs to go then. I wonder why it is there.

As I remember we had a check to avoid waking up the current cpu we are running on, and there was something about the value for the current cpu coming from BIOS tables etc. Because of all of the silliness in how we get the apicid for the current cpu I seem to remember the case where we start all of our secondary processors to periodically regress.

Quoted 4 lines
Quoted 4 lines

the boot_cpu_physical_apicid would be just a boot cpu now, which could be non-BSP on the 2nd kernel; equal to crashing cpu on the 1st kernel. It would need to check explicitly whether a given cpu is BSP or not and this would be better fix here. This is suggested by Eric.

Yes. My two reasons for designing the crash path the way I did was that we might have offlined/hot-unplugged the boot cpu, or the bootcpu might have gotten itself into a terrible state. Since we want as much reliability in kexec on panic we don't bother trying to restore it.

The normal reboot path, and a normal kexec tries to reboot on the boot cpu primarily because there are a lot of BIOS's out there that if you talk to them on anything other than the boot cpu they get confused.

Eric

#8 lore raw

Re: [PATCH 2/2] Enter 2nd kernel with BSP

In reply to HATAYAMA Daisuke

HATAYAMA Daisuke <[email protected]>

Quoted 4 lines · writes:

Split logic into BSP's and AP's: BSP waits for AP halting.

Don't remove variable crashing_cpu for debugging use; useful for determining one what CPU crash happens.

Fenghua has this patchkit to allow offlining the BSP. What happens then?

It would be good to understand why initializing APs from other APs should not work. Not aware of such a limitation in x86.

AFAIK when we online a random cpu after boot there is no effort to do it from BSP. Why does that work and not from the kexec boot?

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw

From: "Eric W. Biederman" <[email protected]>
Subject: Re: [PATCH 0/2] kdump: Enter 2nd kernel with BSP for enabling multiple CPUs
Date: Sun, 15 Apr 2012 19:59:52 -0700

Quoted 9 lines
Quoted 3 lines · HATAYAMA Daisuke

Currently, booting up 2nd kernel with multiple CPUs fails in most cases since it enters 2nd kernel with AP if the crash happens on the AP. The problem is to signal startup IPI from AP to BSP.

If so, then we need to fix our code that sends startupIPIs. And perhaps the code that attempts to shutdown the other cpus.

Now maxcpus=1 is set at default, in which configuration, 2nd kernel doesn't try to wake up secondary and later CPUs. So there's no startup IPI on the 2nd kernel now.

Quoted 1 line

It is not ok to switch cpus during kdump (reducing the reliability) just so you can write crash dumps faster. Better would be to cope with secondary cpus not booting.

Even the current implememntation uses NMI to stop other CPUs. The reliability you are concerned about here is the possibility where the non-crashing BSP doesn't go into machine_kexec() due to some failures of interrupt processing, right?

Alternative idea is:
1) try to go into 2nd kernel with BSP,
2) after some seconds, then try to go into 2nd kernel with crashing CPU. Then, think of CPUs except for the crashing cpu as abnormal, and use crashing cpu only on the 2nd kernel.

This seems as reliable as the current one.

Quoted 3 lines

I do like the direction of pounding on things so we can get multiple cpus going in large configurations. Although I am surprised you are cpu bound and not disk bound in the time to write your crash dumps.

What do you indicate in the 1st sentence? I don't understand around ``pounding on thing'', sorry.

For the 2nd, it depends on data. If data is sparse enough, the data size is significantly reduced, and so IO size is also reduced. If data is randomized enough, compression takes much time and the data remains the same size, resulting in cpu bound processing.

Thanks.
HATAYAMA, Daisuke

#11 lore raw
In reply to HATAYAMA Daisuke

So, the reason why you want to have multiple CPU's enabled in the 2nd kernel is to speed up the compression of the core dump?

The first question is that, why the speed is important? Given the fact that the whole kdump progress happens automatically nowdays, there should be very few guys waiting for a kdump to complete, so the speed is not that important.

Second, currently we use nr_cpus=1 for the 2nd kernel on RHEL6, to reduce the memory usage in the 2nd kernel. You mentioned 512M is a limit, but we want to make it even less, even 512M is not a good choice for us on x86. Bringing up more than 1 CPU will of course need more memory in the 2nd kernel.

The limit is not only the size, but also the max address of loading initrd, which is 896M on x86 IIRC. A contiguos memory area larger than 512M usually sit above 869M, due to the fragmentation in the lower memory, so I am afraid you need to do (much?) more work.

So, I am afraid you spend too much effort to fix a not-that-important issue, but I may miss something here...

Thanks.

#12 lore raw
In reply to HATAYAMA Daisuke
Quoted 9 lines · On Mon, 16 Apr 2012 11:21:28 +0900 HATAYAMA Daisuke…

Currently, booting up 2nd kernel with multiple CPUs fails in most cases since it enters 2nd kernel with AP if the crash happens on the AP. The problem is to signal startup IPI from AP to BSP. Typical result of the operation I saw is the machine hanging during the 2nd kernel boot.

To solve this issue, always enter 2nd kernel with BSP. To do this, I modify logic for shooting down CPUs. I use simple existing logic only in this mechanism, not complicating crash path to machine_kexec().

These patches looked pretty good. I seem to recall that Fenghua (from Intel) had an alternative solution for booting from AP. Unfortunately I can't find his mails in my kexec mailbox...

Anyway, what's the latest upstream status?

Petr

Quoted 84 lines

I did stress tests about 100 in total on the processors below:

Intel(R) Xeon(R) CPU E7- 4820 @ 2.00GHz Socket x 4, Core x 8, Thread x 16 (160 LCPUS in total)

Intel(R) Xeon(R) CPU E7- 8870 @ 2.40GHz Socket x 8, Core x 10, Thread x 20 (64 LCPUS in total)

* Motivation of enabling multiple CPUs on the 2nd kernel

This patch is aimed at doing parallel compression on the 2nd kernel. The machine that has more than tera bytes memory requires several hours to generate crash dump.

There are several ways to reduce generation time of crash time, but they have different pros and cons:

Fast I/O devices
pros
- Can obtain high-speed stably
cons
- Big financial cost for good performance I/O devices. It's difficult financially to prepare these for all environments as dump devices.

Filtering
pros
- No financial cost.
- Large reduction of crash dump size

cons
- Some data is definitely lost. So, we cannot use this on some situations:

1) High availability configuration where application triggers OS to crash and users want to debug the application later by retrieving the application's user process image from the system's crash dump.

2) KVM virtualization configuration where KVM host machine contains KVM guest machine images as user processes.

3) Page cache is needed for debugging filesystem related bugs.

Compression
pros
- No financial cost.
- No data lost.

cons
- Compression doesn't always reduce crash dump size.
- take heavy CPU time. Slow if CPU is weak in speed.

Machines with large memory tend to have a lot of CPUs. Parallel compression is sutable for parallel processing. My goal is to make compression as for free as possible.

* TODO

- Extend 512MB limit of reserved memory size for 2nd kernel for multiple CPUs.

- Intel microcode patch loading on the 2nd kenrel is slow for the 2nd and later CPUs: about one or more minutes per one CPU.

- There are a limited number of irq vectors for TLB flush IPI on
x86_64: 32 for recent 3.x kernels and 8 for around 2.6.x kernels. So compression doesn't scale if a lot of page reclaim happens when reading kernel image larger than memory. Special handling without page cache could be applicable to parallel dump mechanism, but more investigation is needed.

---

HATAYAMA Daisuke (2):
Enter 2nd kernel with BSP
Introduce crash ipi helpers to wait for APs to stop

 arch/x86/include/asm/reboot.h |    4 +++
 arch/x86/kernel/crash.c       |   15 +++++++++-
 arch/x86/kernel/reboot.c      |   63 +++++++++++++++++++++++++++++------------
 3 files changed, 62 insertions(+), 20 deletions(-)
#13 lore raw
Quoted 13 lines · Petr Tesarik
Quoted 9 lines · On Mon, 16 Apr 2012 11:21:28 +0900 HATAYAMA Daisuke…

Currently, booting up 2nd kernel with multiple CPUs fails in most cases since it enters 2nd kernel with AP if the crash happens on the AP. The problem is to signal startup IPI from AP to BSP. Typical result of the operation I saw is the machine hanging during the 2nd kernel boot.

To solve this issue, always enter 2nd kernel with BSP. To do this, I modify logic for shooting down CPUs. I use simple existing logic only in this mechanism, not complicating crash path to machine_kexec().

Quoted 5 lines

These patches looked pretty good. I seem to recall that Fenghua (from Intel) had an alternative solution for booting from AP. Unfortunately I can't find his mails in my kexec mailbox...

Anyway, what's the latest upstream status?

It's still in experimental state.

The patch itself was nacked by Erick since switching the CPU that entered 2nd kenrel through NMI reduced reliability of kdump.

At the discussion of my 2nd patch set that tried to reset BSP flag at boot on the 2nd kernel, Erick suggested that BSP flag could be changed at runtime and then behaviour when INIT was received varied and first we should discuss how unsetting BSP flag affects system.

I'm now going in this direction and the patch I posted a month ago is:

[PATCH] x86, apic: Add unset_bsp parameter to unset BSP flag at boot time
https://lkml.org/lkml/2013/3/18/107

According to Fenghua, some kind of firmware assumes that BSP flag is being kept throughout system is running. I have yet to see difference of behaviour when unsetting BSP flag on top of the patch on my machine. I think this is system dependent and it might be better to assign each user to decide whether to unset BSP flag or not.

BTW, the work of software cpu hotplug for BSP by Fenghua is orthogonal to my case. His work is for system including firmware that is affected if BSP flag is unset and assumes healthy system that cpu#0 is always BSP. On the other hand, our case is for crash kernel and we can no longer assume cpu#0 is BSP and can no longer use NMI to wake up other CPUs since we cannot use logic that depends on the state of CPUs sleeping in the 1st kernel.

Signature · 2 lines
Thanks.
HATAYAMA, Daisuke