Skip to contentSkip to conversation

x86, apic, kexec, Documentation: Add disable_cpu_apic kernel parameter

PATCH Superseded HATAYAMA Daisuke 4 participants 9 messages Dec 3, 2013 – Dec 9, 2013 1 message not in the archive

Subsystems (4)

v9 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

PATCH v9 1/1 x86, apic, kexec, Documentation: Add disable_cpu_apic kernel parameter

This patch set is to allow kdump 2nd kernel to wake up multiple CPUs, a continueing work from:

[PATCH v3 0/2] x86, apic, kdump: Disable BSP if boot cpu is AP
https://lkml.org/lkml/2013/10/16/300.

At v4, basic design has changed. Now users need to figure out initial APIC ID of BSP in the 1st kernel and configures kernel parameter for the 2nd kernel manually using disable_cpu_apicid kernel parameter, which is newly introduced by this patch set. This design is more flexible than the previous version in that we no longer have to rely on ACPI/MP table to get initial APIC ID of BSP.

Sorry, this patch set have not include in-source documentation requested by Borislav Petkov yet, but I'll post it later separately, which would be better to focus on documentation reviewing.

ChangeLog

- Rebased on top of v3.13-rc2

- Call read_apic_id() in case of disable_cpu_apiid only in generic_processor_info().

v7 => v8)

- Merge the patch that adds new descption in Documentations/kernel-parameters.txt in the first patch.

- Call read_apic_id() directly in generic_processor_info() instead of overriding global boot_cpu_physical_apicid with the local variable with the same name.

v6 => v7)

- Rebased on top of v3.13-rc1

- Remove a patch that cleans up the code around x86_cpu_physical_apicid. Instead, use read_apic_id() directly in generic_processor_info(). The clean up removed here will be done in different patch set.

v5 => v6)

- Remove use of rdmsr(IA32_APIC_BASE) to initialize bsp_physical_apicid since the MSR doesn't work well on some cluster systems, suggested by HPA. Also, current users of the variable expects the initial apicid reported through MP table only; removing the use of the MSR is not problematic.

- Rename bsp_physical_apicid as bios_bsp_physical_apicid to make it clear that the apidid contained there is the one reported from some BIOS table. Also, initialize it not only by MP table but also by ACPI MADT.

- Change message displayed when specified cpu is disabled from:

ACPI: Disable specified CPU. Processor 0/0x0 ignored.

to:

ACPI: Disabling requested cpu. Processor 0/0x0 ignored.

v4 => v5)

- Rebased on top of v3.12

- Introduce bsp_physical_apicid that has the initial APIC ID for the processor with BSP flag on IA32_APIC_BASE MSR. Without this, boot_cpu_physical_apicid has temporarilly the value around MP table related codes, although it's designed to have the initial APIC ID for the processor that is doing the boot up. Use the bsp_physical_apicid in MP table related codes; no impact on semantics at runtime there.

v3 => v4)

- Rebased on top of v3.12-rc6

- Basic design has been changed. Now users need to figure out initial APIC ID of BSP in the 1st kernel and configures kernel parameter for the 2nd kernel manually using disable_cpu_apic kernel parameter to be newly introduced in this patch set. This design is more flexible than the previous version in that we no longer have to rely on ACPI/MP table to get initial APIC ID of BSP.

v2 => v3)

- Change default value of boot_cpu_is_bsp to true.

- Before executing rdmsr(MSR_IA32_APICBASE), check if the number of processor family is larger than or equal to 6 in order to avoid invalid opcode exception on processors where MSR_IA32_APICBASE is not supported.

v1 => v2)

- Rebased on top of v3.12-rc5.

- Fix linking time error of boot_cpu_is_bsp_init() in case of CONFIG_LOCAL_APIC disabled by adding empty static inline function instead.

- Fix missing feature check by means of cpu_has_apic macro in boot_cpu_is_bsp_init() before calling rdmsr_safe(MSR_IA32_APICBASE).

NOTE: I've checked local apic-present case only; I don't have any x86 processor without local apic.

- Add __init annotation to boot_cpu_is_bsp_init().

Test

- built with and without CONFIG_LOCAL_APIC
- tested x86_64 in case of acpi and MP table
- tested disable_cpu_apicid=<n> to disable both AP and BSP

Quoted 1 line

From 2cd100eb618a36a82bae2db98f0f4b7d3c1dfe89 Mon Sep 17 00:00:00 2001

From: HATAYAMA Daisuke <[email protected]>
Date: Tue, 3 Dec 2013 09:21:56 +0900
Subject: [PATCH] x86, apic, kexec, Documentation: Add disable_cpu_apicid kernel parameter

Add disable_cpu_apicid kernel parameter. To use this kernel parameter, specify an initial APIC ID of the corresponding CPU you want to disable.

This is mostly used for the kdump 2nd kernel to disable BSP to wake up multiple CPUs without causing system reset or hang due to sending INIT from AP to BSP.

Kdump users first figure out initial APIC ID of the BSP, CPU0 in the 1st kernel, for example from /proc/cpuinfo and then set up this kernel parameter for the 2nd kernel using the obtained APIC ID.

However, doing this procedure at each boot time manually is awkward, which should be automatically done by user-land service scripts, for example, kexec-tools on fedora/RHEL distributions.

This design is more flexible than disabling BSP in kernel boot time automatically in that in kernel boot time we have no choice but referring to ACPI/MP table to obtain initial APIC ID for BSP, meaning that the method is not applicable to the systems without such BIOS tables.

One assumption behind this design is that users get initial APIC ID of the BSP in still healthy state and so BSP is uniquely kept in CPU0. Thus, through the kernel parameter, only one initial APIC ID can be specified.

In a comparison with disabled_cpu_apicid, we use read_apic_id(), not boot_cpu_physical_apicid, because on some platforms, the variable is modified to the apicid reported as BSP through MP table and this function is executed with the temporarily modified boot_cpu_physical_apicid. As a result, disabled_cpu_apicid kernel parameter doesn't work well for apicids of APs.

Fixing the wrong handling of boot_cpu_physical_apicid requires some reviews and tests beyond some platforms and it could take some time. The fix here is a kind of workaround to focus on the main topic of this patch.

---
 Documentation/kernel-parameters.txt |  9 +++++++
 arch/x86/kernel/apic/apic.c         | 49 +++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
Documentation/kernel-parameters.txt +9 -0
index 50680a5..dd77bec 100644
@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 	disable=	[IPV6]
 			See Documentation/networking/ipv6.txt.
 
+	disable_cpu_apicid= [X86,APIC,KEXEC,SMP]
+			Format: <int>
+			The number of initial APIC ID for the
+			corresponding CPU to be disabled at boot,
+			mostly used for the kdump 2nd kernel to
+			disable BSP to wake up multiple CPUs without
+			causing system reset or hang due to sending
+			INIT from AP to BSP.
+
 	disable_ddw     [PPC/PSERIES]
 			Disable Dynamic DMA Window support. Use this if
 			to workaround buggy firmware.
arch/x86/kernel/apic/apic.c +49 -0

64 lines not on this page · raw message

Signature · 5 lines
1.8.3.1

-- 
Thanks.
HATAYAMA, Daisuke
#1 lore raw

On Tue, Dec 03, 2013 at 10:32:26AM +0900, HATAYAMA Daisuke wrote:

[..]

Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 777

Quoted 10 lines
Documentation/kernel-parameters.txt +1 -0
index 50680a5..dd77bec 100644
@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 	disable=	[IPV6]
 			See Documentation/networking/ipv6.txt.
 
+	disable_cpu_apicid= [X86,APIC,KEXEC,SMP]

Hi Hatayama,

We are almost there. A minor nit. Why have we specified KEXEC here. This parameter disabled_cpu_apicid does not seem to dependon CONFIG_KEXEC?

Jerry, this patch looks good to me. Does it work on your system?

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw
Quoted 12 lines · Vivek Goyal

On Tue, Dec 03, 2013 at 10:32:26AM +0900, HATAYAMA Daisuke wrote:

[..]

Quoted 9 lines
Documentation/kernel-parameters.txt +0 -0
index 50680a5..dd77bec 100644
@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
  	disable=	[IPV6]
  			See Documentation/networking/ipv6.txt.
 
Quoted 9 lines

Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 777

Quoted 1 line
Documentation/kernel-parameters.txt +1 -0
+	disable_cpu_apicid= [X86,APIC,KEXEC,SMP]

Hi Hatayama,

We are almost there. A minor nit. Why have we specified KEXEC here. This parameter disabled_cpu_apicid does not seem to dependon CONFIG_KEXEC?

Jerry, this patch looks good to me. Does it work on your system?

Because primary user for the option is currently kexec/kdump only.

I referred to acpi_rsdp description:

acpi_rsdp= [ACPI,EFI,KEXEC]
Pass the RSDP address to the kernel, mostly used on machines running EFI runtime service to boot the second kernel for kdump.

Signature · 2 lines
Thanks.
HATAYAMA, Daisuke
#3 lore raw
Quoted 23 lines · HATAYAMA Daisuke
Quoted 12 lines · Vivek Goyal

On Tue, Dec 03, 2013 at 10:32:26AM +0900, HATAYAMA Daisuke wrote:

[..]

Quoted 9 lines
Documentation/kernel-parameters.txt +0 -0
index 50680a5..dd77bec 100644
@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
      disable=    [IPV6]
              See Documentation/networking/ipv6.txt.
 
Quoted 9 lines

Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 777

Quoted 1 line
Documentation/kernel-parameters.txt +1 -0
+    disable_cpu_apicid= [X86,APIC,KEXEC,SMP]

Hi Hatayama,

We are almost there. A minor nit. Why have we specified KEXEC here. This parameter disabled_cpu_apicid does not seem to dependon CONFIG_KEXEC?

Jerry, this patch looks good to me. Does it work on your system?

Quoted 9 lines

Because primary user for the option is currently kexec/kdump only.

I referred to acpi_rsdp description:

acpi_rsdp= [ACPI,EFI,KEXEC]
Pass the RSDP address to the kernel, mostly used on machines running EFI runtime service to boot the second kernel for kdump.

Indo-san, who introduced acpi_rsdp and KEXEC tag, told me that historical reason why KEXEC tag was introduced. disable_cpu_apicid is generic at least in the current version, so the tag doesn't need to be specified here.

I'll post a new version soon.

Signature · 2 lines
Thanks.
HATAYAMA, Daisuke
#4 lore raw
Quoted 34 lines · HATAYAMA Daisuke
Quoted 23 lines · HATAYAMA Daisuke
Quoted 12 lines · Vivek Goyal

On Tue, Dec 03, 2013 at 10:32:26AM +0900, HATAYAMA Daisuke wrote:

[..]

Quoted 9 lines
Documentation/kernel-parameters.txt +0 -0
index 50680a5..dd77bec 100644
@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
     disable=    [IPV6]
             See Documentation/networking/ipv6.txt.
 
Quoted 9 lines

Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 777

Quoted 1 line
Documentation/kernel-parameters.txt +1 -0
+    disable_cpu_apicid= [X86,APIC,KEXEC,SMP]

Hi Hatayama,

We are almost there. A minor nit. Why have we specified KEXEC here. This parameter disabled_cpu_apicid does not seem to dependon CONFIG_KEXEC?

Jerry, this patch looks good to me. Does it work on your system?

Quoted 9 lines

Because primary user for the option is currently kexec/kdump only.

I referred to acpi_rsdp description:

acpi_rsdp= [ACPI,EFI,KEXEC]
Pass the RSDP address to the kernel, mostly used on machines running EFI runtime service to boot the second kernel for kdump.

Quoted 3 lines

Indo-san, who introduced acpi_rsdp and KEXEC tag, told me that historical reason why KEXEC tag was introduced. disable_cpu_apicid is generic at least in the current version, so the tag doesn't need to be specified here.

Also Documentation/kernel-parameters.txt says following.

"The parameters listed below are only valid if certain kernel build options
were enabled and if respective hardware is present. The text in square brackets at the beginning of each description states the restrictions within which a parameter is applicable:"

As above parameter is not dependent on CONFIG_KEXEC, we should remove KEXEC from the brackets.

acpi_rsdp is defined only if CONFIG_KEXEC is present, so there specifying kexec makes sense.

#ifdef CONFIG_KEXEC
static unsigned long acpi_rsdp;
static int __init setup_acpi_rsdp(char *arg)

{
        acpi_rsdp = simple_strtoul(arg, NULL, 16);
        return 0;
}

early_param("acpi_rsdp", setup_acpi_rsdp);
#endif

Thanks
Vivek

Quoted 6 lines

I'll post a new version soon.

Signature · 2 lines
Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#5 lore raw
In reply to Vivek Goyal
Quoted 10 lines · Vivek Goyal

Hi Hatayama,

We are almost there. A minor nit. Why have we specified KEXEC here. This parameter disabled_cpu_apicid does not seem to dependon CONFIG_KEXEC?

Jerry, this patch looks good to me. Does it work on your system?

Thanks
Vivek

Vivek, Hatayama,

I've back ported v9 of this patch to 2.6.32 and 3.0.80 based kernels to test with existing distros.

I've tested on our smaller prototype server specifying nr_cpus=8/maxcpus=8 to the capture kernel. One hundred iterations (echo c > /proc/sysrq-trigger) varying target cpu and system load to each kernel.

The 2.6.32 based distro kernel showed the < 5% soft lockup (still unresolved) during boot of capture kernel. This is something i've seen on all versions of the patch that i've tested.

The 3.0.80 based distro kernel has had zero failures.

I have not had a chance to test upstream kernels or on our larger prototype configuration.

We still plan to test on our larger prototype. Testing of prior versions of the patch on the larger systems didn't show problems w/ this functionality and I don't anticipate we'll find anything this time either.

I am okay with this patch being accepted upstream and working the intermittent 2.6.32 failures separately.

Jerry

Signature · 7 lines
----------------------------------------------------------------------------
Jerry Hoemann            Software Engineer              Hewlett-Packard

3404 E Harmony Rd. MS 57                        phone:  (970) 898-1022
Ft. Collins, CO 80528                           FAX:    (970) 898-XXXX
                                                email:  [email protected]
----------------------------------------------------------------------------
#7 lore raw
In reply to a message not in the archive

Sorry if you're getting multiple copies, but i have had problems with my subscription to the kexec mailing list and am resending.

Quoted 10 lines · Vivek Goyal

Hi Hatayama,

We are almost there. A minor nit. Why have we specified KEXEC here. This parameter disabled_cpu_apicid does not seem to dependon CONFIG_KEXEC?

Jerry, this patch looks good to me. Does it work on your system?

Thanks
Vivek

Vivek, Hatayama,

I've back ported v9 of this patch to 2.6.32 and 3.0.80 based kernels to test with existing distros.

I've tested on our smaller prototype server specifying nr_cpus=8/maxcpus=8 to the capture kernel. One hundred iterations (echo c > /proc/sysrq-trigger) varying target cpu and system load to each kernel.

The 2.6.32 based distro kernel showed the < 5% soft lockup (still unresolved) during boot of capture kernel. This is something i've seen on all versions of the patch that i've tested.

The 3.0.80 based distro kernel has had zero failures.

I have not had a chance to test upstream kernels or on our larger prototype configuration.

We still plan to test on our larger prototype. Testing of prior versions of the patch on the larger systems didn't show problems w/ this functionality and I don't anticipate we'll find anything this time either.

I am okay with this patch being accepted upstream and working the intermittent 2.6.32 failures separately.

Jerry

Signature · 14 lines
----------------------------------------------------------------------------
Jerry Hoemann            Software Engineer              Hewlett-Packard

3404 E Harmony Rd. MS 57                        phone:  (970) 898-1022
Ft. Collins, CO 80528                           FAX:    (970) 898-XXXX
                                                email:  [email protected]
----------------------------------------------------------------------------



_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#8 lore raw
Quoted 42 lines · Jerry

Sorry if you're getting multiple copies, but i have had problems with my subscription to the kexec mailing list and am resending.

Quoted 10 lines · Vivek Goyal

Hi Hatayama,

We are almost there. A minor nit. Why have we specified KEXEC here. This parameter disabled_cpu_apicid does not seem to dependon CONFIG_KEXEC?

Jerry, this patch looks good to me. Does it work on your system?

Thanks
Vivek

Vivek, Hatayama,

I've back ported v9 of this patch to 2.6.32 and 3.0.80 based kernels to test with existing distros.

I've tested on our smaller prototype server specifying nr_cpus=8/maxcpus=8 to the capture kernel. One hundred iterations (echo c > /proc/sysrq-trigger) varying target cpu and system load to each kernel.

The 2.6.32 based distro kernel showed the < 5% soft lockup (still unresolved) during boot of capture kernel. This is something i've seen on all versions of the patch that i've tested.

The 3.0.80 based distro kernel has had zero failures.

I have not had a chance to test upstream kernels or on our larger prototype configuration.

We still plan to test on our larger prototype. Testing of prior versions of the patch on the larger systems didn't show problems w/ this functionality and I don't anticipate we'll find anything this time either.

Quoted 6 lines

I am okay with this patch being accepted upstream and working the intermittent 2.6.32 failures separately.

Jerry

Another update, I have tested our max cpu configuration prototype, with a fairly large IO config with the backported 3.0.80 kernel Jerry mentions above with the v9 patch backported, and this system got 7 out of 7 successful dumps, with max_cpus=8, so far in testing. This system has a a fairly large IO configuration too, such that intermittently booting the crashkernel with 1 cpu the crashkernel boot would hang due
to IRQ for system disk not getting assigned. So this is early
indication that this patch is working well on larger IO configurations.

I plan to test with the 2.6.32 base with v9 patch over the weekend on
this large configuration. The system takes much longer to dump than
the minimum config, so it is harder to build up as many iterations over a short time, plus I have to share the system with others.

Thanks,

Lisa Mitchell

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw
Quoted 59 lines · Lisa Mitchell
Quoted 42 lines · Jerry

Sorry if you're getting multiple copies, but i have had problems with my subscription to the kexec mailing list and am resending.

Quoted 10 lines · Vivek Goyal

Hi Hatayama,

We are almost there. A minor nit. Why have we specified KEXEC here. This parameter disabled_cpu_apicid does not seem to dependon CONFIG_KEXEC?

Jerry, this patch looks good to me. Does it work on your system?

Thanks
Vivek

Vivek, Hatayama,

I've back ported v9 of this patch to 2.6.32 and 3.0.80 based kernels to test with existing distros.

I've tested on our smaller prototype server specifying nr_cpus=8/maxcpus=8 to the capture kernel. One hundred iterations (echo c > /proc/sysrq-trigger) varying target cpu and system load to each kernel.

The 2.6.32 based distro kernel showed the < 5% soft lockup (still unresolved) during boot of capture kernel. This is something i've seen on all versions of the patch that i've tested.

The 3.0.80 based distro kernel has had zero failures.

I have not had a chance to test upstream kernels or on our larger prototype configuration.

We still plan to test on our larger prototype. Testing of prior versions of the patch on the larger systems didn't show problems w/ this functionality and I don't anticipate we'll find anything this time either.

Quoted 6 lines

I am okay with this patch being accepted upstream and working the intermittent 2.6.32 failures separately.

Jerry

Another update, I have tested our max cpu configuration prototype, with a fairly large IO config with the backported 3.0.80 kernel Jerry mentions above with the v9 patch backported, and this system got 7 out of 7 successful dumps, with max_cpus=8, so far in testing. This system has a a fairly large IO configuration too, such that intermittently booting the crashkernel with 1 cpu the crashkernel boot would hang due
to IRQ for system disk not getting assigned. So this is early
indication that this patch is working well on larger IO configurations.

Quoted 9 lines

I plan to test with the 2.6.32 base with v9 patch over the weekend on
this large configuration. The system takes much longer to dump than
the minimum config, so it is harder to build up as many iterations over a short time, plus I have to share the system with others.

Thanks,

Lisa Mitchell

I have now tested the version 9 patch on a 2.6.32 base on the max cpu/large IO configuration, and got 8 out of 8 successful dumps, (7 with nr_cpus=8, and one with nr_cpus=16).

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec