x86, apic, kexec, Documentation: Add disable_cpu_apic kernel parameter
Subsystems (4)
PATCH v9 1/1 x86, apic, kexec, Documentation: Add disable_cpu_apic kernel parameter
This patch set is to allow kdump 2nd kernel to wake up multiple CPUs, a continueing work from:
[PATCH v3 0/2] x86, apic, kdump: Disable BSP if boot cpu is AP
https://lkml.org/lkml/2013/10/16/300.
At v4, basic design has changed. Now users need to figure out initial APIC ID of BSP in the 1st kernel and configures kernel parameter for the 2nd kernel manually using disable_cpu_apicid kernel parameter, which is newly introduced by this patch set. This design is more flexible than the previous version in that we no longer have to rely on ACPI/MP table to get initial APIC ID of BSP.
Sorry, this patch set have not include in-source documentation requested by Borislav Petkov yet, but I'll post it later separately, which would be better to focus on documentation reviewing.
ChangeLog
- Rebased on top of v3.13-rc2
- Call read_apic_id() in case of disable_cpu_apiid only in generic_processor_info().
v7 => v8)
- Merge the patch that adds new descption in Documentations/kernel-parameters.txt in the first patch.
- Call read_apic_id() directly in generic_processor_info() instead of overriding global boot_cpu_physical_apicid with the local variable with the same name.
v6 => v7)
- Rebased on top of v3.13-rc1
- Remove a patch that cleans up the code around x86_cpu_physical_apicid. Instead, use read_apic_id() directly in generic_processor_info(). The clean up removed here will be done in different patch set.
v5 => v6)
- Remove use of rdmsr(IA32_APIC_BASE) to initialize bsp_physical_apicid since the MSR doesn't work well on some cluster systems, suggested by HPA. Also, current users of the variable expects the initial apicid reported through MP table only; removing the use of the MSR is not problematic.
- Rename bsp_physical_apicid as bios_bsp_physical_apicid to make it clear that the apidid contained there is the one reported from some BIOS table. Also, initialize it not only by MP table but also by ACPI MADT.
- Change message displayed when specified cpu is disabled from:
ACPI: Disable specified CPU. Processor 0/0x0 ignored.
to:
ACPI: Disabling requested cpu. Processor 0/0x0 ignored.
v4 => v5)
- Rebased on top of v3.12
- Introduce bsp_physical_apicid that has the initial APIC ID for the processor with BSP flag on IA32_APIC_BASE MSR. Without this, boot_cpu_physical_apicid has temporarilly the value around MP table related codes, although it's designed to have the initial APIC ID for the processor that is doing the boot up. Use the bsp_physical_apicid in MP table related codes; no impact on semantics at runtime there.
v3 => v4)
- Rebased on top of v3.12-rc6
- Basic design has been changed. Now users need to figure out initial APIC ID of BSP in the 1st kernel and configures kernel parameter for the 2nd kernel manually using disable_cpu_apic kernel parameter to be newly introduced in this patch set. This design is more flexible than the previous version in that we no longer have to rely on ACPI/MP table to get initial APIC ID of BSP.
v2 => v3)
- Change default value of boot_cpu_is_bsp to true.
- Before executing rdmsr(MSR_IA32_APICBASE), check if the number of processor family is larger than or equal to 6 in order to avoid invalid opcode exception on processors where MSR_IA32_APICBASE is not supported.
v1 => v2)
- Rebased on top of v3.12-rc5.
- Fix linking time error of boot_cpu_is_bsp_init() in case of CONFIG_LOCAL_APIC disabled by adding empty static inline function instead.
- Fix missing feature check by means of cpu_has_apic macro in boot_cpu_is_bsp_init() before calling rdmsr_safe(MSR_IA32_APICBASE).
NOTE: I've checked local apic-present case only; I don't have any x86 processor without local apic.
- Add __init annotation to boot_cpu_is_bsp_init().
Test
- built with and without CONFIG_LOCAL_APIC
- tested x86_64 in case of acpi and MP table
- tested disable_cpu_apicid=<n> to disable both AP and BSP
Quoted 1 line
From
2cd100eb618a36a82bae2db98f0f4b7d3c1dfe89Mon Sep 17 00:00:00 2001
From: HATAYAMA Daisuke <[email protected]>
Date: Tue, 3 Dec 2013 09:21:56 +0900
Subject: [PATCH] x86, apic, kexec, Documentation: Add disable_cpu_apicid kernel parameter
Add disable_cpu_apicid kernel parameter. To use this kernel parameter, specify an initial APIC ID of the corresponding CPU you want to disable.
This is mostly used for the kdump 2nd kernel to disable BSP to wake up multiple CPUs without causing system reset or hang due to sending INIT from AP to BSP.
Kdump users first figure out initial APIC ID of the BSP, CPU0 in the 1st kernel, for example from /proc/cpuinfo and then set up this kernel parameter for the 2nd kernel using the obtained APIC ID.
However, doing this procedure at each boot time manually is awkward, which should be automatically done by user-land service scripts, for example, kexec-tools on fedora/RHEL distributions.
This design is more flexible than disabling BSP in kernel boot time automatically in that in kernel boot time we have no choice but referring to ACPI/MP table to obtain initial APIC ID for BSP, meaning that the method is not applicable to the systems without such BIOS tables.
One assumption behind this design is that users get initial APIC ID of the BSP in still healthy state and so BSP is uniquely kept in CPU0. Thus, through the kernel parameter, only one initial APIC ID can be specified.
In a comparison with disabled_cpu_apicid, we use read_apic_id(), not boot_cpu_physical_apicid, because on some platforms, the variable is modified to the apicid reported as BSP through MP table and this function is executed with the temporarily modified boot_cpu_physical_apicid. As a result, disabled_cpu_apicid kernel parameter doesn't work well for apicids of APs.
Fixing the wrong handling of boot_cpu_physical_apicid requires some reviews and tests beyond some platforms and it could take some time. The fix here is a kind of workaround to focus on the main topic of this patch.
--- Documentation/kernel-parameters.txt | 9 +++++++ arch/x86/kernel/apic/apic.c | 49 +++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)
Documentation/kernel-parameters.txt +9 -0
@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted. disable= [IPV6] See Documentation/networking/ipv6.txt. + disable_cpu_apicid= [X86,APIC,KEXEC,SMP] + Format: <int> + The number of initial APIC ID for the + corresponding CPU to be disabled at boot, + mostly used for the kdump 2nd kernel to + disable BSP to wake up multiple CPUs without + causing system reset or hang due to sending + INIT from AP to BSP. + disable_ddw [PPC/PSERIES] Disable Dynamic DMA Window support. Use this if to workaround buggy firmware.
arch/x86/kernel/apic/apic.c +49 -0
64 lines not on this page · raw message
Signature · 5 lines
1.8.3.1 -- Thanks. HATAYAMA, Daisuke
On Tue, Dec 03, 2013 at 10:32:26AM +0900, HATAYAMA Daisuke wrote:
[..]
Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 777
Quoted 10 lines
Documentation/kernel-parameters.txt +1
-0@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted. disable= [IPV6] See Documentation/networking/ipv6.txt. + disable_cpu_apicid= [X86,APIC,KEXEC,SMP]
Hi Hatayama,
We are almost there. A minor nit. Why have we specified KEXEC here. This parameter disabled_cpu_apicid does not seem to dependon CONFIG_KEXEC?
Jerry, this patch looks good to me. Does it work on your system?
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 12 lines · Vivek Goyal
On Tue, Dec 03, 2013 at 10:32:26AM +0900, HATAYAMA Daisuke wrote:
[..]
Quoted 9 lines
Documentation/kernel-parameters.txt +0
-0@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted. disable= [IPV6] See Documentation/networking/ipv6.txt.
Quoted 9 lines
Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 777
Quoted 1 line
Documentation/kernel-parameters.txt +1
-0+ disable_cpu_apicid= [X86,APIC,KEXEC,SMP]Hi Hatayama,
We are almost there. A minor nit. Why have we specified KEXEC here. This parameter
disabled_cpu_apiciddoes not seem to dependonCONFIG_KEXEC?Jerry, this patch looks good to me. Does it work on your system?
Because primary user for the option is currently kexec/kdump only.
I referred to acpi_rsdp description:
acpi_rsdp= [ACPI,EFI,KEXEC]
Pass the RSDP address to the kernel, mostly used on machines running EFI runtime service to boot the second kernel for kdump.
Signature · 2 lines
Thanks. HATAYAMA, Daisuke
Quoted 23 lines · HATAYAMA Daisuke
Quoted 12 lines · Vivek Goyal
On Tue, Dec 03, 2013 at 10:32:26AM +0900, HATAYAMA Daisuke wrote:
[..]
Quoted 9 lines
Documentation/kernel-parameters.txt +0
-0@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted. disable= [IPV6] See Documentation/networking/ipv6.txt.Quoted 9 lines
Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 777
Quoted 1 line
Documentation/kernel-parameters.txt +1
-0+ disable_cpu_apicid= [X86,APIC,KEXEC,SMP]Hi Hatayama,
We are almost there. A minor nit. Why have we specified KEXEC here. This parameter
disabled_cpu_apiciddoes not seem to dependonCONFIG_KEXEC?Jerry, this patch looks good to me. Does it work on your system?
Quoted 9 lines
Because primary user for the option is currently kexec/kdump only.
I referred to
acpi_rsdpdescription:acpi_rsdp= [ACPI,EFI,KEXEC]
Pass the RSDP address to the kernel, mostly used on machines running EFI runtime service to boot the second kernel for kdump.
Indo-san, who introduced acpi_rsdp and KEXEC tag, told me that historical reason why KEXEC tag was introduced. disable_cpu_apicid is generic at least in the current version, so the tag doesn't need to be specified here.
I'll post a new version soon.
Signature · 2 lines
Thanks. HATAYAMA, Daisuke
Quoted 34 lines · HATAYAMA Daisuke
Quoted 23 lines · HATAYAMA Daisuke
Quoted 12 lines · Vivek Goyal
On Tue, Dec 03, 2013 at 10:32:26AM +0900, HATAYAMA Daisuke wrote:
[..]
Quoted 9 lines
Documentation/kernel-parameters.txt +0
-0@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted. disable= [IPV6] See Documentation/networking/ipv6.txt.Quoted 9 lines
Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 777
Quoted 1 line
Documentation/kernel-parameters.txt +1
-0+ disable_cpu_apicid= [X86,APIC,KEXEC,SMP]Hi Hatayama,
We are almost there. A minor nit. Why have we specified KEXEC here. This parameter
disabled_cpu_apiciddoes not seem to dependonCONFIG_KEXEC?Jerry, this patch looks good to me. Does it work on your system?
Quoted 9 lines
Because primary user for the option is currently kexec/kdump only.
I referred to
acpi_rsdpdescription:acpi_rsdp= [ACPI,EFI,KEXEC]
Pass the RSDP address to the kernel, mostly used on machines running EFI runtime service to boot the second kernel for kdump.
Quoted 3 lines
Indo-san, who introduced
acpi_rsdpand KEXEC tag, told me that historical reason why KEXEC tag was introduced.disable_cpu_apicidis generic at least in the current version, so the tag doesn't need to be specified here.
Also Documentation/kernel-parameters.txt says following.
"The parameters listed below are only valid if certain kernel build options
were enabled and if respective hardware is present. The text in square brackets at the beginning of each description states the restrictions within which a parameter is applicable:"
As above parameter is not dependent on CONFIG_KEXEC, we should remove KEXEC from the brackets.
acpi_rsdp is defined only if CONFIG_KEXEC is present, so there specifying kexec makes sense.
#ifdef CONFIG_KEXEC
static unsigned long acpi_rsdp;
static int __init setup_acpi_rsdp(char *arg)
{
acpi_rsdp = simple_strtoul(arg, NULL, 16);
return 0;
}early_param("acpi_rsdp", setup_acpi_rsdp);
#endif
Thanks
Vivek
Quoted 6 lines
I'll post a new version soon.
Signature · 2 lines
Thanks. HATAYAMA, Daisuke
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 10 lines · Vivek Goyal
Hi Hatayama,
We are almost there. A minor nit. Why have we specified KEXEC here. This parameter
disabled_cpu_apiciddoes not seem to dependonCONFIG_KEXEC?Jerry, this patch looks good to me. Does it work on your system?
Thanks
Vivek
Vivek, Hatayama,
I've back ported v9 of this patch to 2.6.32 and 3.0.80 based kernels to test with existing distros.
I've tested on our smaller prototype server specifying nr_cpus=8/maxcpus=8 to the capture kernel. One hundred iterations (echo c > /proc/sysrq-trigger) varying target cpu and system load to each kernel.
The 2.6.32 based distro kernel showed the < 5% soft lockup (still unresolved) during boot of capture kernel. This is something i've seen on all versions of the patch that i've tested.
The 3.0.80 based distro kernel has had zero failures.
I have not had a chance to test upstream kernels or on our larger prototype configuration.
We still plan to test on our larger prototype. Testing of prior versions of the patch on the larger systems didn't show problems w/ this functionality and I don't anticipate we'll find anything this time either.
I am okay with this patch being accepted upstream and working the intermittent 2.6.32 failures separately.
Jerry
Signature · 7 lines
----------------------------------------------------------------------------
Jerry Hoemann Software Engineer Hewlett-Packard
3404 E Harmony Rd. MS 57 phone: (970) 898-1022
Ft. Collins, CO 80528 FAX: (970) 898-XXXX
email: [email protected]
----------------------------------------------------------------------------A message that is not in the archive (sent off-list or not yet received): [email protected]
Sorry if you're getting multiple copies, but i have had problems with my subscription to the kexec mailing list and am resending.
Quoted 10 lines · Vivek Goyal
Hi Hatayama,
We are almost there. A minor nit. Why have we specified KEXEC here. This parameter
disabled_cpu_apiciddoes not seem to dependonCONFIG_KEXEC?Jerry, this patch looks good to me. Does it work on your system?
Thanks
Vivek
Vivek, Hatayama,
I've back ported v9 of this patch to 2.6.32 and 3.0.80 based kernels to test with existing distros.
I've tested on our smaller prototype server specifying nr_cpus=8/maxcpus=8 to the capture kernel. One hundred iterations (echo c > /proc/sysrq-trigger) varying target cpu and system load to each kernel.
The 2.6.32 based distro kernel showed the < 5% soft lockup (still unresolved) during boot of capture kernel. This is something i've seen on all versions of the patch that i've tested.
The 3.0.80 based distro kernel has had zero failures.
I have not had a chance to test upstream kernels or on our larger prototype configuration.
We still plan to test on our larger prototype. Testing of prior versions of the patch on the larger systems didn't show problems w/ this functionality and I don't anticipate we'll find anything this time either.
I am okay with this patch being accepted upstream and working the intermittent 2.6.32 failures separately.
Jerry
Signature · 14 lines
----------------------------------------------------------------------------
Jerry Hoemann Software Engineer Hewlett-Packard
3404 E Harmony Rd. MS 57 phone: (970) 898-1022
Ft. Collins, CO 80528 FAX: (970) 898-XXXX
email: [email protected]
----------------------------------------------------------------------------
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexecQuoted 42 lines · Jerry
Sorry if you're getting multiple copies, but i have had problems with my subscription to the kexec mailing list and am resending.
Quoted 10 lines · Vivek Goyal
Hi Hatayama,
We are almost there. A minor nit. Why have we specified KEXEC here. This parameter
disabled_cpu_apiciddoes not seem to dependonCONFIG_KEXEC?Jerry, this patch looks good to me. Does it work on your system?
Thanks
VivekVivek, Hatayama,
I've back ported v9 of this patch to 2.6.32 and 3.0.80 based kernels to test with existing distros.
I've tested on our smaller prototype server specifying nr_cpus=8/maxcpus=8 to the capture kernel. One hundred iterations (echo c >
/proc/sysrq-trigger) varying target cpu and system load to each kernel.The 2.6.32 based distro kernel showed the < 5% soft lockup (still unresolved) during boot of capture kernel. This is something i've seen on all versions of the patch that i've tested.
The 3.0.80 based distro kernel has had zero failures.
I have not had a chance to test upstream kernels or on our larger prototype configuration.
We still plan to test on our larger prototype. Testing of prior versions of the patch on the larger systems didn't show problems w/ this functionality and I don't anticipate we'll find anything this time either.
Quoted 6 lines
I am okay with this patch being accepted upstream and working the intermittent 2.6.32 failures separately.
Jerry
Another update, I have tested our max cpu configuration prototype, with a fairly large IO config with the backported 3.0.80 kernel Jerry mentions above with the v9 patch backported, and this system got 7 out of 7 successful dumps, with max_cpus=8, so far in testing. This system has a a fairly large IO configuration too, such that intermittently booting the crashkernel with 1 cpu the crashkernel boot would hang due
to IRQ for system disk not getting assigned. So this is early
indication that this patch is working well on larger IO configurations.
I plan to test with the 2.6.32 base with v9 patch over the weekend on
this large configuration. The system takes much longer to dump than
the minimum config, so it is harder to build up as many iterations over a short time, plus I have to share the system with others.
Thanks,
Lisa Mitchell
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 59 lines · Lisa Mitchell
Quoted 42 lines · Jerry
Sorry if you're getting multiple copies, but i have had problems with my subscription to the kexec mailing list and am resending.
Quoted 10 lines · Vivek Goyal
Hi Hatayama,
We are almost there. A minor nit. Why have we specified KEXEC here. This parameter
disabled_cpu_apiciddoes not seem to dependonCONFIG_KEXEC?Jerry, this patch looks good to me. Does it work on your system?
Thanks
VivekVivek, Hatayama,
I've back ported v9 of this patch to 2.6.32 and 3.0.80 based kernels to test with existing distros.
I've tested on our smaller prototype server specifying nr_cpus=8/maxcpus=8 to the capture kernel. One hundred iterations (echo c >
/proc/sysrq-trigger) varying target cpu and system load to each kernel.The 2.6.32 based distro kernel showed the < 5% soft lockup (still unresolved) during boot of capture kernel. This is something i've seen on all versions of the patch that i've tested.
The 3.0.80 based distro kernel has had zero failures.
I have not had a chance to test upstream kernels or on our larger prototype configuration.
We still plan to test on our larger prototype. Testing of prior versions of the patch on the larger systems didn't show problems w/ this functionality and I don't anticipate we'll find anything this time either.
Quoted 6 lines
I am okay with this patch being accepted upstream and working the intermittent 2.6.32 failures separately.
Jerry
Another update, I have tested our max cpu configuration prototype, with a fairly large IO config with the backported 3.0.80 kernel Jerry mentions above with the v9 patch backported, and this system got 7 out of 7 successful dumps, with max_cpus=8, so far in testing. This system has a a fairly large IO configuration too, such that intermittently booting the crashkernel with 1 cpu the crashkernel boot would hang due
to IRQ for system disk not getting assigned. So this is early
indication that this patch is working well on larger IO configurations.
Quoted 9 lines
I plan to test with the 2.6.32 base with v9 patch over the weekend on
this large configuration. The system takes much longer to dump than
the minimum config, so it is harder to build up as many iterations over a short time, plus I have to share the system with others.Thanks,
Lisa Mitchell
I have now tested the version 9 patch on a 2.6.32 base on the max cpu/large IO configuration, and got 8 out of 8 successful dumps, (7 with nr_cpus=8, and one with nr_cpus=16).
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec