x86, apic, kexec: Add disable_cpu_apic kernel parameter
Subsystems (6)
v6 cover x86, apic, kexec: Add disable_cpu_apic kernel parameter
This patch set is to allow kdump 2nd kernel to wake up multiple CPUs, a continueing work from:
[PATCH v3 0/2] x86, apic, kdump: Disable BSP if boot cpu is AP
https://lkml.org/lkml/2013/10/16/300.
At v4, basic design has changed. Now users need to figure out initial APIC ID of BSP in the 1st kernel and configures kernel parameter for the 2nd kernel manually using disable_cpu_apic kernel parameter, which is newly introduced by this patch set. This design is more flexible than the previous version in that we no longer have to rely on ACPI/MP table to get initial APIC ID of BSP.
Sorry, this patch set have not include in-source documentation requested by Borislav Petkov yet, but I'll post it later separately, which would be better to focus on documentation reviewing.
ChangeLog
v5 => v6)
- Remove use of rdmsr(IA32_APIC_BASE) to initialize bsp_physical_apicid since the MSR doesn't work well on some cluster systems, suggested by HPA. Also, current users of the variable expects the initial apicid reported through MP table only; removing the use of the MSR is not problematic.
- Rename bsp_physical_apicid as bios_bsp_physical_apicid to make it clear that the apidid contained there is the one reported from some BIOS table. Also, initialize it not only by MP table but also by ACPI MADT.
- Change message displayed when specified cpu is disabled from:
ACPI: Disable specified CPU. Processor 0/0x0 ignored.
to:
ACPI: Disabling requested cpu. Processor 0/0x0 ignored.
v4 => v5)
- Rebased on top of v3.12
- Introduce bsp_physical_apicid that has the initial APIC ID for the processor with BSP flag on IA32_APIC_BASE MSR. Without this, boot_cpu_physical_apicid has temporarilly the value around MP table related codes, although it's designed to have the initial APIC ID for the processor that is doing the boot up. Use the bsp_physical_apicid in MP table related codes; no impact on semantics at runtime there.
v3 => v4)
- Rebased on top of v3.12-rc6
- Basic design has been changed. Now users need to figure out initial APIC ID of BSP in the 1st kernel and configures kernel parameter for the 2nd kernel manually using disable_cpu_apic kernel parameter to be newly introduced in this patch set. This design is more flexible than the previous version in that we no longer have to rely on ACPI/MP table to get initial APIC ID of BSP.
v2 => v3)
- Change default value of boot_cpu_is_bsp to true.
- Before executing rdmsr(MSR_IA32_APICBASE), check if the number of processor family is larger than or equal to 6 in order to avoid invalid opcode exception on processors where MSR_IA32_APICBASE is not supported.
v1 => v2)
- Rebased on top of v3.12-rc5.
- Fix linking time error of boot_cpu_is_bsp_init() in case of CONFIG_LOCAL_APIC disabled by adding empty static inline function instead.
- Fix missing feature check by means of cpu_has_apic macro in boot_cpu_is_bsp_init() before calling rdmsr_safe(MSR_IA32_APICBASE).
NOTE: I've checked local apic-present case only; I don't have any x86 processor without local apic.
- Add __init annotation to boot_cpu_is_bsp_init().
Test
- built with and without CONFIG_LOCAL_APIC
- tested x86_64 in case of acpi and MP table
- tested disable_cpu_apicid=<n> to disable both AP and BSP
---
HATAYAMA Daisuke (3):
x86, apic: add bios_bsp_physical_apicid
x86, apic: Add disable_cpu_apicid kernel parameter Documentation, x86, apic, kexec: Add disable_cpu_apicid kernel parameter
Documentation/kernel-parameters.txt | 9 ++++++++ arch/x86/include/asm/mpspec.h | 1 + arch/x86/kernel/acpi/boot.c | 8 +++++++ arch/x86/kernel/apic/apic.c | 38 ++++++++++++++++++++++++++++++++ arch/x86/kernel/mpparse.c | 2 +- arch/x86/mm/amdtopology.c | 6 +++-- arch/x86/platform/visws/visws_quirks.c | 2 +- 7 files changed, 61 insertions(+), 5 deletions(-)
Signature · 2 lines
Thanks. HATAYAMA, Daisuke
PATCH v6 1/3 x86, apic: add bios_bsp_physical_apicid
Add bios_bsp_physical_apicid variable. This variable is initialized with the value reported by BIOS tables such as ACPI MADT or MP table.
Without this variable, boot_cpu_physical_apicid temporarilly has the value around MP table related codes such as kernel/mpparse.c, mm/amdtopology.c and platform/visws/visws_quirks.c until it is rewritten back in init_apic_mappings() to the initial APIC ID for the processor that is doing boot up.
This change is also required by the next commit introducing new disable_cpu_apicid kernel parameter, where the value specified via disable_cpu_apicid is compared with boot_cpu_physical_apicid in generic_processor_info(), but in case of using MP table, the boot_cpu_physical_apicid is rewiriten by MP_processor_info() to the initial APIC ID of the BSP reported by BIOS, which is problematic if some AP is specified in disable_cpu_apic.
There's no functional change intended on kernel/mpparse.c, mm/amdtopology.c and platform/visws/visws_quirks.c by the introduction of bios_bsp_physical_apicid.
The reason why we don't use cpuid is that we cannot use IA32_APIC_BASE MSR to determine if the booting up cpu is BSP since rdmsr() to the MSR could not work well even if it exists on some system, e.g. some cluster system.
--- arch/x86/include/asm/mpspec.h | 1 + arch/x86/kernel/acpi/boot.c | 8 ++++++++ arch/x86/kernel/apic/apic.c | 9 +++++++++ arch/x86/kernel/mpparse.c | 2 +- arch/x86/mm/amdtopology.c | 6 +++--- arch/x86/platform/visws/visws_quirks.c | 2 +- 6 files changed, 23 insertions(+), 5 deletions(-)
arch/x86/include/asm/mpspec.h +1 -0
@@ -47,6 +47,7 @@ extern int mp_bus_id_to_type[MAX_MP_BUSSES]; extern DECLARE_BITMAP(mp_bus_not_pci, MAX_MP_BUSSES);extern unsigned int boot_cpu_physical_apicid; +extern unsigned int bios_bsp_physical_apicid;extern unsigned int max_physical_apicid;extern int mpc_default_type;extern unsigned long mp_lapic_addr;
arch/x86/kernel/acpi/boot.c +8 -0
@@ -206,6 +206,14 @@ static void acpi_register_lapic(int id, u8 enabled)if (boot_cpu_physical_apicid !=-1U ) ver = apic_version[boot_cpu_physical_apicid]; +/* +* ACPI specification describes that platform firmware should +* list the BSP processor as the first LAPIC entry in the +* MADT. +*/ +if (!num_processors && !disabled_cpus) + bios_bsp_physical_apicid = id; +generic_processor_info (id, ver); }
arch/x86/kernel/apic/apic.c +9 -0
@@ -63,6 +63,15 @@ unsigned disabled_cpus;/* Processor that is doing the boot up */ unsigned int boot_cpu_physical_apicid =-1U ; +/* Processor with BP flag on IA32_APIC_BASE MSR to be initialized with +* the value reported by BIOS tables such as ACPI MADT or MP table, +* not with the value obtained by rdmsr(IA32_APIC_MSR) since it +* possibly doesn't work well on some clustered system. +* +* In case of kexec, this can differ from the processor that is doing +* the boot up. */ +unsigned int bios_bsp_physical_apicid =BAD_APICID ; +/* * The highest APIC ID seen during enumeration. */
arch/x86/kernel/mpparse.c +1 -1
@@ -64,7 +64,7 @@ static void __init MP_processor_info(struct mpc_cpu *m)if (m->cpuflag &CPU_BOOTPROCESSOR ) { bootup_cpu =" (Bootup-CPU)" ; - boot_cpu_physical_apicid = m->apicid; + bios_bsp_physical_apicid = m->apicid; }printk (KERN_INFO "Processor #%d%s\n" , m->apicid, bootup_cpu);
arch/x86/mm/amdtopology.c +3 -3
@@ -183,9 +183,9 @@ int __init amd_numa_init(void)/* get the APIC ID of the BSP early for systems with apicid lifting */ early_get_boot_cpu_id (); -if (boot_cpu_physical_apicid >0 ) { -pr_info ("BSP APIC ID: %02x\n" , boot_cpu_physical_apicid); - apicid_base = boot_cpu_physical_apicid; +if (bios_bsp_physical_apicid >0 ) { +pr_info ("BSP APIC ID: %02x\n" , bios_bsp_physical_apicid); + apicid_base = bios_bsp_physical_apicid; }for_each_node_mask (i, numa_nodes_parsed)
arch/x86/platform/visws/visws_quirks.c +1 -1
@@ -166,7 +166,7 @@ static void __init MP_processor_info(struct mpc_cpu *m) (m->cpufeature & CPU_MODEL_MASK) >> 4, m->apicver);if (m->cpuflag &CPU_BOOTPROCESSOR ) - boot_cpu_physical_apicid = m->apicid; + bios_bsp_physical_apicid = m->apicid; ver = m->apicver;if ((ver >=0x14 && m->apicid >=0xff ) || m->apicid >=0xf ) {
PATCH v6 2/3 x86, apic: Add disable_cpu_apicid kernel parameter
In reply to HATAYAMA DaisukeAdd disable_cpu_apicid kernel parameter. To use this kernel parameter, specify an initial APIC ID of the corresponding CPU you want to disable.
This is mostly used for the kdump 2nd kernel to disable BSP to wake up multiple CPUs without causing system reset or hang due to sending INIT from AP to BSP.
Kdump users first figure out initial APIC ID of the BSP, CPU0 in the 1st kernel, for example from /proc/cpuinfo and then set up this kernel parameter for the 2nd kernel using the obtained APIC ID.
However, doing this procedure at each boot time manually is awkward, which should be automatically done by user-land service scripts, for example, kexec-tools on fedora/RHEL distributions.
This design is more flexible than disabling BSP in kernel boot time automatically in that in kernel boot time we have no choice but referring to ACPI/MP table to obtain initial APIC ID for BSP, meaning that the method is not applicable to the systems without such BIOS tables.
One assumption behind this design is that users get initial APIC ID of the BSP in still healthy state and so BSP is uniquely kept in CPU0. Thus, through the kernel parameter, only one initial APIC ID can be specified.
--- arch/x86/kernel/apic/apic.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
arch/x86/kernel/apic/apic.c +29 -0
@@ -83,6 +83,13 @@ unsigned int max_physical_apicid;physid_mask_t phys_cpu_present_map;/* +* Processor to be disabled specified by kernel parameter +* disable_cpu_apicid=<int>, mostly used for the kdump 2nd kernel to +* avoid undefined behaviour caused by sending INIT from AP to BSP. +*/ +unsigned int disabled_cpu_apicid =BAD_APICID ; + +/* * Map cpu index to physical APIC ID */ DEFINE_EARLY_PER_CPU_READ_MOSTLY (u16, x86_cpu_to_apicid,BAD_APICID );
@@ -2122,6 +2129,19 @@ void generic_processor_info(int apicid, int version)bool boot_cpu_detected =physid_isset (boot_cpu_physical_apicid, phys_cpu_present_map); +if (disabled_cpu_apicid !=BAD_APICID && + disabled_cpu_apicid != boot_cpu_physical_apicid && + disabled_cpu_apicid == apicid) { +int thiscpu = num_processors + disabled_cpus; + +pr_warning ("ACPI: Disabling requested cpu." +" Processor %d/0x%x ignored.\n" , + thiscpu, apicid); + + disabled_cpus++; +return ; + } +/* * If boot cpu has not been detected yet, then only allow upto * nr_cpu_ids - 1 processors and keep one slot free for boot cpu
@@ -2598,3 +2618,12 @@ static int __init lapic_insert_resource(void)* that is using request_resource */ late_initcall (lapic_insert_resource); + +static int __init apic_set_disabled_cpu_apicid (char *arg) +{ +if (!arg || !get_option (&arg, &disabled_cpu_apicid)) +return -EINVAL ; + +return 0 ; +} +early_param ("disable_cpu_apicid" , apic_set_disabled_cpu_apicid);
Re: [PATCH v6 2/3] x86, apic: Add disable_cpu_apicid kernel parameter
On Thu, Nov 21, 2013 at 11:00:44AM +0900, HATAYAMA Daisuke wrote:
[..]
arch/x86/kernel/apic/apic.c void generic_processor_info(int apicid, int version) · line 2133
Quoted 6 lines
arch/x86/kernel/apic/apic.c +2
-0@@ -2122,6 +2129,19 @@ void generic_processor_info(int apicid, int version)bool boot_cpu_detected =physid_isset (boot_cpu_physical_apicid, phys_cpu_present_map); + if (disabled_cpu_apicid != BAD_APICID && + disabled_cpu_apicid != boot_cpu_physical_apicid &&
Hi Hatayama,
So we are comparing disabled_cpu_apicid with boot_cpu_physical_apicid to make sure that one can not disable the cpu we are booting on. Can we just read the apic id of booting cpu in local variable and compare against that?
Something like as follows.
if (disabled_cpu_apicid != BAD_APICID &&
disabled_cpu_apicid == apicid &&
disabled_cpu_apicid != read_apic_id()) {
/* Disable cpu */
}If above works, you will not need first patch in the series?
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v6 2/3] x86, apic: Add disable_cpu_apicid kernel parameter
Quoted 19 lines · Vivek Goyal
On Thu, Nov 21, 2013 at 11:00:44AM +0900, HATAYAMA Daisuke wrote:
[..]
arch/x86/kernel/apic/apic.c void generic_processor_info(int apicid, int version) · line 2133
Quoted 6 lines
arch/x86/kernel/apic/apic.c +2
-0@@ -2122,6 +2129,19 @@ void generic_processor_info(int apicid, int version)bool boot_cpu_detected =physid_isset (boot_cpu_physical_apicid, phys_cpu_present_map); + if (disabled_cpu_apicid != BAD_APICID && + disabled_cpu_apicid != boot_cpu_physical_apicid &&Hi Hatayama,
So we are comparing
disabled_cpu_apicidwithboot_cpu_physical_apicidto make sure that one can not disable the cpu we are booting on. Can we just read the apic id of booting cpu in local variable and compare against that?Something like as follows.
Quoted 8 lines
if (disabled_cpu_apicid != BAD_APICID && disabled_cpu_apicid == apicid && disabled_cpu_apicid != read_apic_id()) { /* Disable cpu */ }If above works, you will not need first patch in the series?
Yes, I came up with the idea, too. But doing this means we leave two different ways boot_cpu_physical_apicid is used at boot, which seemed incomplete as a patch. Also, then we could even lost the reason why boot_cpu_physical_apicid exists.
But, it's true that the 1st patch causes one more reviewing point. I'll remove it and fix the 2nd patch just as you suggested here.
Signature · 2 lines
Thanks. HATAYAMA, Daisuke
Re: [PATCH v6 2/3] x86, apic: Add disable_cpu_apicid kernel parameter
Quoted 29 lines · HATAYAMA Daisuke
Quoted 19 lines · Vivek Goyal
On Thu, Nov 21, 2013 at 11:00:44AM +0900, HATAYAMA Daisuke wrote:
[..]
arch/x86/kernel/apic/apic.c void generic_processor_info(int apicid, int version) · line 2133
Quoted 6 lines
arch/x86/kernel/apic/apic.c +2
-0@@ -2122,6 +2129,19 @@ void generic_processor_info(int apicid, int version)bool boot_cpu_detected =physid_isset (boot_cpu_physical_apicid, phys_cpu_present_map); + if (disabled_cpu_apicid != BAD_APICID && + disabled_cpu_apicid != boot_cpu_physical_apicid &&Hi Hatayama,
So we are comparing
disabled_cpu_apicidwithboot_cpu_physical_apicidto make sure that one can not disable the cpu we are booting on. Can we just read the apic id of booting cpu in local variable and compare against that?Something like as follows.
Quoted 8 lines
if (disabled_cpu_apicid != BAD_APICID && disabled_cpu_apicid == apicid && disabled_cpu_apicid != read_apic_id()) { /* Disable cpu */ }If above works, you will not need first patch in the series?
Quoted 7 lines
Yes, I came up with the idea, too. But doing this means we leave two different ways
boot_cpu_physical_apicidis used at boot, which seemed incomplete as a patch. Also, then we could even lost the reason whyboot_cpu_physical_apicidexists.But, it's true that the 1st patch causes one more reviewing point. I'll remove it and fix the 2nd patch just as you suggested here.
I would say that do the boot_cpu_physical_id related cleanup in a separate series. If we keep this patch really simple without fear of breaking anything else, acking and committing the patch becomes easier.
Feel free to post boot cpu related cleanup in a separate series.
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
PATCH v6 3/3 Documentation, x86, apic, kexec: Add disable_cpu_apicid kernel parameter
In reply to HATAYAMA DaisukeAdd disable_cpu_apicid kernel parameter to disable the CPU with the specified number of initial APIC ID, mostly used for the kdump 2nd kernel to disable BSP to wake up multiple CPUs without causing system reset or hang due to sending INIT from AP to BSP.
--- Documentation/kernel-parameters.txt | 9 +++++++++ 1 file changed, 9 insertions(+)
Documentation/kernel-parameters.txt +9 -0
@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted. disable= [IPV6] See Documentation/networking/ipv6.txt. + disable_cpu_apicid= [X86,APIC,KEXEC,SMP] + Format: <int> + The number of initial APIC ID for the + corresponding CPU to be disabled at boot, + mostly used for the kdump 2nd kernel to + disable BSP to wake up multiple CPUs without + causing system reset or hang due to sending + INIT from AP to BSP. + disable_ddw [PPC/PSERIES] Disable Dynamic DMA Window support. Use this if to workaround buggy firmware.