Skip to contentSkip to conversation

x86, apic, kexec: Add disable_cpu_apic kernel parameter

PATCH Superseded HATAYAMA Daisuke 2 participants 6 messages Nov 27, 2013 – Nov 28, 2013

Subsystems (4)

v7 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

v7 cover x86, apic, kexec: Add disable_cpu_apic kernel parameter

This patch set is to allow kdump 2nd kernel to wake up multiple CPUs, a continueing work from:

[PATCH v3 0/2] x86, apic, kdump: Disable BSP if boot cpu is AP
https://lkml.org/lkml/2013/10/16/300.

At v4, basic design has changed. Now users need to figure out initial APIC ID of BSP in the 1st kernel and configures kernel parameter for the 2nd kernel manually using disable_cpu_apicid kernel parameter, which is newly introduced by this patch set. This design is more flexible than the previous version in that we no longer have to rely on ACPI/MP table to get initial APIC ID of BSP.

Sorry, this patch set have not include in-source documentation requested by Borislav Petkov yet, but I'll post it later separately, which would be better to focus on documentation reviewing.

ChangeLog

v6 => v7)

- Rebased on top of v3.13-rc1

- Remove a patch that cleans up the code around x86_cpu_physical_apicid. Instead, use read_apic_id() directly in generic_processor_info(). The clean up removed here will be done in different patch set.

v5 => v6)

- Remove use of rdmsr(IA32_APIC_BASE) to initialize bsp_physical_apicid since the MSR doesn't work well on some cluster systems, suggested by HPA. Also, current users of the variable expects the initial apicid reported through MP table only; removing the use of the MSR is not problematic.

- Rename bsp_physical_apicid as bios_bsp_physical_apicid to make it clear that the apidid contained there is the one reported from some BIOS table. Also, initialize it not only by MP table but also by ACPI MADT.

- Change message displayed when specified cpu is disabled from:

ACPI: Disable specified CPU. Processor 0/0x0 ignored.

to:

ACPI: Disabling requested cpu. Processor 0/0x0 ignored.

v4 => v5)

- Rebased on top of v3.12

- Introduce bsp_physical_apicid that has the initial APIC ID for the processor with BSP flag on IA32_APIC_BASE MSR. Without this, boot_cpu_physical_apicid has temporarilly the value around MP table related codes, although it's designed to have the initial APIC ID for the processor that is doing the boot up. Use the bsp_physical_apicid in MP table related codes; no impact on semantics at runtime there.

v3 => v4)

- Rebased on top of v3.12-rc6

- Basic design has been changed. Now users need to figure out initial APIC ID of BSP in the 1st kernel and configures kernel parameter for the 2nd kernel manually using disable_cpu_apic kernel parameter to be newly introduced in this patch set. This design is more flexible than the previous version in that we no longer have to rely on ACPI/MP table to get initial APIC ID of BSP.

v2 => v3)

- Change default value of boot_cpu_is_bsp to true.

- Before executing rdmsr(MSR_IA32_APICBASE), check if the number of processor family is larger than or equal to 6 in order to avoid invalid opcode exception on processors where MSR_IA32_APICBASE is not supported.

v1 => v2)

- Rebased on top of v3.12-rc5.

- Fix linking time error of boot_cpu_is_bsp_init() in case of CONFIG_LOCAL_APIC disabled by adding empty static inline function instead.

- Fix missing feature check by means of cpu_has_apic macro in boot_cpu_is_bsp_init() before calling rdmsr_safe(MSR_IA32_APICBASE).

NOTE: I've checked local apic-present case only; I don't have any x86 processor without local apic.

- Add __init annotation to boot_cpu_is_bsp_init().

Test

- built with and without CONFIG_LOCAL_APIC
- tested x86_64 in case of acpi and MP table
- tested disable_cpu_apicid=<n> to disable both AP and BSP

---

HATAYAMA Daisuke (2):
x86, apic: Add disable_cpu_apicid kernel parameter Documentation, x86, apic, kexec: Add disable_cpu_apicid kernel parameter

 Documentation/kernel-parameters.txt |    9 ++++++
 arch/x86/kernel/apic/apic.c         |   49 +++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
Signature · 7 lines
Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#1 lore raw

PATCH v7 1/2 x86, apic: Add disable_cpu_apicid kernel parameter

Add disable_cpu_apicid kernel parameter. To use this kernel parameter, specify an initial APIC ID of the corresponding CPU you want to disable.

This is mostly used for the kdump 2nd kernel to disable BSP to wake up multiple CPUs without causing system reset or hang due to sending INIT from AP to BSP.

Kdump users first figure out initial APIC ID of the BSP, CPU0 in the 1st kernel, for example from /proc/cpuinfo and then set up this kernel parameter for the 2nd kernel using the obtained APIC ID.

However, doing this procedure at each boot time manually is awkward, which should be automatically done by user-land service scripts, for example, kexec-tools on fedora/RHEL distributions.

This design is more flexible than disabling BSP in kernel boot time automatically in that in kernel boot time we have no choice but referring to ACPI/MP table to obtain initial APIC ID for BSP, meaning that the method is not applicable to the systems without such BIOS tables.

One assumption behind this design is that users get initial APIC ID of the BSP in still healthy state and so BSP is uniquely kept in CPU0. Thus, through the kernel parameter, only one initial APIC ID can be specified.

boot_cpu_physical_apicid is designed to have the apicid returned by read_apic_id(). However, on some platforms, it is temporarilly modified by the apicid reported as BSP through MP table. This function is called with the modified boot_cpu_physical_apicid.

On the platforms, boot_cpu_physical_apicid is always forced to be the apicid of the BSP. Then, disabled_cpu_apicid kernel parameter doesn't work well for apicids for APs.

Here we leave improvement of handling boot_cpu_physical_apicid as another work for now since the temporary modification is done on differnet platforms and tests on each platform are required. As a workaround, we directly update boot_cpu_physical_apicid by read_apic_id().

---
 arch/x86/kernel/apic/apic.c |   49 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)
arch/x86/kernel/apic/apic.c +49 -0
index d278736..a94f618 100644
@@ -75,6 +75,13 @@ unsigned int max_physical_apicid;
 physid_mask_t phys_cpu_present_map;
 
 /*
+ * Processor to be disabled specified by kernel parameter
+ * disable_cpu_apicid=<int>, mostly used for the kdump 2nd kernel to
+ * avoid undefined behaviour caused by sending INIT from AP to BSP.
+ */
+unsigned int disabled_cpu_apicid = BAD_APICID;
+
+/*
  * Map cpu index to physical APIC ID
  */
 DEFINE_EARLY_PER_CPU_READ_MOSTLY(u16, x86_cpu_to_apicid, BAD_APICID);
@@ -2111,9 +2118,42 @@ void disconnect_bsp_APIC(int virt_wire_setup)
 int generic_processor_info(int apicid, int version)
 {
 	int cpu, max = nr_cpu_ids;
+	/*
+	 * boot_cpu_physical_apicid is designed to have the apicid
+	 * returned by read_apic_id(). However, on some platforms, it
+	 * is temporarilly modified by the apicid reported as BSP
+	 * through MP table. Concretely:
+	 *
+	 * - arch/x86/kernel/mpparse.c: MP_processor_info()
+	 * - arch/x86/mm/amdtopology.c: amd_numa_init()
+	 * - arch/x86/platform/visws/visws_quirks.c: MP_processor_info()
+	 *
+	 * This function is executed with the modified
+	 * boot_cpu_physical_apicid. So, disabled_cpu_apicid kernel
+	 * parameter doesn't work.
+	 *
+	 * Since fixing handling of boot_cpu_physical_apicid requires
+	 * another discussion and tests on each platform, we leave it
+	 * for now and here we use read_apic_id() directly by updating
+	 * global scope of boot_cpu_physical_id with the local one.
+	 */
+	unsigned int boot_cpu_physical_apicid = read_apic_id();
 	bool boot_cpu_detected = physid_isset(boot_cpu_physical_apicid,
 				phys_cpu_present_map);
 
+	if (disabled_cpu_apicid != BAD_APICID &&
+	    disabled_cpu_apicid != boot_cpu_physical_apicid &&
+	    disabled_cpu_apicid == apicid) {
+		int thiscpu = num_processors + disabled_cpus;
+
+		pr_warning("ACPI: Disabling requested cpu."
+			   " Processor %d/0x%x ignored.\n",
+			   thiscpu, apicid);
+
+		disabled_cpus++;
+		return -ENODEV;
+	}
+
 	/*
 	 * If boot cpu has not been detected yet, then only allow upto
 	 * nr_cpu_ids - 1 processors and keep one slot free for boot cpu
@@ -2592,3 +2632,12 @@ static int __init lapic_insert_resource(void)
  * that is using request_resource
  */
 late_initcall(lapic_insert_resource);
+
+static int __init apic_set_disabled_cpu_apicid(char *arg)
+{
+	if (!arg || !get_option(&arg, &disabled_cpu_apicid))
+		return -EINVAL;
+
+	return 0;
+}
+early_param("disable_cpu_apicid", apic_set_disabled_cpu_apicid);

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw

Re: [PATCH v7 1/2] x86, apic: Add disable_cpu_apicid kernel parameter

Quoted 79 lines · HATAYAMA Daisuke

Add disable_cpu_apicid kernel parameter. To use this kernel parameter, specify an initial APIC ID of the corresponding CPU you want to disable.

This is mostly used for the kdump 2nd kernel to disable BSP to wake up multiple CPUs without causing system reset or hang due to sending INIT from AP to BSP.

Kdump users first figure out initial APIC ID of the BSP, CPU0 in the 1st kernel, for example from /proc/cpuinfo and then set up this kernel parameter for the 2nd kernel using the obtained APIC ID.

However, doing this procedure at each boot time manually is awkward, which should be automatically done by user-land service scripts, for example, kexec-tools on fedora/RHEL distributions.

This design is more flexible than disabling BSP in kernel boot time automatically in that in kernel boot time we have no choice but referring to ACPI/MP table to obtain initial APIC ID for BSP, meaning that the method is not applicable to the systems without such BIOS tables.

One assumption behind this design is that users get initial APIC ID of the BSP in still healthy state and so BSP is uniquely kept in CPU0. Thus, through the kernel parameter, only one initial APIC ID can be specified.

boot_cpu_physical_apicid is designed to have the apicid returned by read_apic_id(). However, on some platforms, it is temporarilly modified by the apicid reported as BSP through MP table. This function is called with the modified boot_cpu_physical_apicid.

On the platforms, boot_cpu_physical_apicid is always forced to be the apicid of the BSP. Then, disabled_cpu_apicid kernel parameter doesn't work well for apicids for APs.

Here we leave improvement of handling boot_cpu_physical_apicid as another work for now since the temporary modification is done on differnet platforms and tests on each platform are required. As a workaround, we directly update boot_cpu_physical_apicid by read_apic_id().

---
 arch/x86/kernel/apic/apic.c |   49 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)
arch/x86/kernel/apic/apic.c +17 -0
index d278736..a94f618 100644
@@ -75,6 +75,13 @@ unsigned int max_physical_apicid;
 physid_mask_t phys_cpu_present_map;
 
 /*
+ * Processor to be disabled specified by kernel parameter
+ * disable_cpu_apicid=<int>, mostly used for the kdump 2nd kernel to
+ * avoid undefined behaviour caused by sending INIT from AP to BSP.
+ */
+unsigned int disabled_cpu_apicid = BAD_APICID;
+
+/*
  * Map cpu index to physical APIC ID
  */
 DEFINE_EARLY_PER_CPU_READ_MOSTLY(u16, x86_cpu_to_apicid, BAD_APICID);
@@ -2111,9 +2118,42 @@ void disconnect_bsp_APIC(int virt_wire_setup)
 int generic_processor_info(int apicid, int version)
 {
 	int cpu, max = nr_cpu_ids;
+	/*
+	 * boot_cpu_physical_apicid is designed to have the apicid
+	 * returned by read_apic_id(). However, on some platforms, it
+	 * is temporarilly modified by the apicid reported as BSP
+	 * through MP table. Concretely:
+	 *
+	 * - arch/x86/kernel/mpparse.c: MP_processor_info()
+	 * - arch/x86/mm/amdtopology.c: amd_numa_init()
+	 * - arch/x86/platform/visws/visws_quirks.c: MP_processor_info()
+	 *

arch/x86/kernel/apic/apic.c void disconnect_bsp_APIC(int virt_wire_setup) · line 2140

Quoted 10 lines
arch/x86/kernel/apic/apic.c +10 -0
+	 * This function is executed with the modified
+	 * boot_cpu_physical_apicid. So, disabled_cpu_apicid kernel
+	 * parameter doesn't work.
+	 *
+	 * Since fixing handling of boot_cpu_physical_apicid requires
+	 * another discussion and tests on each platform, we leave it
+	 * for now and here we use read_apic_id() directly by updating
+	 * global scope of boot_cpu_physical_id with the local one.
+	 */
+	unsigned int boot_cpu_physical_apicid = read_apic_id();

I think this is confusing. Why are you trying to define a local variable with same name as global variable. There is no need. I think if we simply put the comment there that why are we not making use of boot_cpu_physical_apicid, that is good enough and directly read the apic id.

	if (disabled_cpu_apicid != BAD_APICID &&
	    disabled_cpu_apicid != read_apic_id() &&
	    disabled_cpu_apicid == apicid) {

	}

Thanks
Vivek

Quoted 22 lines
arch/x86/kernel/apic/apic.c +13 -0
 	bool boot_cpu_detected = physid_isset(boot_cpu_physical_apicid,
 				phys_cpu_present_map);
 
+	if (disabled_cpu_apicid != BAD_APICID &&
+	    disabled_cpu_apicid != boot_cpu_physical_apicid &&
+	    disabled_cpu_apicid == apicid) {
+		int thiscpu = num_processors + disabled_cpus;
+
+		pr_warning("ACPI: Disabling requested cpu."
+			   " Processor %d/0x%x ignored.\n",
+			   thiscpu, apicid);
+
+		disabled_cpus++;
+		return -ENODEV;
+	}
+
 	/*
 	 * If boot cpu has not been detected yet, then only allow upto
 	 * nr_cpu_ids - 1 processors and keep one slot free for boot cpu
@@ -2592,3 +2632,12 @@ static int __init lapic_insert_resource(void)
  * that is using request_resource
  */

arch/x86/kernel/apic/apic.c static int __init lapic_insert_resource(void) · line 2643

Quoted 10 lines
arch/x86/kernel/apic/apic.c +9 -0
 late_initcall(lapic_insert_resource);
+
+static int __init apic_set_disabled_cpu_apicid(char *arg)
+{
+	if (!arg || !get_option(&arg, &disabled_cpu_apicid))
+		return -EINVAL;
+
+	return 0;
+}
+early_param("disable_cpu_apicid", apic_set_disabled_cpu_apicid);

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw

Re: [PATCH v7 1/2] x86, apic: Add disable_cpu_apicid kernel parameter

Quoted 97 lines · Vivek Goyal
Quoted 79 lines · HATAYAMA Daisuke

Add disable_cpu_apicid kernel parameter. To use this kernel parameter, specify an initial APIC ID of the corresponding CPU you want to disable.

This is mostly used for the kdump 2nd kernel to disable BSP to wake up multiple CPUs without causing system reset or hang due to sending INIT from AP to BSP.

Kdump users first figure out initial APIC ID of the BSP, CPU0 in the 1st kernel, for example from /proc/cpuinfo and then set up this kernel parameter for the 2nd kernel using the obtained APIC ID.

However, doing this procedure at each boot time manually is awkward, which should be automatically done by user-land service scripts, for example, kexec-tools on fedora/RHEL distributions.

This design is more flexible than disabling BSP in kernel boot time automatically in that in kernel boot time we have no choice but referring to ACPI/MP table to obtain initial APIC ID for BSP, meaning that the method is not applicable to the systems without such BIOS tables.

One assumption behind this design is that users get initial APIC ID of the BSP in still healthy state and so BSP is uniquely kept in CPU0. Thus, through the kernel parameter, only one initial APIC ID can be specified.

boot_cpu_physical_apicid is designed to have the apicid returned by read_apic_id(). However, on some platforms, it is temporarilly modified by the apicid reported as BSP through MP table. This function is called with the modified boot_cpu_physical_apicid.

On the platforms, boot_cpu_physical_apicid is always forced to be the apicid of the BSP. Then, disabled_cpu_apicid kernel parameter doesn't work well for apicids for APs.

Here we leave improvement of handling boot_cpu_physical_apicid as another work for now since the temporary modification is done on differnet platforms and tests on each platform are required. As a workaround, we directly update boot_cpu_physical_apicid by read_apic_id().

---
  arch/x86/kernel/apic/apic.c |   49 +++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 49 insertions(+)
arch/x86/kernel/apic/apic.c +17 -0
index d278736..a94f618 100644
@@ -75,6 +75,13 @@ unsigned int max_physical_apicid;
  physid_mask_t phys_cpu_present_map;
 
  /*
+ * Processor to be disabled specified by kernel parameter
+ * disable_cpu_apicid=<int>, mostly used for the kdump 2nd kernel to
+ * avoid undefined behaviour caused by sending INIT from AP to BSP.
+ */
+unsigned int disabled_cpu_apicid = BAD_APICID;
+
+/*
   * Map cpu index to physical APIC ID
   */
  DEFINE_EARLY_PER_CPU_READ_MOSTLY(u16, x86_cpu_to_apicid, BAD_APICID);
@@ -2111,9 +2118,42 @@ void disconnect_bsp_APIC(int virt_wire_setup)
  int generic_processor_info(int apicid, int version)
  {
  	int cpu, max = nr_cpu_ids;
+	/*
+	 * boot_cpu_physical_apicid is designed to have the apicid
+	 * returned by read_apic_id(). However, on some platforms, it
+	 * is temporarilly modified by the apicid reported as BSP
+	 * through MP table. Concretely:
+	 *
+	 * - arch/x86/kernel/mpparse.c: MP_processor_info()
+	 * - arch/x86/mm/amdtopology.c: amd_numa_init()
+	 * - arch/x86/platform/visws/visws_quirks.c: MP_processor_info()
+	 *

arch/x86/kernel/apic/apic.c void disconnect_bsp_APIC(int virt_wire_setup) · line 2140

Quoted 10 lines
arch/x86/kernel/apic/apic.c +10 -0
+	 * This function is executed with the modified
+	 * boot_cpu_physical_apicid. So, disabled_cpu_apicid kernel
+	 * parameter doesn't work.
+	 *
+	 * Since fixing handling of boot_cpu_physical_apicid requires
+	 * another discussion and tests on each platform, we leave it
+	 * for now and here we use read_apic_id() directly by updating
+	 * global scope of boot_cpu_physical_id with the local one.
+	 */
+	unsigned int boot_cpu_physical_apicid = read_apic_id();

I think this is confusing. Why are you trying to define a local variable with same name as global variable. There is no need. I think if we simply put the comment there that why are we not making use of boot_cpu_physical_apicid, that is good enough and directly read the apic id.

Quoted 6 lines
	if (disabled_cpu_apicid != BAD_APICID &&
	    disabled_cpu_apicid != read_apic_id() &&
	    disabled_cpu_apicid == apicid) {

	}

Because there are also other occurrences of boot_cpu_physical_apicid in the same function, and defining the local variable with the same name needs one line only, and it's a temporary workaround change, I think it unnatural to replace boot_cpu_physical_apicid even temporarily.

Signature · 2 lines
Thanks.
HATAYAMA, Daisuke
#4 lore raw

PATCH v7 2/2 Documentation, x86, apic, kexec: Add disable_cpu_apicid kernel parameter

In reply to HATAYAMA Daisuke

Add disable_cpu_apicid kernel parameter to disable the CPU with the specified number of initial APIC ID, mostly used for the kdump 2nd kernel to disable BSP to wake up multiple CPUs without causing system reset or hang due to sending INIT from AP to BSP.

---
 Documentation/kernel-parameters.txt |    9 +++++++++
 1 file changed, 9 insertions(+)
Documentation/kernel-parameters.txt +9 -0
index 50680a5..dd77bec 100644
@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 	disable=	[IPV6]
 			See Documentation/networking/ipv6.txt.
 
+	disable_cpu_apicid= [X86,APIC,KEXEC,SMP]
+			Format: <int>
+			The number of initial APIC ID for the
+			corresponding CPU to be disabled at boot,
+			mostly used for the kdump 2nd kernel to
+			disable BSP to wake up multiple CPUs without
+			causing system reset or hang due to sending
+			INIT from AP to BSP.
+
 	disable_ddw     [PPC/PSERIES]
 			Disable Dynamic DMA Window support. Use this if
 			to workaround buggy firmware.
#5 lore raw

Re: [PATCH v7 2/2] Documentation, x86, apic, kexec: Add disable_cpu_apicid kernel parameter

Quoted 6 lines · HATAYAMA Daisuke

Add disable_cpu_apicid kernel parameter to disable the CPU with the specified number of initial APIC ID, mostly used for the kdump 2nd kernel to disable BSP to wake up multiple CPUs without causing system reset or hang due to sending INIT from AP to BSP.

I guess you don't require a separate patch for documentation. Just merge it with previous patch. So this becomes a single patch fix and not a series.

Thanks
Vivek

Quoted 14 lines
---
 Documentation/kernel-parameters.txt |    9 +++++++++
 1 file changed, 9 insertions(+)
Documentation/kernel-parameters.txt +2 -0
index 50680a5..dd77bec 100644
@@ -774,6 +774,15 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 	disable=	[IPV6]
 			See Documentation/networking/ipv6.txt.
 
+	disable_cpu_apicid= [X86,APIC,KEXEC,SMP]
+			Format: <int>

Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 788

Quoted 10 lines
Documentation/kernel-parameters.txt +7 -0
+			The number of initial APIC ID for the
+			corresponding CPU to be disabled at boot,
+			mostly used for the kdump 2nd kernel to
+			disable BSP to wake up multiple CPUs without
+			causing system reset or hang due to sending
+			INIT from AP to BSP.
+
 	disable_ddw     [PPC/PSERIES]
 			Disable Dynamic DMA Window support. Use this if
 			to workaround buggy firmware.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec