Skip to contentSkip to conversation

BSP or CPU0 online/offline

PATCH Superseded Fenghua Yu 3 participants 16 messages Nov 4, 2011 – Nov 7, 2011

Subsystems (5)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Fenghua Yu

author
#0 lore raw

cover BSP or CPU0 online/offline

From: Fenghua Yu <[email protected]>

BSP or CPU0 has been the last obstacle to CPU hotplug on x86. This patch set implements BSP online and offline and removes this obstacle to CPU hotplug.

RAS needs the feature. If socket0 needs to be hotplugged for any reason (any thread on socket0 is bad, shared cache issue, uncore issue, etc), CPU0 is required to be offline or hot replaced to keep the system run.

Fenghua Yu (9):
include/linux/cpu.h: Define architecture dependent cpu map update and
state check functions
kernel/cpu.c: Add arch dependent cpu map update functions x86/i387.c: Thread xstate is initialized only on BSP once x86/common.c: Init BSP data during BSP online x86/mtrr/main.c: Ask the first online CPU to save mtrr kernel/power/suspend.c,hibernate.c: Don't hibernate/suspend if CPU0
is offline
x86/topology.c: Support functions for BSP online/offline x86/smpboot.c: Don't offline BSP if any irq can not be migrated out
of it
Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0
online/offline feature

 Documentation/cpu-hotplug.txt       |   19 +++++++++++++++
 Documentation/kernel-parameters.txt |   13 ++++++++++
 arch/x86/include/asm/processor.h    |    1 +
 arch/x86/kernel/cpu/common.c        |   13 ++++++++--
 arch/x86/kernel/cpu/mtrr/main.c     |    9 +++++-
 arch/x86/kernel/i387.c              |    9 ++++++-
 arch/x86/kernel/smpboot.c           |   43 ++++++++++++++++++++++++++++-----
 arch/x86/kernel/topology.c          |   44 +++++++++++++++++++++++++++++-----
 include/linux/cpu.h                 |    6 ++++
 kernel/cpu.c                        |   12 +++++++++
 kernel/power/hibernate.c            |    5 ++++
 kernel/power/main.c                 |    5 ++++
 kernel/power/suspend.c              |    4 +++
 13 files changed, 163 insertions(+), 20 deletions(-)
author
#1 lore raw

PATCH 1/9 include/linux/cpu.h: Define architecture dependent cpu map update and state check functions

From: Fenghua Yu <[email protected]>

arch_cpu_maps_update_begin() and arch_cpu_maps_update_done() are architecture dependent cpu map update functions which are called during cpu_up() and cpu_down().

arch_state_check() is architecture dependent state check function which is called during hibernate and suspend.

They are empty __weak functions and are overriden in x86.

---
 include/linux/cpu.h |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)
include/linux/cpu.h +6 -0
index b1a635a..88a6390 100644
@@ -137,6 +137,8 @@ int cpu_up(unsigned int cpu);
 void notify_cpu_starting(unsigned int cpu);
 extern void cpu_maps_update_begin(void);
 extern void cpu_maps_update_done(void);
+extern void arch_cpu_maps_update_begin(void);
+extern void arch_cpu_maps_update_done(void);
 
 #else	/* CONFIG_SMP */
 
@@ -207,4 +209,8 @@ static inline int disable_nonboot_cpus(void) { return 0; }
 static inline void enable_nonboot_cpus(void) {}
 #endif /* !CONFIG_PM_SLEEP_SMP */
 
+#ifdef CONFIG_PM
+extern int  arch_state_check(void);
+#endif
+
 #endif /* _LINUX_CPU_H_ */
Signature · 1 line
1.6.0.3
author
#2 lore raw

PATCH 2/9 kernel/cpu.c: Add arch dependent cpu map update functions

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

Because resume from hibernate and suspend always starts from CPU0 on x86 BIOS, we need to check if CPU0 is online before hibernate or suspend. This causes a race condition on cpu_online_map.

To cope with the race condition, we add arch_cpu_maps_update_begin() and arch_cpu_maps_update_done() during cpu_down() and cpu_up(). The functions are empty on non x86 platforms and are overriden on x86 platforms with real functions to deal with the race condition.

---
 kernel/cpu.c |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)
kernel/cpu.c +12 -0
index 12b7458..4d80365 100644
@@ -20,6 +20,14 @@
 /* Serializes the updates to cpu_online_mask, cpu_present_mask */
 static DEFINE_MUTEX(cpu_add_remove_lock);
 
+void __weak arch_cpu_maps_update_begin(void)
+{
+}
+
+void __weak arch_cpu_maps_update_done(void)
+{
+}
+
 /*
  * The following two API's must be used when attempting
  * to serialize the updates to cpu_online_mask, cpu_present_mask.
@@ -274,6 +282,7 @@ int __ref cpu_down(unsigned int cpu)
 	int err;
 
 	cpu_maps_update_begin();
+	arch_cpu_maps_update_begin();
 
 	if (cpu_hotplug_disabled) {
 		err = -EBUSY;
@@ -284,6 +293,7 @@ int __ref cpu_down(unsigned int cpu)
 
 out:
 	cpu_maps_update_done();
+	arch_cpu_maps_update_done();
 	return err;
 }
 EXPORT_SYMBOL(cpu_down);
@@ -367,6 +377,7 @@ int __cpuinit cpu_up(unsigned int cpu)
 #endif
 
 	cpu_maps_update_begin();
+	arch_cpu_maps_update_begin();
 
 	if (cpu_hotplug_disabled) {
 		err = -EBUSY;
@@ -377,6 +388,7 @@ int __cpuinit cpu_up(unsigned int cpu)
 
 out:
 	cpu_maps_update_done();
+	arch_cpu_maps_update_done();
 	return err;
 }
 
Signature · 1 line
1.6.0.3
#3 lore raw

Re: [PATCH 2/9] kernel/cpu.c: Add arch dependent cpu map update functions

Quoted 11 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

Because resume from hibernate and suspend always starts from CPU0 on x86 BIOS, we need to check if CPU0 is online before hibernate or suspend. This causes a race condition on cpu_online_map.

To cope with the race condition, we add arch_cpu_maps_update_begin() and arch_cpu_maps_update_done() during cpu_down() and cpu_up(). The functions are empty on non x86 platforms and are overriden on x86 platforms with real functions to deal with the race condition.

The race between CPU Hotplug and suspend/hibernate has been taken care of by my patch here (for another usecase): https://lkml.org/lkml/2011/11/2/487
This is not yet in mainline, but in linux-pm/linux-next.
Please see if this solves your case too. And please CC linux-pm mailing list ([email protected]) on patches related to power management.

Quoted 33 lines
---
 kernel/cpu.c |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)
kernel/cpu.c +9 -0
index 12b7458..4d80365 100644
@@ -20,6 +20,14 @@
 /* Serializes the updates to cpu_online_mask, cpu_present_mask */
 static DEFINE_MUTEX(cpu_add_remove_lock);
 
+void __weak arch_cpu_maps_update_begin(void)
+{
+}
+
+void __weak arch_cpu_maps_update_done(void)
+{
+}
+
 /*
  * The following two API's must be used when attempting
  * to serialize the updates to cpu_online_mask, cpu_present_mask.
@@ -274,6 +282,7 @@ int __ref cpu_down(unsigned int cpu)
 	int err;
 
 	cpu_maps_update_begin();
+	arch_cpu_maps_update_begin();
 
 	if (cpu_hotplug_disabled) {
 		err = -EBUSY;

kernel/cpu.c int __ref cpu_down(unsigned int cpu) · line 298

Quoted 6 lines
kernel/cpu.c +1 -0
@@ -284,6 +293,7 @@ int __ref cpu_down(unsigned int cpu)
 
 out:
 	cpu_maps_update_done();
+	arch_cpu_maps_update_done();
 	return err;
 }

See my comments above about whether this is really necessary. By the way, the locking/unlocking order here seems rather weird to me. Why have you not chosen to do something like:

cpu_maps_update_begin()
arch_cpu_maps_update_begin()
...
arch_cpu_maps_update_done()
cpu_maps_update_done()

Quoted 10 lines
kernel/cpu.c +1 -0
 EXPORT_SYMBOL(cpu_down);
@@ -367,6 +377,7 @@ int __cpuinit cpu_up(unsigned int cpu)
 #endif
 
 	cpu_maps_update_begin();
+	arch_cpu_maps_update_begin();
 
 	if (cpu_hotplug_disabled) {
 		err = -EBUSY;

kernel/cpu.c int __cpuinit cpu_up(unsigned int cpu) · line 392

Quoted 5 lines
kernel/cpu.c +1 -0
@@ -377,6 +388,7 @@ int __cpuinit cpu_up(unsigned int cpu)
 
 out:
 	cpu_maps_update_done();
+	arch_cpu_maps_update_done();
 	return err;

Same here. See my comments above.

Thanks,
Srivatsa S. Bhat

author
#4 lore raw

PATCH 3/9 x86/i387.c: Thread xstate is initialized only on BSP once

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

init_thread_xstate() is only called on BSP once to avoid to override xstate_size.

---
 arch/x86/kernel/i387.c |    9 ++++++++-
 1 files changed, 8 insertions(+), 1 deletions(-)
arch/x86/kernel/i387.c +8 -1
index 739d859..f721a61 100644
@@ -93,6 +93,7 @@ void __cpuinit fpu_init(void)
 {
 	unsigned long cr0;
 	unsigned long cr4_mask = 0;
+	static int once = 1;
 
 	if (cpu_has_fxsr)
 		cr4_mask |= X86_CR4_OSFXSR;
@@ -107,8 +108,14 @@ void __cpuinit fpu_init(void)
 		cr0 |= X86_CR0_EM;
 	write_cr0(cr0);
 
-	if (!smp_processor_id())
+	/*
+	 * init_thread_xstate is only called on BSP once to avoid to override
+	 * xstate_size.
+	 */
+	if (!smp_processor_id() && once) {
+		once = 0;
 		init_thread_xstate();
+	}
 
 	mxcsr_feature_mask_init();
 	/* clean state in init */
Signature · 1 line
1.6.0.3
author
#5 lore raw

PATCH 4/9 x86/common.c: Init BSP data during BSP online

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

During BSP online, enable x2apic, set_numa_node, add numa mask, and enable sep cpu for BSP.

---
 arch/x86/include/asm/processor.h |    1 +
 arch/x86/kernel/cpu/common.c     |   13 ++++++++++---
 2 files changed, 11 insertions(+), 3 deletions(-)
arch/x86/include/asm/processor.h +1 -0
index b650435..4057161 100644
@@ -162,6 +162,7 @@ extern struct pt_regs *idle_regs(struct pt_regs *);
 
 extern void early_cpu_init(void);
 extern void identify_boot_cpu(void);
+extern void identify_boot_cpu_online(void);
 extern void identify_secondary_cpu(struct cpuinfo_x86 *);
 extern void print_cpu_info(struct cpuinfo_x86 *);
 extern void init_scattered_cpuid_features(struct cpuinfo_x86 *c);
arch/x86/kernel/cpu/common.c +10 -3
index aa003b1..faa3bda 100644
@@ -916,6 +916,14 @@ void __init identify_boot_cpu(void)
 #endif
 }
 
+void __cpuinit identify_boot_cpu_online(void)
+{
+	numa_add_cpu(smp_processor_id());
+#ifdef CONFIG_X86_32
+	enable_sep_cpu();
+#endif
+}
+
 void __cpuinit identify_secondary_cpu(struct cpuinfo_x86 *c)
 {
 	BUG_ON(c == &boot_cpu_data);
@@ -1163,7 +1171,7 @@ void __cpuinit cpu_init(void)
 	oist = &per_cpu(orig_ist, cpu);
 
 #ifdef CONFIG_NUMA
-	if (cpu != 0 && percpu_read(numa_node) == 0 &&
+	if (percpu_read(numa_node) == 0 &&
 	    early_cpu_to_node(cpu) != NUMA_NO_NODE)
 		set_numa_node(early_cpu_to_node(cpu));
 #endif
@@ -1195,8 +1203,7 @@ void __cpuinit cpu_init(void)
 	barrier();
 
 	x86_configure_nx();
-	if (cpu != 0)
-		enable_x2apic();
+	enable_x2apic();
 
 	/*
 	 * set up and load the per-CPU TSS
Signature · 1 line
1.6.0.3
author
#6 lore raw

PATCH 5/9 x86/mtrr/main.c: Ask the first online CPU to save mtrr

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

Ask the first online CPU to save mtrr instead of asking BSP. BSP could be offline when mtrr_save_state() is called.

---
 arch/x86/kernel/cpu/mtrr/main.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mtrr/main.c +7 -2
index 6b96110..e4c1a41 100644
@@ -695,11 +695,16 @@ void mtrr_ap_init(void)
 }
 
 /**
- * Save current fixed-range MTRR state of the BSP
+ * Save current fixed-range MTRR state of the first cpu in cpu_online_mask.
  */
 void mtrr_save_state(void)
 {
-	smp_call_function_single(0, mtrr_save_fixed_ranges, NULL, 1);
+	int first_cpu;
+
+	get_online_cpus();
+	first_cpu = cpumask_first(cpu_online_mask);
+	smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1);
+	put_online_cpus();
 }
 
 void set_mtrr_aps_delayed_init(void)
Signature · 1 line
1.6.0.3
author
#7 lore raw

PATCH 6/9 kernel/power/suspend.c,hibernate.c: Don't hibernate/suspend if CPU0 is offline

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

Because x86 BIOS requires CPU0 to resume from sleep, suspend/resume or hibernate can't be executed if CPU0 is detected offline.

---
 kernel/power/hibernate.c |    5 +++++
 kernel/power/main.c      |    5 +++++
 kernel/power/suspend.c   |    4 ++++
 3 files changed, 14 insertions(+), 0 deletions(-)
kernel/power/hibernate.c +5 -0
index 1c53f7f..b9d1a37 100644
@@ -612,6 +612,11 @@ int hibernate(void)
 	int error;
 
 	mutex_lock(&pm_mutex);
+
+	error = arch_state_check();
+	if (error)
+		goto Unlock;
+
 	/* The snapshot device should not be opened while we're running */
 	if (!atomic_add_unless(&snapshot_device_available, -1, 0)) {
 		error = -EBUSY;
kernel/power/main.c +5 -0
index a52e884..863d05d 100644
@@ -447,4 +447,9 @@ static int __init pm_init(void)
 	return sysfs_create_group(power_kobj, &attr_group);
 }
 
+int __weak arch_state_check(void)
+{
+	return 0;
+}
+
 core_initcall(pm_init);
kernel/power/suspend.c +4 -0
index fdd4263..f86ff6b 100644
@@ -284,6 +284,10 @@ int enter_state(suspend_state_t state)
 	if (!mutex_trylock(&pm_mutex))
 		return -EBUSY;
 
+	error = arch_state_check();
+	if (error)
+		goto Unlock;
+
 	printk(KERN_INFO "PM: Syncing filesystems ... ");
 	sys_sync();
 	printk("done.\n");
Signature · 1 line
1.6.0.3
author
#8 lore raw

PATCH 7/9 x86/topology.c: Support functions for BSP online/offline

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

By default, BSP can't be hotpluggable because bsp_hotpluggable is 0. Kernel parameter bsp_hotplug can enable BSP hotplug feature.

arch_cpu_maps_update_begin() and arch_cpu_maps_update_done() locks/unlocks pm_mutex. This solves cpu maps race condition between BSP online check during hibernate/suspend and BSP online/offline operations.

arch_state_check() checks if BSP is still the first online CPU.

---
 arch/x86/kernel/topology.c |   44 +++++++++++++++++++++++++++++++++++++-------
 1 files changed, 37 insertions(+), 7 deletions(-)
arch/x86/kernel/topology.c +37 -7
index 8927486..287d7b6 100644
@@ -29,23 +29,53 @@
 #include <linux/mmzone.h>
 #include <linux/init.h>
 #include <linux/smp.h>
+#include <linux/suspend.h>
 #include <asm/cpu.h>
 
 static DEFINE_PER_CPU(struct x86_cpu, cpu_devices);
 
 #ifdef CONFIG_HOTPLUG_CPU
+
+static int bsp_hotpluggable;
+
+static int __init enable_bsp_hotplug(char *str)
+{
+	bsp_hotpluggable = 1;
+	return 0;
+}
+
+__setup("bsp_hotplug", enable_bsp_hotplug);
+
+void arch_cpu_maps_update_begin(void)
+{
+	lock_system_sleep();
+}
+
+void arch_cpu_maps_update_done(void)
+{
+	unlock_system_sleep();
+}
+
+int arch_state_check(void)
+{
+	if (cpumask_first(cpu_online_mask) != 0) {
+		printk(KERN_WARNING "No CPU0.\n");
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
 int __ref arch_register_cpu(int num)
 {
 	/*
-	 * CPU0 cannot be offlined due to several
-	 * restrictions and assumptions in kernel. This basically
-	 * doesn't add a control file, one cannot attempt to offline
-	 * BSP.
+	 * Suspend/resume depends on BSP. PIC interrupts depend on BSP.
 	 *
-	 * Also certain PCI quirks require not to enable hotplug control
-	 * for all CPU's.
+	 * If the BSP depencies are under control, tell kernel to
+	 * enable BSP hotplug. This basically adds a control file and
+	 * one can attempt to offline BSP.
 	 */
-	if (num)
+	if (num || bsp_hotpluggable)
 		per_cpu(cpu_devices, num).cpu.hotpluggable = 1;
 
 	return register_cpu(&per_cpu(cpu_devices, num).cpu, num);
Signature · 1 line
1.6.0.3
#9 lore raw

Re: [PATCH 7/9] x86/topology.c: Support functions for BSP online/offline

Quoted 45 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

By default, BSP can't be hotpluggable because bsp_hotpluggable is 0. Kernel parameter bsp_hotplug can enable BSP hotplug feature.

arch_cpu_maps_update_begin() and arch_cpu_maps_update_done() locks/unlocks pm_mutex. This solves cpu maps race condition between BSP online check during hibernate/suspend and BSP online/offline operations.

arch_state_check() checks if BSP is still the first online CPU.

---
 arch/x86/kernel/topology.c |   44 +++++++++++++++++++++++++++++++++++++-------
 1 files changed, 37 insertions(+), 7 deletions(-)
arch/x86/kernel/topology.c +16 -0
index 8927486..287d7b6 100644
@@ -29,23 +29,53 @@
 #include <linux/mmzone.h>
 #include <linux/init.h>
 #include <linux/smp.h>
+#include <linux/suspend.h>
 #include <asm/cpu.h>
 
 static DEFINE_PER_CPU(struct x86_cpu, cpu_devices);
 
 #ifdef CONFIG_HOTPLUG_CPU
+
+static int bsp_hotpluggable;
+
+static int __init enable_bsp_hotplug(char *str)
+{
+	bsp_hotpluggable = 1;
+	return 0;
+}
+
+__setup("bsp_hotplug", enable_bsp_hotplug);
+
+void arch_cpu_maps_update_begin(void)
+{
+	lock_system_sleep();
+}

arch/x86/kernel/topology.c line 62

Quoted 10 lines
arch/x86/kernel/topology.c +10 -0
+
+void arch_cpu_maps_update_done(void)
+{
+	unlock_system_sleep();
+}
+
+int arch_state_check(void)
+{
+	if (cpumask_first(cpu_online_mask) != 0) {
+		printk(KERN_WARNING "No CPU0.\n");

How about a better warning, perhaps like "CPU0 is offline" or something like that?

Quoted 16 lines
arch/x86/kernel/topology.c +7 -5
+		return -ENODEV;
+	}
+
+	return 0;
+}
+
 int __ref arch_register_cpu(int num)
 {
 	/*
-	 * CPU0 cannot be offlined due to several
-	 * restrictions and assumptions in kernel. This basically
-	 * doesn't add a control file, one cannot attempt to offline
-	 * BSP.
+	 * Suspend/resume depends on BSP. PIC interrupts depend on BSP.
 	 *
-	 * Also certain PCI quirks require not to enable hotplug control

arch/x86/kernel/topology.c line 81

Quoted 10 lines
arch/x86/kernel/topology.c +4 -2
-	 * for all CPU's.
+	 * If the BSP depencies are under control, tell kernel to
+	 * enable BSP hotplug. This basically adds a control file and
+	 * one can attempt to offline BSP.
 	 */
-	if (num)
+	if (num || bsp_hotpluggable)
 		per_cpu(cpu_devices, num).cpu.hotpluggable = 1;
 
 	return register_cpu(&per_cpu(cpu_devices, num).cpu, num);

Thanks,
Srivatsa S. Bhat

#10 lore raw

Re: [PATCH 7/9] x86/topology.c: Support functions for BSP online/offline

In reply to Fenghua Yu
Quoted 27 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

By default, BSP can't be hotpluggable because bsp_hotpluggable is 0. Kernel parameter bsp_hotplug can enable BSP hotplug feature.

arch_cpu_maps_update_begin() and arch_cpu_maps_update_done() locks/unlocks pm_mutex. This solves cpu maps race condition between BSP online check during hibernate/suspend and BSP online/offline operations.

arch_state_check() checks if BSP is still the first online CPU.

---
 arch/x86/kernel/topology.c |   44 +++++++++++++++++++++++++++++++++++++-------
 1 files changed, 37 insertions(+), 7 deletions(-)
arch/x86/kernel/topology.c +1 -0
index 8927486..287d7b6 100644
@@ -29,23 +29,53 @@
 #include <linux/mmzone.h>
 #include <linux/init.h>
 #include <linux/smp.h>
+#include <linux/suspend.h>
 #include <asm/cpu.h>
 

arch/x86/kernel/topology.c line 44

Quoted 10 lines
arch/x86/kernel/topology.c +7 -0
 static DEFINE_PER_CPU(struct x86_cpu, cpu_devices);
 
 #ifdef CONFIG_HOTPLUG_CPU
+
+static int bsp_hotpluggable;
+
+static int __init enable_bsp_hotplug(char *str)
+{
+	bsp_hotpluggable = 1;
+	return 0;

Any reason why you return 0 here? Most code I have seen similar to this, return 1. I understand that anything declared using early_param() would generate a warning if it returns non-zero, but I am not exactly sure about how it behaves with __setup(). Kindly give this some thought.

arch/x86/kernel/topology.c line 48

Quoted 4 lines
arch/x86/kernel/topology.c +4 -0
+}
+
+__setup("bsp_hotplug", enable_bsp_hotplug);
+

Thanks,
Srivatsa S. Bhat

#11 lore raw

RE: [PATCH 7/9] x86/topology.c: Support functions for BSP online/offline

Quoted 11 lines

arch/x86/kernel/topology.c line 44

Quoted 6 lines
arch/x86/kernel/topology.c +6 -0
+static int bsp_hotpluggable;
+
+static int __init enable_bsp_hotplug(char *str)
+{
+	bsp_hotpluggable = 1;
+	return 0;

Any reason why you return 0 here? Most code I have seen similar to this, return 1. I understand that anything declared using early_param() would generate a warning if it returns non-zero, but I am not exactly sure about how it behaves with __setup(). Kindly give this some thought.

You're right. This function should return 1 here. The parameter setup function returns 1 for success and 0 for failure. The return value is checked in obsolete_checksetup():

                        } else if (p->setup_func(line + n))
                                return 1;

I will change this return value to 1 in the next version of the patch set.

BTW, some places in the current upstream do return 0 for success and return an error number for failure. Maybe we need to clean up them.

Thanks.

-Fenghua

author
#12 lore raw

PATCH 8/9 x86/smpboot.c: Don't offline BSP if any irq can not be migrated out of it

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

Don't offline BSP if any irq can not be migrated out of it.

Call identify_boot_cpu_online() for BSP in smp_callin() and continue to online BSP in native_cpu_up().

---
 arch/x86/kernel/smpboot.c |   43 ++++++++++++++++++++++++++++++++++++-------
 1 files changed, 36 insertions(+), 7 deletions(-)
arch/x86/kernel/smpboot.c +36 -7
index 9f548cb..83838c5 100644
@@ -136,8 +136,8 @@ EXPORT_PER_CPU_SYMBOL(cpu_info);
 atomic_t init_deasserted;
 
 /*
- * Report back to the Boot Processor.
- * Running on AP.
+ * Report back to the Boot Processor during boot time or to the caller processor
+ * during CPU online.
  */
 static void __cpuinit smp_callin(void)
 {
@@ -224,6 +224,13 @@ static void __cpuinit smp_callin(void)
 	smp_store_cpu_info(cpuid);
 
 	/*
+	 * This function won't run on the BSP during boot time. It run
+	 * on BSP only when BSP is offlined and onlined again.
+	 */
+	if (cpuid == 0)
+		identify_boot_cpu_online();
+
+	/*
 	 * This must be done before setting cpu_online_mask
 	 * or calling notify_cpu_starting.
 	 */
@@ -839,7 +846,7 @@ int __cpuinit native_cpu_up(unsigned int cpu)
 
 	pr_debug("++++++++++++++++++++=_---CPU UP  %u\n", cpu);
 
-	if (apicid == BAD_APICID || apicid == boot_cpu_physical_apicid ||
+	if (apicid == BAD_APICID ||
 	    !physid_isset(apicid, phys_cpu_present_map)) {
 		printk(KERN_ERR "%s: bad cpu %d\n", __func__, cpu);
 		return -EINVAL;
@@ -1283,12 +1290,34 @@ int native_cpu_disable(void)
 	 * Perhaps use cpufreq to drop frequency, but that could go
 	 * into generic code.
 	 *
-	 * We won't take down the boot processor on i386 due to some
+	 * We won't take down the boot processor on i386 if some
 	 * interrupts only being able to be serviced by the BSP.
-	 * Especially so if we're not using an IOAPIC	-zwane
+	 * Especially so if we're not using an IOAPIC
+	 * -original comment from zwane, changed by fyu
 	 */
-	if (cpu == 0)
-		return -EBUSY;
+	if (cpu == 0) {
+		int irq;
+		struct irq_desc *desc;
+		struct irq_data *data;
+		struct irq_chip *chip;
+
+		for_each_irq_desc(irq, desc) {
+			raw_spin_lock(&desc->lock);
+			if (!irq_has_action(irq)) {
+				raw_spin_unlock(&desc->lock);
+				continue;
+			}
+
+			data = irq_desc_get_irq_data(desc);
+			chip = irq_data_get_irq_chip(data);
+			if (!chip->irq_set_affinity) {
+				pr_debug("irq%d can't move out of BSP\n", irq);
+				raw_spin_unlock(&desc->lock);
+				return -EBUSY;
+			}
+			raw_spin_unlock(&desc->lock);
+		}
+	}
 
 	clear_local_APIC();
 
Signature · 1 line
1.6.0.3
author
#13 lore raw

PATCH 9/9 Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0 online/offline feature

In reply to Fenghua Yu

From: Fenghua Yu <[email protected]>

Add x86 CPU0 online/offline feature in the two documentations.

By default BSP is not pluggable. Kernel parameter bsp_hotplug enables BSP online/offline feature.

The documentations point out two BSP dependencies. First, resume from hibernate or suspend always starts from BSP. So hibernate and suspend are prevented if BSP is offline. Another dependency is PIC interrupts always go to BSP.

---

---
 Documentation/cpu-hotplug.txt       |   19 +++++++++++++++++++
 Documentation/kernel-parameters.txt |   13 +++++++++++++
 2 files changed, 32 insertions(+), 0 deletions(-)
Documentation/cpu-hotplug.txt +19 -0
index a20bfd4..ebd41e7 100644
@@ -207,6 +207,25 @@ by making it not-removable.
 
 In such cases you will also notice that the online file is missing under cpu0.
 
+Q: Is CPU0 removable on X86?
+A: Yes. If given kernel option bsp_hotplug, CPU0 is removable.
+
+But some features may depend on BSP. Known dependencies are:
+1. Hibernate/suspend depends on BSP. Hibernate/suspend will fail if BSP is
+offline and you need to online BSP before hibernate/suspend can continue.
+2. PIC interrupts also depend on BSP. BSP can't be removed if a PIC interrupt
+is detected.
+
+It's said poweroff/reboot may depend on BSP on some machines although I haven't
+seen any poweroff/reboot failure so far after BSP is offline on a few tested
+machines.
+
+Please let me know if you know or see any other dependencies of BSP.
+
+If the dependencies are under your control, you can turn on bsp_hotplug.
+
+--Fenghua Yu
+
 Q: How do i find out if a particular CPU is not removable?
 A: Depending on the implementation, some architectures may show this by the
 absence of the "online" file. This is done if it can be determined ahead of
Documentation/kernel-parameters.txt +13 -0
index a0c5c5f..b1c7e68 100644
@@ -1842,6 +1842,19 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 
 	nox2apic	[X86-64,APIC] Do not enable x2APIC mode.
 
+	bsp_hotplug	[X86] BSP (aka CPU0) is hotpluggable.
+			Some features may depend on BSP. Known dependencies are
+			1. Suspend/resume depends on BSP. Suspend will fail if
+			BSP is offline and you need to online BSP before
+			suspend/resume.
+			2. PIC interrupts also depend on BSP. BSP can't be
+			removed if a PIC interrupt is detected.
+			It's said poweroff/reboot may depend on BSP on some
+			machines although I haven't seen such issues so far
+			after BSP is offline on a few tested machines.
+			If the dependencies are under your control, you can
+			turn on bsp_hotplug.
+
 	nptcg=		[IA-64] Override max number of concurrent global TLB
 			purges which is reported from either PAL_VM_SUMMARY or
 			SAL PALO.
Signature · 1 line
1.6.0.3
#14 lore raw

Re: [PATCH 9/9] Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0 online/offline feature

Quoted 23 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

Add x86 CPU0 online/offline feature in the two documentations.

By default BSP is not pluggable. Kernel parameter bsp_hotplug enables BSP online/offline feature.

The documentations point out two BSP dependencies. First, resume from hibernate or suspend always starts from BSP. So hibernate and suspend are prevented if BSP is offline. Another dependency is PIC interrupts always go to BSP.

---

---
 Documentation/cpu-hotplug.txt       |   19 +++++++++++++++++++
 Documentation/kernel-parameters.txt |   13 +++++++++++++
 2 files changed, 32 insertions(+), 0 deletions(-)
Documentation/cpu-hotplug.txt +0 -0
index a20bfd4..ebd41e7 100644

Documentation/cpu-hotplug.txt by making it not-removable. · line 213

Quoted 7 lines
Documentation/cpu-hotplug.txt +4 -0
@@ -207,6 +207,25 @@ by making it not-removable.
 
 In such cases you will also notice that the online file is missing under cpu0.
 
+Q: Is CPU0 removable on X86?
+A: Yes. If given kernel option bsp_hotplug, CPU0 is removable.
+
+But some features may depend on BSP. Known dependencies are:

Drop "may".

Quoted 23 lines
Documentation/cpu-hotplug.txt +15 -0
+1. Hibernate/suspend depends on BSP. Hibernate/suspend will fail if BSP is
+offline and you need to online BSP before hibernate/suspend can continue.
+2. PIC interrupts also depend on BSP. BSP can't be removed if a PIC interrupt
+is detected.
+
+It's said poweroff/reboot may depend on BSP on some machines although I haven't
+seen any poweroff/reboot failure so far after BSP is offline on a few tested
+machines.
+
+Please let me know if you know or see any other dependencies of BSP.
+
+If the dependencies are under your control, you can turn on bsp_hotplug.
+
+--Fenghua Yu
+
 Q: How do i find out if a particular CPU is not removable?
 A: Depending on the implementation, some architectures may show this by the
 absence of the "online" file. This is done if it can be determined ahead of
Documentation/kernel-parameters.txt +0 -0
index a0c5c5f..b1c7e68 100644

Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 1846

Quoted 5 lines
Documentation/kernel-parameters.txt +2 -0
@@ -1842,6 +1842,19 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 
 	nox2apic	[X86-64,APIC] Do not enable x2APIC mode.
 
+	bsp_hotplug	[X86] BSP (aka CPU0) is hotpluggable.
+			Some features may depend on BSP. Known dependencies are

Drop "may". Then add ":" after "are". I.e.,

Some features depend on BSP. Known dependencies are:

Quoted 14 lines
Documentation/kernel-parameters.txt +11 -0
+			1. Suspend/resume depends on BSP. Suspend will fail if
+			BSP is offline and you need to online BSP before
+			suspend/resume.
+			2. PIC interrupts also depend on BSP. BSP can't be
+			removed if a PIC interrupt is detected.
+			It's said poweroff/reboot may depend on BSP on some
+			machines although I haven't seen such issues so far
+			after BSP is offline on a few tested machines.
+			If the dependencies are under your control, you can
+			turn on bsp_hotplug.
+
 	nptcg=		[IA-64] Override max number of concurrent global TLB
 			purges which is reported from either PAL_VM_SUMMARY or
 			SAL PALO.
Signature · 2 lines
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
#15 lore raw
In reply to Fenghua Yu
Quoted 9 lines · Fenghua Yu

From: Fenghua Yu <[email protected]>

BSP or CPU0 has been the last obstacle to CPU hotplug on x86. This patch set implements BSP online and offline and removes this obstacle to CPU hotplug.

RAS needs the feature. If socket0 needs to be hotplugged for any reason (any thread on socket0 is bad, shared cache issue, uncore issue, etc), CPU0 is required to be offline or hot replaced to keep the system run.

Quoted 15 lines

Fenghua Yu (9):
include/linux/cpu.h: Define architecture dependent cpu map update and
state check functions
kernel/cpu.c: Add arch dependent cpu map update functions x86/i387.c: Thread xstate is initialized only on BSP once x86/common.c: Init BSP data during BSP online x86/mtrr/main.c: Ask the first online CPU to save mtrr kernel/power/suspend.c,hibernate.c: Don't hibernate/suspend if CPU0
is offline
x86/topology.c: Support functions for BSP online/offline x86/smpboot.c: Don't offline BSP if any irq can not be migrated out
of it
Documentations/cpu-hotplug.tx, kernel-parameters.txt: Add x86 CPU0
online/offline feature

The norm usually followed is to update the version number of the patch or patchset at each iteration and mention the significant changes between the versions in the changelog/cover-letter (and possibly give links to older versions). This would make reviewing patches much more easier.

Quoted 6 lines
 Documentation/cpu-hotplug.txt       |   19 +++++++++++++++
 Documentation/kernel-parameters.txt |   13 ++++++++++
 arch/x86/include/asm/processor.h    |    1 +
 arch/x86/kernel/cpu/common.c        |   13 ++++++++--
 arch/x86/kernel/cpu/mtrr/main.c     |    9 +++++-
 arch/x86/kernel/i387.c              |    9 ++++++-
Quoted 9 lines
 arch/x86/kernel/smpboot.c           |   43 ++++++++++++++++++++++++++++-----
 arch/x86/kernel/topology.c          |   44 +++++++++++++++++++++++++++++-----
 include/linux/cpu.h                 |    6 ++++
 kernel/cpu.c                        |   12 +++++++++
 kernel/power/hibernate.c            |    5 ++++
 kernel/power/main.c                 |    5 ++++
 kernel/power/suspend.c              |    4 +++
 13 files changed, 163 insertions(+), 20 deletions(-)

Thanks,
Srivatsa S. Bhat