Skip to contentSkip to conversation

cpu/hotplug: Keep cpu hotplug disabled until the rebooting cpu is stable

PATCH In review Pingfan Liu 2 participants 4 messages May 12, 2022 – May 15, 2022

Subsystems (8)
Arm (32-bit) PowerPC RISC-V arm64 +4 more subsystems
lore
Share & tools
lore
#0 lore raw

v4 cover cpu/hotplug: Keep cpu hotplug disabled until the rebooting cpu is stable

For the architectures (arm/arm64/ia64/riscv), which relies on the cpu hot-removing mechanism to implement "kexec -e", it is important to ensure that the rebooting happens on a valid online cpu. And this logic should have been guaranteed in migrate_to_reboot_cpu().

But the current code has either contradict (resolved by [2/2]) or redundancy (resolved by [1/2]) about the logic.

V3 -> V4:
Fix the cpu hotplug enable before kexec-rebooting in powerpc. Improve commit log and the code's note.

V2 -> V3:
Taking in [2/2], which also has problem with the valid rebooting cpu. (I had sent three patches for different arches. But maybe it is better to collapse them into one and collect acks from different arches' maintainers )

Cc: Eric Biederman <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Vincent Donnefort <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: YueHaibing <[email protected]>
Cc: Baokun Li <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: kexec at lists.infradead.org
Cc: Geert Uytterhoeven <[email protected]>
Cc: Kees Cook <[email protected]>
Cc: Jens Axboe <[email protected]>
To: [email protected]
Cc: Russell King <[email protected]>
Cc: Guenter Roeck <[email protected]>

Pingfan Liu (2):
cpu/hotplug: Keep cpu hotplug disabled until the rebooting cpu is
stable
cpu/hotplug: Remove the input parameter primary_cpu of
smp_shutdown_nonboot_cpus()

 arch/arm/kernel/reboot.c          |  2 +-
 arch/arm64/kernel/process.c       |  2 +-
 arch/ia64/kernel/process.c        |  2 +-
 arch/powerpc/kexec/core_64.c      |  1 +
 arch/riscv/kernel/machine_kexec.c |  2 +-
 include/linux/cpu.h               |  4 ++--
 kernel/cpu.c                      | 14 +++++++-------
 kernel/kexec_core.c               | 11 +++++------
 8 files changed, 19 insertions(+), 19 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH v4 1/2 cpu/hotplug: Keep cpu hotplug disabled until the rebooting cpu is stable

smp_shutdown_nonboot_cpus() repeats the same code chunk as migrate_to_reboot_cpu() to ensure that the rebooting happens on a valid cpu.

	if (!cpu_online(primary_cpu))
		primary_cpu = cpumask_first(cpu_online_mask);

This is due to an unexpected cpu-down event like the following:
kernel_kexec()
migrate_to_reboot_cpu();

   cpu_hotplug_enable();
                        -----------> comes a cpu_down(this_cpu) on other cpu
   machine_shutdown();
     smp_shutdown_nonboot_cpus();    which needs to re-check "if (!cpu_online(primary_cpu))"

Although the kexec-reboot task can get through a cpu_down() on its cpu, this code looks a little confusing.

Tracing down the git history, the cpu_hotplug_enable() called by kernel_kexec() is introduced by commit 011e4b02f1da ("powerpc, kexec:
Fix "Processor X is stuck" issue during kexec from ST mode"), which wakes up all offline cpu by cpu_up(cpu). Later, it is required by the architectures(arm/arm64/ia64/riscv) which resort to cpu hot-removing to achieve kexec-reboot by smp_shutdown_nonboot_cpus()->cpu_down_maps_locked().

Hence, the cpu_hotplug_enable() in kernel_kexec() is an architecture requirement.

By deferring the cpu hotplug enable to a more proper point, where smp_shutdown_nonboot_cpus() holds cpu_add_remove_lock, the unexpected cpu-down event is squashed out and the rebooting cpu can keep unchanged. (For powerpc, no gains from this change.)

As a result, the repeated code chunk can be removed and in [2/2], the callsites of smp_shutdown_nonboot_cpus() can be consistent.

Cc Eric Biederman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Vincent Donnefort <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Mark Rutland <[email protected]>
Cc YueHaibing <[email protected]>
Cc Baokun Li <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc kexec at lists.infradead.org

To: [email protected]
To: [email protected]

---
 arch/powerpc/kexec/core_64.c |  1 +
 kernel/cpu.c                 | 10 +++++-----
 kernel/kexec_core.c          | 11 +++++------
 3 files changed, 11 insertions(+), 11 deletions(-)
arch/powerpc/kexec/core_64.c +1 -0
index 6cc7793b8420..8ccf22197f08 100644
@@ -224,6 +224,7 @@ static void wake_offline_cpus(void)
 
 static void kexec_prepare_cpus(void)
 {
+	cpu_hotplug_enable();
 	wake_offline_cpus();
 	smp_call_function(kexec_smp_down, NULL, /* wait */0);
 	local_irq_disable();
kernel/cpu.c +5 -5
index d0a9aa0b42e8..4415370f0e91 100644
@@ -1236,12 +1236,12 @@ void smp_shutdown_nonboot_cpus(unsigned int primary_cpu)
 	cpu_maps_update_begin();
 
 	/*
-	 * Make certain the cpu I'm about to reboot on is online.
-	 *
-	 * This is inline to what migrate_to_reboot_cpu() already do.
+	 * At this point, the cpu hotplug is still disabled by
+	 * migrate_to_reboot_cpu() to guarantee that the rebooting happens on
+	 * the selected CPU.  But cpu_down_maps_locked() returns -EBUSY, if
+	 * cpu_hotplug_disabled. So re-enable CPU hotplug here.
 	 */
-	if (!cpu_online(primary_cpu))
-		primary_cpu = cpumask_first(cpu_online_mask);
+	__cpu_hotplug_enable();
 
 	for_each_online_cpu(cpu) {
 		if (cpu == primary_cpu)
kernel/kexec_core.c +5 -6
index 68480f731192..1bd5a8c95a20 100644
@@ -1168,14 +1168,13 @@ int kernel_kexec(void)
 		kexec_in_progress = true;
 		kernel_restart_prepare("kexec reboot");
 		migrate_to_reboot_cpu();
-
 		/*
-		 * migrate_to_reboot_cpu() disables CPU hotplug assuming that
-		 * no further code needs to use CPU hotplug (which is true in
-		 * the reboot case). However, the kexec path depends on using
-		 * CPU hotplug again; so re-enable it here.
+		 * migrate_to_reboot_cpu() disables CPU hotplug and pin the
+		 * rebooting thread on the selected CPU. If an architecture
+		 * requires CPU hotplug to achieve kexec reboot, it should
+		 * enable the hotplug in the architecture specific code
 		 */
-		cpu_hotplug_enable();
+
 		pr_notice("Starting new kernel\n");
 		machine_shutdown();
 	}
Signature · 1 line
2.31.1
#2 lore raw

PATCH v4 2/2 cpu/hotplug: Remove the input parameter primary_cpu of smp_shutdown_nonboot_cpus()

In reply to Pingfan Liu

For all call sites of smp_shutdown_nonboot_cpus(),

$git grep smp_shutdown_nonboot_cpus -- arch | grep -v \*
arch/arm/kernel/reboot.c:94: smp_shutdown_nonboot_cpus(reboot_cpu);
arch/arm64/kernel/process.c:89: smp_shutdown_nonboot_cpus(reboot_cpu); arch/ia64/kernel/process.c:578: smp_shutdown_nonboot_cpus(reboot_cpu); arch/riscv/kernel/machine_kexec.c:135: smp_shutdown_nonboot_cpus(smp_processor_id());

As it shows, the input parameter seems a little arbitrary.

Since kernel_kexec()->migrate_to_reboot_cpu() has already pinned the rebooting thread on the selected CPU and the CPU hotplug keeps disabled before smp_shutdown_nonboot_cpus(). Let smp_shutdown_nonboot_cpus() deduce the rebooting CPU by smp_processor_id(), instead of passing the parameter primary_cpu to it.

As a result, all call sites look consistent.

Cc Thomas Gleixner <[email protected]>
Cc Geert Uytterhoeven <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Kees Cook <[email protected]>
Cc Jens Axboe <[email protected]>

To: [email protected]
Cc: Russell King <[email protected]>
Cc: Guenter Roeck <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Wolfram Sang <[email protected]>
Cc: Lee Jones <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Peter Collingbourne <[email protected]>
Cc: Marc Zyngier <[email protected]>
To: [email protected]
Cc: [email protected]

---
 arch/arm/kernel/reboot.c          | 2 +-
 arch/arm64/kernel/process.c       | 2 +-
 arch/ia64/kernel/process.c        | 2 +-
 arch/riscv/kernel/machine_kexec.c | 2 +-
 include/linux/cpu.h               | 4 ++--
 kernel/cpu.c                      | 4 ++--
 6 files changed, 8 insertions(+), 8 deletions(-)
arch/arm/kernel/reboot.c +1 -1
index 3044fcb8d073..98c961f0b377 100644
@@ -91,7 +91,7 @@ void soft_restart(unsigned long addr)
  */
 void machine_shutdown(void)
 {
-	smp_shutdown_nonboot_cpus(reboot_cpu);
+	smp_shutdown_nonboot_cpus();
 }
 
 /*
arch/arm64/kernel/process.c +1 -1
index 7fa97df55e3a..1f6498d44740 100644
@@ -86,7 +86,7 @@ void arch_cpu_idle_dead(void)
  */
 void machine_shutdown(void)
 {
-	smp_shutdown_nonboot_cpus(reboot_cpu);
+	smp_shutdown_nonboot_cpus();
 }
 
 /*
arch/ia64/kernel/process.c +1 -1
index d7a256bd9d6b..9af53ff43983 100644
@@ -575,7 +575,7 @@ cpu_halt (void)
 
 void machine_shutdown(void)
 {
-	smp_shutdown_nonboot_cpus(reboot_cpu);
+	smp_shutdown_nonboot_cpus();
 
 #ifdef CONFIG_KEXEC
 	kexec_disable_iosapic();
arch/riscv/kernel/machine_kexec.c +1 -1
index cbef0fc73afa..a30c545714cc 100644
@@ -132,7 +132,7 @@ void machine_shutdown(void)
 	local_irq_disable();
 
 #if defined(CONFIG_HOTPLUG_CPU)
-	smp_shutdown_nonboot_cpus(smp_processor_id());
+	smp_shutdown_nonboot_cpus();
 #endif
 }
 
include/linux/cpu.h +2 -2
index 54dc2f9a2d56..0506ba3ac85e 100644
@@ -127,7 +127,7 @@ extern void cpu_hotplug_enable(void);
 void clear_tasks_mm_cpumask(int cpu);
 int remove_cpu(unsigned int cpu);
 int cpu_device_down(struct device *dev);
-extern void smp_shutdown_nonboot_cpus(unsigned int primary_cpu);
+extern void smp_shutdown_nonboot_cpus(void);
 
 #else /* CONFIG_HOTPLUG_CPU */
 
@@ -140,7 +140,7 @@ static inline void lockdep_assert_cpus_held(void) { }
 static inline void cpu_hotplug_disable(void) { }
 static inline void cpu_hotplug_enable(void) { }
 static inline int remove_cpu(unsigned int cpu) { return -EPERM; }
-static inline void smp_shutdown_nonboot_cpus(unsigned int primary_cpu) { }
+static inline void smp_shutdown_nonboot_cpus(void) { }
 #endif	/* !CONFIG_HOTPLUG_CPU */
 
 #ifdef CONFIG_PM_SLEEP_SMP
kernel/cpu.c +2 -2
index 4415370f0e91..66b8af6b8a3e 100644
@@ -1228,7 +1228,7 @@ int remove_cpu(unsigned int cpu)
 }
 EXPORT_SYMBOL_GPL(remove_cpu);
 
-void smp_shutdown_nonboot_cpus(unsigned int primary_cpu)
+void smp_shutdown_nonboot_cpus(void)
 {
 	unsigned int cpu;
 	int error;
@@ -1244,7 +1244,7 @@ void smp_shutdown_nonboot_cpus(unsigned int primary_cpu)
 	__cpu_hotplug_enable();
 
 	for_each_online_cpu(cpu) {
-		if (cpu = primary_cpu)
+		if (cpu = smp_processor_id())
 			continue;
 
 		error = cpu_down_maps_locked(cpu, CPUHP_OFFLINE);
Signature · 1 line
2.31.1
maintainer Acked-by on v4 2/2
#3 lore raw

Re: [PATCHv4 2/2] cpu/hotplug: Remove the input parameter primary_cpu of smp_shutdown_nonboot_cpus()

Quoted 24 lines · Pingfan Liu

For all call sites of smp_shutdown_nonboot_cpus(),

$git grep smp_shutdown_nonboot_cpus -- arch | grep -v \*
arch/arm/kernel/reboot.c:94: smp_shutdown_nonboot_cpus(reboot_cpu);
arch/arm64/kernel/process.c:89: smp_shutdown_nonboot_cpus(reboot_cpu); arch/ia64/kernel/process.c:578: smp_shutdown_nonboot_cpus(reboot_cpu); arch/riscv/kernel/machine_kexec.c:135: smp_shutdown_nonboot_cpus(smp_processor_id());

As it shows, the input parameter seems a little arbitrary.

Since kernel_kexec()->migrate_to_reboot_cpu() has already pinned the rebooting thread on the selected CPU and the CPU hotplug keeps disabled before smp_shutdown_nonboot_cpus(). Let smp_shutdown_nonboot_cpus() deduce the rebooting CPU by smp_processor_id(), instead of passing the parameter primary_cpu to it.

As a result, all call sites look consistent.

Cc Thomas Gleixner <[email protected]>
Cc Geert Uytterhoeven <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Kees Cook <[email protected]>
Cc Jens Axboe <[email protected]>
Quoted 13 lines

To: [email protected]
Cc: Russell King <[email protected]>
Cc: Guenter Roeck <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Wolfram Sang <[email protected]>
Cc: Lee Jones <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Peter Collingbourne <[email protected]>
Cc: Marc Zyngier <[email protected]>
To: [email protected]
Cc: [email protected]

For arm64: