Skip to contentSkip to conversation

cpu/hotplug: Keep cpu hotplug disabled until the rebooting cpu is stable

PATCH Superseded Pingfan Liu 2 participants 8 messages May 9, 2022 – May 11, 2022

Subsystems (7)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover cpu/hotplug: Keep cpu hotplug disabled until the rebooting cpu is stable

For the arches (arm/arm64/ia64/riscv), which relies on the cpu hot-removing mechanism to implement "kexec -e", it is important to make sure that the rebooting happens on a valid online cpu. And this logic should have been guaranteed in migrate_to_reboot_cpu().

But the current code has either contradict (resolved by [2/2]) or redundancy (resolved by [1/2]) about the logic.

V2 -> V3:
Taking in [2/2], which also has problem with the valid rebooting cpu. (I had sent three patches for different arches. But maybe it is better to collapse them into one and collect acks from different arches' maintainers )

Cc: Eric Biederman <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Vincent Donnefort <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: YueHaibing <[email protected]>
Cc: Baokun Li <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: kexec at lists.infradead.org
Cc: Geert Uytterhoeven <[email protected]>
Cc: Kees Cook <[email protected]>
Cc: Jens Axboe <[email protected]>
To: [email protected]
Cc: Russell King <[email protected]>
Cc: Guenter Roeck <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Wolfram Sang <[email protected]>
Cc: Lee Jones <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Peter Collingbourne <[email protected]>
Cc: Marc Zyngier <[email protected]>
To: [email protected]
To: [email protected]

Pingfan Liu (2):
cpu/hotplug: Keep cpu hotplug disabled until the rebooting cpu is
stable
arm/arm64/ia64: kexec: fix the primary cpu passed to
smp_shutdown_nonboot_cpus()

 arch/arm/kernel/reboot.c    |  2 +-
 arch/arm64/kernel/process.c |  2 +-
 arch/ia64/kernel/process.c  |  2 +-
 kernel/cpu.c                | 16 ++++++++++------
 kernel/kexec_core.c         | 10 ++++------
 5 files changed, 17 insertions(+), 15 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH v3 1/2 cpu/hotplug: Keep cpu hotplug disabled until the rebooting cpu is stable

The following code chunk repeats in both migrate_to_reboot_cpu() and smp_shutdown_nonboot_cpus():

	if (!cpu_online(primary_cpu))
		primary_cpu = cpumask_first(cpu_online_mask);

This is due to a breakage like the following:
kernel_kexec()
migrate_to_reboot_cpu();

   cpu_hotplug_enable();
                        -----------> comes a cpu_down(this_cpu) on other cpu
   machine_shutdown();
     smp_shutdown_nonboot_cpus(); // re-check "if (!cpu_online(primary_cpu))" to protect against the former breakin

Although the kexec-reboot task can get through a cpu_down() on its cpu, this code looks a little confusing.

Make things straight forward by keeping cpu hotplug disabled until smp_shutdown_nonboot_cpus() holds cpu_add_remove_lock. By this way, the breakage is squashed out and the rebooting cpu can keep unchanged.

Note: this patch only affects the kexec-reboot on arches(arm/arm64/ia64/riscv) , which rely on the cpu hot-removing mechanism.

Cc Eric Biederman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc Vincent Donnefort <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Mark Rutland <[email protected]>
Cc YueHaibing <[email protected]>
Cc Baokun Li <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc Baoquan He <[email protected]>
Cc kexec at lists.infradead.org

To: [email protected]

---
 kernel/cpu.c        | 16 ++++++++++------
 kernel/kexec_core.c | 10 ++++------
 2 files changed, 14 insertions(+), 12 deletions(-)
kernel/cpu.c +10 -6
index d0a9aa0b42e8..bf901fc90329 100644
@@ -1228,20 +1228,24 @@ int remove_cpu(unsigned int cpu)
 }
 EXPORT_SYMBOL_GPL(remove_cpu);
 
+/* primary_cpu keeps unchanged after migrate_to_reboot_cpu() */
 void smp_shutdown_nonboot_cpus(unsigned int primary_cpu)
 {
 	unsigned int cpu;
 	int error;
 
+	/*
+	 * Block other cpu hotplug event, so primary_cpu is always online if
+	 * it is not touched by us
+	 */
 	cpu_maps_update_begin();
-
 	/*
-	 * Make certain the cpu I'm about to reboot on is online.
-	 *
-	 * This is inline to what migrate_to_reboot_cpu() already do.
+	 * migrate_to_reboot_cpu() disables CPU hotplug assuming that
+	 * no further code needs to use CPU hotplug (which is true in
+	 * the reboot case). However, the kexec path depends on using
+	 * CPU hotplug again; so re-enable it here.
 	 */
-	if (!cpu_online(primary_cpu))
-		primary_cpu = cpumask_first(cpu_online_mask);
+	__cpu_hotplug_enable();
 
 	for_each_online_cpu(cpu) {
 		if (cpu == primary_cpu)
kernel/kexec_core.c +4 -6
index 68480f731192..db4fa6b174e3 100644
@@ -1168,14 +1168,12 @@ int kernel_kexec(void)
 		kexec_in_progress = true;
 		kernel_restart_prepare("kexec reboot");
 		migrate_to_reboot_cpu();
-
 		/*
-		 * migrate_to_reboot_cpu() disables CPU hotplug assuming that
-		 * no further code needs to use CPU hotplug (which is true in
-		 * the reboot case). However, the kexec path depends on using
-		 * CPU hotplug again; so re-enable it here.
+		 * migrate_to_reboot_cpu() disables CPU hotplug. If an arch
+		 * relies on the cpu teardown to achieve reboot, it needs to
+		 * re-enable CPU hotplug there.
 		 */
-		cpu_hotplug_enable();
+
 		pr_notice("Starting new kernel\n");
 		machine_shutdown();
 	}
Signature · 1 line
2.31.1
#7 lore raw

PATCH v3 2/2 arm/arm64/ia64: kexec: fix the primary cpu passed to smp_shutdown_nonboot_cpus()

In reply to Pingfan Liu

kernel_kexec()->migrate_to_reboot_cpu() has already pinned the reboot thread on a valid online cpu, either reboot_cpu or the first online cpu.

So machine_shutdown() should pass smp_processor_id() of the pinned thread as the primary cpu to smp_shutdown_nonboot_cpus().

Cc Geert Uytterhoeven <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Kees Cook <[email protected]>
Cc Jens Axboe <[email protected]>

To: [email protected]
Cc: Russell King <[email protected]>
Cc: Guenter Roeck <[email protected]>
Cc: Arnd Bergmann <[email protected]>
Cc: Wolfram Sang <[email protected]>
Cc: Lee Jones <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Peter Collingbourne <[email protected]>
Cc: Marc Zyngier <[email protected]>
To: [email protected]
Cc: [email protected]

---
 arch/arm/kernel/reboot.c    | 2 +-
 arch/arm64/kernel/process.c | 2 +-
 arch/ia64/kernel/process.c  | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
arch/arm/kernel/reboot.c +1 -1
index 3044fcb8d073..898405682e08 100644
@@ -91,7 +91,7 @@ void soft_restart(unsigned long addr)
  */
 void machine_shutdown(void)
 {
-	smp_shutdown_nonboot_cpus(reboot_cpu);
+	smp_shutdown_nonboot_cpus(smp_processor_id());
 }
 
 /*
arch/arm64/kernel/process.c +1 -1
index 7fa97df55e3a..e5c130582cc6 100644
@@ -86,7 +86,7 @@ void arch_cpu_idle_dead(void)
  */
 void machine_shutdown(void)
 {
-	smp_shutdown_nonboot_cpus(reboot_cpu);
+	smp_shutdown_nonboot_cpus(smp_processor_id());
 }
 
 /*
arch/ia64/kernel/process.c +1 -1
index d7a256bd9d6b..ae732e35e7da 100644
@@ -575,7 +575,7 @@ cpu_halt (void)
 
 void machine_shutdown(void)
 {
-	smp_shutdown_nonboot_cpus(reboot_cpu);
+	smp_shutdown_nonboot_cpus(smp_processor_id());
 
 #ifdef CONFIG_KEXEC
 	kexec_disable_iosapic();
Signature · 1 line
2.31.1
#2 lore raw
In reply to Pingfan Liu
Quoted 7 lines · Pingfan Liu

The following code chunk repeats in both migrate_to_reboot_cpu() and smp_shutdown_nonboot_cpus():

	if (!cpu_online(primary_cpu))
		primary_cpu = cpumask_first(cpu_online_mask);

This is due to a breakage like the following:

I don't see what's broken here.

Quoted 9 lines

kernel_kexec()
migrate_to_reboot_cpu();

   cpu_hotplug_enable();
                        -----------> comes a cpu_down(this_cpu) on other cpu
   machine_shutdown();
     smp_shutdown_nonboot_cpus(); // re-check "if (!cpu_online(primary_cpu))" to protect against the former breakin

Although the kexec-reboot task can get through a cpu_down() on its cpu, this code looks a little confusing.

Confusing != broken.

Quoted 1 line

+/* primary_cpu keeps unchanged after migrate_to_reboot_cpu() */

This comment makes no sense.

Quoted 9 lines

void smp_shutdown_nonboot_cpus(unsigned int primary_cpu) {

kernel/cpu.c +4 -0
 	unsigned int cpu;
 	int error;
 
+	/*
+	 * Block other cpu hotplug event, so primary_cpu is always online if
+	 * it is not touched by us
+	 */

kernel/cpu.c int remove_cpu(unsigned int cpu) · line 1246

Quoted 10 lines
kernel/cpu.c +4 -4
 	cpu_maps_update_begin();
-
 	/*
-	 * Make certain the cpu I'm about to reboot on is online.
-	 *
-	 * This is inline to what migrate_to_reboot_cpu() already do.
+	 * migrate_to_reboot_cpu() disables CPU hotplug assuming that
+	 * no further code needs to use CPU hotplug (which is true in
+	 * the reboot case). However, the kexec path depends on using
+	 * CPU hotplug again; so re-enable it here.

You want to reduce confusion, but in reality this is even more confusing than before.

kernel/cpu.c int remove_cpu(unsigned int cpu) · line 1248

Quoted 4 lines
kernel/cpu.c +1 -2
 	 */
-	if (!cpu_online(primary_cpu))
-		primary_cpu = cpumask_first(cpu_online_mask);
+	__cpu_hotplug_enable();

How is this decrement solving anything? At the end of this function, the counter is incremented again. So what's the point of this exercise?

Quoted 9 lines
kernel/cpu.c +0 -0
 	for_each_online_cpu(cpu) {
 		if (cpu == primary_cpu)
kernel/kexec_core.c +0 -0
index 68480f731192..db4fa6b174e3 100644
@@ -1168,14 +1168,12 @@ int kernel_kexec(void)
 		kexec_in_progress = true;
 		kernel_restart_prepare("kexec reboot");

kernel/kexec_core.c int kernel_kexec(void) · line 1174

Quoted 10 lines
kernel/kexec_core.c +3 -5
 		migrate_to_reboot_cpu();
-
 		/*
-		 * migrate_to_reboot_cpu() disables CPU hotplug assuming that
-		 * no further code needs to use CPU hotplug (which is true in
-		 * the reboot case). However, the kexec path depends on using
-		 * CPU hotplug again; so re-enable it here.
+		 * migrate_to_reboot_cpu() disables CPU hotplug. If an arch
+		 * relies on the cpu teardown to achieve reboot, it needs to
+		 * re-enable CPU hotplug there.

What does that for arch/powerpc/kernel/kexec_machine64.c now?

Nothing, as far as I can tell. Which means you basically reverted 011e4b02f1da ("powerpc, kexec: Fix "Processor X is stuck" issue during kexec from ST mode") unless I'm completely confused.

kernel/kexec_core.c int kernel_kexec(void) · line 1178

Quoted 2 lines
kernel/kexec_core.c +0 -1
 		 */
-		cpu_hotplug_enable();

This is tinkering at best. Can we please sit down and rethink this whole machinery instead of applying random duct tape to it?

Thanks,

tglx

#3 lore raw
Quoted 5 lines · Thomas Gleixner

kernel/kexec_core.c int kernel_kexec(void) · line 1178

Quoted 1 line · Pingfan Liu
kernel/kexec_core.c +0 -1
-		cpu_hotplug_enable();

This is tinkering at best. Can we please sit down and rethink this whole machinery instead of applying random duct tape to it?

That said, I still have not figured out which real world problem you are actually trying to solve.

Thanks,

tglx

#4 lore raw
In reply to Thomas Gleixner
Quoted 11 lines · Thomas Gleixner
Quoted 7 lines · Pingfan Liu

The following code chunk repeats in both migrate_to_reboot_cpu() and smp_shutdown_nonboot_cpus():

	if (!cpu_online(primary_cpu))
		primary_cpu = cpumask_first(cpu_online_mask);

This is due to a breakage like the following:

I don't see what's broken here.

No, no broken. Could it be better to replace 'breakage' with 'breakin'?

Quoted 7 lines
Quoted 7 lines

kernel_kexec()
migrate_to_reboot_cpu();

   cpu_hotplug_enable();
                        -----------> comes a cpu_down(this_cpu) on other cpu
   machine_shutdown();
     smp_shutdown_nonboot_cpus(); // re-check "if (!cpu_online(primary_cpu))" to protect against the former breakin
Quoted 5 lines
Quoted 2 lines

Although the kexec-reboot task can get through a cpu_down() on its cpu, this code looks a little confusing.

Confusing != broken.

Yes. And it only recurs confusing.

Quoted 4 lines
Quoted 1 line

+/* primary_cpu keeps unchanged after migrate_to_reboot_cpu() */

This comment makes no sense.

Since migrate_to_reboot_cpu() disables cpu hotplug, so the selected valid online cpu -- primary_cpu keeps unchange.

Quoted 14 lines
Quoted 14 lines

void smp_shutdown_nonboot_cpus(unsigned int primary_cpu) {

(quoted diff) +4 -3
 	unsigned int cpu;
 	int error;
 
+	/*
+	 * Block other cpu hotplug event, so primary_cpu is always online if
+	 * it is not touched by us
+	 */
 	cpu_maps_update_begin();
-
 	/*
-	 * Make certain the cpu I'm about to reboot on is online.
-	 *
Quoted 9 lines

Review

Quoted 5 lines
(quoted diff) +4 -1
-	 * This is inline to what migrate_to_reboot_cpu() already do.
+	 * migrate_to_reboot_cpu() disables CPU hotplug assuming that
+	 * no further code needs to use CPU hotplug (which is true in
+	 * the reboot case). However, the kexec path depends on using
+	 * CPU hotplug again; so re-enable it here.

You want to reduce confusion, but in reality this is even more confusing than before.

This __cpu_hotplug_enable() can be considered to defer from kernel_kexec() to arch-dependent code chunk (here), which is a more proper point.

Could it make things better by rephrasing the words as the following?
migrate_to_reboot_cpu() disables CPU hotplug to prevent the selected reboot cpu from disappearing. But arches need cpu_down to hot remove cpus except rebooting-cpu, so re-enabling cpu hotplug again.

Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +1 -2
 	 */
-	if (!cpu_online(primary_cpu))
-		primary_cpu = cpumask_first(cpu_online_mask);
+	__cpu_hotplug_enable();

How is this decrement solving anything? At the end of this function, the counter is incremented again. So what's the point of this exercise?

This decrement enables the cpu hot-removing. Since smp_shutdown_nonboot_cpus()->cpu_down_maps_locked(), if cpu_hotplug_disabled, it returns -EBUSY.

On the other hand, at the end of this function, cpu_hotplug_disabled++, which aims to prevent any new coming cpu, since machine_kexec() assumes to execute on the only rebooting-cpu, any dangling cpu has unexpected result.

Quoted 22 lines
Quoted 9 lines
(quoted diff) +0 -0
 	for_each_online_cpu(cpu) {
 		if (cpu == primary_cpu)
kernel/kexec_core.c +0 -0
index 68480f731192..db4fa6b174e3 100644
@@ -1168,14 +1168,12 @@ int kernel_kexec(void)
 		kexec_in_progress = true;
 		kernel_restart_prepare("kexec reboot");

kernel/kexec_core.c int kernel_kexec(void) · line 1174

Quoted 10 lines
kernel/kexec_core.c +3 -5
 		migrate_to_reboot_cpu();
-
 		/*
-		 * migrate_to_reboot_cpu() disables CPU hotplug assuming that
-		 * no further code needs to use CPU hotplug (which is true in
-		 * the reboot case). However, the kexec path depends on using
-		 * CPU hotplug again; so re-enable it here.
+		 * migrate_to_reboot_cpu() disables CPU hotplug. If an arch
+		 * relies on the cpu teardown to achieve reboot, it needs to
+		 * re-enable CPU hotplug there.

What does that for arch/powerpc/kernel/kexec_machine64.c now?

Quoted 4 lines

Nothing, as far as I can tell. Which means you basically reverted 011e4b02f1da ("powerpc, kexec: Fix "Processor X is stuck" issue during kexec from ST mode") unless I'm completely confused.

Oops. Forget about powerpc. Considering the cpu hotplug is an arch-dependent feature in machine_shutdown(), as x86 does not need it.

What about supplying an cpu hotplug enable in the powerpc machine_kexec implement.

arch/powerpc/kexec/core_64.c +1 -0
index 6cc7793b8420..8ccf22197f08 100644
@@ -224,6 +224,7 @@ static void wake_offline_cpus(void)
 
 static void kexec_prepare_cpus(void)
 {
+       cpu_hotplug_enable();
        wake_offline_cpus();
        smp_call_function(kexec_smp_down, NULL, /* wait */0);
        local_irq_disable();
Quoted 6 lines

kernel/kexec_core.c int kernel_kexec(void) · line 1178

Quoted 2 lines
kernel/kexec_core.c +0 -1
 		 */
-		cpu_hotplug_enable();

This is tinkering at best. Can we please sit down and rethink this whole machinery instead of applying random duct tape to it?

I try to make code look consistent. As in the current source tree. There are three sequential sites trying to resolve the valid rebooting cpu:

1.
void migrate_to_reboot_cpu(void)

{
	/* The boot cpu is always logical cpu 0 */
	int cpu = reboot_cpu;

	cpu_hotplug_disable();

	/* Make certain the cpu I'm about to reboot on is online */
	if (!cpu_online(cpu))
		cpu = cpumask_first(cpu_online_mask);

...
}

2. In each arches machine_shutdown(), the input param primary_cpu for smp_shutdown_nonboot_cpus() looks a little arbitrary.

$git grep smp_shutdown_nonboot_cpus -- arch | grep -v \*
arch/arm/kernel/reboot.c:94: smp_shutdown_nonboot_cpus(reboot_cpu);
arch/arm64/kernel/process.c:89: smp_shutdown_nonboot_cpus(reboot_cpu); arch/ia64/kernel/process.c:578: smp_shutdown_nonboot_cpus(reboot_cpu); arch/riscv/kernel/machine_kexec.c:135: smp_shutdown_nonboot_cpus(smp_processor_id());

3. finally
smp_shutdown_nonboot_cpus(primary_cpu)
{
...

	if (!cpu_online(primary_cpu))
		primary_cpu = cpumask_first(cpu_online_mask);

...
}

With this series, the 3rd can be removed, and the 2nd can be unified to smp_processor_id().

Thanks,

Pingfan

#5 lore raw
Quoted 11 lines · Pingfan Liu
Quoted 7 lines · Thomas Gleixner
Quoted 3 lines · Pingfan Liu

The following code chunk repeats in both migrate_to_reboot_cpu() and smp_shutdown_nonboot_cpus():
This is due to a breakage like the following:

I don't see what's broken here.

No, no broken. Could it be better to replace 'breakage' with 'breakin'?

There is no break-in. There is a phase where CPU hotplug is reenabled, which might be avoided.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

+/* primary_cpu keeps unchanged after migrate_to_reboot_cpu() */

This comment makes no sense.

Since migrate_to_reboot_cpu() disables cpu hotplug, so the selected valid online cpu -- primary_cpu keeps unchange.

So what is that parameter for then? If migrate_to_reboot_cpu() ensured that the current task is on the reboot CPU then this parameter is useless, no?

Quoted 24 lines
Quoted 14 lines
Quoted 14 lines

void smp_shutdown_nonboot_cpus(unsigned int primary_cpu) {

(quoted diff) +4 -3
 	unsigned int cpu;
 	int error;
 
+	/*
+	 * Block other cpu hotplug event, so primary_cpu is always online if
+	 * it is not touched by us
+	 */
 	cpu_maps_update_begin();
-
 	/*
-	 * Make certain the cpu I'm about to reboot on is online.
-	 *
Quoted 9 lines

Review

Quoted 5 lines
(quoted diff) +4 -1
-	 * This is inline to what migrate_to_reboot_cpu() already do.
+	 * migrate_to_reboot_cpu() disables CPU hotplug assuming that
+	 * no further code needs to use CPU hotplug (which is true in
+	 * the reboot case). However, the kexec path depends on using
+	 * CPU hotplug again; so re-enable it here.

You want to reduce confusion, but in reality this is even more confusing than before.

Quoted 7 lines

This __cpu_hotplug_enable() can be considered to defer from kernel_kexec() to arch-dependent code chunk (here), which is a more proper point.

Could it make things better by rephrasing the words as the following?
migrate_to_reboot_cpu() disables CPU hotplug to prevent the selected reboot cpu from disappearing. But arches need cpu_down to hot remove cpus except rebooting-cpu, so re-enabling cpu hotplug again.

Can you please use proper words. arches is not a word and it's closer to the plural of arch, than to the word architecture. This is not twitter.

And no, the architectures do not need cpu_down() at all. This very function smp_shutdown_nonboot_cpus() invokes cpu_down_maps_locked() to shut down the non boot CPUs. That fails when cpu_hotplug_disabled != 0.

Quoted 11 lines
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +1 -2
 	 */
-	if (!cpu_online(primary_cpu))
-		primary_cpu = cpumask_first(cpu_online_mask);
+	__cpu_hotplug_enable();

How is this decrement solving anything? At the end of this function, the counter is incremented again. So what's the point of this exercise?

This decrement enables the cpu hot-removing. Since smp_shutdown_nonboot_cpus()->cpu_down_maps_locked(), if cpu_hotplug_disabled, it returns -EBUSY.

Correct, so why can't you spell that out in concise words in the first place right at that comment which reenables hotplug?

Quoted 9 lines
Quoted 6 lines

What does that for arch/powerpc/kernel/kexec_machine64.c now?

Nothing, as far as I can tell. Which means you basically reverted 011e4b02f1da ("powerpc, kexec: Fix "Processor X is stuck" issue during kexec from ST mode") unless I'm completely confused.

Oops. Forget about powerpc. Considering the cpu hotplug is an arch-dependent feature in machine_shutdown(), as x86 does not need it.

It's not a feature, it's a architecture specific requirement. x86 is irrelevant here because this is a powerpc requirement.

Quoted 4 lines
Quoted 3 lines

This is tinkering at best. Can we please sit down and rethink this whole machinery instead of applying random duct tape to it?

I try to make code look consistent.

Emphasis on try. So far the attempt failed and resulted in a regression.

Thanks,

tglx

#6 lore raw
Quoted 7 lines · Thomas Gleixner
Quoted 6 lines · Pingfan Liu
Quoted 5 lines · Thomas Gleixner
Quoted 3 lines · Pingfan Liu

The following code chunk repeats in both migrate_to_reboot_cpu() and smp_shutdown_nonboot_cpus():
This is due to a breakage like the following:

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

I don't see what's broken here.

No, no broken. Could it be better to replace 'breakage' with 'breakin'?

There is no break-in. There is a phase where CPU hotplug is reenabled, which might be avoided.

OK, I will rephrase like this.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

+/* primary_cpu keeps unchanged after migrate_to_reboot_cpu() */

This comment makes no sense.

Quoted 7 lines
Quoted 2 lines

Since migrate_to_reboot_cpu() disables cpu hotplug, so the selected valid online cpu -- primary_cpu keeps unchange.

So what is that parameter for then? If migrate_to_reboot_cpu() ensured that the current task is on the reboot CPU then this parameter is useless, no?

Yes, it is useless after this patch. I will post V4 to kill it.

Quoted 27 lines
Quoted 27 lines
Quoted 14 lines
Quoted 14 lines

void smp_shutdown_nonboot_cpus(unsigned int primary_cpu) {

(quoted diff) +4 -3
 	unsigned int cpu;
 	int error;
 
+	/*
+	 * Block other cpu hotplug event, so primary_cpu is always online if
+	 * it is not touched by us
+	 */
 	cpu_maps_update_begin();
-
 	/*
-	 * Make certain the cpu I'm about to reboot on is online.
-	 *
Quoted 9 lines

Review

Quoted 5 lines
(quoted diff) +4 -1
-	 * This is inline to what migrate_to_reboot_cpu() already do.
+	 * migrate_to_reboot_cpu() disables CPU hotplug assuming that
+	 * no further code needs to use CPU hotplug (which is true in
+	 * the reboot case). However, the kexec path depends on using
+	 * CPU hotplug again; so re-enable it here.

You want to reduce confusion, but in reality this is even more confusing than before.

This __cpu_hotplug_enable() can be considered to defer from kernel_kexec() to arch-dependent code chunk (here), which is a more proper point.

Quoted 8 lines
Quoted 4 lines

Could it make things better by rephrasing the words as the following?
migrate_to_reboot_cpu() disables CPU hotplug to prevent the selected reboot cpu from disappearing. But arches need cpu_down to hot remove cpus except rebooting-cpu, so re-enabling cpu hotplug again.

Can you please use proper words. arches is not a word and it's closer to the plural of arch, than to the word architecture. This is not twitter.

OK, I will correct it.

Quoted 4 lines

And no, the architectures do not need cpu_down() at all. This very function smp_shutdown_nonboot_cpus() invokes cpu_down_maps_locked() to shut down the non boot CPUs. That fails when cpu_hotplug_disabled != 0.

Yes. I will pay attention to the accuracy of the description.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +1 -2
 	 */
-	if (!cpu_online(primary_cpu))
-		primary_cpu = cpumask_first(cpu_online_mask);
+	__cpu_hotplug_enable();

How is this decrement solving anything? At the end of this function, the counter is incremented again. So what's the point of this exercise?

Quoted 7 lines
Quoted 3 lines

This decrement enables the cpu hot-removing. Since smp_shutdown_nonboot_cpus()->cpu_down_maps_locked(), if cpu_hotplug_disabled, it returns -EBUSY.

Correct, so why can't you spell that out in concise words in the first place right at that comment which reenables hotplug?

OK, thanks for the suggestion.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

What does that for arch/powerpc/kernel/kexec_machine64.c now?

Nothing, as far as I can tell. Which means you basically reverted 011e4b02f1da ("powerpc, kexec: Fix "Processor X is stuck" issue during kexec from ST mode") unless I'm completely confused.

Quoted 6 lines
Quoted 2 lines

Oops. Forget about powerpc. Considering the cpu hotplug is an arch-dependent feature in machine_shutdown(), as x86 does not need it.

It's not a feature, it's a architecture specific requirement. x86 is irrelevant here because this is a powerpc requirement.

Yes.

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

This is tinkering at best. Can we please sit down and rethink this whole machinery instead of applying random duct tape to it?

I try to make code look consistent.

Emphasis on try. So far the attempt failed and resulted in a regression.

I will fix the powerpc issue and post V4 after a test.

Thanks for your precious time.

Best Regards,

Pingfan