Skip to contentSkip to conversation

kernel/cpu: restart cpu_up when hotplug is disabled

RFC PATCH In review Joel Savitz 3 participants 5 messages Apr 18, 2022 – Apr 21, 2022 CPU hotplug

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 kernel/cpu: restart cpu_up when hotplug is disabled

The cpu hotplug path may be utilized while hotplug is disabled for a brief moment leading to failures. As an example, attempts to perform cpu hotplug by userspace soon after boot may race with pci_device_probe leading to inconsistent results.

Proposed idea:
Call restart_syscall instead of returning -EBUSY since cpu_hotplug_disabled seems to only have a positive value for short, temporary amounts of time.

Does anyone see any serious problems with this?

---
 kernel/cpu.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/cpu.c +4 -1
index 5797c2a7a93f..2992c7d1d24e 100644
@@ -35,6 +35,7 @@
 #include <linux/percpu-rwsem.h>
 #include <linux/cpuset.h>
 #include <linux/random.h>
+#include <linux/delay.h>
 
 #include <trace/events/power.h>
 #define CREATE_TRACE_POINTS
@@ -1401,7 +1402,9 @@ static int cpu_up(unsigned int cpu, enum cpuhp_state target)
 	cpu_maps_update_begin();
 
 	if (cpu_hotplug_disabled) {
-		err = -EBUSY;
+		/* avoid busy looping (5ms of sleep should be enough) */
+		msleep(5);
+		err = restart_syscall();
 		goto out;
 	}
 	if (!cpu_smt_allowed(cpu)) {
Signature · 1 line
2.27.0
#1 lore raw
Quoted 4 lines · Joel Savitz

The cpu hotplug path may be utilized while hotplug is disabled for a brief moment leading to failures. As an example, attempts to perform cpu hotplug by userspace soon after boot may race with pci_device_probe leading to inconsistent results.

You might want to extend a bit in which situation we observed that issue fairly reliably.

When restricting the number of boot cpus on the kernel cmdline, e.g., via "maxcpus=2", udev will find the offline cpus when enumerating all cpus and try onlining them. Due to the race, onlining of some cpus fails e.g., when racing with pci_device_probe().

While teaching udev to not online coldplugged CPUs when "maxcpus" was specified ("policy"), it revealed the underlying issue that onlining a CPU can fail with -EBUSY in corner cases when cpu hotplug is temporarily disabled.

Quoted 26 lines

Proposed idea:
Call restart_syscall instead of returning -EBUSY since cpu_hotplug_disabled seems to only have a positive value for short, temporary amounts of time.

Does anyone see any serious problems with this?

---
 kernel/cpu.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/cpu.c +1 -0
index 5797c2a7a93f..2992c7d1d24e 100644
@@ -35,6 +35,7 @@
 #include <linux/percpu-rwsem.h>
 #include <linux/cpuset.h>
 #include <linux/random.h>
+#include <linux/delay.h>
 
 #include <trace/events/power.h>
 #define CREATE_TRACE_POINTS

kernel/cpu.c static int cpu_up(unsigned int cpu, enum cpuhp_state target) · line 1407

Quoted 7 lines
kernel/cpu.c +3 -1
@@ -1401,7 +1402,9 @@ static int cpu_up(unsigned int cpu, enum cpuhp_state target)
 	cpu_maps_update_begin();
 
 	if (cpu_hotplug_disabled) {
-		err = -EBUSY;
+		/* avoid busy looping (5ms of sleep should be enough) */
+		msleep(5);
+		err = restart_syscall();

It's worth noting that we use the same approach in lock_device_hotplug_sysfs(). It's far from perfect I would say, but we really wanted to avoid letting user space having to deal with retry logic.

For example, while memory onlining can fail with -EBUSY, it's not expected to fail during memory onlining (we only fail in very rare cases, when a memory notifier fails -- for example when kasan fails to allocate memory).

Signature · 3 lines
Thanks,

David / dhildenb
#2 lore raw
Quoted 6 lines · David Hildenbrand
Quoted 4 lines · Joel Savitz

The cpu hotplug path may be utilized while hotplug is disabled for a brief moment leading to failures. As an example, attempts to perform cpu hotplug by userspace soon after boot may race with pci_device_probe leading to inconsistent results.

Quoted 7 lines

You might want to extend a bit in which situation we observed that issue fairly reliably.

When restricting the number of boot cpus on the kernel cmdline, e.g., via "maxcpus=2", udev will find the offline cpus when enumerating all cpus and try onlining them. Due to the race, onlining of some cpus fails e.g., when racing with pci_device_probe().

maxcpus is a horrible hack and broken vs. MCE broadcasting on x86.

Quoted 4 lines

While teaching udev to not online coldplugged CPUs when "maxcpus" was specified ("policy"), it revealed the underlying issue that onlining a CPU can fail with -EBUSY in corner cases when cpu hotplug is temporarily disabled.

Right. It can fail with -EBUSY and because userspace fails to handle it gracefully we need to hack around it?

Quoted 6 lines
Quoted 6 lines

Proposed idea:
Call restart_syscall instead of returning -EBUSY since cpu_hotplug_disabled seems to only have a positive value for short, temporary amounts of time.

Does anyone see any serious problems with this?

Yes. It's a horrible hack and wrong...

Quoted 5 lines

kernel/cpu.c static int cpu_up(unsigned int cpu, enum cpuhp_state target) · line 1407

Quoted 5 lines
kernel/cpu.c +3 -1
 	if (cpu_hotplug_disabled) {
-		err = -EBUSY;
+		/* avoid busy looping (5ms of sleep should be enough) */
+		msleep(5);
+		err = restart_syscall();

... as it sleeps with cpu_add_remove_lock held, which protects cpu_hotplug_disabled. IOW, cpu_hotplug_enable() is blocked until msleep() returns.

Quoted 2 lines

It's worth noting that we use the same approach in lock_device_hotplug_sysfs().

That does not make it any better, really.

Quoted 2 lines

It's far from perfect I would say, but we really wanted to avoid letting user space having to deal with retry logic.

What's so hard with retry logic in user space?

If you can come up with a reasonable argument why user space cannot be fixed, then there is certainly a better solution than slapping a msleep(5) at some random place into the code.

Thanks,

tglx

#3 lore raw
Quoted 8 lines
Quoted 2 lines

It's far from perfect I would say, but we really wanted to avoid letting user space having to deal with retry logic.

What's so hard with retry logic in user space?

If you can come up with a reasonable argument why user space cannot be fixed, then there is certainly a better solution than slapping a msleep(5) at some random place into the code.

Most probably you're right and we should just retry in udev. Staring at the history, it looks like the -EBUSY might have been returned forever, so user space just never really triggered it on actual CPU hotplug because it doesn't usually happen that cpu hotplug is disabled.

Signature · 3 lines
Thanks,

David / dhildenb
#4 lore raw
Quoted 5 lines · David Hildenbrand
Quoted 5 lines
Quoted 2 lines

It's far from perfect I would say, but we really wanted to avoid letting user space having to deal with retry logic.

What's so hard with retry logic in user space?

Quoted 7 lines
Quoted 3 lines

If you can come up with a reasonable argument why user space cannot be fixed, then there is certainly a better solution than slapping a msleep(5) at some random place into the code.

Most probably you're right and we should just retry in udev. Staring at the history, it looks like the -EBUSY might have been returned forever, so user space just never really triggered it on actual CPU hotplug because it doesn't usually happen that cpu hotplug is disabled.

The last part was confusing: "on actual CPU hotplug because it doesn't usually happen that cpu hotplug is disabled when onlining a CPU that was just hotplugged."

Signature · 3 lines
Thanks,

David / dhildenb