reboot: hotplug cpus in migrate_to_reboot_cpu()
RFC PATCH v2 1/1 reboot: hotplug cpus in migrate_to_reboot_cpu()
Currently system reboots would use arch specific codes (eg. smp_send_stop) to offline non reboot cpus. Some arch like arm64, arm, and x86... would set offline masks to cpu without really offline them. Thus causing some race condition and kernel warning comes out sometimes when system reboots. We can do cpu hotplug in migrate_to_reboot_cpu() to avoid this issue.
---
change log:
v1 --> v2:
Call hotplug function in #ifdef CONFIG instead of IS_ENABLED. (Reported-by: kbuild test robot <[email protected]>)
kernel warnings at reboot:
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
--- include/linux/cpu.h | 1 + kernel/cpu.c | 17 +++++++++++++++++ kernel/reboot.c | 8 ++++++-- 3 files changed, 24 insertions(+), 2 deletions(-)
include/linux/cpu.h +1 -0
@@ -113,6 +113,7 @@ extern void cpu_hotplug_disable(void);extern void cpu_hotplug_enable (void );void clear_tasks_mm_cpumask (int cpu);int cpu_down (unsigned int cpu); +extern void offline_secondary_cpus (int primary);#else /* CONFIG_HOTPLUG_CPU */
kernel/cpu.c +17 -0
@@ -1058,6 +1058,23 @@ int cpu_down(unsigned int cpu) }EXPORT_SYMBOL (cpu_down); +void offline_secondary_cpus (int primary) +{ +int i, err; + +cpu_maps_update_begin (); + +for_each_online_cpu (i) { +if (i == primary) +continue ; + err =_cpu_down (i,0 ,CPUHP_OFFLINE ); +if (err) +pr_warn ("Failed to offline cpu %d\n" , i); + } + cpu_hotplug_disabled++; + +cpu_maps_update_done (); +}#else #define takedown_cpu NULL#endif /*CONFIG_HOTPLUG_CPU*/
kernel/reboot.c +6 -2
@@ -7,6 +7,7 @@#define pr_fmt (fmt) "reboot: " fmt +#include <linux/cpu.h> #include <linux/ctype.h> #include <linux/export.h> #include <linux/kexec.h>
@@ -220,8 +221,6 @@ void migrate_to_reboot_cpu(void)/* The boot cpu is always logical cpu 0 */ int cpu = reboot_cpu; -cpu_hotplug_disable (); -/* Make certain the cpu I'm about to reboot on is online */ if (!cpu_online (cpu)) cpu =cpumask_first (cpu_online_mask);
@@ -231,6 +230,11 @@ void migrate_to_reboot_cpu(void)/* Make certain I only run on the appropriate processor */ set_cpus_allowed_ptr (current,cpumask_of (cpu)); + +/* Hotplug other cpus if possible */ +#ifdef CONFIG_HOTPLUG_CPU +offline_secondary_cpus (cpu); +#endif }/**
Signature · 1 line
2.23.0.700.g56cf767bdb-goog
Quoted 10 lines · Hsin-Yi Wang
kernel/reboot.c +0
-2@@ -220,8 +221,6 @@ void migrate_to_reboot_cpu(void)/* The boot cpu is always logical cpu 0 */ int cpu = reboot_cpu; -cpu_hotplug_disable (); -/* Make certain the cpu I'm about to reboot on is online */ if (!cpu_online (cpu)) cpu =cpumask_first (cpu_online_mask);
kernel/reboot.c void migrate_to_reboot_cpu(void) · line 237
Quoted 8 lines
kernel/reboot.c +5
-0@@ -231,6 +230,11 @@ void migrate_to_reboot_cpu(void)/* Make certain I only run on the appropriate processor */ set_cpus_allowed_ptr (current,cpumask_of (cpu)); + +/* Hotplug other cpus if possible */ +#ifdef CONFIG_HOTPLUG_CPU +offline_secondary_cpus (cpu); +#endif
In general I like the idea, but shouldn't this remove the architecture code as a follow up?
Also this needs to be explicitely enabled per architecture (opt-in) and not as an unconditional operation for all architectures which support CPU hotplug.
Thanks,
tglx
Quoted 25 lines · Thomas Gleixner
Quoted 10 lines · Hsin-Yi Wang
kernel/reboot.c +0
-2@@ -220,8 +221,6 @@ void migrate_to_reboot_cpu(void)/* The boot cpu is always logical cpu 0 */ int cpu = reboot_cpu; -cpu_hotplug_disable (); -/* Make certain the cpu I'm about to reboot on is online */ if (!cpu_online (cpu)) cpu =cpumask_first (cpu_online_mask);kernel/reboot.c void migrate_to_reboot_cpu(void) · line 237
Quoted 8 lines
kernel/reboot.c +5
-0@@ -231,6 +230,11 @@ void migrate_to_reboot_cpu(void)/* Make certain I only run on the appropriate processor */ set_cpus_allowed_ptr (current,cpumask_of (cpu)); + +/* Hotplug other cpus if possible */ +#ifdef CONFIG_HOTPLUG_CPU +offline_secondary_cpus (cpu); +#endif In general I like the idea, but shouldn't this remove the architecture code as a follow up?
Quoted 7 lines
Also this needs to be explicitely enabled per architecture (opt-in) and not as an unconditional operation for all architectures which support CPU hotplug.
Thanks,
tglx
Thanks for your comment.
I'll make this another config and depends on HOTPLUG_CPU.
I did some check on how architectures implement smp_send_stop. Most architecture would loop through all other cpus in cpu_online_mask and call ipi stop to each. If they enable this config, the loop would be empty loop. In case they don't enable this config, they still need original smp_send_stop (or as a fallback if some cpus fails to offline).
Currently there are 2 architectures (csky, alpha) that call ipi on possible cpus (instead of online cpus). Besides this, I'm not sure if there's other architecture code that should be cleaned up.
Thanks,
Hsin-Yi