MIPS: smp: report dying CPU to RCU in stop_this_cpu()
Subsystems (3)
PATCH v2 1/1 MIPS: smp: report dying CPU to RCU in stop_this_cpu()
smp_send_stop() parks all secondary CPUs in stop_this_cpu(). The function marks the CPU offline for the scheduler via set_cpu_online(false) but never informs RCU, so RCU keeps expecting a quiescent state from CPUs that are now spinning forever with interrupts disabled.
As long as nothing waits for an RCU grace period after smp_send_stop() this is harmless, which is why it went unnoticed. Since commit 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT") however, irq_work_sync() calls synchronize_rcu() on architectures without an irq_work self-IPI, i.e. where arch_irq_work_has_interrupt() returns false. That is the asm-generic default used by MIPS. Any irq_work_sync() issued in the reboot/shutdown path after smp_send_stop() then blocks on a grace period that can never complete, hanging the reboot:
WARNING: CPU: 0 PID: 15 at kernel/irq_work.c:144 irq_work_queue_on ... rcu: INFO: rcu_sched detected stalls on CPUs/tasks: rcu: Offline CPU 1 blocking current GP. rcu: Offline CPU 2 blocking current GP. rcu: Offline CPU 3 blocking current GP.
This issue was noticed on several Realtek MIPS switch SoCs (MIPS interAptiv) and came up during kernel bump downstream in OpenWrt from 6.18.33 to 6.18.34, after the backport of the patch to the 6.18 stable branch. The patch also has been backported all the way back to 6.1.
Call rcutree_report_cpu_dead() once interrupts are disabled, mirroring the generic CPU-hotplug offline path, so RCU stops waiting on the parked CPUs and grace periods can still complete. MIPS shuts down all CPUs here without going through the CPU-hotplug mechanism, so this report is not otherwise issued. Reporting a dying CPU to RCU outside the regular hotplug offline path is not unprecedented: arm64 does the same in cpu_die_early(). There it is an exception for a CPU that was coming online and is aborting bringup, rather than the default shutdown action as on MIPS.
arch/mips/kernel/smp.c +2 -0
@@ -20,6 +20,7 @@#include <linux/sched/mm.h> #include <linux/cpumask.h> #include <linux/cpu.h> +#include <linux/rcupdate.h> #include <linux/err.h> #include <linux/ftrace.h> #include <linux/irqdomain.h>
@@ -422,6 +423,7 @@ static void stop_this_cpu(void *dummy)set_cpu_online (smp_processor_id (), false);calculate_cpu_foreign_map ();local_irq_disable (); +rcutree_report_cpu_dead ();while (1 ); }
Signature · 1 line
2.51.0
Hi, Jonas,
Quoted 15 lines · Jonas Jelonek
smp_send_stop()parks all secondary CPUs instop_this_cpu(). The function marks the CPU offline for the scheduler viaset_cpu_online(false)but never informs RCU, so RCU keeps expecting a quiescent state from CPUs that are now spinning forever with interrupts disabled.As long as nothing waits for an RCU grace period after
smp_send_stop()this is harmless, which is why it went unnoticed. Since commit91840be8f710("irq_work: Fix use-after-free inirq_work_single()onPREEMPT_RT") however,irq_work_sync()callssynchronize_rcu()on architectures without anirq_workself-IPI, i.e. wherearch_irq_work_has_interrupt()returns false. That is the asm-generic default used by MIPS. Anyirq_work_sync()issued in the reboot/shutdown path aftersmp_send_stop()then blocks on a grace period that can never complete, hanging the reboot:
Quoted 6 lines
WARNING: CPU: 0 PID: 15 at kernel/irq_work.c:144 irq_work_queue_on ... rcu: INFO: rcu_sched detected stalls on CPUs/tasks: rcu: Offline CPU 1 blocking current GP. rcu: Offline CPU 2 blocking current GP. rcu: Offline CPU 3 blocking current GP.
In theory LoongArch has the same problem, but I cannot reproduce, should I enable PREEMPT_RT? Or there are some special configurations?
Huacai
Quoted 43 lines
This issue was noticed on several Realtek MIPS switch SoCs (MIPS interAptiv) and came up during kernel bump downstream in OpenWrt from 6.18.33 to 6.18.34, after the backport of the patch to the 6.18 stable branch. The patch also has been backported all the way back to 6.1.
Call
rcutree_report_cpu_dead()once interrupts are disabled, mirroring the generic CPU-hotplug offline path, so RCU stops waiting on the parked CPUs and grace periods can still complete. MIPS shuts down all CPUs here without going through the CPU-hotplug mechanism, so this report is not otherwise issued. Reporting a dying CPU to RCU outside the regular hotplug offline path is not unprecedented: arm64 does the same incpu_die_early(). There it is an exception for a CPU that was coming online and is aborting bringup, rather than the default shutdown action as on MIPS.Fixes 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT")Signed-off-by Jonas Jelonek <[email protected]>arch/mips/kernel/smp.c +2
-1@@ -20,6 +20,7 @@#include <linux/sched/mm.h> #include <linux/cpumask.h> #include <linux/cpu.h> +#include <linux/rcupdate.h> #include <linux/err.h> #include <linux/ftrace.h> #include <linux/irqdomain.h> @@ -422,6 +423,7 @@ static void stop_this_cpu(void *dummy)set_cpu_online (smp_processor_id (), false);calculate_cpu_foreign_map ();local_irq_disable (); +rcutree_report_cpu_dead ();while (1 ); } --2.51.0
Hi Huacai,
sorry for the reply delay.
Quoted 3 lines · Huacai Chen
[...]
In theory LoongArch has the same problem, but I cannot reproduce, should I enablePREEMPT_RT? Or there are some special configurations?
Sadly I cannot help with that. For MIPS, this seems to be the default behavior.
Quoted 40 lines
Huacai
Quoted 38 lines
This issue was noticed on several Realtek MIPS switch SoCs (MIPS interAptiv) and came up during kernel bump downstream in OpenWrt from 6.18.33 to 6.18.34, after the backport of the patch to the 6.18 stable branch. The patch also has been backported all the way back to 6.1.
Call
rcutree_report_cpu_dead()once interrupts are disabled, mirroring the generic CPU-hotplug offline path, so RCU stops waiting on the parked CPUs and grace periods can still complete. MIPS shuts down all CPUs here without going through the CPU-hotplug mechanism, so this report is not otherwise issued. Reporting a dying CPU to RCU outside the regular hotplug offline path is not unprecedented: arm64 does the same incpu_die_early(). There it is an exception for a CPU that was coming online and is aborting bringup, rather than the default shutdown action as on MIPS.Fixes 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT")Signed-off-by Jonas Jelonek <[email protected]>arch/mips/kernel/smp.c +2
-0@@ -20,6 +20,7 @@#include <linux/sched/mm.h> #include <linux/cpumask.h> #include <linux/cpu.h> +#include <linux/rcupdate.h> #include <linux/err.h> #include <linux/ftrace.h> #include <linux/irqdomain.h> @@ -422,6 +423,7 @@ static void stop_this_cpu(void *dummy)set_cpu_online (smp_processor_id (), false);calculate_cpu_foreign_map ();local_irq_disable (); +rcutree_report_cpu_dead ();while (1 ); }
Quoted 4 lines
Quoted 4 lines
--
2.51.0
Best,
Jonas
Quoted 5 lines · Jonas Jelonek
Hi Huacai,
sorry for the reply delay.
Quoted 7 lines
Quoted 3 lines · Huacai Chen
[...]
In theory LoongArch has the same problem, but I cannot reproduce, should I enablePREEMPT_RT? Or there are some special configurations?Sadly I cannot help with that. For MIPS, this seems to be the default behavior.
This patch fixes 91840be8f710, and 91840be8f710 adds synchronize_rcu() in irq_work_sync(). Your problem is caused by this synchronize_rcu(), right?
However, synchronize_rcu() only gets called in the IS_ENABLED(CONFIG_PREEMPT_RT) case, so I think your configuration needs PREEMPT_RT, right?
You said this is the default behavior, but PREEMPT_RT is not enabled by default.
Huacai
Quoted 48 lines
Quoted 41 lines
Huacai
Quoted 38 lines
This issue was noticed on several Realtek MIPS switch SoCs (MIPS interAptiv) and came up during kernel bump downstream in OpenWrt from 6.18.33 to 6.18.34, after the backport of the patch to the 6.18 stable branch. The patch also has been backported all the way back to 6.1.
Call
rcutree_report_cpu_dead()once interrupts are disabled, mirroring the generic CPU-hotplug offline path, so RCU stops waiting on the parked CPUs and grace periods can still complete. MIPS shuts down all CPUs here without going through the CPU-hotplug mechanism, so this report is not otherwise issued. Reporting a dying CPU to RCU outside the regular hotplug offline path is not unprecedented: arm64 does the same incpu_die_early(). There it is an exception for a CPU that was coming online and is aborting bringup, rather than the default shutdown action as on MIPS.Fixes 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT")Signed-off-by Jonas Jelonek <[email protected]>arch/mips/kernel/smp.c +2
-0@@ -20,6 +20,7 @@#include <linux/sched/mm.h> #include <linux/cpumask.h> #include <linux/cpu.h> +#include <linux/rcupdate.h> #include <linux/err.h> #include <linux/ftrace.h> #include <linux/irqdomain.h> @@ -422,6 +423,7 @@ static void stop_this_cpu(void *dummy)set_cpu_online (smp_processor_id (), false);calculate_cpu_foreign_map ();local_irq_disable (); +rcutree_report_cpu_dead ();while (1 ); }Quoted 4 lines
Quoted 4 lines
--
2.51.0Best,
Jonas
Hi Huacai,
Quoted 6 lines · Huacai Chen
Quoted 5 lines · Jonas Jelonek
Hi Huacai,
sorry for the reply delay.
On 10.06.26 08:05, Huacai Chen wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
[...]
In theory LoongArch has the same problem, but I cannot reproduce, should I enablePREEMPT_RT? Or there are some special configurations?Sadly I cannot help with that. For MIPS, this seems to be the default behavior.
This patch fixes
91840be8f710, and91840be8f710addssynchronize_rcu()inirq_work_sync(). Your problem is caused by thissynchronize_rcu(), right?
Yes it is.
Quoted 5 lines
However,
synchronize_rcu()only gets called in theIS_ENABLED(CONFIG_PREEMPT_RT)case, so I think your configuration needsPREEMPT_RT, right?You said this is the default behavior, but
PREEMPT_RTis not enabled by default.
The condition where this is added has two parts, see [1]. While PREEMPT_RT isn't active for MIPS, arch_irq_work_has_interrupt gives false for MIPS (since there is no implementation and it falls back to the generic one). This then also calls synchronize_rcu.
Quoted 2 lines
Huacai
Best,
Jonas
[1] https://elixir.bootlin.com/linux/v7.1-rc7/source/kernel/irq_work.c#L291-L302
Quoted 25 lines · Jonas Jelonek
Hi Huacai,
Quoted 6 lines · Huacai Chen
Quoted 5 lines · Jonas Jelonek
Hi Huacai,
sorry for the reply delay.
On 10.06.26 08:05, Huacai Chen wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
[...]
In theory LoongArch has the same problem, but I cannot reproduce, should I enablePREEMPT_RT? Or there are some special configurations?Sadly I cannot help with that. For MIPS, this seems to be the default behavior.
This patch fixes
91840be8f710, and91840be8f710addssynchronize_rcu()inirq_work_sync(). Your problem is caused by thissynchronize_rcu(), right?Yes it is.
Quoted 4 lines
However,
synchronize_rcu()only gets called in theIS_ENABLED(CONFIG_PREEMPT_RT)case, so I think your configuration needsPREEMPT_RT, right?
Quoted 6 lines
Quoted 1 line
You said this is the default behavior, but
PREEMPT_RTis not enabled by default.The condition where this is added has two parts, see [1]. While
PREEMPT_RTisn't active for MIPS,arch_irq_work_has_interruptgives false for MIPS (since there is no implementation and it falls back to the generic one). This then also callssynchronize_rcu.
Sorry, this is my mistake, then what's your preemption model? There are too many config files for MIPS now.
Huacai
Quoted 8 lines
Quoted 3 lines
Huacai
Best,
Jonas[1] https://elixir.bootlin.com/linux/v7.1-rc7/source/kernel/irq_work.c#L291-L302
Quoted 6 lines · Huacai Chen
Quoted 6 lines
[...]
Quoted 4 lines
However,
synchronize_rcu()only gets called in theIS_ENABLED(CONFIG_PREEMPT_RT)case, so I think your configuration needsPREEMPT_RT, right?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
You said this is the default behavior, but
PREEMPT_RTis not enabled by default.The condition where this is added has two parts, see [1]. While
PREEMPT_RTisn't active for MIPS,arch_irq_work_has_interruptgives false for MIPS (since there is no implementation and it falls back to the generic one). This then also callssynchronize_rcu.Sorry, this is my mistake, then what's your preemption model? There are too many config files for MIPS now.
I'm using PREEMPT_NONE, apparently default for all targets in OpenWrt.
Quoted 8 lines
Huacai
Quoted 6 lines
Quoted 2 lines
Huacai
Best,
Jonas[1] https://elixir.bootlin.com/linux/v7.1-rc7/source/kernel/irq_work.c#L291-L302
Best,
Jonas
Huacai Chen
Reviewed-byQuoted 9 lines · Jonas Jelonek
Quoted 7 lines · Huacai Chen
Quoted 7 lines
[...]
Quoted 5 lines
However,
synchronize_rcu()only gets called in theIS_ENABLED(CONFIG_PREEMPT_RT)case, so I think your configuration needsPREEMPT_RT, right?You said this is the default behavior, but
PREEMPT_RTis not enabled by default.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
The condition where this is added has two parts, see [1]. While
PREEMPT_RTisn't active for MIPS,arch_irq_work_has_interruptgives false for MIPS (since there is no implementation and it falls back to the generic one). This then also callssynchronize_rcu.Sorry, this is my mistake, then what's your preemption model? There are too many config files for MIPS now.
I'm using
PREEMPT_NONE, apparently default for all targets in OpenWrt.
With some hack I also reproduce a similar issue on LoongArch in the PREEMPT_NONE model.
Huacai
Quoted 12 lines
Quoted 9 lines
Huacai
Quoted 6 lines
Quoted 2 lines
Huacai
Best,
Jonas[1] https://elixir.bootlin.com/linux/v7.1-rc7/source/kernel/irq_work.c#L291-L302
Best,
Jonas
Quoted 4 lines · Jonas Jelonek
smp_send_stop()parks all secondary CPUs instop_this_cpu(). The function marks the CPU offline for the scheduler viaset_cpu_online(false)but never informs RCU, so RCU keeps expecting a quiescent state from CPUs that are now spinning forever with interrupts disabled.
…
Quoted 4 lines
Fixes 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT")Signed-off-by Jonas Jelonek <[email protected]>
Sorry for being late.
Sebastian
Hi,
Quoted 6 lines · Sebastian Andrzej Siewior
Quoted 4 lines · Jonas Jelonek
smp_send_stop()parks all secondary CPUs instop_this_cpu(). The function marks the CPU offline for the scheduler viaset_cpu_online(false)but never informs RCU, so RCU keeps expecting a quiescent state from CPUs that are now spinning forever with interrupts disabled.…
Quoted 8 lines
Quoted 3 lines
Fixes 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT")Signed-off-by Jonas Jelonek <[email protected]>Sorry for being late.
Reviewed-by Sebastian Andrzej Siewior <[email protected]>Sebastian
Thanks to both of you. The patch has been accepted and merged already (silently) [1]. But I appreciate your reviews a lot.
Best,
Jonas