Skip to contentSkip to conversation

MIPS: smp: report dying CPU to RCU in stop_this_cpu()

PATCH In review Jonas Jelonek 3 participants 10 messages Jun 8 – Jun 19

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 MIPS: smp: report dying CPU to RCU in stop_this_cpu()

smp_send_stop() parks all secondary CPUs in stop_this_cpu(). The function marks the CPU offline for the scheduler via set_cpu_online(false) but never informs RCU, so RCU keeps expecting a quiescent state from CPUs that are now spinning forever with interrupts disabled.

As long as nothing waits for an RCU grace period after smp_send_stop() this is harmless, which is why it went unnoticed. Since commit 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT") however, irq_work_sync() calls synchronize_rcu() on architectures without an irq_work self-IPI, i.e. where arch_irq_work_has_interrupt() returns false. That is the asm-generic default used by MIPS. Any irq_work_sync() issued in the reboot/shutdown path after smp_send_stop() then blocks on a grace period that can never complete, hanging the reboot:

  WARNING: CPU: 0 PID: 15 at kernel/irq_work.c:144 irq_work_queue_on
  ...
  rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
  rcu: Offline CPU 1 blocking current GP.
  rcu: Offline CPU 2 blocking current GP.
  rcu: Offline CPU 3 blocking current GP.

This issue was noticed on several Realtek MIPS switch SoCs (MIPS interAptiv) and came up during kernel bump downstream in OpenWrt from 6.18.33 to 6.18.34, after the backport of the patch to the 6.18 stable branch. The patch also has been backported all the way back to 6.1.

Call rcutree_report_cpu_dead() once interrupts are disabled, mirroring the generic CPU-hotplug offline path, so RCU stops waiting on the parked CPUs and grace periods can still complete. MIPS shuts down all CPUs here without going through the CPU-hotplug mechanism, so this report is not otherwise issued. Reporting a dying CPU to RCU outside the regular hotplug offline path is not unprecedented: arm64 does the same in cpu_die_early(). There it is an exception for a CPU that was coming online and is aborting bringup, rather than the default shutdown action as on MIPS.

Fixes 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT")
arch/mips/kernel/smp.c +2 -0
index 4868e79f3b30..0f28b4a62e72 100644
@@ -20,6 +20,7 @@
 #include <linux/sched/mm.h>
 #include <linux/cpumask.h>
 #include <linux/cpu.h>
+#include <linux/rcupdate.h>
 #include <linux/err.h>
 #include <linux/ftrace.h>
 #include <linux/irqdomain.h>
@@ -422,6 +423,7 @@ static void stop_this_cpu(void *dummy)
 	set_cpu_online(smp_processor_id(), false);
 	calculate_cpu_foreign_map();
 	local_irq_disable();
+	rcutree_report_cpu_dead();
 	while (1);
 }
 
Signature · 1 line
2.51.0
#1 lore raw

Hi, Jonas,

Quoted 15 lines · Jonas Jelonek

smp_send_stop() parks all secondary CPUs in stop_this_cpu(). The function marks the CPU offline for the scheduler via set_cpu_online(false) but never informs RCU, so RCU keeps expecting a quiescent state from CPUs that are now spinning forever with interrupts disabled.

As long as nothing waits for an RCU grace period after smp_send_stop() this is harmless, which is why it went unnoticed. Since commit 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT") however, irq_work_sync() calls synchronize_rcu() on architectures without an irq_work self-IPI, i.e. where arch_irq_work_has_interrupt() returns false. That is the asm-generic default used by MIPS. Any irq_work_sync() issued in the reboot/shutdown path after smp_send_stop() then blocks on a grace period that can never complete, hanging the reboot:

Quoted 6 lines
  WARNING: CPU: 0 PID: 15 at kernel/irq_work.c:144 irq_work_queue_on
  ...
  rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
  rcu: Offline CPU 1 blocking current GP.
  rcu: Offline CPU 2 blocking current GP.
  rcu: Offline CPU 3 blocking current GP.

In theory LoongArch has the same problem, but I cannot reproduce, should I enable PREEMPT_RT? Or there are some special configurations?

Huacai

Quoted 43 lines

This issue was noticed on several Realtek MIPS switch SoCs (MIPS interAptiv) and came up during kernel bump downstream in OpenWrt from 6.18.33 to 6.18.34, after the backport of the patch to the 6.18 stable branch. The patch also has been backported all the way back to 6.1.

Call rcutree_report_cpu_dead() once interrupts are disabled, mirroring the generic CPU-hotplug offline path, so RCU stops waiting on the parked CPUs and grace periods can still complete. MIPS shuts down all CPUs here without going through the CPU-hotplug mechanism, so this report is not otherwise issued. Reporting a dying CPU to RCU outside the regular hotplug offline path is not unprecedented: arm64 does the same in cpu_die_early(). There it is an exception for a CPU that was coming online and is aborting bringup, rather than the default shutdown action as on MIPS.

Fixes 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT")
arch/mips/kernel/smp.c +2 -1
index 4868e79f3b30..0f28b4a62e72 100644
@@ -20,6 +20,7 @@
 #include <linux/sched/mm.h>
 #include <linux/cpumask.h>
 #include <linux/cpu.h>
+#include <linux/rcupdate.h>
 #include <linux/err.h>
 #include <linux/ftrace.h>
 #include <linux/irqdomain.h>
@@ -422,6 +423,7 @@ static void stop_this_cpu(void *dummy)
        set_cpu_online(smp_processor_id(), false);
        calculate_cpu_foreign_map();
        local_irq_disable();
+       rcutree_report_cpu_dead();
        while (1);
 }
 
--

2.51.0

#2 lore raw

Hi Huacai,

sorry for the reply delay.

Quoted 3 lines · Huacai Chen

[...]
In theory LoongArch has the same problem, but I cannot reproduce, should I enable PREEMPT_RT? Or there are some special configurations?

Sadly I cannot help with that. For MIPS, this seems to be the default behavior.

Quoted 40 lines

Huacai

Quoted 38 lines

This issue was noticed on several Realtek MIPS switch SoCs (MIPS interAptiv) and came up during kernel bump downstream in OpenWrt from 6.18.33 to 6.18.34, after the backport of the patch to the 6.18 stable branch. The patch also has been backported all the way back to 6.1.

Call rcutree_report_cpu_dead() once interrupts are disabled, mirroring the generic CPU-hotplug offline path, so RCU stops waiting on the parked CPUs and grace periods can still complete. MIPS shuts down all CPUs here without going through the CPU-hotplug mechanism, so this report is not otherwise issued. Reporting a dying CPU to RCU outside the regular hotplug offline path is not unprecedented: arm64 does the same in cpu_die_early(). There it is an exception for a CPU that was coming online and is aborting bringup, rather than the default shutdown action as on MIPS.

Fixes 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT")
arch/mips/kernel/smp.c +2 -0
index 4868e79f3b30..0f28b4a62e72 100644
@@ -20,6 +20,7 @@
 #include <linux/sched/mm.h>
 #include <linux/cpumask.h>
 #include <linux/cpu.h>
+#include <linux/rcupdate.h>
 #include <linux/err.h>
 #include <linux/ftrace.h>
 #include <linux/irqdomain.h>
@@ -422,6 +423,7 @@ static void stop_this_cpu(void *dummy)
        set_cpu_online(smp_processor_id(), false);
        calculate_cpu_foreign_map();
        local_irq_disable();
+       rcutree_report_cpu_dead();
        while (1);
 }
 
Quoted 4 lines
Quoted 4 lines

--
2.51.0

Best,
Jonas

#3 lore raw
Quoted 5 lines · Jonas Jelonek

Hi Huacai,

sorry for the reply delay.

Quoted 7 lines
Quoted 3 lines · Huacai Chen

[...]
In theory LoongArch has the same problem, but I cannot reproduce, should I enable PREEMPT_RT? Or there are some special configurations?

Sadly I cannot help with that. For MIPS, this seems to be the default behavior.

This patch fixes 91840be8f710, and 91840be8f710 adds synchronize_rcu() in irq_work_sync(). Your problem is caused by this synchronize_rcu(), right?

However, synchronize_rcu() only gets called in the IS_ENABLED(CONFIG_PREEMPT_RT) case, so I think your configuration needs PREEMPT_RT, right?

You said this is the default behavior, but PREEMPT_RT is not enabled by default.

Huacai

Quoted 48 lines
Quoted 41 lines

Huacai

Quoted 38 lines

This issue was noticed on several Realtek MIPS switch SoCs (MIPS interAptiv) and came up during kernel bump downstream in OpenWrt from 6.18.33 to 6.18.34, after the backport of the patch to the 6.18 stable branch. The patch also has been backported all the way back to 6.1.

Call rcutree_report_cpu_dead() once interrupts are disabled, mirroring the generic CPU-hotplug offline path, so RCU stops waiting on the parked CPUs and grace periods can still complete. MIPS shuts down all CPUs here without going through the CPU-hotplug mechanism, so this report is not otherwise issued. Reporting a dying CPU to RCU outside the regular hotplug offline path is not unprecedented: arm64 does the same in cpu_die_early(). There it is an exception for a CPU that was coming online and is aborting bringup, rather than the default shutdown action as on MIPS.

Fixes 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT")
arch/mips/kernel/smp.c +2 -0
index 4868e79f3b30..0f28b4a62e72 100644
@@ -20,6 +20,7 @@
 #include <linux/sched/mm.h>
 #include <linux/cpumask.h>
 #include <linux/cpu.h>
+#include <linux/rcupdate.h>
 #include <linux/err.h>
 #include <linux/ftrace.h>
 #include <linux/irqdomain.h>
@@ -422,6 +423,7 @@ static void stop_this_cpu(void *dummy)
        set_cpu_online(smp_processor_id(), false);
        calculate_cpu_foreign_map();
        local_irq_disable();
+       rcutree_report_cpu_dead();
        while (1);
 }
 
Quoted 4 lines
Quoted 4 lines

--
2.51.0

Best,
Jonas

#4 lore raw

Hi Huacai,

Quoted 6 lines · Huacai Chen
Quoted 5 lines · Jonas Jelonek

Hi Huacai,

sorry for the reply delay.

On 10.06.26 08:05, Huacai Chen wrote:

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

[...]
In theory LoongArch has the same problem, but I cannot reproduce, should I enable PREEMPT_RT? Or there are some special configurations?

Sadly I cannot help with that. For MIPS, this seems to be the default behavior.

This patch fixes 91840be8f710, and 91840be8f710 adds synchronize_rcu() in irq_work_sync(). Your problem is caused by this synchronize_rcu(), right?

Yes it is.

Quoted 5 lines

However, synchronize_rcu() only gets called in the IS_ENABLED(CONFIG_PREEMPT_RT) case, so I think your configuration needs PREEMPT_RT, right?

You said this is the default behavior, but PREEMPT_RT is not enabled by default.

The condition where this is added has two parts, see [1]. While PREEMPT_RT isn't active for MIPS, arch_irq_work_has_interrupt gives false for MIPS (since there is no implementation and it falls back to the generic one). This then also calls synchronize_rcu.

Quoted 2 lines

Huacai

Best,
Jonas

[1] https://elixir.bootlin.com/linux/v7.1-rc7/source/kernel/irq_work.c#L291-L302

#5 lore raw
Quoted 25 lines · Jonas Jelonek

Hi Huacai,

Quoted 6 lines · Huacai Chen
Quoted 5 lines · Jonas Jelonek

Hi Huacai,

sorry for the reply delay.

On 10.06.26 08:05, Huacai Chen wrote:

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

[...]
In theory LoongArch has the same problem, but I cannot reproduce, should I enable PREEMPT_RT? Or there are some special configurations?

Sadly I cannot help with that. For MIPS, this seems to be the default behavior.

This patch fixes 91840be8f710, and 91840be8f710 adds synchronize_rcu() in irq_work_sync(). Your problem is caused by this synchronize_rcu(), right?

Yes it is.

Quoted 4 lines

However, synchronize_rcu() only gets called in the IS_ENABLED(CONFIG_PREEMPT_RT) case, so I think your configuration needs PREEMPT_RT, right?

Quoted 6 lines
Quoted 1 line

You said this is the default behavior, but PREEMPT_RT is not enabled by default.

The condition where this is added has two parts, see [1]. While PREEMPT_RT isn't active for MIPS, arch_irq_work_has_interrupt gives false for MIPS (since there is no implementation and it falls back to the generic one). This then also calls synchronize_rcu.

Sorry, this is my mistake, then what's your preemption model? There are too many config files for MIPS now.

Huacai

Quoted 8 lines
Quoted 3 lines

Huacai

Best,
Jonas

[1] https://elixir.bootlin.com/linux/v7.1-rc7/source/kernel/irq_work.c#L291-L302

#6 lore raw
Quoted 6 lines · Huacai Chen
Quoted 6 lines

[...]

Quoted 4 lines

However, synchronize_rcu() only gets called in the IS_ENABLED(CONFIG_PREEMPT_RT) case, so I think your configuration needs PREEMPT_RT, right?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

You said this is the default behavior, but PREEMPT_RT is not enabled by default.

The condition where this is added has two parts, see [1]. While PREEMPT_RT isn't active for MIPS, arch_irq_work_has_interrupt gives false for MIPS (since there is no implementation and it falls back to the generic one). This then also calls synchronize_rcu.

Sorry, this is my mistake, then what's your preemption model? There are too many config files for MIPS now.

I'm using PREEMPT_NONE, apparently default for all targets in OpenWrt.

Quoted 8 lines

Huacai

Quoted 6 lines
Quoted 2 lines

Huacai

Best,
Jonas

[1] https://elixir.bootlin.com/linux/v7.1-rc7/source/kernel/irq_work.c#L291-L302

Best,
Jonas

Reviewed-by
#7 lore raw
Quoted 9 lines · Jonas Jelonek
Quoted 7 lines · Huacai Chen
Quoted 7 lines

[...]

Quoted 5 lines

However, synchronize_rcu() only gets called in the IS_ENABLED(CONFIG_PREEMPT_RT) case, so I think your configuration needs PREEMPT_RT, right?

You said this is the default behavior, but PREEMPT_RT is not enabled by default.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

The condition where this is added has two parts, see [1]. While PREEMPT_RT isn't active for MIPS, arch_irq_work_has_interrupt gives false for MIPS (since there is no implementation and it falls back to the generic one). This then also calls synchronize_rcu.

Sorry, this is my mistake, then what's your preemption model? There are too many config files for MIPS now.

I'm using PREEMPT_NONE, apparently default for all targets in OpenWrt.

With some hack I also reproduce a similar issue on LoongArch in the PREEMPT_NONE model.

Huacai

Quoted 12 lines
Quoted 9 lines

Huacai

Quoted 6 lines
Quoted 2 lines

Huacai

Best,
Jonas

[1] https://elixir.bootlin.com/linux/v7.1-rc7/source/kernel/irq_work.c#L291-L302

Best,
Jonas

maintainer Reviewed-by on v2 1/1
#8 lore raw
In reply to Jonas Jelonek
Quoted 4 lines · Jonas Jelonek

smp_send_stop() parks all secondary CPUs in stop_this_cpu(). The function marks the CPU offline for the scheduler via set_cpu_online(false) but never informs RCU, so RCU keeps expecting a quiescent state from CPUs that are now spinning forever with interrupts disabled.

…

Quoted 4 lines
Fixes 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT")

Sorry for being late.

Sebastian

#9 lore raw

Hi,

Quoted 6 lines · Sebastian Andrzej Siewior
Quoted 4 lines · Jonas Jelonek

smp_send_stop() parks all secondary CPUs in stop_this_cpu(). The function marks the CPU offline for the scheduler via set_cpu_online(false) but never informs RCU, so RCU keeps expecting a quiescent state from CPUs that are now spinning forever with interrupts disabled.

…

Quoted 8 lines
Quoted 3 lines
Fixes 91840be8f710 ("irq_work: Fix use-after-free in irq_work_single() on PREEMPT_RT")

Sorry for being late.

Sebastian

Thanks to both of you. The patch has been accepted and merged already (silently) [1]. But I appreciate your reviews a lot.

Best,
Jonas

[1] https://git.kernel.org/pub/scm/linux/kernel/git/mips/linux.git/commit/?id=9f3f3bdc6d9dac1a5a8262ee7ad0f2ff1527a7e7