Skip to contentSkip to conversation

timers/migration: Fix livelock in tmigr_handle_remote_up()

PATCH Applied Amit Matityahu 4 participants 4 messages Jun 3 – Jun 4

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 timers/migration: Fix livelock in tmigr_handle_remote_up()

tmigr_handle_remote_cpu() skips timer_expire_remote() when cpu == smp_processor_id(), assuming the local softirq path already handled this CPU's timers.

This assumption breaks when jiffies advances between run_timer_base(BASE_GLOBAL) and tmigr_handle_remote() in the same softirq invocation - a timer expires after the wheel ran but before the hierarchy snapshot is taken.

The stranded timer is never collected, fetch_next_timer_interrupt_remote() keeps reporting it as expired, and the event is re-queued with expires == now on each iteration. The goto-again loop spins indefinitely.

Fix by calling timer_expire_remote() unconditionally. __run_timer_base() already returns early when there is nothing to expire, making this a no-op in the common case.

Fixes 7ee988770326 ("timers: Implement the hierarchical pull model")
Cc Jonathan Chocron <[email protected]>
Cc Akram Baransi <[email protected]>
Cc David Woodhouse <[email protected]>

---

Questions for maintainers:

1. What was the original rationale for the cpu != smp_processor_id() check? There is no code comment, commit message explanation or anything in the original patch's email discussion as to why timer_expire_remote() is skipped for the local CPU.

2. There seems to be a design tension where a CPU can have timers visible in the migration hierarchy while simultaneously running its own local softirq. Is the expectation that run_timer_base() always drains everything before tmigr_handle_remote() sees it, or should the remote path handle local-CPU timers as a fallback?

 kernel/time/timer_migration.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/time/timer_migration.c +1 -2
index 1d0d3a4058d5..298c34c942ae 100644
@@ -978,8 +978,7 @@ static void tmigr_handle_remote_cpu(unsigned int cpu, u64 now,
 	/* Drop the lock to allow the remote CPU to exit idle */
 	raw_spin_unlock_irq(&tmc->lock);
 
-	if (cpu != smp_processor_id())
-		timer_expire_remote(cpu);
+	timer_expire_remote(cpu);
 
 	/*
 	 * Lock ordering needs to be preserved - timer_base locks before tmigr

base-commit: e43ffb69e0438cddd72aaa30898b4dc446f664f8

Signature · 1 line
2.47.3
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 18 lines · Amit Matityahu

tmigr_handle_remote_cpu() skips timer_expire_remote() when cpu == smp_processor_id(), assuming the local softirq path already handled this CPU's timers.

This assumption breaks when jiffies advances between run_timer_base(BASE_GLOBAL) and tmigr_handle_remote() in the same softirq invocation - a timer expires after the wheel ran but before the hierarchy snapshot is taken.

The stranded timer is never collected, fetch_next_timer_interrupt_remote() keeps reporting it as expired, and the event is re-queued with expires == now on each iteration. The goto-again loop spins indefinitely.

Fix by calling timer_expire_remote() unconditionally. __run_timer_base() already returns early when there is nothing to expire, making this a no-op in the common case.

Quoted 7 lines
Fixes 7ee988770326 ("timers: Implement the hierarchical pull model")
Cc Jonathan Chocron <[email protected]>
Cc Akram Baransi <[email protected]>
Cc David Woodhouse <[email protected]>

That's quite serious indeed!

Quoted 8 lines

---

Questions for maintainers:

1. What was the original rationale for the cpu != smp_processor_id() check? There is no code comment, commit message explanation or anything in the original patch's email discussion as to why timer_expire_remote() is skipped for the local CPU.

The rationale was about assuming that such an expired timerqueue actually reflected a timer that was handled locally already and so it could be safely discarded. So we could spare some locking.

Quoted 6 lines

2. There seems to be a design tension where a CPU can have timers visible in the migration hierarchy while simultaneously running its own local softirq. Is the expectation that run_timer_base() always drains everything before tmigr_handle_remote() sees it, or should the remote path handle local-CPU timers as a fallback?

That's not easy to defer all global timers handling to remote expiration because the current CPU may or may not be the migrator.

Quoted 9 lines
 kernel/time/timer_migration.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/time/timer_migration.c +0 -0
index 1d0d3a4058d5..298c34c942ae 100644

kernel/time/timer_migration.c static void tmigr_handle_remote_cpu(unsigned int cpu, u64 now, · line 981

Quoted 6 lines
kernel/time/timer_migration.c +1 -2
@@ -978,8 +978,7 @@ static void tmigr_handle_remote_cpu(unsigned int cpu, u64 now,
 	/* Drop the lock to allow the remote CPU to exit idle */
 	raw_spin_unlock_irq(&tmc->lock);
 
-	if (cpu != smp_processor_id())
-		timer_expire_remote(cpu);
+	timer_expire_remote(cpu);

Thanks!

Quoted 8 lines
kernel/time/timer_migration.c +0 -0
 	/*
 	 * Lock ordering needs to be preserved - timer_base locks before tmigr
 

base-commit: e43ffb69e0438cddd72aaa30898b4dc446f664f8

Signature · 1 line
2.47.3
Signature · 2 lines
Frederic Weisbecker
SUSE Labs
maintainer
#2 lore raw
Quoted 11 lines · Frederic Weisbecker
Quoted 6 lines · Amit Matityahu

Questions for maintainers:

1. What was the original rationale for the cpu != smp_processor_id() check? There is no code comment, commit message explanation or anything in the original patch's email discussion as to why timer_expire_remote() is skipped for the local CPU.

The rationale was about assuming that such an expired timerqueue actually reflected a timer that was handled locally already and so it could be safely discarded. So we could spare some locking.

Right, but the assumption would only be valid _if_ the jiffies value which was used in run_timers(GLOBAL) is propagated into the remote handling.

Quoted 5 lines

kernel/time/timer_migration.c static void tmigr_handle_remote_cpu(unsigned int cpu, u64 now, · line 981

Quoted 3 lines
kernel/time/timer_migration.c +1 -2
-	if (cpu != smp_processor_id())
-		timer_expire_remote(cpu);
+	timer_expire_remote(cpu);

I'll add a comment to that for posterity.

merge bot Applied
#3 lore raw
In reply to Amit Matityahu
The following commit has been merged into the timers/urgent branch of tip:

The following commit has been merged into the timers/urgent branch of tip:

Commit-ID:     d486b4934a8e504376b85cdb3766f306d57aff5b
Gitweb:        https://git.kernel.org/tip/d486b4934a8e504376b85cdb3766f306d57aff5b
Author:        Amit Matityahu <[email protected]>
AuthorDate:    Wed, 03 Jun 2026 17:01:39 
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Thu, 04 Jun 2026 14:35:33 +02:00

timers/migration: Fix livelock in tmigr_handle_remote_up()

tmigr_handle_remote_cpu() skips timer_expire_remote() when cpu == smp_processor_id(), assuming the local softirq path already handled this CPU's timers.

This assumption is wrong because jiffies can advance after the handling of the CPU's global timers in run_timer_base(BASE_GLOBAL) and before tmigr_handle_remote() evaluates the expiry times.

As a consequence a timer which expires after the CPU local timer wheel advanced and becomes expired in the remote handling is ignored and the callback is never invoked and removed from the timer wheel.

What's worse is that fetch_next_timer_interrupt_remote() keeps reporting it as expired, and the event is re-queued with expires == now on each iteration. The goto-again loop spins indefinitely.

Fix this by calling timer_expire_remote() unconditionally. That's minimal overhead for the common case as __run_timer_base() returns immediately if there is nothing to expire in the local wheel.

[ tglx: Amend change log and add a comment ]

Fixes 7ee988770326 ("timers: Implement the hierarchical pull model")
---
 kernel/time/timer_migration.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
kernel/time/timer_migration.c +6 -2
index 1d0d3a4..52c15af 100644
@@ -978,8 +978,12 @@ static void tmigr_handle_remote_cpu(unsigned int cpu, u64 now,
 	/* Drop the lock to allow the remote CPU to exit idle */
 	raw_spin_unlock_irq(&tmc->lock);
 
-	if (cpu != smp_processor_id())
-		timer_expire_remote(cpu);
+	/*
+	 * This can't exclude the local CPU because jiffies might have advanced
+	 * after the timer softirq invoked run_timer_base(BASE_GLOBAL) and the
+	 * point where the jiffies snapshot @jif was taken in tmigr_handle_remote().
+	 */
+	timer_expire_remote(cpu);
 
 	/*
 	 * Lock ordering needs to be preserved - timer_base locks before tmigr