Skip to contentSkip to conversation

hrtimer/sched: Improve hrtick

PATCH In review Peter Zijlstra 6 participants 25 messages Jan 21 – Feb 5

Subsystems (4)
lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

v2 cover hrtimer/sched: Improve hrtick

Hi!

At long last a new version of the hrtick rework! The previous version had a mysterious deadlock which has been resolved. So far (weeks) the 0-day robot has not found more issues with these patches.

These patches aim to reduce the hrtick overhead to such an extent that it can be default enabled. Decoupling the preemption behaviour from CONFIG_HZ, leaving only load-balancing and time keeping dependent on HZ.

Some (limited) performance runs from 0-day have also not found any regressions from enabling HRTICK, but it has not run the full suite yet (AFAIU).

Patches also at:

git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/hrtick

author maintainer
#1 lore raw

PATCH v2 1/6 sched/eevdf: Fix HRTICK duration

The nominal duration for an EEVDF task to run is until its deadline. At which point the deadline is moved ahead and a new task selection is done.

Try and predict the time 'lost' to higher scheduling classes. Since this is an estimate, the timer can be both early or late. In case it is early task_tick_fair() will take the !need_resched() path and restarts the timer.

---
 kernel/sched/fair.c |   55 +++++++++++++++++++++++++++++-----------------------
 1 file changed, 31 insertions(+), 24 deletions(-)
kernel/sched/fair.c +31 -24
@@ -5511,7 +5511,7 @@ static void put_prev_entity(struct cfs_r
 }
 
 static void
-entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
+entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 {
 	/*
 	 * Update run-time statistics of the 'current'.
@@ -5523,17 +5523,6 @@ entity_tick(struct cfs_rq *cfs_rq, struc
 	 */
 	update_load_avg(cfs_rq, curr, UPDATE_TG);
 	update_cfs_group(curr);
-
-#ifdef CONFIG_SCHED_HRTICK
-	/*
-	 * queued ticks are scheduled to match the slice, so don't bother
-	 * validating it and just reschedule.
-	 */
-	if (queued) {
-		resched_curr_lazy(rq_of(cfs_rq));
-		return;
-	}
-#endif
 }
 
 
@@ -6735,21 +6724,39 @@ static inline void sched_fair_update_sto
 static void hrtick_start_fair(struct rq *rq, struct task_struct *p)
 {
 	struct sched_entity *se = &p->se;
+	unsigned long scale = 1024;
+	unsigned long util = 0;
+	u64 vdelta;
+	u64 delta;
 
 	WARN_ON_ONCE(task_rq(p) != rq);
 
-	if (rq->cfs.h_nr_queued > 1) {
-		u64 ran = se->sum_exec_runtime - se->prev_sum_exec_runtime;
-		u64 slice = se->slice;
-		s64 delta = slice - ran;
-
-		if (delta < 0) {
-			if (task_current_donor(rq, p))
-				resched_curr(rq);
-			return;
-		}
-		hrtick_start(rq, delta);
+	if (rq->cfs.h_nr_queued <= 1)
+		return;
+
+	/*
+	 * Compute time until virtual deadline
+	 */
+	vdelta = se->deadline - se->vruntime;
+	if ((s64)vdelta < 0) {
+		if (task_current_donor(rq, p))
+			resched_curr(rq);
+		return;
+	}
+	delta = (se->load.weight * vdelta) / NICE_0_LOAD;
+
+	/*
+	 * Correct for instantaneous load of other classes.
+	 */
+	util += cpu_util_dl(rq);
+	util += cpu_util_rt(rq);
+	util += cpu_util_irq(rq);
+	if (util && util < 1024) {
+		scale *= 1024;
+		scale /= (1024 - util);
 	}
+
+	hrtick_start(rq, (scale * delta) / 1024);
 }
 
 /*
@@ -13373,7 +13380,7 @@ static void task_tick_fair(struct rq *rq
 
 	for_each_sched_entity(se) {
 		cfs_rq = cfs_rq_of(se);
-		entity_tick(cfs_rq, se, queued);
+		entity_tick(cfs_rq, se);
 	}
 
 	if (queued) {
maintainer Reviewed-by on v2 1/6
#2 lore raw

Re: [PATCH v2 1/6] sched/eevdf: Fix HRTICK duration

Hello,

Quoted 11 lines · Peter Zijlstra

The nominal duration for an EEVDF task to run is until its deadline. At which point the deadline is moved ahead and a new task selection is done.

Try and predict the time 'lost' to higher scheduling classes. Since this is an estimate, the timer can be both early or late. In case it is early task_tick_fair() will take the !need_resched() path and restarts the timer.

---

...

Quoted 25 lines
kernel/sched/fair.c +7 -11
@@ -6735,21 +6724,39 @@ static inline void sched_fair_update_sto
 static void hrtick_start_fair(struct rq *rq, struct task_struct *p)
 {
 	struct sched_entity *se = &p->se;
+	unsigned long scale = 1024;
+	unsigned long util = 0;
+	u64 vdelta;
+	u64 delta;
 
 	WARN_ON_ONCE(task_rq(p) != rq);
 
-	if (rq->cfs.h_nr_queued > 1) {
-		u64 ran = se->sum_exec_runtime - se->prev_sum_exec_runtime;
-		u64 slice = se->slice;
-		s64 delta = slice - ran;
-
-		if (delta < 0) {
-			if (task_current_donor(rq, p))
-				resched_curr(rq);
-			return;
-		}
-		hrtick_start(rq, delta);
+	if (rq->cfs.h_nr_queued <= 1)
+		return;
+

kernel/sched/fair.c static inline void sched_fair_update_sto · line 6746

Quoted 10 lines
kernel/sched/fair.c +10 -0
+	/*
+	 * Compute time until virtual deadline
+	 */
+	vdelta = se->deadline - se->vruntime;
+	if ((s64)vdelta < 0) {
+		if (task_current_donor(rq, p))
+			resched_curr(rq);
+		return;
+	}
+	delta = (se->load.weight * vdelta) / NICE_0_LOAD;

Nit.. guess we don't fear overflow since vdelta should be bounded anyway.

Thanks,
Juri

author maintainer
#3 lore raw

Re: [PATCH v2 1/6] sched/eevdf: Fix HRTICK duration

In reply to Peter Zijlstra
Quoted 22 lines · Peter Zijlstra
kernel/sched/fair.c +13 -0
@@ -6735,21 +6724,39 @@ static inline void sched_fair_update_sto
 static void hrtick_start_fair(struct rq *rq, struct task_struct *p)
 {
 	struct sched_entity *se = &p->se;
+	unsigned long scale = 1024;
+	unsigned long util = 0;
+	u64 vdelta;
+	u64 delta;
 
 	WARN_ON_ONCE(task_rq(p) != rq);
 
+	if (rq->cfs.h_nr_queued <= 1)
+		return;
+
+	/*
+	 * Compute time until virtual deadline
+	 */
+	vdelta = se->deadline - se->vruntime;
+	if ((s64)vdelta < 0) {
+		if (task_current_donor(rq, p))

kernel/sched/fair.c static inline void sched_fair_update_sto · line 6752

Quoted 10 lines
kernel/sched/fair.c +10 -0
+			resched_curr(rq);
+		return;
+	}
+	delta = (se->load.weight * vdelta) / NICE_0_LOAD;
+
+	/*
+	 * Correct for instantaneous load of other classes.
+	 */
+	util += cpu_util_dl(rq);
+	util += cpu_util_rt(rq);

Since this is all about current, other scheduling classes are irrelevant, they cannot run without causing schedule() which will cause the hrtick to be reprogrammed anyway.

So I'm thinking those two lines above ought to go.

Quoted 40 lines
kernel/sched/fair.c +7 -12
+	util += cpu_util_irq(rq);
+	if (util && util < 1024) {
+		scale *= 1024;
+		scale /= (1024 - util);
 	}
+
+	hrtick_start(rq, (scale * delta) / 1024);
 }
 
 /*
@@ -5511,7 +5511,7 @@ static void put_prev_entity(struct cfs_r
 }
 
 static void
-entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued)
+entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 {
 	/*
 	 * Update run-time statistics of the 'current'.
@@ -5523,17 +5523,6 @@ entity_tick(struct cfs_rq *cfs_rq, struc
 	 */
 	update_load_avg(cfs_rq, curr, UPDATE_TG);
 	update_cfs_group(curr);
-
-#ifdef CONFIG_SCHED_HRTICK
-	/*
-	 * queued ticks are scheduled to match the slice, so don't bother
-	 * validating it and just reschedule.
-	 */
-	if (queued) {
-		resched_curr_lazy(rq_of(cfs_rq));
-		return;
-	}
-#endif
 }
 
 

kernel/sched/fair.c static void task_tick_fair(struct rq *rq · line 13386

Quoted 8 lines
kernel/sched/fair.c +1 -1
@@ -13373,7 +13380,7 @@ static void task_tick_fair(struct rq *rq
 
 	for_each_sched_entity(se) {
 		cfs_rq = cfs_rq_of(se);
-		entity_tick(cfs_rq, se, queued);
+		entity_tick(cfs_rq, se);
 	}
 
 	if (queued) {

So Thomas did observe some really small hrtimer reprogramming because of this. If we just miss the normal deadline, it will re-try with a stupid sliver of time.

Perhaps it makes sense to leave these two hunks, and simply hard preempt when the hrtick goes, irrespective of slightly missing the vruntime due to the approximation on task-clock.

author maintainer
#4 lore raw

PATCH v2 2/6 hrtimer: Optimize __hrtimer_start_range_ns()

In reply to Peter Zijlstra

Much like hrtimer_reprogram(), skip programming if the cpu_base is running the hrtimer interrupt.

---
 kernel/time/hrtimer.c |    8 ++++++++
 1 file changed, 8 insertions(+)
kernel/time/hrtimer.c +8 -0
@@ -1261,6 +1261,14 @@ static int __hrtimer_start_range_ns(stru
 	}
 
 	first = enqueue_hrtimer(timer, new_base, mode);
+
+	/*
+	 * If the hrtimer interrupt is running, then it will reevaluate the
+	 * clock bases and reprogram the clock event device.
+	 */
+	if (new_base->cpu_base->in_hrtirq)
+		return 0;
+
 	if (!force_local) {
 		/*
 		 * If the current CPU base is online, then the timer is
author maintainer
#7 lore raw

PATCH v2 3/6 hrtimer,sched: Add fuzzy hrtimer mode for HRTICK

In reply to Peter Zijlstra

Upon schedule() HRTICK will cancel the current timer, pick the next task and reprogram the timer. When schedule() consistently triggers due to blocking conditions instead of the timer, this leads to endless reprogramming without ever firing.

Mitigate this with a new hrtimer mode: fuzzy (not really happy with that name); this mode does two things:

- skip reprogramming the hardware on timer remove;
- skip reprogramming the hardware when the new timer is after cpu_base->expires_next

Both things are already possible;

- removing a remote timer will leave the hardware programmed and cause a spurious interrupt.
- this remote CPU adding a timer can skip the reprogramming when the timer's expiration is after the (spurious) expiration.

This new timer mode simply causes more of this 'fuzzy' behaviour; it causes a few spurious interrupts, but similarly avoids endlessly reprogramming the timer.

This makes the HRTICK match the NO_HRTICK hackbench runs -- the case where a task never runs until its slice is complete but always goes sleep early.

---
 include/linux/hrtimer.h       |    1 +
 include/linux/hrtimer_types.h |    1 +
 kernel/sched/core.c           |    3 ++-
 kernel/time/hrtimer.c         |   16 +++++++++++++++-
 4 files changed, 19 insertions(+), 2 deletions(-)
include/linux/hrtimer.h +1 -0
@@ -38,6 +38,7 @@ enum hrtimer_mode {
 	HRTIMER_MODE_PINNED	= 0x02,
 	HRTIMER_MODE_SOFT	= 0x04,
 	HRTIMER_MODE_HARD	= 0x08,
+	HRTIMER_MODE_FUZZY	= 0x10,
 
 	HRTIMER_MODE_ABS_PINNED = HRTIMER_MODE_ABS | HRTIMER_MODE_PINNED,
 	HRTIMER_MODE_REL_PINNED = HRTIMER_MODE_REL | HRTIMER_MODE_PINNED,
include/linux/hrtimer_types.h +1 -0
@@ -45,6 +45,7 @@ struct hrtimer {
 	u8				is_rel;
 	u8				is_soft;
 	u8				is_hard;
+	u8				is_fuzzy;
 };
 
 #endif /* _LINUX_HRTIMER_TYPES_H */
kernel/sched/core.c +2 -1
@@ -928,7 +928,8 @@ void hrtick_start(struct rq *rq, u64 del
 static void hrtick_rq_init(struct rq *rq)
 {
 	INIT_CSD(&rq->hrtick_csd, __hrtick_start, rq);
-	hrtimer_setup(&rq->hrtick_timer, hrtick, CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD);
+	hrtimer_setup(&rq->hrtick_timer, hrtick, CLOCK_MONOTONIC,
+		      HRTIMER_MODE_REL_HARD | HRTIMER_MODE_FUZZY);
 }
 #else /* !CONFIG_SCHED_HRTICK: */
 static inline void hrtick_clear(struct rq *rq)
kernel/time/hrtimer.c +15 -1
@@ -1122,7 +1122,7 @@ static void __remove_hrtimer(struct hrti
 	 * an superfluous call to hrtimer_force_reprogram() on the
 	 * remote cpu later on if the same timer gets enqueued again.
 	 */
-	if (reprogram && timer == cpu_base->next_timer)
+	if (!timer->is_fuzzy && reprogram && timer == cpu_base->next_timer)
 		hrtimer_force_reprogram(cpu_base, 1);
 }
 
@@ -1269,6 +1269,19 @@ static int __hrtimer_start_range_ns(stru
 	if (new_base->cpu_base->in_hrtirq)
 		return 0;
 
+	if (timer->is_fuzzy) {
+		/*
+		 * XXX fuzzy implies pinned!  not sure how to deal with
+		 * retrigger_next_event() for the !local case.
+		 */
+		WARN_ON_ONCE(!(mode & HRTIMER_MODE_PINNED));
+		/*
+		 * Notably, by going into hrtimer_reprogram(), it will
+		 * not reprogram if cpu_base->expires_next is earlier.
+		 */
+		return first;
+	}
+
 	if (!force_local) {
 		/*
 		 * If the current CPU base is online, then the timer is
@@ -1645,6 +1658,7 @@ static void __hrtimer_setup(struct hrtim
 	base += hrtimer_clockid_to_base(clock_id);
 	timer->is_soft = softtimer;
 	timer->is_hard = !!(mode & HRTIMER_MODE_HARD);
+	timer->is_fuzzy = !!(mode & HRTIMER_MODE_FUZZY);
 	timer->base = &cpu_base->clock_base[base];
 	timerqueue_init(&timer->node);
 
maintainer Reviewed-by on v2 3/6
#8 lore raw

Re: [PATCH v2 3/6] hrtimer,sched: Add fuzzy hrtimer mode for HRTICK

Hello,

Quoted 7 lines · Peter Zijlstra

Upon schedule() HRTICK will cancel the current timer, pick the next task and reprogram the timer. When schedule() consistently triggers due to blocking conditions instead of the timer, this leads to endless reprogramming without ever firing.

Mitigate this with a new hrtimer mode: fuzzy (not really happy with that name); this mode does two things:

Does the more common (lazier :) 'lazy' work better?

Quoted 22 lines

- skip reprogramming the hardware on timer remove;
- skip reprogramming the hardware when the new timer is after cpu_base->expires_next

Both things are already possible;

- removing a remote timer will leave the hardware programmed and cause a spurious interrupt.
- this remote CPU adding a timer can skip the reprogramming when the timer's expiration is after the (spurious) expiration.

This new timer mode simply causes more of this 'fuzzy' behaviour; it causes a few spurious interrupts, but similarly avoids endlessly reprogramming the timer.

This makes the HRTICK match the NO_HRTICK hackbench runs -- the case where a task never runs until its slice is complete but always goes sleep early.

---

...

kernel/time/hrtimer.c static int __hrtimer_start_range_ns(stru · line 1277

Quoted 10 lines
kernel/time/hrtimer.c +6 -0
@@ -1269,6 +1269,19 @@ static int __hrtimer_start_range_ns(stru
 	if (new_base->cpu_base->in_hrtirq)
 		return 0;
 
+	if (timer->is_fuzzy) {
+		/*
+		 * XXX fuzzy implies pinned!  not sure how to deal with
+		 * retrigger_next_event() for the !local case.
+		 */
+		WARN_ON_ONCE(!(mode & HRTIMER_MODE_PINNED));

Not sure either, but since it's improving things for local already, maybe it's an acceptable first step?

Thanks,
Juri

#9 lore raw

Re: [PATCH v2 3/6] hrtimer,sched: Add fuzzy hrtimer mode for HRTICK

Quoted 8 lines · On Thu, 22 Jan 2026 14:12:28 +0100 Juri Lelli <[email protected]> wrote:

Hello,

Quoted 5 lines · Peter Zijlstra

Upon schedule() HRTICK will cancel the current timer, pick the next task and reprogram the timer. When schedule() consistently triggers due to blocking conditions instead of the timer, this leads to endless reprogramming without ever firing.

Quoted 4 lines
Quoted 2 lines

Mitigate this with a new hrtimer mode: fuzzy (not really happy with that name); this mode does two things:

Does the more common (lazier :) 'lazy' work better?

I don't like either fuzzy or lazy.

Fuzzy makes me think of just random entries (for fuzz testing and such).

Lazy is to postpone things to do things less often.

What about "speculative"? Like branch prediction and such. Where a timer is expected to be used at a certain time but it may not be?

-- Steve

Reviewed-by on v2 3/6
#10 lore raw

Re: [PATCH v2 3/6] hrtimer,sched: Add fuzzy hrtimer mode for HRTICK

In reply to Peter Zijlstra

kernel/sched/core.c void hrtick_start(struct rq *rq, u64 del · line 932

Quoted 9 lines · Peter Zijlstra
kernel/sched/core.c +2 -1
@@ -928,7 +928,8 @@ void hrtick_start(struct rq *rq, u64 del
 static void hrtick_rq_init(struct rq *rq)
 {
 	INIT_CSD(&rq->hrtick_csd, __hrtick_start, rq);
-	hrtimer_setup(&rq->hrtick_timer, hrtick, CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD);
+	hrtimer_setup(&rq->hrtick_timer, hrtick, CLOCK_MONOTONIC,
+		      HRTIMER_MODE_REL_HARD | HRTIMER_MODE_FUZZY);

SHouldn't this be HRTIMER_MODE_REL_PINNED_HARD? I know it's set when starting the timer, but I had to double check it.

Quoted 15 lines
kernel/sched/core.c +0 -0
 }
 #else /* !CONFIG_SCHED_HRTICK: */
 static inline void hrtick_clear(struct rq *rq)
kernel/time/hrtimer.c +1 -1
@@ -1122,7 +1122,7 @@ static void __remove_hrtimer(struct hrti
 	 * an superfluous call to hrtimer_force_reprogram() on the
 	 * remote cpu later on if the same timer gets enqueued again.
 	 */
-	if (reprogram && timer == cpu_base->next_timer)
+	if (!timer->is_fuzzy && reprogram && timer == cpu_base->next_timer)
 		hrtimer_force_reprogram(cpu_base, 1);
 }
 

kernel/time/hrtimer.c static int __hrtimer_start_range_ns(stru · line 1275

Quoted 7 lines
kernel/time/hrtimer.c +4 -0
@@ -1269,6 +1269,19 @@ static int __hrtimer_start_range_ns(stru
 	if (new_base->cpu_base->in_hrtirq)
 		return 0;
 
+	if (timer->is_fuzzy) {
+		/*
+		 * XXX fuzzy implies pinned!  not sure how to deal with
+		 * retrigger_next_event() for the !local case.

I'd rather say:

Fuzzy requires pinned as the lazy reprogramming only works for CPU local timers.

kernel/time/hrtimer.c static int __hrtimer_start_range_ns(stru · line 1277

Quoted 2 lines
kernel/time/hrtimer.c +2 -0
+		 */
+		WARN_ON_ONCE(!(mode & HRTIMER_MODE_PINNED));

Other than that:

Thanks,

tglx

author maintainer
#11 lore raw

PATCH v2 4/6 hrtimer: Re-arrange hrtimer_interrupt()

In reply to Peter Zijlstra

Rework hrtimer_interrupt() such that reprogramming is split out into an independent function at the end of the interrupt.

This prepares for reprogramming getting delayed beyond the end of hrtimer_interrupt().

Notably, this changes the hang handling to always wait 100ms instead of trying to keep it proportional to the actual delay. This simplifies the state, also this really shouldn't be happening.

---
 kernel/time/hrtimer.c |   87 ++++++++++++++++++++++----------------------------
 1 file changed, 39 insertions(+), 48 deletions(-)
kernel/time/hrtimer.c +39 -48
@@ -1889,6 +1889,29 @@ static __latent_entropy void hrtimer_run
 #ifdef CONFIG_HIGH_RES_TIMERS
 
 /*
+ * Very similar to hrtimer_force_reprogram(), except it deals with
+ * in_hrirq and hang_detected.
+ */
+static void __hrtimer_rearm(struct hrtimer_cpu_base *cpu_base, ktime_t now)
+{
+	ktime_t expires_next = hrtimer_update_next_event(cpu_base);
+
+	cpu_base->expires_next = expires_next;
+	cpu_base->in_hrtirq = 0;
+
+	if (unlikely(cpu_base->hang_detected)) {
+		/*
+		 * Give the system a chance to do something else than looping
+		 * on hrtimer interrupts.
+		 */
+		expires_next = ktime_add_ns(now, 100 * NSEC_PER_MSEC);
+		cpu_base->hang_detected = 0;
+	}
+
+	tick_program_event(expires_next, 1);
+}
+
+/*
  * High resolution timer interrupt
  * Called with interrupts disabled
  */
@@ -1924,63 +1947,31 @@ void hrtimer_interrupt(struct clock_even
 
 	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
 
-	/* Reevaluate the clock bases for the [soft] next expiry */
-	expires_next = hrtimer_update_next_event(cpu_base);
-	/*
-	 * Store the new expiry value so the migration code can verify
-	 * against it.
-	 */
-	cpu_base->expires_next = expires_next;
-	cpu_base->in_hrtirq = 0;
-	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
-
-	/* Reprogramming necessary ? */
-	if (!tick_program_event(expires_next, 0)) {
-		cpu_base->hang_detected = 0;
-		return;
-	}
-
 	/*
 	 * The next timer was already expired due to:
 	 * - tracing
 	 * - long lasting callbacks
 	 * - being scheduled away when running in a VM
 	 *
-	 * We need to prevent that we loop forever in the hrtimer
-	 * interrupt routine. We give it 3 attempts to avoid
-	 * overreacting on some spurious event.
-	 *
-	 * Acquire base lock for updating the offsets and retrieving
-	 * the current time.
+	 * We need to prevent that we loop forever in the hrtiner interrupt
+	 * routine. We give it 3 attempts to avoid overreacting on some
+	 * spurious event.
 	 */
-	raw_spin_lock_irqsave(&cpu_base->lock, flags);
+	expires_next = hrtimer_update_next_event(cpu_base);
 	now = hrtimer_update_base(cpu_base);
-	cpu_base->nr_retries++;
-	if (++retries < 3)
-		goto retry;
-	/*
-	 * Give the system a chance to do something else than looping
-	 * here. We stored the entry time, so we know exactly how long
-	 * we spent here. We schedule the next event this amount of
-	 * time away.
-	 */
-	cpu_base->nr_hangs++;
-	cpu_base->hang_detected = 1;
-	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
+	if (expires_next < now) {
+		if (++retries < 3)
+			goto retry;
+
+		delta = ktime_sub(now, entry_time);
+		cpu_base->max_hang_time = max_t(unsigned int,
+						cpu_base->max_hang_time, delta);
+		cpu_base->nr_hangs++;
+		cpu_base->hang_detected = 1;
+	}
 
-	delta = ktime_sub(now, entry_time);
-	if ((unsigned int)delta > cpu_base->max_hang_time)
-		cpu_base->max_hang_time = (unsigned int) delta;
-	/*
-	 * Limit it to a sensible value as we enforce a longer
-	 * delay. Give the CPU at least 100ms to catch up.
-	 */
-	if (delta > 100 * NSEC_PER_MSEC)
-		expires_next = ktime_add_ns(now, 100 * NSEC_PER_MSEC);
-	else
-		expires_next = ktime_add(now, delta);
-	tick_program_event(expires_next, 1);
-	pr_warn_once("hrtimer: interrupt took %llu ns\n", ktime_to_ns(delta));
+	__hrtimer_rearm(cpu_base, now);
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 }
 #endif /* !CONFIG_HIGH_RES_TIMERS */
 
Reviewed-by on v2 4/6
#12 lore raw

Re: [PATCH v2 4/6] hrtimer: Re-arrange hrtimer_interrupt()

Quoted 9 lines · Peter Zijlstra

Rework hrtimer_interrupt() such that reprogramming is split out into an independent function at the end of the interrupt.

This prepares for reprogramming getting delayed beyond the end of hrtimer_interrupt().

Notably, this changes the hang handling to always wait 100ms instead of trying to keep it proportional to the actual delay. This simplifies the state, also this really shouldn't be happening.

Indeed.

Quoted 3 lines

/*
+ * Very similar to hrtimer_force_reprogram(), except it deals with
+ * in_hrirq and hang_detected.

in_hrtirq

author maintainer
#13 lore raw

PATCH v2 5/6 entry,hrtimer: Push reprogramming timers into the interrupt return path

In reply to Peter Zijlstra

Currently hrtimer_interrupt() runs expired timers, which can re-arm themselves, after which it computes the next expiration time and re-programs the hardware.

However, things like HRTICK, a highres timer driving preemption, cannot re-arm itself at the point of running, since the next task has not been determined yet. The schedule() in the interrupt return path will switch to the next task, which then causes a new hrtimer to be programmed.

This then results in reprogramming the hardware at least twice, once after running the timers, and once upon selecting the new task.

Notably, *both* events happen in the interrupt.

By pushing the hrtimer reprogram all the way into the interrupt return path, it runs after schedule() and this double reprogram can be avoided.

---
 include/asm-generic/thread_info_tif.h |    5 ++++-
 include/linux/hrtimer.h               |   17 +++++++++++++++++
 include/linux/irq-entry-common.h      |    2 ++
 kernel/entry/common.c                 |   13 +++++++++++++
 kernel/sched/core.c                   |   10 ++++++++++
 kernel/time/hrtimer.c                 |   28 ++++++++++++++++++++++++----
 6 files changed, 70 insertions(+), 5 deletions(-)
include/asm-generic/thread_info_tif.h +4 -1
@@ -41,11 +41,14 @@
 #define _TIF_PATCH_PENDING	BIT(TIF_PATCH_PENDING)
 
 #ifdef HAVE_TIF_RESTORE_SIGMASK
-# define TIF_RESTORE_SIGMASK	10	// Restore signal mask in do_signal() */
+# define TIF_RESTORE_SIGMASK	10	// Restore signal mask in do_signal()
 # define _TIF_RESTORE_SIGMASK	BIT(TIF_RESTORE_SIGMASK)
 #endif
 
 #define TIF_RSEQ		11	// Run RSEQ fast path
 #define _TIF_RSEQ		BIT(TIF_RSEQ)
 
+#define TIF_HRTIMER_REARM	12       // re-arm the timer
+#define _TIF_HRTIMER_REARM	BIT(TIF_HRTIMER_REARM)
+
 #endif /* _ASM_GENERIC_THREAD_INFO_TIF_H_ */
include/linux/hrtimer.h +17 -0
@@ -175,10 +175,27 @@ extern void hrtimer_interrupt(struct clo
 
 extern unsigned int hrtimer_resolution;
 
+#ifdef TIF_HRTIMER_REARM
+extern void _hrtimer_rearm(void);
+/*
+ * This is to be called on all irqentry_exit() paths that will enable
+ * interrupts; as well as in the context switch path before switch_to().
+ */
+static inline void hrtimer_rearm(void)
+{
+	if (test_thread_flag(TIF_HRTIMER_REARM))
+		_hrtimer_rearm();
+}
+#else
+static inline void hrtimer_rearm(void) { }
+#endif /* TIF_HRTIMER_REARM */
+
 #else
 
 #define hrtimer_resolution	(unsigned int)LOW_RES_NSEC
 
+static inline void hrtimer_rearm(void) { }
+
 #endif
 
 static inline ktime_t
include/linux/irq-entry-common.h +2 -0
@@ -224,6 +224,8 @@ static __always_inline void __exit_to_us
 	ti_work = read_thread_flags();
 	if (unlikely(ti_work & EXIT_TO_USER_MODE_WORK))
 		ti_work = exit_to_user_mode_loop(regs, ti_work);
+	else
+		hrtimer_rearm();
 
 	arch_exit_to_user_mode_prepare(regs, ti_work);
 }
kernel/entry/common.c +13 -0
@@ -7,6 +7,7 @@
 #include <linux/kmsan.h>
 #include <linux/livepatch.h>
 #include <linux/tick.h>
+#include <linux/hrtimer.h>
 
 /* Workaround to allow gradual conversion of architecture code */
 void __weak arch_do_signal_or_restart(struct pt_regs *regs) { }
@@ -26,6 +27,16 @@ static __always_inline unsigned long __e
 	 */
 	while (ti_work & EXIT_TO_USER_MODE_WORK_LOOP) {
 
+		/*
+		 * If hrtimer need re-arming, do so before enabling IRQs,
+		 * except when a reschedule is needed, in that case schedule()
+		 * will do this.
+		 */
+		if ((ti_work & (_TIF_NEED_RESCHED |
+				_TIF_NEED_RESCHED_LAZY |
+				_TIF_HRTIMER_REARM)) == _TIF_HRTIMER_REARM)
+			hrtimer_rearm();
+
 		local_irq_enable_exit_to_user(ti_work);
 
 		if (ti_work & (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY))
@@ -202,6 +213,7 @@ noinstr void irqentry_exit(struct pt_reg
 		 */
 		if (state.exit_rcu) {
 			instrumentation_begin();
+			hrtimer_rearm();
 			/* Tell the tracer that IRET will enable interrupts */
 			trace_hardirqs_on_prepare();
 			lockdep_hardirqs_on_prepare();
@@ -215,6 +227,7 @@ noinstr void irqentry_exit(struct pt_reg
 		if (IS_ENABLED(CONFIG_PREEMPTION))
 			irqentry_exit_cond_resched();
 
+		hrtimer_rearm();
 		/* Covers both tracing and lockdep */
 		trace_hardirqs_on();
 		instrumentation_end();
kernel/sched/core.c +10 -0
@@ -6814,6 +6814,16 @@ static void __sched notrace __schedule(i
 keep_resched:
 	rq->last_seen_need_resched_ns = 0;
 
+	/*
+	 * Notably, this must be called after pick_next_task() but before
+	 * switch_to(), since the new task need not be on the return from
+	 * interrupt path. Additionally, exit_to_user_mode_loop() relies on
+	 * any schedule() call to imply this call, so do it unconditionally.
+	 *
+	 * We've just cleared TIF_NEED_RESCHED, TIF word should be in cache.
+	 */
+	hrtimer_rearm();
+
 	is_switch = prev != next;
 	if (likely(is_switch)) {
 		rq->nr_switches++;
kernel/time/hrtimer.c +24 -4
@@ -1892,10 +1892,9 @@ static __latent_entropy void hrtimer_run
  * Very similar to hrtimer_force_reprogram(), except it deals with
  * in_hrirq and hang_detected.
  */
-static void __hrtimer_rearm(struct hrtimer_cpu_base *cpu_base, ktime_t now)
+static void __hrtimer_rearm(struct hrtimer_cpu_base *cpu_base,
+			    ktime_t now, ktime_t expires_next)
 {
-	ktime_t expires_next = hrtimer_update_next_event(cpu_base);
-
 	cpu_base->expires_next = expires_next;
 	cpu_base->in_hrtirq = 0;
 
@@ -1970,9 +1969,30 @@ void hrtimer_interrupt(struct clock_even
 		cpu_base->hang_detected = 1;
 	}
 
-	__hrtimer_rearm(cpu_base, now);
+#ifdef TIF_HRTIMER_REARM
+	set_thread_flag(TIF_HRTIMER_REARM);
+#else
+	__hrtimer_rearm(cpu_base, now, expires_next);
+#endif
 	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 }
+
+#ifdef TIF_HRTIMER_REARM
+void _hrtimer_rearm(void)
+{
+	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	ktime_t now, expires_next;
+
+	lockdep_assert_irqs_disabled();
+
+	scoped_guard (raw_spinlock, &cpu_base->lock) {
+		now = hrtimer_update_base(cpu_base);
+		expires_next = hrtimer_update_next_event(cpu_base);
+		__hrtimer_rearm(cpu_base, now, expires_next);
+		clear_thread_flag(TIF_HRTIMER_REARM);
+	}
+}
+#endif /* TIF_HRTIMER_REARM */
 #endif /* !CONFIG_HIGH_RES_TIMERS */
 
 /*
#14 lore raw

Re: [PATCH v2 5/6] entry,hrtimer: Push reprogramming timers into the interrupt return path

Quoted 5 lines · On Wed, 21 Jan 2026 17:20:15 +0100 Peter Zijlstra <[email protected]> wrote:
kernel/time/hrtimer.c +5 -0
+#ifdef TIF_HRTIMER_REARM
+void _hrtimer_rearm(void)
+{
+	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	ktime_t now, expires_next;

kernel/time/hrtimer.c void hrtimer_interrupt(struct clock_even · line 1994

Quoted 10 lines
kernel/time/hrtimer.c +10 -0
+
+	lockdep_assert_irqs_disabled();
+
+	scoped_guard (raw_spinlock, &cpu_base->lock) {
+		now = hrtimer_update_base(cpu_base);
+		expires_next = hrtimer_update_next_event(cpu_base);
+		__hrtimer_rearm(cpu_base, now, expires_next);
+		clear_thread_flag(TIF_HRTIMER_REARM);
+	}
+}

I'm curious to why you decided to use scoped_guard() here and not just guard() and not add the extra indentation? The function is small enough where everything is expected to be protected by the spinlock.

-- Steve

author maintainer
#15 lore raw

Re: [PATCH v2 5/6] entry,hrtimer: Push reprogramming timers into the interrupt return path

Quoted 14 lines · Steven Rostedt
Quoted 11 lines · On Wed, 21 Jan 2026 17:20:15 +0100 Peter Zijlstra <[email protected]> wrote:
kernel/time/hrtimer.c +11 -0
+#ifdef TIF_HRTIMER_REARM
+void _hrtimer_rearm(void)
+{
+	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	ktime_t now, expires_next;
+
+	lockdep_assert_irqs_disabled();
+
+	scoped_guard (raw_spinlock, &cpu_base->lock) {
+		now = hrtimer_update_base(cpu_base);
+		expires_next = hrtimer_update_next_event(cpu_base);
Quoted 8 lines

kernel/time/hrtimer.c void hrtimer_interrupt(struct clock_even · line 1994

Quoted 4 lines
kernel/time/hrtimer.c +4 -0
+		__hrtimer_rearm(cpu_base, now, expires_next);
+		clear_thread_flag(TIF_HRTIMER_REARM);
+	}
+}

I'm curious to why you decided to use scoped_guard() here and not just guard() and not add the extra indentation? The function is small enough where everything is expected to be protected by the spinlock.

Yeah, I'm not entirely sure... its been over 6 months since I wrote this code :-/

#16 lore raw

Re: [PATCH v2 5/6] entry,hrtimer: Push reprogramming timers into the interrupt return path

In reply to Peter Zijlstra

kernel/entry/common.c static __always_inline unsigned long __e · line 38

Quoted 11 lines · Peter Zijlstra
kernel/entry/common.c +9 -0
 	while (ti_work & EXIT_TO_USER_MODE_WORK_LOOP) {
 
+		/*
+		 * If hrtimer need re-arming, do so before enabling IRQs,
+		 * except when a reschedule is needed, in that case schedule()
+		 * will do this.
+		 */
+		if ((ti_work & (_TIF_NEED_RESCHED |
+				_TIF_NEED_RESCHED_LAZY |
+				_TIF_HRTIMER_REARM)) == _TIF_HRTIMER_REARM)
+			hrtimer_rearm();

Two things I'm not convinced that they are handled correctly:

1) Interrupts

After reenabling interrupts and before reaching schedule() an interrupt comes in and runs soft interrupt processing for a while on the way back, which delays the update until that processing completes.

2) Time slice extension

When the time slice is granted this will not rearm the clockevent device unless the slice hrtimer becomes the first expiring timer on that CPU, but even then that misses the full reevaluation of the next timer event.

Quoted 10 lines
kernel/time/hrtimer.c +2 -3
-static void __hrtimer_rearm(struct hrtimer_cpu_base *cpu_base, ktime_t now)
+static void __hrtimer_rearm(struct hrtimer_cpu_base *cpu_base,
+			    ktime_t now, ktime_t expires_next)
 {
-	ktime_t expires_next = hrtimer_update_next_event(cpu_base);
-
 	cpu_base->expires_next = expires_next;
 	cpu_base->in_hrtirq = 0;

kernel/time/hrtimer.c void hrtimer_interrupt(struct clock_even · line 1976

Quoted 9 lines
kernel/time/hrtimer.c +5 -1
@@ -1970,9 +1969,30 @@ void hrtimer_interrupt(struct clock_even
 		cpu_base->hang_detected = 1;
 	}
 
-	__hrtimer_rearm(cpu_base, now);
+#ifdef TIF_HRTIMER_REARM
+	set_thread_flag(TIF_HRTIMER_REARM);
+#else
+	__hrtimer_rearm(cpu_base, now, expires_next);
+#endif

in hrtimer.h where you already have the #ifdef TIF_HRTIMER_REARM section:

static inline bool hrtimer_set_rearm_delayed()

{
	set_thread_flag(TIF_HRTIMER_REARM);
        return true;
}

and a empty stub returning false for the other case then this becomes:

        if (!hrtimer_set_rearm_delayed())
        	hrtimer_rearm(...);

and the ugly ifdef in the code goes away.

kernel/time/hrtimer.c void hrtimer_interrupt(struct clock_even · line 1981

Quoted 5 lines
kernel/time/hrtimer.c +3 -0
 	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 }
+
+#ifdef TIF_HRTIMER_REARM
+void _hrtimer_rearm(void)

Grr. I had to read this five times to figure out that we now have

hrtimer_rearm()
_hrtimer_rearm()
__hrtimer_rearm()

You clearly ran out of characters to make that obvious:

hrtimer_rearm_delayed()
hrtimer_rearm()
hrtimer_do_rearm()

or something like that.

Thanks,

tglx

author maintainer
#17 lore raw

Re: [PATCH v2 5/6] entry,hrtimer: Push reprogramming timers into the interrupt return path

Quoted 15 lines · Thomas Gleixner

kernel/entry/common.c static __always_inline unsigned long __e · line 38

Quoted 11 lines · Peter Zijlstra
kernel/entry/common.c +9 -0
 	while (ti_work & EXIT_TO_USER_MODE_WORK_LOOP) {
 
+		/*
+		 * If hrtimer need re-arming, do so before enabling IRQs,
+		 * except when a reschedule is needed, in that case schedule()
+		 * will do this.
+		 */
+		if ((ti_work & (_TIF_NEED_RESCHED |
+				_TIF_NEED_RESCHED_LAZY |
+				_TIF_HRTIMER_REARM)) == _TIF_HRTIMER_REARM)
+			hrtimer_rearm();

Two things I'm not convinced that they are handled correctly:

Quoted 6 lines

1) Interrupts

After reenabling interrupts and before reaching schedule() an interrupt comes in and runs soft interrupt processing for a while on the way back, which delays the update until that processing completes.

So the basic thing looks like:

      <USER-MODE>
	irqentry_enter()
	run_irq_on_irqstack_cond()
	  if (user_mode() || hardirq_stack_inuse)
	    irq_enter_rcu();
	    func_c();
	    irq_exit_rcu()
	      __irq_exit_rcu()
	        invoke_softirq()
	irqentry_exit()
	  irqentry_exit_to_user_mode()
	    irqentry_exit_to_user_mode_prepare()
	      __exit_to_user_mode_prepare()
	        exit_to_user_mode_loop() 
		  ...here...

So a nested IRQ at this point will have !user_mode(), but I think it can still end up in softirqs due to that hardirq_stack_inuse. Should we perhaps make sure only user_mode() ends up in softirqs?

Quoted 6 lines

2) Time slice extension

When the time slice is granted this will not rearm the clockevent device unless the slice hrtimer becomes the first expiring timer on that CPU, but even then that misses the full reevaluation of the next timer event.

Oh crud yes, that should be something like:

		if (!rseq_grant_slice_extension(ti_work & TIF_SLICE_EXT_DENY))
			schedule();
		else
			hrtimer_rearm();
maintainer
#18 lore raw

Re: [PATCH v2 5/6] entry,hrtimer: Push reprogramming timers into the interrupt return path

Quoted 41 lines · Peter Zijlstra
Quoted 15 lines · Thomas Gleixner

kernel/entry/common.c static __always_inline unsigned long __e · line 38

Quoted 11 lines · Peter Zijlstra
kernel/entry/common.c +9 -0
 	while (ti_work & EXIT_TO_USER_MODE_WORK_LOOP) {
 
+		/*
+		 * If hrtimer need re-arming, do so before enabling IRQs,
+		 * except when a reschedule is needed, in that case schedule()
+		 * will do this.
+		 */
+		if ((ti_work & (_TIF_NEED_RESCHED |
+				_TIF_NEED_RESCHED_LAZY |
+				_TIF_HRTIMER_REARM)) == _TIF_HRTIMER_REARM)
+			hrtimer_rearm();

Two things I'm not convinced that they are handled correctly:

Quoted 6 lines

1) Interrupts

After reenabling interrupts and before reaching schedule() an interrupt comes in and runs soft interrupt processing for a while on the way back, which delays the update until that processing completes.

So the basic thing looks like:

      <USER-MODE>
	irqentry_enter()
	run_irq_on_irqstack_cond()
	  if (user_mode() || hardirq_stack_inuse)
	    irq_enter_rcu();
	    func_c();
	    irq_exit_rcu()
	      __irq_exit_rcu()
	        invoke_softirq()
	irqentry_exit()
	  irqentry_exit_to_user_mode()
	    irqentry_exit_to_user_mode_prepare()
	      __exit_to_user_mode_prepare()
	        exit_to_user_mode_loop() 
		  ...here...
Quoted 3 lines

So a nested IRQ at this point will have !user_mode(), but I think it can still end up in softirqs due to that hardirq_stack_inuse. Should we perhaps make sure only user_mode() ends up in softirqs?

All interrupts independent of the mode they hit are ending up in irq_exit_rcu() and therefore in __irq_exit_rcu()

run_irq_on_irqstack_cond()
if (user_mode() || hardirq_stack_inuse)

            // Stay on user or hardirq stack
 	    irq_enter_rcu();
 	    func_c();
 	    irq_exit_rcu()
         else
            // MAGIC ASM to switch to hardirq stack
            call irq_enter_rcu
            call func_c
            call irq_exit_rcu

The only reason why invoke_softirq() won't be called is when the interrupt hits into the softirq processing region of the previous interrupt, which means it's already on the hardirq stack.

But looking at this there is already a problem without interrupt nesting:

        irq_enter_rcu();
        timer_interrupt()
          hrtimer_interrupt()
            delay_rearm();
        irq_exit_rcu()
          __irq_exit_rcu()
            invoke_softirq()  <- Here

Soft interrupts can run for quite some time, which means this already can cause timers being delayed for way too long. I think in __irq_exit_rcu() you want to do:

       if (!in_interrupt() && local_softirq_pending()) {
         hrtimer_rearm();
         invoke_softirq();
       }

Thanks,

tglx

maintainer
#19 lore raw

Re: [PATCH v2 5/6] entry,hrtimer: Push reprogramming timers into the interrupt return path

Quoted 20 lines · Thomas Gleixner
Quoted 3 lines · Peter Zijlstra

So a nested IRQ at this point will have !user_mode(), but I think it can still end up in softirqs due to that hardirq_stack_inuse. Should we perhaps make sure only user_mode() ends up in softirqs?

All interrupts independent of the mode they hit are ending up in irq_exit_rcu() and therefore in __irq_exit_rcu()

(quoted diff) +0 -0
 	run_irq_on_irqstack_cond()
 	  if (user_mode() || hardirq_stack_inuse)
            // Stay on user or hardirq stack
 	    irq_enter_rcu();
 	    func_c();
 	    irq_exit_rcu()
         else
            // MAGIC ASM to switch to hardirq stack
            call irq_enter_rcu
            call func_c
            call irq_exit_rcu
Quoted 3 lines

The only reason why invoke_softirq() won't be called is when the interrupt hits into the softirq processing region of the previous interrupt, which means it's already on the hardirq stack.

In the case I pointed out where the second interrupt hits right after exit to user enabled interupts, there is no nesting and it will happily take the second path which switches to the hardirq stack and then on return processes soft interrupts.

Quoted 11 lines

But looking at this there is already a problem without interrupt nesting:

        irq_enter_rcu();
        timer_interrupt()
          hrtimer_interrupt()
            delay_rearm();
        irq_exit_rcu()
          __irq_exit_rcu()
            invoke_softirq()  <- Here
Quoted 8 lines

Soft interrupts can run for quite some time, which means this already can cause timers being delayed for way too long. I think in __irq_exit_rcu() you want to do:

       if (!in_interrupt() && local_softirq_pending()) {
         hrtimer_rearm();
         invoke_softirq();
       }

Actually it's worse. Assume the CPU on which this happens has the jiffies duty. As the timer does not fire, jiffies become stale. So anything which relies on jiffies going forward will get stuck until some other condition breaks the tie. That's going to be fun to debug :)

Thanks,

tglx

author maintainer
#20 lore raw

Re: [PATCH v2 5/6] entry,hrtimer: Push reprogramming timers into the interrupt return path

In reply to Thomas Gleixner
Quoted 11 lines · Thomas Gleixner

But looking at this there is already a problem without interrupt nesting:

        irq_enter_rcu();
        timer_interrupt()
          hrtimer_interrupt()
            delay_rearm();
        irq_exit_rcu()
          __irq_exit_rcu()
            invoke_softirq()  <- Here
Quoted 8 lines

Soft interrupts can run for quite some time, which means this already can cause timers being delayed for way too long. I think in __irq_exit_rcu() you want to do:

       if (!in_interrupt() && local_softirq_pending()) {
         hrtimer_rearm();
         invoke_softirq();
       }

Right, and we can do the same on (nested) IRQ entry. Something like so:

---

kernel/entry/common.c +2 -0
@@ -63,6 +63,8 @@ static __always_inline unsigned long __e
 		if (ti_work & (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY)) {
 			if (!rseq_grant_slice_extension(ti_work & TIF_SLICE_EXT_DENY))
 				schedule();
+			else
+				hrtimer_rearm();
 		}
 
 		if (ti_work & _TIF_UPROBE)
kernel/softirq.c +14 -1
@@ -663,6 +663,13 @@ void irq_enter_rcu(void)
 {
 	__irq_enter_raw();
 
+	/*
+	 * If this is a nested IRQ that hits the exit_to_user_mode_loop
+	 * where it has enabled IRQs but before it has hit schedule()
+	 * we could have hrtimers in an undefined state. Fix it up here.
+	 */
+	hrtimer_rearm();
+
 	if (tick_nohz_full_cpu(smp_processor_id()) ||
 	    (is_idle_task(current) && (irq_count() == HARDIRQ_OFFSET)))
 		tick_irq_enter();
@@ -719,8 +726,14 @@ static inline void __irq_exit_rcu(void)
 #endif
 	account_hardirq_exit(current);
 	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt() && local_softirq_pending())
+	if (!in_interrupt() && local_softirq_pending()) {
+		/*
+		 * If we left hrtimers unarmed, make sure to arm them now,
+		 * before enabling interrupts to run SoftIRQ.
+		 */
+		hrtimer_rearm();
 		invoke_softirq();
+	}
 
 	if (IS_ENABLED(CONFIG_IRQ_FORCED_THREADING) && force_irqthreads() &&
 	    local_timers_pending_force_th() && !(in_nmi() | in_hardirq()))
kernel/time/hrtimer.c +3 -3
@@ -1279,8 +1279,8 @@ static int __hrtimer_start_range_ns(stru
 
 	if (timer->is_fuzzy) {
 		/*
-		 * XXX fuzzy implies pinned!  not sure how to deal with
-		 * retrigger_next_event() for the !local case.
+		 * Fuzzy requires pinned as the lazy programming only works
+		 * for CPU local timers.
 		 */
 		WARN_ON_ONCE(!(mode & HRTIMER_MODE_PINNED));
 		/*
@@ -1898,7 +1898,7 @@ static __latent_entropy void hrtimer_run
 
 /*
  * Very similar to hrtimer_force_reprogram(), except it deals with
- * in_hrirq and hang_detected.
+ * in_hrtirq and hang_detected.
  */
 static void __hrtimer_rearm(struct hrtimer_cpu_base *cpu_base,
 			    ktime_t now, ktime_t expires_next)
author maintainer
#21 lore raw

PATCH v2 6/6 sched: Default enable HRTICK

In reply to Peter Zijlstra

... for generic entry architectures. This decouples preemption from CONFIG_HZ, leaving only the periodic load-balancer and various accounting things relying on the tick.

---
 kernel/sched/features.h |    5 +++++
 1 file changed, 5 insertions(+)
kernel/sched/features.h +5 -0
@@ -63,8 +63,13 @@ SCHED_FEAT(DELAY_ZERO, true)
  */
 SCHED_FEAT(WAKEUP_PREEMPTION, true)
 
+#ifdef TIF_HRTIMER_REARM
+SCHED_FEAT(HRTICK, true)
+SCHED_FEAT(HRTICK_DL, true)
+#else
 SCHED_FEAT(HRTICK, false)
 SCHED_FEAT(HRTICK_DL, false)
+#endif
 
 /*
  * Decrement CPU capacity based on time not spent running tasks
#22 lore raw

Re: [PATCH v2 6/6] sched: Default enable HRTICK

Hi Peter,

Quoted 12 lines · On Wed, Jan 21, 2026 at 05:20:16PM +0100 Peter Zijlstra wrote:

... for generic entry architectures. This decouples preemption from CONFIG_HZ, leaving only the periodic load-balancer and various accounting things relying on the tick.

---
 kernel/sched/features.h |    5 +++++
 1 file changed, 5 insertions(+)

kernel/sched/features.h SCHED_FEAT(DELAY_ZERO, true) · line 72

Quoted 10 lines
kernel/sched/features.h +5 -0
@@ -63,8 +63,13 @@ SCHED_FEAT(DELAY_ZERO, true)
  */
 SCHED_FEAT(WAKEUP_PREEMPTION, true)
 
+#ifdef TIF_HRTIMER_REARM
+SCHED_FEAT(HRTICK, true)
+SCHED_FEAT(HRTICK_DL, true)
+#else
 SCHED_FEAT(HRTICK, false)
 SCHED_FEAT(HRTICK_DL, false)
+#endif

I maybe be missing something. But the title of this patch and the above code do not seem to match.

Cheers,
Phil

Quoted 6 lines
kernel/sched/features.h +0 -0
 /*
  * Decrement CPU capacity based on time not spent running tasks
 
author maintainer
#23 lore raw

Re: [PATCH v2 6/6] sched: Default enable HRTICK

Quoted 12 lines · Phil Auld

Hi Peter,

Quoted 9 lines · On Wed, Jan 21, 2026 at 05:20:16PM +0100 Peter Zijlstra wrote:

... for generic entry architectures. This decouples preemption from CONFIG_HZ, leaving only the periodic load-balancer and various accounting things relying on the tick.

---
 kernel/sched/features.h |    5 +++++
 1 file changed, 5 insertions(+)
Quoted 7 lines

kernel/sched/features.h SCHED_FEAT(DELAY_ZERO, true) · line 66

Quoted 7 lines
kernel/sched/features.h +1 -0
@@ -63,8 +63,13 @@ SCHED_FEAT(DELAY_ZERO, true)
  */
 SCHED_FEAT(WAKEUP_PREEMPTION, true)
 
+#ifdef TIF_HRTIMER_REARM

Arguably this should be CONFIG_GENERIC_ENTRY I suppose

Quoted 9 lines

kernel/sched/features.h SCHED_FEAT(DELAY_ZERO, true) · line 72

Quoted 6 lines
kernel/sched/features.h +4 -0
+SCHED_FEAT(HRTICK, true)
+SCHED_FEAT(HRTICK_DL, true)
+#else
 SCHED_FEAT(HRTICK, false)
 SCHED_FEAT(HRTICK_DL, false)
+#endif

I maybe be missing something. But the title of this patch and the above code do not seem to match.

You mean it only default enables it for a subset of architectures?

#24 lore raw

Re: [PATCH v2 6/6] sched: Default enable HRTICK

Quoted 33 lines · On Thu, Jan 22, 2026 at 12:40:54PM +0100 Peter Zijlstra wrote:
Quoted 12 lines · Phil Auld

Hi Peter,

Quoted 9 lines · On Wed, Jan 21, 2026 at 05:20:16PM +0100 Peter Zijlstra wrote:

... for generic entry architectures. This decouples preemption from CONFIG_HZ, leaving only the periodic load-balancer and various accounting things relying on the tick.

---
 kernel/sched/features.h |    5 +++++
 1 file changed, 5 insertions(+)
Quoted 7 lines

kernel/sched/features.h SCHED_FEAT(DELAY_ZERO, true) · line 66

Quoted 7 lines
kernel/sched/features.h +1 -0
@@ -63,8 +63,13 @@ SCHED_FEAT(DELAY_ZERO, true)
  */
 SCHED_FEAT(WAKEUP_PREEMPTION, true)
 
+#ifdef TIF_HRTIMER_REARM

Arguably this should be CONFIG_GENERIC_ENTRY I suppose

Quoted 9 lines

kernel/sched/features.h SCHED_FEAT(DELAY_ZERO, true) · line 72

Quoted 6 lines
kernel/sched/features.h +4 -0
+SCHED_FEAT(HRTICK, true)
+SCHED_FEAT(HRTICK_DL, true)
+#else
 SCHED_FEAT(HRTICK, false)
 SCHED_FEAT(HRTICK_DL, false)
+#endif

I maybe be missing something. But the title of this patch and the above code do not seem to match.

Quoted 2 lines

You mean it only default enables it for a subset of architectures?

Nope, I mean I can't read... nevermind.

Cheers,
Phil