Skip to contentSkip to conversation

hrtimer/sched: Improve hrtick

PATCH Applied Peter Zijlstra 5 participants 22 messages Sep 18, 2025 – Dec 14, 2025

Subsystems (4)

v1 of 2 Applied Latest is v2 →

lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

cover hrtimer/sched: Improve hrtick

Hi!

A few patches trying to improve the sched hrtick feature, which is currently disabled due to overhead.

I wrote these after the last OSPM, but I've been sitting on them because 0-day is not having a good time with the second to last patch. I've not been able to reproduce :-(

Anyway, the first few patches should be 'good' and hopefully one of you will spot my 'obvious' fail in that late patch.

For those of you rocking ARM64 systems, if you pick up the generic entry patches:

https://lkml.kernel.org/r/[email protected]

this late patch should also 'work' (or not as the case might be) for you.

Patches also here:

git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/hrtick

author maintainer
#1 lore raw

PATCH 1/8 sched: Fix hrtick() vs scheduling context

The sched_class::task_tick() method is called on the donor sched_class, and sched_tick() hands it rq->donor as argument, which is consistent.

However, while hrtick() uses the donor sched_class, it then passes rq->curr, which is inconsistent. Fix it.

---
 kernel/sched/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -875,7 +875,7 @@ static enum hrtimer_restart hrtick(struc
 
 	rq_lock(rq, &rf);
 	update_rq_clock(rq);
-	rq->donor->sched_class->task_tick(rq, rq->curr, 1);
+	rq->donor->sched_class->task_tick(rq, rq->donor, 1);
 	rq_unlock(rq, &rf);
 
 	return HRTIMER_NORESTART;
reviewer Reviewed-by on v1 1/8
#2 lore raw

Re: [PATCH 1/8] sched: Fix hrtick() vs scheduling context

Hello Peter,

Quoted 6 lines · Peter Zijlstra

The sched_class::task_tick() method is called on the donor sched_class, and sched_tick() hands it rq->donor as argument, which is consistent.

However, while hrtick() uses the donor sched_class, it then passes rq->curr, which is inconsistent. Fix it.

Can we add either a:

Fixes 7de9d4f94638 ("sched: Start blocked_on chain processing in find_proxy_task()")

where this starts making a difference functionally since single CPU proxy can have rq->curr != rq->donor, or we can target the same commit where task_tick(0 in sched_tick() was updated with:

Fixes af0c8b2bf67b ("sched: Split scheduler and execution contexts")

Other than that, this looks good to me. Feel free to include:

Quoted 20 lines
---
 kernel/sched/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -875,7 +875,7 @@ static enum hrtimer_restart hrtick(struc
 
 	rq_lock(rq, &rf);
 	update_rq_clock(rq);
-	rq->donor->sched_class->task_tick(rq, rq->curr, 1);
+	rq->donor->sched_class->task_tick(rq, rq->donor, 1);
 	rq_unlock(rq, &rf);
 
 	return HRTIMER_NORESTART;
Signature · 2 lines
Thanks and Regards,
Prateek
Acked-by on v1 1/8
#3 lore raw

Re: [PATCH 1/8] sched: Fix hrtick() vs scheduling context

In reply to Peter Zijlstra
Quoted 16 lines · Peter Zijlstra

The sched_class::task_tick() method is called on the donor sched_class, and sched_tick() hands it rq->donor as argument, which is consistent.

However, while hrtick() uses the donor sched_class, it then passes rq->curr, which is inconsistent. Fix it.

---
 kernel/sched/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/sched/core.c static enum hrtimer_restart hrtick(struc · line 879

Quoted 6 lines
kernel/sched/core.c +1 -1
@@ -875,7 +875,7 @@ static enum hrtimer_restart hrtick(struc
 
        rq_lock(rq, &rf);
        update_rq_clock(rq);
-       rq->donor->sched_class->task_tick(rq, rq->curr, 1);
+       rq->donor->sched_class->task_tick(rq, rq->donor, 1);
        rq_unlock(rq, &rf);

Ah. Thanks for catching this! I've run through with some stress testing on this and haven't seen any problems so far.

thanks
-john

merge bot Applied
#4 lore raw

[tip: sched/urgent] sched/hrtick: Fix hrtick() vs. scheduling context

In reply to Peter Zijlstra
The following commit has been merged into the sched/urgent branch of tip:

The following commit has been merged into the sched/urgent branch of tip:

Commit-ID:     8720ba2d028f1aff08a55d8fe1a124dd5a6cfb0a
Gitweb:        https://git.kernel.org/tip/8720ba2d028f1aff08a55d8fe1a124dd5a6cfb0a
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Mon, 01 Sep 2025 22:46:29 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 02 Dec 2025 15:37:52 +01:00

sched/hrtick: Fix hrtick() vs. scheduling context

The sched_class::task_tick() method is called on the donor sched_class, and sched_tick() hands it rq->donor as argument, which is consistent.

However, while hrtick() uses the donor sched_class, it then passes rq->curr, which is inconsistent. Fix it.

---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index 7dfb6a9..be55f95 100644
@@ -878,7 +878,7 @@ static enum hrtimer_restart hrtick(struct hrtimer *timer)
 
 	rq_lock(rq, &rf);
 	update_rq_clock(rq);
-	rq->donor->sched_class->task_tick(rq, rq->curr, 1);
+	rq->donor->sched_class->task_tick(rq, rq->donor, 1);
 	rq_unlock(rq, &rf);
 
 	return HRTIMER_NORESTART;
merge bot Applied
#5 lore raw

[tip: sched/urgent] sched/hrtick: Fix hrtick() vs. scheduling context

In reply to Peter Zijlstra
The following commit has been merged into the sched/urgent branch of tip:

The following commit has been merged into the sched/urgent branch of tip:

Commit-ID:     40671f3f91986844df8947b65b9af5e770752047
Gitweb:        https://git.kernel.org/tip/40671f3f91986844df8947b65b9af5e770752047
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Mon, 01 Sep 2025 22:46:29 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Wed, 03 Dec 2025 19:26:00 +01:00

sched/hrtick: Fix hrtick() vs. scheduling context

The sched_class::task_tick() method is called on the donor sched_class, and sched_tick() hands it rq->donor as argument, which is consistent.

However, while hrtick() uses the donor sched_class, it then passes rq->curr, which is inconsistent. Fix it.

---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index fc358c1..1711e9e 100644
@@ -878,7 +878,7 @@ static enum hrtimer_restart hrtick(struct hrtimer *timer)
 
 	rq_lock(rq, &rf);
 	update_rq_clock(rq);
-	rq->donor->sched_class->task_tick(rq, rq->curr, 1);
+	rq->donor->sched_class->task_tick(rq, rq->donor, 1);
 	rq_unlock(rq, &rf);
 
 	return HRTIMER_NORESTART;
merge bot Applied
#6 lore raw

[tip: sched/urgent] sched/hrtick: Fix hrtick() vs. scheduling context

In reply to Peter Zijlstra
The following commit has been merged into the sched/urgent branch of tip:

The following commit has been merged into the sched/urgent branch of tip:

Commit-ID:     e38e5299747b23015b00b0109891815db44a2f30
Gitweb:        https://git.kernel.org/tip/e38e5299747b23015b00b0109891815db44a2f30
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Mon, 01 Sep 2025 22:46:29 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Sat, 06 Dec 2025 10:03:13 +01:00

sched/hrtick: Fix hrtick() vs. scheduling context

The sched_class::task_tick() method is called on the donor sched_class, and sched_tick() hands it rq->donor as argument, which is consistent.

However, while hrtick() uses the donor sched_class, it then passes rq->curr, which is inconsistent. Fix it.

---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index fc358c1..1711e9e 100644
@@ -878,7 +878,7 @@ static enum hrtimer_restart hrtick(struct hrtimer *timer)
 
 	rq_lock(rq, &rf);
 	update_rq_clock(rq);
-	rq->donor->sched_class->task_tick(rq, rq->curr, 1);
+	rq->donor->sched_class->task_tick(rq, rq->donor, 1);
 	rq_unlock(rq, &rf);
 
 	return HRTIMER_NORESTART;
author maintainer
#7 lore raw

PATCH 2/8 sched/fair: Limit hrtick work

In reply to Peter Zijlstra

The task_tick_fair() function does:

- update the hierarchical runtimes
- drive numa-balancing
- update load-balance statistics
- drive force-idle preemption

All but the very first can be limited to the periodic tick. Let hrtick only update accounting and drive preemption, not load-balancing and other bits.

---
 kernel/sched/fair.c |    6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/fair.c +6 -0
@@ -13119,6 +13119,12 @@ static void task_tick_fair(struct rq *rq
 		entity_tick(cfs_rq, se, queued);
 	}
 
+	if (queued) {
+		if (!need_resched())
+			hrtick_start_fair(rq, curr);
+		return;
+	}
+
 	if (static_branch_unlikely(&sched_numa_balancing))
 		task_tick_numa(rq, curr);
 
#8 lore raw

Re: [PATCH 2/8] sched/fair: Limit hrtick work

Hello Peter,

kernel/sched/fair.c static void task_tick_fair(struct rq *rq · line 13124

Quoted 7 lines · Peter Zijlstra
kernel/sched/fair.c +3 -0
@@ -13119,6 +13119,12 @@ static void task_tick_fair(struct rq *rq
 		entity_tick(cfs_rq, se, queued);
 	}
 
+	if (queued) {
+		if (!need_resched())
+			hrtick_start_fair(rq, curr);

Do we need a hrtick_start_fair() here? Queued tick will always do a resched_curr_lazy() - if another HRTICK fires before the next tick, all it'll do is resched_curr_lazy() again and the next opportunity to resched is either exit to userspace or the periodic tick firing and promoting that LAZY to a full NEED_RESCHED.

The early return does makes sense.

Quoted 6 lines
kernel/sched/fair.c +3 -0
+		return;
+	}
+
 	if (static_branch_unlikely(&sched_numa_balancing))
 		task_tick_numa(rq, curr);
 
Signature · 2 lines
Thanks and Regards,
Prateek
author maintainer
#9 lore raw

Re: [PATCH 2/8] sched/fair: Limit hrtick work

Quoted 7 lines · K Prateek Nayak

Hello Peter,

Quoted 4 lines · Peter Zijlstra
kernel/sched/fair.c +0 -0
@@ -13119,6 +13119,12 @@ static void task_tick_fair(struct rq *rq
 		entity_tick(cfs_rq, se, queued);
 	}
 
Quoted 9 lines

kernel/sched/fair.c static void task_tick_fair(struct rq *rq · line 13124

Quoted 3 lines
kernel/sched/fair.c +3 -0
+	if (queued) {
+		if (!need_resched())
+			hrtick_start_fair(rq, curr);

Do we need a hrtick_start_fair() here? Queued tick will always do a resched_curr_lazy() - if another HRTICK fires before the next tick, all it'll do is resched_curr_lazy() again and the next opportunity to resched is either exit to userspace or the periodic tick firing and promoting that LAZY to a full NEED_RESCHED.

I think I had a version where entity_tick() doesn't force need_resched on queue, and in that case the timer, which is wallclock, and update_curr(), which is task_clock, might disagree and we might not have reached the deadline, and so we need to try again.

merge bot Applied
#10 lore raw

[tip: sched/core] sched/fair: Limit hrtick work

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     95a0155224a658965f34ed4b1943b238d9be1fea
Gitweb:        https://git.kernel.org/tip/95a0155224a658965f34ed4b1943b238d9be1fea
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Mon, 01 Sep 2025 22:50:56 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Sun, 14 Dec 2025 08:25:02 +01:00

sched/fair: Limit hrtick work

The task_tick_fair() function does:

- update the hierarchical runtimes
- drive NUMA-balancing
- update load-balance statistics
- drive force-idle preemption

All but the very first can be limited to the periodic tick. Let hrtick only update accounting and drive preemption, not load-balancing and other bits.

---
 kernel/sched/fair.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/fair.c +6 -0
index 496a30a..f79951f 100644
@@ -13332,6 +13332,12 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 		entity_tick(cfs_rq, se, queued);
 	}
 
+	if (queued) {
+		if (!need_resched())
+			hrtick_start_fair(rq, curr);
+		return;
+	}
+
 	if (static_branch_unlikely(&sched_numa_balancing))
 		task_tick_numa(rq, curr);
 
author maintainer
#11 lore raw

PATCH 3/8 sched/eevdf: Fix HRTICK duration

In reply to Peter Zijlstra

The nominal duration for an EEVDF task to run is until its deadline. At which point the deadline is moved ahead and a new task selection is done.

---
 kernel/sched/fair.c |   40 +++++++++++++++++++++++++++++-----------
 1 file changed, 29 insertions(+), 11 deletions(-)
kernel/sched/fair.c +29 -11
@@ -6775,21 +6775,39 @@ static inline void sched_fair_update_sto
 static void hrtick_start_fair(struct rq *rq, struct task_struct *p)
 {
 	struct sched_entity *se = &p->se;
+	unsigned long scale = 1024;
+	unsigned long util = 0;
+	u64 vdelta;
+	u64 delta;
 
 	WARN_ON_ONCE(task_rq(p) != rq);
 
-	if (rq->cfs.h_nr_queued > 1) {
-		u64 ran = se->sum_exec_runtime - se->prev_sum_exec_runtime;
-		u64 slice = se->slice;
-		s64 delta = slice - ran;
-
-		if (delta < 0) {
-			if (task_current_donor(rq, p))
-				resched_curr(rq);
-			return;
-		}
-		hrtick_start(rq, delta);
+	if (rq->cfs.h_nr_queued <= 1)
+		return;
+
+	/*
+	 * Compute time until virtual deadline
+	 */
+	vdelta = se->deadline - se->vruntime;
+	if ((s64)vdelta < 0) {
+		if (task_current_donor(rq, p))
+			resched_curr(rq);
+		return;
+	}
+	delta = (se->load.weight * vdelta) / NICE_0_LOAD;
+
+	/*
+	 * Correct for instantaneous load of other classes.
+	 */
+	util += cpu_util_dl(rq);
+	util += cpu_util_rt(rq);
+	util += cpu_util_irq(rq);
+	if (util && util < 1024) {
+		scale *= 1024;
+		scale /= (1024 - util);
 	}
+
+	hrtick_start(rq, (scale * delta) / 1024);
 }
 
 /*
#12 lore raw

Re: [PATCH 3/8] sched/eevdf: Fix HRTICK duration

Hello Peter,

kernel/sched/fair.c static inline void sched_fair_update_sto · line 6794

Quoted 7 lines · Peter Zijlstra
kernel/sched/fair.c +7 -0
+	/*
+	 * Compute time until virtual deadline
+	 */
+	vdelta = se->deadline - se->vruntime;
+	if ((s64)vdelta < 0) {
+		if (task_current_donor(rq, p))
+			resched_curr(rq);

Why the task_current_donor() check? If the scheduling context has run out of gas, shouldn't we reschedule curr even if we were proxied?

Quoted 4 lines
kernel/sched/fair.c +4 -0
+		return;
+	}
+	delta = (se->load.weight * vdelta) / NICE_0_LOAD;
+

kernel/sched/fair.c static inline void sched_fair_update_sto · line 6808

Quoted 10 lines
kernel/sched/fair.c +9 -0
+	/*
+	 * Correct for instantaneous load of other classes.
+	 */
+	util += cpu_util_dl(rq);
+	util += cpu_util_rt(rq);
+	util += cpu_util_irq(rq);
+	if (util && util < 1024) {
+		scale *= 1024;
+		scale /= (1024 - util);
 	}

Could it be possible that we arrive here from the dl_server's pick and end up inflating the HRTICK duration despite having an uninterrupted period for fair tasks ahead?

Quoted 5 lines
kernel/sched/fair.c +2 -0
+
+	hrtick_start(rq, (scale * delta) / 1024);
 }
 
 /*
Signature · 2 lines
Thanks and Regards,
Prateek
author maintainer
#13 lore raw

Re: [PATCH 3/8] sched/eevdf: Fix HRTICK duration

Quoted 13 lines · K Prateek Nayak

Hello Peter,

kernel/sched/fair.c static inline void sched_fair_update_sto · line 6794

Quoted 7 lines · Peter Zijlstra
kernel/sched/fair.c +7 -0
+	/*
+	 * Compute time until virtual deadline
+	 */
+	vdelta = se->deadline - se->vruntime;
+	if ((s64)vdelta < 0) {
+		if (task_current_donor(rq, p))
+			resched_curr(rq);

Why the task_current_donor() check? If the scheduling context has run out of gas, shouldn't we reschedule curr even if we were proxied?

task_current_donor() is the current scheduling context, right? So we've just determined that vruntime is ahead of deadline, which means we should reschedule now.

Quoted 10 lines
Quoted 10 lines
kernel/sched/fair.c +10 -0
+		return;
+	}
+	delta = (se->load.weight * vdelta) / NICE_0_LOAD;
+
+	/*
+	 * Correct for instantaneous load of other classes.
+	 */
+	util += cpu_util_dl(rq);
+	util += cpu_util_rt(rq);
+	util += cpu_util_irq(rq);
Quoted 8 lines

kernel/sched/fair.c static inline void sched_fair_update_sto · line 6808

Quoted 4 lines
kernel/sched/fair.c +3 -0
+	if (util && util < 1024) {
+		scale *= 1024;
+		scale /= (1024 - util);
 	}

Could it be possible that we arrive here from the dl_server's pick and end up inflating the HRTICK duration despite having an uninterrupted period for fair tasks ahead?

Yes, but since this is all approximation anyway, how many correction terms do we want to stack on top ? :-)

Quoted 8 lines
Quoted 4 lines
kernel/sched/fair.c +1 -0
+	hrtick_start(rq, (scale * delta) / 1024);
 }
 
 /*
Signature · 2 lines
Thanks and Regards,
Prateek
author maintainer
#14 lore raw

PATCH 4/8 hrtimer: Optimize __hrtimer_start_range_ns()

In reply to Peter Zijlstra

Much like hrtimer_reprogram(), skip programming is the cpu_base is running the hrtimer interrupt.

---
 kernel/time/hrtimer.c |    8 ++++++++
 1 file changed, 8 insertions(+)
kernel/time/hrtimer.c +8 -0
@@ -1261,6 +1261,14 @@ static int __hrtimer_start_range_ns(stru
 	}
 
 	first = enqueue_hrtimer(timer, new_base, mode);
+
+	/*
+	 * If the hrtimer interrupt is running, then it will reevaluate the
+	 * clock bases and reprogram the clock event device.
+	 */
+	if (new_base->cpu_base->in_hrtirq)
+		return 0;
+
 	if (!force_local) {
 		/*
 		 * If the current CPU base is online, then the timer is
author maintainer
#15 lore raw

PATCH 5/8 hrtimer,sched: Add fuzzy hrtimer mode for HRTICK

In reply to Peter Zijlstra

Upon schedule() HRTICK will cancel the current timer, pick the next task and reprogram the timer. When schedule() consistently triggers due to blocking conditions instead of the timer, this leads to endless reprogramming without ever firing.

Mitigate this with a new hrtimer mode: fuzzy (not really happy with that name); this mode does two things:

- skip reprogramming the hardware on timer remove;
- skip reprogramming the hardware when the new timer is after cpu_base->expires_next

Both things are already possible;

- removing a remote timer will leave the hardware programmed and cause a spurious interrupt.
- this remote CPU adding a timer can skip the reprogramming when the timer's expiration is after the (spurious) expiration.

This new timer mode simply causes more of this 'fuzzy' behaviour; it causes a few spurious interrupts, but similarly avoids endlessly reprogramming the timer.

This makes the HRTICK match the NO_HRTICK hackbench runs.

---
 include/linux/hrtimer.h       |    1 +
 include/linux/hrtimer_types.h |    1 +
 kernel/sched/core.c           |    3 ++-
 kernel/time/hrtimer.c         |   16 +++++++++++++++-
 4 files changed, 19 insertions(+), 2 deletions(-)
include/linux/hrtimer.h +1 -0
@@ -38,6 +38,7 @@ enum hrtimer_mode {
 	HRTIMER_MODE_PINNED	= 0x02,
 	HRTIMER_MODE_SOFT	= 0x04,
 	HRTIMER_MODE_HARD	= 0x08,
+	HRTIMER_MODE_FUZZY	= 0x10,
 
 	HRTIMER_MODE_ABS_PINNED = HRTIMER_MODE_ABS | HRTIMER_MODE_PINNED,
 	HRTIMER_MODE_REL_PINNED = HRTIMER_MODE_REL | HRTIMER_MODE_PINNED,
include/linux/hrtimer_types.h +1 -0
@@ -45,6 +45,7 @@ struct hrtimer {
 	u8				is_rel;
 	u8				is_soft;
 	u8				is_hard;
+	u8				is_fuzzy;
 };
 
 #endif /* _LINUX_HRTIMER_TYPES_H */
kernel/sched/core.c +2 -1
@@ -928,7 +928,8 @@ void hrtick_start(struct rq *rq, u64 del
 static void hrtick_rq_init(struct rq *rq)
 {
 	INIT_CSD(&rq->hrtick_csd, __hrtick_start, rq);
-	hrtimer_setup(&rq->hrtick_timer, hrtick, CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD);
+	hrtimer_setup(&rq->hrtick_timer, hrtick, CLOCK_MONOTONIC,
+		      HRTIMER_MODE_REL_HARD | HRTIMER_MODE_FUZZY);
 }
 #else /* !CONFIG_SCHED_HRTICK: */
 static inline void hrtick_clear(struct rq *rq)
kernel/time/hrtimer.c +15 -1
@@ -1122,7 +1122,7 @@ static void __remove_hrtimer(struct hrti
 	 * an superfluous call to hrtimer_force_reprogram() on the
 	 * remote cpu later on if the same timer gets enqueued again.
 	 */
-	if (reprogram && timer == cpu_base->next_timer)
+	if (!timer->is_fuzzy && reprogram && timer == cpu_base->next_timer)
 		hrtimer_force_reprogram(cpu_base, 1);
 }
 
@@ -1269,6 +1269,19 @@ static int __hrtimer_start_range_ns(stru
 	if (new_base->cpu_base->in_hrtirq)
 		return 0;
 
+	if (timer->is_fuzzy) {
+		/*
+		 * XXX fuzzy implies pinned!  not sure how to deal with
+		 * retrigger_next_event() for the !local case.
+		 */
+		WARN_ON_ONCE(!(mode & HRTIMER_MODE_PINNED));
+		/*
+		 * Notably, by going into hrtimer_reprogram(), it will
+		 * not reprogram if cpu_base->expires_next is earlier.
+		 */
+		return first;
+	}
+
 	if (!force_local) {
 		/*
 		 * If the current CPU base is online, then the timer is
@@ -1645,6 +1658,7 @@ static void __hrtimer_setup(struct hrtim
 	base += hrtimer_clockid_to_base(clock_id);
 	timer->is_soft = softtimer;
 	timer->is_hard = !!(mode & HRTIMER_MODE_HARD);
+	timer->is_fuzzy = !!(mode & HRTIMER_MODE_FUZZY);
 	timer->base = &cpu_base->clock_base[base];
 	timerqueue_init(&timer->node);
 
author maintainer
#16 lore raw

PATCH 6/8 hrtimer: Re-arrange hrtimer_interrupt()

In reply to Peter Zijlstra

Rework hrtimer_interrupt() such that reprogramming is split out into an independent function at the end of the interrupt.

This prepares for reprogramming getting delayed beyond the end of hrtimer_interrupt().

Notably, this changes the hang handling to always wait 100ms instead of trying to keep it proportional to the actual delay. This simplifies the state, also this really shouldn't be happening.

---
 kernel/time/hrtimer.c |   87 ++++++++++++++++++++++----------------------------
 1 file changed, 39 insertions(+), 48 deletions(-)
kernel/time/hrtimer.c +39 -48
@@ -1889,6 +1889,29 @@ static __latent_entropy void hrtimer_run
 #ifdef CONFIG_HIGH_RES_TIMERS
 
 /*
+ * Very similar to hrtimer_force_reprogram(), except it deals with
+ * in_hrirq and hang_detected.
+ */
+static void __hrtimer_rearm(struct hrtimer_cpu_base *cpu_base, ktime_t now)
+{
+	ktime_t expires_next = hrtimer_update_next_event(cpu_base);
+
+	cpu_base->expires_next = expires_next;
+	cpu_base->in_hrtirq = 0;
+
+	if (unlikely(cpu_base->hang_detected)) {
+		/*
+		 * Give the system a chance to do something else than looping
+		 * on hrtimer interrupts.
+		 */
+		expires_next = ktime_add_ns(now, 100 * NSEC_PER_MSEC);
+		cpu_base->hang_detected = 0;
+	}
+
+	tick_program_event(expires_next, 1);
+}
+
+/*
  * High resolution timer interrupt
  * Called with interrupts disabled
  */
@@ -1924,63 +1947,31 @@ void hrtimer_interrupt(struct clock_even
 
 	__hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
 
-	/* Reevaluate the clock bases for the [soft] next expiry */
-	expires_next = hrtimer_update_next_event(cpu_base);
-	/*
-	 * Store the new expiry value so the migration code can verify
-	 * against it.
-	 */
-	cpu_base->expires_next = expires_next;
-	cpu_base->in_hrtirq = 0;
-	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
-
-	/* Reprogramming necessary ? */
-	if (!tick_program_event(expires_next, 0)) {
-		cpu_base->hang_detected = 0;
-		return;
-	}
-
 	/*
 	 * The next timer was already expired due to:
 	 * - tracing
 	 * - long lasting callbacks
 	 * - being scheduled away when running in a VM
 	 *
-	 * We need to prevent that we loop forever in the hrtimer
-	 * interrupt routine. We give it 3 attempts to avoid
-	 * overreacting on some spurious event.
-	 *
-	 * Acquire base lock for updating the offsets and retrieving
-	 * the current time.
+	 * We need to prevent that we loop forever in the hrtiner interrupt
+	 * routine. We give it 3 attempts to avoid overreacting on some
+	 * spurious event.
 	 */
-	raw_spin_lock_irqsave(&cpu_base->lock, flags);
+	expires_next = hrtimer_update_next_event(cpu_base);
 	now = hrtimer_update_base(cpu_base);
-	cpu_base->nr_retries++;
-	if (++retries < 3)
-		goto retry;
-	/*
-	 * Give the system a chance to do something else than looping
-	 * here. We stored the entry time, so we know exactly how long
-	 * we spent here. We schedule the next event this amount of
-	 * time away.
-	 */
-	cpu_base->nr_hangs++;
-	cpu_base->hang_detected = 1;
-	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
+	if (expires_next < now) {
+		if (++retries < 3)
+			goto retry;
+
+		delta = ktime_sub(now, entry_time);
+		cpu_base->max_hang_time = max_t(unsigned int,
+						cpu_base->max_hang_time, delta);
+		cpu_base->nr_hangs++;
+		cpu_base->hang_detected = 1;
+	}
 
-	delta = ktime_sub(now, entry_time);
-	if ((unsigned int)delta > cpu_base->max_hang_time)
-		cpu_base->max_hang_time = (unsigned int) delta;
-	/*
-	 * Limit it to a sensible value as we enforce a longer
-	 * delay. Give the CPU at least 100ms to catch up.
-	 */
-	if (delta > 100 * NSEC_PER_MSEC)
-		expires_next = ktime_add_ns(now, 100 * NSEC_PER_MSEC);
-	else
-		expires_next = ktime_add(now, delta);
-	tick_program_event(expires_next, 1);
-	pr_warn_once("hrtimer: interrupt took %llu ns\n", ktime_to_ns(delta));
+	__hrtimer_rearm(cpu_base, now);
+	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 }
 #endif /* !CONFIG_HIGH_RES_TIMERS */
 
author maintainer
#17 lore raw

RFC PATCH 7/8 entry,hrtimer: Push reprogramming timers into the interrupt return path

In reply to Peter Zijlstra

Currently hrtimer_interrupt() runs expired timers, which can re-arm themselves, after which it computes the next expiration time and re-programs the hardware.

However, things like HRTICK, a highres timer driving preemption, cannot re-arm itself at the point of running, since the next task has not been determined yet. The schedule() in the interrupt return path will switch to the next task, which then causes a new hrtimer to be programmed.

This then results in reprogramming the hardware at least twice, once after running the timers, and once upon selecting the new task.

Notably, *both* events happen in the interrupt.

By pushing the hrtimer reprogram all the way into the interrupt return path, it runs after schedule() and this double reprogram can be avoided.

XXX: 0-day is unhappy with this patch -- it is reporting lockups that very much look like a timer goes missing. Am unable to reproduce.
Notable: the lockup goes away when the workloads are ran without perf monitors.

---
 include/asm-generic/thread_info_tif.h |    5 ++++-
 include/linux/hrtimer.h               |   17 +++++++++++++++++
 kernel/entry/common.c                 |    7 +++++++
 kernel/sched/core.c                   |    6 ++++++
 kernel/time/hrtimer.c                 |   28 ++++++++++++++++++++++++----
 5 files changed, 58 insertions(+), 5 deletions(-)
include/asm-generic/thread_info_tif.h +4 -1
@@ -41,8 +41,11 @@
 #define _TIF_PATCH_PENDING	BIT(TIF_PATCH_PENDING)
 
 #ifdef HAVE_TIF_RESTORE_SIGMASK
-# define TIF_RESTORE_SIGMASK	10	// Restore signal mask in do_signal() */
+# define TIF_RESTORE_SIGMASK	10	// Restore signal mask in do_signal()
 # define _TIF_RESTORE_SIGMASK	BIT(TIF_RESTORE_SIGMASK)
 #endif
 
+#define TIF_HRTIMER_REARM              11       // re-arm the timer
+#define _TIF_HRTIMER_REARM             BIT(TIF_HRTIMER_REARM)
+
 #endif /* _ASM_GENERIC_THREAD_INFO_TIF_H_ */
include/linux/hrtimer.h +17 -0
@@ -175,10 +175,27 @@ extern void hrtimer_interrupt(struct clo
 
 extern unsigned int hrtimer_resolution;
 
+#ifdef TIF_HRTIMER_REARM
+extern void _hrtimer_rearm(void);
+/*
+ * This is to be called on all irqentry_exit() paths; as well as in the context
+ * switch path before switch_to().
+ */
+static inline void hrtimer_rearm(void)
+{
+	if (test_thread_flag(TIF_HRTIMER_REARM))
+		_hrtimer_rearm();
+}
+#else
+static inline void hrtimer_rearm(void) { }
+#endif /* TIF_HRTIMER_REARM */
+
 #else
 
 #define hrtimer_resolution	(unsigned int)LOW_RES_NSEC
 
+static inline void hrtimer_rearm(void) { }
+
 #endif
 
 static inline ktime_t
kernel/entry/common.c +7 -0
@@ -7,6 +7,7 @@
 #include <linux/kmsan.h>
 #include <linux/livepatch.h>
 #include <linux/tick.h>
+#include <linux/hrtimer.h>
 
 /* Workaround to allow gradual conversion of architecture code */
 void __weak arch_do_signal_or_restart(struct pt_regs *regs) { }
@@ -71,6 +72,7 @@ noinstr void irqentry_exit_to_user_mode(
 {
 	instrumentation_begin();
 	exit_to_user_mode_prepare(regs);
+	hrtimer_rearm();
 	instrumentation_end();
 	exit_to_user_mode();
 }
@@ -183,6 +185,7 @@ noinstr void irqentry_exit(struct pt_reg
 		 */
 		if (state.exit_rcu) {
 			instrumentation_begin();
+			hrtimer_rearm();
 			/* Tell the tracer that IRET will enable interrupts */
 			trace_hardirqs_on_prepare();
 			lockdep_hardirqs_on_prepare();
@@ -196,10 +199,14 @@ noinstr void irqentry_exit(struct pt_reg
 		if (IS_ENABLED(CONFIG_PREEMPTION))
 			irqentry_exit_cond_resched();
 
+		hrtimer_rearm();
 		/* Covers both tracing and lockdep */
 		trace_hardirqs_on();
 		instrumentation_end();
 	} else {
+		instrumentation_begin();
+		hrtimer_rearm();
+		instrumentation_end();
 		/*
 		 * IRQ flags state is correct already. Just tell RCU if it
 		 * was not watching on entry.
kernel/sched/core.c +6 -0
@@ -5161,6 +5161,12 @@ prepare_task_switch(struct rq *rq, struc
 	fire_sched_out_preempt_notifiers(prev, next);
 	kmap_local_sched_out();
 	prepare_task(next);
+	/*
+	 * Notably, this must be called after pick_next_task() but before
+	 * switch_to(), since the new task need not be on the return from
+	 * interrupt path.
+	 */
+	hrtimer_rearm();
 	prepare_arch_switch(next);
 }
 
kernel/time/hrtimer.c +24 -4
@@ -1892,10 +1892,9 @@ static __latent_entropy void hrtimer_run
  * Very similar to hrtimer_force_reprogram(), except it deals with
  * in_hrirq and hang_detected.
  */
-static void __hrtimer_rearm(struct hrtimer_cpu_base *cpu_base, ktime_t now)
+static void __hrtimer_rearm(struct hrtimer_cpu_base *cpu_base,
+			    ktime_t now, ktime_t expires_next)
 {
-	ktime_t expires_next = hrtimer_update_next_event(cpu_base);
-
 	cpu_base->expires_next = expires_next;
 	cpu_base->in_hrtirq = 0;
 
@@ -1970,9 +1969,30 @@ void hrtimer_interrupt(struct clock_even
 		cpu_base->hang_detected = 1;
 	}
 
-	__hrtimer_rearm(cpu_base, now);
+#ifdef TIF_HRTIMER_REARM
+	set_thread_flag(TIF_HRTIMER_REARM);
+#else
+	__hrtimer_rearm(cpu_base, now, expires_next);
+#endif
 	raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
 }
+
+#ifdef TIF_HRTIMER_REARM
+void _hrtimer_rearm(void)
+{
+	struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases);
+	ktime_t now, expires_next;
+
+	lockdep_assert_irqs_disabled();
+
+	scoped_guard (raw_spinlock, &cpu_base->lock) {
+		now = hrtimer_update_base(cpu_base);
+		expires_next = hrtimer_update_next_event(cpu_base);
+		__hrtimer_rearm(cpu_base, now, expires_next);
+		clear_thread_flag(TIF_HRTIMER_REARM);
+	}
+}
+#endif /* TIF_HRTIMER_REARM */
 #endif /* !CONFIG_HIGH_RES_TIMERS */
 
 /*
#18 lore raw

Re: [RFC][PATCH 7/8] entry,hrtimer: Push reprogramming timers into the interrupt return path

Quoted 15 lines · Peter Zijlstra

Currently hrtimer_interrupt() runs expired timers, which can re-arm themselves, after which it computes the next expiration time and re-programs the hardware.

However, things like HRTICK, a highres timer driving preemption, cannot re-arm itself at the point of running, since the next task has not been determined yet. The schedule() in the interrupt return path will switch to the next task, which then causes a new hrtimer to be programmed.

This then results in reprogramming the hardware at least twice, once after running the timers, and once upon selecting the new task.

Notably, *both* events happen in the interrupt.

Quoted 8 lines

By pushing the hrtimer reprogram all the way into the interrupt return path, it runs after schedule() and this double reprogram can be avoided.

XXX: 0-day is unhappy with this patch -- it is reporting lockups that very much look like a timer goes missing. Am unable to reproduce.
Notable: the lockup goes away when the workloads are ran without perf monitors.

After staring at it for a while, I have two observations.

1) In the 0-day report the lockup detector triggers on a spinlock contention in futex_wait_setup()

I'm not really seeing how that's related to a missing timer.

Without knowing what the other CPUs are doing and what holds the lock, it's pretty much impossible to tell what the hell is going on.

So that might need a back trace triggered on all CPUs and perhaps some debug output in the backtrace about the hrtimer state.

On the CPU where the lockup is detected, the timer is working.

2) I came up with the following scenario, which is broken with this delayed rearm.

Assume this happens on the timekeeping CPU.

hrtimer_interrupt()
expire_timers();
set(TIF_REARM);

exit_to_user_mode_prepare()
handle_tif_muck()
...
to = jiffies + 2;
while (!cond() && time_before(jiffies, to))
relax();

If cond() does not become true for whatever reason, then this won't make progress ever because the tick hrtimer which increments jiffies is not happening.

It can also be a wait on a remote CPU preventing progress indirectly or a subtle dependency on a timer (timer list or hrtimer) to expire.

I have no idea whether that's related to the reported 0-day fallout, but it definitely is a real problem lurking in the dark.

Thanks,

tglx

author maintainer
#19 lore raw

Re: [RFC][PATCH 7/8] entry,hrtimer: Push reprogramming timers into the interrupt return path

Quoted 32 lines · Thomas Gleixner
Quoted 15 lines · Peter Zijlstra

Currently hrtimer_interrupt() runs expired timers, which can re-arm themselves, after which it computes the next expiration time and re-programs the hardware.

However, things like HRTICK, a highres timer driving preemption, cannot re-arm itself at the point of running, since the next task has not been determined yet. The schedule() in the interrupt return path will switch to the next task, which then causes a new hrtimer to be programmed.

This then results in reprogramming the hardware at least twice, once after running the timers, and once upon selecting the new task.

Notably, *both* events happen in the interrupt.

Quoted 8 lines

By pushing the hrtimer reprogram all the way into the interrupt return path, it runs after schedule() and this double reprogram can be avoided.

XXX: 0-day is unhappy with this patch -- it is reporting lockups that very much look like a timer goes missing. Am unable to reproduce.
Notable: the lockup goes away when the workloads are ran without perf monitors.

After staring at it for a while, I have two observations.

1) In the 0-day report the lockup detector triggers on a spinlock contention in futex_wait_setup()

I'm not really seeing how that's related to a missing timer.

Quoted 7 lines

Without knowing what the other CPUs are doing and what holds the lock, it's pretty much impossible to tell what the hell is going on.

So that might need a back trace triggered on all CPUs and perhaps some debug output in the backtrace about the hrtimer state.

On the CPU where the lockup is detected, the timer is working.

Fair enough; I was thinking it got stuck on a missing timeout, but indeed, that needs verifying.

Quoted 20 lines

2) I came up with the following scenario, which is broken with this delayed rearm.

Assume this happens on the timekeeping CPU.

hrtimer_interrupt()
expire_timers();
set(TIF_REARM);

exit_to_user_mode_prepare()
handle_tif_muck()
...
to = jiffies + 2;
while (!cond() && time_before(jiffies, to))
relax();

If cond() does not become true for whatever reason, then this won't make progress ever because the tick hrtimer which increments jiffies is not happening.

Quoted 6 lines

It can also be a wait on a remote CPU preventing progress indirectly or a subtle dependency on a timer (timer list or hrtimer) to expire.

I have no idea whether that's related to the reported 0-day fallout, but it definitely is a real problem lurking in the dark.

Argh... that exit_to_user_mode_loop() thing enables IRQs. Yes, buggered something mighty.

Let me haz a poke.

author maintainer
#20 lore raw

Re: [RFC][PATCH 7/8] entry,hrtimer: Push reprogramming timers into the interrupt return path

Quoted 24 lines · Peter Zijlstra
Quoted 24 lines

2) I came up with the following scenario, which is broken with this delayed rearm.

Assume this happens on the timekeeping CPU.

hrtimer_interrupt()
expire_timers();
set(TIF_REARM);

exit_to_user_mode_prepare()
handle_tif_muck()
...
to = jiffies + 2;
while (!cond() && time_before(jiffies, to))
relax();

If cond() does not become true for whatever reason, then this won't make progress ever because the tick hrtimer which increments jiffies is not happening.

It can also be a wait on a remote CPU preventing progress indirectly or a subtle dependency on a timer (timer list or hrtimer) to expire.

Quoted 7 lines
Quoted 2 lines

I have no idea whether that's related to the reported 0-day fallout, but it definitely is a real problem lurking in the dark.

Argh... that exit_to_user_mode_loop() thing enables IRQs. Yes, buggered something mighty.

Let me haz a poke.

Bah. So schedule() is first in the TIF loop. Delaying hrtimer_rearm() until that first schedule() call might just be enough, but that also means running all of sched_submit_work() without timers... it might just work, but urgh.

Let me try that anyway. I'll push it out to the robot, we'll see what happens.

author maintainer
#21 lore raw

RFC PATCH 8/8 sched: Default enable HRTICK

In reply to Peter Zijlstra

For the robots.. let us find regressions.

---
 kernel/sched/features.h |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/features.h +2 -2
@@ -63,8 +63,8 @@ SCHED_FEAT(DELAY_ZERO, true)
  */
 SCHED_FEAT(WAKEUP_PREEMPTION, true)
 
-SCHED_FEAT(HRTICK, false)
-SCHED_FEAT(HRTICK_DL, false)
+SCHED_FEAT(HRTICK, true)
+SCHED_FEAT(HRTICK_DL, true)
 
 /*
  * Decrement CPU capacity based on time not spent running tasks