hrtimer/sched: Improve hrtick
Subsystems (4)
Peter Zijlstra
maintainercover hrtimer/sched: Improve hrtick
Hi!
A few patches trying to improve the sched hrtick feature, which is currently disabled due to overhead.
I wrote these after the last OSPM, but I've been sitting on them because 0-day is not having a good time with the second to last patch. I've not been able to reproduce :-(
Anyway, the first few patches should be 'good' and hopefully one of you will spot my 'obvious' fail in that late patch.
For those of you rocking ARM64 systems, if you pick up the generic entry patches:
https://lkml.kernel.org/r/[email protected]
this late patch should also 'work' (or not as the case might be) for you.
Patches also here:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/hrtick
Peter Zijlstra
maintainerPATCH 1/8 sched: Fix hrtick() vs scheduling context
The sched_class::task_tick() method is called on the donor sched_class, and sched_tick() hands it rq->donor as argument, which is consistent.
However, while hrtick() uses the donor sched_class, it then passes rq->curr, which is inconsistent. Fix it.
--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -875,7 +875,7 @@ static enum hrtimer_restart hrtick(strucrq_lock (rq, &rf);update_rq_clock (rq); - rq->donor->sched_class->task_tick (rq, rq->curr,1 ); + rq->donor->sched_class->task_tick (rq, rq->donor,1 );rq_unlock (rq, &rf);return HRTIMER_NORESTART ;
Re: [PATCH 1/8] sched: Fix hrtick() vs scheduling context
Hello Peter,
Quoted 6 lines · Peter Zijlstra
The
sched_class::task_tick()method is called on the donorsched_class, andsched_tick()hands itrq->donoras argument, which is consistent.However, while
hrtick()uses the donorsched_class, it then passesrq->curr, which is inconsistent. Fix it.
Can we add either a:
where this starts making a difference functionally since single CPU proxy can have rq->curr != rq->donor, or we can target the same commit where task_tick(0 in sched_tick() was updated with:
Other than that, this looks good to me. Feel free to include:
Quoted 20 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/core.c +1
-1@@ -875,7 +875,7 @@ static enum hrtimer_restart hrtick(strucrq_lock (rq, &rf);update_rq_clock (rq); - rq->donor->sched_class->task_tick (rq, rq->curr,1 ); + rq->donor->sched_class->task_tick (rq, rq->donor,1 );rq_unlock (rq, &rf);return HRTIMER_NORESTART ;
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 1/8] sched: Fix hrtick() vs scheduling context
In reply to Peter ZijlstraQuoted 16 lines · Peter Zijlstra
The
sched_class::task_tick()method is called on the donorsched_class, andsched_tick()hands itrq->donoras argument, which is consistent.However, while
hrtick()uses the donorsched_class, it then passesrq->curr, which is inconsistent. Fix it.Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c static enum hrtimer_restart hrtick(struc · line 879
Quoted 6 lines
kernel/sched/core.c +1
-1@@ -875,7 +875,7 @@ static enum hrtimer_restart hrtick(strucrq_lock (rq, &rf);update_rq_clock (rq); - rq->donor->sched_class->task_tick (rq, rq->curr,1 ); + rq->donor->sched_class->task_tick (rq, rq->donor,1 );rq_unlock (rq, &rf);
Ah. Thanks for catching this! I've run through with some stress testing on this and haven't seen any problems so far.
thanks
-john
[tip: sched/urgent] sched/hrtick: Fix hrtick() vs. scheduling context
In reply to Peter ZijlstraThe following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: 8720ba2d028f1aff08a55d8fe1a124dd5a6cfb0a Gitweb: https://git.kernel.org/tip/8720ba2d028f1aff08a55d8fe1a124dd5a6cfb0a Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 01 Sep 2025 22:46:29 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 02 Dec 2025 15:37:52 +01:00
sched/hrtick: Fix hrtick() vs. scheduling context
The sched_class::task_tick() method is called on the donor sched_class, and sched_tick() hands it rq->donor as argument, which is consistent.
However, while hrtick() uses the donor sched_class, it then passes rq->curr, which is inconsistent. Fix it.
--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -878,7 +878,7 @@ static enum hrtimer_restart hrtick(struct hrtimer *timer)rq_lock (rq, &rf);update_rq_clock (rq); - rq->donor->sched_class->task_tick (rq, rq->curr,1 ); + rq->donor->sched_class->task_tick (rq, rq->donor,1 );rq_unlock (rq, &rf);return HRTIMER_NORESTART ;
[tip: sched/urgent] sched/hrtick: Fix hrtick() vs. scheduling context
In reply to Peter ZijlstraThe following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: 40671f3f91986844df8947b65b9af5e770752047 Gitweb: https://git.kernel.org/tip/40671f3f91986844df8947b65b9af5e770752047 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 01 Sep 2025 22:46:29 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 03 Dec 2025 19:26:00 +01:00
sched/hrtick: Fix hrtick() vs. scheduling context
The sched_class::task_tick() method is called on the donor sched_class, and sched_tick() hands it rq->donor as argument, which is consistent.
However, while hrtick() uses the donor sched_class, it then passes rq->curr, which is inconsistent. Fix it.
--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -878,7 +878,7 @@ static enum hrtimer_restart hrtick(struct hrtimer *timer)rq_lock (rq, &rf);update_rq_clock (rq); - rq->donor->sched_class->task_tick (rq, rq->curr,1 ); + rq->donor->sched_class->task_tick (rq, rq->donor,1 );rq_unlock (rq, &rf);return HRTIMER_NORESTART ;
[tip: sched/urgent] sched/hrtick: Fix hrtick() vs. scheduling context
In reply to Peter ZijlstraThe following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: e38e5299747b23015b00b0109891815db44a2f30 Gitweb: https://git.kernel.org/tip/e38e5299747b23015b00b0109891815db44a2f30 Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 01 Sep 2025 22:46:29 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 06 Dec 2025 10:03:13 +01:00
sched/hrtick: Fix hrtick() vs. scheduling context
The sched_class::task_tick() method is called on the donor sched_class, and sched_tick() hands it rq->donor as argument, which is consistent.
However, while hrtick() uses the donor sched_class, it then passes rq->curr, which is inconsistent. Fix it.
--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -878,7 +878,7 @@ static enum hrtimer_restart hrtick(struct hrtimer *timer)rq_lock (rq, &rf);update_rq_clock (rq); - rq->donor->sched_class->task_tick (rq, rq->curr,1 ); + rq->donor->sched_class->task_tick (rq, rq->donor,1 );rq_unlock (rq, &rf);return HRTIMER_NORESTART ;
Peter Zijlstra
maintainerPATCH 2/8 sched/fair: Limit hrtick work
In reply to Peter ZijlstraThe task_tick_fair() function does:
- update the hierarchical runtimes
- drive numa-balancing
- update load-balance statistics
- drive force-idle preemption
All but the very first can be limited to the periodic tick. Let hrtick only update accounting and drive preemption, not load-balancing and other bits.
--- kernel/sched/fair.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/sched/fair.c +6 -0
@@ -13119,6 +13119,12 @@ static void task_tick_fair(struct rq *rqentity_tick (cfs_rq, se, queued); } +if (queued) { +if (!need_resched ()) +hrtick_start_fair (rq, curr); +return ; + } +if (static_branch_unlikely (&sched_numa_balancing))task_tick_numa (rq, curr);
K Prateek Nayak
reviewerRe: [PATCH 2/8] sched/fair: Limit hrtick work
Hello Peter,
kernel/sched/fair.c static void task_tick_fair(struct rq *rq · line 13124
Quoted 7 lines · Peter Zijlstra
kernel/sched/fair.c +3
-0@@ -13119,6 +13119,12 @@ static void task_tick_fair(struct rq *rqentity_tick (cfs_rq, se, queued); } +if (queued) { +if (!need_resched ()) +hrtick_start_fair (rq, curr);
Do we need a hrtick_start_fair() here? Queued tick will always do a resched_curr_lazy() - if another HRTICK fires before the next tick, all it'll do is resched_curr_lazy() again and the next opportunity to resched is either exit to userspace or the periodic tick firing and promoting that LAZY to a full NEED_RESCHED.
The early return does makes sense.
Quoted 6 lines
kernel/sched/fair.c +3
-0+return ; + } +if (static_branch_unlikely (&sched_numa_balancing))task_tick_numa (rq, curr);
Signature · 2 lines
Thanks and Regards, Prateek
Peter Zijlstra
maintainerRe: [PATCH 2/8] sched/fair: Limit hrtick work
Quoted 7 lines · K Prateek Nayak
Hello Peter,
Quoted 4 lines · Peter Zijlstra
kernel/sched/fair.c +0
-0@@ -13119,6 +13119,12 @@ static void task_tick_fair(struct rq *rqentity_tick (cfs_rq, se, queued); }
Quoted 9 lines
kernel/sched/fair.c static void task_tick_fair(struct rq *rq · line 13124
Quoted 3 lines
kernel/sched/fair.c +3
-0+if (queued) { +if (!need_resched ()) +hrtick_start_fair (rq, curr);Do we need a
hrtick_start_fair()here? Queued tick will always do aresched_curr_lazy()- if another HRTICK fires before the next tick, all it'll do isresched_curr_lazy()again and the next opportunity to resched is either exit to userspace or the periodic tick firing and promoting that LAZY to a fullNEED_RESCHED.
I think I had a version where entity_tick() doesn't force need_resched on queue, and in that case the timer, which is wallclock, and update_curr(), which is task_clock, might disagree and we might not have reached the deadline, and so we need to try again.
[tip: sched/core] sched/fair: Limit hrtick work
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 95a0155224a658965f34ed4b1943b238d9be1fea Gitweb: https://git.kernel.org/tip/95a0155224a658965f34ed4b1943b238d9be1fea Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 01 Sep 2025 22:50:56 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 14 Dec 2025 08:25:02 +01:00
sched/fair: Limit hrtick work
The task_tick_fair() function does:
- update the hierarchical runtimes
- drive NUMA-balancing
- update load-balance statistics
- drive force-idle preemption
All but the very first can be limited to the periodic tick. Let hrtick only update accounting and drive preemption, not load-balancing and other bits.
--- kernel/sched/fair.c | 6 ++++++ 1 file changed, 6 insertions(+)
kernel/sched/fair.c +6 -0
@@ -13332,6 +13332,12 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)entity_tick (cfs_rq, se, queued); } +if (queued) { +if (!need_resched ()) +hrtick_start_fair (rq, curr); +return ; + } +if (static_branch_unlikely (&sched_numa_balancing))task_tick_numa (rq, curr);
Peter Zijlstra
maintainerPATCH 3/8 sched/eevdf: Fix HRTICK duration
In reply to Peter ZijlstraThe nominal duration for an EEVDF task to run is until its deadline. At which point the deadline is moved ahead and a new task selection is done.
--- kernel/sched/fair.c | 40 +++++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 11 deletions(-)
kernel/sched/fair.c +29 -11
@@ -6775,21 +6775,39 @@ static inline void sched_fair_update_stostatic void hrtick_start_fair (struct rq *rq,struct task_struct *p) {struct sched_entity *se = &p->se; +unsigned long scale =1024 ; +unsigned long util =0 ; +u64 vdelta; +u64 delta;WARN_ON_ONCE (task_rq (p) != rq); -if (rq->cfs.h_nr_queued >1 ) { -u64 ran = se->sum_exec_runtime - se->prev_sum_exec_runtime; -u64 slice = se->slice; -s64 delta = slice - ran; - -if (delta <0 ) { -if (task_current_donor (rq, p)) -resched_curr (rq); -return ; - } -hrtick_start (rq, delta); +if (rq->cfs.h_nr_queued <=1 ) +return ; + +/* +* Compute time until virtual deadline +*/ + vdelta = se->deadline - se->vruntime; +if ((s64 )vdelta <0 ) { +if (task_current_donor (rq, p)) +resched_curr (rq); +return ; + } + delta = (se->load.weight * vdelta) /NICE_0_LOAD ; + +/* +* Correct for instantaneous load of other classes. +*/ + util +=cpu_util_dl (rq); + util +=cpu_util_rt (rq); + util +=cpu_util_irq (rq); +if (util && util <1024 ) { + scale *=1024 ; + scale /= (1024 - util); } + +hrtick_start (rq, (scale * delta) /1024 ); }/*
K Prateek Nayak
reviewerRe: [PATCH 3/8] sched/eevdf: Fix HRTICK duration
Hello Peter,
kernel/sched/fair.c static inline void sched_fair_update_sto · line 6794
Quoted 7 lines · Peter Zijlstra
kernel/sched/fair.c +7
-0+/* +* Compute time until virtual deadline +*/ + vdelta = se->deadline - se->vruntime; +if ((s64 )vdelta <0 ) { +if (task_current_donor (rq, p)) +resched_curr (rq);
Why the task_current_donor() check? If the scheduling context has run out of gas, shouldn't we reschedule curr even if we were proxied?
Quoted 4 lines
kernel/sched/fair.c +4
-0+return ; + } + delta = (se->load.weight * vdelta) /NICE_0_LOAD ; +
kernel/sched/fair.c static inline void sched_fair_update_sto · line 6808
Quoted 10 lines
kernel/sched/fair.c +9
-0+/* +* Correct for instantaneous load of other classes. +*/ + util +=cpu_util_dl (rq); + util +=cpu_util_rt (rq); + util +=cpu_util_irq (rq); +if (util && util <1024 ) { + scale *=1024 ; + scale /= (1024 - util); }
Could it be possible that we arrive here from the dl_server's pick and end up inflating the HRTICK duration despite having an uninterrupted period for fair tasks ahead?
Quoted 5 lines
kernel/sched/fair.c +2
-0+ +hrtick_start (rq, (scale * delta) /1024 ); }/*
Signature · 2 lines
Thanks and Regards, Prateek
Peter Zijlstra
maintainerRe: [PATCH 3/8] sched/eevdf: Fix HRTICK duration
Quoted 13 lines · K Prateek Nayak
Hello Peter,
kernel/sched/fair.c static inline void sched_fair_update_sto · line 6794
Quoted 7 lines · Peter Zijlstra
kernel/sched/fair.c +7
-0+/* +* Compute time until virtual deadline +*/ + vdelta = se->deadline - se->vruntime; +if ((s64 )vdelta <0 ) { +if (task_current_donor (rq, p)) +resched_curr (rq);Why the
task_current_donor()check? If the scheduling context has run out of gas, shouldn't we reschedule curr even if we were proxied?
task_current_donor() is the current scheduling context, right? So we've just determined that vruntime is ahead of deadline, which means we should reschedule now.
Quoted 10 lines
Quoted 10 lines
kernel/sched/fair.c +10
-0+return ; + } + delta = (se->load.weight * vdelta) /NICE_0_LOAD ; + +/* +* Correct for instantaneous load of other classes. +*/ + util +=cpu_util_dl (rq); + util +=cpu_util_rt (rq); + util +=cpu_util_irq (rq);
Quoted 8 lines
kernel/sched/fair.c static inline void sched_fair_update_sto · line 6808
Quoted 4 lines
kernel/sched/fair.c +3
-0+if (util && util <1024 ) { + scale *=1024 ; + scale /= (1024 - util); }Could it be possible that we arrive here from the dl_server's pick and end up inflating the HRTICK duration despite having an uninterrupted period for fair tasks ahead?
Yes, but since this is all approximation anyway, how many correction terms do we want to stack on top ? :-)
Quoted 8 lines
Quoted 4 lines
kernel/sched/fair.c +1
-0+hrtick_start (rq, (scale * delta) /1024 ); }/* Signature · 2 lines
Thanks and Regards, Prateek
Peter Zijlstra
maintainerPATCH 4/8 hrtimer: Optimize __hrtimer_start_range_ns()
In reply to Peter ZijlstraMuch like hrtimer_reprogram(), skip programming is the cpu_base is running the hrtimer interrupt.
--- kernel/time/hrtimer.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/time/hrtimer.c +8 -0
@@ -1261,6 +1261,14 @@ static int __hrtimer_start_range_ns(stru } first =enqueue_hrtimer (timer, new_base, mode); + +/* +* If the hrtimer interrupt is running, then it will reevaluate the +* clock bases and reprogram the clock event device. +*/ +if (new_base->cpu_base->in_hrtirq) +return 0 ; +if (!force_local) {/* * If the current CPU base is online, then the timer is
Peter Zijlstra
maintainerPATCH 5/8 hrtimer,sched: Add fuzzy hrtimer mode for HRTICK
In reply to Peter ZijlstraUpon schedule() HRTICK will cancel the current timer, pick the next task and reprogram the timer. When schedule() consistently triggers due to blocking conditions instead of the timer, this leads to endless reprogramming without ever firing.
Mitigate this with a new hrtimer mode: fuzzy (not really happy with that name); this mode does two things:
- skip reprogramming the hardware on timer remove;
- skip reprogramming the hardware when the new timer is after cpu_base->expires_next
Both things are already possible;
- removing a remote timer will leave the hardware programmed and cause a spurious interrupt.
- this remote CPU adding a timer can skip the reprogramming when the timer's expiration is after the (spurious) expiration.
This new timer mode simply causes more of this 'fuzzy' behaviour; it causes a few spurious interrupts, but similarly avoids endlessly reprogramming the timer.
This makes the HRTICK match the NO_HRTICK hackbench runs.
--- include/linux/hrtimer.h | 1 + include/linux/hrtimer_types.h | 1 + kernel/sched/core.c | 3 ++- kernel/time/hrtimer.c | 16 +++++++++++++++- 4 files changed, 19 insertions(+), 2 deletions(-)
include/linux/hrtimer.h +1 -0
@@ -38,6 +38,7 @@ enum hrtimer_mode {HRTIMER_MODE_PINNED =0x02 ,HRTIMER_MODE_SOFT =0x04 ,HRTIMER_MODE_HARD =0x08 , +HRTIMER_MODE_FUZZY =0x10 ,HRTIMER_MODE_ABS_PINNED =HRTIMER_MODE_ABS |HRTIMER_MODE_PINNED , HRTIMER_MODE_REL_PINNED = HRTIMER_MODE_REL | HRTIMER_MODE_PINNED,
include/linux/hrtimer_types.h +1 -0
@@ -45,6 +45,7 @@ struct hrtimer {u8 is_rel;u8 is_soft;u8 is_hard; + u8 is_fuzzy; };#endif /* _LINUX_HRTIMER_TYPES_H */
kernel/sched/core.c +2 -1
@@ -928,7 +928,8 @@ void hrtick_start(struct rq *rq, u64 delstatic void hrtick_rq_init (struct rq *rq) {INIT_CSD (&rq->hrtick_csd, __hrtick_start, rq); -hrtimer_setup (&rq->hrtick_timer, hrtick,CLOCK_MONOTONIC ,HRTIMER_MODE_REL_HARD ); +hrtimer_setup (&rq->hrtick_timer, hrtick,CLOCK_MONOTONIC , +HRTIMER_MODE_REL_HARD |HRTIMER_MODE_FUZZY ); }#else /* !CONFIG_SCHED_HRTICK: */ static inline void hrtick_clear (struct rq *rq)
kernel/time/hrtimer.c +15 -1
@@ -1122,7 +1122,7 @@ static void __remove_hrtimer(struct hrti* an superfluous call to hrtimer_force_reprogram() on the * remote cpu later on if the same timer gets enqueued again. */ -if (reprogram && timer == cpu_base->next_timer) +if (!timer->is_fuzzy && reprogram && timer == cpu_base->next_timer)hrtimer_force_reprogram (cpu_base,1 ); }
@@ -1269,6 +1269,19 @@ static int __hrtimer_start_range_ns(struif (new_base->cpu_base->in_hrtirq)return 0 ; +if (timer->is_fuzzy) { +/* +* XXX fuzzy implies pinned! not sure how to deal with +* retrigger_next_event() for the !local case. +*/ +WARN_ON_ONCE (!(mode &HRTIMER_MODE_PINNED )); +/* +* Notably, by going into hrtimer_reprogram(), it will +* not reprogram if cpu_base->expires_next is earlier. +*/ +return first; + } +if (!force_local) {/* * If the current CPU base is online, then the timer is
@@ -1645,6 +1658,7 @@ static void __hrtimer_setup(struct hrtim base +=hrtimer_clockid_to_base (clock_id); timer->is_soft = softtimer; timer->is_hard = !!(mode &HRTIMER_MODE_HARD ); + timer->is_fuzzy = !!(mode &HRTIMER_MODE_FUZZY ); timer->base = &cpu_base->clock_base[base];timerqueue_init (&timer->node);
Peter Zijlstra
maintainerPATCH 6/8 hrtimer: Re-arrange hrtimer_interrupt()
In reply to Peter ZijlstraRework hrtimer_interrupt() such that reprogramming is split out into an independent function at the end of the interrupt.
This prepares for reprogramming getting delayed beyond the end of hrtimer_interrupt().
Notably, this changes the hang handling to always wait 100ms instead of trying to keep it proportional to the actual delay. This simplifies the state, also this really shouldn't be happening.
--- kernel/time/hrtimer.c | 87 ++++++++++++++++++++++---------------------------- 1 file changed, 39 insertions(+), 48 deletions(-)
kernel/time/hrtimer.c +39 -48
@@ -1889,6 +1889,29 @@ static __latent_entropy void hrtimer_run#ifdef CONFIG_HIGH_RES_TIMERS /* +* Very similar to hrtimer_force_reprogram(), except it deals with +* in_hrirq and hang_detected. +*/ +static void __hrtimer_rearm (struct hrtimer_cpu_base *cpu_base,ktime_t now) +{ +ktime_t expires_next =hrtimer_update_next_event (cpu_base); + + cpu_base->expires_next = expires_next; + cpu_base->in_hrtirq =0 ; + +if (unlikely (cpu_base->hang_detected)) { +/* +* Give the system a chance to do something else than looping +* on hrtimer interrupts. +*/ + expires_next =ktime_add_ns (now,100 *NSEC_PER_MSEC ); + cpu_base->hang_detected =0 ; + } + +tick_program_event (expires_next,1 ); +} + +/* * High resolution timer interrupt * Called with interrupts disabled */
@@ -1924,63 +1947,31 @@ void hrtimer_interrupt(struct clock_even__hrtimer_run_queues (cpu_base, now, flags,HRTIMER_ACTIVE_HARD ); -/* Reevaluate the clock bases for the [soft] next expiry */ - expires_next =hrtimer_update_next_event (cpu_base); -/* -* Store the new expiry value so the migration code can verify -* against it. -*/ - cpu_base->expires_next = expires_next; - cpu_base->in_hrtirq =0 ; -raw_spin_unlock_irqrestore (&cpu_base->lock, flags); - -/* Reprogramming necessary ? */ -if (!tick_program_event (expires_next,0 )) { - cpu_base->hang_detected =0 ; -return ; - } -/* * The next timer was already expired due to: * - tracing * - long lasting callbacks * - being scheduled away when running in a VM * -* We need to prevent that we loop forever in the hrtimer -* interrupt routine. We give it 3 attempts to avoid -* overreacting on some spurious event. -* -* Acquire base lock for updating the offsets and retrieving -* the current time. +* We need to prevent that we loop forever in the hrtiner interrupt +* routine. We give it 3 attempts to avoid overreacting on some +* spurious event. */ -raw_spin_lock_irqsave (&cpu_base->lock, flags); + expires_next =hrtimer_update_next_event (cpu_base); now =hrtimer_update_base (cpu_base); - cpu_base->nr_retries++; -if (++retries <3 ) - goto retry; -/* -* Give the system a chance to do something else than looping -* here. We stored the entry time, so we know exactly how long -* we spent here. We schedule the next event this amount of -* time away. -*/ - cpu_base->nr_hangs++; - cpu_base->hang_detected =1 ; -raw_spin_unlock_irqrestore (&cpu_base->lock, flags); +if (expires_next < now) { +if (++retries <3 ) + goto retry; + + delta =ktime_sub (now, entry_time); + cpu_base->max_hang_time = max_t(unsigned int, + cpu_base->max_hang_time, delta); + cpu_base->nr_hangs++; + cpu_base->hang_detected =1 ; + } - delta =ktime_sub (now, entry_time); -if ((unsigned int )delta > cpu_base->max_hang_time) - cpu_base->max_hang_time = (unsigned int ) delta; -/* -* Limit it to a sensible value as we enforce a longer -* delay. Give the CPU at least 100ms to catch up. -*/ -if (delta >100 *NSEC_PER_MSEC ) - expires_next =ktime_add_ns (now,100 *NSEC_PER_MSEC ); -else - expires_next =ktime_add (now, delta); -tick_program_event (expires_next,1 ); - pr_warn_once("hrtimer: interrupt took %llu ns\n", ktime_to_ns(delta)); +__hrtimer_rearm (cpu_base, now); + raw_spin_unlock_irqrestore(&cpu_base->lock, flags); }#endif /* !CONFIG_HIGH_RES_TIMERS */
Peter Zijlstra
maintainerRFC PATCH 7/8 entry,hrtimer: Push reprogramming timers into the interrupt return path
In reply to Peter ZijlstraCurrently hrtimer_interrupt() runs expired timers, which can re-arm themselves, after which it computes the next expiration time and re-programs the hardware.
However, things like HRTICK, a highres timer driving preemption, cannot re-arm itself at the point of running, since the next task has not been determined yet. The schedule() in the interrupt return path will switch to the next task, which then causes a new hrtimer to be programmed.
This then results in reprogramming the hardware at least twice, once after running the timers, and once upon selecting the new task.
Notably, *both* events happen in the interrupt.
By pushing the hrtimer reprogram all the way into the interrupt return path, it runs after schedule() and this double reprogram can be avoided.
XXX: 0-day is unhappy with this patch -- it is reporting lockups that very much look like a timer goes missing. Am unable to reproduce.
Notable: the lockup goes away when the workloads are ran without perf monitors.
--- include/asm-generic/thread_info_tif.h | 5 ++++- include/linux/hrtimer.h | 17 +++++++++++++++++ kernel/entry/common.c | 7 +++++++ kernel/sched/core.c | 6 ++++++ kernel/time/hrtimer.c | 28 ++++++++++++++++++++++++---- 5 files changed, 58 insertions(+), 5 deletions(-)
include/asm-generic/thread_info_tif.h +4 -1
@@ -41,8 +41,11 @@#define _TIF_PATCH_PENDING BIT(TIF_PATCH_PENDING)#ifdef HAVE_TIF_RESTORE_SIGMASK -# define TIF_RESTORE_SIGMASK 10 // Restore signal mask in do_signal() */ +# define TIF_RESTORE_SIGMASK 10 // Restore signal mask in do_signal()# define _TIF_RESTORE_SIGMASK BIT(TIF_RESTORE_SIGMASK)#endif +#define TIF_HRTIMER_REARM 11 // re-arm the timer +#define _TIF_HRTIMER_REARM BIT(TIF_HRTIMER_REARM) +#endif /* _ASM_GENERIC_THREAD_INFO_TIF_H_ */
include/linux/hrtimer.h +17 -0
@@ -175,10 +175,27 @@ extern void hrtimer_interrupt(struct cloextern unsigned int hrtimer_resolution; +#ifdef TIF_HRTIMER_REARM +extern void _hrtimer_rearm (void ); +/* +* This is to be called on all irqentry_exit() paths; as well as in the context +* switch path before switch_to(). +*/ +static inline void hrtimer_rearm (void ) +{ +if (test_thread_flag (TIF_HRTIMER_REARM )) +_hrtimer_rearm (); +} +#else +static inline void hrtimer_rearm (void ) { } +#endif /* TIF_HRTIMER_REARM */ +#else #define hrtimer_resolution (unsigned int)LOW_RES_NSEC +static inline void hrtimer_rearm (void ) { } +#endif static inline ktime_t
kernel/entry/common.c +7 -0
@@ -7,6 +7,7 @@#include <linux/kmsan.h> #include <linux/livepatch.h> #include <linux/tick.h> +#include <linux/hrtimer.h> /* Workaround to allow gradual conversion of architecture code */ void __weak arch_do_signal_or_restart (struct pt_regs *regs) { }
@@ -71,6 +72,7 @@ noinstr void irqentry_exit_to_user_mode( {instrumentation_begin ();exit_to_user_mode_prepare (regs); +hrtimer_rearm ();instrumentation_end ();exit_to_user_mode (); }
@@ -183,6 +185,7 @@ noinstr void irqentry_exit(struct pt_reg*/ if (state.exit_rcu) {instrumentation_begin (); +hrtimer_rearm ();/* Tell the tracer that IRET will enable interrupts */ trace_hardirqs_on_prepare ();lockdep_hardirqs_on_prepare ();
@@ -196,10 +199,14 @@ noinstr void irqentry_exit(struct pt_regif (IS_ENABLED (CONFIG_PREEMPTION ))irqentry_exit_cond_resched (); +hrtimer_rearm ();/* Covers both tracing and lockdep */ trace_hardirqs_on ();instrumentation_end (); }else { +instrumentation_begin (); +hrtimer_rearm (); +instrumentation_end ();/* * IRQ flags state is correct already. Just tell RCU if it * was not watching on entry.
kernel/sched/core.c +6 -0
@@ -5161,6 +5161,12 @@ prepare_task_switch(struct rq *rq, strucfire_sched_out_preempt_notifiers (prev, next);kmap_local_sched_out ();prepare_task (next); +/* +* Notably, this must be called after pick_next_task() but before +* switch_to(), since the new task need not be on the return from +* interrupt path. +*/ +hrtimer_rearm ();prepare_arch_switch (next); }
kernel/time/hrtimer.c +24 -4
@@ -1892,10 +1892,9 @@ static __latent_entropy void hrtimer_run* Very similar to hrtimer_force_reprogram(), except it deals with * in_hrirq and hang_detected. */ -static void __hrtimer_rearm (struct hrtimer_cpu_base *cpu_base,ktime_t now) +static void __hrtimer_rearm (struct hrtimer_cpu_base *cpu_base, +ktime_t now,ktime_t expires_next) { -ktime_t expires_next =hrtimer_update_next_event (cpu_base); - cpu_base->expires_next = expires_next; cpu_base->in_hrtirq =0 ;
@@ -1970,9 +1969,30 @@ void hrtimer_interrupt(struct clock_even cpu_base->hang_detected = 1; } -__hrtimer_rearm (cpu_base, now); +#ifdef TIF_HRTIMER_REARM +set_thread_flag (TIF_HRTIMER_REARM ); +#else +__hrtimer_rearm (cpu_base, now, expires_next); +#endif raw_spin_unlock_irqrestore (&cpu_base->lock, flags); } + +#ifdef TIF_HRTIMER_REARM +void _hrtimer_rearm (void ) +{ +struct hrtimer_cpu_base *cpu_base =this_cpu_ptr (&hrtimer_bases); +ktime_t now, expires_next; + +lockdep_assert_irqs_disabled (); + + scoped_guard (raw_spinlock, &cpu_base->lock) { + now =hrtimer_update_base (cpu_base); + expires_next =hrtimer_update_next_event (cpu_base); +__hrtimer_rearm (cpu_base, now, expires_next); +clear_thread_flag (TIF_HRTIMER_REARM ); + } +} +#endif /* TIF_HRTIMER_REARM */ #endif /* !CONFIG_HIGH_RES_TIMERS */ /*
Re: [RFC][PATCH 7/8] entry,hrtimer: Push reprogramming timers into the interrupt return path
Quoted 15 lines · Peter Zijlstra
Currently
hrtimer_interrupt()runs expired timers, which can re-arm themselves, after which it computes the next expiration time and re-programs the hardware.However, things like HRTICK, a highres timer driving preemption, cannot re-arm itself at the point of running, since the next task has not been determined yet. The
schedule()in the interrupt return path will switch to the next task, which then causes a new hrtimer to be programmed.This then results in reprogramming the hardware at least twice, once after running the timers, and once upon selecting the new task.
Notably, *both* events happen in the interrupt.
Quoted 8 lines
By pushing the hrtimer reprogram all the way into the interrupt return path, it runs after
schedule()and this double reprogram can be avoided.XXX: 0-day is unhappy with this patch -- it is reporting lockups that very much look like a timer goes missing. Am unable to reproduce.
Notable: the lockup goes away when the workloads are ran without perf monitors.
After staring at it for a while, I have two observations.
1) In the 0-day report the lockup detector triggers on a spinlock contention in futex_wait_setup()
I'm not really seeing how that's related to a missing timer.
Without knowing what the other CPUs are doing and what holds the lock, it's pretty much impossible to tell what the hell is going on.
So that might need a back trace triggered on all CPUs and perhaps some debug output in the backtrace about the hrtimer state.
On the CPU where the lockup is detected, the timer is working.
2) I came up with the following scenario, which is broken with this delayed rearm.
Assume this happens on the timekeeping CPU.
hrtimer_interrupt()
expire_timers();
set(TIF_REARM);
exit_to_user_mode_prepare()
handle_tif_muck()
...
to = jiffies + 2;
while (!cond() && time_before(jiffies, to))
relax();
If cond() does not become true for whatever reason, then this won't make progress ever because the tick hrtimer which increments jiffies is not happening.
It can also be a wait on a remote CPU preventing progress indirectly or a subtle dependency on a timer (timer list or hrtimer) to expire.
I have no idea whether that's related to the reported 0-day fallout, but it definitely is a real problem lurking in the dark.
Thanks,
tglx
Peter Zijlstra
maintainerRe: [RFC][PATCH 7/8] entry,hrtimer: Push reprogramming timers into the interrupt return path
Quoted 32 lines · Thomas Gleixner
Quoted 15 lines · Peter Zijlstra
Currently
hrtimer_interrupt()runs expired timers, which can re-arm themselves, after which it computes the next expiration time and re-programs the hardware.However, things like HRTICK, a highres timer driving preemption, cannot re-arm itself at the point of running, since the next task has not been determined yet. The
schedule()in the interrupt return path will switch to the next task, which then causes a new hrtimer to be programmed.This then results in reprogramming the hardware at least twice, once after running the timers, and once upon selecting the new task.
Notably, *both* events happen in the interrupt.
Quoted 8 lines
By pushing the hrtimer reprogram all the way into the interrupt return path, it runs after
schedule()and this double reprogram can be avoided.XXX: 0-day is unhappy with this patch -- it is reporting lockups that very much look like a timer goes missing. Am unable to reproduce.
Notable: the lockup goes away when the workloads are ran without perf monitors.After staring at it for a while, I have two observations.
1) In the 0-day report the lockup detector triggers on a spinlock contention in
futex_wait_setup()I'm not really seeing how that's related to a missing timer.
Quoted 7 lines
Without knowing what the other CPUs are doing and what holds the lock, it's pretty much impossible to tell what the hell is going on.
So that might need a back trace triggered on all CPUs and perhaps some debug output in the backtrace about the hrtimer state.
On the CPU where the lockup is detected, the timer is working.
Fair enough; I was thinking it got stuck on a missing timeout, but indeed, that needs verifying.
Quoted 20 lines
2) I came up with the following scenario, which is broken with this delayed rearm.
Assume this happens on the timekeeping CPU.
hrtimer_interrupt()
expire_timers();
set(TIF_REARM);
exit_to_user_mode_prepare()
handle_tif_muck()
...
to = jiffies + 2;
while (!cond() &&time_before(jiffies, to))
relax();If
cond()does not become true for whatever reason, then this won't make progress ever because the tick hrtimer which increments jiffies is not happening.
Quoted 6 lines
It can also be a wait on a remote CPU preventing progress indirectly or a subtle dependency on a timer (timer list or hrtimer) to expire.
I have no idea whether that's related to the reported 0-day fallout, but it definitely is a real problem lurking in the dark.
Argh... that exit_to_user_mode_loop() thing enables IRQs. Yes, buggered something mighty.
Let me haz a poke.
Peter Zijlstra
maintainerRe: [RFC][PATCH 7/8] entry,hrtimer: Push reprogramming timers into the interrupt return path
Quoted 24 lines · Peter Zijlstra
Quoted 24 lines
2) I came up with the following scenario, which is broken with this delayed rearm.
Assume this happens on the timekeeping CPU.
hrtimer_interrupt()
expire_timers();
set(TIF_REARM);
exit_to_user_mode_prepare()
handle_tif_muck()
...
to = jiffies + 2;
while (!cond() &&time_before(jiffies, to))
relax();If
cond()does not become true for whatever reason, then this won't make progress ever because the tick hrtimer which increments jiffies is not happening.It can also be a wait on a remote CPU preventing progress indirectly or a subtle dependency on a timer (timer list or hrtimer) to expire.
Quoted 7 lines
Quoted 2 lines
I have no idea whether that's related to the reported 0-day fallout, but it definitely is a real problem lurking in the dark.
Argh... that
exit_to_user_mode_loop()thing enables IRQs. Yes, buggered something mighty.Let me haz a poke.
Bah. So schedule() is first in the TIF loop. Delaying hrtimer_rearm() until that first schedule() call might just be enough, but that also means running all of sched_submit_work() without timers... it might just work, but urgh.
Let me try that anyway. I'll push it out to the robot, we'll see what happens.
Peter Zijlstra
maintainerRFC PATCH 8/8 sched: Default enable HRTICK
In reply to Peter ZijlstraFor the robots.. let us find regressions.
--- kernel/sched/features.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/features.h +2 -2
@@ -63,8 +63,8 @@ SCHED_FEAT(DELAY_ZERO, true)*/ SCHED_FEAT(WAKEUP_PREEMPTION, true) -SCHED_FEAT(HRTICK, false) -SCHED_FEAT (HRTICK_DL , false) +SCHED_FEAT(HRTICK, true) +SCHED_FEAT (HRTICK_DL , true)/* * Decrement CPU capacity based on time not spent running tasks