sched/psi: skip irqtime accounting when no new irq time has elapsed
Subsystems (3)
cover sched/psi: skip irqtime accounting when no new irq time has elapsed
psi_account_irqtime() reads irq_time_read() into a per-rq cumulative counter and only bails out when the delta vs. the previously accounted amount is negative. A delta of exactly zero is treated as "do the work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor chain is walked to add zero to every group's PSI_IRQ_FULL bucket.
The zero-delta case is common in practice -- it fires every time a context switch crosses a PSI group boundary on a CPU that hasn't serviced an interrupt between the two switches.
To find out how often this actually fires in the wild, I attached a bpftrace probe to psi_account_irqtime() on a 176-thread AMD EPYC 9D64 server running an compute intensive workload.
The probe also reads irq_time_read(cpu) directly from the per-CPU cpu_irqtime variable so it can separate delta == 0 from delta < 0. The bpftrace script was generated by claude and is at the end of the coverletter.
Over a 30 s window under steady-state load:
@total 17,229,311 (100.0%)
@ret_curr_swapper 7,864,195 ( 45.6%) curr->pid == 0
@ret_samegrp 323,299 ( 1.9%) same cgroup as prev
@reached_delta 9,041,817 ( 52.5%)
@delta_positive 6,358,192 ( 36.9%) real work
@delta_zero 2,683,625 ( 15.6%) work wasted (this patch)
@delta_negative (0) ( 0.0%) monotonic clock15.6 % of all psi_account_irqtime() calls -- and 29.7 % of the calls that get past the early returns -- hit the delta == 0 case. delta < 0 did not occur once in the 30 s window. That works out to ~89 k calls/sec on this host that today take the full seqcount write + cpu_clock() + cgroup-chain walk purely to add 0 to every group's PSI_IRQ_FULL counter.
Extend the early-return to also cover delta == 0. rq->psi_irq_time does not need updating in that case (it would store the same value back) and no PSI bucket would change. The existing behaviour for delta > 0 is untouched.
--------- psi_delta_exact.bt -------
#!/usr/bin/env bpftrace
#include <linux/sched.h>
kprobe:psi_account_irqtime
{
$rq = (struct rq *)arg0;
$curr = (struct task_struct *)arg1;
$prev = (struct task_struct *)arg2;
@total = count();
if ($curr->pid == 0) {
@ret_curr_swapper = count();
return;
}
if ($prev != 0) {
$pg_curr = $curr->cgroups->dfl_cgrp;
$pg_prev = $prev->cgroups->dfl_cgrp;
if ($pg_curr == $pg_prev) {
@ret_samegrp = count();
return;
}
}
@reached_delta = count();
$pcpu_off = *(uint64 *)(kaddr("__per_cpu_offset") + cpu * 8);
$irq_time = *(uint64 *)(kaddr("cpu_irqtime") + $pcpu_off);
$prev_time = $rq->psi_irq_time;
$delta = (int64)($irq_time - $prev_time);
if ($delta == 0) {
@delta_zero = count();
} else if ($delta > 0) {
@delta_positive = count();
} else {
@delta_negative = count();
}
}interval:s:30 { exit(); }
--------- end bpftrace ---------
Usama Arif (1):
sched/psi: skip irqtime accounting when no new irq time has elapsed
kernel/sched/psi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 1 line
2.53.0-Meta
PATCH 1/1 sched/psi: skip irqtime accounting when no new irq time has elapsed
psi_account_irqtime() reads irq_time_read() into a per-rq cumulative counter and only bails out when the delta vs. the previously accounted amount is negative. A delta of exactly zero is treated as "do the work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor chain is walked to add zero to every group's PSI_IRQ_FULL bucket.
The zero-delta case is common in practice -- it fires every time a context switch crosses a PSI group boundary on a CPU that hasn't serviced an interrupt between the two switches.
Measured on a 176-thread AMD EPYC 9D64 server running a compute intensive production workload, instrumented with bpftrace over a 30s window (irq_time_read() read directly from the per-CPU cpu_irqtime so that delta == 0 and delta < 0 could be separated):
@total 17,229,311 (100.0%)
@ret_curr_swapper 7,864,195 ( 45.6%) curr->pid == 0
@ret_samegrp 323,299 ( 1.9%) same cgroup as prev
@reached_delta 9,041,817 ( 52.5%)
@delta_positive 6,358,192 ( 36.9%) real work
@delta_zero 2,683,625 ( 15.6%) work wasted (this patch)
@delta_negative (0) ( 0.0%) monotonic clockSo 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the calls that get past the early returns - hit the delta == 0 case; delta < 0 did not occur once in the 30 s window. Under the current code each of those ~89 k calls per second performs the full seqcount write + cpu_clock() read + cgroup-chain walk just to add 0 to every group's PSI_IRQ_FULL counter.
Extend the early-return to also cover delta == 0. rq->psi_irq_time does not need updating in that case (it would store the same value back) and no PSI bucket would change. The existing behaviour for delta > 0 is untouched.
--- kernel/sched/psi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/psi.c +1 -1
@@ -1023,7 +1023,7 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st irq =irq_time_read (cpu); delta = (s64)(irq - rq->psi_irq_time); -if (delta <0 ) +if (delta <=0 )return ; rq->psi_irq_time = irq;
Signature · 1 line
2.53.0-Meta
Quoted 31 lines · Usama Arif
psi_account_irqtime()readsirq_time_read()into a per-rq cumulative counter and only bails out when the delta vs. the previously accounted amount is negative. A delta of exactly zero is treated as "do the work":psi_write_begin()is taken,cpu_clock(cpu)is read (which on x86 ends up innative_sched_clock()/ rdtsc) and the cgroup ancestor chain is walked to add zero to every group'sPSI_IRQ_FULLbucket.The zero-delta case is common in practice -- it fires every time a context switch crosses a PSI group boundary on a CPU that hasn't serviced an interrupt between the two switches.
Measured on a 176-thread AMD EPYC 9D64 server running a compute intensive production workload, instrumented with bpftrace over a 30s window (
irq_time_read()read directly from the per-CPUcpu_irqtimeso that delta == 0 and delta < 0 could be separated):@total 17,229,311 (100.0%) @ret_curr_swapper 7,864,195 ( 45.6%) curr->pid == 0 @ret_samegrp 323,299 ( 1.9%) same cgroup as prev @reached_delta 9,041,817 ( 52.5%) @delta_positive 6,358,192 ( 36.9%) real work @delta_zero 2,683,625 ( 15.6%) work wasted (this patch) @delta_negative (0) ( 0.0%) monotonic clockSo 15.6 % of all
psi_account_irqtime()calls - and 29.7 % of the calls that get past the early returns - hit the delta == 0 case; delta < 0 did not occur once in the 30 s window. Under the current code each of those ~89 k calls per second performs the full seqcount write +cpu_clock()read + cgroup-chain walk just to add 0 to every group'sPSI_IRQ_FULLcounter.
Quoted 6 lines
Extend the early-return to also cover delta == 0.
rq->psi_irq_timedoes not need updating in that case (it would store the same value back) and no PSI bucket would change. The existing behaviour for delta > 0 is untouched.Signed-off-by Usama Arif <[email protected]>
CCing Chengming as well, quote untrimmed.
Quoted 20 lines
--- kernel/sched/psi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/psi.c +1
-1@@ -1023,7 +1023,7 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st irq =irq_time_read (cpu); delta = (s64)(irq - rq->psi_irq_time); -if (delta <0 ) +if (delta <=0 )return ; rq->psi_irq_time = irq;Signature · 1 line
2.53.0-Meta
Quoted 31 lines · Usama Arif
psi_account_irqtime()readsirq_time_read()into a per-rq cumulative counter and only bails out when the delta vs. the previously accounted amount is negative. A delta of exactly zero is treated as "do the work":psi_write_begin()is taken,cpu_clock(cpu)is read (which on x86 ends up innative_sched_clock()/ rdtsc) and the cgroup ancestor chain is walked to add zero to every group'sPSI_IRQ_FULLbucket.The zero-delta case is common in practice -- it fires every time a context switch crosses a PSI group boundary on a CPU that hasn't serviced an interrupt between the two switches.
Measured on a 176-thread AMD EPYC 9D64 server running a compute intensive production workload, instrumented with bpftrace over a 30s window (
irq_time_read()read directly from the per-CPUcpu_irqtimeso that delta == 0 and delta < 0 could be separated):@total 17,229,311 (100.0%) @ret_curr_swapper 7,864,195 ( 45.6%) curr->pid == 0 @ret_samegrp 323,299 ( 1.9%) same cgroup as prev @reached_delta 9,041,817 ( 52.5%) @delta_positive 6,358,192 ( 36.9%) real work @delta_zero 2,683,625 ( 15.6%) work wasted (this patch) @delta_negative (0) ( 0.0%) monotonic clockSo 15.6 % of all
psi_account_irqtime()calls - and 29.7 % of the calls that get past the early returns - hit the delta == 0 case; delta < 0 did not occur once in the 30 s window. Under the current code each of those ~89 k calls per second performs the full seqcount write +cpu_clock()read + cgroup-chain walk just to add 0 to every group'sPSI_IRQ_FULLcounter.
Quoted 6 lines
Extend the early-return to also cover delta == 0.
rq->psi_irq_timedoes not need updating in that case (it would store the same value back) and no PSI bucket would change. The existing behaviour for delta > 0 is untouched.Signed-off-by Usama Arif <[email protected]>
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: b0bc1c75f304099347cfc0cc880b556966b54138 Gitweb: https://git.kernel.org/tip/b0bc1c75f304099347cfc0cc880b556966b54138 Author: Usama Arif <[email protected]> AuthorDate: Wed, 17 Jun 2026 10:50:06 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 30 Jun 2026 10:56:56 +02:00
sched/psi: skip irqtime accounting when no new irq time has elapsed
psi_account_irqtime() reads irq_time_read() into a per-rq cumulative counter and only bails out when the delta vs. the previously accounted amount is negative. A delta of exactly zero is treated as "do the work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor chain is walked to add zero to every group's PSI_IRQ_FULL bucket.
The zero-delta case is common in practice -- it fires every time a context switch crosses a PSI group boundary on a CPU that hasn't serviced an interrupt between the two switches.
Measured on a 176-thread AMD EPYC 9D64 server running a compute intensive production workload, instrumented with bpftrace over a 30s window (irq_time_read() read directly from the per-CPU cpu_irqtime so that delta == 0 and delta < 0 could be separated):
@total 17,229,311 (100.0%)
@ret_curr_swapper 7,864,195 ( 45.6%) curr->pid == 0
@ret_samegrp 323,299 ( 1.9%) same cgroup as prev
@reached_delta 9,041,817 ( 52.5%)
@delta_positive 6,358,192 ( 36.9%) real work
@delta_zero 2,683,625 ( 15.6%) work wasted (this patch)
@delta_negative (0) ( 0.0%) monotonic clockSo 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the calls that get past the early returns - hit the delta == 0 case; delta < 0 did not occur once in the 30 s window. Under the current code each of those ~89 k calls per second performs the full seqcount write + cpu_clock() read + cgroup-chain walk just to add 0 to every group's PSI_IRQ_FULL counter.
Extend the early-return to also cover delta == 0. rq->psi_irq_time does not need updating in that case (it would store the same value back) and no PSI bucket would change. The existing behaviour for delta > 0 is untouched.
--- kernel/sched/psi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/psi.c +1 -1
@@ -1023,7 +1023,7 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st irq =irq_time_read (cpu); delta = (s64)(irq - rq->psi_irq_time); -if (delta <0 ) +if (delta <=0 )return ; rq->psi_irq_time = irq;