Skip to contentSkip to conversation

sched/psi: skip irqtime accounting when no new irq time has elapsed

PATCH Applied Usama Arif 4 participants 5 messages Jun 17 – Jun 30

Subsystems (3)
lore
Share & tools
lore

Usama Arif

author
#0 lore raw

cover sched/psi: skip irqtime accounting when no new irq time has elapsed

psi_account_irqtime() reads irq_time_read() into a per-rq cumulative counter and only bails out when the delta vs. the previously accounted amount is negative. A delta of exactly zero is treated as "do the work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor chain is walked to add zero to every group's PSI_IRQ_FULL bucket.

The zero-delta case is common in practice -- it fires every time a context switch crosses a PSI group boundary on a CPU that hasn't serviced an interrupt between the two switches.

To find out how often this actually fires in the wild, I attached a bpftrace probe to psi_account_irqtime() on a 176-thread AMD EPYC 9D64 server running an compute intensive workload.
The probe also reads irq_time_read(cpu) directly from the per-CPU cpu_irqtime variable so it can separate delta == 0 from delta < 0. The bpftrace script was generated by claude and is at the end of the coverletter.

Over a 30 s window under steady-state load:

  @total              17,229,311    (100.0%)
  @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
  @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
  @reached_delta       9,041,817    ( 52.5%)
    @delta_positive    6,358,192    ( 36.9%)  real work
    @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
    @delta_negative          (0)    (  0.0%)  monotonic clock

15.6 % of all psi_account_irqtime() calls -- and 29.7 % of the calls that get past the early returns -- hit the delta == 0 case. delta < 0 did not occur once in the 30 s window. That works out to ~89 k calls/sec on this host that today take the full seqcount write + cpu_clock() + cgroup-chain walk purely to add 0 to every group's PSI_IRQ_FULL counter.

Extend the early-return to also cover delta == 0. rq->psi_irq_time does not need updating in that case (it would store the same value back) and no PSI bucket would change. The existing behaviour for delta > 0 is untouched.

--------- psi_delta_exact.bt -------
#!/usr/bin/env bpftrace

#include <linux/sched.h>

kprobe:psi_account_irqtime

{
	$rq = (struct rq *)arg0;
	$curr = (struct task_struct *)arg1;
	$prev = (struct task_struct *)arg2;

	@total = count();

	if ($curr->pid == 0) {
		@ret_curr_swapper = count();
		return;
	}

	if ($prev != 0) {
		$pg_curr = $curr->cgroups->dfl_cgrp;
		$pg_prev = $prev->cgroups->dfl_cgrp;
		if ($pg_curr == $pg_prev) {
			@ret_samegrp = count();
			return;
		}
	}

	@reached_delta = count();

	$pcpu_off  = *(uint64 *)(kaddr("__per_cpu_offset") + cpu * 8);
	$irq_time  = *(uint64 *)(kaddr("cpu_irqtime") + $pcpu_off);
	$prev_time = $rq->psi_irq_time;
	$delta     = (int64)($irq_time - $prev_time);

	if ($delta == 0) {
		@delta_zero = count();
	} else if ($delta > 0) {
		@delta_positive    = count();
	} else {
		@delta_negative = count();
	}
}

interval:s:30 { exit(); }
--------- end bpftrace ---------

Usama Arif (1):
sched/psi: skip irqtime accounting when no new irq time has elapsed

 kernel/sched/psi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 1 line
2.53.0-Meta
author
#1 lore raw

PATCH 1/1 sched/psi: skip irqtime accounting when no new irq time has elapsed

psi_account_irqtime() reads irq_time_read() into a per-rq cumulative counter and only bails out when the delta vs. the previously accounted amount is negative. A delta of exactly zero is treated as "do the work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor chain is walked to add zero to every group's PSI_IRQ_FULL bucket.

The zero-delta case is common in practice -- it fires every time a context switch crosses a PSI group boundary on a CPU that hasn't serviced an interrupt between the two switches.

Measured on a 176-thread AMD EPYC 9D64 server running a compute intensive production workload, instrumented with bpftrace over a 30s window (irq_time_read() read directly from the per-CPU cpu_irqtime so that delta == 0 and delta < 0 could be separated):

  @total              17,229,311    (100.0%)
  @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
  @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
  @reached_delta       9,041,817    ( 52.5%)
    @delta_positive    6,358,192    ( 36.9%)  real work
    @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
    @delta_negative          (0)    (  0.0%)  monotonic clock

So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the calls that get past the early returns - hit the delta == 0 case; delta < 0 did not occur once in the 30 s window. Under the current code each of those ~89 k calls per second performs the full seqcount write + cpu_clock() read + cgroup-chain walk just to add 0 to every group's PSI_IRQ_FULL counter.

Extend the early-return to also cover delta == 0. rq->psi_irq_time does not need updating in that case (it would store the same value back) and no PSI bucket would change. The existing behaviour for delta > 0 is untouched.

---
 kernel/sched/psi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/psi.c +1 -1
index d9c9d9480a45..848955f8893d 100644
@@ -1023,7 +1023,7 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st
 
 	irq = irq_time_read(cpu);
 	delta = (s64)(irq - rq->psi_irq_time);
-	if (delta < 0)
+	if (delta <= 0)
 		return;
 	rq->psi_irq_time = irq;
 
Signature · 1 line
2.53.0-Meta
maintainer Acked-by on v1 1/1
#2 lore raw
Quoted 31 lines · Usama Arif

psi_account_irqtime() reads irq_time_read() into a per-rq cumulative counter and only bails out when the delta vs. the previously accounted amount is negative. A delta of exactly zero is treated as "do the work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor chain is walked to add zero to every group's PSI_IRQ_FULL bucket.

The zero-delta case is common in practice -- it fires every time a context switch crosses a PSI group boundary on a CPU that hasn't serviced an interrupt between the two switches.

Measured on a 176-thread AMD EPYC 9D64 server running a compute intensive production workload, instrumented with bpftrace over a 30s window (irq_time_read() read directly from the per-CPU cpu_irqtime so that delta == 0 and delta < 0 could be separated):

  @total              17,229,311    (100.0%)
  @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
  @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
  @reached_delta       9,041,817    ( 52.5%)
    @delta_positive    6,358,192    ( 36.9%)  real work
    @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
    @delta_negative          (0)    (  0.0%)  monotonic clock

So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the calls that get past the early returns - hit the delta == 0 case; delta < 0 did not occur once in the 30 s window. Under the current code each of those ~89 k calls per second performs the full seqcount write + cpu_clock() read + cgroup-chain walk just to add 0 to every group's PSI_IRQ_FULL counter.

Quoted 6 lines

Extend the early-return to also cover delta == 0. rq->psi_irq_time does not need updating in that case (it would store the same value back) and no PSI bucket would change. The existing behaviour for delta > 0 is untouched.

CCing Chengming as well, quote untrimmed.

Quoted 20 lines
---
 kernel/sched/psi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/psi.c +1 -1
index d9c9d9480a45..848955f8893d 100644
@@ -1023,7 +1023,7 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st
 
 	irq = irq_time_read(cpu);
 	delta = (s64)(irq - rq->psi_irq_time);
-	if (delta < 0)
+	if (delta <= 0)
 		return;
 	rq->psi_irq_time = irq;
 
Signature · 1 line
2.53.0-Meta
Reviewed-by on v1 1/1
#3 lore raw
In reply to Usama Arif
Quoted 31 lines · Usama Arif

psi_account_irqtime() reads irq_time_read() into a per-rq cumulative counter and only bails out when the delta vs. the previously accounted amount is negative. A delta of exactly zero is treated as "do the work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor chain is walked to add zero to every group's PSI_IRQ_FULL bucket.

The zero-delta case is common in practice -- it fires every time a context switch crosses a PSI group boundary on a CPU that hasn't serviced an interrupt between the two switches.

Measured on a 176-thread AMD EPYC 9D64 server running a compute intensive production workload, instrumented with bpftrace over a 30s window (irq_time_read() read directly from the per-CPU cpu_irqtime so that delta == 0 and delta < 0 could be separated):

  @total              17,229,311    (100.0%)
  @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
  @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
  @reached_delta       9,041,817    ( 52.5%)
    @delta_positive    6,358,192    ( 36.9%)  real work
    @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
    @delta_negative          (0)    (  0.0%)  monotonic clock

So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the calls that get past the early returns - hit the delta == 0 case; delta < 0 did not occur once in the 30 s window. Under the current code each of those ~89 k calls per second performs the full seqcount write + cpu_clock() read + cgroup-chain walk just to add 0 to every group's PSI_IRQ_FULL counter.

Quoted 6 lines

Extend the early-return to also cover delta == 0. rq->psi_irq_time does not need updating in that case (it would store the same value back) and no PSI bucket would change. The existing behaviour for delta > 0 is untouched.

merge bot Applied
#4 lore raw
In reply to Usama Arif
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     b0bc1c75f304099347cfc0cc880b556966b54138
Gitweb:        https://git.kernel.org/tip/b0bc1c75f304099347cfc0cc880b556966b54138
Author:        Usama Arif <[email protected]>
AuthorDate:    Wed, 17 Jun 2026 10:50:06 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Tue, 30 Jun 2026 10:56:56 +02:00

sched/psi: skip irqtime accounting when no new irq time has elapsed

psi_account_irqtime() reads irq_time_read() into a per-rq cumulative counter and only bails out when the delta vs. the previously accounted amount is negative. A delta of exactly zero is treated as "do the work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor chain is walked to add zero to every group's PSI_IRQ_FULL bucket.

The zero-delta case is common in practice -- it fires every time a context switch crosses a PSI group boundary on a CPU that hasn't serviced an interrupt between the two switches.

Measured on a 176-thread AMD EPYC 9D64 server running a compute intensive production workload, instrumented with bpftrace over a 30s window (irq_time_read() read directly from the per-CPU cpu_irqtime so that delta == 0 and delta < 0 could be separated):

  @total              17,229,311    (100.0%)
  @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
  @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
  @reached_delta       9,041,817    ( 52.5%)
    @delta_positive    6,358,192    ( 36.9%)  real work
    @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
    @delta_negative          (0)    (  0.0%)  monotonic clock

So 15.6 % of all psi_account_irqtime() calls - and 29.7 % of the calls that get past the early returns - hit the delta == 0 case; delta < 0 did not occur once in the 30 s window. Under the current code each of those ~89 k calls per second performs the full seqcount write + cpu_clock() read + cgroup-chain walk just to add 0 to every group's PSI_IRQ_FULL counter.

Extend the early-return to also cover delta == 0. rq->psi_irq_time does not need updating in that case (it would store the same value back) and no PSI bucket would change. The existing behaviour for delta > 0 is untouched.

---
 kernel/sched/psi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/psi.c +1 -1
index d9c9d94..848955f 100644
@@ -1023,7 +1023,7 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st
 
 	irq = irq_time_read(cpu);
 	delta = (s64)(irq - rq->psi_irq_time);
-	if (delta < 0)
+	if (delta <= 0)
 		return;
 	rq->psi_irq_time = irq;