Skip to content

sched/psi: skip irqtime accounting when no new irq time has elapsed

Usama Arif linux-mm lkml 1 version Applied

Status from this message: The following commit has been merged into the sched/core branch of tip:

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest1 patch 2 repliesApplied

Patches in v1 Applied

#Subject and filesSizeRepliesApplied
1/1sched/psi: skip irqtime accounting when no new irq time has elapsedkernel/sched/psi.c+1 -12 replies

Reviews trailers per patch

The patch has a Reviewed-by or Acked-by.

PatchJohannes WeinerShakeel Butt
1/1A-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 0/1] sched/psi: skip irqtime accounting when no new irq time has elapsed

psi_account_irqtime() reads irq_time_read() into a per-rq cumulative counter and only bails out when the delta vs. the previously accounted amount is negative. A delta of exactly zero is treated as "do the work": psi_write_begin() is taken, cpu_clock(cpu) is read (which on x86 ends up in native_sched_clock() / rdtsc) and the cgroup ancestor chain is walked to add zero to every group's PSI_IRQ_FULL bucket.

The zero-delta case is common in practice -- it fires every time a context switch crosses a PSI group boundary on a CPU that hasn't serviced an interrupt between the two switches.

To find out how often this actually fires in the wild, I attached a bpftrace probe to psi_account_irqtime() on a 176-thread AMD EPYC 9D64 server running an compute intensive workload.
The probe also reads irq_time_read(cpu) directly from the per-CPU cpu_irqtime variable so it can separate delta == 0 from delta < 0. The bpftrace script was generated by claude and is at the end of the coverletter.

Over a 30 s window under steady-state load:

  @total              17,229,311    (100.0%)
  @ret_curr_swapper    7,864,195    ( 45.6%)  curr->pid == 0
  @ret_samegrp           323,299    (  1.9%)  same cgroup as prev
  @reached_delta       9,041,817    ( 52.5%)
    @delta_positive    6,358,192    ( 36.9%)  real work
    @delta_zero        2,683,625    ( 15.6%)  work wasted (this patch)
    @delta_negative          (0)    (  0.0%)  monotonic clock

15.6 % of all psi_account_irqtime() calls -- and 29.7 % of the calls that get past the early returns -- hit the delta == 0 case. delta < 0 did not occur once in the 30 s window. That works out to ~89 k calls/sec on this host that today take the full seqcount write + cpu_clock() + cgroup-chain walk purely to add 0 to every group's PSI_IRQ_FULL counter.

Extend the early-return to also cover delta == 0. rq->psi_irq_time does not need updating in that case (it would store the same value back) and no PSI bucket would change. The existing behaviour for delta > 0 is untouched.

--------- psi_delta_exact.bt -------
#!/usr/bin/env bpftrace

#include <linux/sched.h>

kprobe:psi_account_irqtime

{
	$rq = (struct rq *)arg0;
	$curr = (struct task_struct *)arg1;
	$prev = (struct task_struct *)arg2;

	@total = count();

	if ($curr->pid == 0) {
		@ret_curr_swapper = count();
		return;
	}

	if ($prev != 0) {
		$pg_curr = $curr->cgroups->dfl_cgrp;
		$pg_prev = $prev->cgroups->dfl_cgrp;
		if ($pg_curr == $pg_prev) {
			@ret_samegrp = count();
			return;
		}
	}

	@reached_delta = count();

	$pcpu_off  = *(uint64 *)(kaddr("__per_cpu_offset") + cpu * 8);
	$irq_time  = *(uint64 *)(kaddr("cpu_irqtime") + $pcpu_off);
	$prev_time = $rq->psi_irq_time;
	$delta     = (int64)($irq_time - $prev_time);

	if ($delta == 0) {
		@delta_zero = count();
	} else if ($delta > 0) {
		@delta_positive    = count();
	} else {
		@delta_negative = count();
	}
}

interval:s:30 { exit(); }
--------- end bpftrace ---------

Usama Arif (1):
sched/psi: skip irqtime accounting when no new irq time has elapsed

 kernel/sched/psi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Signature · 1 line
2.53.0-Meta