Skip to content

alpha/perf: Enable duration-based sampling throttling

Magnus Lindholm linux-alpha lkml 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest1 patch 1 replyIn review
  2. v11 patch 2 repliesSuperseded

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/1alpha/perf: Account for time spent handling PMU interruptsarch/alpha/kernel/perf_event.c+33 -21 reply

Reviews trailers per patch

The patch has a Reviewed-by or Acked-by.

PatchMatt Turner
1/1R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH v2 0/1] alpha/perf: Enable duration-based sampling throttling

Hi,

Changes in v2, following Matt's review:
- Describe the unresponsive-machine/RCU-starvation symptom in the commit.
- Include proto.h for est_cycle_freq and use cycles_t without the cast.
- Precompute the cycles-to-nanoseconds multiplier/shift and mark both __ro_after_init, removing division from the interrupt path.
- Add Matt's Reviewed-by.

Both v2 mainline and ES40 integration builds passed (vmlinux, boot image and modules), with only the existing saved_config COMMON-symbol warning. Disassembly confirms no division call in either interrupt wrapper. The multiplier conversion checks cover rounding, overflow and RPCC wrap. The historical hardware results below apply to v1's division-based conversion. V2 also passed seven bounded tests on the ES40 with the alpha-devel integration: the 1024/s and 4096/s CPU0 controls, 1024/s controls on CPUs 1 and 2, and period-17 tests starting at 100000/s on each CPU. Duration feedback reduced the limits to 7168/s, 5120/s and 5120/s respectively. All 26156 samples were recorded without reported loss, and no new RCU stalls or kernel faults appeared, including a 30-second observation after the final run. This does not repeat the full v1 stress matrix below. The selected-frequency cleanup remains outside this patch.

While testing Matt's v7 ProfileMe series [1] on my EV67 AlphaServer ES40, I found that short-period sampling could make the machine unresponsive. Once serial logging was enabled, the console showed RCU grace-period kthread starvation reports. The failure also reproduced without my generic-entry follow-up patches.

Count-based throttling was working: at HZ=1024 and the default limit of 100000 samples/s, the observed cadence was approximately 98 interrupts per throttle cycle. However, Alpha never called perf_sample_event_took(), so perf could not reduce that limit based on interrupt handling cost. Setting perf_cpu_time_max_percent to 25 did not provide that feedback.

Keeping period 17 but lowering the limit to 1024 or 4096 samples/s made the tests pass while still exercising repeated stop/park/restart cycles. Adding duration feedback also made the original high-rate test pass, without changing those counter-state operations. The minimal fix measured average handler costs of 24-26 us against a 2.5 us budget and reduced the limit to 7168-8192 samples/s.

Reproducer
==========

The reproducer requires Matt's ProfileMe kernel series and matching perf tools. On the ES40 I used the following one-second workload, built natively as "busy":

  #include <stdint.h>
  #include <time.h>

  static double now(void)
  {
          struct timespec t;
          clock_gettime(CLOCK_MONOTONIC, &t);
          return t.tv_sec + t.tv_nsec * 1e-9;

}

  int main(void)
  {
          uint64_t x = 1;
          double end = now() + 1;
          do {
                  for (int i = 0; i < 1000; i++)
                          asm volatile("mulq %0,3,%0" : "+r" (x));
          } while (now() < end);
          return 0;

}

cc -O2 -o busy busy.c

As root, with serial capture running:

dmesg -n 7
sysctl -w kernel.perf_event_max_sample_rate=100000 sysctl -w kernel.perf_cpu_time_max_percent=25

The period-10007 control completed successfully:

  taskset -c 0 perf record -B -R \
      -e cpu/instructions,profileme,randomize/ -c 10007 \
      -o control.data -- ./busy

Changing only the period reproduced the failure:

  taskset -c 0 perf record -B -R \
      -e cpu/instructions,profileme,randomize/ -c 17 \
      -o period17.data -- ./busy

The unpatched runs required a manual reset; a userspace timeout did not reliably recover the system. The commands above are the essential recording commands from the controlled tests, which also used a timeout supervisor and saved logs on another machine.

Validation and submission scope
===============================

The v1 minimal fix, without diagnostic instrumentation, passed three repeats of the high-rate reproducer and the two low-rate controls. Broader testing covered 21 cases on all three CPUs: randomized ProfileMe instruction sampling at periods 16, 17 and 31, ProfileMe and Aggregate cycle sampling, and concurrent ProfileMe and mixed-mode workloads. All completed, with 1352488 samples, no recorded losses and no new RCU stalls or kernel faults. This was bounded stress testing, not a long soak.

The missing feedback predates ProfileMe. This patch is therefore based on mainline and can be reviewed and applied independently of Matt's series. I cross-built vmlinux, the Alpha boot image and modules on mainline d48723711bae with the ES40 configuration and GCC 15.0.1. The only build warning was the existing saved_config COMMON-symbol warning.

The hardware results above used Matt's v7 series with the v1 fix applied; the mainline-only build has not been boot-tested. RPCC supplies a short, same-CPU interval because Alpha's tick-based sched_clock() is too coarse for this measurement. Counter programming and event handling are unchanged.

[1] https://lore.kernel.org/linux-alpha/[email protected]/

Magnus Lindholm (1):
alpha/perf: Account for time spent handling PMU interrupts

 arch/alpha/kernel/perf_event.c | 35 ++++++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0