Skip to content

perf tests: Robustness and performance improvements

Ian Rogers linux-perf-users lkml 4 versions – Applied

Status from this message: Applied to perf-tools-next, thanks!

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest13 patches 1 replyApplied
  2. v313 patches 6 repliesSuperseded
  3. v212 patches 3 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v112 patches0 repliesSuperseded

Changes in v4 from the cover letter

- Rebased on top of latest ptn/perf-tools-next branch.
- Resolved merge conflicts in util/metricgroup.c, util/parse-events.c, and util/python.c from upstream evlist reference counting changes.
- Fixed a ShellCheck error in trace_record_replay.sh where local was used outside of a function.
- Cleaned up line length warnings from checkpatch.pl in test_brstack.sh.
- Simplified the perf kvm record test loop in kvm.sh to run sleep directly as a workload (instead of running in the background and killing), using -e cpu-clock explicitly to bypass the perf kvm option handling bug on x86/PowerPC platforms while remaining portable to architectures lacking a hardware cycles event (e.g. s390x).

Patches in v4 Applied

#Subject and filesSizeRepliesApplied
1/13perf parse-events: Restrict core PMU bypass to --cputype optiontools/perf/builtin-script.c tools/perf/builtin-stat.c tools/perf/tests/expand-cgroup.c +8 more+87 -430 replies
2/13perf test: Truncate test description to fit terminal widthtools/perf/tests/builtin-test.c+89 -740 replies
3/13perf tests workloads: Support sub-second durations in noploop and thlooptools/perf/tests/workloads/noploop.c tools/perf/tests/workloads/thloop.c+25 -80 replies
4/13perf tests: Add robust record retry helper and use subsecond workloadstools/perf/tests/shell/kvm.sh tools/perf/tests/shell/lib/perf_record.sh tools/perf/tests/shell/pipe_test.sh +2 more+214 -1270 replies
5/13perf tests: Skip metrics validation if system-wide recording lacks permissiontools/perf/tests/shell/lib/perf_metric_validation.py tools/perf/tests/shell/stat_all_metrics.sh tools/perf/tests/shell/stat_metrics_values.sh+60 -330 replies
6/13perf tests: Fix Python JIT dump profiling test failuretools/perf/tests/shell/jitdump-python.sh+53 -260 replies
7/13perf tests: Fix flakiness in trace record and replay testtools/perf/tests/shell/lib/perf_record.sh tools/perf/tests/shell/trace_record_replay.sh+40 -50 replies
8/13perf tests: Fix flakiness in BPF counters test on hybrid systemstools/perf/tests/shell/stat_bpf_counters.sh+18 -100 replies
9/13perf tests: Fix flakiness in branch stack sampling teststools/perf/tests/shell/test_brstack.sh+66 -410 replies
10/13perf tests: Speed up off-cpu profiling teststools/perf/tests/shell/record_offcpu.sh+7 -70 replies
11/13perf tests: Speed up lock contention analysis shell testtools/perf/tests/shell/lock_contention.sh+19 -130 replies
12/13perf tests: Speed up metrics checking shell teststools/perf/tests/shell/stat_all_metrics.sh tools/perf/tests/shell/stat_metrics_values.sh+2 -20 replies
13/13perf tests: Include error output for skipped tests in JUnit XMLtools/perf/tests/builtin-test.c+8 -20 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v4 yet.

Read the v4 thread

Cover letter [PATCH v4 00/13] perf tests: Robustness and performance improvements

This patch series introduces several robustness and performance improvements to the perf test suite, alongside fixes for long-standing flakiness.

Key changes across the series:
- Introduces robust retry logic (perf_record_with_retry) to prevent spurious test failures due to transient EPERM issues during system-wide background profiling, safely extracting error messages from the PERF_RECORD_LOGS array.
- Significantly speeds up test execution across multiple scripts (such as kvm, record, trace, off-cpu, and lock contention) by supporting sub-second durations in the noploop and thloop workloads, reducing unnecessary wait times.
- Fixes flakiness in the BPF counters hybrid test by parsing taskset to determine valid CPUs rather than relying on shuf, which can fail under cgroups or missing binaries.
- Fixes JIT dump file leaks and TOCTOU vulnerabilities by securely staging with mktemp -d to guarantee cleanup.
- Decouples format alignments in builtin-test.c to gracefully truncate test descriptions without losing alignment or failing under non-TTY environments.
- Restricts uncore PMU bypass in event parsing strictly to --cputype to prevent unintended metric evaluation impacts.
- Injects trace output into JUnit XML <skipped> blocks to aid debugging.

Changes in v4:
- Rebased on top of latest ptn/perf-tools-next branch.
- Resolved merge conflicts in util/metricgroup.c, util/parse-events.c, and util/python.c from upstream evlist reference counting changes.
- Fixed a ShellCheck error in trace_record_replay.sh where local was used outside of a function.
- Cleaned up line length warnings from checkpatch.pl in test_brstack.sh.
- Simplified the perf kvm record test loop in kvm.sh to run sleep directly as a workload (instead of running in the background and killing), using -e cpu-clock explicitly to bypass the perf kvm option handling bug on x86/PowerPC platforms while remaining portable to architectures lacking a hardware cycles event (e.g. s390x).

Changes in v3:
- Fix line length warnings in checkpatch.pl.
- Fix wrap commit descriptions in checkpatch.pl.
- Re-aligned parse_events_state struct initializations.
- Added explanatory comment for the 10000 magic number in get_term_width().
- Added inline comments (e.g. /*cputype_filter=*/false) to boolean literal arguments.
- Fixed patch 4 to securely read from PERF_RECORD_LOGS array instead of unsafe wildcard cleanup.
- Fixed patch 6 TOCTOU vulnerability via mktemp -d staging directory.
- Implemented patch 13 to inject trace output into JUnit XML <skipped> blocks for debugging.
- Add Assisted-by tags to all commits.

Changes in v2:
- Drop the sleep 0 patch from v1.
- Introduce 'perf_record_with_retry' helper to encapsulate the retry logic and cleanly output error messages (patch 4).
- Fix issue with perf record lacking sufficient permissions on some setups (patch 5).
- Improve subsecond duration support in noploop/thloop (patch 3).
- Reduce test durations to avoid excessive overall runtimes.

Ian Rogers (13):
perf parse-events: Restrict core PMU bypass to --cputype option perf test: Truncate test description to fit terminal width perf tests workloads: Support sub-second durations in noploop and
thloop
perf tests: Add robust record retry helper and use subsecond workloads perf tests: Skip metrics validation if system-wide recording lacks
permission
perf tests: Fix Python JIT dump profiling test failure perf tests: Fix flakiness in trace record and replay test perf tests: Fix flakiness in BPF counters test on hybrid systems perf tests: Fix flakiness in branch stack sampling tests perf tests: Speed up off-cpu profiling tests
perf tests: Speed up lock contention analysis shell test perf tests: Speed up metrics checking shell tests
perf tests: Include error output for skipped tests in JUnit XML

 tools/perf/builtin-script.c                   |   1 +
 tools/perf/builtin-stat.c                     |  20 +-
 tools/perf/tests/builtin-test.c               | 173 ++++++++++--------
 tools/perf/tests/expand-cgroup.c              |   3 +-
 tools/perf/tests/parse-events.c               |  11 +-
 tools/perf/tests/parse-metric.c               |   3 +-
 tools/perf/tests/pmu-events.c                 |  10 +-
 tools/perf/tests/shell/jitdump-python.sh      |  79 +++++---
 tools/perf/tests/shell/kvm.sh                 |  61 +++---
 .../tests/shell/lib/perf_metric_validation.py |  11 +-
 tools/perf/tests/shell/lib/perf_record.sh     |  58 ++++++
 tools/perf/tests/shell/lock_contention.sh     |  32 ++--
 tools/perf/tests/shell/pipe_test.sh           |   4 +-
 tools/perf/tests/shell/record.sh              | 173 +++++++++---------
 tools/perf/tests/shell/record_lbr.sh          |  50 +++--
 tools/perf/tests/shell/record_offcpu.sh       |  14 +-
 tools/perf/tests/shell/stat_all_metrics.sh    |  77 +++++---
 tools/perf/tests/shell/stat_bpf_counters.sh   |  28 ++-
 tools/perf/tests/shell/stat_metrics_values.sh |   9 +-
 tools/perf/tests/shell/test_brstack.sh        | 107 ++++++-----
 tools/perf/tests/shell/trace_record_replay.sh |  38 +++-
 tools/perf/tests/workloads/noploop.c          |  17 +-
 tools/perf/tests/workloads/thloop.c           |  16 +-
 tools/perf/util/metricgroup.c                 |  26 ++-
 tools/perf/util/metricgroup.h                 |   4 +-
 tools/perf/util/parse-events.c                |  32 ++--
 tools/perf/util/parse-events.h                |  17 +-
 tools/perf/util/python.c                      |   3 +-
 28 files changed, 687 insertions(+), 390 deletions(-)
 create mode 100644 tools/perf/tests/shell/lib/perf_record.sh
Signature · 1 line
2.55.0.rc0.786.g65d90a0328-goog