perf python: Follow-up fixes and shell test deflaking
Ian Rogers
reviewercover perf python: Follow-up fixes and shell test deflaking
This series contains follow-up fixes for the standalone Python scripts and their shell tests, particularly when running 'perf test' as root under heavy parallel load:
- Clamp sample_id size calculation for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE records to work around kernels without "perf/core: Restore header fields in sideband output callbacks" [1].
- Fix offline interval printing in sctop.py and live signal races in stat-cpi.py.
- Deflake Intel PT, failed-syscalls, and other Python shell tests under parallel 'perf test' load by scoping tracepoint checks to 'perf list tracepoint', passing '-B -N --no-bpf-event' to 'perf record', and avoiding unnecessary system-wide ('-a') recordings.
- Default event_analyzing_sample.py to an in-memory SQLite database (':memory:') when '--db' is not specified.
- Initialize perf debug output ('perf_debug_setup()') when loading the 'perf' Python extension module.
- Fix races in 'perf list' ('perf_pmus__print_pmu_events()' and 'tp_pmu.c') when concurrent tests dynamically create and remove kprobe/uprobes tracepoints.
[1] https://lore.kernel.org/r/[email protected]
Ian Rogers (9):
perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke
events
perf python sctop: Fix offline interval printing and test flakiness perf python stat-cpi: Fix live mode signal races and test flakiness perf test: Deflake Intel PT Python shell tests under load perf test: Deflake failed-syscalls Python shell tests under load perf test: Reduce overhead and contention in Python shell tests perf python event_analyzing_sample: Default to in-memory SQLite
database
perf python: Initialize debug output on module load
perf pmu: Fix race with concurrent tracepoint creation and removal in
perf list
tools/perf/python/event_analyzing_sample.py | 35 +++------- tools/perf/python/sctop.py | 9 ++- tools/perf/python/stat-cpi.py | 19 +++--- .../shell/test_check_perf_trace_python.sh | 8 ++- .../shell/test_compaction_times_python.sh | 8 ++- .../test_event_analyzing_sample_python.sh | 11 ++- .../shell/test_export_to_postgresql_python.sh | 49 +++++++------ .../shell/test_export_to_sqlite_python.sh | 50 ++++++++------ .../test_failed_syscalls_by_pid_python.sh | 68 +++++++++++-------- .../shell/test_failed_syscalls_python.sh | 44 +++++++----- .../tests/shell/test_flamegraph_python.sh | 5 +- .../shell/test_futex_contention_python.sh | 7 +- tools/perf/tests/shell/test_gecko_python.sh | 3 +- .../shell/test_intel_pt_events_python.sh | 43 ++++++------ .../tests/shell/test_mem_phys_addr_python.sh | 8 ++- .../shell/test_net_dropmonitor_python.sh | 9 +-- .../tests/shell/test_netdev_times_python.sh | 6 +- .../tests/shell/test_powerpc_hcalls_python.sh | 4 +- .../tests/shell/test_rw_by_file_python.sh | 5 +- .../perf/tests/shell/test_rw_by_pid_python.sh | 4 +- tools/perf/tests/shell/test_rwtop_python.sh | 4 +- .../shell/test_sched_migration_python.sh | 4 +- tools/perf/tests/shell/test_sctop_python.sh | 48 ++++++------- .../tests/shell/test_stackcollapse_python.sh | 2 +- .../perf/tests/shell/test_stat_cpi_python.sh | 12 +++- .../test_syscall_counts_by_pid_python.sh | 6 +- .../tests/shell/test_syscall_counts_python.sh | 5 +- tools/perf/tests/shell/test_task_analyzer.sh | 3 +- .../tests/shell/test_wakeup_latency_python.sh | 7 +- tools/perf/util/debug.c | 2 +- tools/perf/util/evlist.c | 6 +- tools/perf/util/evsel.c | 59 +++++++++++++++- tools/perf/util/evsel.h | 1 + tools/perf/util/pmus.c | 20 ++++-- tools/perf/util/python.c | 1 + tools/perf/util/tp_pmu.c | 8 ++- 36 files changed, 358 insertions(+), 225 deletions(-)
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Ian Rogers
reviewerPATCH 1/9 perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
On kernels without the fix ("perf/core: Restore header fields in sideband output callbacks") to restore event_id.header.size in perf_event_ksymbol_output(), perf_event_bpf_output(), and perf_event_text_poke_output(), concurrent perf sessions cause those sideband records to be emitted with header.size inflated by multiple id_header_size increments while the single id_sample is written immediately after the event payload. Indexing backwards from event->header.size reads uninitialized ring-buffer bytes at the end of the record, causing evlist__event2evsel() to fail with -EFAULT.
Add evsel__event_size() to clamp the effective size used to locate the trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving event->header.size intact for advancing the ring-buffer/file stream.
--- tools/perf/util/evlist.c | 6 ++-- tools/perf/util/evsel.c | 59 +++++++++++++++++++++++++++++++++++++++- tools/perf/util/evsel.h | 1 + 3 files changed, 63 insertions(+), 3 deletions(-)
tools/perf/util/evlist.c +4 -2
@@ -973,13 +973,15 @@ static int evlist__event2id(struct evlist *evlist, union perf_event *event, u64const __u64 *array = event->sample.array;ssize_t n; - n = (event->header.size -sizeof (event->header)) >>3 ; -if (event->header.type ==PERF_RECORD_SAMPLE ) { + n = (event->header.size -sizeof (event->header)) >>3 ;if (evlist__id_pos (evlist) >= n)return -1 ; *id = array[evlist__id_pos (evlist)]; }else { +u16 size =evsel__event_size (evlist__first (evlist), event); + + n = (size -sizeof (event->header)) >>3 ;if (evlist__is_pos (evlist) > n)return -1 ; n -=evlist__is_pos (evlist);
tools/perf/util/evsel.c +58 -1
71 lines not on this page · raw message
tools/perf/util/evsel.h +1 -0
@@ -469,6 +469,7 @@ int evsel__parse_sample_timestamp(struct evsel *evsel, union perf_event *event, u64 *timestamp);u16 evsel__id_hdr_size (const struct evsel *evsel); +u16 evsel__event_size (const struct evsel *evsel,const union perf_event *event);static inline struct evsel *evsel__next (struct evsel *evsel) {
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v1 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf evsel: Incorrect PERF_RECORD_TEXT_POKE payload size calculation --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf evsel: Incorrect PERF_RECORD_TEXT_POKE payload size calculation --
commit fc6474e9b3305fb280e04ce2b30b2a815e2f3a7e
Author: Ian Rogers <[email protected]>
perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
This commit adds evsel__event_size() to clamp the effective size used to locate the trailing id_sample for specific events. This prevents user-space tools from reading uninitialized memory when processing data from unpatched kernels.
Quoted 4 lines
tools/perf/util/evsel.c +0
-0
[ ... ]
Quoted 6 lines
tools/perf/util/evsel.c +2
-0@@ -3966,6 +3966,63 @@ u16 evsel__id_hdr_size(const struct evsel *evsel)return size; } +/* +* Prior to kernel fix, perf_event_ksymbol_output(), perf_event_bpf_output(),
[ ... ]
tools/perf/util/evsel.c u16 evsel__id_hdr_size(const struct evsel *evsel) · line 4013
Quoted 7 lines
tools/perf/util/evsel.c +7
-0+case PERF_RECORD_TEXT_POKE : { +size_t fixed = offsetof(struct perf_record_text_poke_event , bytes); + +if (size < fixed) +return size; + payload = fixed +PERF_ALIGN ((size_t )event->text_poke.old_len + + event->text_poke.new_len,sizeof (u64));
[Severity: Medium]
Does this misapply the 8-byte alignment only to the variable lengths rather than the entire structure?
The kernel generates PERF_RECORD_TEXT_POKE events by padding the entire event size to an 8-byte boundary. The payload size before id_sample is 16 + ALIGN(old_len + new_len + 4, 8).
This calculates the user-space payload as
fixed + PERF_ALIGN(old_len + new_len, 8), where fixed = 20. Because 20 is not a multiple of 8, adding it outside the alignment yields a mathematically incorrect size.
For example, if old_len = 1 and new_len = 1, the kernel calculates 24 bytes, but the code here calculates 28 bytes. Calculating the wrong payload size causes perf to index incorrectly into event->sample.array, leading to parsing failures or out-of-bounds accesses.
Quoted 5 lines
tools/perf/util/evsel.c +5
-0+break ; + } +default : +return size; + }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Ian Rogers
reviewerPATCH 2/9 perf python sctop: Fix offline interval printing and test flakiness
In reply to Ian RogersIn sctop.py, process_event() updated self.last_print_time on every sample in offline mode even when skip was true for non-matching comm events. When filtering by comm with a short interval (such as 'sleep 1' in test_sctop_python.sh) on a system-wide recording spanning more than 1 second under heavy load, non-matching samples could trigger print_current_totals() and clear self.syscalls, followed by finally:
printing a trailing empty table. In addition, analyzer.e_machine was initialized after session.process_events() instead of before.
Fix sctop.py by initializing analyzer.e_machine before session.process_events(), only advancing self.last_print_time when a sample is not skipped, and only calling print_current_totals() in
finally: if no table has been printed yet or remaining syscalls are pending.
In test_sctop_python.sh, use 'perf list tracepoint' instead of unfiltered 'perf list', record the child workload directly without '-a' and with '-B -N --no-bpf-event' to avoid system-wide ringbuffer overflow and synthesis contention, and retry up to 5 times if needed.
--- tools/perf/python/sctop.py | 9 ++-- tools/perf/tests/shell/test_sctop_python.sh | 48 +++++++++++---------- 2 files changed, 31 insertions(+), 26 deletions(-)
tools/perf/python/sctop.py +6 -3
@@ -37,6 +37,7 @@ class SCTopAnalyzer: self.offline = offline self.own_pid = os.getpid() self.last_print_time: Optional[int] = None + self.printed = False self.session: Optional[perf.session] = None self.e_machine: Optional[int] = None
@@ -127,7 +128,7 @@ class SCTopAnalyzer: if not skip and is_enter and 0 <= (syscall_id & ~0x40000000) <= 0xffff: self.syscalls[syscall_id] += 1 - if self.offline and hasattr(sample, "sample_time"): + if not skip and self.offline and hasattr(sample, "sample_time"): interval_ns = self.interval * (10 ** 9) if self.last_print_time is None: self.last_print_time = sample.sample_time
@@ -137,6 +138,7 @@ class SCTopAnalyzer: def print_current_totals(self): """Print current syscall totals.""" + self.printed = True # Clear terminal if not self.offline: print("\x1b[2J\x1b[H", end="")
@@ -217,8 +219,8 @@ def main(): if args.input: session = perf.session(perf.data(args.input), sample=analyzer.process_event) analyzer.session = session - session.process_events() analyzer.e_machine = getattr(session, "e_machine", None) + session.process_events() else: try: live_session = LiveSession(
@@ -237,7 +239,8 @@ def main(): sys.exit(1) finally: if args.input: - analyzer.print_current_totals() + if not analyzer.printed or analyzer.syscalls: + analyzer.print_current_totals() # Break the reference cycle between perf.session and analyzer.process_event # because perf.session lacks cyclic GC support (tp_traverse). analyzer.session = None
tools/perf/tests/shell/test_sctop_python.sh +25 -23
62 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v1 2/9] perf python sctop: Fix offline interval printing and test flakiness
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf python: sctop offline interval bucketting broken by comm filter --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf python: sctop offline interval bucketting broken by comm filter --
commit b0674a2bbd03e3e74148a12d9d8a3c74b4f35ce5
Author: Ian Rogers <[email protected]>
perf python sctop: Fix offline interval printing and test flakiness
This commit addresses sctop offline interval printing by only advancing the logical clock on non-skipped samples. It also reorders e_machine initialization and updates the shell test with retries and different recording parameters.
Quoted 4 lines
tools/perf/python/sctop.py +0
-0
[ ... ]
tools/perf/python/sctop.py class SCTopAnalyzer: · line 131
Quoted 6 lines
tools/perf/python/sctop.py +1
-1@@ -127,7 +128,7 @@ class SCTopAnalyzer: if not skip and is_enter and 0 <= (syscall_id & ~0x40000000) <= 0xffff: self.syscalls[syscall_id] += 1 - if self.offline and hasattr(sample, "sample_time"): + if not skip and self.offline and hasattr(sample, "sample_time"):
[Severity: Medium]
Does adding 'not skip' to this condition in sctop.py's SCTopAnalyzer.process_event() introduce a regression by blocking the offline clock from advancing during periods where the target process is idle?
By ignoring the sample_time of non-matching events, the script's internal logical clock fails to advance. This prevents the current interval bucket from flushing until the target comm executes another syscall, merging events from multiple intervals into a single bucket and resulting in incorrect interval sums.
Does this also cause the script to miss printing periodic empty tables during idle periods, breaking consistency with live mode?
Quoted 3 lines
tools/perf/python/sctop.py +0
-0interval_ns = self.interval * (10 ** 9) if self.last_print_time is None: self.last_print_time = sample.sample_time
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Ian Rogers
reviewerPATCH 3/9 perf python stat-cpi: Fix live mode signal races and test flakiness
In reply to Ian RogersIn test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a 1-second duration and can exit under heavy parallel load before 'perf stat -p' and 'perf script stat-cpi' finish starting up. In addition, the fixed 'sleep 0.5' before sending SIGINT can fire before Python finishes importing the perf module, opening the live evlist, and flushing the first interval.
In stat-cpi.py, register SIGINT and SIGTERM handlers before calling _open_live_evlist() and pass flush=True when printing live output so redirected stdout is flushed immediately after each interval.
In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the target workload stays alive until killed, and poll the output file for 'cpi' (up to 5 seconds) before sending SIGINT.
--- tools/perf/python/stat-cpi.py | 19 ++++++++++--------- .../perf/tests/shell/test_stat_cpi_python.sh | 12 +++++++++--- 2 files changed, 19 insertions(+), 12 deletions(-)
tools/perf/python/stat-cpi.py +10 -9
@@ -106,7 +106,8 @@ class StatCpiAnalyzer: if ins != 0: cpi = cyc / float(ins) t_sec = timestamp / 1000000000.0 - print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})") + print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})", + flush=True) def read_counters(self, evlist: Any) -> None: """Read counters live."""
@@ -173,19 +174,19 @@ class StatCpiAnalyzer: def run_live(self) -> None: """Read counters live.""" - try: - evlist = self._open_live_evlist() - except OSError as e: - print(f"Failed to open events: {e}", file=sys.stderr) - sys.exit(1) - def handle_signal(_signum: int, _frame: Any) -> None: raise KeyboardInterrupt signal.signal(signal.SIGINT, signal.default_int_handler) signal.signal(signal.SIGTERM, handle_signal) - print("Live mode started. Press Ctrl+C to stop.") + try: + evlist = self._open_live_evlist() + except OSError as e: + print(f"Failed to open events: {e}", file=sys.stderr) + sys.exit(1) + + print("Live mode started. Press Ctrl+C to stop.", flush=True) try: while True: time.sleep(self.args.interval)
@@ -195,7 +196,7 @@ class StatCpiAnalyzer: self.data.clear() self.recorded_pairs.clear() except KeyboardInterrupt: - print("\nStopped.") + print("\nStopped.", flush=True) finally: evlist.close()
tools/perf/tests/shell/test_stat_cpi_python.sh +9 -3
@@ -50,7 +50,7 @@ test_live_mode() { echo "perf stat failed (permissions?), skipping live mode test." return 0 fi - perf test -w noploop & + perf test -w noploop 60 & workload_pid=$! if ! perf stat -e cycles,instructions -p "$workload_pid" -- sleep 0.05 2>/dev/null && \ ! perf stat -e cycles:u,instructions:u -p "$workload_pid" -- sleep 0.05 2>/dev/null; then
@@ -61,10 +61,16 @@ test_live_mode() { fi ran=1 - # Run live mode for 1 interval in the background, give it a tiny sleep, then interrupt + # Run live mode in the background, wait until at least one interval is + # printed, then interrupt. perf script stat-cpi -I 0.1 -p "$workload_pid" > "${temp_out}" & pid=$! - sleep 0.5 + for _ in $(seq 1 50); do + if grep -q "cpi" "${temp_out}"; then + break + fi + sleep 0.1 + done kill -INT "$pid" 2>/dev/null || true set +e wait "$pid"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v1 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
In
test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a 1-second duration and can exit under heavy parallel load before 'perf stat -p' and 'perf script stat-cpi' finish starting up. In addition, the fixed 'sleep 0.5' before sending SIGINT can fire before Python finishes importing the perf module, opening the live evlist, and flushing the first interval.In
stat-cpi.py, register SIGINT and SIGTERM handlers before calling_open_live_evlist()and pass flush=True when printing live output so redirected stdout is flushed immediately after each interval.
Quoted 7 lines
In
test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the target workload stays alive until killed, and poll the output file for 'cpi' (up to 5 seconds) before sending SIGINT.Fixes 4425182d426b ("perf python: Port stat-cpi to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Ian Rogers
reviewerPATCH 4/9 perf test: Deflake Intel PT Python shell tests under load
In reply to Ian RogersIn test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).
Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh to avoid build-id cache and BPF synthesis overhead.
--- .../shell/test_export_to_postgresql_python.sh | 49 ++++++++++-------- .../shell/test_export_to_sqlite_python.sh | 50 +++++++++++-------- .../shell/test_intel_pt_events_python.sh | 43 +++++++++------- 3 files changed, 79 insertions(+), 63 deletions(-)
tools/perf/tests/shell/test_export_to_postgresql_python.sh +27 -22
@@ -61,9 +61,10 @@ test_file_mode() { fi # Generate events with callchains and context switches - if ! perf record -g --switch-events -o "${temp_data}" \ + if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 && \ - ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
@@ -92,29 +93,33 @@ test_file_mode() { test_intel_pt() { echo "Testing export-to-postgresql.py with intel_pt..." - psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true - rm -f "${temp_data}" - # Generate some intel_pt events; use a subshell that waits for uname - if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \ - -- sh -c "uname; true" >/dev/null 2>&1; then - echo "Skipping intel_pt test, intel_pt not available." - return 0 - fi + # Generate some intel_pt events; sleep briefly after uname in the subshell + # so uname's AUX buffer is flushed before SIGCHLD stops perf record. + passed=0 + for _ in 1 2 3 4 5; do + psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true + rm -f "${temp_data}" + if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \ + -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then + echo "Skipping intel_pt test, intel_pt not available." + return 0 + fi - # Run the script with --itrace cr to synthesize call_returns - if ! perf script export-to-postgresql -i "${temp_data}" \ - -o "${temp_db}" --itrace cr >/dev/null; then - echo "intel_pt file mode test failed." - err=1 - else - # Check DB for calls - if ! psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \ + # Run the script with --itrace cr to synthesize call_returns + if perf script export-to-postgresql -i "${temp_data}" \ + -o "${temp_db}" --itrace cr >/dev/null && \ + psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \ grep -q '[1-9]'; then - echo "PostgreSQL intel_pt validation failed (no calls found)." - err=1 - else - echo "intel_pt test passed (cr validated)." + passed=1 + break fi + done + + if [ "$passed" -eq 0 ]; then + echo "PostgreSQL intel_pt validation failed (no calls found)." + err=1 + else + echo "intel_pt test passed (cr validated)." fi }
tools/perf/tests/shell/test_export_to_sqlite_python.sh +28 -22
66 lines not on this page · raw message
tools/perf/tests/shell/test_intel_pt_events_python.sh +24 -19
58 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v1 4/9] perf test: Deflake Intel PT Python shell tests under load
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines
In
test_intel_pt_events_python.sh,test_export_to_sqlite_python.sh, andtest_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in
test_export_to_sqlite_python.shandtest_export_to_postgresql_python.shto avoid build-id cache and BPF synthesis overhead.
Quoted 5 lines
Fixes d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")Fixes 62d350135e67 ("perf python: Port export-to-sqlite to perf module")Fixes b1f968c9656a ("perf python: Port export-to-postgresql to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Ian Rogers
reviewerPATCH 5/9 perf test: Deflake failed-syscalls Python shell tests under load
In reply to Ian RogersWhen running 'perf test' in parallel under heavy system load, a one-shot 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event' can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and ENOENT sys_exit events are captured, while also contending on ~/.debug build-id caching and BPF event synthesis.
Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell after 'ls' so its events are flushed before the parent subshell exits, and wrap the record and check steps in a bounded retry loop (up to 5 attempts).
--- .../test_failed_syscalls_by_pid_python.sh | 68 +++++++++++-------- .../shell/test_failed_syscalls_python.sh | 44 +++++++----- 2 files changed, 67 insertions(+), 45 deletions(-)
tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh +39 -29
@@ -41,8 +41,10 @@ test_file_mode() { echo "Testing failed-syscalls-by-pid.py..." # Check if syscalls:sys_exit is supported/readable - if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then - if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \ + -o /dev/null -- true >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \ + -o /dev/null -- true >/dev/null 2>&1; then echo "Skipping test, no syscalls:sys_exit or raw_syscalls:sys_exit event" exit 2 else
@@ -52,41 +54,49 @@ test_file_mode() { EVENT="syscalls:sys_exit" fi - # Generate some events by running a command that should fail at least some syscall - # (e.g. failing stat on non-existent file). - # Using '|| true' because 'perf record' returns the exit code of 'ls', - # which fails with ENOENT - perf record -e "${EVENT}" -o "${temp_data}" -- ls /does_not_exist >/dev/null 2>&1 || true + # Generate some events by running a command that fails a syscall + # (e.g. failing stat on non-existent file), sleeping briefly in the + # subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed. + passed=0 + for _ in 1 2 3 4 5; do + rm -f "${temp_data}" "${temp_out}" "${temp_out}.comm" "${temp_out}.pid" + perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \ + -- sh -c "ls /does_not_exist 2>/dev/null; sleep 0.05 || true" \ + >/dev/null 2>&1 || true + if [ ! -s "${temp_data}" ]; then + continue + fi + + if perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}" && \ + grep -n -q "err = ENOENT" "${temp_out}" && \ + perf script failed-syscalls-by-pid -i "${temp_data}" \ + "ls" > "${temp_out}.comm" && \ + grep -q "err = ENOENT" "${temp_out}.comm"; then + ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' \ + "${temp_out}.comm" | head -n 1) + if [ -n "${ls_pid}" ] && \ + perf script failed-syscalls-by-pid -i "${temp_data}" \ + "${ls_pid}" > "${temp_out}.pid" && \ + grep -q "err = ENOENT" "${temp_out}.pid"; then + passed=1 + break + fi + fi + done + if [ ! -s "${temp_data}" ]; then echo "Skipping test, perf record failed to create data" exit 2 fi - # Run the script and check output - if ! perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}"; then + if [ "$passed" -eq 0 ]; then echo "failed-syscalls-by-pid test failed." - err=1 - elif ! grep -n -q "err = ENOENT" "${temp_out}"; then - echo "Failed to find expected failed syscalls" - cat "${temp_out}" - err=1 - elif ! perf script failed-syscalls-by-pid -i "${temp_data}" "ls" > "${temp_out}.comm" || \ - ! grep -q "err = ENOENT" "${temp_out}.comm"; then - echo "failed-syscalls-by-pid comm filter test failed." - cat "${temp_out}.comm" + cat "${temp_out}" 2>/dev/null || true + cat "${temp_out}.comm" 2>/dev/null || true + cat "${temp_out}.pid" 2>/dev/null || true err=1 else - ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' "${temp_out}.comm" | head -n 1) - if [ -z "${ls_pid}" ] || \ - ! perf script failed-syscalls-by-pid -i "${temp_data}" \ - "${ls_pid}" > "${temp_out}.pid" || \ - ! grep -q "err = ENOENT" "${temp_out}.pid"; then - echo "failed-syscalls-by-pid PID filter test failed." - cat "${temp_out}.pid" 2>/dev/null || true - err=1 - else - echo "failed-syscalls-by-pid test passed." - fi + echo "failed-syscalls-by-pid test passed." fi rm -f "${temp_out}" "${temp_out}.comm" "${temp_out}.pid" }
tools/perf/tests/shell/test_failed_syscalls_python.sh +28 -16
64 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v1 5/9] perf test: Deflake failed-syscalls Python shell tests under load
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] perf test: TOCTOU symlink vulnerability via rm in /tmp --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] perf test: TOCTOU symlink vulnerability via rm in /tmp
--
commit 2edcd77bcabafd7550e8af4c57dda0ca7cfe9278
Author: Ian Rogers <[email protected]>
perf test: Deflake failed-syscalls Python shell tests under load
This commit updates the failed-syscalls python shell tests to handle heavy system load by adding a bounded retry loop and short sleep intervals. It also explicitly passes the no-bpf-event flags to perf record to prevent early exits during BPF event synthesis.
Quoted 4 lines
tools/perf/tests/shell/test_failed_syscalls_python.sh +0
-0
[ ... ]
tools/perf/tests/shell/test_failed_syscalls_python.sh else · line 61
Quoted 13 lines
tools/perf/tests/shell/test_failed_syscalls_python.sh +5
-4@@ -52,28 +54,38 @@ else EVENT="raw_syscalls:sys_exit" fi -# Run perf record with a command that fails a syscall (ls non-existent file). -# ls exits with non-zero, so perf record returns non-zero exit code of the workload. -perf record -e "${EVENT}" -o "${temp_data}" \ - -- ls /nonexistent_file_for_test >/dev/null 2>&1 || true +# Run perf record with a command that fails a syscall (ls non-existent file), +# sleeping briefly in the subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed. +passed=0 +for _ in 1 2 3 4 5; do + rm -f "${temp_data}" "${temp_out}"
[Severity: High]
Does this rm command expose the script to a time-of-check to time-of-use symlink vulnerability?
The files temp_data and temp_out are initially created securely in the world-writable /tmp directory using mktemp. By explicitly deleting them here in the retry loop, the filenames are freed.
During the window between this deletion and the subsequent shell redirection, a local attacker observing /tmp could recreate these names as symlinks. When the test redirects output or writes to them later, it will follow the symlink and could overwrite arbitrary files on the system. This could lead to local privilege escalation if the test is run with elevated privileges.
Quoted 3 lines
tools/perf/tests/shell/test_failed_syscalls_python.sh +3
-0+ perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \ + -- sh -c "ls /nonexistent_file_for_test 2>/dev/null; sleep 0.05 || true" \ + >/dev/null 2>&1 || true
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Ian Rogers
reviewerPATCH 6/9 perf test: Reduce overhead and contention in Python shell tests
In reply to Ian RogersWhen all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh, test_rw_by_file_python.sh, test_rw_by_pid_python.sh, test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and test_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in test_compaction_times_python.sh.
--- tools/perf/tests/shell/test_check_perf_trace_python.sh | 8 +++++--- tools/perf/tests/shell/test_compaction_times_python.sh | 8 +++++--- .../tests/shell/test_event_analyzing_sample_python.sh | 3 ++- tools/perf/tests/shell/test_flamegraph_python.sh | 5 +++-- tools/perf/tests/shell/test_futex_contention_python.sh | 7 ++++--- tools/perf/tests/shell/test_gecko_python.sh | 3 ++- tools/perf/tests/shell/test_mem_phys_addr_python.sh | 8 +++++--- tools/perf/tests/shell/test_net_dropmonitor_python.sh | 9 +++++---- tools/perf/tests/shell/test_netdev_times_python.sh | 6 +++--- tools/perf/tests/shell/test_powerpc_hcalls_python.sh | 4 ++-- tools/perf/tests/shell/test_rw_by_file_python.sh | 5 +++-- tools/perf/tests/shell/test_rw_by_pid_python.sh | 4 ++-- tools/perf/tests/shell/test_rwtop_python.sh | 4 ++-- tools/perf/tests/shell/test_sched_migration_python.sh | 4 ++-- tools/perf/tests/shell/test_stackcollapse_python.sh | 2 +- .../tests/shell/test_syscall_counts_by_pid_python.sh | 6 +++--- tools/perf/tests/shell/test_syscall_counts_python.sh | 5 +++-- tools/perf/tests/shell/test_task_analyzer.sh | 3 ++- tools/perf/tests/shell/test_wakeup_latency_python.sh | 7 ++++--- 19 files changed, 58 insertions(+), 43 deletions(-)
tools/perf/tests/shell/test_check_perf_trace_python.sh +5 -3
@@ -45,10 +45,11 @@ test_file_mode() { echo "Testing check-perf-trace.py..." events="" - if perf list | grep -q "irq:softirq_entry"; then + tp_list=$(perf list tracepoint) + if echo "$tp_list" | grep -q "irq:softirq_entry"; then events="irq:softirq_entry" fi - if perf list | grep -q "kmem:kmalloc"; then + if echo "$tp_list" | grep -q "kmem:kmalloc"; then if [ -n "$events" ]; then events="$events,kmem:kmalloc,kmem:kfree" else
@@ -62,7 +63,8 @@ test_file_mode() { fi # Generate events - if ! perf record -e "$events" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e "$events" -o "${temp_data}" \ + -- sh -c "sleep 0.1" >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
tools/perf/tests/shell/test_compaction_times_python.sh +5 -3
@@ -44,15 +44,17 @@ test_file_mode() { echo "Testing compaction-times.py..." # Check for any compaction events to see if kernel supports it - if ! perf list | grep -q "compaction:mm_compaction_begin"; then + if ! perf list tracepoint | grep -q "compaction:mm_compaction_begin"; then echo "Skipping test, compaction tracepoints not found" exit 2 fi # Generate some events - # We might not naturally trigger compaction in 0.5s sleep, but the script + # We might not naturally trigger compaction in 0.1s sleep, but the script # should parse the empty or sparse file correctly without crashing. - if ! perf record -e "compaction:*" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event \ + -e "compaction:mm_compaction_begin,compaction:mm_compaction_end" \ + -a -o "${temp_data}" -- sleep 0.1 >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +2 -1
@@ -39,7 +39,8 @@ test_file_mode() { echo "Testing event_analyzing_sample.py..." # Generate some events - if ! perf record -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
tools/perf/tests/shell/test_flamegraph_python.sh +3 -2
@@ -60,7 +60,8 @@ test_file_mode() { echo "Testing flamegraph.py..." # Generate some events with callchains - if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record -g failed (permissions or lack of support)" exit 2 fi
@@ -75,7 +76,7 @@ test_file_mode() { # Run the script in pipe ('-') mode and validate JSON output rm -f "${temp_json}" - if ! perf record -g -o - -- perf test -w noploop 2>/dev/null | \ + if ! perf record -B -N --no-bpf-event -g -o - -- perf test -w noploop 2>/dev/null | \ perf script flamegraph -i - -f json -o "${temp_json}" >/dev/null; then echo "Pipe stdin JSON mode test failed." err=1
tools/perf/tests/shell/test_futex_contention_python.sh +4 -3
@@ -89,14 +89,15 @@ EOF test_file_mode() { echo "Testing futex-contention.py..." # Some systems might not have syscalls:sys_enter_futex - if ! perf list | grep -q syscalls:sys_enter_futex; then + if ! perf list tracepoint | grep -q syscalls:sys_enter_futex; then echo "Skipping file mode test, syscalls:sys_enter_futex not found" return fi # Generate some futex events - if ! perf record -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \ - -- sleep 0.5 2>/dev/null; then + if ! perf record -B -N --no-bpf-event \ + -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \ + -- sleep 0.1 2>/dev/null; then echo "Skipping file mode test (record failed)" return fi
tools/perf/tests/shell/test_gecko_python.sh +2 -1
@@ -39,7 +39,8 @@ test_file_mode() { echo "Testing gecko.py..." # Generate some events with callchains - if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record -g failed (permissions or lack of support)" exit 2 fi
tools/perf/tests/shell/test_mem_phys_addr_python.sh +5 -3
@@ -78,10 +78,12 @@ test_file_mode() { echo "Testing mem-phys-addr.py file mode..." # Generate memory access events (try unprivileged user-space first, then system-wide) - if ! perf record --phys-data -d -o "${temp_data}" \ + if ! perf record -B -N --no-bpf-event --phys-data -d -o "${temp_data}" \ -- perf test -w datasym >/dev/null 2>&1 && \ - ! perf record -d -o "${temp_data}" -- perf test -w datasym >/dev/null 2>&1 && \ - ! perf record -d -a -o "${temp_data}" -- sleep 0.2 >/dev/null 2>&1; then + ! perf record -B -N --no-bpf-event -d -o "${temp_data}" \ + -- perf test -w datasym >/dev/null 2>&1 && \ + ! perf record -B -N --no-bpf-event -d -a -o "${temp_data}" \ + -- sleep 0.1 >/dev/null 2>&1; then echo "Skipping file mode record test, perf record -d not supported" return 0 fi
tools/perf/tests/shell/test_net_dropmonitor_python.sh +5 -4
@@ -42,11 +42,12 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing net_dropmonitor.py..." # Create a perf.data file. Force dropping a packet if tracepoint is available! -if ! perf record -e skb:kfree_skb -o "${temp_data}" -a \ - -- ping -c 1 255.255.255.255 >/dev/null 2>&1; then - if ! perf record -e skb:kfree_skb -o "${temp_data}" \ +if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" -a \ + -- ping -c 1 -W 1 255.255.255.255 >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" \ -- sleep 0.1 >/dev/null 2>&1; then - if ! perf record -o "${temp_data}" -- uname >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -o "${temp_data}" \ + -- uname >/dev/null 2>&1; then echo "Skipping test, cannot record perf events" exit 2 fi
tools/perf/tests/shell/test_netdev_times_python.sh +3 -3
@@ -85,9 +85,9 @@ then fi # Create a perf.data file. Force dropping a packet if tracepoint is available! -if ! perf record -e skb:kfree_skb -a -o "${temp_data}" \ - -- ping -c 1 127.0.0.1 >/dev/null 2>&1; then - perf record -e cycles -o "${temp_data}" \ +if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -a -o "${temp_data}" \ + -- ping -c 1 -W 1 127.0.0.1 >/dev/null 2>&1; then + perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } fi
tools/perf/tests/shell/test_powerpc_hcalls_python.sh +2 -2
@@ -67,8 +67,8 @@ if ! grep -q "H_REMOVE.*1.*1500.*1500.*1500" "${temp_out}"; then fi # Create a perf.data file if powerpc hcall tracepoints are available on this host. -if ! perf record -e powerpc:hcall_entry,powerpc:hcall_exit -a -o "${temp_data}" \ - -- perf test -w noploop >/dev/null 2>&1; then +if ! perf record -B -N --no-bpf-event -e powerpc:hcall_entry,powerpc:hcall_exit \ + -a -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping live record test, powerpc hcall tracepoints not available" exit 0 fi
tools/perf/tests/shell/test_rw_by_file_python.sh +3 -2
@@ -37,8 +37,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing rw-by-file.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "syscalls:sys_enter_read"; then - perf record -e syscalls:sys_enter_read,syscalls:sys_enter_write -a -o "${temp_data}" \ +if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then + perf record -B -N --no-bpf-event -e syscalls:sys_enter_read,syscalls:sys_enter_write \ + -o "${temp_data}" \ -- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
tools/perf/tests/shell/test_rw_by_pid_python.sh +2 -2
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing rw-by-pid.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "syscalls:sys_enter_read"; then +if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then ev="syscalls:sys_enter_read,syscalls:sys_exit_read" ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write" - perf record -e "$ev" -a -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \ -- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
tools/perf/tests/shell/test_rwtop_python.sh +2 -2
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing rwtop.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "syscalls:sys_enter_read"; then +if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then ev="syscalls:sys_enter_read,syscalls:sys_exit_read" ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write" - perf record -e "$ev" -a -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \ -- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
tools/perf/tests/shell/test_sched_migration_python.sh +2 -2
@@ -44,10 +44,10 @@ echo "Testing sched-migration.py..." ev="sched:sched_switch,sched:sched_migrate_task" ev="${ev},sched:sched_wakeup_new,sched:sched_wakeup" has_sched=1 -if ! perf record -e "$ev" -a -o "${temp_data}" \ +if ! perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \ -- sleep 0.1 >/dev/null 2>&1; then has_sched=0 - perf record -e cycles -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } fi
tools/perf/tests/shell/test_stackcollapse_python.sh +1 -1
@@ -37,7 +37,7 @@ echo "Testing stackcollapse.py..." # Create a perf.data file with callchains. Use a busy workload rather than # sleep, as an idle system may not generate any samples at all. -perf record -g -o "${temp_data}" \ +perf record -B -N --no-bpf-event -g -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; }
tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh +3 -3
@@ -43,14 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) test_file_mode() { echo "Testing syscall-counts-by-pid.py..." # Some systems might not have raw_syscalls:sys_enter - if ! perf list | grep -q raw_syscalls:sys_enter; then + if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then echo "Skipping test, raw_syscalls:sys_enter not found" exit 2 fi # Generate some syscall events - perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \ - -- sleep 0.5 >/dev/null 2>&1 || \ + perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \ + -- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } if ! perf script syscall-counts-by-pid -i "${temp_data}" > "${temp_out}"; then
tools/perf/tests/shell/test_syscall_counts_python.sh +3 -2
@@ -43,13 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) test_file_mode() { echo "Testing syscall-counts.py..." # Some systems might not have raw_syscalls:sys_enter (e.g. stripped kernels or permissions) - if ! perf list | grep -q raw_syscalls:sys_enter; then + if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then echo "Skipping test, raw_syscalls:sys_enter not found" exit 2 fi # Generate some syscall events - if ! perf record -e raw_syscalls:sys_enter -o "${temp_data}" -- sleep 0.5 2>/dev/null; then + if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \ + -- sh -c "sleep 0.1; sleep 0.05" 2>/dev/null; then echo "perf record failed (permissions?), skipping file mode test." exit 2 fi
tools/perf/tests/shell/test_task_analyzer.sh +2 -1
@@ -60,7 +60,8 @@ skip_no_probe_record_support() { prepare_perf_data() { # 1s should be sufficient to catch at least some switches - perf record -e sched:sched_switch -a -o "${perfdata}" -- sleep 1 > /dev/null 2>&1 + perf record -B -N --no-bpf-event -e sched:sched_switch -a -o "${perfdata}" \ + -- sleep 1 > /dev/null 2>&1 # check if perf data file got created in above step. if [ ! -e "${perfdata}" ]; then printf "FAIL: perf record failed to create \"${perfdata}\" \n"
tools/perf/tests/shell/test_wakeup_latency_python.sh +4 -3
@@ -41,9 +41,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing wakeup-latency.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "sched:sched_wakeup"; then +if perf list tracepoint | grep -q "sched:sched_wakeup"; then ev="sched:sched_wakeup,sched:sched_wakeup_new,sched:sched_switch" - perf record -e "$ev" -a -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \ -- sleep 0.1 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
@@ -70,7 +70,8 @@ else fi # Also test zero-wakeups / unhandled events path to verify division-by-zero protection -if perf record -e cycles -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then +if perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then if ! perf script wakeup-latency -i "${temp_data}" > "${temp_out}" || \ ! grep -q "avg_wakeup_latency (ns): N/A" "${temp_out}"; then echo "wakeup-latency zero-wakeups guard test failed"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v1 6/9] perf test: Reduce overhead and contention in Python shell tests
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 18 lines
When all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' intest_check_perf_trace_python.sh,test_rw_by_file_python.sh,test_rw_by_pid_python.sh,test_rwtop_python.sh, andtest_syscall_counts_by_pid_python.shwhere only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' intest_net_dropmonitor_python.shandtest_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' intest_compaction_times_python.sh.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Ian Rogers
reviewerPATCH 7/9 perf python event_analyzing_sample: Default to in-memory SQLite database
In reply to Ian RogersWhen -d/--db is not specified, event_analyzing_sample.py created a temporary file in /tmp via tempfile.mkstemp() and deleted it in trace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.
Default to sqlite3.connect(":memory:") when db_path is not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode in test_event_analyzing_sample_python.sh.
--- tools/perf/python/event_analyzing_sample.py | 35 ++++++------------- .../test_event_analyzing_sample_python.sh | 8 ++++- 2 files changed, 17 insertions(+), 26 deletions(-)
tools/perf/python/event_analyzing_sample.py +10 -25
@@ -17,10 +17,8 @@ from __future__ import annotations import argparse import math -import os import sqlite3 import struct -import tempfile from typing import Any import perf
@@ -117,16 +115,11 @@ session: Any = None class _DB: con: sqlite3.Connection | None = None - temp_path: str | None = None def trace_begin(db_path: str | None = None) -> None: """Initialize database tables.""" print("In trace_begin:\n") - if not db_path: - fd, db_path = tempfile.mkstemp(prefix="perf_events_", suffix=".db") - os.close(fd) - _DB.temp_path = db_path - con = sqlite3.connect(db_path) + con = sqlite3.connect(db_path or ":memory:") try: # Drop any pre-existing tables so repeated runs do not accumulate duplicate events. con.execute("drop table if exists gen_events;")
@@ -297,28 +290,20 @@ def show_pebs_ll() -> None: def trace_end() -> None: """Called at the end of trace processing.""" print("In trace_end:\n") - try: - if _DB.con: - try: - _DB.con.commit() - show_general_events() - show_pebs_ll() - finally: - _DB.con.close() - _DB.con = None - finally: - if _DB.temp_path and os.path.exists(_DB.temp_path): - try: - os.remove(_DB.temp_path) - except OSError: - pass - _DB.temp_path = None + if _DB.con: + try: + _DB.con.commit() + show_general_events() + show_pebs_ll() + finally: + _DB.con.close() + _DB.con = None if __name__ == "__main__": ap = argparse.ArgumentParser(description="Analyze events with SQLite") ap.add_argument("-i", "--input", default="perf.data", help="Input file name") ap.add_argument("-d", "--db", "--database", dest="database", default=None, - help="Database file name (defaults to a temporary file cleaned up on exit)") + help="Database file name (defaults to an in-memory database)") args = ap.parse_args() try:
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +7 -1
@@ -45,8 +45,14 @@ test_file_mode() { exit 2 fi - # Run the script + # Run the script with default (:memory:) database and with explicit -d path if ! perf script event_analyzing_sample -i "${temp_data}" \ + > "${temp_dir}/perf.mem.out" 2>&1 || \ + ! grep -q "Statistics about the general events" "${temp_dir}/perf.mem.out" || \ + grep -q "Error creating/inserting event" "${temp_dir}/perf.mem.out"; then + echo "Default in-memory database mode test failed." + err=1 + elif ! perf script event_analyzing_sample -i "${temp_data}" \ -d "${temp_db}" > "${temp_dir}/perf.out" 2>&1; then echo "File mode test failed." err=1
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v1 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines
When -d/--db is not specified,
event_analyzing_sample.pycreated a temporary file in /tmp viatempfile.mkstemp()and deleted it intrace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.
Quoted 8 lines
Default to
sqlite3.connect(":memory:")whendb_pathis not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode intest_event_analyzing_sample_python.sh.Fixes eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Ian Rogers
reviewerPATCH 8/9 perf python: Initialize debug output on module load
In reply to Ian RogersUnlike the perf binary's main(), PyInit_perf() does not call perf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hits debug_file() with _debug_file == NULL and emits 'debug_file not set' without a trailing newline before the actual diagnostic message.
Call perf_debug_setup() in PyInit_perf() and add the missing trailing newline to the fallback warning in debug_file().
--- tools/perf/util/debug.c | 2 +- tools/perf/util/python.c | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
tools/perf/util/debug.c +1 -1
@@ -55,7 +55,7 @@ FILE *debug_file(void) {if (!_debug_file) {debug_set_file (stderr); -pr_warning_once ("debug_file not set" ); +pr_warning_once ("%s not set\n" , __func__); }return _debug_file; }
tools/perf/util/python.c +1 -0
@@ -5157,6 +5157,7 @@ PyMODINIT_FUNC PyInit_perf(void)/* The page_size is placed in util object. */ page_size =sysconf (_SC_PAGE_SIZE); +perf_debug_setup ();Py_INCREF (&pyrf_evlist__type);PyModule_AddObject (module,"evlist" , (PyObject *)&pyrf_evlist__type);
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v1 8/9] perf python: Initialize debug output on module load
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines
Unlike the perf binary's
main(),PyInit_perf()does not callperf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hitsdebug_file()with_debug_file== NULL and emits 'debug_filenot set' without a trailing newline before the actual diagnostic message.
Quoted 6 lines
Call
perf_debug_setup()inPyInit_perf()and add the missing trailing newline to the fallback warning indebug_file().Fixes ec49230cf6dd ("perf debug: Expose debug file")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Ian Rogers
reviewerPATCH 9/9 perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
In reply to Ian Rogersperf_pmus__print_pmu_events() first counts events via perf_pmu__num_events() to allocate the aliases array, then populates it via perf_pmu__for_each_event(). When dynamic tracepoints (such as kprobes or uprobes during 'perf test' runs) are concurrently created or removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len, leaving trailing zeroed entries in aliases[] with name == NULL and pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning /sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns -ENOENT and aborts enumeration of all remaining tracepoint subsystems.
- If an event is added, perf_pmus__print_pmu_events__callback() aborts enumeration when state->index reaches state->aliases_len.
Grow state->aliases dynamically via realloc() when needed, use state.index as the actual populated length for sorting and printing, and ignore -ENOENT when a tracepoint subsystem directory disappears during enumeration.
--- tools/perf/util/pmus.c | 20 ++++++++++++++++---- tools/perf/util/tp_pmu.c | 8 ++++++-- 2 files changed, 22 insertions(+), 6 deletions(-)
tools/perf/util/pmus.c +16 -4
@@ -571,7 +571,7 @@ static int cmp_sevent(const void *a, const void *b) }/* Order by event name. */ -return strcmp (as->name, bs->name); +return strcmp (as->name ?:"" , bs->name ?:"" ); }static bool pmu_alias_is_duplicate (struct sevent *a,struct sevent *b)
@@ -581,7 +581,7 @@ static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)return false;/* Don't remove duplicates for different PMUs */ -return strcmp (a->pmu_name, b->pmu_name) ==0 ; +return strcmp (a->pmu_name ?:"" , b->pmu_name ?:"" ) ==0 ; }struct events_callback_state {
@@ -597,8 +597,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,struct sevent *s;if (state->index >= state->aliases_len) { -pr_err ("Unexpected event %s/%s/\n" , info->pmu->name, info->name); -return 1 ; +size_t new_len =max_t (size_t,16 , state->aliases_len *2 ); +struct sevent *new_aliases; + + new_aliases =realloc (state->aliases, new_len *sizeof (struct sevent )); +if (!new_aliases) { +pr_err ("Unexpected event %s/%s/\n" , info->pmu->name, info->name); +return 1 ; + } +memset (&new_aliases[state->aliases_len],0 , + (new_len - state->aliases_len) *sizeof (struct sevent )); + state->aliases = new_aliases; + state->aliases_len = new_len; }assert (info->pmu !=NULL || info->name !=NULL ); s = &state->aliases[state->index];
@@ -654,6 +664,8 @@ void perf_pmus__print_pmu_events(const struct print_callbacks *print_cb, void *pperf_pmu__for_each_event (pmu, skip_duplicate_pmus, &state, perf_pmus__print_pmu_events__callback); } + aliases = state.aliases; + len = state.index;qsort (aliases, len,sizeof (struct sevent ), cmp_sevent);for (int j =0 ; j < len; j++) {/* Skip duplicates */
tools/perf/util/tp_pmu.c +6 -2
@@ -151,7 +151,9 @@ static int for_each_event_cb(void *state, const char *sys_name, const char *evt_static int for_each_event_sys_cb (void *state,const char *sys_name) { -return tp_pmu__for_each_tp_event (sys_name, state, for_each_event_cb); +int ret =tp_pmu__for_each_tp_event (sys_name, state, for_each_event_cb); + +return ret == -ENOENT ?0 : ret; }int tp_pmu__for_each_event (struct perf_pmu *pmu,void *state,pmu_event_callback cb)
@@ -176,7 +178,9 @@ static int num_events_cb(void *state, const char *sys_name __maybe_unused,static int num_events_sys_cb (void *state,const char *sys_name) { -return tp_pmu__for_each_tp_event (sys_name, state, num_events_cb); +int ret =tp_pmu__for_each_tp_event (sys_name, state, num_events_cb); + +return ret == -ENOENT ?0 : ret; }size_t tp_pmu__num_events (struct perf_pmu *pmu__maybe_unused )
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v1 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] perf: missing explicit <stdlib.h> inclusion for realloc() --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] perf: missing explicit <stdlib.h> inclusion for realloc()
--
commit 156bff4d99dfcb5bfd8c7cc38a4671357c5113e3
Author: Ian Rogers <[email protected]>
perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
This commit modifies the perf tool to handle concurrent addition and removal of tracepoint events by dynamically reallocating the aliases array and handling -ENOENT gracefully.
Quoted 4 lines
tools/perf/util/pmus.c +0
-0
[ ... ]
tools/perf/util/pmus.c static int perf_pmus__print_pmu_events__callback(void *vstate, · line 603
Quoted 10 lines
tools/perf/util/pmus.c +4
-2@@ -597,8 +597,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,struct sevent *s;if (state->index >= state->aliases_len) { -pr_err ("Unexpected event %s/%s/\n" , info->pmu->name, info->name); -return 1 ; +size_t new_len =max_t (size_t,16 , state->aliases_len *2 ); +struct sevent *new_aliases; + + new_aliases =realloc (state->aliases, new_len *sizeof (struct sevent ));
[Severity: Low]
Does this code in perf_pmus__print_pmu_events__callback() rely on implicit header inclusions for realloc()?
Since the perf tool can be built against musl libc, which strictly separates declarations and does not implicitly include <stdlib.h> through other headers, should <stdlib.h> be explicitly included at the top of tools/perf/util/pmus.c to prevent compilation failures?
Quoted 4 lines
tools/perf/util/pmus.c +4
-0+if (!new_aliases) { +pr_err ("Unexpected event %s/%s/\n" , info->pmu->name, info->name); +return 1 ; + }
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Ian Rogers
reviewerv2 cover perf python: Follow-up fixes and shell test deflaking
In reply to Ian RogersThis series contains follow-up fixes for the standalone Python scripts and their shell tests, particularly when running 'perf test' as root under heavy parallel load:
- Clamp sample_id size calculation for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE records to work around kernels without "perf/core: Restore header fields in sideband output callbacks" [1].
- Fix offline interval printing in sctop.py and live signal races in stat-cpi.py.
- Deflake Intel PT, failed-syscalls, and other Python shell tests under parallel 'perf test' load by scoping tracepoint checks to 'perf list tracepoint', passing '-B -N --no-bpf-event' to 'perf record', and avoiding unnecessary system-wide ('-a') recordings.
- Default event_analyzing_sample.py to an in-memory SQLite database (':memory:') when '--db' is not specified.
- Initialize perf debug output ('perf_debug_setup()') when loading the 'perf' Python extension module.
- Fix races in 'perf list' ('perf_pmus__print_pmu_events()' and 'tp_pmu.c') when concurrent tests dynamically create and remove kprobe/uprobes tracepoints.
v2:
- Fix 8-byte alignment calculation for PERF_RECORD_TEXT_POKE payload in evsel__event_size() by aligning fixed + old_len + new_len.
- Keep offline interval clock advancing on all samples in sctop.py while flushing any remaining syscalls at EOF.
- Use 'mktemp -d' private directories in test_sctop_python.sh and test_failed_syscalls_python.sh so temporary files are not unlinked and recreated directly in /tmp.
- Explicitly include <stdlib.h> in tools/perf/util/pmus.c.
[1] https://lore.kernel.org/r/[email protected]
Ian Rogers (9):
perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke
events
perf python sctop: Fix offline interval printing and test flakiness perf python stat-cpi: Fix live mode signal races and test flakiness perf test: Deflake Intel PT Python shell tests under load perf test: Deflake failed-syscalls Python shell tests under load perf test: Reduce overhead and contention in Python shell tests perf python event_analyzing_sample: Default to in-memory SQLite
database
perf python: Initialize debug output on module load
perf pmu: Fix race with concurrent tracepoint creation and removal in
perf list
tools/perf/python/event_analyzing_sample.py | 35 +++------- tools/perf/python/sctop.py | 7 +- tools/perf/python/stat-cpi.py | 19 +++--- .../shell/test_check_perf_trace_python.sh | 8 ++- .../shell/test_compaction_times_python.sh | 8 ++- .../test_event_analyzing_sample_python.sh | 11 ++- .../shell/test_export_to_postgresql_python.sh | 49 +++++++------ .../shell/test_export_to_sqlite_python.sh | 50 ++++++++------ .../test_failed_syscalls_by_pid_python.sh | 68 +++++++++++-------- .../shell/test_failed_syscalls_python.sh | 54 +++++++++------ .../tests/shell/test_flamegraph_python.sh | 5 +- .../shell/test_futex_contention_python.sh | 7 +- tools/perf/tests/shell/test_gecko_python.sh | 3 +- .../shell/test_intel_pt_events_python.sh | 43 ++++++------ .../tests/shell/test_mem_phys_addr_python.sh | 8 ++- .../shell/test_net_dropmonitor_python.sh | 9 +-- .../tests/shell/test_netdev_times_python.sh | 6 +- .../tests/shell/test_powerpc_hcalls_python.sh | 4 +- .../tests/shell/test_rw_by_file_python.sh | 5 +- .../perf/tests/shell/test_rw_by_pid_python.sh | 4 +- tools/perf/tests/shell/test_rwtop_python.sh | 4 +- .../shell/test_sched_migration_python.sh | 4 +- tools/perf/tests/shell/test_sctop_python.sh | 58 ++++++++-------- .../tests/shell/test_stackcollapse_python.sh | 2 +- .../perf/tests/shell/test_stat_cpi_python.sh | 12 +++- .../test_syscall_counts_by_pid_python.sh | 6 +- .../tests/shell/test_syscall_counts_python.sh | 5 +- tools/perf/tests/shell/test_task_analyzer.sh | 3 +- .../tests/shell/test_wakeup_latency_python.sh | 7 +- tools/perf/util/debug.c | 2 +- tools/perf/util/evlist.c | 6 +- tools/perf/util/evsel.c | 59 +++++++++++++++- tools/perf/util/evsel.h | 1 + tools/perf/util/pmus.c | 21 ++++-- tools/perf/util/python.c | 1 + tools/perf/util/tp_pmu.c | 8 ++- 36 files changed, 366 insertions(+), 236 deletions(-)
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Ian Rogers
reviewerPATCH v2 1/9 perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
On kernels without the fix ("perf/core: Restore header fields in sideband output callbacks") to restore event_id.header.size in perf_event_ksymbol_output(), perf_event_bpf_output(), and perf_event_text_poke_output(), concurrent perf sessions cause those sideband records to be emitted with header.size inflated by multiple id_header_size increments while the single id_sample is written immediately after the event payload. Indexing backwards from event->header.size reads uninitialized ring-buffer bytes at the end of the record, causing evlist__event2evsel() to fail with -EFAULT.
Add evsel__event_size() to clamp the effective size used to locate the trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving event->header.size intact for advancing the ring-buffer/file stream.
--- tools/perf/util/evlist.c | 6 ++-- tools/perf/util/evsel.c | 59 +++++++++++++++++++++++++++++++++++++++- tools/perf/util/evsel.h | 1 + 3 files changed, 63 insertions(+), 3 deletions(-)
tools/perf/util/evlist.c +4 -2
@@ -973,13 +973,15 @@ static int evlist__event2id(struct evlist *evlist, union perf_event *event, u64const __u64 *array = event->sample.array;ssize_t n; - n = (event->header.size -sizeof (event->header)) >>3 ; -if (event->header.type ==PERF_RECORD_SAMPLE ) { + n = (event->header.size -sizeof (event->header)) >>3 ;if (evlist__id_pos (evlist) >= n)return -1 ; *id = array[evlist__id_pos (evlist)]; }else { +u16 size =evsel__event_size (evlist__first (evlist), event); + + n = (size -sizeof (event->header)) >>3 ;if (evlist__is_pos (evlist) > n)return -1 ; n -=evlist__is_pos (evlist);
tools/perf/util/evsel.c +58 -1
71 lines not on this page · raw message
tools/perf/util/evsel.h +1 -0
@@ -469,6 +469,7 @@ int evsel__parse_sample_timestamp(struct evsel *evsel, union perf_event *event, u64 *timestamp);u16 evsel__id_hdr_size (const struct evsel *evsel); +u16 evsel__event_size (const struct evsel *evsel,const union perf_event *event);static inline struct evsel *evsel__next (struct evsel *evsel) {
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v2 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines
On kernels without the fix ("perf/core: Restore header fields in sideband output callbacks") to restore event_id.header.size in
perf_event_ksymbol_output(),perf_event_bpf_output(), andperf_event_text_poke_output(), concurrent perf sessions cause those sideband records to be emitted with header.size inflated by multipleid_header_sizeincrements while the singleid_sampleis written immediately after the event payload. Indexing backwards fromevent->header.sizereads uninitialized ring-buffer bytes at the end of the record, causingevlist__event2evsel()to fail with-EFAULT.Add
evsel__event_size()to clamp the effective size used to locate the trailingid_sampleforPERF_RECORD_KSYMBOL,PERF_RECORD_BPF_EVENT, andPERF_RECORD_TEXT_POKEto payload +id_hdr_sizewhile leavingevent->header.sizeintact for advancing the ring-buffer/file stream.
Quoted 6 lines
Fixes 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")Fixes 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")Fixes 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Ian Rogers
reviewerPATCH v2 2/9 perf python sctop: Fix offline interval printing and test flakiness
In reply to Ian RogersIn sctop.py:
- If an earlier interval elapsed and printed an empty table before the target comm ('sleep') executed any syscalls, analyzer.printed became True and the final partial interval containing the target comm's syscalls was never flushed at EOF. Flush print_current_totals() in finally when analyzer.syscalls is non-empty as well as when nothing has been printed yet.
- Initialize analyzer.e_machine after creating perf.session rather than when session is still None.
In test_sctop_python.sh:
- Use a private temporary directory via 'mktemp -d'.
- Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a bounded retry loop so 'sleep's PERF_RECORD_COMM and sys_enter events are reliably captured under heavy load.
--- tools/perf/python/sctop.py | 7 ++- tools/perf/tests/shell/test_sctop_python.sh | 58 ++++++++++----------- 2 files changed, 34 insertions(+), 31 deletions(-)
tools/perf/python/sctop.py +5 -2
@@ -37,6 +37,7 @@ class SCTopAnalyzer: self.offline = offline self.own_pid = os.getpid() self.last_print_time: Optional[int] = None + self.printed = False self.session: Optional[perf.session] = None self.e_machine: Optional[int] = None
@@ -137,6 +138,7 @@ class SCTopAnalyzer: def print_current_totals(self): """Print current syscall totals.""" + self.printed = True # Clear terminal if not self.offline: print("\x1b[2J\x1b[H", end="")
@@ -217,8 +219,8 @@ def main(): if args.input: session = perf.session(perf.data(args.input), sample=analyzer.process_event) analyzer.session = session - session.process_events() analyzer.e_machine = getattr(session, "e_machine", None) + session.process_events() else: try: live_session = LiveSession(
@@ -237,7 +239,8 @@ def main(): sys.exit(1) finally: if args.input: - analyzer.print_current_totals() + if not analyzer.printed or analyzer.syscalls: + analyzer.print_current_totals() # Break the reference cycle between perf.session and analyzer.process_event # because perf.session lacks cyclic GC support (tp_traverse). analyzer.session = None
tools/perf/tests/shell/test_sctop_python.sh +29 -29
80 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v2 2/9] perf python sctop: Fix offline interval printing and test flakiness
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines
In
sctop.py:
- If an earlier interval elapsed and printed an empty table before the target comm ('sleep') executed any syscalls, analyzer.printed became True and the final partial interval containing the target comm's syscalls was never flushed at EOF. Flushprint_current_totals()in finally when analyzer.syscalls is non-empty as well as when nothing has been printed yet.
- Initialize analyzer.e_machine after creating perf.session rather than when session is still None.In
test_sctop_python.sh:
- Use a private temporary directory via 'mktemp -d'.
- Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a bounded retry loop so 'sleep'sPERF_RECORD_COMMandsys_enterevents are reliably captured under heavy load.
Quoted 3 lines
Fixes b83f0bacf5e4 ("perf python: Port sctop to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Ian Rogers
reviewerPATCH v2 3/9 perf python stat-cpi: Fix live mode signal races and test flakiness
In reply to Ian RogersIn test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a 1-second duration and can exit under heavy parallel load before 'perf stat -p' and 'perf script stat-cpi' finish starting up. In addition, the fixed 'sleep 0.5' before sending SIGINT can fire before Python finishes importing the perf module, opening the live evlist, and flushing the first interval.
In stat-cpi.py, register SIGINT and SIGTERM handlers before calling _open_live_evlist() and pass flush=True when printing live output so redirected stdout is flushed immediately after each interval.
In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the target workload stays alive until killed, and poll the output file for 'cpi' (up to 5 seconds) before sending SIGINT.
--- tools/perf/python/stat-cpi.py | 19 ++++++++++--------- .../perf/tests/shell/test_stat_cpi_python.sh | 12 +++++++++--- 2 files changed, 19 insertions(+), 12 deletions(-)
tools/perf/python/stat-cpi.py +10 -9
@@ -106,7 +106,8 @@ class StatCpiAnalyzer: if ins != 0: cpi = cyc / float(ins) t_sec = timestamp / 1000000000.0 - print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})") + print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})", + flush=True) def read_counters(self, evlist: Any) -> None: """Read counters live."""
@@ -173,19 +174,19 @@ class StatCpiAnalyzer: def run_live(self) -> None: """Read counters live.""" - try: - evlist = self._open_live_evlist() - except OSError as e: - print(f"Failed to open events: {e}", file=sys.stderr) - sys.exit(1) - def handle_signal(_signum: int, _frame: Any) -> None: raise KeyboardInterrupt signal.signal(signal.SIGINT, signal.default_int_handler) signal.signal(signal.SIGTERM, handle_signal) - print("Live mode started. Press Ctrl+C to stop.") + try: + evlist = self._open_live_evlist() + except OSError as e: + print(f"Failed to open events: {e}", file=sys.stderr) + sys.exit(1) + + print("Live mode started. Press Ctrl+C to stop.", flush=True) try: while True: time.sleep(self.args.interval)
@@ -195,7 +196,7 @@ class StatCpiAnalyzer: self.data.clear() self.recorded_pairs.clear() except KeyboardInterrupt: - print("\nStopped.") + print("\nStopped.", flush=True) finally: evlist.close()
tools/perf/tests/shell/test_stat_cpi_python.sh +9 -3
@@ -50,7 +50,7 @@ test_live_mode() { echo "perf stat failed (permissions?), skipping live mode test." return 0 fi - perf test -w noploop & + perf test -w noploop 60 & workload_pid=$! if ! perf stat -e cycles,instructions -p "$workload_pid" -- sleep 0.05 2>/dev/null && \ ! perf stat -e cycles:u,instructions:u -p "$workload_pid" -- sleep 0.05 2>/dev/null; then
@@ -61,10 +61,16 @@ test_live_mode() { fi ran=1 - # Run live mode for 1 interval in the background, give it a tiny sleep, then interrupt + # Run live mode in the background, wait until at least one interval is + # printed, then interrupt. perf script stat-cpi -I 0.1 -p "$workload_pid" > "${temp_out}" & pid=$! - sleep 0.5 + for _ in $(seq 1 50); do + if grep -q "cpi" "${temp_out}"; then + break + fi + sleep 0.1 + done kill -INT "$pid" 2>/dev/null || true set +e wait "$pid"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v2 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf python: unhandled KeyboardInterrupt in stat-cpi init --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf python: unhandled KeyboardInterrupt in stat-cpi init
--
commit 1a071f63d2155dda63d4eb450e190a0e0b0b2876
Author: Ian Rogers <[email protected]>
perf python stat-cpi: Fix live mode signal races and test flakiness
This commit attempts to fix live mode signal races by registering signal handlers earlier and flushing output to prevent test flakiness. It also modifies the shell test to run the workload longer and poll for output before sending SIGINT.
Quoted 4 lines
tools/perf/python/stat-cpi.py +0
-0
[ ... ]
Quoted 17 lines
tools/perf/python/stat-cpi.py +0
-7@@ -173,19 +174,19 @@ class StatCpiAnalyzer: def run_live(self) -> None: """Read counters live.""" - try: - evlist = self._open_live_evlist() - except OSError as e: - print(f"Failed to open events: {e}", file=sys.stderr) - sys.exit(1) - def handle_signal(_signum: int, _frame: Any) -> None: raise KeyboardInterrupt signal.signal(signal.SIGINT, signal.default_int_handler) signal.signal(signal.SIGTERM, handle_signal) - print("Live mode started. Press Ctrl+C to stop.")
tools/perf/python/stat-cpi.py class StatCpiAnalyzer: · line 192
Quoted 10 lines
tools/perf/python/stat-cpi.py +7
-0+ try: + evlist = self._open_live_evlist() + except OSError as e: + print(f"Failed to open events: {e}", file=sys.stderr) + sys.exit(1) + + print("Live mode started. Press Ctrl+C to stop.", flush=True) try: while True: time.sleep(self.args.interval)
[Severity: Medium]
Will this crash if a signal is received during _open_live_evlist()?
The signal handlers are registered to raise KeyboardInterrupt before _open_live_evlist() is called, but the try...except KeyboardInterrupt block below only wraps the while loop.
If SIGINT or SIGTERM is received during this initialization phase, wouldn't it result in an unhandled KeyboardInterrupt exception, printing a traceback and bypassing graceful cleanup?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Ian Rogers
reviewerPATCH v2 4/9 perf test: Deflake Intel PT Python shell tests under load
In reply to Ian RogersIn test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).
Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh to avoid build-id cache and BPF synthesis overhead.
--- .../shell/test_export_to_postgresql_python.sh | 49 ++++++++++-------- .../shell/test_export_to_sqlite_python.sh | 50 +++++++++++-------- .../shell/test_intel_pt_events_python.sh | 43 +++++++++------- 3 files changed, 79 insertions(+), 63 deletions(-)
tools/perf/tests/shell/test_export_to_postgresql_python.sh +27 -22
@@ -61,9 +61,10 @@ test_file_mode() { fi # Generate events with callchains and context switches - if ! perf record -g --switch-events -o "${temp_data}" \ + if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 && \ - ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
@@ -92,29 +93,33 @@ test_file_mode() { test_intel_pt() { echo "Testing export-to-postgresql.py with intel_pt..." - psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true - rm -f "${temp_data}" - # Generate some intel_pt events; use a subshell that waits for uname - if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \ - -- sh -c "uname; true" >/dev/null 2>&1; then - echo "Skipping intel_pt test, intel_pt not available." - return 0 - fi + # Generate some intel_pt events; sleep briefly after uname in the subshell + # so uname's AUX buffer is flushed before SIGCHLD stops perf record. + passed=0 + for _ in 1 2 3 4 5; do + psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true + rm -f "${temp_data}" + if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \ + -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then + echo "Skipping intel_pt test, intel_pt not available." + return 0 + fi - # Run the script with --itrace cr to synthesize call_returns - if ! perf script export-to-postgresql -i "${temp_data}" \ - -o "${temp_db}" --itrace cr >/dev/null; then - echo "intel_pt file mode test failed." - err=1 - else - # Check DB for calls - if ! psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \ + # Run the script with --itrace cr to synthesize call_returns + if perf script export-to-postgresql -i "${temp_data}" \ + -o "${temp_db}" --itrace cr >/dev/null && \ + psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \ grep -q '[1-9]'; then - echo "PostgreSQL intel_pt validation failed (no calls found)." - err=1 - else - echo "intel_pt test passed (cr validated)." + passed=1 + break fi + done + + if [ "$passed" -eq 0 ]; then + echo "PostgreSQL intel_pt validation failed (no calls found)." + err=1 + else + echo "intel_pt test passed (cr validated)." fi }
tools/perf/tests/shell/test_export_to_sqlite_python.sh +28 -22
66 lines not on this page · raw message
tools/perf/tests/shell/test_intel_pt_events_python.sh +24 -19
58 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v2 4/9] perf test: Deflake Intel PT Python shell tests under load
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines
In
test_intel_pt_events_python.sh,test_export_to_sqlite_python.sh, andtest_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in
test_export_to_sqlite_python.shandtest_export_to_postgresql_python.shto avoid build-id cache and BPF synthesis overhead.
Quoted 5 lines
Fixes d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")Fixes 62d350135e67 ("perf python: Port export-to-sqlite to perf module")Fixes b1f968c9656a ("perf python: Port export-to-postgresql to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Ian Rogers
reviewerPATCH v2 5/9 perf test: Deflake failed-syscalls Python shell tests under load
In reply to Ian RogersWhen running 'perf test' in parallel under heavy system load, a one-shot 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event' can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and ENOENT sys_exit events are captured, while also contending on ~/.debug build-id caching and BPF event synthesis.
Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell after 'ls' so its events are flushed before the parent subshell exits, and wrap the record and check steps in a bounded retry loop (up to 5 attempts).
--- .../test_failed_syscalls_by_pid_python.sh | 68 +++++++++++-------- .../shell/test_failed_syscalls_python.sh | 54 +++++++++------ 2 files changed, 71 insertions(+), 51 deletions(-)
tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh +39 -29
@@ -41,8 +41,10 @@ test_file_mode() { echo "Testing failed-syscalls-by-pid.py..." # Check if syscalls:sys_exit is supported/readable - if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then - if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \ + -o /dev/null -- true >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \ + -o /dev/null -- true >/dev/null 2>&1; then echo "Skipping test, no syscalls:sys_exit or raw_syscalls:sys_exit event" exit 2 else
@@ -52,41 +54,49 @@ test_file_mode() { EVENT="syscalls:sys_exit" fi - # Generate some events by running a command that should fail at least some syscall - # (e.g. failing stat on non-existent file). - # Using '|| true' because 'perf record' returns the exit code of 'ls', - # which fails with ENOENT - perf record -e "${EVENT}" -o "${temp_data}" -- ls /does_not_exist >/dev/null 2>&1 || true + # Generate some events by running a command that fails a syscall + # (e.g. failing stat on non-existent file), sleeping briefly in the + # subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed. + passed=0 + for _ in 1 2 3 4 5; do + rm -f "${temp_data}" "${temp_out}" "${temp_out}.comm" "${temp_out}.pid" + perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \ + -- sh -c "ls /does_not_exist 2>/dev/null; sleep 0.05 || true" \ + >/dev/null 2>&1 || true + if [ ! -s "${temp_data}" ]; then + continue + fi + + if perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}" && \ + grep -n -q "err = ENOENT" "${temp_out}" && \ + perf script failed-syscalls-by-pid -i "${temp_data}" \ + "ls" > "${temp_out}.comm" && \ + grep -q "err = ENOENT" "${temp_out}.comm"; then + ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' \ + "${temp_out}.comm" | head -n 1) + if [ -n "${ls_pid}" ] && \ + perf script failed-syscalls-by-pid -i "${temp_data}" \ + "${ls_pid}" > "${temp_out}.pid" && \ + grep -q "err = ENOENT" "${temp_out}.pid"; then + passed=1 + break + fi + fi + done + if [ ! -s "${temp_data}" ]; then echo "Skipping test, perf record failed to create data" exit 2 fi - # Run the script and check output - if ! perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}"; then + if [ "$passed" -eq 0 ]; then echo "failed-syscalls-by-pid test failed." - err=1 - elif ! grep -n -q "err = ENOENT" "${temp_out}"; then - echo "Failed to find expected failed syscalls" - cat "${temp_out}" - err=1 - elif ! perf script failed-syscalls-by-pid -i "${temp_data}" "ls" > "${temp_out}.comm" || \ - ! grep -q "err = ENOENT" "${temp_out}.comm"; then - echo "failed-syscalls-by-pid comm filter test failed." - cat "${temp_out}.comm" + cat "${temp_out}" 2>/dev/null || true + cat "${temp_out}.comm" 2>/dev/null || true + cat "${temp_out}.pid" 2>/dev/null || true err=1 else - ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' "${temp_out}.comm" | head -n 1) - if [ -z "${ls_pid}" ] || \ - ! perf script failed-syscalls-by-pid -i "${temp_data}" \ - "${ls_pid}" > "${temp_out}.pid" || \ - ! grep -q "err = ENOENT" "${temp_out}.pid"; then - echo "failed-syscalls-by-pid PID filter test failed." - cat "${temp_out}.pid" 2>/dev/null || true - err=1 - else - echo "failed-syscalls-by-pid test passed." - fi + echo "failed-syscalls-by-pid test passed." fi rm -f "${temp_out}" "${temp_out}.comm" "${temp_out}.pid" }
tools/perf/tests/shell/test_failed_syscalls_python.sh +32 -22
82 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v2 5/9] perf test: Deflake failed-syscalls Python shell tests under load
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
When running 'perf test' in parallel under heavy system load, a one-shot 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event' can exit in under 300 microseconds before 'ls's
PERF_RECORD_COMMand ENOENTsys_exitevents are captured, while also contending on ~/.debug build-id caching and BPF event synthesis.Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell after 'ls' so its events are flushed before the parent subshell exits, and wrap the record and check steps in a bounded retry loop (up to 5 attempts).
Quoted 4 lines
Fixes b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")Fixes 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Ian Rogers
reviewerPATCH v2 6/9 perf test: Reduce overhead and contention in Python shell tests
In reply to Ian RogersWhen all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh, test_rw_by_file_python.sh, test_rw_by_pid_python.sh, test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and test_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in test_compaction_times_python.sh.
--- tools/perf/tests/shell/test_check_perf_trace_python.sh | 8 +++++--- tools/perf/tests/shell/test_compaction_times_python.sh | 8 +++++--- .../tests/shell/test_event_analyzing_sample_python.sh | 3 ++- tools/perf/tests/shell/test_flamegraph_python.sh | 5 +++-- tools/perf/tests/shell/test_futex_contention_python.sh | 7 ++++--- tools/perf/tests/shell/test_gecko_python.sh | 3 ++- tools/perf/tests/shell/test_mem_phys_addr_python.sh | 8 +++++--- tools/perf/tests/shell/test_net_dropmonitor_python.sh | 9 +++++---- tools/perf/tests/shell/test_netdev_times_python.sh | 6 +++--- tools/perf/tests/shell/test_powerpc_hcalls_python.sh | 4 ++-- tools/perf/tests/shell/test_rw_by_file_python.sh | 5 +++-- tools/perf/tests/shell/test_rw_by_pid_python.sh | 4 ++-- tools/perf/tests/shell/test_rwtop_python.sh | 4 ++-- tools/perf/tests/shell/test_sched_migration_python.sh | 4 ++-- tools/perf/tests/shell/test_stackcollapse_python.sh | 2 +- .../tests/shell/test_syscall_counts_by_pid_python.sh | 6 +++--- tools/perf/tests/shell/test_syscall_counts_python.sh | 5 +++-- tools/perf/tests/shell/test_task_analyzer.sh | 3 ++- tools/perf/tests/shell/test_wakeup_latency_python.sh | 7 ++++--- 19 files changed, 58 insertions(+), 43 deletions(-)
tools/perf/tests/shell/test_check_perf_trace_python.sh +5 -3
@@ -45,10 +45,11 @@ test_file_mode() { echo "Testing check-perf-trace.py..." events="" - if perf list | grep -q "irq:softirq_entry"; then + tp_list=$(perf list tracepoint) + if echo "$tp_list" | grep -q "irq:softirq_entry"; then events="irq:softirq_entry" fi - if perf list | grep -q "kmem:kmalloc"; then + if echo "$tp_list" | grep -q "kmem:kmalloc"; then if [ -n "$events" ]; then events="$events,kmem:kmalloc,kmem:kfree" else
@@ -62,7 +63,8 @@ test_file_mode() { fi # Generate events - if ! perf record -e "$events" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e "$events" -o "${temp_data}" \ + -- sh -c "sleep 0.1" >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
tools/perf/tests/shell/test_compaction_times_python.sh +5 -3
@@ -44,15 +44,17 @@ test_file_mode() { echo "Testing compaction-times.py..." # Check for any compaction events to see if kernel supports it - if ! perf list | grep -q "compaction:mm_compaction_begin"; then + if ! perf list tracepoint | grep -q "compaction:mm_compaction_begin"; then echo "Skipping test, compaction tracepoints not found" exit 2 fi # Generate some events - # We might not naturally trigger compaction in 0.5s sleep, but the script + # We might not naturally trigger compaction in 0.1s sleep, but the script # should parse the empty or sparse file correctly without crashing. - if ! perf record -e "compaction:*" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event \ + -e "compaction:mm_compaction_begin,compaction:mm_compaction_end" \ + -a -o "${temp_data}" -- sleep 0.1 >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +2 -1
@@ -39,7 +39,8 @@ test_file_mode() { echo "Testing event_analyzing_sample.py..." # Generate some events - if ! perf record -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
tools/perf/tests/shell/test_flamegraph_python.sh +3 -2
@@ -60,7 +60,8 @@ test_file_mode() { echo "Testing flamegraph.py..." # Generate some events with callchains - if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record -g failed (permissions or lack of support)" exit 2 fi
@@ -75,7 +76,7 @@ test_file_mode() { # Run the script in pipe ('-') mode and validate JSON output rm -f "${temp_json}" - if ! perf record -g -o - -- perf test -w noploop 2>/dev/null | \ + if ! perf record -B -N --no-bpf-event -g -o - -- perf test -w noploop 2>/dev/null | \ perf script flamegraph -i - -f json -o "${temp_json}" >/dev/null; then echo "Pipe stdin JSON mode test failed." err=1
tools/perf/tests/shell/test_futex_contention_python.sh +4 -3
@@ -89,14 +89,15 @@ EOF test_file_mode() { echo "Testing futex-contention.py..." # Some systems might not have syscalls:sys_enter_futex - if ! perf list | grep -q syscalls:sys_enter_futex; then + if ! perf list tracepoint | grep -q syscalls:sys_enter_futex; then echo "Skipping file mode test, syscalls:sys_enter_futex not found" return fi # Generate some futex events - if ! perf record -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \ - -- sleep 0.5 2>/dev/null; then + if ! perf record -B -N --no-bpf-event \ + -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \ + -- sleep 0.1 2>/dev/null; then echo "Skipping file mode test (record failed)" return fi
tools/perf/tests/shell/test_gecko_python.sh +2 -1
@@ -39,7 +39,8 @@ test_file_mode() { echo "Testing gecko.py..." # Generate some events with callchains - if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record -g failed (permissions or lack of support)" exit 2 fi
tools/perf/tests/shell/test_mem_phys_addr_python.sh +5 -3
@@ -78,10 +78,12 @@ test_file_mode() { echo "Testing mem-phys-addr.py file mode..." # Generate memory access events (try unprivileged user-space first, then system-wide) - if ! perf record --phys-data -d -o "${temp_data}" \ + if ! perf record -B -N --no-bpf-event --phys-data -d -o "${temp_data}" \ -- perf test -w datasym >/dev/null 2>&1 && \ - ! perf record -d -o "${temp_data}" -- perf test -w datasym >/dev/null 2>&1 && \ - ! perf record -d -a -o "${temp_data}" -- sleep 0.2 >/dev/null 2>&1; then + ! perf record -B -N --no-bpf-event -d -o "${temp_data}" \ + -- perf test -w datasym >/dev/null 2>&1 && \ + ! perf record -B -N --no-bpf-event -d -a -o "${temp_data}" \ + -- sleep 0.1 >/dev/null 2>&1; then echo "Skipping file mode record test, perf record -d not supported" return 0 fi
tools/perf/tests/shell/test_net_dropmonitor_python.sh +5 -4
@@ -42,11 +42,12 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing net_dropmonitor.py..." # Create a perf.data file. Force dropping a packet if tracepoint is available! -if ! perf record -e skb:kfree_skb -o "${temp_data}" -a \ - -- ping -c 1 255.255.255.255 >/dev/null 2>&1; then - if ! perf record -e skb:kfree_skb -o "${temp_data}" \ +if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" -a \ + -- ping -c 1 -W 1 255.255.255.255 >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" \ -- sleep 0.1 >/dev/null 2>&1; then - if ! perf record -o "${temp_data}" -- uname >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -o "${temp_data}" \ + -- uname >/dev/null 2>&1; then echo "Skipping test, cannot record perf events" exit 2 fi
tools/perf/tests/shell/test_netdev_times_python.sh +3 -3
@@ -85,9 +85,9 @@ then fi # Create a perf.data file. Force dropping a packet if tracepoint is available! -if ! perf record -e skb:kfree_skb -a -o "${temp_data}" \ - -- ping -c 1 127.0.0.1 >/dev/null 2>&1; then - perf record -e cycles -o "${temp_data}" \ +if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -a -o "${temp_data}" \ + -- ping -c 1 -W 1 127.0.0.1 >/dev/null 2>&1; then + perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } fi
tools/perf/tests/shell/test_powerpc_hcalls_python.sh +2 -2
@@ -67,8 +67,8 @@ if ! grep -q "H_REMOVE.*1.*1500.*1500.*1500" "${temp_out}"; then fi # Create a perf.data file if powerpc hcall tracepoints are available on this host. -if ! perf record -e powerpc:hcall_entry,powerpc:hcall_exit -a -o "${temp_data}" \ - -- perf test -w noploop >/dev/null 2>&1; then +if ! perf record -B -N --no-bpf-event -e powerpc:hcall_entry,powerpc:hcall_exit \ + -a -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping live record test, powerpc hcall tracepoints not available" exit 0 fi
tools/perf/tests/shell/test_rw_by_file_python.sh +3 -2
@@ -37,8 +37,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing rw-by-file.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "syscalls:sys_enter_read"; then - perf record -e syscalls:sys_enter_read,syscalls:sys_enter_write -a -o "${temp_data}" \ +if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then + perf record -B -N --no-bpf-event -e syscalls:sys_enter_read,syscalls:sys_enter_write \ + -o "${temp_data}" \ -- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
tools/perf/tests/shell/test_rw_by_pid_python.sh +2 -2
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing rw-by-pid.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "syscalls:sys_enter_read"; then +if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then ev="syscalls:sys_enter_read,syscalls:sys_exit_read" ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write" - perf record -e "$ev" -a -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \ -- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
tools/perf/tests/shell/test_rwtop_python.sh +2 -2
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing rwtop.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "syscalls:sys_enter_read"; then +if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then ev="syscalls:sys_enter_read,syscalls:sys_exit_read" ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write" - perf record -e "$ev" -a -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \ -- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
tools/perf/tests/shell/test_sched_migration_python.sh +2 -2
@@ -44,10 +44,10 @@ echo "Testing sched-migration.py..." ev="sched:sched_switch,sched:sched_migrate_task" ev="${ev},sched:sched_wakeup_new,sched:sched_wakeup" has_sched=1 -if ! perf record -e "$ev" -a -o "${temp_data}" \ +if ! perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \ -- sleep 0.1 >/dev/null 2>&1; then has_sched=0 - perf record -e cycles -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } fi
tools/perf/tests/shell/test_stackcollapse_python.sh +1 -1
@@ -37,7 +37,7 @@ echo "Testing stackcollapse.py..." # Create a perf.data file with callchains. Use a busy workload rather than # sleep, as an idle system may not generate any samples at all. -perf record -g -o "${temp_data}" \ +perf record -B -N --no-bpf-event -g -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; }
tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh +3 -3
@@ -43,14 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) test_file_mode() { echo "Testing syscall-counts-by-pid.py..." # Some systems might not have raw_syscalls:sys_enter - if ! perf list | grep -q raw_syscalls:sys_enter; then + if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then echo "Skipping test, raw_syscalls:sys_enter not found" exit 2 fi # Generate some syscall events - perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \ - -- sleep 0.5 >/dev/null 2>&1 || \ + perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \ + -- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } if ! perf script syscall-counts-by-pid -i "${temp_data}" > "${temp_out}"; then
tools/perf/tests/shell/test_syscall_counts_python.sh +3 -2
@@ -43,13 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) test_file_mode() { echo "Testing syscall-counts.py..." # Some systems might not have raw_syscalls:sys_enter (e.g. stripped kernels or permissions) - if ! perf list | grep -q raw_syscalls:sys_enter; then + if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then echo "Skipping test, raw_syscalls:sys_enter not found" exit 2 fi # Generate some syscall events - if ! perf record -e raw_syscalls:sys_enter -o "${temp_data}" -- sleep 0.5 2>/dev/null; then + if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \ + -- sh -c "sleep 0.1; sleep 0.05" 2>/dev/null; then echo "perf record failed (permissions?), skipping file mode test." exit 2 fi
tools/perf/tests/shell/test_task_analyzer.sh +2 -1
@@ -60,7 +60,8 @@ skip_no_probe_record_support() { prepare_perf_data() { # 1s should be sufficient to catch at least some switches - perf record -e sched:sched_switch -a -o "${perfdata}" -- sleep 1 > /dev/null 2>&1 + perf record -B -N --no-bpf-event -e sched:sched_switch -a -o "${perfdata}" \ + -- sleep 1 > /dev/null 2>&1 # check if perf data file got created in above step. if [ ! -e "${perfdata}" ]; then printf "FAIL: perf record failed to create \"${perfdata}\" \n"
tools/perf/tests/shell/test_wakeup_latency_python.sh +4 -3
@@ -41,9 +41,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing wakeup-latency.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "sched:sched_wakeup"; then +if perf list tracepoint | grep -q "sched:sched_wakeup"; then ev="sched:sched_wakeup,sched:sched_wakeup_new,sched:sched_switch" - perf record -e "$ev" -a -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \ -- sleep 0.1 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
@@ -70,7 +70,8 @@ else fi # Also test zero-wakeups / unhandled events path to verify division-by-zero protection -if perf record -e cycles -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then +if perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then if ! perf script wakeup-latency -i "${temp_data}" > "${temp_out}" || \ ! grep -q "avg_wakeup_latency (ns): N/A" "${temp_out}"; then echo "wakeup-latency zero-wakeups guard test failed"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v2 6/9] perf test: Reduce overhead and contention in Python shell tests
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 18 lines
When all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' intest_check_perf_trace_python.sh,test_rw_by_file_python.sh,test_rw_by_pid_python.sh,test_rwtop_python.sh, andtest_syscall_counts_by_pid_python.shwhere only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' intest_net_dropmonitor_python.shandtest_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' intest_compaction_times_python.sh.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Ian Rogers
reviewerPATCH v2 7/9 perf python event_analyzing_sample: Default to in-memory SQLite database
In reply to Ian RogersWhen -d/--db is not specified, event_analyzing_sample.py created a temporary file in /tmp via tempfile.mkstemp() and deleted it in trace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.
Default to sqlite3.connect(":memory:") when db_path is not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode in test_event_analyzing_sample_python.sh.
--- tools/perf/python/event_analyzing_sample.py | 35 ++++++------------- .../test_event_analyzing_sample_python.sh | 8 ++++- 2 files changed, 17 insertions(+), 26 deletions(-)
tools/perf/python/event_analyzing_sample.py +10 -25
@@ -17,10 +17,8 @@ from __future__ import annotations import argparse import math -import os import sqlite3 import struct -import tempfile from typing import Any import perf
@@ -117,16 +115,11 @@ session: Any = None class _DB: con: sqlite3.Connection | None = None - temp_path: str | None = None def trace_begin(db_path: str | None = None) -> None: """Initialize database tables.""" print("In trace_begin:\n") - if not db_path: - fd, db_path = tempfile.mkstemp(prefix="perf_events_", suffix=".db") - os.close(fd) - _DB.temp_path = db_path - con = sqlite3.connect(db_path) + con = sqlite3.connect(db_path or ":memory:") try: # Drop any pre-existing tables so repeated runs do not accumulate duplicate events. con.execute("drop table if exists gen_events;")
@@ -297,28 +290,20 @@ def show_pebs_ll() -> None: def trace_end() -> None: """Called at the end of trace processing.""" print("In trace_end:\n") - try: - if _DB.con: - try: - _DB.con.commit() - show_general_events() - show_pebs_ll() - finally: - _DB.con.close() - _DB.con = None - finally: - if _DB.temp_path and os.path.exists(_DB.temp_path): - try: - os.remove(_DB.temp_path) - except OSError: - pass - _DB.temp_path = None + if _DB.con: + try: + _DB.con.commit() + show_general_events() + show_pebs_ll() + finally: + _DB.con.close() + _DB.con = None if __name__ == "__main__": ap = argparse.ArgumentParser(description="Analyze events with SQLite") ap.add_argument("-i", "--input", default="perf.data", help="Input file name") ap.add_argument("-d", "--db", "--database", dest="database", default=None, - help="Database file name (defaults to a temporary file cleaned up on exit)") + help="Database file name (defaults to an in-memory database)") args = ap.parse_args() try:
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +7 -1
@@ -45,8 +45,14 @@ test_file_mode() { exit 2 fi - # Run the script + # Run the script with default (:memory:) database and with explicit -d path if ! perf script event_analyzing_sample -i "${temp_data}" \ + > "${temp_dir}/perf.mem.out" 2>&1 || \ + ! grep -q "Statistics about the general events" "${temp_dir}/perf.mem.out" || \ + grep -q "Error creating/inserting event" "${temp_dir}/perf.mem.out"; then + echo "Default in-memory database mode test failed." + err=1 + elif ! perf script event_analyzing_sample -i "${temp_data}" \ -d "${temp_db}" > "${temp_dir}/perf.out" 2>&1; then echo "File mode test failed." err=1
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v2 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines
When -d/--db is not specified,
event_analyzing_sample.pycreated a temporary file in /tmp viatempfile.mkstemp()and deleted it intrace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.
Quoted 8 lines
Default to
sqlite3.connect(":memory:")whendb_pathis not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode intest_event_analyzing_sample_python.sh.Fixes eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Ian Rogers
reviewerPATCH v2 8/9 perf python: Initialize debug output on module load
In reply to Ian RogersUnlike the perf binary's main(), PyInit_perf() does not call perf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hits debug_file() with _debug_file == NULL and emits 'debug_file not set' without a trailing newline before the actual diagnostic message.
Call perf_debug_setup() in PyInit_perf() and add the missing trailing newline to the fallback warning in debug_file().
--- tools/perf/util/debug.c | 2 +- tools/perf/util/python.c | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
tools/perf/util/debug.c +1 -1
@@ -55,7 +55,7 @@ FILE *debug_file(void) {if (!_debug_file) {debug_set_file (stderr); -pr_warning_once ("debug_file not set" ); +pr_warning_once ("%s not set\n" , __func__); }return _debug_file; }
tools/perf/util/python.c +1 -0
@@ -5157,6 +5157,7 @@ PyMODINIT_FUNC PyInit_perf(void)/* The page_size is placed in util object. */ page_size =sysconf (_SC_PAGE_SIZE); +perf_debug_setup ();Py_INCREF (&pyrf_evlist__type);PyModule_AddObject (module,"evlist" , (PyObject *)&pyrf_evlist__type);
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v2 8/9] perf python: Initialize debug output on module load
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines
Unlike the perf binary's
main(),PyInit_perf()does not callperf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hitsdebug_file()with_debug_file== NULL and emits 'debug_filenot set' without a trailing newline before the actual diagnostic message.
Quoted 6 lines
Call
perf_debug_setup()inPyInit_perf()and add the missing trailing newline to the fallback warning indebug_file().Fixes ec49230cf6dd ("perf debug: Expose debug file")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Ian Rogers
reviewerPATCH v2 9/9 perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
In reply to Ian Rogersperf_pmus__print_pmu_events() first counts events via perf_pmu__num_events() to allocate the aliases array, then populates it via perf_pmu__for_each_event(). When dynamic tracepoints (such as kprobes or uprobes during 'perf test' runs) are concurrently created or removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len, leaving trailing zeroed entries in aliases[] with name == NULL and pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning /sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns -ENOENT and aborts enumeration of all remaining tracepoint subsystems.
- If an event is added, perf_pmus__print_pmu_events__callback() aborts enumeration when state->index reaches state->aliases_len.
Grow state->aliases dynamically via realloc() when needed, use state.index as the actual populated length for sorting and printing, and ignore -ENOENT when a tracepoint subsystem directory disappears during enumeration.
--- tools/perf/util/pmus.c | 21 +++++++++++++++++---- tools/perf/util/tp_pmu.c | 8 ++++++-- 2 files changed, 23 insertions(+), 6 deletions(-)
tools/perf/util/pmus.c +17 -4
@@ -8,6 +8,7 @@#include <sys/types.h> #include <ctype.h> #include <pthread.h> +#include <stdlib.h> #include <string.h> #include <unistd.h> #include "cpumap.h"
@@ -571,7 +572,7 @@ static int cmp_sevent(const void *a, const void *b) }/* Order by event name. */ -return strcmp (as->name, bs->name); +return strcmp (as->name ?:"" , bs->name ?:"" ); }static bool pmu_alias_is_duplicate (struct sevent *a,struct sevent *b)
@@ -581,7 +582,7 @@ static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)return false;/* Don't remove duplicates for different PMUs */ -return strcmp (a->pmu_name, b->pmu_name) ==0 ; +return strcmp (a->pmu_name ?:"" , b->pmu_name ?:"" ) ==0 ; }struct events_callback_state {
@@ -597,8 +598,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,struct sevent *s;if (state->index >= state->aliases_len) { -pr_err ("Unexpected event %s/%s/\n" , info->pmu->name, info->name); -return 1 ; +size_t new_len =max_t (size_t,16 , state->aliases_len *2 ); +struct sevent *new_aliases; + + new_aliases =realloc (state->aliases, new_len *sizeof (struct sevent )); +if (!new_aliases) { +pr_err ("Unexpected event %s/%s/\n" , info->pmu->name, info->name); +return 1 ; + } +memset (&new_aliases[state->aliases_len],0 , + (new_len - state->aliases_len) *sizeof (struct sevent )); + state->aliases = new_aliases; + state->aliases_len = new_len; }assert (info->pmu !=NULL || info->name !=NULL ); s = &state->aliases[state->index];
@@ -654,6 +665,8 @@ void perf_pmus__print_pmu_events(const struct print_callbacks *print_cb, void *pperf_pmu__for_each_event (pmu, skip_duplicate_pmus, &state, perf_pmus__print_pmu_events__callback); } + aliases = state.aliases; + len = state.index;qsort (aliases, len,sizeof (struct sevent ), cmp_sevent);for (int j =0 ; j < len; j++) {/* Skip duplicates */
tools/perf/util/tp_pmu.c +6 -2
@@ -151,7 +151,9 @@ static int for_each_event_cb(void *state, const char *sys_name, const char *evt_static int for_each_event_sys_cb (void *state,const char *sys_name) { -return tp_pmu__for_each_tp_event (sys_name, state, for_each_event_cb); +int ret =tp_pmu__for_each_tp_event (sys_name, state, for_each_event_cb); + +return ret == -ENOENT ?0 : ret; }int tp_pmu__for_each_event (struct perf_pmu *pmu,void *state,pmu_event_callback cb)
@@ -176,7 +178,9 @@ static int num_events_cb(void *state, const char *sys_name __maybe_unused,static int num_events_sys_cb (void *state,const char *sys_name) { -return tp_pmu__for_each_tp_event (sys_name, state, num_events_cb); +int ret =tp_pmu__for_each_tp_event (sys_name, state, num_events_cb); + +return ret == -ENOENT ?0 : ret; }size_t tp_pmu__num_events (struct perf_pmu *pmu__maybe_unused )
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v2 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 20 lines
perf_pmus__print_pmu_events()first counts events viaperf_pmu__num_events()to allocate the aliases array, then populates it viaperf_pmu__for_each_event(). When dynamic tracepoints (such as kprobes or uprobes during 'perf test' runs) are concurrently created or removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len, leaving trailing zeroed entries in aliases[] with name == NULL and pmu == NULL, which causesqsort(cmp_sevent)or the print loop to crash with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning/sys/kernel/tracing/events,tp_pmu__for_each_tp_event()returns-ENOENTand aborts enumeration of all remaining tracepoint subsystems.
- If an event is added,perf_pmus__print_pmu_events__callback()aborts enumeration whenstate->indexreachesstate->aliases_len.Grow
state->aliasesdynamically viarealloc()when needed, use state.index as the actual populated length for sorting and printing, and ignore-ENOENTwhen a tracepoint subsystem directory disappears during enumeration.
Quoted 4 lines
Fixes c3245d2093c1 ("perf pmu: Abstract alias/event struct")Fixes 45b6e281cb06 ("perf tp_pmu: Add event APIs")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Ian Rogers
reviewerv3 cover perf python: Follow-up fixes and shell test deflaking
In reply to Ian RogersThis series contains follow-up fixes for the standalone Python scripts and their shell tests, particularly when running 'perf test' as root under heavy parallel load:
- Clamp sample_id size calculation for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE records to work around kernels without "perf/core: Restore header fields in sideband output callbacks" [1].
- Fix offline interval printing in sctop.py and live signal races in stat-cpi.py.
- Deflake Intel PT, failed-syscalls, and other Python shell tests under parallel 'perf test' load by scoping tracepoint checks to 'perf list tracepoint', passing '-B -N --no-bpf-event' to 'perf record', and avoiding unnecessary system-wide ('-a') recordings.
- Default event_analyzing_sample.py to an in-memory SQLite database (':memory:') when '--db' is not specified.
- Initialize perf debug output ('perf_debug_setup()') when loading the 'perf' Python extension module.
- Fix races in 'perf list' ('perf_pmus__print_pmu_events()' and 'tp_pmu.c') when concurrent tests dynamically create and remove kprobe/uprobes tracepoints.
v3:
- Wrap _open_live_evlist() inside the try/except KeyboardInterrupt block in stat-cpi.py so signals arriving during event initialization are caught and cleaned up gracefully.
v2:
- Fix 8-byte alignment calculation for PERF_RECORD_TEXT_POKE payload in evsel__event_size() by aligning fixed + old_len + new_len.
- Keep offline interval clock advancing on all samples in sctop.py while flushing any remaining syscalls at EOF.
- Use 'mktemp -d' private directories in test_sctop_python.sh and test_failed_syscalls_python.sh so temporary files are not unlinked and recreated directly in /tmp.
- Explicitly include <stdlib.h> in tools/perf/util/pmus.c.
[1] https://lore.kernel.org/r/[email protected]
Ian Rogers (9):
perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke
events
perf python sctop: Fix offline interval printing and test flakiness perf python stat-cpi: Fix live mode signal races and test flakiness perf test: Deflake Intel PT Python shell tests under load perf test: Deflake failed-syscalls Python shell tests under load perf test: Reduce overhead and contention in Python shell tests perf python event_analyzing_sample: Default to in-memory SQLite
database
perf python: Initialize debug output on module load
perf pmu: Fix race with concurrent tracepoint creation and removal in
perf list
tools/perf/python/event_analyzing_sample.py | 35 +++------- tools/perf/python/sctop.py | 7 +- tools/perf/python/stat-cpi.py | 32 ++++++--- .../shell/test_check_perf_trace_python.sh | 8 ++- .../shell/test_compaction_times_python.sh | 8 ++- .../test_event_analyzing_sample_python.sh | 11 ++- .../shell/test_export_to_postgresql_python.sh | 49 +++++++------ .../shell/test_export_to_sqlite_python.sh | 50 ++++++++------ .../test_failed_syscalls_by_pid_python.sh | 68 +++++++++++-------- .../shell/test_failed_syscalls_python.sh | 54 +++++++++------ .../tests/shell/test_flamegraph_python.sh | 5 +- .../shell/test_futex_contention_python.sh | 7 +- tools/perf/tests/shell/test_gecko_python.sh | 3 +- .../shell/test_intel_pt_events_python.sh | 43 ++++++------ .../tests/shell/test_mem_phys_addr_python.sh | 8 ++- .../shell/test_net_dropmonitor_python.sh | 9 +-- .../tests/shell/test_netdev_times_python.sh | 6 +- .../tests/shell/test_powerpc_hcalls_python.sh | 4 +- .../tests/shell/test_rw_by_file_python.sh | 5 +- .../perf/tests/shell/test_rw_by_pid_python.sh | 4 +- tools/perf/tests/shell/test_rwtop_python.sh | 4 +- .../shell/test_sched_migration_python.sh | 4 +- tools/perf/tests/shell/test_sctop_python.sh | 58 ++++++++-------- .../tests/shell/test_stackcollapse_python.sh | 2 +- .../perf/tests/shell/test_stat_cpi_python.sh | 12 +++- .../test_syscall_counts_by_pid_python.sh | 6 +- .../tests/shell/test_syscall_counts_python.sh | 5 +- tools/perf/tests/shell/test_task_analyzer.sh | 3 +- .../tests/shell/test_wakeup_latency_python.sh | 7 +- tools/perf/util/debug.c | 2 +- tools/perf/util/evlist.c | 6 +- tools/perf/util/evsel.c | 59 +++++++++++++++- tools/perf/util/evsel.h | 1 + tools/perf/util/pmus.c | 21 ++++-- tools/perf/util/python.c | 1 + tools/perf/util/tp_pmu.c | 8 ++- 36 files changed, 378 insertions(+), 237 deletions(-)
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
Ian Rogers
reviewerPATCH v3 1/9 perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
On kernels without the fix ("perf/core: Restore header fields in sideband output callbacks") to restore event_id.header.size in perf_event_ksymbol_output(), perf_event_bpf_output(), and perf_event_text_poke_output(), concurrent perf sessions cause those sideband records to be emitted with header.size inflated by multiple id_header_size increments while the single id_sample is written immediately after the event payload. Indexing backwards from event->header.size reads uninitialized ring-buffer bytes at the end of the record, causing evlist__event2evsel() to fail with -EFAULT.
Add evsel__event_size() to clamp the effective size used to locate the trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving event->header.size intact for advancing the ring-buffer/file stream.
--- tools/perf/util/evlist.c | 6 ++-- tools/perf/util/evsel.c | 59 +++++++++++++++++++++++++++++++++++++++- tools/perf/util/evsel.h | 1 + 3 files changed, 63 insertions(+), 3 deletions(-)
tools/perf/util/evlist.c +4 -2
@@ -973,13 +973,15 @@ static int evlist__event2id(struct evlist *evlist, union perf_event *event, u64const __u64 *array = event->sample.array;ssize_t n; - n = (event->header.size -sizeof (event->header)) >>3 ; -if (event->header.type ==PERF_RECORD_SAMPLE ) { + n = (event->header.size -sizeof (event->header)) >>3 ;if (evlist__id_pos (evlist) >= n)return -1 ; *id = array[evlist__id_pos (evlist)]; }else { +u16 size =evsel__event_size (evlist__first (evlist), event); + + n = (size -sizeof (event->header)) >>3 ;if (evlist__is_pos (evlist) > n)return -1 ; n -=evlist__is_pos (evlist);
tools/perf/util/evsel.c +58 -1
71 lines not on this page · raw message
tools/perf/util/evsel.h +1 -0
@@ -469,6 +469,7 @@ int evsel__parse_sample_timestamp(struct evsel *evsel, union perf_event *event, u64 *timestamp);u16 evsel__id_hdr_size (const struct evsel *evsel); +u16 evsel__event_size (const struct evsel *evsel,const union perf_event *event);static inline struct evsel *evsel__next (struct evsel *evsel) {
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines
On kernels without the fix ("perf/core: Restore header fields in sideband output callbacks") to restore event_id.header.size in
perf_event_ksymbol_output(),perf_event_bpf_output(), andperf_event_text_poke_output(), concurrent perf sessions cause those sideband records to be emitted with header.size inflated by multipleid_header_sizeincrements while the singleid_sampleis written immediately after the event payload. Indexing backwards fromevent->header.sizereads uninitialized ring-buffer bytes at the end of the record, causingevlist__event2evsel()to fail with-EFAULT.Add
evsel__event_size()to clamp the effective size used to locate the trailingid_sampleforPERF_RECORD_KSYMBOL,PERF_RECORD_BPF_EVENT, andPERF_RECORD_TEXT_POKEto payload +id_hdr_sizewhile leavingevent->header.sizeintact for advancing the ring-buffer/file stream.
Quoted 6 lines
Fixes 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")Fixes 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")Fixes 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Ian Rogers
reviewerPATCH v3 2/9 perf python sctop: Fix offline interval printing and test flakiness
In reply to Ian RogersIn sctop.py:
- If an earlier interval elapsed and printed an empty table before the target comm ('sleep') executed any syscalls, analyzer.printed became True and the final partial interval containing the target comm's syscalls was never flushed at EOF. Flush print_current_totals() in finally when analyzer.syscalls is non-empty as well as when nothing has been printed yet.
- Initialize analyzer.e_machine after creating perf.session rather than when session is still None.
In test_sctop_python.sh:
- Use a private temporary directory via 'mktemp -d'.
- Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a bounded retry loop so 'sleep's PERF_RECORD_COMM and sys_enter events are reliably captured under heavy load.
--- tools/perf/python/sctop.py | 7 ++- tools/perf/tests/shell/test_sctop_python.sh | 58 ++++++++++----------- 2 files changed, 34 insertions(+), 31 deletions(-)
tools/perf/python/sctop.py +5 -2
@@ -37,6 +37,7 @@ class SCTopAnalyzer: self.offline = offline self.own_pid = os.getpid() self.last_print_time: Optional[int] = None + self.printed = False self.session: Optional[perf.session] = None self.e_machine: Optional[int] = None
@@ -137,6 +138,7 @@ class SCTopAnalyzer: def print_current_totals(self): """Print current syscall totals.""" + self.printed = True # Clear terminal if not self.offline: print("\x1b[2J\x1b[H", end="")
@@ -217,8 +219,8 @@ def main(): if args.input: session = perf.session(perf.data(args.input), sample=analyzer.process_event) analyzer.session = session - session.process_events() analyzer.e_machine = getattr(session, "e_machine", None) + session.process_events() else: try: live_session = LiveSession(
@@ -237,7 +239,8 @@ def main(): sys.exit(1) finally: if args.input: - analyzer.print_current_totals() + if not analyzer.printed or analyzer.syscalls: + analyzer.print_current_totals() # Break the reference cycle between perf.session and analyzer.process_event # because perf.session lacks cyclic GC support (tp_traverse). analyzer.session = None
tools/perf/tests/shell/test_sctop_python.sh +29 -29
80 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 2/9] perf python sctop: Fix offline interval printing and test flakiness
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines
In
sctop.py:
- If an earlier interval elapsed and printed an empty table before the target comm ('sleep') executed any syscalls, analyzer.printed became True and the final partial interval containing the target comm's syscalls was never flushed at EOF. Flushprint_current_totals()in finally when analyzer.syscalls is non-empty as well as when nothing has been printed yet.
- Initialize analyzer.e_machine after creating perf.session rather than when session is still None.In
test_sctop_python.sh:
- Use a private temporary directory via 'mktemp -d'.
- Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a bounded retry loop so 'sleep'sPERF_RECORD_COMMandsys_enterevents are reliably captured under heavy load.
Quoted 3 lines
Fixes b83f0bacf5e4 ("perf python: Port sctop to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Ian Rogers
reviewerPATCH v3 3/9 perf python stat-cpi: Fix live mode signal races and test flakiness
In reply to Ian RogersIn test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a 1-second duration and can exit under heavy parallel load before 'perf stat -p' and 'perf script stat-cpi' finish starting up. In addition, the fixed 'sleep 0.5' before sending SIGINT can fire before Python finishes importing the perf module, opening the live evlist, and flushing the first interval.
In stat-cpi.py, register SIGINT and SIGTERM handlers before calling _open_live_evlist() and pass flush=True when printing live output so redirected stdout is flushed immediately after each interval.
In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the target workload stays alive until killed, and poll the output file for 'cpi' (up to 5 seconds) before sending SIGINT.
--- tools/perf/python/stat-cpi.py | 32 +++++++++++++------ .../perf/tests/shell/test_stat_cpi_python.sh | 12 +++++-- 2 files changed, 31 insertions(+), 13 deletions(-)
tools/perf/python/stat-cpi.py +22 -10
@@ -106,7 +106,8 @@ class StatCpiAnalyzer: if ins != 0: cpi = cyc / float(ins) t_sec = timestamp / 1000000000.0 - print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})") + print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})", + flush=True) def read_counters(self, evlist: Any) -> None: """Read counters live."""
@@ -151,6 +152,7 @@ class StatCpiAnalyzer: last_err: Optional[OSError] = None for events, tmap in candidates: + evlist = None try: evlist = perf.parse_events(events, None, tmap) for evsel in evlist:
@@ -161,32 +163,41 @@ class StatCpiAnalyzer: evlist.enable() return evlist except PermissionError as e: + if evlist is not None: + evlist.close() last_err = e except OSError as e: + if evlist is not None: + evlist.close() if e.errno == 13: last_err = e else: raise + except BaseException: + if evlist is not None: + evlist.close() + raise if last_err is not None: raise last_err raise RuntimeError("Failed to open events") def run_live(self) -> None: """Read counters live.""" - try: - evlist = self._open_live_evlist() - except OSError as e: - print(f"Failed to open events: {e}", file=sys.stderr) - sys.exit(1) - def handle_signal(_signum: int, _frame: Any) -> None: raise KeyboardInterrupt signal.signal(signal.SIGINT, signal.default_int_handler) signal.signal(signal.SIGTERM, handle_signal) - print("Live mode started. Press Ctrl+C to stop.") + evlist = None try: + try: + evlist = self._open_live_evlist() + except OSError as e: + print(f"Failed to open events: {e}", file=sys.stderr) + sys.exit(1) + + print("Live mode started. Press Ctrl+C to stop.", flush=True) while True: time.sleep(self.args.interval) timestamp = time.time_ns()
@@ -195,9 +206,10 @@ class StatCpiAnalyzer: self.data.clear() self.recorded_pairs.clear() except KeyboardInterrupt: - print("\nStopped.") + print("\nStopped.", flush=True) finally: - evlist.close() + if evlist is not None: + evlist.close() def main() -> None: """Main function."""
tools/perf/tests/shell/test_stat_cpi_python.sh +9 -3
@@ -50,7 +50,7 @@ test_live_mode() { echo "perf stat failed (permissions?), skipping live mode test." return 0 fi - perf test -w noploop & + perf test -w noploop 60 & workload_pid=$! if ! perf stat -e cycles,instructions -p "$workload_pid" -- sleep 0.05 2>/dev/null && \ ! perf stat -e cycles:u,instructions:u -p "$workload_pid" -- sleep 0.05 2>/dev/null; then
@@ -61,10 +61,16 @@ test_live_mode() { fi ran=1 - # Run live mode for 1 interval in the background, give it a tiny sleep, then interrupt + # Run live mode in the background, wait until at least one interval is + # printed, then interrupt. perf script stat-cpi -I 0.1 -p "$workload_pid" > "${temp_out}" & pid=$! - sleep 0.5 + for _ in $(seq 1 50); do + if grep -q "cpi" "${temp_out}"; then + break + fi + sleep 0.1 + done kill -INT "$pid" 2>/dev/null || true set +e wait "$pid"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
In
test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a 1-second duration and can exit under heavy parallel load before 'perf stat -p' and 'perf script stat-cpi' finish starting up. In addition, the fixed 'sleep 0.5' before sending SIGINT can fire before Python finishes importing the perf module, opening the live evlist, and flushing the first interval.In
stat-cpi.py, register SIGINT and SIGTERM handlers before calling_open_live_evlist()and pass flush=True when printing live output so redirected stdout is flushed immediately after each interval.
Quoted 7 lines
In
test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the target workload stays alive until killed, and poll the output file for 'cpi' (up to 5 seconds) before sending SIGINT.Fixes 4425182d426b ("perf python: Port stat-cpi to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Ian Rogers
reviewerPATCH v3 4/9 perf test: Deflake Intel PT Python shell tests under load
In reply to Ian RogersIn test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).
Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh to avoid build-id cache and BPF synthesis overhead.
--- .../shell/test_export_to_postgresql_python.sh | 49 ++++++++++-------- .../shell/test_export_to_sqlite_python.sh | 50 +++++++++++-------- .../shell/test_intel_pt_events_python.sh | 43 +++++++++------- 3 files changed, 79 insertions(+), 63 deletions(-)
tools/perf/tests/shell/test_export_to_postgresql_python.sh +27 -22
@@ -61,9 +61,10 @@ test_file_mode() { fi # Generate events with callchains and context switches - if ! perf record -g --switch-events -o "${temp_data}" \ + if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 && \ - ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
@@ -92,29 +93,33 @@ test_file_mode() { test_intel_pt() { echo "Testing export-to-postgresql.py with intel_pt..." - psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true - rm -f "${temp_data}" - # Generate some intel_pt events; use a subshell that waits for uname - if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \ - -- sh -c "uname; true" >/dev/null 2>&1; then - echo "Skipping intel_pt test, intel_pt not available." - return 0 - fi + # Generate some intel_pt events; sleep briefly after uname in the subshell + # so uname's AUX buffer is flushed before SIGCHLD stops perf record. + passed=0 + for _ in 1 2 3 4 5; do + psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true + rm -f "${temp_data}" + if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \ + -- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then + echo "Skipping intel_pt test, intel_pt not available." + return 0 + fi - # Run the script with --itrace cr to synthesize call_returns - if ! perf script export-to-postgresql -i "${temp_data}" \ - -o "${temp_db}" --itrace cr >/dev/null; then - echo "intel_pt file mode test failed." - err=1 - else - # Check DB for calls - if ! psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \ + # Run the script with --itrace cr to synthesize call_returns + if perf script export-to-postgresql -i "${temp_data}" \ + -o "${temp_db}" --itrace cr >/dev/null && \ + psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \ grep -q '[1-9]'; then - echo "PostgreSQL intel_pt validation failed (no calls found)." - err=1 - else - echo "intel_pt test passed (cr validated)." + passed=1 + break fi + done + + if [ "$passed" -eq 0 ]; then + echo "PostgreSQL intel_pt validation failed (no calls found)." + err=1 + else + echo "intel_pt test passed (cr validated)." fi }
tools/perf/tests/shell/test_export_to_sqlite_python.sh +28 -22
66 lines not on this page · raw message
tools/perf/tests/shell/test_intel_pt_events_python.sh +24 -19
58 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 4/9] perf test: Deflake Intel PT Python shell tests under load
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines
In
test_intel_pt_events_python.sh,test_export_to_sqlite_python.sh, andtest_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in
test_export_to_sqlite_python.shandtest_export_to_postgresql_python.shto avoid build-id cache and BPF synthesis overhead.
Quoted 5 lines
Fixes d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")Fixes 62d350135e67 ("perf python: Port export-to-sqlite to perf module")Fixes b1f968c9656a ("perf python: Port export-to-postgresql to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Ian Rogers
reviewerPATCH v3 5/9 perf test: Deflake failed-syscalls Python shell tests under load
In reply to Ian RogersWhen running 'perf test' in parallel under heavy system load, a one-shot 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event' can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and ENOENT sys_exit events are captured, while also contending on ~/.debug build-id caching and BPF event synthesis.
Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell after 'ls' so its events are flushed before the parent subshell exits, and wrap the record and check steps in a bounded retry loop (up to 5 attempts).
--- .../test_failed_syscalls_by_pid_python.sh | 68 +++++++++++-------- .../shell/test_failed_syscalls_python.sh | 54 +++++++++------ 2 files changed, 71 insertions(+), 51 deletions(-)
tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh +39 -29
@@ -41,8 +41,10 @@ test_file_mode() { echo "Testing failed-syscalls-by-pid.py..." # Check if syscalls:sys_exit is supported/readable - if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then - if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \ + -o /dev/null -- true >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \ + -o /dev/null -- true >/dev/null 2>&1; then echo "Skipping test, no syscalls:sys_exit or raw_syscalls:sys_exit event" exit 2 else
@@ -52,41 +54,49 @@ test_file_mode() { EVENT="syscalls:sys_exit" fi - # Generate some events by running a command that should fail at least some syscall - # (e.g. failing stat on non-existent file). - # Using '|| true' because 'perf record' returns the exit code of 'ls', - # which fails with ENOENT - perf record -e "${EVENT}" -o "${temp_data}" -- ls /does_not_exist >/dev/null 2>&1 || true + # Generate some events by running a command that fails a syscall + # (e.g. failing stat on non-existent file), sleeping briefly in the + # subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed. + passed=0 + for _ in 1 2 3 4 5; do + rm -f "${temp_data}" "${temp_out}" "${temp_out}.comm" "${temp_out}.pid" + perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \ + -- sh -c "ls /does_not_exist 2>/dev/null; sleep 0.05 || true" \ + >/dev/null 2>&1 || true + if [ ! -s "${temp_data}" ]; then + continue + fi + + if perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}" && \ + grep -n -q "err = ENOENT" "${temp_out}" && \ + perf script failed-syscalls-by-pid -i "${temp_data}" \ + "ls" > "${temp_out}.comm" && \ + grep -q "err = ENOENT" "${temp_out}.comm"; then + ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' \ + "${temp_out}.comm" | head -n 1) + if [ -n "${ls_pid}" ] && \ + perf script failed-syscalls-by-pid -i "${temp_data}" \ + "${ls_pid}" > "${temp_out}.pid" && \ + grep -q "err = ENOENT" "${temp_out}.pid"; then + passed=1 + break + fi + fi + done + if [ ! -s "${temp_data}" ]; then echo "Skipping test, perf record failed to create data" exit 2 fi - # Run the script and check output - if ! perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}"; then + if [ "$passed" -eq 0 ]; then echo "failed-syscalls-by-pid test failed." - err=1 - elif ! grep -n -q "err = ENOENT" "${temp_out}"; then - echo "Failed to find expected failed syscalls" - cat "${temp_out}" - err=1 - elif ! perf script failed-syscalls-by-pid -i "${temp_data}" "ls" > "${temp_out}.comm" || \ - ! grep -q "err = ENOENT" "${temp_out}.comm"; then - echo "failed-syscalls-by-pid comm filter test failed." - cat "${temp_out}.comm" + cat "${temp_out}" 2>/dev/null || true + cat "${temp_out}.comm" 2>/dev/null || true + cat "${temp_out}.pid" 2>/dev/null || true err=1 else - ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' "${temp_out}.comm" | head -n 1) - if [ -z "${ls_pid}" ] || \ - ! perf script failed-syscalls-by-pid -i "${temp_data}" \ - "${ls_pid}" > "${temp_out}.pid" || \ - ! grep -q "err = ENOENT" "${temp_out}.pid"; then - echo "failed-syscalls-by-pid PID filter test failed." - cat "${temp_out}.pid" 2>/dev/null || true - err=1 - else - echo "failed-syscalls-by-pid test passed." - fi + echo "failed-syscalls-by-pid test passed." fi rm -f "${temp_out}" "${temp_out}.comm" "${temp_out}.pid" }
tools/perf/tests/shell/test_failed_syscalls_python.sh +32 -22
82 lines not on this page · raw message
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 5/9] perf test: Deflake failed-syscalls Python shell tests under load
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
When running 'perf test' in parallel under heavy system load, a one-shot 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event' can exit in under 300 microseconds before 'ls's
PERF_RECORD_COMMand ENOENTsys_exitevents are captured, while also contending on ~/.debug build-id caching and BPF event synthesis.Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell after 'ls' so its events are flushed before the parent subshell exits, and wrap the record and check steps in a bounded retry loop (up to 5 attempts).
Quoted 4 lines
Fixes b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")Fixes 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Ian Rogers
reviewerPATCH v3 6/9 perf test: Reduce overhead and contention in Python shell tests
In reply to Ian RogersWhen all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh, test_rw_by_file_python.sh, test_rw_by_pid_python.sh, test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and test_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in test_compaction_times_python.sh.
--- tools/perf/tests/shell/test_check_perf_trace_python.sh | 8 +++++--- tools/perf/tests/shell/test_compaction_times_python.sh | 8 +++++--- .../tests/shell/test_event_analyzing_sample_python.sh | 3 ++- tools/perf/tests/shell/test_flamegraph_python.sh | 5 +++-- tools/perf/tests/shell/test_futex_contention_python.sh | 7 ++++--- tools/perf/tests/shell/test_gecko_python.sh | 3 ++- tools/perf/tests/shell/test_mem_phys_addr_python.sh | 8 +++++--- tools/perf/tests/shell/test_net_dropmonitor_python.sh | 9 +++++---- tools/perf/tests/shell/test_netdev_times_python.sh | 6 +++--- tools/perf/tests/shell/test_powerpc_hcalls_python.sh | 4 ++-- tools/perf/tests/shell/test_rw_by_file_python.sh | 5 +++-- tools/perf/tests/shell/test_rw_by_pid_python.sh | 4 ++-- tools/perf/tests/shell/test_rwtop_python.sh | 4 ++-- tools/perf/tests/shell/test_sched_migration_python.sh | 4 ++-- tools/perf/tests/shell/test_stackcollapse_python.sh | 2 +- .../tests/shell/test_syscall_counts_by_pid_python.sh | 6 +++--- tools/perf/tests/shell/test_syscall_counts_python.sh | 5 +++-- tools/perf/tests/shell/test_task_analyzer.sh | 3 ++- tools/perf/tests/shell/test_wakeup_latency_python.sh | 7 ++++--- 19 files changed, 58 insertions(+), 43 deletions(-)
tools/perf/tests/shell/test_check_perf_trace_python.sh +5 -3
@@ -45,10 +45,11 @@ test_file_mode() { echo "Testing check-perf-trace.py..." events="" - if perf list | grep -q "irq:softirq_entry"; then + tp_list=$(perf list tracepoint) + if echo "$tp_list" | grep -q "irq:softirq_entry"; then events="irq:softirq_entry" fi - if perf list | grep -q "kmem:kmalloc"; then + if echo "$tp_list" | grep -q "kmem:kmalloc"; then if [ -n "$events" ]; then events="$events,kmem:kmalloc,kmem:kfree" else
@@ -62,7 +63,8 @@ test_file_mode() { fi # Generate events - if ! perf record -e "$events" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e "$events" -o "${temp_data}" \ + -- sh -c "sleep 0.1" >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
tools/perf/tests/shell/test_compaction_times_python.sh +5 -3
@@ -44,15 +44,17 @@ test_file_mode() { echo "Testing compaction-times.py..." # Check for any compaction events to see if kernel supports it - if ! perf list | grep -q "compaction:mm_compaction_begin"; then + if ! perf list tracepoint | grep -q "compaction:mm_compaction_begin"; then echo "Skipping test, compaction tracepoints not found" exit 2 fi # Generate some events - # We might not naturally trigger compaction in 0.5s sleep, but the script + # We might not naturally trigger compaction in 0.1s sleep, but the script # should parse the empty or sparse file correctly without crashing. - if ! perf record -e "compaction:*" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event \ + -e "compaction:mm_compaction_begin,compaction:mm_compaction_end" \ + -a -o "${temp_data}" -- sleep 0.1 >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +2 -1
@@ -39,7 +39,8 @@ test_file_mode() { echo "Testing event_analyzing_sample.py..." # Generate some events - if ! perf record -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record failed" exit 2 fi
tools/perf/tests/shell/test_flamegraph_python.sh +3 -2
@@ -60,7 +60,8 @@ test_file_mode() { echo "Testing flamegraph.py..." # Generate some events with callchains - if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record -g failed (permissions or lack of support)" exit 2 fi
@@ -75,7 +76,7 @@ test_file_mode() { # Run the script in pipe ('-') mode and validate JSON output rm -f "${temp_json}" - if ! perf record -g -o - -- perf test -w noploop 2>/dev/null | \ + if ! perf record -B -N --no-bpf-event -g -o - -- perf test -w noploop 2>/dev/null | \ perf script flamegraph -i - -f json -o "${temp_json}" >/dev/null; then echo "Pipe stdin JSON mode test failed." err=1
tools/perf/tests/shell/test_futex_contention_python.sh +4 -3
@@ -89,14 +89,15 @@ EOF test_file_mode() { echo "Testing futex-contention.py..." # Some systems might not have syscalls:sys_enter_futex - if ! perf list | grep -q syscalls:sys_enter_futex; then + if ! perf list tracepoint | grep -q syscalls:sys_enter_futex; then echo "Skipping file mode test, syscalls:sys_enter_futex not found" return fi # Generate some futex events - if ! perf record -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \ - -- sleep 0.5 2>/dev/null; then + if ! perf record -B -N --no-bpf-event \ + -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \ + -- sleep 0.1 2>/dev/null; then echo "Skipping file mode test (record failed)" return fi
tools/perf/tests/shell/test_gecko_python.sh +2 -1
@@ -39,7 +39,8 @@ test_file_mode() { echo "Testing gecko.py..." # Generate some events with callchains - if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping test, perf record -g failed (permissions or lack of support)" exit 2 fi
tools/perf/tests/shell/test_mem_phys_addr_python.sh +5 -3
@@ -78,10 +78,12 @@ test_file_mode() { echo "Testing mem-phys-addr.py file mode..." # Generate memory access events (try unprivileged user-space first, then system-wide) - if ! perf record --phys-data -d -o "${temp_data}" \ + if ! perf record -B -N --no-bpf-event --phys-data -d -o "${temp_data}" \ -- perf test -w datasym >/dev/null 2>&1 && \ - ! perf record -d -o "${temp_data}" -- perf test -w datasym >/dev/null 2>&1 && \ - ! perf record -d -a -o "${temp_data}" -- sleep 0.2 >/dev/null 2>&1; then + ! perf record -B -N --no-bpf-event -d -o "${temp_data}" \ + -- perf test -w datasym >/dev/null 2>&1 && \ + ! perf record -B -N --no-bpf-event -d -a -o "${temp_data}" \ + -- sleep 0.1 >/dev/null 2>&1; then echo "Skipping file mode record test, perf record -d not supported" return 0 fi
tools/perf/tests/shell/test_net_dropmonitor_python.sh +5 -4
@@ -42,11 +42,12 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing net_dropmonitor.py..." # Create a perf.data file. Force dropping a packet if tracepoint is available! -if ! perf record -e skb:kfree_skb -o "${temp_data}" -a \ - -- ping -c 1 255.255.255.255 >/dev/null 2>&1; then - if ! perf record -e skb:kfree_skb -o "${temp_data}" \ +if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" -a \ + -- ping -c 1 -W 1 255.255.255.255 >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" \ -- sleep 0.1 >/dev/null 2>&1; then - if ! perf record -o "${temp_data}" -- uname >/dev/null 2>&1; then + if ! perf record -B -N --no-bpf-event -o "${temp_data}" \ + -- uname >/dev/null 2>&1; then echo "Skipping test, cannot record perf events" exit 2 fi
tools/perf/tests/shell/test_netdev_times_python.sh +3 -3
@@ -85,9 +85,9 @@ then fi # Create a perf.data file. Force dropping a packet if tracepoint is available! -if ! perf record -e skb:kfree_skb -a -o "${temp_data}" \ - -- ping -c 1 127.0.0.1 >/dev/null 2>&1; then - perf record -e cycles -o "${temp_data}" \ +if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -a -o "${temp_data}" \ + -- ping -c 1 -W 1 127.0.0.1 >/dev/null 2>&1; then + perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } fi
tools/perf/tests/shell/test_powerpc_hcalls_python.sh +2 -2
@@ -67,8 +67,8 @@ if ! grep -q "H_REMOVE.*1.*1500.*1500.*1500" "${temp_out}"; then fi # Create a perf.data file if powerpc hcall tracepoints are available on this host. -if ! perf record -e powerpc:hcall_entry,powerpc:hcall_exit -a -o "${temp_data}" \ - -- perf test -w noploop >/dev/null 2>&1; then +if ! perf record -B -N --no-bpf-event -e powerpc:hcall_entry,powerpc:hcall_exit \ + -a -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then echo "Skipping live record test, powerpc hcall tracepoints not available" exit 0 fi
tools/perf/tests/shell/test_rw_by_file_python.sh +3 -2
@@ -37,8 +37,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing rw-by-file.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "syscalls:sys_enter_read"; then - perf record -e syscalls:sys_enter_read,syscalls:sys_enter_write -a -o "${temp_data}" \ +if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then + perf record -B -N --no-bpf-event -e syscalls:sys_enter_read,syscalls:sys_enter_write \ + -o "${temp_data}" \ -- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
tools/perf/tests/shell/test_rw_by_pid_python.sh +2 -2
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing rw-by-pid.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "syscalls:sys_enter_read"; then +if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then ev="syscalls:sys_enter_read,syscalls:sys_exit_read" ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write" - perf record -e "$ev" -a -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \ -- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
tools/perf/tests/shell/test_rwtop_python.sh +2 -2
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing rwtop.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "syscalls:sys_enter_read"; then +if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then ev="syscalls:sys_enter_read,syscalls:sys_exit_read" ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write" - perf record -e "$ev" -a -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \ -- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
tools/perf/tests/shell/test_sched_migration_python.sh +2 -2
@@ -44,10 +44,10 @@ echo "Testing sched-migration.py..." ev="sched:sched_switch,sched:sched_migrate_task" ev="${ev},sched:sched_wakeup_new,sched:sched_wakeup" has_sched=1 -if ! perf record -e "$ev" -a -o "${temp_data}" \ +if ! perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \ -- sleep 0.1 >/dev/null 2>&1; then has_sched=0 - perf record -e cycles -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } fi
tools/perf/tests/shell/test_stackcollapse_python.sh +1 -1
@@ -37,7 +37,7 @@ echo "Testing stackcollapse.py..." # Create a perf.data file with callchains. Use a busy workload rather than # sleep, as an idle system may not generate any samples at all. -perf record -g -o "${temp_data}" \ +perf record -B -N --no-bpf-event -g -o "${temp_data}" \ -- perf test -w noploop >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; }
tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh +3 -3
@@ -43,14 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) test_file_mode() { echo "Testing syscall-counts-by-pid.py..." # Some systems might not have raw_syscalls:sys_enter - if ! perf list | grep -q raw_syscalls:sys_enter; then + if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then echo "Skipping test, raw_syscalls:sys_enter not found" exit 2 fi # Generate some syscall events - perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \ - -- sleep 0.5 >/dev/null 2>&1 || \ + perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \ + -- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } if ! perf script syscall-counts-by-pid -i "${temp_data}" > "${temp_out}"; then
tools/perf/tests/shell/test_syscall_counts_python.sh +3 -2
@@ -43,13 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) test_file_mode() { echo "Testing syscall-counts.py..." # Some systems might not have raw_syscalls:sys_enter (e.g. stripped kernels or permissions) - if ! perf list | grep -q raw_syscalls:sys_enter; then + if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then echo "Skipping test, raw_syscalls:sys_enter not found" exit 2 fi # Generate some syscall events - if ! perf record -e raw_syscalls:sys_enter -o "${temp_data}" -- sleep 0.5 2>/dev/null; then + if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \ + -- sh -c "sleep 0.1; sleep 0.05" 2>/dev/null; then echo "perf record failed (permissions?), skipping file mode test." exit 2 fi
tools/perf/tests/shell/test_task_analyzer.sh +2 -1
@@ -60,7 +60,8 @@ skip_no_probe_record_support() { prepare_perf_data() { # 1s should be sufficient to catch at least some switches - perf record -e sched:sched_switch -a -o "${perfdata}" -- sleep 1 > /dev/null 2>&1 + perf record -B -N --no-bpf-event -e sched:sched_switch -a -o "${perfdata}" \ + -- sleep 1 > /dev/null 2>&1 # check if perf data file got created in above step. if [ ! -e "${perfdata}" ]; then printf "FAIL: perf record failed to create \"${perfdata}\" \n"
tools/perf/tests/shell/test_wakeup_latency_python.sh +4 -3
@@ -41,9 +41,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX) echo "Testing wakeup-latency.py..." # Create a perf.data file. Try to get tracepoint data. -if perf list | grep -q "sched:sched_wakeup"; then +if perf list tracepoint | grep -q "sched:sched_wakeup"; then ev="sched:sched_wakeup,sched:sched_wakeup_new,sched:sched_switch" - perf record -e "$ev" -a -o "${temp_data}" \ + perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \ -- sleep 0.1 >/dev/null 2>&1 || \ { echo "Skipping test, perf record failed"; exit 2; } else
@@ -70,7 +70,8 @@ else fi # Also test zero-wakeups / unhandled events path to verify division-by-zero protection -if perf record -e cycles -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then +if perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \ + -- perf test -w noploop >/dev/null 2>&1; then if ! perf script wakeup-latency -i "${temp_data}" > "${temp_out}" || \ ! grep -q "avg_wakeup_latency (ns): N/A" "${temp_out}"; then echo "wakeup-latency zero-wakeups guard test failed"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 6/9] perf test: Reduce overhead and contention in Python shell tests
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 18 lines
When all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' intest_check_perf_trace_python.sh,test_rw_by_file_python.sh,test_rw_by_pid_python.sh,test_rwtop_python.sh, andtest_syscall_counts_by_pid_python.shwhere only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' intest_net_dropmonitor_python.shandtest_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' intest_compaction_times_python.sh.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Ian Rogers
reviewerPATCH v3 7/9 perf python event_analyzing_sample: Default to in-memory SQLite database
In reply to Ian RogersWhen -d/--db is not specified, event_analyzing_sample.py created a temporary file in /tmp via tempfile.mkstemp() and deleted it in trace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.
Default to sqlite3.connect(":memory:") when db_path is not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode in test_event_analyzing_sample_python.sh.
--- tools/perf/python/event_analyzing_sample.py | 35 ++++++------------- .../test_event_analyzing_sample_python.sh | 8 ++++- 2 files changed, 17 insertions(+), 26 deletions(-)
tools/perf/python/event_analyzing_sample.py +10 -25
@@ -17,10 +17,8 @@ from __future__ import annotations import argparse import math -import os import sqlite3 import struct -import tempfile from typing import Any import perf
@@ -117,16 +115,11 @@ session: Any = None class _DB: con: sqlite3.Connection | None = None - temp_path: str | None = None def trace_begin(db_path: str | None = None) -> None: """Initialize database tables.""" print("In trace_begin:\n") - if not db_path: - fd, db_path = tempfile.mkstemp(prefix="perf_events_", suffix=".db") - os.close(fd) - _DB.temp_path = db_path - con = sqlite3.connect(db_path) + con = sqlite3.connect(db_path or ":memory:") try: # Drop any pre-existing tables so repeated runs do not accumulate duplicate events. con.execute("drop table if exists gen_events;")
@@ -297,28 +290,20 @@ def show_pebs_ll() -> None: def trace_end() -> None: """Called at the end of trace processing.""" print("In trace_end:\n") - try: - if _DB.con: - try: - _DB.con.commit() - show_general_events() - show_pebs_ll() - finally: - _DB.con.close() - _DB.con = None - finally: - if _DB.temp_path and os.path.exists(_DB.temp_path): - try: - os.remove(_DB.temp_path) - except OSError: - pass - _DB.temp_path = None + if _DB.con: + try: + _DB.con.commit() + show_general_events() + show_pebs_ll() + finally: + _DB.con.close() + _DB.con = None if __name__ == "__main__": ap = argparse.ArgumentParser(description="Analyze events with SQLite") ap.add_argument("-i", "--input", default="perf.data", help="Input file name") ap.add_argument("-d", "--db", "--database", dest="database", default=None, - help="Database file name (defaults to a temporary file cleaned up on exit)") + help="Database file name (defaults to an in-memory database)") args = ap.parse_args() try:
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +7 -1
@@ -45,8 +45,14 @@ test_file_mode() { exit 2 fi - # Run the script + # Run the script with default (:memory:) database and with explicit -d path if ! perf script event_analyzing_sample -i "${temp_data}" \ + > "${temp_dir}/perf.mem.out" 2>&1 || \ + ! grep -q "Statistics about the general events" "${temp_dir}/perf.mem.out" || \ + grep -q "Error creating/inserting event" "${temp_dir}/perf.mem.out"; then + echo "Default in-memory database mode test failed." + err=1 + elif ! perf script event_analyzing_sample -i "${temp_data}" \ -d "${temp_db}" > "${temp_dir}/perf.out" 2>&1; then echo "File mode test failed." err=1
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines
When -d/--db is not specified,
event_analyzing_sample.pycreated a temporary file in /tmp viatempfile.mkstemp()and deleted it intrace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.
Quoted 8 lines
Default to
sqlite3.connect(":memory:")whendb_pathis not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode intest_event_analyzing_sample_python.sh.Fixes eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Ian Rogers
reviewerPATCH v3 8/9 perf python: Initialize debug output on module load
In reply to Ian RogersUnlike the perf binary's main(), PyInit_perf() does not call perf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hits debug_file() with _debug_file == NULL and emits 'debug_file not set' without a trailing newline before the actual diagnostic message.
Call perf_debug_setup() in PyInit_perf() and add the missing trailing newline to the fallback warning in debug_file().
--- tools/perf/util/debug.c | 2 +- tools/perf/util/python.c | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
tools/perf/util/debug.c +1 -1
@@ -55,7 +55,7 @@ FILE *debug_file(void) {if (!_debug_file) {debug_set_file (stderr); -pr_warning_once ("debug_file not set" ); +pr_warning_once ("%s not set\n" , __func__); }return _debug_file; }
tools/perf/util/python.c +1 -0
@@ -5157,6 +5157,7 @@ PyMODINIT_FUNC PyInit_perf(void)/* The page_size is placed in util object. */ page_size =sysconf (_SC_PAGE_SIZE); +perf_debug_setup ();Py_INCREF (&pyrf_evlist__type);PyModule_AddObject (module,"evlist" , (PyObject *)&pyrf_evlist__type);
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 8/9] perf python: Initialize debug output on module load
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines
Unlike the perf binary's
main(),PyInit_perf()does not callperf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hitsdebug_file()with_debug_file== NULL and emits 'debug_filenot set' without a trailing newline before the actual diagnostic message.
Quoted 6 lines
Call
perf_debug_setup()inPyInit_perf()and add the missing trailing newline to the fallback warning indebug_file().Fixes ec49230cf6dd ("perf debug: Expose debug file")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Ian Rogers
reviewerPATCH v3 9/9 perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
In reply to Ian Rogersperf_pmus__print_pmu_events() first counts events via perf_pmu__num_events() to allocate the aliases array, then populates it via perf_pmu__for_each_event(). When dynamic tracepoints (such as kprobes or uprobes during 'perf test' runs) are concurrently created or removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len, leaving trailing zeroed entries in aliases[] with name == NULL and pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning /sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns -ENOENT and aborts enumeration of all remaining tracepoint subsystems.
- If an event is added, perf_pmus__print_pmu_events__callback() aborts enumeration when state->index reaches state->aliases_len.
Grow state->aliases dynamically via realloc() when needed, use state.index as the actual populated length for sorting and printing, and ignore -ENOENT when a tracepoint subsystem directory disappears during enumeration.
--- tools/perf/util/pmus.c | 21 +++++++++++++++++---- tools/perf/util/tp_pmu.c | 8 ++++++-- 2 files changed, 23 insertions(+), 6 deletions(-)
tools/perf/util/pmus.c +17 -4
@@ -8,6 +8,7 @@#include <sys/types.h> #include <ctype.h> #include <pthread.h> +#include <stdlib.h> #include <string.h> #include <unistd.h> #include "cpumap.h"
@@ -571,7 +572,7 @@ static int cmp_sevent(const void *a, const void *b) }/* Order by event name. */ -return strcmp (as->name, bs->name); +return strcmp (as->name ?:"" , bs->name ?:"" ); }static bool pmu_alias_is_duplicate (struct sevent *a,struct sevent *b)
@@ -581,7 +582,7 @@ static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)return false;/* Don't remove duplicates for different PMUs */ -return strcmp (a->pmu_name, b->pmu_name) ==0 ; +return strcmp (a->pmu_name ?:"" , b->pmu_name ?:"" ) ==0 ; }struct events_callback_state {
@@ -597,8 +598,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,struct sevent *s;if (state->index >= state->aliases_len) { -pr_err ("Unexpected event %s/%s/\n" , info->pmu->name, info->name); -return 1 ; +size_t new_len =max_t (size_t,16 , state->aliases_len *2 ); +struct sevent *new_aliases; + + new_aliases =realloc (state->aliases, new_len *sizeof (struct sevent )); +if (!new_aliases) { +pr_err ("Unexpected event %s/%s/\n" , info->pmu->name, info->name); +return 1 ; + } +memset (&new_aliases[state->aliases_len],0 , + (new_len - state->aliases_len) *sizeof (struct sevent )); + state->aliases = new_aliases; + state->aliases_len = new_len; }assert (info->pmu !=NULL || info->name !=NULL ); s = &state->aliases[state->index];
@@ -654,6 +665,8 @@ void perf_pmus__print_pmu_events(const struct print_callbacks *print_cb, void *pperf_pmu__for_each_event (pmu, skip_duplicate_pmus, &state, perf_pmus__print_pmu_events__callback); } + aliases = state.aliases; + len = state.index;qsort (aliases, len,sizeof (struct sevent ), cmp_sevent);for (int j =0 ; j < len; j++) {/* Skip duplicates */
tools/perf/util/tp_pmu.c +6 -2
@@ -151,7 +151,9 @@ static int for_each_event_cb(void *state, const char *sys_name, const char *evt_static int for_each_event_sys_cb (void *state,const char *sys_name) { -return tp_pmu__for_each_tp_event (sys_name, state, for_each_event_cb); +int ret =tp_pmu__for_each_tp_event (sys_name, state, for_each_event_cb); + +return ret == -ENOENT ?0 : ret; }int tp_pmu__for_each_event (struct perf_pmu *pmu,void *state,pmu_event_callback cb)
@@ -176,7 +178,9 @@ static int num_events_cb(void *state, const char *sys_name __maybe_unused,static int num_events_sys_cb (void *state,const char *sys_name) { -return tp_pmu__for_each_tp_event (sys_name, state, num_events_cb); +int ret =tp_pmu__for_each_tp_event (sys_name, state, num_events_cb); + +return ret == -ENOENT ?0 : ret; }size_t tp_pmu__num_events (struct perf_pmu *pmu__maybe_unused )
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
[email protected]
AI reviewRe: [PATCH v3 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 20 lines
perf_pmus__print_pmu_events()first counts events viaperf_pmu__num_events()to allocate the aliases array, then populates it viaperf_pmu__for_each_event(). When dynamic tracepoints (such as kprobes or uprobes during 'perf test' runs) are concurrently created or removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len, leaving trailing zeroed entries in aliases[] with name == NULL and pmu == NULL, which causesqsort(cmp_sevent)or the print loop to crash with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning/sys/kernel/tracing/events,tp_pmu__for_each_tp_event()returns-ENOENTand aborts enumeration of all remaining tracepoint subsystems.
- If an event is added,perf_pmus__print_pmu_events__callback()aborts enumeration whenstate->indexreachesstate->aliases_len.Grow
state->aliasesdynamically viarealloc()when needed, use state.index as the actual populated length for sorting and printing, and ignore-ENOENTwhen a tracepoint subsystem directory disappears during enumeration.
Quoted 4 lines
Fixes c3245d2093c1 ("perf pmu: Abstract alias/event struct")Fixes 45b6e281cb06 ("perf tp_pmu: Add event APIs")Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Quoted 3 lines · Ian Rogers
This series contains follow-up fixes for the standalone Python scripts and their shell tests, particularly when running 'perf test' as root under heavy parallel load:
Thanks, applied to perf-tools-next, for v7.4.
- Arnaldo