Skip to contentSkip to conversation

perf python: Follow-up fixes and shell test deflaking

PATCH Superseded Ian Rogers 3 participants 58 messages Sep 29 perf events

lore
Share & tools
lore

Ian Rogers

author reviewer
#0 lore raw

cover perf python: Follow-up fixes and shell test deflaking

This series contains follow-up fixes for the standalone Python scripts and their shell tests, particularly when running 'perf test' as root under heavy parallel load:

- Clamp sample_id size calculation for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE records to work around kernels without "perf/core: Restore header fields in sideband output callbacks" [1].
- Fix offline interval printing in sctop.py and live signal races in stat-cpi.py.
- Deflake Intel PT, failed-syscalls, and other Python shell tests under parallel 'perf test' load by scoping tracepoint checks to 'perf list tracepoint', passing '-B -N --no-bpf-event' to 'perf record', and avoiding unnecessary system-wide ('-a') recordings.
- Default event_analyzing_sample.py to an in-memory SQLite database (':memory:') when '--db' is not specified.
- Initialize perf debug output ('perf_debug_setup()') when loading the 'perf' Python extension module.
- Fix races in 'perf list' ('perf_pmus__print_pmu_events()' and 'tp_pmu.c') when concurrent tests dynamically create and remove kprobe/uprobes tracepoints.

[1] https://lore.kernel.org/r/[email protected]

Ian Rogers (9):
perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke
events
perf python sctop: Fix offline interval printing and test flakiness perf python stat-cpi: Fix live mode signal races and test flakiness perf test: Deflake Intel PT Python shell tests under load perf test: Deflake failed-syscalls Python shell tests under load perf test: Reduce overhead and contention in Python shell tests perf python event_analyzing_sample: Default to in-memory SQLite
database
perf python: Initialize debug output on module load
perf pmu: Fix race with concurrent tracepoint creation and removal in
perf list

 tools/perf/python/event_analyzing_sample.py   | 35 +++-------
 tools/perf/python/sctop.py                    |  9 ++-
 tools/perf/python/stat-cpi.py                 | 19 +++---
 .../shell/test_check_perf_trace_python.sh     |  8 ++-
 .../shell/test_compaction_times_python.sh     |  8 ++-
 .../test_event_analyzing_sample_python.sh     | 11 ++-
 .../shell/test_export_to_postgresql_python.sh | 49 +++++++------
 .../shell/test_export_to_sqlite_python.sh     | 50 ++++++++------
 .../test_failed_syscalls_by_pid_python.sh     | 68 +++++++++++--------
 .../shell/test_failed_syscalls_python.sh      | 44 +++++++-----
 .../tests/shell/test_flamegraph_python.sh     |  5 +-
 .../shell/test_futex_contention_python.sh     |  7 +-
 tools/perf/tests/shell/test_gecko_python.sh   |  3 +-
 .../shell/test_intel_pt_events_python.sh      | 43 ++++++------
 .../tests/shell/test_mem_phys_addr_python.sh  |  8 ++-
 .../shell/test_net_dropmonitor_python.sh      |  9 +--
 .../tests/shell/test_netdev_times_python.sh   |  6 +-
 .../tests/shell/test_powerpc_hcalls_python.sh |  4 +-
 .../tests/shell/test_rw_by_file_python.sh     |  5 +-
 .../perf/tests/shell/test_rw_by_pid_python.sh |  4 +-
 tools/perf/tests/shell/test_rwtop_python.sh   |  4 +-
 .../shell/test_sched_migration_python.sh      |  4 +-
 tools/perf/tests/shell/test_sctop_python.sh   | 48 ++++++-------
 .../tests/shell/test_stackcollapse_python.sh  |  2 +-
 .../perf/tests/shell/test_stat_cpi_python.sh  | 12 +++-
 .../test_syscall_counts_by_pid_python.sh      |  6 +-
 .../tests/shell/test_syscall_counts_python.sh |  5 +-
 tools/perf/tests/shell/test_task_analyzer.sh  |  3 +-
 .../tests/shell/test_wakeup_latency_python.sh |  7 +-
 tools/perf/util/debug.c                       |  2 +-
 tools/perf/util/evlist.c                      |  6 +-
 tools/perf/util/evsel.c                       | 59 +++++++++++++++-
 tools/perf/util/evsel.h                       |  1 +
 tools/perf/util/pmus.c                        | 20 ++++--
 tools/perf/util/python.c                      |  1 +
 tools/perf/util/tp_pmu.c                      |  8 ++-
 36 files changed, 358 insertions(+), 225 deletions(-)
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
author reviewer
#1 lore raw

PATCH 1/9 perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events

On kernels without the fix ("perf/core: Restore header fields in sideband output callbacks") to restore event_id.header.size in perf_event_ksymbol_output(), perf_event_bpf_output(), and perf_event_text_poke_output(), concurrent perf sessions cause those sideband records to be emitted with header.size inflated by multiple id_header_size increments while the single id_sample is written immediately after the event payload. Indexing backwards from event->header.size reads uninitialized ring-buffer bytes at the end of the record, causing evlist__event2evsel() to fail with -EFAULT.

Add evsel__event_size() to clamp the effective size used to locate the trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving event->header.size intact for advancing the ring-buffer/file stream.

Fixes 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")
Fixes 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")
Fixes 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/util/evlist.c |  6 ++--
 tools/perf/util/evsel.c  | 59 +++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/evsel.h  |  1 +
 3 files changed, 63 insertions(+), 3 deletions(-)
tools/perf/util/evlist.c +4 -2
index 9392d912d254..c2402e4791b6 100644
@@ -973,13 +973,15 @@ static int evlist__event2id(struct evlist *evlist, union perf_event *event, u64
 	const __u64 *array = event->sample.array;
 	ssize_t n;
 
-	n = (event->header.size - sizeof(event->header)) >> 3;
-
 	if (event->header.type == PERF_RECORD_SAMPLE) {
+		n = (event->header.size - sizeof(event->header)) >> 3;
 		if (evlist__id_pos(evlist) >= n)
 			return -1;
 		*id = array[evlist__id_pos(evlist)];
 	} else {
+		u16 size = evsel__event_size(evlist__first(evlist), event);
+
+		n = (size - sizeof(event->header)) >> 3;
 		if (evlist__is_pos(evlist) > n)
 			return -1;
 		n -= evlist__is_pos(evlist);
tools/perf/util/evsel.c +58 -1

71 lines not on this page · raw message

tools/perf/util/evsel.h +1 -0
index 5c5799cee601..174f3414fd3c 100644
@@ -469,6 +469,7 @@ int evsel__parse_sample_timestamp(struct evsel *evsel, union perf_event *event,
 				  u64 *timestamp);
 
 u16 evsel__id_hdr_size(const struct evsel *evsel);
+u16 evsel__event_size(const struct evsel *evsel, const union perf_event *event);
 
 static inline struct evsel *evsel__next(struct evsel *evsel)
 {
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#2 lore raw

Re: [PATCH v1 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf evsel: Incorrect PERF_RECORD_TEXT_POKE payload size calculation --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf evsel: Incorrect PERF_RECORD_TEXT_POKE payload size calculation --

commit fc6474e9b3305fb280e04ce2b30b2a815e2f3a7e
Author: Ian Rogers <[email protected]>

perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events

This commit adds evsel__event_size() to clamp the effective size used to locate the trailing id_sample for specific events. This prevents user-space tools from reading uninitialized memory when processing data from unpatched kernels.

Quoted 4 lines
tools/perf/util/evsel.c +0 -0
index 3367242c57640..70da6be798cca 100644

[ ... ]

Quoted 6 lines
tools/perf/util/evsel.c +2 -0
@@ -3966,6 +3966,63 @@ u16 evsel__id_hdr_size(const struct evsel *evsel)
 	return size;
 }
 
+/*
+ * Prior to kernel fix, perf_event_ksymbol_output(), perf_event_bpf_output(),

[ ... ]

tools/perf/util/evsel.c u16 evsel__id_hdr_size(const struct evsel *evsel) · line 4013

Quoted 7 lines
tools/perf/util/evsel.c +7 -0
+	case PERF_RECORD_TEXT_POKE: {
+		size_t fixed = offsetof(struct perf_record_text_poke_event, bytes);
+
+		if (size < fixed)
+			return size;
+		payload = fixed + PERF_ALIGN((size_t)event->text_poke.old_len +
+					     event->text_poke.new_len, sizeof(u64));

[Severity: Medium]
Does this misapply the 8-byte alignment only to the variable lengths rather than the entire structure?

The kernel generates PERF_RECORD_TEXT_POKE events by padding the entire event size to an 8-byte boundary. The payload size before id_sample is 16 + ALIGN(old_len + new_len + 4, 8).

This calculates the user-space payload as
fixed + PERF_ALIGN(old_len + new_len, 8), where fixed = 20. Because 20 is not a multiple of 8, adding it outside the alignment yields a mathematically incorrect size.

For example, if old_len = 1 and new_len = 1, the kernel calculates 24 bytes, but the code here calculates 28 bytes. Calculating the wrong payload size causes perf to index incorrectly into event->sample.array, leading to parsing failures or out-of-bounds accesses.

Quoted 5 lines
tools/perf/util/evsel.c +5 -0
+		break;
+	}
+	default:
+		return size;
+	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author reviewer
#3 lore raw

PATCH 2/9 perf python sctop: Fix offline interval printing and test flakiness

In reply to Ian Rogers

In sctop.py, process_event() updated self.last_print_time on every sample in offline mode even when skip was true for non-matching comm events. When filtering by comm with a short interval (such as 'sleep 1' in test_sctop_python.sh) on a system-wide recording spanning more than 1 second under heavy load, non-matching samples could trigger print_current_totals() and clear self.syscalls, followed by finally:
printing a trailing empty table. In addition, analyzer.e_machine was initialized after session.process_events() instead of before.

Fix sctop.py by initializing analyzer.e_machine before session.process_events(), only advancing self.last_print_time when a sample is not skipped, and only calling print_current_totals() in
finally: if no table has been printed yet or remaining syscalls are pending.

In test_sctop_python.sh, use 'perf list tracepoint' instead of unfiltered 'perf list', record the child workload directly without '-a' and with '-B -N --no-bpf-event' to avoid system-wide ringbuffer overflow and synthesis contention, and retry up to 5 times if needed.

Fixes b83f0bacf5e4 ("perf python: Port sctop to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/python/sctop.py                  |  9 ++--
 tools/perf/tests/shell/test_sctop_python.sh | 48 +++++++++++----------
 2 files changed, 31 insertions(+), 26 deletions(-)
tools/perf/python/sctop.py +6 -3
index 42e95ecfffad..c524e08e6440 100755
@@ -37,6 +37,7 @@ class SCTopAnalyzer:
         self.offline = offline
         self.own_pid = os.getpid()
         self.last_print_time: Optional[int] = None
+        self.printed = False
         self.session: Optional[perf.session] = None
         self.e_machine: Optional[int] = None
 
@@ -127,7 +128,7 @@ class SCTopAnalyzer:
             if not skip and is_enter and 0 <= (syscall_id & ~0x40000000) <= 0xffff:
                 self.syscalls[syscall_id] += 1
 
-        if self.offline and hasattr(sample, "sample_time"):
+        if not skip and self.offline and hasattr(sample, "sample_time"):
             interval_ns = self.interval * (10 ** 9)
             if self.last_print_time is None:
                 self.last_print_time = sample.sample_time
@@ -137,6 +138,7 @@ class SCTopAnalyzer:
 
     def print_current_totals(self):
         """Print current syscall totals."""
+        self.printed = True
         # Clear terminal
         if not self.offline:
             print("\x1b[2J\x1b[H", end="")
@@ -217,8 +219,8 @@ def main():
         if args.input:
             session = perf.session(perf.data(args.input), sample=analyzer.process_event)
             analyzer.session = session
-            session.process_events()
             analyzer.e_machine = getattr(session, "e_machine", None)
+            session.process_events()
         else:
             try:
                 live_session = LiveSession(
@@ -237,7 +239,8 @@ def main():
         sys.exit(1)
     finally:
         if args.input:
-            analyzer.print_current_totals()
+            if not analyzer.printed or analyzer.syscalls:
+                analyzer.print_current_totals()
             # Break the reference cycle between perf.session and analyzer.process_event
             # because perf.session lacks cyclic GC support (tp_traverse).
             analyzer.session = None
tools/perf/tests/shell/test_sctop_python.sh +25 -23

62 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#4 lore raw

Re: [PATCH v1 2/9] perf python sctop: Fix offline interval printing and test flakiness

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf python: sctop offline interval bucketting broken by comm filter --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf python: sctop offline interval bucketting broken by comm filter --

commit b0674a2bbd03e3e74148a12d9d8a3c74b4f35ce5
Author: Ian Rogers <[email protected]>

perf python sctop: Fix offline interval printing and test flakiness

This commit addresses sctop offline interval printing by only advancing the logical clock on non-skipped samples. It also reorders e_machine initialization and updates the shell test with retries and different recording parameters.

Quoted 4 lines
tools/perf/python/sctop.py +0 -0
index 42e95ecfffad7..c524e08e64405 100755

[ ... ]

tools/perf/python/sctop.py class SCTopAnalyzer: · line 131

Quoted 6 lines
tools/perf/python/sctop.py +1 -1
@@ -127,7 +128,7 @@ class SCTopAnalyzer:
             if not skip and is_enter and 0 <= (syscall_id & ~0x40000000) <= 0xffff:
                 self.syscalls[syscall_id] += 1
 
-        if self.offline and hasattr(sample, "sample_time"):
+        if not skip and self.offline and hasattr(sample, "sample_time"):

[Severity: Medium]
Does adding 'not skip' to this condition in sctop.py's SCTopAnalyzer.process_event() introduce a regression by blocking the offline clock from advancing during periods where the target process is idle?

By ignoring the sample_time of non-matching events, the script's internal logical clock fails to advance. This prevents the current interval bucket from flushing until the target comm executes another syscall, merging events from multiple intervals into a single bucket and resulting in incorrect interval sums.

Does this also cause the script to miss printing periodic empty tables during idle periods, breaking consistency with live mode?

Quoted 3 lines
tools/perf/python/sctop.py +0 -0
             interval_ns = self.interval * (10 ** 9)
             if self.last_print_time is None:
                 self.last_print_time = sample.sample_time
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author reviewer
#5 lore raw

PATCH 3/9 perf python stat-cpi: Fix live mode signal races and test flakiness

In reply to Ian Rogers

In test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a 1-second duration and can exit under heavy parallel load before 'perf stat -p' and 'perf script stat-cpi' finish starting up. In addition, the fixed 'sleep 0.5' before sending SIGINT can fire before Python finishes importing the perf module, opening the live evlist, and flushing the first interval.

In stat-cpi.py, register SIGINT and SIGTERM handlers before calling _open_live_evlist() and pass flush=True when printing live output so redirected stdout is flushed immediately after each interval.

In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the target workload stays alive until killed, and poll the output file for 'cpi' (up to 5 seconds) before sending SIGINT.

Fixes 4425182d426b ("perf python: Port stat-cpi to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/python/stat-cpi.py                 | 19 ++++++++++---------
 .../perf/tests/shell/test_stat_cpi_python.sh  | 12 +++++++++---
 2 files changed, 19 insertions(+), 12 deletions(-)
tools/perf/python/stat-cpi.py +10 -9
index 0b7d76876a6c..da92cf560067 100755
@@ -106,7 +106,8 @@ class StatCpiAnalyzer:
             if ins != 0:
                 cpi = cyc / float(ins)
             t_sec = timestamp / 1000000000.0
-            print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})")
+            print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})",
+                  flush=True)
 
     def read_counters(self, evlist: Any) -> None:
         """Read counters live."""
@@ -173,19 +174,19 @@ class StatCpiAnalyzer:
 
     def run_live(self) -> None:
         """Read counters live."""
-        try:
-            evlist = self._open_live_evlist()
-        except OSError as e:
-            print(f"Failed to open events: {e}", file=sys.stderr)
-            sys.exit(1)
-
         def handle_signal(_signum: int, _frame: Any) -> None:
             raise KeyboardInterrupt
 
         signal.signal(signal.SIGINT, signal.default_int_handler)
         signal.signal(signal.SIGTERM, handle_signal)
 
-        print("Live mode started. Press Ctrl+C to stop.")
+        try:
+            evlist = self._open_live_evlist()
+        except OSError as e:
+            print(f"Failed to open events: {e}", file=sys.stderr)
+            sys.exit(1)
+
+        print("Live mode started. Press Ctrl+C to stop.", flush=True)
         try:
             while True:
                 time.sleep(self.args.interval)
@@ -195,7 +196,7 @@ class StatCpiAnalyzer:
                 self.data.clear()
                 self.recorded_pairs.clear()
         except KeyboardInterrupt:
-            print("\nStopped.")
+            print("\nStopped.", flush=True)
         finally:
             evlist.close()
 
tools/perf/tests/shell/test_stat_cpi_python.sh +9 -3
index fe7562307634..6cb376c92e2f 100755
@@ -50,7 +50,7 @@ test_live_mode() {
 		echo "perf stat failed (permissions?), skipping live mode test."
 		return 0
 	fi
-	perf test -w noploop &
+	perf test -w noploop 60 &
 	workload_pid=$!
 	if ! perf stat -e cycles,instructions -p "$workload_pid" -- sleep 0.05 2>/dev/null && \
 	   ! perf stat -e cycles:u,instructions:u -p "$workload_pid" -- sleep 0.05 2>/dev/null; then
@@ -61,10 +61,16 @@ test_live_mode() {
 	fi
 	ran=1
 
-	# Run live mode for 1 interval in the background, give it a tiny sleep, then interrupt
+	# Run live mode in the background, wait until at least one interval is
+	# printed, then interrupt.
 	perf script stat-cpi -I 0.1 -p "$workload_pid" > "${temp_out}" &
 	pid=$!
-	sleep 0.5
+	for _ in $(seq 1 50); do
+		if grep -q "cpi" "${temp_out}"; then
+			break
+		fi
+		sleep 0.1
+	done
 	kill -INT "$pid" 2>/dev/null || true
 	set +e
 	wait "$pid"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#6 lore raw

Re: [PATCH v1 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines

In test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a 1-second duration and can exit under heavy parallel load before 'perf stat -p' and 'perf script stat-cpi' finish starting up. In addition, the fixed 'sleep 0.5' before sending SIGINT can fire before Python finishes importing the perf module, opening the live evlist, and flushing the first interval.

In stat-cpi.py, register SIGINT and SIGTERM handlers before calling _open_live_evlist() and pass flush=True when printing live output so redirected stdout is flushed immediately after each interval.

Quoted 7 lines

In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the target workload stays alive until killed, and poll the output file for 'cpi' (up to 5 seconds) before sending SIGINT.

Fixes 4425182d426b ("perf python: Port stat-cpi to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author reviewer
#7 lore raw

PATCH 4/9 perf test: Deflake Intel PT Python shell tests under load

In reply to Ian Rogers

In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).

Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh to avoid build-id cache and BPF synthesis overhead.

Fixes d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
Fixes 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
Fixes b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 .../shell/test_export_to_postgresql_python.sh | 49 ++++++++++--------
 .../shell/test_export_to_sqlite_python.sh     | 50 +++++++++++--------
 .../shell/test_intel_pt_events_python.sh      | 43 +++++++++-------
 3 files changed, 79 insertions(+), 63 deletions(-)
tools/perf/tests/shell/test_export_to_postgresql_python.sh +27 -22
index 350813466700..835f48f002e9 100755
@@ -61,9 +61,10 @@ test_file_mode() {
 	fi
 
 	# Generate events with callchains and context switches
-	if ! perf record -g --switch-events -o "${temp_data}" \
+	if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \
 	     -- perf test -w noploop >/dev/null 2>&1 && \
-	   ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	   ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+	     -- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
@@ -92,29 +93,33 @@ test_file_mode() {
 test_intel_pt() {
 	echo "Testing export-to-postgresql.py with intel_pt..."
 
-	psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
-	rm -f "${temp_data}"
-	# Generate some intel_pt events; use a subshell that waits for uname
-	if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
-		-- sh -c "uname; true" >/dev/null 2>&1; then
-		echo "Skipping intel_pt test, intel_pt not available."
-		return 0
-	fi
+	# Generate some intel_pt events; sleep briefly after uname in the subshell
+	# so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+	passed=0
+	for _ in 1 2 3 4 5; do
+		psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
+		rm -f "${temp_data}"
+		if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+			-- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+			echo "Skipping intel_pt test, intel_pt not available."
+			return 0
+		fi
 
-	# Run the script with --itrace cr to synthesize call_returns
-	if ! perf script export-to-postgresql -i "${temp_data}" \
-		-o "${temp_db}" --itrace cr >/dev/null; then
-		echo "intel_pt file mode test failed."
-		err=1
-	else
-		# Check DB for calls
-		if ! psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
+		# Run the script with --itrace cr to synthesize call_returns
+		if perf script export-to-postgresql -i "${temp_data}" \
+			-o "${temp_db}" --itrace cr >/dev/null && \
+		   psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
 			grep -q '[1-9]'; then
-			echo "PostgreSQL intel_pt validation failed (no calls found)."
-			err=1
-		else
-			echo "intel_pt test passed (cr validated)."
+			passed=1
+			break
 		fi
+	done
+
+	if [ "$passed" -eq 0 ]; then
+		echo "PostgreSQL intel_pt validation failed (no calls found)."
+		err=1
+	else
+		echo "intel_pt test passed (cr validated)."
 	fi
 }
 
tools/perf/tests/shell/test_export_to_sqlite_python.sh +28 -22

66 lines not on this page · raw message

tools/perf/tests/shell/test_intel_pt_events_python.sh +24 -19

58 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#8 lore raw

Re: [PATCH v1 4/9] perf test: Deflake Intel PT Python shell tests under load

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines

In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).

Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh to avoid build-id cache and BPF synthesis overhead.

Quoted 5 lines
Fixes d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
Fixes 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
Fixes b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author reviewer
#9 lore raw

PATCH 5/9 perf test: Deflake failed-syscalls Python shell tests under load

In reply to Ian Rogers

When running 'perf test' in parallel under heavy system load, a one-shot 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event' can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and ENOENT sys_exit events are captured, while also contending on ~/.debug build-id caching and BPF event synthesis.

Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell after 'ls' so its events are flushed before the parent subshell exits, and wrap the record and check steps in a bounded retry loop (up to 5 attempts).

Fixes b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")
Fixes 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 .../test_failed_syscalls_by_pid_python.sh     | 68 +++++++++++--------
 .../shell/test_failed_syscalls_python.sh      | 44 +++++++-----
 2 files changed, 67 insertions(+), 45 deletions(-)
tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh +39 -29
index 215372d4e1c5..070a7ef6b8ad 100755
@@ -41,8 +41,10 @@ test_file_mode() {
 	echo "Testing failed-syscalls-by-pid.py..."
 
 	# Check if syscalls:sys_exit is supported/readable
-	if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
-		if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \
+		-o /dev/null -- true >/dev/null 2>&1; then
+		if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \
+			-o /dev/null -- true >/dev/null 2>&1; then
 			echo "Skipping test, no syscalls:sys_exit or raw_syscalls:sys_exit event"
 			exit 2
 		else
@@ -52,41 +54,49 @@ test_file_mode() {
 		EVENT="syscalls:sys_exit"
 	fi
 
-	# Generate some events by running a command that should fail at least some syscall
-	# (e.g. failing stat on non-existent file).
-	# Using '|| true' because 'perf record' returns the exit code of 'ls',
-	# which fails with ENOENT
-	perf record -e "${EVENT}" -o "${temp_data}" -- ls /does_not_exist >/dev/null 2>&1 || true
+	# Generate some events by running a command that fails a syscall
+	# (e.g. failing stat on non-existent file), sleeping briefly in the
+	# subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed.
+	passed=0
+	for _ in 1 2 3 4 5; do
+		rm -f "${temp_data}" "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
+		perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \
+			-- sh -c "ls /does_not_exist 2>/dev/null; sleep 0.05 || true" \
+			>/dev/null 2>&1 || true
+		if [ ! -s "${temp_data}" ]; then
+			continue
+		fi
+
+		if perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}" && \
+		   grep -n -q "err = ENOENT" "${temp_out}" && \
+		   perf script failed-syscalls-by-pid -i "${temp_data}" \
+			"ls" > "${temp_out}.comm" && \
+		   grep -q "err = ENOENT" "${temp_out}.comm"; then
+			ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' \
+				"${temp_out}.comm" | head -n 1)
+			if [ -n "${ls_pid}" ] && \
+			   perf script failed-syscalls-by-pid -i "${temp_data}" \
+				"${ls_pid}" > "${temp_out}.pid" && \
+			   grep -q "err = ENOENT" "${temp_out}.pid"; then
+				passed=1
+				break
+			fi
+		fi
+	done
+
 	if [ ! -s "${temp_data}" ]; then
 		echo "Skipping test, perf record failed to create data"
 		exit 2
 	fi
 
-	# Run the script and check output
-	if ! perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}"; then
+	if [ "$passed" -eq 0 ]; then
 		echo "failed-syscalls-by-pid test failed."
-		err=1
-	elif ! grep -n -q "err = ENOENT" "${temp_out}"; then
-		echo "Failed to find expected failed syscalls"
-		cat "${temp_out}"
-		err=1
-	elif ! perf script failed-syscalls-by-pid -i "${temp_data}" "ls" > "${temp_out}.comm" || \
-	     ! grep -q "err = ENOENT" "${temp_out}.comm"; then
-		echo "failed-syscalls-by-pid comm filter test failed."
-		cat "${temp_out}.comm"
+		cat "${temp_out}" 2>/dev/null || true
+		cat "${temp_out}.comm" 2>/dev/null || true
+		cat "${temp_out}.pid" 2>/dev/null || true
 		err=1
 	else
-		ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' "${temp_out}.comm" | head -n 1)
-		if [ -z "${ls_pid}" ] || \
-		   ! perf script failed-syscalls-by-pid -i "${temp_data}" \
-			"${ls_pid}" > "${temp_out}.pid" || \
-		   ! grep -q "err = ENOENT" "${temp_out}.pid"; then
-			echo "failed-syscalls-by-pid PID filter test failed."
-			cat "${temp_out}.pid" 2>/dev/null || true
-			err=1
-		else
-			echo "failed-syscalls-by-pid test passed."
-		fi
+		echo "failed-syscalls-by-pid test passed."
 	fi
 	rm -f "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
 }
tools/perf/tests/shell/test_failed_syscalls_python.sh +28 -16

64 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#10 lore raw

Re: [PATCH v1 5/9] perf test: Deflake failed-syscalls Python shell tests under load

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] perf test: TOCTOU symlink vulnerability via rm in /tmp --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] perf test: TOCTOU symlink vulnerability via rm in /tmp
--

commit 2edcd77bcabafd7550e8af4c57dda0ca7cfe9278
Author: Ian Rogers <[email protected]>

perf test: Deflake failed-syscalls Python shell tests under load

This commit updates the failed-syscalls python shell tests to handle heavy system load by adding a bounded retry loop and short sleep intervals. It also explicitly passes the no-bpf-event flags to perf record to prevent early exits during BPF event synthesis.

Quoted 4 lines
tools/perf/tests/shell/test_failed_syscalls_python.sh +0 -0
index 861c2ba71c0a2..88a42e3de4f35 100755

[ ... ]

tools/perf/tests/shell/test_failed_syscalls_python.sh else · line 61

Quoted 13 lines
tools/perf/tests/shell/test_failed_syscalls_python.sh +5 -4
@@ -52,28 +54,38 @@ else
 	EVENT="raw_syscalls:sys_exit"
 fi
 
-# Run perf record with a command that fails a syscall (ls non-existent file).
-# ls exits with non-zero, so perf record returns non-zero exit code of the workload.
-perf record -e "${EVENT}" -o "${temp_data}" \
-	-- ls /nonexistent_file_for_test >/dev/null 2>&1 || true
+# Run perf record with a command that fails a syscall (ls non-existent file),
+# sleeping briefly in the subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed.
+passed=0
+for _ in 1 2 3 4 5; do
+	rm -f "${temp_data}" "${temp_out}"

[Severity: High]
Does this rm command expose the script to a time-of-check to time-of-use symlink vulnerability?

The files temp_data and temp_out are initially created securely in the world-writable /tmp directory using mktemp. By explicitly deleting them here in the retry loop, the filenames are freed.

During the window between this deletion and the subsequent shell redirection, a local attacker observing /tmp could recreate these names as symlinks. When the test redirects output or writes to them later, it will follow the symlink and could overwrite arbitrary files on the system. This could lead to local privilege escalation if the test is run with elevated privileges.

Quoted 3 lines
tools/perf/tests/shell/test_failed_syscalls_python.sh +3 -0
+	perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \
+		-- sh -c "ls /nonexistent_file_for_test 2>/dev/null; sleep 0.05 || true" \
+		>/dev/null 2>&1 || true
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author reviewer
#11 lore raw

PATCH 6/9 perf test: Reduce overhead and contention in Python shell tests

In reply to Ian Rogers

When all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh, test_rw_by_file_python.sh, test_rw_by_pid_python.sh, test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and test_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in test_compaction_times_python.sh.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/tests/shell/test_check_perf_trace_python.sh   | 8 +++++---
 tools/perf/tests/shell/test_compaction_times_python.sh   | 8 +++++---
 .../tests/shell/test_event_analyzing_sample_python.sh    | 3 ++-
 tools/perf/tests/shell/test_flamegraph_python.sh         | 5 +++--
 tools/perf/tests/shell/test_futex_contention_python.sh   | 7 ++++---
 tools/perf/tests/shell/test_gecko_python.sh              | 3 ++-
 tools/perf/tests/shell/test_mem_phys_addr_python.sh      | 8 +++++---
 tools/perf/tests/shell/test_net_dropmonitor_python.sh    | 9 +++++----
 tools/perf/tests/shell/test_netdev_times_python.sh       | 6 +++---
 tools/perf/tests/shell/test_powerpc_hcalls_python.sh     | 4 ++--
 tools/perf/tests/shell/test_rw_by_file_python.sh         | 5 +++--
 tools/perf/tests/shell/test_rw_by_pid_python.sh          | 4 ++--
 tools/perf/tests/shell/test_rwtop_python.sh              | 4 ++--
 tools/perf/tests/shell/test_sched_migration_python.sh    | 4 ++--
 tools/perf/tests/shell/test_stackcollapse_python.sh      | 2 +-
 .../tests/shell/test_syscall_counts_by_pid_python.sh     | 6 +++---
 tools/perf/tests/shell/test_syscall_counts_python.sh     | 5 +++--
 tools/perf/tests/shell/test_task_analyzer.sh             | 3 ++-
 tools/perf/tests/shell/test_wakeup_latency_python.sh     | 7 ++++---
 19 files changed, 58 insertions(+), 43 deletions(-)
tools/perf/tests/shell/test_check_perf_trace_python.sh +5 -3
index 92cc7b1f0033..baf55945d429 100755
@@ -45,10 +45,11 @@ test_file_mode() {
 	echo "Testing check-perf-trace.py..."
 
 	events=""
-	if perf list | grep -q "irq:softirq_entry"; then
+	tp_list=$(perf list tracepoint)
+	if echo "$tp_list" | grep -q "irq:softirq_entry"; then
 		events="irq:softirq_entry"
 	fi
-	if perf list | grep -q "kmem:kmalloc"; then
+	if echo "$tp_list" | grep -q "kmem:kmalloc"; then
 		if [ -n "$events" ]; then
 			events="$events,kmem:kmalloc,kmem:kfree"
 		else
@@ -62,7 +63,8 @@ test_file_mode() {
 	fi
 
 	# Generate events
-	if ! perf record -e "$events" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -e "$events" -o "${temp_data}" \
+		-- sh -c "sleep 0.1" >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
tools/perf/tests/shell/test_compaction_times_python.sh +5 -3
index 50397afaafed..3f5a71e52388 100755
@@ -44,15 +44,17 @@ test_file_mode() {
 	echo "Testing compaction-times.py..."
 
 	# Check for any compaction events to see if kernel supports it
-	if ! perf list | grep -q "compaction:mm_compaction_begin"; then
+	if ! perf list tracepoint | grep -q "compaction:mm_compaction_begin"; then
 		echo "Skipping test, compaction tracepoints not found"
 		exit 2
 	fi
 
 	# Generate some events
-	# We might not naturally trigger compaction in 0.5s sleep, but the script
+	# We might not naturally trigger compaction in 0.1s sleep, but the script
 	# should parse the empty or sparse file correctly without crashing.
-	if ! perf record -e "compaction:*" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event \
+		-e "compaction:mm_compaction_begin,compaction:mm_compaction_end" \
+		-a -o "${temp_data}" -- sleep 0.1 >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +2 -1
index a7a8ded003db..5f0a41080acf 100755
@@ -39,7 +39,8 @@ test_file_mode() {
 	echo "Testing event_analyzing_sample.py..."
 
 	# Generate some events
-	if ! perf record -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+		-- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
tools/perf/tests/shell/test_flamegraph_python.sh +3 -2
index 86313c72f53e..787d6bbdf04a 100755
@@ -60,7 +60,8 @@ test_file_mode() {
 	echo "Testing flamegraph.py..."
 
 	# Generate some events with callchains
-	if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+		-- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record -g failed (permissions or lack of support)"
 		exit 2
 	fi
@@ -75,7 +76,7 @@ test_file_mode() {
 
 	# Run the script in pipe ('-') mode and validate JSON output
 	rm -f "${temp_json}"
-	if ! perf record -g -o - -- perf test -w noploop 2>/dev/null | \
+	if ! perf record -B -N --no-bpf-event -g -o - -- perf test -w noploop 2>/dev/null | \
 		perf script flamegraph -i - -f json -o "${temp_json}" >/dev/null; then
 		echo "Pipe stdin JSON mode test failed."
 		err=1
tools/perf/tests/shell/test_futex_contention_python.sh +4 -3
index a8846ed68ac2..41ce8470a75f 100755
@@ -89,14 +89,15 @@ EOF
 test_file_mode() {
 	echo "Testing futex-contention.py..."
 	# Some systems might not have syscalls:sys_enter_futex
-	if ! perf list | grep -q syscalls:sys_enter_futex; then
+	if ! perf list tracepoint | grep -q syscalls:sys_enter_futex; then
 		echo "Skipping file mode test, syscalls:sys_enter_futex not found"
 		return
 	fi
 
 	# Generate some futex events
-	if ! perf record -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
-		-- sleep 0.5 2>/dev/null; then
+	if ! perf record -B -N --no-bpf-event \
+		-e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
+		-- sleep 0.1 2>/dev/null; then
 		echo "Skipping file mode test (record failed)"
 		return
 	fi
tools/perf/tests/shell/test_gecko_python.sh +2 -1
index de1d7fcf88b7..18e9f5cd2ab0 100755
@@ -39,7 +39,8 @@ test_file_mode() {
 	echo "Testing gecko.py..."
 
 	# Generate some events with callchains
-	if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+		-- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record -g failed (permissions or lack of support)"
 		exit 2
 	fi
tools/perf/tests/shell/test_mem_phys_addr_python.sh +5 -3
index 451692d5e089..cbb9f572be9d 100755
@@ -78,10 +78,12 @@ test_file_mode() {
 	echo "Testing mem-phys-addr.py file mode..."
 
 	# Generate memory access events (try unprivileged user-space first, then system-wide)
-	if ! perf record --phys-data -d -o "${temp_data}" \
+	if ! perf record -B -N --no-bpf-event --phys-data -d -o "${temp_data}" \
 	     -- perf test -w datasym >/dev/null 2>&1 && \
-	   ! perf record -d -o "${temp_data}" -- perf test -w datasym >/dev/null 2>&1 && \
-	   ! perf record -d -a -o "${temp_data}" -- sleep 0.2 >/dev/null 2>&1; then
+	   ! perf record -B -N --no-bpf-event -d -o "${temp_data}" \
+	     -- perf test -w datasym >/dev/null 2>&1 && \
+	   ! perf record -B -N --no-bpf-event -d -a -o "${temp_data}" \
+	     -- sleep 0.1 >/dev/null 2>&1; then
 		echo "Skipping file mode record test, perf record -d not supported"
 		return 0
 	fi
tools/perf/tests/shell/test_net_dropmonitor_python.sh +5 -4
index 05056a897bc9..2e37994418fc 100755
@@ -42,11 +42,12 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing net_dropmonitor.py..."
 
 # Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -o "${temp_data}" -a \
-	-- ping -c 1 255.255.255.255 >/dev/null 2>&1; then
-	if ! perf record -e skb:kfree_skb -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" -a \
+	-- ping -c 1 -W 1 255.255.255.255 >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" \
 		-- sleep 0.1 >/dev/null 2>&1; then
-		if ! perf record -o "${temp_data}" -- uname >/dev/null 2>&1; then
+		if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+			-- uname >/dev/null 2>&1; then
 			echo "Skipping test, cannot record perf events"
 			exit 2
 		fi
tools/perf/tests/shell/test_netdev_times_python.sh +3 -3
index 393bf446efb2..0d8d10a42cab 100755
@@ -85,9 +85,9 @@ then
 fi
 
 # Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -a -o "${temp_data}" \
-	-- ping -c 1 127.0.0.1 >/dev/null 2>&1; then
-	perf record -e cycles -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -a -o "${temp_data}" \
+	-- ping -c 1 -W 1 127.0.0.1 >/dev/null 2>&1; then
+	perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
 		-- perf test -w noploop >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 fi
tools/perf/tests/shell/test_powerpc_hcalls_python.sh +2 -2
index 19569c6d3613..b140ccf00ffa 100755
@@ -67,8 +67,8 @@ if ! grep -q "H_REMOVE.*1.*1500.*1500.*1500" "${temp_out}"; then
 fi
 
 # Create a perf.data file if powerpc hcall tracepoints are available on this host.
-if ! perf record -e powerpc:hcall_entry,powerpc:hcall_exit -a -o "${temp_data}" \
-	-- perf test -w noploop >/dev/null 2>&1; then
+if ! perf record -B -N --no-bpf-event -e powerpc:hcall_entry,powerpc:hcall_exit \
+	-a -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
 	echo "Skipping live record test, powerpc hcall tracepoints not available"
 	exit 0
 fi
tools/perf/tests/shell/test_rw_by_file_python.sh +3 -2
index f121597f418f..e3c538c3ace3 100755
@@ -37,8 +37,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing rw-by-file.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
-	perf record -e syscalls:sys_enter_read,syscalls:sys_enter_write -a -o "${temp_data}" \
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
+	perf record -B -N --no-bpf-event -e syscalls:sys_enter_read,syscalls:sys_enter_write \
+		-o "${temp_data}" \
 		-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
tools/perf/tests/shell/test_rw_by_pid_python.sh +2 -2
index 41eaa29097a2..cd93c1d9714c 100755
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing rw-by-pid.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
 	ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
 	ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
-	perf record -e "$ev" -a -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
 		-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
tools/perf/tests/shell/test_rwtop_python.sh +2 -2
index 05897e384702..d33807923c92 100755
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing rwtop.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
 	ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
 	ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
-	perf record -e "$ev" -a -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
 		-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
tools/perf/tests/shell/test_sched_migration_python.sh +2 -2
index 75465bcd3791..714275b0e492 100755
@@ -44,10 +44,10 @@ echo "Testing sched-migration.py..."
 ev="sched:sched_switch,sched:sched_migrate_task"
 ev="${ev},sched:sched_wakeup_new,sched:sched_wakeup"
 has_sched=1
-if ! perf record -e "$ev" -a -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
 	-- sleep 0.1 >/dev/null 2>&1; then
 	has_sched=0
-	perf record -e cycles -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
 		-- perf test -w noploop >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 fi
tools/perf/tests/shell/test_stackcollapse_python.sh +1 -1
index ce0f80407eec..2048734c5e0d 100755
@@ -37,7 +37,7 @@ echo "Testing stackcollapse.py..."
 
 # Create a perf.data file with callchains. Use a busy workload rather than
 # sleep, as an idle system may not generate any samples at all.
-perf record -g -o "${temp_data}" \
+perf record -B -N --no-bpf-event -g -o "${temp_data}" \
 		-- perf test -w noploop >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 
tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh +3 -3
index 45bbdc554c8f..300289fe33b4 100755
@@ -43,14 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 test_file_mode() {
 	echo "Testing syscall-counts-by-pid.py..."
 	# Some systems might not have raw_syscalls:sys_enter
-	if ! perf list | grep -q raw_syscalls:sys_enter; then
+	if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
 		echo "Skipping test, raw_syscalls:sys_enter not found"
 		exit 2
 	fi
 
 	# Generate some syscall events
-	perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \
-		-- sleep 0.5 >/dev/null 2>&1 || \
+	perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+		-- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 
 	if ! perf script syscall-counts-by-pid -i "${temp_data}" > "${temp_out}"; then
tools/perf/tests/shell/test_syscall_counts_python.sh +3 -2
index 310e05d399f9..86463f14391b 100755
@@ -43,13 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 test_file_mode() {
 	echo "Testing syscall-counts.py..."
 	# Some systems might not have raw_syscalls:sys_enter (e.g. stripped kernels or permissions)
-	if ! perf list | grep -q raw_syscalls:sys_enter; then
+	if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
 		echo "Skipping test, raw_syscalls:sys_enter not found"
 		exit 2
 	fi
 
 	# Generate some syscall events
-	if ! perf record -e raw_syscalls:sys_enter -o "${temp_data}" -- sleep 0.5 2>/dev/null; then
+	if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+		-- sh -c "sleep 0.1; sleep 0.05" 2>/dev/null; then
 		echo "perf record failed (permissions?), skipping file mode test."
 		exit 2
 	fi
tools/perf/tests/shell/test_task_analyzer.sh +2 -1
index f99a637a6099..f53fa4a27c79 100755
@@ -60,7 +60,8 @@ skip_no_probe_record_support() {
 
 prepare_perf_data() {
 	# 1s should be sufficient to catch at least some switches
-	perf record -e sched:sched_switch -a -o "${perfdata}" -- sleep 1 > /dev/null 2>&1
+	perf record -B -N --no-bpf-event -e sched:sched_switch -a -o "${perfdata}" \
+		-- sleep 1 > /dev/null 2>&1
 	# check if perf data file got created in above step.
 	if [ ! -e "${perfdata}" ]; then
 		printf "FAIL: perf record failed to create \"${perfdata}\" \n"
tools/perf/tests/shell/test_wakeup_latency_python.sh +4 -3
index cb2571449168..b43a86996be2 100755
@@ -41,9 +41,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing wakeup-latency.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "sched:sched_wakeup"; then
+if perf list tracepoint | grep -q "sched:sched_wakeup"; then
 	ev="sched:sched_wakeup,sched:sched_wakeup_new,sched:sched_switch"
-	perf record -e "$ev" -a -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
 		-- sleep 0.1 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
@@ -70,7 +70,8 @@ else
 fi
 
 # Also test zero-wakeups / unhandled events path to verify division-by-zero protection
-if perf record -e cycles -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+if perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
+	-- perf test -w noploop >/dev/null 2>&1; then
 	if ! perf script wakeup-latency -i "${temp_data}" > "${temp_out}" || \
 	   ! grep -q "avg_wakeup_latency (ns): N/A" "${temp_out}"; then
 		echo "wakeup-latency zero-wakeups guard test failed"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#12 lore raw

Re: [PATCH v1 6/9] perf test: Reduce overhead and contention in Python shell tests

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 18 lines

When all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh, test_rw_by_file_python.sh, test_rw_by_pid_python.sh, test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and test_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in test_compaction_times_python.sh.

Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author reviewer
#13 lore raw

PATCH 7/9 perf python event_analyzing_sample: Default to in-memory SQLite database

In reply to Ian Rogers

When -d/--db is not specified, event_analyzing_sample.py created a temporary file in /tmp via tempfile.mkstemp() and deleted it in trace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.

Default to sqlite3.connect(":memory:") when db_path is not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode in test_event_analyzing_sample_python.sh.

Fixes eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/python/event_analyzing_sample.py   | 35 ++++++-------------
 .../test_event_analyzing_sample_python.sh     |  8 ++++-
 2 files changed, 17 insertions(+), 26 deletions(-)
tools/perf/python/event_analyzing_sample.py +10 -25
index 15b47cff9fa3..b4efee88d94b 100755
@@ -17,10 +17,8 @@ from __future__ import annotations
 
 import argparse
 import math
-import os
 import sqlite3
 import struct
-import tempfile
 from typing import Any
 import perf
 
@@ -117,16 +115,11 @@ session: Any = None
 
 class _DB:
     con: sqlite3.Connection | None = None
-    temp_path: str | None = None
 
 def trace_begin(db_path: str | None = None) -> None:
     """Initialize database tables."""
     print("In trace_begin:\n")
-    if not db_path:
-        fd, db_path = tempfile.mkstemp(prefix="perf_events_", suffix=".db")
-        os.close(fd)
-        _DB.temp_path = db_path
-    con = sqlite3.connect(db_path)
+    con = sqlite3.connect(db_path or ":memory:")
     try:
         # Drop any pre-existing tables so repeated runs do not accumulate duplicate events.
         con.execute("drop table if exists gen_events;")
@@ -297,28 +290,20 @@ def show_pebs_ll() -> None:
 def trace_end() -> None:
     """Called at the end of trace processing."""
     print("In trace_end:\n")
-    try:
-        if _DB.con:
-            try:
-                _DB.con.commit()
-                show_general_events()
-                show_pebs_ll()
-            finally:
-                _DB.con.close()
-                _DB.con = None
-    finally:
-        if _DB.temp_path and os.path.exists(_DB.temp_path):
-            try:
-                os.remove(_DB.temp_path)
-            except OSError:
-                pass
-            _DB.temp_path = None
+    if _DB.con:
+        try:
+            _DB.con.commit()
+            show_general_events()
+            show_pebs_ll()
+        finally:
+            _DB.con.close()
+            _DB.con = None
 
 if __name__ == "__main__":
     ap = argparse.ArgumentParser(description="Analyze events with SQLite")
     ap.add_argument("-i", "--input", default="perf.data", help="Input file name")
     ap.add_argument("-d", "--db", "--database", dest="database", default=None,
-                    help="Database file name (defaults to a temporary file cleaned up on exit)")
+                    help="Database file name (defaults to an in-memory database)")
     args = ap.parse_args()
 
     try:
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +7 -1
index 5f0a41080acf..cddb12b67698 100755
@@ -45,8 +45,14 @@ test_file_mode() {
 		exit 2
 	fi
 
-	# Run the script
+	# Run the script with default (:memory:) database and with explicit -d path
 	if ! perf script event_analyzing_sample -i "${temp_data}" \
+		> "${temp_dir}/perf.mem.out" 2>&1 || \
+	   ! grep -q "Statistics about the general events" "${temp_dir}/perf.mem.out" || \
+	   grep -q "Error creating/inserting event" "${temp_dir}/perf.mem.out"; then
+		echo "Default in-memory database mode test failed."
+		err=1
+	elif ! perf script event_analyzing_sample -i "${temp_data}" \
 		-d "${temp_db}" > "${temp_dir}/perf.out" 2>&1; then
 		echo "File mode test failed."
 		err=1
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#14 lore raw

Re: [PATCH v1 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines

When -d/--db is not specified, event_analyzing_sample.py created a temporary file in /tmp via tempfile.mkstemp() and deleted it in trace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.

Quoted 8 lines

Default to sqlite3.connect(":memory:") when db_path is not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode in test_event_analyzing_sample_python.sh.

Fixes eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
author reviewer
#15 lore raw

PATCH 8/9 perf python: Initialize debug output on module load

In reply to Ian Rogers

Unlike the perf binary's main(), PyInit_perf() does not call perf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hits debug_file() with _debug_file == NULL and emits 'debug_file not set' without a trailing newline before the actual diagnostic message.

Call perf_debug_setup() in PyInit_perf() and add the missing trailing newline to the fallback warning in debug_file().

Fixes ec49230cf6dd ("perf debug: Expose debug file")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/util/debug.c  | 2 +-
 tools/perf/util/python.c | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
tools/perf/util/debug.c +1 -1
index 6b5ffe81f141..b7095519419e 100644
@@ -55,7 +55,7 @@ FILE *debug_file(void)
 {
 	if (!_debug_file) {
 		debug_set_file(stderr);
-		pr_warning_once("debug_file not set");
+		pr_warning_once("%s not set\n", __func__);
 	}
 	return _debug_file;
 }
tools/perf/util/python.c +1 -0
index 690a94fb4f62..95140dfaef9c 100644
@@ -5157,6 +5157,7 @@ PyMODINIT_FUNC PyInit_perf(void)
 
 	/* The page_size is placed in util object. */
 	page_size = sysconf(_SC_PAGE_SIZE);
+	perf_debug_setup();
 
 	Py_INCREF(&pyrf_evlist__type);
 	PyModule_AddObject(module, "evlist", (PyObject *)&pyrf_evlist__type);
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#16 lore raw

Re: [PATCH v1 8/9] perf python: Initialize debug output on module load

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines

Unlike the perf binary's main(), PyInit_perf() does not call perf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hits debug_file() with _debug_file == NULL and emits 'debug_file not set' without a trailing newline before the actual diagnostic message.

Quoted 6 lines

Call perf_debug_setup() in PyInit_perf() and add the missing trailing newline to the fallback warning in debug_file().

Fixes ec49230cf6dd ("perf debug: Expose debug file")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
author reviewer
#17 lore raw

PATCH 9/9 perf pmu: Fix race with concurrent tracepoint creation and removal in perf list

In reply to Ian Rogers

perf_pmus__print_pmu_events() first counts events via perf_pmu__num_events() to allocate the aliases array, then populates it via perf_pmu__for_each_event(). When dynamic tracepoints (such as kprobes or uprobes during 'perf test' runs) are concurrently created or removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len, leaving trailing zeroed entries in aliases[] with name == NULL and pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning /sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns -ENOENT and aborts enumeration of all remaining tracepoint subsystems.
- If an event is added, perf_pmus__print_pmu_events__callback() aborts enumeration when state->index reaches state->aliases_len.

Grow state->aliases dynamically via realloc() when needed, use state.index as the actual populated length for sorting and printing, and ignore -ENOENT when a tracepoint subsystem directory disappears during enumeration.

Fixes c3245d2093c1 ("perf pmu: Abstract alias/event struct")
Fixes 45b6e281cb06 ("perf tp_pmu: Add event APIs")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/util/pmus.c   | 20 ++++++++++++++++----
 tools/perf/util/tp_pmu.c |  8 ++++++--
 2 files changed, 22 insertions(+), 6 deletions(-)
tools/perf/util/pmus.c +16 -4
index e0a4cb2428ca..137abad37c33 100644
@@ -571,7 +571,7 @@ static int cmp_sevent(const void *a, const void *b)
 	}
 
 	/* Order by event name. */
-	return strcmp(as->name, bs->name);
+	return strcmp(as->name ?: "", bs->name ?: "");
 }
 
 static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
@@ -581,7 +581,7 @@ static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
 		return false;
 
 	/* Don't remove duplicates for different PMUs */
-	return strcmp(a->pmu_name, b->pmu_name) == 0;
+	return strcmp(a->pmu_name ?: "", b->pmu_name ?: "") == 0;
 }
 
 struct events_callback_state {
@@ -597,8 +597,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,
 	struct sevent *s;
 
 	if (state->index >= state->aliases_len) {
-		pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
-		return 1;
+		size_t new_len = max_t(size_t, 16, state->aliases_len * 2);
+		struct sevent *new_aliases;
+
+		new_aliases = realloc(state->aliases, new_len * sizeof(struct sevent));
+		if (!new_aliases) {
+			pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
+			return 1;
+		}
+		memset(&new_aliases[state->aliases_len], 0,
+		       (new_len - state->aliases_len) * sizeof(struct sevent));
+		state->aliases = new_aliases;
+		state->aliases_len = new_len;
 	}
 	assert(info->pmu != NULL || info->name != NULL);
 	s = &state->aliases[state->index];
@@ -654,6 +664,8 @@ void perf_pmus__print_pmu_events(const struct print_callbacks *print_cb, void *p
 		perf_pmu__for_each_event(pmu, skip_duplicate_pmus, &state,
 					 perf_pmus__print_pmu_events__callback);
 	}
+	aliases = state.aliases;
+	len = state.index;
 	qsort(aliases, len, sizeof(struct sevent), cmp_sevent);
 	for (int j = 0; j < len; j++) {
 		/* Skip duplicates */
tools/perf/util/tp_pmu.c +6 -2
index c2be8c9f9084..5ac732e06841 100644
@@ -151,7 +151,9 @@ static int for_each_event_cb(void *state, const char *sys_name, const char *evt_
 
 static int for_each_event_sys_cb(void *state, const char *sys_name)
 {
-	return tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+	int ret = tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+
+	return ret == -ENOENT ? 0 : ret;
 }
 
 int tp_pmu__for_each_event(struct perf_pmu *pmu, void *state, pmu_event_callback cb)
@@ -176,7 +178,9 @@ static int num_events_cb(void *state, const char *sys_name __maybe_unused,
 
 static int num_events_sys_cb(void *state, const char *sys_name)
 {
-	return tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+	int ret = tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+
+	return ret == -ENOENT ? 0 : ret;
 }
 
 size_t tp_pmu__num_events(struct perf_pmu *pmu __maybe_unused)
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#18 lore raw

Re: [PATCH v1 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] perf: missing explicit <stdlib.h> inclusion for realloc() --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] perf: missing explicit <stdlib.h> inclusion for realloc()
--

commit 156bff4d99dfcb5bfd8c7cc38a4671357c5113e3
Author: Ian Rogers <[email protected]>

perf pmu: Fix race with concurrent tracepoint creation and removal in perf list

This commit modifies the perf tool to handle concurrent addition and removal of tracepoint events by dynamically reallocating the aliases array and handling -ENOENT gracefully.

Quoted 4 lines
tools/perf/util/pmus.c +0 -0
index e0a4cb2428ca4..137abad37c33b 100644

[ ... ]

tools/perf/util/pmus.c static int perf_pmus__print_pmu_events__callback(void *vstate, · line 603

Quoted 10 lines
tools/perf/util/pmus.c +4 -2
@@ -597,8 +597,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,
 	struct sevent *s;
 
 	if (state->index >= state->aliases_len) {
-		pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
-		return 1;
+		size_t new_len = max_t(size_t, 16, state->aliases_len * 2);
+		struct sevent *new_aliases;
+
+		new_aliases = realloc(state->aliases, new_len * sizeof(struct sevent));

[Severity: Low]
Does this code in perf_pmus__print_pmu_events__callback() rely on implicit header inclusions for realloc()?

Since the perf tool can be built against musl libc, which strictly separates declarations and does not implicitly include <stdlib.h> through other headers, should <stdlib.h> be explicitly included at the top of tools/perf/util/pmus.c to prevent compilation failures?

Quoted 4 lines
tools/perf/util/pmus.c +4 -0
+		if (!new_aliases) {
+			pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
+			return 1;
+		}

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
author reviewer
#19 lore raw

v2 cover perf python: Follow-up fixes and shell test deflaking

In reply to Ian Rogers

This series contains follow-up fixes for the standalone Python scripts and their shell tests, particularly when running 'perf test' as root under heavy parallel load:

- Clamp sample_id size calculation for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE records to work around kernels without "perf/core: Restore header fields in sideband output callbacks" [1].
- Fix offline interval printing in sctop.py and live signal races in stat-cpi.py.
- Deflake Intel PT, failed-syscalls, and other Python shell tests under parallel 'perf test' load by scoping tracepoint checks to 'perf list tracepoint', passing '-B -N --no-bpf-event' to 'perf record', and avoiding unnecessary system-wide ('-a') recordings.
- Default event_analyzing_sample.py to an in-memory SQLite database (':memory:') when '--db' is not specified.
- Initialize perf debug output ('perf_debug_setup()') when loading the 'perf' Python extension module.
- Fix races in 'perf list' ('perf_pmus__print_pmu_events()' and 'tp_pmu.c') when concurrent tests dynamically create and remove kprobe/uprobes tracepoints.

v2:
- Fix 8-byte alignment calculation for PERF_RECORD_TEXT_POKE payload in evsel__event_size() by aligning fixed + old_len + new_len.
- Keep offline interval clock advancing on all samples in sctop.py while flushing any remaining syscalls at EOF.
- Use 'mktemp -d' private directories in test_sctop_python.sh and test_failed_syscalls_python.sh so temporary files are not unlinked and recreated directly in /tmp.
- Explicitly include <stdlib.h> in tools/perf/util/pmus.c.

[1] https://lore.kernel.org/r/[email protected]

Ian Rogers (9):
perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke
events
perf python sctop: Fix offline interval printing and test flakiness perf python stat-cpi: Fix live mode signal races and test flakiness perf test: Deflake Intel PT Python shell tests under load perf test: Deflake failed-syscalls Python shell tests under load perf test: Reduce overhead and contention in Python shell tests perf python event_analyzing_sample: Default to in-memory SQLite
database
perf python: Initialize debug output on module load
perf pmu: Fix race with concurrent tracepoint creation and removal in
perf list

 tools/perf/python/event_analyzing_sample.py   | 35 +++-------
 tools/perf/python/sctop.py                    |  7 +-
 tools/perf/python/stat-cpi.py                 | 19 +++---
 .../shell/test_check_perf_trace_python.sh     |  8 ++-
 .../shell/test_compaction_times_python.sh     |  8 ++-
 .../test_event_analyzing_sample_python.sh     | 11 ++-
 .../shell/test_export_to_postgresql_python.sh | 49 +++++++------
 .../shell/test_export_to_sqlite_python.sh     | 50 ++++++++------
 .../test_failed_syscalls_by_pid_python.sh     | 68 +++++++++++--------
 .../shell/test_failed_syscalls_python.sh      | 54 +++++++++------
 .../tests/shell/test_flamegraph_python.sh     |  5 +-
 .../shell/test_futex_contention_python.sh     |  7 +-
 tools/perf/tests/shell/test_gecko_python.sh   |  3 +-
 .../shell/test_intel_pt_events_python.sh      | 43 ++++++------
 .../tests/shell/test_mem_phys_addr_python.sh  |  8 ++-
 .../shell/test_net_dropmonitor_python.sh      |  9 +--
 .../tests/shell/test_netdev_times_python.sh   |  6 +-
 .../tests/shell/test_powerpc_hcalls_python.sh |  4 +-
 .../tests/shell/test_rw_by_file_python.sh     |  5 +-
 .../perf/tests/shell/test_rw_by_pid_python.sh |  4 +-
 tools/perf/tests/shell/test_rwtop_python.sh   |  4 +-
 .../shell/test_sched_migration_python.sh      |  4 +-
 tools/perf/tests/shell/test_sctop_python.sh   | 58 ++++++++--------
 .../tests/shell/test_stackcollapse_python.sh  |  2 +-
 .../perf/tests/shell/test_stat_cpi_python.sh  | 12 +++-
 .../test_syscall_counts_by_pid_python.sh      |  6 +-
 .../tests/shell/test_syscall_counts_python.sh |  5 +-
 tools/perf/tests/shell/test_task_analyzer.sh  |  3 +-
 .../tests/shell/test_wakeup_latency_python.sh |  7 +-
 tools/perf/util/debug.c                       |  2 +-
 tools/perf/util/evlist.c                      |  6 +-
 tools/perf/util/evsel.c                       | 59 +++++++++++++++-
 tools/perf/util/evsel.h                       |  1 +
 tools/perf/util/pmus.c                        | 21 ++++--
 tools/perf/util/python.c                      |  1 +
 tools/perf/util/tp_pmu.c                      |  8 ++-
 36 files changed, 366 insertions(+), 236 deletions(-)
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
author reviewer
#20 lore raw

PATCH v2 1/9 perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events

On kernels without the fix ("perf/core: Restore header fields in sideband output callbacks") to restore event_id.header.size in perf_event_ksymbol_output(), perf_event_bpf_output(), and perf_event_text_poke_output(), concurrent perf sessions cause those sideband records to be emitted with header.size inflated by multiple id_header_size increments while the single id_sample is written immediately after the event payload. Indexing backwards from event->header.size reads uninitialized ring-buffer bytes at the end of the record, causing evlist__event2evsel() to fail with -EFAULT.

Add evsel__event_size() to clamp the effective size used to locate the trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving event->header.size intact for advancing the ring-buffer/file stream.

Fixes 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")
Fixes 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")
Fixes 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/util/evlist.c |  6 ++--
 tools/perf/util/evsel.c  | 59 +++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/evsel.h  |  1 +
 3 files changed, 63 insertions(+), 3 deletions(-)
tools/perf/util/evlist.c +4 -2
index 9392d912d254..c2402e4791b6 100644
@@ -973,13 +973,15 @@ static int evlist__event2id(struct evlist *evlist, union perf_event *event, u64
 	const __u64 *array = event->sample.array;
 	ssize_t n;
 
-	n = (event->header.size - sizeof(event->header)) >> 3;
-
 	if (event->header.type == PERF_RECORD_SAMPLE) {
+		n = (event->header.size - sizeof(event->header)) >> 3;
 		if (evlist__id_pos(evlist) >= n)
 			return -1;
 		*id = array[evlist__id_pos(evlist)];
 	} else {
+		u16 size = evsel__event_size(evlist__first(evlist), event);
+
+		n = (size - sizeof(event->header)) >> 3;
 		if (evlist__is_pos(evlist) > n)
 			return -1;
 		n -= evlist__is_pos(evlist);
tools/perf/util/evsel.c +58 -1

71 lines not on this page · raw message

tools/perf/util/evsel.h +1 -0
index 5c5799cee601..174f3414fd3c 100644
@@ -469,6 +469,7 @@ int evsel__parse_sample_timestamp(struct evsel *evsel, union perf_event *event,
 				  u64 *timestamp);
 
 u16 evsel__id_hdr_size(const struct evsel *evsel);
+u16 evsel__event_size(const struct evsel *evsel, const union perf_event *event);
 
 static inline struct evsel *evsel__next(struct evsel *evsel)
 {
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#21 lore raw

Re: [PATCH v2 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines

On kernels without the fix ("perf/core: Restore header fields in sideband output callbacks") to restore event_id.header.size in perf_event_ksymbol_output(), perf_event_bpf_output(), and perf_event_text_poke_output(), concurrent perf sessions cause those sideband records to be emitted with header.size inflated by multiple id_header_size increments while the single id_sample is written immediately after the event payload. Indexing backwards from event->header.size reads uninitialized ring-buffer bytes at the end of the record, causing evlist__event2evsel() to fail with -EFAULT.

Add evsel__event_size() to clamp the effective size used to locate the trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving event->header.size intact for advancing the ring-buffer/file stream.

Quoted 6 lines
Fixes 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")
Fixes 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")
Fixes 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author reviewer
#22 lore raw

PATCH v2 2/9 perf python sctop: Fix offline interval printing and test flakiness

In reply to Ian Rogers

In sctop.py:
- If an earlier interval elapsed and printed an empty table before the target comm ('sleep') executed any syscalls, analyzer.printed became True and the final partial interval containing the target comm's syscalls was never flushed at EOF. Flush print_current_totals() in finally when analyzer.syscalls is non-empty as well as when nothing has been printed yet.
- Initialize analyzer.e_machine after creating perf.session rather than when session is still None.

In test_sctop_python.sh:
- Use a private temporary directory via 'mktemp -d'.
- Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a bounded retry loop so 'sleep's PERF_RECORD_COMM and sys_enter events are reliably captured under heavy load.

Fixes b83f0bacf5e4 ("perf python: Port sctop to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/python/sctop.py                  |  7 ++-
 tools/perf/tests/shell/test_sctop_python.sh | 58 ++++++++++-----------
 2 files changed, 34 insertions(+), 31 deletions(-)
tools/perf/python/sctop.py +5 -2
index 42e95ecfffad..48320af8b755 100755
@@ -37,6 +37,7 @@ class SCTopAnalyzer:
         self.offline = offline
         self.own_pid = os.getpid()
         self.last_print_time: Optional[int] = None
+        self.printed = False
         self.session: Optional[perf.session] = None
         self.e_machine: Optional[int] = None
 
@@ -137,6 +138,7 @@ class SCTopAnalyzer:
 
     def print_current_totals(self):
         """Print current syscall totals."""
+        self.printed = True
         # Clear terminal
         if not self.offline:
             print("\x1b[2J\x1b[H", end="")
@@ -217,8 +219,8 @@ def main():
         if args.input:
             session = perf.session(perf.data(args.input), sample=analyzer.process_event)
             analyzer.session = session
-            session.process_events()
             analyzer.e_machine = getattr(session, "e_machine", None)
+            session.process_events()
         else:
             try:
                 live_session = LiveSession(
@@ -237,7 +239,8 @@ def main():
         sys.exit(1)
     finally:
         if args.input:
-            analyzer.print_current_totals()
+            if not analyzer.printed or analyzer.syscalls:
+                analyzer.print_current_totals()
             # Break the reference cycle between perf.session and analyzer.process_event
             # because perf.session lacks cyclic GC support (tp_traverse).
             analyzer.session = None
tools/perf/tests/shell/test_sctop_python.sh +29 -29

80 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#23 lore raw

Re: [PATCH v2 2/9] perf python sctop: Fix offline interval printing and test flakiness

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines

In sctop.py:
- If an earlier interval elapsed and printed an empty table before the target comm ('sleep') executed any syscalls, analyzer.printed became True and the final partial interval containing the target comm's syscalls was never flushed at EOF. Flush print_current_totals() in finally when analyzer.syscalls is non-empty as well as when nothing has been printed yet.
- Initialize analyzer.e_machine after creating perf.session rather than when session is still None.

In test_sctop_python.sh:
- Use a private temporary directory via 'mktemp -d'.
- Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a bounded retry loop so 'sleep's PERF_RECORD_COMM and sys_enter events are reliably captured under heavy load.

Quoted 3 lines
Fixes b83f0bacf5e4 ("perf python: Port sctop to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author reviewer
#24 lore raw

PATCH v2 3/9 perf python stat-cpi: Fix live mode signal races and test flakiness

In reply to Ian Rogers

In test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a 1-second duration and can exit under heavy parallel load before 'perf stat -p' and 'perf script stat-cpi' finish starting up. In addition, the fixed 'sleep 0.5' before sending SIGINT can fire before Python finishes importing the perf module, opening the live evlist, and flushing the first interval.

In stat-cpi.py, register SIGINT and SIGTERM handlers before calling _open_live_evlist() and pass flush=True when printing live output so redirected stdout is flushed immediately after each interval.

In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the target workload stays alive until killed, and poll the output file for 'cpi' (up to 5 seconds) before sending SIGINT.

Fixes 4425182d426b ("perf python: Port stat-cpi to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/python/stat-cpi.py                 | 19 ++++++++++---------
 .../perf/tests/shell/test_stat_cpi_python.sh  | 12 +++++++++---
 2 files changed, 19 insertions(+), 12 deletions(-)
tools/perf/python/stat-cpi.py +10 -9
index 0b7d76876a6c..da92cf560067 100755
@@ -106,7 +106,8 @@ class StatCpiAnalyzer:
             if ins != 0:
                 cpi = cyc / float(ins)
             t_sec = timestamp / 1000000000.0
-            print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})")
+            print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})",
+                  flush=True)
 
     def read_counters(self, evlist: Any) -> None:
         """Read counters live."""
@@ -173,19 +174,19 @@ class StatCpiAnalyzer:
 
     def run_live(self) -> None:
         """Read counters live."""
-        try:
-            evlist = self._open_live_evlist()
-        except OSError as e:
-            print(f"Failed to open events: {e}", file=sys.stderr)
-            sys.exit(1)
-
         def handle_signal(_signum: int, _frame: Any) -> None:
             raise KeyboardInterrupt
 
         signal.signal(signal.SIGINT, signal.default_int_handler)
         signal.signal(signal.SIGTERM, handle_signal)
 
-        print("Live mode started. Press Ctrl+C to stop.")
+        try:
+            evlist = self._open_live_evlist()
+        except OSError as e:
+            print(f"Failed to open events: {e}", file=sys.stderr)
+            sys.exit(1)
+
+        print("Live mode started. Press Ctrl+C to stop.", flush=True)
         try:
             while True:
                 time.sleep(self.args.interval)
@@ -195,7 +196,7 @@ class StatCpiAnalyzer:
                 self.data.clear()
                 self.recorded_pairs.clear()
         except KeyboardInterrupt:
-            print("\nStopped.")
+            print("\nStopped.", flush=True)
         finally:
             evlist.close()
 
tools/perf/tests/shell/test_stat_cpi_python.sh +9 -3
index fe7562307634..6cb376c92e2f 100755
@@ -50,7 +50,7 @@ test_live_mode() {
 		echo "perf stat failed (permissions?), skipping live mode test."
 		return 0
 	fi
-	perf test -w noploop &
+	perf test -w noploop 60 &
 	workload_pid=$!
 	if ! perf stat -e cycles,instructions -p "$workload_pid" -- sleep 0.05 2>/dev/null && \
 	   ! perf stat -e cycles:u,instructions:u -p "$workload_pid" -- sleep 0.05 2>/dev/null; then
@@ -61,10 +61,16 @@ test_live_mode() {
 	fi
 	ran=1
 
-	# Run live mode for 1 interval in the background, give it a tiny sleep, then interrupt
+	# Run live mode in the background, wait until at least one interval is
+	# printed, then interrupt.
 	perf script stat-cpi -I 0.1 -p "$workload_pid" > "${temp_out}" &
 	pid=$!
-	sleep 0.5
+	for _ in $(seq 1 50); do
+		if grep -q "cpi" "${temp_out}"; then
+			break
+		fi
+		sleep 0.1
+	done
 	kill -INT "$pid" 2>/dev/null || true
 	set +e
 	wait "$pid"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#25 lore raw

Re: [PATCH v2 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf python: unhandled KeyboardInterrupt in stat-cpi init --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf python: unhandled KeyboardInterrupt in stat-cpi init
--

commit 1a071f63d2155dda63d4eb450e190a0e0b0b2876
Author: Ian Rogers <[email protected]>

perf python stat-cpi: Fix live mode signal races and test flakiness

This commit attempts to fix live mode signal races by registering signal handlers earlier and flushing output to prevent test flakiness. It also modifies the shell test to run the workload longer and poll for output before sending SIGINT.

Quoted 4 lines
tools/perf/python/stat-cpi.py +0 -0
index 0b7d76876a6c6..da92cf560067a 100755

[ ... ]

Quoted 17 lines
tools/perf/python/stat-cpi.py +0 -7
@@ -173,19 +174,19 @@ class StatCpiAnalyzer:
 
     def run_live(self) -> None:
         """Read counters live."""
-        try:
-            evlist = self._open_live_evlist()
-        except OSError as e:
-            print(f"Failed to open events: {e}", file=sys.stderr)
-            sys.exit(1)
-
         def handle_signal(_signum: int, _frame: Any) -> None:
             raise KeyboardInterrupt
 
         signal.signal(signal.SIGINT, signal.default_int_handler)
         signal.signal(signal.SIGTERM, handle_signal)
 
-        print("Live mode started. Press Ctrl+C to stop.")

tools/perf/python/stat-cpi.py class StatCpiAnalyzer: · line 192

Quoted 10 lines
tools/perf/python/stat-cpi.py +7 -0
+        try:
+            evlist = self._open_live_evlist()
+        except OSError as e:
+            print(f"Failed to open events: {e}", file=sys.stderr)
+            sys.exit(1)
+
+        print("Live mode started. Press Ctrl+C to stop.", flush=True)
         try:
             while True:
                 time.sleep(self.args.interval)

[Severity: Medium]
Will this crash if a signal is received during _open_live_evlist()?

The signal handlers are registered to raise KeyboardInterrupt before _open_live_evlist() is called, but the try...except KeyboardInterrupt block below only wraps the while loop.

If SIGINT or SIGTERM is received during this initialization phase, wouldn't it result in an unhandled KeyboardInterrupt exception, printing a traceback and bypassing graceful cleanup?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author reviewer
#26 lore raw

PATCH v2 4/9 perf test: Deflake Intel PT Python shell tests under load

In reply to Ian Rogers

In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).

Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh to avoid build-id cache and BPF synthesis overhead.

Fixes d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
Fixes 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
Fixes b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 .../shell/test_export_to_postgresql_python.sh | 49 ++++++++++--------
 .../shell/test_export_to_sqlite_python.sh     | 50 +++++++++++--------
 .../shell/test_intel_pt_events_python.sh      | 43 +++++++++-------
 3 files changed, 79 insertions(+), 63 deletions(-)
tools/perf/tests/shell/test_export_to_postgresql_python.sh +27 -22
index 350813466700..835f48f002e9 100755
@@ -61,9 +61,10 @@ test_file_mode() {
 	fi
 
 	# Generate events with callchains and context switches
-	if ! perf record -g --switch-events -o "${temp_data}" \
+	if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \
 	     -- perf test -w noploop >/dev/null 2>&1 && \
-	   ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	   ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+	     -- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
@@ -92,29 +93,33 @@ test_file_mode() {
 test_intel_pt() {
 	echo "Testing export-to-postgresql.py with intel_pt..."
 
-	psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
-	rm -f "${temp_data}"
-	# Generate some intel_pt events; use a subshell that waits for uname
-	if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
-		-- sh -c "uname; true" >/dev/null 2>&1; then
-		echo "Skipping intel_pt test, intel_pt not available."
-		return 0
-	fi
+	# Generate some intel_pt events; sleep briefly after uname in the subshell
+	# so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+	passed=0
+	for _ in 1 2 3 4 5; do
+		psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
+		rm -f "${temp_data}"
+		if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+			-- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+			echo "Skipping intel_pt test, intel_pt not available."
+			return 0
+		fi
 
-	# Run the script with --itrace cr to synthesize call_returns
-	if ! perf script export-to-postgresql -i "${temp_data}" \
-		-o "${temp_db}" --itrace cr >/dev/null; then
-		echo "intel_pt file mode test failed."
-		err=1
-	else
-		# Check DB for calls
-		if ! psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
+		# Run the script with --itrace cr to synthesize call_returns
+		if perf script export-to-postgresql -i "${temp_data}" \
+			-o "${temp_db}" --itrace cr >/dev/null && \
+		   psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
 			grep -q '[1-9]'; then
-			echo "PostgreSQL intel_pt validation failed (no calls found)."
-			err=1
-		else
-			echo "intel_pt test passed (cr validated)."
+			passed=1
+			break
 		fi
+	done
+
+	if [ "$passed" -eq 0 ]; then
+		echo "PostgreSQL intel_pt validation failed (no calls found)."
+		err=1
+	else
+		echo "intel_pt test passed (cr validated)."
 	fi
 }
 
tools/perf/tests/shell/test_export_to_sqlite_python.sh +28 -22

66 lines not on this page · raw message

tools/perf/tests/shell/test_intel_pt_events_python.sh +24 -19

58 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#27 lore raw

Re: [PATCH v2 4/9] perf test: Deflake Intel PT Python shell tests under load

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines

In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).

Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh to avoid build-id cache and BPF synthesis overhead.

Quoted 5 lines
Fixes d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
Fixes 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
Fixes b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author reviewer
#28 lore raw

PATCH v2 5/9 perf test: Deflake failed-syscalls Python shell tests under load

In reply to Ian Rogers

When running 'perf test' in parallel under heavy system load, a one-shot 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event' can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and ENOENT sys_exit events are captured, while also contending on ~/.debug build-id caching and BPF event synthesis.

Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell after 'ls' so its events are flushed before the parent subshell exits, and wrap the record and check steps in a bounded retry loop (up to 5 attempts).

Fixes b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")
Fixes 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 .../test_failed_syscalls_by_pid_python.sh     | 68 +++++++++++--------
 .../shell/test_failed_syscalls_python.sh      | 54 +++++++++------
 2 files changed, 71 insertions(+), 51 deletions(-)
tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh +39 -29
index 215372d4e1c5..070a7ef6b8ad 100755
@@ -41,8 +41,10 @@ test_file_mode() {
 	echo "Testing failed-syscalls-by-pid.py..."
 
 	# Check if syscalls:sys_exit is supported/readable
-	if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
-		if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \
+		-o /dev/null -- true >/dev/null 2>&1; then
+		if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \
+			-o /dev/null -- true >/dev/null 2>&1; then
 			echo "Skipping test, no syscalls:sys_exit or raw_syscalls:sys_exit event"
 			exit 2
 		else
@@ -52,41 +54,49 @@ test_file_mode() {
 		EVENT="syscalls:sys_exit"
 	fi
 
-	# Generate some events by running a command that should fail at least some syscall
-	# (e.g. failing stat on non-existent file).
-	# Using '|| true' because 'perf record' returns the exit code of 'ls',
-	# which fails with ENOENT
-	perf record -e "${EVENT}" -o "${temp_data}" -- ls /does_not_exist >/dev/null 2>&1 || true
+	# Generate some events by running a command that fails a syscall
+	# (e.g. failing stat on non-existent file), sleeping briefly in the
+	# subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed.
+	passed=0
+	for _ in 1 2 3 4 5; do
+		rm -f "${temp_data}" "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
+		perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \
+			-- sh -c "ls /does_not_exist 2>/dev/null; sleep 0.05 || true" \
+			>/dev/null 2>&1 || true
+		if [ ! -s "${temp_data}" ]; then
+			continue
+		fi
+
+		if perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}" && \
+		   grep -n -q "err = ENOENT" "${temp_out}" && \
+		   perf script failed-syscalls-by-pid -i "${temp_data}" \
+			"ls" > "${temp_out}.comm" && \
+		   grep -q "err = ENOENT" "${temp_out}.comm"; then
+			ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' \
+				"${temp_out}.comm" | head -n 1)
+			if [ -n "${ls_pid}" ] && \
+			   perf script failed-syscalls-by-pid -i "${temp_data}" \
+				"${ls_pid}" > "${temp_out}.pid" && \
+			   grep -q "err = ENOENT" "${temp_out}.pid"; then
+				passed=1
+				break
+			fi
+		fi
+	done
+
 	if [ ! -s "${temp_data}" ]; then
 		echo "Skipping test, perf record failed to create data"
 		exit 2
 	fi
 
-	# Run the script and check output
-	if ! perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}"; then
+	if [ "$passed" -eq 0 ]; then
 		echo "failed-syscalls-by-pid test failed."
-		err=1
-	elif ! grep -n -q "err = ENOENT" "${temp_out}"; then
-		echo "Failed to find expected failed syscalls"
-		cat "${temp_out}"
-		err=1
-	elif ! perf script failed-syscalls-by-pid -i "${temp_data}" "ls" > "${temp_out}.comm" || \
-	     ! grep -q "err = ENOENT" "${temp_out}.comm"; then
-		echo "failed-syscalls-by-pid comm filter test failed."
-		cat "${temp_out}.comm"
+		cat "${temp_out}" 2>/dev/null || true
+		cat "${temp_out}.comm" 2>/dev/null || true
+		cat "${temp_out}.pid" 2>/dev/null || true
 		err=1
 	else
-		ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' "${temp_out}.comm" | head -n 1)
-		if [ -z "${ls_pid}" ] || \
-		   ! perf script failed-syscalls-by-pid -i "${temp_data}" \
-			"${ls_pid}" > "${temp_out}.pid" || \
-		   ! grep -q "err = ENOENT" "${temp_out}.pid"; then
-			echo "failed-syscalls-by-pid PID filter test failed."
-			cat "${temp_out}.pid" 2>/dev/null || true
-			err=1
-		else
-			echo "failed-syscalls-by-pid test passed."
-		fi
+		echo "failed-syscalls-by-pid test passed."
 	fi
 	rm -f "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
 }
tools/perf/tests/shell/test_failed_syscalls_python.sh +32 -22

82 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#29 lore raw

Re: [PATCH v2 5/9] perf test: Deflake failed-syscalls Python shell tests under load

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines

When running 'perf test' in parallel under heavy system load, a one-shot 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event' can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and ENOENT sys_exit events are captured, while also contending on ~/.debug build-id caching and BPF event synthesis.

Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell after 'ls' so its events are flushed before the parent subshell exits, and wrap the record and check steps in a bounded retry loop (up to 5 attempts).

Quoted 4 lines
Fixes b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")
Fixes 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author reviewer
#30 lore raw

PATCH v2 6/9 perf test: Reduce overhead and contention in Python shell tests

In reply to Ian Rogers

When all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh, test_rw_by_file_python.sh, test_rw_by_pid_python.sh, test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and test_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in test_compaction_times_python.sh.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/tests/shell/test_check_perf_trace_python.sh   | 8 +++++---
 tools/perf/tests/shell/test_compaction_times_python.sh   | 8 +++++---
 .../tests/shell/test_event_analyzing_sample_python.sh    | 3 ++-
 tools/perf/tests/shell/test_flamegraph_python.sh         | 5 +++--
 tools/perf/tests/shell/test_futex_contention_python.sh   | 7 ++++---
 tools/perf/tests/shell/test_gecko_python.sh              | 3 ++-
 tools/perf/tests/shell/test_mem_phys_addr_python.sh      | 8 +++++---
 tools/perf/tests/shell/test_net_dropmonitor_python.sh    | 9 +++++----
 tools/perf/tests/shell/test_netdev_times_python.sh       | 6 +++---
 tools/perf/tests/shell/test_powerpc_hcalls_python.sh     | 4 ++--
 tools/perf/tests/shell/test_rw_by_file_python.sh         | 5 +++--
 tools/perf/tests/shell/test_rw_by_pid_python.sh          | 4 ++--
 tools/perf/tests/shell/test_rwtop_python.sh              | 4 ++--
 tools/perf/tests/shell/test_sched_migration_python.sh    | 4 ++--
 tools/perf/tests/shell/test_stackcollapse_python.sh      | 2 +-
 .../tests/shell/test_syscall_counts_by_pid_python.sh     | 6 +++---
 tools/perf/tests/shell/test_syscall_counts_python.sh     | 5 +++--
 tools/perf/tests/shell/test_task_analyzer.sh             | 3 ++-
 tools/perf/tests/shell/test_wakeup_latency_python.sh     | 7 ++++---
 19 files changed, 58 insertions(+), 43 deletions(-)
tools/perf/tests/shell/test_check_perf_trace_python.sh +5 -3
index 92cc7b1f0033..baf55945d429 100755
@@ -45,10 +45,11 @@ test_file_mode() {
 	echo "Testing check-perf-trace.py..."
 
 	events=""
-	if perf list | grep -q "irq:softirq_entry"; then
+	tp_list=$(perf list tracepoint)
+	if echo "$tp_list" | grep -q "irq:softirq_entry"; then
 		events="irq:softirq_entry"
 	fi
-	if perf list | grep -q "kmem:kmalloc"; then
+	if echo "$tp_list" | grep -q "kmem:kmalloc"; then
 		if [ -n "$events" ]; then
 			events="$events,kmem:kmalloc,kmem:kfree"
 		else
@@ -62,7 +63,8 @@ test_file_mode() {
 	fi
 
 	# Generate events
-	if ! perf record -e "$events" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -e "$events" -o "${temp_data}" \
+		-- sh -c "sleep 0.1" >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
tools/perf/tests/shell/test_compaction_times_python.sh +5 -3
index 50397afaafed..3f5a71e52388 100755
@@ -44,15 +44,17 @@ test_file_mode() {
 	echo "Testing compaction-times.py..."
 
 	# Check for any compaction events to see if kernel supports it
-	if ! perf list | grep -q "compaction:mm_compaction_begin"; then
+	if ! perf list tracepoint | grep -q "compaction:mm_compaction_begin"; then
 		echo "Skipping test, compaction tracepoints not found"
 		exit 2
 	fi
 
 	# Generate some events
-	# We might not naturally trigger compaction in 0.5s sleep, but the script
+	# We might not naturally trigger compaction in 0.1s sleep, but the script
 	# should parse the empty or sparse file correctly without crashing.
-	if ! perf record -e "compaction:*" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event \
+		-e "compaction:mm_compaction_begin,compaction:mm_compaction_end" \
+		-a -o "${temp_data}" -- sleep 0.1 >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +2 -1
index a7a8ded003db..5f0a41080acf 100755
@@ -39,7 +39,8 @@ test_file_mode() {
 	echo "Testing event_analyzing_sample.py..."
 
 	# Generate some events
-	if ! perf record -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+		-- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
tools/perf/tests/shell/test_flamegraph_python.sh +3 -2
index 86313c72f53e..787d6bbdf04a 100755
@@ -60,7 +60,8 @@ test_file_mode() {
 	echo "Testing flamegraph.py..."
 
 	# Generate some events with callchains
-	if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+		-- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record -g failed (permissions or lack of support)"
 		exit 2
 	fi
@@ -75,7 +76,7 @@ test_file_mode() {
 
 	# Run the script in pipe ('-') mode and validate JSON output
 	rm -f "${temp_json}"
-	if ! perf record -g -o - -- perf test -w noploop 2>/dev/null | \
+	if ! perf record -B -N --no-bpf-event -g -o - -- perf test -w noploop 2>/dev/null | \
 		perf script flamegraph -i - -f json -o "${temp_json}" >/dev/null; then
 		echo "Pipe stdin JSON mode test failed."
 		err=1
tools/perf/tests/shell/test_futex_contention_python.sh +4 -3
index a8846ed68ac2..41ce8470a75f 100755
@@ -89,14 +89,15 @@ EOF
 test_file_mode() {
 	echo "Testing futex-contention.py..."
 	# Some systems might not have syscalls:sys_enter_futex
-	if ! perf list | grep -q syscalls:sys_enter_futex; then
+	if ! perf list tracepoint | grep -q syscalls:sys_enter_futex; then
 		echo "Skipping file mode test, syscalls:sys_enter_futex not found"
 		return
 	fi
 
 	# Generate some futex events
-	if ! perf record -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
-		-- sleep 0.5 2>/dev/null; then
+	if ! perf record -B -N --no-bpf-event \
+		-e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
+		-- sleep 0.1 2>/dev/null; then
 		echo "Skipping file mode test (record failed)"
 		return
 	fi
tools/perf/tests/shell/test_gecko_python.sh +2 -1
index de1d7fcf88b7..18e9f5cd2ab0 100755
@@ -39,7 +39,8 @@ test_file_mode() {
 	echo "Testing gecko.py..."
 
 	# Generate some events with callchains
-	if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+		-- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record -g failed (permissions or lack of support)"
 		exit 2
 	fi
tools/perf/tests/shell/test_mem_phys_addr_python.sh +5 -3
index 451692d5e089..cbb9f572be9d 100755
@@ -78,10 +78,12 @@ test_file_mode() {
 	echo "Testing mem-phys-addr.py file mode..."
 
 	# Generate memory access events (try unprivileged user-space first, then system-wide)
-	if ! perf record --phys-data -d -o "${temp_data}" \
+	if ! perf record -B -N --no-bpf-event --phys-data -d -o "${temp_data}" \
 	     -- perf test -w datasym >/dev/null 2>&1 && \
-	   ! perf record -d -o "${temp_data}" -- perf test -w datasym >/dev/null 2>&1 && \
-	   ! perf record -d -a -o "${temp_data}" -- sleep 0.2 >/dev/null 2>&1; then
+	   ! perf record -B -N --no-bpf-event -d -o "${temp_data}" \
+	     -- perf test -w datasym >/dev/null 2>&1 && \
+	   ! perf record -B -N --no-bpf-event -d -a -o "${temp_data}" \
+	     -- sleep 0.1 >/dev/null 2>&1; then
 		echo "Skipping file mode record test, perf record -d not supported"
 		return 0
 	fi
tools/perf/tests/shell/test_net_dropmonitor_python.sh +5 -4
index 05056a897bc9..2e37994418fc 100755
@@ -42,11 +42,12 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing net_dropmonitor.py..."
 
 # Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -o "${temp_data}" -a \
-	-- ping -c 1 255.255.255.255 >/dev/null 2>&1; then
-	if ! perf record -e skb:kfree_skb -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" -a \
+	-- ping -c 1 -W 1 255.255.255.255 >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" \
 		-- sleep 0.1 >/dev/null 2>&1; then
-		if ! perf record -o "${temp_data}" -- uname >/dev/null 2>&1; then
+		if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+			-- uname >/dev/null 2>&1; then
 			echo "Skipping test, cannot record perf events"
 			exit 2
 		fi
tools/perf/tests/shell/test_netdev_times_python.sh +3 -3
index 393bf446efb2..0d8d10a42cab 100755
@@ -85,9 +85,9 @@ then
 fi
 
 # Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -a -o "${temp_data}" \
-	-- ping -c 1 127.0.0.1 >/dev/null 2>&1; then
-	perf record -e cycles -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -a -o "${temp_data}" \
+	-- ping -c 1 -W 1 127.0.0.1 >/dev/null 2>&1; then
+	perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
 		-- perf test -w noploop >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 fi
tools/perf/tests/shell/test_powerpc_hcalls_python.sh +2 -2
index 19569c6d3613..b140ccf00ffa 100755
@@ -67,8 +67,8 @@ if ! grep -q "H_REMOVE.*1.*1500.*1500.*1500" "${temp_out}"; then
 fi
 
 # Create a perf.data file if powerpc hcall tracepoints are available on this host.
-if ! perf record -e powerpc:hcall_entry,powerpc:hcall_exit -a -o "${temp_data}" \
-	-- perf test -w noploop >/dev/null 2>&1; then
+if ! perf record -B -N --no-bpf-event -e powerpc:hcall_entry,powerpc:hcall_exit \
+	-a -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
 	echo "Skipping live record test, powerpc hcall tracepoints not available"
 	exit 0
 fi
tools/perf/tests/shell/test_rw_by_file_python.sh +3 -2
index f121597f418f..e3c538c3ace3 100755
@@ -37,8 +37,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing rw-by-file.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
-	perf record -e syscalls:sys_enter_read,syscalls:sys_enter_write -a -o "${temp_data}" \
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
+	perf record -B -N --no-bpf-event -e syscalls:sys_enter_read,syscalls:sys_enter_write \
+		-o "${temp_data}" \
 		-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
tools/perf/tests/shell/test_rw_by_pid_python.sh +2 -2
index 41eaa29097a2..cd93c1d9714c 100755
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing rw-by-pid.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
 	ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
 	ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
-	perf record -e "$ev" -a -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
 		-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
tools/perf/tests/shell/test_rwtop_python.sh +2 -2
index 05897e384702..d33807923c92 100755
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing rwtop.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
 	ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
 	ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
-	perf record -e "$ev" -a -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
 		-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
tools/perf/tests/shell/test_sched_migration_python.sh +2 -2
index 75465bcd3791..714275b0e492 100755
@@ -44,10 +44,10 @@ echo "Testing sched-migration.py..."
 ev="sched:sched_switch,sched:sched_migrate_task"
 ev="${ev},sched:sched_wakeup_new,sched:sched_wakeup"
 has_sched=1
-if ! perf record -e "$ev" -a -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
 	-- sleep 0.1 >/dev/null 2>&1; then
 	has_sched=0
-	perf record -e cycles -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
 		-- perf test -w noploop >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 fi
tools/perf/tests/shell/test_stackcollapse_python.sh +1 -1
index ce0f80407eec..2048734c5e0d 100755
@@ -37,7 +37,7 @@ echo "Testing stackcollapse.py..."
 
 # Create a perf.data file with callchains. Use a busy workload rather than
 # sleep, as an idle system may not generate any samples at all.
-perf record -g -o "${temp_data}" \
+perf record -B -N --no-bpf-event -g -o "${temp_data}" \
 		-- perf test -w noploop >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 
tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh +3 -3
index 45bbdc554c8f..300289fe33b4 100755
@@ -43,14 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 test_file_mode() {
 	echo "Testing syscall-counts-by-pid.py..."
 	# Some systems might not have raw_syscalls:sys_enter
-	if ! perf list | grep -q raw_syscalls:sys_enter; then
+	if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
 		echo "Skipping test, raw_syscalls:sys_enter not found"
 		exit 2
 	fi
 
 	# Generate some syscall events
-	perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \
-		-- sleep 0.5 >/dev/null 2>&1 || \
+	perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+		-- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 
 	if ! perf script syscall-counts-by-pid -i "${temp_data}" > "${temp_out}"; then
tools/perf/tests/shell/test_syscall_counts_python.sh +3 -2
index 310e05d399f9..86463f14391b 100755
@@ -43,13 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 test_file_mode() {
 	echo "Testing syscall-counts.py..."
 	# Some systems might not have raw_syscalls:sys_enter (e.g. stripped kernels or permissions)
-	if ! perf list | grep -q raw_syscalls:sys_enter; then
+	if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
 		echo "Skipping test, raw_syscalls:sys_enter not found"
 		exit 2
 	fi
 
 	# Generate some syscall events
-	if ! perf record -e raw_syscalls:sys_enter -o "${temp_data}" -- sleep 0.5 2>/dev/null; then
+	if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+		-- sh -c "sleep 0.1; sleep 0.05" 2>/dev/null; then
 		echo "perf record failed (permissions?), skipping file mode test."
 		exit 2
 	fi
tools/perf/tests/shell/test_task_analyzer.sh +2 -1
index f99a637a6099..f53fa4a27c79 100755
@@ -60,7 +60,8 @@ skip_no_probe_record_support() {
 
 prepare_perf_data() {
 	# 1s should be sufficient to catch at least some switches
-	perf record -e sched:sched_switch -a -o "${perfdata}" -- sleep 1 > /dev/null 2>&1
+	perf record -B -N --no-bpf-event -e sched:sched_switch -a -o "${perfdata}" \
+		-- sleep 1 > /dev/null 2>&1
 	# check if perf data file got created in above step.
 	if [ ! -e "${perfdata}" ]; then
 		printf "FAIL: perf record failed to create \"${perfdata}\" \n"
tools/perf/tests/shell/test_wakeup_latency_python.sh +4 -3
index cb2571449168..b43a86996be2 100755
@@ -41,9 +41,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing wakeup-latency.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "sched:sched_wakeup"; then
+if perf list tracepoint | grep -q "sched:sched_wakeup"; then
 	ev="sched:sched_wakeup,sched:sched_wakeup_new,sched:sched_switch"
-	perf record -e "$ev" -a -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
 		-- sleep 0.1 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
@@ -70,7 +70,8 @@ else
 fi
 
 # Also test zero-wakeups / unhandled events path to verify division-by-zero protection
-if perf record -e cycles -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+if perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
+	-- perf test -w noploop >/dev/null 2>&1; then
 	if ! perf script wakeup-latency -i "${temp_data}" > "${temp_out}" || \
 	   ! grep -q "avg_wakeup_latency (ns): N/A" "${temp_out}"; then
 		echo "wakeup-latency zero-wakeups guard test failed"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#31 lore raw

Re: [PATCH v2 6/9] perf test: Reduce overhead and contention in Python shell tests

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 18 lines

When all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh, test_rw_by_file_python.sh, test_rw_by_pid_python.sh, test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and test_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in test_compaction_times_python.sh.

Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author reviewer
#32 lore raw

PATCH v2 7/9 perf python event_analyzing_sample: Default to in-memory SQLite database

In reply to Ian Rogers

When -d/--db is not specified, event_analyzing_sample.py created a temporary file in /tmp via tempfile.mkstemp() and deleted it in trace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.

Default to sqlite3.connect(":memory:") when db_path is not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode in test_event_analyzing_sample_python.sh.

Fixes eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/python/event_analyzing_sample.py   | 35 ++++++-------------
 .../test_event_analyzing_sample_python.sh     |  8 ++++-
 2 files changed, 17 insertions(+), 26 deletions(-)
tools/perf/python/event_analyzing_sample.py +10 -25
index 15b47cff9fa3..b4efee88d94b 100755
@@ -17,10 +17,8 @@ from __future__ import annotations
 
 import argparse
 import math
-import os
 import sqlite3
 import struct
-import tempfile
 from typing import Any
 import perf
 
@@ -117,16 +115,11 @@ session: Any = None
 
 class _DB:
     con: sqlite3.Connection | None = None
-    temp_path: str | None = None
 
 def trace_begin(db_path: str | None = None) -> None:
     """Initialize database tables."""
     print("In trace_begin:\n")
-    if not db_path:
-        fd, db_path = tempfile.mkstemp(prefix="perf_events_", suffix=".db")
-        os.close(fd)
-        _DB.temp_path = db_path
-    con = sqlite3.connect(db_path)
+    con = sqlite3.connect(db_path or ":memory:")
     try:
         # Drop any pre-existing tables so repeated runs do not accumulate duplicate events.
         con.execute("drop table if exists gen_events;")
@@ -297,28 +290,20 @@ def show_pebs_ll() -> None:
 def trace_end() -> None:
     """Called at the end of trace processing."""
     print("In trace_end:\n")
-    try:
-        if _DB.con:
-            try:
-                _DB.con.commit()
-                show_general_events()
-                show_pebs_ll()
-            finally:
-                _DB.con.close()
-                _DB.con = None
-    finally:
-        if _DB.temp_path and os.path.exists(_DB.temp_path):
-            try:
-                os.remove(_DB.temp_path)
-            except OSError:
-                pass
-            _DB.temp_path = None
+    if _DB.con:
+        try:
+            _DB.con.commit()
+            show_general_events()
+            show_pebs_ll()
+        finally:
+            _DB.con.close()
+            _DB.con = None
 
 if __name__ == "__main__":
     ap = argparse.ArgumentParser(description="Analyze events with SQLite")
     ap.add_argument("-i", "--input", default="perf.data", help="Input file name")
     ap.add_argument("-d", "--db", "--database", dest="database", default=None,
-                    help="Database file name (defaults to a temporary file cleaned up on exit)")
+                    help="Database file name (defaults to an in-memory database)")
     args = ap.parse_args()
 
     try:
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +7 -1
index 5f0a41080acf..cddb12b67698 100755
@@ -45,8 +45,14 @@ test_file_mode() {
 		exit 2
 	fi
 
-	# Run the script
+	# Run the script with default (:memory:) database and with explicit -d path
 	if ! perf script event_analyzing_sample -i "${temp_data}" \
+		> "${temp_dir}/perf.mem.out" 2>&1 || \
+	   ! grep -q "Statistics about the general events" "${temp_dir}/perf.mem.out" || \
+	   grep -q "Error creating/inserting event" "${temp_dir}/perf.mem.out"; then
+		echo "Default in-memory database mode test failed."
+		err=1
+	elif ! perf script event_analyzing_sample -i "${temp_data}" \
 		-d "${temp_db}" > "${temp_dir}/perf.out" 2>&1; then
 		echo "File mode test failed."
 		err=1
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#33 lore raw

Re: [PATCH v2 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines

When -d/--db is not specified, event_analyzing_sample.py created a temporary file in /tmp via tempfile.mkstemp() and deleted it in trace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.

Quoted 8 lines

Default to sqlite3.connect(":memory:") when db_path is not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode in test_event_analyzing_sample_python.sh.

Fixes eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
author reviewer
#34 lore raw

PATCH v2 8/9 perf python: Initialize debug output on module load

In reply to Ian Rogers

Unlike the perf binary's main(), PyInit_perf() does not call perf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hits debug_file() with _debug_file == NULL and emits 'debug_file not set' without a trailing newline before the actual diagnostic message.

Call perf_debug_setup() in PyInit_perf() and add the missing trailing newline to the fallback warning in debug_file().

Fixes ec49230cf6dd ("perf debug: Expose debug file")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/util/debug.c  | 2 +-
 tools/perf/util/python.c | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
tools/perf/util/debug.c +1 -1
index 6b5ffe81f141..b7095519419e 100644
@@ -55,7 +55,7 @@ FILE *debug_file(void)
 {
 	if (!_debug_file) {
 		debug_set_file(stderr);
-		pr_warning_once("debug_file not set");
+		pr_warning_once("%s not set\n", __func__);
 	}
 	return _debug_file;
 }
tools/perf/util/python.c +1 -0
index 690a94fb4f62..95140dfaef9c 100644
@@ -5157,6 +5157,7 @@ PyMODINIT_FUNC PyInit_perf(void)
 
 	/* The page_size is placed in util object. */
 	page_size = sysconf(_SC_PAGE_SIZE);
+	perf_debug_setup();
 
 	Py_INCREF(&pyrf_evlist__type);
 	PyModule_AddObject(module, "evlist", (PyObject *)&pyrf_evlist__type);
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#35 lore raw

Re: [PATCH v2 8/9] perf python: Initialize debug output on module load

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines

Unlike the perf binary's main(), PyInit_perf() does not call perf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hits debug_file() with _debug_file == NULL and emits 'debug_file not set' without a trailing newline before the actual diagnostic message.

Quoted 6 lines

Call perf_debug_setup() in PyInit_perf() and add the missing trailing newline to the fallback warning in debug_file().

Fixes ec49230cf6dd ("perf debug: Expose debug file")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
author reviewer
#36 lore raw

PATCH v2 9/9 perf pmu: Fix race with concurrent tracepoint creation and removal in perf list

In reply to Ian Rogers

perf_pmus__print_pmu_events() first counts events via perf_pmu__num_events() to allocate the aliases array, then populates it via perf_pmu__for_each_event(). When dynamic tracepoints (such as kprobes or uprobes during 'perf test' runs) are concurrently created or removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len, leaving trailing zeroed entries in aliases[] with name == NULL and pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning /sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns -ENOENT and aborts enumeration of all remaining tracepoint subsystems.
- If an event is added, perf_pmus__print_pmu_events__callback() aborts enumeration when state->index reaches state->aliases_len.

Grow state->aliases dynamically via realloc() when needed, use state.index as the actual populated length for sorting and printing, and ignore -ENOENT when a tracepoint subsystem directory disappears during enumeration.

Fixes c3245d2093c1 ("perf pmu: Abstract alias/event struct")
Fixes 45b6e281cb06 ("perf tp_pmu: Add event APIs")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/util/pmus.c   | 21 +++++++++++++++++----
 tools/perf/util/tp_pmu.c |  8 ++++++--
 2 files changed, 23 insertions(+), 6 deletions(-)
tools/perf/util/pmus.c +17 -4
index e0a4cb2428ca..1355b317f3ed 100644
@@ -8,6 +8,7 @@
 #include <sys/types.h>
 #include <ctype.h>
 #include <pthread.h>
+#include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
 #include "cpumap.h"
@@ -571,7 +572,7 @@ static int cmp_sevent(const void *a, const void *b)
 	}
 
 	/* Order by event name. */
-	return strcmp(as->name, bs->name);
+	return strcmp(as->name ?: "", bs->name ?: "");
 }
 
 static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
@@ -581,7 +582,7 @@ static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
 		return false;
 
 	/* Don't remove duplicates for different PMUs */
-	return strcmp(a->pmu_name, b->pmu_name) == 0;
+	return strcmp(a->pmu_name ?: "", b->pmu_name ?: "") == 0;
 }
 
 struct events_callback_state {
@@ -597,8 +598,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,
 	struct sevent *s;
 
 	if (state->index >= state->aliases_len) {
-		pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
-		return 1;
+		size_t new_len = max_t(size_t, 16, state->aliases_len * 2);
+		struct sevent *new_aliases;
+
+		new_aliases = realloc(state->aliases, new_len * sizeof(struct sevent));
+		if (!new_aliases) {
+			pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
+			return 1;
+		}
+		memset(&new_aliases[state->aliases_len], 0,
+		       (new_len - state->aliases_len) * sizeof(struct sevent));
+		state->aliases = new_aliases;
+		state->aliases_len = new_len;
 	}
 	assert(info->pmu != NULL || info->name != NULL);
 	s = &state->aliases[state->index];
@@ -654,6 +665,8 @@ void perf_pmus__print_pmu_events(const struct print_callbacks *print_cb, void *p
 		perf_pmu__for_each_event(pmu, skip_duplicate_pmus, &state,
 					 perf_pmus__print_pmu_events__callback);
 	}
+	aliases = state.aliases;
+	len = state.index;
 	qsort(aliases, len, sizeof(struct sevent), cmp_sevent);
 	for (int j = 0; j < len; j++) {
 		/* Skip duplicates */
tools/perf/util/tp_pmu.c +6 -2
index c2be8c9f9084..5ac732e06841 100644
@@ -151,7 +151,9 @@ static int for_each_event_cb(void *state, const char *sys_name, const char *evt_
 
 static int for_each_event_sys_cb(void *state, const char *sys_name)
 {
-	return tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+	int ret = tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+
+	return ret == -ENOENT ? 0 : ret;
 }
 
 int tp_pmu__for_each_event(struct perf_pmu *pmu, void *state, pmu_event_callback cb)
@@ -176,7 +178,9 @@ static int num_events_cb(void *state, const char *sys_name __maybe_unused,
 
 static int num_events_sys_cb(void *state, const char *sys_name)
 {
-	return tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+	int ret = tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+
+	return ret == -ENOENT ? 0 : ret;
 }
 
 size_t tp_pmu__num_events(struct perf_pmu *pmu __maybe_unused)
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#37 lore raw

Re: [PATCH v2 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 20 lines

perf_pmus__print_pmu_events() first counts events via perf_pmu__num_events() to allocate the aliases array, then populates it via perf_pmu__for_each_event(). When dynamic tracepoints (such as kprobes or uprobes during 'perf test' runs) are concurrently created or removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len, leaving trailing zeroed entries in aliases[] with name == NULL and pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning /sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns -ENOENT and aborts enumeration of all remaining tracepoint subsystems.
- If an event is added, perf_pmus__print_pmu_events__callback() aborts enumeration when state->index reaches state->aliases_len.

Grow state->aliases dynamically via realloc() when needed, use state.index as the actual populated length for sorting and printing, and ignore -ENOENT when a tracepoint subsystem directory disappears during enumeration.

Quoted 4 lines
Fixes c3245d2093c1 ("perf pmu: Abstract alias/event struct")
Fixes 45b6e281cb06 ("perf tp_pmu: Add event APIs")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
author reviewer
#38 lore raw

v3 cover perf python: Follow-up fixes and shell test deflaking

In reply to Ian Rogers

This series contains follow-up fixes for the standalone Python scripts and their shell tests, particularly when running 'perf test' as root under heavy parallel load:

- Clamp sample_id size calculation for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE records to work around kernels without "perf/core: Restore header fields in sideband output callbacks" [1].
- Fix offline interval printing in sctop.py and live signal races in stat-cpi.py.
- Deflake Intel PT, failed-syscalls, and other Python shell tests under parallel 'perf test' load by scoping tracepoint checks to 'perf list tracepoint', passing '-B -N --no-bpf-event' to 'perf record', and avoiding unnecessary system-wide ('-a') recordings.
- Default event_analyzing_sample.py to an in-memory SQLite database (':memory:') when '--db' is not specified.
- Initialize perf debug output ('perf_debug_setup()') when loading the 'perf' Python extension module.
- Fix races in 'perf list' ('perf_pmus__print_pmu_events()' and 'tp_pmu.c') when concurrent tests dynamically create and remove kprobe/uprobes tracepoints.

v3:
- Wrap _open_live_evlist() inside the try/except KeyboardInterrupt block in stat-cpi.py so signals arriving during event initialization are caught and cleaned up gracefully.

v2:
- Fix 8-byte alignment calculation for PERF_RECORD_TEXT_POKE payload in evsel__event_size() by aligning fixed + old_len + new_len.
- Keep offline interval clock advancing on all samples in sctop.py while flushing any remaining syscalls at EOF.
- Use 'mktemp -d' private directories in test_sctop_python.sh and test_failed_syscalls_python.sh so temporary files are not unlinked and recreated directly in /tmp.
- Explicitly include <stdlib.h> in tools/perf/util/pmus.c.

[1] https://lore.kernel.org/r/[email protected]

Ian Rogers (9):
perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke
events
perf python sctop: Fix offline interval printing and test flakiness perf python stat-cpi: Fix live mode signal races and test flakiness perf test: Deflake Intel PT Python shell tests under load perf test: Deflake failed-syscalls Python shell tests under load perf test: Reduce overhead and contention in Python shell tests perf python event_analyzing_sample: Default to in-memory SQLite
database
perf python: Initialize debug output on module load
perf pmu: Fix race with concurrent tracepoint creation and removal in
perf list

 tools/perf/python/event_analyzing_sample.py   | 35 +++-------
 tools/perf/python/sctop.py                    |  7 +-
 tools/perf/python/stat-cpi.py                 | 32 ++++++---
 .../shell/test_check_perf_trace_python.sh     |  8 ++-
 .../shell/test_compaction_times_python.sh     |  8 ++-
 .../test_event_analyzing_sample_python.sh     | 11 ++-
 .../shell/test_export_to_postgresql_python.sh | 49 +++++++------
 .../shell/test_export_to_sqlite_python.sh     | 50 ++++++++------
 .../test_failed_syscalls_by_pid_python.sh     | 68 +++++++++++--------
 .../shell/test_failed_syscalls_python.sh      | 54 +++++++++------
 .../tests/shell/test_flamegraph_python.sh     |  5 +-
 .../shell/test_futex_contention_python.sh     |  7 +-
 tools/perf/tests/shell/test_gecko_python.sh   |  3 +-
 .../shell/test_intel_pt_events_python.sh      | 43 ++++++------
 .../tests/shell/test_mem_phys_addr_python.sh  |  8 ++-
 .../shell/test_net_dropmonitor_python.sh      |  9 +--
 .../tests/shell/test_netdev_times_python.sh   |  6 +-
 .../tests/shell/test_powerpc_hcalls_python.sh |  4 +-
 .../tests/shell/test_rw_by_file_python.sh     |  5 +-
 .../perf/tests/shell/test_rw_by_pid_python.sh |  4 +-
 tools/perf/tests/shell/test_rwtop_python.sh   |  4 +-
 .../shell/test_sched_migration_python.sh      |  4 +-
 tools/perf/tests/shell/test_sctop_python.sh   | 58 ++++++++--------
 .../tests/shell/test_stackcollapse_python.sh  |  2 +-
 .../perf/tests/shell/test_stat_cpi_python.sh  | 12 +++-
 .../test_syscall_counts_by_pid_python.sh      |  6 +-
 .../tests/shell/test_syscall_counts_python.sh |  5 +-
 tools/perf/tests/shell/test_task_analyzer.sh  |  3 +-
 .../tests/shell/test_wakeup_latency_python.sh |  7 +-
 tools/perf/util/debug.c                       |  2 +-
 tools/perf/util/evlist.c                      |  6 +-
 tools/perf/util/evsel.c                       | 59 +++++++++++++++-
 tools/perf/util/evsel.h                       |  1 +
 tools/perf/util/pmus.c                        | 21 ++++--
 tools/perf/util/python.c                      |  1 +
 tools/perf/util/tp_pmu.c                      |  8 ++-
 36 files changed, 378 insertions(+), 237 deletions(-)
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
author reviewer
#39 lore raw

PATCH v3 1/9 perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events

On kernels without the fix ("perf/core: Restore header fields in sideband output callbacks") to restore event_id.header.size in perf_event_ksymbol_output(), perf_event_bpf_output(), and perf_event_text_poke_output(), concurrent perf sessions cause those sideband records to be emitted with header.size inflated by multiple id_header_size increments while the single id_sample is written immediately after the event payload. Indexing backwards from event->header.size reads uninitialized ring-buffer bytes at the end of the record, causing evlist__event2evsel() to fail with -EFAULT.

Add evsel__event_size() to clamp the effective size used to locate the trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving event->header.size intact for advancing the ring-buffer/file stream.

Fixes 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")
Fixes 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")
Fixes 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/util/evlist.c |  6 ++--
 tools/perf/util/evsel.c  | 59 +++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/evsel.h  |  1 +
 3 files changed, 63 insertions(+), 3 deletions(-)
tools/perf/util/evlist.c +4 -2
index 9392d912d254..c2402e4791b6 100644
@@ -973,13 +973,15 @@ static int evlist__event2id(struct evlist *evlist, union perf_event *event, u64
 	const __u64 *array = event->sample.array;
 	ssize_t n;
 
-	n = (event->header.size - sizeof(event->header)) >> 3;
-
 	if (event->header.type == PERF_RECORD_SAMPLE) {
+		n = (event->header.size - sizeof(event->header)) >> 3;
 		if (evlist__id_pos(evlist) >= n)
 			return -1;
 		*id = array[evlist__id_pos(evlist)];
 	} else {
+		u16 size = evsel__event_size(evlist__first(evlist), event);
+
+		n = (size - sizeof(event->header)) >> 3;
 		if (evlist__is_pos(evlist) > n)
 			return -1;
 		n -= evlist__is_pos(evlist);
tools/perf/util/evsel.c +58 -1

71 lines not on this page · raw message

tools/perf/util/evsel.h +1 -0
index 5c5799cee601..174f3414fd3c 100644
@@ -469,6 +469,7 @@ int evsel__parse_sample_timestamp(struct evsel *evsel, union perf_event *event,
 				  u64 *timestamp);
 
 u16 evsel__id_hdr_size(const struct evsel *evsel);
+u16 evsel__event_size(const struct evsel *evsel, const union perf_event *event);
 
 static inline struct evsel *evsel__next(struct evsel *evsel)
 {
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#40 lore raw

Re: [PATCH v3 1/9] perf evsel: Clamp sample_id size for ksymbol, bpf, and text_poke events

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines

On kernels without the fix ("perf/core: Restore header fields in sideband output callbacks") to restore event_id.header.size in perf_event_ksymbol_output(), perf_event_bpf_output(), and perf_event_text_poke_output(), concurrent perf sessions cause those sideband records to be emitted with header.size inflated by multiple id_header_size increments while the single id_sample is written immediately after the event payload. Indexing backwards from event->header.size reads uninitialized ring-buffer bytes at the end of the record, causing evlist__event2evsel() to fail with -EFAULT.

Add evsel__event_size() to clamp the effective size used to locate the trailing id_sample for PERF_RECORD_KSYMBOL, PERF_RECORD_BPF_EVENT, and PERF_RECORD_TEXT_POKE to payload + id_hdr_size while leaving event->header.size intact for advancing the ring-buffer/file stream.

Quoted 6 lines
Fixes 9aa0bfa370b2 ("perf tools: Handle PERF_RECORD_KSYMBOL")
Fixes 45178a928a4b ("perf tools: Handle PERF_RECORD_BPF_EVENT")
Fixes 246eba8e9041 ("perf tools: Add support for PERF_RECORD_TEXT_POKE")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author reviewer
#41 lore raw

PATCH v3 2/9 perf python sctop: Fix offline interval printing and test flakiness

In reply to Ian Rogers

In sctop.py:
- If an earlier interval elapsed and printed an empty table before the target comm ('sleep') executed any syscalls, analyzer.printed became True and the final partial interval containing the target comm's syscalls was never flushed at EOF. Flush print_current_totals() in finally when analyzer.syscalls is non-empty as well as when nothing has been printed yet.
- Initialize analyzer.e_machine after creating perf.session rather than when session is still None.

In test_sctop_python.sh:
- Use a private temporary directory via 'mktemp -d'.
- Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a bounded retry loop so 'sleep's PERF_RECORD_COMM and sys_enter events are reliably captured under heavy load.

Fixes b83f0bacf5e4 ("perf python: Port sctop to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/python/sctop.py                  |  7 ++-
 tools/perf/tests/shell/test_sctop_python.sh | 58 ++++++++++-----------
 2 files changed, 34 insertions(+), 31 deletions(-)
tools/perf/python/sctop.py +5 -2
index 42e95ecfffad..48320af8b755 100755
@@ -37,6 +37,7 @@ class SCTopAnalyzer:
         self.offline = offline
         self.own_pid = os.getpid()
         self.last_print_time: Optional[int] = None
+        self.printed = False
         self.session: Optional[perf.session] = None
         self.e_machine: Optional[int] = None
 
@@ -137,6 +138,7 @@ class SCTopAnalyzer:
 
     def print_current_totals(self):
         """Print current syscall totals."""
+        self.printed = True
         # Clear terminal
         if not self.offline:
             print("\x1b[2J\x1b[H", end="")
@@ -217,8 +219,8 @@ def main():
         if args.input:
             session = perf.session(perf.data(args.input), sample=analyzer.process_event)
             analyzer.session = session
-            session.process_events()
             analyzer.e_machine = getattr(session, "e_machine", None)
+            session.process_events()
         else:
             try:
                 live_session = LiveSession(
@@ -237,7 +239,8 @@ def main():
         sys.exit(1)
     finally:
         if args.input:
-            analyzer.print_current_totals()
+            if not analyzer.printed or analyzer.syscalls:
+                analyzer.print_current_totals()
             # Break the reference cycle between perf.session and analyzer.process_event
             # because perf.session lacks cyclic GC support (tp_traverse).
             analyzer.session = None
tools/perf/tests/shell/test_sctop_python.sh +29 -29

80 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#42 lore raw

Re: [PATCH v3 2/9] perf python sctop: Fix offline interval printing and test flakiness

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines

In sctop.py:
- If an earlier interval elapsed and printed an empty table before the target comm ('sleep') executed any syscalls, analyzer.printed became True and the final partial interval containing the target comm's syscalls was never flushed at EOF. Flush print_current_totals() in finally when analyzer.syscalls is non-empty as well as when nothing has been printed yet.
- Initialize analyzer.e_machine after creating perf.session rather than when session is still None.

In test_sctop_python.sh:
- Use a private temporary directory via 'mktemp -d'.
- Drop '-a' and pass '-B -N --no-bpf-event' to 'perf record', and sleep briefly in the subshell ('sh -c "sleep 0.1; sleep 0.05"') with a bounded retry loop so 'sleep's PERF_RECORD_COMM and sys_enter events are reliably captured under heavy load.

Quoted 3 lines
Fixes b83f0bacf5e4 ("perf python: Port sctop to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author reviewer
#43 lore raw

PATCH v3 3/9 perf python stat-cpi: Fix live mode signal races and test flakiness

In reply to Ian Rogers

In test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a 1-second duration and can exit under heavy parallel load before 'perf stat -p' and 'perf script stat-cpi' finish starting up. In addition, the fixed 'sleep 0.5' before sending SIGINT can fire before Python finishes importing the perf module, opening the live evlist, and flushing the first interval.

In stat-cpi.py, register SIGINT and SIGTERM handlers before calling _open_live_evlist() and pass flush=True when printing live output so redirected stdout is flushed immediately after each interval.

In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the target workload stays alive until killed, and poll the output file for 'cpi' (up to 5 seconds) before sending SIGINT.

Fixes 4425182d426b ("perf python: Port stat-cpi to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/python/stat-cpi.py                 | 32 +++++++++++++------
 .../perf/tests/shell/test_stat_cpi_python.sh  | 12 +++++--
 2 files changed, 31 insertions(+), 13 deletions(-)
tools/perf/python/stat-cpi.py +22 -10
index 0b7d76876a6c..b4ec87b07938 100755
@@ -106,7 +106,8 @@ class StatCpiAnalyzer:
             if ins != 0:
                 cpi = cyc / float(ins)
             t_sec = timestamp / 1000000000.0
-            print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})")
+            print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})",
+                  flush=True)
 
     def read_counters(self, evlist: Any) -> None:
         """Read counters live."""
@@ -151,6 +152,7 @@ class StatCpiAnalyzer:
 
         last_err: Optional[OSError] = None
         for events, tmap in candidates:
+            evlist = None
             try:
                 evlist = perf.parse_events(events, None, tmap)
                 for evsel in evlist:
@@ -161,32 +163,41 @@ class StatCpiAnalyzer:
                 evlist.enable()
                 return evlist
             except PermissionError as e:
+                if evlist is not None:
+                    evlist.close()
                 last_err = e
             except OSError as e:
+                if evlist is not None:
+                    evlist.close()
                 if e.errno == 13:
                     last_err = e
                 else:
                     raise
+            except BaseException:
+                if evlist is not None:
+                    evlist.close()
+                raise
         if last_err is not None:
             raise last_err
         raise RuntimeError("Failed to open events")
 
     def run_live(self) -> None:
         """Read counters live."""
-        try:
-            evlist = self._open_live_evlist()
-        except OSError as e:
-            print(f"Failed to open events: {e}", file=sys.stderr)
-            sys.exit(1)
-
         def handle_signal(_signum: int, _frame: Any) -> None:
             raise KeyboardInterrupt
 
         signal.signal(signal.SIGINT, signal.default_int_handler)
         signal.signal(signal.SIGTERM, handle_signal)
 
-        print("Live mode started. Press Ctrl+C to stop.")
+        evlist = None
         try:
+            try:
+                evlist = self._open_live_evlist()
+            except OSError as e:
+                print(f"Failed to open events: {e}", file=sys.stderr)
+                sys.exit(1)
+
+            print("Live mode started. Press Ctrl+C to stop.", flush=True)
             while True:
                 time.sleep(self.args.interval)
                 timestamp = time.time_ns()
@@ -195,9 +206,10 @@ class StatCpiAnalyzer:
                 self.data.clear()
                 self.recorded_pairs.clear()
         except KeyboardInterrupt:
-            print("\nStopped.")
+            print("\nStopped.", flush=True)
         finally:
-            evlist.close()
+            if evlist is not None:
+                evlist.close()
 
 def main() -> None:
     """Main function."""
tools/perf/tests/shell/test_stat_cpi_python.sh +9 -3
index fe7562307634..6cb376c92e2f 100755
@@ -50,7 +50,7 @@ test_live_mode() {
 		echo "perf stat failed (permissions?), skipping live mode test."
 		return 0
 	fi
-	perf test -w noploop &
+	perf test -w noploop 60 &
 	workload_pid=$!
 	if ! perf stat -e cycles,instructions -p "$workload_pid" -- sleep 0.05 2>/dev/null && \
 	   ! perf stat -e cycles:u,instructions:u -p "$workload_pid" -- sleep 0.05 2>/dev/null; then
@@ -61,10 +61,16 @@ test_live_mode() {
 	fi
 	ran=1
 
-	# Run live mode for 1 interval in the background, give it a tiny sleep, then interrupt
+	# Run live mode in the background, wait until at least one interval is
+	# printed, then interrupt.
 	perf script stat-cpi -I 0.1 -p "$workload_pid" > "${temp_out}" &
 	pid=$!
-	sleep 0.5
+	for _ in $(seq 1 50); do
+		if grep -q "cpi" "${temp_out}"; then
+			break
+		fi
+		sleep 0.1
+	done
 	kill -INT "$pid" 2>/dev/null || true
 	set +e
 	wait "$pid"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#44 lore raw

Re: [PATCH v3 3/9] perf python stat-cpi: Fix live mode signal races and test flakiness

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines

In test_stat_cpi_python.sh, 'perf test -w noploop &' defaults to a 1-second duration and can exit under heavy parallel load before 'perf stat -p' and 'perf script stat-cpi' finish starting up. In addition, the fixed 'sleep 0.5' before sending SIGINT can fire before Python finishes importing the perf module, opening the live evlist, and flushing the first interval.

In stat-cpi.py, register SIGINT and SIGTERM handlers before calling _open_live_evlist() and pass flush=True when printing live output so redirected stdout is flushed immediately after each interval.

Quoted 7 lines

In test_stat_cpi_python.sh, run 'perf test -w noploop 60 &' so the target workload stays alive until killed, and poll the output file for 'cpi' (up to 5 seconds) before sending SIGINT.

Fixes 4425182d426b ("perf python: Port stat-cpi to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author reviewer
#45 lore raw

PATCH v3 4/9 perf test: Deflake Intel PT Python shell tests under load

In reply to Ian Rogers

In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).

Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh to avoid build-id cache and BPF synthesis overhead.

Fixes d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
Fixes 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
Fixes b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 .../shell/test_export_to_postgresql_python.sh | 49 ++++++++++--------
 .../shell/test_export_to_sqlite_python.sh     | 50 +++++++++++--------
 .../shell/test_intel_pt_events_python.sh      | 43 +++++++++-------
 3 files changed, 79 insertions(+), 63 deletions(-)
tools/perf/tests/shell/test_export_to_postgresql_python.sh +27 -22
index 350813466700..835f48f002e9 100755
@@ -61,9 +61,10 @@ test_file_mode() {
 	fi
 
 	# Generate events with callchains and context switches
-	if ! perf record -g --switch-events -o "${temp_data}" \
+	if ! perf record -B -N --no-bpf-event -g --switch-events -o "${temp_data}" \
 	     -- perf test -w noploop >/dev/null 2>&1 && \
-	   ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	   ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+	     -- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
@@ -92,29 +93,33 @@ test_file_mode() {
 test_intel_pt() {
 	echo "Testing export-to-postgresql.py with intel_pt..."
 
-	psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
-	rm -f "${temp_data}"
-	# Generate some intel_pt events; use a subshell that waits for uname
-	if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
-		-- sh -c "uname; true" >/dev/null 2>&1; then
-		echo "Skipping intel_pt test, intel_pt not available."
-		return 0
-	fi
+	# Generate some intel_pt events; sleep briefly after uname in the subshell
+	# so uname's AUX buffer is flushed before SIGCHLD stops perf record.
+	passed=0
+	for _ in 1 2 3 4 5; do
+		psql -c "DROP DATABASE IF EXISTS ${temp_db}" postgres >/dev/null 2>&1 || true
+		rm -f "${temp_data}"
+		if ! perf record -B -N --no-bpf-event -e intel_pt//u -o "${temp_data}" \
+			-- sh -c "uname; sleep 0.05" >/dev/null 2>&1; then
+			echo "Skipping intel_pt test, intel_pt not available."
+			return 0
+		fi
 
-	# Run the script with --itrace cr to synthesize call_returns
-	if ! perf script export-to-postgresql -i "${temp_data}" \
-		-o "${temp_db}" --itrace cr >/dev/null; then
-		echo "intel_pt file mode test failed."
-		err=1
-	else
-		# Check DB for calls
-		if ! psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
+		# Run the script with --itrace cr to synthesize call_returns
+		if perf script export-to-postgresql -i "${temp_data}" \
+			-o "${temp_db}" --itrace cr >/dev/null && \
+		   psql -d "${temp_db}" -t -c 'SELECT COUNT(*) FROM calls WHERE id > 0;' | \
 			grep -q '[1-9]'; then
-			echo "PostgreSQL intel_pt validation failed (no calls found)."
-			err=1
-		else
-			echo "intel_pt test passed (cr validated)."
+			passed=1
+			break
 		fi
+	done
+
+	if [ "$passed" -eq 0 ]; then
+		echo "PostgreSQL intel_pt validation failed (no calls found)."
+		err=1
+	else
+		echo "intel_pt test passed (cr validated)."
 	fi
 }
 
tools/perf/tests/shell/test_export_to_sqlite_python.sh +28 -22

66 lines not on this page · raw message

tools/perf/tests/shell/test_intel_pt_events_python.sh +24 -19

58 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#46 lore raw

Re: [PATCH v3 4/9] perf test: Deflake Intel PT Python shell tests under load

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines

In test_intel_pt_events_python.sh, test_export_to_sqlite_python.sh, and test_export_to_postgresql_python.sh, 'sh -c "uname; true"' uses the shell builtin 'true' and exits within microseconds of 'uname', sending SIGCHLD to 'perf record' before the Intel PT AUX buffer is always flushed under heavy parallel load (~5-10% drop rate).

Sleep 0.05s in the subshell after 'uname' ('sh -c "uname; sleep 0.05"') so 'uname' completely exits and flushes its AUX trace before 'sh' exits, and wrap the record and verification step in a bounded retry loop (up to 5 attempts). Also pass '-B -N --no-bpf-event' to 'perf record -g' in test_export_to_sqlite_python.sh and test_export_to_postgresql_python.sh to avoid build-id cache and BPF synthesis overhead.

Quoted 5 lines
Fixes d4ce72e9e238 ("perf python: Port intel-pt-events and libxed to perf module")
Fixes 62d350135e67 ("perf python: Port export-to-sqlite to perf module")
Fixes b1f968c9656a ("perf python: Port export-to-postgresql to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author reviewer
#47 lore raw

PATCH v3 5/9 perf test: Deflake failed-syscalls Python shell tests under load

In reply to Ian Rogers

When running 'perf test' in parallel under heavy system load, a one-shot 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event' can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and ENOENT sys_exit events are captured, while also contending on ~/.debug build-id caching and BPF event synthesis.

Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell after 'ls' so its events are flushed before the parent subshell exits, and wrap the record and check steps in a bounded retry loop (up to 5 attempts).

Fixes b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")
Fixes 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 .../test_failed_syscalls_by_pid_python.sh     | 68 +++++++++++--------
 .../shell/test_failed_syscalls_python.sh      | 54 +++++++++------
 2 files changed, 71 insertions(+), 51 deletions(-)
tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh +39 -29
index 215372d4e1c5..070a7ef6b8ad 100755
@@ -41,8 +41,10 @@ test_file_mode() {
 	echo "Testing failed-syscalls-by-pid.py..."
 
 	# Check if syscalls:sys_exit is supported/readable
-	if ! perf record -e syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
-		if ! perf record -e raw_syscalls:sys_exit -o /dev/null -- true >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -e syscalls:sys_exit \
+		-o /dev/null -- true >/dev/null 2>&1; then
+		if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_exit \
+			-o /dev/null -- true >/dev/null 2>&1; then
 			echo "Skipping test, no syscalls:sys_exit or raw_syscalls:sys_exit event"
 			exit 2
 		else
@@ -52,41 +54,49 @@ test_file_mode() {
 		EVENT="syscalls:sys_exit"
 	fi
 
-	# Generate some events by running a command that should fail at least some syscall
-	# (e.g. failing stat on non-existent file).
-	# Using '|| true' because 'perf record' returns the exit code of 'ls',
-	# which fails with ENOENT
-	perf record -e "${EVENT}" -o "${temp_data}" -- ls /does_not_exist >/dev/null 2>&1 || true
+	# Generate some events by running a command that fails a syscall
+	# (e.g. failing stat on non-existent file), sleeping briefly in the
+	# subshell so ls's PERF_RECORD_COMM and sys_exit events are flushed.
+	passed=0
+	for _ in 1 2 3 4 5; do
+		rm -f "${temp_data}" "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
+		perf record -B -N --no-bpf-event -e "${EVENT}" -o "${temp_data}" \
+			-- sh -c "ls /does_not_exist 2>/dev/null; sleep 0.05 || true" \
+			>/dev/null 2>&1 || true
+		if [ ! -s "${temp_data}" ]; then
+			continue
+		fi
+
+		if perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}" && \
+		   grep -n -q "err = ENOENT" "${temp_out}" && \
+		   perf script failed-syscalls-by-pid -i "${temp_data}" \
+			"ls" > "${temp_out}.comm" && \
+		   grep -q "err = ENOENT" "${temp_out}.comm"; then
+			ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' \
+				"${temp_out}.comm" | head -n 1)
+			if [ -n "${ls_pid}" ] && \
+			   perf script failed-syscalls-by-pid -i "${temp_data}" \
+				"${ls_pid}" > "${temp_out}.pid" && \
+			   grep -q "err = ENOENT" "${temp_out}.pid"; then
+				passed=1
+				break
+			fi
+		fi
+	done
+
 	if [ ! -s "${temp_data}" ]; then
 		echo "Skipping test, perf record failed to create data"
 		exit 2
 	fi
 
-	# Run the script and check output
-	if ! perf script failed-syscalls-by-pid -i "${temp_data}" > "${temp_out}"; then
+	if [ "$passed" -eq 0 ]; then
 		echo "failed-syscalls-by-pid test failed."
-		err=1
-	elif ! grep -n -q "err = ENOENT" "${temp_out}"; then
-		echo "Failed to find expected failed syscalls"
-		cat "${temp_out}"
-		err=1
-	elif ! perf script failed-syscalls-by-pid -i "${temp_data}" "ls" > "${temp_out}.comm" || \
-	     ! grep -q "err = ENOENT" "${temp_out}.comm"; then
-		echo "failed-syscalls-by-pid comm filter test failed."
-		cat "${temp_out}.comm"
+		cat "${temp_out}" 2>/dev/null || true
+		cat "${temp_out}.comm" 2>/dev/null || true
+		cat "${temp_out}.pid" 2>/dev/null || true
 		err=1
 	else
-		ls_pid=$(sed -n 's/^ls \[\([0-9][0-9]*\)\].*/\1/p' "${temp_out}.comm" | head -n 1)
-		if [ -z "${ls_pid}" ] || \
-		   ! perf script failed-syscalls-by-pid -i "${temp_data}" \
-			"${ls_pid}" > "${temp_out}.pid" || \
-		   ! grep -q "err = ENOENT" "${temp_out}.pid"; then
-			echo "failed-syscalls-by-pid PID filter test failed."
-			cat "${temp_out}.pid" 2>/dev/null || true
-			err=1
-		else
-			echo "failed-syscalls-by-pid test passed."
-		fi
+		echo "failed-syscalls-by-pid test passed."
 	fi
 	rm -f "${temp_out}" "${temp_out}.comm" "${temp_out}.pid"
 }
tools/perf/tests/shell/test_failed_syscalls_python.sh +32 -22

82 lines not on this page · raw message

Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#48 lore raw

Re: [PATCH v3 5/9] perf test: Deflake failed-syscalls Python shell tests under load

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines

When running 'perf test' in parallel under heavy system load, a one-shot 'perf record -e ... -- ls /does_not_exist' without '-B -N --no-bpf-event' can exit in under 300 microseconds before 'ls's PERF_RECORD_COMM and ENOENT sys_exit events are captured, while also contending on ~/.debug build-id caching and BPF event synthesis.

Pass '-B -N --no-bpf-event' to 'perf record', sleep 0.05s in a subshell after 'ls' so its events are flushed before the parent subshell exits, and wrap the record and check steps in a bounded retry loop (up to 5 attempts).

Quoted 4 lines
Fixes b76c43d09b06 ("perf python: Port failed-syscalls-by-pid to perf module")
Fixes 4e3fe6987cba ("perf python: Port failed-syscalls from Perl to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author reviewer
#49 lore raw

PATCH v3 6/9 perf test: Reduce overhead and contention in Python shell tests

In reply to Ian Rogers

When all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh, test_rw_by_file_python.sh, test_rw_by_pid_python.sh, test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and test_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in test_compaction_times_python.sh.

Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/tests/shell/test_check_perf_trace_python.sh   | 8 +++++---
 tools/perf/tests/shell/test_compaction_times_python.sh   | 8 +++++---
 .../tests/shell/test_event_analyzing_sample_python.sh    | 3 ++-
 tools/perf/tests/shell/test_flamegraph_python.sh         | 5 +++--
 tools/perf/tests/shell/test_futex_contention_python.sh   | 7 ++++---
 tools/perf/tests/shell/test_gecko_python.sh              | 3 ++-
 tools/perf/tests/shell/test_mem_phys_addr_python.sh      | 8 +++++---
 tools/perf/tests/shell/test_net_dropmonitor_python.sh    | 9 +++++----
 tools/perf/tests/shell/test_netdev_times_python.sh       | 6 +++---
 tools/perf/tests/shell/test_powerpc_hcalls_python.sh     | 4 ++--
 tools/perf/tests/shell/test_rw_by_file_python.sh         | 5 +++--
 tools/perf/tests/shell/test_rw_by_pid_python.sh          | 4 ++--
 tools/perf/tests/shell/test_rwtop_python.sh              | 4 ++--
 tools/perf/tests/shell/test_sched_migration_python.sh    | 4 ++--
 tools/perf/tests/shell/test_stackcollapse_python.sh      | 2 +-
 .../tests/shell/test_syscall_counts_by_pid_python.sh     | 6 +++---
 tools/perf/tests/shell/test_syscall_counts_python.sh     | 5 +++--
 tools/perf/tests/shell/test_task_analyzer.sh             | 3 ++-
 tools/perf/tests/shell/test_wakeup_latency_python.sh     | 7 ++++---
 19 files changed, 58 insertions(+), 43 deletions(-)
tools/perf/tests/shell/test_check_perf_trace_python.sh +5 -3
index 92cc7b1f0033..baf55945d429 100755
@@ -45,10 +45,11 @@ test_file_mode() {
 	echo "Testing check-perf-trace.py..."
 
 	events=""
-	if perf list | grep -q "irq:softirq_entry"; then
+	tp_list=$(perf list tracepoint)
+	if echo "$tp_list" | grep -q "irq:softirq_entry"; then
 		events="irq:softirq_entry"
 	fi
-	if perf list | grep -q "kmem:kmalloc"; then
+	if echo "$tp_list" | grep -q "kmem:kmalloc"; then
 		if [ -n "$events" ]; then
 			events="$events,kmem:kmalloc,kmem:kfree"
 		else
@@ -62,7 +63,8 @@ test_file_mode() {
 	fi
 
 	# Generate events
-	if ! perf record -e "$events" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -e "$events" -o "${temp_data}" \
+		-- sh -c "sleep 0.1" >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
tools/perf/tests/shell/test_compaction_times_python.sh +5 -3
index 50397afaafed..3f5a71e52388 100755
@@ -44,15 +44,17 @@ test_file_mode() {
 	echo "Testing compaction-times.py..."
 
 	# Check for any compaction events to see if kernel supports it
-	if ! perf list | grep -q "compaction:mm_compaction_begin"; then
+	if ! perf list tracepoint | grep -q "compaction:mm_compaction_begin"; then
 		echo "Skipping test, compaction tracepoints not found"
 		exit 2
 	fi
 
 	# Generate some events
-	# We might not naturally trigger compaction in 0.5s sleep, but the script
+	# We might not naturally trigger compaction in 0.1s sleep, but the script
 	# should parse the empty or sparse file correctly without crashing.
-	if ! perf record -e "compaction:*" -a -o "${temp_data}" -- sleep 0.5 >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event \
+		-e "compaction:mm_compaction_begin,compaction:mm_compaction_end" \
+		-a -o "${temp_data}" -- sleep 0.1 >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +2 -1
index a7a8ded003db..5f0a41080acf 100755
@@ -39,7 +39,8 @@ test_file_mode() {
 	echo "Testing event_analyzing_sample.py..."
 
 	# Generate some events
-	if ! perf record -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+		-- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record failed"
 		exit 2
 	fi
tools/perf/tests/shell/test_flamegraph_python.sh +3 -2
index 86313c72f53e..787d6bbdf04a 100755
@@ -60,7 +60,8 @@ test_file_mode() {
 	echo "Testing flamegraph.py..."
 
 	# Generate some events with callchains
-	if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+		-- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record -g failed (permissions or lack of support)"
 		exit 2
 	fi
@@ -75,7 +76,7 @@ test_file_mode() {
 
 	# Run the script in pipe ('-') mode and validate JSON output
 	rm -f "${temp_json}"
-	if ! perf record -g -o - -- perf test -w noploop 2>/dev/null | \
+	if ! perf record -B -N --no-bpf-event -g -o - -- perf test -w noploop 2>/dev/null | \
 		perf script flamegraph -i - -f json -o "${temp_json}" >/dev/null; then
 		echo "Pipe stdin JSON mode test failed."
 		err=1
tools/perf/tests/shell/test_futex_contention_python.sh +4 -3
index a8846ed68ac2..41ce8470a75f 100755
@@ -89,14 +89,15 @@ EOF
 test_file_mode() {
 	echo "Testing futex-contention.py..."
 	# Some systems might not have syscalls:sys_enter_futex
-	if ! perf list | grep -q syscalls:sys_enter_futex; then
+	if ! perf list tracepoint | grep -q syscalls:sys_enter_futex; then
 		echo "Skipping file mode test, syscalls:sys_enter_futex not found"
 		return
 	fi
 
 	# Generate some futex events
-	if ! perf record -e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
-		-- sleep 0.5 2>/dev/null; then
+	if ! perf record -B -N --no-bpf-event \
+		-e syscalls:sys_enter_futex,syscalls:sys_exit_futex -a -o "${temp_data}" \
+		-- sleep 0.1 2>/dev/null; then
 		echo "Skipping file mode test (record failed)"
 		return
 	fi
tools/perf/tests/shell/test_gecko_python.sh +2 -1
index de1d7fcf88b7..18e9f5cd2ab0 100755
@@ -39,7 +39,8 @@ test_file_mode() {
 	echo "Testing gecko.py..."
 
 	# Generate some events with callchains
-	if ! perf record -g -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -g -o "${temp_data}" \
+		-- perf test -w noploop >/dev/null 2>&1; then
 		echo "Skipping test, perf record -g failed (permissions or lack of support)"
 		exit 2
 	fi
tools/perf/tests/shell/test_mem_phys_addr_python.sh +5 -3
index 451692d5e089..cbb9f572be9d 100755
@@ -78,10 +78,12 @@ test_file_mode() {
 	echo "Testing mem-phys-addr.py file mode..."
 
 	# Generate memory access events (try unprivileged user-space first, then system-wide)
-	if ! perf record --phys-data -d -o "${temp_data}" \
+	if ! perf record -B -N --no-bpf-event --phys-data -d -o "${temp_data}" \
 	     -- perf test -w datasym >/dev/null 2>&1 && \
-	   ! perf record -d -o "${temp_data}" -- perf test -w datasym >/dev/null 2>&1 && \
-	   ! perf record -d -a -o "${temp_data}" -- sleep 0.2 >/dev/null 2>&1; then
+	   ! perf record -B -N --no-bpf-event -d -o "${temp_data}" \
+	     -- perf test -w datasym >/dev/null 2>&1 && \
+	   ! perf record -B -N --no-bpf-event -d -a -o "${temp_data}" \
+	     -- sleep 0.1 >/dev/null 2>&1; then
 		echo "Skipping file mode record test, perf record -d not supported"
 		return 0
 	fi
tools/perf/tests/shell/test_net_dropmonitor_python.sh +5 -4
index 05056a897bc9..2e37994418fc 100755
@@ -42,11 +42,12 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing net_dropmonitor.py..."
 
 # Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -o "${temp_data}" -a \
-	-- ping -c 1 255.255.255.255 >/dev/null 2>&1; then
-	if ! perf record -e skb:kfree_skb -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" -a \
+	-- ping -c 1 -W 1 255.255.255.255 >/dev/null 2>&1; then
+	if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -o "${temp_data}" \
 		-- sleep 0.1 >/dev/null 2>&1; then
-		if ! perf record -o "${temp_data}" -- uname >/dev/null 2>&1; then
+		if ! perf record -B -N --no-bpf-event -o "${temp_data}" \
+			-- uname >/dev/null 2>&1; then
 			echo "Skipping test, cannot record perf events"
 			exit 2
 		fi
tools/perf/tests/shell/test_netdev_times_python.sh +3 -3
index 393bf446efb2..0d8d10a42cab 100755
@@ -85,9 +85,9 @@ then
 fi
 
 # Create a perf.data file. Force dropping a packet if tracepoint is available!
-if ! perf record -e skb:kfree_skb -a -o "${temp_data}" \
-	-- ping -c 1 127.0.0.1 >/dev/null 2>&1; then
-	perf record -e cycles -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e skb:kfree_skb -a -o "${temp_data}" \
+	-- ping -c 1 -W 1 127.0.0.1 >/dev/null 2>&1; then
+	perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
 		-- perf test -w noploop >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 fi
tools/perf/tests/shell/test_powerpc_hcalls_python.sh +2 -2
index 19569c6d3613..b140ccf00ffa 100755
@@ -67,8 +67,8 @@ if ! grep -q "H_REMOVE.*1.*1500.*1500.*1500" "${temp_out}"; then
 fi
 
 # Create a perf.data file if powerpc hcall tracepoints are available on this host.
-if ! perf record -e powerpc:hcall_entry,powerpc:hcall_exit -a -o "${temp_data}" \
-	-- perf test -w noploop >/dev/null 2>&1; then
+if ! perf record -B -N --no-bpf-event -e powerpc:hcall_entry,powerpc:hcall_exit \
+	-a -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
 	echo "Skipping live record test, powerpc hcall tracepoints not available"
 	exit 0
 fi
tools/perf/tests/shell/test_rw_by_file_python.sh +3 -2
index f121597f418f..e3c538c3ace3 100755
@@ -37,8 +37,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing rw-by-file.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
-	perf record -e syscalls:sys_enter_read,syscalls:sys_enter_write -a -o "${temp_data}" \
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
+	perf record -B -N --no-bpf-event -e syscalls:sys_enter_read,syscalls:sys_enter_write \
+		-o "${temp_data}" \
 		-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
tools/perf/tests/shell/test_rw_by_pid_python.sh +2 -2
index 41eaa29097a2..cd93c1d9714c 100755
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing rw-by-pid.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
 	ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
 	ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
-	perf record -e "$ev" -a -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
 		-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
tools/perf/tests/shell/test_rwtop_python.sh +2 -2
index 05897e384702..d33807923c92 100755
@@ -41,10 +41,10 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing rwtop.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "syscalls:sys_enter_read"; then
+if perf list tracepoint | grep -q "syscalls:sys_enter_read"; then
 	ev="syscalls:sys_enter_read,syscalls:sys_exit_read"
 	ev="${ev},syscalls:sys_enter_write,syscalls:sys_exit_write"
-	perf record -e "$ev" -a -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e "$ev" -o "${temp_data}" \
 		-- dd if=/dev/urandom of=/dev/null bs=1M count=10 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
tools/perf/tests/shell/test_sched_migration_python.sh +2 -2
index 75465bcd3791..714275b0e492 100755
@@ -44,10 +44,10 @@ echo "Testing sched-migration.py..."
 ev="sched:sched_switch,sched:sched_migrate_task"
 ev="${ev},sched:sched_wakeup_new,sched:sched_wakeup"
 has_sched=1
-if ! perf record -e "$ev" -a -o "${temp_data}" \
+if ! perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
 	-- sleep 0.1 >/dev/null 2>&1; then
 	has_sched=0
-	perf record -e cycles -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
 		-- perf test -w noploop >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 fi
tools/perf/tests/shell/test_stackcollapse_python.sh +1 -1
index ce0f80407eec..2048734c5e0d 100755
@@ -37,7 +37,7 @@ echo "Testing stackcollapse.py..."
 
 # Create a perf.data file with callchains. Use a busy workload rather than
 # sleep, as an idle system may not generate any samples at all.
-perf record -g -o "${temp_data}" \
+perf record -B -N --no-bpf-event -g -o "${temp_data}" \
 		-- perf test -w noploop >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 
tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh +3 -3
index 45bbdc554c8f..300289fe33b4 100755
@@ -43,14 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 test_file_mode() {
 	echo "Testing syscall-counts-by-pid.py..."
 	# Some systems might not have raw_syscalls:sys_enter
-	if ! perf list | grep -q raw_syscalls:sys_enter; then
+	if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
 		echo "Skipping test, raw_syscalls:sys_enter not found"
 		exit 2
 	fi
 
 	# Generate some syscall events
-	perf record -e raw_syscalls:sys_enter -a -o "${temp_data}" \
-		-- sleep 0.5 >/dev/null 2>&1 || \
+	perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+		-- sh -c "sleep 0.1; sleep 0.05" >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 
 	if ! perf script syscall-counts-by-pid -i "${temp_data}" > "${temp_out}"; then
tools/perf/tests/shell/test_syscall_counts_python.sh +3 -2
index 310e05d399f9..86463f14391b 100755
@@ -43,13 +43,14 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 test_file_mode() {
 	echo "Testing syscall-counts.py..."
 	# Some systems might not have raw_syscalls:sys_enter (e.g. stripped kernels or permissions)
-	if ! perf list | grep -q raw_syscalls:sys_enter; then
+	if ! perf list tracepoint | grep -q raw_syscalls:sys_enter; then
 		echo "Skipping test, raw_syscalls:sys_enter not found"
 		exit 2
 	fi
 
 	# Generate some syscall events
-	if ! perf record -e raw_syscalls:sys_enter -o "${temp_data}" -- sleep 0.5 2>/dev/null; then
+	if ! perf record -B -N --no-bpf-event -e raw_syscalls:sys_enter -o "${temp_data}" \
+		-- sh -c "sleep 0.1; sleep 0.05" 2>/dev/null; then
 		echo "perf record failed (permissions?), skipping file mode test."
 		exit 2
 	fi
tools/perf/tests/shell/test_task_analyzer.sh +2 -1
index f99a637a6099..f53fa4a27c79 100755
@@ -60,7 +60,8 @@ skip_no_probe_record_support() {
 
 prepare_perf_data() {
 	# 1s should be sufficient to catch at least some switches
-	perf record -e sched:sched_switch -a -o "${perfdata}" -- sleep 1 > /dev/null 2>&1
+	perf record -B -N --no-bpf-event -e sched:sched_switch -a -o "${perfdata}" \
+		-- sleep 1 > /dev/null 2>&1
 	# check if perf data file got created in above step.
 	if [ ! -e "${perfdata}" ]; then
 		printf "FAIL: perf record failed to create \"${perfdata}\" \n"
tools/perf/tests/shell/test_wakeup_latency_python.sh +4 -3
index cb2571449168..b43a86996be2 100755
@@ -41,9 +41,9 @@ temp_out=$(mktemp /tmp/perf.out.XXXXXX)
 echo "Testing wakeup-latency.py..."
 
 # Create a perf.data file. Try to get tracepoint data.
-if perf list | grep -q "sched:sched_wakeup"; then
+if perf list tracepoint | grep -q "sched:sched_wakeup"; then
 	ev="sched:sched_wakeup,sched:sched_wakeup_new,sched:sched_switch"
-	perf record -e "$ev" -a -o "${temp_data}" \
+	perf record -B -N --no-bpf-event -e "$ev" -a -o "${temp_data}" \
 		-- sleep 0.1 >/dev/null 2>&1 || \
 		{ echo "Skipping test, perf record failed"; exit 2; }
 else
@@ -70,7 +70,8 @@ else
 fi
 
 # Also test zero-wakeups / unhandled events path to verify division-by-zero protection
-if perf record -e cycles -o "${temp_data}" -- perf test -w noploop >/dev/null 2>&1; then
+if perf record -B -N --no-bpf-event -e cycles -o "${temp_data}" \
+	-- perf test -w noploop >/dev/null 2>&1; then
 	if ! perf script wakeup-latency -i "${temp_data}" > "${temp_out}" || \
 	   ! grep -q "avg_wakeup_latency (ns): N/A" "${temp_out}"; then
 		echo "wakeup-latency zero-wakeups guard test failed"
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#50 lore raw

Re: [PATCH v3 6/9] perf test: Reduce overhead and contention in Python shell tests

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 18 lines

When all Python shell tests run concurrently during 'perf test' under heavy load:
1. Unfiltered 'perf list' initializes all hardware PMUs and metrics in each test process; use 'perf list tracepoint' when checking for tracepoint events.
2. Running 'perf record' without '-B -N --no-bpf-event' synthesizes BPF events across the system and caches build-ids into ~/.debug; add '-B -N --no-bpf-event' to all Python shell test recordings.
3. Drop '-a' in test_check_perf_trace_python.sh, test_rw_by_file_python.sh, test_rw_by_pid_python.sh, test_rwtop_python.sh, and test_syscall_counts_by_pid_python.sh where only a single child workload ('dd' or 'sleep') is tested, avoiding system-wide /proc synthesis and ringbuffer contention.
4. Add '-W 1' to 'ping -c 1' in test_net_dropmonitor_python.sh and test_netdev_times_python.sh, and narrow 'compaction:*' to 'compaction:mm_compaction_begin,compaction:mm_compaction_end' in test_compaction_times_python.sh.

Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author reviewer
#51 lore raw

PATCH v3 7/9 perf python event_analyzing_sample: Default to in-memory SQLite database

In reply to Ian Rogers

When -d/--db is not specified, event_analyzing_sample.py created a temporary file in /tmp via tempfile.mkstemp() and deleted it in trace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.

Default to sqlite3.connect(":memory:") when db_path is not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode in test_event_analyzing_sample_python.sh.

Fixes eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/python/event_analyzing_sample.py   | 35 ++++++-------------
 .../test_event_analyzing_sample_python.sh     |  8 ++++-
 2 files changed, 17 insertions(+), 26 deletions(-)
tools/perf/python/event_analyzing_sample.py +10 -25
index 15b47cff9fa3..b4efee88d94b 100755
@@ -17,10 +17,8 @@ from __future__ import annotations
 
 import argparse
 import math
-import os
 import sqlite3
 import struct
-import tempfile
 from typing import Any
 import perf
 
@@ -117,16 +115,11 @@ session: Any = None
 
 class _DB:
     con: sqlite3.Connection | None = None
-    temp_path: str | None = None
 
 def trace_begin(db_path: str | None = None) -> None:
     """Initialize database tables."""
     print("In trace_begin:\n")
-    if not db_path:
-        fd, db_path = tempfile.mkstemp(prefix="perf_events_", suffix=".db")
-        os.close(fd)
-        _DB.temp_path = db_path
-    con = sqlite3.connect(db_path)
+    con = sqlite3.connect(db_path or ":memory:")
     try:
         # Drop any pre-existing tables so repeated runs do not accumulate duplicate events.
         con.execute("drop table if exists gen_events;")
@@ -297,28 +290,20 @@ def show_pebs_ll() -> None:
 def trace_end() -> None:
     """Called at the end of trace processing."""
     print("In trace_end:\n")
-    try:
-        if _DB.con:
-            try:
-                _DB.con.commit()
-                show_general_events()
-                show_pebs_ll()
-            finally:
-                _DB.con.close()
-                _DB.con = None
-    finally:
-        if _DB.temp_path and os.path.exists(_DB.temp_path):
-            try:
-                os.remove(_DB.temp_path)
-            except OSError:
-                pass
-            _DB.temp_path = None
+    if _DB.con:
+        try:
+            _DB.con.commit()
+            show_general_events()
+            show_pebs_ll()
+        finally:
+            _DB.con.close()
+            _DB.con = None
 
 if __name__ == "__main__":
     ap = argparse.ArgumentParser(description="Analyze events with SQLite")
     ap.add_argument("-i", "--input", default="perf.data", help="Input file name")
     ap.add_argument("-d", "--db", "--database", dest="database", default=None,
-                    help="Database file name (defaults to a temporary file cleaned up on exit)")
+                    help="Database file name (defaults to an in-memory database)")
     args = ap.parse_args()
 
     try:
tools/perf/tests/shell/test_event_analyzing_sample_python.sh +7 -1
index 5f0a41080acf..cddb12b67698 100755
@@ -45,8 +45,14 @@ test_file_mode() {
 		exit 2
 	fi
 
-	# Run the script
+	# Run the script with default (:memory:) database and with explicit -d path
 	if ! perf script event_analyzing_sample -i "${temp_data}" \
+		> "${temp_dir}/perf.mem.out" 2>&1 || \
+	   ! grep -q "Statistics about the general events" "${temp_dir}/perf.mem.out" || \
+	   grep -q "Error creating/inserting event" "${temp_dir}/perf.mem.out"; then
+		echo "Default in-memory database mode test failed."
+		err=1
+	elif ! perf script event_analyzing_sample -i "${temp_data}" \
 		-d "${temp_db}" > "${temp_dir}/perf.out" 2>&1; then
 		echo "File mode test failed."
 		err=1
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#52 lore raw

Re: [PATCH v3 7/9] perf python event_analyzing_sample: Default to in-memory SQLite database

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines

When -d/--db is not specified, event_analyzing_sample.py created a temporary file in /tmp via tempfile.mkstemp() and deleted it in trace_end(). As noted during review, creating the database in a shared /tmp directory does not reserve SQLite's auxiliary sidecar filenames (-journal or -wal), and a temporary on-disk file is unnecessary when the caller did not ask to persist the database.

Quoted 8 lines

Default to sqlite3.connect(":memory:") when db_path is not provided, removing the temporary file creation and cleanup logic, and test both the default in-memory mode and explicit -d file mode in test_event_analyzing_sample_python.sh.

Fixes eeb70645a809 ("perf python: Port event_analyzing_sample to perf module")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
author reviewer
#53 lore raw

PATCH v3 8/9 perf python: Initialize debug output on module load

In reply to Ian Rogers

Unlike the perf binary's main(), PyInit_perf() does not call perf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hits debug_file() with _debug_file == NULL and emits 'debug_file not set' without a trailing newline before the actual diagnostic message.

Call perf_debug_setup() in PyInit_perf() and add the missing trailing newline to the fallback warning in debug_file().

Fixes ec49230cf6dd ("perf debug: Expose debug file")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/util/debug.c  | 2 +-
 tools/perf/util/python.c | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
tools/perf/util/debug.c +1 -1
index 6b5ffe81f141..b7095519419e 100644
@@ -55,7 +55,7 @@ FILE *debug_file(void)
 {
 	if (!_debug_file) {
 		debug_set_file(stderr);
-		pr_warning_once("debug_file not set");
+		pr_warning_once("%s not set\n", __func__);
 	}
 	return _debug_file;
 }
tools/perf/util/python.c +1 -0
index 690a94fb4f62..95140dfaef9c 100644
@@ -5157,6 +5157,7 @@ PyMODINIT_FUNC PyInit_perf(void)
 
 	/* The page_size is placed in util object. */
 	page_size = sysconf(_SC_PAGE_SIZE);
+	perf_debug_setup();
 
 	Py_INCREF(&pyrf_evlist__type);
 	PyModule_AddObject(module, "evlist", (PyObject *)&pyrf_evlist__type);
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#54 lore raw

Re: [PATCH v3 8/9] perf python: Initialize debug output on module load

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines

Unlike the perf binary's main(), PyInit_perf() does not call perf_debug_setup(). As a result, the first warning or error printed from the perf Python C extension hits debug_file() with _debug_file == NULL and emits 'debug_file not set' without a trailing newline before the actual diagnostic message.

Quoted 6 lines

Call perf_debug_setup() in PyInit_perf() and add the missing trailing newline to the fallback warning in debug_file().

Fixes ec49230cf6dd ("perf debug: Expose debug file")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
author reviewer
#55 lore raw

PATCH v3 9/9 perf pmu: Fix race with concurrent tracepoint creation and removal in perf list

In reply to Ian Rogers

perf_pmus__print_pmu_events() first counts events via perf_pmu__num_events() to allocate the aliases array, then populates it via perf_pmu__for_each_event(). When dynamic tracepoints (such as kprobes or uprobes during 'perf test' runs) are concurrently created or removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len, leaving trailing zeroed entries in aliases[] with name == NULL and pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning /sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns -ENOENT and aborts enumeration of all remaining tracepoint subsystems.
- If an event is added, perf_pmus__print_pmu_events__callback() aborts enumeration when state->index reaches state->aliases_len.

Grow state->aliases dynamically via realloc() when needed, use state.index as the actual populated length for sorting and printing, and ignore -ENOENT when a tracepoint subsystem directory disappears during enumeration.

Fixes c3245d2093c1 ("perf pmu: Abstract alias/event struct")
Fixes 45b6e281cb06 ("perf tp_pmu: Add event APIs")
Assisted-by Antigravity:gemini-3.1-pro
---
 tools/perf/util/pmus.c   | 21 +++++++++++++++++----
 tools/perf/util/tp_pmu.c |  8 ++++++--
 2 files changed, 23 insertions(+), 6 deletions(-)
tools/perf/util/pmus.c +17 -4
index e0a4cb2428ca..1355b317f3ed 100644
@@ -8,6 +8,7 @@
 #include <sys/types.h>
 #include <ctype.h>
 #include <pthread.h>
+#include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
 #include "cpumap.h"
@@ -571,7 +572,7 @@ static int cmp_sevent(const void *a, const void *b)
 	}
 
 	/* Order by event name. */
-	return strcmp(as->name, bs->name);
+	return strcmp(as->name ?: "", bs->name ?: "");
 }
 
 static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
@@ -581,7 +582,7 @@ static bool pmu_alias_is_duplicate(struct sevent *a, struct sevent *b)
 		return false;
 
 	/* Don't remove duplicates for different PMUs */
-	return strcmp(a->pmu_name, b->pmu_name) == 0;
+	return strcmp(a->pmu_name ?: "", b->pmu_name ?: "") == 0;
 }
 
 struct events_callback_state {
@@ -597,8 +598,18 @@ static int perf_pmus__print_pmu_events__callback(void *vstate,
 	struct sevent *s;
 
 	if (state->index >= state->aliases_len) {
-		pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
-		return 1;
+		size_t new_len = max_t(size_t, 16, state->aliases_len * 2);
+		struct sevent *new_aliases;
+
+		new_aliases = realloc(state->aliases, new_len * sizeof(struct sevent));
+		if (!new_aliases) {
+			pr_err("Unexpected event %s/%s/\n", info->pmu->name, info->name);
+			return 1;
+		}
+		memset(&new_aliases[state->aliases_len], 0,
+		       (new_len - state->aliases_len) * sizeof(struct sevent));
+		state->aliases = new_aliases;
+		state->aliases_len = new_len;
 	}
 	assert(info->pmu != NULL || info->name != NULL);
 	s = &state->aliases[state->index];
@@ -654,6 +665,8 @@ void perf_pmus__print_pmu_events(const struct print_callbacks *print_cb, void *p
 		perf_pmu__for_each_event(pmu, skip_duplicate_pmus, &state,
 					 perf_pmus__print_pmu_events__callback);
 	}
+	aliases = state.aliases;
+	len = state.index;
 	qsort(aliases, len, sizeof(struct sevent), cmp_sevent);
 	for (int j = 0; j < len; j++) {
 		/* Skip duplicates */
tools/perf/util/tp_pmu.c +6 -2
index c2be8c9f9084..5ac732e06841 100644
@@ -151,7 +151,9 @@ static int for_each_event_cb(void *state, const char *sys_name, const char *evt_
 
 static int for_each_event_sys_cb(void *state, const char *sys_name)
 {
-	return tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+	int ret = tp_pmu__for_each_tp_event(sys_name, state, for_each_event_cb);
+
+	return ret == -ENOENT ? 0 : ret;
 }
 
 int tp_pmu__for_each_event(struct perf_pmu *pmu, void *state, pmu_event_callback cb)
@@ -176,7 +178,9 @@ static int num_events_cb(void *state, const char *sys_name __maybe_unused,
 
 static int num_events_sys_cb(void *state, const char *sys_name)
 {
-	return tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+	int ret = tp_pmu__for_each_tp_event(sys_name, state, num_events_cb);
+
+	return ret == -ENOENT ? 0 : ret;
 }
 
 size_t tp_pmu__num_events(struct perf_pmu *pmu __maybe_unused)
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog
#56 lore raw

Re: [PATCH v3 9/9] perf pmu: Fix race with concurrent tracepoint creation and removal in perf list

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 20 lines

perf_pmus__print_pmu_events() first counts events via perf_pmu__num_events() to allocate the aliases array, then populates it via perf_pmu__for_each_event(). When dynamic tracepoints (such as kprobes or uprobes during 'perf test' runs) are concurrently created or removed between the two passes:
- If an event is removed, state.index is smaller than the allocated len, leaving trailing zeroed entries in aliases[] with name == NULL and pmu == NULL, which causes qsort(cmp_sevent) or the print loop to crash with SIGSEGV.
- If a dynamic tracepoint subsystem directory is removed while scanning /sys/kernel/tracing/events, tp_pmu__for_each_tp_event() returns -ENOENT and aborts enumeration of all remaining tracepoint subsystems.
- If an event is added, perf_pmus__print_pmu_events__callback() aborts enumeration when state->index reaches state->aliases_len.

Grow state->aliases dynamically via realloc() when needed, use state.index as the actual populated length for sorting and printing, and ignore -ENOENT when a tracepoint subsystem directory disappears during enumeration.

Quoted 4 lines
Fixes c3245d2093c1 ("perf pmu: Abstract alias/event struct")
Fixes 45b6e281cb06 ("perf tp_pmu: Add event APIs")
Assisted-by Antigravity:gemini-3.1-pro

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
maintainer Applied
#57 lore raw
In reply to Ian Rogers
Quoted 3 lines · Ian Rogers

This series contains follow-up fixes for the standalone Python scripts and their shell tests, particularly when running 'perf test' as root under heavy parallel load:

Thanks, applied to perf-tools-next, for v7.4.

- Arnaldo