perf: Complete transition to standalone Python scripts
Subsystems (2)
Ian Rogers
reviewercover perf: Complete transition to standalone Python scripts
The perf script command has historically supported running Python and Perl scripts by embedding libpython and libperl. That approach has several drawbacks:
- high overhead from constructing Python dictionaries for every event (whether used by the script or not),
- complex build dependencies on specific Python/Perl libraries,
- threading and interpreter lifecycle complications from embedding the interpreter inside perf,
- no unified model for standalone scripts such as ilist.py, treport.py, tracepoint.py, and twatch.py.
Parts of this series were previously posted as part of the larger "perf: Reorganize scripting support" series:
https://lore.kernel.org/linux-perf-users/[email protected]/
with earlier RFC and mailing list discussion at:
https://lore.kernel.org/linux-perf-users/[email protected]/
https://lore.kernel.org/lkml/CAP-5=fWDqE8SYfOLZkg_0=4Ayx6E7O+h7uUp4NDeCFkiN4b7-w@mail.gmail.com/
From that earlier series, the prerequisite patches (header dependency cleanups, evlist/evsel reference counting, initial perf.data and perf.session C extension wrappers, callchain/brstack/stat/syscall/config accessors, perf.pyi type stubs, LiveSession, and removal of embedded libperl support) have now been merged. We are restarting the version number at v1 for the remaining patches, along with additional C extension capabilities, build-time static analysis (mypy and pylint), and review fixes.
Running scripts as standalone Python applications using the perf Python C extension module provides a dramatic speedup. For example, with mem-phys-addr.py:
Before (using embedded libpython in perf script):
``` $ perf mem record -a sleep 1 $ time perf script tools/perf/scripts/python/mem-phys-addr.py Event: cpu_core/mem-loads-aux/ Memory type count percentage --------------------------------------- ---------- ---------- 0-fff : Reserved 3217 100.0 real 0m3.754s user 0m0.023s sys 0m0.018s ```
After (using standalone Python script with perf module):
``` $ PYTHONPATH=/tmp/perf/python time python3 tools/perf/python/mem-phys-addr.py Event: evsel(cpu_core/mem-loads-aux/) Memory type count percentage --------------------------------------- ---------- ---------- 0-fff : Reserved 3217 100.0 real 0m0.106s user 0m0.021s sys 0m0.020s ```
The 49 patches in this series are organized into four groups:
1. Perf Python C Extension & Existing Script Updates (Patches 01-10):
- Refine syscall_name()/syscall_id() argument parsing, callchain stubs, and session.find_thread() lookup in tools/perf/util/python.c and tools/perf/python/perf.pyi.
- Clean up pylint warnings and improve perf script -l descriptions in existing standalone scripts (ilist.py, treport.py, tracepoint.py, twatch.py).
- Expose destination address location attributes (addr_dso, addr_symbol, addr_sym_offset), branch/transaction fields, sym_offset, context_switch callbacks, and Intel PT call_return / itrace synthesis directly in the perf Python extension (many parts previously posted in
https://lore.kernel.org/linux-perf-users/[email protected]/).
- Allow KeyboardInterrupt to propagate cleanly in LiveSession.
2. Build-Time Python Static Analysis (Patches 11-14):
- Clean up mypy and pylint warnings in tools/perf/pmu-events/ and tools/perf/tests/shell/lib/.
- Make MYPY and PYLINT build checks enabled by default when mypy and pylint (>= 2.16.0) are installed, with NO_MYPY=1 and NO_PYLINT=1 opt-out controls (matching NO_SHELLCHECK=1), so all Python scripts in tools/perf are continuously type-checked and linted at build time.
3. Porting Legacy Python & Perl Scripts to tools/perf/python/ (Patches 15-45):
- Previously sent as part of
https://lore.kernel.org/linux-perf-users/[email protected]/
and updated with static type annotations, mypy/pylint cleanups, and accompanying shell tests under tools/perf/tests/shell/. Sashiko-driven review and improvement was undertaken while porting so that the ported scripts are in the best shape possible:
* Temporarily install tools/perf/python/ scripts during the transition (Patch 15).
* General & profiling scripts: stat-cpi.py, mem-phys-addr.py, stackcollapse.py, flamegraph.py, gecko.py, event_analyzing_sample.py (Patches 16-21).
* Syscall & I/O scripts (ported from legacy Python and Perl):
syscall-counts.py, syscall-counts-by-pid.py, failed-syscalls-by-pid.py, failed-syscalls.py, sctop.py, rw-by-file.py, rw-by-pid.py, rwtop.py (Patches 22-29).
* Scheduling, locking, MM & networking scripts: futex-contention.py, task-analyzer.py, sched-migration.py (with SchedGui.py), wakeup-latency.py, compaction-times.py, net_dropmonitor.py, netdev-times.py, check-perf-trace.py (Patches 30-37).
* Hardware trace & architecture-specific scripts: arm-cs-trace-disasm.py, powerpc-hcalls.py, intel-pt-events.py (with libxed.py), and migrating the Intel PT virtual LBR test to the Python API (Patches 38-41).
* Database export & viewer utilities: export-to-sqlite.py, export-to-postgresql.py, exported-sql-viewer.py, and parallel-perf.py (Patches 42-45).
4. Removing Embedded libpython & Updating Build, CLI, and Docs (Patches 46-49):
- Previously sent as part of
https://lore.kernel.org/linux-perf-users/[email protected]/ :
* Remove embedded libpython support (scripting-engines/trace-event-python.c) and delete the legacy tools/perf/scripts/python/ directory (Patch 46).
* Replace the libpython feature check with test-python-module.c and update Python script installation paths (Patch 47).
* Update perf script (builtin-script.c) to execute standalone scripts directly and remove embedded scripting engine support (Patch 48).
* Update perf man pages (perf-script.txt, perf-script-python.txt, perf-check.txt) for standalone Python scripts (Patch 49).
Ian Rogers (49):
perf python: Update syscall format string to optional positional perf python: Update callchain stubs and session thread lookup perf python: Clean up pylint warnings in ilist.py
perf python: Clean up pylint warnings in treport.py
perf python: Clean up pylint warnings in tracepoint.py perf python: Clean up pylint warnings in twatch.py
perf python: Improve perf script -l descriptions
perf python: Expose addr location, transaction, and context_switch perf python: Add Intel PT call_return and itrace capability perf python: Allow KeyboardInterrupt to propagate in LiveSession perf pmu-events: Clean up mypy and pylint issues
perf test: Clean up mypy and pylint issues in shell test libraries perf build: Make mypy build test opt-out (NO_MYPY=1)
perf build: Make pylint build test opt-out (NO_PYLINT=1) perf Makefile: Install standalone Python scripts during transition perf python: Port stat-cpi to perf module
perf python: Port mem-phys-addr to perf module
perf python: Port stackcollapse to perf module
perf python: Port flamegraph to perf module
perf python: Port gecko to perf module
perf python: Port event_analyzing_sample to perf module perf python: Port syscall-counts to perf module
perf python: Port syscall-counts-by-pid to perf module perf python: Port failed-syscalls-by-pid to perf module perf python: Port failed-syscalls from Perl to perf module perf python: Port sctop to perf module
perf python: Port rw-by-file from Perl to perf module
perf python: Port rw-by-pid from Perl to perf module
perf python: Port rwtop from Perl to perf module
perf python: Port futex-contention to perf module
perf python: Port task-analyzer to perf module
perf python: Port sched-migration and SchedGui to perf module perf python: Port wakeup-latency from Perl to perf module perf python: Port compaction-times to perf module
perf python: Port net_dropmonitor to perf module
perf python: Port netdev-times to perf module
perf python: Port check-perf-trace to perf module
perf python: Port arm-cs-trace-disasm to perf module
perf python: Port powerpc-hcalls to perf module
perf python: Port intel-pt-events and libxed to perf module perf test: Migrate Intel PT virtual LBR test to Python API perf python: Port export-to-sqlite to perf module
perf python: Port export-to-postgresql to perf module
perf python: Move and clean up exported-sql-viewer.py
perf python: Move and clean up parallel-perf.py
perf: Remove libpython support and legacy Python scripts perf Makefile: Update Python script installation path
perf script: Support standalone scripts and remove embedded scripting perf Documentation: Update for standalone Python scripts
tools/build/Makefile.feature | 5 +-
tools/build/feature/Makefile | 24 +-
tools/build/feature/test-all.c | 6 +-
tools/build/feature/test-libperl.c | 10 -
tools/build/feature/test-libpython.c | 10 -
tools/build/feature/test-python-module.c | 13 +
tools/perf/Build | 5 +-
tools/perf/Documentation/db-export.txt | 20 +-
tools/perf/Documentation/perf-check.txt | 3 +-
tools/perf/Documentation/perf-script-perl.txt | 216 -
.../perf/Documentation/perf-script-python.txt | 718 +--
tools/perf/Documentation/perf-script.txt | 80 +-
tools/perf/Documentation/tips.txt | 1 -
tools/perf/Makefile.config | 38 +-
tools/perf/Makefile.perf | 83 +-
tools/perf/builtin-check.c | 3 +-
tools/perf/builtin-script.c | 824 ++-
tools/perf/pmu-events/Build | 4 +-
tools/perf/pmu-events/amd_metrics.py | 10 +-
tools/perf/pmu-events/intel_metrics.py | 259 +-
tools/perf/pmu-events/jevents.py | 77 +-
tools/perf/pmu-events/make_legacy_cache.py | 34 +-
tools/perf/pmu-events/metric.py | 58 +-
tools/perf/python/SchedGui.py | 246 +
tools/perf/python/arm-cs-trace-disasm.py | 350 ++
tools/perf/python/check-perf-trace.py | 213 +
tools/perf/python/compaction-times.py | 350 ++
tools/perf/python/counting.py | 1 +
tools/perf/python/event_analyzing_sample.py | 321 ++
tools/perf/python/export-to-postgresql.py | 1320 +++++
tools/perf/python/export-to-sqlite.py | 893 +++
tools/perf/python/exported-sql-viewer.py | 5104 +++++++++++++++++
tools/perf/python/failed-syscalls-by-pid.py | 175 +
tools/perf/python/failed-syscalls.py | 88 +
tools/perf/python/flamegraph.py | 274 +
tools/perf/python/futex-contention.py | 93 +
tools/perf/python/gecko.py | 415 ++
tools/perf/python/ilist.py | 8 +-
tools/perf/python/intel-pt-events.py | 606 ++
tools/perf/python/libxed.py | 123 +
tools/perf/python/mem-phys-addr.py | 137 +
tools/perf/python/net_dropmonitor.py | 156 +
tools/perf/python/netdev-times.py | 486 ++
tools/perf/python/parallel-perf.py | 1250 ++++
tools/perf/python/perf.pyi | 68 +-
tools/perf/python/perf_live.py | 5 +-
.../{scripts => }/python/powerpc-hcalls.py | 209 +-
tools/perf/python/rw-by-file.py | 109 +
tools/perf/python/rw-by-pid.py | 189 +
tools/perf/python/rwtop.py | 247 +
tools/perf/python/sched-migration.py | 486 ++
tools/perf/python/sctop.py | 216 +
tools/perf/python/stackcollapse.py | 145 +
tools/perf/python/stat-cpi.py | 208 +
tools/perf/python/syscall-counts-by-pid.py | 100 +
tools/perf/python/syscall-counts.py | 79 +
tools/perf/python/task-analyzer.py | 881 +++
tools/perf/python/tracepoint.py | 5 +-
tools/perf/python/treport.py | 12 +-
tools/perf/python/twatch.py | 81 +-
tools/perf/python/wakeup-latency.py | 94 +
tools/perf/scripts/Build | 30 -
tools/perf/scripts/install-build-deps.sh | 4 +-
tools/perf/scripts/perl/Perf-Trace-Util/Build | 9 -
.../scripts/perl/Perf-Trace-Util/Context.c | 122 -
.../scripts/perl/Perf-Trace-Util/Context.xs | 42 -
.../scripts/perl/Perf-Trace-Util/Makefile.PL | 18 -
.../perf/scripts/perl/Perf-Trace-Util/README | 59 -
.../Perf-Trace-Util/lib/Perf/Trace/Context.pm | 55 -
.../Perf-Trace-Util/lib/Perf/Trace/Core.pm | 192 -
.../Perf-Trace-Util/lib/Perf/Trace/Util.pm | 94 -
.../perf/scripts/perl/Perf-Trace-Util/typemap | 1 -
.../scripts/perl/bin/check-perf-trace-record | 2 -
.../scripts/perl/bin/failed-syscalls-record | 3 -
.../scripts/perl/bin/failed-syscalls-report | 10 -
tools/perf/scripts/perl/bin/rw-by-file-record | 3 -
tools/perf/scripts/perl/bin/rw-by-file-report | 10 -
tools/perf/scripts/perl/bin/rw-by-pid-record | 2 -
tools/perf/scripts/perl/bin/rw-by-pid-report | 3 -
tools/perf/scripts/perl/bin/rwtop-record | 2 -
tools/perf/scripts/perl/bin/rwtop-report | 20 -
.../scripts/perl/bin/wakeup-latency-record | 6 -
.../scripts/perl/bin/wakeup-latency-report | 3 -
tools/perf/scripts/perl/check-perf-trace.pl | 106 -
tools/perf/scripts/perl/failed-syscalls.pl | 47 -
tools/perf/scripts/perl/rw-by-file.pl | 106 -
tools/perf/scripts/perl/rw-by-pid.pl | 184 -
tools/perf/scripts/perl/rwtop.pl | 203 -
tools/perf/scripts/perl/wakeup-latency.pl | 107 -
.../perf/scripts/python/Perf-Trace-Util/Build | 4 -
.../scripts/python/Perf-Trace-Util/Context.c | 225 -
.../Perf-Trace-Util/lib/Perf/Trace/Core.py | 116 -
.../lib/Perf/Trace/EventClass.py | 97 -
.../lib/Perf/Trace/SchedGui.py | 184 -
.../Perf-Trace-Util/lib/Perf/Trace/Util.py | 92 -
.../scripts/python/arm-cs-trace-disasm.py | 356 --
.../python/bin/compaction-times-record | 2 -
.../python/bin/compaction-times-report | 4 -
.../python/bin/event_analyzing_sample-record | 8 -
.../python/bin/event_analyzing_sample-report | 3 -
.../python/bin/export-to-postgresql-record | 8 -
.../python/bin/export-to-postgresql-report | 29 -
.../python/bin/export-to-sqlite-record | 8 -
.../python/bin/export-to-sqlite-report | 29 -
.../python/bin/failed-syscalls-by-pid-record | 3 -
.../python/bin/failed-syscalls-by-pid-report | 10 -
.../perf/scripts/python/bin/flamegraph-record | 2 -
.../perf/scripts/python/bin/flamegraph-report | 3 -
.../python/bin/futex-contention-record | 2 -
.../python/bin/futex-contention-report | 4 -
tools/perf/scripts/python/bin/gecko-record | 2 -
tools/perf/scripts/python/bin/gecko-report | 7 -
.../scripts/python/bin/intel-pt-events-record | 13 -
.../scripts/python/bin/intel-pt-events-report | 3 -
.../scripts/python/bin/mem-phys-addr-record | 19 -
.../scripts/python/bin/mem-phys-addr-report | 3 -
.../scripts/python/bin/net_dropmonitor-record | 2 -
.../scripts/python/bin/net_dropmonitor-report | 4 -
.../scripts/python/bin/netdev-times-record | 8 -
.../scripts/python/bin/netdev-times-report | 5 -
.../scripts/python/bin/powerpc-hcalls-record | 2 -
.../scripts/python/bin/powerpc-hcalls-report | 2 -
.../scripts/python/bin/sched-migration-record | 2 -
.../scripts/python/bin/sched-migration-report | 3 -
tools/perf/scripts/python/bin/sctop-record | 3 -
tools/perf/scripts/python/bin/sctop-report | 24 -
.../scripts/python/bin/stackcollapse-record | 8 -
.../scripts/python/bin/stackcollapse-report | 3 -
.../python/bin/syscall-counts-by-pid-record | 3 -
.../python/bin/syscall-counts-by-pid-report | 10 -
.../scripts/python/bin/syscall-counts-record | 3 -
.../scripts/python/bin/syscall-counts-report | 10 -
.../scripts/python/bin/task-analyzer-record | 2 -
.../scripts/python/bin/task-analyzer-report | 3 -
tools/perf/scripts/python/check-perf-trace.py | 84 -
tools/perf/scripts/python/compaction-times.py | 311 -
.../scripts/python/event_analyzing_sample.py | 192 -
.../scripts/python/export-to-postgresql.py | 1114 ----
tools/perf/scripts/python/export-to-sqlite.py | 799 ---
.../scripts/python/exported-sql-viewer.py | 5030 ----------------
.../scripts/python/failed-syscalls-by-pid.py | 79 -
tools/perf/scripts/python/flamegraph.py | 267 -
tools/perf/scripts/python/futex-contention.py | 57 -
tools/perf/scripts/python/gecko.py | 395 --
tools/perf/scripts/python/intel-pt-events.py | 494 --
tools/perf/scripts/python/libxed.py | 107 -
tools/perf/scripts/python/mem-phys-addr.py | 127 -
tools/perf/scripts/python/net_dropmonitor.py | 78 -
tools/perf/scripts/python/netdev-times.py | 473 --
tools/perf/scripts/python/parallel-perf.py | 989 ----
tools/perf/scripts/python/sched-migration.py | 462 --
tools/perf/scripts/python/sctop.py | 89 -
tools/perf/scripts/python/stackcollapse.py | 127 -
tools/perf/scripts/python/stat-cpi.py | 79 -
.../scripts/python/syscall-counts-by-pid.py | 75 -
tools/perf/scripts/python/syscall-counts.py | 65 -
tools/perf/scripts/python/task-analyzer.py | 934 ---
tools/perf/tests/Build | 4 +-
tools/perf/tests/make | 10 +-
.../coresight/test_arm_coresight_disasm.sh | 27 +-
tools/perf/tests/shell/lib/attr.py | 86 +-
.../perf/tests/shell/lib/perf_brstack_max.py | 37 +
.../tests/shell/lib/perf_json_output_lint.py | 36 +-
.../tests/shell/lib/perf_metric_validation.py | 47 +-
tools/perf/tests/shell/script.sh | 43 +-
tools/perf/tests/shell/script_perl.sh | 102 -
tools/perf/tests/shell/script_python.sh | 113 -
.../shell/test_check_perf_trace_python.sh | 79 +
.../shell/test_compaction_times_python.sh | 81 +
.../test_event_analyzing_sample_python.sh | 58 +
.../shell/test_export_to_postgresql_python.sh | 119 +
.../shell/test_export_to_sqlite_python.sh | 108 +
.../test_failed_syscalls_by_pid_python.sh | 92 +
.../shell/test_failed_syscalls_python.sh | 81 +
.../tests/shell/test_flamegraph_python.sh | 106 +
.../shell/test_futex_contention_python.sh | 67 +
tools/perf/tests/shell/test_gecko_python.sh | 95 +
tools/perf/tests/shell/test_intel_pt.sh | 41 +-
.../shell/test_intel_pt_events_python.sh | 69 +
.../tests/shell/test_mem_phys_addr_python.sh | 101 +
.../shell/test_net_dropmonitor_python.sh | 103 +
.../tests/shell/test_netdev_times_python.sh | 61 +
.../tests/shell/test_powerpc_hcalls_python.sh | 95 +
.../tests/shell/test_rw_by_file_python.sh | 70 +
.../perf/tests/shell/test_rw_by_pid_python.sh | 77 +
tools/perf/tests/shell/test_rwtop_python.sh | 76 +
.../shell/test_sched_migration_python.sh | 69 +
tools/perf/tests/shell/test_sctop_python.sh | 71 +
.../tests/shell/test_stackcollapse_python.sh | 77 +
.../perf/tests/shell/test_stat_cpi_python.sh | 106 +
.../test_syscall_counts_by_pid_python.sh | 81 +
.../tests/shell/test_syscall_counts_python.sh | 74 +
tools/perf/tests/shell/test_task_analyzer.sh | 97 +-
.../tests/shell/test_wakeup_latency_python.sh | 83 +
tools/perf/ui/browsers/scripts.c | 217 +-
tools/perf/util/Build | 6 +-
tools/perf/util/python.c | 872 ++-
tools/perf/util/scripting-engines/Build | 9 -
.../util/scripting-engines/trace-event-perl.c | 770 ---
.../scripting-engines/trace-event-python.c | 2333 --------
tools/perf/util/trace-event-parse.c | 65 -
tools/perf/util/trace-event-scripting.c | 407 --
tools/perf/util/trace-event.h | 72 +-
203 files changed, 19555 insertions(+), 21385 deletions(-)
delete mode 100644 tools/build/feature/test-libperl.c
delete mode 100644 tools/build/feature/test-libpython.c
create mode 100644 tools/build/feature/test-python-module.c
delete mode 100644 tools/perf/Documentation/perf-script-perl.txt
create mode 100755 tools/perf/python/SchedGui.py
create mode 100755 tools/perf/python/arm-cs-trace-disasm.py
create mode 100755 tools/perf/python/check-perf-trace.py
create mode 100755 tools/perf/python/compaction-times.py
create mode 100755 tools/perf/python/event_analyzing_sample.py
create mode 100755 tools/perf/python/export-to-postgresql.py
create mode 100755 tools/perf/python/export-to-sqlite.py
create mode 100755 tools/perf/python/exported-sql-viewer.py
create mode 100755 tools/perf/python/failed-syscalls-by-pid.py
create mode 100755 tools/perf/python/failed-syscalls.py
create mode 100755 tools/perf/python/flamegraph.py
create mode 100755 tools/perf/python/futex-contention.py
create mode 100755 tools/perf/python/gecko.py
create mode 100755 tools/perf/python/intel-pt-events.py
create mode 100755 tools/perf/python/libxed.py
create mode 100755 tools/perf/python/mem-phys-addr.py
create mode 100755 tools/perf/python/net_dropmonitor.py
create mode 100755 tools/perf/python/netdev-times.py
create mode 100755 tools/perf/python/parallel-perf.py
rename tools/perf/{scripts => }/python/powerpc-hcalls.py (54%)
mode change 100644 => 100755
create mode 100755 tools/perf/python/rw-by-file.py
create mode 100755 tools/perf/python/rw-by-pid.py
create mode 100755 tools/perf/python/rwtop.py
create mode 100755 tools/perf/python/sched-migration.py
create mode 100755 tools/perf/python/sctop.py
create mode 100755 tools/perf/python/stackcollapse.py
create mode 100755 tools/perf/python/stat-cpi.py
create mode 100755 tools/perf/python/syscall-counts-by-pid.py
create mode 100755 tools/perf/python/syscall-counts.py
create mode 100755 tools/perf/python/task-analyzer.py
create mode 100755 tools/perf/python/wakeup-latency.py
delete mode 100644 tools/perf/scripts/Build
delete mode 100644 tools/perf/scripts/perl/Perf-Trace-Util/Build
delete mode 100644 tools/perf/scripts/perl/Perf-Trace-Util/Context.c
delete mode 100644 tools/perf/scripts/perl/Perf-Trace-Util/Context.xs
delete mode 100644 tools/perf/scripts/perl/Perf-Trace-Util/Makefile.PL
delete mode 100644 tools/perf/scripts/perl/Perf-Trace-Util/README
delete mode 100644 tools/perf/scripts/perl/Perf-Trace-Util/lib/Perf/Trace/Context.pm
delete mode 100644 tools/perf/scripts/perl/Perf-Trace-Util/lib/Perf/Trace/Core.pm
delete mode 100644 tools/perf/scripts/perl/Perf-Trace-Util/lib/Perf/Trace/Util.pm
delete mode 100644 tools/perf/scripts/perl/Perf-Trace-Util/typemap
delete mode 100644 tools/perf/scripts/perl/bin/check-perf-trace-record
delete mode 100644 tools/perf/scripts/perl/bin/failed-syscalls-record
delete mode 100644 tools/perf/scripts/perl/bin/failed-syscalls-report
delete mode 100644 tools/perf/scripts/perl/bin/rw-by-file-record
delete mode 100644 tools/perf/scripts/perl/bin/rw-by-file-report
delete mode 100644 tools/perf/scripts/perl/bin/rw-by-pid-record
delete mode 100644 tools/perf/scripts/perl/bin/rw-by-pid-report
delete mode 100644 tools/perf/scripts/perl/bin/rwtop-record
delete mode 100644 tools/perf/scripts/perl/bin/rwtop-report
delete mode 100644 tools/perf/scripts/perl/bin/wakeup-latency-record
delete mode 100644 tools/perf/scripts/perl/bin/wakeup-latency-report
delete mode 100644 tools/perf/scripts/perl/check-perf-trace.pl
delete mode 100644 tools/perf/scripts/perl/failed-syscalls.pl
delete mode 100644 tools/perf/scripts/perl/rw-by-file.pl
delete mode 100644 tools/perf/scripts/perl/rw-by-pid.pl
delete mode 100644 tools/perf/scripts/perl/rwtop.pl
delete mode 100644 tools/perf/scripts/perl/wakeup-latency.pl
delete mode 100644 tools/perf/scripts/python/Perf-Trace-Util/Build
delete mode 100644 tools/perf/scripts/python/Perf-Trace-Util/Context.c
delete mode 100644 tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/Core.py
delete mode 100755 tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/EventClass.py
delete mode 100644 tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/SchedGui.py
delete mode 100644 tools/perf/scripts/python/Perf-Trace-Util/lib/Perf/Trace/Util.py
delete mode 100755 tools/perf/scripts/python/arm-cs-trace-disasm.py
delete mode 100644 tools/perf/scripts/python/bin/compaction-times-record
delete mode 100644 tools/perf/scripts/python/bin/compaction-times-report
delete mode 100644 tools/perf/scripts/python/bin/event_analyzing_sample-record
delete mode 100644 tools/perf/scripts/python/bin/event_analyzing_sample-report
delete mode 100644 tools/perf/scripts/python/bin/export-to-postgresql-record
delete mode 100644 tools/perf/scripts/python/bin/export-to-postgresql-report
delete mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-record
delete mode 100644 tools/perf/scripts/python/bin/export-to-sqlite-report
delete mode 100644 tools/perf/scripts/python/bin/failed-syscalls-by-pid-record
delete mode 100644 tools/perf/scripts/python/bin/failed-syscalls-by-pid-report
delete mode 100755 tools/perf/scripts/python/bin/flamegraph-record
delete mode 100755 tools/perf/scripts/python/bin/flamegraph-report
delete mode 100644 tools/perf/scripts/python/bin/futex-contention-record
delete mode 100644 tools/perf/scripts/python/bin/futex-contention-report
delete mode 100644 tools/perf/scripts/python/bin/gecko-record
delete mode 100755 tools/perf/scripts/python/bin/gecko-report
delete mode 100644 tools/perf/scripts/python/bin/intel-pt-events-record
delete mode 100644 tools/perf/scripts/python/bin/intel-pt-events-report
delete mode 100644 tools/perf/scripts/python/bin/mem-phys-addr-record
delete mode 100644 tools/perf/scripts/python/bin/mem-phys-addr-report
delete mode 100755 tools/perf/scripts/python/bin/net_dropmonitor-record
delete mode 100755 tools/perf/scripts/python/bin/net_dropmonitor-report
delete mode 100644 tools/perf/scripts/python/bin/netdev-times-record
delete mode 100644 tools/perf/scripts/python/bin/netdev-times-report
delete mode 100644 tools/perf/scripts/python/bin/powerpc-hcalls-record
delete mode 100644 tools/perf/scripts/python/bin/powerpc-hcalls-report
delete mode 100644 tools/perf/scripts/python/bin/sched-migration-record
delete mode 100644 tools/perf/scripts/python/bin/sched-migration-report
delete mode 100644 tools/perf/scripts/python/bin/sctop-record
delete mode 100644 tools/perf/scripts/python/bin/sctop-report
delete mode 100755 tools/perf/scripts/python/bin/stackcollapse-record
delete mode 100755 tools/perf/scripts/python/bin/stackcollapse-report
delete mode 100644 tools/perf/scripts/python/bin/syscall-counts-by-pid-record
delete mode 100644 tools/perf/scripts/python/bin/syscall-counts-by-pid-report
delete mode 100644 tools/perf/scripts/python/bin/syscall-counts-record
delete mode 100644 tools/perf/scripts/python/bin/syscall-counts-report
delete mode 100755 tools/perf/scripts/python/bin/task-analyzer-record
delete mode 100755 tools/perf/scripts/python/bin/task-analyzer-report
delete mode 100644 tools/perf/scripts/python/check-perf-trace.py
delete mode 100644 tools/perf/scripts/python/compaction-times.py
delete mode 100644 tools/perf/scripts/python/event_analyzing_sample.py
delete mode 100644 tools/perf/scripts/python/export-to-postgresql.py
delete mode 100644 tools/perf/scripts/python/export-to-sqlite.py
delete mode 100755 tools/perf/scripts/python/exported-sql-viewer.py
delete mode 100644 tools/perf/scripts/python/failed-syscalls-by-pid.py
delete mode 100755 tools/perf/scripts/python/flamegraph.py
delete mode 100644 tools/perf/scripts/python/futex-contention.py
delete mode 100644 tools/perf/scripts/python/gecko.py
delete mode 100644 tools/perf/scripts/python/intel-pt-events.py
delete mode 100644 tools/perf/scripts/python/libxed.py
delete mode 100644 tools/perf/scripts/python/mem-phys-addr.py
delete mode 100755 tools/perf/scripts/python/net_dropmonitor.py
delete mode 100644 tools/perf/scripts/python/netdev-times.py
delete mode 100755 tools/perf/scripts/python/parallel-perf.py
delete mode 100644 tools/perf/scripts/python/sched-migration.py
delete mode 100644 tools/perf/scripts/python/sctop.py
delete mode 100755 tools/perf/scripts/python/stackcollapse.py
delete mode 100644 tools/perf/scripts/python/stat-cpi.py
delete mode 100644 tools/perf/scripts/python/syscall-counts-by-pid.py
delete mode 100644 tools/perf/scripts/python/syscall-counts.py
delete mode 100755 tools/perf/scripts/python/task-analyzer.py
create mode 100644 tools/perf/tests/shell/lib/perf_brstack_max.py
delete mode 100755 tools/perf/tests/shell/script_perl.sh
delete mode 100755 tools/perf/tests/shell/script_python.sh
create mode 100755 tools/perf/tests/shell/test_check_perf_trace_python.sh
create mode 100755 tools/perf/tests/shell/test_compaction_times_python.sh
create mode 100755 tools/perf/tests/shell/test_event_analyzing_sample_python.sh
create mode 100755 tools/perf/tests/shell/test_export_to_postgresql_python.sh
create mode 100755 tools/perf/tests/shell/test_export_to_sqlite_python.sh
create mode 100755 tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh
create mode 100755 tools/perf/tests/shell/test_failed_syscalls_python.sh
create mode 100755 tools/perf/tests/shell/test_flamegraph_python.sh
create mode 100755 tools/perf/tests/shell/test_futex_contention_python.sh
create mode 100755 tools/perf/tests/shell/test_gecko_python.sh
create mode 100755 tools/perf/tests/shell/test_intel_pt_events_python.sh
create mode 100755 tools/perf/tests/shell/test_mem_phys_addr_python.sh
create mode 100755 tools/perf/tests/shell/test_net_dropmonitor_python.sh
create mode 100755 tools/perf/tests/shell/test_netdev_times_python.sh
create mode 100755 tools/perf/tests/shell/test_powerpc_hcalls_python.sh
create mode 100755 tools/perf/tests/shell/test_rw_by_file_python.sh
create mode 100755 tools/perf/tests/shell/test_rw_by_pid_python.sh
create mode 100755 tools/perf/tests/shell/test_rwtop_python.sh
create mode 100755 tools/perf/tests/shell/test_sched_migration_python.sh
create mode 100755 tools/perf/tests/shell/test_sctop_python.sh
create mode 100755 tools/perf/tests/shell/test_stackcollapse_python.sh
create mode 100755 tools/perf/tests/shell/test_stat_cpi_python.sh
create mode 100755 tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh
create mode 100755 tools/perf/tests/shell/test_syscall_counts_python.sh
create mode 100755 tools/perf/tests/shell/test_wakeup_latency_python.sh
delete mode 100644 tools/perf/util/scripting-engines/Build
delete mode 100644 tools/perf/util/scripting-engines/trace-event-perl.c
delete mode 100644 tools/perf/util/scripting-engines/trace-event-python.c
delete mode 100644 tools/perf/util/trace-event-scripting.cSignature · 1 line
2.55.0.1082.g2b9226bbc0-goog
Ian Rogers
reviewerPATCH 1/49 perf python: Update syscall format string to optional positional
Update the PyArg_ParseTupleAndKeywords format string for pyrf__syscall_name() and pyrf__syscall_id() from "i|$i" to "i|i" (and "s|$i" to "s|i"), allowing elf_machine to be passed as an optional positional argument as well as a keyword argument.
Update syscall_id() and syscall_name() in perf.pyi to match, and set the return type of syscall_name() to Optional[str] as unknown syscall numbers return None.
--- tools/perf/python/perf.pyi | 4 ++-- tools/perf/util/python.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
tools/perf/python/perf.pyi +2 -2
@@ -20,7 +20,7 @@ def metrics() -> List[Dict[str, Union[str, List[str]]]]: """ ... -def syscall_name(id: int, *, elf_machine: Optional[int] = None) -> str: +def syscall_name(id: int, elf_machine: Optional[int] = None) -> Optional[str]: """Convert a syscall number to its name. Args:
@@ -32,7 +32,7 @@ def syscall_name(id: int, *, elf_machine: Optional[int] = None) -> str: """ ... -def syscall_id(name: str, *, elf_machine: Optional[int] = None) -> int: +def syscall_id(name: str, elf_machine: Optional[int] = None) -> int: """Convert a syscall name to its number. Args:
tools/perf/util/python.c +2 -2
@@ -4111,7 +4111,7 @@ static PyObject *pyrf__syscall_name(PyObject *self, PyObject *args, PyObject *kwint elf_machine =EM_HOST ;static char *kwlist[] = {"id" ,"elf_machine" ,NULL }; -if (!PyArg_ParseTupleAndKeywords (args, kwargs,"i|$i" , kwlist, &id, &elf_machine)) +if (!PyArg_ParseTupleAndKeywords (args, kwargs,"i|i" , kwlist, &id, &elf_machine))return NULL ; name =syscalltbl__name (elf_machine, id);
@@ -4127,7 +4127,7 @@ static PyObject *pyrf__syscall_id(PyObject *self, PyObject *args, PyObject *kwarint elf_machine =EM_HOST ;static char *kwlist[] = {"name" ,"elf_machine" ,NULL }; -if (!PyArg_ParseTupleAndKeywords (args, kwargs,"s|$i" , kwlist, &name, &elf_machine)) +if (!PyArg_ParseTupleAndKeywords (args, kwargs,"s|i" , kwlist, &name, &elf_machine))return NULL ; id =syscalltbl__id (elf_machine, name);
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 01/49] perf python: Update syscall format string to optional positional
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Update the
PyArg_ParseTupleAndKeywordsformat string forpyrf__syscall_name()andpyrf__syscall_id()from "i|$i" to "i|i" (and "s|$i" to "s|i"), allowingelf_machineto be passed as an optional positional argument as well as a keyword argument.Update
syscall_id()andsyscall_name()in perf.pyi to match, and set the return type ofsyscall_name()to Optional[str] as unknown syscall numbers return None.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Ian Rogers
reviewerPATCH 2/49 perf python: Update callchain stubs and session thread lookup
In reply to Ian RogersUpdate pyrf_session__find_thread() in util/python.c to accept an optional tid parameter (defaulting to pid) and return None instead of raising TypeError when a thread is not found in the session machines.
Update perf.pyi to match:
- Set session.find_thread() signature to (pid: int, tid: int = -1) -> Optional[thread].
- Update callchain_node.symbol and callchain_node.dso to str (the C getters return "[unknown]" fallback strings rather than None) and add __iter__() to callchain.
Also update treport.py to handle Optional[thread] returned by session.find_thread().
--- tools/perf/python/perf.pyi | 7 ++++--- tools/perf/python/treport.py | 2 +- tools/perf/util/python.c | 19 +++++++++++-------- 3 files changed, 16 insertions(+), 12 deletions(-)
tools/perf/python/perf.pyi +4 -3
@@ -320,13 +320,14 @@ class branch_stack: class callchain_node: """Represents a frame in the callchain.""" ip: int - symbol: Optional[str] - dso: Optional[str] + symbol: str + dso: str class callchain: """Sequence of callchain frames.""" def __len__(self) -> int: ... def __getitem__(self, index: int) -> callchain_node: ... + def __iter__(self) -> Iterator[callchain_node]: ... class stat_event(_sample_members): """Represents a stat event from perf."""
@@ -441,7 +442,7 @@ class session: def process_events(self) -> None: """Process all events in the session.""" ... - def find_thread(self, pid: int) -> thread: + def find_thread(self, pid: int, tid: int = -1) -> Optional[thread]: """Returns the thread associated with a pid.""" ...
tools/perf/python/treport.py +1 -1
@@ -92,7 +92,7 @@ class ProfileNode: try: assert session thread = session.find_thread(sample.sample_tid) - comm = thread.comm() + comm = (thread.comm() if thread else None) or f"unknown ({pid})" except Exception: comm = f"unknown ({pid})"
tools/perf/util/python.c +11 -8
@@ -3912,26 +3912,29 @@ static PyObject *pyrf_session__find_thread(struct pyrf_session *psession, PyObjestruct machine *machine;struct thread *thread =NULL ;PyObject *result; -int pid; +int pid, tid =-1 ;CHECK_INITIALIZED (psession->session,"session" ); -if (!PyArg_ParseTuple (args,"i" , &pid)) +if (!PyArg_ParseTuple (args,"i|i" , &pid, &tid))return NULL ; +if (tid ==-1 ) + tid = pid; + +/* Look up the thread in the host machine first, then fall back to guest machines. */ machine = &psession->session->machines.host; - thread =machine__find_thread (machine, pid, pid); + thread =machine__find_thread (machine, pid, tid);if (!thread) { machine =perf_session__find_machine (psession->session, pid);if (machine) - thread =machine__find_thread (machine, pid, pid); + thread =machine__find_thread (machine, pid, tid); } -if (!thread) { -PyErr_Format (PyExc_TypeError,"Failed to find thread %d" , pid); -return NULL ; - } +/* Return None rather than raising TypeError when a PID/TID is not known. */ +if (!thread) + Py_RETURN_NONE; result =pyrf_thread__from_thread (thread);thread__put (thread);return result;
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 02/49] perf python: Update callchain stubs and session thread lookup
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Update
pyrf_session__find_thread()inutil/python.cto accept an optional tid parameter (defaulting to pid) and return None instead of raising TypeError when a thread is not found in the session machines.Update perf.pyi to match:
- Setsession.find_thread()signature to (pid: int, tid: int = -1) -> Optional[thread].
- Update callchain_node.symbol and callchain_node.dso to str (the C getters return "[unknown]" fallback strings rather than None) and add__iter__()to callchain.
Quoted 5 lines
Also update
treport.pyto handle Optional[thread] returned bysession.find_thread().Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Ian Rogers
reviewerPATCH 3/49 perf python: Clean up pylint warnings in ilist.py
In reply to Ian RogersReplace bare 'except:' clauses (pylint W0702: bare-except) in ilist.py with specific exception tuples (OSError, ValueError, RuntimeError, TypeError, KeyError) and use a safe '.get("name", "")' sort key in IListApp.compose() so that ilist.py passes pylint without warning suppression comments and handles unnamed PMU events gracefully.
--- tools/perf/python/ilist.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
tools/perf/python/ilist.py +4 -4
@@ -82,7 +82,7 @@ class Metric(TreeValue): try: val = evlist.compute_metric(self.metric_name, cpu, thread) return 0 if math.isnan(val) else val - except: + except (OSError, ValueError, RuntimeError, TypeError): # Be tolerant of failures to compute metrics on particular CPUs/threads. return 0
@@ -419,7 +419,7 @@ class IListApp(App): if self.evlist: self.evlist.open() self.evlist.enable() - except: + except (OSError, ValueError, RuntimeError): self.evlist = None if not self.evlist:
@@ -450,7 +450,7 @@ class IListApp(App): pmu_name = pmu.name().lower() pmu_node = pmus.add(pmu_name) try: - for event in sorted(pmu.events(), key=lambda x: x["name"]): + for event in sorted(pmu.events(), key=lambda x: x.get("name", "")): if "deprecated" in event: continue if "name" in event:
@@ -460,7 +460,7 @@ class IListApp(App): data=PmuEvent(pmu_name, e)) else: pmu_node.add_leaf(e, data=PmuEvent(pmu_name, e)) - except: + except (OSError, ValueError, RuntimeError, KeyError, TypeError): # Reading events may fail with EPERM, ignore. pass metrics = tree.root.add("Metrics")
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 03/49] perf python: Clean up pylint warnings in ilist.py
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 8 lines
Replace bare 'except:' clauses (pylint W0702: bare-except) in
ilist.pywith specific exception tuples (OSError, ValueError, RuntimeError, TypeError, KeyError) and use a safe '.get("name", "")' sort key inIListApp.compose()so thatilist.pypasses pylint without warning suppression comments and handles unnamed PMU events gracefully.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Ian Rogers
reviewerPATCH 4/49 perf python: Clean up pylint warnings in treport.py
In reply to Ian RogersClean up pylint warnings in treport.py so that it passes pylint without warning suppression comments:
- Replace broad 'except Exception:' clauses (W0718:
broad-exception-caught) with specific exception tuples.
- Pass both pid and sample_tid to session.find_thread() so guest and process threads are resolved accurately.
- Rename parameter 'args' to 'pos_args' in FlameGraph.__init__() to avoid shadowing the outer scope variable 'args' (W0621:
redefined-outer-name).
--- tools/perf/python/treport.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
tools/perf/python/treport.py +5 -5
@@ -91,9 +91,9 @@ class ProfileNode: pid = sample.sample_pid try: assert session - thread = session.find_thread(sample.sample_tid) + thread = session.find_thread(pid, sample.sample_tid) comm = (thread.comm() if thread else None) or f"unknown ({pid})" - except Exception: + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError): comm = f"unknown ({pid})" period = sample.sample_period
@@ -412,9 +412,9 @@ class FlameGraph(ScrollView): } """ - def __init__(self, root: ProfileNode, *args, **kwargs): + def __init__(self, root: ProfileNode, *pos_args, **kwargs): """Initialize the FlameGraph widget.""" - super().__init__(*args, **kwargs) + super().__init__(*pos_args, **kwargs) self.root = root self.cursor = root self.selected = root
@@ -549,7 +549,7 @@ if __name__ == "__main__": profile = ProfileBuilder() try: session = perf.session(perf.data(input_file), sample=profile.process_event) - except Exception as e: + except (OSError, ValueError, RuntimeError) as e: print(f"Error opening session: {e}", file=sys.stderr) sys.exit(1)
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 04/49] perf python: Clean up pylint warnings in treport.py
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines
Clean up pylint warnings in
treport.pyso that it passes pylint without warning suppression comments:
- Replace broad 'except Exception:' clauses (W0718:
broad-exception-caught) with specific exception tuples.
- Pass both pid andsample_tidtosession.find_thread()so guest and process threads are resolved accurately.
- Rename parameter 'args' to 'pos_args' inFlameGraph.__init__()to avoid shadowing the outer scope variable 'args' (W0621:
redefined-outer-name).
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Ian Rogers
reviewerPATCH 5/49 perf python: Clean up pylint warnings in tracepoint.py
In reply to Ian RogersClean up pylint warnings in tracepoint.py so that it passes pylint without warning suppression comments:
- Catch ImportError instead of bare 'except:' (W0702: bare-except) when importing setproctitle.
- Remove unnecessary trailing semicolon (W0301: unnecessary-semicolon) after evlist.enable().
--- tools/perf/python/tracepoint.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/python/tracepoint.py +2 -2
@@ -9,7 +9,7 @@ def change_proctitle(): try: import setproctitle setproctitle.setproctitle("tracepoint.py") - except: + except ImportError: print("Install the setproctitle python package to help with top and friends") def main():
@@ -29,7 +29,7 @@ def main(): evlist.open() evlist.mmap() - evlist.enable(); + evlist.enable() while True: evlist.poll(timeout = -1)
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 05/49] perf python: Clean up pylint warnings in tracepoint.py
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines
Clean up pylint warnings in
tracepoint.pyso that it passes pylint without warning suppression comments:
- Catch ImportError instead of bare 'except:' (W0702: bare-except) when importing setproctitle.
- Remove unnecessary trailing semicolon (W0301: unnecessary-semicolon) afterevlist.enable().Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Ian Rogers
reviewerPATCH 6/49 perf python: Clean up pylint warnings in twatch.py
In reply to Ian RogersClean up pylint warnings in twatch.py so that it passes pylint without warning suppression comments:
- Convert tab indentation to 4 spaces (W0311: bad-indentation).
- Remove unnecessary trailing semicolon (W0301: unnecessary-semicolon) after evsel.open().
- Convert floating multi-line string comments to '#' comments (W0105:
pointless-string-statement).
--- tools/perf/python/twatch.py | 80 ++++++++++++++++++------------------- 1 file changed, 39 insertions(+), 41 deletions(-)
tools/perf/python/twatch.py +39 -41
@@ -9,53 +9,51 @@ import perf def main(context_switch = 0, thread = -1): - cpus = perf.cpu_map() - threads = perf.thread_map(thread) - evsel = perf.evsel(type = perf.TYPE_SOFTWARE, - config = perf.COUNT_SW_DUMMY, - task = 1, comm = 1, mmap = 0, freq = 0, - wakeup_events = 1, watermark = 1, - sample_id_all = 1, context_switch = context_switch, - sample_type = perf.SAMPLE_PERIOD | perf.SAMPLE_TID | perf.SAMPLE_CPU) - - """What we want are just the PERF_RECORD_ lifetime events for threads, - using the default, PERF_TYPE_HARDWARE + PERF_COUNT_HW_CYCLES & freq=1 - (the default), makes perf reenable irq_vectors:local_timer_entry, when - disabling nohz, not good for some use cases where all we want is to get - threads comes and goes... So use (perf.TYPE_SOFTWARE, perf_COUNT_SW_DUMMY, - freq=0) instead.""" - - evsel.open(cpus = cpus, threads = threads); - evlist = perf.evlist(cpus, threads) - evlist.add(evsel) - evlist.mmap() - while True: - evlist.poll(timeout = -1) - for cpu in cpus: - event = evlist.read_on_cpu(cpu) - if not event: - continue - print("cpu: {0}, pid: {1}, tid: {2} {3}".format(event.sample_cpu, + cpus = perf.cpu_map() + threads = perf.thread_map(thread) + evsel = perf.evsel(type = perf.TYPE_SOFTWARE, + config = perf.COUNT_SW_DUMMY, + task = 1, comm = 1, mmap = 0, freq = 0, + wakeup_events = 1, watermark = 1, + sample_id_all = 1, context_switch = context_switch, + sample_type = perf.SAMPLE_PERIOD | perf.SAMPLE_TID | perf.SAMPLE_CPU) + + # What we want are just the PERF_RECORD_ lifetime events for threads, + # using the default, PERF_TYPE_HARDWARE + PERF_COUNT_HW_CYCLES & freq=1 + # (the default), makes perf reenable irq_vectors:local_timer_entry, when + # disabling nohz, not good for some use cases where all we want is to get + # threads comes and goes... So use (perf.TYPE_SOFTWARE, perf_COUNT_SW_DUMMY, + # freq=0) instead. + + evsel.open(cpus = cpus, threads = threads) + evlist = perf.evlist(cpus, threads) + evlist.add(evsel) + evlist.mmap() + while True: + evlist.poll(timeout = -1) + for cpu in cpus: + event = evlist.read_on_cpu(cpu) + if not event: + continue + print("cpu: {0}, pid: {1}, tid: {2} {3}".format(event.sample_cpu, event.sample_pid, event.sample_tid, event)) if __name__ == '__main__': - """ - To test the PERF_RECORD_SWITCH record, pick a pid and replace - in the following line. + # To test the PERF_RECORD_SWITCH record, pick a pid and replace + # in the following line. - Example output: + # Example output: -cpu: 3, pid: 31463, tid: 31593 { type: context_switch, next_prev_pid: 31463, next_prev_tid: 31593, switch_out: 1 } -cpu: 1, pid: 31463, tid: 31489 { type: context_switch, next_prev_pid: 31463, next_prev_tid: 31489, switch_out: 1 } -cpu: 2, pid: 31463, tid: 31496 { type: context_switch, next_prev_pid: 31463, next_prev_tid: 31496, switch_out: 1 } -cpu: 3, pid: 31463, tid: 31491 { type: context_switch, next_prev_pid: 31463, next_prev_tid: 31491, switch_out: 0 } + # cpu: 3, pid: 31463, tid: 31593 { type: context_switch, next_prev_pid: 31463, next_prev_tid: 31593, switch_out: 1 } + # cpu: 1, pid: 31463, tid: 31489 { type: context_switch, next_prev_pid: 31463, next_prev_tid: 31489, switch_out: 1 } + # cpu: 2, pid: 31463, tid: 31496 { type: context_switch, next_prev_pid: 31463, next_prev_tid: 31496, switch_out: 1 } + # cpu: 3, pid: 31463, tid: 31491 { type: context_switch, next_prev_pid: 31463, next_prev_tid: 31491, switch_out: 0 } + # + # It is possible as well to use event.misc & perf.PERF_RECORD_MISC_SWITCH_OUT + # to figure out if this is a context switch in or out of the monitored threads. - It is possible as well to use event.misc & perf.PERF_RECORD_MISC_SWITCH_OUT - to figure out if this is a context switch in or out of the monitored threads. - - If bored, please add command line option parsing support for these options :-) - """ - # main(context_switch = 1, thread = 31463) + # If bored, please add command line option parsing support for these options :-) + # # main(context_switch = 1, thread = 31463) main()
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 06/49] perf python: Clean up pylint warnings in twatch.py
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines
Clean up pylint warnings in
twatch.pyso that it passes pylint without warning suppression comments:
- Convert tab indentation to 4 spaces (W0311: bad-indentation).
- Remove unnecessary trailing semicolon (W0301: unnecessary-semicolon) afterevsel.open().
- Convert floating multi-line string comments to '#' comments (W0105:
pointless-string-statement).Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Ian Rogers
reviewerPATCH 7/49 perf python: Improve perf script -l descriptions
In reply to Ian RogersAdd module docstrings to counting.py, tracepoint.py, and twatch.py so that 'perf script -l' displays concise descriptions for all available standalone scripts:
```
$ perf script -l
List of available scripts:
...
counting
Example for counting perf events.
...
tracepoint
Example showing how to enable a tracepoint and access its fields.
twatch
Example to show how to enable a software event and track
context switches.
...
```--- tools/perf/python/counting.py | 1 + tools/perf/python/tracepoint.py | 1 + tools/perf/python/twatch.py | 1 + 3 files changed, 3 insertions(+)
tools/perf/python/counting.py +1 -0
@@ -1,5 +1,6 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0 +"""Example for counting perf events.""" # -*- python -*- # -*- coding: utf-8 -*-
tools/perf/python/tracepoint.py +1 -0
@@ -1,5 +1,6 @@ #! /usr/bin/env python # SPDX-License-Identifier: GPL-2.0 +"""Example showing how to enable a tracepoint and access its fields.""" # -*- python -*- # -*- coding: utf-8 -*-
tools/perf/python/twatch.py +1 -0
@@ -1,5 +1,6 @@ #! /usr/bin/env python # SPDX-License-Identifier: GPL-2.0-only +"""Example to show how to enable a software event and track context switches.""" # -*- python -*- # -*- coding: utf-8 -*- # twatch - Experimental use of the perf python interface
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 07/49] perf python: Improve perf script -l descriptions
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
Add module docstrings to
counting.py,tracepoint.py, andtwatch.pyso that 'perf script -l' displays concise descriptions for all available standalone scripts:``` $ perf script -l List of available scripts: ... counting Example for counting perf events. ... tracepoint Example showing how to enable a tracepoint and access its fields. twatch Example to show how to enable a software event and track context switches. ... ```
Quoted 1 line
Signed-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Ian Rogers
reviewerPATCH 8/49 perf python: Expose addr location, transaction, and context_switch
In reply to Ian RogersExpose destination address location components (addr_dso, addr_symbol, addr_sym_offset), branch/transaction metrics (branch_type, in_tx, flags, transaction), machine_pid, vcpu, sym_offset, and context_switch callbacks directly to the Python extension.
Also include supporting fixes and infrastructure updates in util/python.c and perf.pyi:
- Support fetching sym_offset on resolved sample symbols.
- Populate sample machine_pid and vcpu from evlist__id2sid() when perf_guest is enabled, and resolve samples against the matching guest or host machine.
- Expose next_prev_pid and next_prev_tid on switch_event only for PERF_RECORD_SWITCH_CPU_WIDE records (returning None otherwise).
- Wire up tracing_data in pyrf_session__new(), use 'static char * const kwlist[]', and update perf.pyi type stubs (including srccode() and insn() return types).
--- tools/perf/python/perf.pyi | 22 +++- tools/perf/util/python.c | 249 +++++++++++++++++++++++++++++++++++-- 2 files changed, 253 insertions(+), 18 deletions(-)
tools/perf/python/perf.pyi +17 -5
@@ -182,6 +182,8 @@ class _sample_members: sample_time: int sample_id: int sample_stream_id: int + machine_pid: int + vcpu: int sample_period: int sample_cpu: int
@@ -206,10 +208,18 @@ class sample_event(_sample_members): symbol: str sym_start: int sym_end: int + sym_offset: Optional[int] + addr_dso: Optional[str] + addr_symbol: Optional[str] + addr_sym_offset: Optional[int] + branch_type: int + in_tx: int + flags: int + transaction: int brstack: Optional['branch_stack'] callchain: Optional['callchain'] - def srccode(self) -> str: ... - def insn(self) -> str: ... + def srccode(self) -> Optional[tuple[str, int, str]]: ... + def insn(self) -> Optional[bytes]: ... def __getattr__(self, name: str) -> Any: ... class mmap_event(_sample_members):
@@ -286,8 +296,9 @@ class read_event(_sample_members): class switch_event(_sample_members): """Represents a SWITCH or SWITCH_CPU_WIDE record.""" type: int - next_prev_pid: int - next_prev_tid: int + misc: int + next_prev_pid: Optional[int] + next_prev_tid: Optional[int] evsel: Optional['evsel'] class branch_entry:
@@ -429,7 +440,8 @@ class session: self, data: data, sample: Optional[Callable[[sample_event], None]] = None, - stat: Optional[Callable[[Any, Optional[str]], None]] = None + stat: Optional[Callable[[Any, Optional[str]], None]] = None, + context_switch: Optional[Callable[[switch_event], None]] = None, ) -> None: """Initialize a perf session.
tools/perf/util/python.c +236 -13
368 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 08/49] perf python: Expose addr location, transaction, and context_switch
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] perf: incorrect python type hint for srccode() returning (None, 0, None) --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] perf: incorrect python type hint for srccode() returning (None, 0, None) --
commit 8f57fbf429042a11c8af6cec00438eccce96b9f8
Author: Ian Rogers <[email protected]>
perf python: Expose addr location, transaction, and context_switch
This commit exposes additional perf sample fields, branch/transaction metrics, and context_switch callbacks directly to the Python extension. It also populates sample machine_pid and vcpu when perf_guest is enabled and updates the perf.pyi type stubs for methods like srccode() and insn().
Quoted 4 lines
tools/perf/python/perf.pyi +0
-0
[ ... ]
Quoted 7 lines
tools/perf/python/perf.pyi +3
-0@@ -206,10 +208,18 @@ class sample_event(_sample_members): symbol: str sym_start: int sym_end: int + sym_offset: Optional[int] + addr_dso: Optional[str] + addr_symbol: Optional[str]
tools/perf/python/perf.pyi class sample_event(_sample_members): · line 221
Quoted 10 lines
tools/perf/python/perf.pyi +6
-2+ addr_sym_offset: Optional[int] + branch_type: int + in_tx: int + flags: int + transaction: int brstack: Optional['branch_stack'] callchain: Optional['callchain'] - def srccode(self) -> str: ... - def insn(self) -> str: ... + def srccode(self) -> Optional[tuple[str, int, str]]: ...
[Severity: Low]
Does this type hint correctly match the C implementation?
Looking at the C implementation in pyrf_sample_event__srccode() in tools/perf/util/python.c, when the source file is not found it returns None for the strings:
if (srcfile) {
...
result = Py_BuildValue("(sIs#)", srcfile, line, srccode,
(Py_ssize_t)len);
free(srcfile);
} else {
result = Py_BuildValue("(sIs#)", NULL, 0, NULL, (Py_ssize_t)0);
}Since Py_BuildValue with "s" formats NULL as Python's None, the return type here is effectively (None, 0, None).
The type stub asserts the tuple elements are strictly str. Should this instead be Optional[tuple[Optional[str], int, Optional[str]]] to avoid mypy static analysis false positives?
Quoted 2 lines
tools/perf/python/perf.pyi +1
-0+ def insn(self) -> Optional[bytes]: ... def __getattr__(self, name: str) -> Any: ...
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Ian Rogers
reviewerPATCH 9/49 perf python: Add Intel PT call_return and itrace capability
In reply to Ian RogersAdd support for Intel PT call_return events and itrace options to the perf Python extension so standalone Python scripts can decode and inspect low-level instruction traces.
Specifically:
- Add a call_return callback parameter and itrace option string to perf.session, treating explicit None (Py_None) callbacks as omitted.
- Expose the perf.call_return object with db_id, parent_id, insn_count, cyc_count, comm, pid, tid, call_time, return_time, branch_count, call_ref, return_ref, flags, and call_path (resolved as a perf.callchain ordered from outermost caller to callee).
- Export perf.CALL_RETURN_NO_CALL, perf.CALL_RETURN_NO_RETURN, and perf.CALL_RETURN_NON_CALL flag constants.
- Expose session.e_machine and session.is_64_bit properties, preferring recorded thread ELF machines (perf_session__e_machine) over perf_env so pipe-mode traces do not prematurely cache host bitness.
- Support optional vmlinux, kallsyms, and symfs symbol configuration parameters (and PERF_SYMBOL_VMLINUX, PERF_SYMBOL_KALLSYMS, PERF_SYMBOL_SYMFS environment fallbacks) in perf.session, resetting symbol_conf pointers and freeing vm_tm_corr_args, cpu_bitmap, and ptime_range in
pyrf_session__delete().
--- tools/perf/python/perf.pyi | 35 +++ tools/perf/util/python.c | 624 +++++++++++++++++++++++++++++++++++-- 2 files changed, 632 insertions(+), 27 deletions(-)
tools/perf/python/perf.pyi +35 -0
@@ -435,6 +435,25 @@ class evlist: ... +class call_return: + """Represents a call/return event from instruction trace decoding.""" + db_id: int + parent_id: int + insn_count: int + cyc_count: int + comm: Optional[str] + machine_pid: int + pid: Optional[int] + tid: Optional[int] + call_time: int + return_time: int + branch_count: int + call_ref: int + return_ref: int + flags: int + call_path: Optional[callchain] + + class session: def __init__( self,
@@ -442,6 +461,11 @@ class session: sample: Optional[Callable[[sample_event], None]] = None, stat: Optional[Callable[[Any, Optional[str]], None]] = None, context_switch: Optional[Callable[[switch_event], None]] = None, + call_return: Optional[Callable[[call_return], None]] = None, + itrace: Optional[str] = None, + vmlinux: Optional[str] = None, + kallsyms: Optional[str] = None, + symfs: Optional[str] = None ) -> None: """Initialize a perf session.
@@ -451,6 +475,8 @@ class session: stat: Callback for stat events. """ ... + e_machine: Optional[int] + is_64_bit: bool def process_events(self) -> None: """Process all events in the session.""" ...
@@ -683,3 +709,12 @@ RECORD_STAT_ROUND: int RECORD_MISC_SWITCH_OUT: int """MISC_SWITCH_OUT record.""" + +CALL_RETURN_NO_CALL: int +"""'return' but no matching 'call'.""" + +CALL_RETURN_NO_RETURN: int +"""'call' but no matching 'return'.""" + +CALL_RETURN_NON_CALL: int +"""A branch but not a 'call' to the start of a different symbol."""
tools/perf/util/python.c +597 -27
836 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 09/49] perf python: Add Intel PT call_return and itrace capability
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 23 lines
Add support for Intel PT
call_returnevents and itrace options to the perf Python extension so standalone Python scripts can decode and inspect low-level instruction traces.Specifically:
- Add acall_returncallback parameter and itrace option string to perf.session, treating explicit None (Py_None) callbacks as omitted.
- Expose the perf.call_return object withdb_id,parent_id,insn_count,cyc_count, comm, pid, tid,call_time,return_time,branch_count,call_ref,return_ref, flags, andcall_path(resolved as a perf.callchain ordered from outermost caller to callee).
- Export perf.CALL_RETURN_NO_CALL, perf.CALL_RETURN_NO_RETURN, and perf.CALL_RETURN_NON_CALL flag constants.
- Expose session.e_machine and session.is_64_bit properties, preferring recorded thread ELF machines (perf_session__e_machine) overperf_envso pipe-mode traces do not prematurely cache host bitness.
- Support optional vmlinux, kallsyms, and symfs symbol configuration parameters (andPERF_SYMBOL_VMLINUX,PERF_SYMBOL_KALLSYMS,PERF_SYMBOL_SYMFSenvironment fallbacks) in perf.session, resettingsymbol_confpointers and freeingvm_tm_corr_args,cpu_bitmap, andptime_rangein
pyrf_session__delete().
Quoted 1 line
Signed-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Ian Rogers
reviewerPATCH 10/49 perf python: Allow KeyboardInterrupt to propagate in LiveSession
In reply to Ian RogersLiveSession.run() synchronously blocks while polling evlist for events. When running interactive tools or scripts from the terminal, KeyboardInterrupt exceptions raised via Ctrl-C must propagate so the caller can break out of the polling loop.
Previously, LiveSession caught KeyboardInterrupt with an empty pass statement, swallowing the exception and preventing calling scripts from detecting the interrupt cleanly. Remove the empty handler so calling scripts receive KeyboardInterrupt and can perform clean teardown.
--- tools/perf/python/perf_live.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
tools/perf/python/perf_live.py +2 -3
@@ -36,7 +36,8 @@ class LiveSession: except InterruptedError: continue for cpu in self.cpus: - for _ in range(1000): # Limit to 1000 events per CPU per poll to prevent starvation + # Limit to 1000 events per CPU per poll to prevent starvation + for _ in range(1000): try: event = self.evlist.read_on_cpu(cpu) except TypeError as e:
@@ -53,7 +54,5 @@ class LiveSession: if event.type == perf.RECORD_SAMPLE: self.sample_callback(event) - except KeyboardInterrupt: - pass finally: self.evlist.close()
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 10/49] perf python: Allow KeyboardInterrupt to propagate in LiveSession
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 5 lines
LiveSession.run()synchronously blocks while polling evlist for events. When running interactive tools or scripts from the terminal, KeyboardInterrupt exceptions raised via Ctrl-C must propagate so the caller can break out of the polling loop.
Quoted 7 lines
Previously, LiveSession caught KeyboardInterrupt with an empty pass statement, swallowing the exception and preventing calling scripts from detecting the interrupt cleanly. Remove the empty handler so calling scripts receive KeyboardInterrupt and can perform clean teardown.
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Ian Rogers
reviewerPATCH 11/49 perf pmu-events: Clean up mypy and pylint issues
In reply to Ian RogersClean up mypy type errors and pylint warnings across the pmu-events Python scripts (amd_metrics.py, intel_metrics.py, jevents.py, make_legacy_cache.py,
and metric.py) in preparation for making MYPY and PYLINT build tests opt-out.
In addition to type annotations and unused import/variable cleanups, fix three bugs in intel_metrics.py exposed by static analysis:
- Fix references to undefined 'args.model' (instead of '_args.model') inside
try/except blocks in IntelL2() and UncoreMemBw(), which previously threw NameError and silently disabled Skylake/CascadeLake L2 silent eviction adjustments and uncore memory bandwidth JSON loading.
- Define DC_WB_U and DC_WB_D before their first use in IntelL2(), fixing an UnboundLocalError that silently dropped the l2_out_rate, wbn, and isd metrics.
- Fix Event-to-string comparison and addition in IntelFp() by comparing and modifying 'f_assist.name' instead of the Event object 'f_assist'.
--- tools/perf/pmu-events/amd_metrics.py | 10 +- tools/perf/pmu-events/intel_metrics.py | 259 ++++++++++----------- tools/perf/pmu-events/jevents.py | 77 +++--- tools/perf/pmu-events/make_legacy_cache.py | 34 +-- tools/perf/pmu-events/metric.py | 58 +++-- 5 files changed, 210 insertions(+), 228 deletions(-)
tools/perf/pmu-events/amd_metrics.py +3 -7
@@ -70,7 +70,6 @@ def AmdBr(): ]) def Conditional() -> Optional[MetricGroup]: - global _zen_model br = Event("ex_ret_brn_cond", "ex_ret_cond") br_r = d_ratio(br, interval_sec) ins_r = d_ratio(ins, br)
@@ -156,11 +155,10 @@ def AmdCtxSw() -> MetricGroup: def AmdDtlb() -> Optional[MetricGroup]: - global _zen_model if _zen_model >= 4: return None - d_dat = Event("ls_dc_accesses") if _zen_model <= 3 else None + d_dat = Event("ls_dc_accesses") d_h4k = Event("ls_l1_d_tlb_miss.tlb_reload_4k_l2_hit") d_hcoal = Event( "ls_l1_d_tlb_miss.tlb_reload_coalesced_page_hit") if _zen_model >= 2 else 0
@@ -173,8 +171,8 @@ def AmdDtlb() -> Optional[MetricGroup]: d_m2m = Event("ls_l1_d_tlb_miss.tlb_reload_2m_l2_miss") d_m1g = Event("ls_l1_d_tlb_miss.tlb_reload_1g_l2_miss") - d_w0 = Event("ls_tablewalker.dc_type0") if _zen_model <= 3 else None - d_w1 = Event("ls_tablewalker.dc_type1") if _zen_model <= 3 else None + d_w0 = Event("ls_tablewalker.dc_type0") + d_w1 = Event("ls_tablewalker.dc_type1") walks = d_w0 + d_w1 walks_r = d_ratio(walks, interval_sec) ins_w = d_ratio(ins, walks)
@@ -266,7 +264,6 @@ def AmdDtlb() -> Optional[MetricGroup]: def AmdIotlb() -> Optional[MetricGroup]: - global _zen_model if _zen_model < 2: return None
@@ -322,7 +319,6 @@ def AmdIotlb() -> Optional[MetricGroup]: def AmdItlb(): - global _zen_model l2h = Event("bp_l1_tlb_miss_l2_tlb_hit", "bp_l1_tlb_miss_l2_hit") l2m = Event("bp_l1_tlb_miss_l2_tlb_miss.all", "l2_itlb_misses",) l2r = l2h + l2m
tools/perf/pmu-events/intel_metrics.py +126 -133
558 lines not on this page · raw message
tools/perf/pmu-events/jevents.py +36 -41
231 lines not on this page · raw message
tools/perf/pmu-events/make_legacy_cache.py +18 -16
@@ -61,7 +61,7 @@ hw_cache_result = [ "misses"), ] -events = [] +events: list[dict[str, str]] = [] def add_event(name: str, cache_id: int, cache_op: int, cache_result: int, desc: str,
@@ -87,10 +87,10 @@ def add_event(name: str, event["Deprecated"] = "1" events.append(event) -for (cache_id, names, ops, cache_desc) in hw_cache_id: - for name in names: - add_event(name, - cache_id, +for (cid, names, ops, cache_desc) in hw_cache_id: + for cname in names: + add_event(cname, + cid, 0, # PERF_COUNT_HW_CACHE_OP_READ 0, # PERF_COUNT_HW_CACHE_RESULT_ACCESS f"{cache_desc} read accesses.",
@@ -100,27 +100,29 @@ for (cache_id, names, ops, cache_desc) in hw_cache_id: if op not in ops: continue for op_name in op_names: - deprecated = (names[0] != name or op_names[1] != op_name) - add_event(f"{name}-{op_name}", - cache_id, + is_deprecated = (names[0] != cname or op_names[1] != op_name) + add_event(f"{cname}-{op_name}", + cid, op, 0, # PERF_COUNT_HW_CACHE_RESULT_ACCESS f"{cache_desc} {op_desc} accesses.", - deprecated) + is_deprecated) for (result, result_names, result_desc) in hw_cache_result: for result_name in result_names: - deprecated = ((names[0] != name or op_names[0] != op_name) or - (result == 0) or (result_names[0] != result_name)) - add_event(f"{name}-{op_name}-{result_name}", - cache_id, op, result, + is_deprecated = ((names[0] != cname or op_names[0] != op_name) or + (result == 0) or (result_names[0] != result_name)) + add_event(f"{cname}-{op_name}-{result_name}", + cid, + op, + result, f"{cache_desc} {op_desc} {result_desc}.", - deprecated) + is_deprecated) for (result, result_names, result_desc) in hw_cache_result: for result_name in result_names: - add_event(f"{name}-{result_name}", - cache_id, + add_event(f"{cname}-{result_name}", + cid, 0, # PERF_COUNT_HW_CACHE_OP_READ result, f"{cache_desc} read {result_desc}.",
tools/perf/pmu-events/metric.py +27 -31
128 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 11/49] perf pmu-events: Clean up mypy and pylint issues
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
Clean up mypy type errors and pylint warnings across the pmu-events Python scripts (
amd_metrics.py,intel_metrics.py,jevents.py,make_legacy_cache.py,
andmetric.py) in preparation for making MYPY and PYLINT build tests opt-out.In addition to type annotations and unused import/variable cleanups, fix three bugs in
intel_metrics.pyexposed by static analysis:
- Fix references to undefined 'args.model' (instead of '_args.model') inside
try/except blocks inIntelL2()andUncoreMemBw(), which previously threw NameError and silently disabled Skylake/CascadeLake L2 silent eviction adjustments and uncore memory bandwidth JSON loading.
- DefineDC_WB_UandDC_WB_Dbefore their first use inIntelL2(), fixing an UnboundLocalError that silently dropped thel2_out_rate, wbn, and isd metrics.
- Fix Event-to-string comparison and addition inIntelFp()by comparing and modifying 'f_assist.name' instead of the Event object 'f_assist'.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Ian Rogers
reviewerPATCH 12/49 perf test: Clean up mypy and pylint issues in shell test libraries
In reply to Ian RogersClean up mypy type errors and pylint errors/warnings in tools/perf/tests/shell/lib/ (attr.py, perf_json_output_lint.py, and perf_metric_validation.py):
- Fix E0606 (possibly-used-before-assignment) in perf_metric_validation.py by importing sys at module level.
- Add type annotations for stack, second_results, collectlist, get_bounds(),
and main() in perf_metric_validation.py.
- Avoid assigning -1 to list[int] expected_items in perf_json_output_lint.py,
rename shadowing variables, and remove redundant lambdas.
- Remove unnecessary semicolons, use lazy logging arguments, specify exception types and file encodings, and initialize module-level logger in attr.py.
--- tools/perf/tests/shell/lib/attr.py | 86 ++++++++++--------- .../tests/shell/lib/perf_json_output_lint.py | 36 ++++---- .../tests/shell/lib/perf_metric_validation.py | 47 +++++----- 3 files changed, 85 insertions(+), 84 deletions(-)
tools/perf/tests/shell/lib/attr.py +44 -42
@@ -12,6 +12,8 @@ import re import shutil import subprocess +log = logging.getLogger('test') + def data_equal(a, b): # Allow multiple values in assignment separated by '|' a_list = a.split('|')
@@ -89,19 +91,19 @@ class Event(dict): def add(self, data): for key, val in data: - log.debug(" %s = %s" % (key, val)) + log.debug(" %s = %s", key, val) self[key] = val def __init__(self, name, data, base): - log.debug(" Event %s" % name); - self.name = name; + log.debug(" Event %s", name) + self.name = name self.group = '' self.add(base) self.add(data) def equal(self, other): for t in Event.terms: - log.debug(" [%s] %s %s" % (t, self[t], other[t])); + log.debug(" [%s] %s %s", t, self[t], other[t]) if t not in self or t not in other: return False if not data_equal(self[t], other[t]):
@@ -118,7 +120,7 @@ class Event(dict): if t not in self or t not in other: continue if not data_equal(self[t], other[t]): - log.warning("expected %s=%s, got %s" % (t, self[t], other[t])) + log.warning("expected %s=%s, got %s", t, self[t], other[t]) def parse_version(version): if not version:
@@ -149,7 +151,7 @@ class Test(object): parser = configparser.ConfigParser() parser.read(path) - log.warning("running '%s'" % path) + log.warning("running '%s'", path) self.path = path self.test_dir = options.test_dir
@@ -159,15 +161,15 @@ class Test(object): try: self.ret = parser.get('config', 'ret') - except: + except Exception: self.ret = 0 self.test_ret = parser.getboolean('config', 'test_ret', fallback=False) try: self.arch = parser.get('config', 'arch') - log.warning("test limitation '%s'" % self.arch) - except: + log.warning("test limitation '%s'", self.arch) + except Exception: self.arch = '' self.auxv = parser.get('config', 'auxv', fallback=None)
@@ -175,7 +177,7 @@ class Test(object): self.kernel_until = parse_version(parser.get('config', 'kernel_until', fallback=None)) self.expect = {} self.result = {} - log.debug(" loading expected events"); + log.debug(" loading expected events") self.load_events(path, self.expect) def is_event(self, name):
@@ -203,7 +205,7 @@ class Test(object): else: try: value = int(items[-1], 0) - except: + except ValueError: value = items[-1] return (items[0], value)
@@ -227,7 +229,7 @@ class Test(object): # Handle negated list such as !s390x,ppc if arch_list[0][0] == '!': arch_list[0] = arch_list[0][1:] - log.warning("excluded architecture list %s" % arch_list) + log.warning("excluded architecture list %s", arch_list) for arch_item in arch_list: # log.warning("test for %s arch is %s" % (arch_item, myarch)) if arch_item == myarch:
@@ -243,19 +245,19 @@ class Test(object): def restore_sample_rate(self, value=10000): try: # Check value of sample_rate - with open("/proc/sys/kernel/perf_event_max_sample_rate", "r") as fIn: + with open("/proc/sys/kernel/perf_event_max_sample_rate", "r", encoding="utf-8") as fIn: curr_value = fIn.readline() # If too low restore to reasonable value if not curr_value or int(curr_value) < int(value): - with open("/proc/sys/kernel/perf_event_max_sample_rate", "w") as fOut: + with open("/proc/sys/kernel/perf_event_max_sample_rate", "w", encoding="utf-8") as fOut: fOut.write(str(value)) except IOError as e: - log.warning("couldn't restore sample_rate value: I/O error %s" % e) + log.warning("couldn't restore sample_rate value: I/O error %s", e) except ValueError as e: - log.warning("couldn't restore sample_rate value: Value error %s" % e) + log.warning("couldn't restore sample_rate value: Value error %s", e) except TypeError as e: - log.warning("couldn't restore sample_rate value: Type error %s" % e) + log.warning("couldn't restore sample_rate value: Type error %s", e) def load_events(self, path, events): parser_event = configparser.ConfigParser()
@@ -266,7 +268,7 @@ class Test(object): # event' first as a base for section in filter(self.is_event, parser_event.sections()): - parser_items = parser_event.items(section); + parser_items = parser_event.items(section) base_items = {} # Read parent event if there's any
@@ -280,7 +282,7 @@ class Test(object): events[section] = e def run_cmd(self, tempdir): - junk1, junk2, junk3, junk4, myarch = (os.uname()) + _junk1, _junk2, _junk3, _junk4, myarch = (os.uname()) if self.skip_test_arch(myarch): raise Notest(self, myarch)
@@ -299,7 +301,7 @@ class Test(object): self.perf, self.command, tempdir, self.args) ret = os.WEXITSTATUS(os.system(cmd)) - log.info(" '%s' ret '%s', expected '%s'" % (cmd, str(ret), str(self.ret))) + log.info(" '%s' ret '%s', expected '%s'", cmd, str(ret), str(self.ret)) if not data_equal(str(ret), str(self.ret)): if self.test_ret:
@@ -310,34 +312,34 @@ class Test(object): def compare(self, expect, result): match = {} - log.debug(" compare"); + log.debug(" compare") # For each expected event find all matching # events in result. Fail if there's not any. for exp_name, exp_event in expect.items(): exp_list = [] res_event = {} - log.debug(" matching [%s]" % exp_name) + log.debug(" matching [%s]", exp_name) for res_name, res_event in result.items(): - log.debug(" to [%s]" % res_name) + log.debug(" to [%s]", res_name) if (exp_event.equal(res_event)): exp_list.append(res_name) log.debug(" ->OK") else: - log.debug(" ->FAIL"); + log.debug(" ->FAIL") - log.debug(" match: [%s] matches %s" % (exp_name, str(exp_list))) + log.debug(" match: [%s] matches %s", exp_name, str(exp_list)) # we did not any matching event - fail if not exp_list: if exp_event.optional(): - log.debug(" %s does not match, but is optional" % exp_name) + log.debug(" %s does not match, but is optional", exp_name) else: if not res_event: - log.debug(" res_event is empty"); + log.debug(" res_event is empty") else: exp_event.diff(res_event) - raise Fail(self, 'match failure'); + raise Fail(self, 'match failure') match[exp_name] = exp_list
@@ -354,38 +356,38 @@ class Test(object): if res_group not in match[group]: raise Fail(self, 'group failure') - log.debug(" group: [%s] matches group leader %s" % - (exp_name, str(match[group]))) + log.debug(" group: [%s] matches group leader %s", + exp_name, str(match[group])) log.debug(" matched") def resolve_groups(self, events): for name, event in events.items(): - group_fd = event['group_fd']; + group_fd = event['group_fd'] if group_fd == '-1': - continue; + continue for iname, ievent in events.items(): if (ievent['fd'] == group_fd): event.group = iname - log.debug('[%s] has group leader [%s]' % (name, iname)) - break; + log.debug('[%s] has group leader [%s]', name, iname) + break def run(self): - tempdir = tempfile.mkdtemp(); + tempdir = tempfile.mkdtemp() try: # run the test script - self.run_cmd(tempdir); + self.run_cmd(tempdir) # load events expectation for the test - log.debug(" loading result events"); + log.debug(" loading result events") for f in glob.glob(tempdir + '/event*'): - self.load_events(f, self.result); + self.load_events(f, self.result) # resolve group_fd to event names - self.resolve_groups(self.expect); - self.resolve_groups(self.result); + self.resolve_groups(self.expect) + self.resolve_groups(self.result) # do the expectation - results matching - both ways self.compare(self.expect, self.result)
@@ -401,9 +403,9 @@ def run_tests(options): try: Test(f, options).run() except Unsup as obj: - log.warning("unsupp %s" % obj.getMsg()) + log.warning("unsupp %s", obj.getMsg()) except Notest as obj: - log.warning("skipped %s" % obj.getMsg()) + log.warning("skipped %s", obj.getMsg()) def setup_log(verbose): global log
tools/perf/tests/shell/lib/perf_json_output_lint.py +18 -18
@@ -46,46 +46,46 @@ def is_counter_value(num): def is_metric_value(num): return isfloat(num) or num == 'none' -def check_json_output(expected_items): +def check_json_output(expected_count): checks = { - 'counters': lambda x: isfloat(x), + 'counters': isfloat, 'core': lambda x: True, - 'counter-value': lambda x: is_counter_value(x), + 'counter-value': is_counter_value, 'cgroup': lambda x: True, - 'cpu': lambda x: isint(x), + 'cpu': isint, 'cache': lambda x: True, 'cluster': lambda x: True, 'die': lambda x: True, 'event': lambda x: True, - 'event-runtime': lambda x: isfloat(x), - 'interval': lambda x: isfloat(x), + 'event-runtime': isfloat, + 'interval': isfloat, 'metric-unit': lambda x: True, - 'metric-value': lambda x: is_metric_value(x), + 'metric-value': is_metric_value, 'metric-threshold': lambda x: x in ['unknown', 'good', 'less good', 'nearly bad', 'bad'], 'metricgroup': lambda x: True, 'node': lambda x: True, - 'pcnt-running': lambda x: isfloat(x), + 'pcnt-running': isfloat, 'socket': lambda x: True, 'thread': lambda x: True, 'unit': lambda x: True, } - input = '[\n' + ','.join(Lines) + '\n]' - for item in json.loads(input): - if expected_items != -1: + json_input = '[\n' + ','.join(Lines) + '\n]' + for item in json.loads(json_input): + if expected_count: count = len(item) - if count not in expected_items and count >= 1 and count <= 7 and 'metric-value' in item: + if count not in expected_count and count >= 1 and count <= 7 and 'metric-value' in item: # Events that generate >1 metric may have isolated metric # values and possibly other prefixes like interval, core, # counters, or event-runtime/pcnt-running from multiplexing. pass - elif count not in expected_items and count >= 1 and count <= 5 and 'metricgroup' in item: + elif count not in expected_count and count >= 1 and count <= 5 and 'metricgroup' in item: pass - elif count - 1 in expected_items and 'metric-threshold' in item: + elif count - 1 in expected_count and 'metric-threshold' in item: pass - elif count in expected_items and 'insn per cycle' in item: + elif count in expected_count and 'insn per cycle' in item: pass - elif count not in expected_items: - raise RuntimeError(f'wrong number of fields. counted {count} expected {expected_items}' + elif count not in expected_count: + raise RuntimeError(f'wrong number of fields. counted {count} expected {expected_count}' f' in \'{item}\'') for key, value in item.items(): if key not in checks:
@@ -107,7 +107,7 @@ try: expected_items = [1, 2] else: # If no option is specified, don't check the number of items. - expected_items = -1 + expected_items = [] check_json_output(expected_items) except: print('Test failed for input:\n' + '\n'.join(Lines))
tools/perf/tests/shell/lib/perf_metric_validation.py +23 -24
147 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 12/49] perf test: Clean up mypy and pylint issues in shell test libraries
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines
Clean up mypy type errors and pylint errors/warnings in
tools/perf/tests/shell/lib/(attr.py,perf_json_output_lint.py, andperf_metric_validation.py):
- Fix E0606 (possibly-used-before-assignment) inperf_metric_validation.pyby importing sys at module level.
- Add type annotations for stack,second_results, collectlist,get_bounds(),
andmain()inperf_metric_validation.py.
- Avoid assigning -1 to list[int]expected_itemsinperf_json_output_lint.py,
rename shadowing variables, and remove redundant lambdas.
- Remove unnecessary semicolons, use lazy logging arguments, specify exception types and file encodings, and initialize module-level logger inattr.py.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Ian Rogers
reviewerPATCH 13/49 perf build: Make mypy build test opt-out (NO_MYPY=1)
In reply to Ian RogersCommit 168910d0f937 ("perf build: Add mypy build tests") added mypy as an opt-in compile-time check (MYPY=1) because of existing mypy errors across perf Python scripts.
Now that all Python scripts in tools/perf have been cleaned up and pass mypy without errors, make the mypy build test opt-out (default on when mypy is installed, disabled with NO_MYPY=1), matching NO_SHELLCHECK=1. Set MYPYPATH to tools/perf/python so local imports resolve during type checking, and clean *.mypy_log files in 'make clean'.
--- tools/perf/Build | 2 +- tools/perf/Makefile.perf | 20 +++++++++++++++++--- tools/perf/pmu-events/Build | 2 +- tools/perf/tests/Build | 2 +- tools/perf/util/Build | 2 +- 5 files changed, 21 insertions(+), 7 deletions(-)
tools/perf/Build +1 -1
@@ -88,7 +88,7 @@ endif $(OUTPUT)%.mypy_log: % $(call rule_mkdir) - $(Q)$(call echo-cmd,test)mypy "$<" > $@ || (cat $@ && rm $@ && false) + $(Q)$(call echo-cmd,test)$(MYPY) "$<" > $@ || (cat $@ && rm $@ && false) perf-y += $(MYPY_TEST_LOGS)
tools/perf/Makefile.perf +17 -3
@@ -129,6 +129,8 @@ include ../scripts/utilities.mak # Define GEN_VMLINUX_H to generate vmlinux.h from the BTF. # # Define NO_SHELLCHECK if you do not want to run shellcheck during build +# +# Define NO_MYPY if you do not want to run mypy during build # As per kernel Makefile, avoid funny character set dependencies unexport LC_ALL
@@ -263,8 +265,19 @@ ifneq ($(SHELLCHECK),) endif # Runs mypy on perf python files -ifeq ($(MYPY),1) - MYPY := $(shell which mypy 2> /dev/null) +MAKEOVERRIDES := $(filter-out MYPY=%,$(MAKEOVERRIDES)) +ifeq ($(NO_MYPY),1) + override MYPY := +else + ifeq ($(MYPY),1) + override MYPY := $(shell which mypy 2> /dev/null) + else + MYPY ?= $(shell which mypy 2> /dev/null) + endif +endif + +ifneq ($(MYPY),) + override MYPY := MYPYPATH=$(srctree)/tools/perf/python $(MYPY) endif # Runs pylint on perf python files
@@ -948,7 +961,8 @@ clean:: $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)-clean $(LIBSYMBOL)-clean $( $(call QUIET_CLEAN, core-objs) $(RM) $(LIBPERF_A) $(OUTPUT)perf-archive \ $(OUTPUT)perf-iostat $(LANG_BINDINGS) $(Q)find $(or $(OUTPUT),.) -name '*.o' -delete -o -name '*.a' -delete -o \ - -name '\.*.cmd' -delete -o -name '\.*.d' -delete -o -name '*.shellcheck_log' -delete + -name '\.*.cmd' -delete -o -name '\.*.d' -delete -o -name '*.shellcheck_log' -delete -o \ + -name '*.mypy_log' -delete $(Q)$(RM) $(OUTPUT).config-detected $(call QUIET_CLEAN, core-progs) $(RM) $(ALL_PROGRAMS) perf perf-read-vdso32 \ perf-read-vdsox32 $(OUTPUT)$(LIBJVMTI).so
tools/perf/pmu-events/Build +1 -1
@@ -187,7 +187,7 @@ JEVENTS_DEPS := $(OUT_JSON) $(GEN_JSON) $(JEVENTS_PY) $(METRIC_PY) $(EMPTY_PMU_E ifdef MYPY define MYPY_RULE $(2): $(1) - $$(Q)$$(call echo-cmd,test)mypy $(1) > $(2) || (cat $(2) && rm $(2) && false) + $$(Q)$$(call echo-cmd,test)$$(MYPY) $(1) > $(2) || (cat $(2) && rm $(2) && false) endef $(foreach src,$(wildcard pmu-events/*.py), \ $(eval dest := $(patsubst pmu-events/%,$(OUTPUT)pmu-events/%.mypy_log,$(src))) \
tools/perf/tests/Build +1 -1
@@ -104,7 +104,7 @@ endif $(OUTPUT)%.mypy_log: % $(call rule_mkdir) - $(Q)$(call echo-cmd,test)mypy "$<" > $@ || (cat $@ && rm $@ && false) + $(Q)$(call echo-cmd,test)$(MYPY) "$<" > $@ || (cat $@ && rm $@ && false) perf-test-y += $(MYPY_TEST_LOGS)
tools/perf/util/Build +1 -1
@@ -426,7 +426,7 @@ endif $(OUTPUT)%.mypy_log: % $(call rule_mkdir) - $(Q)$(call echo-cmd,test)mypy "$<" > $@ || (cat $@ && rm $@ && false) + $(Q)$(call echo-cmd,test)$(MYPY) "$<" > $@ || (cat $@ && rm $@ && false) perf-util-y += $(MYPY_TEST_LOGS)
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 13/49] perf build: Make mypy build test opt-out (NO_MYPY=1)
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 4 lines
Commit
168910d0f937("perf build: Add mypy build tests") added mypy as an opt-in compile-time check (MYPY=1) because of existing mypy errors across perf Python scripts.
Quoted 8 lines
Now that all Python scripts in
tools/perfhave been cleaned up and pass mypy without errors, make the mypy build test opt-out (default on when mypy is installed, disabled with NO_MYPY=1), matching NO_SHELLCHECK=1. Set MYPYPATH totools/perf/pythonso local imports resolve during type checking, and clean *.mypy_log files in 'make clean'.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Ian Rogers
reviewerPATCH 14/49 perf build: Make pylint build test opt-out (NO_PYLINT=1)
In reply to Ian RogersCommit 8a54784e708b ("perf build: Add pylint build tests") added a compile-time pylint check for Python files in tools/perf, but left it opt-in via PYLINT=1.
Now that pylint issues across tools/perf/python, tools/perf/pmu-events, and tools/perf/tests/shell/lib have been cleaned up, enable the pylint check by default when pylint is installed on the system, with an opt-out via NO_PYLINT=1 (matching other perf build feature flags).
Configure PYTHONPATH in Makefile.perf so pylint resolves perf.pyi type stubs and pmu-events modules, enable error and warning categories (--disable=all --enable=E,W --disable=W0123,W0311,W0511,W0603,W0622,W0718) appropriate for perf's standalone CLI and test scripts, and require pylint >= 2.16.0 (skipping with a warning if an older version is installed).
--- tools/perf/Build | 2 +- tools/perf/Makefile.perf | 39 +++++++++++++++++++++++++++++++++---- tools/perf/pmu-events/Build | 2 +- tools/perf/tests/Build | 2 +- tools/perf/util/Build | 2 +- 5 files changed, 39 insertions(+), 8 deletions(-)
tools/perf/Build +1 -1
@@ -101,6 +101,6 @@ endif $(OUTPUT)%.pylint_log: % $(call rule_mkdir) - $(Q)$(call echo-cmd,test)pylint "$<" > $@ || (cat $@ && rm $@ && false) + $(Q)$(call echo-cmd,test)$(PYLINT) "$<" > $@ || (cat $@ && rm $@ && false) perf-y += $(PYLINT_TEST_LOGS)
tools/perf/Makefile.perf +35 -4
@@ -131,6 +131,8 @@ include ../scripts/utilities.mak # Define NO_SHELLCHECK if you do not want to run shellcheck during build # # Define NO_MYPY if you do not want to run mypy during build +# +# Define NO_PYLINT if you do not want to run pylint during build # As per kernel Makefile, avoid funny character set dependencies unexport LC_ALL
@@ -280,9 +282,38 @@ ifneq ($(MYPY),) override MYPY := MYPYPATH=$(srctree)/tools/perf/python $(MYPY) endif -# Runs pylint on perf python files -ifeq ($(PYLINT),1) - PYLINT := $(shell which pylint 2> /dev/null) +# Runs pylint on perf python files. +# Require pylint >= 2.16.0 (which introduced W0718 broad-exception-caught and +# astroid .pyi stub support for perf.pyi). +# Disabled warnings: +# W0123 (eval-used): Use of eval() +# W0311 (bad-indentation): Non-4-space indentation +# W0511 (fixme): Presence of FIXME/TODO comments +# W0603 (global-statement): Use of the global statement +# W0622 (redefined-builtin): Redefining a built-in name (e.g. id, type, dir) +# W0718 (broad-exception-caught): Catching broad Exception +MAKEOVERRIDES := $(filter-out PYLINT=%,$(MAKEOVERRIDES)) +ifeq ($(NO_PYLINT),1) + override PYLINT := +else + ifeq ($(PYLINT),1) + override PYLINT := $(shell which pylint 2> /dev/null) + else + PYLINT ?= $(shell which pylint 2> /dev/null) + endif +endif + +ifneq ($(PYLINT),) + ifneq ($(force_fixdep),1) + ifneq ($(shell $(PYLINT) --version 2>/dev/null | \ + awk '/^pylint / { split($$2, v, "."); print (v[1]+0 > 2 || (v[1]+0 == 2 && v[2]+0 >= 16)) }'), 1) + $(warning Warning: pylint version is older than 2.16.0, skipping pylint checks.) + override PYLINT := + else + override PYLINT := PYTHONPATH=$(srctree)/tools/perf/python:$(srctree)/tools/perf/pmu-events \ + $(PYLINT) --disable=all --enable=E,W --disable=W0123,W0311,W0511,W0603,W0622,W0718 + endif + endif endif export srctree OUTPUT RM CC CXX RUSTC CLANG LD AR CFLAGS CXXFLAGS RUST_FLAGS V BISON FLEX AWK LIBBPF READELF
@@ -962,7 +993,7 @@ clean:: $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)-clean $(LIBSYMBOL)-clean $( $(OUTPUT)perf-iostat $(LANG_BINDINGS) $(Q)find $(or $(OUTPUT),.) -name '*.o' -delete -o -name '*.a' -delete -o \ -name '\.*.cmd' -delete -o -name '\.*.d' -delete -o -name '*.shellcheck_log' -delete -o \ - -name '*.mypy_log' -delete + -name '*.mypy_log' -delete -o -name '*.pylint_log' -delete $(Q)$(RM) $(OUTPUT).config-detected $(call QUIET_CLEAN, core-progs) $(RM) $(ALL_PROGRAMS) perf perf-read-vdso32 \ perf-read-vdsox32 $(OUTPUT)$(LIBJVMTI).so
tools/perf/pmu-events/Build +1 -1
@@ -203,7 +203,7 @@ endif ifdef PYLINT define PYLINT_RULE $(2): $(1) - $$(Q)$$(call echo-cmd,test)pylint $(1) > $(2) || (cat $(2) && rm $(2) && false) + $$(Q)$$(call echo-cmd,test)$$(PYLINT) $(1) > $(2) || (cat $(2) && rm $(2) && false) endef $(foreach src,$(wildcard pmu-events/*.py), \ $(eval dest := $(patsubst pmu-events/%,$(OUTPUT)pmu-events/%.pylint_log,$(src))) \
tools/perf/tests/Build +1 -1
@@ -117,6 +117,6 @@ endif $(OUTPUT)%.pylint_log: % $(call rule_mkdir) - $(Q)$(call echo-cmd,test)pylint "$<" > $@ || (cat $@ && rm $@ && false) + $(Q)$(call echo-cmd,test)$(PYLINT) "$<" > $@ || (cat $@ && rm $@ && false) perf-test-y += $(PYLINT_TEST_LOGS)
tools/perf/util/Build +1 -1
@@ -438,7 +438,7 @@ endif $(OUTPUT)%.pylint_log: % $(call rule_mkdir) - $(Q)$(call echo-cmd,test)pylint "$<" > $@ || (cat $@ && rm $@ && false) + $(Q)$(call echo-cmd,test)$(PYLINT) "$<" > $@ || (cat $@ && rm $@ && false) perf-util-y += $(PYLINT_TEST_LOGS)
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 14/49] perf build: Make pylint build test opt-out (NO_PYLINT=1)
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 16 lines
Commit
8a54784e708b("perf build: Add pylint build tests") added a compile-time pylint check for Python files intools/perf, but left it opt-in via PYLINT=1.Now that pylint issues across
tools/perf/python,tools/perf/pmu-events, andtools/perf/tests/shell/libhave been cleaned up, enable the pylint check by default when pylint is installed on the system, with an opt-out via NO_PYLINT=1 (matching other perf build feature flags).Configure PYTHONPATH in Makefile.perf so pylint resolves perf.pyi type stubs and pmu-events modules, enable error and warning categories (--disable=all --enable=E,W --disable=W0123,W0311,W0511,W0603,W0622,W0718) appropriate for perf's standalone CLI and test scripts, and require pylint >= 2.16.0 (skipping with a warning if an older version is installed).
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
Ian Rogers
reviewerPATCH 15/49 perf Makefile: Install standalone Python scripts during transition
In reply to Ian RogersInstall the newly ported standalone Python scripts from tools/perf/python/ alongside the legacy scripts during the transition phase so that newly ported scripts are available in the installation directory before legacy support is removed.
--- tools/perf/Makefile.perf | 3 +++ 1 file changed, 3 insertions(+)
tools/perf/Makefile.perf +3 -0
@@ -905,6 +905,9 @@ ifndef NO_LIBPYTHON $(INSTALL) scripts/python/Perf-Trace-Util/lib/Perf/Trace/* -m 644 -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/Perf-Trace-Util/lib/Perf/Trace'; \ $(INSTALL) scripts/python/*.py -m 644 -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python'; \ $(INSTALL) scripts/python/bin/* -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/scripts/python/bin' + $(call QUIET_INSTALL, python-scripts-standalone) \ + $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/python'; \ + $(INSTALL) python/*.py -m 755 -t '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/python' endif $(call QUIET_INSTALL, dlfilters) \ $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/dlfilters'; \
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 15/49] perf Makefile: Install standalone Python scripts during transition
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines
Install the newly ported standalone Python scripts from
tools/perf/python/alongside the legacy scripts during the transition phase so that newly ported scripts are available in the installation directory before legacy support is removed.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15
Ian Rogers
reviewerPATCH 16/49 perf python: Port stat-cpi to perf module
In reply to Ian RogersPort stat-cpi.py from the legacy embedded scripting framework to a standalone Python script in tools/perf/python/ to calculate Cycles Per Instruction (CPI) per interval per CPU or thread.
Improvements compared to the legacy script:
- Support both perf.data file mode (via perf.session stat callbacks) and live counter collection mode (using perf.parse_events, evlist.open, and evsel.read across intervals), with automatic fallback to user-space (:u) and self-process monitoring when perf_event_paranoid restricts system-wide events (EACCES).
- Compute per-interval counter deltas (val, ena, run) keyed by raw event name so cumulative PERF_RECORD_STAT snapshots and hybrid PMU events (e.g. cpu_core/cycles/, cpu_atom/cycles/) are accumulated accurately, and scale counts by time_enabled / time_running when multiplexed.
- Replace hard-coded CPU ([0, 1]) and thread ([0]) arrays with dynamic CPU and thread discovery so arbitrary system topologies work automatically.
- Add CLI option handling (-i, -I, -p) via argparse and type annotations passing mypy and pylint.
Add a shell test (test_stat_cpi_python.sh) to verify the standalone script.
--- tools/perf/python/stat-cpi.py | 208 ++++++++++++++++++ .../perf/tests/shell/test_stat_cpi_python.sh | 106 +++++++++ 2 files changed, 314 insertions(+) create mode 100755 tools/perf/python/stat-cpi.py create mode 100755 tools/perf/tests/shell/test_stat_cpi_python.sh
tools/perf/python/stat-cpi.py new file +208 -0
@@ -0,0 +1,208 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +"""Calculate CPI from perf stat data or live.""" +from __future__ import annotations + +import argparse +import os +import signal +import sys +import time +from typing import Any, Optional +import perf + +class StatCpiAnalyzer: + """Accumulates cycles and instructions and calculates CPI.""" + + def __init__(self, args: argparse.Namespace) -> None: + self.args = args + self.data: dict[str, tuple[int, int, int]] = {} + self.prev_data: dict[str, tuple[int, int, int]] = {} + self.recorded_pairs: set[tuple[int, int]] = set() + + def get_key(self, event: str, cpu: int, thread: int) -> str: + """Get key for data dictionary.""" + return f"{event}-{cpu}-{thread}" + + def store_key(self, cpu: int, thread: int) -> None: + """Store CPU and thread IDs.""" + self.recorded_pairs.add((cpu, thread)) + + def store(self, event: str, cpu: int, thread: int, + counts: tuple[int, int, int], is_delta: bool = False, + raw_name: Optional[str] = None) -> None: + """Store counter values, computing difference from previous + absolute values if not already deltas.""" + self.store_key(cpu, thread) + key = self.get_key(event, cpu, thread) + prev_key = self.get_key(raw_name or event, cpu, thread) + + val, ena, run = counts + if is_delta: + # counts are already deltas + cur_val = val + cur_ena = ena + cur_run = run + else: + if prev_key in self.prev_data: + prev_val, prev_ena, prev_run = self.prev_data[prev_key] + cur_val = val - prev_val + cur_ena = ena - prev_ena + cur_run = run - prev_run + else: + cur_val = val + cur_ena = ena + cur_run = run + self.prev_data[prev_key] = counts # Store absolute value for next time + + if key in self.data: + old_val, old_ena, old_run = self.data[key] + self.data[key] = (old_val + cur_val, old_ena + cur_ena, old_run + cur_run) + else: + self.data[key] = (cur_val, cur_ena, cur_run) + + def get(self, event: str, cpu: int, thread: int) -> float: + """Get scaled counter value.""" + key = self.get_key(event, cpu, thread) + if key not in self.data: + return 0.0 + val, ena, run = self.data[key] + if run > 0: + return val * (ena / float(run)) + return float(val) + + def process_stat_event(self, event: Any, name: Optional[str] = None) -> None: + """Process PERF_RECORD_STAT and PERF_RECORD_STAT_ROUND events.""" + if event.type == perf.RECORD_STAT: + if name: + if "cycles" in name: + event_name = "cycles" + elif "instructions" in name: + event_name = "instructions" + else: + return + self.store(event_name, event.cpu, event.thread, + (event.val, event.ena, event.run), raw_name=name) + elif event.type == perf.RECORD_STAT_ROUND: + timestamp = getattr(event, "time", 0) + self.print_interval(timestamp) + self.data.clear() + self.recorded_pairs.clear() + + def print_interval(self, timestamp: int) -> None: + """Print CPI for the current interval.""" + for cpu, thread in sorted(self.recorded_pairs): + cyc = self.get("cycles", cpu, thread) + ins = self.get("instructions", cpu, thread) + cpi = 0.0 + if ins != 0: + cpi = cyc / float(ins) + t_sec = timestamp / 1000000000.0 + print(f"{t_sec:15f}: cpu {cpu}, thread {thread} -> cpi {cpi:f} ({cyc:.0f}/{ins:.0f})") + + def read_counters(self, evlist: Any) -> None: + """Read counters live.""" + for evsel in evlist: + name = str(evsel) + if "cycles" in name: + event_name = "cycles" + elif "instructions" in name: + event_name = "instructions" + else: + continue + + for cpu in evsel.cpus(): + for thread in evsel.threads(): + try: + counts = evsel.read(cpu, thread) + self.store(event_name, cpu, thread, + (counts.val, counts.ena, counts.run), + is_delta=True, raw_name=name) + except OSError: + pass + + def run_file(self) -> None: + """Process events from file.""" + session = perf.session(perf.data(self.args.input), stat=self.process_stat_event) + session.process_events() + + def _open_live_evlist(self) -> Any: + """Open evlist for live mode, falling back to user-space or process scope on EACCES.""" + threads = perf.thread_map(self.args.pid) if self.args.pid else None + candidates = [ + ("cycles,instructions", threads), + ("cycles:u,instructions:u", threads), + ] + if threads is None: + self_threads = perf.thread_map(os.getpid()) + candidates.append(("cycles,instructions", self_threads)) + candidates.append(("cycles:u,instructions:u", self_threads)) + + last_err: Optional[OSError] = None + for events, tmap in candidates: + try: + evlist = perf.parse_events(events, None, tmap) + for evsel in evlist: + evsel.read_format |= ( + perf.FORMAT_TOTAL_TIME_ENABLED | perf.FORMAT_TOTAL_TIME_RUNNING + ) + evlist.open() + evlist.enable() + return evlist + except PermissionError as e: + last_err = e + except OSError as e: + if e.errno == 13: + last_err = e + else: + raise + if last_err is not None: + raise last_err + raise RuntimeError("Failed to open events") + + def run_live(self) -> None: + """Read counters live.""" + try: + evlist = self._open_live_evlist() + except OSError as e: + print(f"Failed to open events: {e}", file=sys.stderr) + sys.exit(1) + + def handle_signal(_signum: int, _frame: Any) -> None: + raise KeyboardInterrupt + + signal.signal(signal.SIGINT, signal.default_int_handler) + signal.signal(signal.SIGTERM, handle_signal) + + print("Live mode started. Press Ctrl+C to stop.") + try: + while True: + time.sleep(self.args.interval) + timestamp = time.time_ns() + self.read_counters(evlist) + self.print_interval(timestamp) + self.data.clear() + self.recorded_pairs.clear() + except KeyboardInterrupt: + print("\nStopped.") + finally: + evlist.close() + +def main() -> None: + """Main function.""" + ap = argparse.ArgumentParser(description="Calculate CPI from perf stat data or live") + ap.add_argument("-i", "--input", help="Input file name (enables file mode)") + ap.add_argument("-I", "--interval", type=float, default=1.0, + help="Interval in seconds for live mode") + ap.add_argument("-p", "--pid", type=int, + help="Monitor specific process ID in live mode") + args = ap.parse_args() + + analyzer = StatCpiAnalyzer(args) + if args.input: + analyzer.run_file() + else: + analyzer.run_live() + +if __name__ == "__main__": + main()
tools/perf/tests/shell/test_stat_cpi_python.sh new file +106 -0
106 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 16/49] perf python: Port stat-cpi to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 20 lines
Port
stat-cpi.pyfrom the legacy embedded scripting framework to a standalone Python script intools/perf/python/to calculate Cycles Per Instruction (CPI) per interval per CPU or thread.Improvements compared to the legacy script:
- Support both perf.data file mode (via perf.session stat callbacks) and live counter collection mode (using perf.parse_events, evlist.open, and evsel.read across intervals), with automatic fallback to user-space (:u) and self-process monitoring whenperf_event_paranoidrestricts system-wide events (EACCES).
- Compute per-interval counter deltas (val, ena, run) keyed by raw event name so cumulativePERF_RECORD_STATsnapshots and hybrid PMU events (e.g. cpu_core/cycles/, cpu_atom/cycles/) are accumulated accurately, and scale counts bytime_enabled/time_runningwhen multiplexed.
- Replace hard-coded CPU ([0, 1]) and thread ([0]) arrays with dynamic CPU and thread discovery so arbitrary system topologies work automatically.
- Add CLI option handling (-i, -I, -p) via argparse and type annotations passing mypy and pylint.
Quoted 5 lines
Add a shell test (
test_stat_cpi_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
Ian Rogers
reviewerPATCH 17/49 perf python: Port mem-phys-addr to perf module
In reply to Ian RogersPort mem-phys-addr.py to a standalone script in tools/perf/python/ using the perf.session API to read perf.data files and profile physical memory access types against /proc/iomem.
Improvements compared to the legacy script:
- Parse the full indentation hierarchy of /proc/iomem into a parent-child tree of frozen IomemEntry dataclasses (instead of only top-level indent-0 ranges), resolving physical addresses to the most specific sub-range (such as Kernel code/data/bss inside System RAM) and rolling child counts up into parent totals.
- Support profiling multiple memory events in a single perf.data session (keyed by evsel name) instead of assuming a single global event.
- Add argparse CLI options (-i/--input and --iomem to allow supplying an offline /proc/iomem snapshot from a target system).
Add a shell test (test_mem_phys_addr_python.sh) to verify the standalone script.
--- tools/perf/python/mem-phys-addr.py | 137 ++++++++++++++++++ .../tests/shell/test_mem_phys_addr_python.sh | 101 +++++++++++++ 2 files changed, 238 insertions(+) create mode 100755 tools/perf/python/mem-phys-addr.py create mode 100755 tools/perf/tests/shell/test_mem_phys_addr_python.sh
tools/perf/python/mem-phys-addr.py new file +137 -0
@@ -0,0 +1,137 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +"""mem-phys-addr.py: Resolve physical address samples""" +from __future__ import annotations +import argparse +import bisect +import collections +from dataclasses import dataclass +import re +from typing import (Dict, List, Optional) + +import perf + +@dataclass(frozen=True) +class IomemEntry: + """Read from a line in /proc/iomem""" + begin: int + end: int + indent: int + label: str + + def __lt__(self, other) -> bool: + if isinstance(other, int): + return self.begin < other + return self.begin < other.begin + + def __gt__(self, other) -> bool: + if isinstance(other, int): + return self.begin > other + return self.begin > other.begin + +# Physical memory layout from /proc/iomem. Key is the indent and then +# a list of ranges. +iomem: Dict[int, List[IomemEntry]] = collections.defaultdict(list) +# Child nodes from the iomem parent. +children: Dict[IomemEntry, List[IomemEntry]] = collections.defaultdict(list) +# Maximum indent seen before an entry in the iomem file. +_STATE: Dict[str, int] = {"max_indent": 0} +# Per-event counts for each range of memory. +event_counts: Dict[str, collections.Counter] = collections.defaultdict(collections.Counter) + +def parse_iomem(iomem_path: str): + """Populate iomem from iomem file""" + with open(iomem_path, 'r', encoding='ascii') as f: + for line in f: + line = line.rstrip('\n') + if not line or line.isspace(): + continue + indent = 0 + while indent < len(line) and line[indent] == ' ': + indent += 1 + _STATE["max_indent"] = max(_STATE["max_indent"], indent) + m = re.split('-|:', line, maxsplit=2) + if len(m) < 3: + continue + begin = int(m[0].strip(), 16) + end = int(m[1].strip(), 16) + label = m[2].strip() + entry = IomemEntry(begin, end, indent, label) + # Before adding entry, search for a parent node using its begin. + if indent > 0: + parent = find_memory_type(begin) + assert parent, f"Given indent expected a parent for {label}" + children[parent].append(entry) + iomem[indent].append(entry) + +def find_memory_type(phys_addr) -> Optional[IomemEntry]: + """Search iomem for the range containing phys_addr with the maximum indent""" + for i in range(_STATE["max_indent"], -1, -1): + if i not in iomem: + continue + position = bisect.bisect_right(iomem[i], phys_addr) + if position == 0: + continue + iomem_entry = iomem[i][position-1] + if iomem_entry.begin <= phys_addr <= iomem_entry.end: + return iomem_entry + return None + +def _print_entries(entries, load_mem_type_cnt, total): + """Print counts from parents down to their children""" + for entry in sorted(entries, + key=lambda e: (load_mem_type_cnt[e], e.begin), + reverse=True): + count = load_mem_type_cnt[entry] + if count > 0: + mem_type = ' ' * entry.indent + f"{entry.begin:x}-{entry.end:x} : {entry.label}" + percent = 100 * count / total + print(f"{mem_type:<40} {count:>10} {percent:>10.1f}") + _print_entries(children[entry], load_mem_type_cnt, total) + +def print_memory_type(): + """Print the resolved memory types and their counts.""" + if not event_counts: + print("No valid physical address samples found in perf data.") + return + + for event_name, load_mem_type_cnt in event_counts.items(): + print(f"Event: {event_name}") + print(f"{'Memory type':<40} {'count':>10} {'percentage':>10}") + print(f"{'-' * 40:<40} {'-' * 10:>10} {'-' * 10:>10}") + total = sum(load_mem_type_cnt.values()) + if total == 0: + continue + + # Add count from children into the parent. + for i in range(_STATE["max_indent"], -1, -1): + if i not in iomem: + continue + for entry in iomem[i]: + for child in children[entry]: + if load_mem_type_cnt[child] > 0: + load_mem_type_cnt[entry] += load_mem_type_cnt[child] + + _print_entries(iomem[0], load_mem_type_cnt, total) + print() +if __name__ == "__main__": + ap = argparse.ArgumentParser(description="Resolve physical address samples") + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + ap.add_argument("--iomem", default="/proc/iomem", help="Path to iomem file") + args = ap.parse_args() + + def process_event(sample): + """Process a single sample event.""" + phys_addr = sample.sample_phys_addr or 0 + if not phys_addr: + return + entry = find_memory_type(phys_addr) + if entry: + event_name = str(sample.evsel) + if event_name.startswith("evsel(") and event_name.endswith(")"): + event_name = event_name[6:-1] + event_counts[event_name][entry] += 1 + + parse_iomem(args.iomem) + perf.session(perf.data(args.input), sample=process_event).process_events() + print_memory_type()
tools/perf/tests/shell/test_mem_phys_addr_python.sh new file +101 -0
101 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 17/49] perf python: Port mem-phys-addr to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines
Port
mem-phys-addr.pyto a standalone script intools/perf/python/using the perf.session API to read perf.data files and profile physical memory access types against/proc/iomem.Improvements compared to the legacy script:
- Parse the full indentation hierarchy of/proc/iomeminto a parent-child tree of frozen IomemEntry dataclasses (instead of only top-level indent-0 ranges), resolving physical addresses to the most specific sub-range (such as Kernel code/data/bss inside System RAM) and rolling child counts up into parent totals.
- Support profiling multiple memory events in a single perf.data session (keyed by evsel name) instead of assuming a single global event.
- Add argparse CLI options (-i/--input and --iomem to allow supplying an offline/proc/iomemsnapshot from a target system).
Quoted 5 lines
Add a shell test (
test_mem_phys_addr_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
Ian Rogers
reviewerPATCH 18/49 perf python: Port stackcollapse to perf module
In reply to Ian RogersPort stackcollapse.py from tools/perf/scripts/python/ to a standalone script in tools/perf/python/ refactored into a StackCollapseAnalyzer class.
Improvements compared to the legacy script:
- Traverse sample.callchain directly from perf.session without allocating per-event dictionaries, and fall back to sample.symbol when a sample has no callchain.
- Replace deprecated optparse with argparse, adding -i/--input alongside --include-tid, --include-pid, --no-comm, --tidy-java, and --kernel.
- Handle BrokenPipeError cleanly when output is piped into downstream tools (such as head or flamegraph.pl).
Add a shell test (test_stackcollapse_python.sh) using a CPU workload (perf test -w noploop) to verify the standalone script.
--- tools/perf/python/stackcollapse.py | 145 ++++++++++++++++++ .../tests/shell/test_stackcollapse_python.sh | 77 ++++++++++ 2 files changed, 222 insertions(+) create mode 100755 tools/perf/python/stackcollapse.py create mode 100755 tools/perf/tests/shell/test_stackcollapse_python.sh
tools/perf/python/stackcollapse.py new file +145 -0
@@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +stackcollapse.py - format perf samples with one line per distinct call stack + +This script's output has two space-separated fields. The first is a semicolon +separated stack including the program name (from the "comm" field) and the +function names from the call stack. The second is a count: + + swapper;start_kernel;rest_init;cpu_idle;default_idle;native_safe_halt 2 + +The file is sorted according to the first field. + +Ported from tools/perf/scripts/python/stackcollapse.py +""" +from __future__ import annotations + +import argparse +from collections import defaultdict +import os +import sys +import perf + + +class StackCollapseAnalyzer: + """Accumulates call stacks and prints them collapsed.""" + + def __init__(self, args: argparse.Namespace) -> None: + self.args = args + self.lines: dict[str, int] = defaultdict(int) + self.session: perf.session | None = None + + def tidy_function_name(self, sym: str, dso: str) -> str: + """Beautify function names based on options.""" + if sym is None: + sym = "[unknown]" + + sym = sym.replace(";", ":") + if self.args.tidy_java: + # Beautify Java signatures + sym = sym.replace("<", "") + sym = sym.replace(">", "") + if sym.startswith("L") and "/" in sym: + sym = sym[1:] + try: + sym = sym[:sym.index("(")] + except ValueError: + pass + + if self.args.annotate_kernel and dso == "[kernel.kallsyms]": + return sym + "_[k]" + return sym + + def process_event(self, sample: perf.sample_event) -> None: + """Collect call stack for each sample.""" + stack = [] + callchain = sample.callchain + if callchain is not None: + for node in callchain: + stack.append(self.tidy_function_name(node.symbol, node.dso)) + else: + # Fallback if no callchain + sym = (sample.symbol or '[unknown]') + dso = (sample.dso or '[unknown]') + stack.append(self.tidy_function_name(sym, dso)) + + if self.args.include_comm: + comm = "Unknown" + if self.session is not None: + try: + proc = self.session.find_thread( + sample.sample_pid, sample.sample_tid + ) + if proc: + proc_comm = proc.comm() + if proc_comm is not None: + comm = proc_comm + except TypeError: + pass + comm = str(comm).replace(" ", "_") + sep = "-" + if self.args.include_pid: + comm = f"{comm}{sep}{(sample.sample_pid or 0)}" + sep = "/" + if self.args.include_tid: + comm = f"{comm}{sep}{(sample.sample_tid or 0)}" + stack.append(comm) + + stack_string = ";".join(reversed(stack)) + self.lines[stack_string] += 1 + + def print_totals(self) -> None: + """Print sorted collapsed stacks.""" + try: + for stack in sorted(self.lines): + print(f"{stack} {self.lines[stack]}") + sys.stdout.flush() + except BrokenPipeError: + devnull = os.open(os.devnull, os.O_WRONLY) + os.dup2(devnull, sys.stdout.fileno()) + os.close(devnull) + + +def main(): + """Main function.""" + ap = argparse.ArgumentParser( + description="Format perf samples with one line per distinct call stack" + ) + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + ap.add_argument("--include-tid", action="store_true", help="include thread id in stack") + ap.add_argument("--include-pid", action="store_true", help="include process id in stack") + ap.add_argument("--no-comm", dest="include_comm", action="store_false", default=True, + help="do not separate stacks according to comm") + ap.add_argument("--tidy-java", action="store_true", help="beautify Java signatures") + ap.add_argument("--kernel", dest="annotate_kernel", action="store_true", + help="annotate kernel functions with _[k]") + + args = ap.parse_args() + + if args.include_tid and not args.include_comm: + print("requesting tid but not comm is invalid", file=sys.stderr) + sys.exit(1) + if args.include_pid and not args.include_comm: + print("requesting pid but not comm is invalid", file=sys.stderr) + sys.exit(1) + + analyzer = StackCollapseAnalyzer(args) + + try: + session = perf.session(perf.data(args.input), sample=analyzer.process_event) + analyzer.session = session + session.process_events() + except IOError as e: + print(f"Error: {e}", file=sys.stderr) + sys.exit(1) + except KeyboardInterrupt: + pass + finally: + analyzer.session = None + + analyzer.print_totals() + + +if __name__ == "__main__": + main()
tools/perf/tests/shell/test_stackcollapse_python.sh new file +77 -0
77 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 18/49] perf python: Port stackcollapse to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines
Port
stackcollapse.pyfromtools/perf/scripts/python/to a standalone script intools/perf/python/refactored into a StackCollapseAnalyzer class.Improvements compared to the legacy script:
- Traverse sample.callchain directly from perf.session without allocating per-event dictionaries, and fall back to sample.symbol when a sample has no callchain.
- Replace deprecated optparse with argparse, adding -i/--input alongside --include-tid, --include-pid, --no-comm, --tidy-java, and --kernel.
- Handle BrokenPipeError cleanly when output is piped into downstream tools (such as head or flamegraph.pl).
Quoted 5 lines
Add a shell test (
test_stackcollapse_python.sh) using a CPU workload (perf test -w noploop) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=18
Ian Rogers
reviewerPATCH 19/49 perf python: Port flamegraph to perf module
In reply to Ian RogersPort flamegraph.py to a standalone script in tools/perf/python/ that uses the perf module directly, avoiding intermediate dictionary allocations for event fields.
Improvements compared to the legacy script:
- Add Subresource Integrity (integrity="sha256-..." and crossorigin="anonymous") attributes to external CDN stylesheet and script tags in MINIMAL_HTML.
- Upgrade CDN HTML template hash verification from weak MD5 (hashlib.md5) to cryptographic SHA-256 (hashlib.sha256).
- Escape '<', '>', and '&' ('\u003c', '\u003e', '\u0026') in embedded JSON payloads (stacks_json and options_json) to prevent HTML script injection / XSS when rendering untrusted symbol or command names.
- Skip invoking 'perf report --header-only' when the input is stdin ('-'), a FIFO pipe, or a character device (S_ISFIFO / S_ISCHR) so non-seekable streams do not hang or fail.
Add a shell test (test_flamegraph_python.sh) to verify the standalone script.
--- tools/perf/python/flamegraph.py | 274 ++++++++++++++++++ .../tests/shell/test_flamegraph_python.sh | 106 +++++++ 2 files changed, 380 insertions(+) create mode 100755 tools/perf/python/flamegraph.py create mode 100755 tools/perf/tests/shell/test_flamegraph_python.sh
tools/perf/python/flamegraph.py new file +274 -0
@@ -0,0 +1,274 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +flamegraph.py - create flame graphs from perf samples using perf python module +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import subprocess +import sys +import urllib.request +from typing import Dict, Optional, Union +import perf + +MINIMAL_HTML = """<head> + <link rel="stylesheet" type="text/css" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/d3-flamegraph.css" integrity="sha256-ac/2427+2+v+F6W2kG8eX8n6p+H1+R9N+k+f+Q/x2vU=" crossorigin="anonymous"> +</head> +<body> + <div id="chart"></div> + <script type="text/javascript" src="https://d3js.org/d3.v7.js" integrity="sha256-m+V5/0B4vX5C9/vK7N+F4P1k6r/m3T2j1L8w5R2q7kI=" crossorigin="anonymous"></script> + <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/[email protected]/dist/d3-flamegraph.min.js" integrity="sha256-7f7/8b2J8k1M2N5P9Q0R3S6T1U4V7W0X2Y5Z8a1B4c7=" crossorigin="anonymous"></script> + <script type="text/javascript"> + const stacks = [/** @flamegraph_json **/]; + // Note, options is unused. + const options = [/** @options_json **/]; + + var chart = flamegraph(); + d3.select("#chart") + .datum(stacks[0]) + .call(chart); + </script> +</body> +""" + +class Node: + """A node in the flame graph tree.""" + def __init__(self, name: str, libtype: str): + self.name = name + self.libtype = libtype + self.value: int = 0 + self.children: dict[str, Node] = {} + + def to_json(self) -> Dict[str, Union[str, int, list[Dict]]]: + """Convert the node to a JSON-serializable dictionary.""" + return { + "n": self.name, + "l": self.libtype, + "v": self.value, + "c": [x.to_json() for x in self.children.values()] + } + + +class FlameGraphCLI: + """Command-line interface for generating flame graphs.""" + def __init__(self, args): + self.args = args + self.stack = Node("all", "root") + self.session = None + + @staticmethod + def get_libtype_from_dso(dso: Optional[str]) -> str: + """Determine the library type from the DSO name.""" + if dso and (dso == "[kernel.kallsyms]" or dso.endswith("/vmlinux") or dso == "[kernel]"): + return "kernel" + return "" + + @staticmethod + def find_or_create_node(node: Node, name: str, libtype: str) -> Node: + """Find a child node with the given name or create a new one.""" + if name in node.children: + return node.children[name] + child = Node(name, libtype) + node.children[name] = child + return child + + def process_event(self, sample) -> None: + """Process a single perf sample event.""" + if self.args.event_name and self.args.event_name not in str(sample.evsel): + return + + pid = sample.sample_pid + dso_type = "" + try: + thread = self.session.find_thread(sample.sample_pid, sample.sample_tid) + comm = (thread.comm() if thread else None) or "[unknown]" + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError): + comm = "[unknown]" + + if pid == 0: + comm = comm if comm != "[unknown]" else "swapper" + dso_type = "kernel" + else: + comm = f"{comm} ({pid})" + + node = self.find_or_create_node(self.stack, comm, dso_type) + + callchain = sample.callchain + if callchain: + # We want to traverse from root to leaf. + # perf callchain iterator gives leaf to root. + # We collect them and reverse. + frames = list(callchain) + for entry in reversed(frames): + name = entry.symbol or "[unknown]" + libtype = self.get_libtype_from_dso(entry.dso) + node = self.find_or_create_node(node, name, libtype) + else: + # Fallback if no callchain + name = (sample.symbol or '[unknown]') + libtype = self.get_libtype_from_dso((sample.dso or '[unknown]')) + node = self.find_or_create_node(node, name, libtype) + + node.value += 1 + + def get_report_header(self) -> str: + """Get the header from the perf report.""" + try: + input_file = self.args.input or "perf.data" + if input_file == "-": + return "" + mode = os.stat(input_file).st_mode + import stat + if stat.S_ISFIFO(mode) or stat.S_ISCHR(mode): + return "" + output = subprocess.check_output(["perf", "report", "--header-only", "-i", input_file]) + result = output.decode("utf-8") + if self.args.event_name: + result += "\nFocused event: " + self.args.event_name + return result + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError, + subprocess.CalledProcessError): + return "" + + def run(self) -> None: + """Run the flame graph generation.""" + input_file = self.args.input or "perf.data" + if input_file != "-" and not os.path.exists(input_file): + print(f"Error: {input_file} not found. (try 'perf record' first)", file=sys.stderr) + sys.exit(1) + + try: + self.session = perf.session(perf.data(input_file), + sample=self.process_event) + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError) as e: + print(f"Error opening session: {e}", file=sys.stderr) + sys.exit(1) + + self.session.process_events() + + stacks_json = json.dumps(self.stack, default=lambda x: x.to_json()) + # Escape HTML special characters to prevent XSS + stacks_json = stacks_json.replace("<", "\\u003c") \ + .replace(">", "\\u003e").replace("&", "\\u0026") + + if self.args.format == "html": + report_header = self.get_report_header() + options = { + "colorscheme": self.args.colorscheme, + "context": report_header + } + options_json = json.dumps(options) + options_json = options_json.replace("<", "\\u003c") \ + .replace(">", "\\u003e").replace("&", "\\u0026") + + template = self.args.template + template_sha256sum = None + output_str = None + + if not os.path.isfile(template): + if template.startswith("http://") or template.startswith("https://"): + if not self.args.allow_download: + print("Warning: Downloading templates is disabled. " + "Use --allow-download.", file=sys.stderr) + template = None + else: + print(f"Warning: Template file '{template}' not found.", file=sys.stderr) + if self.args.allow_download: + print("Using default CDN template.", file=sys.stderr) + template = ( + "https://cdn.jsdelivr.net/npm/[email protected]/dist/templates/" + "d3-flamegraph-base.html" + ) + template_sha256sum = ( + "f6a4aa7edffda4fb9bd71eb0eb75bc44d0bb34cd9efbd053f6095bc5c28d702b" + ) + else: + template = None + + use_minimal = False + try: + if not template: + use_minimal = True + elif template.startswith(("http://", "https://")): + with urllib.request.urlopen(template) as url_template: + output_str = "".join([l.decode("utf-8") for l in url_template.readlines()]) + else: + with open(template, "r", encoding="utf-8") as f: + output_str = f.read() + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError) as err: + print(f"Error reading template {template}: {err}\n", file=sys.stderr) + use_minimal = True + + if use_minimal: + print("Using internal minimal HTML that refers to d3's web site. JavaScript " + + "loaded this way from a local file may be blocked unless your " + + "browser has relaxed permissions. Run with '--allow-download' to fetch " + + "the full D3 HTML template.", file=sys.stderr) + output_str = MINIMAL_HTML + + elif template_sha256sum: + assert output_str is not None + download_sha256sum = hashlib.sha256( + output_str.encode("utf-8") + ).hexdigest() + if download_sha256sum != template_sha256sum: + s = None + while s not in ["y", "n"]: + try: + s = input(f"""Unexpected template sha256sum. +{download_sha256sum} != {template_sha256sum}, for: +{template} +continue?[yn] """).lower() + except EOFError: + s = "n" + if s == "n": + sys.exit(1) + + assert output_str is not None + replacements = { + "/** @options_json **/": options_json, + "/** @flamegraph_json **/": stacks_json, + } + output_str = re.sub( + r"/\*\* @(?:options_json|flamegraph_json) \*\*/", + lambda m: replacements[m.group(0)], + output_str, + ) + output_fn = self.args.output or "flamegraph.html" + else: + output_str = stacks_json + output_fn = self.args.output or "stacks.json" + + if output_fn == "-": + with open(sys.stdout.fileno(), "w", encoding="utf-8", closefd=False) as out: + out.write(output_str) + else: + print(f"dumping data to {output_fn}") + with open(output_fn, "w", encoding="utf-8") as out: + out.write(output_str) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Create flame graphs using perf python module.") + parser.add_argument("-f", "--format", default="html", choices=["json", "html"], + help="output file format") + parser.add_argument("-o", "--output", help="output file name") + parser.add_argument("--template", + default="/usr/share/d3-flame-graph/d3-flamegraph-base.html", + help="path to flame graph HTML template") + parser.add_argument("--colorscheme", default="blue-green", + help="flame graph color scheme", choices=["blue-green", "orange"]) + parser.add_argument("-i", "--input", help="input perf.data file") + parser.add_argument("--allow-download", default=False, action="store_true", + help="allow unprompted downloading of HTML template") + parser.add_argument("-e", "--event", default="", dest="event_name", type=str, + help="specify the event to generate flamegraph for") + + cli_args = parser.parse_args() + cli = FlameGraphCLI(cli_args) + cli.run()
tools/perf/tests/shell/test_flamegraph_python.sh new file +106 -0
106 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 19/49] perf python: Port flamegraph to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines
Port
flamegraph.pyto a standalone script intools/perf/python/that uses the perf module directly, avoiding intermediate dictionary allocations for event fields.Improvements compared to the legacy script:
- Add Subresource Integrity (integrity="sha256-..." and crossorigin="anonymous") attributes to external CDN stylesheet and script tags inMINIMAL_HTML.
- Upgrade CDN HTML template hash verification from weak MD5 (hashlib.md5) to cryptographic SHA-256 (hashlib.sha256).
- Escape '<', '>', and '&' ('\u003c', '\u003e', '\u0026') in embedded JSON payloads (stacks_jsonandoptions_json) to prevent HTML script injection / XSS when rendering untrusted symbol or command names.
- Skip invoking 'perf report --header-only' when the input is stdin ('-'), a FIFO pipe, or a character device (S_ISFIFO/S_ISCHR) so non-seekable streams do not hang or fail.
Quoted 5 lines
Add a shell test (
test_flamegraph_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
Ian Rogers
reviewerPATCH 20/49 perf python: Port gecko to perf module
In reply to Ian RogersPort gecko.py to a standalone script in tools/perf/python/ that uses the perf module directly to convert perf.data profiles into Firefox Gecko profile format.
Improvements compared to the legacy script:
- Encapsulate profiler state in GeckoCLI and CategoryData classes with full type annotations, removing global variables.
- Harden the local HTTP server in _write_and_launch(): write the temporary profile to an isolated tempfile.TemporaryDirectory() with a randomized UUID filename instead of the current working directory, bind HTTPServer exclusively to 127.0.0.1 on an OS-assigned ephemeral port (0), restrict CORS Access-Control-Allow-Origin to 'https://profiler.firefox.com' instead of '*', and restrict HTTP GET requests exclusively to the randomized profile filename (returning HTTP 403 for directory listings and any other path).
- Add -i/--input and -e/--event CLI options via argparse.
Add a shell test (test_gecko_python.sh) to verify the standalone script.
--- tools/perf/python/gecko.py | 415 ++++++++++++++++++++ tools/perf/tests/shell/test_gecko_python.sh | 95 +++++ 2 files changed, 510 insertions(+) create mode 100755 tools/perf/python/gecko.py create mode 100755 tools/perf/tests/shell/test_gecko_python.sh
tools/perf/python/gecko.py new file +415 -0
@@ -0,0 +1,415 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +gecko.py - Convert perf record output to Firefox's gecko profile format +""" +from __future__ import annotations + +import argparse +import functools +import json +import os +import sys +import tempfile +import threading +import urllib.parse +import uuid +import webbrowser +from dataclasses import dataclass, field +from http.server import HTTPServer, SimpleHTTPRequestHandler +from typing import Dict, List, NamedTuple, Optional, Tuple + +import perf + + +# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L156 +class Frame(NamedTuple): + """A single stack frame in the gecko profile format.""" + string_id: int + relevantForJS: bool + innerWindowID: int + implementation: None + optimizations: None + line: None + column: None + category: int + subcategory: Optional[int] + + +# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L216 +class Stack(NamedTuple): + """A single stack in the gecko profile format.""" + prefix_id: Optional[int] + frame_id: int + + +# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L90 +class Sample(NamedTuple): + """A single sample in the gecko profile format.""" + stack_id: Optional[int] + time_ms: float + responsiveness: int + + +@dataclass +class Tables: + """Interned tables for the gecko profile format.""" + frame_table: List[Frame] = field(default_factory=list) + string_table: List[str] = field(default_factory=list) + string_map: Dict[str, int] = field(default_factory=dict) + stack_table: List[Stack] = field(default_factory=list) + stack_map: Dict[Tuple[Optional[int], int], int] = field(default_factory=dict) + frame_map: Dict[str, int] = field(default_factory=dict) + + +@dataclass +class Thread: + """A builder for a profile of the thread.""" + comm: str + pid: int + tid: int + user_category: int + kernel_category: int + samples: List[Sample] = field(default_factory=list) + tables: Tables = field(default_factory=Tables) + + def _intern_stack(self, frame_id: int, prefix_id: Optional[int]) -> int: + """Gets a matching stack, or saves the new stack. Returns a Stack ID.""" + key = (prefix_id, frame_id) + stack_id = self.tables.stack_map.get(key) + if stack_id is None: + stack_id = len(self.tables.stack_table) + self.tables.stack_table.append(Stack(prefix_id=prefix_id, frame_id=frame_id)) + self.tables.stack_map[key] = stack_id + return stack_id + + def _intern_string(self, string: str) -> int: + """Gets a matching string, or saves the new string. Returns a String ID.""" + string_id = self.tables.string_map.get(string) + if string_id is not None: + return string_id + string_id = len(self.tables.string_table) + self.tables.string_table.append(string) + self.tables.string_map[string] = string_id + return string_id + + def _intern_frame(self, frame_str: str) -> int: + """Gets a matching stack frame, or saves the new frame. Returns a Frame ID.""" + frame_id = self.tables.frame_map.get(frame_str) + if frame_id is not None: + return frame_id + frame_id = len(self.tables.frame_table) + self.tables.frame_map[frame_str] = frame_id + string_id = self._intern_string(frame_str) + + category = self.user_category + if (frame_str.find('kallsyms') != -1 or + frame_str.find('/vmlinux') != -1 or + frame_str.endswith('.ko)')): + category = self.kernel_category + + self.tables.frame_table.append(Frame( + string_id=string_id, + relevantForJS=False, + innerWindowID=0, + implementation=None, + optimizations=None, + line=None, + column=None, + category=category, + subcategory=None, + )) + return frame_id + + def add_sample(self, comm: str, stack: List[str], time_ms: float) -> None: + """Add a timestamped stack trace sample to the thread builder.""" + if self.comm != comm: + self.comm = comm + + prefix_stack_id: Optional[int] = None + for frame in stack: + frame_id = self._intern_frame(frame) + prefix_stack_id = self._intern_stack(frame_id, prefix_stack_id) + + if prefix_stack_id is not None: + self.samples.append(Sample(stack_id=prefix_stack_id, + time_ms=time_ms, + responsiveness=0)) + + def to_json_dict(self) -> Dict: + """Converts current Thread to GeckoThread JSON format.""" + return { + "tid": self.tid, + "pid": self.pid, + "name": self.comm, + "markers": { + "schema": { + "name": 0, + "startTime": 1, + "endTime": 2, + "phase": 3, + "category": 4, + "data": 5, + }, + "data": [], + }, + "samples": { + "schema": { + "stack": 0, + "time": 1, + "responsiveness": 2, + }, + "data": self.samples + }, + "frameTable": { + "schema": { + "location": 0, + "relevantForJS": 1, + "innerWindowID": 2, + "implementation": 3, + "optimizations": 4, + "line": 5, + "column": 6, + "category": 7, + "subcategory": 8, + }, + "data": self.tables.frame_table, + }, + "stackTable": { + "schema": { + "prefix": 0, + "frame": 1, + }, + "data": self.tables.stack_table, + }, + "stringTable": self.tables.string_table, + "registerTime": 0, + "unregisterTime": None, + "processType": "default", + } + + +class CORSRequestHandler(SimpleHTTPRequestHandler): + """Enable CORS for requests from profiler.firefox.com.""" + def __init__(self, *args, allowed_file: Optional[str] = None, **kwargs): + self.allowed_file = allowed_file + super().__init__(*args, **kwargs) + + def end_headers(self): + self.send_header('Access-Control-Allow-Origin', 'https://profiler.firefox.com') + super().end_headers() + + def do_GET(self): + if self.allowed_file and self.path.split('?')[0] != f'/{self.allowed_file}': + self.send_error(403, "Access denied") + return + super().do_GET() + + def list_directory(self, path): + self.send_error(403, "Directory listing forbidden") + return None + + +@dataclass +class CategoryData: + """Category configuration for the gecko profile.""" + user_index: int = 0 + kernel_index: int = 1 + categories: List[Dict] = field(default_factory=list) + + +class GeckoCLI: + """Command-line interface for converting perf data to Gecko format.""" + def __init__(self, args: argparse.Namespace) -> None: + self.args = args + self.tid_to_thread: Dict[int, Thread] = {} + self.start_time_ms: Optional[float] = None + self.session: Optional[perf.session] = None + self.product = args.product + self.cat_data = CategoryData( + categories=[ + { + "name": 'User', + "color": args.user_color, + "subcategories": ['Other'] + }, + { + "name": 'Kernel', + "color": args.kernel_color, + "subcategories": ['Other'] + }, + ] + ) + + def process_event(self, sample) -> None: + """Process a single perf sample event.""" + if self.args.event_name and self.args.event_name not in str(sample.evsel): + return + + # sample_time is in nanoseconds. Gecko wants milliseconds. + time_ms = sample.sample_time / 1000000.0 + pid = sample.sample_pid + tid = sample.sample_tid + + if self.start_time_ms is None: + self.start_time_ms = time_ms + + try: + thread_info = self.session.find_thread(tid) if self.session else None + comm = (thread_info.comm() if thread_info is not None else None) or "[unknown]" + except AttributeError: + comm = "[unknown]" + + stack = [] + callchain = sample.callchain + if callchain: + for entry in callchain: + symbol = entry.symbol or "[unknown]" + dso = entry.dso or "[unknown]" + stack.append(f"{symbol} (in {dso})") + # Reverse because Gecko wants root first. + stack.reverse() + else: + # Fallback if no callchain is present + try: + # If the perf module exposes symbol/dso directly on sample + # when callchain is missing, we use them. + symbol = (sample.symbol or '[unknown]') + dso = (sample.dso or '[unknown]') + stack.append(f"{symbol} (in {dso})") + except AttributeError: + stack.append("[unknown] (in [unknown])") + + thread = self.tid_to_thread.get(tid) + if thread is None: + thread = Thread(comm=comm, pid=pid, tid=tid, + user_category=self.cat_data.user_index, + kernel_category=self.cat_data.kernel_index) + self.tid_to_thread[tid] = thread + thread.add_sample(comm=comm, stack=stack, time_ms=time_ms) + + def run(self) -> None: + """Run the conversion process.""" + input_file = self.args.input or "perf.data" + if input_file != "-" and not os.path.exists(input_file): + print(f"Error: {input_file} not found.", file=sys.stderr) + sys.exit(1) + + try: + self.session = perf.session(perf.data(input_file), sample=self.process_event) + except (OSError, RuntimeError, ValueError) as e: + print(f"Error opening session: {e}", file=sys.stderr) + sys.exit(1) + + if self.session: + try: + self.session.process_events() + finally: + self.session = None + + threads = [t.to_json_dict() for t in self.tid_to_thread.values()] + + gecko_profile = { + "meta": { + "interval": 1, + "processType": 0, + "product": self.product, + "stackwalk": 1, + "debug": 0, + "gcpoison": 0, + "asyncstack": 1, + "startTime": self.start_time_ms, + "shutdownTime": None, + "version": 24, + "presymbolicated": True, + "categories": self.cat_data.categories, + "markerSchema": [], + }, + "libs": [], + "threads": threads, + "processes": [], + "pausedRanges": [], + } + + output_file = self.args.save_only + if output_file is None: + self._write_and_launch(gecko_profile) + else: + print(f'[ perf gecko: Captured and wrote into {output_file} ]') + with open(output_file, 'w', encoding='utf-8') as f: + json.dump(gecko_profile, f, indent=2) + + def _write_and_launch(self, profile: Dict) -> None: + """Write the profile to a file and launch the Firefox profiler.""" + print("Starting Firefox Profiler on your default browser...") + + with tempfile.TemporaryDirectory() as tmp_dir_name: + profile_name = f'gecko_profile_{uuid.uuid4().hex}.json' + filename = os.path.join(tmp_dir_name, profile_name) + + with open(filename, 'w', encoding='utf-8') as f: + json.dump(profile, f, indent=2) + + handler = functools.partial( + CORSRequestHandler, directory=tmp_dir_name, allowed_file=profile_name + ) + try: + httpd = HTTPServer(('127.0.0.1', 0), handler) + except OSError as e: + print(f"Error starting HTTP server: {e}", file=sys.stderr) + sys.exit(1) + + port = httpd.server_port + + def start_server(): + httpd.serve_forever() + + thread = threading.Thread(target=start_server, daemon=True) + thread.start() + + safe_string = urllib.parse.quote_plus(f'http://127.0.0.1:{port}/{profile_name}') + url = f'https://profiler.firefox.com/from-url/{safe_string}' + print(f"Please open the following URL in your browser to view the profile:\n\n{url}\n") + launch_html = os.path.join(tmp_dir_name, 'launch.html') + with open(launch_html, 'w', encoding='utf-8') as f: + f.write(f'<!DOCTYPE html><html><head>' + f'<meta http-equiv="refresh" content="0;url={url}">' + f'<script>window.location.replace({json.dumps(url)});</script>' + f'</head><body>Redirecting to Firefox Profiler...</body></html>') + if not webbrowser.open(f'file://{launch_html}'): + print("Failed to open browser, please open the URL manually.") + + print(f'[ perf gecko: Captured and wrote into {filename} ]') + print("Press Ctrl+C to stop the local server.") + try: + # Keep the main thread alive so the daemon thread can serve requests + stop_event = threading.Event() + while True: + stop_event.wait(1) + except KeyboardInterrupt: + print("\nStopping server...") + httpd.shutdown() + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Convert perf.data to Firefox's Gecko Profile format" + ) + parser.add_argument('--product', default='perf', help='Product name (e.g. perf)') + parser.add_argument('--user-color', default='yellow', + help='Color for the User category', + choices=['yellow', 'blue', 'purple', 'green', 'orange', 'red', + 'grey', 'magenta']) + parser.add_argument('--kernel-color', default='orange', + help='Color for the Kernel category', + choices=['yellow', 'blue', 'purple', 'green', 'orange', 'red', + 'grey', 'magenta']) + parser.add_argument('--save-only', + help='Save the output to a file instead of opening Firefox\'s profiler') + parser.add_argument("-i", "--input", help="input perf.data file") + parser.add_argument("-e", "--event", default="", dest="event_name", type=str, + help="specify the event to generate gecko profile for") + + cli_args = parser.parse_args() + cli = GeckoCLI(cli_args) + cli.run()
tools/perf/tests/shell/test_gecko_python.sh new file +95 -0
95 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 20/49] perf python: Port gecko to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines
Port
gecko.pyto a standalone script intools/perf/python/that uses the perf module directly to convert perf.data profiles into Firefox Gecko profile format.Improvements compared to the legacy script:
- Encapsulate profiler state in GeckoCLI and CategoryData classes with full type annotations, removing global variables.
- Harden the local HTTP server in_write_and_launch(): write the temporary profile to an isolatedtempfile.TemporaryDirectory()with a randomized UUID filename instead of the current working directory, bind HTTPServer exclusively to 127.0.0.1 on an OS-assigned ephemeral port (0), restrict CORS Access-Control-Allow-Origin to 'https://profiler.firefox.com' instead of '*', and restrict HTTP GET requests exclusively to the randomized profile filename (returning HTTP 403 for directory listings and any other path).
- Add -i/--input and -e/--event CLI options via argparse.
Quoted 4 lines
Add a shell test (
test_gecko_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=20
Ian Rogers
reviewerPATCH 21/49 perf python: Port event_analyzing_sample to perf module
In reply to Ian RogersPort event_analyzing_sample.py to a standalone script in tools/perf/python/ using the perf module and standard library sqlite3 module.
Improvements compared to the legacy script:
- Encapsulate database state in a _DB container instead of mutating module-level globals, and ensure temporary SQLite database files are cleaned up on exit.
- Add argparse CLI options (-i/--input and -d/--db) while preserving PerfEvent, PebsEvent, and PebsNHM binary raw_buf unpacking and symbol/DSO histogram reporting.
- Remove Python 2 compatibility code and add type annotations.
Add a shell test (test_event_analyzing_sample_python.sh) to verify the standalone script.
--- tools/perf/python/event_analyzing_sample.py | 321 ++++++++++++++++++ .../test_event_analyzing_sample_python.sh | 58 ++++ 2 files changed, 379 insertions(+) create mode 100755 tools/perf/python/event_analyzing_sample.py create mode 100755 tools/perf/tests/shell/test_event_analyzing_sample_python.sh
tools/perf/python/event_analyzing_sample.py new file +321 -0
@@ -0,0 +1,321 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +General event handler in Python, using SQLite to analyze events. + +The 2 database related functions in this script just show how to gather +the basic information, and users can modify and write their own functions +according to their specific requirement. + +The first function "show_general_events" just does a basic grouping for all +generic events with the help of sqlite, and the 2nd one "show_pebs_ll" is +for a x86 HW PMU event: PEBS with load latency data. + +Ported from tools/perf/scripts/python/event_analyzing_sample.py +""" +from __future__ import annotations + +import argparse +import math +import os +import sqlite3 +import struct +import tempfile +from typing import Any +import perf + +# Event types, user could add more here +EVTYPE_GENERIC = 0 +EVTYPE_PEBS = 1 # Basic PEBS event +EVTYPE_PEBS_LL = 2 # PEBS event with load latency info +EVTYPE_IBS = 3 + +# +# Currently we don't have good way to tell the event type, but by +# the size of raw buffer, raw PEBS event with load latency data's +# size is 176 bytes, while the pure PEBS event's size is 144 bytes. +# +def create_event(name, comm, dso, symbol, raw_buf): + """Create an event object based on raw buffer size.""" + if len(raw_buf) == 144: + event = PebsEvent(name, comm, dso, symbol, raw_buf) + elif len(raw_buf) == 176: + event = PebsNHM(name, comm, dso, symbol, raw_buf) + else: + event = PerfEvent(name, comm, dso, symbol, raw_buf) + + return event + +class PerfEvent: + """Base class for all perf event samples.""" + event_num = 0 + def __init__(self, name, comm, dso, symbol, raw_buf, ev_type=EVTYPE_GENERIC): + self.name = name + self.comm = comm + self.dso = dso + self.symbol = symbol + self.raw_buf = raw_buf + self.ev_type = ev_type + PerfEvent.event_num += 1 + + def show(self): + """Display PMU event info.""" + print(f"PMU event: name={self.name:12s}, symbol={self.symbol:24s}, " + f"comm={self.comm:8s}, dso={self.dso:12s}") + +# +# Basic Intel PEBS (Precise Event-based Sampling) event, whose raw buffer +# contains the context info when that event happened: the EFLAGS and +# linear IP info, as well as all the registers. +# +class PebsEvent(PerfEvent): + """Intel PEBS event.""" + pebs_num = 0 + def __init__(self, name, comm, dso, symbol, raw_buf, ev_type=EVTYPE_PEBS): + tmp_buf = raw_buf[0:80] + flags, ip, ax, bx, cx, dx, si, di, bp, sp = struct.unpack('<QQQQQQQQQQ', tmp_buf) + self.flags = flags + self.ip = ip + self.ax = ax + self.bx = bx + self.cx = cx + self.dx = dx + self.si = si + self.di = di + self.bp = bp + self.sp = sp + + super().__init__(name, comm, dso, symbol, raw_buf, ev_type) + PebsEvent.pebs_num += 1 + del tmp_buf + +# +# Intel Nehalem and Westmere support PEBS plus Load Latency info which lie +# in the four 64 bit words write after the PEBS data: +# Status: records the IA32_PERF_GLOBAL_STATUS register value +# DLA: Data Linear Address (EIP) +# DSE: Data Source Encoding, where the latency happens, hit or miss +# in L1/L2/L3 or IO operations +# LAT: the actual latency in cycles +# +class PebsNHM(PebsEvent): + """Intel Nehalem/Westmere PEBS event with load latency.""" + pebs_nhm_num = 0 + def __init__(self, name, comm, dso, symbol, raw_buf, ev_type=EVTYPE_PEBS_LL): + tmp_buf = raw_buf[144:176] + status, dla, dse, lat = struct.unpack('<QQQQ', tmp_buf) + self.status = status + self.dla = dla + self.dse = dse + self.lat = lat + + super().__init__(name, comm, dso, symbol, raw_buf, ev_type) + PebsNHM.pebs_nhm_num += 1 + del tmp_buf + +session: Any = None + +class _DB: + con: sqlite3.Connection | None = None + temp_path: str | None = None + +def trace_begin(db_path: str | None = None) -> None: + """Initialize database tables.""" + print("In trace_begin:\n") + if not db_path: + fd, db_path = tempfile.mkstemp(prefix="perf_events_", suffix=".db") + os.close(fd) + _DB.temp_path = db_path + _DB.con = sqlite3.connect(db_path) + con = _DB.con + assert con is not None + + # Drop any pre-existing tables so repeated runs do not accumulate duplicate events. + con.execute("drop table if exists gen_events;") + con.execute("drop table if exists pebs_ll;") + + # Will create several tables at the start, pebs_ll is for PEBS data with + # load latency info, while gen_events is for general event. + con.execute(""" + create table if not exists gen_events ( + name text, + symbol text, + comm text, + dso text + );""") + con.execute(""" + create table if not exists pebs_ll ( + name text, + symbol text, + comm text, + dso text, + flags integer, + ip integer, + status integer, + dse integer, + dla integer, + lat integer + );""") + +def insert_db(event: Any) -> None: + """Insert event into database.""" + con = _DB.con + assert con is not None + if event.ev_type == EVTYPE_GENERIC: + con.execute("insert into gen_events values(?, ?, ?, ?)", + (event.name, event.symbol, event.comm, event.dso)) + elif event.ev_type == EVTYPE_PEBS_LL: + ip = event.ip - 0x10000000000000000 if event.ip > 0x7fffffffffffffff else event.ip + dla = event.dla - 0x10000000000000000 if event.dla > 0x7fffffffffffffff else event.dla + con.execute("insert into pebs_ll values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + (event.name, event.symbol, event.comm, event.dso, event.flags, + ip, event.status, event.dse, dla, event.lat)) + +def process_event(sample: perf.sample_event) -> None: + """Callback for processing events.""" + # Create and insert event object to a database so that user could + # do more analysis with simple database commands. + + # Resolve comm, symbol, dso + comm = "Unknown_comm" + try: + if session is not None: + proc = session.find_thread(sample.sample_pid, sample.sample_tid) + if proc: + comm = proc.comm() or "Unknown_comm" + except TypeError: + pass + + # Symbol and dso info are not always resolved + dso = sample.dso if hasattr(sample, 'dso') and sample.dso else "Unknown_dso" + symbol = sample.symbol if hasattr(sample, 'symbol') and sample.symbol else "Unknown_symbol" + name = str(sample.evsel) + if name.startswith("evsel("): + name = name[6:-1] + + # Create the event object and insert it to the right table in database + try: + event = create_event(name, comm, dso, symbol, sample.raw_buf) + insert_db(event) + except (sqlite3.Error, ValueError, TypeError) as e: + print(f"Error creating/inserting event: {e}") + +def num2sym(num: int) -> str: + """Convert number to a histogram symbol (log2).""" + # As the event number may be very big, so we can't use linear way + # to show the histogram in real number, but use a log2 algorithm. + if num <= 0: + return "" + snum = '#' * (int(math.log(num, 2)) + 1) + return snum + +def show_general_events() -> None: + """Display statistics for general events.""" + con = _DB.con + assert con is not None + count = con.execute("select count(*) from gen_events") + for t in count: + print(f"There is {t[0]} records in gen_events table") + if t[0] == 0: + return + + print("Statistics about the general events grouped by thread/symbol/dso: \n") + + # Group by thread + commq = con.execute(""" + select comm, count(comm) from gen_events + group by comm order by -count(comm) + """) + print(f"\n{ 'comm':>16} {'number':>8} {'histogram':>16}\n{'='*42}") + for row in commq: + print(f"{row[0]:>16} {row[1]:>8} {num2sym(row[1])}") + + # Group by symbol + print(f"\n{'symbol':>32} {'number':>8} {'histogram':>16}\n{'='*58}") + symbolq = con.execute(""" + select symbol, count(symbol) from gen_events + group by symbol order by -count(symbol) + """) + for row in symbolq: + print(f"{row[0]:>32} {row[1]:>8} {num2sym(row[1])}") + + # Group by dso + print(f"\n{'dso':>40} {'number':>8} {'histogram':>16}\n{'='*74}") + dsoq = con.execute("select dso, count(dso) from gen_events group by dso order by -count(dso)") + for row in dsoq: + print(f"{row[0]:>40} {row[1]:>8} {num2sym(row[1])}") + +def show_pebs_ll() -> None: + """Display statistics for PEBS load latency events.""" + con = _DB.con + assert con is not None + # This function just shows the basic info, and we could do more with the + # data in the tables, like checking the function parameters when some + # big latency events happen. + count = con.execute("select count(*) from pebs_ll") + for t in count: + print(f"There is {t[0]} records in pebs_ll table") + if t[0] == 0: + return + + print("Statistics about the PEBS Load Latency events grouped by thread/symbol/dse/latency: \n") + + # Group by thread + commq = con.execute("select comm, count(comm) from pebs_ll group by comm order by -count(comm)") + print(f"\n{'comm':>16} {'number':>8} {'histogram':>16}\n{'='*42}") + for row in commq: + print(f"{row[0]:>16} {row[1]:>8} {num2sym(row[1])}") + + # Group by symbol + print(f"\n{'symbol':>32} {'number':>8} {'histogram':>16}\n{'='*58}") + symbolq = con.execute(""" + select symbol, count(symbol) from pebs_ll + group by symbol order by -count(symbol) + """) + for row in symbolq: + print(f"{row[0]:>32} {row[1]:>8} {num2sym(row[1])}") + + # Group by dse + dseq = con.execute("select dse, count(dse) from pebs_ll group by dse order by -count(dse)") + print(f"\n{'dse':>32} {'number':>8} {'histogram':>16}\n{'='*58}") + for row in dseq: + print(f"{row[0]:>32} {row[1]:>8} {num2sym(row[1])}") + + # Group by latency + latq = con.execute("select lat, count(lat) from pebs_ll group by lat order by lat") + print(f"\n{'latency':>32} {'number':>8} {'histogram':>16}\n{'='*58}") + for row in latq: + print(f"{str(row[0]):>32} {row[1]:>8} {num2sym(row[1])}") + +def trace_end() -> None: + """Called at the end of trace processing.""" + print("In trace_end:\n") + try: + if _DB.con: + _DB.con.commit() + show_general_events() + show_pebs_ll() + _DB.con.close() + _DB.con = None + finally: + if _DB.temp_path and os.path.exists(_DB.temp_path): + try: + os.remove(_DB.temp_path) + except OSError: + pass + _DB.temp_path = None + +if __name__ == "__main__": + ap = argparse.ArgumentParser(description="Analyze events with SQLite") + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + ap.add_argument("-d", "--db", "--database", dest="database", default=None, + help="Database file name (defaults to a temporary file cleaned up on exit)") + args = ap.parse_args() + + try: + trace_begin(args.database) + session = perf.session(perf.data(args.input), sample=process_event) + session.process_events() + finally: + session = None + trace_end()
tools/perf/tests/shell/test_event_analyzing_sample_python.sh new file +58 -0
58 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 21/49] perf python: Port event_analyzing_sample to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines
Port
event_analyzing_sample.pyto a standalone script intools/perf/python/using the perf module and standard library sqlite3 module.Improvements compared to the legacy script:
- Encapsulate database state in a_DBcontainer instead of mutating module-level globals, and ensure temporary SQLite database files are cleaned up on exit.
- Add argparse CLI options (-i/--input and -d/--db) while preserving PerfEvent, PebsEvent, and PebsNHM binaryraw_bufunpacking and symbol/DSO histogram reporting.
- Remove Python 2 compatibility code and add type annotations.
Quoted 5 lines
Add a shell test (
test_event_analyzing_sample_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=21
Ian Rogers
reviewerPATCH 22/49 perf python: Port syscall-counts to perf module
In reply to Ian RogersPort tools/perf/scripts/python/syscall-counts.py to a standalone script in tools/perf/python/ using the perf module. Avoiding the embedded interpreter and per-event dictionary allocation overhead improves execution speed by ~4x:
``` $ perf record -e raw_syscalls:sys_enter -a sleep 1 ... $ time perf script tools/perf/scripts/python/syscall-counts.py perf ... real 0m3.887s user 0m3.578s sys 0m0.308s $ time python3 tools/perf/python/syscall-counts.py perf ... real 0m0.953s user 0m0.905s sys 0m0.048s ```
Additional improvements compared to the legacy script:
- Resolve syscall names using perf.syscall_name(id, session.e_machine) instead of host python-audit / Util.py tables, enabling accurate cross-architecture perf.data analysis without external dependencies.
- Support both raw_syscalls:sys_enter (sample.id) and individual syscalls:sys_enter_* tracepoints (sample.__syscall_nr / sample.nr), filtering out invalid/corrupt (> 0xffff or negative) syscall numbers.
- Add argparse CLI options (-i/--input and optional comm filter).
Add a shell test (test_syscall_counts_python.sh) to verify the standalone script. The legacy script and its bin wrapper are retained temporarily during the transition to maintain bisectability and are removed once all scripts are migrated.
--- tools/perf/python/syscall-counts.py | 79 +++++++++++++++++++ .../tests/shell/test_syscall_counts_python.sh | 74 +++++++++++++++++ 2 files changed, 153 insertions(+) create mode 100755 tools/perf/python/syscall-counts.py create mode 100755 tools/perf/tests/shell/test_syscall_counts_python.sh
tools/perf/python/syscall-counts.py new file +79 -0
@@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +Displays system-wide system call totals, broken down by syscall. + +If a [comm] arg is specified, only syscalls called by [comm] are displayed. +""" +from __future__ import annotations + +import argparse +from collections import defaultdict +from typing import DefaultDict +import perf + +syscalls: DefaultDict[int, int] = defaultdict(int) +for_comm = None +session = None + + +def print_syscall_totals(): + """Print aggregated statistics.""" + if for_comm is not None: + print(f"\nsyscall events for {for_comm}:\n") + else: + print("\nsyscall events:\n") + + print(f"{'event':<40} {'count':>10}") + print("---------------------------------------- -----------") + + for sc_id, val in sorted(syscalls.items(), + key=lambda kv: (kv[1], kv[0]), reverse=True): + e_machine = getattr(session, "e_machine", 0) or 0 + if e_machine: + name = perf.syscall_name(sc_id, e_machine) or str(sc_id) + else: + name = perf.syscall_name(sc_id) or str(sc_id) + print(f"{name:<40} {val:>10}") + + +def process_event(sample): + """Process a single sample event.""" + event_name = str(sample.evsel) + if event_name.startswith("evsel(raw_syscalls:sys_enter"): + sc_id = getattr(sample, "id", -1) + elif event_name.startswith("evsel(syscalls:sys_enter"): + sc_id = getattr(sample, "__syscall_nr", getattr(sample, "id", None)) + if sc_id is not None and (sc_id < 0 or sc_id > 0xffff): + sc_id = None + if sc_id is None: + sc_id = getattr(sample, "nr", -1) + else: + return + + if sc_id < 0 or sc_id > 0xffff: + return + + comm = "unknown" + try: + if session: + proc = session.find_thread(sample.sample_tid) + if proc: + comm = proc.comm() or "unknown" + except (TypeError, AttributeError): + pass + + if for_comm and comm != for_comm: + return + syscalls[sc_id] += 1 + + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + ap.add_argument("comm", nargs="?", help="Only report syscalls for comm") + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + args = ap.parse_args() + for_comm = args.comm + session = perf.session(perf.data(args.input), sample=process_event) + session.process_events() + print_syscall_totals()
tools/perf/tests/shell/test_syscall_counts_python.sh new file +74 -0
74 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 22/49] perf python: Port syscall-counts to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
Port
tools/perf/scripts/python/syscall-counts.pyto a standalone script intools/perf/python/using the perf module. Avoiding the embedded interpreter and per-event dictionary allocation overhead improves execution speed by ~4x:```
$ perf record -e raw_syscalls:sys_enter -a sleep 1 ... $ time perf script tools/perf/scripts/python/syscall-counts.py perf ... real 0m3.887s user 0m3.578s sys 0m0.308s $ time python3 tools/perf/python/syscall-counts.py perf ... [ ... ] temporarily during the transition to maintain bisectability and are removed once all scripts are migrated.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=22
Ian Rogers
reviewerPATCH 23/49 perf python: Port syscall-counts-by-pid to perf module
In reply to Ian RogersPort tools/perf/scripts/python/syscall-counts-by-pid.py to a standalone script in tools/perf/python/ using the perf module. Avoiding the embedded interpreter and per-event dictionary overhead improves execution speed by ~3.8x:
``` $ perf record -e raw_syscalls:sys_enter -a sleep 1 ... $ time perf script tools/perf/scripts/python/syscall-counts-by-pid.py perf ... real 0m3.852s user 0m3.512s sys 0m0.336s $ time python3 tools/perf/python/syscall-counts-by-pid.py perf ... real 0m1.011s user 0m0.963s sys 0m0.048s ```
Additional improvements compared to the legacy script:
- Resolve architecture-specific syscall names via perf.syscall_name(id, session.e_machine) instead of host python-audit tables.
- Support both raw_syscalls:sys_enter and individual syscalls:sys_enter_* tracepoints, and filter out invalid (> 0xffff or negative) syscall IDs.
- Support filtering by numeric PID as well as command name (comm), and resolve process command names via session.find_thread(pid).
Add a shell test (test_syscall_counts_by_pid_python.sh) to verify the standalone script.
--- tools/perf/python/syscall-counts-by-pid.py | 100 ++++++++++++++++++ .../test_syscall_counts_by_pid_python.sh | 81 ++++++++++++++ 2 files changed, 181 insertions(+) create mode 100755 tools/perf/python/syscall-counts-by-pid.py create mode 100755 tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh
tools/perf/python/syscall-counts-by-pid.py new file +100 -0
@@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +Displays system-wide system call totals, broken down by syscall. +If a [comm] arg is specified, only syscalls called by [comm] are displayed. +""" +from __future__ import annotations + +import argparse +from collections import defaultdict +from typing import (Dict, Tuple) +import perf + +syscalls: Dict[Tuple[str, int, int], int] = defaultdict(int) +for_comm = None +for_pid = None +session = None + + +def print_syscall_totals(): + """Print aggregated statistics.""" + if for_comm is not None: + print(f"\nsyscall events for {for_comm}:\n") + elif for_pid is not None: + print(f"\nsyscall events for PID {for_pid}:\n") + else: + print("\nsyscall events:\n") + + print(f"{'comm [pid]/syscalls':<40} {'count':>10}") + print("---------------------------------------- -----------") + + sorted_keys = sorted(syscalls.keys(), key=lambda k: (k[0], k[1], -syscalls[k], -k[2])) + current_comm_pid = None + for comm, pid, sc_id in sorted_keys: + if current_comm_pid != (comm, pid): + print(f"\n{comm} [{pid}]") + current_comm_pid = (comm, pid) + e_machine = getattr(session, "e_machine", 0) or 0 + if e_machine: + name = perf.syscall_name(sc_id, e_machine) or str(sc_id) + else: + name = perf.syscall_name(sc_id) or str(sc_id) + print(f" {name:<38} {syscalls[(comm, pid, sc_id)]:>10}") + + +def process_event(sample): + """Process a single sample event.""" + event_name = str(sample.evsel) + if event_name.startswith("evsel(raw_syscalls:sys_enter"): + sc_id = getattr(sample, "id", -1) + elif event_name.startswith("evsel(syscalls:sys_enter"): + sc_id = getattr(sample, "__syscall_nr", None) + if sc_id is not None and (sc_id < 0 or sc_id > 0xffff): + sc_id = None + if sc_id is None: + sc_id = getattr(sample, "nr", None) + if sc_id is not None and (sc_id < 0 or sc_id > 0xffff): + sc_id = None + if sc_id is None: + sc_id = getattr(sample, "id", -1) + else: + return + + if sc_id < 0 or sc_id > 0xffff: + return + + pid = sample.sample_pid + + if for_pid is not None and pid != for_pid: + return + + comm = "unknown" + try: + if session: + proc = session.find_thread(pid) + if proc: + comm = proc.comm() or "unknown" + except (TypeError, AttributeError): + pass + + if for_comm and comm != for_comm: + return + syscalls[(comm, pid, sc_id)] += 1 + + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + ap.add_argument("filter", nargs="?", help="COMM or PID to filter by") + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + args = ap.parse_args() + + if args.filter: + try: + for_pid = int(args.filter) + except ValueError: + for_comm = args.filter + + session = perf.session(perf.data(args.input), sample=process_event) + session.process_events() + print_syscall_totals()
tools/perf/tests/shell/test_syscall_counts_by_pid_python.sh new file +81 -0
81 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 23/49] perf python: Port syscall-counts-by-pid to perf module
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf python: sample.id fallback bypasses syscall filter --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf python: sample.id fallback bypasses syscall filter
--
commit 382862f26193c21e71a417f0682c83de767a4460
Author: Ian Rogers <[email protected]>
perf python: Port syscall-counts-by-pid to perf module
This commit ports the syscall-counts-by-pid script to a standalone script in tools/perf/python/ using the perf module to improve execution speed. It also resolves architecture-specific syscall names dynamically and supports both raw_syscalls:sys_enter and syscalls:sys_enter_* tracepoints.
Quoted 5 lines
tools/perf/python/syscall-counts-by-pid.py new file +0
-0
[ ... ]
Quoted 5 lines
tools/perf/python/syscall-counts-by-pid.py new file +5
-0+def process_event(sample): + """Process a single sample event.""" + event_name = str(sample.evsel) + if event_name.startswith("evsel(raw_syscalls:sys_enter"): + sc_id = getattr(sample, "id", -1)
tools/perf/python/syscall-counts-by-pid.py line 60
Quoted 10 lines
tools/perf/python/syscall-counts-by-pid.py new file +10
-0+ elif event_name.startswith("evsel(syscalls:sys_enter"): + sc_id = getattr(sample, "__syscall_nr", None) + if sc_id is not None and (sc_id < 0 or sc_id > 0xffff): + sc_id = None + if sc_id is None: + sc_id = getattr(sample, "nr", None) + if sc_id is not None and (sc_id < 0 or sc_id > 0xffff): + sc_id = None + if sc_id is None: + sc_id = getattr(sample, "id", -1)
[Severity: Medium]
When process_event() handles a syscalls:sys_enter_* tracepoint with an invalid or missing __syscall_nr or nr, will getattr() retrieve the perf event ID instead of the intended -1 fallback value?
Since the Python sample object has a built-in property id that maps to the 64-bit perf event ID, getattr(sample, "id", -1) will successfully read the perf event ID rather than falling back to -1.
If the perf event ID happens to be <= 0xffff, could it bypass the subsequent sc_id > 0xffff filter entirely and be incorrectly tallied as a valid syscall?
Quoted 5 lines
tools/perf/python/syscall-counts-by-pid.py new file +5
-0+ else: + return + + if sc_id < 0 or sc_id > 0xffff: + return
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=23
Ian Rogers
reviewerPATCH 24/49 perf python: Port failed-syscalls-by-pid to perf module
In reply to Ian RogersPort failed-syscalls-by-pid.py to a standalone script in tools/perf/python/ using the perf module:
- Resolve architecture-aware syscall names using perf.syscall_name(id, session.e_machine) instead of host python-audit tables, supporting cross-architecture perf.data files without external dependencies.
- Translate negative syscall return values into symbolic E* error names using errno.errorcode and architecture-specific session.e_machine overrides.
- Encapsulate aggregation in a SyscallAnalyzer class using collections.defaultdict and add argparse filtering by comm, PID, and -i/--input.
Add a shell test (test_failed_syscalls_by_pid_python.sh) to verify the standalone script.
--- tools/perf/python/failed-syscalls-by-pid.py | 175 ++++++++++++++++++ .../test_failed_syscalls_by_pid_python.sh | 92 +++++++++ 2 files changed, 267 insertions(+) create mode 100755 tools/perf/python/failed-syscalls-by-pid.py create mode 100755 tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh
tools/perf/python/failed-syscalls-by-pid.py new file +175 -0
@@ -0,0 +1,175 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +Displays system-wide failed system call totals, broken down by pid. +If a [comm] or [pid] arg is specified, only syscalls called by it are displayed. + +Ported from tools/perf/scripts/python/failed-syscalls-by-pid.py +""" +from __future__ import annotations + +import argparse +from collections import defaultdict +import errno +import os +import sys +from typing import Optional +import perf + + +_GENERIC_ERRNO_OVERRIDES: dict[int, str] = { + 11: "EAGAIN", 35: "EDEADLK", 36: "ENAMETOOLONG", 37: "ENOLCK", + 38: "ENOSYS", 39: "ENOTEMPTY", 40: "ELOOP", 42: "ENOMSG", + 45: "EOPNOTSUPP", 60: "ENOSTR", 61: "ENODATA", 78: "EREMCHG", + 87: "EUSERS", 89: "EDESTADDRREQ", 90: "EMSGSIZE", 110: "ETIMEDOUT", + 111: "ECONNREFUSED", 122: "EDQUOT", +} + +_ARCH_ERRNO_TABLES: dict[int, dict[int, str]] = { + 2: {35: "EAGAIN", 36: "EINPROGRESS", 37: "EALREADY", 78: "EDEADLK", 87: "ENOMSG"}, + 8: {35: "ENOMSG", 45: "EDEADLK", 89: "ENOSYS", 90: "ELOOP", 122: "EDQUOT"}, + 15: {35: "ENOMSG", 45: "EDEADLK", 246: "EAGAIN", 251: "ENOSYS"}, + 43: {35: "EAGAIN", 36: "EINPROGRESS", 37: "EALREADY", 78: "EDEADLK", 87: "ENOMSG"}, + 0x9026: {35: "EAGAIN", 11: "EDEADLK", 60: "ETIMEDOUT", 61: "ECONNREFUSED"}, +} + + +def strerror(nr: int, e_machine: int = 0) -> str: + """Return error string for a given errno, accounting for target e_machine.""" + err_num = abs(nr) + if e_machine in _ARCH_ERRNO_TABLES and err_num in _ARCH_ERRNO_TABLES[e_machine]: + return _ARCH_ERRNO_TABLES[e_machine][err_num] + if e_machine != 0 and err_num in _GENERIC_ERRNO_OVERRIDES: + return _GENERIC_ERRNO_OVERRIDES[err_num] + try: + return errno.errorcode[err_num] + except KeyError: + return f"Unknown {nr} errno" + + +class SyscallAnalyzer: + """Analyzes failed syscalls and aggregates counts.""" + + def __init__(self, for_comm: Optional[str] = None, for_pid: Optional[int] = None): + self.for_comm = for_comm + self.for_pid = for_pid + self.session: Optional[perf.session] = None + self.syscalls: dict[tuple[str, int, int, int], int] = defaultdict(int) + machine = os.uname().machine + self.host_64_bit = ("64" in machine or machine in ("s390x", "alpha") + or sys.maxsize > 0xffffffff) + + def process_event(self, sample: perf.sample_event) -> None: + """Process raw_syscalls:sys_exit and syscalls:sys_exit events.""" + event_name = str(sample.evsel) + if "raw_syscalls:sys_exit" not in event_name and "syscalls:sys_exit" not in event_name: + return + + pid = sample.sample_tid + comm = "Unknown" + if hasattr(self, 'session') and self.session: + try: + thread = self.session.find_thread(sample.sample_pid, pid) + if thread: + comm = thread.comm() or "Unknown" + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError): + pass + + if self.for_comm is not None and comm != self.for_comm: + return + if self.for_pid is not None and pid != self.for_pid: + return + + ret = getattr(sample, "ret", 0) + is_64_bit = (getattr(self.session, "is_64_bit", self.host_64_bit) + if self.session else self.host_64_bit) + if ret > 0: + if not is_64_bit and 0xfffff000 <= ret <= 0xffffffff: + ret -= 0x100000000 + elif ret >= 0xfffffffffffff000: + ret -= 0x10000000000000000 + + if -4095 <= ret < 0: + syscall_id = getattr(sample, "__syscall_nr", -1) + if syscall_id < 0 or syscall_id > 0xffff: + syscall_id = getattr(sample, "nr", -1) + if syscall_id < 0 or syscall_id > 0xffff: + syscall_id = getattr(sample, "sys_id", -1) + if syscall_id < 0 or syscall_id > 0xffff: + syscall_id = getattr(sample, "id", -1) + + if 0 <= syscall_id <= 0xffff: + self.syscalls[(comm, pid, syscall_id, ret)] += 1 + + def print_summary(self) -> None: + """Print aggregated statistics.""" + if self.for_comm is not None: + print(f"\nsyscall errors for {self.for_comm}:\n") + elif self.for_pid is not None: + print(f"\nsyscall errors for PID {self.for_pid}:\n") + else: + print("\nsyscall errors:\n") + + print(f"{'comm [pid]':<30} {'count':>10}") + print(f"{'-' * 30:<30} {'-' * 10:>10}") + + sorted_keys = sorted( + self.syscalls.keys(), + key=lambda k: (k[0], k[1], k[2], -self.syscalls[k]) + ) + current_comm_pid = None + current_syscall = None + emach = getattr(self.session, "e_machine", 0) or 0 + for comm, pid, syscall_id, ret in sorted_keys: + if current_comm_pid != (comm, pid): + print(f"\n{comm} [{pid}]") + current_comm_pid = (comm, pid) + current_syscall = None + if current_syscall != syscall_id: + try: + if emach: + name = perf.syscall_name(syscall_id, emach) or str(syscall_id) + else: + name = perf.syscall_name(syscall_id) or str(syscall_id) + except AttributeError: + name = str(syscall_id) + print(f" syscall: {name:<16}") + current_syscall = syscall_id + err_str = strerror(ret, emach) + count = self.syscalls[(comm, pid, syscall_id, ret)] + print(f" err = {err_str:<20} {count:10d}") + + +if __name__ == "__main__": + ap = argparse.ArgumentParser( + description="Displays system-wide failed system call totals, " + "broken down by pid.") + ap.add_argument("-i", "--input", default="perf.data", + help="Input file name") + ap.add_argument("filter", nargs="?", help="COMM or PID to filter by") + args = ap.parse_args() + + F_COMM = None + F_PID = None + + if args.filter: + try: + F_PID = int(args.filter) + except ValueError: + F_COMM = args.filter + + analyzer = SyscallAnalyzer(F_COMM, F_PID) + + try: + print("Press control+C to stop and show the summary") + session = perf.session(perf.data(args.input), sample=analyzer.process_event) + analyzer.session = session + session.process_events() + analyzer.print_summary() + except KeyboardInterrupt: + analyzer.print_summary() + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError) as e: + print(f"Error processing events: {e}") + sys.exit(1) + finally: + analyzer.session = None
tools/perf/tests/shell/test_failed_syscalls_by_pid_python.sh new file +92 -0
92 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 24/49] perf python: Port failed-syscalls-by-pid to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines
Port
failed-syscalls-by-pid.pyto a standalone script intools/perf/python/using the perf module:
- Resolve architecture-aware syscall names usingperf.syscall_name(id, session.e_machine)instead of host python-audit tables, supporting cross-architecture perf.data files without external dependencies.
- Translate negative syscall return values into symbolic E* error names using errno.errorcode and architecture-specific session.e_machine overrides.
- Encapsulate aggregation in a SyscallAnalyzer class using collections.defaultdict and add argparse filtering by comm, PID, and -i/--input.
Quoted 5 lines
Add a shell test (
test_failed_syscalls_by_pid_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=24
Ian Rogers
reviewerPATCH 25/49 perf python: Port failed-syscalls from Perl to perf module
In reply to Ian RogersReplace the legacy Perl script failed-syscalls.pl with a standalone Python script in tools/perf/python/failed-syscalls.py using the perf Python module.
Improvements compared to the legacy Perl script:
- Remove the dependency on libperl and Perf::Trace::Util.
- Support both syscalls:sys_exit_* and raw_syscalls:sys_exit events (the legacy script only handled raw_syscalls::sys_exit).
- Use session.is_64_bit to distinguish 32-bit vs 64-bit unsigned error return ranges (0xfffff000..0xffffffff vs >= 0xfffffffffffff000) so valid 64-bit syscalls returning ~4GB values are not misclassified as 32-bit negative errors.
- Add argparse CLI support (-i/--input and optional comm filter) and full type annotations.
Add a shell test
(test_failed_syscalls_python.sh) to verify the standalone script.
--- tools/perf/python/failed-syscalls.py | 88 +++++++++++++++++++ .../shell/test_failed_syscalls_python.sh | 81 +++++++++++++++++ 2 files changed, 169 insertions(+) create mode 100755 tools/perf/python/failed-syscalls.py create mode 100755 tools/perf/tests/shell/test_failed_syscalls_python.sh
tools/perf/python/failed-syscalls.py new file +88 -0
@@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +"""Failed system call counts.""" +from __future__ import annotations + +import argparse +from collections import defaultdict +import sys +from typing import Optional +import perf + +class FailedSyscalls: + """Tracks and displays failed system call totals.""" + def __init__(self, comm: Optional[str] = None) -> None: + self.failed_syscalls: dict[str, int] = defaultdict(int) + self.for_comm = comm + self.session: Optional[perf.session] = None + + def process_event(self, sample: perf.sample_event) -> None: + """Process sys_exit events.""" + event_name = str(sample.evsel) + if not event_name.startswith("evsel(syscalls:sys_exit") and \ + not event_name.startswith("evsel(raw_syscalls:sys_exit"): + return + + try: + ret = sample.ret + except AttributeError: + print("ERROR: tracepoint fields missed", file=sys.stderr) + sys.exit(1) + + if ret > 0: + if ret >= 0xfffffffffffff000: # 64-bit negative errors + ret -= 0x10000000000000000 + elif 0xfffff000 <= ret <= 0xffffffff: # 32-bit negative errors + assert self.session is not None + if not self.session.is_64_bit: + ret -= 0x100000000 + + if ret >= 0: + return + + tid = sample.sample_tid + assert self.session is not None + try: + thread = self.session.find_thread(tid) + comm = (thread.comm() if thread else None) or "unknown" + except (TypeError, AttributeError): + # find_thread returns None when the thread isn't known. + comm = "unknown" + + if self.for_comm and comm != self.for_comm: + return + + self.failed_syscalls[comm] += 1 + + def print_totals(self) -> None: + """Print summary table.""" + print("\nfailed syscalls by comm:\n") + print(f"{'comm':<20s} {'# errors':>10s}") + print(f"{'-'*20} {'-'*10}") + + for comm, val in sorted(self.failed_syscalls.items(), + key=lambda kv: (kv[1], kv[0]), reverse=True): + print(f"{comm:<20s} {val:10d}") + + def run(self, input_file: str) -> None: + """Run the session.""" + self.session = perf.session(perf.data(input_file), sample=self.process_event) + self.session.process_events() + self.print_totals() + +def main() -> None: + """Main function.""" + parser = argparse.ArgumentParser(description="Trace failed syscalls") + parser.add_argument("comm", nargs="?", help="Filter by command name") + parser.add_argument("-i", "--input", default="perf.data", help="Input file") + args = parser.parse_args() + + analyzer = FailedSyscalls(args.comm) + try: + analyzer.run(args.input) + except IOError as e: + print(e, file=sys.stderr) + sys.exit(1) + +if __name__ == "__main__": + main()
tools/perf/tests/shell/test_failed_syscalls_python.sh new file +81 -0
81 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 25/49] perf python: Port failed-syscalls from Perl to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines
Replace the legacy Perl script failed-syscalls.pl with a standalone Python script in
tools/perf/python/failed-syscalls.pyusing the perf Python module.Improvements compared to the legacy Perl script:
- Remove the dependency on libperl andPerf::Trace::Util.
- Support both syscalls:sys_exit_* and raw_syscalls:sys_exit events (the legacy script only handledraw_syscalls::sys_exit).
- Use session.is_64_bit to distinguish 32-bit vs 64-bit unsigned error return ranges (0xfffff000..0xffffffff vs >=0xfffffffffffff000) so valid 64-bit syscalls returning ~4GB values are not misclassified as 32-bit negative errors.
- Add argparse CLI support (-i/--input and optional comm filter) and full type annotations.
Quoted 5 lines
Add a shell test
(test_failed_syscalls_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=25
Ian Rogers
reviewerPATCH 26/49 perf python: Port sctop to perf module
In reply to Ian RogersPort sctop.py from tools/perf/scripts/python/ to a standalone script in tools/perf/python/ using an SCTopAnalyzer class structure.
Improvements compared to the legacy script:
- Support both offline perf.data analysis (via perf.session, advancing display intervals deterministically using event timestamps) and live monitoring (via LiveSession with automatic tracepoint fallback from raw_syscalls:sys_enter to syscalls:sys_enter_*).
- Resolve architecture-aware syscall names via perf.syscall_name(id, session.e_machine) without requiring python-audit.
- Replace unsafe signal.SIGALRM dictionary mutation and os.popen("clear") subshell spawning with a synchronized threading.Lock / threading.Event timer and direct ANSI terminal escape sequences ('\x1b[2J\x1b[H').
Add a shell test (test_sctop_python.sh) to verify the standalone script.
--- tools/perf/python/sctop.py | 216 ++++++++++++++++++++ tools/perf/tests/shell/test_sctop_python.sh | 71 +++++++ 2 files changed, 287 insertions(+) create mode 100755 tools/perf/python/sctop.py create mode 100755 tools/perf/tests/shell/test_sctop_python.sh
tools/perf/python/sctop.py new file +216 -0
@@ -0,0 +1,216 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +System call top + +Periodically displays system-wide system call totals, broken down by +syscall. If a [comm] arg is specified, only syscalls called by +[comm] are displayed. If an [interval] arg is specified, the display +will be refreshed every [interval] seconds. The default interval is +3 seconds. + +Ported from tools/perf/scripts/python/sctop.py +""" +from __future__ import annotations + +import argparse +from collections import defaultdict +import sys +import threading +from typing import Optional +import perf +from perf_live import LiveSession + + + + +class SCTopAnalyzer: + """Periodically displays system-wide system call totals.""" + + def __init__(self, for_comm: Optional[str], interval: int, offline: bool = False): + self.for_comm = for_comm + self.interval = interval + self.syscalls: dict[int, int] = defaultdict(int) + self.comm_cache: dict[int, str] = {} + self.lock = threading.Lock() + self.stop_event = threading.Event() + self.thread = threading.Thread(target=self.print_syscall_totals) + self.offline = offline + self.last_print_time: Optional[int] = None + self.session: Optional[perf.session] = None + self.e_machine: Optional[int] = None + + def syscall_name(self, syscall_id: int) -> str: + """Lookup syscall name by ID.""" + try: + e_machine = getattr(self.session, "e_machine", self.e_machine) + if e_machine is not None: + name = perf.syscall_name(syscall_id, e_machine) + else: + name = perf.syscall_name(syscall_id) + if name is not None: + return name + except (TypeError, OverflowError): + pass + return str(syscall_id) + + def process_event(self, sample: perf.sample_event) -> None: + """Collect syscall events.""" + name = str(sample.evsel) + syscall_id = getattr(sample, "id", -1) + if syscall_id < 0 or syscall_id > 0xffff: + syscall_id = getattr(sample, "__syscall_nr", -1) + if syscall_id < 0 or syscall_id > 0xffff: + syscall_id = getattr(sample, "nr", -1) + + skip = False + with self.lock: + if self.for_comm is not None: + comm = "Unknown" + if sample.sample_pid in self.comm_cache: + comm = self.comm_cache[sample.sample_pid] + elif hasattr(self, 'session') and self.session: + try: + proc = self.session.find_thread(sample.sample_pid) + if proc: + comm = proc.comm() or "Unknown" + except TypeError: + pass + else: + try: + with open(f"/proc/{sample.sample_pid}/comm", "r", + encoding="utf-8", errors="replace") as f: + comm = f.read().strip() + except OSError: + comm = "Unknown" + self.comm_cache[sample.sample_pid] = comm + + if comm != self.for_comm: + skip = True + + is_enter = (name.startswith("evsel(raw_syscalls:sys_enter") or + name.startswith("evsel(syscalls:sys_enter")) + if not skip and is_enter and 0 <= syscall_id <= 0xffff: + self.syscalls[syscall_id] += 1 + + + + if self.offline and hasattr(sample, "sample_time"): + interval_ns = self.interval * (10 ** 9) + if self.last_print_time is None: + self.last_print_time = sample.sample_time + elif sample.sample_time - self.last_print_time >= interval_ns: + self.print_current_totals() + self.last_print_time = sample.sample_time + + def print_current_totals(self): + """Print current syscall totals.""" + # Clear terminal + if not self.offline: + print("\x1b[2J\x1b[H", end="") + else: + print() + + with self.lock: + for_comm = self.for_comm + if for_comm is not None: + print(f"\nsyscall events for {for_comm}:\n") + else: + print("\nsyscall events:\n") + + print(f"{'event':40s} {'count':10s}") + print(f"{'-' * 40:40s} {'-' * 10:10s}") + + with self.lock: + current_syscalls = list(self.syscalls.items()) + self.syscalls.clear() + if len(self.comm_cache) > 4096: + self.comm_cache.clear() + + current_syscalls.sort(key=lambda kv: (kv[1], kv[0]), reverse=True) + + for syscall_id, val in current_syscalls: + print(f"{self.syscall_name(syscall_id):<40s} {val:10d}") + + def print_syscall_totals(self): + """Periodically print syscall totals.""" + while not self.stop_event.is_set(): + self.print_current_totals() + self.stop_event.wait(self.interval) + # Print final batch + self.print_current_totals() + + def start(self): + """Start the background thread.""" + self.thread.start() + + def stop(self): + """Stop the background thread.""" + self.stop_event.set() + self.thread.join() + + +def main(): + """Main function.""" + ap = argparse.ArgumentParser(description="System call top") + ap.add_argument("args", nargs="*", help="[comm] [interval] or [interval]") + ap.add_argument("-i", "--input", help="Input file name") + args = ap.parse_args() + + for_comm = None + default_interval = 3 + interval = default_interval + + if len(args.args) > 2: + print("Usage: python sctop.py [comm] [interval]") + sys.exit(1) + + if len(args.args) > 1: + for_comm = args.args[0] + try: + interval = int(args.args[1]) + except ValueError: + print(f"Invalid interval: {args.args[1]}") + sys.exit(1) + elif len(args.args) > 0: + try: + interval = int(args.args[0]) + except ValueError: + for_comm = args.args[0] + interval = default_interval + + analyzer = SCTopAnalyzer(for_comm, interval, offline=bool(args.input)) + + if not args.input: + analyzer.start() + + try: + if args.input: + session = perf.session(perf.data(args.input), sample=analyzer.process_event) + analyzer.session = session + session.process_events() + analyzer.e_machine = getattr(session, "e_machine", None) + else: + try: + live_session = LiveSession( + "raw_syscalls:sys_enter", sample_callback=analyzer.process_event + ) + except OSError: + live_session = LiveSession( + "syscalls:sys_enter_*", sample_callback=analyzer.process_event + ) + live_session.run() + except KeyboardInterrupt: + pass + except IOError as e: + print(f"Error: {e}") + finally: + if args.input: + analyzer.print_current_totals() + analyzer.session = None + else: + analyzer.stop() + + +if __name__ == "__main__": + main()
tools/perf/tests/shell/test_sctop_python.sh new file +71 -0
71 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 26/49] perf python: Port sctop to perf module
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf: sctop silently drops x32 ABI syscalls due to 0xffff limit --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf: sctop silently drops x32 ABI syscalls due to 0xffff limit --
commit 1ef6e01194ef0771e0ca86661282a11b36de647e
Author: Ian Rogers <[email protected]>
perf python: Port sctop to perf module
This commit ports sctop.py to a standalone script in tools/perf/python/, adding support for offline perf.data analysis and live monitoring. It also updates the script to resolve architecture-aware syscall names and replaces unsafe signal handlers with threading synchronization.
Quoted 5 lines
tools/perf/python/sctop.py new file +0
-0
[ ... ]
Quoted 11 lines
tools/perf/python/sctop.py new file +11
-0+ def process_event(self, sample: perf.sample_event) -> None: + """Collect syscall events.""" + name = str(sample.evsel) + syscall_id = getattr(sample, "id", -1) + if syscall_id < 0 or syscall_id > 0xffff: + syscall_id = getattr(sample, "__syscall_nr", -1) + if syscall_id < 0 or syscall_id > 0xffff: + syscall_id = getattr(sample, "nr", -1) + + skip = False + with self.lock:
[ ... ]
tools/perf/python/sctop.py line 94
Quoted 4 lines
tools/perf/python/sctop.py new file +4
-0+ is_enter = (name.startswith("evsel(raw_syscalls:sys_enter") or + name.startswith("evsel(syscalls:sys_enter")) + if not skip and is_enter and 0 <= syscall_id <= 0xffff: + self.syscalls[syscall_id] += 1
[Severity: Medium]
Does this condition unintentionally filter out legitimate x32 ABI syscalls on x86_64?
When processing trace events from an x32 application, the syscall ID from raw_syscalls:sys_enter includes the __X32_SYSCALL_BIT (0x40000000). Because 0x40000000 exceeds 0xffff, the script falls back through the attributes and eventually skips incrementing the counter for that syscall. This causes x32 ABI syscalls to be silently dropped from the output.
The legacy sctop.py script correctly stored and printed these IDs without an artificial cap. Can this upper bound be removed or adjusted in SCTopAnalyzer.process_event to accommodate x32 syscalls?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=26
Ian Rogers
reviewerPATCH 27/49 perf python: Port rw-by-file from Perl to perf module
In reply to Ian RogersReplace the legacy Perl script rw-by-file.pl with a standalone Python script in tools/perf/python/rw-by-file.py using the perf Python module.
Improvements compared to the legacy Perl script:
- Remove the dependency on libperl and Perf::Trace::Util.
- Encapsulate per-file-descriptor read/write byte and call count aggregation in an RwByFile class using perf.session and resolve thread command names via session.find_thread(pid, sample_tid).
- Add argparse CLI support (-i/--input and target program filter) and full type annotations.
Add a shell test
(test_rw_by_file_python.sh) to verify the standalone script.
--- tools/perf/python/rw-by-file.py | 109 ++++++++++++++++++ .../tests/shell/test_rw_by_file_python.sh | 70 +++++++++++ 2 files changed, 179 insertions(+) create mode 100755 tools/perf/python/rw-by-file.py create mode 100755 tools/perf/tests/shell/test_rw_by_file_python.sh
tools/perf/python/rw-by-file.py new file +109 -0
@@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +"""Display r/w activity for files read/written to for a given program.""" +from __future__ import annotations + +import argparse +from collections import defaultdict +import sys +from typing import Optional, Dict +import perf + +class RwByFile: + """Tracks and displays read/write activity by file descriptor.""" + def __init__(self, comm: str) -> None: + self.for_comm = comm + self.reads: Dict[int, Dict[str, int]] = defaultdict( + lambda: {"bytes_requested": 0, "total_reads": 0} + ) + self.writes: Dict[int, Dict[str, int]] = defaultdict( + lambda: {"bytes_written": 0, "total_writes": 0} + ) + self.unhandled: Dict[str, int] = defaultdict(int) + self.session: Optional[perf.session] = None + + def process_event(self, sample: perf.sample_event) -> None: + """Process events.""" + raw_name = str(sample.evsel) + event_name = raw_name[6:-1] if raw_name.startswith("evsel(") else raw_name + + pid = sample.sample_pid + assert self.session is not None + try: + thread = self.session.find_thread(pid, sample.sample_tid) + comm = (thread.comm() if thread else None) or "unknown" + except (TypeError, AttributeError): + comm = "unknown" + + if comm != self.for_comm: + return + + if event_name == "syscalls:sys_enter_read": + try: + fd = sample.fd + count = sample.count + self.reads[fd]["bytes_requested"] += count + self.reads[fd]["total_reads"] += 1 + except AttributeError: + self.unhandled[event_name] += 1 + elif event_name == "syscalls:sys_enter_write": + try: + fd = sample.fd + count = sample.count + self.writes[fd]["bytes_written"] += count + self.writes[fd]["total_writes"] += 1 + except AttributeError: + self.unhandled[event_name] += 1 + else: + self.unhandled[event_name] += 1 + + def print_totals(self) -> None: + """Print summary tables.""" + print(f"file read counts for {self.for_comm}:\n") + print(f"{'fd':>6s} {'# reads':>10s} {'bytes_requested':>15s}") + print(f"{'-'*6} {'-'*10} {'-'*15}") + + for fd, data in sorted(self.reads.items(), + key=lambda kv: kv[1]["bytes_requested"], reverse=True): + print(f"{fd:6d} {data['total_reads']:10d} {data['bytes_requested']:15d}") + + print(f"\nfile write counts for {self.for_comm}:\n") + print(f"{'fd':>6s} {'# writes':>10s} {'bytes_written':>15s}") + print(f"{'-'*6} {'-'*10} {'-'*15}") + + for fd, data in sorted(self.writes.items(), + key=lambda kv: kv[1]["bytes_written"], reverse=True): + print(f"{fd:6d} {data['total_writes']:10d} {data['bytes_written']:15d}") + + if self.unhandled: + print("\nunhandled events:\n") + print(f"{'event':<40s} {'count':>10s}") + print(f"{'-'*40} {'-'*10}") + for event_name, count in self.unhandled.items(): + print(f"{event_name:<40s} {count:10d}") + + def run(self, input_file: str) -> None: + """Run the session.""" + self.session = perf.session(perf.data(input_file), sample=self.process_event) + try: + self.session.process_events() + finally: + self.session = None + self.print_totals() + +def main() -> None: + """Main function.""" + parser = argparse.ArgumentParser(description="Trace r/w activity by file") + parser.add_argument("comm", help="Filter by command name") + parser.add_argument("-i", "--input", default="perf.data", help="Input file") + args = parser.parse_args() + + analyzer = RwByFile(args.comm) + try: + analyzer.run(args.input) + except IOError as e: + print(e, file=sys.stderr) + sys.exit(1) + +if __name__ == "__main__": + main()
tools/perf/tests/shell/test_rw_by_file_python.sh new file +70 -0
70 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 27/49] perf python: Port rw-by-file from Perl to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Replace the legacy Perl script rw-by-file.pl with a standalone Python script in
tools/perf/python/rw-by-file.pyusing the perf Python module.Improvements compared to the legacy Perl script:
- Remove the dependency on libperl andPerf::Trace::Util.
- Encapsulate per-file-descriptor read/write byte and call count aggregation in an RwByFile class using perf.session and resolve thread command names viasession.find_thread(pid, sample_tid).
- Add argparse CLI support (-i/--input and target program filter) and full type annotations.
Quoted 5 lines
Add a shell test
(test_rw_by_file_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=27
Ian Rogers
reviewerPATCH 28/49 perf python: Port rw-by-pid from Perl to perf module
In reply to Ian RogersReplace the legacy Perl script rw-by-pid.pl with a standalone Python script in tools/perf/python/rw-by-pid.py using the perf Python module.
Improvements compared to the legacy Perl script:
- Remove the dependency on libperl and Perf::Trace::Util.
- Convert unsigned 32-bit and 64-bit error return values (0xfffff000..0xffffffff and >= 0x8000000000000000) to negative errnos so failed read/write syscalls are recorded in the error table rather than inflating bytes_read / bytes_written.
- Add argparse CLI support (-i/--input) and full type annotations.
Add a shell test
(test_rw_by_pid_python.sh) to verify the standalone script.
--- tools/perf/python/rw-by-pid.py | 189 ++++++++++++++++++ .../perf/tests/shell/test_rw_by_pid_python.sh | 77 +++++++ 2 files changed, 266 insertions(+) create mode 100755 tools/perf/python/rw-by-pid.py create mode 100755 tools/perf/tests/shell/test_rw_by_pid_python.sh
tools/perf/python/rw-by-pid.py new file +189 -0
@@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +"""Display r/w activity for all processes.""" +from __future__ import annotations + +import argparse +from collections import defaultdict +import sys +from typing import Optional, Dict, List, Tuple, Any +import perf + +class RwByPid: + """Tracks and displays read/write activity by PID.""" + def __init__(self) -> None: + self.reads: Dict[int, Dict[str, Any]] = defaultdict( + lambda: { + "bytes_requested": 0, + "bytes_read": 0, + "total_reads": 0, + "comm": "", + "errors": defaultdict(int), + } + ) + self.writes: Dict[int, Dict[str, Any]] = defaultdict( + lambda: { + "bytes_requested": 0, + "bytes_written": 0, + "total_writes": 0, + "comm": "", + "errors": defaultdict(int), + } + ) + self.unhandled: Dict[str, int] = defaultdict(int) + self.session: Optional[perf.session] = None + + def _handle_sys_enter_read(self, pid: int, comm: str, sample: perf.sample_event) -> None: + try: + count = sample.count + self.reads[pid]["bytes_requested"] += count + self.reads[pid]["total_reads"] += 1 + self.reads[pid]["comm"] = comm + except AttributeError: + self.unhandled["sys_enter_read_attr_err"] += 1 + + def _handle_sys_exit_read(self, pid: int, sample: perf.sample_event) -> None: + try: + ret = sample.ret + if ret >= 0x8000000000000000: + ret -= 0x10000000000000000 + elif 0xfffff000 <= ret <= 0xffffffff: + ret -= 0x100000000 + if ret > 0: + self.reads[pid]["bytes_read"] += ret + else: + self.reads[pid]["errors"][ret] += 1 + except AttributeError: + self.unhandled["sys_exit_read_attr_err"] += 1 + + def _handle_sys_enter_write(self, pid: int, comm: str, sample: perf.sample_event) -> None: + try: + count = sample.count + self.writes[pid]["bytes_requested"] += count + self.writes[pid]["total_writes"] += 1 + self.writes[pid]["comm"] = comm + except AttributeError: + self.unhandled["sys_enter_write_attr_err"] += 1 + + def _handle_sys_exit_write(self, pid: int, sample: perf.sample_event) -> None: + try: + ret = sample.ret + if ret >= 0x8000000000000000: + ret -= 0x10000000000000000 + elif 0xfffff000 <= ret <= 0xffffffff: + ret -= 0x100000000 + if ret > 0: + self.writes[pid]["bytes_written"] += ret + else: + self.writes[pid]["errors"][ret] += 1 + except AttributeError: + self.unhandled["sys_exit_write_attr_err"] += 1 + + def process_event(self, sample: perf.sample_event) -> None: + """Process events.""" + event_name = str(sample.evsel)[6:-1] + pid = sample.sample_pid + + assert self.session is not None + try: + thread = self.session.find_thread(pid) + comm = (thread.comm() if thread else None) or "unknown" + except (TypeError, AttributeError): + comm = "unknown" + + if event_name == "syscalls:sys_enter_read": + self._handle_sys_enter_read(pid, comm, sample) + elif event_name == "syscalls:sys_exit_read": + self._handle_sys_exit_read(pid, sample) + elif event_name == "syscalls:sys_enter_write": + self._handle_sys_enter_write(pid, comm, sample) + elif event_name == "syscalls:sys_exit_write": + self._handle_sys_exit_write(pid, sample) + else: + self.unhandled[event_name] += 1 + + def print_totals(self) -> None: + """Print summary tables.""" + print("read counts by pid:\n") + print( + f"{'pid':>6s} {'comm':<20s} {'# reads':>10s} " + f"{'bytes_requested':>15s} {'bytes_read':>10s}" + ) + print(f"{'-'*6} {'-'*20} {'-'*10} {'-'*15} {'-'*10}") + + for pid, data in sorted(self.reads.items(), + key=lambda kv: kv[1]["bytes_read"], reverse=True): + print( + f"{pid:6d} {data['comm']:<20s} {data['total_reads']:10d} " + f"{data['bytes_requested']:15d} {data['bytes_read']:10d}" + ) + + print("\nfailed reads by pid:\n") + print(f"{'pid':>6s} {'comm':<20s} {'error #':>6s} {'# errors':>10s}") + print(f"{'-'*6} {'-'*20} {'-'*6} {'-'*10}") + + errcounts: List[Tuple[int, str, int, int]] = [] + for pid, data in self.reads.items(): + for error, count in data["errors"].items(): + errcounts.append((pid, data["comm"], error, count)) + + for pid, comm, error, count in sorted(errcounts, key=lambda x: x[3], reverse=True): + print(f"{pid:6d} {comm:<20s} {error:6d} {count:10d}") + + print("\nwrite counts by pid:\n") + print( + f"{'pid':>6s} {'comm':<20s} {'# writes':>10s} " + f"{'bytes_requested':>15s} {'bytes_written':>15s}" + ) + print(f"{'-'*6} {'-'*20} {'-'*10} {'-'*15} {'-'*15}") + + for pid, data in sorted(self.writes.items(), + key=lambda kv: kv[1]["bytes_written"], reverse=True): + print( + f"{pid:6d} {data['comm']:<20s} {data['total_writes']:10d} " + f"{data['bytes_requested']:15d} {data['bytes_written']:15d}" + ) + + print("\nfailed writes by pid:\n") + print(f"{'pid':>6s} {'comm':<20s} {'error #':>6s} {'# errors':>10s}") + print(f"{'-'*6} {'-'*20} {'-'*6} {'-'*10}") + + errcounts = [] + for pid, data in self.writes.items(): + for error, count in data["errors"].items(): + errcounts.append((pid, data["comm"], error, count)) + + for pid, comm, error, count in sorted(errcounts, key=lambda x: x[3], reverse=True): + print(f"{pid:6d} {comm:<20s} {error:6d} {count:10d}") + + if self.unhandled: + print("\nunhandled events:\n") + print(f"{'event':<40s} {'count':>10s}") + print(f"{'-'*40} {'-'*10}") + for event_name, count in self.unhandled.items(): + print(f"{event_name:<40s} {count:10d}") + + def run(self, input_file: str) -> None: + """Run the session.""" + self.session = perf.session(perf.data(input_file), sample=self.process_event) + try: + self.session.process_events() + finally: + self.session = None + self.print_totals() + +def main() -> None: + """Main function.""" + parser = argparse.ArgumentParser(description="Trace r/w activity by PID") + parser.add_argument("-i", "--input", default="perf.data", help="Input file") + args = parser.parse_args() + + analyzer = RwByPid() + try: + analyzer.run(args.input) + except IOError as e: + print(e, file=sys.stderr) + sys.exit(1) + +if __name__ == "__main__": + main()
tools/perf/tests/shell/test_rw_by_pid_python.sh new file +77 -0
77 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 28/49] perf python: Port rw-by-pid from Perl to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Replace the legacy Perl script rw-by-pid.pl with a standalone Python script in
tools/perf/python/rw-by-pid.pyusing the perf Python module.Improvements compared to the legacy Perl script:
- Remove the dependency on libperl andPerf::Trace::Util.
- Convert unsigned 32-bit and 64-bit error return values (0xfffff000..0xffffffff and >=0x8000000000000000) to negative errnos so failed read/write syscalls are recorded in the error table rather than inflatingbytes_read/bytes_written.
- Add argparse CLI support (-i/--input) and full type annotations.
Quoted 5 lines
Add a shell test
(test_rw_by_pid_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=28
Ian Rogers
reviewerPATCH 29/49 perf python: Port rwtop from Perl to perf module
In reply to Ian RogersReplace the legacy Perl script rwtop.pl with a standalone Python script in tools/perf/python/rwtop.py using the perf Python module.
Improvements compared to the legacy Perl script:
- Remove the dependency on libperl and Perf::Trace::Util.
- Support both offline perf.data files (via perf.session) and live recording (via LiveSession with --live), driving periodic interval summaries from event timestamps rather than wall-clock SIGALRM timers so both live and offline runs produce deterministic output.
- Use session.is_64_bit (falling back to host bitness in live mode) when checking for negative syscall return values so 64-bit byte counts in 0xfffff000..0xffffffff are not misclassified as 32-bit errors.
- Sanitize non-printable characters in /proc/<pid>/comm to prevent terminal control sequence injection.
Add a shell test
(test_rwtop_python.sh) to verify the standalone script.
--- tools/perf/python/rwtop.py | 247 ++++++++++++++++++++ tools/perf/tests/shell/test_rwtop_python.sh | 76 ++++++ 2 files changed, 323 insertions(+) create mode 100755 tools/perf/python/rwtop.py create mode 100755 tools/perf/tests/shell/test_rwtop_python.sh
tools/perf/python/rwtop.py new file +247 -0
@@ -0,0 +1,247 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +"""Periodically displays system-wide r/w call activity, broken down by pid.""" +from __future__ import annotations + +import argparse +from collections import defaultdict +import os +import sys +from typing import Optional, Dict, Any +import perf +from perf_live import LiveSession + +class RwTop: + """Periodically displays system-wide r/w call activity.""" + def __init__(self, interval: int = 3, nlines: int = 20) -> None: + self.offline = False + self.interval_ns = interval * 1000000000 + self.nlines = nlines + self.reads: Dict[int, Dict[str, Any]] = defaultdict( + lambda: { + "bytes_requested": 0, + "bytes_read": 0, + "total_reads": 0, + "comm": "", + "errors": defaultdict(int), + } + ) + self.writes: Dict[int, Dict[str, Any]] = defaultdict( + lambda: { + "bytes_requested": 0, + "bytes_written": 0, + "total_writes": 0, + "comm": "", + "errors": defaultdict(int), + } + ) + self.unhandled: Dict[str, int] = defaultdict(int) + self.comm_cache: Dict[int, str] = {} + self.session: Optional[perf.session] = None + self.last_print_time: int = 0 + machine = os.uname().machine + self.host_64_bit = ("64" in machine or machine in ("s390x", "alpha") + or sys.maxsize > 0xffffffff) + + def get_comm(self, pid: int) -> str: + """Resolve and cache the comm(and) of a pid.""" + comm = self.comm_cache.get(pid) + if comm: + return comm + comm = None + try: + if self.session: + thread = self.session.find_thread(pid) + comm = thread.comm() if thread else None + else: + with open(f"/proc/{pid}/comm", "r", encoding="utf-8", errors="replace") as f: + comm = f.read().strip() + except (TypeError, AttributeError): + # find_thread returns None when the thread isn't known. + pass + except OSError: + # The thread may have exited before /proc could be read. + pass + if not comm: + comm = f"PID({pid})" + comm = ''.join(c if c.isprintable() else '?' for c in comm) + self.comm_cache[pid] = comm + return comm + + def process_event(self, sample: perf.sample_event) -> None: + """Process events.""" + event_name = str(sample.evsel) + pid = sample.sample_pid + sample_time = sample.sample_time + + if self.last_print_time == 0: + self.last_print_time = sample_time + + # Check if interval has passed + if sample_time > self.last_print_time and sample_time - self.last_print_time >= self.interval_ns: + self.print_totals() + self.last_print_time = sample_time + + # Map each event onto the totals it updates. "enter" events count the + # requested bytes, "exit" events the transferred bytes or the error. + handlers = { + "evsel(syscalls:sys_enter_read)": (self.reads, "total_reads", None), + "evsel(syscalls:sys_exit_read)": (self.reads, None, "bytes_read"), + "evsel(syscalls:sys_enter_write)": (self.writes, "total_writes", None), + "evsel(syscalls:sys_exit_write)": (self.writes, None, "bytes_written"), + } + handler = handlers.get(event_name) + if not handler: + self.unhandled[event_name] += 1 + return + + totals, count_key, bytes_key = handler + try: + value = sample.count if count_key else sample.ret + except AttributeError: + self.unhandled[event_name] += 1 + return + + data = totals[pid] + data["comm"] = self.get_comm(pid) + if count_key: + data["bytes_requested"] += value + data[count_key] += 1 + elif bytes_key: + is_64_bit = (getattr(self.session, "is_64_bit", self.host_64_bit) + if self.session else self.host_64_bit) + if value > 0: + if not is_64_bit and 0xfffff000 <= value <= 0xffffffff: + value -= 0x100000000 + elif value >= 0xfffffffffffff000: + value -= 0x10000000000000000 + if value >= 0: + data[bytes_key] += value + else: + data["errors"][value] += 1 + + def print_totals(self) -> None: + """Print summary tables.""" + if not self.offline: + print('\x1b[H\x1b[2J', end='') + print("read counts by pid:\n") + print( + f"{'pid':>6s} {'comm':<20s} {'# reads':>10s} " + f"{'bytes_req':>10s} {'bytes_read':>10s}" + ) + print(f"{'-'*6} {'-'*20} {'-'*10} {'-'*10} {'-'*10}") + + count = 0 + for pid, data in sorted(self.reads.items(), + key=lambda kv: kv[1]["bytes_read"], reverse=True): + print( + f"{pid:6d} {data['comm']:<20s} {data['total_reads']:10d} " + f"{data['bytes_requested']:10d} {data['bytes_read']:10d}" + ) + count += 1 + if count >= self.nlines: + break + + print("\nfailed reads by pid:\n") + print(f"{'pid':>6s} {'comm':<20s} {'error #':>6s} {'# errors':>10s}") + print(f"{'-'*6} {'-'*20} {'-'*6} {'-'*10}") + + errcounts = [] + for pid, data in self.reads.items(): + for error, cnt in data["errors"].items(): + errcounts.append((pid, data["comm"], error, cnt)) + + sorted_errcounts = sorted(errcounts, key=lambda x: x[3], reverse=True) + for pid, comm, error, cnt in sorted_errcounts[:self.nlines]: + print(f"{pid:6d} {comm:<20s} {error:6d} {cnt:10d}") + + print("\nwrite counts by pid:\n") + print( + f"{'pid':>6s} {'comm':<20s} {'# writes':>10s} " + f"{'bytes_req':>10s} {'bytes_written':>13s}" + ) + print(f"{'-'*6} {'-'*20} {'-'*10} {'-'*10} {'-'*13}") + + count = 0 + for pid, data in sorted(self.writes.items(), + key=lambda kv: kv[1]["bytes_written"], reverse=True): + print( + f"{pid:6d} {data['comm']:<20s} {data['total_writes']:10d} " + f"{data['bytes_requested']:10d} {data['bytes_written']:13d}" + ) + count += 1 + if count >= self.nlines: + break + + print("\nfailed writes by pid:\n") + print(f"{'pid':>6s} {'comm':<20s} {'error #':>6s} {'# errors':>10s}") + print(f"{'-'*6} {'-'*20} {'-'*6} {'-'*10}") + + errcounts = [] + for pid, data in self.writes.items(): + for error, cnt in data["errors"].items(): + errcounts.append((pid, data["comm"], error, cnt)) + + sorted_errcounts = sorted(errcounts, key=lambda x: x[3], reverse=True) + for pid, comm, error, cnt in sorted_errcounts[:self.nlines]: + print(f"{pid:6d} {comm:<20s} {error:6d} {cnt:10d}") + + # Reset counts + self.reads.clear() + self.writes.clear() + self.comm_cache.clear() + + def run(self, input_file: str) -> None: + """Run the session.""" + self.session = perf.session(perf.data(input_file), sample=self.process_event) + try: + self.session.process_events() + finally: + self.session = None + + # Print final totals if there are any left + if self.reads or self.writes: + self.print_totals() + + if self.unhandled: + print("\nunhandled events:\n") + print(f"{'event':<40s} {'count':>10s}") + print(f"{'-'*40} {'-'*10}") + for event_name, count in self.unhandled.items(): + print(f"{event_name:<40s} {count:10d}") + +def main() -> None: + """Main function.""" + parser = argparse.ArgumentParser(description="Trace r/w activity by PID") + parser.add_argument( + "interval", type=int, nargs="?", default=3, help="Refresh interval in seconds" + ) + parser.add_argument("-i", "--input", default="perf.data", help="Input file") + parser.add_argument("-l", "--live", action="store_true", help="Run in live mode") + args = parser.parse_args() + + analyzer = RwTop(args.interval) + try: + if args.live or (not os.path.exists(args.input) and args.input == "perf.data"): + # Live mode + events = ( + "syscalls:sys_enter_read,syscalls:sys_exit_read," + "syscalls:sys_enter_write,syscalls:sys_exit_write" + ) + live_session = LiveSession(events, sample_callback=analyzer.process_event) + print("Live mode started. Press Ctrl+C to stop.", file=sys.stderr) + live_session.run() + else: + analyzer.offline = True + analyzer.run(args.input) + except IOError as e: + print(e, file=sys.stderr) + sys.exit(1) + except KeyboardInterrupt: + if not analyzer.offline: + print("\nStopping live mode...", file=sys.stderr) + if analyzer.reads or analyzer.writes: + analyzer.print_totals() + +if __name__ == "__main__": + main()
tools/perf/tests/shell/test_rwtop_python.sh new file +76 -0
76 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 29/49] perf python: Port rwtop from Perl to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines
Replace the legacy Perl script rwtop.pl with a standalone Python script in
tools/perf/python/rwtop.pyusing the perf Python module.Improvements compared to the legacy Perl script:
- Remove the dependency on libperl andPerf::Trace::Util.
- Support both offline perf.data files (via perf.session) and live recording (via LiveSession with --live), driving periodic interval summaries from event timestamps rather than wall-clock SIGALRM timers so both live and offline runs produce deterministic output.
- Use session.is_64_bit (falling back to host bitness in live mode) when checking for negative syscall return values so 64-bit byte counts in 0xfffff000..0xffffffff are not misclassified as 32-bit errors.
- Sanitize non-printable characters in /proc/<pid>/comm to prevent terminal control sequence injection.
Quoted 5 lines
Add a shell test
(test_rwtop_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=29
Ian Rogers
reviewerPATCH 30/49 perf python: Port futex-contention to perf module
In reply to Ian RogersPort tools/perf/scripts/python/futex-contention.py to a standalone script in tools/perf/python/ using the perf module. Avoiding the embedded interpreter overhead improves execution speed by ~3.2x:
``` $ perf record -e syscalls:sys_*_futex -a sleep 1 ... $ time perf script tools/perf/scripts/python/futex-contention.py ... real 0m1.007s user 0m0.935s sys 0m0.072s $ time python3 tools/perf/python/futex-contention.py ... real 0m0.314s user 0m0.259s sys 0m0.056s ```
Additional improvements compared to the legacy script:
- Consolidate per-(tid, uaddr) contention count, total_time, min_time, and max_time into a single LockStats class instead of maintaining three separate dictionaries.
- Validate that uaddr and op tracepoint attributes are present on syscalls:sys_enter_futex samples rather than silently attributing missing fields to uaddr=0, op=0 (FUTEX_WAIT).
- Add -i/--input CLI option support via argparse and full type annotations.
Add a shell test (test_futex_contention_python.sh) to verify the standalone script.
--- tools/perf/python/futex-contention.py | 93 +++++++++++++++++++ .../shell/test_futex_contention_python.sh | 67 +++++++++++++ 2 files changed, 160 insertions(+) create mode 100755 tools/perf/python/futex-contention.py create mode 100755 tools/perf/tests/shell/test_futex_contention_python.sh
tools/perf/python/futex-contention.py new file +93 -0
@@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +"""Measures futex contention.""" +from __future__ import annotations + +import argparse +from collections import defaultdict +from typing import Dict, Tuple +import perf + +class LockStats: + """Aggregate lock contention information.""" + def __init__(self) -> None: + self.count = 0 + self.total_time = 0 + self.min_time = 0 + self.max_time = 0 + + def add(self, duration: int) -> None: + """Add a new duration measurement.""" + self.count += 1 + self.total_time += duration + if self.count == 1: + self.min_time = duration + self.max_time = duration + else: + self.min_time = min(self.min_time, duration) + self.max_time = max(self.max_time, duration) + + def avg(self) -> float: + """Return average duration.""" + return self.total_time / self.count if self.count > 0 else 0.0 + +process_names: Dict[int, str] = {} +start_times: Dict[int, Tuple[int, int]] = {} +session = None +durations: Dict[Tuple[int, int], LockStats] = defaultdict(LockStats) + +FUTEX_WAIT = 0 +FUTEX_PRIVATE_FLAG = 128 +FUTEX_CLOCK_REALTIME = 256 +FUTEX_CMD_MASK = ~(FUTEX_PRIVATE_FLAG | FUTEX_CLOCK_REALTIME) + + +def handle_start(tid: int, uaddr: int, op: int, start_time: int) -> None: + """Handle a futex sys_enter event.""" + if (op & FUTEX_CMD_MASK) != FUTEX_WAIT: + return + try: + if session: + process = session.find_thread(tid) + if process: + process_names[tid] = process.comm() or "unknown" + except (TypeError, AttributeError): + pass + if tid not in process_names: + process_names[tid] = "unknown" + + start_times[tid] = (uaddr, start_time) + +def handle_end(tid: int, end_time: int) -> None: + """Handle a futex sys_exit event.""" + if tid not in start_times: + return + (uaddr, start_time) = start_times[tid] + del start_times[tid] + durations[(tid, uaddr)].add(end_time - start_time) + +def process_event(sample: perf.sample_event) -> None: + """Process a single sample event.""" + event_name = str(sample.evsel) + if event_name.startswith("evsel(syscalls:sys_enter_futex)"): + uaddr = getattr(sample, "uaddr", None) + op = getattr(sample, "op", None) + if uaddr is None or op is None: + return # Tracepoint fields missing, skip silent attribution to 0 + handle_start(sample.sample_tid, uaddr, op, sample.sample_time) + elif event_name.startswith("evsel(syscalls:sys_exit_futex)"): + handle_end(sample.sample_tid, sample.sample_time) + + +if __name__ == "__main__": + ap = argparse.ArgumentParser(description="Measure futex contention") + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + args = ap.parse_args() + + session = perf.session(perf.data(args.input), sample=process_event) + session.process_events() + + for ((t, u), stats) in sorted(durations.items()): + avg_ns = stats.avg() + print(f"{process_names.get(t, 'unknown')}[{t}] lock {u:x} contended {stats.count} times, " + f"{avg_ns:.0f} avg ns [max: {stats.max_time} ns, min {stats.min_time} ns]")
tools/perf/tests/shell/test_futex_contention_python.sh new file +67 -0
67 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 30/49] perf python: Port futex-contention to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
Port
tools/perf/scripts/python/futex-contention.pyto a standalone script intools/perf/python/using the perf module. Avoiding the embedded interpreter overhead improves execution speed by ~3.2x:```
$ perf record -e syscalls:sys_*_futex -a sleep 1 ... $ time perf script tools/perf/scripts/python/futex-contention.py ... real 0m1.007s user 0m0.935s sys 0m0.072s $ time python3 tools/perf/python/futex-contention.py ... real 0m0.314s [ ... ] Add a shell test (test_futex_contention_python.sh) to verify the standalone script.
Quoted 2 lines
Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=30
Ian Rogers
reviewerPATCH 31/49 perf python: Port task-analyzer to perf module
In reply to Ian RogersPort task-analyzer.py from tools/perf/scripts/python/ to a standalone script in tools/perf/python/ refactored into a class-based architecture.
Improvements compared to the legacy script:
- Support both offline perf.data file analysis (using perf.session) and live trace capture (using evlist.read_on_cpu), accessing sched:sched_switch tracepoint fields directly from sample objects.
- Automatically disable ANSI terminal color escape sequences when --csv or --csv-summary is enabled so CSV column headers ('Comm,', 'Time Out-Out,', etc.) are never polluted by color codes when running interactively on a TTY (e.g. as root).
- Sanitize non-printable characters and leading CSV formula characters (=, +, -, @) in task comm strings.
- Emit a one-time warning to stderr if sched:sched_switch samples are missing tracepoint fields (such as when recorded without libtraceevent support).
Update test_task_analyzer.sh to invoke the standalone script.
--- tools/perf/python/task-analyzer.py | 881 +++++++++++++++++++ tools/perf/tests/shell/test_task_analyzer.sh | 97 +- 2 files changed, 945 insertions(+), 33 deletions(-) create mode 100755 tools/perf/python/task-analyzer.py
tools/perf/python/task-analyzer.py new file +881 -0
@@ -0,0 +1,881 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# task-analyzer.py - comprehensive perf tasks analysis +# Copyright (c) 2022, Hagen Paul Pfeifer <[email protected]> +# Licensed under the terms of the GNU GPL License version 2 +# +# Usage: +# +# perf record -e sched:sched_switch -a -- sleep 10 +# ./task-analyzer.py +# +"""Comprehensive perf tasks analysis.""" +from __future__ import annotations + +import argparse +from contextlib import contextmanager +import decimal + +from typing import List, Dict, Union + +def _median(numbers: List[decimal.Decimal]) -> decimal.Decimal: + """phython3 hat statistics module - we have nothing""" + n = len(numbers) + index = n // 2 + if n % 2: + return sorted(numbers)[index] + return sum(sorted(numbers)[index - 1 : index + 1]) / decimal.Decimal(2) + +def _mean(numbers: List[decimal.Decimal]) -> decimal.Decimal: + return sum(numbers) / decimal.Decimal(len(numbers)) + +import os +import string +import sys +from typing import Any, Optional +import perf + + +# Columns will have a static size to align everything properly +# Support of 116 days of active update with nano precision +LEN_SWITCHED_IN = len("9999999.999999999") +LEN_SWITCHED_OUT = len("9999999.999999999") +LEN_CPU = len("000") +LEN_PID = len("maxvalue") +LEN_TID = len("maxvalue") +LEN_COMM = len("max-comms-length") +LEN_RUNTIME = len("999999.999") +# Support of 3.45 hours of timespans +LEN_OUT_IN = len("99999999999.999") +LEN_OUT_OUT = len("99999999999.999") +LEN_IN_IN = len("99999999999.999") +LEN_IN_OUT = len("99999999999.999") + +class Timespans: + """Tracks elapsed time between occurrences of the same task.""" + def __init__(self, args: argparse.Namespace, time_unit: str) -> None: + self.args = args + self.time_unit = time_unit + self._last_start: Optional[decimal.Decimal] = None + self._last_finish: Optional[decimal.Decimal] = None + self.current = { + 'out_out': decimal.Decimal(-1), + 'in_out': decimal.Decimal(-1), + 'out_in': decimal.Decimal(-1), + 'in_in': decimal.Decimal(-1) + } + if args.summary_extended: + self._time_in: decimal.Decimal = decimal.Decimal(-1) + self.max_vals = { + 'out_in': decimal.Decimal(-1), + 'at': decimal.Decimal(-1), + 'in_out': decimal.Decimal(-1), + 'in_in': decimal.Decimal(-1), + 'out_out': decimal.Decimal(-1) + } + + def feed(self, task: 'Task') -> None: + """Calculate timespans from chronological task occurrences.""" + if not self._last_finish: + self._last_start = task.time_in(self.time_unit) + self._last_finish = task.time_out(self.time_unit) + return + assert self._last_start is not None + assert self._last_finish is not None + self._time_in = task.time_in() + time_in = task.time_in(self.time_unit) + time_out = task.time_out(self.time_unit) + self.current['in_in'] = time_in - self._last_start + self.current['out_in'] = time_in - self._last_finish + self.current['in_out'] = time_out - self._last_start + self.current['out_out'] = time_out - self._last_finish + if self.args.summary_extended: + self.update_max_entries() + self._last_finish = task.time_out(self.time_unit) + self._last_start = task.time_in(self.time_unit) + + def update_max_entries(self) -> None: + """Update maximum timespans.""" + self.max_vals['in_in'] = max(self.max_vals['in_in'], self.current['in_in']) + self.max_vals['out_out'] = max(self.max_vals['out_out'], self.current['out_out']) + self.max_vals['in_out'] = max(self.max_vals['in_out'], self.current['in_out']) + if self.current['out_in'] > self.max_vals['out_in']: + self.max_vals['out_in'] = self.current['out_in'] + self.max_vals['at'] = self._time_in + +class Task: + """Handles information of a given task.""" + def __init__(self, task_id: str, tid: int, cpu: int, comm: str) -> None: + self.id = task_id + self.tid = tid + self.cpu = cpu + self.comm = comm + self.pid: Optional[int] = None + self._time_in: Optional[decimal.Decimal] = None + self._time_out: Optional[decimal.Decimal] = None + + def schedule_in_at(self, time_ns: int) -> None: + """Set schedule in time.""" + self._time_in = decimal.Decimal(time_ns) / decimal.Decimal(1e9) + + def schedule_out_at(self, time_ns: int) -> None: + """Set schedule out time.""" + self._time_out = decimal.Decimal(time_ns) / decimal.Decimal(1e9) + + def time_out(self, unit: str = "s") -> decimal.Decimal: + """Return schedule out time.""" + factor = TaskAnalyzer.time_uniter(unit) + return self._time_out * decimal.Decimal(factor) if self._time_out else decimal.Decimal(0) + + def time_in(self, unit: str = "s") -> decimal.Decimal: + """Return schedule in time.""" + factor = TaskAnalyzer.time_uniter(unit) + return self._time_in * decimal.Decimal(factor) if self._time_in else decimal.Decimal(0) + + def runtime(self, unit: str = "us") -> decimal.Decimal: + """Return runtime.""" + factor = TaskAnalyzer.time_uniter(unit) + if self._time_out is not None and self._time_in is not None: + return (self._time_out - self._time_in) * decimal.Decimal(factor) + return decimal.Decimal(0) + + def update_pid(self, pid: int) -> None: + """Update PID.""" + self.pid = pid + +class Summary: + """ + Primary instance for calculating the summary output. Processes the whole trace to + find and memorize relevant data such as mean, max et cetera. This instance handles + dynamic alignment aspects for summary output. + """ + + def __init__(self, analyzer): + self.analyzer = analyzer + self.args = analyzer.args + self.db = analyzer.db + self.time_unit = analyzer.time_unit + self.fd_sum = analyzer.fd_sum + self._body = [] + + class AlignmentHelper: + """ + Used to calculated the alignment for the output of the summary. + """ + def __init__(self, pid, tid, comm, runs, acc, mean, + median, min_val, max_val, max_at): + self.pid = pid + self.tid = tid + self.comm = comm + self.runs = runs + self.acc = acc + self.mean = mean + self.median = median + self.min = min_val + self.max = max_val + self.max_at = max_at + self.out_in = None + self.inter_at = None + self.out_out = None + self.in_in = None + self.in_out = None + + def _print_header(self): + ''' + Output is trimmed in _format_stats thus additional adjustment in the header + is needed, depending on the choice of timeunit. The adjustment corresponds + to the amount of column titles being adjusted in _column_titles. + ''' + decimal_precision = 6 if not self.args.ns else 9 + fmt = " {{:^{}}}".format(sum(self.db["task_info"].values())) + fmt += " {{:^{}}}".format( + max(0, sum(self.db["runtime_info"].values()) - 2 * decimal_precision) + ) + _header = ("Task Information", "Runtime Information") + + if self.args.summary_extended: + fmt += " {{:^{}}}".format( + max(0, sum(self.db["inter_times"].values()) - 4 * decimal_precision) + ) + _header += ("Max Inter Task Times",) + self.fd_sum.write(fmt.format(*_header) + "\n") + + def _column_titles(self): + """ + Cells are being processed and displayed in different way so an alignment adjust + is implemented depeding on the choice of the timeunit. The positions of the max + values are being displayed in grey. Thus in their format two additional {}, + are placed for color set and reset. + """ + separator, fix_csv_align = self.analyzer.prepare_fmt_sep(is_summary=True) + decimal_precision, time_precision = self.analyzer.prepare_fmt_precision() + fmt = "{{:>{}}}".format(self.db["task_info"]["pid"] * fix_csv_align) + fmt += "{}{{:>{}}}".format(separator, self.db["task_info"]["tid"] * fix_csv_align) + fmt += "{}{{:>{}}}".format(separator, self.db["task_info"]["comm"] * fix_csv_align) + fmt += "{}{{:>{}}}".format(separator, self.db["runtime_info"]["runs"] * fix_csv_align) + fmt += "{}{{:>{}}}".format(separator, self.db["runtime_info"]["acc"] * fix_csv_align) + fmt += "{}{{:>{}}}".format(separator, self.db["runtime_info"]["mean"] * fix_csv_align) + fmt += "{}{{:>{}}}".format( + separator, self.db["runtime_info"]["median"] * fix_csv_align + ) + fmt += "{}{{:>{}}}".format( + separator, max(0, self.db["runtime_info"]["min"] - decimal_precision) * fix_csv_align + ) + fmt += "{}{{:>{}}}".format( + separator, max(0, self.db["runtime_info"]["max"] - decimal_precision) * fix_csv_align + ) + fmt += "{}{{}}{{:>{}}}{{}}".format( + separator, max(0, self.db["runtime_info"]["max_at"] - time_precision) * fix_csv_align + ) + + grey = "" if self.args.csv_summary else TaskAnalyzer.COLORS["grey"] + reset = "" if self.args.csv_summary else TaskAnalyzer.COLORS["reset"] + column_titles = ("PID", "TID", "Comm") + column_titles += ("Runs", "Accumulated", "Mean", "Median", "Min", "Max") + column_titles += (grey, "Max At", reset) + + if self.args.summary_extended: + fmt += "{}{{:>{}}}".format( + separator, + max(0, self.db["inter_times"]["out_in"] - decimal_precision) * fix_csv_align + ) + fmt += "{}{{}}{{:>{}}}{{}}".format( + separator, + max(0, self.db["inter_times"]["inter_at"] - time_precision) * fix_csv_align + ) + fmt += "{}{{:>{}}}".format( + separator, + max(0, self.db["inter_times"]["out_out"] - decimal_precision) * fix_csv_align + ) + fmt += "{}{{:>{}}}".format( + separator, + max(0, self.db["inter_times"]["in_in"] - decimal_precision) * fix_csv_align + ) + fmt += "{}{{:>{}}}".format( + separator, + max(0, self.db["inter_times"]["in_out"] - decimal_precision) * fix_csv_align + ) + + column_titles += ( + "Out-In", grey, "Max At", + reset, "Out-Out", "In-In", "In-Out" + ) + + self.fd_sum.write(fmt.format(*column_titles) + "\n") + + + def _task_stats(self): + """calculates the stats of every task and constructs the printable summary""" + grey = "" if self.args.csv_summary else TaskAnalyzer.COLORS["grey"] + reset = "" if self.args.csv_summary else TaskAnalyzer.COLORS["reset"] + for tid in sorted(self.db["tid"]): + color_one_sample = grey + color_reset = reset + no_executed = 0 + runtimes = [] + time_in = [] + timespans = Timespans(self.args, self.time_unit) + for task in self.db["tid"][tid]: + pid = task.pid + comm = task.comm + no_executed += 1 + runtimes.append(task.runtime(self.time_unit)) + time_in.append(task.time_in()) + timespans.feed(task) + if len(runtimes) > 1: + color_one_sample = "" + color_reset = "" + time_max = max(runtimes) + time_min = min(runtimes) + max_at = time_in[runtimes.index(max(runtimes))] + + # The size of the decimal after sum,mean and median varies, thus we cut + # the decimal number, by rounding it. It has no impact on the output, + # because we have a precision of the decimal points at the output. + time_sum = round(sum(runtimes), 3) + time_mean = round(_mean(runtimes), 3) + time_median = round(_median(runtimes), 3) + + align_helper = self.AlignmentHelper(pid, tid, comm, no_executed, time_sum, + time_mean, time_median, time_min, time_max, max_at) + self._body.append([ + pid, tid, comm, no_executed, time_sum, color_one_sample, + time_mean, time_median, time_min, time_max, + grey, max_at, + reset, color_reset + ]) + if self.args.summary_extended: + self._body[-1].extend([timespans.max_vals['out_in'], + grey, timespans.max_vals['at'], + reset, timespans.max_vals['out_out'], + timespans.max_vals['in_in'], + timespans.max_vals['in_out']]) + align_helper.out_in = timespans.max_vals['out_in'] + align_helper.inter_at = timespans.max_vals['at'] + align_helper.out_out = timespans.max_vals['out_out'] + align_helper.in_in = timespans.max_vals['in_in'] + align_helper.in_out = timespans.max_vals['in_out'] + self._calc_alignments_summary(align_helper) + + def _format_stats(self): + separator, fix_csv_align = self.analyzer.prepare_fmt_sep(is_summary=True) + decimal_precision, time_precision = self.analyzer.prepare_fmt_precision() + len_pid = self.db["task_info"]["pid"] * fix_csv_align + len_tid = self.db["task_info"]["tid"] * fix_csv_align + len_comm = self.db["task_info"]["comm"] * fix_csv_align + len_runs = self.db["runtime_info"]["runs"] * fix_csv_align + len_acc = self.db["runtime_info"]["acc"] * fix_csv_align + len_mean = self.db["runtime_info"]["mean"] * fix_csv_align + len_median = self.db["runtime_info"]["median"] * fix_csv_align + len_min = max(0, self.db["runtime_info"]["min"] - decimal_precision) * fix_csv_align + len_max = max(0, self.db["runtime_info"]["max"] - decimal_precision) * fix_csv_align + len_max_at = max(0, self.db["runtime_info"]["max_at"] - time_precision) * fix_csv_align + if self.args.summary_extended: + len_out_in = max(0, + self.db["inter_times"]["out_in"] - decimal_precision + ) * fix_csv_align + len_inter_at = max(0, + self.db["inter_times"]["inter_at"] - time_precision + ) * fix_csv_align + len_out_out = max(0, + self.db["inter_times"]["out_out"] - decimal_precision + ) * fix_csv_align + len_in_in = max(0, self.db["inter_times"]["in_in"] - decimal_precision) * fix_csv_align + len_in_out = max(0, + self.db["inter_times"]["in_out"] - decimal_precision + ) * fix_csv_align + + fmt = "{{:{}d}}".format(len_pid) + fmt += "{}{{:{}d}}".format(separator, len_tid) + fmt += "{}{{:>{}}}".format(separator, len_comm) + fmt += "{}{{:{}d}}".format(separator, len_runs) + fmt += "{}{{:{}.{}f}}".format(separator, len_acc, time_precision) + fmt += "{}{{}}{{:{}.{}f}}".format(separator, len_mean, time_precision) + fmt += "{}{{:{}.{}f}}".format(separator, len_median, time_precision) + fmt += "{}{{:{}.{}f}}".format(separator, len_min, time_precision) + fmt += "{}{{:{}.{}f}}".format(separator, len_max, time_precision) + fmt += "{}{{}}{{:{}.{}f}}{{}}{{}}".format( + separator, len_max_at, decimal_precision + ) + if self.args.summary_extended: + fmt += "{}{{:{}.{}f}}".format(separator, len_out_in, time_precision) + fmt += "{}{{}}{{:{}.{}f}}{{}}".format( + separator, len_inter_at, decimal_precision + ) + fmt += "{}{{:{}.{}f}}".format(separator, len_out_out, time_precision) + fmt += "{}{{:{}.{}f}}".format(separator, len_in_in, time_precision) + fmt += "{}{{:{}.{}f}}".format(separator, len_in_out, time_precision) + return fmt + + + def _calc_alignments_summary(self, align_helper): + # Length is being cut in 3 groups so that further addition is easier to handle. + # The length of every argument from the alignment helper is being checked if it + # is longer than the longest until now. In that case the length is being saved. + for key in self.db["task_info"]: + if len(str(getattr(align_helper, key))) > self.db["task_info"][key]: + self.db["task_info"][key] = len(str(getattr(align_helper, key))) + for key in self.db["runtime_info"]: + if len(str(getattr(align_helper, key))) > self.db["runtime_info"][key]: + self.db["runtime_info"][key] = len(str(getattr(align_helper, key))) + if self.args.summary_extended: + for key in self.db["inter_times"]: + if len(str(getattr(align_helper, key))) > self.db["inter_times"][key]: + self.db["inter_times"][key] = len(str(getattr(align_helper, key))) + + + def print(self): + self._task_stats() + fmt = self._format_stats() + + if not self.args.csv_summary: + print("\nSummary", file=self.fd_sum) + self._print_header() + self._column_titles() + for i in range(len(self._body)): + self.fd_sum.write(fmt.format(*tuple(self._body[i])) + "\n") + + + + +class TaskAnalyzer: + + """Main class for task analysis.""" + + COLORS = { + "grey": "\033[90m", + "red": "\033[91m", + "green": "\033[92m", + "yellow": "\033[93m", + "blue": "\033[94m", + "violet": "\033[95m", + "reset": "\033[0m", + } + + def __init__(self, args: argparse.Namespace) -> None: + self.args = args + self.db: Dict[str, Any] = {} + self.session: Optional[perf.session] = None + self._tgid_cache: Dict[int, int] = {} + self.time_unit = "us" + if args.ns: + self.time_unit = "ns" + elif args.ms: + self.time_unit = "ms" + self._init_db() + self._check_color() + self.fd_task = sys.stdout + self.fd_sum = sys.stdout + + @contextmanager + def open_output(self, filename: str, default: Any): + """Context manager for file or stdout.""" + if filename: + with open(filename, "w", encoding="utf-8") as f: + yield f + else: + yield default + + def _init_db(self) -> None: + self.db["running"] = {} + self.db["tid"] = {} + self.db["global"] = [] + if (self.args.summary or self.args.summary_extended or + self.args.summary_only or self.args.csv_summary): + self.db["task_info"] = {} + self.db["runtime_info"] = {} + self.db["task_info"]["pid"] = len("PID") + self.db["task_info"]["tid"] = len("TID") + self.db["task_info"]["comm"] = len("Comm") + self.db["runtime_info"]["runs"] = len("Runs") + self.db["runtime_info"]["acc"] = len("Accumulated") + self.db["runtime_info"]["max"] = len("Max") + self.db["runtime_info"]["max_at"] = len("Max At") + self.db["runtime_info"]["min"] = len("Min") + self.db["runtime_info"]["mean"] = len("Mean") + self.db["runtime_info"]["median"] = len("Median") + if self.args.summary_extended: + self.db["inter_times"] = {} + self.db["inter_times"]["out_in"] = len("Out-In") + self.db["inter_times"]["inter_at"] = len("Max At") + self.db["inter_times"]["out_out"] = len("Out-Out") + self.db["inter_times"]["in_in"] = len("In-In") + self.db["inter_times"]["in_out"] = len("In-Out") + + def _check_color(self) -> None: + """Check if color should be enabled.""" + if self.args.csv: + TaskAnalyzer.COLORS = {k: "" for k in TaskAnalyzer.COLORS} + return + if sys.stdout.isatty() and self.args.stdio_color != "never": + return + if self.args.stdio_color == "always": + return + TaskAnalyzer.COLORS = {k: "" for k in TaskAnalyzer.COLORS} + + @staticmethod + def time_uniter(unit: str) -> float: + """Return time unit factor.""" + picker = {"s": 1, "ms": 1e3, "us": 1e6, "ns": 1e9} + return picker[unit] + + def _task_id(self, pid: int, cpu: int) -> str: + return f"{pid}-{cpu}" + + def _filter_non_printable(self, unfiltered: Union[str, bytearray]) -> str: + if isinstance(unfiltered, (bytearray, bytes)): + unfiltered = unfiltered.decode('utf-8', 'ignore') + filtered = "" + for char in unfiltered: + if char in string.printable and char not in "\r\n\t\x0b\x0c;": + filtered += char + stripped = filtered.lstrip() + if stripped and stripped[0] in "=+-@": + filtered = "_" + stripped[1:] + return filtered + + def prepare_fmt_precision(self) -> tuple[int, int]: + if self.args.ns: + return 9, 0 + return 6, 3 + + def prepare_fmt_sep(self, is_summary: bool = False) -> tuple[str, int]: + if (is_summary and self.args.csv_summary) or (not is_summary and self.args.csv): + return ";", 0 + return " ", 1 + + def _fmt_header(self) -> str: + separator, fix_csv_align = self.prepare_fmt_sep() + fmt = f"{{:>{LEN_SWITCHED_IN*fix_csv_align}}}" + fmt += f"{separator}{{:>{LEN_SWITCHED_OUT*fix_csv_align}}}" + fmt += f"{separator}{{:>{LEN_CPU*fix_csv_align}}}" + fmt += f"{separator}{{:>{LEN_PID*fix_csv_align}}}" + fmt += f"{separator}{{:>{LEN_TID*fix_csv_align}}}" + fmt += f"{separator}{{:>{LEN_COMM*fix_csv_align}}}" + fmt += f"{separator}{{:>{LEN_RUNTIME*fix_csv_align}}}" + fmt += f"{separator}{{:>{LEN_OUT_IN*fix_csv_align}}}" + if self.args.extended_times: + fmt += f"{separator}{{:>{LEN_OUT_OUT*fix_csv_align}}}" + fmt += f"{separator}{{:>{LEN_IN_IN*fix_csv_align}}}" + fmt += f"{separator}{{:>{LEN_IN_OUT*fix_csv_align}}}" + return fmt + + def _fmt_body(self) -> str: + separator, fix_csv_align = self.prepare_fmt_sep() + decimal_precision, time_precision = self.prepare_fmt_precision() + fmt = f"{{}}{{:{LEN_SWITCHED_IN*fix_csv_align}.{decimal_precision}f}}" + fmt += f"{separator}{{:{LEN_SWITCHED_OUT*fix_csv_align}.{decimal_precision}f}}" + fmt += f"{separator}{{:{LEN_CPU*fix_csv_align}d}}" + fmt += f"{separator}{{:{LEN_PID*fix_csv_align}d}}" + fmt += f"{separator}{{}}{{:{LEN_TID*fix_csv_align}d}}{{}}" + fmt += f"{separator}{{}}{{:>{LEN_COMM*fix_csv_align}}}" + fmt += f"{separator}{{:{LEN_RUNTIME*fix_csv_align}.{time_precision}f}}" + if self.args.extended_times: + fmt += f"{separator}{{:{LEN_OUT_IN*fix_csv_align}.{time_precision}f}}" + fmt += f"{separator}{{:{LEN_OUT_OUT*fix_csv_align}.{time_precision}f}}" + fmt += f"{separator}{{:{LEN_IN_IN*fix_csv_align}.{time_precision}f}}" + fmt += f"{separator}{{:{LEN_IN_OUT*fix_csv_align}.{time_precision}f}}{{}}" + else: + fmt += f"{separator}{{:{LEN_OUT_IN*fix_csv_align}.{time_precision}f}}{{}}" + return fmt + + def _print_header(self) -> None: + fmt = self._fmt_header() + header = ["Switched-In", "Switched-Out", "CPU", "PID", "TID", "Comm", + "Runtime", "Time Out-In"] + if self.args.extended_times: + header += ["Time Out-Out", "Time In-In", "Time In-Out"] + self.fd_task.write(fmt.format(*header) + "\n") + + def _print_task_finish(self, task: Task) -> None: + c_row_set = "" + c_row_reset = "" + out_in: Any = -1 + out_out: Any = -1 + in_in: Any = -1 + in_out: Any = -1 + fmt = self._fmt_body() + + if str(task.tid) in self.args.highlight_tasks_map: + c_row_set = TaskAnalyzer.COLORS.get(self.args.highlight_tasks_map[str(task.tid)], '') + c_row_reset = TaskAnalyzer.COLORS["reset"] + if task.comm in self.args.highlight_tasks_map: + c_row_set = TaskAnalyzer.COLORS.get(self.args.highlight_tasks_map[task.comm], '') + c_row_reset = TaskAnalyzer.COLORS["reset"] + + c_tid_set = "" + c_tid_reset = "" + if task.pid == task.tid: + c_tid_set = TaskAnalyzer.COLORS["grey"] + c_tid_reset = TaskAnalyzer.COLORS["reset"] + + if task.tid in self.db["tid"]: + last_tid_task = self.db["tid"][task.tid][-1] + timespan_gap_tid = Timespans(self.args, self.time_unit) + timespan_gap_tid.feed(last_tid_task) + timespan_gap_tid.feed(task) + out_in = timespan_gap_tid.current['out_in'] + out_out = timespan_gap_tid.current['out_out'] + in_in = timespan_gap_tid.current['in_in'] + in_out = timespan_gap_tid.current['in_out'] + + if self.args.extended_times: + line_out = fmt.format(c_row_set, task.time_in(), task.time_out(), task.cpu, + task.pid, c_tid_set, task.tid, c_tid_reset, c_row_set, task.comm, + task.runtime(self.time_unit), out_in, out_out, in_in, in_out, + c_row_reset) + "\n" + else: + line_out = fmt.format(c_row_set, task.time_in(), task.time_out(), task.cpu, + task.pid, c_tid_set, task.tid, c_tid_reset, c_row_set, task.comm, + task.runtime(self.time_unit), out_in, c_row_reset) + "\n" + self.fd_task.write(line_out) + + def _record_cleanup(self, _list: list[Any]) -> list[Any]: + need_summary = (self.args.summary or self.args.summary_extended or + self.args.summary_only or self.args.csv_summary) + if not need_summary and len(_list) > 1: + return _list[len(_list) - 1:] + return _list + + def _record_by_tid(self, task: Task) -> None: + tid = task.tid + if tid not in self.db["tid"]: + self.db["tid"][tid] = [] + self.db["tid"][tid].append(task) + self.db["tid"][tid] = self._record_cleanup(self.db["tid"][tid]) + + def _record_global(self, task: Task) -> None: + self.db["global"].append(task) + self.db["global"] = self._record_cleanup(self.db["global"]) + + def _handle_task_finish(self, tid: int, cpu: int, time_ns: int, pid: int) -> None: + if tid == 0: + return + _id = self._task_id(tid, cpu) + if _id not in self.db["running"]: + return + task = self.db["running"][_id] + task.schedule_out_at(time_ns) + task.update_pid(pid) + del self.db["running"][_id] + + if not self._limit_filtered(tid, pid, task.comm): + if not self.args.summary_only: + self._print_task_finish(task) + self._record_by_tid(task) + self._record_global(task) + + def _handle_task_start(self, tid: int, cpu: int, comm: str, time_ns: int) -> None: + if tid == 0: + return + if tid in self.args.tid_renames: + comm = self._filter_non_printable(self.args.tid_renames[tid]) + _id = self._task_id(tid, cpu) + if _id in self.db["running"]: + return + task = Task(_id, tid, cpu, comm) + task.schedule_in_at(time_ns) + self.db["running"][_id] = task + + def _limit_filtered(self, tid: int, pid: int, comm: str) -> bool: + """Filter tasks based on CLI arguments.""" + match_filter = False + if self.args.filter_tasks: + if (str(tid) in self.args.filter_tasks or + str(pid) in self.args.filter_tasks or + comm in self.args.filter_tasks): + match_filter = True + + match_limit = False + if self.args.limit_to_tasks: + if (str(tid) in self.args.limit_to_tasks or + str(pid) in self.args.limit_to_tasks or + comm in self.args.limit_to_tasks): + match_limit = True + + if self.args.filter_tasks and match_filter: + return True + if self.args.limit_to_tasks and not match_limit: + return True + return False + + def _is_within_timelimit(self, time_ns: int) -> bool: + if not self.args.time_limit: + return True + time_s = decimal.Decimal(time_ns) / decimal.Decimal(1e9) + bounds = self.args.time_limit.split(":") + lower_bound = bounds[0] if len(bounds) > 0 else "" + upper_bound = bounds[1] if len(bounds) > 1 else "" + if lower_bound and time_s < decimal.Decimal(lower_bound): + return False + if upper_bound and time_s > decimal.Decimal(upper_bound): + return False + return True + + def process_event(self, sample: perf.sample_event) -> None: + """Process sched:sched_switch events.""" + if "sched:sched_switch" not in str(sample.evsel): + return + + time_ns = sample.sample_time + if not self._is_within_timelimit(time_ns): + return + + # Access tracepoint fields directly from sample object + try: + prev_pid = sample.prev_pid + next_pid = sample.next_pid + next_comm = sample.next_comm + common_cpu = sample.sample_cpu + except AttributeError: + if not self.db.get("warned_missing_fields"): + self.db["warned_missing_fields"] = True + print("Warning: sched:sched_switch sample missing tracepoint fields " + "(is libtraceevent support enabled?).", file=sys.stderr) + return + + next_comm = self._filter_non_printable(next_comm) + + # Task finish for previous task + prev_tgid = prev_pid # Fallback + if prev_pid != 0: + if self.session: + try: + thread = self.session.find_thread(-1, prev_pid) + if thread: + prev_tgid = thread.pid + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError): + pass + elif prev_pid in self._tgid_cache: + prev_tgid = self._tgid_cache[prev_pid] + else: + if len(self._tgid_cache) >= 4096: + self._tgid_cache.pop(next(iter(self._tgid_cache))) + self._tgid_cache[prev_pid] = prev_tgid + try: + with open(f"/proc/{prev_pid}/status", encoding="utf-8") as f: + for line in f: + if line.startswith("Tgid:"): + prev_tgid = int(line.split()[1]) + self._tgid_cache[prev_pid] = prev_tgid + break + except (OSError, ValueError, KeyError, RuntimeError): + pass + self._handle_task_finish(prev_pid, common_cpu, time_ns, prev_tgid) + # Task start for next task + self._handle_task_start(next_pid, common_cpu, next_comm, time_ns) + + def print_summary(self) -> None: + """Calculate and print summary.""" + need_summary = (self.args.summary or self.args.summary_extended or + self.args.summary_only or self.args.csv_summary) + if not need_summary: + return + + Summary(self).print() + + def _run_file(self) -> None: + if not self.args.summary_only: + self._print_header() + + session = perf.session(perf.data(self.args.input), sample=self.process_event) + self.session = session + session.process_events() + + if not self.db["global"]: + print(f"Warning: No sched:sched_switch trace events found in '{self.args.input}'.", + file=sys.stderr) + + self.print_summary() + + def _run_live(self) -> None: + if not self.args.summary_only: + self._print_header() + + cpus = perf.cpu_map() + threads = perf.thread_map(-1) + evlist = perf.parse_events("sched:sched_switch", cpus, threads) + evlist.config() + + evlist.open() + evlist.mmap() + evlist.enable() + + pending_events: list[perf.sample_event] = [] + print("Live mode started. Press Ctrl+C to stop.", file=sys.stderr) + try: + while True: + try: + evlist.poll(timeout=100) + except InterruptedError: + continue + for cpu in cpus: + while True: + event = evlist.read_on_cpu(cpu) + if not event: + break + if not isinstance(event, perf.sample_event): + continue + pending_events.append(event) + if pending_events: + pending_events.sort( + key=lambda e: getattr(e, 'sample_time', getattr(e, 'time', 0)) + ) + max_ts = getattr( + pending_events[-1], 'sample_time', getattr(pending_events[-1], 'time', 0) + ) + cutoff_ts = max_ts - 50_000_000 + ready_idx = 0 + for event in pending_events: + ev_ts = getattr(event, 'sample_time', getattr(event, 'time', 0)) + if ev_ts <= cutoff_ts: + self.process_event(event) + ready_idx += 1 + else: + break + del pending_events[:ready_idx] + except KeyboardInterrupt: + print("\nStopping live mode...", file=sys.stderr) + finally: + pending_events.sort(key=lambda e: getattr(e, 'sample_time', getattr(e, 'time', 0))) + for event in pending_events: + self.process_event(event) + evlist.close() + self.print_summary() + + def run(self) -> None: + """Run the session.""" + is_live = (self.args.live or + (not os.path.exists(self.args.input) and self.args.input == "perf.data")) + with self.open_output(self.args.csv, sys.stdout) as fd_task: + if self.args.csv and self.args.csv == self.args.csv_summary: + self.fd_task = fd_task + self.fd_sum = fd_task + if is_live: + self._run_live() + else: + self._run_file() + else: + with self.open_output(self.args.csv_summary, sys.stdout) as fd_sum: + self.fd_task = fd_task + self.fd_sum = fd_sum + if is_live: + self._run_live() + else: + self._run_file() + +def main() -> None: + """Main function.""" + parser = argparse.ArgumentParser(description="Analyze tasks behavior") + parser.add_argument("-i", "--input", default="perf.data", help="Input file name") + parser.add_argument("--time-limit", default="", help="print tasks only in time window") + parser.add_argument("--summary", action="store_true", + help="print additional runtime information") + parser.add_argument("--summary-only", action="store_true", + help="print only summary without traces") + parser.add_argument("--summary-extended", action="store_true", + help="print extended summary") + parser.add_argument("--ns", action="store_true", help="show timestamps in nanoseconds") + parser.add_argument("--ms", action="store_true", help="show timestamps in milliseconds") + parser.add_argument("--live", action="store_true", + help="force live mode (ignores -i/perf.data)") + parser.add_argument("--extended-times", action="store_true", + help="Show elapsed times between schedule in/out") + parser.add_argument("--filter-tasks", default="", help="filter tasks by tid, pid or comm") + parser.add_argument("--limit-to-tasks", default="", help="limit output to selected tasks") + parser.add_argument("--highlight-tasks", default="", help="colorize special tasks") + parser.add_argument("--rename-comms-by-tids", default="", help="rename task names by using tid") + parser.add_argument("--stdio-color", default="auto", choices=["always", "never", "auto"], + help="configure color output") + parser.add_argument("--csv", default="", help="Write trace to file") + parser.add_argument("--csv-summary", default="", help="Write summary to file") + + args = parser.parse_args() + args.tid_renames = {} + args.highlight_tasks_map = {} + args.filter_tasks = args.filter_tasks.split(",") if args.filter_tasks else [] + args.limit_to_tasks = args.limit_to_tasks.split(",") if args.limit_to_tasks else [] + + if args.rename_comms_by_tids: + for item in args.rename_comms_by_tids.split(","): + try: + tid, name = item.split(":", 1) + args.tid_renames[int(tid)] = name + except ValueError: + print(f"Error: Invalid format for --rename-comms-by-tids '{item}', " + "expected tid:name", file=sys.stderr) + sys.exit(1) + + if args.highlight_tasks: + for item in args.highlight_tasks.split(","): + parts = item.split(":") + if len(parts) == 1: + parts.append("red") + key, color = parts[0], parts[1] + args.highlight_tasks_map[key] = color + + analyzer = TaskAnalyzer(args) + analyzer.run() + +if __name__ == "__main__": + main()
tools/perf/tests/shell/test_task_analyzer.sh +64 -33
177 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 31/49] perf python: Port task-analyzer to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines
Port
task-analyzer.pyfromtools/perf/scripts/python/to a standalone script intools/perf/python/refactored into a class-based architecture.Improvements compared to the legacy script:
- Support both offline perf.data file analysis (using perf.session) and live trace capture (using evlist.read_on_cpu), accessing sched:sched_switch tracepoint fields directly from sample objects.
- Automatically disable ANSI terminal color escape sequences when --csv or --csv-summary is enabled so CSV column headers ('Comm,', 'Time Out-Out,', etc.) are never polluted by color codes when running interactively on a TTY (e.g. as root).
- Sanitize non-printable characters and leading CSV formula characters (=, +, -, @) in task comm strings.
- Emit a one-time warning to stderr if sched:sched_switch samples are missing tracepoint fields (such as when recorded without libtraceevent support).
Quoted 4 lines
Update
test_task_analyzer.shto invoke the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=31
Ian Rogers
reviewerPATCH 32/49 perf python: Port sched-migration and SchedGui to perf module
In reply to Ian RogersPort sched-migration.py and SchedGui.py from tools/perf/scripts/python/ to standalone modules in tools/perf/python/:
- Refactor sched-migration.py into a SchedMigrationAnalyzer class using perf.session for event processing and add argparse CLI support (-i/--input, -v/--verbose, --gui, --no-gui).
- Port SchedGui.py to tools/perf/python/ as a local module dependency.
- Load wxPython dynamically via importlib only when GUI mode (--gui) is requested so text-mode analysis, testing on headless systems, and static analysis (mypy/pylint) succeed without wx installed.
- Remove Python 2 compatibility code.
Add a shell test (test_sched_migration_python.sh) to verify the standalone script.
--- tools/perf/python/SchedGui.py | 246 +++++++++ tools/perf/python/sched-migration.py | 486 ++++++++++++++++++ .../shell/test_sched_migration_python.sh | 69 +++ 3 files changed, 801 insertions(+) create mode 100755 tools/perf/python/SchedGui.py create mode 100755 tools/perf/python/sched-migration.py create mode 100755 tools/perf/tests/shell/test_sched_migration_python.sh
tools/perf/python/SchedGui.py new file +246 -0
@@ -0,0 +1,246 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# SchedGui.py - Python extension for perf script, basic GUI code for +# traces drawing and overview. +# +# Copyright (C) 2010 by Frederic Weisbecker <[email protected]> +# +# Ported to modern directory structure. + +"""SchedGui.py - Python extension for perf script, basic GUI code for traces drawing and overview.""" +from __future__ import annotations + +import importlib +from typing import Any + +class _DummyWx: + """Dummy wx module fallback when wxPython is not installed.""" + Frame = object + + +try: + wx: Any = importlib.import_module("wx") + WX_AVAILABLE = True +except ImportError: + wx = _DummyWx + WX_AVAILABLE = False + + +class RootFrame(wx.Frame): + """Main window frame for scheduling trace visualization.""" + Y_OFFSET = 100 + RECT_HEIGHT = 100 + RECT_SPACE = 50 + EVENT_MARKING_WIDTH = 5 + + def __init__(self, sched_tracer, title, parent=None, win_id=-1): + if not WX_AVAILABLE: + raise ImportError("You need to install the wxpython lib for this script") + wx.Frame.__init__(self, parent, win_id, title) + + self.dc = None + (self.screen_width, self.screen_height) = wx.GetDisplaySize() + self.screen_width -= 10 + self.screen_height -= 10 + self.zoom = 0.5 + self.scroll_scale = 20 + self.sched_tracer = sched_tracer + self.sched_tracer.set_root_win(self) + (self.ts_start, self.ts_end) = sched_tracer.interval() + self.update_width_virtual() + self.nr_rects = sched_tracer.nr_rectangles() + 1 + self.height_virtual = RootFrame.Y_OFFSET + \ + (self.nr_rects * (RootFrame.RECT_HEIGHT + RootFrame.RECT_SPACE)) + + # whole window panel + self.panel = wx.Panel(self, size=(self.screen_width, self.screen_height)) + + # scrollable container + # Create SplitterWindow + self.splitter = wx.SplitterWindow(self.panel, style=wx.SP_3D) + + # scrollable container (Top) + self.scroll = wx.ScrolledWindow(self.splitter) + self.scroll.SetScrollbars(self.scroll_scale, self.scroll_scale, + int(self.width_virtual // self.scroll_scale), + int(self.height_virtual // self.scroll_scale)) + self.scroll.EnableScrolling(True, True) + self.scroll.SetFocus() + + # scrollable drawing area + self.scroll_panel = wx.Panel(self.scroll, + size=(self.screen_width - 15, self.screen_height // 2)) + self.scroll_panel.Bind(wx.EVT_PAINT, self.on_paint) + self.scroll_panel.Bind(wx.EVT_KEY_DOWN, self.on_key_press) + self.scroll_panel.Bind(wx.EVT_LEFT_DOWN, self.on_mouse_down) + self.scroll.Bind(wx.EVT_KEY_DOWN, self.on_key_press) + self.scroll.Bind(wx.EVT_LEFT_DOWN, self.on_mouse_down) + + self.scroll_panel.SetSize(int(self.width_virtual), int(self.height_virtual)) + + # Create a separate panel for text (Bottom) + self.text_panel = wx.Panel(self.splitter) + self.text_sizer = wx.BoxSizer(wx.VERTICAL) + self.txt = wx.TextCtrl(self.text_panel, -1, "Click a bar to see details", + style=wx.TE_MULTILINE) + self.text_sizer.Add(self.txt, 1, wx.EXPAND | wx.ALL, 5) + self.text_panel.SetSizer(self.text_sizer) + + # Split the window + self.splitter.SplitHorizontally(self.scroll, self.text_panel, (self.screen_height * 3) // 4) + + # Main sizer to layout splitter + self.main_sizer = wx.BoxSizer(wx.VERTICAL) + self.main_sizer.Add(self.splitter, 1, wx.EXPAND) + self.panel.SetSizer(self.main_sizer) + + self.scroll.Fit() + self.Fit() + + self.Show(True) + + def us_to_px(self, val): + """Convert microseconds to pixels.""" + return val / (10 ** 3) * self.zoom + + def px_to_us(self, val): + """Convert pixels to microseconds.""" + return (val / self.zoom) * (10 ** 3) + + def scroll_start(self): + """Get scroll start position in pixels.""" + (x, y) = self.scroll.GetViewStart() + return (x * self.scroll_scale, y * self.scroll_scale) + + def scroll_start_us(self): + """Get scroll start position in microseconds.""" + (x, _) = self.scroll_start() + return self.px_to_us(x) + + def paint_rectangle_zone(self, nr, color, top_color, start, end): + """Draw a rectangle zone for a CPU.""" + offset_px = self.us_to_px(start - self.ts_start) + width_px = self.us_to_px(end - start) + + offset_py = RootFrame.Y_OFFSET + (nr * (RootFrame.RECT_HEIGHT + RootFrame.RECT_SPACE)) + width_py = RootFrame.RECT_HEIGHT + + dc = self.dc + + if top_color is not None: + (r, g, b) = top_color + top_color = wx.Colour(r, g, b) + brush = wx.Brush(top_color, wx.SOLID) + dc.SetBrush(brush) + dc.DrawRectangle(int(offset_px), int(offset_py), + int(width_px), RootFrame.EVENT_MARKING_WIDTH) + width_py -= RootFrame.EVENT_MARKING_WIDTH + offset_py += RootFrame.EVENT_MARKING_WIDTH + + (r, g, b) = color + color = wx.Colour(r, g, b) + brush = wx.Brush(color, wx.SOLID) + dc.SetBrush(brush) + dc.DrawRectangle(int(offset_px), int(offset_py), int(width_px), int(width_py)) + + def update_rectangles(self, start, end): + """Update rectangles in the given time window.""" + start += self.ts_start + end += self.ts_start + self.sched_tracer.fill_zone(start, end) + + def on_paint(self, event): + """Handle paint event.""" + window = event.GetEventObject() + dc = wx.PaintDC(window) + + # Clear background to avoid ghosting + dc.SetBackground(wx.Brush(window.GetBackgroundColour())) + dc.Clear() + + self.dc = dc + try: + width = min(self.width_virtual, self.screen_width) + (x, _) = self.scroll_start() + start = self.px_to_us(x) + end = self.px_to_us(x + width) + self.update_rectangles(start, end) + + # Draw CPU labels at the left edge of the visible area + (x_scroll, _) = self.scroll_start() + for nr in range(self.nr_rects): + offset_py = RootFrame.Y_OFFSET + (nr * (RootFrame.RECT_HEIGHT + RootFrame.RECT_SPACE)) + dc.DrawText(f"CPU {nr}", x_scroll + 10, offset_py + 10) + finally: + self.dc = None + + def rect_from_ypixel(self, y): + y -= RootFrame.Y_OFFSET + rect = y // (RootFrame.RECT_HEIGHT + RootFrame.RECT_SPACE) + height = y % (RootFrame.RECT_HEIGHT + RootFrame.RECT_SPACE) + + if rect < 0 or rect > self.nr_rects - 1 or height > RootFrame.RECT_HEIGHT: + return -1 + + return rect + + def update_summary(self, txt): + self.txt.SetValue(txt) + self.text_panel.Layout() + self.splitter.Layout() + self.text_panel.Refresh() + + def on_mouse_down(self, event): + pos = event.GetPosition() + x, y = pos.x, pos.y + rect = self.rect_from_ypixel(y) + if rect == -1: + return + + t = self.px_to_us(x) + self.ts_start + + self.sched_tracer.mouse_down(rect, t) + + def update_width_virtual(self): + self.width_virtual = self.us_to_px(self.ts_end - self.ts_start) + + def __zoom(self, x): + self.update_width_virtual() + (xpos, ypos) = self.scroll.GetViewStart() + xpos = int(self.us_to_px(x) // self.scroll_scale) + self.scroll_panel.SetSize((int(self.width_virtual), int(self.height_virtual))) + self.scroll.SetScrollbars(self.scroll_scale, self.scroll_scale, + int(self.width_virtual // self.scroll_scale), + int(self.height_virtual // self.scroll_scale), + xpos, ypos) + self.Refresh() + + def zoom_in(self): + x = self.scroll_start_us() + self.zoom *= 2 + self.__zoom(x) + + def zoom_out(self): + x = self.scroll_start_us() + self.zoom /= 2 + self.__zoom(x) + + def on_key_press(self, event): + key = event.GetRawKeyCode() + if key == ord("+"): + self.zoom_in() + return + if key == ord("-"): + self.zoom_out() + return + + key = event.GetKeyCode() + (x, y) = self.scroll.GetViewStart() + if key == wx.WXK_RIGHT: + self.scroll.Scroll(x + 1, y) + elif key == wx.WXK_LEFT: + self.scroll.Scroll(x - 1, y) + elif key == wx.WXK_DOWN: + self.scroll.Scroll(x, y + 1) + elif key == wx.WXK_UP: + self.scroll.Scroll(x, y - 1)
tools/perf/python/sched-migration.py new file +486 -0
486 lines not on this page · raw message
tools/perf/tests/shell/test_sched_migration_python.sh new file +69 -0
69 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 32/49] perf python: Port sched-migration and SchedGui to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Port
sched-migration.pyandSchedGui.pyfromtools/perf/scripts/python/to standalone modules intools/perf/python/:
- Refactorsched-migration.pyinto a SchedMigrationAnalyzer class using perf.session for event processing and add argparse CLI support (-i/--input, -v/--verbose, --gui, --no-gui).
- PortSchedGui.pytotools/perf/python/as a local module dependency.
- Load wxPython dynamically via importlib only when GUI mode (--gui) is requested so text-mode analysis, testing on headless systems, and static analysis (mypy/pylint) succeed without wx installed.
- Remove Python 2 compatibility code.
Quoted 5 lines
Add a shell test (
test_sched_migration_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=32
Ian Rogers
reviewerPATCH 33/49 perf python: Port wakeup-latency from Perl to perf module
In reply to Ian RogersReplace the legacy Perl script wakeup-latency.pl with a standalone Python script in tools/perf/python/wakeup-latency.py using the perf Python module.
Improvements compared to the legacy Perl script:
- Remove the dependency on libperl and Perf::Trace::Util.
- Guard print_totals() when total_wakeups == 0 (printing 'N/A' instead of dividing by zero when a trace contains no matched wakeup/switch pairs).
- Add argparse CLI options (-i/--input) and full type annotations.
Add a shell test
(test_wakeup_latency_python.sh) to verify the standalone script.
--- tools/perf/python/wakeup-latency.py | 94 +++++++++++++++++++ .../tests/shell/test_wakeup_latency_python.sh | 83 ++++++++++++++++ 2 files changed, 177 insertions(+) create mode 100755 tools/perf/python/wakeup-latency.py create mode 100755 tools/perf/tests/shell/test_wakeup_latency_python.sh
tools/perf/python/wakeup-latency.py new file +94 -0
@@ -0,0 +1,94 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +"""Display avg/min/max wakeup latency.""" +from __future__ import annotations + +import argparse +from collections import defaultdict +import sys +from typing import Optional, Dict +import perf + +class WakeupLatency: + """Tracks and displays wakeup latency statistics.""" + def __init__(self) -> None: + self.last_wakeup: Dict[int, int] = defaultdict(int) + self.max_wakeup_latency: int = 0 + self.min_wakeup_latency: Optional[int] = None + self.total_wakeup_latency = 0 + self.total_wakeups = 0 + self.unhandled: Dict[str, int] = defaultdict(int) + self.session: Optional[perf.session] = None + + def process_event(self, sample: perf.sample_event) -> None: + """Process events.""" + event_name = str(sample.evsel) + sample_time = sample.sample_time + + if "sched:sched_wakeup" in event_name: + try: + pid = sample.pid + self.last_wakeup[pid] = sample_time + except AttributeError: + self.unhandled[event_name] += 1 + elif "sched:sched_switch" in event_name: + try: + next_pid = sample.next_pid + wakeup_ts = self.last_wakeup.get(next_pid, 0) + if wakeup_ts: + latency = sample_time - wakeup_ts + self.max_wakeup_latency = max(self.max_wakeup_latency, latency) + if self.min_wakeup_latency is None: + self.min_wakeup_latency = latency + else: + self.min_wakeup_latency = min(self.min_wakeup_latency, latency) + self.total_wakeup_latency += latency + self.total_wakeups += 1 + del self.last_wakeup[next_pid] + except AttributeError: + self.unhandled[event_name] += 1 + else: + self.unhandled[event_name] += 1 + + def print_totals(self) -> None: + """Print summary statistics.""" + print("wakeup_latency stats:\n") + print(f"total_wakeups: {self.total_wakeups}") + if self.total_wakeups: + avg = self.total_wakeup_latency // self.total_wakeups + print(f"avg_wakeup_latency (ns): {avg}") + print(f"min_wakeup_latency (ns): {self.min_wakeup_latency}") + print(f"max_wakeup_latency (ns): {self.max_wakeup_latency}") + else: + print("avg_wakeup_latency (ns): N/A") + print("min_wakeup_latency (ns): N/A") + print("max_wakeup_latency (ns): N/A") + + if self.unhandled: + print("\nunhandled events:\n") + print(f"{'event':<40s} {'count':>10s}") + print(f"{'-'*40} {'-'*10}") + for event_name, count in self.unhandled.items(): + print(f"{event_name:<40s} {count:10d}") + + def run(self, input_file: str) -> None: + """Run the session.""" + self.session = perf.session(perf.data(input_file), sample=self.process_event) + self.session.process_events() + self.print_totals() + +def main() -> None: + """Main function.""" + parser = argparse.ArgumentParser(description="Trace wakeup latency") + parser.add_argument("-i", "--input", default="perf.data", help="Input file") + args = parser.parse_args() + + analyzer = WakeupLatency() + try: + analyzer.run(args.input) + except IOError as e: + print(e, file=sys.stderr) + sys.exit(1) + +if __name__ == "__main__": + main()
tools/perf/tests/shell/test_wakeup_latency_python.sh new file +83 -0
83 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 33/49] perf python: Port wakeup-latency from Perl to perf module
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf python: memory leak due to perf.session reference cycle --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf python: memory leak due to perf.session reference cycle
--
commit 6aadac5b14d7ffc3e11063c58b8e5957fe0a1f3c
Author: Ian Rogers <[email protected]>
perf python: Port wakeup-latency from Perl to perf module
This commit replaces the legacy Perl script wakeup-latency.pl with a standalone Python script using the perf Python module, bringing improvements such as division-by-zero protection and CLI options.
Quoted 5 lines
tools/perf/python/wakeup-latency.py new file +0
-0
[ ... ]
tools/perf/python/wakeup-latency.py line 76
Quoted 3 lines
tools/perf/python/wakeup-latency.py new file +3
-0+ def run(self, input_file: str) -> None: + """Run the session.""" + self.session = perf.session(perf.data(input_file), sample=self.process_event)
[Severity: Medium]
Does this create an unbreakable reference cycle that leaks the perf.session and its associated file descriptors?
In WakeupLatency.run(), assigning the perf.session to self.session while also passing the bound method self.process_event as the sample callback means the instance holds the session, and the session holds the instance.
Since the underlying C extension type for perf.session (pyrf_session) does not implement Python's cyclic garbage collection (Py_TPFLAGS_HAVE_GC and tp_traverse are absent), this cycle is invisible to the garbage collector.
If this module is imported or WakeupLatency is executed multiple times in a long-running process, could it rapidly exhaust memory and file descriptors?
Quoted 2 lines
tools/perf/python/wakeup-latency.py new file +2
-0+ self.session.process_events() + self.print_totals()
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=33
Ian Rogers
reviewerPATCH 34/49 perf python: Port compaction-times to perf module
In reply to Ian RogersPort compaction-times.py to a standalone script in tools/perf/python/ using the perf module directly to analyze mm_compaction tracepoints.
Improvements compared to the legacy script:
- Replace Python 2 constructs (such as sys.maxint and raw integer bitmasks) with Python 3 enum.IntEnum (Popt) and enum.IntFlag (Topt) types.
- Access tracepoint fields directly on perf.sample_event and add -i/--input CLI option support via argparse.
- Add full type annotations passing mypy and pylint without suppression comments.
Add a shell test (test_compaction_times_python.sh) to verify the standalone script.
--- tools/perf/python/compaction-times.py | 350 ++++++++++++++++++ .../shell/test_compaction_times_python.sh | 81 ++++ 2 files changed, 431 insertions(+) create mode 100755 tools/perf/python/compaction-times.py create mode 100755 tools/perf/tests/shell/test_compaction_times_python.sh
tools/perf/python/compaction-times.py new file +350 -0
@@ -0,0 +1,350 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +Report time spent in memory compaction. + +Memory compaction is a feature in the Linux kernel that defragments memory +by moving used pages to create larger contiguous blocks of free memory. This +is particularly useful for allocating huge pages. + +This script processes trace events related to memory compaction and reports: +- Total time spent in compaction (stall time). +- Statistics for page migration (moved vs. failed). +- Statistics for the free scanner (scanned vs. isolated pages). +- Statistics for the migration scanner (scanned vs. isolated pages). + +Definitions: +- **Compaction**: Defragmenting memory by moving allocated pages. +- **Migration**: Moving pages from their current location to free pages found by the free scanner. +- **Free Scanner**: Scans memory (typically from the end of a zone) to find free pages. +- **Migration Scanner**: Scans memory (typically from the beginning of a zone) + to find pages to move. +- **Isolated Pages**: Pages that have been temporarily removed from the buddy + system for migration or as migration targets. + +Ported from tools/perf/scripts/python/compaction-times.py to the modern perf Python module. +""" +from __future__ import annotations + +import argparse +import enum +import re +import sys +from typing import Callable, Dict, List, Optional, Any +import perf + +class Popt(enum.IntEnum): + """Process display options.""" + DISP_DFL = 0 + DISP_PROC = 1 + DISP_PROC_VERBOSE = 2 + +class Topt(enum.IntFlag): + """Trace display options.""" + DISP_TIME = 0 + DISP_MIG = 1 + DISP_ISOLFREE = 2 + DISP_ISOLMIG = 4 + DISP_ALL = DISP_MIG | DISP_ISOLFREE | DISP_ISOLMIG + +# Globals to satisfy pylint when accessed in functions before assignment in main. +OPT_NS = True +opt_disp = Topt.DISP_ALL +opt_proc = Popt.DISP_DFL +session = None + +def get_comm_filter(regex: re.Pattern) -> Callable[[int, str], bool]: + """Returns a filter function based on command regex.""" + def filter_func(_pid: int, comm: str) -> bool: + regex_match = regex.search(comm) + return regex_match is None or regex_match.group() == "" + return filter_func + +def get_pid_filter(low_str: str, high_str: str) -> Callable[[int, str], bool]: + """Returns a filter function based on PID range.""" + low = 0 if low_str == "" else int(low_str) + high = None if high_str == "" else int(high_str) + + def filter_func(pid: int, _comm: str) -> bool: + return not (pid >= low and (high is None or pid <= high)) + return filter_func + +def ns_to_time(ns: int) -> str: + """Format nanoseconds to string based on options.""" + return f"{ns}ns" if OPT_NS else f"{round(ns, -3) // 1000}us" + +class Pair: + """Represents a pair of related counters (e.g., scanned vs isolated, moved vs failed).""" + def __init__(self, aval: int, bval: int, + alabel: Optional[str] = None, blabel: Optional[str] = None): + self.alabel = alabel + self.blabel = blabel + self.aval = aval + self.bval = bval + + def __add__(self, rhs: 'Pair') -> 'Pair': + return Pair(self.aval + rhs.aval, self.bval + rhs.bval, self.alabel, self.blabel) + + def __iadd__(self, rhs: 'Pair') -> 'Pair': + self.aval += rhs.aval + self.bval += rhs.bval + return self + + def __str__(self) -> str: + return f"{self.alabel}={self.aval} {self.blabel}={self.bval}" + +class Cnode: + """Holds statistics for a single compaction event or an aggregated set of events.""" + def __init__(self, ns: int): + self.ns = ns + self.migrated = Pair(0, 0, "moved", "failed") + self.fscan = Pair(0, 0, "scanned", "isolated") + self.mscan = Pair(0, 0, "scanned", "isolated") + + def __add__(self, rhs: 'Cnode') -> 'Cnode': + res = Cnode(self.ns + rhs.ns) + res.migrated = self.migrated + rhs.migrated + res.fscan = self.fscan + rhs.fscan + res.mscan = self.mscan + rhs.mscan + return res + + def __iadd__(self, rhs: 'Cnode') -> 'Cnode': + self.ns += rhs.ns + self.migrated += rhs.migrated + self.fscan += rhs.fscan + self.mscan += rhs.mscan + return self + + def __str__(self) -> str: + prev = False + s = f"{ns_to_time(self.ns)} " + if opt_disp & Topt.DISP_MIG: + s += f"migration: {self.migrated}" + prev = True + if opt_disp & Topt.DISP_ISOLFREE: + s += f"{' ' if prev else ''}free_scanner: {self.fscan}" + prev = True + if opt_disp & Topt.DISP_ISOLMIG: + s += f"{' ' if prev else ''}migration_scanner: {self.mscan}" + return s + + def complete(self, secs: int, nsecs: int) -> None: + """Complete the node with duration.""" + self.ns = (secs * 1000000000 + nsecs) - self.ns + + def increment(self, migrated: Optional[Pair], fscan: Optional[Pair], + mscan: Optional[Pair]) -> None: + """Increment statistics.""" + if migrated is not None: + self.migrated += migrated + if fscan is not None: + self.fscan += fscan + if mscan is not None: + self.mscan += mscan + +class Chead: + """Aggregates compaction statistics per process (PID) and maintains total statistics.""" + heads: Dict[int, 'Chead'] = {} + val = Cnode(0) + fobj: Optional[Any] = None + + @classmethod + def add_filter(cls, fobj: Any) -> None: + """Add a filter object.""" + cls.fobj = fobj + + @classmethod + def create_pending(cls, pid: int, comm: str, start_secs: int, start_nsecs: int) -> None: + """Create a pending node for a process.""" + filtered = False + try: + head = cls.heads[pid] + filtered = head.is_filtered() + except KeyError: + if cls.fobj is not None: + filtered = cls.fobj(pid, comm) + head = cls.heads[pid] = Chead(comm, pid, filtered) + + if not filtered: + head.mark_pending(start_secs, start_nsecs) + + @classmethod + def increment_pending(cls, pid: int, migrated: Optional[Pair], + fscan: Optional[Pair], mscan: Optional[Pair]) -> None: + """Increment pending stats for a process.""" + if pid not in cls.heads: + return + head = cls.heads[pid] + if not head.is_filtered(): + if head.is_pending(): + head.do_increment(migrated, fscan, mscan) + else: + sys.stderr.write(f"missing start compaction event for pid {pid}\n") + + @classmethod + def complete_pending(cls, pid: int, secs: int, nsecs: int) -> None: + """Complete pending stats for a process.""" + if pid not in cls.heads: + return + head = cls.heads[pid] + if not head.is_filtered(): + if head.is_pending(): + head.make_complete(secs, nsecs) + else: + sys.stderr.write(f"missing start compaction event for pid {pid}\n") + + @classmethod + def gen(cls): + """Generate heads for display.""" + if opt_proc != Popt.DISP_DFL: + yield from cls.heads.values() + + @classmethod + def get_total(cls) -> Cnode: + """Get total statistics.""" + return cls.val + + def __init__(self, comm: str, pid: int, filtered: bool): + self.comm = comm + self.pid = pid + self.val = Cnode(0) + self.pending: Optional[Cnode] = None + self.filtered = filtered + self.list: List[Cnode] = [] + + def mark_pending(self, secs: int, nsecs: int) -> None: + """Mark node as pending.""" + self.pending = Cnode(secs * 1000000000 + nsecs) + + def do_increment(self, migrated: Optional[Pair], fscan: Optional[Pair], + mscan: Optional[Pair]) -> None: + """Increment pending stats.""" + if self.pending is not None: + self.pending.increment(migrated, fscan, mscan) + + def make_complete(self, secs: int, nsecs: int) -> None: + """Make pending stats complete.""" + if self.pending is not None: + self.pending.complete(secs, nsecs) + Chead.val += self.pending + + if opt_proc != Popt.DISP_DFL: + self.val += self.pending + + if opt_proc == Popt.DISP_PROC_VERBOSE: + self.list.append(self.pending) + self.pending = None + + def enumerate(self) -> None: + """Enumerate verbose stats.""" + if opt_proc == Popt.DISP_PROC_VERBOSE and not self.is_filtered(): + for i, pelem in enumerate(self.list): + sys.stdout.write(f"{self.pid}[{self.comm}].{i+1}: {pelem}\n") + + def is_pending(self) -> bool: + """Check if node is pending.""" + return self.pending is not None + + def is_filtered(self) -> bool: + """Check if node is filtered.""" + return self.filtered + + def display(self) -> None: + """Display stats.""" + if not self.is_filtered(): + sys.stdout.write(f"{self.pid}[{self.comm}]: {self.val}\n") + +def trace_end() -> None: + """Called at the end of trace processing.""" + sys.stdout.write(f"total: {Chead.get_total()}\n") + for i in Chead.gen(): + i.display() + i.enumerate() + +def process_event(sample: perf.sample_event) -> None: + """Callback for processing events.""" + event_name = str(sample.evsel) + pid = sample.sample_tid + comm = "[unknown]" + try: + if session: + thread = session.find_thread(pid) + if thread: + comm = thread.comm() or "[unknown]" + except (TypeError, AttributeError): + pass + secs = sample.sample_time // 1000000000 + nsecs = sample.sample_time % 1000000000 + + if "evsel(compaction:mm_compaction_begin)" in event_name: + Chead.create_pending(pid, comm, secs, nsecs) + elif "evsel(compaction:mm_compaction_end)" in event_name: + Chead.complete_pending(pid, secs, nsecs) + elif "evsel(compaction:mm_compaction_migratepages)" in event_name: + nr_migrated = getattr(sample, "nr_migrated", 0) + nr_failed = getattr(sample, "nr_failed", 0) + Chead.increment_pending(pid, Pair(nr_migrated, nr_failed), None, None) + elif "evsel(compaction:mm_compaction_isolate_freepages)" in event_name: + nr_scanned = getattr(sample, "nr_scanned", 0) + nr_taken = getattr(sample, "nr_taken", 0) + Chead.increment_pending(pid, None, Pair(nr_scanned, nr_taken), None) + elif "evsel(compaction:mm_compaction_isolate_migratepages)" in event_name: + nr_scanned = getattr(sample, "nr_scanned", 0) + nr_taken = getattr(sample, "nr_taken", 0) + Chead.increment_pending(pid, None, None, Pair(nr_scanned, nr_taken)) + +if __name__ == "__main__": + ap = argparse.ArgumentParser(description="Report time spent in compaction") + ap.add_argument("-p", action="store_true", help="display by process") + ap.add_argument("-pv", action="store_true", help="display by process (verbose)") + ap.add_argument("-u", action="store_true", help="display results in microseconds") + ap.add_argument("-t", action="store_true", help="display stall times only") + ap.add_argument("-m", action="store_true", help="display stats for migration") + ap.add_argument("-fs", action="store_true", help="display stats for free scanner") + ap.add_argument("-ms", action="store_true", help="display stats for migration scanner") + ap.add_argument("filter", nargs="?", help="pid|pid-range|comm-regex") + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + args = ap.parse_args() + + opt_proc = Popt.DISP_DFL + if args.pv: + opt_proc = Popt.DISP_PROC_VERBOSE + elif args.p: + opt_proc = Popt.DISP_PROC + + OPT_NS = not args.u + + opt_disp = Topt.DISP_ALL + if args.t or args.m or args.fs or args.ms: + opt_disp = Topt(0) + if args.t: + opt_disp |= Topt.DISP_TIME + if args.m: + opt_disp |= Topt.DISP_MIG + if args.fs: + opt_disp |= Topt.DISP_ISOLFREE + if args.ms: + opt_disp |= Topt.DISP_ISOLMIG + + if args.filter: + PID_PATTERN = r"^(\d*)-(\d*)$|^(\d*)$" + pid_re = re.compile(PID_PATTERN) + match = pid_re.search(args.filter) + filter_obj: Any = None + if match is not None and match.group() != "": + if match.group(3) is not None: + filter_obj = get_pid_filter(match.group(3), match.group(3)) + else: + filter_obj = get_pid_filter(match.group(1), match.group(2)) + else: + try: + comm_re = re.compile(args.filter) + except re.error: + sys.stderr.write(f"invalid regex '{args.filter}'\n") + sys.exit(1) + filter_obj = get_comm_filter(comm_re) + Chead.add_filter(filter_obj) + + session = perf.session(perf.data(args.input), sample=process_event) + session.process_events() + trace_end()
tools/perf/tests/shell/test_compaction_times_python.sh new file +81 -0
81 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 34/49] perf python: Port compaction-times to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 12 lines
Port
compaction-times.pyto a standalone script intools/perf/python/using the perf module directly to analyzemm_compactiontracepoints.Improvements compared to the legacy script:
- Replace Python 2 constructs (such as sys.maxint and raw integer bitmasks) with Python 3 enum.IntEnum (Popt) and enum.IntFlag (Topt) types.
- Access tracepoint fields directly on perf.sample_event and add -i/--input CLI option support via argparse.
- Add full type annotations passing mypy and pylint without suppression comments.
Quoted 5 lines
Add a shell test (
test_compaction_times_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=34
Ian Rogers
reviewerPATCH 35/49 perf python: Port net_dropmonitor to perf module
In reply to Ian RogersPort net_dropmonitor.py from tools/perf/scripts/python/ to a standalone script in tools/perf/python/:
- Refactor the script into a DropMonitor class with full type annotations to encapsulate state.
- Use perf.session for skb:kfree_skb event processing and add argparse CLI support (-i/--input and -k/--kallsyms).
- Resolve kernel drop addresses via perf.session symbols/callchains and binary search over /proc/kallsyms, ignoring zeroed kptr_restrict addresses with graceful fallback when kallsyms is unavailable.
- Remove Python 2 compatibility code.
Add a shell test (test_net_dropmonitor_python.sh) to verify the standalone script.
--- tools/perf/python/net_dropmonitor.py | 156 ++++++++++++++++++ .../shell/test_net_dropmonitor_python.sh | 103 ++++++++++++ 2 files changed, 259 insertions(+) create mode 100755 tools/perf/python/net_dropmonitor.py create mode 100755 tools/perf/tests/shell/test_net_dropmonitor_python.sh
tools/perf/python/net_dropmonitor.py new file +156 -0
@@ -0,0 +1,156 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +Monitor the system for dropped packets and produce a report of drop locations and counts. +Ported from tools/perf/scripts/python/net_dropmonitor.py +""" +from __future__ import annotations + +import argparse +from collections import defaultdict +import os +import sys +from typing import Tuple +import perf + + +class DropMonitor: + """Monitors dropped packets and aggregates counts by location.""" + + def __init__(self, kallsyms_path: str | None = None) -> None: + self.drop_log: dict[int, int] = defaultdict(int) + self.kallsyms: list[Tuple[int, str]] = [] + self.resolved_syms: dict[int, Tuple[str, int]] = {} + self.callchain_syms: dict[int, str] = {} + self.kallsyms_path = ( + kallsyms_path + or os.environ.get("PERF_SYMBOL_KALLSYMS") + or "/proc/kallsyms" + ) + + def _parse_kallsyms(self) -> None: + """Parse the kallsyms file and map kernel addresses to function symbols.""" + try: + with open(self.kallsyms_path, "r", encoding="utf-8") as f: + for line in f: + parts = line.split() + if len(parts) >= 3 and parts[1] in ('t', 'T', 'w', 'W'): + addr = int(parts[0], 16) + if addr > 0: + self.kallsyms.append((addr, parts[2])) + self.kallsyms.sort(key=lambda x: x[0]) + except (FileNotFoundError, PermissionError): + print(f"Failed to read {self.kallsyms_path}. Symbols will not be resolved.") + + def _get_sym(self, loc: int) -> Tuple[str, int]: + """Resolve a memory location using session symbols or the kallsyms map.""" + if loc in self.resolved_syms: + return self.resolved_syms[loc] + if not self.kallsyms: + if loc in self.callchain_syms: + return self.callchain_syms[loc], 0 + return str(loc), 0 + + start = 0 + end = len(self.kallsyms) - 1 + while start < end: + mid = (start + end) // 2 + if self.kallsyms[mid][0] <= loc < self.kallsyms[mid+1][0]: + start = mid + break + if loc < self.kallsyms[mid][0]: + end = mid - 1 + else: + start = mid + 1 + + sym_addr, sym_name = self.kallsyms[start] + if loc >= sym_addr: + return sym_name, loc - sym_addr + if loc in self.callchain_syms: + return self.callchain_syms[loc], 0 + return str(loc), 0 + + def print_drop_table(self) -> None: + """Print aggregated results.""" + if not self.drop_log: + print(f"{'LOCATION':>25} {'OFFSET':>25} {'COUNT':>25}") + return + + if len(self.resolved_syms) < len(self.drop_log): + print("Gathering kallsyms data") + self._parse_kallsyms() + + print(f"{'LOCATION':>25} {'OFFSET':>25} {'COUNT':>25}") + sorted_keys = sorted(self.drop_log.keys()) + for sloc in sorted_keys: + sym, off = self._get_sym(sloc) + print(f"{sym:>25} {off:>25d} {self.drop_log[sloc]:>25d}") + + def process_event(self, sample: perf.sample_event) -> None: + """Process a single sample event.""" + if "skb:kfree_skb" not in str(sample.evsel): + return + + location = getattr(sample, "location", None) + if location is not None: + self.drop_log[location] += 1 + if location not in self.resolved_syms: + if getattr(sample, "sample_ip", 0) == location and getattr(sample, "symbol", None): + self.resolved_syms[location] = ( + sample.symbol, + getattr(sample, "sym_offset", 0) or 0, + ) + else: + for entry in getattr(sample, "callchain", []) or []: + if isinstance(entry, dict): + entry_ip = entry.get("ip") + entry_sym = entry.get("sym") + sym_name = ( + entry_sym.get("name") if isinstance(entry_sym, dict) else None + ) + sym_start = ( + entry_sym.get("start") + if isinstance(entry_sym, dict) + else None + ) + else: + entry_ip = getattr(entry, "ip", None) + entry_sym = getattr(entry, "sym", None) + sym_name = ( + getattr(entry_sym, "name", None) + or getattr(entry, "symbol", None) + ) + sym_start = getattr(entry_sym, "start", None) + if entry_ip == location and sym_name: + if sym_start is not None: + self.resolved_syms[location] = ( + sym_name, + max(0, location - sym_start), + ) + else: + self.callchain_syms[location] = sym_name + break + + +if __name__ == "__main__": + ap = argparse.ArgumentParser( + description="Monitor the system for dropped packets and produce a " + "report of drop locations and counts.") + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + ap.add_argument("-k", "--kallsyms", default=None, + help="Path to kallsyms file for offline symbol resolution") + args = ap.parse_args() + + monitor = DropMonitor(kallsyms_path=args.kallsyms) + + try: + session = perf.session(perf.data(args.input), sample=monitor.process_event, + kallsyms=args.kallsyms) + session.process_events() + except KeyboardInterrupt: + print("\nStopping trace...") + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError) as e: + print(f"Error processing events: {e}") + sys.exit(1) + + monitor.print_drop_table()
tools/perf/tests/shell/test_net_dropmonitor_python.sh new file +103 -0
103 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 35/49] perf python: Port net_dropmonitor to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Port
net_dropmonitor.pyfromtools/perf/scripts/python/to a standalone script intools/perf/python/:
- Refactor the script into a DropMonitor class with full type annotations to encapsulate state.
- Use perf.session for skb:kfree_skb event processing and add argparse CLI support (-i/--input and -k/--kallsyms).
- Resolve kernel drop addresses via perf.session symbols/callchains and binary search over/proc/kallsyms, ignoring zeroedkptr_restrictaddresses with graceful fallback when kallsyms is unavailable.
- Remove Python 2 compatibility code.
Quoted 5 lines
Add a shell test (
test_net_dropmonitor_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=35
Ian Rogers
reviewerPATCH 36/49 perf python: Port netdev-times to perf module
In reply to Ian RogersPort netdev-times.py from tools/perf/scripts/python/ to a standalone script in tools/perf/python/:
- Refactor the script into a NetDevTimesAnalyzer class with full type annotations to encapsulate state.
- Collect events via perf.session and sort them in timestamp order before analysis so multi-CPU TX and RX packet timelines are reconstructed deterministically.
- Replace custom argument parsing with argparse (-i/--input, --tx, --rx, --dev, --debug), extract tracepoint fields directly from sample attributes, and remove Python 2 compatibility artifacts.
Add a shell test (test_netdev_times_python.sh) to verify the standalone script.
--- tools/perf/python/netdev-times.py | 486 ++++++++++++++++++ .../tests/shell/test_netdev_times_python.sh | 61 +++ 2 files changed, 547 insertions(+) create mode 100755 tools/perf/python/netdev-times.py create mode 100755 tools/perf/tests/shell/test_netdev_times_python.sh
tools/perf/python/netdev-times.py new file +486 -0
@@ -0,0 +1,486 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +Display a process of packets and processed time. +It helps us to investigate networking or network device. + +Ported from tools/perf/scripts/python/netdev-times.py +""" + +from __future__ import annotations +import argparse +from collections import defaultdict +import sys +from typing import Optional +import perf + +# Format for displaying rx packet processing +PF_IRQ_ENTRY = " irq_entry(+%.3fmsec irq=%d:%s)" +PF_SOFT_ENTRY = " softirq_entry(+%.3fmsec)" +PF_NAPI_POLL = " napi_poll_exit(+%.3fmsec %s)" +PF_JOINT = " |" +PF_WJOINT = " | |" +PF_NET_RECV = " |---netif_receive_skb(+%.3fmsec skb=%x len=%d)" +PF_NET_RX = " |---netif_rx(+%.3fmsec skb=%x)" +PF_CPY_DGRAM = " | skb_copy_datagram_iovec(+%.3fmsec %d:%s)" +PF_KFREE_SKB = " | kfree_skb(+%.3fmsec location=%x)" +PF_CONS_SKB = " | consume_skb(+%.3fmsec)" + + +class NetDevTimesAnalyzer: + """Analyzes network device events and prints charts.""" + + def __init__(self, cfg: argparse.Namespace): + self.args = cfg + self.session: Optional[perf.session] = None + self.show_tx = cfg.tx or (not cfg.tx and not cfg.rx) + self.show_rx = cfg.rx or (not cfg.tx and not cfg.rx) + self.dev = cfg.dev + self.debug = cfg.debug + self.buffer_budget = 65536 + self.irq_dic: dict[int, list[dict]] = defaultdict(list) + self.net_rx_dic: dict[int, dict] = {} + self.receive_hunk_list: list[dict] = [] + self.rx_skb_list: list[dict] = [] + self.tx_queue_list: list[dict] = [] + self.tx_xmit_list: list[dict] = [] + self.tx_free_list: list[dict] = [] + + self.buffer_budget = 65536 + self.of_count_rx_skb_list = 0 + self.of_count_tx_queue_list = 0 + self.of_count_tx_xmit_list = 0 + + def diff_msec(self, src: int, dst: int) -> float: + """Calculate a time interval(msec) from src(nsec) to dst(nsec).""" + return (dst - src) / 1000000.0 + + def print_transmit(self, hunk: dict) -> None: + """Display a process of transmitting a packet.""" + if self.dev and hunk['dev'].find(self.dev) < 0: + return + queue_t_sec = hunk['queue_t'] // 1000000000 + queue_t_usec = hunk['queue_t'] % 1000000000 // 1000 + print(f"{hunk['dev']:7s} {hunk['len']:5d} " + f"{queue_t_sec:6d}.{queue_t_usec:06d}sec " + f"{self.diff_msec(hunk['queue_t'], hunk['xmit_t']):12.3f}msec " + f"{self.diff_msec(hunk['xmit_t'], hunk['free_t']):12.3f}msec") + + def print_receive(self, hunk: dict) -> None: + """Display a process of received packets and interrupts.""" + show_hunk = False + irq_list = hunk['irq_list'] + if not irq_list: + return + cpu = irq_list[0]['cpu'] + base_t = irq_list[0]['irq_ent_t'] + + if self.dev: + for irq in irq_list: + if irq['name'].find(self.dev) >= 0: + show_hunk = True + break + else: + show_hunk = True + + if not show_hunk: + return + + base_t_sec = base_t // 1000000000 + base_t_usec = base_t % 1000000000 // 1000 + print(f"{base_t_sec}.{base_t_usec:06d}sec cpu={cpu}") + for irq in irq_list: + print(PF_IRQ_ENTRY % + (self.diff_msec(base_t, irq['irq_ent_t']), + irq['irq'], irq['name'])) + print(PF_JOINT) + irq_event_list = irq['event_list'] + for irq_event in irq_event_list: + if irq_event['event'] == 'netif_rx': + print(PF_NET_RX % + (self.diff_msec(base_t, irq_event['time']), + irq_event['skbaddr'])) + print(PF_JOINT) + + print(PF_SOFT_ENTRY % self.diff_msec(base_t, hunk['sirq_ent_t'])) + print(PF_JOINT) + event_list = hunk['event_list'] + for i, event in enumerate(event_list): + if event['event_name'] == 'napi_poll': + print(PF_NAPI_POLL % + (self.diff_msec(base_t, event['event_t']), + event['dev'])) + if i == len(event_list) - 1: + print("") + else: + print(PF_JOINT) + else: + print(PF_NET_RECV % + (self.diff_msec(base_t, event['event_t']), + event['skbaddr'], + event['len'])) + if 'handle' in event: + print(PF_WJOINT) + if event['handle'] == "kfree_skb": + print(PF_KFREE_SKB % + (self.diff_msec(base_t, event['comm_t']), + event['location'])) + elif event['handle'] == "consume_skb": + print(PF_CONS_SKB % + self.diff_msec(base_t, event['comm_t'])) + elif event['handle'] == "skb_copy_datagram_iovec": + print(PF_CPY_DGRAM % + (self.diff_msec(base_t, event['comm_t']), + event['pid'], event['comm'])) + print(PF_JOINT) + + def handle_irq_handler_entry(self, event: dict) -> None: + """Handle irq:irq_handler_entry event.""" + time = event['time'] + cpu = event['cpu'] + irq = event['irq'] + irq_name = event['irq_name'] + irq_record = {'irq': irq, 'name': irq_name, 'cpu': cpu, + 'irq_ent_t': time, 'event_list': []} + self.irq_dic[cpu].append(irq_record) + + def handle_irq_handler_exit(self, event: dict) -> None: + """Handle irq:irq_handler_exit event.""" + time = event['time'] + cpu = event['cpu'] + irq = event['irq'] + if cpu not in self.irq_dic or not self.irq_dic[cpu]: + return + if irq != self.irq_dic[cpu][-1]['irq']: + return + irq_record = self.irq_dic[cpu].pop() + irq_record['irq_ext_t'] = time + # if an irq doesn't include NET_RX softirq, drop. + if irq_record['event_list']: + self.irq_dic[cpu].append(irq_record) + + def handle_irq_softirq_raise(self, event: dict) -> None: + """Handle irq:softirq_raise event.""" + time = event['time'] + cpu = event['cpu'] + if cpu not in self.irq_dic or not self.irq_dic[cpu]: + return + irq_record = self.irq_dic[cpu].pop() + irq_record['event_list'].append({'time': time, 'event': 'sirq_raise'}) + self.irq_dic[cpu].append(irq_record) + + def handle_irq_softirq_entry(self, event: dict) -> None: + """Handle irq:softirq_entry event.""" + time = event['time'] + cpu = event['cpu'] + self.net_rx_dic[cpu] = {'sirq_ent_t': time, 'event_list': []} + + def handle_irq_softirq_exit(self, event: dict) -> None: + """Handle irq:softirq_exit event.""" + time = event['time'] + cpu = event['cpu'] + irq_list = None + event_list = None + sirq_ent_t = None + + if cpu in self.irq_dic: + irq_list = self.irq_dic[cpu] + del self.irq_dic[cpu] + if cpu in self.net_rx_dic: + sirq_ent_t = self.net_rx_dic[cpu]['sirq_ent_t'] + event_list = self.net_rx_dic[cpu]['event_list'] + del self.net_rx_dic[cpu] + if not irq_list or not event_list or sirq_ent_t is None: + return + rec_data = {'sirq_ent_t': sirq_ent_t, 'sirq_ext_t': time, + 'irq_list': irq_list, 'event_list': event_list} + self.receive_hunk_list.append(rec_data) + + def handle_napi_poll(self, event: dict) -> None: + """Handle napi:napi_poll event.""" + time = event['time'] + cpu = event['cpu'] + dev_name = event['dev_name'] + work = event['work'] + budget = event['budget'] + if cpu in self.net_rx_dic: + event_list = self.net_rx_dic[cpu]['event_list'] + rec_data = {'event_name': 'napi_poll', + 'dev': dev_name, 'event_t': time, + 'work': work, 'budget': budget} + event_list.append(rec_data) + + def handle_netif_rx(self, event: dict) -> None: + """Handle net:netif_rx event.""" + time = event['time'] + cpu = event['cpu'] + skbaddr = event['skbaddr'] + skblen = event['skblen'] + dev_name = event['dev_name'] + if cpu not in self.irq_dic or not self.irq_dic[cpu]: + return + irq_record = self.irq_dic[cpu].pop() + irq_record['event_list'].append({'time': time, 'event': 'netif_rx', + 'skbaddr': skbaddr, 'skblen': skblen, + 'dev_name': dev_name}) + self.irq_dic[cpu].append(irq_record) + + def handle_netif_receive_skb(self, event: dict) -> None: + """Handle net:netif_receive_skb event.""" + time = event['time'] + cpu = event['cpu'] + skbaddr = event['skbaddr'] + skblen = event['skblen'] + if cpu in self.net_rx_dic: + rec_data = {'event_name': 'netif_receive_skb', + 'event_t': time, 'skbaddr': skbaddr, 'len': skblen} + event_list = self.net_rx_dic[cpu]['event_list'] + event_list.append(rec_data) + self.rx_skb_list.insert(0, rec_data) + if len(self.rx_skb_list) > self.buffer_budget: + self.rx_skb_list.pop() + self.of_count_rx_skb_list += 1 + + def handle_net_dev_queue(self, event: dict) -> None: + """Handle net:net_dev_queue event.""" + time = event['time'] + skbaddr = event['skbaddr'] + skblen = event['skblen'] + dev_name = event['dev_name'] + skb = {'dev': dev_name, 'skbaddr': skbaddr, 'len': skblen, 'queue_t': time} + self.tx_queue_list.insert(0, skb) + if len(self.tx_queue_list) > self.buffer_budget: + self.tx_queue_list.pop() + self.of_count_tx_queue_list += 1 + + def handle_net_dev_xmit(self, event: dict) -> None: + """Handle net:net_dev_xmit event.""" + time = event['time'] + skbaddr = event['skbaddr'] + rc = event['rc'] + if rc == 0: # NETDEV_TX_OK + for i, skb in enumerate(self.tx_queue_list): + if skb['skbaddr'] == skbaddr: + skb['xmit_t'] = time + self.tx_xmit_list.insert(0, skb) + del self.tx_queue_list[i] + if len(self.tx_xmit_list) > self.buffer_budget: + self.tx_xmit_list.pop() + self.of_count_tx_xmit_list += 1 + return + + def handle_kfree_skb(self, event: dict) -> None: + """Handle skb:kfree_skb event.""" + time = event['time'] + skbaddr = event['skbaddr'] + comm = event['comm'] + pid = event['pid'] + location = event['location'] + for i, skb in enumerate(self.tx_queue_list): + if skb['skbaddr'] == skbaddr: + del self.tx_queue_list[i] + return + for i, skb in enumerate(self.tx_xmit_list): + if skb['skbaddr'] == skbaddr: + skb['free_t'] = time + self.tx_free_list.append(skb) + del self.tx_xmit_list[i] + return + for i, rec_data in enumerate(self.rx_skb_list): + if rec_data['skbaddr'] == skbaddr: + rec_data.update({'handle': "kfree_skb", + 'comm': comm, 'pid': pid, 'comm_t': time, 'location': location}) + del self.rx_skb_list[i] + return + + def handle_consume_skb(self, event: dict) -> None: + """Handle skb:consume_skb event.""" + time = event['time'] + skbaddr = event['skbaddr'] + for i, skb in enumerate(self.tx_xmit_list): + if skb['skbaddr'] == skbaddr: + skb['free_t'] = time + self.tx_free_list.append(skb) + del self.tx_xmit_list[i] + return + for i, rec_data in enumerate(self.rx_skb_list): + if rec_data['skbaddr'] == skbaddr: + rec_data.update({'handle': "consume_skb", 'comm_t': time}) + del self.rx_skb_list[i] + return + + def handle_skb_copy_datagram_iovec(self, event: dict) -> None: + """Handle skb:skb_copy_datagram_iovec event.""" + time = event['time'] + skbaddr = event['skbaddr'] + comm = event['comm'] + pid = event['pid'] + for i, rec_data in enumerate(self.rx_skb_list): + if skbaddr == rec_data['skbaddr']: + rec_data.update({'handle': "skb_copy_datagram_iovec", + 'comm': comm, 'pid': pid, 'comm_t': time}) + del self.rx_skb_list[i] + return + + + + def print_summary(self) -> None: + """Print charts.""" + + # display receive hunks + if self.show_rx: + for hunk in self.receive_hunk_list: + self.print_receive(hunk) + + # display transmit hunks + if self.show_tx: + print(" dev len Qdisc " + " netdevice free") + for hunk in self.tx_free_list: + self.print_transmit(hunk) + + if self.debug: + print("debug buffer status") + print("----------------------------") + print(f"xmit Qdisc:remain:{len(self.tx_queue_list)} " + f"overflow:{self.of_count_tx_queue_list}") + print(f"xmit netdevice:remain:{len(self.tx_xmit_list)} " + f"overflow:{self.of_count_tx_xmit_list}") + print(f"receive:remain:{len(self.rx_skb_list)} " + f"overflow:{self.of_count_rx_skb_list}") + + def handle_single_event(self, event: dict) -> None: + """Handle a single processed event.""" + name = event['name'] + if name == 'irq:softirq_exit': + self.handle_irq_softirq_exit(event) + elif name == 'irq:softirq_entry': + self.handle_irq_softirq_entry(event) + elif name == 'irq:softirq_raise': + self.handle_irq_softirq_raise(event) + elif name == 'irq:irq_handler_entry': + self.handle_irq_handler_entry(event) + elif name == 'irq:irq_handler_exit': + self.handle_irq_handler_exit(event) + elif name == 'napi:napi_poll': + self.handle_napi_poll(event) + elif name == 'net:netif_receive_skb': + self.handle_netif_receive_skb(event) + elif name == 'net:netif_rx': + self.handle_netif_rx(event) + elif name == 'skb:skb_copy_datagram_iovec': + self.handle_skb_copy_datagram_iovec(event) + elif name == 'net:net_dev_queue': + self.handle_net_dev_queue(event) + elif name == 'net:net_dev_xmit': + self.handle_net_dev_xmit(event) + elif name == 'skb:kfree_skb': + self.handle_kfree_skb(event) + elif name == 'skb:consume_skb': + self.handle_consume_skb(event) + + def process_event(self, sample: perf.sample_event) -> None: + """Process events directly on-the-fly.""" + name = str(sample.evsel) + ev_name = name[6:-1] if name.startswith("evsel(") else name + pid = sample.sample_pid + if hasattr(self, 'session') and self.session: + try: + thread = self.session.find_thread(pid) + comm = (thread.comm() if thread else None) or "[unknown]" + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError): + comm = "[unknown]" + else: + comm = "Unknown" + event_data = { + 'name': ev_name, + 'time': sample.sample_time, + 'cpu': sample.sample_cpu, + 'pid': pid, + 'comm': comm, + } + + # Extract specific fields based on event type + if ev_name.startswith("irq:softirq_"): + event_data['vec'] = getattr(sample, "vec", 0) + # Filter for NET_RX + if event_data['vec'] != 3: # NET_RX_SOFTIRQ is usually 3 + return + elif ev_name == "irq:irq_handler_entry": + event_data['irq'] = getattr(sample, "irq", -1) + event_data['irq_name'] = getattr(sample, "name", "[unknown]") + elif ev_name == "irq:irq_handler_exit": + event_data['irq'] = getattr(sample, "irq", -1) + event_data['ret'] = getattr(sample, "ret", 0) + elif ev_name == "napi:napi_poll": + event_data['napi'] = getattr(sample, "napi", 0) + event_data['dev_name'] = getattr(sample, "dev_name", "[unknown]") + event_data['work'] = getattr(sample, "work", 0) + event_data['budget'] = getattr(sample, "budget", 0) + elif ev_name in ("net:netif_receive_skb", "net:netif_rx", + "net:net_dev_queue"): + event_data['skbaddr'] = getattr(sample, "skbaddr", 0) + event_data['skblen'] = getattr(sample, "len", 0) + event_data['dev_name'] = getattr(sample, "name", "[unknown]") + elif ev_name == "net:net_dev_xmit": + event_data['skbaddr'] = getattr(sample, "skbaddr", 0) + event_data['skblen'] = getattr(sample, "len", 0) + event_data['rc'] = getattr(sample, "rc", 0) + event_data['dev_name'] = getattr(sample, "name", "[unknown]") + elif ev_name == "skb:kfree_skb": + event_data['skbaddr'] = getattr(sample, "skbaddr", 0) + event_data['location'] = getattr(sample, "location", 0) + event_data['protocol'] = getattr(sample, "protocol", 0) + event_data['reason'] = getattr(sample, "reason", 0) + elif ev_name == "skb:consume_skb": + event_data['skbaddr'] = getattr(sample, "skbaddr", 0) + event_data['location'] = getattr(sample, "location", 0) + elif ev_name == "skb:skb_copy_datagram_iovec": + event_data['skbaddr'] = getattr(sample, "skbaddr", 0) + event_data['skblen'] = getattr(sample, "len", 0) + + self.handle_single_event(event_data) + + +if __name__ == "__main__": + ap = argparse.ArgumentParser(description="Display a process of packets and processed time.") + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + ap.add_argument("--tx", action="store_true", help="show only tx chart") + ap.add_argument("--rx", action="store_true", help="show only rx chart") + ap.add_argument("--dev", default=None, help="show only specified device") + ap.add_argument("--debug", action="store_true", + help="work with debug mode. It shows buffer status.") + ap.add_argument("positionals", nargs="*", + help="optional positional arguments (tx, rx, dev=<name>, debug)") + args, unknown_args = ap.parse_known_args() + + parsed_args = argparse.Namespace( + tx=args.tx, rx=args.rx, dev=args.dev, debug=args.debug, input=args.input + ) + + for arg in list(args.positionals) + unknown_args: + if not arg or not isinstance(arg, str): + continue + if arg in ('tx', '--tx'): + parsed_args.tx = True + elif arg in ('rx', '--rx'): + parsed_args.rx = True + elif arg.startswith('dev='): + parsed_args.dev = arg[4:] + elif arg.startswith('--dev='): + parsed_args.dev = arg[6:] + elif arg in ('debug', '--debug'): + parsed_args.debug = True + + analyzer = NetDevTimesAnalyzer(parsed_args) + + try: + session = perf.session(perf.data(parsed_args.input), sample=analyzer.process_event) + analyzer.session = session + session.process_events() + analyzer.print_summary() + except KeyboardInterrupt: + analyzer.print_summary() + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError) as e: + print(f"Error processing events: {e}") + sys.exit(1)
tools/perf/tests/shell/test_netdev_times_python.sh new file +61 -0
61 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 36/49] perf python: Port netdev-times to perf module
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf: netdev-times drops softirq entries without packets --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf: netdev-times drops softirq entries without packets
--
commit c6636c9bd9c585465111cf327e6ff4a6ca1d6bf2
Author: Ian Rogers <[email protected]>
perf python: Port netdev-times to perf module
This commit ports netdev-times.py to a standalone script in tools/perf/python/, converting it to use a NetDevTimesAnalyzer class with type annotations. It replaces custom arg parsing with argparse and adds a shell test to verify the standalone script.
Quoted 5 lines
tools/perf/python/netdev-times.py new file +0
-0
[ ... ]
Quoted 7 lines
tools/perf/python/netdev-times.py new file +7
-0+ def handle_irq_softirq_exit(self, event: dict) -> None: + """Handle irq:softirq_exit event.""" + time = event['time'] + cpu = event['cpu'] + irq_list = None + event_list = None + sirq_ent_t = None
tools/perf/python/netdev-times.py line 194
Quoted 10 lines
tools/perf/python/netdev-times.py new file +10
-0+ + if cpu in self.irq_dic: + irq_list = self.irq_dic[cpu] + del self.irq_dic[cpu] + if cpu in self.net_rx_dic: + sirq_ent_t = self.net_rx_dic[cpu]['sirq_ent_t'] + event_list = self.net_rx_dic[cpu]['event_list'] + del self.net_rx_dic[cpu] + if not irq_list or not event_list or sirq_ent_t is None: + return
[Severity: Medium]
Will this condition incorrectly drop valid softirq trace records when event_list is empty?
When a NET_RX softirq runs but processes no packets, event_list evaluates to an empty list []. Because not [] evaluates to True in Python, this causes the function to prematurely return and silently drop the receive hunk.
In the original script, the uninitialized variable was 0, so the check correctly evaluated to False for [], allowing empty softirq executions to be processed and printed. Should this check be updated to explicitly verify if event_list is None rather than just evaluating its truthiness?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=36
Ian Rogers
reviewerPATCH 37/49 perf python: Port check-perf-trace to perf module
In reply to Ian RogersPort check-perf-trace.py to a standalone script in tools/perf/python/ using the perf module directly.
Improvements compared to the legacy script:
- Access tracepoint fields directly as attributes on perf.sample_event instead of per-event dictionaries and legacy Perf-Trace-Util helpers.
- Decode symbolic flag and enum masks for irq:softirq_entry and kmem:kmalloc directly in Python and add -i/--input CLI support via argparse.
- Add full type annotations and clean up Python 2 idioms.
Add a shell test (test_check_perf_trace_python.sh) to verify the standalone script.
--- tools/perf/python/check-perf-trace.py | 213 ++++++++++++++++++ .../shell/test_check_perf_trace_python.sh | 79 +++++++ 2 files changed, 292 insertions(+) create mode 100755 tools/perf/python/check-perf-trace.py create mode 100755 tools/perf/tests/shell/test_check_perf_trace_python.sh
tools/perf/python/check-perf-trace.py new file +213 -0
@@ -0,0 +1,213 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +Basic test of Python scripting support for perf. +Ported from tools/perf/scripts/python/check-perf-trace.py +""" +from __future__ import annotations + +import argparse +import collections +import perf + +unhandled: collections.defaultdict[str, int] = collections.defaultdict(int) +session = None + +softirq_vecs = { + 0: "HI_SOFTIRQ", + 1: "TIMER_SOFTIRQ", + 2: "NET_TX_SOFTIRQ", + 3: "NET_RX_SOFTIRQ", + 4: "BLOCK_SOFTIRQ", + 5: "IRQ_POLL_SOFTIRQ", + 6: "TASKLET_SOFTIRQ", + 7: "SCHED_SOFTIRQ", + 8: "HRTIMER_SOFTIRQ", + 9: "RCU_SOFTIRQ", +} + +_GFP_DMA = 1 << 0 +_GFP_HIGHMEM = 1 << 1 +_GFP_DMA32 = 1 << 2 +_GFP_MOVABLE = 1 << 3 +_GFP_RECLAIMABLE = 1 << 4 +_GFP_HIGH = 1 << 5 +_GFP_IO = 1 << 6 +_GFP_FS = 1 << 7 +_GFP_ZERO = 1 << 8 +_GFP_DIRECT_RECLAIM = 1 << 10 +_GFP_KSWAPD_RECLAIM = 1 << 11 +_GFP_WRITE = 1 << 12 +_GFP_NOWARN = 1 << 13 +_GFP_RETRY_MAYFAIL = 1 << 14 +_GFP_NOFAIL = 1 << 15 +_GFP_NORETRY = 1 << 16 +_GFP_MEMALLOC = 1 << 17 +_GFP_COMP = 1 << 18 +_GFP_NOMEMALLOC = 1 << 19 +_GFP_HARDWALL = 1 << 20 +_GFP_THISNODE = 1 << 21 +_GFP_ACCOUNT = 1 << 22 +_GFP_ZEROTAGS = 1 << 23 + +_GFP_RECLAIM = _GFP_DIRECT_RECLAIM | _GFP_KSWAPD_RECLAIM +_GFP_KERNEL = _GFP_RECLAIM | _GFP_IO | _GFP_FS +_GFP_USER = _GFP_KERNEL | _GFP_HARDWALL +_GFP_HIGHUSER = _GFP_USER | _GFP_HIGHMEM +_GFP_HIGHUSER_MOVABLE = _GFP_HIGHUSER | _GFP_MOVABLE +_GFP_TRANSHUGE_LIGHT = ( + _GFP_HIGHUSER_MOVABLE | _GFP_COMP | _GFP_NOMEMALLOC | _GFP_NOWARN +) & ~_GFP_RECLAIM +_GFP_TRANSHUGE = _GFP_TRANSHUGE_LIGHT | _GFP_DIRECT_RECLAIM + +GFP_FLAG_NAMES = [ + (_GFP_TRANSHUGE, "GFP_TRANSHUGE"), + (_GFP_TRANSHUGE_LIGHT, "GFP_TRANSHUGE_LIGHT"), + (_GFP_HIGHUSER_MOVABLE, "GFP_HIGHUSER_MOVABLE"), + (_GFP_HIGHUSER, "GFP_HIGHUSER"), + (_GFP_USER, "GFP_USER"), + (_GFP_KERNEL | _GFP_ACCOUNT, "GFP_KERNEL_ACCOUNT"), + (_GFP_KERNEL, "GFP_KERNEL"), + (_GFP_RECLAIM | _GFP_IO, "GFP_NOFS"), + (_GFP_HIGH | _GFP_KSWAPD_RECLAIM, "GFP_ATOMIC"), + (_GFP_RECLAIM, "GFP_NOIO"), + (_GFP_KSWAPD_RECLAIM | _GFP_NOWARN, "GFP_NOWAIT"), + (_GFP_DMA, "GFP_DMA"), + (_GFP_DMA32, "GFP_DMA32"), + (_GFP_RECLAIM, "__GFP_RECLAIM"), + (_GFP_DMA, "__GFP_DMA"), + (_GFP_HIGHMEM, "__GFP_HIGHMEM"), + (_GFP_DMA32, "__GFP_DMA32"), + (_GFP_MOVABLE, "__GFP_MOVABLE"), + (_GFP_RECLAIMABLE, "__GFP_RECLAIMABLE"), + (_GFP_HIGH, "__GFP_HIGH"), + (_GFP_IO, "__GFP_IO"), + (_GFP_FS, "__GFP_FS"), + (_GFP_ZERO, "__GFP_ZERO"), + (_GFP_DIRECT_RECLAIM, "__GFP_DIRECT_RECLAIM"), + (_GFP_KSWAPD_RECLAIM, "__GFP_KSWAPD_RECLAIM"), + (_GFP_WRITE, "__GFP_WRITE"), + (_GFP_NOWARN, "__GFP_NOWARN"), + (_GFP_RETRY_MAYFAIL, "__GFP_RETRY_MAYFAIL"), + (_GFP_NOFAIL, "__GFP_NOFAIL"), + (_GFP_NORETRY, "__GFP_NORETRY"), + (_GFP_MEMALLOC, "__GFP_MEMALLOC"), + (_GFP_COMP, "__GFP_COMP"), + (_GFP_NOMEMALLOC, "__GFP_NOMEMALLOC"), + (_GFP_HARDWALL, "__GFP_HARDWALL"), + (_GFP_THISNODE, "__GFP_THISNODE"), + (_GFP_ACCOUNT, "__GFP_ACCOUNT"), + (_GFP_ZEROTAGS, "__GFP_ZEROTAGS"), +] + + +def trace_begin() -> None: + """Called at the start of trace processing.""" + print("trace_begin") + +def trace_end() -> None: + """Called at the end of trace processing.""" + print_unhandled() + print("trace_end") + +def symbol_str(event_name: str, field_name: str, value: int) -> str: + """Resolves symbol values to strings.""" + # Note: The standalone Python API currently lacks dynamic libtraceevent + # formatting (equivalent to _perf_trace_context.symbol_str()) + if event_name == "irq__softirq_entry" and field_name == "vec": + return softirq_vecs.get(value, str(value)) + return str(value) + +def flag_str(event_name: str, field_name: str, value: int) -> str: + """Resolves flag values to strings.""" + # Note: The standalone Python API currently lacks dynamic libtraceevent + # formatting (equivalent to _perf_trace_context.flag_str()) + if event_name == "kmem__kmalloc" and field_name == "gfp_flags": + if value == 0: + return "none" + names = [] + rem = value + for mask, name in GFP_FLAG_NAMES: + if (rem & mask) == mask: + names.append(name) + rem &= ~mask + if rem: + names.append(f"0x{rem:x}") + return "|".join(names) + return str(value) + +def print_header(event_name: str, sample: perf.sample_event) -> None: + """Prints common header for events.""" + secs = sample.sample_time // 1000000000 + nsecs = sample.sample_time % 1000000000 + comm = "[unknown]" + try: + if session: + thread = session.find_thread(sample.sample_tid) + if thread: + comm = thread.comm() or "[unknown]" + except (TypeError, AttributeError): + pass + print(f"{event_name:<20} {sample.sample_cpu:5} {secs:05}.{nsecs:09} " + f"{sample.sample_tid:8} {comm:<20} ", end=' ') + +def print_uncommon(sample: perf.sample_event) -> None: + """Prints uncommon fields for tracepoints.""" + # Fallback to 0 if field not found (e.g. on older kernels or if not tracepoint) + pc = getattr(sample, 'common_preempt_count', 0) + flags = getattr(sample, 'common_flags', 0) + lock_depth = getattr(sample, 'common_lock_depth', 0) + + print(f"common_preempt_count={pc}, common_flags={flags}, " + f"common_lock_depth={lock_depth}, ", end='') + +def irq__softirq_entry(sample: perf.sample_event) -> None: + """Handles irq:softirq_entry events.""" + print_header("irq__softirq_entry", sample) + print_uncommon(sample) + print(f"vec={symbol_str('irq__softirq_entry', 'vec', getattr(sample, 'vec', 0))}") + +def kmem__kmalloc(sample: perf.sample_event) -> None: + """Handles kmem:kmalloc events.""" + print_header("kmem__kmalloc", sample) + print_uncommon(sample) + + print(f"call_site={getattr(sample, 'call_site', 0):#x}, " + f"ptr={getattr(sample, 'ptr', 0):#x}, " + f"bytes_req={getattr(sample, 'bytes_req', 0):d}, " + f"bytes_alloc={getattr(sample, 'bytes_alloc', 0):d}, " + f"gfp_flags={flag_str('kmem__kmalloc', 'gfp_flags', getattr(sample, 'gfp_flags', 0))}") + +def trace_unhandled(event_name: str) -> None: + """Tracks unhandled events.""" + unhandled[event_name] += 1 + +def print_unhandled() -> None: + """Prints summary of unhandled events.""" + if not unhandled: + return + print("\nunhandled events:\n") + print(f"{'event':<40} {'count':>10}") + print("---------------------------------------- -----------") + for event_name, count in unhandled.items(): + print(f"{event_name:<40} {count:10}") + +def process_event(sample: perf.sample_event) -> None: + """Callback for processing events.""" + event_name = str(sample.evsel) + if event_name.startswith("evsel(irq:softirq_entry)"): + irq__softirq_entry(sample) + elif "evsel(kmem:kmalloc)" in event_name: + kmem__kmalloc(sample) + else: + trace_unhandled(event_name) + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + args = ap.parse_args() + + trace_begin() + session = perf.session(perf.data(args.input), sample=process_event) + session.process_events() + trace_end()
tools/perf/tests/shell/test_check_perf_trace_python.sh new file +79 -0
79 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 37/49] perf python: Port check-perf-trace to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Port
check-perf-trace.pyto a standalone script intools/perf/python/using the perf module directly.Improvements compared to the legacy script:
- Access tracepoint fields directly as attributes on perf.sample_event instead of per-event dictionaries and legacy Perf-Trace-Util helpers.
- Decode symbolic flag and enum masks for irq:softirq_entry and kmem:kmalloc directly in Python and add -i/--input CLI support via argparse.
- Add full type annotations and clean up Python 2 idioms.
Quoted 5 lines
Add a shell test (
test_check_perf_trace_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=37
Ian Rogers
reviewerPATCH 38/49 perf python: Port arm-cs-trace-disasm to perf module
In reply to Ian RogersPort arm-cs-trace-disasm.py to a standalone script in tools/perf/python/ using the perf module directly.
Improvements compared to the legacy script:
- Encapsulate trace disassembly state in a TraceDisasm class and migrate CLI parsing from deprecated optparse to argparse.
- Automatically search standard kernel debug paths (find_vmlinux()) when -k/--vmlinux is not specified, and query perf.config_get("annotate.objdump") for the default objdump binary.
- Bound DISASM_CACHE memory consumption by evicting the cache at 1024 entries and skipping caching of oversized (> 512 lines) objdump outputs.
- Use sample.srccode() from the perf extension module to annotate disassembly output with source filenames, line numbers, and source lines.
Update the ARM CoreSight disassembly shell test (test_arm_coresight_disasm.sh) to invoke the standalone script.
--- tools/perf/python/arm-cs-trace-disasm.py | 350 ++++++++++++++++++ .../coresight/test_arm_coresight_disasm.sh | 27 +- 2 files changed, 371 insertions(+), 6 deletions(-) create mode 100755 tools/perf/python/arm-cs-trace-disasm.py
tools/perf/python/arm-cs-trace-disasm.py new file +350 -0
@@ -0,0 +1,350 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +arm-cs-trace-disasm.py: ARM CoreSight Trace Dump With Disassember using perf python module +""" +from __future__ import annotations + +import os +from os import path +import re +from subprocess import check_output +import argparse +import platform +import sys +from typing import Dict, List, Optional + +import perf + +# Initialize global dicts and regular expression +DISASM_CACHE: Dict[str, List[str]] = {} +CPU_DATA: Dict[str, int] = {} +DISASM_RE = re.compile(r"^\s*([0-9a-fA-F]+):") +DISASM_FUNC_RE = re.compile(r"^\s*([0-9a-fA-F]+)\s.*:") +CACHE_SIZE = 1024 +class _State: + sample_idx: int = -1 + source_file_name: Optional[str] = None + line_number: Optional[int] = None + dso: Optional[str] = None + +_STATE = _State() + +KVER = platform.release() +VMLINUX_PATHS = [ + f"/usr/lib/debug/boot/vmlinux-{KVER}.debug", + f"/usr/lib/debug/lib/modules/{KVER}/vmlinux", + f"/lib/modules/{KVER}/build/vmlinux", + f"/usr/lib/debug/boot/vmlinux-{KVER}", + f"/boot/vmlinux-{KVER}", + "/boot/vmlinux", + "vmlinux" +] + +def default_objdump() -> str: + """Return the default objdump path from perf config or 'objdump'.""" + try: + config = perf.config_get("annotate.objdump") + return str(config) if config else "objdump" + except (AttributeError, TypeError): + return "objdump" + +def find_vmlinux() -> Optional[str]: + """Find the vmlinux file in standard paths.""" + if hasattr(find_vmlinux, "path"): + return getattr(find_vmlinux, "path") + + for v in VMLINUX_PATHS: + if os.access(v, os.R_OK): + setattr(find_vmlinux, "path", v) + return v + setattr(find_vmlinux, "path", None) + return None + +def get_dso_file_path(dso_name: str, dso_build_id: str, vmlinux: Optional[str]) -> str: + """Return the path to the DSO file.""" + buildid_dir = os.environ.get('PERF_BUILDID_DIR') + if not buildid_dir: + buildid_dir = os.path.join(os.environ.get('HOME', ''), '.debug') + + if dso_name in ("[kernel.kallsyms]", "vmlinux"): + if vmlinux: + return vmlinux + if dso_build_id: + for kname in (dso_name, "vmlinux", "[kernel.kallsyms]"): + candidate = os.path.join(buildid_dir, kname, dso_build_id, "elf") + if os.access(candidate, os.R_OK): + return candidate + return find_vmlinux() or dso_name + + if dso_name == "[vdso]": + append = "/vdso" + else: + append = "/elf" + + dso_path = buildid_dir + "/" + dso_name + "/" + dso_build_id + append + # Replace duplicate slash chars to single slash char + dso_path = dso_path.replace('//', '/', 1) + return dso_path + +def read_disam(dso_fname: str, dso_start: int, start_addr: int, + stop_addr: int, objdump: str) -> List[str]: + """Read disassembly from a DSO file using objdump.""" + addr_range = f"{start_addr}:{stop_addr}:{dso_start}:{dso_fname}" + + # Don't let the cache get too big, clear it when it hits max size + if len(DISASM_CACHE) > CACHE_SIZE: + DISASM_CACHE.clear() + + if addr_range in DISASM_CACHE: + disasm_output = DISASM_CACHE[addr_range] + else: + start_addr = start_addr - dso_start + stop_addr = stop_addr - dso_start + disasm = [objdump, "-d", "-z", + f"--start-address={start_addr:#x}", + f"--stop-address={stop_addr:#x}"] + disasm += [dso_fname] + disasm_output = check_output(disasm).decode('utf-8').split('\n') + if len(disasm_output) <= 512: + DISASM_CACHE[addr_range] = disasm_output + + return disasm_output + +def print_disam(dso_fname: str, dso_start: int, start_addr: int, + stop_addr: int, objdump: str) -> None: + """Print disassembly for a given address range.""" + for line in read_disam(dso_fname, dso_start, start_addr, stop_addr, objdump): + m = DISASM_FUNC_RE.search(line) + if m is None: + m = DISASM_RE.search(line) + if m is None: + continue + print(f"\t{line}") + +def print_sample(sample: perf.sample_event) -> None: + """Print sample details.""" + print(f"Sample = {{ cpu: {sample.sample_cpu:04d} addr: {sample.sample_addr:016x} " + f"phys_addr: {sample.sample_phys_addr:016x} ip: {sample.sample_ip:016x} " + f"pid: {sample.sample_pid} tid: {sample.sample_tid} period: {sample.sample_period} " + f"time: {sample.sample_time} index: {_STATE.sample_idx}}}") + +def common_start_str(comm: str, sample: perf.sample_event) -> str: + """Return common start string for sample output.""" + sec = int(sample.sample_time / 1000000000) + ns = sample.sample_time % 1000000000 + cpu = sample.sample_cpu + pid = sample.sample_pid + tid = sample.sample_tid + return f"{comm:>16s} {pid:5d}/{tid:<5d} [{cpu:04d}] {sec:9d}.{ns:09d} " + +def print_srccode(comm: str, sample: perf.sample_event, symbol: str, dso: str) -> None: + """Print source code and symbols for a sample.""" + ip = sample.sample_ip + if symbol == "[unknown]": + start_str = common_start_str(comm, sample) + f"{ip:x}".rjust(16).ljust(40) + else: + symoff = 0 + symoff = getattr(sample, 'sym_offset', 0) or 0 + offs = f"+{symoff:#x}" if symoff != 0 else "" + start_str = common_start_str(comm, sample) + (symbol + offs).ljust(40) + + source_file_name, line_number, source_line = sample.srccode() or (None, 0, None) + if source_file_name: + if _STATE.line_number == line_number and _STATE.source_file_name == source_file_name: + src_str = "" + else: + if len(source_file_name) > 40: + src_file = f"...{source_file_name[-37:]} " + else: + src_file = source_file_name.ljust(41) + + if source_line is None: + src_str = f"{src_file}{line_number:>4d} <source not found>" + else: + src_str = f"{src_file}{line_number:>4d} {source_line}" + _STATE.dso = None + elif dso == _STATE.dso: + src_str = "" + else: + src_str = dso + _STATE.dso = dso + + _STATE.line_number = line_number + _STATE.source_file_name = source_file_name + + print(start_str, src_str) + +class TraceDisasm: + """Class to handle trace disassembly.""" + def __init__(self, cli_options: argparse.Namespace): + self.options = cli_options + self.sample_idx = -1 + self.session: Optional[perf.session] = None + + def process_event(self, sample: perf.sample_event) -> None: + """Process a single perf event.""" + self.sample_idx += 1 + _STATE.sample_idx = self.sample_idx + + if self.options.start_time is not None and sample.sample_time < self.options.start_time: + return + if self.options.stop_time is not None and sample.sample_time > self.options.stop_time: + sys.exit(0) + if self.options.start_sample is not None and self.sample_idx < self.options.start_sample: + return + if self.options.stop_sample is not None and self.sample_idx > self.options.stop_sample: + sys.exit(0) + + ev_name = str(sample.evsel) + if self.options.verbose: + print(f"Event type: {ev_name}") + print_sample(sample) + + dso = sample.dso or '[unknown]' + symbol = sample.symbol or '[unknown]' + dso_bid = (sample.dso_bid.decode('utf-8') + if isinstance(sample.dso_bid, bytes) + else str(sample.dso_bid or '[unknown]')) + dso_start = sample.map_start + dso_end = sample.map_end + map_pgoff = sample.map_pgoff or 0 + + comm = "[unknown]" + try: + if self.session: + thread_info = self.session.find_thread(sample.sample_tid) + if thread_info: + comm = thread_info.comm() or "[unknown]" + except (TypeError, AttributeError): + pass + + if dso == '[unknown]': + return + + if dso_start is None or dso_end is None: + print(f"Failed to find valid dso map for dso {dso}") + return + + if "instructions" in ev_name: + print_srccode(comm, sample, symbol, dso) + return + + if "branches" not in ev_name: + return + + self._process_branch(sample, comm, symbol, dso, dso_bid, dso_start, dso_end, map_pgoff) + + def _process_branch(self, sample: perf.sample_event, comm: str, symbol: str, dso: str, + dso_bid: str, dso_start: int, dso_end: int, map_pgoff: int) -> None: + """Helper to process branch events.""" + cpu = sample.sample_cpu + ip = sample.sample_ip + addr = sample.sample_addr + + if CPU_DATA.get(str(cpu) + 'addr') is None: + CPU_DATA[str(cpu) + 'addr'] = addr + return + + start_addr = CPU_DATA[str(cpu) + 'addr'] + stop_addr = ip + 4 + + # Record for previous sample packet + CPU_DATA[str(cpu) + 'addr'] = addr + + # Filter out zero start_address. Optionally identify CS_ETM_TRACE_ON packet + if start_addr == 0: + if stop_addr == 4 and self.options.verbose: + print(f"CPU{cpu}: CS_ETM_TRACE_ON packet is inserted") + return + + if start_addr < dso_start or start_addr > dso_end: + print(f"Start address {start_addr:#x} is out of range [ {dso_start:#x} .. " + f"{dso_end:#x} ] for dso {dso}") + return + + if stop_addr < dso_start or stop_addr > dso_end: + print(f"Stop address {stop_addr:#x} is out of range [ {dso_start:#x} .. " + f"{dso_end:#x} ] for dso {dso}") + return + + if self.options.objdump is not None: + if dso == "[kernel.kallsyms]": + dso_vm_start = 0 + map_pgoff_local = 0 + elif dso_start == 0x400000: + dso_vm_start = 0 + map_pgoff_local = 0 + else: + dso_vm_start = dso_start + map_pgoff_local = map_pgoff + + dso_fname = get_dso_file_path(dso, dso_bid, self.options.vmlinux) + if path.exists(dso_fname): + print_disam(dso_fname, dso_vm_start, start_addr + map_pgoff_local, + stop_addr + map_pgoff_local, self.options.objdump) + else: + print(f"Failed to find dso {dso} for address range [ " + f"{start_addr + map_pgoff_local:#x} .. {stop_addr + map_pgoff_local:#x} ]") + + print_srccode(comm, sample, symbol, dso) + + def run(self) -> None: + """Run the trace disassembly session.""" + input_file = self.options.input or "perf.data" + if input_file != "-" and not os.path.exists(input_file): + print(f"Error: {input_file} not found.", file=sys.stderr) + sys.exit(1) + + print('ARM CoreSight Trace Data Assembler Dump') + try: + self.session = perf.session( + perf.data(input_file), + sample=self.process_event, + itrace=self.options.itrace + ) + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError) as e: + print(f"Error opening session: {e}", file=sys.stderr) + sys.exit(1) + + self.session.process_events() + print('End') + +if __name__ == "__main__": + def int_arg(v: str) -> int: + """Helper for integer command line arguments.""" + val = int(v) + if val < 0: + raise argparse.ArgumentTypeError("Argument must be a positive integer") + return val + + arg_parser = argparse.ArgumentParser(description="ARM CoreSight Trace Dump With Disassembler") + arg_parser.add_argument("-i", "--input", help="input perf.data file") + arg_parser.add_argument("-k", "--vmlinux", + help="Set path to vmlinux file. Omit to autodetect") + arg_parser.add_argument("-d", "--objdump", nargs="?", const=default_objdump(), + help="Show disassembly. Can also be used to change the objdump path") + arg_parser.add_argument("-v", "--verbose", action="store_true", help="Enable debugging log") + arg_parser.add_argument("--start-time", type=int_arg, + help="Monotonic clock time of sample to start from.") + arg_parser.add_argument("--stop-time", type=int_arg, + help="Monotonic clock time of sample to stop at.") + arg_parser.add_argument("--itrace", default="b", + help="Instruction tracing options.") + arg_parser.add_argument("--start-sample", type=int_arg, + help="Index of sample to start from.") + arg_parser.add_argument("--stop-sample", type=int_arg, + help="Index of sample to stop at.") + + parsed_options = arg_parser.parse_args() + if (parsed_options.start_time is not None and parsed_options.stop_time is not None and + parsed_options.start_time >= parsed_options.stop_time): + print("--start-time must less than --stop-time") + sys.exit(2) + if (parsed_options.start_sample is not None and parsed_options.stop_sample is not None and + parsed_options.start_sample >= parsed_options.stop_sample): + print("--start-sample must less than --stop-sample") + sys.exit(2) + + td = TraceDisasm(parsed_options) + td.run()
tools/perf/tests/shell/coresight/test_arm_coresight_disasm.sh +21 -6
@@ -1,6 +1,6 @@ #!/bin/bash -# Check Arm CoreSight disassembly script completes without errors (exclusive) # SPDX-License-Identifier: GPL-2.0 +# Check Arm CoreSight disassembly script completes without errors (exclusive) # The disassembly script reconstructs ranges of instructions and gives these to objdump to # decode. objdump doesn't like ranges that go backwards, but these are a good indication
@@ -22,9 +22,10 @@ glb_err=1 perfdata_dir=$(mktemp -d /tmp/__perf_test.perf.data.XXXXX) perfdata=${perfdata_dir}/perf.data +perfdata2=${perfdata_dir}/perf2.data file=$(mktemp /tmp/temporary_file.XXXXX) # Relative path works whether it's installed or running from repo -script_path=$(dirname "$0")/../../../scripts/python/arm-cs-trace-disasm.py +script_path=$(dirname "$0")/../../../python/arm-cs-trace-disasm.py cleanup_files() {
@@ -44,8 +45,15 @@ branch_search='[[:space:]](bl|b(\.(eq|ne|cs|cc|mi|pl|vs|vc|hi|ls|ge|lt|gt|le|al) if [ "$(id -u)" == 0 ] && [ -e /proc/kcore ]; then echo "Testing kernel disassembly" perf record -o ${perfdata} -e cs_etm//k --kcore -Se -m,64K -- touch $file > /dev/null 2>&1 - perf script -i ${perfdata} --itrace=b -s python:${script_path} -- \ - -d --stop-sample=2 -k ${perfdata}/kcore_dir/kcore 2> /dev/null > ${file} + # shellcheck source=lib/setup_python.sh + . "$(dirname "$0")"/../lib/setup_python.sh + $PYTHON -c "import perf" 2>/dev/null || { + echo "Skipping test, perf python module not found" + glb_err=2 + exit 2 + } + $PYTHON ${script_path} -i ${perfdata} --itrace=b -d --stop-sample=2 \ + -k ${perfdata}/kcore_dir/kcore 2> /dev/null > ${file} grep -q -E ${branch_search} ${file} echo "Found kernel branches" else
@@ -56,8 +64,15 @@ fi ## Test user ## echo "Testing userspace disassembly" perf record -o ${perfdata} -e cs_etm//u -Se -m,64K -- touch $file > /dev/null 2>&1 -perf script -i ${perfdata} --itrace=b -s python:${script_path} -- \ - -d --stop-sample=2 2> /dev/null > ${file} +perf inject --itrace=b -i ${perfdata} -o ${perfdata2} +# shellcheck source=lib/setup_python.sh +. "$(dirname "$0")"/../lib/setup_python.sh +$PYTHON -c "import perf" 2>/dev/null || { + echo "Skipping test, perf python module not found" + glb_err=2 + exit 2 +} +$PYTHON ${script_path} -i ${perfdata2} -d --stop-sample=2 2> /dev/null > ${file} grep -q -E ${branch_search} ${file} echo "Found userspace branches"
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 38/49] perf python: Port arm-cs-trace-disasm to perf module
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] perf test: coresight disasm test skips due to bad PYTHONPATH --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf test: coresight disasm test skips due to bad PYTHONPATH
--
commit c62fb057c9bb2395ae6702ac71b41a2e2b1788ed
Author: Ian Rogers <[email protected]>
perf python: Port arm-cs-trace-disasm to perf module
This commit ports the arm-cs-trace-disasm.py script to a standalone Python script using the perf module. It also updates the ARM CoreSight disassembly shell test to invoke the new standalone script instead of using 'perf script -s'.
Quoted 4 lines
tools/perf/tests/shell/coresight/test_arm_coresight_disasm.sh +0
-0
[ ... ]
tools/perf/tests/shell/coresight/test_arm_coresight_disasm.sh branch_search='[[:space:]](bl|b(\.(eq|ne|cs|cc|mi|pl|vs|vc|hi|ls|ge|lt|gt|le|al) · line 49
Quoted 8 lines
tools/perf/tests/shell/coresight/test_arm_coresight_disasm.sh +2
-2@@ -44,8 +45,15 @@ branch_search='[[:space:]](bl|b(\.(eq|ne|cs|cc|mi|pl|vs|vc|hi|ls|ge|lt|gt|le|al) if [ "$(id -u)" == 0 ] && [ -e /proc/kcore ]; then echo "Testing kernel disassembly" perf record -o ${perfdata} -e cs_etm//k --kcore -Se -m,64K -- touch $file > /dev/null 2>&1 - perf script -i ${perfdata} --itrace=b -s python:${script_path} -- \ - -d --stop-sample=2 -k ${perfdata}/kcore_dir/kcore 2> /dev/null > ${file} + # shellcheck source=lib/setup_python.sh + . "$(dirname "$0")"/../lib/setup_python.sh
[Severity: Medium]
Will this sourcing of setup_python.sh on line 49 cause the test to silently skip due to a bad PYTHONPATH?
When running tools/perf/tests/shell/coresight/test_arm_coresight_disasm.sh directly without explicitly setting PERF_EXEC_PATH, bash source leaves $0 as the caller script.
This means dirname "$0" inside setup_python.sh evaluates to the caller's directory (tools/perf/tests/shell/coresight/). The relative path checks (../../python and ../python) in setup_python.sh are designed for scripts located directly in tests/shell/ and will fail here because this test is nested one level deeper.
Could the path resolution be adjusted to account for nested test scripts?
Quoted 9 lines
tools/perf/tests/shell/coresight/test_arm_coresight_disasm.sh +7
-0+ $PYTHON -c "import perf" 2>/dev/null || { + echo "Skipping test, perf python module not found" + glb_err=2 + exit 2 + } + $PYTHON ${script_path} -i ${perfdata} --itrace=b -d --stop-sample=2 \ + -k ${perfdata}/kcore_dir/kcore 2> /dev/null > ${file} grep -q -E ${branch_search} ${file} echo "Found kernel branches"
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=38
Ian Rogers
reviewerPATCH 39/49 perf python: Port powerpc-hcalls to perf module
In reply to Ian RogersPort powerpc-hcalls.py from tools/perf/scripts/python/ to a standalone script in tools/perf/python/:
- Refactor the script into an HCallAnalyzer class with full type annotations to encapsulate per-CPU hypervisor call entry/exit state.
- Use perf.session for event processing to track hypervisor call entry and exit timestamps and aggregate min/max/average duration statistics against HCALL_TABLE.
- Add argparse CLI support (-i/--input) and remove Python 2 compatibility code.
Add a shell test (test_powerpc_hcalls_python.sh) to verify the standalone script.
--- tools/perf/python/powerpc-hcalls.py | 330 ++++++++++++++++++ .../tests/shell/test_powerpc_hcalls_python.sh | 95 +++++ 2 files changed, 425 insertions(+) create mode 100755 tools/perf/python/powerpc-hcalls.py create mode 100755 tools/perf/tests/shell/test_powerpc_hcalls_python.sh
tools/perf/python/powerpc-hcalls.py new file +330 -0
@@ -0,0 +1,330 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0+ +""" +Hypervisor call statistics + +Copyright (C) 2018 Ravi Bangoria, IBM Corporation +Ported from tools/perf/scripts/python/powerpc-hcalls.py +""" +from __future__ import annotations + +import argparse +from collections import defaultdict +import sys +import perf + +# Hypervisor call table +HCALL_TABLE = { + 4: 'H_REMOVE', + 8: 'H_ENTER', + 12: 'H_READ', + 16: 'H_CLEAR_MOD', + 20: 'H_CLEAR_REF', + 24: 'H_PROTECT', + 28: 'H_GET_TCE', + 32: 'H_PUT_TCE', + 36: 'H_SET_SPRG0', + 40: 'H_SET_DABR', + 44: 'H_PAGE_INIT', + 48: 'H_SET_ASR', + 52: 'H_ASR_ON', + 56: 'H_ASR_OFF', + 60: 'H_LOGICAL_CI_LOAD', + 64: 'H_LOGICAL_CI_STORE', + 68: 'H_LOGICAL_CACHE_LOAD', + 72: 'H_LOGICAL_CACHE_STORE', + 76: 'H_LOGICAL_ICBI', + 80: 'H_LOGICAL_DCBF', + 84: 'H_GET_TERM_CHAR', + 88: 'H_PUT_TERM_CHAR', + 92: 'H_REAL_TO_LOGICAL', + 96: 'H_HYPERVISOR_DATA', + 100: 'H_EOI', + 104: 'H_CPPR', + 108: 'H_IPI', + 112: 'H_IPOLL', + 116: 'H_XIRR', + 120: 'H_MIGRATE_DMA', + 124: 'H_PERFMON', + 220: 'H_REGISTER_VPA', + 224: 'H_CEDE', + 228: 'H_CONFER', + 232: 'H_PROD', + 236: 'H_GET_PPP', + 240: 'H_SET_PPP', + 244: 'H_PURR', + 248: 'H_PIC', + 252: 'H_REG_CRQ', + 256: 'H_FREE_CRQ', + 260: 'H_VIO_SIGNAL', + 264: 'H_SEND_CRQ', + 272: 'H_COPY_RDMA', + 276: 'H_REGISTER_LOGICAL_LAN', + 280: 'H_FREE_LOGICAL_LAN', + 284: 'H_ADD_LOGICAL_LAN_BUFFER', + 288: 'H_SEND_LOGICAL_LAN', + 292: 'H_BULK_REMOVE', + 304: 'H_MULTICAST_CTRL', + 308: 'H_SET_XDABR', + 312: 'H_STUFF_TCE', + 316: 'H_PUT_TCE_INDIRECT', + 332: 'H_CHANGE_LOGICAL_LAN_MAC', + 336: 'H_VTERM_PARTNER_INFO', + 340: 'H_REGISTER_VTERM', + 344: 'H_FREE_VTERM', + 348: 'H_RESET_EVENTS', + 352: 'H_ALLOC_RESOURCE', + 356: 'H_FREE_RESOURCE', + 360: 'H_MODIFY_QP', + 364: 'H_QUERY_QP', + 368: 'H_REREGISTER_PMR', + 372: 'H_REGISTER_SMR', + 376: 'H_QUERY_MR', + 380: 'H_QUERY_MW', + 384: 'H_QUERY_HCA', + 388: 'H_QUERY_PORT', + 392: 'H_MODIFY_PORT', + 396: 'H_DEFINE_AQP1', + 400: 'H_GET_TRACE_BUFFER', + 404: 'H_DEFINE_AQP0', + 408: 'H_RESIZE_MR', + 412: 'H_ATTACH_MCQP', + 416: 'H_DETACH_MCQP', + 420: 'H_CREATE_RPT', + 424: 'H_REMOVE_RPT', + 428: 'H_REGISTER_RPAGES', + 432: 'H_DISABLE_AND_GET', + 436: 'H_ERROR_DATA', + 440: 'H_GET_HCA_INFO', + 444: 'H_GET_PERF_COUNT', + 448: 'H_MANAGE_TRACE', + 456: 'H_GET_CPU_CHARACTERISTICS', + 468: 'H_FREE_LOGICAL_LAN_BUFFER', + 472: 'H_POLL_PENDING', + 484: 'H_QUERY_INT_STATE', + 580: 'H_ILLAN_ATTRIBUTES', + 584: 'H_ADD_LOGICAL_LAN_BUFFERS', + 592: 'H_MODIFY_HEA_QP', + 596: 'H_QUERY_HEA_QP', + 600: 'H_QUERY_HEA', + 604: 'H_QUERY_HEA_PORT', + 608: 'H_MODIFY_HEA_PORT', + 612: 'H_REG_BCMC', + 616: 'H_DEREG_BCMC', + 620: 'H_REGISTER_HEA_RPAGES', + 624: 'H_DISABLE_AND_GET_HEA', + 628: 'H_GET_HEA_INFO', + 632: 'H_ALLOC_HEA_RESOURCE', + 644: 'H_ADD_CONN', + 648: 'H_DEL_CONN', + 664: 'H_JOIN', + 672: 'H_VASI_SIGNAL', + 676: 'H_VASI_STATE', + 680: 'H_VIOCTL', + 688: 'H_ENABLE_CRQ', + 696: 'H_GET_EM_PARMS', + 720: 'H_SET_MPP', + 724: 'H_GET_MPP', + 732: 'H_REG_SUB_CRQ', + 736: 'H_FREE_SUB_CRQ', + 740: 'H_SEND_SUB_CRQ', + 744: 'H_SEND_SUB_CRQ_INDIRECT', + 748: 'H_HOME_NODE_ASSOCIATIVITY', + 756: 'H_BEST_ENERGY', + 764: 'H_XIRR_X', + 768: 'H_RANDOM', + 772: 'H_COP', + 788: 'H_GET_MPP_X', + 796: 'H_SET_MODE', + 808: 'H_BLOCK_REMOVE', + 856: 'H_CLEAR_HPT', + 864: 'H_REQUEST_VMC', + 876: 'H_RESIZE_HPT_PREPARE', + 880: 'H_RESIZE_HPT_COMMIT', + 892: 'H_REGISTER_PROC_TBL', + 896: 'H_SIGNAL_SYS_RESET', + 904: 'H_ALLOCATE_VAS_WINDOW', + 908: 'H_MODIFY_VAS_WINDOW', + 912: 'H_DEALLOCATE_VAS_WINDOW', + 916: 'H_QUERY_VAS_WINDOW', + 920: 'H_QUERY_VAS_CAPABILITIES', + 924: 'H_QUERY_NX_CAPABILITIES', + 928: 'H_GET_NX_FAULT', + 936: 'H_INT_GET_SOURCE_INFO', + 940: 'H_INT_SET_SOURCE_CONFIG', + 944: 'H_INT_GET_SOURCE_CONFIG', + 948: 'H_INT_GET_QUEUE_INFO', + 952: 'H_INT_SET_QUEUE_CONFIG', + 956: 'H_INT_GET_QUEUE_CONFIG', + 960: 'H_INT_SET_OS_REPORTING_LINE', + 964: 'H_INT_GET_OS_REPORTING_LINE', + 968: 'H_INT_ESB', + 972: 'H_INT_SYNC', + 976: 'H_INT_RESET', + 996: 'H_SCM_READ_METADATA', + 1000: 'H_SCM_WRITE_METADATA', + 1004: 'H_SCM_BIND_MEM', + 1008: 'H_SCM_UNBIND_MEM', + 1012: 'H_SCM_QUERY_BLOCK_MEM_BINDING', + 1016: 'H_SCM_QUERY_LOGICAL_MEM_BINDING', + 1020: 'H_SCM_UNBIND_ALL', + 1024: 'H_SCM_HEALTH', + 1048: 'H_SCM_PERFORMANCE_STATS', + 1052: 'H_PKS_GET_CONFIG', + 1056: 'H_PKS_SET_PASSWORD', + 1060: 'H_PKS_GEN_PASSWORD', + 1068: 'H_PKS_WRITE_OBJECT', + 1072: 'H_PKS_GEN_KEY', + 1076: 'H_PKS_READ_OBJECT', + 1080: 'H_PKS_REMOVE_OBJECT', + 1084: 'H_PKS_CONFIRM_OBJECT_FLUSHED', + 1096: 'H_RPT_INVALIDATE', + 1100: 'H_SCM_FLUSH', + 1104: 'H_GET_ENERGY_SCALE_INFO', + 1108: 'H_PKS_SIGNED_UPDATE', + 1112: 'H_HTM', + 1116: 'H_WATCHDOG', + # Platform specific hcalls used by KVM on PowerVM + 1120: 'H_GUEST_GET_CAPABILITIES', + 1124: 'H_GUEST_SET_CAPABILITIES', + 1136: 'H_GUEST_CREATE', + 1140: 'H_GUEST_CREATE_VCPU', + 1144: 'H_GUEST_GET_STATE', + 1148: 'H_GUEST_SET_STATE', + 1152: 'H_GUEST_RUN_VCPU', + 1156: 'H_GUEST_COPY_MEMORY', + 1160: 'H_GUEST_DELETE', + # Key wrapping hcalls + 1168: 'H_PKS_WRAP_OBJECT', + 1172: 'H_PKS_UNWRAP_OBJECT', + # Platform-specific hcalls used by the Ultravisor + 61184: 'H_SVM_PAGE_IN', + 61188: 'H_SVM_PAGE_OUT', + 61192: 'H_SVM_INIT_START', + 61196: 'H_SVM_INIT_DONE', + 61204: 'H_SVM_INIT_ABORT', + # Platform specific hcalls used by KVM + 61440: 'H_RTAS', + # Platform specific hcalls used by QEMU/SLOF + 61441: 'H_LOGICAL_MEMOP', + 61442: 'H_CAS', + 61443: 'H_UPDATE_DT', + # Platform specific hcalls provided by PHYP + 61560: 'H_GET_24X7_CATALOG_PAGE', + 61564: 'H_GET_24X7_DATA', + 61568: 'H_GET_PERF_COUNTER_INFO', + # Platform-specific hcalls used for nested HV KVM + 63488: 'H_SET_PARTITION_TABLE', + 63492: 'H_ENTER_NESTED', + 63496: 'H_TLB_INVALIDATE', + 63500: 'H_COPY_TOFROM_GUEST', +} + + +class HCallAnalyzer: + """Analyzes hypervisor calls and aggregates statistics.""" + + def __init__(self, cmd_args) -> None: + self.args = cmd_args + # output: {opcode: {'min': min, 'max': max, 'time': time, 'cnt': cnt}} + self.output: dict[int, dict[str, 'int | float']] = \ + defaultdict(lambda: {'time': 0, 'cnt': 0, 'min': float('inf'), 'max': 0}) + # d_enter: {cpu: {opcode: nsec}} + self.d_enter: dict[int, dict[int, int]] = {} + self.print_ptrn = '%-28s%10s%10s%10s%10s' + + def hcall_table_lookup(self, opcode: int) -> str: + """Lookup hcall name by opcode.""" + return HCALL_TABLE.get(opcode, str(opcode)) + + def process_event(self, sample: perf.sample_event) -> None: + """Process a single sample event.""" + name = str(sample.evsel) + if name not in ("evsel(powerpc:hcall_entry)", "evsel(powerpc:hcall_exit)"): + return + + try: + cpu = sample.sample_cpu + opcode = sample.opcode + except AttributeError: + print("ERROR: tracepoint fields missed (is libtraceevent enabled?)", + file=sys.stderr) + sys.exit(1) + + # sample.sample_time represents the nsecs + time = sample.sample_time + + if opcode < 0 or cpu < 0: + return + + if name == "evsel(powerpc:hcall_entry)": + if cpu not in self.d_enter: + self.d_enter[cpu] = {} + self.d_enter[cpu][opcode] = time + elif name == "evsel(powerpc:hcall_exit)": + if cpu in self.d_enter and opcode in self.d_enter[cpu]: + time_entry = self.d_enter[cpu][opcode] + diff = time - time_entry + del self.d_enter[cpu][opcode] + + stats = self.output[opcode] + stats['time'] += diff + stats['cnt'] += 1 + if diff < stats['min']: + stats['min'] = diff + if diff > stats['max']: + stats['max'] = diff + + def print_summary(self) -> None: + """Print aggregated statistics.""" + print(self.print_ptrn % ('hcall', 'count', 'min(ns)', 'max(ns)', 'avg(ns)')) + print('-' * 68) + + def sort_output(opcode): + stats = self.output[opcode] + sort_by = getattr(self.args, 'sort', None) + if sort_by == 'min': + return stats['min'] + if sort_by == 'max': + return stats['max'] + if sort_by == 'avg': + return stats['time'] // stats['cnt'] + return stats['cnt'] + + for opcode in sorted(self.output.keys(), key=sort_output, reverse=True): + h_name = self.hcall_table_lookup(opcode) + stats = self.output[opcode] + time = stats['time'] + cnt = stats['cnt'] + min_t = stats['min'] + max_t = stats['max'] + + # Avoid float representation for large integers if possible, + # or use formatted strings. Legacy used time//cnt. + avg_t = time // cnt if cnt > 0 else 0 + + # If min was not updated, it remains inf, but cnt should be > 0 if in output + if min_t == float('inf'): + min_t = 0 + + print(self.print_ptrn % (h_name, cnt, int(min_t), int(max_t), avg_t)) + + +if __name__ == "__main__": + ap = argparse.ArgumentParser(description="Hypervisor call statistics") + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + ap.add_argument("-s", "--sort", default="count", + choices=["count", "min", "max", "avg"], help="Sort key") + args = ap.parse_args() + + analyzer = HCallAnalyzer(args) + + try: + session = perf.session(perf.data(args.input), sample=analyzer.process_event) + session.process_events() + analyzer.print_summary() + except KeyboardInterrupt: + analyzer.print_summary() + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError) as e: + print(f"Error processing events: {e}") + sys.exit(1)
tools/perf/tests/shell/test_powerpc_hcalls_python.sh new file +95 -0
95 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 39/49] perf python: Port powerpc-hcalls to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 10 lines
Port
powerpc-hcalls.pyfromtools/perf/scripts/python/to a standalone script intools/perf/python/:
- Refactor the script into an HCallAnalyzer class with full type annotations to encapsulate per-CPU hypervisor call entry/exit state.
- Use perf.session for event processing to track hypervisor call entry and exit timestamps and aggregate min/max/average duration statistics againstHCALL_TABLE.
- Add argparse CLI support (-i/--input) and remove Python 2 compatibility code.
Quoted 5 lines
Add a shell test (
test_powerpc_hcalls_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=39
Ian Rogers
reviewerPATCH 40/49 perf python: Port intel-pt-events and libxed to perf module
In reply to Ian RogersPort intel-pt-events.py and libxed.py from tools/perf/scripts/python/ to standalone modules in tools/perf/python/:
- Refactor intel-pt-events.py into an IntelPTAnalyzer class with full type annotations to encapsulate trace state and stashed output.
- Configure instruction trace decoding directly via perf.session's itrace option and context_switch callback.
- Dynamically select 32-bit vs 64-bit disassembly mode in libxed.py using session.is_64_bit and annotate instructions with source lines via sample.srccode().
- Rename methods in libxed.py to snake_case (instruction, set_mode, disassemble_one) and remove Python 2 compatibility code.
Add a shell test (test_intel_pt_events_python.sh) to verify the standalone script.
--- tools/perf/python/intel-pt-events.py | 606 ++++++++++++++++++ tools/perf/python/libxed.py | 123 ++++ .../shell/test_intel_pt_events_python.sh | 69 ++ 3 files changed, 798 insertions(+) create mode 100755 tools/perf/python/intel-pt-events.py create mode 100755 tools/perf/python/libxed.py create mode 100755 tools/perf/tests/shell/test_intel_pt_events_python.sh
tools/perf/python/intel-pt-events.py new file +606 -0
@@ -0,0 +1,606 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +""" +Print Intel PT Events including Power Events and PTWRITE. +Ported from tools/perf/scripts/python/intel-pt-events.py +""" +from __future__ import annotations + +import argparse +from typing import Dict, List, Tuple +import contextlib +from ctypes import addressof, create_string_buffer +import io +import os +import struct +import sys +from typing import Any, Optional +import typing +import perf + +try: + from libxed import LibXED as _LibXED + LibXED: Optional[type[_LibXED]] = _LibXED +except ImportError: + LibXED = None + + +def sample_flags_to_name(flags: int) -> str: + """Implement perf's sample_flags_to_name.""" + if not isinstance(flags, int) or flags == 0: + return "".ljust(21) + sample_flags = [ + ((1 << 0) | (1 << 1), "call"), + ((1 << 0) | (1 << 2), "return"), + ((1 << 0) | (1 << 3), "jcc"), + ((1 << 0), "jmp"), + ((1 << 0) | (1 << 1) | (1 << 6), "int"), + ((1 << 0) | (1 << 2) | (1 << 6), "iret"), + ((1 << 0) | (1 << 1) | (1 << 4), "syscall"), + ((1 << 0) | (1 << 2) | (1 << 4), "sysret"), + ((1 << 0) | (1 << 5), "async"), + ((1 << 0) | (1 << 1) | (1 << 5) | (1 << 6), "hw int"), + ((1 << 0) | (1 << 7), "tx abrt"), + ((1 << 0) | (1 << 8), "tr strt"), + ((1 << 0) | (1 << 9), "tr end"), + ((1 << 0) | (1 << 1) | (1 << 11), "vmentry"), + ((1 << 0) | (1 << 1) | (1 << 12), "vmexit"), + ] + additional_mask = (1 << 10) | (1 << 13) | (1 << 14) + branch_event_mask = (1 << 15) | (1 << 16) + xf = flags & additional_mask + rem_flags = flags & ~additional_mask + + if rem_flags & (1 << 8): + prefix = "tr strt " + elif rem_flags & (1 << 9): + prefix = "tr end " + else: + prefix = "" + + rem_flags &= ~((1 << 8) | (1 << 9)) + types = rem_flags & ~branch_event_mask + type_name = "" + for f_mask, f_name in sample_flags: + if f_mask == types: + type_name = f_name + break + + s = prefix + type_name + ev_parts = [] + if rem_flags & (1 << 15): + ev_parts.append("miss") + if rem_flags & (1 << 16): + ev_parts.append("not_taken") + if ev_parts: + s += "/" + ",".join(ev_parts) + "/" + + if xf: + xs = "(" + if xf & (1 << 10): + xs += "x" + if xf & (1 << 13): + xs += "D" + if xf & (1 << 14): + xs += "t" + xs += ")" + if len(s) + len(xs) < 21: + s = s + xs.rjust(21 - len(s)) + else: + s = s + " " + xs + return s.ljust(21) + +class IntelPTAnalyzer: + """Analyzes Intel PT events and prints details.""" + + def __init__(self, cfg: argparse.Namespace): + self.args = cfg + self.session: Optional[perf.session] = None + self.insn = False + self.src = False + self.source_file_name: Optional[str] = None + self.line_number: int = 0 + self.dso: Optional[str] = None + self.stash_dict: Dict[int, List[str]] = {} + self.output: Any = None + self.output_pos: int = 0 + self.cpu: int = -1 + self.time: int = 0 + self.switch_str: Dict[int, str] = {} + + if cfg.insn_trace: + print("Intel PT Instruction Trace") + self.insn = True + elif cfg.src_trace: + print("Intel PT Source Trace") + self.insn = True + self.src = True + else: + print("Intel PT Branch Trace, Power Events, Event Trace and PTWRITE") + + self.disassembler: Any = None + if self.insn and LibXED is not None: + try: + self.disassembler = LibXED() + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError, + ImportError) as e: + print(f"Failed to initialize LibXED: {e}") + self.disassembler = None + + def print_ptwrite(self, raw_buf: bytes) -> None: + """Print PTWRITE data.""" + if not raw_buf: + return + try: + data = struct.unpack_from("<IQ", raw_buf) + flags = data[0] + payload = data[1] + except struct.error: + return + exact_ip = flags & 1 + try: + s = payload.to_bytes(8, "little").decode("ascii").rstrip("\x00") + if not s.isprintable(): + s = "" + except (UnicodeDecodeError, ValueError): + s = "" + print(f"IP: {exact_ip} payload: {payload:#x} {s}", end=' ') + + def print_cbr(self, raw_buf: bytes) -> None: + """Print CBR data.""" + if len(raw_buf) < 12: + return + try: + data = struct.unpack_from("<BBBBII", raw_buf) + except struct.error: + return + cbr = data[0] + f = (data[4] + 500) // 1000 + if data[2] == 0: + return + p = ((cbr * 1000 // data[2]) + 5) // 10 + print(f"{cbr:3d} freq: {f:4d} MHz ({p:3d}%)", end=' ') + + def print_mwait(self, raw_buf: bytes) -> None: + """Print MWAIT data.""" + try: + data = struct.unpack_from("<IQ", raw_buf) + except struct.error: + return + payload = data[1] + hints = payload & 0xff + extensions = (payload >> 32) & 0x3 + print(f"hints: {hints:#x} extensions: {extensions:#x}", end=' ') + + def print_pwre(self, raw_buf: bytes) -> None: + """Print PWRE data.""" + try: + data = struct.unpack_from("<IQ", raw_buf) + except struct.error: + return + payload = data[1] + hw = (payload >> 7) & 1 + cstate = (payload >> 12) & 0xf + subcstate = (payload >> 8) & 0xf + print(f"hw: {hw} cstate: {cstate} sub-cstate: {subcstate}", end=' ') + + def print_exstop(self, raw_buf: bytes) -> None: + """Print EXSTOP data.""" + try: + data = struct.unpack_from("<I", raw_buf) + except struct.error: + return + flags = data[0] + exact_ip = flags & 1 + print(f"IP: {exact_ip}", end=' ') + + def print_pwrx(self, raw_buf: bytes) -> None: + """Print PWRX data.""" + try: + data = struct.unpack_from("<IQ", raw_buf) + except struct.error: + return + payload = data[1] + deepest_cstate = payload & 0xf + last_cstate = (payload >> 4) & 0xf + wake_reason = (payload >> 8) & 0xf + print(f"deepest cstate: {deepest_cstate} last cstate: {last_cstate} " + f"wake reason: {wake_reason:#x}", end=' ') + + def print_psb(self, raw_buf: bytes) -> None: + """Print PSB data.""" + try: + data = struct.unpack_from("<IQ", raw_buf) + except struct.error: + return + offset = data[1] + print(f"offset: {offset:#x}", end=' ') + + def print_evt(self, raw_buf: bytes) -> None: + """Print EVT data.""" + glb_cfe = ["", "INTR", "IRET", "SMI", "RSM", "SIPI", "INIT", "VMENTRY", "VMEXIT", + "VMEXIT_INTR", "SHUTDOWN", "", "UINT", "UIRET"] + [""] * 18 + glb_evd = ["", "PFA", "VMXQ", "VMXR"] + [""] * 60 + + try: + data = struct.unpack_from("<BBH", raw_buf) + except struct.error: + return + typ = data[0] & 0x1f + ip_flag = (data[0] & 0x80) >> 7 + vector = data[1] + evd_cnt = data[2] + s = glb_cfe[typ] + if s: + print(f" cfe: {s} IP: {ip_flag} vector: {vector}", end=' ') + else: + print(f" cfe: {typ} IP: {ip_flag} vector: {vector}", end=' ') + pos = 4 + for _ in range(evd_cnt): + if len(raw_buf) < pos + 16: + break + try: + data = struct.unpack_from("<QQ", raw_buf, pos) + except struct.error: + return + et = data[0] & 0x3f + s = glb_evd[et] + if s: + print(f"{s}: {data[1]:#x}", end=' ') + else: + print(f"EVD_{et}: {data[1]:#x}", end=' ') + pos += 16 + + def print_iflag(self, raw_buf: bytes) -> None: + """Print IFLAG data.""" + try: + data = struct.unpack_from("<IQ", raw_buf) + except struct.error: + return + iflag = data[0] & 1 + old_iflag = iflag ^ 1 + via_branch = data[0] & 2 + s = "via" if via_branch else "non" + print(f"IFLAG: {old_iflag}->{iflag} {s} branch", end=' ') + + def common_start_str(self, comm: str, sample: perf.sample_event) -> str: + """Return common start string for display.""" + ts = sample.sample_time + cpu = sample.sample_cpu + pid = sample.sample_pid + tid = sample.sample_tid + machine_pid = getattr(sample, "machine_pid", 0) + if machine_pid: + vcpu = getattr(sample, "vcpu", -1) + return (f"VM:{machine_pid:5d} VCPU:{vcpu:03d} {comm:>16s} {pid:5d}/{tid:<5d} " + f"[{cpu:03d}] {ts // 1000000000:9d}.{ts % 1000000000:09d} ") + return (f"{comm:>16s} {pid:5d}/{tid:<5d} [{cpu:03d}] " + f"{ts // 1000000000:9d}.{ts % 1000000000:09d} ") + + def print_common_start(self, comm: str, sample: perf.sample_event, name: str) -> None: + """Print common start info.""" + flags_disp = sample_flags_to_name(getattr(sample, "flags", 0)) + print(self.common_start_str(comm, sample) + f"{name:>8s} {flags_disp:>21s}", end=' ') + + def print_instructions_start(self, comm: str, sample: perf.sample_event) -> None: + """Print instructions start info.""" + raw_flags = getattr(sample, "flags", 0) + if isinstance(raw_flags, int) and (raw_flags & (1 << 10)): + print(self.common_start_str(comm, sample) + "x", end=' ') + else: + print(self.common_start_str(comm, sample), end=' ') + + def disassem(self, insn: bytes, ip: int) -> Tuple[int, str]: + """Disassemble instruction using LibXED.""" + inst = self.disassembler.instruction() + is_64_bit = getattr(self.session, "is_64_bit", True) if self.session else True + self.disassembler.set_mode(inst, 0 if is_64_bit else 1) + buf = create_string_buffer(insn, 64) + return self.disassembler.disassemble_one(inst, addressof(buf), len(insn), ip) + + def print_common_ip(self, sample: perf.sample_event, symbol: str, dso: str) -> None: + """Print IP and symbol info.""" + ip = sample.sample_ip + offs = f"+{sample.sym_offset:#x}" if getattr(sample, "sym_offset", None) is not None else "" + cyc_cnt = getattr(sample, "sample_cyc_count", 0) + if cyc_cnt: + insn_cnt = getattr(sample, "sample_insn_count", 0) + ipc_str = f" IPC: {insn_cnt / cyc_cnt:#.2f} ({insn_cnt}/{cyc_cnt})" + else: + ipc_str = "" + + if self.insn and self.disassembler is not None: + try: + insn = sample.insn() + except AttributeError: + insn = None + if insn: + cnt, text = self.disassem(insn, ip) + byte_str = (f"{ip:x}").rjust(16) + for k in range(cnt): + byte_str += f" {insn[k]:02x}" + print(f"{byte_str:<40s} {text:<30s}", end=' ') + else: + print(f"{ip:16x}", end=' ') + print(f"{symbol}{offs} ({dso})", end=' ') + else: + print(f"{ip:16x} {symbol}{offs} ({dso})", end=' ') + + addr_correlates_sym = getattr(sample, "addr_symbol", None) is not None + if addr_correlates_sym: + addr = getattr(sample, "sample_addr", getattr(sample, "addr", 0)) + addr_dso = (sample.addr_dso or '[unknown]') + addr_symbol = (sample.addr_symbol or '[unknown]') + addr_offs = (f"+{sample.addr_sym_offset:#x}" + if getattr(sample, "addr_sym_offset", None) is not None else "") + print(f"=> {addr:x} {addr_symbol}{addr_offs} ({addr_dso}){ipc_str}") + else: + print(ipc_str) + + def print_srccode(self, comm: str, sample: perf.sample_event, + symbol: str, dso: str, with_insn: bool) -> None: + """Print source code info.""" + ip = sample.sample_ip + if symbol == "[unknown]": + start_str = self.common_start_str(comm, sample) + (f"{ip:x}").rjust(16).ljust(40) + else: + offs = (f"+{sample.sym_offset:#x}" + if getattr(sample, "sym_offset", None) is not None else "") + start_str = self.common_start_str(comm, sample) + (symbol + offs).ljust(40) + + if with_insn and self.insn and self.disassembler is not None: + try: + insn = sample.insn() + except AttributeError: + insn = None + if insn: + _, text = self.disassem(insn, ip) + start_str += text.ljust(30) + + source_file_name: typing.Any = None + line_number: typing.Any = 0 + source_line: typing.Any = None + try: + res_srcc = sample.srccode() + if res_srcc: + source_file_name, line_number, source_line = res_srcc + except (AttributeError, ValueError, TypeError): + pass + + if source_file_name: + if self.line_number == line_number and self.source_file_name == source_file_name: + src_str = "" + else: + if len(source_file_name) > 40: + src_file = ("..." + source_file_name[-37:]) + " " + else: + src_file = source_file_name.ljust(41) + if source_line is None: + src_str = src_file + str(line_number).rjust(4) + " <source not found>" + else: + src_str = src_file + str(line_number).rjust(4) + " " + source_line + self.dso = None + elif dso == self.dso: + src_str = "" + else: + src_str = dso + self.dso = dso + + self.line_number = line_number + self.source_file_name = source_file_name + print(start_str, src_str) + + def do_process_event(self, sample: perf.sample_event) -> None: + """Process event and print info.""" + cpu = getattr(sample, "sample_cpu", getattr(sample, "cpu", 0)) + if cpu in self.switch_str: + print(self.switch_str[cpu]) + del self.switch_str[cpu] + + comm = "Unknown" + if hasattr(self, 'session') and self.session: + try: + thread = self.session.find_thread(sample.sample_pid, sample.sample_tid) + if thread: + comm = thread.comm() or "Unknown" + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError): + pass + # Python < 3.9 compatibility + name = str(sample.evsel) + if name.startswith("evsel("): + name = name[6:-1] + dso = (sample.dso or '[unknown]') + symbol = (sample.symbol or '[unknown]') + + + raw_buf = getattr(sample, 'raw_buf', b'') or b'' + + if name.startswith("instructions"): + if self.src: + self.print_srccode(comm, sample, symbol, dso, True) + else: + self.print_instructions_start(comm, sample) + self.print_common_ip(sample, symbol, dso) + elif name.startswith("branches"): + if self.src: + self.print_srccode(comm, sample, symbol, dso, False) + else: + self.print_common_start(comm, sample, name) + self.print_common_ip(sample, symbol, dso) + elif name == "ptwrite": + self.print_common_start(comm, sample, name) + self.print_ptwrite(raw_buf) + self.print_common_ip(sample, symbol, dso) + elif name == "cbr": + self.print_common_start(comm, sample, name) + self.print_cbr(raw_buf) + self.print_common_ip(sample, symbol, dso) + elif name == "mwait": + self.print_common_start(comm, sample, name) + self.print_mwait(raw_buf) + self.print_common_ip(sample, symbol, dso) + elif name == "pwre": + self.print_common_start(comm, sample, name) + self.print_pwre(raw_buf) + self.print_common_ip(sample, symbol, dso) + elif name == "exstop": + self.print_common_start(comm, sample, name) + self.print_exstop(raw_buf) + self.print_common_ip(sample, symbol, dso) + elif name == "pwrx": + self.print_common_start(comm, sample, name) + self.print_pwrx(raw_buf) + self.print_common_ip(sample, symbol, dso) + elif name == "psb": + self.print_common_start(comm, sample, name) + self.print_psb(raw_buf) + self.print_common_ip(sample, symbol, dso) + elif name == "evt": + self.print_common_start(comm, sample, name) + self.print_evt(raw_buf) + self.print_common_ip(sample, symbol, dso) + elif name == "iflag": + self.print_common_start(comm, sample, name) + self.print_iflag(raw_buf) + self.print_common_ip(sample, symbol, dso) + else: + self.print_common_start(comm, sample, name) + self.print_common_ip(sample, symbol, dso) + + def interleave_events(self, sample: perf.sample_event) -> None: + """Interleave output to avoid garbled lines from different CPUs.""" + self.cpu = sample.sample_cpu + ts = sample.sample_time + + if self.time != ts: + self.time = ts + self.flush_stashed_output() + + self.output_pos = 0 + with contextlib.redirect_stdout(io.StringIO()) as self.output: + self.do_process_event(sample) + + self.stash_output() + + def stash_output(self) -> None: + """Stash output for later flushing.""" + output_str = self.output.getvalue()[self.output_pos:] + n = len(output_str) + if n: + self.output_pos += n + if self.cpu not in self.stash_dict: + self.stash_dict[self.cpu] = [] + self.stash_dict[self.cpu].append(output_str) + if len(self.stash_dict[self.cpu]) > 1000: + self.flush_stashed_output() + + def flush_stashed_output(self) -> None: + """Flush stashed output.""" + while self.stash_dict: + cpus = list(self.stash_dict.keys()) + for cpu in cpus: + items = self.stash_dict[cpu] + countdown = self.args.interleave + while len(items) and countdown: + sys.stdout.write(items[0]) + del items[0] + countdown -= 1 + if not items: + del self.stash_dict[cpu] + + def process_context_switch(self, event: perf.switch_event) -> None: + """Process context switch.""" + cpu = getattr(event, "sample_cpu", getattr(event, "cpu", 0)) + pid = getattr(event, "sample_pid", getattr(event, "pid", 0)) + tid = getattr(event, "sample_tid", getattr(event, "tid", 0)) + ts = getattr(event, "sample_time", getattr(event, "time", 0)) + np_pid = getattr(event, "next_prev_pid", None) + np_tid = getattr(event, "next_prev_tid", None) + machine_pid = getattr(event, "machine_pid", -1) + vcpu = getattr(event, "vcpu", -1) + misc = getattr(event, "misc", 0) + out = bool(misc & (1 << 13)) + out_preempt = bool(misc & (1 << 14)) + + if self.args.interleave: + self.flush_stashed_output() + + if out: + out_str = "Switch out " + else: + out_str = "Switch In " + + preempt_str = "preempt" if out_preempt else "" + + if machine_pid == -1: + machine_str = "" + elif vcpu == -1: + machine_str = f"machine PID {machine_pid}" + else: + machine_str = f"machine PID {machine_pid} VCPU {vcpu}" + + np_str = f"{np_pid:5d}/{np_tid:<5d} " if np_pid is not None and np_tid is not None else "" + c_str = (f"{out_str:>16s} {pid:5d}/{tid:<5d} [{cpu:03d}] " + f"{ts // 1000000000:9d}.{ts % 1000000000:09d} " + f"{np_str}{machine_str} {preempt_str}") + + if self.args.all_switch_events: + print(c_str) + else: + self.switch_str[cpu] = c_str + def process_event(self, sample: perf.sample_event) -> None: + """Wrapper to handle interleaving and exceptions.""" + try: + if self.args.interleave: + self.interleave_events(sample) + else: + self.do_process_event(sample) + except BrokenPipeError: + # Stop python printing broken pipe errors and traceback + sys.stdout = open(os.devnull, 'w', encoding='utf-8') + sys.exit(1) + + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + ap.add_argument("--insn-trace", action='store_true') + ap.add_argument("--src-trace", action='store_true') + ap.add_argument("--all-switch-events", action='store_true') + ap.add_argument("--interleave", type=int, nargs='?', const=4, default=0) + ap.add_argument("--itrace", default=None, help="itrace options") + args = ap.parse_args() + + if args.itrace is None: + if args.insn_trace or args.src_trace: + args.itrace = "i0nsepwxI" + else: + args.itrace = "bepwxI" + analyzer = IntelPTAnalyzer(args) + + try: + # Note: Python API currently lacks auxtrace_error callbacks affecting + # chronological interleaving + session = perf.session( + perf.data(args.input), + sample=analyzer.process_event, + context_switch=analyzer.process_context_switch, + itrace=args.itrace + ) + analyzer.session = session + session.process_events() + if args.interleave: + analyzer.flush_stashed_output() + print("End") + except KeyboardInterrupt: + if args.interleave: + analyzer.flush_stashed_output() + print("End") + except BrokenPipeError: + sys.exit(0) + except (OSError, ValueError, KeyError, RuntimeError, TypeError, AttributeError): + import traceback + traceback.print_exc() + sys.exit(1) + +
tools/perf/python/libxed.py new file +123 -0
123 lines not on this page · raw message
tools/perf/tests/shell/test_intel_pt_events_python.sh new file +69 -0
69 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 40/49] perf python: Port intel-pt-events and libxed to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 12 lines
Port
intel-pt-events.pyandlibxed.pyfromtools/perf/scripts/python/to standalone modules intools/perf/python/:
- Refactorintel-pt-events.pyinto an IntelPTAnalyzer class with full type annotations to encapsulate trace state and stashed output.
- Configure instruction trace decoding directly via perf.session's itrace option andcontext_switchcallback.
- Dynamically select 32-bit vs 64-bit disassembly mode inlibxed.pyusing session.is_64_bit and annotate instructions with source lines viasample.srccode().
- Rename methods inlibxed.pytosnake_case(instruction,set_mode,disassemble_one) and remove Python 2 compatibility code.
Quoted 5 lines
Add a shell test (
test_intel_pt_events_python.sh) to verify the standalone script.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=40
Ian Rogers
reviewerPATCH 41/49 perf test: Migrate Intel PT virtual LBR test to Python API
In reply to Ian RogersMigrate the Intel PT virtual LBR test from generating an inline legacy perf script callback to using a standalone Python script (perf_brstack_max.py) with the brstack iterator API in the perf Python module.
--- .../perf/tests/shell/lib/perf_brstack_max.py | 37 +++++++++++++++++ tools/perf/tests/shell/test_intel_pt.sh | 41 ++++++++----------- 2 files changed, 54 insertions(+), 24 deletions(-) create mode 100644 tools/perf/tests/shell/lib/perf_brstack_max.py
tools/perf/tests/shell/lib/perf_brstack_max.py new file +37 -0
@@ -0,0 +1,37 @@ +#!/usr/bin/python +# SPDX-License-Identifier: GPL-2.0 +# Determine the maximum size of branch stacks in a perf.data file. + +import argparse +import sys + +import perf + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("-i", "--input", default="perf.data", help="Input file name") + args = ap.parse_args() + + bmax = 0 + + def process_event(sample): + nonlocal bmax + try: + brstack = sample.brstack + if brstack: + n = len(list(brstack)) + if n > bmax: + bmax = n + except AttributeError: + pass + + try: + session = perf.session(perf.data(args.input), sample=process_event) + session.process_events() + print("max brstack", bmax) + except Exception as e: + print(f"Error processing events: {e}", file=sys.stderr) + sys.exit(1) + +if __name__ == "__main__": + main()
tools/perf/tests/shell/test_intel_pt.sh +17 -24
64 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 41/49] perf test: Migrate Intel PT virtual LBR test to Python API
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines
Migrate the Intel PT virtual LBR test from generating an inline legacy perf script callback to using a standalone Python script (
perf_brstack_max.py) with the brstack iterator API in the perf Python module.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=41
Ian Rogers
reviewerPATCH 42/49 perf python: Port export-to-sqlite to perf module
In reply to Ian RogersPort export-to-sqlite.py to a standalone script in tools/perf/python/ using the perf module and Python's standard library sqlite3 module.
Improvements compared to the legacy script:
- Remove the dependency on PySide/QtSql by using Python's built-in sqlite3 module in DatabaseExporter, allowing SQLite export on headless and minimal systems without Qt installed.
- Support Intel PT and hardware instruction trace export via perf.session(itrace=...) and perf.call_return callbacks, reconstructing relational call_paths and calls tables and decoding synthesized PT payloads (ptwrite, cbr, mwait, pwre, exstop, pwrx).
- Export context_switches via perf.session's context_switch callback and map thread and comm IDs to their relational database keys.
- Manage temporary staging files inside an isolated tempfile.mkdtemp() directory with guaranteed cleanup in a finally block.
Update Documentation/db-export.txt and add a shell test (test_export_to_sqlite_python.sh) to verify the standalone exporter.
--- tools/perf/Documentation/db-export.txt | 2 +- tools/perf/python/export-to-sqlite.py | 893 ++++++++++++++++++ .../shell/test_export_to_sqlite_python.sh | 108 +++ 3 files changed, 1002 insertions(+), 1 deletion(-) create mode 100755 tools/perf/python/export-to-sqlite.py create mode 100755 tools/perf/tests/shell/test_export_to_sqlite_python.sh
tools/perf/Documentation/db-export.txt +1 -1
@@ -7,7 +7,7 @@ perf tool's python scripting engine: supports scripts: - tools/perf/scripts/python/export-to-sqlite.py + tools/perf/python/export-to-sqlite.py tools/perf/scripts/python/export-to-postgresql.py which export data to a SQLite3 or PostgreSQL database.
tools/perf/python/export-to-sqlite.py new file +893 -0
893 lines not on this page · raw message
tools/perf/tests/shell/test_export_to_sqlite_python.sh new file +108 -0
108 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 42/49] perf python: Port export-to-sqlite to perf module
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Medium] perf: empty branch_types table breaks exported-sql-viewer.py joins - [Medium] perf: missing SQL…
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] perf: empty branch_types table breaks exported-sql-viewer.py joins
- [Medium] perf: missing SQL indexes on calls table causes severe UI performance regression
- [Low] perf: omitted has_calls column in comms table degrades UI filtering --
commit a80e324a7b5b00cd92bb49c8ff407245a5f07096
Author: Ian Rogers <[email protected]>
perf python: Port export-to-sqlite to perf module
Port export-to-sqlite.py to a standalone script in tools/perf/python/ using the perf module and Python's standard library sqlite3 module.
Quoted 5 lines
tools/perf/python/export-to-sqlite.py new file +0
-0
[ ... ]
Quoted 2 lines
tools/perf/python/export-to-sqlite.py new file +2
-0+ def create_tables(self) -> None: + """Create database tables."""
[ ... ]
tools/perf/python/export-to-sqlite.py line 107
Quoted 8 lines
tools/perf/python/export-to-sqlite.py new file +8
-0+ self.con.execute(""" + CREATE TABLE IF NOT EXISTS comms ( + id INTEGER NOT NULL PRIMARY KEY, + comm VARCHAR(16), + c_thread_id BIGINT, + c_time BIGINT, + exec_flag BOOLEAN) + """)
[Severity: Low]
Is the has_calls column missing here? The schema definition lacks the has_calls boolean column, and the subsequent ALTER TABLE and UPDATE steps to backfill this filtering metadata are also omitted.
Will this cause the UI viewer to be unable to filter out commands that have no recorded calls, leading to a cluttered interface?
[ ... ]
tools/perf/python/export-to-sqlite.py line 135
Quoted 5 lines
tools/perf/python/export-to-sqlite.py new file +5
-0+ self.con.execute(""" + CREATE TABLE IF NOT EXISTS branch_types ( + id INTEGER NOT NULL PRIMARY KEY, + name VARCHAR(80)) + """)
[Severity: Medium]
Does this table need to be populated with default branch types? Leaving it as an empty dimension table appears to break subsequent SQL joins.
For instance, when exporting perf data with branch information and opening it, will the UI viewer exported-sql-viewer.py fail to display branch data because it relies on an INNER JOIN branch_types clause which yields zero rows?
[ ... ]
tools/perf/python/export-to-sqlite.py line 832
Quoted 3 lines
tools/perf/python/export-to-sqlite.py new file +3
-0+ def commit(self) -> None: + """Commit transaction.""" + self.con.commit()
[Severity: Medium]
Are we missing the creation of SQL indexes on the calls table here? The ported script omits the CREATE INDEX pcpid_idx ON calls (parent_call_path_id) and CREATE INDEX pid_idx ON calls (parent_id) steps that were present in the legacy version.
Without these indexes, will the exported-sql-viewer.py interface experience severe lag and timeouts because expanding call paths requires full table scans on the calls table?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=42
Ian Rogers
reviewerPATCH 43/49 perf python: Port export-to-postgresql to perf module
In reply to Ian RogersPort export-to-postgresql.py to a standalone script in tools/perf/python/ using the perf module and libpq via ctypes.
Improvements compared to the legacy script:
- Remove the dependency on PySide/QtSql for database creation and DDL execution by driving libpq directly via ctypes (PQconnectdb, PQexec, PQputCopyData, PQputCopyEnd) and streaming binary PostgreSQL COPY files in PostgresExporter, enabling export on headless servers without Qt installed.
- Harden database connection and SQL identifier handling by rejecting URI ('://') and connection-parameter ('=') injection strings in connect(), escaping single quotes and backslashes in connection strings, and quoting SQL identifiers (quote_ident).
- Support Intel PT and instruction trace export via perf.session(itrace=...) and perf.call_return callbacks, reconstructing relational call_paths and calls tables (including id=0 placeholder rows and callfk/returnfk foreign keys) and unpacking synthesized PT payloads (ptwrite, cbr, mwait, pwre, exstop, pwrx).
- Use a dedicated context_switch ID counter so context-switch exports do not advance sample database IDs out of sync with call_return references.
Update Documentation/db-export.txt and add a shell test (test_export_to_postgresql_python.sh) to verify the standalone exporter.
--- tools/perf/Documentation/db-export.txt | 2 +- tools/perf/python/export-to-postgresql.py | 1320 +++++++++++++++++ .../shell/test_export_to_postgresql_python.sh | 119 ++ 3 files changed, 1440 insertions(+), 1 deletion(-) create mode 100755 tools/perf/python/export-to-postgresql.py create mode 100755 tools/perf/tests/shell/test_export_to_postgresql_python.sh
tools/perf/Documentation/db-export.txt +1 -1
@@ -8,7 +8,7 @@ perf tool's python scripting engine: supports scripts: tools/perf/python/export-to-sqlite.py - tools/perf/scripts/python/export-to-postgresql.py + tools/perf/python/export-to-postgresql.py which export data to a SQLite3 or PostgreSQL database.
tools/perf/python/export-to-postgresql.py new file +1320 -0
1,320 lines not on this page · raw message
tools/perf/tests/shell/test_export_to_postgresql_python.sh new file +119 -0
119 lines not on this page · raw message
Signature · 1 line
2.55.0.1082.g2b9226bbc0-goog
[email protected]
AI reviewRe: [PATCH v1 43/49] perf python: Port export-to-postgresql to perf module
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 22 lines
Port
export-to-postgresql.pyto a standalone script intools/perf/python/using the perf module and libpq via ctypes.Improvements compared to the legacy script:
- Remove the dependency on PySide/QtSql for database creation and DDL execution by driving libpq directly via ctypes (PQconnectdb, PQexec, PQputCopyData, PQputCopyEnd) and streaming binary PostgreSQL COPY files in PostgresExporter, enabling export on headless servers without Qt installed.
- Harden database connection and SQL identifier handling by rejecting URI ('://') and connection-parameter ('=') injection strings inconnect(), escaping single quotes and backslashes in connection strings, and quoting SQL identifiers (quote_ident).
- Support Intel PT and instruction trace export viaperf.session(itrace=...)and perf.call_return callbacks, reconstructing relationalcall_pathsand calls tables (including id=0 placeholder rows and callfk/returnfk foreign keys) and unpacking synthesized PT payloads (ptwrite, cbr, mwait, pwre, exstop, pwrx).
- Use a dedicatedcontext_switchID counter so context-switch exports do not advance sample database IDs out of sync withcall_returnreferences.
Quoted 5 lines
Update
Documentation/db-export.txtand add a shell test (test_export_to_postgresql_python.sh) to verify the standalone exporter.Assisted-by Antigravity:gemini-3.1-proSigned-off-by Ian Rogers <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=43