perf tools: Annotate fixes, stdio progress indication, debuginfo-client in more places
Arnaldo Carvalho de Melo
maintainercover perf tools: Annotate fixes, stdio progress indication, debuginfo-client in more places
Hi,
This came out of work on data type profiling, but none of them depends on that work and nothing in this series needs anything from it, so sending them separately.
The fixes:
- 'perf report -s type' spins forever, burning all of a CPU with no output, on the dwz compressed debug info of zlib-ng (libz.so.1):
die_collect_vars() saves the dwarf_dieoffset() of the type DIE, which is relative to the file that DIE lives in, the dwz common file for the types shared by more than one CU, and resolving one of those offsets in the main debug file does not fail, it parses whatever is at that offset there, in this case a typedef whose DW_AT_type refers to itself, which is what makes the "follow the typedefs and qualifiers until a pointer or an array type" loop in die_get_pointer_type() spin. The offset is now resolved in the file it was recorded as coming from, which the CU of the collected DIE tells exactly rather than having to be inferred from what is at the offset, and the type chases and the struct/union member nesting are bounded, so a debug info file broken in some other way makes perf give up on a type with a pr_debug instead of looking like it hung (patch 7);
- the data type browser prints, in its samples view (-n, annotate.show_nr_samples), a local variable initialized to zero and never updated, so every member shows up as having no samples while the period and percent columns for the same entry are filled in (patch 4);
- the 'perf data type profiling' shell test turns "this PMU cannot record these events" into a test failure: the script runs under 'set -e', so the bare 'perf mem record' aborts it through the EXIT trap, which reports a signal that never happened, instead of reaching the code right below meant to report the failure (patch 1).
The features:
- 'perf report --progress' prints the phase it is in and how far along it is to stderr: the TUI shows that with a progress bar, but in the stdio case the ui_progress updates that are already there are dropped on the floor. It is what makes a slow run tell itself apart from a stuck one (patch 5);
- the debuginfo for a DSO, and the kernel symbols, can now be fetched keyed by the build ID recorded in the perf.data file, using the debuginfod client, for the cases where they are not available locally under the name the DSO was opened with: a vmlinux for a kernel that since got upgraded, or a profile recorded on another machine. Both are off with --no-debuginfod, with core.debuginfod=false, and when the build-id cache is turned off (patches 2 and 3);
- 'perf mem record' asks for PERF_SAMPLE_CPU by default, as without it the cpu field is the (u32)-1 "no CPU info" sentinel and per-sample analysis cannot tell accesses from different cores apart from same-CPU traffic (patch 8).
Which file a saved type DIE offset belongs to is settled with dwarf_cu_getdwarf(), new in elfutils 0.160, so the libdw feature test now probes for it and the message in Makefile.config says 0.160 where it said 0.157: 0.157 to 0.159 is from 2014 and has no such symbol, and those versions now lose dwarf support with that message instead of failing to link util/dwarf-aux.c.
Best regards,
- Arnaldo
Arnaldo Carvalho de Melo (8):
perf test: Skip data_type_profiling when the PMU cannot record memory
events
perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod perf symbol: Fall back to fetching the vmlinux by build ID perf annotate-data: Show the sample count in the data-type browser perf report: Add --progress option
perf scripts: Add perf-stuck, to tell where a running perf is stuck perf annotate-data: Resolve type DIEs in the debug file they came from perf mem record: Request PERF_SAMPLE_CPU by default
tools/build/feature/test-libdw.c | 15 +- tools/perf/Documentation/perf-annotate.txt | 10 + tools/perf/Documentation/perf-config.txt | 19 + tools/perf/Documentation/perf-mem.txt | 4 + tools/perf/Documentation/perf-report.txt | 28 ++ tools/perf/Documentation/perf-top.txt | 14 + tools/perf/Makefile.config | 2 +- tools/perf/builtin-annotate.c | 2 + tools/perf/builtin-mem.c | 9 + tools/perf/builtin-report.c | 17 + tools/perf/builtin-top.c | 6 + tools/perf/scripts/perf-stuck.gdb | 104 +++++ tools/perf/scripts/perf-stuck.sh | 165 ++++++++ tools/perf/tests/shell/data_type_profiling.sh | 46 ++- tools/perf/ui/Build | 1 + tools/perf/ui/browsers/annotate-data.c | 2 +- tools/perf/ui/progress.h | 2 + tools/perf/ui/stdio/progress.c | 162 ++++++++ tools/perf/util/annotate-data.c | 55 ++- tools/perf/util/annotate-data.h | 3 + tools/perf/util/config.c | 3 + tools/perf/util/debuginfo.c | 372 ++++++++++++++++++ tools/perf/util/debuginfo.h | 36 ++ tools/perf/util/dso.c | 19 + tools/perf/util/dwarf-aux.c | 161 +++++++- tools/perf/util/dwarf-aux.h | 33 ++ tools/perf/util/ordered-events.c | 17 +- tools/perf/util/session.c | 12 +- tools/perf/util/symbol.c | 40 +- tools/perf/util/symbol_conf.h | 1 + 30 files changed, 1307 insertions(+), 53 deletions(-)
base-commit: aa18964dd64511305de0711fed912054da6f5d18
Arnaldo Carvalho de Melo
maintainerPATCH 1/8 perf test: Skip data_type_profiling when the PMU cannot record memory events
From: Arnaldo Carvalho de Melo <[email protected]>
The test records with 'perf mem record' in per-thread mode and its only guard matches one specific message:
perf mem record -o /dev/null -- true 2>&1 | \
grep -q "failed: no PMU supports the memory events" && exit 2A PMU that has memory events but refuses them per-thread falls through it, and AMD IBS does:
$ perf mem record -o /dev/null -- true Error: Failure to open event 'ibs_op/ldlat=0/u' on PMU 'ibs_op' which will be removed. Invalid event (ibs_op/ldlat=0/u) in per-thread mode, enable system wide with '-a'. Error: Failure to open any events for recording.
What follows is not a skip either. The script runs under 'set -e', so the bare 'perf mem record' in test_basic_annotate() aborts it through the EXIT trap, which reports a signal that never happened and exits 1:
Basic Rust perf annotate test
Unexpected signal in test_basic_annotate
so 'perf test' turns "this PMU cannot record these events" into a test failure:
$ perf test "data type profiling" 87: perf data type profiling tests : FAILED!
The 'if [ "x$?" != "x0" ]' right below the record was meant to report "[Failed: perf record]" and carry on with the other cases, but under 'set -e' it is dead code: the shell is gone before it is reached. The same goes for the 'perf annotate' call below it.
Skip when a trivial per-thread 'perf mem record' fails, and put both commands in the condition of an 'if', where 'set -e' leaves them alone, so that the
reporting works for the cases where only one workload cannot be recorded or annotated:
$ perf test "data type profiling" 87: perf data type profiling tests : Skip
and, with just the new skip guard disabled on the same box, every case reports its failure instead of the first one aborting the run:
Basic Rust perf annotate test
Basic annotate [Failed: perf record]
Pipe Rust perf annotate test
Pipe annotate [Failed: perf record]
Basic C perf annotate test
Basic annotate [Failed: perf record]
Pipe C perf annotate test
Pipe annotate [Failed: perf record]
--- tools/perf/tests/shell/data_type_profiling.sh | 46 +++++++++++++++---- 1 file changed, 36 insertions(+), 10 deletions(-)
tools/perf/tests/shell/data_type_profiling.sh +36 -10
@@ -19,6 +19,15 @@ perfout=$(mktemp /tmp/__perf_test.perf.out.XXXXX) perf mem record -o /dev/null -- true 2>&1 | \ grep -q "failed: no PMU supports the memory events" && exit 2 +# Skip if per-thread mem record is not supported on this PMU (e.g. AMD IBS +# needs system-wide '-a'): it is what the test records with below, and a +# failing record must not be reported as a test failure. +if ! perf mem record -o /dev/null -- true 2>/dev/null +then + echo "Skip: cannot record memory events on this PMU" + exit 2 +fi + cleanup() { rm -rf "${perfdata}" "${perfout}" rm -rf "${perfdata}".old
@@ -52,25 +61,42 @@ test_basic_annotate() { index=1 ;; esac + # Under 'set -e' a bare failing command aborts the script through the EXIT + # trap, so the commands that report a failure have to be the condition of + # an 'if' for that reporting to ever happen. if [ "x${mode}" == "xBasic" ] then - perf mem record -o "${perfdata}" ${testprogs[$index]} 2> /dev/null + if ! perf mem record -o "${perfdata}" ${testprogs[$index]} 2> /dev/null + then + echo "${mode} annotate [Failed: perf record]" + err=1 + return + fi else - perf mem record -o - ${testprogs[$index]} 2> /dev/null > "${perfdata}" - fi - if [ "x$?" != "x0" ] - then - echo "${mode} annotate [Failed: perf record]" - err=1 - return + if ! perf mem record -o - ${testprogs[$index]} 2> /dev/null > "${perfdata}" + then + echo "${mode} annotate [Failed: perf record]" + err=1 + return + fi fi # Generate the annotated output file if [ "x${mode}" == "xBasic" ] then - perf annotate --code-with-type -i "${perfdata}" --stdio --percent-limit 1 2> /dev/null > "${perfout}" + if ! perf annotate --code-with-type -i "${perfdata}" --stdio --percent-limit 1 2> /dev/null > "${perfout}" + then + echo "${mode} annotate [Failed: perf annotate]" + err=1 + return + fi else - perf annotate --code-with-type -i - --stdio 2> /dev/null --percent-limit 1 < "${perfdata}" > "${perfout}" + if ! perf annotate --code-with-type -i - --stdio 2> /dev/null --percent-limit 1 < "${perfdata}" > "${perfout}" + then + echo "${mode} annotate [Failed: perf annotate]" + err=1 + return + fi fi # check if it has the target data type
Signature · 1 line
2.55.0
Arnaldo Carvalho de Melo
maintainerPATCH 2/8 perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod
In reply to Arnaldo Carvalho de MeloFrom: Arnaldo Carvalho de Melo <[email protected]>
perf already uses debuginfod to fetch source files when annotating (via probe-finder.c) and 'perf probe' has open_from_debuginfod(), which queries debuginfo keyed by build ID when a module's debuginfo isn't found locally, largely the same thing this adds; eventually that one could be moved over to the new helper. For the analysis tools there was no way to obtain the debuginfo for a DSO in a profile when it isn't available locally under the name the DSO was opened with, for instance the vmlinux for the kernel a profile was recorded on when processing it on another machine, or after the kernel and its debuginfo package got upgraded in between.
Add debuginfo__find_build_id(), that uses the debuginfod client to locate a debuginfo file keyed by the build ID, checking its local cache first and then querying the servers in DEBUGINFOD_URLS, and debuginfo__new_build_id(), that opens the DWARF in the file it finds.
The debuginfod client fails when DEBUGINFOD_URLS isn't set even when what it wants is in its local cache, and the distro setup scripts that populate it from /etc/debuginfod don't reach cron jobs, systemd services and other environments that don't source the profile scripts, so also set it from the .urls files in /etc/debuginfod when not set.
Querying servers, possibly third party ones, sends off-box the build IDs of the binaries being analysed and a fetch can take a while, so this is opt-out: on by default, off with --no-debuginfod, with core.debuginfod=false, per tool with report.debuginfod and top.debuginfod, and, since users that set buildid.dir to /dev/null (e.g. Linus) or otherwise turn the local build-id cache off clearly don't want fetched files stored on the box, off too in that case.
When a fetch is in progress in a terminal, stdio, the way it prints progress is how one gets out of it: 's' aborts the current fetch via the debuginfod client's progress callback protocol, 'd' additionally disables debuginfod for the rest of the session and points at 'perf config core.debuginfod=false' to make that permanent -- rewriting the user's ~/.perfconfig from a keypress would silently drop its comments -- and SIGINT/SIGTERM are intercepted while the terminal is in raw mode, so that it is restored and the signal is re-raised when the user interrupts a fetch.
Make dso__debuginfo() use debuginfo__new_build_id() as a fallback, keyed by the build ID recorded in the perf.data file, so that consumers such as the data type profiler can resolve the types of DSOs whose debuginfo can be fetched this way. Do the fetch outside dso__lock and remember the build IDs that were a miss, so that consumers revisiting a set of DSOs repeatedly, such as the data type profiler on every hist entry DSO switch, don't pay server round trips per attempt, a user cancelled search isn't recorded as a miss.
debuginfo__new_build_id() needs libdw to open the DWARF, so it lives in debuginfo.o, built only with CONFIG_LIBDW, while the libdebuginfod feature check is independent of NO_LIBDW; keep the new build ID prototypes under HAVE_LIBDW_SUPPORT too, with stubs otherwise, so that make NO_LIBDW=1 on a system that has the debuginfod client keeps linking.
--- tools/perf/Documentation/perf-annotate.txt | 10 + tools/perf/Documentation/perf-config.txt | 19 ++ tools/perf/Documentation/perf-report.txt | 15 + tools/perf/Documentation/perf-top.txt | 14 + tools/perf/builtin-annotate.c | 2 + tools/perf/builtin-report.c | 6 + tools/perf/builtin-top.c | 6 + tools/perf/util/config.c | 3 + tools/perf/util/debuginfo.c | 372 +++++++++++++++++++++ tools/perf/util/debuginfo.h | 36 ++ tools/perf/util/dso.c | 19 ++ tools/perf/util/symbol.c | 2 + tools/perf/util/symbol_conf.h | 1 + 13 files changed, 505 insertions(+)
tools/perf/Documentation/perf-annotate.txt +10 -0
@@ -58,6 +58,16 @@ OPTIONS --ignore-vmlinux:: Ignore vmlinux files. +--debuginfod:: +--no-debuginfod:: + Fetch debuginfo keyed by build ID from the debuginfod servers + configured in DEBUGINFOD_URLS, checking the local debuginfod + client cache first, when it is not available locally, on for + these commands by default. See the --debuginfod option of + 'perf report' for how to turn it off, including the 's' and + 'd' keys that skip a fetch in progress while 'perf' is + waiting for it. + --itrace:: Options for decoding instruction tracing data. The options are:
tools/perf/Documentation/perf-config.txt +19 -0
@@ -216,6 +216,14 @@ core.*:: addr2line-timeout:: Sets a timeout (in milliseconds) for parsing 'addr2line' output. The default timeout is 5s. + debuginfod:: + When set to 'false', disable fetching debuginfo keyed by + build ID from the debuginfod servers configured in + DEBUGINFOD_URLS. It is on by default, can be overridden per + tool with the 'report.debuginfod' and 'top.debuginfod' + options and per invocation with --no-debuginfod; it is off + too when the local build-id cache is disabled, e.g. + 'buildid.dir' set to /dev/null. tui.*, gtk.*:: Subcommands that can be configured here are 'top', 'report' and 'annotate'.
@@ -562,6 +570,14 @@ report.*:: This option can change default stat behavior with empty results. If it's set true, 'perf report --stat' will not show 0 stats. + report.debuginfod:: + Fetch debuginfo keyed by build ID from the debuginfod + servers configured in DEBUGINFOD_URLS, checking the local + debuginfod client cache first, when it is not available + locally. On by default, set to 'false' to disable it for + 'perf report', globally with 'core.debuginfod=false' or per + invocation with --no-debuginfod. + top.*:: top.children:: Same as 'report.children'. So if it is enabled, the output of 'top'
@@ -569,6 +585,9 @@ top.*:: column by default. The default is 'true'. + top.debuginfod:: + Same as 'report.debuginfod', for 'perf top'. + top.call-graph:: This is identical to 'call-graph.record-mode', except it is applicable only for 'top' subcommand. This option ONLY setup
tools/perf/Documentation/perf-report.txt +15 -0
@@ -366,6 +366,21 @@ OPTIONS --ignore-vmlinux:: Ignore vmlinux files. +--debuginfod:: +--no-debuginfod:: + Fetch debuginfo keyed by build ID from the debuginfod servers + configured in DEBUGINFOD_URLS, checking the local debuginfod + client cache first, when it is not available locally, on for + these commands by default. It can be turned off per invocation + with --no-debuginfod, per tool with the "report.debuginfod" + config option or globally with "core.debuginfod" set to false. + While a fetch is in progress in the stdio interface, 's' skips + the current fetch and 'd' skips it and disables debuginfod for + the rest of the session, pointing at 'perf config' to make that + permanent. It is disabled as well when the local build-id cache + is turned off, e.g. "buildid.dir" set to /dev/null, as that + asks for fetched files not to be kept on the box. + --kallsyms=<file>:: kallsyms pathname
tools/perf/Documentation/perf-top.txt +14 -0
@@ -83,6 +83,20 @@ Default is to monitor all CPUS. --ignore-vmlinux:: Ignore vmlinux files. +--debuginfod:: +--no-debuginfod:: + Fetch debuginfo keyed by build ID from the debuginfod servers + configured in DEBUGINFOD_URLS, checking the local debuginfod + client cache first, when it is not available locally, on for + these commands by default. Turn it off per invocation with + --no-debuginfod, with the "top.debuginfod" config option or + globally with "core.debuginfod" set to false. While a fetch is + in progress in the stdio interface, 's' skips the current fetch + and 'd' skips it and disables debuginfod for the rest of the + session, pointing at 'perf config' to make that permanent. + Disabled as well when the build-id cache is off, e.g. + "buildid.dir" set to /dev/null. + --kallsyms=<file>:: kallsyms pathname
tools/perf/builtin-annotate.c +2 -0
@@ -733,6 +733,8 @@ int cmd_annotate(int argc, const char **argv)OPT_BOOLEAN (0 ,"stdio2" , &annotate.use_stdio2,"Use the stdio interface" ),OPT_BOOLEAN (0 ,"ignore-vmlinux" , &symbol_conf.ignore_vmlinux,"don't load vmlinux even if found" ), +OPT_BOOLEAN (0 ,"debuginfod" , &symbol_conf.debuginfod, +"fetch debuginfo keyed by build ID from the debuginfod servers, on by default, use --no-debuginfod to turn off" ),OPT_STRING ('k' ,"vmlinux" , &symbol_conf.vmlinux_name,"file" ,"vmlinux pathname" ), OPT_BOOLEAN('m', "modules", &symbol_conf.use_modules,
tools/perf/builtin-report.c +6 -0
@@ -134,6 +134,10 @@ static int report__config(const char *var, const char *value, void *cb) symbol_conf.event_group =perf_config_bool (var, value);return 0 ; } +if (!strcmp (var,"report.debuginfod" )) { + symbol_conf.debuginfod =perf_config_bool (var, value); +return 0 ; + }if (!strcmp (var,"report.percent-limit" )) {double pcnt =strtof (value,NULL );
@@ -1346,6 +1350,8 @@ int cmd_report(int argc, const char **argv) "file", "vmlinux pathname"),OPT_BOOLEAN (0 ,"ignore-vmlinux" , &symbol_conf.ignore_vmlinux,"don't load vmlinux even if found" ), +OPT_BOOLEAN (0 ,"debuginfod" , &symbol_conf.debuginfod, +"fetch debuginfo keyed by build ID from the debuginfod servers, on by default, use --no-debuginfod to turn off" ),OPT_STRING (0 ,"kallsyms" , &symbol_conf.kallsyms_name,"file" ,"kallsyms pathname" ), OPT_BOOLEAN('f', "force", &symbol_conf.force, "don't complain, do it"),
tools/perf/builtin-top.c +6 -0
@@ -1436,6 +1436,10 @@ static int perf_top_config(const char *var, const char *value, void *cb __maybe_ symbol_conf.cumulate_callchain =perf_config_bool (var, value);return 0 ; } +if (!strcmp (var,"top.debuginfod" )) { + symbol_conf.debuginfod =perf_config_bool (var, value); +return 0 ; + }return 0 ; }
@@ -1508,6 +1512,8 @@ int cmd_top(int argc, const char **argv) "file", "vmlinux pathname"),OPT_BOOLEAN (0 ,"ignore-vmlinux" , &symbol_conf.ignore_vmlinux,"don't load vmlinux even if found" ), +OPT_BOOLEAN (0 ,"debuginfod" , &symbol_conf.debuginfod, +"fetch debuginfo keyed by build ID from the debuginfod servers, on by default, use --no-debuginfod to turn off" ),OPT_STRING (0 ,"kallsyms" , &symbol_conf.kallsyms_name,"file" ,"kallsyms pathname" ), OPT_BOOLEAN('K', "hide_kernel_symbols", &top.hide_kernel_symbols,
tools/perf/util/config.c +3 -0
@@ -470,6 +470,9 @@ static int perf_default_core_config(const char *var, const char *value)if (!strcmp (var,"core.addr2line-disable-warn" )) symbol_conf.addr2line_disable_warn =perf_config_bool (var, value); +if (!strcmp (var,"core.debuginfod" )) + symbol_conf.debuginfod =perf_config_bool (var, value); +/* Add other config variables here. */ return 0 ; }
tools/perf/util/debuginfo.c +372 -0
395 lines not on this page · raw message
tools/perf/util/debuginfo.h +36 -0
@@ -5,6 +5,8 @@#include <errno.h> #include <linux/compiler.h> +struct build_id ; +#ifdef HAVE_LIBDW_SUPPORT #include "dwarf-aux.h"
@@ -54,6 +56,28 @@ static inline int debuginfo__get_text_offset(struct debuginfo *dbg __maybe_unuse#ifdef HAVE_DEBUGINFOD_SUPPORT int get_source_from_debuginfod (const char *raw_path,const char *sbuild_id,char **new_path); + +/* +* Finding a debuginfo file keyed by build ID uses the debuginfod client, +* but opening the DWARF in it needs libdw, i.e. these live in +* debuginfo.o, which is only built with CONFIG_LIBDW. +*/ +#ifdef HAVE_LIBDW_SUPPORT +int debuginfo__find_build_id (const struct build_id *bid,char **path); +struct debuginfo *debuginfo__new_build_id (const struct build_id *bid); +#else +static inline int debuginfo__find_build_id (const struct build_id *bid__maybe_unused , +char **path__maybe_unused ) +{ +return -ENOTSUP ; +} + +static inline struct debuginfo * +debuginfo__new_build_id (const struct build_id *bid__maybe_unused ) +{ +return NULL ; +} +#endif /* HAVE_LIBDW_SUPPORT */ #else /* HAVE_DEBUGINFOD_SUPPORT */ static inline int get_source_from_debuginfod(const char *raw_path __maybe_unused, const char *sbuild_id __maybe_unused,
@@ -61,6 +85,18 @@ static inline int get_source_from_debuginfod(const char *raw_path __maybe_unused {return -ENOTSUP ; } + +static inline int debuginfo__find_build_id (const struct build_id *bid__maybe_unused , +char **path__maybe_unused ) +{ +return -ENOTSUP ; +} + +static inline struct debuginfo * +debuginfo__new_build_id (const struct build_id *bid__maybe_unused ) +{ +return NULL ; +}#endif /* HAVE_DEBUGINFOD_SUPPORT */ #endif /* _PERF_DEBUGINFO_H */
tools/perf/util/dso.c +19 -0
@@ -32,6 +32,7 @@#include "string2.h" #include "vdso.h" #include "annotate-data.h" +#include "debuginfo.h" #include "libdw.h" static const char *const debuglink_paths[] = {
@@ -2073,5 +2074,23 @@ struct debuginfo *dso__debuginfo(struct dso *dso)mutex_unlock (dso__lock (dso));free (name); + +/* +* The debuginfo for a DSO in the profile may not be installed +* locally, for instance the vmlinux for the kernel the profile was +* recorded on when processing it on another machine, or after the +* kernel and its debuginfo got upgraded in between. Fall back to +* fetching it keyed by the build ID recorded in the perf.data file, +* using the debuginfod client, which checks its local cache first. +* +* Do it outside dso__lock, a fetch from a remote debuginfod server +* can take a while and would otherwise block anything else using +* this dso, and honour the opt-out, the user may have asked for +* no debuginfod via --no-debuginfod, core.debuginfod=false or by +* disabling the build-id cache. +*/ +if (dinfo ==NULL ) + dinfo =debuginfo__new_build_id (dso__bid (dso)); +return dinfo; }
tools/perf/util/symbol.c +2 -0
@@ -76,6 +76,8 @@ struct symbol_conf symbol_conf = { .inline_name = true, .res_sample = 0, .addr2line_timeout_ms = 5 * 1000, + /* Fetching debuginfo by build ID, off via --no-debuginfod, etc */ + .debuginfod = true, };struct map_list_node {
tools/perf/util/symbol_conf.h +1 -0
@@ -45,6 +45,7 @@ struct symbol_conf { force, ignore_vmlinux, ignore_vmlinux_buildid, + debuginfod, show_kernel_path, use_modules, allow_aliases,
Signature · 1 line
2.55.0
Arnaldo Carvalho de Melo
maintainerPATCH 3/8 perf symbol: Fall back to fetching the vmlinux by build ID
In reply to Arnaldo Carvalho de MeloFrom: Arnaldo Carvalho de Melo <[email protected]>
A profile recorded on a kernel that is no longer installed, be it because the machine was rebooted into a new kernel or because the profile is being processed on another machine, can't have its kernel symbols resolved: /proc/kallsyms matches the running kernel, not the one in the profile, and is refused when restricted, e.g. with kernel.perf_event_paranoid > 1, while the build-id cache may carry just a kallsyms copy with zeroed addresses. With no kernel symbols the kernel samples can't be annotated, so they all end up in the '(unknown)' data type and the kernel is missing from the data type profile JSON "dsos" entry.
As a last resort, when the kernel symbols can't be found locally and the user didn't specify a kallsyms file, fetch the vmlinux keyed by the kernel build ID recorded in the perf.data file using the debuginfod client and use it for symbols, which also provides the DWARF debuginfo needed for data type profiling.
Like the other vmlinux sources this honors --ignore-vmlinux and --ignore-vmlinux_buildid: a fetched vmlinux is still a vmlinux, and the fetch is keyed by the build ID, so both flags skip it, keeping 'perf record' and 'perf probe', that set ignore_vmlinux_buildid internally, away from it. The fetch itself follows the opt-out controls added in the previous patch, --no-debuginfod, core.debuginfod=false and the build-id-cache-is-off case, and can be skipped with the 's'/'d' keys while it runs. Build IDs that were a miss are remembered, not queried again on every dso__load() retry.
With a profile recorded on a system running kernel 7.1.10, later processed after it was upgraded to 7.1.13, 'perf report -s type ' went from having all 24.91% of the kernel samples as '(unknown)' data types to resolving struct task_struct, struct rq, struct tty_struct, struct qspinlock, etc, with the kernel DSO, keyed by its build ID, appearing in the data type profile JSON.
--- tools/perf/util/symbol.c | 38 +++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-)
tools/perf/util/symbol.c +37 -1
@@ -20,6 +20,7 @@#include "cap.h" #include "cpumap.h" #include "debug.h" +#include "debuginfo.h" #include "demangle-cxx.h" #include "demangle-java.h" #include "demangle-ocaml.h"
@@ -2191,12 +2192,35 @@ static char *dso__find_kallsyms(struct dso *dso, struct map *map)return strdup (path); } +/* +* Last resort when the symbols for the kernel the profile was recorded +* on can't be found locally: fetch the vmlinux keyed by the build ID +* recorded in the perf.data file using the debuginfod client, which +* checks its local cache first, e.g. when processing the profile on +* another machine or after the kernel and its debuginfo package got +* upgraded in between. +*/ +static int dso__load_vmlinux_build_id (struct dso *dso,struct map *map) +{ +char *path =NULL ; + +if (!dso__has_build_id (dso)) +return -1 ; + +if (debuginfo__find_build_id (dso__bid (dso), &path)) +return -1 ; + +/* Takes ownership of 'path' even when it fails */ +return dso__load_vmlinux (dso, map, path, true); +} +static int dso__load_kernel_sym (struct dso *dso,struct map *map) {int err;const char *kallsyms_filename =NULL ;char *kallsyms_allocated_filename =NULL ;char *filename =NULL ; +bool user_kallsyms = false;/* * Step 1: if the user specified a kallsyms or vmlinux filename, use
@@ -2215,6 +2239,7 @@ static int dso__load_kernel_sym(struct dso *dso, struct map *map)*/ if (symbol_conf.kallsyms_name !=NULL ) { kallsyms_filename = symbol_conf.kallsyms_name; + user_kallsyms = true; goto do_kallsyms; }
@@ -2257,7 +2282,18 @@ static int dso__load_kernel_sym(struct dso *dso, struct map *map)pr_debug ("Using %s for symbols\n" , kallsyms_filename);free (kallsyms_allocated_filename); -if (err >0 && !dso__is_kcore (dso)) { +/* +* The kallsyms may be unavailable or restricted, e.g. +* /proc/kallsyms with kernel.perf_event_paranoid > 1, try to fetch +* the vmlinux keyed by the build ID using debuginfod as a last +* resort, honoring --ignore-vmlinux and --ignore-vmlinux_buildid +* like the other vmlinux sources above. +*/ +if (err <=0 && !user_kallsyms && + !symbol_conf.ignore_vmlinux && + !symbol_conf.ignore_vmlinux_buildid) { + err =dso__load_vmlinux_build_id (dso, map); + }else if (err >0 && !dso__is_kcore (dso)) {struct maps *kmaps =map__kmaps (map);dso__set_binary_type (dso,DSO_BINARY_TYPE__KALLSYMS );
Signature · 1 line
2.55.0
Arnaldo Carvalho de Melo
maintainerPATCH 4/8 perf annotate-data: Show the sample count in the data-type browser
In reply to Arnaldo Carvalho de MeloFrom: Arnaldo Carvalho de Melo <[email protected]>
The data-type browser has a samples view, selected with -n (or with annotate.show_nr_samples), in which browser__write_overhead() prints a local nr_samples variable that is initialized to zero and never updated, so every member is listed as having no samples while the period and percent columns for the same entry are filled in.
Print the histogram entry's own count instead.
This predates the load/store counter split, so fix it ahead of that
patch: the split then only has to adapt a line that is already correct, and this fix can be picked on its own.
--- tools/perf/ui/browsers/annotate-data.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/ui/browsers/annotate-data.c +1 -1
@@ -370,7 +370,7 @@ static void browser__write_overhead(struct ui_browser *uib,u64 period = hist->period;double percent = total->period ? (100.0 * period / total->period) :0 ;bool current =ui_browser__is_current_entry (uib, row); -int nr_samples =0 ; +int nr_samples = hist->nr_samples;ui_browser__set_percent_color (uib, percent, current);
Signature · 1 line
2.55.0
Arnaldo Carvalho de Melo
maintainerPATCH 5/8 perf report: Add --progress option
In reply to Arnaldo Carvalho de MeloFrom: Arnaldo Carvalho de Melo <[email protected]>
Processing a large data type profiling session, e.g. an AMD IBS one, can take a long time and, when using the stdio output, 'perf report' gives no feedback about which phase it is in nor about how far along it is: the TUI has a progress bar for that, but in the stdio case the ui_progress updates, that are already there, are dropped on the floor.
Add a --progress option that installs a stdio backend for ui_progress, ui/stdio/progress.c, printing the phase title, the percentage done and the current/total counts to stderr:
Processing events... [ 42.3%] 317M / 746M Merging related events... [ 61.0%] 309026 / 506686 Sorting events for output... [ 98.2%] 14132 / 14387
The first one is the perf.data file size based progress already kept while reading events, sized with the same unit_number__scnprintf() used by the TUI progress bar title; the other two are the hist entry based ones for the hist entry merging (collapse) and output sorting phases, that print raw counts.
Steps are 1% of the phase total, the default (total / 16) is tuned for the character cell based TUI bar, and the last update normally stops short of the total, so finishing a phase prints it as complete.
When stderr is a tty the line is updated in place, with each phase getting a line of its own, otherwise one line is printed per update, so that redirecting stderr to a file leaves a readable log of the phases.
Phases can be nested, e.g. the ordered events flushes that take place while the "Processing events..." phase is still in progress, so the backend keeps track of the ones started so far to be able to complete the right one when a phase finishes, as ui_progress__finish() gets no arguments.
That bookkeeping requires ui_progress__init() and ui_progress__finish() to be paired, and a few callers had paths returning early without the
finish: do_flush() in ordered-events.c on session_done() and on deliver() errors, and the ENOMEM paths right after the init in __perf_session__process_pipe_events() and __perf_session__process_dir_events().
Fix those, with a backend that kept the stale entries it would print through dangling pointers to returned stack frames, and have the update() method ignore anything that doesn't match the innermost running phase, printing nothing is better than printing another phase's numbers or reading the stack array with index -1.
Committer testing:
⬢ [acme@toolbx perf-tools-next]$ perf record -o perf.data.small -- sleep 0.2 [ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.002 MB perf.data.small (7 samples) ] ⬢ [acme@toolbx perf-tools-next]$ perf report --progress -i perf.data.small > /dev/null Processing events... [ 0.0%] 0B / 2K
Processing events... [ 99.6%] 2040B / 2K
Processing time ordered events... [100.0%] 13 / 13
Processing events... [100.0%] 2K / 2K
Sorting events for output... [100.0%] 5 / 5
⬢ [acme@toolbx perf-tools-next]$
Nothing goes to stderr without --progress:
⬢ [acme@toolbx perf-tools-next]$ perf report --stdio -i perf.data.small > /dev/null 2> stderr.txt ⬢ [acme@toolbx perf-tools-next]$ wc -c stderr.txt 0 stderr.txt ⬢ [acme@toolbx perf-tools-next]$
With a 321 MB perf.data (perf mem record, i7-14700K):
⬢ [acme@toolbx perf-tools-next]$ perf report --progress -i perf.data.i7 > /dev/null Processing events... [ 0.0%] 0B / 306M [...] Sorting events for output... [100.0%] 29996 / 29996 ⬢ [acme@toolbx perf-tools-next]$
Which is what prompted this: an AMD IBS data type profile session hangs in the DWARF type resolution done when merging hist entries, with --progress it now is visible that it is not the event loading that is stuck, but the merging, at a specific hist entry:
⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > /dev/null Processing events... [ 0.0%] 0B / 746M [...] Processing events... [100.0%] 746M / 746M Merging related events... [ 58.0%] 293828 / 506686 Merging related events... [ 59.0%] 298894 / 506686 Merging related events... [ 60.0%] 303960 / 506686 Merging related events... [ 61.0%] 309026 / 506686 ⬢ [acme@toolbx perf-tools-next]$ gdb -p $(pidof perf) -batch -ex 'bt 6' -ex detach #0 0x0000000000773f31 in die_get_pointer_type () #1 0x000000000077b817 in find_data_type () #2 0x0000000000636f20 in __hist_entry__get_data_type () #3 0x0000000000639c65 in hist_entry__get_data_type () #4 0x00000000006e57a8 in sort__type_collapse () #5 0x00000000006ef0ee in hists__collapse_resort () ⬢ [acme@toolbx perf-tools-next]$
The report related entries in 'perf test' pass:
⬢ [acme@toolbx perf-tools-next]$ perf test 17 27 30 31 85 88
17: Match and link multiple hists : Ok 27: Filter hist entries : Ok 30: Sort output of hist entries : Ok 31: Cumulate child hist entries : Ok
85: Test that perf report includes file offsets and event type names in diagnostic messages. : Skip 88: Test that perf report handles truncated perf.data gracefully (no crash, no segfault — clean error exit).: Ok ⬢ [acme@toolbx perf-tools-next]$
--- tools/perf/Documentation/perf-report.txt | 13 ++ tools/perf/builtin-report.c | 11 ++ tools/perf/ui/Build | 1 + tools/perf/ui/progress.h | 2 + tools/perf/ui/stdio/progress.c | 162 +++++++++++++++++++++++ tools/perf/util/ordered-events.c | 17 ++- tools/perf/util/session.c | 12 +- 7 files changed, 211 insertions(+), 7 deletions(-) create mode 100644 tools/perf/ui/stdio/progress.c
tools/perf/Documentation/perf-report.txt +13 -0
@@ -29,6 +29,19 @@ OPTIONS --quiet:: Do not show any warnings or messages. (Suppress -v) +--progress:: + Show progress for each of the processing phases, printing the + percentage done and the current/total counts: the first phase + counts the bytes of the perf.data file processed so far, the + merge and sort phases count hist entries, e.g.: + + Processing events... [ 42.3%] 4G / 10G + Merging related events... [ 7.1%] 1024 / 14387 + Sorting events for output... [ 98.2%] 14132 / 14387 + + It is a no-op when using the TUI or GTK browsers, that already + present progress information. + -n:: --show-nr-samples:: Show the number of samples for each symbol
tools/perf/builtin-report.c +11 -0
@@ -87,6 +87,7 @@ struct report {bool use_gtk;#endif bool use_stdio; +bool progress;bool show_full_info;bool show_threads;bool inverted_callchain;
@@ -1373,6 +1374,8 @@ int cmd_report(int argc, const char **argv) "Use the stdio interface"),OPT_BOOLEAN (0 ,"weights" , &symbol_conf.annotate_weight,"Show or hide weight columns in annotation. Default show if non-zero." ), +OPT_BOOLEAN (0 ,"progress" , &report.progress, +"Show progress while processing the perf.data file" ),OPT_BOOLEAN (0 ,"header" , &report.header,"Show data header." ),OPT_BOOLEAN (0 ,"header-only" , &report.header_only, "Show only data header."),
@@ -1765,6 +1768,14 @@ int cmd_report(int argc, const char **argv)else use_browser =0 ; +/* +* The TUI/GTK browsers already show progress, this is for the stdio +* case, where we print the percentage of the perf.data file that was +* processed so far, for each of the processing phases. +*/ +if (report.progress && use_browser ==0 ) +stdio_progress__init (); +if (report.data_type && use_browser ==1 ) { symbol_conf.annotate_data_member = true; symbol_conf.annotate_data_sample = true;
tools/perf/ui/Build +1 -0
@@ -4,6 +4,7 @@ perf-ui-y += progress.o perf-ui-y += util.o perf-ui-y += hist.o perf-ui-y += stdio/hist.o +perf-ui-y += stdio/progress.o CFLAGS_setup.o += -DLIBDIR="BUILD_STR($(LIBDIR))"
tools/perf/ui/progress.h +2 -0
@@ -23,6 +23,8 @@ void __ui_progress__init(struct ui_progress *p, u64 total,void ui_progress__update (struct ui_progress *p,u64 adv); +void stdio_progress__init (void ); +struct ui_progress_ops {void (*init)(struct ui_progress *p);void (*update)(struct ui_progress *p);
tools/perf/ui/stdio/progress.c new file +162 -0
162 lines not on this page · raw message
tools/perf/util/ordered-events.c +13 -4
@@ -237,14 +237,16 @@ static int do_flush(struct ordered_events *oe, bool show_progress)ui_progress__init (&prog, oe->nr_events,"Processing time ordered events..." );list_for_each_entry_safe (iter, tmp, head, list) { -if (session_done ()) -return 0 ; +if (session_done ()) { + ret =0 ; + goto out_progress; + }if (iter->timestamp > limit)break ; ret = oe->deliver (oe, iter);if (ret <0 ) -return ret; + goto out_progress;ordered_events__delete (oe, iter); oe->last_flush = iter->timestamp;
@@ -258,10 +260,17 @@ static int do_flush(struct ordered_events *oe, bool show_progress)else if (last_ts <= limit) oe->last = list_entry(head->prev, struct ordered_event, list); + ret =0 ; +out_progress: +/* +* Always pair ui_progress__init() with ui_progress__finish(), the +* stdio progress backend tracks phases on a stack and an early +* return that skipped the finish would leave the dead 'prog' on it. +*/ if (show_progress)ui_progress__finish (); -return 0 ; +return ret; } static int __ordered_events__flush(struct ordered_events *oe, enum oe_flush how,
tools/perf/util/session.c +9 -3
@@ -3146,8 +3146,12 @@ static int __perf_session__process_pipe_events(struct perf_session *session) cur_size =sizeof (union perf_event ); buf =malloc (cur_size); -if (!buf) -return -errno; +if (!buf) { + err = -errno; +if (update_prog) +ui_progress__finish (); +return err; + }ordered_events__set_copy_on_queue (oe, true); more: event = buf;
@@ -3646,8 +3650,10 @@ static int __perf_session__process_dir_events(struct perf_session *session) } rd =calloc (nr_readers,sizeof (struct reader )); -if (!rd) +if (!rd) { +ui_progress__finish ();return -ENOMEM ; + } rd[0 ] = (struct reader ) { .fd =perf_data__fd (session->data),
Signature · 1 line
2.55.0
Arnaldo Carvalho de Melo
maintainerPATCH 6/8 perf scripts: Add perf-stuck, to tell where a running perf is stuck
In reply to Arnaldo Carvalho de MeloFrom: Arnaldo Carvalho de Melo <[email protected]>
A perf that takes forever is hard to tell apart from one that is stuck in a loop, and when it is stuck there is no way to know where without attaching gdb to it and looking around, which is what this does, from the outside, sampling /proc/<pid> at a fixed interval:
⬢ [acme@toolbx perf-tools-next]$ tools/perf/scripts/perf-stuck.sh -i 15 -n 4 -l ibs.log $(pgrep -x perf) watching 2297313 (perf report --progress -s type -i perf.data.ibs) every 15s 09:39:34 state=R cpu=+0 (0.00s) rss=664295kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB Merging related events... [ 39.0%] 197574 / 506686 09:39:49 state=R cpu=+1496 (14.96s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=1 Merging related events... [ 39.0%] 197574 / 506686 09:40:04 state=R cpu=+1497 (14.97s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=2 Merging related events... [ 39.0%] 197574 / 506686 09:40:19 state=R cpu=+1497 (14.97s) rss=502250kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=0 Merging related events... [ 61.0%] 309026 / 506686
The CPU time used grows by a whole interval on every sample while the [stack] mapping, that would be moving down if this was recursion, stays put, so that one is spinning, and the last line of the progress log of 'perf report --progress' tells in which phase.
With -g it runs gdb when no progress is made for two consecutive samples, using the perf-stuck.gdb that sits next to it, which adds the perf-die-chain command used by the next patch, to print the DIE chain a DWARF type chasing loop is walking when the perf being watched is stuck in one of those.
It is a prototype: this wants to become a first class 'perf stuck' command, sampling a running process from inside perf, instead of this shell script poking at /proc and shelling out to gdb.
Example:
⬢ [acme@toolbx perf-tools-next]$ tools/perf/scripts/perf-stuck.sh -i 1 -n 3 -g $(pgrep -x sleep) watching 2342826 (sleep 45 ) every 1s
11:58:36 state=S cpu=+0 (0.00s) rss=492kB stack=7ffe21f89000-7ffe21faa000 size=132kB stuck=2 (no progress log) ... gdb output of 2342826 in /tmp/perf-stuck-gdb.Uz9i5p
⬢ [acme@toolbx perf-tools-next]$ tail -4 /tmp/perf-stuck-gdb.Uz9i5p
#3 0x0000555bc6ddc28f in main ()
not in a DWARF type chaser, try: bt
not in find_data_type()
[Inferior 1 (process 2342826) detached]
⬢ [acme@toolbx perf-tools-next]$
--- tools/perf/scripts/perf-stuck.gdb | 104 +++++++++++++++++++ tools/perf/scripts/perf-stuck.sh | 165 ++++++++++++++++++++++++++++++ 2 files changed, 269 insertions(+) create mode 100644 tools/perf/scripts/perf-stuck.gdb create mode 100755 tools/perf/scripts/perf-stuck.sh
tools/perf/scripts/perf-stuck.gdb new file +104 -0
@@ -0,0 +1,104 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# gdb commands for a perf that is stuck, used by perf-stuck.sh -g and usable +# directly: +# +# gdb -p $(pgrep -x perf) -batch -x perf-stuck.gdb -ex bt +# +# PROTOTYPE: part of the perf-stuck.sh stopgap, see the note at the start of +# that script: this wants to move into a first class 'perf stuck' command, +# which would print these DIE chains by itself, without gdb. +# +# The settings are the ones that keep a batch attach from stopping to ask +# questions (debuginfod, pagination) and that make the output readable. +# +# The commands are for the DWARF type chasers in util/dwarf-aux.c, the +# functions a data type profiling 'perf report -s type' spins in when a +# debug info file has a type chain that got into a cycle: +# +# perf-die-chain <function> <die variable> [iterations] +# perf-die-chain-all [iterations] +# perf-dso +# +# For each iteration of the chasing loop they print the DIE address, the +# CU it came from, its offset in the debug file, its tag and its name: a +# cycle shows up as the same handful of (addr, cu) pairs repeating, and a +# CU that changes from one iteration to the next means the chase is +# hopping between a debug file and its dwz common file. + +set pagination off +set confirm off +set debuginfod enabled off +set print pretty on +set height 0 +set width 0 + +define perf-die-chain + if $argc < 2 + printf "usage: perf-die-chain <function> <die variable> [iterations]\n" + else + frame function $arg0 + if $argc == 3 + set $perf_die_chain_n = $arg2 + else + set $perf_die_chain_n = 10 + end + set $perf_die_chain_head = $pc + set $perf_die_chain_i = 0 + while $perf_die_chain_i < $perf_die_chain_n + printf "chain[%d] die=%p addr=%p cu=%p off=0x%lx tag=%d name=%s\n", $perf_die_chain_i, $arg1, $arg1->addr, $arg1->cu, ((Dwarf_Off) dwarf_dieoffset($arg1)), ((int) dwarf_tag($arg1)), ((char *) dwarf_diename($arg1)) + until *$perf_die_chain_head + set $perf_die_chain_i = $perf_die_chain_i + 1 + end + end +end + +document perf-die-chain +Print the DIE chain being walked by a DWARF type chasing loop. +usage: perf-die-chain <function> <die variable> [iterations] + perf-die-chain die_get_pointer_type type_die + perf-die-chain __die_get_real_type vr_die + perf-die-chain die_get_real_type vr_die +end + +define perf-die-chain-all + if $argc == 0 + set $perf_die_chain_n = 10 + else + set $perf_die_chain_n = $arg0 + end + if $_any_caller_is("die_get_pointer_type", 20) + printf "stuck in die_get_pointer_type():\n" + perf-die-chain die_get_pointer_type type_die $perf_die_chain_n + else + if $_any_caller_is("__die_get_real_type", 20) + printf "stuck in __die_get_real_type():\n" + perf-die-chain __die_get_real_type vr_die $perf_die_chain_n + else + if $_any_caller_is("die_get_real_type", 20) + printf "stuck in die_get_real_type():\n" + perf-die-chain die_get_real_type vr_die $perf_die_chain_n + else + printf "not in a DWARF type chaser, try: bt\n" + end + end + end +end + +document perf-die-chain-all +Find which DWARF type chaser the process is in and print the DIE chain. +usage: perf-die-chain-all [iterations] +end + +define perf-dso + if $_any_caller_is("find_data_type", 20) + frame function find_data_type + printf "dso=%s ip=0x%lx sym=%s\n", dloc->ms.map->dso->name, dloc->ip, dloc->ms.sym->name + else + printf "not in find_data_type()\n" + end +end + +document perf-dso +Print the dso, ip and symbol of the data location being resolved. +end
tools/perf/scripts/perf-stuck.sh new file +165 -0
165 lines not on this page · raw message
Signature · 1 line
2.55.0
Arnaldo Carvalho de Melo
maintainerPATCH 7/8 perf annotate-data: Resolve type DIEs in the debug file they came from
In reply to Arnaldo Carvalho de MeloFrom: Arnaldo Carvalho de Melo <[email protected]>
A 'perf report -s type' on a 783 MB AMD IBS data type profiling session hangs, burning all of a CPU and producing no output:
⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > /dev/null Processing events... [100.0%] 746M / 746M Merging related events... [ 58.0%] 293828 / 506686 Merging related events... [ 59.0%] 298894 / 506686 Merging related events... [ 60.0%] 303960 / 506686 Merging related events... [ 61.0%] 309026 / 506686 [ ... nothing else, ever ... ]
It is a spin and not a slow path: with perf-stuck the CPU time used grows by a whole interval on every sample, while the [stack] mapping, which would be moving down if this was recursion, stays put:
⬢ [acme@toolbx perf-tools-next]$ tools/perf/scripts/perf-stuck.sh -i 15 -n 4 -l ibs.log $(pgrep -x perf) watching 2297313 (perf report --progress -s type -i perf.data.ibs) every 15s 09:39:34 state=R cpu=+0 (0.00s) rss=664295kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB Merging related events... [ 39.0%] 197574 / 506686 09:39:49 state=R cpu=+1496 (14.96s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=1 Merging related events... [ 39.0%] 197574 / 506686 09:40:04 state=R cpu=+1497 (14.97s) rss=665383kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=2 Merging related events... [ 39.0%] 197574 / 506686 09:40:19 state=R cpu=+1497 (14.97s) rss=502250kB stack=7ffc3ae31000-7ffc3ae52000 size=132kB stuck=0 Merging related events... [ 61.0%] 309026 / 506686
Attaching gdb and stepping the loop, printing dwarf_dieoffset() and dwarf_tag() for the DIE being chased on each trip round it, shows the chase never moving, on a typedef that refers to itself:
⬢ [acme@toolbx perf-tools-next]$ gdb -p $(pgrep -x perf) -batch -x tools/perf/scripts/perf-stuck.gdb -ex 'perf-die-chain die_get_pointer_type type_die 8' -ex perf-dso -ex detach stuck in die_get_pointer_type(): #3 0x0000000000772421 in die_get_pointer_type (type_die=0x7ffc3ae4b710, type_die@entry=0x7ffc3ae4b6f0, die_mem=die_mem@entry=0x7ffc3ae4b710) at util/dwarf-aux.c:327 327 type_die = die_get_type(type_die, die_mem); chain[0] die=0x7ffc3ae4b710 addr=0x7ff6811591ef cu=0x44ef0878 off=0x1f tag=22 name=(null) chain[1] die=0x7ffc3ae4b710 addr=0x7ff6811591ef cu=0x44ef0878 off=0x1f tag=22 name=(null) [ ... the very same DIE, forever ... ] dso=/usr/lib64/libz.so.1.3.1.zlib-ng ip=0xe2e sym=build_tree
The DIE is at offset 0x1f of the debug info of libz.so.1, which is zlib-ng, and is one of the dwz compressed ones: the type DIEs shared by more than one CU live in the common file, where 0x1f is a perfectly good DW_TAG_base_type:
⬢ [acme@toolbx perf-tools-next]$ readelf --debug-dump=info /usr/lib/debug/.dwz/zlib-ng-2.3.3-3.fc44.x86_64 | sed -n '/Compilation Unit @ offset 0:/,/Compilation Unit @ offset 0x5f:/p'
Compilation Unit @ offset 0:
Length: 0x5b (32-bit)
Version: 5
Unit Type: DW_UT_partial (3)
<0><c>: Abbrev Number: 1 (DW_TAG_partial_unit)
[...]
<1><1f>: Abbrev Number: 62 (DW_TAG_base_type)
<20> DW_AT_byte_size : 4
<21> DW_AT_encoding : 5 (signed)
<22> DW_AT_name : intwhile in the main debug file that same offset is not a DIE at all, it is the start of another unit's header:
⬢ [acme@toolbx perf-tools-next]$ readelf --debug-dump=info /usr/lib/debug/usr/lib64/libz.so.1.3.1.zlib-ng-2.3.3-3.fc44.x86_64.debug | grep 'Compilation Unit @' | head -2
Compilation Unit @ offset 0:
Compilation Unit @ offset 0x1f:
die_collect_vars() saves the dwarf_dieoffset() of the type DIE, which is relative to the file that DIE lives in, and update_var_state() then hands that offset to dwarf_offdie() together with the main debug file, where dwarf_offdie() parses whatever is there as a DIE, in this case the one typedef that refers to itself, which is what makes the "follow the typedefs and qualifiers until a pointer or an array type" loop in die_get_pointer_type() spin.
So record, next to the offset, whether the type DIE was in the file the variable DIE came from or in the dwz common one, and resolve the offset in the file it came from, which is what the new die_get_type_die() does.
Which of the two that is does not have to be guessed from what is at the
offset: dwz encodes the references into its common file as DW_FORM_GNU_ref_alt, so elfutils resolves them into the alt Dwarf and the CU of the resulting type DIE belongs to that other file, so comparing the Dwarf each of the two CUs belongs to, die_same_file(), settles it exactly, for any number of hops from the variable DIE.
Then bound the chases themselves: no sane chain of typedefs and qualifiers is 32 DIEs long, no sane nesting for the struct and union members that __add_member_cb() follows recursively is 8 deep, and the same recursion bound covers the type names die_get_typename_from_type() builds by following pointers and arrays, so that a debug info file broken in some other way makes perf give up on a type, telling about it with pr_debug, visible with -v, instead of looking like it hung. The member nesting bound is reported with pr_debug_dtp (visible with -vvv or -D type-profile).
Cycles can't occur in member trees from valid DWARF, embedded members can't be recursive in C, so the nesting bound only ever bites legitimate depth and the member where the recursion was cut is marked 'truncated', which the JSON exporter added in a subsequent series reports to its consumers, so that they can tell a truncated tree from one that really ends there.
die_get_type_die() then has no fallback to the other file: resolving an alt file offset in the main file is the misparse above, so it resolves in the file the offset was recorded as belonging to, and gives up on the type when the offset does not resolve there. The tag the type DIE had is kept as a sanity check, a mismatch means the debug info changed under perf or is broken in yet another way, and the bounds on the type chasers remain the backstop: a debug info file broken in some other way still makes perf give up on a type with a pr_debug instead of hanging.
Testing:
Before, on the 783 MB AMD IBS session, killed after some 8 minutes stuck at 61%:
⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > /dev/null Processing events... [100.0%] 746M / 746M Merging related events... [ 61.0%] 309026 / 506686 ⬢ [acme@toolbx perf-tools-next]$
After, the whole session is processed, and the zlib-ng types, from the build_tree() hist entry that used to hang it, show up:
⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > ibs.out Processing events... [100.0%] 746M / 746M
Merging related events... [100.0%] 506686 / 506686
Sorting events for output... [100.0%] 1428 / 1428
⬢ [acme@toolbx perf-tools-next]$ grep -E "deflate_state|inflate_state|internal_state" ibs.out
0.00% deflate_state
0.00% deflate_state*
0.00% struct inflate_state
0.00% struct inflate_state*
0.00% struct internal_state
⬢ [acme@toolbx perf-tools-next]$ perf test 17 27 30 31 85 88
17: Match and link multiple hists : Ok
27: Filter hist entries : Ok
30: Sort output of hist entries : Ok
31: Cumulate child hist entries : Ok85: Test that perf report includes file offsets and event type names in diagnostic messages. : Ok 88: Test that perf report handles truncated perf.data gracefully (no crash, no segfault — clean error exit).: Skip ⬢ [acme@toolbx perf-tools-next]$
Requiring elfutils 0.160 for this: dwarf_cu_getdwarf(), the function that tells which Dwarf a CU belongs to, first appeared in 0.160 ("libdw:
New functions dwarf_cu_getdwarf, dwarf_cu_die", elfutils NEWS), so the libdw feature test now probes for it, in tools/build/feature/test-libdw.c, and Makefile.config says 0.160 where it said 0.157. The probe takes the address instead of calling it, as that is all that is needed to make a 0.157-0.159 elfutils, from 2014 and without the symbol, disable dwarf support with the existing message rather than fail to link dwarf-aux.c.
--- tools/build/feature/test-libdw.c | 15 ++- tools/perf/Makefile.config | 2 +- tools/perf/util/annotate-data.c | 55 ++++++++--- tools/perf/util/annotate-data.h | 3 + tools/perf/util/dwarf-aux.c | 161 +++++++++++++++++++++++++++---- tools/perf/util/dwarf-aux.h | 33 +++++++ 6 files changed, 235 insertions(+), 34 deletions(-)
tools/build/feature/test-libdw.c +14 -1
@@ -49,8 +49,21 @@ int test_elfutils(void)return 0 ; } +/* +* elfutils 0.160 and later: used to tell which debug file a DIE lives in, +* the dwz alt file or the main one, see die_same_file() in +* tools/perf/util/dwarf-aux.c. Only the symbol is needed, so take its +* address instead of calling it. +*/ +int test_libdw_cu_getdwarf (void ) +{ +void *sym = (void *)dwarf_cu_getdwarf; + +return sym ==NULL ; +} +int main (void ) {return test_libdw () +test_libdw_unwind () +test_libdw_getlocations () + -test_libdw_getcfi () +test_elfutils (); +test_libdw_getcfi () +test_libdw_cu_getdwarf () +test_elfutils (); }
tools/perf/Makefile.config +1 -1
@@ -470,7 +470,7 @@ else else ifneq ($(feature-libdw), 1) ifndef NO_LIBDW - $(warning No libdw.h found or old libdw.h found or elfutils is older than 0.157, disables dwarf support. Please install new elfutils-devel/libdw-dev) + $(warning No libdw.h found or old libdw.h found or elfutils is older than 0.160, disables dwarf support. Please install new elfutils-devel/libdw-dev) NO_LIBDW := 1 endif endif # Dwarf support
tools/perf/util/annotate-data.c +43 -12
129 lines not on this page · raw message
tools/perf/util/annotate-data.h +3 -0
@@ -57,6 +57,9 @@ struct annotated_member {char *var_name;int offset;int size; +unsigned int depth; +/* Children not expanded because the nesting limit was reached */ +bool truncated; };/**
tools/perf/util/dwarf-aux.c +141 -20
244 lines not on this page · raw message
tools/perf/util/dwarf-aux.h +33 -0
@@ -152,6 +152,8 @@ int die_get_scopes(Dwarf_Die *cu_die, Dwarf_Addr pc, Dwarf_Die **scopes);struct die_var_type {struct die_var_type *next;u64 die_off; +int die_tag; +bool from_alt;/* die_off is relative to the alt (dwz) file */ u64 addr;u64 end;/* end address of location range */ int reg;
@@ -183,6 +185,37 @@ Dwarf_Die *die_find_variable_by_addr(Dwarf_Die *sc_die, Dwarf_Addr addr,/* Save all variables and parameters in this scope */ void die_collect_vars (Dwarf_Die *sc_die,struct die_var_type **var_types); +/* +* Get the type DIE saved by die_collect_vars()/die_collect_global_vars(). +* +* The offsets those save are the dwarf_dieoffset() of the type DIE, which is +* relative to the debug file that DIE lives in: the dwz common file, the alt +* file in libdw terms, for the types shared by more than one CU, the main +* file for the rest. Resolving an alt file offset in the main file does not +* fail: dwarf_offdie() parses whatever is at that offset there, and an offset +* that is a CU header in the main file reads back as a typedef whose +* DW_AT_type refers to itself, which is what hung 'perf report -s type' on +* the dwz compressed debug info of zlib-ng (libz.so.1). +* +* So @from_alt, recorded when the offset was saved, says which of the two +* files to resolve it in. It is not inferred from the DIE contents: dwz +* encodes references into the common file as DW_FORM_GNU_ref_alt, so +* elfutils resolves them into the alt Dwarf and the CU of the type DIE then +* belongs to that other file, which die_same_file() compares exactly. +* +* There is deliberately no fallback to the other file when the offset does +* not resolve: that fallback is the misparse above. +* +* @die_tag is then only a sanity check: the offset is of a DIE that had this +* tag when it was saved, so a mismatch means the debug info changed under us, +* or is broken, and giving up on the type is the right answer. +*/ +Dwarf_Die *die_get_type_die (Dwarf *dbg,u64 die_off,int die_tag,bool from_alt, +Dwarf_Die *die_mem); + +/* Whether two DIEs live in the same debug file */ +bool die_same_file (Dwarf_Die *die_a,Dwarf_Die *die_b); +/* Save all global variables in this CU */ void die_collect_global_vars (Dwarf_Die *cu_die,struct die_var_type **var_types);
Signature · 1 line
2.55.0
Arnaldo Carvalho de Melo
maintainerPATCH 8/8 perf mem record: Request PERF_SAMPLE_CPU by default
In reply to Arnaldo Carvalho de MeloFrom: Arnaldo Carvalho de Melo <[email protected]>
The data-type profiling per-sample stream keys cross-CPU contention on sample->cpu: without PERF_SAMPLE_CPU the cpu field is the (u32)-1 "no CPU info" sentinel, documented as such in perf_session__deliver_event(), and same-instance reads and writes from different cores are indistinguishable from same-CPU traffic, so pahole's false-sharing detector cannot tell them apart.
builtin-record.c already defines --sample-cpu and 'perf mem record' forwards unknown options to the record parser, so passing it explicitly works today; make it the default, next to the -d (addr) and -W (weight) the command already requests, documenting it in perf-mem(1).
--- tools/perf/Documentation/perf-mem.txt | 4 ++++ tools/perf/builtin-mem.c | 9 +++++++++ 2 files changed, 13 insertions(+)
tools/perf/Documentation/perf-mem.txt +4 -0
@@ -14,6 +14,10 @@ DESCRIPTION ----------- "perf mem record" runs a command and gathers memory operation data from it, into perf.data. Perf record options are accepted and are passed through. +It also requests the address (-d), the weight (-W, where supported) and the +CPU id (--sample-cpu) of every sampled access by default; the CPU id is what +lets per-sample analysis tell reads and writes to the same data from +different cores apart from same-CPU traffic. "perf mem report" displays the result. It invokes perf report with the right set of options to display a memory access profile. By default, loads
tools/perf/builtin-mem.c +9 -0
@@ -135,6 +135,15 @@ static int __cmd_record(int argc, const char **argv, struct perf_mem *mem, rec_argv[i++] ="-d" ; +/* +* The data-type profiling per-sample stream keys cross-CPU +* contention on sample->cpu (PERF_SAMPLE_CPU); without it the cpu +* field is the (u32)-1 'no CPU info' sentinel and same-instance +* reads and writes from different cores are indistinguishable +* from same-CPU traffic. +*/ + rec_argv[i++] ="--sample-cpu" ; +if (mem->phys_addr) rec_argv[i++] ="--phys-data" ;
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH v1 1/8] perf test: Skip data_type_profiling when the PMU cannot record memory events
In reply to Arnaldo Carvalho de MeloSashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
From: Arnaldo Carvalho de Melo <[email protected]>
The test records with 'perf mem record' in per-thread mode and its only guard matches one specific message:
perf mem record -o /dev/null -- true 2>&1 | \ grep -q "failed: no PMU supports the memory events" && exit 2A PMU that has memory events but refuses them per-thread falls through it, and AMD IBS does:
$ perf mem record -o /dev/null -- true Error: Failure to open event 'ibs_op/ldlat=0/u' on PMU 'ibs_op' which will be removed. Invalid event (ibs_op/ldlat=0/u) in per-thread mode, enable system wide with '-a'. [ ... ] Pipe C perf annotate test Pipe annotate [Failed: perf record]
Quoted 2 lines
Assisted-by LLMSigned-off-by Arnaldo Carvalho de Melo <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [PATCH v1 5/8] perf report: Add --progress option
In reply to Arnaldo Carvalho de MeloSashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
From: Arnaldo Carvalho de Melo <[email protected]>
Processing a large data type profiling session, e.g. an AMD IBS one, can take a long time and, when using the stdio output, 'perf report' gives no feedback about which phase it is in nor about how far along it is: the TUI has a progress bar for that, but in the stdio case the
ui_progressupdates, that are already there, are dropped on the floor.Add a --progress option that installs a stdio backend for
ui_progress,ui/stdio/progress.c, printing the phase title, the percentage done and the current/total counts to stderr:Processing events... [ 42.3%] 317M / 746M
Merging related events... [ 61.0%] 309026 / 506686
Sorting events for output... [ 98.2%] 14132 / 14387
[ ... ]
88: Test that perf report handles truncated perf.data gracefully (no crash, no segfault — clean error exit).: Ok ⬢ [acme@toolbx perf-tools-next]$
Quoted 2 lines
Assisted-by LLMSigned-off-by Arnaldo Carvalho de Melo <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
[email protected]
AI reviewRe: [PATCH v1 3/8] perf symbol: Fall back to fetching the vmlinux by build ID
In reply to Arnaldo Carvalho de MeloSashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines
From: Arnaldo Carvalho de Melo <[email protected]>
A profile recorded on a kernel that is no longer installed, be it because the machine was rebooted into a new kernel or because the profile is being processed on another machine, can't have its kernel symbols resolved:
/proc/kallsymsmatches the running kernel, not the one in the profile, and is refused when restricted, e.g. with kernel.perf_event_paranoid > 1, while the build-id cache may carry just a kallsyms copy with zeroed addresses. With no kernel symbols the kernel samples can't be annotated, so they all end up in the '(unknown)' data type and the kernel is missing from the data type profile JSON "dsos" entry.
Quoted 8 lines
As a last resort, when the kernel symbols can't be found locally and the user didn't specify a kallsyms file, fetch the vmlinux keyed by [ ... ]
structtty_struct, struct qspinlock, etc, with the kernel DSO, keyed by its build ID, appearing in the data type profile JSON.Assisted-by LLMSigned-off-by Arnaldo Carvalho de Melo <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
[email protected]
AI reviewRe: [PATCH v1 4/8] perf annotate-data: Show the sample count in the data-type browser
In reply to Arnaldo Carvalho de MeloThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but child nodes and their 'hists' arrays are leaked during browser entries deletion.
- [Medium] This problem wasn't introduced by this patch, but there are missing POSIX libc header inclusions for snprintf, calloc, and free. --
commit 5a4ac13ab552e8559bedbbcd49a2b0cf582b03bc
Author: Arnaldo Carvalho de Melo <[email protected]>
perf annotate-data: Show the sample count in the data-type browser
The data-type browser has a samples view, selected with -n (or with annotate.show_nr_samples), in which browser__write_overhead() prints a local nr_samples variable that is initialized to zero and never updated, so every member is listed as having no samples while the period and percent columns for the same entry are filled in.
Print the histogram entry's own count instead.
This predates the load/store counter split, so fix it ahead of that
patch: the split then only has to adapt a line that is already correct, and this fix can be picked on its own.
tools/perf/ui/browsers/annotate-data.c
Quoted 4 lines
tools/perf/ui/browsers/annotate-data.c +0
-0
[ ... ]
[Severity: High]
This isn't a bug introduced by this patch, but child nodes and their hists arrays are leaked when browser entries are deleted:
tools/perf/ui/browsers/annotate-data.c:annotated_data_browser__delete_entries() {
list_for_each_entry_safe(pos, tmp, &browser->entries, node) {
list_del_init(&pos->node);
zfree(&pos->hists);
free(pos);
}
}The loop exclusively iterates over the top-level nodes in the browser->entries list. Does this leak the memory for all nested child nodes in pos->children and their associated hists arrays?
If a user repeatedly opens and closes the data type annotation TUI, could this unbounded leak eventually exhaust system memory and crash the tool during a long-lived interactive session?
[Severity: Medium]
This is a pre-existing issue, but does this file fail to build under musl libc?
tools/perf/ui/browsers/annotate-data.c
#include <inttypes.h> #include <string.h> #include <linux/zalloc.h> #include <sys/ttydefaults.h>
The code uses functions from stdio.h (snprintf) and stdlib.h (calloc, free) without explicitly including these headers. Could we add these missing POSIX libc header inclusions to ensure musl compilation compatibility?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
[email protected]
AI reviewRe: [PATCH v1 6/8] perf scripts: Add perf-stuck, to tell where a running perf is stuck
In reply to Arnaldo Carvalho de MeloThank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider: - [Medium] Time-of-check to time-of-use race condition when accessing process statistics leads to script…
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Medium] Time-of-check to time-of-use race condition when accessing process statistics leads to script crash.
- [Medium] Incorrect parsing of /proc/[pid]/stat for processes with spaces in their name.
- [Medium] Unsanitized output of process command lines allows terminal escape sequence injection.
- [Medium] Arithmetic evaluation failure due to awk outputting scientific notation for large CPU times. --
commit bb6bb44a87ff31de8100a51a6f3b35196458dff1
Author: Arnaldo Carvalho de Melo <[email protected]>
perf scripts: Add perf-stuck, to tell where a running perf is stuck
This commit introduces the perf-stuck.sh script and an accompanying gdb script to help identify where a running perf process is stuck. It samples /proc/<pid> at fixed intervals to distinguish between spinning, blocking, and recursion, and can automatically attach gdb if no progress is made.
Quoted 6 lines
tools/perf/scripts/perf-stuck.sh new file +0
-0@@ -0,0 +1,165 @@
[ ... ]
tools/perf/scripts/perf-stuck.sh line 99
Quoted 10 lines
tools/perf/scripts/perf-stuck.sh new file +10
-0+hz=$(getconf CLK_TCK) +psz=$(getconf PAGESIZE) +prev_cpu= +prev_stack= +prev_progress= +stuck=0 +gdb_done= +nsample=0 + +echo "watching $pid ($(tr '\0' ' ' < /proc/"$pid"/cmdline)) every ${interval}s"
[Severity: Medium]
Does directly printing the process command line allow for terminal escape sequence injection?
The tr command does not sanitize non-printable characters, which could allow a process launched with malicious ANSI escape sequences in its command line arguments to corrupt the local terminal when monitored.
tools/perf/scripts/perf-stuck.sh line 107
Quoted 8 lines
tools/perf/scripts/perf-stuck.sh new file +8
-0+ +while :; do + if [ ! -d /proc/"$pid" ]; then + echo "$(date +%T) process gone" + break + fi + + stat=($(awk '{print $3, $14 + $15, $24}' /proc/"$pid"/stat))
[Severity: Medium]
If the monitored process has a space in its command name (e.g., "(my script)"), will awk split the name into multiple fields using its default whitespace delimiter, incorrectly shifting positional variables like $14 and $24?
[Severity: Medium]
If the process terminates immediately after the directory check on the previous lines, but before the awk command reads the file, will awk return an empty output?
Because set -u is enabled, accessing the unbound stat[0] variable below would cause the script to abruptly crash instead of cleanly exiting.
Quoted 6 lines
tools/perf/scripts/perf-stuck.sh new file +6
-0+ state=${stat[0]} + cpu=${stat[1]} + # field 24 is the resident set size in pages + rss=$(( stat[2] * psz / 1024 )) + + stack=$(awk '/\[stack\]/{print $1; exit}' /proc/"$pid"/maps)
[ ... ]
tools/perf/scripts/perf-stuck.sh line 127
Quoted 2 lines
tools/perf/scripts/perf-stuck.sh new file +2
-0+ if [ -n "$prev_cpu" ]; then + cpu_delta=$(( cpu - prev_cpu ))
[Severity: Medium]
If the sum of utime and stime ($14 + $15) calculated by awk above exceeds 1,000,000 ticks, will awk format the output in scientific notation (e.g., 1e+06) based on the default OFMT?
If so, the bash arithmetic context here will be unable to parse this format, causing a fatal syntax error and crashing the script when monitoring long-running processes.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
[email protected]
AI reviewRe: [PATCH v1 7/8] perf annotate-data: Resolve type DIEs in the debug file they came from
In reply to Arnaldo Carvalho de MeloSashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines
From: Arnaldo Carvalho de Melo <[email protected]>
A 'perf report -s type' on a 783 MB AMD IBS data type profiling session hangs, burning all of a CPU and producing no output:
⬢ [acme@toolbx perf-tools-next]$ perf report --progress -s type -i perf.data.ibs > /dev/null Processing events... [100.0%] 746M / 746M Merging related events... [ 58.0%] 293828 / 506686 Merging related events... [ 59.0%] 298894 / 506686 Merging related events... [ 60.0%] 303960 / 506686 Merging related events... [ 61.0%] 309026 / 506686 [ ... nothing else, ever ... ]
Quoted 8 lines
It is a spin and not a slow path: with perf-stuck the CPU time used grows by a whole interval on every sample, while the [stack] mapping, [ ... ]
Fixes 06b2ce75386df04b ("perf annotate-data: Maintain variable type info")Fixes 55ee3d005d62279d ("perf annotate-data: Add a cache for global variable types")Assisted-by LLMSigned-off-by Arnaldo Carvalho de Melo <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
[email protected]
AI reviewRe: [PATCH v1 2/8] perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod
In reply to Arnaldo Carvalho de MeloThank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider: - [High] Concurrent calls to `setenv()` cause undefined behavior and Use-After-Free crashes. - [High]…
Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [High] Concurrent calls to setenv() cause undefined behavior and Use-After-Free crashes.
- [High] Concurrent modifications to process-global terminal settings and signal handlers cause permanent corruption.
- [Medium] Data races on static variables (debuginfod_fetch_cancelled, debuginfod_signal, debuginfod_progress_started) corrupt concurrent fetches.
- [Medium] Intercepted signals (e.g., SIGINT) are swallowed on the success path, preventing graceful termination.
- [Low] Missing explicit inclusion of <limits.h> for PATH_MAX, causing musl libc compilation failures.
--
commit 93a135dd9f8083fd8291b38e49e5952c517a78da
Author: Arnaldo Carvalho de Melo <[email protected]>
perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod
This commit adds support for fetching debuginfo keyed by build ID from debuginfod servers. It integrates this into the DSO debuginfo resolution path as a fallback when local debuginfo is missing.
Quoted 3 lines
tools/perf/util/debuginfo.c +0
-0
[ ... ]
Quoted 1 line
tools/perf/util/debuginfo.c +0
-0@@ -139,6 +148,369 @@ struct debuginfo *debuginfo__new(const char *path)
[ ... ]
Quoted 7 lines
tools/perf/util/debuginfo.c +7
-0+static void debuginfod__setup_urls_env (void ) +{ +char *urls =NULL ; +DIR *dir; +struct dirent *dent; + +if (getenv ("DEBUGINFOD_URLS" ) !=NULL )
tools/perf/util/debuginfo.c struct debuginfo *debuginfo__new(const char *path) · line 266
Quoted 10 lines
tools/perf/util/debuginfo.c +10
-0+return ; + + dir =opendir ("/etc/debuginfod" ); +if (dir ==NULL ) +return ; + +while ((dent =readdir (dir)) !=NULL ) { +char *content =NULL ; +char *new_urls; +char path[PATH_MAX ];
[Severity: Low]
Does the use of PATH_MAX without an explicit inclusion of <limits.h> cause build failures on systems using musl libc?
Quoted 2 lines
tools/perf/util/debuginfo.c +2
-0+size_t len =strlen (dent->d_name), i, size; +int n;
[ ... ]
tools/perf/util/debuginfo.c struct debuginfo *debuginfo__new(const char *path) · line 297
Quoted 2 lines
tools/perf/util/debuginfo.c +2
-0+if (urls !=NULL ) { +setenv ("DEBUGINFOD_URLS" , urls,1 );
[Severity: High]
Since dso__debuginfo() explicitly drops the dso__lock before falling back to fetch debuginfo, we can have a call chain like this executed concurrently:
Thread 1 & 2:dso__debuginfo() -> drops lock
debuginfo__new_build_id()
debuginfo__find_build_id()
debuginfod__setup_urls_env()
setenv()
Can concurrent threads resolving DSOs call setenv() simultaneously here? Because setenv() is thread-unsafe, this concurrent access could corrupt the process-global environment array and cause a crash.
Quoted 4 lines
tools/perf/util/debuginfo.c +4
-0+pr_debug ("Set DEBUGINFOD_URLS from /etc/debuginfod: %s\n" , urls); + } +free (urls); +}
[ ... ]
Quoted 8 lines
tools/perf/util/debuginfo.c +8
-0+int debuginfo__find_build_id (const struct build_id *bid,char **path) +{ +char sbuild_id[SBUILD_ID_SIZE ]; +struct termios orig_termios; +struct sigaction sa, orig_sigint, orig_sigterm; +bool term_set = false, sigint_set = false, sigterm_set = false; +debuginfod_client *c; +int fd;
[ ... ]
tools/perf/util/debuginfo.c struct debuginfo *debuginfo__new(const char *path) · line 433
Quoted 2 lines
tools/perf/util/debuginfo.c +2
-0+ debuginfod_fetch_cancelled = false; + debuginfod_signal =0 ;
[Severity: Medium]
These static global variables are shared across all threads but modified here without synchronization. If a second thread starts a fetch, could it unconditionally reset debuginfod_fetch_cancelled to false and cause a concurrent thread attempting to cancel its blocked fetch to hang indefinitely?
Quoted 4 lines
tools/perf/util/debuginfo.c +4
-0+ +/* +* Make stdin deliver keypresses without waiting for a newline, +* the progress callback above polls it for the 's'/'d' keys,
[ ... ]
tools/perf/util/debuginfo.c struct debuginfo *debuginfo__new(const char *path) · line 455
Quoted 12 lines
tools/perf/util/debuginfo.c +12
-0+if (isatty (STDIN_FILENO ) &&isatty (STDERR_FILENO ) && !use_browser) { +set_term_quiet_input (&orig_termios); + term_set = true; + +memset (&sa,0 ,sizeof (sa)); + sa.sa_handler = debuginfod_signal_handler; +sigemptyset (&sa.sa_mask); +if (sigaction (SIGINT , &sa, &orig_sigint) ==0 ) + sigint_set = true; +if (sigaction (SIGTERM , &sa, &orig_sigterm) ==0 ) + sigterm_set = true; + }
[Severity: High]
Following the same concurrent execution path from dso__debuginfo(), can multiple threads concurrently modify the process-wide terminal attributes and signal handlers here?
If Thread B captures Thread A's temporary raw state into orig_termios, could it eventually restore that state permanently and leave the process terminal broken after the fetches complete?
Quoted 27 lines
tools/perf/util/debuginfo.c +27
-0+ + fd =debuginfod_find_debuginfo (c, bid->data, bid->size, path); + +if (term_set) +tcsetattr (STDIN_FILENO ,TCSANOW , &orig_termios); +if (sigint_set) +sigaction (SIGINT , &orig_sigint,NULL ); +if (sigterm_set) +sigaction (SIGTERM , &orig_sigterm,NULL ); + +debuginfod_end (c); +if (debuginfod_progress_started) { +fputc ('\n' , stderr); + debuginfod_progress_started = false; + } +if (fd <0 ) { +build_id__snprintf (bid, sbuild_id,sizeof (sbuild_id)); +if (debuginfod_fetch_cancelled || debuginfod_signal) { +pr_debug ("debuginfod search for build ID %s cancelled by the user\n" , + sbuild_id); +/* +* The terminal is restored, die as the user asked; +* the original dispositions are back in place. +*/ +if (debuginfod_signal) +raise (debuginfod_signal); +return -1 ;
tools/perf/util/debuginfo.c struct debuginfo *debuginfo__new(const char *path) · line 492
Quoted 10 lines
tools/perf/util/debuginfo.c +10
-0+ } +pr_debug ("No debuginfo found for build ID %s in debuginfod\n" , + sbuild_id); +debuginfod__miss_add (bid); +return -1 ; + } + +close (fd); +return 0 ; +}
[Severity: Medium]
On the success path when fd is valid, the code returns 0 and skips evaluating and re-raising debuginfod_signal. Does this mean user interrupt requests like SIGINT are silently swallowed if the user presses Ctrl-C right as the fetch is concluding successfully?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewRe: [PATCH v1 8/8] perf mem record: Request PERF_SAMPLE_CPU by default
In reply to Arnaldo Carvalho de MeloThank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
Pre-existing issues:
- [High] Insufficient allocation for rec_argv array in __cmd_record causes a heap buffer overflow.
- [High] Stack buffer overflow in get_sort_order due to unsafe strcat calls after a bounded scnprintf.
- [Medium] Missing .feature event callback in perf_tool during pipe mode raw event processing.
- [Medium] Memory and reference count leak of cpu_map on error paths in perf_mem_events__record_args. --
commit a9d649273551560b359198abf2d0080276d35896
Author: Arnaldo Carvalho de Melo <[email protected]>
perf mem record: Request PERF_SAMPLE_CPU by default
The data-type profiling per-sample stream requires PERF_SAMPLE_CPU to distinguish reads and writes from different cores. This patch modifies perf mem record to request the CPU id by default by passing the --sample-cpu option.
Quoted 4 lines
tools/perf/builtin-mem.c +0
-0
[ ... ]
tools/perf/builtin-mem.c static int __cmd_record(int argc, const char **argv, struct perf_mem *mem, · line 146
Quoted 13 lines
tools/perf/builtin-mem.c +9
-0@@ -135,6 +135,15 @@ static int __cmd_record(int argc, const char **argv, struct perf_mem *mem, rec_argv[i++] ="-d" ; +/* +* The data-type profiling per-sample stream keys cross-CPU +* contention on sample->cpu (PERF_SAMPLE_CPU); without it the cpu +* field is the (u32)-1 'no CPU info' sentinel and same-instance +* reads and writes from different cores are indistinguishable +* from same-CPU traffic. +*/ + rec_argv[i++] ="--sample-cpu" ; +
[Severity: High]
This isn't a bug introduced by this patch, but doesn't adding this unconditional argument worsen a heap buffer overflow in rec_argv?
In __cmd_record(), the rec_argv array is allocated using this formula:
/* Max number of arguments multiplied by number of PMUs that can support them. */ rec_argc = argc + 9 * (perf_pmu__mem_events_num_mem_pmus(pmu) + 1); if (mem->cpu_list) rec_argc += 2; rec_argv = calloc(rec_argc + 1, sizeof(char *));
This calculation doesn't seem to account for all statically and dynamically appended options. When combining optional flags (like --phys-data, --data-page-size, --all-kernel) and discovering one or more PMUs supporting memory events, could the unconditionally written base arguments (which now includes the newly added --sample-cpu) exceed the rec_argc capacity and corrupt heap memory?
While reviewing this code, I also noticed a few other pre-existing issues in related code:
[Severity: Medium]
This is a pre-existing issue, but does report_raw_events() in tools/perf/builtin-mem.c omit the .feature callback during perf_tool initialization?
tools/perf/builtin-mem.c:report_raw_events() {
...
perf_tool__init(&mem->tool, /*ordered_events=*/true);
mem->tool.sample = process_sample_event;
mem->tool.mmap = perf_event__process_mmap;
mem->tool.mmap2 = perf_event__process_mmap2;
mem->tool.comm = perf_event__process_comm;
mem->tool.lost = perf_event__process_lost;
mem->tool.fork = perf_event__process_fork;
mem->tool.attr = perf_event__process_attr;
mem->tool.build_id = perf_event__process_build_id;
mem->tool.namespaces = perf_event__process_namespaces;
mem->tool.auxtrace_info = perf_event__process_auxtrace_info;
mem->tool.auxtrace = perf_event__process_auxtrace;
mem->tool.auxtrace_error = perf_event__process_auxtrace_error;
...
}Without mem->tool.feature = perf_event__process_feature;, does perf.data environmental metadata (like CPU topologies and build features) fail to synthesize in pipe mode streams, leaving struct perf_env unpopulated?
[Severity: Medium]
This is a pre-existing issue, but do the error paths in perf_mem_events__record_args() leak the cpu_map allocation and reference counts?
tools/perf/util/mem-events.c:perf_mem_events__record_args() {
...
if (!e->supported) {
char buf[128];
pr_err("failed: event '%s' not supported\n",
perf_pmu__mem_events_name(pmu, j, buf, sizeof(buf)));
free(event_name_storage);
return -1;
}
...
ret = perf_cpu_map__merge(&cpu_map, pmu->cpus);
if (ret < 0) {
free(event_name_storage);
return ret;
}
...
}If a subsequent PMU iteration encounters an unsupported event or fails merging, do these error paths return directly without calling perf_cpu_map__put() to drop the accumulated cpu_map reference?
[Severity: High]
This is a pre-existing issue, but can the strcat() calls in get_sort_order() overflow the stack buffer?
tools/perf/builtin-mem.c:get_sort_order() {
char sort[128];
if (mem->sort_key)
scnprintf(sort, sizeof(sort), "--sort=%s", mem->sort_key);
...
if (mem->phys_addr)
strcat(sort, ",phys_daddr");
if (mem->data_page_size)
strcat(sort, ",data_page_size");
/* make sure it has 'type' sort key even -s option is used */
if (mem->data_type && !strstr(sort, "type"))
strcat(sort, ",type");
...
}If the user provides a custom sort key via -s or --sort that fills the 128-byte array up to the boundary checked by scnprintf(), do the subsequent strcat() calls conditionally triggered by flags like --phys-data append bytes blindly past the end of the array, causing stack memory corruption?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8